37 lines
15 KiB
HTML
37 lines
15 KiB
HTML
<html><head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
|
|
<title>32. Inter-Application Communication</title><link rel="stylesheet" type="text/css" href="css/manual-multipage.css"><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"><link rel="home" href="multi_spring-cloud.html" title="Spring Cloud"><link rel="up" href="multi__spring_cloud_stream.html" title="Part V. Spring Cloud Stream"><link rel="prev" href="multi_schema-evolution.html" title="31. Schema Evolution Support"><link rel="next" href="multi__testing.html" title="33. Testing"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">32. Inter-Application Communication</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="multi_schema-evolution.html">Prev</a> </td><th width="60%" align="center">Part V. Spring Cloud Stream</th><td width="20%" align="right"> <a accesskey="n" href="multi__testing.html">Next</a></td></tr></table><hr></div><div class="chapter"><div class="titlepage"><div><div><h2 class="title"><a name="_inter_application_communication" href="#_inter_application_communication"></a>32. Inter-Application Communication</h2></div></div></div><p>Spring Cloud Stream enables communication between applications. Inter-application communication is a complex issue spanning several concerns, as described in the following topics:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><span class="quote">“<span class="quote"><a class="xref" href="multi__inter_application_communication.html#spring-cloud-stream-overview-connecting-multiple-application-instances" title="32.1 Connecting Multiple Application Instances">Section 32.1, “Connecting Multiple Application Instances”</a></span>”</span></li><li class="listitem"><span class="quote">“<span class="quote"><a class="xref" href="multi__inter_application_communication.html#spring-cloud-stream-overview-instance-index-instance-count" title="32.2 Instance Index and Instance Count">Section 32.2, “Instance Index and Instance Count”</a></span>”</span></li><li class="listitem"><span class="quote">“<span class="quote"><a class="xref" href="multi__inter_application_communication.html#spring-cloud-stream-overview-partitioning" title="32.3 Partitioning">Section 32.3, “Partitioning”</a></span>”</span></li></ul></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="spring-cloud-stream-overview-connecting-multiple-application-instances" href="#spring-cloud-stream-overview-connecting-multiple-application-instances"></a>32.1 Connecting Multiple Application Instances</h2></div></div></div><p>While Spring Cloud Stream makes it easy for individual Spring Boot applications to connect to messaging systems, the typical scenario for Spring Cloud Stream is the creation of multi-application pipelines, where microservice applications send data to each other.
|
|
You can achieve this scenario by correlating the input and output destinations of <span class="quote">“<span class="quote">adjacent</span>”</span> applications.</p><p>Suppose a design calls for the Time Source application to send data to the Log Sink application. You could use a common destination named <code class="literal">ticktock</code> for bindings within both applications.</p><p>Time Source (that has the channel name <code class="literal">output</code>) would set the following property:</p><pre class="screen">spring.cloud.stream.bindings.output.destination=ticktock</pre><p>Log Sink (that has the channel name <code class="literal">input</code>) would set the following property:</p><pre class="screen">spring.cloud.stream.bindings.input.destination=ticktock</pre></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="spring-cloud-stream-overview-instance-index-instance-count" href="#spring-cloud-stream-overview-instance-index-instance-count"></a>32.2 Instance Index and Instance Count</h2></div></div></div><p>When scaling up Spring Cloud Stream applications, each instance can receive information about how many other instances of the same application exist and what its own instance index is.
|
|
Spring Cloud Stream does this through the <code class="literal">spring.cloud.stream.instanceCount</code> and <code class="literal">spring.cloud.stream.instanceIndex</code> properties.
|
|
For example, if there are three instances of a HDFS sink application, all three instances have <code class="literal">spring.cloud.stream.instanceCount</code> set to <code class="literal">3</code>, and the individual applications have <code class="literal">spring.cloud.stream.instanceIndex</code> set to <code class="literal">0</code>, <code class="literal">1</code>, and <code class="literal">2</code>, respectively.</p><p>When Spring Cloud Stream applications are deployed through Spring Cloud Data Flow, these properties are configured automatically; when Spring Cloud Stream applications are launched independently, these properties must be set correctly.
|
|
By default, <code class="literal">spring.cloud.stream.instanceCount</code> is <code class="literal">1</code>, and <code class="literal">spring.cloud.stream.instanceIndex</code> is <code class="literal">0</code>.</p><p>In a scaled-up scenario, correct configuration of these two properties is important for addressing partitioning behavior (see below) in general, and the two properties are always required by certain binders (for example, the Kafka binder) in order to ensure that data are split correctly across multiple consumer instances.</p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="spring-cloud-stream-overview-partitioning" href="#spring-cloud-stream-overview-partitioning"></a>32.3 Partitioning</h2></div></div></div><p>Partitioning in Spring Cloud Stream consists of two tasks:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><span class="quote">“<span class="quote"><a class="xref" href="multi__inter_application_communication.html#spring-cloud-stream-overview-configuring-output-bindings-partitioning" title="32.3.1 Configuring Output Bindings for Partitioning">Section 32.3.1, “Configuring Output Bindings for Partitioning”</a></span>”</span></li><li class="listitem"><span class="quote">“<span class="quote"><a class="xref" href="multi__inter_application_communication.html#spring-cloud-stream-overview-configuring-input-bindings-partitioning" title="32.3.2 Configuring Input Bindings for Partitioning">Section 32.3.2, “Configuring Input Bindings for Partitioning”</a></span>”</span></li></ul></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="spring-cloud-stream-overview-configuring-output-bindings-partitioning" href="#spring-cloud-stream-overview-configuring-output-bindings-partitioning"></a>32.3.1 Configuring Output Bindings for Partitioning</h3></div></div></div><p>You can configure an output binding to send partitioned data by setting one and only one of its <code class="literal">partitionKeyExpression</code> or <code class="literal">partitionKeyExtractorName</code> properties, as well as its <code class="literal">partitionCount</code> property.</p><p>For example, the following is a valid and typical configuration:</p><pre class="screen">spring.cloud.stream.bindings.output.producer.partitionKeyExpression=payload.id
|
|
spring.cloud.stream.bindings.output.producer.partitionCount=5</pre><p>Based on that example configuration, data is sent to the target partition by using the following logic.</p><p>A partition key’s value is calculated for each message sent to a partitioned output channel based on the <code class="literal">partitionKeyExpression</code>.
|
|
The <code class="literal">partitionKeyExpression</code> is a SpEL expression that is evaluated against the outbound message for extracting the partitioning key.</p><p>If a SpEL expression is not sufficient for your needs, you can instead calculate the partition key value by providing an implementation of <code class="literal">org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy</code> and configuring it as a bean (by using the <code class="literal">@Bean</code> annotation).
|
|
If you have more then one bean of type <code class="literal">org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy</code> available in the Application Context, you can further filter it by specifying its name with the <code class="literal">partitionKeyExtractorName</code> property, as shown in the following example:</p><pre class="screen">--spring.cloud.stream.bindings.output.producer.partitionKeyExtractorName=customPartitionKeyExtractor
|
|
--spring.cloud.stream.bindings.output.producer.partitionCount=5
|
|
. . .
|
|
@Bean
|
|
public CustomPartitionKeyExtractorClass customPartitionKeyExtractor() {
|
|
return new CustomPartitionKeyExtractorClass();
|
|
}</pre><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>In previous versions of Spring Cloud Stream, you could specify the implementation of <code class="literal">org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy</code> by setting the <code class="literal">spring.cloud.stream.bindings.output.producer.partitionKeyExtractorClass</code> property.
|
|
Since version 2.0, this property is deprecated, and support for it will be removed in a future version.</p></td></tr></table></div><p>Once the message key is calculated, the partition selection process determines the target partition as a value between <code class="literal">0</code> and <code class="literal">partitionCount - 1</code>.
|
|
The default calculation, applicable in most scenarios, is based on the following formula: <code class="literal">key.hashCode() % partitionCount</code>.
|
|
This can be customized on the binding, either by setting a SpEL expression to be evaluated against the 'key' (through the <code class="literal">partitionSelectorExpression</code> property) or by configuring an implementation of <code class="literal">org.springframework.cloud.stream.binder.PartitionSelectorStrategy</code> as a bean (by using the @Bean annotation).
|
|
Similar to the <code class="literal">PartitionKeyExtractorStrategy</code>, you can further filter it by using the <code class="literal">spring.cloud.stream.bindings.output.producer.partitionSelectorName</code> property when more than one bean of this type is available in the Application Context, as shown in the following example:</p><pre class="screen">--spring.cloud.stream.bindings.output.producer.partitionSelectorName=customPartitionSelector
|
|
. . .
|
|
@Bean
|
|
public CustomPartitionSelectorClass customPartitionSelector() {
|
|
return new CustomPartitionSelectorClass();
|
|
}</pre><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>In previous versions of Spring Cloud Stream you could specify the implementation of <code class="literal">org.springframework.cloud.stream.binder.PartitionSelectorStrategy</code> by setting the <code class="literal">spring.cloud.stream.bindings.output.producer.partitionSelectorClass</code> property.
|
|
Since version 2.0, this property is deprecated and support for it will be removed in a future version.</p></td></tr></table></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="spring-cloud-stream-overview-configuring-input-bindings-partitioning" href="#spring-cloud-stream-overview-configuring-input-bindings-partitioning"></a>32.3.2 Configuring Input Bindings for Partitioning</h3></div></div></div><p>An input binding (with the channel name <code class="literal">input</code>) is configured to receive partitioned data by setting its <code class="literal">partitioned</code> property, as well as the <code class="literal">instanceIndex</code> and <code class="literal">instanceCount</code> properties on the application itself, as shown in the following example:</p><pre class="screen">spring.cloud.stream.bindings.input.consumer.partitioned=true
|
|
spring.cloud.stream.instanceIndex=3
|
|
spring.cloud.stream.instanceCount=5</pre><p>The <code class="literal">instanceCount</code> value represents the total number of application instances between which the data should be partitioned.
|
|
The <code class="literal">instanceIndex</code> must be a unique value across the multiple instances, with a value between <code class="literal">0</code> and <code class="literal">instanceCount - 1</code>.
|
|
The instance index helps each application instance to identify the unique partition(s) from which it receives data.
|
|
It is required by binders using technology that does not support partitioning natively.
|
|
For example, with RabbitMQ, there is a queue for each partition, with the queue name containing the instance index.
|
|
With Kafka, if <code class="literal">autoRebalanceEnabled</code> is <code class="literal">true</code> (default), Kafka takes care of distributing partitions across instances, and these properties are not required.
|
|
If <code class="literal">autoRebalanceEnabled</code> is set to false, the <code class="literal">instanceCount</code> and <code class="literal">instanceIndex</code> are used by the binder to determine which partition(s) the instance subscribes to (you must have at least as many partitions as there are instances).
|
|
The binder allocates the partitions instead of Kafka.
|
|
This might be useful if you want messages for a particular partition to always go to the same instance.
|
|
When a binder configuration requires them, it is important to set both values correctly in order to ensure that all of the data is consumed and that the application instances receive mutually exclusive datasets.</p><p>While a scenario in which using multiple instances for partitioned data processing may be complex to set up in a standalone case, Spring Cloud Dataflow can simplify the process significantly by populating both the input and output values correctly and by letting you rely on the runtime infrastructure to provide information about the instance index and instance count.</p></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="multi_schema-evolution.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="multi__spring_cloud_stream.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="multi__testing.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">31. Schema Evolution Support </td><td width="20%" align="center"><a accesskey="h" href="multi_spring-cloud.html">Home</a></td><td width="40%" align="right" valign="top"> 33. Testing</td></tr></table></div></body></html> |