26 lines
11 KiB
HTML
26 lines
11 KiB
HTML
<html><head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
|
|
<title>31. Inter-Application Communication</title><link rel="stylesheet" type="text/css" href="css/manual-multipage.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="multi_spring-cloud.html" title="Spring Cloud"><link rel="up" href="multi__spring_cloud_stream.html" title="Part IV. Spring Cloud Stream"><link rel="prev" href="multi_schema-evolution.html" title="30. Schema evolution support"><link rel="next" href="multi__testing.html" title="32. Testing"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">31. Inter-Application Communication</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="multi_schema-evolution.html">Prev</a> </td><th width="60%" align="center">Part IV. Spring Cloud Stream</th><td width="20%" align="right"> <a accesskey="n" href="multi__testing.html">Next</a></td></tr></table><hr></div><div class="chapter"><div class="titlepage"><div><div><h2 class="title"><a name="_inter_application_communication" href="#_inter_application_communication"></a>31. Inter-Application Communication</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_connecting_multiple_application_instances" href="#_connecting_multiple_application_instances"></a>31.1 Connecting Multiple Application Instances</h2></div></div></div><p>While Spring Cloud Stream makes it easy for individual Spring Boot applications to connect to messaging systems, the typical scenario for Spring Cloud Stream is the creation of multi-application pipelines, where microservice applications send data to each other.
|
|
You can achieve this scenario by correlating the input and output destinations of adjacent applications.</p><p>Supposing that a design calls for the Time Source application to send data to the Log Sink application, you can use a common destination named <code class="literal">ticktock</code> for bindings within both applications.</p><p>Time Source (that has the channel name <code class="literal">output</code>) will set the following property:</p><pre class="screen">spring.cloud.stream.bindings.output.destination=ticktock</pre><p>Log Sink (that has the channel name <code class="literal">input</code>) will set the following property:</p><pre class="screen">spring.cloud.stream.bindings.input.destination=ticktock</pre></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_instance_index_and_instance_count" href="#_instance_index_and_instance_count"></a>31.2 Instance Index and Instance Count</h2></div></div></div><p>When scaling up Spring Cloud Stream applications, each instance can receive information about how many other instances of the same application exist and what its own instance index is.
|
|
Spring Cloud Stream does this through the <code class="literal">spring.cloud.stream.instanceCount</code> and <code class="literal">spring.cloud.stream.instanceIndex</code> properties.
|
|
For example, if there are three instances of a HDFS sink application, all three instances will have <code class="literal">spring.cloud.stream.instanceCount</code> set to <code class="literal">3</code>, and the individual applications will have <code class="literal">spring.cloud.stream.instanceIndex</code> set to <code class="literal">0</code>, <code class="literal">1</code>, and <code class="literal">2</code>, respectively.</p><p>When Spring Cloud Stream applications are deployed via Spring Cloud Data Flow, these properties are configured automatically; when Spring Cloud Stream applications are launched independently, these properties must be set correctly.
|
|
By default, <code class="literal">spring.cloud.stream.instanceCount</code> is <code class="literal">1</code>, and <code class="literal">spring.cloud.stream.instanceIndex</code> is <code class="literal">0</code>.</p><p>In a scaled-up scenario, correct configuration of these two properties is important for addressing partitioning behavior (see below) in general, and the two properties are always required by certain binders (e.g., the Kafka binder) in order to ensure that data are split correctly across multiple consumer instances.</p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_partitioning" href="#_partitioning"></a>31.3 Partitioning</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_configuring_output_bindings_for_partitioning" href="#_configuring_output_bindings_for_partitioning"></a>31.3.1 Configuring Output Bindings for Partitioning</h3></div></div></div><p>An output binding is configured to send partitioned data by setting one and only one of its <code class="literal">partitionKeyExpression</code> or <code class="literal">partitionKeyExtractorClass</code> properties, as well as its <code class="literal">partitionCount</code> property.
|
|
For example, the following is a valid and typical configuration:</p><pre class="screen">spring.cloud.stream.bindings.output.producer.partitionKeyExpression=payload.id
|
|
spring.cloud.stream.bindings.output.producer.partitionCount=5</pre><p>Based on the above example configuration, data will be sent to the target partition using the following logic.</p><p>A partition key’s value is calculated for each message sent to a partitioned output channel based on the <code class="literal">partitionKeyExpression</code>.
|
|
The <code class="literal">partitionKeyExpression</code> is a SpEL expression which is evaluated against the outbound message for extracting the partitioning key.</p><p>If a SpEL expression is not sufficient for your needs, you can instead calculate the partition key value by setting the property <code class="literal">partitionKeyExtractorClass</code> to a class which implements the <code class="literal">org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy</code> interface.
|
|
While the SpEL expression should usually suffice, more complex cases may use the custom implementation strategy.
|
|
In that case, the property 'partitionKeyExtractorClass' can be set as follows:</p><pre class="screen">spring.cloud.stream.bindings.output.producer.partitionKeyExtractorClass=com.example.MyKeyExtractor
|
|
spring.cloud.stream.bindings.output.producer.partitionCount=5</pre><p>Once the message key is calculated, the partition selection process will determine the target partition as a value between <code class="literal">0</code> and <code class="literal">partitionCount - 1</code>.
|
|
The default calculation, applicable in most scenarios, is based on the formula <code class="literal">key.hashCode() % partitionCount</code>.
|
|
This can be customized on the binding, either by setting a SpEL expression to be evaluated against the 'key' (via the <code class="literal">partitionSelectorExpression</code> property) or by setting a <code class="literal">org.springframework.cloud.stream.binder.PartitionSelectorStrategy</code> implementation (via the <code class="literal">partitionSelectorClass</code> property).</p><p>The binding level properties for 'partitionSelectorExpression' and 'partitionSelectorClass' can be specified similar to the way 'partitionKeyExpression' and 'partitionKeyExtractorClass' properties are specified in the above examples.
|
|
Additional properties can be configured for more advanced scenarios, as described in the following section.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_spring_managed_custom_literal_partitionkeyextractorclass_literal_implementations" href="#_spring_managed_custom_literal_partitionkeyextractorclass_literal_implementations"></a>Spring-managed custom <code class="literal">PartitionKeyExtractorClass</code> implementations</h4></div></div></div><p>In the example above, a custom strategy such as <code class="literal">MyKeyExtractor</code> is instantiated by the Spring Cloud Stream directly.
|
|
In some cases, it is necessary for such a custom strategy implementation to be created as a Spring bean, for being able to be managed by Spring, so that it can perform dependency injection, property binding, etc.
|
|
This can be done by configuring it as a @Bean in the application context and using the fully qualified class name as the bean’s name, as in the following example.</p><pre class="screen">@Bean(name="com.example.MyKeyExtractor")
|
|
public MyKeyExtractor extractor() {
|
|
return new MyKeyExtractor();
|
|
}</pre><p>As a Spring bean, the custom strategy benefits from the full lifecycle of a Spring bean.
|
|
For example, if the implementation need access to the application context directly, it can make implement 'ApplicationContextAware'.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_configuring_input_bindings_for_partitioning" href="#_configuring_input_bindings_for_partitioning"></a>Configuring Input Bindings for Partitioning</h4></div></div></div><p>An input binding (with the channel name <code class="literal">input</code>) is configured to receive partitioned data by setting its <code class="literal">partitioned</code> property, as well as the <code class="literal">instanceIndex</code> and <code class="literal">instanceCount</code> properties on the application itself, as in the following example:</p><pre class="screen">spring.cloud.stream.bindings.input.consumer.partitioned=true
|
|
spring.cloud.stream.instanceIndex=3
|
|
spring.cloud.stream.instanceCount=5</pre><p>The <code class="literal">instanceCount</code> value represents the total number of application instances between which the data need to be partitioned, and the <code class="literal">instanceIndex</code> must be a unique value across the multiple instances, between <code class="literal">0</code> and <code class="literal">instanceCount - 1</code>.
|
|
The instance index helps each application instance to identify the unique partition (or, in the case of Kafka, the partition set) from which it receives data.
|
|
It is important to set both values correctly in order to ensure that all of the data is consumed and that the application instances receive mutually exclusive datasets.</p><p>While a scenario which using multiple instances for partitioned data processing may be complex to set up in a standalone case, Spring Cloud Dataflow can simplify the process significantly by populating both the input and output values correctly as well as relying on the runtime infrastructure to provide information about the instance index and instance count.</p></div></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="multi_schema-evolution.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="multi__spring_cloud_stream.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="multi__testing.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">30. Schema evolution support </td><td width="20%" align="center"><a accesskey="h" href="multi_spring-cloud.html">Home</a></td><td width="40%" align="right" valign="top"> 32. Testing</td></tr></table></div></body></html> |