34 lines
14 KiB
HTML
34 lines
14 KiB
HTML
<html><head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
|
|
<title>24. Main Concepts</title><link rel="stylesheet" type="text/css" href="css/manual-multipage.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="multi_spring-cloud.html" title="Spring Cloud"><link rel="up" href="multi__spring_cloud_stream.html" title="Part IV. Spring Cloud Stream"><link rel="prev" href="multi__introducing_spring_cloud_stream.html" title="23. Introducing Spring Cloud Stream"><link rel="next" href="multi__programming_model.html" title="25. Programming Model"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">24. Main Concepts</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="multi__introducing_spring_cloud_stream.html">Prev</a> </td><th width="60%" align="center">Part IV. Spring Cloud Stream</th><td width="20%" align="right"> <a accesskey="n" href="multi__programming_model.html">Next</a></td></tr></table><hr></div><div class="chapter"><div class="titlepage"><div><div><h2 class="title"><a name="_main_concepts" href="#_main_concepts"></a>24. Main Concepts</h2></div></div></div><p>Spring Cloud Stream provides a number of abstractions and primitives that simplify the writing of message-driven microservice applications.
|
|
This section gives an overview of the following:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">Spring Cloud Stream’s application model</li><li class="listitem">The Binder abstraction</li><li class="listitem">Persistent publish-subscribe support</li><li class="listitem">Consumer group support</li><li class="listitem">Partitioning support</li><li class="listitem">A pluggable Binder API</li></ul></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_application_model" href="#_application_model"></a>24.1 Application Model</h2></div></div></div><p>A Spring Cloud Stream application consists of a middleware-neutral core.
|
|
The application communicates with the outside world through input and output <span class="emphasis"><em>channels</em></span> injected into it by Spring Cloud Stream.
|
|
Channels are connected to external brokers through middleware-specific Binder implementations.</p><div class="figure"><a name="d0e5821" href="#d0e5821"></a><p class="title"><b>Figure 24.1. Spring Cloud Stream Application</b></p><div class="figure-contents"><div class="mediaobject"><img src="images/SCSt-with-binder.png" alt="SCSt with binder"></div></div></div><br class="figure-break"><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_fat_jar" href="#_fat_jar"></a>24.1.1 Fat JAR</h3></div></div></div><p>Spring Cloud Stream applications can be run in standalone mode from your IDE for testing.
|
|
To run a Spring Cloud Stream application in production, you can create an executable (or "fat") JAR by using the standard Spring Boot tooling provided for Maven or Gradle.</p></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_the_binder_abstraction" href="#_the_binder_abstraction"></a>24.2 The Binder Abstraction</h2></div></div></div><p>Spring Cloud Stream provides Binder implementations for <a class="link" href="https://github.com/spring-cloud/spring-cloud-stream/tree/master/spring-cloud-stream-binders/spring-cloud-stream-binder-kafka" target="_top">Kafka</a> and <a class="link" href="https://github.com/spring-cloud/spring-cloud-stream/tree/master/spring-cloud-stream-binders/spring-cloud-stream-binder-rabbit" target="_top">Rabbit MQ</a>.
|
|
Spring Cloud Stream also includes a <a class="link" href="https://github.com/spring-cloud/spring-cloud-stream/blob/master/spring-cloud-stream-test-support/src/main/java/org/springframework/cloud/stream/test/binder/TestSupportBinder.java" target="_top">TestSupportBinder</a>, which leaves a channel unmodified so that tests can interact with channels directly and reliably assert on what is received.
|
|
You can use the extensible API to write your own Binder.</p><p>Spring Cloud Stream uses Spring Boot for configuration, and the Binder abstraction makes it possible for a Spring Cloud Stream application to be flexible in how it connects to middleware.
|
|
For example, deployers can dynamically choose, at runtime, the destinations (e.g., the Kafka topics or RabbitMQ exchanges) to which channels connect.
|
|
Such configuration can be provided through external configuration properties and in any form supported by Spring Boot (including application arguments, environment variables, and <code class="literal">application.yml</code> or <code class="literal">application.properties</code> files).
|
|
In the sink example from the <a class="xref" href="multi__introducing_spring_cloud_stream.html" title="23. Introducing Spring Cloud Stream">Chapter 23, <i>Introducing Spring Cloud Stream</i></a> section, setting the application property <code class="literal">spring.cloud.stream.bindings.input.destination</code> to <code class="literal">raw-sensor-data</code> will cause it to read from the <code class="literal">raw-sensor-data</code> Kafka topic, or from a queue bound to the <code class="literal">raw-sensor-data</code> RabbitMQ exchange.</p><p>Spring Cloud Stream automatically detects and uses a binder found on the classpath.
|
|
You can easily use different types of middleware with the same code: just include a different binder at build time.
|
|
For more complex use cases, you can also package multiple binders with your application and have it choose the binder, and even whether to use different binders for different channels, at runtime.</p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_persistent_publish_subscribe_support" href="#_persistent_publish_subscribe_support"></a>24.3 Persistent Publish-Subscribe Support</h2></div></div></div><p>Communication between applications follows a publish-subscribe model, where data is broadcast through shared topics.
|
|
This can be seen in the following figure, which shows a typical deployment for a set of interacting Spring Cloud Stream applications.</p><div class="figure"><a name="d0e5878" href="#d0e5878"></a><p class="title"><b>Figure 24.2. Spring Cloud Stream Publish-Subscribe</b></p><div class="figure-contents"><div class="mediaobject"><img src="images/SCSt-sensors.png" alt="SCSt sensors"></div></div></div><br class="figure-break"><p>Data reported by sensors to an HTTP endpoint is sent to a common destination named <code class="literal">raw-sensor-data</code>.
|
|
From the destination, it is independently processed by a microservice application that computes time-windowed averages and by another microservice application that ingests the raw data into HDFS.
|
|
In order to process the data, both applications declare the topic as their input at runtime.</p><p>The publish-subscribe communication model reduces the complexity of both the producer and the consumer, and allows new applications to be added to the topology without disruption of the existing flow.
|
|
For example, downstream from the average-calculating application, you can add an application that calculates the highest temperature values for display and monitoring.
|
|
You can then add another application that interprets the same flow of averages for fault detection.
|
|
Doing all communication through shared topics rather than point-to-point queues reduces coupling between microservices.</p><p>While the concept of publish-subscribe messaging is not new, Spring Cloud Stream takes the extra step of making it an opinionated choice for its application model.
|
|
By using native middleware support, Spring Cloud Stream also simplifies use of the publish-subscribe model across different platforms.</p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="consumer-groups" href="#consumer-groups"></a>24.4 Consumer Groups</h2></div></div></div><p>While the publish-subscribe model makes it easy to connect applications through shared topics, the ability to scale up by creating multiple instances of a given application is equally important.
|
|
When doing this, different instances of an application are placed in a competing consumer relationship, where only one of the instances is expected to handle a given message.</p><p>Spring Cloud Stream models this behavior through the concept of a <span class="emphasis"><em>consumer group</em></span>.
|
|
(Spring Cloud Stream consumer groups are similar to and inspired by Kafka consumer groups.)
|
|
Each consumer binding can use the <code class="literal">spring.cloud.stream.bindings.<channelName>.group</code> property to specify a group name.
|
|
For the consumers shown in the following figure, this property would be set as <code class="literal">spring.cloud.stream.bindings.<channelName>.group=hdfsWrite</code> or <code class="literal">spring.cloud.stream.bindings.<channelName>.group=average</code>.</p><div class="figure"><a name="d0e5915" href="#d0e5915"></a><p class="title"><b>Figure 24.3. Spring Cloud Stream Consumer Groups</b></p><div class="figure-contents"><div class="mediaobject"><img src="images/SCSt-groups.png" alt="SCSt groups"></div></div></div><br class="figure-break"><p>All groups which subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination.
|
|
By default, when a group is not specified, Spring Cloud Stream assigns the application to an anonymous and independent single-member consumer group that is in a publish-subscribe relationship with all other consumer groups.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="durability" href="#durability"></a>24.4.1 Durability</h3></div></div></div><p>Consistent with the opinionated application model of Spring Cloud Stream, consumer group subscriptions are <span class="emphasis"><em>durable</em></span>.
|
|
That is, a binder implementation ensures that group subscriptions are persistent, and once at least one subscription for a group has been created, the group will receive messages, even if they are sent while all applications in the group are stopped.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>Anonymous subscriptions are non-durable by nature.
|
|
For some binder implementations (e.g., RabbitMQ), it is possible to have non-durable group subscriptions.</p></td></tr></table></div><p>In general, it is preferable to always specify a consumer group when binding an application to a given destination.
|
|
When scaling up a Spring Cloud Stream application, you must specify a consumer group for each of its input bindings.
|
|
This prevents the application’s instances from receiving duplicate messages (unless that behavior is desired, which is unusual).</p></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="partitioning" href="#partitioning"></a>24.5 Partitioning Support</h2></div></div></div><p>Spring Cloud Stream provides support for <span class="emphasis"><em>partitioning</em></span> data between multiple instances of a given application.
|
|
In a partitioned scenario, the physical communication medium (e.g., the broker topic) is viewed as being structured into multiple partitions.
|
|
One or more producer application instances send data to multiple consumer application instances and ensure that data identified by common characteristics are processed by the same consumer instance.</p><p>Spring Cloud Stream provides a common abstraction for implementing partitioned processing use cases in a uniform fashion.
|
|
Partitioning can thus be used whether the broker itself is naturally partitioned (e.g., Kafka) or not (e.g., RabbitMQ).</p><div class="figure"><a name="d0e5949" href="#d0e5949"></a><p class="title"><b>Figure 24.4. Spring Cloud Stream Partitioning</b></p><div class="figure-contents"><div class="mediaobject"><img src="images/SCSt-partitioning.png" alt="SCSt partitioning"></div></div></div><br class="figure-break"><p>Partitioning is a critical concept in stateful processing, where it is critiical, for either performance or consistency reasons, to ensure that all related data is processed together.
|
|
For example, in the time-windowed average calculation example, it is important that all measurements from any given sensor are processed by the same application instance.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>To set up a partitioned processing scenario, you must configure both the data-producing and the data-consuming ends.</p></td></tr></table></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="multi__introducing_spring_cloud_stream.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="multi__spring_cloud_stream.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="multi__programming_model.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">23. Introducing Spring Cloud Stream </td><td width="20%" align="center"><a accesskey="h" href="multi_spring-cloud.html">Home</a></td><td width="40%" align="right" valign="top"> 25. Programming Model</td></tr></table></div></body></html> |