diff --git a/multi/multi__binder_implementations.html b/multi/multi__binder_implementations.html index e9d5f4145..034b0353e 100644 --- a/multi/multi__binder_implementations.html +++ b/multi/multi__binder_implementations.html @@ -1,3 +1,3 @@ - 16. Binder Implementations

16. Binder Implementations

The following is the list of available binder implementations

\ No newline at end of file + 16. Binder Implementations

16. Binder Implementations

The following is the list of available binder implementations

\ No newline at end of file diff --git a/multi/multi__configuration_options.html b/multi/multi__configuration_options.html index 9215fadcd..578c90346 100644 --- a/multi/multi__configuration_options.html +++ b/multi/multi__configuration_options.html @@ -56,7 +56,7 @@ When this configuration is being used, the outbound message marshalling is not b When native encoding is used, it is the responsibility of the consumer to use an appropriate decoder (for example, the Kafka consumer value de-serializer) to deserialize the inbound message. Also, when native encoding and decoding is used, the headerMode=embeddedHeaders property is ignored and headers are not embedded in the message. See the consumer property useNativeDecoding.

Default: false.

errorChannelEnabled

When set to true, if the binder supports asynchroous send results, send failures are sent to an error channel for the destination. -See ??? for more information.

Default: false.

8.3 Using Dynamically Bound Destinations

Besides the channels defined by using @EnableBinding, Spring Cloud Stream lets applications send messages to dynamically bound destinations. +See Section 6.4, “Error Handling” for more information.

Default: false.

8.3 Using Dynamically Bound Destinations

Besides the channels defined by using @EnableBinding, Spring Cloud Stream lets applications send messages to dynamically bound destinations. This is useful, for example, when the target destination needs to be determined at runtime. Applications can do so by using the BinderAwareChannelResolver bean, registered automatically by the @EnableBinding annotation.

The 'spring.cloud.stream.dynamicDestinations' property can be used for restricting the dynamic destination names to a known set (whitelisting). If this property is not set, any destination can be bound dynamically.

The BinderAwareChannelResolver can be used directly, as shown in the following example of a REST controller using a path variable to decide the target channel:

@EnableBinding
diff --git a/multi/multi__main_concepts.html b/multi/multi__main_concepts.html
index 90be8765d..813d5d7e4 100644
--- a/multi/multi__main_concepts.html
+++ b/multi/multi__main_concepts.html
@@ -3,7 +3,7 @@
    5. Main Concepts

5. Main Concepts

Spring Cloud Stream provides a number of abstractions and primitives that simplify the writing of message-driven microservice applications. This section gives an overview of the following:

5.1 Application Model

A Spring Cloud Stream application consists of a middleware-neutral core. The application communicates with the outside world through input and output channels injected into it by Spring Cloud Stream. -Channels are connected to external brokers through middleware-specific Binder implementations.

Figure 5.1. Spring Cloud Stream Application

SCSt with binder

5.1.1 Fat JAR

Spring Cloud Stream applications can be run in stand-alone mode from your IDE for testing. +Channels are connected to external brokers through middleware-specific Binder implementations.

Figure 5.1. Spring Cloud Stream Application

SCSt with binder

5.1.1 Fat JAR

Spring Cloud Stream applications can be run in stand-alone mode from your IDE for testing. To run a Spring Cloud Stream application in production, you can create an executable (or fat) JAR by using the standard Spring Boot tooling provided for Maven or Gradle. See the Spring Boot Reference Guide for more details.

5.2 The Binder Abstraction

Spring Cloud Stream provides Binder implementations for Kafka and Rabbit MQ. Spring Cloud Stream also includes a TestSupportBinder, which leaves a channel unmodified so that tests can interact with channels directly and reliably assert on what is received. You can also use the extensible API to write your own Binder.

Spring Cloud Stream uses Spring Boot for configuration, and the Binder abstraction makes it possible for a Spring Cloud Stream application to be flexible in how it connects to middleware. @@ -13,7 +13,7 @@ In the sink example from the

5.3 Persistent Publish-Subscribe Support

Communication between applications follows a publish-subscribe model, where data is broadcast through shared topics. -This can be seen in the following figure, which shows a typical deployment for a set of interacting Spring Cloud Stream applications.

Figure 5.2. Spring Cloud Stream Publish-Subscribe

SCSt sensors

Data reported by sensors to an HTTP endpoint is sent to a common destination named raw-sensor-data. +This can be seen in the following figure, which shows a typical deployment for a set of interacting Spring Cloud Stream applications.

Figure 5.2. Spring Cloud Stream Publish-Subscribe

SCSt sensors

Data reported by sensors to an HTTP endpoint is sent to a common destination named raw-sensor-data. From the destination, it is independently processed by a microservice application that computes time-windowed averages and by another microservice application that ingests the raw data into HDFS (Hadoop Distributed File System). In order to process the data, both applications declare the topic as their input at runtime.

The publish-subscribe communication model reduces the complexity of both the producer and the consumer and lets new applications be added to the topology without disruption of the existing flow. For example, downstream from the average-calculating application, you can add an application that calculates the highest temperature values for display and monitoring. @@ -23,7 +23,7 @@ By using native middleware support, Spring Cloud Stream also simplifies use of t When doing so, different instances of an application are placed in a competing consumer relationship, where only one of the instances is expected to handle a given message.

Spring Cloud Stream models this behavior through the concept of a consumer group. (Spring Cloud Stream consumer groups are similar to and inspired by Kafka consumer groups.) Each consumer binding can use the spring.cloud.stream.bindings.<channelName>.group property to specify a group name. -For the consumers shown in the following figure, this property would be set as spring.cloud.stream.bindings.<channelName>.group=hdfsWrite or spring.cloud.stream.bindings.<channelName>.group=average.

Figure 5.3. Spring Cloud Stream Consumer Groups

SCSt groups

All groups that subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination. +For the consumers shown in the following figure, this property would be set as spring.cloud.stream.bindings.<channelName>.group=hdfsWrite or spring.cloud.stream.bindings.<channelName>.group=average.

Figure 5.3. Spring Cloud Stream Consumer Groups

SCSt groups

All groups that subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination. By default, when a group is not specified, Spring Cloud Stream assigns the application to an anonymous and independent single-member consumer group that is in a publish-subscribe relationship with all other consumer groups.

5.5 Consumer Types

Two types of consumer are supported:

  • Message-driven (sometimes referred to as Asynchronous)
  • Polled (sometimes referred to as Synchronous)

Prior to version 2.0, only asynchronous consumers were supported. A message is delivered as soon as it is available and a thread is available to process it.

When you wish to control the rate at which messages are processed, you might want to use a synchronous consumer.

5.5.1 Durability

Consistent with the opinionated application model of Spring Cloud Stream, consumer group subscriptions are durable. That is, a binder implementation ensures that group subscriptions are persistent and that, once at least one subscription for a group has been created, the group receives messages, even if they are sent while all applications in the group are stopped.

[Note]Note

Anonymous subscriptions are non-durable by nature. For some binder implementations (such as RabbitMQ), it is possible to have non-durable group subscriptions.

In general, it is preferable to always specify a consumer group when binding an application to a given destination. @@ -31,5 +31,5 @@ When scaling up a Spring Cloud Stream application, you must specify a consumer g Doing so prevents the application’s instances from receiving duplicate messages (unless that behavior is desired, which is unusual).

5.6 Partitioning Support

Spring Cloud Stream provides support for partitioning data between multiple instances of a given application. In a partitioned scenario, the physical communication medium (such as the broker topic) is viewed as being structured into multiple partitions. One or more producer application instances send data to multiple consumer application instances and ensure that data identified by common characteristics are processed by the same consumer instance.

Spring Cloud Stream provides a common abstraction for implementing partitioned processing use cases in a uniform fashion. -Partitioning can thus be used whether the broker itself is naturally partitioned (for example, Kafka) or not (for example, RabbitMQ).

Figure 5.4. Spring Cloud Stream Partitioning

SCSt partitioning

Partitioning is a critical concept in stateful processing, where it is critical (for either performance or consistency reasons) to ensure that all related data is processed together. +Partitioning can thus be used whether the broker itself is naturally partitioned (for example, Kafka) or not (for example, RabbitMQ).

Figure 5.4. Spring Cloud Stream Partitioning

SCSt partitioning

Partitioning is a critical concept in stateful processing, where it is critical (for either performance or consistency reasons) to ensure that all related data is processed together. For example, in the time-windowed average calculation example, it is important that all measurements from any given sensor are processed by the same application instance.

[Note]Note

To set up a partitioned processing scenario, you must configure both the data-producing and the data-consuming ends.

\ No newline at end of file diff --git a/multi/multi__programming_model.html b/multi/multi__programming_model.html index ee5a1664b..53f63eaaf 100644 --- a/multi/multi__programming_model.html +++ b/multi/multi__programming_model.html @@ -224,7 +224,8 @@ Consider the following example of a polled consumer:

PollableMessageSource.poll() method takes a MessageHandler argument (often a lambda expression, as shown here).
-It returns true if the message was received and successfully processed.

As with message-driven consumers, if the MessageHandler throws an exception, messages are published to error channels, as discussed in ???.

Normally, the poll() method acknowledges the message when the MessageHandler exits. +It returns true if the message was received and successfully processed.

As with message-driven consumers, if the MessageHandler throws an exception, messages are published to error channels, +as discussed in Section 6.4, “Error Handling”.

Normally, the poll() method acknowledges the message when the MessageHandler exits. If the method exits abnormally, the message is rejected (not re-queued), but see the section called “Handling Errors”. You can override that behavior by taking responsibility for the acknowledgment, as shown in the following example:

@Bean
 public ApplicationRunner poller(PollableMessageSource dest1In, MessageChannel dest2Out) {
@@ -250,7 +251,7 @@ If the service activator throws a RequeueCurrentMessageExc
 The error handling comes in two flavors:

  • application: The error handling is done within the application (custom error handler).
  • system: The error handling is delegated to the binder (re-queue, DL, and others). Note that the techniques are dependent on binder implementation and the capability of the underlying messaging middleware.

Spring Cloud Stream uses the Spring Retry library to facilitate successful message processing. See Section 6.4.3, “Retry Template” for more details. However, when all fails, the exceptions thrown by the message handlers are propagated back to the binder. At that point, binder invokes custom error handler or communicates -the error back to the messaging system (re-queue, DLQ, and others).

6.4.1 Application Error Handling

There are two types of application-level error handling. Errors can be handled at each binding subscription or a global handler can handle all the binding subscription errors. Let’s review the details.

Figure 6.1. A Spring Cloud Stream Sink Application with Custom and Global Error Handlers

custom vs global error channels

For each input binding, Spring Cloud Stream creates a dedicated error channel with the following semantics <destinationName>.errors.

[Note]Note

The <destinationName> consists of the name of the binding (such as input) and the name of the group (such as myGroup).

Consider the following:

spring.cloud.stream.bindings.input.group=myGroup
@StreamListener(Sink.INPUT) // destination name 'input.myGroup'
+the error back to the messaging system (re-queue, DLQ, and others).

6.4.1 Application Error Handling

There are two types of application-level error handling. Errors can be handled at each binding subscription or a global handler can handle all the binding subscription errors. Let’s review the details.

Figure 6.1. A Spring Cloud Stream Sink Application with Custom and Global Error Handlers

custom vs global error channels

For each input binding, Spring Cloud Stream creates a dedicated error channel with the following semantics <destinationName>.errors.

[Note]Note

The <destinationName> consists of the name of the binding (such as input) and the name of the group (such as myGroup).

Consider the following:

spring.cloud.stream.bindings.input.group=myGroup
@StreamListener(Sink.INPUT) // destination name 'input.myGroup'
 public void handle(Person value) {
 	throw new RuntimeException("BOOM!");
 }
diff --git a/multi/multi__quick_start.html b/multi/multi__quick_start.html
index c20ea0a23..d887a8dfd 100644
--- a/multi/multi__quick_start.html
+++ b/multi/multi__quick_start.html
@@ -8,7 +8,7 @@ When the Cloud Stream̶
 We recommend using the one you have already installed or feel more comfortable with installing and running.
 Also, as you can see from the Initilaizer screen, there are a few other options you can choose.
 For example, you can choose Gradle as your build tool instead of Maven (the default).

  • In the Artifact field, type 'logging-consumer'.

    The value of the Artifact field becomes the application name. -If you chose RabbitMQ for the middleware, your Spring Initializr should now be as follows:

    Spring Initializr
  • Click the Generate Project button.

    Doing so downloads the zipped version of the generated project to your hard drive.

  • Unzip the file into the folder you want to use as your project directory.
  • [Tip]Tip

    We encourage you to explore the many possibilities available in the Spring Initializr. +If you chose RabbitMQ for the middleware, your Spring Initializr should now be as follows:

    spring initializr
    1. Click the Generate Project button.

      Doing so downloads the zipped version of the generated project to your hard drive.

    2. Unzip the file into the folder you want to use as your project directory.
    [Tip]Tip

    We encourage you to explore the many possibilities available in the Spring Initializr. It lets you create many different kinds of Spring applications.

    2.2 Importing the Project into Your IDE

    Now you can import the project into your IDE. Keep in mind that, depending on the IDE, you may need to follow a specific import procedure. For example, depending on how the project was generated (Maven or Gradle), you may need to follow specific import procedure (for example, in Eclipse or STS, you need to use File → Import → Maven → Existing Maven Project).

    Once imported, the project must have no errors of any kind. Also, src/main/java should contain com.example.loggingconsumer.LoggingConsumerApplication.

    Technically, at this point, you can run the application’s main class. diff --git a/multi/multi_schema-evolution.html b/multi/multi_schema-evolution.html index 3e28eff72..2daafb4e0 100644 --- a/multi/multi_schema-evolution.html +++ b/multi/multi_schema-evolution.html @@ -79,10 +79,10 @@ If you want to use the Confluent schema registry, you need to create a bean of t return client; }

    [Note]Note

    The ConfluentSchemaRegistryClient is tested against Confluent platform version 4.0.0.

    10.6 Schema Registration and Resolution

    To better understand how Spring Cloud Stream registers and resolves new schemas and its use of Avro schema comparison features, we provide two separate subsections:

    10.6.1 Schema Registration Process (Serialization)

    The first part of the registration process is extracting a schema from the payload that is being sent over a channel. Avro types such as SpecificRecord or GenericRecord already contain a schema, which can be retrieved immediately from the instance. -In the case of POJOs, a schema is inferred if the spring.cloud.stream.schema.avro.dynamicSchemaGenerationEnabled property is set to true (the default).

    Figure 10.1. Schema Writer Resolution Process

    schema resolution

    Ones a schema is obtained, the converter loads its metadata (version) from the remote server. +In the case of POJOs, a schema is inferred if the spring.cloud.stream.schema.avro.dynamicSchemaGenerationEnabled property is set to true (the default).

    Figure 10.1. Schema Writer Resolution Process

    schema resolution

    Ones a schema is obtained, the converter loads its metadata (version) from the remote server. First, it queries a local cache. If no result is found, it submits the data to the server, which replies with versioning information. -The converter always caches the results to avoid the overhead of querying the Schema Server for every new message that needs to be serialized.

    Figure 10.2. Schema Registration Process

    registration

    With the schema version information, the converter sets the contentType header of the message to carry the version information — for example: application/vnd.user.v1+avro.

    10.6.2 Schema Resolution Process (Deserialization)

    When reading messages that contain version information (that is, a contentType header with a scheme like the one described under Section 10.6.1, “Schema Registration Process (Serialization)”), the converter queries the Schema server to fetch the writer schema of the message. -Once it has found the correct schema of the incoming message, it retrieves the reader schema and, by using Avro’s schema resolution support, reads it into the reader definition (setting defaults and any missing properties).

    Figure 10.3. Schema Reading Resolution Process

    schema reading

    [Note]Note

    You should understand the difference between a writer schema (the application that wrote the message) and a reader schema (the receiving application). +The converter always caches the results to avoid the overhead of querying the Schema Server for every new message that needs to be serialized.

    Figure 10.2. Schema Registration Process

    registration

    With the schema version information, the converter sets the contentType header of the message to carry the version information — for example: application/vnd.user.v1+avro.

    10.6.2 Schema Resolution Process (Deserialization)

    When reading messages that contain version information (that is, a contentType header with a scheme like the one described under Section 10.6.1, “Schema Registration Process (Serialization)”), the converter queries the Schema server to fetch the writer schema of the message. +Once it has found the correct schema of the incoming message, it retrieves the reader schema and, by using Avro’s schema resolution support, reads it into the reader definition (setting defaults and any missing properties).

    Figure 10.3. Schema Reading Resolution Process

    schema reading

    [Note]Note

    You should understand the difference between a writer schema (the application that wrote the message) and a reader schema (the receiving application). We suggest taking a moment to read the Avro terminology and understand the process. Spring Cloud Stream always fetches the writer schema to determine how to read a message. If you want to get Avro’s schema evolution support working, you need to make sure that a readerSchema was properly set for your application.

    \ No newline at end of file diff --git a/multi/multi_spring-cloud-stream-overview-binders.html b/multi/multi_spring-cloud-stream-overview-binders.html index 132174267..9d3be77ac 100644 --- a/multi/multi_spring-cloud-stream-overview-binders.html +++ b/multi/multi_spring-cloud-stream-overview-binders.html @@ -1,7 +1,7 @@ 7. Binders

    7. Binders

    Spring Cloud Stream provides a Binder abstraction for use in connecting to physical destinations at the external middleware. -This section provides information about the main concepts behind the Binder SPI, its main components, and implementation-specific details.

    7.1 Producers and Consumers

    The following image shows the general relationship of producers and consumers:

    Figure 7.1. Producers and Consumers

    producers consumers

    A producer is any component that sends messages to a channel. +This section provides information about the main concepts behind the Binder SPI, its main components, and implementation-specific details.

    7.1 Producers and Consumers

    The following image shows the general relationship of producers and consumers:

    Figure 7.1. Producers and Consumers

    producers consumers

    A producer is any component that sends messages to a channel. The channel can be bound to an external message broker with a Binder implementation for that broker. When invoking the bindProducer() method, the first parameter is the name of the destination within the broker, the second parameter is the local channel instance to which the producer sends messages, and the third parameter contains properties (such as a partition key expression) to be used within the adapter that is created for that channel.

    A consumer is any component that receives messages from a channel. As with a producer, the consumer’s channel can be bound to an external message broker. diff --git a/single/spring-cloud-stream.html b/single/spring-cloud-stream.html index 6161b2728..a2696ff59 100644 --- a/single/spring-cloud-stream.html +++ b/single/spring-cloud-stream.html @@ -14,7 +14,7 @@ When the Cloud Stream̶ We recommend using the one you have already installed or feel more comfortable with installing and running. Also, as you can see from the Initilaizer screen, there are a few other options you can choose. For example, you can choose Gradle as your build tool instead of Maven (the default).

  • In the Artifact field, type 'logging-consumer'.

    The value of the Artifact field becomes the application name. -If you chose RabbitMQ for the middleware, your Spring Initializr should now be as follows:

    Spring Initializr
  • Click the Generate Project button.

    Doing so downloads the zipped version of the generated project to your hard drive.

  • Unzip the file into the folder you want to use as your project directory.
  • [Tip]Tip

    We encourage you to explore the many possibilities available in the Spring Initializr. +If you chose RabbitMQ for the middleware, your Spring Initializr should now be as follows:

    spring initializr
    1. Click the Generate Project button.

      Doing so downloads the zipped version of the generated project to your hard drive.

    2. Unzip the file into the folder you want to use as your project directory.
    [Tip]Tip

    We encourage you to explore the many possibilities available in the Spring Initializr. It lets you create many different kinds of Spring applications.

    2.2 Importing the Project into Your IDE

    Now you can import the project into your IDE. Keep in mind that, depending on the IDE, you may need to follow a specific import procedure. For example, depending on how the project was generated (Maven or Gradle), you may need to follow specific import procedure (for example, in Eclipse or STS, you need to use File → Import → Maven → Existing Maven Project).

    Once imported, the project must have no errors of any kind. Also, src/main/java should contain com.example.loggingconsumer.LoggingConsumerApplication.

    Technically, at this point, you can run the application’s main class. @@ -127,7 +127,7 @@ You can use this in the application by autowiring it, as shown in the following }

    5. Main Concepts

    Spring Cloud Stream provides a number of abstractions and primitives that simplify the writing of message-driven microservice applications. This section gives an overview of the following:

    5.1 Application Model

    A Spring Cloud Stream application consists of a middleware-neutral core. The application communicates with the outside world through input and output channels injected into it by Spring Cloud Stream. -Channels are connected to external brokers through middleware-specific Binder implementations.

    Figure 5.1. Spring Cloud Stream Application

    SCSt with binder

    5.1.1 Fat JAR

    Spring Cloud Stream applications can be run in stand-alone mode from your IDE for testing. +Channels are connected to external brokers through middleware-specific Binder implementations.

    Figure 5.1. Spring Cloud Stream Application

    SCSt with binder

    5.1.1 Fat JAR

    Spring Cloud Stream applications can be run in stand-alone mode from your IDE for testing. To run a Spring Cloud Stream application in production, you can create an executable (or fat) JAR by using the standard Spring Boot tooling provided for Maven or Gradle. See the Spring Boot Reference Guide for more details.

    5.2 The Binder Abstraction

    Spring Cloud Stream provides Binder implementations for Kafka and Rabbit MQ. Spring Cloud Stream also includes a TestSupportBinder, which leaves a channel unmodified so that tests can interact with channels directly and reliably assert on what is received. You can also use the extensible API to write your own Binder.

    Spring Cloud Stream uses Spring Boot for configuration, and the Binder abstraction makes it possible for a Spring Cloud Stream application to be flexible in how it connects to middleware. @@ -137,7 +137,7 @@ In the sink example from the

    5.3 Persistent Publish-Subscribe Support

    Communication between applications follows a publish-subscribe model, where data is broadcast through shared topics. -This can be seen in the following figure, which shows a typical deployment for a set of interacting Spring Cloud Stream applications.

    Figure 5.2. Spring Cloud Stream Publish-Subscribe

    SCSt sensors

    Data reported by sensors to an HTTP endpoint is sent to a common destination named raw-sensor-data. +This can be seen in the following figure, which shows a typical deployment for a set of interacting Spring Cloud Stream applications.

    Figure 5.2. Spring Cloud Stream Publish-Subscribe

    SCSt sensors

    Data reported by sensors to an HTTP endpoint is sent to a common destination named raw-sensor-data. From the destination, it is independently processed by a microservice application that computes time-windowed averages and by another microservice application that ingests the raw data into HDFS (Hadoop Distributed File System). In order to process the data, both applications declare the topic as their input at runtime.

    The publish-subscribe communication model reduces the complexity of both the producer and the consumer and lets new applications be added to the topology without disruption of the existing flow. For example, downstream from the average-calculating application, you can add an application that calculates the highest temperature values for display and monitoring. @@ -147,7 +147,7 @@ By using native middleware support, Spring Cloud Stream also simplifies use of t When doing so, different instances of an application are placed in a competing consumer relationship, where only one of the instances is expected to handle a given message.

    Spring Cloud Stream models this behavior through the concept of a consumer group. (Spring Cloud Stream consumer groups are similar to and inspired by Kafka consumer groups.) Each consumer binding can use the spring.cloud.stream.bindings.<channelName>.group property to specify a group name. -For the consumers shown in the following figure, this property would be set as spring.cloud.stream.bindings.<channelName>.group=hdfsWrite or spring.cloud.stream.bindings.<channelName>.group=average.

    Figure 5.3. Spring Cloud Stream Consumer Groups

    SCSt groups

    All groups that subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination. +For the consumers shown in the following figure, this property would be set as spring.cloud.stream.bindings.<channelName>.group=hdfsWrite or spring.cloud.stream.bindings.<channelName>.group=average.

    Figure 5.3. Spring Cloud Stream Consumer Groups

    SCSt groups

    All groups that subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination. By default, when a group is not specified, Spring Cloud Stream assigns the application to an anonymous and independent single-member consumer group that is in a publish-subscribe relationship with all other consumer groups.

    5.5 Consumer Types

    Two types of consumer are supported:

    • Message-driven (sometimes referred to as Asynchronous)
    • Polled (sometimes referred to as Synchronous)

    Prior to version 2.0, only asynchronous consumers were supported. A message is delivered as soon as it is available and a thread is available to process it.

    When you wish to control the rate at which messages are processed, you might want to use a synchronous consumer.

    5.5.1 Durability

    Consistent with the opinionated application model of Spring Cloud Stream, consumer group subscriptions are durable. That is, a binder implementation ensures that group subscriptions are persistent and that, once at least one subscription for a group has been created, the group receives messages, even if they are sent while all applications in the group are stopped.

    [Note]Note

    Anonymous subscriptions are non-durable by nature. For some binder implementations (such as RabbitMQ), it is possible to have non-durable group subscriptions.

    In general, it is preferable to always specify a consumer group when binding an application to a given destination. @@ -155,7 +155,7 @@ When scaling up a Spring Cloud Stream application, you must specify a consumer g Doing so prevents the application’s instances from receiving duplicate messages (unless that behavior is desired, which is unusual).

    5.6 Partitioning Support

    Spring Cloud Stream provides support for partitioning data between multiple instances of a given application. In a partitioned scenario, the physical communication medium (such as the broker topic) is viewed as being structured into multiple partitions. One or more producer application instances send data to multiple consumer application instances and ensure that data identified by common characteristics are processed by the same consumer instance.

    Spring Cloud Stream provides a common abstraction for implementing partitioned processing use cases in a uniform fashion. -Partitioning can thus be used whether the broker itself is naturally partitioned (for example, Kafka) or not (for example, RabbitMQ).

    Figure 5.4. Spring Cloud Stream Partitioning

    SCSt partitioning

    Partitioning is a critical concept in stateful processing, where it is critical (for either performance or consistency reasons) to ensure that all related data is processed together. +Partitioning can thus be used whether the broker itself is naturally partitioned (for example, Kafka) or not (for example, RabbitMQ).

    Figure 5.4. Spring Cloud Stream Partitioning

    SCSt partitioning

    Partitioning is a critical concept in stateful processing, where it is critical (for either performance or consistency reasons) to ensure that all related data is processed together. For example, in the time-windowed average calculation example, it is important that all measurements from any given sensor are processed by the same application instance.

    [Note]Note

    To set up a partitioned processing scenario, you must configure both the data-producing and the data-consuming ends.

    6. Programming Model

    To understand the programming model, you should be familiar with the following core concepts:

    • Destination Binders: Components responsible to provide integration with the external messaging systems.
    • Destination Bindings: Bridge between the external messaging systems and application provided Producers and Consumers of messages (created by the Destination Binders).
    • Message: The canonical data structure used by producers and consumers to communicate with Destination Binders (and thus other applications via external messaging systems).
    SCSt overview

    6.1 Destination Binders

    Destination Binders are extension components of Spring Cloud Stream responsible for providing the necessary configuration and implementation to facilitate integration with external messaging systems. This integration is responsible for connectivity, delegation, and routing of messages to and from producers and consumers, data type conversion, @@ -380,7 +380,8 @@ Consider the following example of a polled consumer:

    PollableMessageSource.poll() method takes a MessageHandler argument (often a lambda expression, as shown here).
    -It returns true if the message was received and successfully processed.

    As with message-driven consumers, if the MessageHandler throws an exception, messages are published to error channels, as discussed in ???.

    Normally, the poll() method acknowledges the message when the MessageHandler exits. +It returns true if the message was received and successfully processed.

    As with message-driven consumers, if the MessageHandler throws an exception, messages are published to error channels, +as discussed in Section 6.4, “Error Handling”.

    Normally, the poll() method acknowledges the message when the MessageHandler exits. If the method exits abnormally, the message is rejected (not re-queued), but see the section called “Handling Errors”. You can override that behavior by taking responsibility for the acknowledgment, as shown in the following example:

    @Bean
     public ApplicationRunner poller(PollableMessageSource dest1In, MessageChannel dest2Out) {
    @@ -406,7 +407,7 @@ If the service activator throws a RequeueCurrentMessageExc
     The error handling comes in two flavors:

    • application: The error handling is done within the application (custom error handler).
    • system: The error handling is delegated to the binder (re-queue, DL, and others). Note that the techniques are dependent on binder implementation and the capability of the underlying messaging middleware.

    Spring Cloud Stream uses the Spring Retry library to facilitate successful message processing. See Section 6.4.3, “Retry Template” for more details. However, when all fails, the exceptions thrown by the message handlers are propagated back to the binder. At that point, binder invokes custom error handler or communicates -the error back to the messaging system (re-queue, DLQ, and others).

    6.4.1 Application Error Handling

    There are two types of application-level error handling. Errors can be handled at each binding subscription or a global handler can handle all the binding subscription errors. Let’s review the details.

    Figure 6.1. A Spring Cloud Stream Sink Application with Custom and Global Error Handlers

    custom vs global error channels

    For each input binding, Spring Cloud Stream creates a dedicated error channel with the following semantics <destinationName>.errors.

    [Note]Note

    The <destinationName> consists of the name of the binding (such as input) and the name of the group (such as myGroup).

    Consider the following:

    spring.cloud.stream.bindings.input.group=myGroup
    @StreamListener(Sink.INPUT) // destination name 'input.myGroup'
    +the error back to the messaging system (re-queue, DLQ, and others).

    6.4.1 Application Error Handling

    There are two types of application-level error handling. Errors can be handled at each binding subscription or a global handler can handle all the binding subscription errors. Let’s review the details.

    Figure 6.1. A Spring Cloud Stream Sink Application with Custom and Global Error Handlers

    custom vs global error channels

    For each input binding, Spring Cloud Stream creates a dedicated error channel with the following semantics <destinationName>.errors.

    [Note]Note

    The <destinationName> consists of the name of the binding (such as input) and the name of the group (such as myGroup).

    Consider the following:

    spring.cloud.stream.bindings.input.group=myGroup
    @StreamListener(Sink.INPUT) // destination name 'input.myGroup'
     public void handle(Person value) {
     	throw new RuntimeException("BOOM!");
     }
    @@ -549,7 +550,7 @@ The Publisher in the following example still uses R
             .toReactivePublisher();
       }
     }

    7. Binders

    Spring Cloud Stream provides a Binder abstraction for use in connecting to physical destinations at the external middleware. -This section provides information about the main concepts behind the Binder SPI, its main components, and implementation-specific details.

    7.1 Producers and Consumers

    The following image shows the general relationship of producers and consumers:

    Figure 7.1. Producers and Consumers

    producers consumers

    A producer is any component that sends messages to a channel. +This section provides information about the main concepts behind the Binder SPI, its main components, and implementation-specific details.

    7.1 Producers and Consumers

    The following image shows the general relationship of producers and consumers:

    Figure 7.1. Producers and Consumers

    producers consumers

    A producer is any component that sends messages to a channel. The channel can be bound to an external message broker with a Binder implementation for that broker. When invoking the bindProducer() method, the first parameter is the name of the destination within the broker, the second parameter is the local channel instance to which the producer sends messages, and the third parameter contains properties (such as a partition key expression) to be used within the adapter that is created for that channel.

    A consumer is any component that receives messages from a channel. As with a producer, the consumer’s channel can be bound to an external message broker. @@ -677,7 +678,7 @@ When this configuration is being used, the outbound message marshalling is not b When native encoding is used, it is the responsibility of the consumer to use an appropriate decoder (for example, the Kafka consumer value de-serializer) to deserialize the inbound message. Also, when native encoding and decoding is used, the headerMode=embeddedHeaders property is ignored and headers are not embedded in the message. See the consumer property useNativeDecoding.

    Default: false.

    errorChannelEnabled

    When set to true, if the binder supports asynchroous send results, send failures are sent to an error channel for the destination. -See ??? for more information.

    Default: false.

    8.3 Using Dynamically Bound Destinations

    Besides the channels defined by using @EnableBinding, Spring Cloud Stream lets applications send messages to dynamically bound destinations. +See Section 6.4, “Error Handling” for more information.

    Default: false.

    8.3 Using Dynamically Bound Destinations

    Besides the channels defined by using @EnableBinding, Spring Cloud Stream lets applications send messages to dynamically bound destinations. This is useful, for example, when the target destination needs to be determined at runtime. Applications can do so by using the BinderAwareChannelResolver bean, registered automatically by the @EnableBinding annotation.

    The 'spring.cloud.stream.dynamicDestinations' property can be used for restricting the dynamic destination names to a known set (whitelisting). If this property is not set, any destination can be bound dynamically.

    The BinderAwareChannelResolver can be used directly, as shown in the following example of a REST controller using a path variable to decide the target channel:

    @EnableBinding
    @@ -899,10 +900,10 @@ If you want to use the Confluent schema registry, you need to create a bean of t
       return client;
     }
    [Note]Note

    The ConfluentSchemaRegistryClient is tested against Confluent platform version 4.0.0.

    10.6 Schema Registration and Resolution

    To better understand how Spring Cloud Stream registers and resolves new schemas and its use of Avro schema comparison features, we provide two separate subsections:

    10.6.1 Schema Registration Process (Serialization)

    The first part of the registration process is extracting a schema from the payload that is being sent over a channel. Avro types such as SpecificRecord or GenericRecord already contain a schema, which can be retrieved immediately from the instance. -In the case of POJOs, a schema is inferred if the spring.cloud.stream.schema.avro.dynamicSchemaGenerationEnabled property is set to true (the default).

    Figure 10.1. Schema Writer Resolution Process

    schema resolution

    Ones a schema is obtained, the converter loads its metadata (version) from the remote server. +In the case of POJOs, a schema is inferred if the spring.cloud.stream.schema.avro.dynamicSchemaGenerationEnabled property is set to true (the default).

    Figure 10.1. Schema Writer Resolution Process

    schema resolution

    Ones a schema is obtained, the converter loads its metadata (version) from the remote server. First, it queries a local cache. If no result is found, it submits the data to the server, which replies with versioning information. -The converter always caches the results to avoid the overhead of querying the Schema Server for every new message that needs to be serialized.

    Figure 10.2. Schema Registration Process

    registration

    With the schema version information, the converter sets the contentType header of the message to carry the version information — for example: application/vnd.user.v1+avro.

    10.6.2 Schema Resolution Process (Deserialization)

    When reading messages that contain version information (that is, a contentType header with a scheme like the one described under Section 10.6.1, “Schema Registration Process (Serialization)”), the converter queries the Schema server to fetch the writer schema of the message. -Once it has found the correct schema of the incoming message, it retrieves the reader schema and, by using Avro’s schema resolution support, reads it into the reader definition (setting defaults and any missing properties).

    Figure 10.3. Schema Reading Resolution Process

    schema reading

    [Note]Note

    You should understand the difference between a writer schema (the application that wrote the message) and a reader schema (the receiving application). +The converter always caches the results to avoid the overhead of querying the Schema Server for every new message that needs to be serialized.

    Figure 10.2. Schema Registration Process

    registration

    With the schema version information, the converter sets the contentType header of the message to carry the version information — for example: application/vnd.user.v1+avro.

    10.6.2 Schema Resolution Process (Deserialization)

    When reading messages that contain version information (that is, a contentType header with a scheme like the one described under Section 10.6.1, “Schema Registration Process (Serialization)”), the converter queries the Schema server to fetch the writer schema of the message. +Once it has found the correct schema of the incoming message, it retrieves the reader schema and, by using Avro’s schema resolution support, reads it into the reader definition (setting defaults and any missing properties).

    Figure 10.3. Schema Reading Resolution Process

    schema reading

    [Note]Note

    You should understand the difference between a writer schema (the application that wrote the message) and a reader schema (the receiving application). We suggest taking a moment to read the Avro terminology and understand the process. Spring Cloud Stream always fetches the writer schema to determine how to read a message. If you want to get Avro’s schema evolution support working, you need to make sure that a readerSchema was properly set for your application.

    11. Inter-Application Communication

    Spring Cloud Stream enables communication between applications. Inter-application communication is a complex issue spanning several concerns, as described in the following topics:

    11.1 Connecting Multiple Application Instances

    While Spring Cloud Stream makes it easy for individual Spring Boot applications to connect to messaging systems, the typical scenario for Spring Cloud Stream is the creation of multi-application pipelines, where microservice applications send data to each other. @@ -1048,4 +1049,4 @@ For example, specifying spring.integration.* captur } ] }

    [Note]Note

    Given that the format of the Metric message has slightly changed after migrating to Micrometer, the published message will also have -a STREAM_CLOUD_STREAM_VERSION header set to 2.x to help distinguish between Metric messages from the older versions of the Spring Cloud Stream.

    15. Samples

    For Spring Cloud Stream samples, see the spring-cloud-stream-samples repository on GitHub.

    15.1 Deploying Stream Applications on CloudFoundry

    On CloudFoundry, services are usually exposed through a special environment variable called VCAP_SERVICES.

    When configuring your binder connections, you can use the values from an environment variable as explained on the dataflow Cloud Foundry Server docs.

    \ No newline at end of file +a STREAM_CLOUD_STREAM_VERSION header set to 2.x to help distinguish between Metric messages from the older versions of the Spring Cloud Stream.

    15. Samples

    For Spring Cloud Stream samples, see the spring-cloud-stream-samples repository on GitHub.

    15.1 Deploying Stream Applications on CloudFoundry

    On CloudFoundry, services are usually exposed through a special environment variable called VCAP_SERVICES.

    When configuring your binder connections, you can use the values from an environment variable as explained on the dataflow Cloud Foundry Server docs.

    \ No newline at end of file diff --git a/spring-cloud-stream.xml b/spring-cloud-stream.xml index 0af2cc3a6..317d8a364 100644 --- a/spring-cloud-stream.xml +++ b/spring-cloud-stream.xml @@ -4,7 +4,7 @@ Spring Cloud Stream Reference Guide -2018-11-16 +2018-12-12 @@ -172,15 +172,17 @@ For example, you can choose Gradle as your build tool instead of Maven (the defa In the Artifact field, type 'logging-consumer'. The value of the Artifact field becomes the application name. If you chose RabbitMQ for the middleware, your Spring Initializr should now be as follows: + + - + -Spring Initializr +spring initializr - + Click the Generate Project button. Doing so downloads the zipped version of the generated project to your hard drive. @@ -993,7 +995,8 @@ public ApplicationRunner poller(PollableMessageSource destIn, MessageChannel des } The PollableMessageSource.poll() method takes a MessageHandler argument (often a lambda expression, as shown here). It returns true if the message was received and successfully processed. -As with message-driven consumers, if the MessageHandler throws an exception, messages are published to error channels, as discussed in . +As with message-driven consumers, if the MessageHandler throws an exception, messages are published to error channels, +as discussed in . Normally, the poll() method acknowledges the message when the MessageHandler exits. If the method exits abnormally, the message is rejected (not re-queued), but see . You can override that behavior by taking responsibility for the acknowledgment, as shown in the following example: @@ -1884,7 +1887,7 @@ See the consumer property useNativeDecoding. errorChannelEnabled When set to true, if the binder supports asynchroous send results, send failures are sent to an error channel for the destination. -See for more information. +See for more information. Default: false. @@ -2874,22 +2877,22 @@ a STREAM_CLOUD_STREAM_VERSION header set to 2.xThe following is the list of available binder implementations -[RabbitMQ](https://cloud.spring.io/spring-cloud-stream-binder-rabbit/) +RabbitMQ -[Apache Kafka](https://cloud.spring.io/spring-cloud-stream-binder-kafka/) +Apache Kafka -[Amazon Kinesis](https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis) +Amazon Kinesis -[Google PubSub (partner maintained)](https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-pubsub-stream-binder) +Google PubSub (partner maintained) -[Solace PubSub+ (partner maintained)](https://github.com/SolaceProducts/spring-cloud-stream-binder-solace) +Solace PubSub+ (partner maintained) -[Azure Event Hubs (partner maintained)](https://github.com/Microsoft/spring-cloud-azure/tree/master/spring-cloud-azure-eventhub-stream-binder) +Azure Event Hubs (partner maintained)