diff --git a/docs/modules/ROOT/pages/kafka/kafka-streams.adoc b/docs/modules/ROOT/pages/kafka/kafka-streams.adoc index b56207830..8befe8a2f 100644 --- a/docs/modules/ROOT/pages/kafka/kafka-streams.adoc +++ b/docs/modules/ROOT/pages/kafka/kafka-streams.adoc @@ -2,2153 +2,3 @@ = Kafka Streams Binder -[[usage]] -== Usage - -For using the Kafka Streams binder, you just need to add it to your Spring Cloud Stream application, using the following maven coordinates: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-kafka-streams - ----- - -A quick way to bootstrap a new project for Kafka Streams binder is to use http://start.spring.io[Spring Initializr] and then select "Cloud Streams" and "Spring for Kafka Streams" as shown below - -image::{github-raw}/docs/src/main/asciidoc/images/spring-initializr-kafka-streams.png[width=800,scaledwidth="75%",align="center"] - -[[overview]] -== Overview - -Spring Cloud Stream includes a binder implementation designed explicitly for https://kafka.apache.org/documentation/streams/[Apache Kafka Streams] binding. -With this native integration, a Spring Cloud Stream "processor" application can directly use the -https://kafka.apache.org/documentation/streams/developer-guide[Apache Kafka Streams] APIs in the core business logic. - -Kafka Streams binder implementation builds on the foundations provided by the https://docs.spring.io/spring-kafka/reference/html/#kafka-streams[Spring for Apache Kafka] project. - -Kafka Streams binder provides binding capabilities for the three major types in Kafka Streams - `KStream`, `KTable` and `GlobalKTable`. - -Kafka Streams applications typically follow a model in which the records are read from an inbound topic, apply business logic, and then write the transformed records to an outbound topic. -Alternatively, a Processor application with no outbound destination can be defined as well. - -In the following sections, we are going to look at the details of Spring Cloud Stream's integration with Kafka Streams. - -[[programming-model]] -== Programming Model - -When using the programming model provided by Kafka Streams binder, both the high-level https://docs.confluent.io/current/streams/developer-guide/dsl-api.html[Streams DSL] and a mix of both the higher level and the lower level https://docs.confluent.io/current/streams/developer-guide/processor-api.html[Processor-API] can be used as options. -When mixing both higher and lower level API's, this is usually achieved by invoking `transform` or `process` API methods on `KStream`. - -[[functional-style]] -=== Functional Style - -Starting with Spring Cloud Stream `3.0.0`, Kafka Streams binder allows the applications to be designed and developed using the functional programming style that is available in Java 8. -This means that the applications can be concisely represented as a lambda expression of types `java.util.function.Function` or `java.util.function.Consumer`. - -Let's take a very basic example. - -[source] ----- -@SpringBootApplication -public class SimpleConsumerApplication { - - @Bean - public java.util.function.Consumer> process() { - - return input -> - input.foreach((key, value) -> { - System.out.println("Key: " + key + " Value: " + value); - }); - } -} ----- - -Albeit simple, this is a complete standalone Spring Boot application that is leveraging Kafka Streams for stream processing. -This is a consumer application with no outbound binding and only a single inbound binding. -The application consumes data and it simply logs the information from the `KStream` key and value on the standard output. -The application contains the `SpringBootApplication` annotation and a method that is marked as `Bean`. -The bean method is of type `java.util.function.Consumer` which is parameterized with `KStream`. -Then in the implementation, we are returning a Consumer object that is essentially a lambda expression. -Inside the lambda expression, the code for processing the data is provided. - -In this application, there is a single input binding that is of type `KStream`. -The binder creates this binding for the application with a name `process-in-0`, i.e. the name of the function bean name followed by a dash character (`-`) and the literal `in` followed by another dash and then the ordinal position of the parameter. -You use this binding name to set other properties such as destination. -For example, `spring.cloud.stream.bindings.process-in-0.destination=my-topic`. - -NOTE: If the destination property is not set on the binding, a topic is created with the same name as the binding (if there are sufficient privileges for the application) or that topic is expected to be already available. - -Once built as a uber-jar (e.g., `kstream-consumer-app.jar`), you can run the above example like the following. - -If the applications choose to define the functional beans using Spring's `Component` annotation, the binder also supports that model. -The above functional bean could be rewritten as below. - -``` -@Component(name = "process") -public class SimpleConsumer implements java.util.function.Consumer> { - - @Override - public void accept(KStream input) { - input.foreach((key, value) -> { - System.out.println("Key: " + key + " Value: " + value); - }); - } -} -``` - -[source] ----- -java -jar kstream-consumer-app.jar --spring.cloud.stream.bindings.process-in-0.destination=my-topic ----- - -Here is another example, where it is a full processor with both input and output bindings. -This is the classic word-count example in which the application receives data from a topic, the number of occurrences for each word is then computed in a tumbling time-window. - -[source] ----- -@SpringBootApplication -public class WordCountProcessorApplication { - - @Bean - public Function, KStream> process() { - - return input -> input - .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+"))) - .map((key, value) -> new KeyValue<>(value, value)) - .groupByKey(Serialized.with(Serdes.String(), Serdes.String())) - .windowedBy(TimeWindows.of(5000)) - .count(Materialized.as("word-counts-state-store")) - .toStream() - .map((key, value) -> new KeyValue<>(key.key(), new WordCount(key.key(), value, - new Date(key.window().start()), new Date(key.window().end())))); - } - - public static void main(String[] args) { - SpringApplication.run(WordCountProcessorApplication.class, args); - } -} ----- - -Here again, this is a complete Spring Boot application. The difference here from the first application is that the bean method is of type `java.util.function.Function`. -The first parameterized type for the `Function` is for the input `KStream` and the second one is for the output. -In the method body, a lambda expression is provided that is of type `Function` and as implementation, the actual business logic is given. -Similar to the previously discussed Consumer based application, the input binding here is named as `process-in-0` by default. For the output, the binding name is automatically also set to `process-out-0`. - -Once built as an uber-jar (e.g., `wordcount-processor.jar`), you can run the above example like the following. - -[source] ----- -java -jar wordcount-processor.jar --spring.cloud.stream.bindings.process-in-0.destination=words --spring.cloud.stream.bindings.process-out-0.destination=counts ----- - -This application will consume messages from the Kafka topic `words` and the computed results are published to an output -topic `counts`. - -Spring Cloud Stream will ensure that the messages from both the incoming and outgoing topics are automatically bound as -KStream objects. As a developer, you can exclusively focus on the business aspects of the code, i.e. writing the logic -required in the processor. Setting up Kafka Streams specific configuration required by the Kafka Streams infrastructure -is automatically handled by the framework. - -The two examples we saw above have a single `KStream` input binding. In both cases, the bindings received the records from a single topic. -If you want to multiplex multiple topics into a single `KStream` binding, you can provide comma separated Kafka topics as destinations below. - -`spring.cloud.stream.bindings.process-in-0.destination=topic-1,topic-2,topic-3` - -In addition, you can also provide topic patterns as destinations if you want to match topics against a regular exression. - -`spring.cloud.stream.bindings.process-in-0.destination=input.*` - -[[multiple-input-bindings]] -==== Multiple Input Bindings - -Many non-trivial Kafka Streams applications often consume data from more than one topic through multiple bindings. -For instance, one topic is consumed as `Kstream` and another as `KTable` or `GlobalKTable`. -There are many reasons why an application might want to receive data as a table type. -Think of a use-case where the underlying topic is populated through a change data capture (CDC) mechanism from a database or perhaps the application only cares about the latest updates for downstream processing. -If the application specifies that the data needs to be bound as `KTable` or `GlobalKTable`, then Kafka Streams binder will properly bind the destination to a `KTable` or `GlobalKTable` and make them available for the application to operate upon. -We will look at a few different scenarios how multiple input bindings are handled in the Kafka Streams binder. - -[[bifunction-in-kafka-streams-binder]] -===== BiFunction in Kafka Streams Binder - -Here is an example where we have two inputs and an output. In this case, the application can leverage on `java.util.function.BiFunction`. - -[source] ----- -@Bean -public BiFunction, KTable, KStream> process() { - return (userClicksStream, userRegionsTable) -> (userClicksStream - .leftJoin(userRegionsTable, (clicks, region) -> new RegionWithClicks(region == null ? - "UNKNOWN" : region, clicks), - Joined.with(Serdes.String(), Serdes.Long(), null)) - .map((user, regionWithClicks) -> new KeyValue<>(regionWithClicks.getRegion(), - regionWithClicks.getClicks())) - .groupByKey(Grouped.with(Serdes.String(), Serdes.Long())) - .reduce(Long::sum) - .toStream()); -} ----- - -Here again, the basic theme is the same as in the previous examples, but here we have two inputs. -Java's `BiFunction` support is used to bind the inputs to the desired destinations. -The default binding names generated by the binder for the inputs are `process-in-0` and `process-in-1` respectively. The default output binding is `process-out-0`. -In this example, the first parameter of `BiFunction` is bound as a `KStream` for the first input and the second parameter is bound as a `KTable` for the second input. - -[[biconsumer-in-kafka-streams-binder]] -===== BiConsumer in Kafka Streams Binder - -If there are two inputs, but no outputs, in that case we can use `java.util.function.BiConsumer` as shown below. - -[source] ----- -@Bean -public BiConsumer, KTable> process() { - return (userClicksStream, userRegionsTable) -> {} -} ----- - -[[beyond-two-inputs]] -===== Beyond two inputs - -What if you have more than two inputs? -There are situations in which you need more than two inputs. In that case, the binder allows you to chain partial functions. -In functional programming jargon, this technique is generally known as currying. -With the functional programming support added as part of Java 8, Java now enables you to write curried functions. -Spring Cloud Stream Kafka Streams binder can make use of this feature to enable multiple input bindings. - -Let's see an example. - -[source] ----- -@Bean -public Function, - Function, - Function, KStream>>> enrichOrder() { - - return orders -> ( - customers -> ( - products -> ( - orders.join(customers, - (orderId, order) -> order.getCustomerId(), - (order, customer) -> new CustomerOrder(customer, order)) - .join(products, - (orderId, customerOrder) -> customerOrder - .productId(), - (customerOrder, product) -> { - EnrichedOrder enrichedOrder = new EnrichedOrder(); - enrichedOrder.setProduct(product); - enrichedOrder.setCustomer(customerOrder.customer); - enrichedOrder.setOrder(customerOrder.order); - return enrichedOrder; - }) - ) - ) - ); -} ----- - -Let's look at the details of the binding model presented above. -In this model, we have 3 partially applied functions on the inbound. Let's call them as `f(x)`, `f(y)` and `f(z)`. -If we expand these functions in the sense of true mathematical functions, it will look like these: `f(x) -> (fy) -> f(z) -> KStream`. -The `x` variable stands for `KStream`, the `y` variable stands for `GlobalKTable` and the `z` variable stands for `GlobalKTable`. -The first function `f(x)` has the first input binding of the application (`KStream`) and its output is the function, f(y). -The function `f(y)` has the second input binding for the application (`GlobalKTable`) and its output is yet another function, `f(z)`. -The input for the function `f(z)` is the third input for the application (`GlobalKTable`) and its output is `KStream` which is the final output binding for the application. -The input from the three partial functions which are `KStream`, `GlobalKTable`, `GlobalKTable` respectively are available for you in the method body for implementing the business logic as part of the lambda expression. - -Input bindings are named as `enrichOrder-in-0`, `enrichOrder-in-1` and `enrichOrder-in-2` respectively. Output binding is named as `enrichOrder-out-0`. - -With curried functions, you can virtually have any number of inputs. However, keep in mind that, anything more than a smaller number of inputs and partially applied functions for them as above in Java might lead to unreadable code. -Therefore if your Kafka Streams application requires more than a reasonably smaller number of input bindings, and you want to use this functional model, then you may want to rethink your design and decompose the application appropriately. - -[[output-bindings]] -==== Output Bindings - -Kafka Streams binder allows types of either `KStream` or `KTable` as output bindings. -Behind the scenes, the binder uses the `to` method on `KStream` to send the resultant records to the output topic. -If the application provides a `KTable` as output in the function, the binder still uses this technique by delegating to the `to` method of `KStream`. - -For example both functions below will work: - -``` -@Bean -public Function, KTable> foo() { - return KStream::toTable; - }; -} - -@Bean -public Function, KStream> bar() { - return KTable::toStream; -} -``` - -[[multiple-output-bindings]] -==== Multiple Output Bindings - -Kafka Streams allows writing outbound data into multiple topics. This feature is known as branching in Kafka Streams. -When using multiple output bindings, you need to provide an array of KStream (`KStream[]`) as the outbound return type. - -Here is an example: - -[source] ----- -@Bean -public Function, KStream[]> process() { - - Predicate isEnglish = (k, v) -> v.word.equals("english"); - Predicate isFrench = (k, v) -> v.word.equals("french"); - Predicate isSpanish = (k, v) -> v.word.equals("spanish"); - - return input -> { - final Map> stringKStreamMap = input - .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+"))) - .groupBy((key, value) -> value) - .windowedBy(TimeWindows.of(Duration.ofSeconds(5))) - .count(Materialized.as("WordCounts-branch")) - .toStream() - .map((key, value) -> new KeyValue<>(null, new WordCount(key.key(), value, - new Date(key.window().start()), new Date(key.window().end())))) - .split() - .branch(isEnglish) - .branch(isFrench) - .branch(isSpanish) - .noDefaultBranch(); - - return stringKStreamMap.values().toArray(new KStream[0]); - }; -} ----- - -The programming model remains the same, however the outbound parameterized type is `KStream[]`. -The default output binding names are `process-out-0`, `process-out-1`, `process-out-2` respectively for the function above. -The reason why the binder generates three output bindings is because it detects the length of the returned `KStream` array as three. -Note that in this example, we provide a `noDefaultBranch()`; if we have used `defaultBranch()` instead, that would have required an extra output binding, essentially returning a `KStream` array of length four. - -[[summary-of-function-based-programming-styles-for-kafka-streams]] -==== Summary of Function based Programming Styles for Kafka Streams - -In summary, the following table shows the various options that can be used in the functional paradigm. - -|=== -|Number of Inputs |Number of Outputs |Component to use - -|1|0|java.util.function.Consumer -|2|0|java.util.function.BiConsumer -|1|1..n |java.util.function.Function -|2|1..n |java.util.function.BiFunction -|>= 3 |0..n |Use curried functions - -|=== - -* In the case of more than one output in this table, the type simply becomes `KStream[]`. - -[[function-composition-in-kafka-streams-binder]] -==== Function composition in Kafka Streams binder - -Kafka Streams binder supports minimal forms of functional composition for linear topologies. -Using the Java functional API support, you can write multiple functions and then compose them on your own using the `andThen` method. -For example, assume that you have the following two functions. - -``` -@Bean -public Function, KStream> foo() { - return input -> input.peek((s, s2) -> {}); -} - -@Bean -public Function, KStream> bar() { - return input -> input.peek((s, s2) -> {}); -} -``` - -Even without the functional composition support in the binder, you can compose these two functions as below. - -``` -@Bean -public Function, KStream> composed() { - foo().andThen(bar()); -} -``` - -Then you can provide definitions of the form `spring.cloud.function.definition=foo;bar;composed`. -With the functional composition support in the binder, you don't need to write this third function in which you are doing explicit function composition. - -You can simply do this instead: - -``` -spring.cloud.function.definition=foo|bar -``` - -You can even do this: - -``` -spring.cloud.function.definition=foo|bar;foo;bar -``` - -The composed function's default binding names in this example becomes `foobar-in-0` and `foobar-out-0`. - -[[limitations-of-functional-composition-in-kafka-streams-bincer]] -===== Limitations of functional composition in Kafka Streams bincer - -When you have `java.util.function.Function` bean, that can be composed with another function or multiple functions. -The same function bean can be composed with a `java.util.function.Consumer` as well. In this case, consumer is the last component composed. -A function can be composed with multiple functions, then end with a `java.util.function.Consumer` bean as well. - -When composing the beans of type `java.util.function.BiFunction`, the `BiFunction` must be the first function in the definition. -The composed entities must be either of type `java.util.function.Function` or `java.util.funciton.Consumer`. -In other words, you cannot take a `BiFunction` bean and then compose with another `BiFunction`. - -You cannot compose with types of `BiConsumer` or definitions where `Consumer` is the first component. -You cannot also compose with functions where the output is an array (`KStream[]` for branching) unless this is the last component in the definition. - -The very first `Function` of `BiFunction` in the function definition may use a curried form also. -For example, the following is possible. - -``` -@Bean -public Function, Function, KStream>> curriedFoo() { - return a -> b -> - a.join(b, (value1, value2) -> value1 + value2); -} - -@Bean -public Function, KStream> bar() { - return input -> input.mapValues(value -> value + "From-anotherFooFunc"); -} -``` - -and the function definition could be `curriedFoo|bar`. -Behind the scenes, the binder will create two input bindings for the curried function, and an output binding based on the final function in the definition. -The default input bindings in this case are going to be `curriedFoobar-in-0` and `curriedFoobar-in-1`. -The default output binding for this example becomes `curriedFoobar-out-0`. - -[[special-note-on-using-ktable-as-output-in-function-composition]] -===== Special note on using `KTable` as output in function composition - -Lets say you have the following two functions. - -``` -@Bean -public Function, KTable> foo() { - return KStream::toTable; - }; -} - -@Bean -public Function, KStream> bar() { - return KTable::toStream; -} -``` - -You can compose them as `foo|bar`, but keep in mind that the second function (`bar` in this case) must have a `KTable` as input since the first function (`foo`) has `KTable` as output. - -[[ancillaries-to-the-programming-model]] -== Ancillaries to the programming model - -[[multiple-kafka-streams-processors-within-a-single-application]] -=== Multiple Kafka Streams processors within a single application - -Binder allows to have multiple Kafka Streams processors within a single Spring Cloud Stream application. -You can have an application as below. - -``` -@Bean -public java.util.function.Function, KStream> process() { - ... -} - -@Bean -public java.util.function.Consumer> anotherProcess() { - ... -} - -@Bean -public java.util.function.BiFunction, KTable, KStream> yetAnotherProcess() { - ... -} - -``` - -In this case, the binder will create 3 separate Kafka Streams objects with different application ID's (more on this below). -However, if you have more than one processor in the application, you have to tell Spring Cloud Stream, which functions need to be activated. -Here is how you activate the functions. - -`spring.cloud.function.definition: process;anotherProcess;yetAnotherProcess` - -If you want certain functions to be not activated right away, you can remove that from this list. - -This is also true when you have a single Kafka Streams processor and other types of `Function` beans in the same application that is handled through a different binder (for e.g., a function bean that is based on the regular Kafka Message Channel binder) - -[[kafka-streams-application-id]] -=== Kafka Streams Application ID - -Application id is a mandatory property that you need to provide for a Kafka Streams application. -Spring Cloud Stream Kafka Streams binder allows you to configure this application id in multiple ways. - -If you only have one single processor in the application, then you can set this at the binder level using the following property: - -`spring.cloud.stream.kafka.streams.binder.applicationId`. - -As a convenience, if you only have a single processor, you can also use `spring.application.name` as the property to delegate the application id. - -If you have multiple Kafka Streams processors in the application, then you need to set the application id per processor. -In the case of the functional model, you can attach it to each function as a property. - -For e.g. imagine that you have the following functions. - -``` -@Bean -public java.util.function.Consumer> process() { - ... -} -``` - -and - -``` -@Bean -public java.util.function.Consumer> anotherProcess() { - ... -} -``` - -Then you can set the application id for each, using the following binder level properties. - -`spring.cloud.stream.kafka.streams.binder.functions.process.applicationId` - -and - -`spring.cloud.stream.kafka.streams.binder.functions.anotherProcess.applicationId` - -For function based model also, this approach of setting application id at the binding level will work. -However, setting per function at the binder level as we have seen above is much easier if you are using the functional model. - -For production deployments, it is highly recommended to explicitly specify the application ID through configuration. -This is especially going to be very critical if you are auto scaling your application in which case you need to make sure that you are deploying each instance with the same application ID. - -If the application does not provide an application ID, then in that case the binder will auto generate a static application ID for you. -This is convenient in development scenarios as it avoids the need for explicitly providing the application ID. -The generated application ID in this manner will be static over application restarts. -In the case of functional model, the generated application ID will be the function bean name followed by the literal `applicationID`, for e.g `process-applicationID` if `process` if the function bean name. - -[[summary-of-setting-application-id]] -===== Summary of setting Application ID - -* By default, binder will auto generate the application ID per function methods. -* If you have a single processor, then you can use `spring.kafka.streams.applicationId`, `spring.application.name` or `spring.cloud.stream.kafka.streams.binder.applicationId`. -* If you have multiple processors, then application ID can be set per function using the property - `spring.cloud.stream.kafka.streams.binder.functions..applicationId`. - -[[overriding-the-default-binding-names-generated-by-the-binder-with-the-functional-style]] -=== Overriding the default binding names generated by the binder with the functional style - -By default, the binder uses the strategy discussed above to generate the binding name when using the functional style, i.e. -|-[0..n], for e.g. process-in-0, process-out-0 etc. -If you want to override those binding names, you can do that by specifying the following properties. - -`spring.cloud.stream.function.bindings.`. Default binding name is the original binding name generated by the binder. - -For e.g. lets say, you have this function. - -[source] ----- -@Bean -public BiFunction, KTable, KStream> process() { -... -} ----- - -Binder will generate bindings with names, `process-in-0`, `process-in-1` and `process-out-0`. -Now, if you want to change them to something else completely, maybe more domain specific binding names, then you can do so as below. - -`spring.cloud.stream.function.bindings.process-in-0=users` - -`spring.cloud.stream.function.bindings.process-in-0=regions` - -and - -`spring.cloud.stream.function.bindings.process-out-0=clicks` - -After that, you must set all the binding level properties on these new binding names. - -Please keep in mind that with the functional programming model described above, adhering to the default binding names make sense in most situations. -The only reason you may still want to do this overriding is when you have larger number of configuration properties and you want to map the bindings to something more domain friendly. - -[[setting-up-bootstrap-server-configuration]] -=== Setting up bootstrap server configuration - -When running Kafka Streams applications, you must provide the Kafka broker server information. -If you don't provide this information, the binder expects that you are running the broker at the default `localhost:9092`. -If that is not the case, then you need to override that. There are a couple of ways to do that. - -* Using the boot property - `spring.kafka.bootstrapServers` -* Binder level property - `spring.cloud.stream.kafka.streams.binder.brokers` - -When it comes to the binder level property, it doesn't matter if you use the broker property provided through the regular Kafka binder - `spring.cloud.stream.kafka.binder.brokers`. -Kafka Streams binder will first check if Kafka Streams binder specific broker property is set (`spring.cloud.stream.kafka.streams.binder.brokers`) and if not found, it looks for `spring.cloud.stream.kafka.binder.brokers`. - -[[record-serialization-and-deserialization]] -== Record serialization and deserialization - -Kafka Streams binder allows you to serialize and deserialize records in two ways. -One is the native serialization and deserialization facilities provided by Kafka and the other one is the message conversion capabilities of Spring Cloud Stream framework. -Lets look at some details. - -[[inbound-deserialization]] -=== Inbound deserialization - -Keys are always deserialized using native Serdes. - -For values, by default, deserialization on the inbound is natively performed by Kafka. -Please note that this is a major change on default behavior from previous versions of Kafka Streams binder where the deserialization was done by the framework. - -Kafka Streams binder will try to infer matching `Serde` types by looking at the type signature of `java.util.function.Function|Consumer`. -Here is the order that it matches the Serdes. - -* If the application provides a bean of type `Serde` and if the return type is parameterized with the actual type of the incoming key or value type, then it will use that `Serde` for inbound deserialization. -For e.g. if you have the following in the application, the binder detects that the incoming value type for the `KStream` matches with a type that is parameterized on a `Serde` bean. -It will use that for inbound deserialization. - - -``` -@Bean -public Serde customSerde() { - ... -} - -@Bean -public Function, KStream> process() { -} -``` - -* Next, it looks at the types and see if they are one of the types exposed by Kafka Streams. If so, use them. -Here are the Serde types that the binder will try to match from Kafka Streams. - - Integer, Long, Short, Double, Float, byte[], UUID and String. - -* If none of the Serdes provided by Kafka Streams don't match the types, then it will use JsonSerde provided by Spring Kafka. In this case, the binder assumes that the types are JSON friendly. -This is useful if you have multiple value objects as inputs since the binder will internally infer them to correct Java types. -Before falling back to the `JsonSerde` though, the binder checks at the default `Serde` s set in the Kafka Streams configuration to see if it is a `Serde` that it can match with the incoming KStream's types. - -If none of the above strategies worked, then the applications must provide the `Serde` s through configuration. -This can be configured in two ways - binding or default. - -First the binder will look if a `Serde` is provided at the binding level. -For e.g. if you have the following processor, - -``` -@Bean -public BiFunction, KTable, KStream> process() {...} -``` - -then, you can provide a binding level `Serde` using the following: - -``` -spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.keySerde=CustomKeySerde -spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde - -spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.keySerde=CustomKeySerde -spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde -``` - -NOTE: If you provide `Serde` as abover per input binding, then that will takes higher precedence and the binder will stay away from any `Serde` inference. - -If you want the default key/value Serdes to be used for inbound deserialization, you can do so at the binder level. - -``` -spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde -spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde -``` - -If you don't want the native decoding provided by Kafka, you can rely on the message conversion features that Spring Cloud Stream provides. -Since native decoding is the default, in order to let Spring Cloud Stream deserialize the inbound value object, you need to explicitly disable native decoding. - -For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process-in-0.consumer.nativeDecoding: false` -You need to disable native decoding for all the inputs individually. Otherwise, native decoding will still be applied for those you do not disable. - -By default, Spring Cloud Stream will use `application/json` as the content type and use an appropriate json message converter. -You can use custom message converters by using the following property and an appropriate `MessageConverter` bean. -``` -spring.cloud.stream.bindings.process-in-0.contentType -``` - -[[outbound-serialization]] -=== Outbound serialization - -Outbound serialization pretty much follows the same rules as above for inbound deserialization. -As with the inbound deserialization, one major change from the previous versions of Spring Cloud Stream is that the serialization on the outbound is handled by Kafka natively. -Before 3.0 versions of the binder, this was done by the framework itself. - -Keys on the outbound are always serialized by Kafka using a matching `Serde` that is inferred by the binder. -If it can't infer the type of the key, then that needs to be specified using configuration. - -Value serdes are inferred using the same rules used for inbound deserialization. -First it matches to see if the outbound type is from a provided bean in the application. -If not, it checks to see if it matches with a `Serde` exposed by Kafka such as - `Integer`, `Long`, `Short`, `Double`, `Float`, `byte[]`, `UUID` and `String`. -If that doesnt't work, then it falls back to `JsonSerde` provided by the Spring Kafka project, but first look at the default `Serde` configuration to see if there is a match. -Keep in mind that all these happen transparently to the application. -If none of these work, then the user has to provide the `Serde` to use by configuration. - -Lets say you are using the same `BiFunction` processor as above. Then you can configure outbound key/value Serdes as following. - -``` -spring.cloud.stream.kafka.streams.bindings.process-out-0.producer.keySerde=CustomKeySerde -spring.cloud.stream.kafka.streams.bindings.process-out-0.producer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde -``` - -If Serde inference fails, and no binding level Serdes are provided, then the binder falls back to the `JsonSerde`, but look at the default Serdes for a match. - -Default serdes are configured in the same way as above where it is described under deserialization. - -`spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde` -`spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde` - -If your application uses the branching feature and has multiple output bindings, then these have to be configured per binding. -Once again, if the binder is capable of inferring the `Serde` types, you don't need to do this configuration. - -If you don't want the native encoding provided by Kafka, but want to use the framework provided message conversion, then you need to explicitly disable native encoding since since native encoding is the default. -For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process-out-0.producer.nativeEncoding: false` -You need to disable native encoding for all the output individually in the case of branching. Otherwise, native encoding will still be applied for those you don't disable. - -When conversion is done by Spring Cloud Stream, by default, it will use `application/json` as the content type and use an appropriate json message converter. -You can use custom message converters by using the following property and a corresponding `MessageConverter` bean. -``` -spring.cloud.stream.bindings.process-out-0.contentType -``` - -When native encoding/decoding is disabled, binder will not do any inference as in the case of native Serdes. -Applications need to explicitly provide all the configuration options. -For that reason, it is generally advised to stay with the default options for de/serialization and stick with native de/serialization provided by Kafka Streams when you write Spring Cloud Stream Kafka Streams applications. -The one scenario in which you must use message conversion capabilities provided by the framework is when your upstream producer is using a specific serialization strategy. -In that case, you want to use a matching deserialization strategy as native mechanisms may fail. -When relying on the default `Serde` mechanism, the applications must ensure that the binder has a way forward with correctly map the inbound and outbound with a proper `Serde`, as otherwise things might fail. - -It is worth to mention that the data de/serialization approaches outlined above are only applicable on the edges of your processors, i.e. - inbound and outbound. -Your business logic might still need to call Kafka Streams API's that explicitly need `Serde` objects. -Those are still the responsibility of the application and must be handled accordingly by the developer. - -[[error-handling]] -== Error Handling - -Apache Kafka Streams provides the capability for natively handling exceptions from deserialization errors. -For details on this support, please see https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers[this]. -Out of the box, Apache Kafka Streams provides two kinds of deserialization exception handlers - `LogAndContinueExceptionHandler` and `LogAndFailExceptionHandler`. -As the name indicates, the former will log the error and continue processing the next records and the latter will log the error and fail. `LogAndFailExceptionHandler` is the default deserialization exception handler. - -[[handling-deserialization-exceptions-in-the-binder]] -=== Handling Deserialization Exceptions in the Binder - -Kafka Streams binder allows to specify the deserialization exception handlers above using the following property. - -[source] ----- -spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler: logAndContinue ----- - -or - -[source] ----- -spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler: logAndFail ----- - -In addition to the above two deserialization exception handlers, the binder also provides a third one for sending the erroneous records (poison pills) to a DLQ (dead letter queue) topic. -Here is how you enable this DLQ exception handler. - -[source] ----- -spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler: sendToDlq ----- - -When the above property is set, all the records in deserialization error are automatically sent to the DLQ topic. - -You can set the topic name where the DLQ messages are published as below. - -You can provide an implementation for `DlqDestinationResolver` which is a functional interface. -`DlqDestinationResolver` takes `ConsumerRecord` and the exception as inputs and then allows to specify a topic name as the output. -By gaining access to the Kafka `ConsumerRecord`, the header records can be introspected in the implementation of the `BiFunction`. - -Here is an example of providing an implementation for `DlqDestinationResolver`. - -[source] ----- -@Bean -public DlqDestinationResolver dlqDestinationResolver() { - return (rec, ex) -> { - if (rec.topic().equals("word1")) { - return "topic1-dlq"; - } - else { - return "topic2-dlq"; - } - }; -} ----- - -One important thing to keep in mind when providing an implementation for `DlqDestinationResolver` is that the provisioner in the binder will not auto create topics for the application. -This is because there is no way for the binder to infer the names of all the DLQ topics the implementation might send to. -Therefore, if you provide DLQ names using this strategy, it is the application's responsibility to ensure that those topics are created beforehand. - -If `DlqDestinationResolver` is present in the application as a bean, that takes higher precedence. -If you do not want to follow this approach and rather provide a static DLQ name using configuration, you can set the following property. - -[source] ----- -spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.dlqName: custom-dlq (Change the binding name accordingly) ----- - -If this is set, then the error records are sent to the topic `custom-dlq`. -If the application is not using either of the above strategies, then it will create a DLQ topic with the name `error..`. -For instance, if your binding's destination topic is `inputTopic` and the application ID is `process-applicationId`, then the default DLQ topic is `error.inputTopic.process-applicationId`. -It is always recommended to explicitly create a DLQ topic for each input binding if it is your intention to enable DLQ. - -[[dlq-per-input-consumer-binding]] -=== DLQ per input consumer binding - -The property `spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler` is applicable for the entire application. -This implies that if there are multiple functions in the same application, this property is applied to all of them. -However, if you have multiple processors or multiple input bindings within a single processor, then you can use the finer-grained DLQ control that the binder provides per input consumer binding. - -If you have the following processor, - -``` -@Bean -public BiFunction, KTable, KStream> process() { -... -} -``` - -and you only want to enable DLQ on the first input binding and skipAndContinue on the second binding, then you can do so on the consumer as below. - -`spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.deserializationExceptionHandler: sendToDlq` -`spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.deserializationExceptionHandler: skipAndContinue` - -Setting deserialization exception handlers this way has a higher precedence than setting at the binder level. - -[[dlq-partitioning]] -=== DLQ partitioning - -By default, records are published to the Dead-Letter topic using the same partition as the original record. -This means the Dead-Letter topic must have at least as many partitions as the original record. - -To change this behavior, add a `DlqPartitionFunction` implementation as a `@Bean` to the application context. -Only one such bean can be present. -The function is provided with the consumer group (which is the same as the application ID in most situations), the failed `ConsumerRecord` and the exception. -For example, if you always want to route to partition 0, you might use: - - -[source, java] ----- -@Bean -public DlqPartitionFunction partitionFunction() { - return (group, record, ex) -> 0; -} ----- - -NOTE: If you set a consumer binding's `dlqPartitions` property to 1 (and the binder's `minPartitionCount` is equal to `1`), there is no need to supply a `DlqPartitionFunction`; the framework will always use partition 0. -If you set a consumer binding's `dlqPartitions` property to a value greater than `1` (or the binder's `minPartitionCount` is greater than `1`), you **must** provide a `DlqPartitionFunction` bean, even if the partition count is the same as the original topic's. - - -A couple of things to keep in mind when using the exception handling feature in Kafka Streams binder. - -* The property `spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler` is applicable for the entire application. -This implies that if there are multiple functions in the same application, this property is applied to all of them. -* The exception handling for deserialization works consistently with native deserialization and framework provided message conversion. - -[[handling-production-exceptions-in-the-binder]] -=== Handling Production Exceptions in the Binder - -Unlike the support for deserialization exception handlers as described above, the binder does not provide such first class mechanisms for handling production exceptions. -However, you still can configure production exception handlers using the `StreamsBuilderFactoryBean` customizer which you can find more details about, in a subsequent section below. - -[[runtime-error-handling]] -=== Runtime Error Handling - -When it comes to handling errors from application code, i.e. from the business logic execution, it is usually up to the application to handle that. -Because, the Kafka Streams binder does not have a way to interfere with the application code. -However, to make things a bit easier for the application, the binder provides a convenient `DltAwareProcessor`, using which, you can dictate how you want to handle the application level errors. - -Consider the following code. - -``` -@Bean -public java.util.function.Function, KStream> process() { - return input -> input - .map(...); -} -``` - -If the business code inside your `map` call above throws an exception, it is your responsibility to handle that error. -This is where `DltAwareProcessor` becomes handy. -Let's say that you want to publish the failed record to a DLT, rather than handling it within the application. -Here is how you can do that. - -``` -@Bean -public java.util.function.Function, KStream> process(DltPublishingContext dltSenderContext) { - return input -> input - .process(() -> new DltAwareProcessor<>((k, v) -> { - throw new RuntimeException("error"); - }, "hello-dlt-1", dltPublishingContext)); -} -``` - -The business logic code from the original `map` call now has been moved as part of `KStream#process` method call, which takes a `ProcessorSupplier`. -We, then, pass in the custom `DltAwareProcessor,` which is capable to publishing to a DLT. -The constructor for `DltAwareProcessor` above takes three parameters - a `BiFunction` that takes the key and value of the input record and then the business logic operation as part of the `BiFunction` body, the DLT topic, and finally a `DltPublishingContext`. When the `BiFunction`'s lambda expression throws an exception, the `DltAwareProcessor` will send the input record to a DLT. The `DltPublishingContext` provides `DltAwareProcessor` the necessary publishing infrastructure beans. -The `DltPublishingContext` is autoconfigured by the binder, so that you can inject directly this into the application. - -If you want to provide a custom timestamp on the record that gets published to the DLT, then you can provide an optional fourth constructor argument which is a `Supplier`. -If this value is provided, then this `Supplier` is invoked each time `DltAwareProcessor` publishes to the DLT. - -If you do not want the binder to publish failed records to a DLT, then you can provide your own recoverer as a `BiConsumer`. -Assume a scenario, in which you do not want to send the record to the DLT, but simply log the message and move on. -For this, it is convenient, if we can override the recovery process in `DltAwareProcessor`. -Here is an example of how you do that. - -``` -@Bean -public java.util.function.Function, KStream> process() { - return input -> input - .process(() -> new DltAwareProcessor<>((k, v) -> { - throw new RuntimeException("error"); - }, - (record, exception) -> { - // log the message - })); -} -``` - -In this case, when the record fails, the `DltAwareProcessor`, instead of using its built-in recoverer which publishes to a DLT, uses the user provided recoverer which is a `BiConsumer` that takes the failed record and the exception thrown as arguments. -In this case also, you can provide an optional `Supplier` to dictate the timestamp used in the record passed in to the `BiConsumer` recoverer. - -[[retrying-critical-business-logic]] -== Retrying critical business logic - -There are scenarios in which you might want to retry parts of your business logic that are critical to the application. -There maybe an external call to a relational database or invoking a REST endpoint from the Kafka Streams processor. -These calls can fail for various reasons such as network issues or remote service unavailability. -More often, these failures may self resolve if you can try them again. -By default, Kafka Streams binder creates `RetryTemplate` beans for all the input bindings. - -If the function has the following signature, -``` -@Bean -public java.util.function.Consumer> process() -``` -and with default binding name, the `RetryTemplate` will be registered as `process-in-0-RetryTemplate`. -This is following the convention of binding name (`process-in-0`) followed by the literal `-RetryTemplate`. -In the case of multiple input bindings, there will be a separate `RetryTemplate` bean available per binding. -If there is a custom `RetryTemplate` bean available in the application and provided through `spring.cloud.stream.bindings..consumer.retryTemplateName`, then that takes precedence over any input binding level retry template configuration properties. - -Once the `RetryTemplate` from the binding is injected into the application, it can be used to retry any critical sections of the application. -Here is an example: - -``` -@Bean -public java.util.function.Consumer> process(@Lazy @Qualifier("process-in-0-RetryTemplate") RetryTemplate retryTemplate) { - - return input -> input - .process(() -> new Processor() { - @Override - public void init(ProcessorContext processorContext) { - } - - @Override - public void process(Object o, String s) { - retryTemplate.execute(context -> { - //Critical business logic goes here. - }); - } - - @Override - public void close() { - } - }); -} -``` - -Or you can use a custom `RetryTemplate` as below. - -``` -@EnableAutoConfiguration -public static class CustomRetryTemplateApp { - - @Bean - @StreamRetryTemplate - RetryTemplate fooRetryTemplate() { - RetryTemplate retryTemplate = new RetryTemplate(); - - RetryPolicy retryPolicy = new SimpleRetryPolicy(4); - FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy(); - backOffPolicy.setBackOffPeriod(1); - - retryTemplate.setBackOffPolicy(backOffPolicy); - retryTemplate.setRetryPolicy(retryPolicy); - - return retryTemplate; - } - - @Bean - public java.util.function.Consumer> process() { - - return input -> input - .process(() -> new Processor() { - @Override - public void init(ProcessorContext processorContext) { - } - - @Override - public void process(Object o, String s) { - fooRetryTemplate().execute(context -> { - //Critical business logic goes here. - }); - - } - - @Override - public void close() { - } - }); - } -} -``` - -Note that when retries are exhausted, by default, the last exception will be thrown, causing the processor to terminate. -If you wish to handle the exception and continue processing, you can add a RecoveryCallback to the `execute` method: -Here is an example. -``` -retryTemplate.execute(context -> { - //Critical business logic goes here. - }, context -> { - //Recovery logic goes here. - return null; - )); -``` -Refer to the https://github.com/spring-projects/spring-retry[Spring Retry] project for more information about the RetryTemplate, retry policies, backoff policies and more. - -[[state-store]] -== State Store - -State stores are created automatically by Kafka Streams when the high level DSL is used and appropriate calls are made those trigger a state store. - -If you want to materialize an incoming `KTable` binding as a named state store, then you can do so by using the following strategy. - -Lets say you have the following function. - -[source] ----- -@Bean -public BiFunction, KTable, KStream> process() { - ... -} ----- - -Then by setting the following property, the incoming `KTable` data will be materialized in to the named state store. - -[source] ----- -spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.materializedAs: incoming-store ----- - -You can define custom state stores as beans in your application and those will be detected and added to the Kafka Streams builder by the binder. -Especially when the processor API is used, you need to register a state store manually. -In order to do so, you can create the StateStore as a bean in the application. -Here are examples of defining such beans. - -[source] ----- -@Bean -public StoreBuilder myStore() { - return Stores.keyValueStoreBuilder( - Stores.persistentKeyValueStore("my-store"), Serdes.Long(), - Serdes.Long()); -} - -@Bean -public StoreBuilder otherStore() { - return Stores.windowStoreBuilder( - Stores.persistentWindowStore("other-store", - 1L, 3, 3L, false), Serdes.Long(), - Serdes.Long()); -} ----- - -These state stores can be then accessed by the applications directly. - -During the bootstrap, the above beans will be processed by the binder and passed on to the Streams builder object. - -Accessing the state store: -[source] ----- -Processor() { - - WindowStore state; - - @Override - public void init(ProcessorContext processorContext) { - state = (WindowStore)processorContext.getStateStore("mystate"); - } - ... -} ----- - -This will not work when it comes to registering global state stores. -In order to register a global state store, please see the section below on customizing `StreamsBuilderFactoryBean`. - -[[interactive-queries]] -== Interactive Queries - -Kafka Streams binder API exposes a class called `InteractiveQueryService` to interactively query the state stores. -You can access this as a Spring bean in your application. An easy way to get access to this bean from your application is to `autowire` the bean. - -[source] ----- -@Autowired -private InteractiveQueryService interactiveQueryService; ----- - -Once you gain access to this bean, then you can query for the particular state-store that you are interested. See below. - -[source] ----- -ReadOnlyKeyValueStore keyValueStore = - interactiveQueryService.getQueryableStoreType("my-store", QueryableStoreTypes.keyValueStore()); ----- - -During the startup, the above method call to retrieve the store might fail. -For example, it might still be in the middle of initializing the state store. -In such cases, it will be useful to retry this operation. -Kafka Streams binder provides a simple retry mechanism to accommodate this. - -Following are the two properties that you can use to control this retrying. - -* spring.cloud.stream.kafka.streams.binder.stateStoreRetry.maxAttempts - Default is `1` . -* spring.cloud.stream.kafka.streams.binder.stateStoreRetry.backOffInterval - Default is `1000` milliseconds. - -If there are multiple instances of the kafka streams application running, then before you can query them interactively, you need to identify which application instance hosts the particular key that you are querying. -`InteractiveQueryService` API provides methods for identifying the host information. - -In order for this to work, you must configure the property `application.server` as below: - -[source] ----- -spring.cloud.stream.kafka.streams.binder.configuration.application.server: : ----- - -Here are some code snippets: - -[source] ----- -org.apache.kafka.streams.state.HostInfo hostInfo = interactiveQueryService.getHostInfo("store-name", - key, keySerializer); - -if (interactiveQueryService.getCurrentHostInfo().equals(hostInfo)) { - - //query from the store that is locally available -} -else { - //query from the remote host -} ----- - -For more information on these host finding methods, please see the Javadoc on the methods. -For these methods also, during startup, if the underlying KafkaStreams objects are not ready, they might throw exceptions. -The aforementioned retry properties are applicable for these methods as well. - -[[other-api-methods-available-through-the-interactivequeryservice]] -=== Other API methods available through the InteractiveQueryService - -Use the following API method to retrieve the `KeyQueryMetadata` object associated with the combination of given store and key. - -``` -public KeyQueryMetadata getKeyQueryMetadata(String store, K key, Serializer serializer) -``` - -Use the following API method to retrieve the `KakfaStreams` object associated with the combination of given store and key. - -``` -public KafkaStreams getKafkaStreams(String store, K key, Serializer serializer) -``` - -[[customizing-store-query-parameters]] -=== Customizing Store Query Parameters - -Sometimes it is necessary that you need to fine tune the store query parameters before querying the store through `InteractiveQueryService`. -For this purpose, starting with the `4.0.1` version of the binder, you can provide a bean for `StoreQueryParametersCustomizer` which is a functional interface with a `customize` method that takes a `StoreQueryParameter` as the argument. -Here is its method signature. - -``` -StoreQueryParameters customize(StoreQueryParameters storeQueryParameters); -``` - -Using this approach, applications can further customize the `StoreQueryParameters` such as enabling stale stores. - -When this bean is present in this application, `InteractiveQueryService` will call its `customize` method before querying the state store. - -NOTE: Keep in mind that, there must be a unique bean for `StoreQueryParametersCustomizer` available in the application. - -[[health-indicator]] -== Health Indicator - -The health indicator requires the dependency `spring-boot-starter-actuator`. For maven use: -[source,xml] ----- - - org.springframework.boot - spring-boot-starter-actuator - ----- - -Spring Cloud Stream Kafka Streams Binder provides a health indicator to check the state of the underlying streams threads. -Spring Cloud Stream defines a property `management.health.binders.enabled` to enable the health indicator. See the -https://docs.spring.io/spring-cloud-stream/docs/current/reference/htmlsingle/#_health_indicator[Spring Cloud Stream documentation]. - -The health indicator provides the following details for each stream thread's metadata: - -* Thread name -* Thread state: `CREATED`, `RUNNING`, `PARTITIONS_REVOKED`, `PARTITIONS_ASSIGNED`, `PENDING_SHUTDOWN` or `DEAD` -* Active tasks: task ID and partitions -* Standby tasks: task ID and partitions - -By default, only the global status is visible (`UP` or `DOWN`). To show the details, the property `management.endpoint.health.show-details` must be set to `ALWAYS` or `WHEN_AUTHORIZED`. -For more details about the health information, see the -https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html#production-ready-health[Spring Boot Actuator documentation]. - -NOTE: The status of the health indicator is `UP` if all the Kafka threads registered are in the `RUNNING` state. - -Since there are three individual binders in Kafka Streams binder (`KStream`, `KTable` and `GlobalKTable`), all of them will report the health status. -When enabling `show-details`, some of the information reported may be redundant. - -When there are multiple Kafka Streams processors present in the same application, then the health checks will be reported for all of them and will be categorized by the application ID of Kafka Streams. - -[[accessing-kafka-streams-metrics]] -== Accessing Kafka Streams Metrics - -Spring Cloud Stream Kafka Streams binder provides Kafka Streams metrics which can be exported through a Micrometer `MeterRegistry`. - -For Spring Boot version 2.2.x, the metrics support is provided through a custom Micrometer metrics implementation by the binder. -For Spring Boot version 2.3.x, the Kafka Streams metrics support is provided natively through Micrometer. - -When accessing metrics through the Boot actuator endpoint, make sure to add `metrics` to the property `management.endpoints.web.exposure.include`. -Then you can access `/acutator/metrics` to get a list of all the available metrics, which then can be individually accessed through the same URI (`/actuator/metrics/`). - -[[mixing-high-level-dsl-and-low-level-processor-api]] -== Mixing high level DSL and low level Processor API - -Kafka Streams provides two variants of APIs. -It has a higher level DSL like API where you can chain various operations that maybe familiar to a lot of functional programmers. -Kafka Streams also gives access to a low level Processor API. -The processor API, although very powerful and gives the ability to control things in a much lower level, is imperative in nature. -Kafka Streams binder for Spring Cloud Stream, allows you to use either the high level DSL or mixing both the DSL and the processor API. -Mixing both of these variants give you a lot of options to control various use cases in an application. -Applications can use the `transform` or `process` method API calls to get access to the processor API. - -Here is a look at how one may combine both the DSL and the processor API in a Spring Cloud Stream application using the `process` API. - -``` -@Bean -public Consumer> process() { - return input -> - input.process(() -> new Processor() { - @Override - @SuppressWarnings("unchecked") - public void init(ProcessorContext context) { - this.context = context; - } - - @Override - public void process(Object key, String value) { - //business logic - } - - @Override - public void close() { - - }); -} -``` - -Here is an example using the `transform` API. - -``` -@Bean -public Consumer> process() { - return (input, a) -> - input.transform(() -> new Transformer>() { - @Override - public void init(ProcessorContext context) { - - } - - @Override - public void close() { - - } - - @Override - public KeyValue transform(Object key, String value) { - // business logic - return transformed KStream; - } - }); -} -``` - -The `process` API method call is a terminal operation while the `transform` API is non terminal and gives you a potentially transformed `KStream` using which you can continue further processing using either the DSL or the processor API. - -[[partition-support-on-the-outbound]] -== Partition support on the outbound - -A Kafka Streams processor usually sends the processed output into an outbound Kafka topic. -If the outbound topic is partitioned and the processor needs to send the outgoing data into particular partitions, the applications needs to provide a bean of type `StreamPartitioner`. -See https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/processor/StreamPartitioner.html[StreamPartitioner] for more details. -Let's see some examples. - -This is the same processor we already saw multiple times, - -``` -@Bean -public Function, KStream> process() { - - ... -} -``` - -Here is the output binding destination: - -``` -spring.cloud.stream.bindings.process-out-0.destination: outputTopic -``` - -If the topic `outputTopic` has 4 partitions, if you don't provide a partitioning strategy, Kafka Streams will use default partitioning strategy which may not be the outcome you want depending on the particular use case. -Let's say, you want to send any key that matches to `spring` to partition 0, `cloud` to partition 1, `stream` to partition 2, and everything else to partition 3. -This is what you need to do in the application. - -``` -@Bean -public StreamPartitioner streamPartitioner() { - return (t, k, v, n) -> { - if (k.equals("spring")) { - return 0; - } - else if (k.equals("cloud")) { - return 1; - } - else if (k.equals("stream")) { - return 2; - } - else { - return 3; - } - }; -} -``` - -This is a rudimentary implementation, however, you have access to the key/value of the record, the topic name and the total number of partitions. -Therefore, you can implement complex partitioning strategies if need be. - -You also need to provide this bean name along with the application configuration. - -``` -spring.cloud.stream.kafka.streams.bindings.process-out-0.producer.streamPartitionerBeanName: streamPartitioner -``` - -Each output topic in the application needs to be configured separately like this. - -[[streamsbuilderfactorybean-customizer]] -== StreamsBuilderFactoryBean customizer - -It is often required to customize the `StreamsBuilderFactoryBean` that creates the `KafkaStreams` objects. -Based on the underlying support provided by Spring Kafka, the binder allows you to customize the `StreamsBuilderFactoryBean`. -You can use the `StreamsBuilderFactoryBeanCustomizer` to customize the `StreamsBuilderFactoryBean` itself. -Then, once you get access to the `StreamsBuilderFactoryBean` through this customizer, you can customize the corresponding `KafkaStreams` using `KafkaStreamsCustomzier`. -Both of these customizers are part of the Spring for Apache Kafka project. - -Here is an example of using the `StreamsBuilderFactoryBeanCustomizer`. - -``` -@Bean -public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() { - return sfb -> sfb.setStateListener((newState, oldState) -> { - //Do some action here! - }); -} -``` - -The above is shown as an illustration of the things you can do to customize the `StreamsBuilderFactoryBean`. -You can essentially call any available mutation operations from `StreamsBuilderFactoryBean` to customize it. -This customizer will be invoked by the binder right before the factory bean is started. - -Once you get access to the `StreamsBuilderFactoryBean`, you can also customize the underlying `KafkaStreams` object. -Here is a blueprint for doing so. - -``` -@Bean -public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() { - return factoryBean -> { - factoryBean.setKafkaStreamsCustomizer(new KafkaStreamsCustomizer() { - @Override - public void customize(KafkaStreams kafkaStreams) { - kafkaStreams.setUncaughtExceptionHandler((t, e) -> { - - }); - } - }); - }; -} -``` - -`KafkaStreamsCustomizer` will be called by the `StreamsBuilderFactoryBeabn` right before the underlying `KafkaStreams` gets started. - -There can only be one `StreamsBuilderFactoryBeanCustomizer` in the entire application. -Then how do we account for multiple Kafka Streams processors as each of them are backed up by individual `StreamsBuilderFactoryBean` objects? -In that case, if the customization needs to be different for those processors, then the application needs to apply some filter based on the application ID. - -For e.g, - -``` -@Bean -public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() { - - return factoryBean -> { - if (factoryBean.getStreamsConfiguration().getProperty(StreamsConfig.APPLICATION_ID_CONFIG) - .equals("processor1-application-id")) { - factoryBean.setKafkaStreamsCustomizer(new KafkaStreamsCustomizer() { - @Override - public void customize(KafkaStreams kafkaStreams) { - kafkaStreams.setUncaughtExceptionHandler((t, e) -> { - - }); - } - }); - } - }; -``` - -[[using-customizer-to-register-a-global-state-store]] -=== Using Customizer to register a global state store - -As mentioned above, the binder does not provide a first class way to register global state stores as a feature. -For that, you need to use the customizer. -Here is how that can be done. - -``` -@Bean -public StreamsBuilderFactoryBeanCustomizer customizer() { - return fb -> { - try { - final StreamsBuilder streamsBuilder = fb.getObject(); - streamsBuilder.addGlobalStore(...); - } - catch (Exception e) { - - } - }; -} -``` - -Again, if you have multiple processors, you want to attach the global state store to the right `StreamsBuilder` by filtering out the other `StreamsBuilderFactoryBean` objects using the application id as outlined above. - -[[using-customizer-to-register-a-production-exception-handler]] -=== Using customizer to register a production exception handler - -In the error handling section, we indicated that the binder does not provide a first class way to deal with production exceptions. -Though that is the case, you can still use the `StreamsBuilderFacotryBean` customizer to register production exception handlers. See below. - -``` -@Bean -public StreamsBuilderFactoryBeanCustomizer customizer() { - return fb -> { - fb.getStreamsConfiguration().put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG, - CustomProductionExceptionHandler.class); - }; -} -``` - -Once again, if you have multiple processors, you may want to set it appropriately against the correct `StreamsBuilderFactoryBean`. -You may also add such production exception handlers using the configuration property (See below for more on that), but this is an option if you choose to go with a programmatic approach. - -[[timestamp-extractor]] -== Timestamp extractor - -Kafka Streams allows you to control the processing of the consumer records based on various notions of timestamp. -By default, Kafka Streams extracts the timestamp metadata embedded in the consumer record. -You can change this default behavior by providing a different `TimestampExtractor` implementation per input binding. -Here are some details on how that can be done. - -``` -@Bean -public Function, - Function, - Function, KStream>>> process() { - return orderStream -> - customers -> - products -> orderStream; -} - -@Bean -public TimestampExtractor timestampExtractor() { - return new WallclockTimestampExtractor(); -} -``` - -Then you set the above `TimestampExtractor` bean name per consumer binding. - -``` -spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.timestampExtractorBeanName=timestampExtractor -spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.timestampExtractorBeanName=timestampExtractor -spring.cloud.stream.kafka.streams.bindings.process-in-2.consumer.timestampExtractorBeanName=timestampExtractor" -``` - -If you skip an input consumer binding for setting a custom timestamp extractor, that consumer will use the default settings. - -[[multi-binders-with-kafka-streams-based-binders-and-regular-kafka-binder]] -== Multi binders with Kafka Streams based binders and regular Kafka Binder - -You can have an application where you have both a function/consumer/supplier that is based on the regular Kafka binder and a Kafka Streams based processor. -However, you cannot mix both of them within a single function or consumer. - -Here is an example, where you have both binder based components within the same application. - -``` -@Bean -public Function process() { - return s -> s; -} - -@Bean -public Function, KStream> kstreamProcess() { - - return input -> input; -} - -``` - -This is the relevant parts from the configuration: - -``` -spring.cloud.function.definition=process;kstreamProcess -spring.cloud.stream.bindings.process-in-0.destination=foo -spring.cloud.stream.bindings.process-out-0.destination=bar -spring.cloud.stream.bindings.kstreamProcess-in-0.destination=bar -spring.cloud.stream.bindings.kstreamProcess-out-0.destination=foobar -``` - -Things become a bit more complex if you have the same application as above, but is dealing with two different Kafka clusters, for e.g. the regular `process` is acting upon both Kafka cluster 1 and cluster 2 (receiving data from cluster-1 and sending to cluster-2) and the Kafka Streams processor is acting upon Kafka cluster 2. -Then you have to use the https://cloud.spring.io/spring-cloud-stream/reference/html/spring-cloud-stream.html#multiple-binders[multi binder] facilities provided by Spring Cloud Stream. - -Here is how your configuration may change in that scenario. - -``` -# multi binder configuration -spring.cloud.stream.binders.kafka1.type: kafka -spring.cloud.stream.binders.kafka1.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-1} #Replace kafkaCluster-1 with the approprate IP of the cluster -spring.cloud.stream.binders.kafka2.type: kafka -spring.cloud.stream.binders.kafka2.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} #Replace kafkaCluster-2 with the approprate IP of the cluster -spring.cloud.stream.binders.kafka3.type: kstream -spring.cloud.stream.binders.kafka3.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} #Replace kafkaCluster-2 with the approprate IP of the cluster - -spring.cloud.function.definition=process;kstreamProcess - -# From cluster 1 to cluster 2 with regular process function -spring.cloud.stream.bindings.process-in-0.destination=foo -spring.cloud.stream.bindings.process-in-0.binder=kafka1 # source from cluster 1 -spring.cloud.stream.bindings.process-out-0.destination=bar -spring.cloud.stream.bindings.process-out-0.binder=kafka2 # send to cluster 2 - -# Kafka Streams processor on cluster 2 -spring.cloud.stream.bindings.kstreamProcess-in-0.destination=bar -spring.cloud.stream.bindings.kstreamProcess-in-0.binder=kafka3 -spring.cloud.stream.bindings.kstreamProcess-out-0.destination=foobar -spring.cloud.stream.bindings.kstreamProcess-out-0.binder=kafka3 -``` - -Pay attention to the above configuration. -We have two kinds of binders, but 3 binders all in all, first one is the regular Kafka binder based on cluster 1 (`kafka1`), then another Kafka binder based on cluster 2 (`kafka2`) and finally the `kstream` one (`kafka3`). -The first processor in the application receives data from `kafka1` and publishes to `kafka2` where both binders are based on regular Kafka binder but differnt clusters. -The second processor, which is a Kafka Streams processor consumes data from `kafka3` which is the same cluster as `kafka2`, but a different binder type. - -Since there are three different binder types available in the Kafka Streams family of binders - `kstream`, `ktable` and `globalktable` - if your application has multiple bindings based on any of these binders, that needs to be explicitly provided as the binder type. - -For e.g if you have a processor as below, - -``` -@Bean -public Function, - Function, - Function, KStream>>> enrichOrder() { - - ... -} -``` - -then, this has to be configured in a multi binder scenario as the following. -Please note that this is only needed if you have a true multi-binder scenario where there are multiple processors dealing with multiple clusters within a single application. -In that case, the binders need to be explicitly provided with the bindings to distinguish from other processor's binder types and clusters. - -``` -spring.cloud.stream.binders.kafka1.type: kstream -spring.cloud.stream.binders.kafka1.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} -spring.cloud.stream.binders.kafka2.type: ktable -spring.cloud.stream.binders.kafka2.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} -spring.cloud.stream.binders.kafka3.type: globalktable -spring.cloud.stream.binders.kafka3.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} - -spring.cloud.stream.bindings.enrichOrder-in-0.binder=kafka1 #kstream -spring.cloud.stream.bindings.enrichOrder-in-1.binder=kafka2 #ktablr -spring.cloud.stream.bindings.enrichOrder-in-2.binder=kafka3 #globalktable -spring.cloud.stream.bindings.enrichOrder-out-0.binder=kafka1 #kstream - -# rest of the configuration is omitted. - -``` - -[[state-cleanup]] -== State Cleanup - -By default, no local state is cleaned up when the binding is stopped. -This is the same behavior effective from Spring Kafka version 2.7. -See https://docs.spring.io/spring-kafka/reference/html/#streams-config[Spring Kafka documentation] for more details. -To modify this behavior simply add a single `CleanupConfig` `@Bean` (configured to clean up on start, stop, or neither) to the application context; the bean will be detected and wired into the factory bean. - -[[kafka-streams-topology-visualization]] -== Kafka Streams topology visualization - -Kafka Streams binder provides the following actuator endpoints for retrieving the topology description using which you can visualize the topology using external tools. - -`/actuator/kafkastreamstopology` - -`/actuator/kafkastreamstopology/` - -You need to include the actuator and web dependencies from Spring Boot to access these endpoints. -Further, you also need to add `kafkastreamstopology` to `management.endpoints.web.exposure.include` property. -By default, the `kafkastreamstopology` endpoint is disabled. - -[[event-type-based-routing-in-kafka-streams-applications]] -== Event type based routing in Kafka Streams applications - -Routing functions available in regular message channel based binders are not supported in Kafka Streams binder. -However, Kafka Streams binder still provides routing capabilities through the event type record header on the inbound records. - -To enable routing based on event types, the application must provide the following property. - -`spring.cloud.stream.kafka.streams.bindings..consumer.eventTypes`. - -This can be a comma separated value. - -For example, lets assume we have this function: - -``` -@Bean -public Function, KStream> process() { - return input -> input; -} -``` - -Let us also assume that we only want the business logic in this function to be executed, if the incoming record has event types as `foo` or `bar`. -That can be expressed as below using the `eventTypes` property on the binding. - -`spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.eventTypes=foo,bar` - -Now, when the application runs, the binder checks each incoming records for the header `event_type` and see if it has value set as `foo` or `bar`. -If it does not find either of them, then the function execution will be skipped. - -By default, the binder expects the record header key to be `event_type`, but that can be changed per binding. -For instance, if we want to change the header key on this binding to `my_event` instead of the default, that can be changed as below. - -`spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.eventTypeHeaderKey=my_event`. - -When using the event routing feature in Kafkfa Streams binder, it uses the byte array `Serde` to deserialze all incoming records. -If the record headers match the event type, then only it uses the actual `Serde` to do a proper deserialization using either the configured or the inferred `Serde`. -This introduces issues if you set a deserialization exception handler on the binding as the expected deserialization only happens down the stack causing unexpected errors. -In order to address this issue, you can set the following property on the binding to force the binder to use the configured or inferred `Serde` instead of byte array `Serde`. - -`spring.cloud.stream.kafka.streams.bindings..consumer.useConfiguredSerdeWhenRoutingEvents` - -This way, the application can detect deserialization issues right away when using the event routing feature and can take appropriate handling decisions. - -[[binding-visualization-and-control-in-kafka-streams-binder]] -== Binding visualization and control in Kafka Streams binder - -Starting with version 3.1.2, Kafka Streams binder supports binding visualization and control. -The only two lifecycle phases supported are `STOPPED` and `STARTED`. -The lifecycle phases `PAUSED` and `RESUMED` are not available in Kafka Streams binder. - -In order to activate binding visualization and control, the application needs to include the following two dependencies. - -``` - - org.springframework.boot - spring-boot-starter-actuator - - - org.springframework.boot - spring-boot-starter-web - -``` - -If you prefer using webflux, you can then include `spring-boot-starter-webflux` instead of the standard web dependency. - -In addition, you also need to set the following property: - -``` -management.endpoints.web.exposure.include=bindings -``` - -To illustrate this feature further, let us use the following application as a guide: - -``` -@SpringBootApplication -public class KafkaStreamsApplication { - - public static void main(String[] args) { - SpringApplication.run(KafkaStreamsApplication.class, args); - } - - @Bean - public Consumer> consumer() { - return s -> s.foreach((key, value) -> System.out.println(value)); - } - - @Bean - public Function, KStream> function() { - return ks -> ks; - } - -} -``` - -As we can see, the application has two Kafka Streams functions - one, a consumer and another a function. -The consumer binding is named by default as `consumer-in-0`. -Similarly, for the function, the input binding is `function-in-0` and the output binding is `function-out-0`. - -Once the application is started, we can find details about the bindings using the following bindings endpoint. - -``` - curl http://localhost:8080/actuator/bindings | jq . -[ - { - "bindingName": "consumer-in-0", - "name": "consumer-in-0", - "group": "consumer-applicationId", - "pausable": false, - "state": "running", - "paused": false, - "input": true, - "extendedInfo": {} - }, - { - "bindingName": "function-in-0", - "name": "function-in-0", - "group": "function-applicationId", - "pausable": false, - "state": "running", - "paused": false, - "input": true, - "extendedInfo": {} - }, - { - "bindingName": "function-out-0", - "name": "function-out-0", - "group": "function-applicationId", - "pausable": false, - "state": "running", - "paused": false, - "input": false, - "extendedInfo": {} - } -] -``` - -The details about all three bindings can be found above. - -Let us now stop the consumer-in-0 binding. - -``` -curl -d '{"state":"STOPPED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/consumer-in-0 -``` - -At this point, no records will be received through this binding. - -Start the binding again. - -``` -curl -d '{"state":"STARTED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/consumer-in-0 -``` - -When there are multiple bindings present on a single function, invoking these operations on any of those bindings will work. -This is because all the bindings on a single function are backed by the same `StreamsBuilderFactoryBean`. -Therefore, for the function above, either `function-in-0` or `function-out-0` will work. - -[[manually-starting-kafka-streams-processors]] -== Manually starting Kafka Streams processors - -Spring Cloud Stream Kafka Streams binder offers an abstraction called `StreamsBuilderFactoryManager` on top of the `StreamsBuilderFactoryBean` from Spring for Apache Kafka. -This manager API is used for controlling the multiple `StreamsBuilderFactoryBean` per processor in a binder based application. -Therefore, when using the binder, if you manually want to control the auto starting of the various `StreamsBuilderFactoryBean` objects in the application, you need to use `StreamsBuilderFactoryManager`. -You can use the property `spring.kafka.streams.auto-startup` and set this to `false` in order to turn off auto starting of the processors. -Then, in the application, you can use something as below to start the processors using `StreamsBuilderFactoryManager`. - -``` -@Bean -public ApplicationRunner runner(StreamsBuilderFactoryManager sbfm) { - return args -> { - sbfm.start(); - }; -} -``` - -This feature is handy, when you want your application to start in the main thread and let Kafka Streams processors start separately. -For example, when you have a large state store that needs to be restored, if the processors are started normally as is the default case, this may block your application to start. -If you are using some sort of liveness probe mechanism (for example on Kubernetes), it may think that the application is down and attempt a restart. -In order to correct this, you can set `spring.kafka.streams.auto-startup` to `false` and follow the approach above. - -Keep in mind that, when using the Spring Cloud Stream binder, you are not directly dealing with `StreamsBuilderFactoryBean` from Spring for Apache Kafka, rather `StreamsBuilderFactoryManager`, as the `StreamsBuilderFactoryBean` objects are internally managed by the binder. - -[[manually-starting-kafka-streams-processors-selectively]] -== Manually starting Kafka Streams processors selectively - -While the approach laid out above will unconditionally apply auto start `false` to all the Kafka Streams processors in the application through `StreamsBuilderFactoryManager`, it is often desirable that only individually selected Kafka Streams processors are not auto started. -For instance, let us assume that you have three different functions (processors) in your application and for one of the processors, you do not want to start it as part of the application startup. -Here is an example of such a situation. - -``` - -@Bean -public Function, KStream> process1() { - -} - -@Bean -public Consumer> process2() { - -} - -@Bean -public BiFunction, KTable, KStream> process3() { - -} - -``` - -In this scenario above, if you set `spring.kafka.streams.auto-startup` to `false`, then none of the processors will auto start during the application startup. -In that case, you have to programmatically start them as described above by calling `start()` on the underlying `StreamsBuilderFactoryManager`. -However, if we have a use case to selectively disable only one processor, then you have to set `auto-startup` on the individual binding for that processor. -Let us assume that we don't want our `process3` function to auto start. -This is a `BiFunction` with two input bindings - `process3-in-0` and `process3-in-1`. -In order to avoid auto start for this processor, you can pick any of these input bindings and set `auto-startup` on them. -It does not matter which binding you pick; if you wish, you can set `auto-startup` to `false` on both of them, but one will be sufficient. -Because they share the same factory bean, you don't have to set autoStartup to false on both bindings, but it probably makes sense to do so, for clarity. - -Here is the Spring Cloud Stream property that you can use to disable auto startup for this processor. - -``` -spring.cloud.stream.bindings.process3-in-0.consumer.auto-startup: false -``` - -or - -``` -spring.cloud.stream.bindings.process3-in-1.consumer.auto-startup: false -``` - -Then, you can manually start the processor either using the REST endpoint or using the `BindingsEndpoint` API as shown below. -For this, you need to ensure that you have the Spring Boot actuator dependency on the classpath. - -``` -curl -d '{"state":"STARTED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/process3-in-0 -``` - -or - -``` -@Autowired -BindingsEndpoint endpoint; - -@Bean -public ApplicationRunner runner() { - return args -> { - endpoint.changeState("process3-in-0", State.STARTED); - }; -} -``` - -See https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/spring-cloud-stream.html#binding_visualization_control[this section] from the reference docs for more details on this mechanism. - -NOTE: When controlling the bindings by disabling `auto-startup` as described in this section, please note that this is only available for consumer bindings. -In other words, if you use the producer binding, `process3-out-0`, that does not have any effect in terms of disabling the auto starting of the processor, although this producer binding uses the same `StreamsBuilderFactoryBean` as the consumer bindings. - -[[tracing-using-spring-cloud-sleuth]] -== Tracing using Spring Cloud Sleuth - -When Spring Cloud Sleuth is on the classpath of a Spring Cloud Stream Kafka Streams binder based application, both its consumer and producer are automatically instrumented with tracing information. -However, in order to trace any application specific operations, those need to be explicitly instrumented by the user code. -This can be done by injecting the `KafkaStreamsTracing` bean from Spring Cloud Sleuth in the application and then invoke various Kafka Streams operations through this injected bean. -Here are some examples of using it. - -``` -@Bean -public BiFunction, KTable, KStream> clicks(KafkaStreamsTracing kafkaStreamsTracing) { - return (userClicksStream, userRegionsTable) -> (userClicksStream - .transformValues(kafkaStreamsTracing.peek("span-1", (key, value) -> LOG.info("key/value: " + key + "/" + value))) - .leftJoin(userRegionsTable, (clicks, region) -> new RegionWithClicks(region == null ? - "UNKNOWN" : region, clicks), - Joined.with(Serdes.String(), Serdes.Long(), null)) - .transform(kafkaStreamsTracing.map("span-2", (key, value) -> { - LOG.info("Click Info: " + value.getRegion() + "/" + value.getClicks()); - return new KeyValue<>(value.getRegion(), - value.getClicks()); - })) - .groupByKey(Grouped.with(Serdes.String(), Serdes.Long())) - .reduce(Long::sum, Materialized.as(CLICK_UPDATES)) - .toStream()); -} -``` - -In the example above, there are two places where it adds explicit tracing instrumentation. -First, we are logging the key/value information from the incoming `KStream`. -When this information is logged, the associated span and trace IDs get logged as well so that a monitoring system can track them and correlate with the same span id. -Second, when we call a `map` operation, instead of calling it directly on the `KStream` class, we wrap it inside a `transform` operation and then call `map` from `KafkaStreamsTracing`. -In this case also, the logged message will contain the span ID and trace ID. - -Here is another example, where we use the low-level transformer API for accessing the various Kafka Streams headers. -When spring-cloud-sleuth is on the classpath, all the tracing headers can also be accessed like this. - -``` -@Bean -public Function, KStream> process(KafkaStreamsTracing kafkaStreamsTracing) { - return input -> input.transform(kafkaStreamsTracing.transformer( - "transformer-1", - () -> new Transformer>() { - ProcessorContext context; - - @Override - public void init(ProcessorContext context) { - this.context = context; - } - - @Override - public KeyValue transform(String key, String value) { - LOG.info("Headers: " + this.context.headers()); - LOG.info("K/V:" + key + "/" + value); - // More transformations, business logic execution, etc. go here. - return KeyValue.pair(key, value); - } - - @Override - public void close() { - } - })); -} -``` - -[[configuration-options]] -== Configuration Options - -This section contains the configuration options used by the Kafka Streams binder. - -For common configuration options and properties pertaining to binder, refer to the <>. - -[[kafka-streams-binder-properties]] -=== Kafka Streams Binder Properties - -The following properties are available at the binder level and must be prefixed with `spring.cloud.stream.kafka.streams.binder.` -Any Kafka binder provided properties re-used in Kafka Streams binder must be prefixed with `spring.cloud.stream.kafka.streams.binder` instead of `spring.cloud.stream.kafka.binder`. -The only exception to this rule is when defining the Kafka bootstrap server property in which case either prefix works. - -configuration:: -Map with a key/value pair containing properties pertaining to Apache Kafka Streams API. -This property must be prefixed with `spring.cloud.stream.kafka.streams.binder.`. -Following are some examples of using this property. - -[source] ----- -spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde -spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde -spring.cloud.stream.kafka.streams.binder.configuration.commit.interval.ms=1000 ----- - -For more information about all the properties that may go into streams configuration, see `StreamsConfig` JavaDocs in Apache Kafka Streams docs. -All configuration that you can set from `StreamsConfig` can be set through this. -When using this property, it is applicable against the entire application since this is a binder level property. -If you have more than processors in the application, all of them will acquire these properties. -In the case of properties like `application.id`, this will become problematic and therefore you have to carefully examine how the properties from `StreamsConfig` are mapped using this binder level `configuration` property. - -functions..applicationId:: -Applicable only for functional style processors. -This can be used for setting application ID per function in the application. -In the case of multiple functions, this is a handy way to set the application ID. - -functions..configuration:: -Applicable only for functional style processors. -Map with a key/value pair containing properties pertaining to Apache Kafka Streams API. -This is similar to the binder level `configuration` property describe above, but this level of `configuration` property is restricted only against the named function. -When you have multiple processors and you want to restrict access to the configuration based on particular functions, you might want to use this. -All `StreamsConfig` properties can be used here. - -brokers:: -Broker URL -+ -Default: `localhost` -zkNodes:: -Zookeeper URL -+ -Default: `localhost` - -deserializationExceptionHandler:: -Deserialization error handler type. -This handler is applied at the binder level and thus applied against all input binding in the application. -There is a way to control it in a more fine-grained way at the consumer binding level. -Possible values are - `logAndContinue`, `logAndFail`, `skipAndContinue` or `sendToDlq` -+ -Default: `logAndFail` - -applicationId:: -Convenient way to set the application.id for the Kafka Streams application globally at the binder level. -If the application contains multiple functions, then the application id should be set differently. -See above where setting the application id is discussed in detail. -+ -Default: application will generate a static application ID. See the application ID section for more details. - -stateStoreRetry.maxAttempts:: -Max attempts for trying to connect to a state store. -+ -Default: 1 - -stateStoreRetry.backoffPeriod:: -Backoff period when trying to connect to a state store on a retry. -+ -Default: 1000 ms - -consumerProperties:: -Arbitrary consumer properties at the binder level. - -producerProperties:: -Arbitrary producer properties at the binder level. - -includeStoppedProcessorsForHealthCheck:: -When bindings for processors are stopped through actuator, then this processor will not participate in the health check by default. -Set this property to `true` to enable health check for all processors including the ones that are currently stopped through bindings actuator endpoint. -+ -Default: false - -[[kafka-streams-producer-properties]] -=== Kafka Streams Producer Properties - -The following properties are _only_ available for Kafka Streams producers and must be prefixed with `spring.cloud.stream.kafka.streams.bindings..producer.` -For convenience, if there are multiple output bindings and they all require a common value, that can be configured by using the prefix `spring.cloud.stream.kafka.streams.default.producer.`. - -keySerde:: -key serde to use -+ -Default: See the above discussion on message de/serialization - -valueSerde:: -value serde to use -+ -Default: See the above discussion on message de/serialization - -useNativeEncoding:: -flag to enable/disable native encoding -+ -Default: `true`. - -streamPartitionerBeanName:: -Custom outbound partitioner bean name to be used at the consumer. -Applications can provide custom `StreamPartitioner` as a Spring bean and the name of this bean can be provided to the producer to use instead of the default one. -+ -Default: See the discussion above on outbound partition support. - -producedAs:: -Custom name for the sink component to which the processor is producing to. -+ -Deafult: `none` (generated by Kafka Streams) - -[[kafka-streams-consumer-properties]] -=== Kafka Streams Consumer Properties - -The following properties are available for Kafka Streams consumers and must be prefixed with `spring.cloud.stream.kafka.streams.bindings..consumer.` -For convenience, if there are multiple input bindings and they all require a common value, that can be configured by using the prefix `spring.cloud.stream.kafka.streams.default.consumer.`. - -applicationId:: -Setting application.id per input binding. -+ -Default: See above. - -keySerde:: -key serde to use -+ -Default: See the above discussion on message de/serialization - -valueSerde:: -value serde to use -+ -Default: See the above discussion on message de/serialization - -materializedAs:: -state store to materialize when using incoming KTable types -+ -Default: `none`. - -useNativeDecoding:: -flag to enable/disable native decoding -+ -Default: `true`. - -dlqName:: -DLQ topic name. -+ -Default: See above on the discussion of error handling and DLQ. - -startOffset:: -Offset to start from if there is no committed offset to consume from. -This is mostly used when the consumer is consuming from a topic for the first time. -Kafka Streams uses `earliest` as the default strategy and the binder uses the same default. -This can be overridden to `latest` using this property. -+ -Default: `earliest`. - -Note: Using `resetOffsets` on the consumer does not have any effect on Kafka Streams binder. -Unlike the message channel based binder, Kafka Streams binder does not seek to beginning or end on demand. - -deserializationExceptionHandler:: -Deserialization error handler type. -This handler is applied per consumer binding as opposed to the binder level property described before. -Possible values are - `logAndContinue`, `logAndFail`, `skipAndContinue` or `sendToDlq` -+ -Default: `logAndFail` - -timestampExtractorBeanName:: -Specific time stamp extractor bean name to be used at the consumer. -Applications can provide `TimestampExtractor` as a Spring bean and the name of this bean can be provided to the consumer to use instead of the default one. -+ -Default: See the discussion above on timestamp extractors. - -eventTypes:: -Comma separated list of supported event types for this binding. -+ -Default: `none` - -eventTypeHeaderKey:: -Event type header key on each incoming records through this binding. -+ -Default: `event_type` - -consumedAs:: -Custom name for the source component from which the processor is consuming from. -+ -Deafult: `none` (generated by Kafka Streams) - -[[special-note-on-concurrency]] -=== Special note on concurrency - -In Kafka Streams, you can control of the number of threads a processor can create using the `num.stream.threads` property. -This, you can do using the various `configuration` options described above under binder, functions, producer or consumer level. -You can also use the `concurrency` property that core Spring Cloud Stream provides for this purpose. -When using this, you need to use it on the consumer. -When you have more than one input binding, set this on the first input binding. -For e.g. when setting `spring.cloud.stream.bindings.process-in-0.consumer.concurrency`, it will be translated as `num.stream.threads` by the binder. -If you have multiple processors and one processor defines binding level concurrency, but not the others, those ones with no binding level concurrency will default back to the binder wide property specified through -`spring.cloud.stream.kafka.streams.binder.configuration.num.stream.threads`. -If this binder configuration is not available, then the application will use the default set by Kafka Streams. diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview.adoc index c0a66e48e..2fbbd46dc 100644 --- a/docs/modules/ROOT/pages/kafka/kafka_overview.adoc +++ b/docs/modules/ROOT/pages/kafka/kafka_overview.adoc @@ -8,1079 +8,3 @@ In addition, this guide explains the Kafka Streams binding capabilities of Sprin [[apache-kafka-binder]] = Apache Kafka Binder -[[usage]] -== Usage - -To use Apache Kafka binder, you need to add `spring-cloud-stream-binder-kafka` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-kafka - ----- - -Alternatively, you can also use the Spring Cloud Stream Kafka Starter, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-starter-stream-kafka - ----- - -[[overview]] -== Overview - -The following image shows a simplified diagram of how the Apache Kafka binder operates: - -.Kafka Binder -image::{github-raw}/docs/src/main/asciidoc/images/kafka-binder.png[width=300,scaledwidth="50%"] - -The Apache Kafka Binder implementation maps each destination to an Apache Kafka topic. -The consumer group maps directly to the same Apache Kafka concept. -Partitioning also maps directly to Apache Kafka partitions as well. - -The binder currently uses the Apache Kafka `kafka-clients` version `3.1.0`. -This client can communicate with older brokers (see the Kafka documentation), but certain features may not be available. -For example, with versions earlier than 0.11.x.x, native headers are not supported. -Also, 0.11.x.x does not support the `autoAddPartitions` property. - -[[configuration-options]] -== Configuration Options - -This section contains the configuration options used by the Apache Kafka binder. - -For common configuration options and properties pertaining to the binder, see the https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/current/reference/html/spring-cloud-stream.html#binding-properties[binding properties] in core documentation. - -[[kafka-binder-properties]] -=== Kafka Binder Properties - -spring.cloud.stream.kafka.binder.brokers:: -A list of brokers to which the Kafka binder connects. -+ -Default: `localhost`. -spring.cloud.stream.kafka.binder.defaultBrokerPort:: -`brokers` allows hosts specified with or without port information (for example, `host1,host2:port2`). -This sets the default port when no port is configured in the broker list. -+ -Default: `9092`. -spring.cloud.stream.kafka.binder.configuration:: -Key/Value map of client properties (both producers and consumer) passed to all clients created by the binder. -Due to the fact that these properties are used by both producers and consumers, usage should be restricted to common properties -- for example, security settings. -Unknown Kafka producer or consumer properties provided through this configuration are filtered out and not allowed to propagate. -Properties here supersede any properties set in boot. -+ -Default: Empty map. -spring.cloud.stream.kafka.binder.consumerProperties:: -Key/Value map of arbitrary Kafka client consumer properties. -In addition to support known Kafka consumer properties, unknown consumer properties are allowed here as well. -Properties here supersede any properties set in boot and in the `configuration` property above. -+ -Default: Empty map. -spring.cloud.stream.kafka.binder.headers:: -The list of custom headers that are transported by the binder. -Only required when communicating with older applications (<= 1.3.x) with a `kafka-clients` version < 0.11.0.0. Newer versions support headers natively. -+ -Default: empty. -spring.cloud.stream.kafka.binder.healthTimeout:: -The time to wait to get partition information, in seconds. -Health reports as down if this timer expires. -+ -Default: 10. -spring.cloud.stream.kafka.binder.requiredAcks:: -The number of required acks on the broker. -See the Kafka documentation for the producer `acks` property. -+ -Default: `1`. -spring.cloud.stream.kafka.binder.minPartitionCount:: -Effective only if `autoCreateTopics` or `autoAddPartitions` is set. -The global minimum number of partitions that the binder configures on topics on which it produces or consumes data. -It can be superseded by the `partitionCount` setting of the producer or by the value of `instanceCount * concurrency` settings of the producer (if either is larger). -+ -Default: `1`. -spring.cloud.stream.kafka.binder.producerProperties:: -Key/Value map of arbitrary Kafka client producer properties. -In addition to support known Kafka producer properties, unknown producer properties are allowed here as well. -Properties here supersede any properties set in boot and in the `configuration` property above. -+ -Default: Empty map. -spring.cloud.stream.kafka.binder.replicationFactor:: -The replication factor of auto-created topics if `autoCreateTopics` is active. -Can be overridden on each binding. -+ -NOTE: If you are using Kafka broker versions prior to 2.4, then this value should be set to at least `1`. -Starting with version 3.0.8, the binder uses `-1` as the default value, which indicates that the broker 'default.replication.factor' property will be used to determine the number of replicas. -Check with your Kafka broker admins to see if there is a policy in place that requires a minimum replication factor, if that's the case then, typically, the `default.replication.factor` will match that value and `-1` should be used, unless you need a replication factor greater than the minimum. -+ -Default: `-1`. -spring.cloud.stream.kafka.binder.autoCreateTopics:: -If set to `true`, the binder creates new topics automatically. -If set to `false`, the binder relies on the topics being already configured. -In the latter case, if the topics do not exist, the binder fails to start. -+ -NOTE: This setting is independent of the `auto.create.topics.enable` setting of the broker and does not influence it. -If the server is set to auto-create topics, they may be created as part of the metadata retrieval request, with default broker settings. -+ -Default: `true`. -spring.cloud.stream.kafka.binder.autoAddPartitions:: -If set to `true`, the binder creates new partitions if required. -If set to `false`, the binder relies on the partition size of the topic being already configured. -If the partition count of the target topic is smaller than the expected value, the binder fails to start. -+ -Default: `false`. -spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix:: -Enables transactions in the binder. See `transaction.id` in the Kafka documentation and https://docs.spring.io/spring-kafka/reference/html/#transactions[Transactions] in the `spring-kafka` documentation. -When transactions are enabled, individual `producer` properties are ignored and all producers use the `spring.cloud.stream.kafka.binder.transaction.producer.*` properties. -+ -Default `null` (no transactions) -spring.cloud.stream.kafka.binder.transaction.producer.*:: -Global producer properties for producers in a transactional binder. -See `spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix` and <> and the general producer properties supported by all binders. -+ -Default: See individual producer properties. - -spring.cloud.stream.kafka.binder.headerMapperBeanName:: -The bean name of a `KafkaHeaderMapper` used for mapping `spring-messaging` headers to and from Kafka headers. -Use this, for example, if you wish to customize the trusted packages in a `BinderHeaderMapper` bean that uses JSON deserialization for the headers. -If this custom `BinderHeaderMapper` bean is not made available to the binder using this property, then the binder will look for a header mapper bean with the name `kafkaBinderHeaderMapper` that is of type `BinderHeaderMapper` before falling back to a default `BinderHeaderMapper` created by the binder. -+ -Default: none. - -spring.cloud.stream.kafka.binder.considerDownWhenAnyPartitionHasNoLeader:: -Flag to set the binder health as `down`, when any partitions on the topic, regardless of the consumer that is receiving data from it, is found without a leader. -+ -Default: `false`. - -spring.cloud.stream.kafka.binder.certificateStoreDirectory:: -When the truststore or keystore certificate location is given as a non-local file system resource (resources supported by org.springframework.core.io.Resource e.g. CLASSPATH, HTTP, etc.), -the binder copies the resource from the path (which is convertible to org.springframework.core.io.Resource) to a location on the filesystem. -This is true for both broker level certificates (`ssl.truststore.location` and `ssl.keystore.location`) and certificates intended for schema registry (`schema.registry.ssl.truststore.location` and `schema.registry.ssl.keystore.location`). -Keep in mind that the truststore and keystore location paths must be provided under `spring.cloud.stream.kafka.binder.configuration...`. -For example, `spring.cloud.stream.kafka.binder.configuration.ssl.truststore.location`, `spring.cloud.stream.kafka.binder.configuration.schema.registry.ssl.truststore.location`, etc. -The file will be copied to the location specified as the value for this property which must be an existing directory on the filesystem that is writable by the process running the application. -If this value is not set and the certificate file is a non-local file system resource, then it will be copied to System's temp directory as returned by `System.getProperty("java.io.tmpdir")`. -This is also true, if this value is present, but the directory cannot be found on the filesystem or is not writable. -+ -Default: none. - -spring.cloud.stream.kafka.binder.metrics.defaultOffsetLagMetricsEnabled:: -When set to true, the offset lag metric of each consumer topic is computed whenever the metric is accessed. -When set to false only the periodically calculated offset lag is used. -+ -Default: true - -spring.cloud.stream.kafka.binder.metrics.offsetLagMetricsInterval:: -The interval in which the offset lag for each consumer topic is computed. -This value is used whenever `metrics.defaultOffsetLagMetricsEnabled` is disabled or its -computation is taking too long. -+ -Default: 60 seconds - -spring.cloud.stream.kafka.binder.enableObservation:: -Enable Micrometer observation registry on all the bindings in this binder. -+ -Default: false - -[[kafka-consumer-properties]] -=== Kafka Consumer Properties - -NOTE: To avoid repetition, Spring Cloud Stream supports setting values for all channels, in the format of `spring.cloud.stream.kafka.default.consumer.=`. - - -The following properties are available for Kafka consumers only and -must be prefixed with `spring.cloud.stream.kafka.bindings..consumer.`. - -admin.configuration:: -Since version 2.1.1, this property is deprecated in favor of `topic.properties`, and support for it will be removed in a future version. - -admin.replicas-assignment:: -Since version 2.1.1, this property is deprecated in favor of `topic.replicas-assignment`, and support for it will be removed in a future version. - -admin.replication-factor:: -Since version 2.1.1, this property is deprecated in favor of `topic.replication-factor`, and support for it will be removed in a future version. - -autoRebalanceEnabled:: -When `true`, topic partitions is automatically rebalanced between the members of a consumer group. -When `false`, each consumer is assigned a fixed set of partitions based on `spring.cloud.stream.instanceCount` and `spring.cloud.stream.instanceIndex`. -This requires both the `spring.cloud.stream.instanceCount` and `spring.cloud.stream.instanceIndex` properties to be set appropriately on each launched instance. -The value of the `spring.cloud.stream.instanceCount` property must typically be greater than 1 in this case. -+ -Default: `true`. -ackEachRecord:: -When `autoCommitOffset` is `true`, this setting dictates whether to commit the offset after each record is processed. -By default, offsets are committed after all records in the batch of records returned by `consumer.poll()` have been processed. -The number of records returned by a poll can be controlled with the `max.poll.records` Kafka property, which is set through the consumer `configuration` property. -Setting this to `true` may cause a degradation in performance, but doing so reduces the likelihood of redelivered records when a failure occurs. -Also, see the binder `requiredAcks` property, which also affects the performance of committing offsets. -This property is deprecated as of 3.1 in favor of using `ackMode`. -If the `ackMode` is not set and batch mode is not enabled, `RECORD` ackMode will be used. -+ -Default: `false`. - -autoCommitOffset:: - -Starting with version 3.1, this property is deprecated. -See `ackMode` for more details on alternatives. -Whether to autocommit offsets when a message has been processed. -If set to `false`, a header with the key `kafka_acknowledgment` of the type `org.springframework.kafka.support.Acknowledgment` header is present in the inbound message. -Applications may use this header for acknowledging messages. -See the examples section for details. -When this property is set to `false`, Kafka binder sets the ack mode to `org.springframework.kafka.listener.AbstractMessageListenerContainer.AckMode.MANUAL` and the application is responsible for acknowledging records. -Also see `ackEachRecord`. -+ -Default: `true`. -ackMode:: -Specify the container ack mode. -This is based on the AckMode enumeration defined in Spring Kafka. -If `ackEachRecord` property is set to `true` and consumer is not in batch mode, then this will use the ack mode of `RECORD`, otherwise, use the provided ack mode using this property. - -autoCommitOnError:: -In pollable consumers, if set to `true`, it always auto commits on error. -If not set (the default) or false, it will not auto commit in pollable consumers. -Note that this property is only applicable for pollable consumers. -+ -Default: not set. -resetOffsets:: -Whether to reset offsets on the consumer to the value provided by startOffset. -Must be false if a `KafkaBindingRebalanceListener` is provided; see <>. -See <> for more information about this property. -+ -Default: `false`. -startOffset:: -The starting offset for new groups. -Allowed values: `earliest` and `latest`. -If the consumer group is set explicitly for the consumer 'binding' (through `spring.cloud.stream.bindings..group`), 'startOffset' is set to `earliest`. Otherwise, it is set to `latest` for the `anonymous` consumer group. -See <> for more information about this property. -+ -Default: null (equivalent to `earliest`). -enableDlq:: -When set to true, it enables DLQ behavior for the consumer. -By default, messages that result in errors are forwarded to a topic named `error..`. -The DLQ topic name can be configurable by setting the `dlqName` property or by defining a `@Bean` of type `DlqDestinationResolver`. -This provides an alternative option to the more common Kafka replay scenario for the case when the number of errors is relatively small and replaying the entire original topic may be too cumbersome. -See <> processing for more information. -Starting with version 2.0, messages sent to the DLQ topic are enhanced with the following headers: `x-original-topic`, `x-exception-message`, and `x-exception-stacktrace` as `byte[]`. -By default, a failed record is sent to the same partition number in the DLQ topic as the original record. -See <> for how to change that behavior. -**Not allowed when `destinationIsPattern` is `true`.** -+ -Default: `false`. -dlqPartitions:: -When `enableDlq` is true, and this property is not set, a dead letter topic with the same number of partitions as the primary topic(s) is created. -Usually, dead-letter records are sent to the same partition in the dead-letter topic as the original record. -This behavior can be changed; see <>. -If this property is set to `1` and there is no `DqlPartitionFunction` bean, all dead-letter records will be written to partition `0`. -If this property is greater than `1`, you **MUST** provide a `DlqPartitionFunction` bean. -Note that the actual partition count is affected by the binder's `minPartitionCount` property. -+ -Default: `none` -configuration:: -Map with a key/value pair containing generic Kafka consumer properties. -In addition to having Kafka consumer properties, other configuration properties can be passed here. -For example some properties needed by the application such as `spring.cloud.stream.kafka.bindings.input.consumer.configuration.foo=bar`. -The `bootstrap.servers` property cannot be set here; use multi-binder support if you need to connect to multiple clusters. -+ -Default: Empty map. -dlqName:: -The name of the DLQ topic to receive the error messages. -+ -Default: null (If not specified, messages that result in errors are forwarded to a topic named `error..`). -dlqProducerProperties:: -Using this, DLQ-specific producer properties can be set. -All the properties available through kafka producer properties can be set through this property. -When native decoding is enabled on the consumer (i.e., useNativeDecoding: true) , the application must provide corresponding key/value serializers for DLQ. -This must be provided in the form of `dlqProducerProperties.configuration.key.serializer` and `dlqProducerProperties.configuration.value.serializer`. -+ -Default: Default Kafka producer properties. -standardHeaders:: -Indicates which standard headers are populated by the inbound channel adapter. -Allowed values: `none`, `id`, `timestamp`, or `both`. -Useful if using native deserialization and the first component to receive a message needs an `id` (such as an aggregator that is configured to use a JDBC message store). -+ -Default: `none` -converterBeanName:: -The name of a bean that implements `RecordMessageConverter`. Used in the inbound channel adapter to replace the default `MessagingMessageConverter`. -+ -Default: `null` -idleEventInterval:: -The interval, in milliseconds, between events indicating that no messages have recently been received. -Use an `ApplicationListener` to receive these events. -See <> for a usage example. -+ -Default: `30000` -destinationIsPattern:: -When true, the destination is treated as a regular expression `Pattern` used to match topic names by the broker. -When true, topics are not provisioned, and `enableDlq` is not allowed, because the binder does not know the topic names during the provisioning phase. -Note, the time taken to detect new topics that match the pattern is controlled by the consumer property `metadata.max.age.ms`, which (at the time of writing) defaults to 300,000ms (5 minutes). -This can be configured using the `configuration` property above. -+ -Default: `false` -topic.properties:: -A `Map` of Kafka topic properties used when provisioning new topics -- for example, `spring.cloud.stream.kafka.bindings.input.consumer.topic.properties.message.format.version=0.9.0.0` -+ -Default: none. -topic.replicas-assignment:: -A Map> of replica assignments, with the key being the partition and the value being the assignments. -Used when provisioning new topics. -See the `NewTopic` Javadocs in the `kafka-clients` jar. -+ -Default: none. -topic.replication-factor:: -The replication factor to use when provisioning topics. Overrides the binder-wide setting. -Ignored if `replicas-assignments` is present. -+ -Default: none (the binder-wide default of -1 is used). -pollTimeout:: -Timeout used for polling in pollable consumers. -+ -Default: 5 seconds. -transactionManager:: -Bean name of a `KafkaAwareTransactionManager` used to override the binder's transaction manager for this binding. -Usually needed if you want to synchronize another transaction with the Kafka transaction, using the `ChainedKafkaTransactionManaager`. -To achieve exactly once consumption and production of records, the consumer and producer bindings must all be configured with the same transaction manager. -+ -Default: none. -txCommitRecovered:: -When using a transactional binder, the offset of a recovered record (e.g. when retries are exhausted and the record is sent to a dead letter topic) will be committed via a new transaction, by default. -Setting this property to `false` suppresses committing the offset of recovered record. -+ -Default: true. -commonErrorHandlerBeanName:: -`CommonErrorHandler` bean name to use per consumer binding. -When present, this user provided `CommonErrorHandler` takes precedence over any other error handlers defined by the binder. -This is a handy way to express error handlers, if the application does not want to use a `ListenerContainerCustomizer` and then check the destination/group combination to set an error handler. -+ -Default: none. - - -[[reset-offsets]] -=== Resetting Offsets - -When an application starts, the initial position in each assigned partition depends on two properties `startOffset` and `resetOffsets`. -If `resetOffsets` is `false`, normal Kafka consumer https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset[`auto.offset.reset`] semantics apply. -i.e. If there is no committed offset for a partition for the binding's consumer group, the position is `earliest` or `latest`. -By default, bindings with an explicit `group` use `earliest`, and anonymous bindings (with no `group`) use `latest`. -These defaults can be overridden by setting the `startOffset` binding property. -There will be no committed offset(s) the first time the binding is started with a particular `group`. -The other condition where no committed offset exists is if the offset has been expired. -With modern brokers (since 2.1), and default broker properties, the offsets are expired 7 days after the last member leaves the group. -See the https://kafka.apache.org/documentation/#brokerconfigs_offsets.retention.minutes[`offsets.retention.minutes`] broker property for more information. - -When `resetOffsets` is `true`, the binder applies similar semantics to those that apply when there is no committed offset on the broker, as if this binding has never consumed from the topic; i.e. any current committed offset is ignored. - -Following are two use cases when this might be used. - -1. Consuming from a compacted topic containing key/value pairs. -Set `resetOffsets` to `true` and `startOffset` to `earliest`; the binding will perform a `seekToBeginning` on all newly assigned partitions. - -2. Consuming from a topic containing events, where you are only interested in events that occur while this binding is running. -Set `resetOffsets` to `true` and `startOffset` to `latest`; the binding will perform a `seekToEnd` on all newly assigned partitions. - -IMPORTANT: If a rebalance occurs after the initial assignment, the seeks will only be performed on any newly assigned partitions that were not assigned during the initial assignment. - -For more control over topic offsets, see <>; when a listener is provided, `resetOffsets` should not be set to `true`, otherwise, that will cause an error. - -[[consuming-batches]] -=== Consuming Batches - -Starting with version 3.0, when `spring.cloud.stream.bindings..consumer.batch-mode` is set to `true`, all of the records received by polling the Kafka `Consumer` will be presented as a `List` to the listener method. -Otherwise, the method will be called with one record at a time. -The size of the batch is controlled by Kafka consumer properties `max.poll.records`, `fetch.min.bytes`, `fetch.max.wait.ms`; refer to the Kafka documentation for more information. - -Starting with version `4.0.2`, the binder supports DLQ capabilities when consuming in batch mode. -Keep in mind that, when using DLQ on a consumer binding that is in batch mode, all the records received from the previous poll will be delivered to the DLQ topic. - -IMPORTANT: Retry within the binder is not supported when using batch mode, so `maxAttempts` will be overridden to 1. -You can configure a `DefaultErrorHandler` (using a `ListenerContainerCustomizer`) to achieve similar functionality to retry in the binder. -You can also use a manual `AckMode` and call `Ackowledgment.nack(index, sleep)` to commit the offsets for a partial batch and have the remaining records redelivered. -Refer to the https://docs.spring.io/spring-kafka/docs/2.3.15.BUILD-SNAPSHOT/reference/html/#committing-offsets[Spring for Apache Kafka documentation] for more information about these techniques. - -[[kafka-producer-properties]] -=== Kafka Producer Properties - -NOTE: To avoid repetition, Spring Cloud Stream supports setting values for all channels, in the format of `spring.cloud.stream.kafka.default.producer.=`. - - -The following properties are available for Kafka producers only and -must be prefixed with `spring.cloud.stream.kafka.bindings..producer.`. - -admin.configuration:: -Since version 2.1.1, this property is deprecated in favor of `topic.properties`, and support for it will be removed in a future version. - -admin.replicas-assignment:: -Since version 2.1.1, this property is deprecated in favor of `topic.replicas-assignment`, and support for it will be removed in a future version. - -admin.replication-factor:: -Since version 2.1.1, this property is deprecated in favor of `topic.replication-factor`, and support for it will be removed in a future version. - -bufferSize:: -Upper limit, in bytes, of how much data the Kafka producer attempts to batch before sending. -+ -Default: `16384`. -sync:: -Whether the producer is synchronous. -+ -Default: `false`. -sendTimeoutExpression:: -A SpEL expression evaluated against the outgoing message used to evaluate the time to wait for ack when synchronous publish is enabled -- for example, `headers['mySendTimeout']`. -The value of the timeout is in milliseconds. -With versions before 3.0, the payload could not be used unless native encoding was being used because, by the time this expression was evaluated, the payload was already in the form of a `byte[]`. -Now, the expression is evaluated before the payload is converted. -+ -Default: `none`. -batchTimeout:: -How long the producer waits to allow more messages to accumulate in the same batch before sending the messages. -(Normally, the producer does not wait at all and simply sends all the messages that accumulated while the previous send was in progress.) A non-zero value may increase throughput at the expense of latency. -+ -Default: `0`. -messageKeyExpression:: -A SpEL expression evaluated against the outgoing message used to populate the key of the produced Kafka message -- for example, `headers['myKey']`. -With versions before 3.0, the payload could not be used unless native encoding was being used because, by the time this expression was evaluated, the payload was already in the form of a `byte[]`. -Now, the expression is evaluated before the payload is converted. -In the case of a regular processor (`Function` or `Function, Message`), if the produced key needs to be same as the incoming key from the topic, this property can be set as below. -`spring.cloud.stream.kafka.bindings..producer.messageKeyExpression: headers['kafka_receivedMessageKey']` -There is an important caveat to keep in mind for reactive functions. -In that case, it is up to the application to manually copy the headers from the incoming messages to outbound messages. -You can set the header, e.g. `myKey` and use `headers['myKey']` as suggested above or, for convenience, simply set the `KafkaHeaders.MESSAGE_KEY` header, and you do not need to set this property at all. -+ -Default: `none`. -headerPatterns:: -A comma-delimited list of simple patterns to match Spring messaging headers to be mapped to the Kafka `Headers` in the `ProducerRecord`. -Patterns can begin or end with the wildcard character (asterisk). -Patterns can be negated by prefixing with `!`. -Matching stops after the first match (positive or negative). -For example `!ask,as*` will pass `ash` but not `ask`. -`id` and `timestamp` are never mapped. -+ -Default: `*` (all headers - except the `id` and `timestamp`) -configuration:: -Map with a key/value pair containing generic Kafka producer properties. -The `bootstrap.servers` property cannot be set here; use multi-binder support if you need to connect to multiple clusters. -+ -Default: Empty map. -topic.properties:: -A `Map` of Kafka topic properties used when provisioning new topics -- for example, `spring.cloud.stream.kafka.bindings.output.producer.topic.properties.message.format.version=0.9.0.0` -+ -topic.replicas-assignment:: -A Map> of replica assignments, with the key being the partition and the value being the assignments. -Used when provisioning new topics. -See the `NewTopic` Javadocs in the `kafka-clients` jar. -+ -Default: none. -topic.replication-factor:: -The replication factor to use when provisioning topics. Overrides the binder-wide setting. -Ignored if `replicas-assignments` is present. -+ -Default: none (the binder-wide default of -1 is used). -useTopicHeader:: -Set to `true` to override the default binding destination (topic name) with the value of the `KafkaHeaders.TOPIC` message header in the outbound message. -If the header is not present, the default binding destination is used. -+ -Default: `false`. -recordMetadataChannel:: -The bean name of a `MessageChannel` to which successful send results should be sent; the bean must exist in the application context. -The message sent to the channel is the sent message (after conversion, if any) with an additional header `KafkaHeaders.RECORD_METADATA`. -The header contains a `RecordMetadata` object provided by the Kafka client; it includes the partition and offset where the record was written in the topic. -+ -`ResultMetadata meta = sendResultMsg.getHeaders().get(KafkaHeaders.RECORD_METADATA, RecordMetadata.class)` -+ -Failed sends go the producer error channel (if configured); see <>. -+ -Default: null. - -NOTE: The Kafka binder uses the `partitionCount` setting of the producer as a hint to create a topic with the given partition count (in conjunction with the `minPartitionCount`, the maximum of the two being the value being used). -Exercise caution when configuring both `minPartitionCount` for a binder and `partitionCount` for an application, as the larger value is used. -If a topic already exists with a smaller partition count and `autoAddPartitions` is disabled (the default), the binder fails to start. -If a topic already exists with a smaller partition count and `autoAddPartitions` is enabled, new partitions are added. -If a topic already exists with a larger number of partitions than the maximum of (`minPartitionCount` or `partitionCount`), the existing partition count is used. - -compression:: -Set the `compression.type` producer property. -Supported values are `none`, `gzip`, `snappy`, `lz4` and `zstd`. -If you override the `kafka-clients` jar to 2.1.0 (or later), as discussed in the https://docs.spring.io/spring-kafka/docs/2.2.x/reference/html/#deps-for-21x[Spring for Apache Kafka documentation], and wish to use `zstd` compression, use `spring.cloud.stream.kafka.bindings..producer.configuration.compression.type=zstd`. -+ -Default: `none`. -transactionManager:: -Bean name of a `KafkaAwareTransactionManager` used to override the binder's transaction manager for this binding. -Usually needed if you want to synchronize another transaction with the Kafka transaction, using the `ChainedKafkaTransactionManaager`. -To achieve exactly once consumption and production of records, the consumer and producer bindings must all be configured with the same transaction manager. -+ -Default: none. - -closeTimeout:: -Timeout in number of seconds to wait for when closing the producer. -+ -Default: `30` - -allowNonTransactional:: -Normally, all output bindings associated with a transactional binder will publish in a new transaction, if one is not already in process. -This property allows you to override that behavior. -If set to true, records published to this output binding will not be run in a transaction, unless one is already in process. -+ -Default: `false` - -[[usage-examples]] -=== Usage examples - -In this section, we show the use of the preceding properties for specific scenarios. - -[[example:-setting-ackmode-to-manual-and-relying-on-manual-acknowledgement]] -==== Example: Setting `ackMode` to `MANUAL` and Relying on Manual Acknowledgement - -This example illustrates how one may manually acknowledge offsets in a consumer application. - -This example requires that `spring.cloud.stream.kafka.bindings.input.consumer.ackMode` be set to `MANUAL`. -Use the corresponding input channel name for your example. - -[source] ----- -@SpringBootApplication -public class ManuallyAcknowdledgingConsumer { - - public static void main(String[] args) { - SpringApplication.run(ManuallyAcknowdledgingConsumer.class, args); - } - - @Bean - public Consumer> process() { - return message -> { - Acknowledgment acknowledgment = message.getHeaders().get(KafkaHeaders.ACKNOWLEDGMENT, Acknowledgment.class); - if (acknowledgment != null) { - System.out.println("Acknowledgment provided"); - acknowledgment.acknowledge(); - } - }; -} ----- - -[[example:-security-configuration]] -==== Example: Security Configuration - -Apache Kafka 0.9 supports secure connections between client and brokers. -To take advantage of this feature, follow the guidelines in the https://kafka.apache.org/090/documentation.html#security_configclients[Apache Kafka Documentation] as well as the Kafka 0.9 https://docs.confluent.io/2.0.0/kafka/security.html[security guidelines from the Confluent documentation]. -Use the `spring.cloud.stream.kafka.binder.configuration` option to set security properties for all clients created by the binder. - -For example, to set `security.protocol` to `SASL_SSL`, set the following property: - -[source] ----- -spring.cloud.stream.kafka.binder.configuration.security.protocol=SASL_SSL ----- - -All the other security properties can be set in a similar manner. - -When using Kerberos, follow the instructions in the https://kafka.apache.org/090/documentation.html#security_sasl_clientconfig[reference documentation] for creating and referencing the JAAS configuration. - -Spring Cloud Stream supports passing JAAS configuration information to the application by using a JAAS configuration file and using Spring Boot properties. - -[[using-jaas-configuration-files]] -===== Using JAAS Configuration Files - -The JAAS and (optionally) krb5 file locations can be set for Spring Cloud Stream applications by using system properties. -The following example shows how to launch a Spring Cloud Stream application with SASL and Kerberos by using a JAAS configuration file: - -[source,bash] ----- - java -Djava.security.auth.login.config=/path.to/kafka_client_jaas.conf -jar log.jar \ - --spring.cloud.stream.kafka.binder.brokers=secure.server:9092 \ - --spring.cloud.stream.bindings.input.destination=stream.ticktock \ - --spring.cloud.stream.kafka.binder.configuration.security.protocol=SASL_PLAINTEXT ----- - -[[using-spring-boot-properties]] -===== Using Spring Boot Properties - -As an alternative to having a JAAS configuration file, Spring Cloud Stream provides a mechanism for setting up the JAAS configuration for Spring Cloud Stream applications by using Spring Boot properties. - -The following properties can be used to configure the login context of the Kafka client: - -spring.cloud.stream.kafka.binder.jaas.loginModule:: -The login module name. Not necessary to be set in normal cases. -+ -Default: `com.sun.security.auth.module.Krb5LoginModule`. -spring.cloud.stream.kafka.binder.jaas.controlFlag:: -The control flag of the login module. -+ -Default: `required`. -spring.cloud.stream.kafka.binder.jaas.options:: -Map with a key/value pair containing the login module options. -+ -Default: Empty map. - -The following example shows how to launch a Spring Cloud Stream application with SASL and Kerberos by using Spring Boot configuration properties: - -[source,bash] ----- - java --spring.cloud.stream.kafka.binder.brokers=secure.server:9092 \ - --spring.cloud.stream.bindings.input.destination=stream.ticktock \ - --spring.cloud.stream.kafka.binder.autoCreateTopics=false \ - --spring.cloud.stream.kafka.binder.configuration.security.protocol=SASL_PLAINTEXT \ - --spring.cloud.stream.kafka.binder.jaas.options.useKeyTab=true \ - --spring.cloud.stream.kafka.binder.jaas.options.storeKey=true \ - --spring.cloud.stream.kafka.binder.jaas.options.keyTab=/etc/security/keytabs/kafka_client.keytab \ - --spring.cloud.stream.kafka.binder.jaas.options.principal=kafka-client-1@EXAMPLE.COM ----- - -The preceding example represents the equivalent of the following JAAS file: - -[source] ----- -KafkaClient { - com.sun.security.auth.module.Krb5LoginModule required - useKeyTab=true - storeKey=true - keyTab="/etc/security/keytabs/kafka_client.keytab" - principal="kafka-client-1@EXAMPLE.COM"; -}; ----- - -If the topics required already exist on the broker or will be created by an administrator, autocreation can be turned off and only client JAAS properties need to be sent. - -NOTE: Do not mix JAAS configuration files and Spring Boot properties in the same application. -If the `-Djava.security.auth.login.config` system property is already present, Spring Cloud Stream ignores the Spring Boot properties. - -NOTE: Be careful when using the `autoCreateTopics` and `autoAddPartitions` with Kerberos. -Usually, applications may use principals that do not have administrative rights in Kafka and Zookeeper. -Consequently, relying on Spring Cloud Stream to create/modify topics may fail. -In secure environments, we strongly recommend creating topics and managing ACLs administratively by using Kafka tooling. - -[[multi-binder-configuration-and-jaas]] -===== Multi-binder configuration and JAAS - -When connecting to multiple clusters in which each one requires separate JAAS configuration, then set the JAAS configuration using the property `sasl.jaas.config`. -When this property is present in the application, it takes precedence over the other strategies mentioned above. -See this https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients[KIP-85] for more details. - -For example, if you have two clusters in your application with separate JAAS configuration, then the following is a template that you can use: - -``` -spring.cloud.stream: - binders: - kafka1: - type: kafka - environment: - spring: - cloud: - stream: - kafka: - binder: - brokers: localhost:9092 - configuration.sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"admin-secret\";" - kafka2: - type: kafka - environment: - spring: - cloud: - stream: - kafka: - binder: - brokers: localhost:9093 - configuration.sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user1\" password=\"user1-secret\";" - kafka.binder: - configuration: - security.protocol: SASL_PLAINTEXT - sasl.mechanism: PLAIN -``` - -Note that both the Kafka clusters, and the `sasl.jaas.config` values for each of them are different in the above configuration. - -See this https://github.com/spring-cloud/spring-cloud-stream-samples/tree/main/multi-binder-samples/kafka-multi-binder-jaas[sample application] for more details on how to setup and run such an application. - -[[pause-resume]] -==== Example: Pausing and Resuming the Consumer - -If you wish to suspend consumption but not cause a partition rebalance, you can pause and resume the consumer. -This is facilitated by managing the binding lifecycle as shown in **Binding visualization and control** in the Spring Cloud Stream documentation, using `State.PAUSED` and `State.RESUMED`. - -To resume, you can use an `ApplicationListener` (or `@EventListener` method) to receive `ListenerContainerIdleEvent` instances. -The frequency at which events are published is controlled by the `idleEventInterval` property. - -[[kafka-transactional-binder]] -== Transactional Binder - -Enable transactions by setting `spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix` to a non-empty value, e.g. `tx-`. -When used in a processor application, the consumer starts the transaction; any records sent on the consumer thread participate in the same transaction. -When the listener exits normally, the listener container will send the offset to the transaction and commit it. -A common producer factory is used for all producer bindings configured using `spring.cloud.stream.kafka.binder.transaction.producer.*` properties; individual binding Kafka producer properties are ignored. - -IMPORTANT: Normal binder retries (and dead lettering) are not supported with transactions because the retries will run in the original transaction, which may be rolled back and any published records will be rolled back too. -When retries are enabled (the common property `maxAttempts` is greater than zero) the retry properties are used to configure a `DefaultAfterRollbackProcessor` to enable retries at the container level. -Similarly, instead of publishing dead-letter records within the transaction, this functionality is moved to the listener container, again via the `DefaultAfterRollbackProcessor` which runs after the main transaction has rolled back. - -If you wish to use transactions in a source application, or from some arbitrary thread for producer-only transaction (e.g. `@Scheduled` method), you must get a reference to the transactional producer factory and define a `KafkaTransactionManager` bean using it. - -==== -[source, java] ----- -@Bean -public PlatformTransactionManager transactionManager(BinderFactory binders, - @Value("${unique.tx.id.per.instance}") String txId) { - - ProducerFactory pf = ((KafkaMessageChannelBinder) binders.getBinder(null, - MessageChannel.class)).getTransactionalProducerFactory(); - KafkaTransactionManager tm = new KafkaTransactionManager<>(pf); - tm.setTransactionId(txId) - return tm; -} ----- -==== - -Notice that we get a reference to the binder using the `BinderFactory`; use `null` in the first argument when there is only one binder configured. -If more than one binder is configured, use the binder name to get the reference. -Once we have a reference to the binder, we can obtain a reference to the `ProducerFactory` and create a transaction manager. - -Then you would use normal Spring transaction support, e.g. `TransactionTemplate` or `@Transactional`, for example: - -==== -[source, java] ----- -public static class Sender { - - @Transactional - public void doInTransaction(MessageChannel output, List stuffToSend) { - stuffToSend.forEach(stuff -> output.send(new GenericMessage<>(stuff))); - } - -} ----- -==== - -If you wish to synchronize producer-only transactions with those from some other transaction manager, use a `ChainedTransactionManager`. - -IMPORTANT: If you deploy multiple instances of your application, each instance needs a unique `transactionIdPrefix`. - -[[kafka-error-channels]] -== Error Channels - -Starting with version 1.3, the binder unconditionally sends exceptions to an error channel for each consumer destination and can also be configured to send async producer send failures to an error channel. -See https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/current/reference/html/spring-cloud-stream.html#spring-cloud-stream-overview-error-handling[this section on error handling] for more information. - -The payload of the `ErrorMessage` for a send failure is a `KafkaSendFailureException` with properties: - -* `failedMessage`: The Spring Messaging `Message` that failed to be sent. -* `record`: The raw `ProducerRecord` that was created from the `failedMessage` - -There is no automatic handling of producer exceptions (such as sending to a <>). -You can consume these exceptions with your own Spring Integration flow. - -[[kafka-metrics]] -== Kafka Metrics - -Kafka binder module exposes the following metrics: - -`spring.cloud.stream.binder.kafka.offset`: This metric indicates how many messages have not been yet consumed from a given binder's topic by a given consumer group. -The metrics provided are based on the Micrometer library. -The binder creates the `KafkaBinderMetrics` bean if Micrometer is on the classpath and no other such beans provided by the application. -The metric contains the consumer group information, topic and the actual lag in committed offset from the latest offset on the topic. -This metric is particularly useful for providing auto-scaling feedback to a PaaS platform. - -The metric collection behaviour can be configured by setting properties in the `spring.cloud.stream.kafka.binder.metrics` namespace, -refer to the <> for more information. - -You can exclude `KafkaBinderMetrics` from creating the necessary infrastructure like consumers and then reporting the metrics by providing the following component in the application. - -``` -@Component -class NoOpBindingMeters { - NoOpBindingMeters(MeterRegistry registry) { - registry.config().meterFilter( - MeterFilter.denyNameStartsWith(KafkaBinderMetrics.OFFSET_LAG_METRIC_NAME)); - } -} -``` - -More details on how to suppress meters selectively can be found https://micrometer.io/docs/concepts#_meter_filters[here]. - -[[kafka-tombstones]] -== Tombstone Records (null record values) - -When using compacted topics, a record with a `null` value (also called a tombstone record) represents the deletion of a key. -To receive such messages in a Spring Cloud Stream function, you can use the following strategy. - -==== -[source, java] ----- -@Bean -public Function, String> myFunction() { - return value -> { - Object v = value.getPayload(); - String className = v.getClass().getName(); - if (className.isEqualTo("org.springframework.kafka.support.KafkaNull")) { - // this is a tombstone record - } - else { - // continue with processing - } - }; -} ----- -==== - -[[rebalance-listener]] -== Using a KafkaBindingRebalanceListener - -Applications may wish to seek topics/partitions to arbitrary offsets when the partitions are initially assigned, or perform other operations on the consumer. -Starting with version 2.1, if you provide a single `KafkaBindingRebalanceListener` bean in the application context, it will be wired into all Kafka consumer bindings. - -==== -[source, java] ----- -public interface KafkaBindingRebalanceListener { - - /** - * Invoked by the container before any pending offsets are committed. - * @param bindingName the name of the binding. - * @param consumer the consumer. - * @param partitions the partitions. - */ - default void onPartitionsRevokedBeforeCommit(String bindingName, Consumer consumer, - Collection partitions) { - - } - - /** - * Invoked by the container after any pending offsets are committed. - * @param bindingName the name of the binding. - * @param consumer the consumer. - * @param partitions the partitions. - */ - default void onPartitionsRevokedAfterCommit(String bindingName, Consumer consumer, Collection partitions) { - - } - - /** - * Invoked when partitions are initially assigned or after a rebalance. - * Applications might only want to perform seek operations on an initial assignment. - * @param bindingName the name of the binding. - * @param consumer the consumer. - * @param partitions the partitions. - * @param initial true if this is the initial assignment. - */ - default void onPartitionsAssigned(String bindingName, Consumer consumer, Collection partitions, - boolean initial) { - - } - -} ----- -==== - -You cannot set the `resetOffsets` consumer property to `true` when you provide a rebalance listener. - -[[retry-and-dlq-processing]] -== Retry and Dead Letter Processing - -By default, when you configure retry (e.g. `maxAttemts`) and `enableDlq` in a consumer binding, these functions are performed within the binder, with no participation by the listener container or Kafka consumer. - -There are situations where it is preferable to move this functionality to the listener container, such as: - -* The aggregate of retries and delays will exceed the consumer's `max.poll.interval.ms` property, potentially causing a partition rebalance. -* You wish to publish the dead letter to a different Kafka cluster. -* You wish to add retry listeners to the error handler. -* ... - -To configure moving this functionality from the binder to the container, define a `@Bean` of type `ListenerContainerWithDlqAndRetryCustomizer`. -This interface has the following methods: - -==== -[source, java] ----- -/** - * Configure the container. - * @param container the container. - * @param destinationName the destination name. - * @param group the group. - * @param dlqDestinationResolver a destination resolver for the dead letter topic (if - * enableDlq). - * @param backOff the backOff using retry properties (if configured). - * @see #retryAndDlqInBinding(String, String) - */ -void configure(AbstractMessageListenerContainer container, String destinationName, String group, - @Nullable BiFunction, Exception, TopicPartition> dlqDestinationResolver, - @Nullable BackOff backOff); - -/** - * Return false to move retries and DLQ from the binding to a customized error handler - * using the retry metadata and/or a {@code DeadLetterPublishingRecoverer} when - * configured via - * {@link #configure(AbstractMessageListenerContainer, String, String, BiFunction, BackOff)}. - * @param destinationName the destination name. - * @param group the group. - * @return false to disable retries and DLQ in the binding - */ -default boolean retryAndDlqInBinding(String destinationName, String group) { - return true; -} ----- -==== - -The destination resolver and `BackOff` are created from the binding properties (if configured). The `KafkaTemplate` uses configuration from `spring.kafka....` properties. You can then use these to create a custom error handler and dead letter publisher; for example: - -==== -[source, java] ----- -@Bean -ListenerContainerWithDlqAndRetryCustomizer cust(KafkaTemplate template) { - return new ListenerContainerWithDlqAndRetryCustomizer() { - - @Override - public void configure(AbstractMessageListenerContainer container, String destinationName, - String group, - @Nullable BiFunction, Exception, TopicPartition> dlqDestinationResolver, - @Nullable BackOff backOff) { - - if (destinationName.equals("topicWithLongTotalRetryConfig")) { - ConsumerRecordRecoverer dlpr = new DeadLetterPublishingRecoverer(template, - dlqDestinationResolver); - container.setCommonErrorHandler(new DefaultErrorHandler(dlpr, backOff)); - } - } - - @Override - public boolean retryAndDlqInBinding(String destinationName, String group) { - return !destinationName.contains("topicWithLongTotalRetryConfig"); - } - - }; -} ----- -==== - -Now, only a single retry delay needs to be greater than the consumer's `max.poll.interval.ms` property. - -When working with several binders, the 'ListenerContainerWithDlqAndRetryCustomizer' bean gets overridden by the 'DefaultBinderFactory'. For the bean -to apply, you need to use a 'BinderCustomizer' to set the container customizer (See <>): -==== -[source, java] ----- -@Bean -public BinderCustomizer binderCustomizer(ListenerContainerWithDlqAndRetryCustomizer containerCustomizer) { - return (binder, binderName) -> { - if (binder instanceof KafkaMessageChannelBinder kafkaMessageChannelBinder) { - kafkaMessageChannelBinder.setContainerCustomizer(containerCustomizer); - } - else if (binder instanceof KStreamBinder) { - ... - } - else if (binder instanceof RabbitMessageChannelBinder) { - ... - } - }; -} ----- -==== - -[[consumer-producer-config-customizer]] -== Customizing Consumer and Producer configuration - -If you want advanced customization of consumer and producer configuration that is used for creating `ConsumerFactory` and `ProducerFactory` in Kafka, -you can implement the following customizers. - -* ConsumerConfigCustomizer -* ProducerConfigCustomizer - -Both of these interfaces provide a way to configure the config map used for consumer and producer properties. -For example, if you want to gain access to a bean that is defined at the application level, you can inject that in the implementation of the `configure` method. -When the binder discovers that these customizers are available as beans, it will invoke the `configure` method right before creating the consumer and producer factories. - -Both of these interfaces also provide access to both the binding and destination names so that they can be accessed while customizing producer and consumer properties. - -[[admin-client-config-customization]] -== Customizing AdminClient Configuration - -As with consumer and producer config customization above, applications can also customize the configuration for admin clients by providing an `AdminClientConfigCustomizer`. -AdminClientConfigCustomizer's configure method provides access to the admin client properties, using which you can define further customization. -Binder's Kafka topic provisioner gives the highest precedence for the properties given through this customizer. -Here is an example of providing this customizer bean. - -``` -@Bean -public AdminClientConfigCustomizer adminClientConfigCustomizer() { - return props -> { - props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL"); - }; -} -``` - -[[custom-kafka-binder-health-indicator]] -== Custom Kafka Binder Health Indicator - -Kafka binder activates a default health indicator when Spring Boot actuator is on the classpath. -This health indicator checks the health of the binder and any communication issues with the Kafka broker. -If an application wants to disable this default health check implementation and include a custom implementation, then it can provide an implementation for `KafkaBinderHealth` interface. -`KafkaBinderHealth` is a marker interface that extends from `HealthIndicator`. -In the custom implementation, it must provide an implementation for the `health()` method. -The custom implementation must be present in the application configuration as a bean. -When the binder discovers the custom implementation, it will use that instead of the default implementation. -Here is an example of such a custom implementation bean in the application. - -``` -@Bean -public KafkaBinderHealth kafkaBinderHealthIndicator() { - return new KafkaBinderHealth() { - @Override - public Health health() { - // custom implementation details. - } - }; -} -``` - -[[custom-kafka-binder-health-indicator-example]] -== Custom kafka Binder Health Indicator Example - -Here is the pseudo-code for writing a custom Kafka binder HealthIndicator. -In this example, we try to override the binder provided Kafka HealthIndicator by specifically checking first for cluster connectivity and then followed by topic-related issues. - - -1. First we need create a custom implementation of the `KafkaBinderHealth` interface. - -``` -public class KafkaBinderHealthImplementation implements KafkaBinderHealth { - @Value("${spring.cloud.bus.destination}") - private String topic; - private final AdminClient client; - - public KafkaBinderHealthImplementation(final KafkaAdmin admin) { - // More about configuring Kafka - // https://docs.spring.io/spring-kafka/reference/html/#configuring-topics - this.client = AdminClient.create(admin.getConfigurationProperties()); - } - - @Override - public Health health() { - if (!checkBrokersConnection()) { - logger.error("Error when connect brokers"); - return Health.down().withDetail("BrokersConnectionError", "Error message").build(); - } - if (!checkTopicConnection()) { - logger.error("Error when trying to connect with specific topic"); - return Health.down().withDetail("TopicError", "Error message with topic name").build(); - } - return Health.up().build(); - } - - public boolean checkBrokersConnection() { - // Your implementation - } - - public boolean checkTopicConnection() { - // Your implementation - } -} -``` - -2. Then we need to create a bean for the custom implementation. - -``` -@Configuration -public class KafkaBinderHealthIndicatorConfiguration { - @Bean - public KafkaBinderHealth kafkaBinderHealthIndicator(final KafkaAdmin admin) { - return new KafkaBinderHealthImplementation(admin); - } -} -``` diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/admin-client-config-customization.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/admin-client-config-customization.adoc new file mode 100644 index 000000000..27e6e8181 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/admin-client-config-customization.adoc @@ -0,0 +1,17 @@ +[[admin-client-config-customization]] += Customizing AdminClient Configuration + +As with consumer and producer config customization above, applications can also customize the configuration for admin clients by providing an `AdminClientConfigCustomizer`. +AdminClientConfigCustomizer's configure method provides access to the admin client properties, using which you can define further customization. +Binder's Kafka topic provisioner gives the highest precedence for the properties given through this customizer. +Here is an example of providing this customizer bean. + +``` +@Bean +public AdminClientConfigCustomizer adminClientConfigCustomizer() { + return props -> { + props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL"); + }; +} +``` + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/configuration-options.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/configuration-options.adoc new file mode 100644 index 000000000..0f02e9fd6 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/configuration-options.adoc @@ -0,0 +1,650 @@ +[[configuration-options]] += Configuration Options + +This section contains the configuration options used by the Apache Kafka binder. + +For common configuration options and properties pertaining to the binder, see the https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/current/reference/html/spring-cloud-stream.html#binding-properties[binding properties] in core documentation. + +[[kafka-binder-properties]] +== Kafka Binder Properties + +spring.cloud.stream.kafka.binder.brokers:: +A list of brokers to which the Kafka binder connects. ++ +Default: `localhost`. +spring.cloud.stream.kafka.binder.defaultBrokerPort:: +`brokers` allows hosts specified with or without port information (for example, `host1,host2:port2`). +This sets the default port when no port is configured in the broker list. ++ +Default: `9092`. +spring.cloud.stream.kafka.binder.configuration:: +Key/Value map of client properties (both producers and consumer) passed to all clients created by the binder. +Due to the fact that these properties are used by both producers and consumers, usage should be restricted to common properties -- for example, security settings. +Unknown Kafka producer or consumer properties provided through this configuration are filtered out and not allowed to propagate. +Properties here supersede any properties set in boot. ++ +Default: Empty map. +spring.cloud.stream.kafka.binder.consumerProperties:: +Key/Value map of arbitrary Kafka client consumer properties. +In addition to support known Kafka consumer properties, unknown consumer properties are allowed here as well. +Properties here supersede any properties set in boot and in the `configuration` property above. ++ +Default: Empty map. +spring.cloud.stream.kafka.binder.headers:: +The list of custom headers that are transported by the binder. +Only required when communicating with older applications (<= 1.3.x) with a `kafka-clients` version < 0.11.0.0. Newer versions support headers natively. ++ +Default: empty. +spring.cloud.stream.kafka.binder.healthTimeout:: +The time to wait to get partition information, in seconds. +Health reports as down if this timer expires. ++ +Default: 10. +spring.cloud.stream.kafka.binder.requiredAcks:: +The number of required acks on the broker. +See the Kafka documentation for the producer `acks` property. ++ +Default: `1`. +spring.cloud.stream.kafka.binder.minPartitionCount:: +Effective only if `autoCreateTopics` or `autoAddPartitions` is set. +The global minimum number of partitions that the binder configures on topics on which it produces or consumes data. +It can be superseded by the `partitionCount` setting of the producer or by the value of `instanceCount * concurrency` settings of the producer (if either is larger). ++ +Default: `1`. +spring.cloud.stream.kafka.binder.producerProperties:: +Key/Value map of arbitrary Kafka client producer properties. +In addition to support known Kafka producer properties, unknown producer properties are allowed here as well. +Properties here supersede any properties set in boot and in the `configuration` property above. ++ +Default: Empty map. +spring.cloud.stream.kafka.binder.replicationFactor:: +The replication factor of auto-created topics if `autoCreateTopics` is active. +Can be overridden on each binding. ++ +NOTE: If you are using Kafka broker versions prior to 2.4, then this value should be set to at least `1`. +Starting with version 3.0.8, the binder uses `-1` as the default value, which indicates that the broker 'default.replication.factor' property will be used to determine the number of replicas. +Check with your Kafka broker admins to see if there is a policy in place that requires a minimum replication factor, if that's the case then, typically, the `default.replication.factor` will match that value and `-1` should be used, unless you need a replication factor greater than the minimum. ++ +Default: `-1`. +spring.cloud.stream.kafka.binder.autoCreateTopics:: +If set to `true`, the binder creates new topics automatically. +If set to `false`, the binder relies on the topics being already configured. +In the latter case, if the topics do not exist, the binder fails to start. ++ +NOTE: This setting is independent of the `auto.create.topics.enable` setting of the broker and does not influence it. +If the server is set to auto-create topics, they may be created as part of the metadata retrieval request, with default broker settings. ++ +Default: `true`. +spring.cloud.stream.kafka.binder.autoAddPartitions:: +If set to `true`, the binder creates new partitions if required. +If set to `false`, the binder relies on the partition size of the topic being already configured. +If the partition count of the target topic is smaller than the expected value, the binder fails to start. ++ +Default: `false`. +spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix:: +Enables transactions in the binder. See `transaction.id` in the Kafka documentation and https://docs.spring.io/spring-kafka/reference/html/#transactions[Transactions] in the `spring-kafka` documentation. +When transactions are enabled, individual `producer` properties are ignored and all producers use the `spring.cloud.stream.kafka.binder.transaction.producer.*` properties. ++ +Default `null` (no transactions) +spring.cloud.stream.kafka.binder.transaction.producer.*:: +Global producer properties for producers in a transactional binder. +See `spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix` and <> and the general producer properties supported by all binders. ++ +Default: See individual producer properties. + +spring.cloud.stream.kafka.binder.headerMapperBeanName:: +The bean name of a `KafkaHeaderMapper` used for mapping `spring-messaging` headers to and from Kafka headers. +Use this, for example, if you wish to customize the trusted packages in a `BinderHeaderMapper` bean that uses JSON deserialization for the headers. +If this custom `BinderHeaderMapper` bean is not made available to the binder using this property, then the binder will look for a header mapper bean with the name `kafkaBinderHeaderMapper` that is of type `BinderHeaderMapper` before falling back to a default `BinderHeaderMapper` created by the binder. ++ +Default: none. + +spring.cloud.stream.kafka.binder.considerDownWhenAnyPartitionHasNoLeader:: +Flag to set the binder health as `down`, when any partitions on the topic, regardless of the consumer that is receiving data from it, is found without a leader. ++ +Default: `false`. + +spring.cloud.stream.kafka.binder.certificateStoreDirectory:: +When the truststore or keystore certificate location is given as a non-local file system resource (resources supported by org.springframework.core.io.Resource e.g. CLASSPATH, HTTP, etc.), +the binder copies the resource from the path (which is convertible to org.springframework.core.io.Resource) to a location on the filesystem. +This is true for both broker level certificates (`ssl.truststore.location` and `ssl.keystore.location`) and certificates intended for schema registry (`schema.registry.ssl.truststore.location` and `schema.registry.ssl.keystore.location`). +Keep in mind that the truststore and keystore location paths must be provided under `spring.cloud.stream.kafka.binder.configuration...`. +For example, `spring.cloud.stream.kafka.binder.configuration.ssl.truststore.location`, `spring.cloud.stream.kafka.binder.configuration.schema.registry.ssl.truststore.location`, etc. +The file will be copied to the location specified as the value for this property which must be an existing directory on the filesystem that is writable by the process running the application. +If this value is not set and the certificate file is a non-local file system resource, then it will be copied to System's temp directory as returned by `System.getProperty("java.io.tmpdir")`. +This is also true, if this value is present, but the directory cannot be found on the filesystem or is not writable. ++ +Default: none. + +spring.cloud.stream.kafka.binder.metrics.defaultOffsetLagMetricsEnabled:: +When set to true, the offset lag metric of each consumer topic is computed whenever the metric is accessed. +When set to false only the periodically calculated offset lag is used. ++ +Default: true + +spring.cloud.stream.kafka.binder.metrics.offsetLagMetricsInterval:: +The interval in which the offset lag for each consumer topic is computed. +This value is used whenever `metrics.defaultOffsetLagMetricsEnabled` is disabled or its +computation is taking too long. ++ +Default: 60 seconds + +spring.cloud.stream.kafka.binder.enableObservation:: +Enable Micrometer observation registry on all the bindings in this binder. ++ +Default: false + +[[kafka-consumer-properties]] +== Kafka Consumer Properties + +NOTE: To avoid repetition, Spring Cloud Stream supports setting values for all channels, in the format of `spring.cloud.stream.kafka.default.consumer.=`. + + +The following properties are available for Kafka consumers only and +must be prefixed with `spring.cloud.stream.kafka.bindings..consumer.`. + +admin.configuration:: +Since version 2.1.1, this property is deprecated in favor of `topic.properties`, and support for it will be removed in a future version. + +admin.replicas-assignment:: +Since version 2.1.1, this property is deprecated in favor of `topic.replicas-assignment`, and support for it will be removed in a future version. + +admin.replication-factor:: +Since version 2.1.1, this property is deprecated in favor of `topic.replication-factor`, and support for it will be removed in a future version. + +autoRebalanceEnabled:: +When `true`, topic partitions is automatically rebalanced between the members of a consumer group. +When `false`, each consumer is assigned a fixed set of partitions based on `spring.cloud.stream.instanceCount` and `spring.cloud.stream.instanceIndex`. +This requires both the `spring.cloud.stream.instanceCount` and `spring.cloud.stream.instanceIndex` properties to be set appropriately on each launched instance. +The value of the `spring.cloud.stream.instanceCount` property must typically be greater than 1 in this case. ++ +Default: `true`. +ackEachRecord:: +When `autoCommitOffset` is `true`, this setting dictates whether to commit the offset after each record is processed. +By default, offsets are committed after all records in the batch of records returned by `consumer.poll()` have been processed. +The number of records returned by a poll can be controlled with the `max.poll.records` Kafka property, which is set through the consumer `configuration` property. +Setting this to `true` may cause a degradation in performance, but doing so reduces the likelihood of redelivered records when a failure occurs. +Also, see the binder `requiredAcks` property, which also affects the performance of committing offsets. +This property is deprecated as of 3.1 in favor of using `ackMode`. +If the `ackMode` is not set and batch mode is not enabled, `RECORD` ackMode will be used. ++ +Default: `false`. + +autoCommitOffset:: + +Starting with version 3.1, this property is deprecated. +See `ackMode` for more details on alternatives. +Whether to autocommit offsets when a message has been processed. +If set to `false`, a header with the key `kafka_acknowledgment` of the type `org.springframework.kafka.support.Acknowledgment` header is present in the inbound message. +Applications may use this header for acknowledging messages. +See the examples section for details. +When this property is set to `false`, Kafka binder sets the ack mode to `org.springframework.kafka.listener.AbstractMessageListenerContainer.AckMode.MANUAL` and the application is responsible for acknowledging records. +Also see `ackEachRecord`. ++ +Default: `true`. +ackMode:: +Specify the container ack mode. +This is based on the AckMode enumeration defined in Spring Kafka. +If `ackEachRecord` property is set to `true` and consumer is not in batch mode, then this will use the ack mode of `RECORD`, otherwise, use the provided ack mode using this property. + +autoCommitOnError:: +In pollable consumers, if set to `true`, it always auto commits on error. +If not set (the default) or false, it will not auto commit in pollable consumers. +Note that this property is only applicable for pollable consumers. ++ +Default: not set. +resetOffsets:: +Whether to reset offsets on the consumer to the value provided by startOffset. +Must be false if a `KafkaBindingRebalanceListener` is provided; see <>. +See <> for more information about this property. ++ +Default: `false`. +startOffset:: +The starting offset for new groups. +Allowed values: `earliest` and `latest`. +If the consumer group is set explicitly for the consumer 'binding' (through `spring.cloud.stream.bindings..group`), 'startOffset' is set to `earliest`. Otherwise, it is set to `latest` for the `anonymous` consumer group. +See <> for more information about this property. ++ +Default: null (equivalent to `earliest`). +enableDlq:: +When set to true, it enables DLQ behavior for the consumer. +By default, messages that result in errors are forwarded to a topic named `error..`. +The DLQ topic name can be configurable by setting the `dlqName` property or by defining a `@Bean` of type `DlqDestinationResolver`. +This provides an alternative option to the more common Kafka replay scenario for the case when the number of errors is relatively small and replaying the entire original topic may be too cumbersome. +See <> processing for more information. +Starting with version 2.0, messages sent to the DLQ topic are enhanced with the following headers: `x-original-topic`, `x-exception-message`, and `x-exception-stacktrace` as `byte[]`. +By default, a failed record is sent to the same partition number in the DLQ topic as the original record. +See <> for how to change that behavior. +**Not allowed when `destinationIsPattern` is `true`.** ++ +Default: `false`. +dlqPartitions:: +When `enableDlq` is true, and this property is not set, a dead letter topic with the same number of partitions as the primary topic(s) is created. +Usually, dead-letter records are sent to the same partition in the dead-letter topic as the original record. +This behavior can be changed; see <>. +If this property is set to `1` and there is no `DqlPartitionFunction` bean, all dead-letter records will be written to partition `0`. +If this property is greater than `1`, you **MUST** provide a `DlqPartitionFunction` bean. +Note that the actual partition count is affected by the binder's `minPartitionCount` property. ++ +Default: `none` +configuration:: +Map with a key/value pair containing generic Kafka consumer properties. +In addition to having Kafka consumer properties, other configuration properties can be passed here. +For example some properties needed by the application such as `spring.cloud.stream.kafka.bindings.input.consumer.configuration.foo=bar`. +The `bootstrap.servers` property cannot be set here; use multi-binder support if you need to connect to multiple clusters. ++ +Default: Empty map. +dlqName:: +The name of the DLQ topic to receive the error messages. ++ +Default: null (If not specified, messages that result in errors are forwarded to a topic named `error..`). +dlqProducerProperties:: +Using this, DLQ-specific producer properties can be set. +All the properties available through kafka producer properties can be set through this property. +When native decoding is enabled on the consumer (i.e., useNativeDecoding: true) , the application must provide corresponding key/value serializers for DLQ. +This must be provided in the form of `dlqProducerProperties.configuration.key.serializer` and `dlqProducerProperties.configuration.value.serializer`. ++ +Default: Default Kafka producer properties. +standardHeaders:: +Indicates which standard headers are populated by the inbound channel adapter. +Allowed values: `none`, `id`, `timestamp`, or `both`. +Useful if using native deserialization and the first component to receive a message needs an `id` (such as an aggregator that is configured to use a JDBC message store). ++ +Default: `none` +converterBeanName:: +The name of a bean that implements `RecordMessageConverter`. Used in the inbound channel adapter to replace the default `MessagingMessageConverter`. ++ +Default: `null` +idleEventInterval:: +The interval, in milliseconds, between events indicating that no messages have recently been received. +Use an `ApplicationListener` to receive these events. +See <> for a usage example. ++ +Default: `30000` +destinationIsPattern:: +When true, the destination is treated as a regular expression `Pattern` used to match topic names by the broker. +When true, topics are not provisioned, and `enableDlq` is not allowed, because the binder does not know the topic names during the provisioning phase. +Note, the time taken to detect new topics that match the pattern is controlled by the consumer property `metadata.max.age.ms`, which (at the time of writing) defaults to 300,000ms (5 minutes). +This can be configured using the `configuration` property above. ++ +Default: `false` +topic.properties:: +A `Map` of Kafka topic properties used when provisioning new topics -- for example, `spring.cloud.stream.kafka.bindings.input.consumer.topic.properties.message.format.version=0.9.0.0` ++ +Default: none. +topic.replicas-assignment:: +A Map> of replica assignments, with the key being the partition and the value being the assignments. +Used when provisioning new topics. +See the `NewTopic` Javadocs in the `kafka-clients` jar. ++ +Default: none. +topic.replication-factor:: +The replication factor to use when provisioning topics. Overrides the binder-wide setting. +Ignored if `replicas-assignments` is present. ++ +Default: none (the binder-wide default of -1 is used). +pollTimeout:: +Timeout used for polling in pollable consumers. ++ +Default: 5 seconds. +transactionManager:: +Bean name of a `KafkaAwareTransactionManager` used to override the binder's transaction manager for this binding. +Usually needed if you want to synchronize another transaction with the Kafka transaction, using the `ChainedKafkaTransactionManaager`. +To achieve exactly once consumption and production of records, the consumer and producer bindings must all be configured with the same transaction manager. ++ +Default: none. +txCommitRecovered:: +When using a transactional binder, the offset of a recovered record (e.g. when retries are exhausted and the record is sent to a dead letter topic) will be committed via a new transaction, by default. +Setting this property to `false` suppresses committing the offset of recovered record. ++ +Default: true. +commonErrorHandlerBeanName:: +`CommonErrorHandler` bean name to use per consumer binding. +When present, this user provided `CommonErrorHandler` takes precedence over any other error handlers defined by the binder. +This is a handy way to express error handlers, if the application does not want to use a `ListenerContainerCustomizer` and then check the destination/group combination to set an error handler. ++ +Default: none. + + +[[reset-offsets]] +== Resetting Offsets + +When an application starts, the initial position in each assigned partition depends on two properties `startOffset` and `resetOffsets`. +If `resetOffsets` is `false`, normal Kafka consumer https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset[`auto.offset.reset`] semantics apply. +i.e. If there is no committed offset for a partition for the binding's consumer group, the position is `earliest` or `latest`. +By default, bindings with an explicit `group` use `earliest`, and anonymous bindings (with no `group`) use `latest`. +These defaults can be overridden by setting the `startOffset` binding property. +There will be no committed offset(s) the first time the binding is started with a particular `group`. +The other condition where no committed offset exists is if the offset has been expired. +With modern brokers (since 2.1), and default broker properties, the offsets are expired 7 days after the last member leaves the group. +See the https://kafka.apache.org/documentation/#brokerconfigs_offsets.retention.minutes[`offsets.retention.minutes`] broker property for more information. + +When `resetOffsets` is `true`, the binder applies similar semantics to those that apply when there is no committed offset on the broker, as if this binding has never consumed from the topic; i.e. any current committed offset is ignored. + +Following are two use cases when this might be used. + +1. Consuming from a compacted topic containing key/value pairs. +Set `resetOffsets` to `true` and `startOffset` to `earliest`; the binding will perform a `seekToBeginning` on all newly assigned partitions. + +2. Consuming from a topic containing events, where you are only interested in events that occur while this binding is running. +Set `resetOffsets` to `true` and `startOffset` to `latest`; the binding will perform a `seekToEnd` on all newly assigned partitions. + +IMPORTANT: If a rebalance occurs after the initial assignment, the seeks will only be performed on any newly assigned partitions that were not assigned during the initial assignment. + +For more control over topic offsets, see <>; when a listener is provided, `resetOffsets` should not be set to `true`, otherwise, that will cause an error. + +[[consuming-batches]] +== Consuming Batches + +Starting with version 3.0, when `spring.cloud.stream.bindings..consumer.batch-mode` is set to `true`, all of the records received by polling the Kafka `Consumer` will be presented as a `List` to the listener method. +Otherwise, the method will be called with one record at a time. +The size of the batch is controlled by Kafka consumer properties `max.poll.records`, `fetch.min.bytes`, `fetch.max.wait.ms`; refer to the Kafka documentation for more information. + +Starting with version `4.0.2`, the binder supports DLQ capabilities when consuming in batch mode. +Keep in mind that, when using DLQ on a consumer binding that is in batch mode, all the records received from the previous poll will be delivered to the DLQ topic. + +IMPORTANT: Retry within the binder is not supported when using batch mode, so `maxAttempts` will be overridden to 1. +You can configure a `DefaultErrorHandler` (using a `ListenerContainerCustomizer`) to achieve similar functionality to retry in the binder. +You can also use a manual `AckMode` and call `Ackowledgment.nack(index, sleep)` to commit the offsets for a partial batch and have the remaining records redelivered. +Refer to the https://docs.spring.io/spring-kafka/docs/2.3.15.BUILD-SNAPSHOT/reference/html/#committing-offsets[Spring for Apache Kafka documentation] for more information about these techniques. + +[[kafka-producer-properties]] +== Kafka Producer Properties + +NOTE: To avoid repetition, Spring Cloud Stream supports setting values for all channels, in the format of `spring.cloud.stream.kafka.default.producer.=`. + + +The following properties are available for Kafka producers only and +must be prefixed with `spring.cloud.stream.kafka.bindings..producer.`. + +admin.configuration:: +Since version 2.1.1, this property is deprecated in favor of `topic.properties`, and support for it will be removed in a future version. + +admin.replicas-assignment:: +Since version 2.1.1, this property is deprecated in favor of `topic.replicas-assignment`, and support for it will be removed in a future version. + +admin.replication-factor:: +Since version 2.1.1, this property is deprecated in favor of `topic.replication-factor`, and support for it will be removed in a future version. + +bufferSize:: +Upper limit, in bytes, of how much data the Kafka producer attempts to batch before sending. ++ +Default: `16384`. +sync:: +Whether the producer is synchronous. ++ +Default: `false`. +sendTimeoutExpression:: +A SpEL expression evaluated against the outgoing message used to evaluate the time to wait for ack when synchronous publish is enabled -- for example, `headers['mySendTimeout']`. +The value of the timeout is in milliseconds. +With versions before 3.0, the payload could not be used unless native encoding was being used because, by the time this expression was evaluated, the payload was already in the form of a `byte[]`. +Now, the expression is evaluated before the payload is converted. ++ +Default: `none`. +batchTimeout:: +How long the producer waits to allow more messages to accumulate in the same batch before sending the messages. +(Normally, the producer does not wait at all and simply sends all the messages that accumulated while the previous send was in progress.) A non-zero value may increase throughput at the expense of latency. ++ +Default: `0`. +messageKeyExpression:: +A SpEL expression evaluated against the outgoing message used to populate the key of the produced Kafka message -- for example, `headers['myKey']`. +With versions before 3.0, the payload could not be used unless native encoding was being used because, by the time this expression was evaluated, the payload was already in the form of a `byte[]`. +Now, the expression is evaluated before the payload is converted. +In the case of a regular processor (`Function` or `Function, Message`), if the produced key needs to be same as the incoming key from the topic, this property can be set as below. +`spring.cloud.stream.kafka.bindings..producer.messageKeyExpression: headers['kafka_receivedMessageKey']` +There is an important caveat to keep in mind for reactive functions. +In that case, it is up to the application to manually copy the headers from the incoming messages to outbound messages. +You can set the header, e.g. `myKey` and use `headers['myKey']` as suggested above or, for convenience, simply set the `KafkaHeaders.MESSAGE_KEY` header, and you do not need to set this property at all. ++ +Default: `none`. +headerPatterns:: +A comma-delimited list of simple patterns to match Spring messaging headers to be mapped to the Kafka `Headers` in the `ProducerRecord`. +Patterns can begin or end with the wildcard character (asterisk). +Patterns can be negated by prefixing with `!`. +Matching stops after the first match (positive or negative). +For example `!ask,as*` will pass `ash` but not `ask`. +`id` and `timestamp` are never mapped. ++ +Default: `*` (all headers - except the `id` and `timestamp`) +configuration:: +Map with a key/value pair containing generic Kafka producer properties. +The `bootstrap.servers` property cannot be set here; use multi-binder support if you need to connect to multiple clusters. ++ +Default: Empty map. +topic.properties:: +A `Map` of Kafka topic properties used when provisioning new topics -- for example, `spring.cloud.stream.kafka.bindings.output.producer.topic.properties.message.format.version=0.9.0.0` ++ +topic.replicas-assignment:: +A Map> of replica assignments, with the key being the partition and the value being the assignments. +Used when provisioning new topics. +See the `NewTopic` Javadocs in the `kafka-clients` jar. ++ +Default: none. +topic.replication-factor:: +The replication factor to use when provisioning topics. Overrides the binder-wide setting. +Ignored if `replicas-assignments` is present. ++ +Default: none (the binder-wide default of -1 is used). +useTopicHeader:: +Set to `true` to override the default binding destination (topic name) with the value of the `KafkaHeaders.TOPIC` message header in the outbound message. +If the header is not present, the default binding destination is used. ++ +Default: `false`. +recordMetadataChannel:: +The bean name of a `MessageChannel` to which successful send results should be sent; the bean must exist in the application context. +The message sent to the channel is the sent message (after conversion, if any) with an additional header `KafkaHeaders.RECORD_METADATA`. +The header contains a `RecordMetadata` object provided by the Kafka client; it includes the partition and offset where the record was written in the topic. ++ +`ResultMetadata meta = sendResultMsg.getHeaders().get(KafkaHeaders.RECORD_METADATA, RecordMetadata.class)` ++ +Failed sends go the producer error channel (if configured); see <>. ++ +Default: null. + +NOTE: The Kafka binder uses the `partitionCount` setting of the producer as a hint to create a topic with the given partition count (in conjunction with the `minPartitionCount`, the maximum of the two being the value being used). +Exercise caution when configuring both `minPartitionCount` for a binder and `partitionCount` for an application, as the larger value is used. +If a topic already exists with a smaller partition count and `autoAddPartitions` is disabled (the default), the binder fails to start. +If a topic already exists with a smaller partition count and `autoAddPartitions` is enabled, new partitions are added. +If a topic already exists with a larger number of partitions than the maximum of (`minPartitionCount` or `partitionCount`), the existing partition count is used. + +compression:: +Set the `compression.type` producer property. +Supported values are `none`, `gzip`, `snappy`, `lz4` and `zstd`. +If you override the `kafka-clients` jar to 2.1.0 (or later), as discussed in the https://docs.spring.io/spring-kafka/docs/2.2.x/reference/html/#deps-for-21x[Spring for Apache Kafka documentation], and wish to use `zstd` compression, use `spring.cloud.stream.kafka.bindings..producer.configuration.compression.type=zstd`. ++ +Default: `none`. +transactionManager:: +Bean name of a `KafkaAwareTransactionManager` used to override the binder's transaction manager for this binding. +Usually needed if you want to synchronize another transaction with the Kafka transaction, using the `ChainedKafkaTransactionManaager`. +To achieve exactly once consumption and production of records, the consumer and producer bindings must all be configured with the same transaction manager. ++ +Default: none. + +closeTimeout:: +Timeout in number of seconds to wait for when closing the producer. ++ +Default: `30` + +allowNonTransactional:: +Normally, all output bindings associated with a transactional binder will publish in a new transaction, if one is not already in process. +This property allows you to override that behavior. +If set to true, records published to this output binding will not be run in a transaction, unless one is already in process. ++ +Default: `false` + +[[usage-examples]] +== Usage examples + +In this section, we show the use of the preceding properties for specific scenarios. + +[[example:-setting-ackmode-to-manual-and-relying-on-manual-acknowledgement]] +=== Example: Setting `ackMode` to `MANUAL` and Relying on Manual Acknowledgement + +This example illustrates how one may manually acknowledge offsets in a consumer application. + +This example requires that `spring.cloud.stream.kafka.bindings.input.consumer.ackMode` be set to `MANUAL`. +Use the corresponding input channel name for your example. + +[source] +---- +@SpringBootApplication +public class ManuallyAcknowdledgingConsumer { + + public static void main(String[] args) { + SpringApplication.run(ManuallyAcknowdledgingConsumer.class, args); + } + + @Bean + public Consumer> process() { + return message -> { + Acknowledgment acknowledgment = message.getHeaders().get(KafkaHeaders.ACKNOWLEDGMENT, Acknowledgment.class); + if (acknowledgment != null) { + System.out.println("Acknowledgment provided"); + acknowledgment.acknowledge(); + } + }; +} +---- + +[[example:-security-configuration]] +=== Example: Security Configuration + +Apache Kafka 0.9 supports secure connections between client and brokers. +To take advantage of this feature, follow the guidelines in the https://kafka.apache.org/090/documentation.html#security_configclients[Apache Kafka Documentation] as well as the Kafka 0.9 https://docs.confluent.io/2.0.0/kafka/security.html[security guidelines from the Confluent documentation]. +Use the `spring.cloud.stream.kafka.binder.configuration` option to set security properties for all clients created by the binder. + +For example, to set `security.protocol` to `SASL_SSL`, set the following property: + +[source] +---- +spring.cloud.stream.kafka.binder.configuration.security.protocol=SASL_SSL +---- + +All the other security properties can be set in a similar manner. + +When using Kerberos, follow the instructions in the https://kafka.apache.org/090/documentation.html#security_sasl_clientconfig[reference documentation] for creating and referencing the JAAS configuration. + +Spring Cloud Stream supports passing JAAS configuration information to the application by using a JAAS configuration file and using Spring Boot properties. + +[[using-jaas-configuration-files]] +==== Using JAAS Configuration Files + +The JAAS and (optionally) krb5 file locations can be set for Spring Cloud Stream applications by using system properties. +The following example shows how to launch a Spring Cloud Stream application with SASL and Kerberos by using a JAAS configuration file: + +[source,bash] +---- + java -Djava.security.auth.login.config=/path.to/kafka_client_jaas.conf -jar log.jar \ + --spring.cloud.stream.kafka.binder.brokers=secure.server:9092 \ + --spring.cloud.stream.bindings.input.destination=stream.ticktock \ + --spring.cloud.stream.kafka.binder.configuration.security.protocol=SASL_PLAINTEXT +---- + +[[using-spring-boot-properties]] +==== Using Spring Boot Properties + +As an alternative to having a JAAS configuration file, Spring Cloud Stream provides a mechanism for setting up the JAAS configuration for Spring Cloud Stream applications by using Spring Boot properties. + +The following properties can be used to configure the login context of the Kafka client: + +spring.cloud.stream.kafka.binder.jaas.loginModule:: +The login module name. Not necessary to be set in normal cases. ++ +Default: `com.sun.security.auth.module.Krb5LoginModule`. +spring.cloud.stream.kafka.binder.jaas.controlFlag:: +The control flag of the login module. ++ +Default: `required`. +spring.cloud.stream.kafka.binder.jaas.options:: +Map with a key/value pair containing the login module options. ++ +Default: Empty map. + +The following example shows how to launch a Spring Cloud Stream application with SASL and Kerberos by using Spring Boot configuration properties: + +[source,bash] +---- + java --spring.cloud.stream.kafka.binder.brokers=secure.server:9092 \ + --spring.cloud.stream.bindings.input.destination=stream.ticktock \ + --spring.cloud.stream.kafka.binder.autoCreateTopics=false \ + --spring.cloud.stream.kafka.binder.configuration.security.protocol=SASL_PLAINTEXT \ + --spring.cloud.stream.kafka.binder.jaas.options.useKeyTab=true \ + --spring.cloud.stream.kafka.binder.jaas.options.storeKey=true \ + --spring.cloud.stream.kafka.binder.jaas.options.keyTab=/etc/security/keytabs/kafka_client.keytab \ + --spring.cloud.stream.kafka.binder.jaas.options.principal=kafka-client-1@EXAMPLE.COM +---- + +The preceding example represents the equivalent of the following JAAS file: + +[source] +---- +KafkaClient { + com.sun.security.auth.module.Krb5LoginModule required + useKeyTab=true + storeKey=true + keyTab="/etc/security/keytabs/kafka_client.keytab" + principal="kafka-client-1@EXAMPLE.COM"; +}; +---- + +If the topics required already exist on the broker or will be created by an administrator, autocreation can be turned off and only client JAAS properties need to be sent. + +NOTE: Do not mix JAAS configuration files and Spring Boot properties in the same application. +If the `-Djava.security.auth.login.config` system property is already present, Spring Cloud Stream ignores the Spring Boot properties. + +NOTE: Be careful when using the `autoCreateTopics` and `autoAddPartitions` with Kerberos. +Usually, applications may use principals that do not have administrative rights in Kafka and Zookeeper. +Consequently, relying on Spring Cloud Stream to create/modify topics may fail. +In secure environments, we strongly recommend creating topics and managing ACLs administratively by using Kafka tooling. + +[[multi-binder-configuration-and-jaas]] +==== Multi-binder configuration and JAAS + +When connecting to multiple clusters in which each one requires separate JAAS configuration, then set the JAAS configuration using the property `sasl.jaas.config`. +When this property is present in the application, it takes precedence over the other strategies mentioned above. +See this https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients[KIP-85] for more details. + +For example, if you have two clusters in your application with separate JAAS configuration, then the following is a template that you can use: + +``` +spring.cloud.stream: + binders: + kafka1: + type: kafka + environment: + spring: + cloud: + stream: + kafka: + binder: + brokers: localhost:9092 + configuration.sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"admin-secret\";" + kafka2: + type: kafka + environment: + spring: + cloud: + stream: + kafka: + binder: + brokers: localhost:9093 + configuration.sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user1\" password=\"user1-secret\";" + kafka.binder: + configuration: + security.protocol: SASL_PLAINTEXT + sasl.mechanism: PLAIN +``` + +Note that both the Kafka clusters, and the `sasl.jaas.config` values for each of them are different in the above configuration. + +See this https://github.com/spring-cloud/spring-cloud-stream-samples/tree/main/multi-binder-samples/kafka-multi-binder-jaas[sample application] for more details on how to setup and run such an application. + +[[pause-resume]] +=== Example: Pausing and Resuming the Consumer + +If you wish to suspend consumption but not cause a partition rebalance, you can pause and resume the consumer. +This is facilitated by managing the binding lifecycle as shown in **Binding visualization and control** in the Spring Cloud Stream documentation, using `State.PAUSED` and `State.RESUMED`. + +To resume, you can use an `ApplicationListener` (or `@EventListener` method) to receive `ListenerContainerIdleEvent` instances. +The frequency at which events are published is controlled by the `idleEventInterval` property. + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/consumer-producer-config-customizer.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/consumer-producer-config-customizer.adoc new file mode 100644 index 000000000..412362d21 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/consumer-producer-config-customizer.adoc @@ -0,0 +1,15 @@ +[[consumer-producer-config-customizer]] += Customizing Consumer and Producer configuration + +If you want advanced customization of consumer and producer configuration that is used for creating `ConsumerFactory` and `ProducerFactory` in Kafka, +you can implement the following customizers. + +* ConsumerConfigCustomizer +* ProducerConfigCustomizer + +Both of these interfaces provide a way to configure the config map used for consumer and producer properties. +For example, if you want to gain access to a bean that is defined at the application level, you can inject that in the implementation of the `configure` method. +When the binder discovers that these customizers are available as beans, it will invoke the `configure` method right before creating the consumer and producer factories. + +Both of these interfaces also provide access to both the binding and destination names so that they can be accessed while customizing producer and consumer properties. + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/custom-binder-health-indicator-example.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/custom-binder-health-indicator-example.adoc new file mode 100644 index 000000000..e3067edfa --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/custom-binder-health-indicator-example.adoc @@ -0,0 +1,55 @@ +[[custom-kafka-binder-health-indicator-example]] += Custom kafka Binder Health Indicator Example + +Here is the pseudo-code for writing a custom Kafka binder HealthIndicator. +In this example, we try to override the binder provided Kafka HealthIndicator by specifically checking first for cluster connectivity and then followed by topic-related issues. + + +1. First we need create a custom implementation of the `KafkaBinderHealth` interface. + +``` +public class KafkaBinderHealthImplementation implements KafkaBinderHealth { + @Value("${spring.cloud.bus.destination}") + private String topic; + private final AdminClient client; + + public KafkaBinderHealthImplementation(final KafkaAdmin admin) { + // More about configuring Kafka + // https://docs.spring.io/spring-kafka/reference/html/#configuring-topics + this.client = AdminClient.create(admin.getConfigurationProperties()); + } + + @Override + public Health health() { + if (!checkBrokersConnection()) { + logger.error("Error when connect brokers"); + return Health.down().withDetail("BrokersConnectionError", "Error message").build(); + } + if (!checkTopicConnection()) { + logger.error("Error when trying to connect with specific topic"); + return Health.down().withDetail("TopicError", "Error message with topic name").build(); + } + return Health.up().build(); + } + + public boolean checkBrokersConnection() { + // Your implementation + } + + public boolean checkTopicConnection() { + // Your implementation + } +} +``` + +2. Then we need to create a bean for the custom implementation. + +``` +@Configuration +public class KafkaBinderHealthIndicatorConfiguration { + @Bean + public KafkaBinderHealth kafkaBinderHealthIndicator(final KafkaAdmin admin) { + return new KafkaBinderHealthImplementation(admin); + } +} +``` diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/custom-binder-health-indicator.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/custom-binder-health-indicator.adoc new file mode 100644 index 000000000..f30ea74f5 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/custom-binder-health-indicator.adoc @@ -0,0 +1,24 @@ +[[custom-kafka-binder-health-indicator]] += Custom Kafka Binder Health Indicator + +Kafka binder activates a default health indicator when Spring Boot actuator is on the classpath. +This health indicator checks the health of the binder and any communication issues with the Kafka broker. +If an application wants to disable this default health check implementation and include a custom implementation, then it can provide an implementation for `KafkaBinderHealth` interface. +`KafkaBinderHealth` is a marker interface that extends from `HealthIndicator`. +In the custom implementation, it must provide an implementation for the `health()` method. +The custom implementation must be present in the application configuration as a bean. +When the binder discovers the custom implementation, it will use that instead of the default implementation. +Here is an example of such a custom implementation bean in the application. + +``` +@Bean +public KafkaBinderHealth kafkaBinderHealthIndicator() { + return new KafkaBinderHealth() { + @Override + public Health health() { + // custom implementation details. + } + }; +} +``` + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/error-channels.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/error-channels.adoc new file mode 100644 index 000000000..5af7ac3c4 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/error-channels.adoc @@ -0,0 +1,14 @@ +[[kafka-error-channels]] += Error Channels + +Starting with version 1.3, the binder unconditionally sends exceptions to an error channel for each consumer destination and can also be configured to send async producer send failures to an error channel. +See https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/current/reference/html/spring-cloud-stream.html#spring-cloud-stream-overview-error-handling[this section on error handling] for more information. + +The payload of the `ErrorMessage` for a send failure is a `KafkaSendFailureException` with properties: + +* `failedMessage`: The Spring Messaging `Message` that failed to be sent. +* `record`: The raw `ProducerRecord` that was created from the `failedMessage` + +There is no automatic handling of producer exceptions (such as sending to a <>). +You can consume these exceptions with your own Spring Integration flow. + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/metrics.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/metrics.adoc new file mode 100644 index 000000000..11f2d30cb --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/metrics.adoc @@ -0,0 +1,28 @@ +[[kafka-metrics]] += Kafka Metrics + +Kafka binder module exposes the following metrics: + +`spring.cloud.stream.binder.kafka.offset`: This metric indicates how many messages have not been yet consumed from a given binder's topic by a given consumer group. +The metrics provided are based on the Micrometer library. +The binder creates the `KafkaBinderMetrics` bean if Micrometer is on the classpath and no other such beans provided by the application. +The metric contains the consumer group information, topic and the actual lag in committed offset from the latest offset on the topic. +This metric is particularly useful for providing auto-scaling feedback to a PaaS platform. + +The metric collection behaviour can be configured by setting properties in the `spring.cloud.stream.kafka.binder.metrics` namespace, +refer to the <> for more information. + +You can exclude `KafkaBinderMetrics` from creating the necessary infrastructure like consumers and then reporting the metrics by providing the following component in the application. + +``` +@Component +class NoOpBindingMeters { + NoOpBindingMeters(MeterRegistry registry) { + registry.config().meterFilter( + MeterFilter.denyNameStartsWith(KafkaBinderMetrics.OFFSET_LAG_METRIC_NAME)); + } +} +``` + +More details on how to suppress meters selectively can be found https://micrometer.io/docs/concepts#_meter_filters[here]. + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/overview.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/overview.adoc new file mode 100644 index 000000000..1cc558d92 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/overview.adoc @@ -0,0 +1,17 @@ +[[overview]] += Overview + +The following image shows a simplified diagram of how the Apache Kafka binder operates: + +.Kafka Binder +image::{github-raw}/docs/src/main/asciidoc/images/kafka-binder.png[width=300,scaledwidth="50%"] + +The Apache Kafka Binder implementation maps each destination to an Apache Kafka topic. +The consumer group maps directly to the same Apache Kafka concept. +Partitioning also maps directly to Apache Kafka partitions as well. + +The binder currently uses the Apache Kafka `kafka-clients` version `3.1.0`. +This client can communicate with older brokers (see the Kafka documentation), but certain features may not be available. +For example, with versions earlier than 0.11.x.x, native headers are not supported. +Also, 0.11.x.x does not support the `autoAddPartitions` property. + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/rebalance-listener.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/rebalance-listener.adoc new file mode 100644 index 000000000..ffc80c169 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/rebalance-listener.adoc @@ -0,0 +1,51 @@ +[[rebalance-listener]] += Using a KafkaBindingRebalanceListener + +Applications may wish to seek topics/partitions to arbitrary offsets when the partitions are initially assigned, or perform other operations on the consumer. +Starting with version 2.1, if you provide a single `KafkaBindingRebalanceListener` bean in the application context, it will be wired into all Kafka consumer bindings. + +==== +[source, java] +---- +public interface KafkaBindingRebalanceListener { + + /** + * Invoked by the container before any pending offsets are committed. + * @param bindingName the name of the binding. + * @param consumer the consumer. + * @param partitions the partitions. + */ + default void onPartitionsRevokedBeforeCommit(String bindingName, Consumer consumer, + Collection partitions) { + + } + + /** + * Invoked by the container after any pending offsets are committed. + * @param bindingName the name of the binding. + * @param consumer the consumer. + * @param partitions the partitions. + */ + default void onPartitionsRevokedAfterCommit(String bindingName, Consumer consumer, Collection partitions) { + + } + + /** + * Invoked when partitions are initially assigned or after a rebalance. + * Applications might only want to perform seek operations on an initial assignment. + * @param bindingName the name of the binding. + * @param consumer the consumer. + * @param partitions the partitions. + * @param initial true if this is the initial assignment. + */ + default void onPartitionsAssigned(String bindingName, Consumer consumer, Collection partitions, + boolean initial) { + + } + +} +---- +==== + +You cannot set the `resetOffsets` consumer property to `true` when you provide a rebalance listener. + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/retry-and-dlq-processing.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/retry-and-dlq-processing.adoc new file mode 100644 index 000000000..7ccfbd33f --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/retry-and-dlq-processing.adoc @@ -0,0 +1,103 @@ +[[retry-and-dlq-processing]] += Retry and Dead Letter Processing + +By default, when you configure retry (e.g. `maxAttemts`) and `enableDlq` in a consumer binding, these functions are performed within the binder, with no participation by the listener container or Kafka consumer. + +There are situations where it is preferable to move this functionality to the listener container, such as: + +* The aggregate of retries and delays will exceed the consumer's `max.poll.interval.ms` property, potentially causing a partition rebalance. +* You wish to publish the dead letter to a different Kafka cluster. +* You wish to add retry listeners to the error handler. +* ... + +To configure moving this functionality from the binder to the container, define a `@Bean` of type `ListenerContainerWithDlqAndRetryCustomizer`. +This interface has the following methods: + +==== +[source, java] +---- +/** + * Configure the container. + * @param container the container. + * @param destinationName the destination name. + * @param group the group. + * @param dlqDestinationResolver a destination resolver for the dead letter topic (if + * enableDlq). + * @param backOff the backOff using retry properties (if configured). + * @see #retryAndDlqInBinding(String, String) + */ +void configure(AbstractMessageListenerContainer container, String destinationName, String group, + @Nullable BiFunction, Exception, TopicPartition> dlqDestinationResolver, + @Nullable BackOff backOff); + +/** + * Return false to move retries and DLQ from the binding to a customized error handler + * using the retry metadata and/or a {@code DeadLetterPublishingRecoverer} when + * configured via + * {@link #configure(AbstractMessageListenerContainer, String, String, BiFunction, BackOff)}. + * @param destinationName the destination name. + * @param group the group. + * @return false to disable retries and DLQ in the binding + */ +default boolean retryAndDlqInBinding(String destinationName, String group) { + return true; +} +---- +==== + +The destination resolver and `BackOff` are created from the binding properties (if configured). The `KafkaTemplate` uses configuration from `spring.kafka....` properties. You can then use these to create a custom error handler and dead letter publisher; for example: + +==== +[source, java] +---- +@Bean +ListenerContainerWithDlqAndRetryCustomizer cust(KafkaTemplate template) { + return new ListenerContainerWithDlqAndRetryCustomizer() { + + @Override + public void configure(AbstractMessageListenerContainer container, String destinationName, + String group, + @Nullable BiFunction, Exception, TopicPartition> dlqDestinationResolver, + @Nullable BackOff backOff) { + + if (destinationName.equals("topicWithLongTotalRetryConfig")) { + ConsumerRecordRecoverer dlpr = new DeadLetterPublishingRecoverer(template, + dlqDestinationResolver); + container.setCommonErrorHandler(new DefaultErrorHandler(dlpr, backOff)); + } + } + + @Override + public boolean retryAndDlqInBinding(String destinationName, String group) { + return !destinationName.contains("topicWithLongTotalRetryConfig"); + } + + }; +} +---- +==== + +Now, only a single retry delay needs to be greater than the consumer's `max.poll.interval.ms` property. + +When working with several binders, the 'ListenerContainerWithDlqAndRetryCustomizer' bean gets overridden by the 'DefaultBinderFactory'. For the bean +to apply, you need to use a 'BinderCustomizer' to set the container customizer (See <>): +==== +[source, java] +---- +@Bean +public BinderCustomizer binderCustomizer(ListenerContainerWithDlqAndRetryCustomizer containerCustomizer) { + return (binder, binderName) -> { + if (binder instanceof KafkaMessageChannelBinder kafkaMessageChannelBinder) { + kafkaMessageChannelBinder.setContainerCustomizer(containerCustomizer); + } + else if (binder instanceof KStreamBinder) { + ... + } + else if (binder instanceof RabbitMessageChannelBinder) { + ... + } + }; +} +---- +==== + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/tombstones.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/tombstones.adoc new file mode 100644 index 000000000..a677317f5 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/tombstones.adoc @@ -0,0 +1,25 @@ +[[kafka-tombstones]] += Tombstone Records (null record values) + +When using compacted topics, a record with a `null` value (also called a tombstone record) represents the deletion of a key. +To receive such messages in a Spring Cloud Stream function, you can use the following strategy. + +==== +[source, java] +---- +@Bean +public Function, String> myFunction() { + return value -> { + Object v = value.getPayload(); + String className = v.getClass().getName(); + if (className.isEqualTo("org.springframework.kafka.support.KafkaNull")) { + // this is a tombstone record + } + else { + // continue with processing + } + }; +} +---- +==== + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/transactional-binder.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/transactional-binder.adoc new file mode 100644 index 000000000..dbc760267 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/transactional-binder.adoc @@ -0,0 +1,54 @@ +[[kafka-transactional-binder]] += Transactional Binder + +Enable transactions by setting `spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix` to a non-empty value, e.g. `tx-`. +When used in a processor application, the consumer starts the transaction; any records sent on the consumer thread participate in the same transaction. +When the listener exits normally, the listener container will send the offset to the transaction and commit it. +A common producer factory is used for all producer bindings configured using `spring.cloud.stream.kafka.binder.transaction.producer.*` properties; individual binding Kafka producer properties are ignored. + +IMPORTANT: Normal binder retries (and dead lettering) are not supported with transactions because the retries will run in the original transaction, which may be rolled back and any published records will be rolled back too. +When retries are enabled (the common property `maxAttempts` is greater than zero) the retry properties are used to configure a `DefaultAfterRollbackProcessor` to enable retries at the container level. +Similarly, instead of publishing dead-letter records within the transaction, this functionality is moved to the listener container, again via the `DefaultAfterRollbackProcessor` which runs after the main transaction has rolled back. + +If you wish to use transactions in a source application, or from some arbitrary thread for producer-only transaction (e.g. `@Scheduled` method), you must get a reference to the transactional producer factory and define a `KafkaTransactionManager` bean using it. + +==== +[source, java] +---- +@Bean +public PlatformTransactionManager transactionManager(BinderFactory binders, + @Value("${unique.tx.id.per.instance}") String txId) { + + ProducerFactory pf = ((KafkaMessageChannelBinder) binders.getBinder(null, + MessageChannel.class)).getTransactionalProducerFactory(); + KafkaTransactionManager tm = new KafkaTransactionManager<>(pf); + tm.setTransactionId(txId) + return tm; +} +---- +==== + +Notice that we get a reference to the binder using the `BinderFactory`; use `null` in the first argument when there is only one binder configured. +If more than one binder is configured, use the binder name to get the reference. +Once we have a reference to the binder, we can obtain a reference to the `ProducerFactory` and create a transaction manager. + +Then you would use normal Spring transaction support, e.g. `TransactionTemplate` or `@Transactional`, for example: + +==== +[source, java] +---- +public static class Sender { + + @Transactional + public void doInTransaction(MessageChannel output, List stuffToSend) { + stuffToSend.forEach(stuff -> output.send(new GenericMessage<>(stuff))); + } + +} +---- +==== + +If you wish to synchronize producer-only transactions with those from some other transaction manager, use a `ChainedTransactionManager`. + +IMPORTANT: If you deploy multiple instances of your application, each instance needs a unique `transactionIdPrefix`. + diff --git a/docs/modules/ROOT/pages/kafka/kafka_overview/usage.adoc b/docs/modules/ROOT/pages/kafka/kafka_overview/usage.adoc new file mode 100644 index 000000000..32c182924 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/kafka_overview/usage.adoc @@ -0,0 +1,23 @@ +[[usage]] += Usage + +To use Apache Kafka binder, you need to add `spring-cloud-stream-binder-kafka` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-kafka + +---- + +Alternatively, you can also use the Spring Cloud Stream Kafka Starter, as shown in the following example for Maven: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-starter-stream-kafka + +---- + diff --git a/docs/modules/ROOT/pages/kafka/streams/accessing-metrics.adoc b/docs/modules/ROOT/pages/kafka/streams/accessing-metrics.adoc new file mode 100644 index 000000000..14e02acee --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/accessing-metrics.adoc @@ -0,0 +1,11 @@ +[[accessing-kafka-streams-metrics]] += Accessing Kafka Streams Metrics + +Spring Cloud Stream Kafka Streams binder provides Kafka Streams metrics which can be exported through a Micrometer `MeterRegistry`. + +For Spring Boot version 2.2.x, the metrics support is provided through a custom Micrometer metrics implementation by the binder. +For Spring Boot version 2.3.x, the Kafka Streams metrics support is provided natively through Micrometer. + +When accessing metrics through the Boot actuator endpoint, make sure to add `metrics` to the property `management.endpoints.web.exposure.include`. +Then you can access `/acutator/metrics` to get a list of all the available metrics, which then can be individually accessed through the same URI (`/actuator/metrics/`). + diff --git a/docs/modules/ROOT/pages/kafka/streams/ancillaries-to-the-programming-model.adoc b/docs/modules/ROOT/pages/kafka/streams/ancillaries-to-the-programming-model.adoc new file mode 100644 index 000000000..6ba426930 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/ancillaries-to-the-programming-model.adoc @@ -0,0 +1,143 @@ +[[ancillaries-to-the-programming-model]] += Ancillaries to the programming model + +[[multiple-kafka-streams-processors-within-a-single-application]] +== Multiple Kafka Streams processors within a single application + +Binder allows to have multiple Kafka Streams processors within a single Spring Cloud Stream application. +You can have an application as below. + +``` +@Bean +public java.util.function.Function, KStream> process() { + ... +} + +@Bean +public java.util.function.Consumer> anotherProcess() { + ... +} + +@Bean +public java.util.function.BiFunction, KTable, KStream> yetAnotherProcess() { + ... +} + +``` + +In this case, the binder will create 3 separate Kafka Streams objects with different application ID's (more on this below). +However, if you have more than one processor in the application, you have to tell Spring Cloud Stream, which functions need to be activated. +Here is how you activate the functions. + +`spring.cloud.function.definition: process;anotherProcess;yetAnotherProcess` + +If you want certain functions to be not activated right away, you can remove that from this list. + +This is also true when you have a single Kafka Streams processor and other types of `Function` beans in the same application that is handled through a different binder (for e.g., a function bean that is based on the regular Kafka Message Channel binder) + +[[kafka-streams-application-id]] +== Kafka Streams Application ID + +Application id is a mandatory property that you need to provide for a Kafka Streams application. +Spring Cloud Stream Kafka Streams binder allows you to configure this application id in multiple ways. + +If you only have one single processor in the application, then you can set this at the binder level using the following property: + +`spring.cloud.stream.kafka.streams.binder.applicationId`. + +As a convenience, if you only have a single processor, you can also use `spring.application.name` as the property to delegate the application id. + +If you have multiple Kafka Streams processors in the application, then you need to set the application id per processor. +In the case of the functional model, you can attach it to each function as a property. + +For e.g. imagine that you have the following functions. + +``` +@Bean +public java.util.function.Consumer> process() { + ... +} +``` + +and + +``` +@Bean +public java.util.function.Consumer> anotherProcess() { + ... +} +``` + +Then you can set the application id for each, using the following binder level properties. + +`spring.cloud.stream.kafka.streams.binder.functions.process.applicationId` + +and + +`spring.cloud.stream.kafka.streams.binder.functions.anotherProcess.applicationId` + +For function based model also, this approach of setting application id at the binding level will work. +However, setting per function at the binder level as we have seen above is much easier if you are using the functional model. + +For production deployments, it is highly recommended to explicitly specify the application ID through configuration. +This is especially going to be very critical if you are auto scaling your application in which case you need to make sure that you are deploying each instance with the same application ID. + +If the application does not provide an application ID, then in that case the binder will auto generate a static application ID for you. +This is convenient in development scenarios as it avoids the need for explicitly providing the application ID. +The generated application ID in this manner will be static over application restarts. +In the case of functional model, the generated application ID will be the function bean name followed by the literal `applicationID`, for e.g `process-applicationID` if `process` if the function bean name. + +[[summary-of-setting-application-id]] +==== Summary of setting Application ID + +* By default, binder will auto generate the application ID per function methods. +* If you have a single processor, then you can use `spring.kafka.streams.applicationId`, `spring.application.name` or `spring.cloud.stream.kafka.streams.binder.applicationId`. +* If you have multiple processors, then application ID can be set per function using the property - `spring.cloud.stream.kafka.streams.binder.functions..applicationId`. + +[[overriding-the-default-binding-names-generated-by-the-binder-with-the-functional-style]] +== Overriding the default binding names generated by the binder with the functional style + +By default, the binder uses the strategy discussed above to generate the binding name when using the functional style, i.e. -|-[0..n], for e.g. process-in-0, process-out-0 etc. +If you want to override those binding names, you can do that by specifying the following properties. + +`spring.cloud.stream.function.bindings.`. Default binding name is the original binding name generated by the binder. + +For e.g. lets say, you have this function. + +[source] +---- +@Bean +public BiFunction, KTable, KStream> process() { +... +} +---- + +Binder will generate bindings with names, `process-in-0`, `process-in-1` and `process-out-0`. +Now, if you want to change them to something else completely, maybe more domain specific binding names, then you can do so as below. + +`spring.cloud.stream.function.bindings.process-in-0=users` + +`spring.cloud.stream.function.bindings.process-in-0=regions` + +and + +`spring.cloud.stream.function.bindings.process-out-0=clicks` + +After that, you must set all the binding level properties on these new binding names. + +Please keep in mind that with the functional programming model described above, adhering to the default binding names make sense in most situations. +The only reason you may still want to do this overriding is when you have larger number of configuration properties and you want to map the bindings to something more domain friendly. + +[[setting-up-bootstrap-server-configuration]] +== Setting up bootstrap server configuration + +When running Kafka Streams applications, you must provide the Kafka broker server information. +If you don't provide this information, the binder expects that you are running the broker at the default `localhost:9092`. +If that is not the case, then you need to override that. There are a couple of ways to do that. + +* Using the boot property - `spring.kafka.bootstrapServers` +* Binder level property - `spring.cloud.stream.kafka.streams.binder.brokers` + +When it comes to the binder level property, it doesn't matter if you use the broker property provided through the regular Kafka binder - `spring.cloud.stream.kafka.binder.brokers`. +Kafka Streams binder will first check if Kafka Streams binder specific broker property is set (`spring.cloud.stream.kafka.streams.binder.brokers`) and if not found, it looks for `spring.cloud.stream.kafka.binder.brokers`. + diff --git a/docs/modules/ROOT/pages/kafka/streams/binding-visualization-and-control-in-binder.adoc b/docs/modules/ROOT/pages/kafka/streams/binding-visualization-and-control-in-binder.adoc new file mode 100644 index 000000000..d2823c565 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/binding-visualization-and-control-in-binder.adoc @@ -0,0 +1,113 @@ +[[binding-visualization-and-control-in-kafka-streams-binder]] += Binding visualization and control in Kafka Streams binder + +Starting with version 3.1.2, Kafka Streams binder supports binding visualization and control. +The only two lifecycle phases supported are `STOPPED` and `STARTED`. +The lifecycle phases `PAUSED` and `RESUMED` are not available in Kafka Streams binder. + +In order to activate binding visualization and control, the application needs to include the following two dependencies. + +``` + + org.springframework.boot + spring-boot-starter-actuator + + + org.springframework.boot + spring-boot-starter-web + +``` + +If you prefer using webflux, you can then include `spring-boot-starter-webflux` instead of the standard web dependency. + +In addition, you also need to set the following property: + +``` +management.endpoints.web.exposure.include=bindings +``` + +To illustrate this feature further, let us use the following application as a guide: + +``` +@SpringBootApplication +public class KafkaStreamsApplication { + + public static void main(String[] args) { + SpringApplication.run(KafkaStreamsApplication.class, args); + } + + @Bean + public Consumer> consumer() { + return s -> s.foreach((key, value) -> System.out.println(value)); + } + + @Bean + public Function, KStream> function() { + return ks -> ks; + } + +} +``` + +As we can see, the application has two Kafka Streams functions - one, a consumer and another a function. +The consumer binding is named by default as `consumer-in-0`. +Similarly, for the function, the input binding is `function-in-0` and the output binding is `function-out-0`. + +Once the application is started, we can find details about the bindings using the following bindings endpoint. + +``` + curl http://localhost:8080/actuator/bindings | jq . +[ + { + "bindingName": "consumer-in-0", + "name": "consumer-in-0", + "group": "consumer-applicationId", + "pausable": false, + "state": "running", + "paused": false, + "input": true, + "extendedInfo": {} + }, + { + "bindingName": "function-in-0", + "name": "function-in-0", + "group": "function-applicationId", + "pausable": false, + "state": "running", + "paused": false, + "input": true, + "extendedInfo": {} + }, + { + "bindingName": "function-out-0", + "name": "function-out-0", + "group": "function-applicationId", + "pausable": false, + "state": "running", + "paused": false, + "input": false, + "extendedInfo": {} + } +] +``` + +The details about all three bindings can be found above. + +Let us now stop the consumer-in-0 binding. + +``` +curl -d '{"state":"STOPPED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/consumer-in-0 +``` + +At this point, no records will be received through this binding. + +Start the binding again. + +``` +curl -d '{"state":"STARTED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/consumer-in-0 +``` + +When there are multiple bindings present on a single function, invoking these operations on any of those bindings will work. +This is because all the bindings on a single function are backed by the same `StreamsBuilderFactoryBean`. +Therefore, for the function above, either `function-in-0` or `function-out-0` will work. + diff --git a/docs/modules/ROOT/pages/kafka/streams/configuration-options.adoc b/docs/modules/ROOT/pages/kafka/streams/configuration-options.adoc new file mode 100644 index 000000000..5c524d78b --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/configuration-options.adoc @@ -0,0 +1,209 @@ +[[configuration-options]] += Configuration Options + +This section contains the configuration options used by the Kafka Streams binder. + +For common configuration options and properties pertaining to binder, refer to the <>. + +[[kafka-streams-binder-properties]] +== Kafka Streams Binder Properties + +The following properties are available at the binder level and must be prefixed with `spring.cloud.stream.kafka.streams.binder.` +Any Kafka binder provided properties re-used in Kafka Streams binder must be prefixed with `spring.cloud.stream.kafka.streams.binder` instead of `spring.cloud.stream.kafka.binder`. +The only exception to this rule is when defining the Kafka bootstrap server property in which case either prefix works. + +configuration:: +Map with a key/value pair containing properties pertaining to Apache Kafka Streams API. +This property must be prefixed with `spring.cloud.stream.kafka.streams.binder.`. +Following are some examples of using this property. + +[source] +---- +spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde +spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde +spring.cloud.stream.kafka.streams.binder.configuration.commit.interval.ms=1000 +---- + +For more information about all the properties that may go into streams configuration, see `StreamsConfig` JavaDocs in Apache Kafka Streams docs. +All configuration that you can set from `StreamsConfig` can be set through this. +When using this property, it is applicable against the entire application since this is a binder level property. +If you have more than processors in the application, all of them will acquire these properties. +In the case of properties like `application.id`, this will become problematic and therefore you have to carefully examine how the properties from `StreamsConfig` are mapped using this binder level `configuration` property. + +functions..applicationId:: +Applicable only for functional style processors. +This can be used for setting application ID per function in the application. +In the case of multiple functions, this is a handy way to set the application ID. + +functions..configuration:: +Applicable only for functional style processors. +Map with a key/value pair containing properties pertaining to Apache Kafka Streams API. +This is similar to the binder level `configuration` property describe above, but this level of `configuration` property is restricted only against the named function. +When you have multiple processors and you want to restrict access to the configuration based on particular functions, you might want to use this. +All `StreamsConfig` properties can be used here. + +brokers:: +Broker URL ++ +Default: `localhost` +zkNodes:: +Zookeeper URL ++ +Default: `localhost` + +deserializationExceptionHandler:: +Deserialization error handler type. +This handler is applied at the binder level and thus applied against all input binding in the application. +There is a way to control it in a more fine-grained way at the consumer binding level. +Possible values are - `logAndContinue`, `logAndFail`, `skipAndContinue` or `sendToDlq` ++ +Default: `logAndFail` + +applicationId:: +Convenient way to set the application.id for the Kafka Streams application globally at the binder level. +If the application contains multiple functions, then the application id should be set differently. +See above where setting the application id is discussed in detail. ++ +Default: application will generate a static application ID. See the application ID section for more details. + +stateStoreRetry.maxAttempts:: +Max attempts for trying to connect to a state store. ++ +Default: 1 + +stateStoreRetry.backoffPeriod:: +Backoff period when trying to connect to a state store on a retry. ++ +Default: 1000 ms + +consumerProperties:: +Arbitrary consumer properties at the binder level. + +producerProperties:: +Arbitrary producer properties at the binder level. + +includeStoppedProcessorsForHealthCheck:: +When bindings for processors are stopped through actuator, then this processor will not participate in the health check by default. +Set this property to `true` to enable health check for all processors including the ones that are currently stopped through bindings actuator endpoint. ++ +Default: false + +[[kafka-streams-producer-properties]] +== Kafka Streams Producer Properties + +The following properties are _only_ available for Kafka Streams producers and must be prefixed with `spring.cloud.stream.kafka.streams.bindings..producer.` +For convenience, if there are multiple output bindings and they all require a common value, that can be configured by using the prefix `spring.cloud.stream.kafka.streams.default.producer.`. + +keySerde:: +key serde to use ++ +Default: See the above discussion on message de/serialization + +valueSerde:: +value serde to use ++ +Default: See the above discussion on message de/serialization + +useNativeEncoding:: +flag to enable/disable native encoding ++ +Default: `true`. + +streamPartitionerBeanName:: +Custom outbound partitioner bean name to be used at the consumer. +Applications can provide custom `StreamPartitioner` as a Spring bean and the name of this bean can be provided to the producer to use instead of the default one. ++ +Default: See the discussion above on outbound partition support. + +producedAs:: +Custom name for the sink component to which the processor is producing to. ++ +Deafult: `none` (generated by Kafka Streams) + +[[kafka-streams-consumer-properties]] +== Kafka Streams Consumer Properties + +The following properties are available for Kafka Streams consumers and must be prefixed with `spring.cloud.stream.kafka.streams.bindings..consumer.` +For convenience, if there are multiple input bindings and they all require a common value, that can be configured by using the prefix `spring.cloud.stream.kafka.streams.default.consumer.`. + +applicationId:: +Setting application.id per input binding. ++ +Default: See above. + +keySerde:: +key serde to use ++ +Default: See the above discussion on message de/serialization + +valueSerde:: +value serde to use ++ +Default: See the above discussion on message de/serialization + +materializedAs:: +state store to materialize when using incoming KTable types ++ +Default: `none`. + +useNativeDecoding:: +flag to enable/disable native decoding ++ +Default: `true`. + +dlqName:: +DLQ topic name. ++ +Default: See above on the discussion of error handling and DLQ. + +startOffset:: +Offset to start from if there is no committed offset to consume from. +This is mostly used when the consumer is consuming from a topic for the first time. +Kafka Streams uses `earliest` as the default strategy and the binder uses the same default. +This can be overridden to `latest` using this property. ++ +Default: `earliest`. + +Note: Using `resetOffsets` on the consumer does not have any effect on Kafka Streams binder. +Unlike the message channel based binder, Kafka Streams binder does not seek to beginning or end on demand. + +deserializationExceptionHandler:: +Deserialization error handler type. +This handler is applied per consumer binding as opposed to the binder level property described before. +Possible values are - `logAndContinue`, `logAndFail`, `skipAndContinue` or `sendToDlq` ++ +Default: `logAndFail` + +timestampExtractorBeanName:: +Specific time stamp extractor bean name to be used at the consumer. +Applications can provide `TimestampExtractor` as a Spring bean and the name of this bean can be provided to the consumer to use instead of the default one. ++ +Default: See the discussion above on timestamp extractors. + +eventTypes:: +Comma separated list of supported event types for this binding. ++ +Default: `none` + +eventTypeHeaderKey:: +Event type header key on each incoming records through this binding. ++ +Default: `event_type` + +consumedAs:: +Custom name for the source component from which the processor is consuming from. ++ +Deafult: `none` (generated by Kafka Streams) + +[[special-note-on-concurrency]] +== Special note on concurrency + +In Kafka Streams, you can control of the number of threads a processor can create using the `num.stream.threads` property. +This, you can do using the various `configuration` options described above under binder, functions, producer or consumer level. +You can also use the `concurrency` property that core Spring Cloud Stream provides for this purpose. +When using this, you need to use it on the consumer. +When you have more than one input binding, set this on the first input binding. +For e.g. when setting `spring.cloud.stream.bindings.process-in-0.consumer.concurrency`, it will be translated as `num.stream.threads` by the binder. +If you have multiple processors and one processor defines binding level concurrency, but not the others, those ones with no binding level concurrency will default back to the binder wide property specified through +`spring.cloud.stream.kafka.streams.binder.configuration.num.stream.threads`. +If this binder configuration is not available, then the application will use the default set by Kafka Streams. diff --git a/docs/modules/ROOT/pages/kafka/streams/error-handling.adoc b/docs/modules/ROOT/pages/kafka/streams/error-handling.adoc new file mode 100644 index 000000000..0a20316c3 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/error-handling.adoc @@ -0,0 +1,195 @@ +[[error-handling]] += Error Handling + +Apache Kafka Streams provides the capability for natively handling exceptions from deserialization errors. +For details on this support, please see https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers[this]. +Out of the box, Apache Kafka Streams provides two kinds of deserialization exception handlers - `LogAndContinueExceptionHandler` and `LogAndFailExceptionHandler`. +As the name indicates, the former will log the error and continue processing the next records and the latter will log the error and fail. `LogAndFailExceptionHandler` is the default deserialization exception handler. + +[[handling-deserialization-exceptions-in-the-binder]] +== Handling Deserialization Exceptions in the Binder + +Kafka Streams binder allows to specify the deserialization exception handlers above using the following property. + +[source] +---- +spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler: logAndContinue +---- + +or + +[source] +---- +spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler: logAndFail +---- + +In addition to the above two deserialization exception handlers, the binder also provides a third one for sending the erroneous records (poison pills) to a DLQ (dead letter queue) topic. +Here is how you enable this DLQ exception handler. + +[source] +---- +spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler: sendToDlq +---- + +When the above property is set, all the records in deserialization error are automatically sent to the DLQ topic. + +You can set the topic name where the DLQ messages are published as below. + +You can provide an implementation for `DlqDestinationResolver` which is a functional interface. +`DlqDestinationResolver` takes `ConsumerRecord` and the exception as inputs and then allows to specify a topic name as the output. +By gaining access to the Kafka `ConsumerRecord`, the header records can be introspected in the implementation of the `BiFunction`. + +Here is an example of providing an implementation for `DlqDestinationResolver`. + +[source] +---- +@Bean +public DlqDestinationResolver dlqDestinationResolver() { + return (rec, ex) -> { + if (rec.topic().equals("word1")) { + return "topic1-dlq"; + } + else { + return "topic2-dlq"; + } + }; +} +---- + +One important thing to keep in mind when providing an implementation for `DlqDestinationResolver` is that the provisioner in the binder will not auto create topics for the application. +This is because there is no way for the binder to infer the names of all the DLQ topics the implementation might send to. +Therefore, if you provide DLQ names using this strategy, it is the application's responsibility to ensure that those topics are created beforehand. + +If `DlqDestinationResolver` is present in the application as a bean, that takes higher precedence. +If you do not want to follow this approach and rather provide a static DLQ name using configuration, you can set the following property. + +[source] +---- +spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.dlqName: custom-dlq (Change the binding name accordingly) +---- + +If this is set, then the error records are sent to the topic `custom-dlq`. +If the application is not using either of the above strategies, then it will create a DLQ topic with the name `error..`. +For instance, if your binding's destination topic is `inputTopic` and the application ID is `process-applicationId`, then the default DLQ topic is `error.inputTopic.process-applicationId`. +It is always recommended to explicitly create a DLQ topic for each input binding if it is your intention to enable DLQ. + +[[dlq-per-input-consumer-binding]] +== DLQ per input consumer binding + +The property `spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler` is applicable for the entire application. +This implies that if there are multiple functions in the same application, this property is applied to all of them. +However, if you have multiple processors or multiple input bindings within a single processor, then you can use the finer-grained DLQ control that the binder provides per input consumer binding. + +If you have the following processor, + +``` +@Bean +public BiFunction, KTable, KStream> process() { +... +} +``` + +and you only want to enable DLQ on the first input binding and skipAndContinue on the second binding, then you can do so on the consumer as below. + +`spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.deserializationExceptionHandler: sendToDlq` +`spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.deserializationExceptionHandler: skipAndContinue` + +Setting deserialization exception handlers this way has a higher precedence than setting at the binder level. + +[[dlq-partitioning]] +== DLQ partitioning + +By default, records are published to the Dead-Letter topic using the same partition as the original record. +This means the Dead-Letter topic must have at least as many partitions as the original record. + +To change this behavior, add a `DlqPartitionFunction` implementation as a `@Bean` to the application context. +Only one such bean can be present. +The function is provided with the consumer group (which is the same as the application ID in most situations), the failed `ConsumerRecord` and the exception. +For example, if you always want to route to partition 0, you might use: + + +[source, java] +---- +@Bean +public DlqPartitionFunction partitionFunction() { + return (group, record, ex) -> 0; +} +---- + +NOTE: If you set a consumer binding's `dlqPartitions` property to 1 (and the binder's `minPartitionCount` is equal to `1`), there is no need to supply a `DlqPartitionFunction`; the framework will always use partition 0. +If you set a consumer binding's `dlqPartitions` property to a value greater than `1` (or the binder's `minPartitionCount` is greater than `1`), you **must** provide a `DlqPartitionFunction` bean, even if the partition count is the same as the original topic's. + + +A couple of things to keep in mind when using the exception handling feature in Kafka Streams binder. + +* The property `spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler` is applicable for the entire application. +This implies that if there are multiple functions in the same application, this property is applied to all of them. +* The exception handling for deserialization works consistently with native deserialization and framework provided message conversion. + +[[handling-production-exceptions-in-the-binder]] +== Handling Production Exceptions in the Binder + +Unlike the support for deserialization exception handlers as described above, the binder does not provide such first class mechanisms for handling production exceptions. +However, you still can configure production exception handlers using the `StreamsBuilderFactoryBean` customizer which you can find more details about, in a subsequent section below. + +[[runtime-error-handling]] +== Runtime Error Handling + +When it comes to handling errors from application code, i.e. from the business logic execution, it is usually up to the application to handle that. +Because, the Kafka Streams binder does not have a way to interfere with the application code. +However, to make things a bit easier for the application, the binder provides a convenient `DltAwareProcessor`, using which, you can dictate how you want to handle the application level errors. + +Consider the following code. + +``` +@Bean +public java.util.function.Function, KStream> process() { + return input -> input + .map(...); +} +``` + +If the business code inside your `map` call above throws an exception, it is your responsibility to handle that error. +This is where `DltAwareProcessor` becomes handy. +Let's say that you want to publish the failed record to a DLT, rather than handling it within the application. +Here is how you can do that. + +``` +@Bean +public java.util.function.Function, KStream> process(DltPublishingContext dltSenderContext) { + return input -> input + .process(() -> new DltAwareProcessor<>((k, v) -> { + throw new RuntimeException("error"); + }, "hello-dlt-1", dltPublishingContext)); +} +``` + +The business logic code from the original `map` call now has been moved as part of `KStream#process` method call, which takes a `ProcessorSupplier`. +We, then, pass in the custom `DltAwareProcessor,` which is capable to publishing to a DLT. +The constructor for `DltAwareProcessor` above takes three parameters - a `BiFunction` that takes the key and value of the input record and then the business logic operation as part of the `BiFunction` body, the DLT topic, and finally a `DltPublishingContext`. When the `BiFunction`'s lambda expression throws an exception, the `DltAwareProcessor` will send the input record to a DLT. The `DltPublishingContext` provides `DltAwareProcessor` the necessary publishing infrastructure beans. +The `DltPublishingContext` is autoconfigured by the binder, so that you can inject directly this into the application. + +If you want to provide a custom timestamp on the record that gets published to the DLT, then you can provide an optional fourth constructor argument which is a `Supplier`. +If this value is provided, then this `Supplier` is invoked each time `DltAwareProcessor` publishes to the DLT. + +If you do not want the binder to publish failed records to a DLT, then you can provide your own recoverer as a `BiConsumer`. +Assume a scenario, in which you do not want to send the record to the DLT, but simply log the message and move on. +For this, it is convenient, if we can override the recovery process in `DltAwareProcessor`. +Here is an example of how you do that. + +``` +@Bean +public java.util.function.Function, KStream> process() { + return input -> input + .process(() -> new DltAwareProcessor<>((k, v) -> { + throw new RuntimeException("error"); + }, + (record, exception) -> { + // log the message + })); +} +``` + +In this case, when the record fails, the `DltAwareProcessor`, instead of using its built-in recoverer which publishes to a DLT, uses the user provided recoverer which is a `BiConsumer` that takes the failed record and the exception thrown as arguments. +In this case also, you can provide an optional `Supplier` to dictate the timestamp used in the record passed in to the `BiConsumer` recoverer. + diff --git a/docs/modules/ROOT/pages/kafka/streams/event-type-based-routing-in-applications.adoc b/docs/modules/ROOT/pages/kafka/streams/event-type-based-routing-in-applications.adoc new file mode 100644 index 000000000..b2cbe5af0 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/event-type-based-routing-in-applications.adoc @@ -0,0 +1,43 @@ +[[event-type-based-routing-in-kafka-streams-applications]] += Event type based routing in Kafka Streams applications + +Routing functions available in regular message channel based binders are not supported in Kafka Streams binder. +However, Kafka Streams binder still provides routing capabilities through the event type record header on the inbound records. + +To enable routing based on event types, the application must provide the following property. + +`spring.cloud.stream.kafka.streams.bindings..consumer.eventTypes`. + +This can be a comma separated value. + +For example, lets assume we have this function: + +``` +@Bean +public Function, KStream> process() { + return input -> input; +} +``` + +Let us also assume that we only want the business logic in this function to be executed, if the incoming record has event types as `foo` or `bar`. +That can be expressed as below using the `eventTypes` property on the binding. + +`spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.eventTypes=foo,bar` + +Now, when the application runs, the binder checks each incoming records for the header `event_type` and see if it has value set as `foo` or `bar`. +If it does not find either of them, then the function execution will be skipped. + +By default, the binder expects the record header key to be `event_type`, but that can be changed per binding. +For instance, if we want to change the header key on this binding to `my_event` instead of the default, that can be changed as below. + +`spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.eventTypeHeaderKey=my_event`. + +When using the event routing feature in Kafkfa Streams binder, it uses the byte array `Serde` to deserialze all incoming records. +If the record headers match the event type, then only it uses the actual `Serde` to do a proper deserialization using either the configured or the inferred `Serde`. +This introduces issues if you set a deserialization exception handler on the binding as the expected deserialization only happens down the stack causing unexpected errors. +In order to address this issue, you can set the following property on the binding to force the binder to use the configured or inferred `Serde` instead of byte array `Serde`. + +`spring.cloud.stream.kafka.streams.bindings..consumer.useConfiguredSerdeWhenRoutingEvents` + +This way, the application can detect deserialization issues right away when using the event routing feature and can take appropriate handling decisions. + diff --git a/docs/modules/ROOT/pages/kafka/streams/health-indicator.adoc b/docs/modules/ROOT/pages/kafka/streams/health-indicator.adoc new file mode 100644 index 000000000..871d711c3 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/health-indicator.adoc @@ -0,0 +1,34 @@ +[[health-indicator]] += Health Indicator + +The health indicator requires the dependency `spring-boot-starter-actuator`. For maven use: +[source,xml] +---- + + org.springframework.boot + spring-boot-starter-actuator + +---- + +Spring Cloud Stream Kafka Streams Binder provides a health indicator to check the state of the underlying streams threads. +Spring Cloud Stream defines a property `management.health.binders.enabled` to enable the health indicator. See the +https://docs.spring.io/spring-cloud-stream/docs/current/reference/htmlsingle/#_health_indicator[Spring Cloud Stream documentation]. + +The health indicator provides the following details for each stream thread's metadata: + +* Thread name +* Thread state: `CREATED`, `RUNNING`, `PARTITIONS_REVOKED`, `PARTITIONS_ASSIGNED`, `PENDING_SHUTDOWN` or `DEAD` +* Active tasks: task ID and partitions +* Standby tasks: task ID and partitions + +By default, only the global status is visible (`UP` or `DOWN`). To show the details, the property `management.endpoint.health.show-details` must be set to `ALWAYS` or `WHEN_AUTHORIZED`. +For more details about the health information, see the +https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html#production-ready-health[Spring Boot Actuator documentation]. + +NOTE: The status of the health indicator is `UP` if all the Kafka threads registered are in the `RUNNING` state. + +Since there are three individual binders in Kafka Streams binder (`KStream`, `KTable` and `GlobalKTable`), all of them will report the health status. +When enabling `show-details`, some of the information reported may be redundant. + +When there are multiple Kafka Streams processors present in the same application, then the health checks will be reported for all of them and will be categorized by the application ID of Kafka Streams. + diff --git a/docs/modules/ROOT/pages/kafka/streams/interactive-queries.adoc b/docs/modules/ROOT/pages/kafka/streams/interactive-queries.adoc new file mode 100644 index 000000000..9a97d4914 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/interactive-queries.adoc @@ -0,0 +1,92 @@ +[[interactive-queries]] += Interactive Queries + +Kafka Streams binder API exposes a class called `InteractiveQueryService` to interactively query the state stores. +You can access this as a Spring bean in your application. An easy way to get access to this bean from your application is to `autowire` the bean. + +[source] +---- +@Autowired +private InteractiveQueryService interactiveQueryService; +---- + +Once you gain access to this bean, then you can query for the particular state-store that you are interested. See below. + +[source] +---- +ReadOnlyKeyValueStore keyValueStore = + interactiveQueryService.getQueryableStoreType("my-store", QueryableStoreTypes.keyValueStore()); +---- + +During the startup, the above method call to retrieve the store might fail. +For example, it might still be in the middle of initializing the state store. +In such cases, it will be useful to retry this operation. +Kafka Streams binder provides a simple retry mechanism to accommodate this. + +Following are the two properties that you can use to control this retrying. + +* spring.cloud.stream.kafka.streams.binder.stateStoreRetry.maxAttempts - Default is `1` . +* spring.cloud.stream.kafka.streams.binder.stateStoreRetry.backOffInterval - Default is `1000` milliseconds. + +If there are multiple instances of the kafka streams application running, then before you can query them interactively, you need to identify which application instance hosts the particular key that you are querying. +`InteractiveQueryService` API provides methods for identifying the host information. + +In order for this to work, you must configure the property `application.server` as below: + +[source] +---- +spring.cloud.stream.kafka.streams.binder.configuration.application.server: : +---- + +Here are some code snippets: + +[source] +---- +org.apache.kafka.streams.state.HostInfo hostInfo = interactiveQueryService.getHostInfo("store-name", + key, keySerializer); + +if (interactiveQueryService.getCurrentHostInfo().equals(hostInfo)) { + + //query from the store that is locally available +} +else { + //query from the remote host +} +---- + +For more information on these host finding methods, please see the Javadoc on the methods. +For these methods also, during startup, if the underlying KafkaStreams objects are not ready, they might throw exceptions. +The aforementioned retry properties are applicable for these methods as well. + +[[other-api-methods-available-through-the-interactivequeryservice]] +== Other API methods available through the InteractiveQueryService + +Use the following API method to retrieve the `KeyQueryMetadata` object associated with the combination of given store and key. + +``` +public KeyQueryMetadata getKeyQueryMetadata(String store, K key, Serializer serializer) +``` + +Use the following API method to retrieve the `KakfaStreams` object associated with the combination of given store and key. + +``` +public KafkaStreams getKafkaStreams(String store, K key, Serializer serializer) +``` + +[[customizing-store-query-parameters]] +== Customizing Store Query Parameters + +Sometimes it is necessary that you need to fine tune the store query parameters before querying the store through `InteractiveQueryService`. +For this purpose, starting with the `4.0.1` version of the binder, you can provide a bean for `StoreQueryParametersCustomizer` which is a functional interface with a `customize` method that takes a `StoreQueryParameter` as the argument. +Here is its method signature. + +``` +StoreQueryParameters customize(StoreQueryParameters storeQueryParameters); +``` + +Using this approach, applications can further customize the `StoreQueryParameters` such as enabling stale stores. + +When this bean is present in this application, `InteractiveQueryService` will call its `customize` method before querying the state store. + +NOTE: Keep in mind that, there must be a unique bean for `StoreQueryParametersCustomizer` available in the application. + diff --git a/docs/modules/ROOT/pages/kafka/streams/manually-starting-processors-selectively.adoc b/docs/modules/ROOT/pages/kafka/streams/manually-starting-processors-selectively.adoc new file mode 100644 index 000000000..3b19c7f98 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/manually-starting-processors-selectively.adoc @@ -0,0 +1,73 @@ +[[manually-starting-kafka-streams-processors-selectively]] += Manually starting Kafka Streams processors selectively + +While the approach laid out above will unconditionally apply auto start `false` to all the Kafka Streams processors in the application through `StreamsBuilderFactoryManager`, it is often desirable that only individually selected Kafka Streams processors are not auto started. +For instance, let us assume that you have three different functions (processors) in your application and for one of the processors, you do not want to start it as part of the application startup. +Here is an example of such a situation. + +``` + +@Bean +public Function, KStream> process1() { + +} + +@Bean +public Consumer> process2() { + +} + +@Bean +public BiFunction, KTable, KStream> process3() { + +} + +``` + +In this scenario above, if you set `spring.kafka.streams.auto-startup` to `false`, then none of the processors will auto start during the application startup. +In that case, you have to programmatically start them as described above by calling `start()` on the underlying `StreamsBuilderFactoryManager`. +However, if we have a use case to selectively disable only one processor, then you have to set `auto-startup` on the individual binding for that processor. +Let us assume that we don't want our `process3` function to auto start. +This is a `BiFunction` with two input bindings - `process3-in-0` and `process3-in-1`. +In order to avoid auto start for this processor, you can pick any of these input bindings and set `auto-startup` on them. +It does not matter which binding you pick; if you wish, you can set `auto-startup` to `false` on both of them, but one will be sufficient. +Because they share the same factory bean, you don't have to set autoStartup to false on both bindings, but it probably makes sense to do so, for clarity. + +Here is the Spring Cloud Stream property that you can use to disable auto startup for this processor. + +``` +spring.cloud.stream.bindings.process3-in-0.consumer.auto-startup: false +``` + +or + +``` +spring.cloud.stream.bindings.process3-in-1.consumer.auto-startup: false +``` + +Then, you can manually start the processor either using the REST endpoint or using the `BindingsEndpoint` API as shown below. +For this, you need to ensure that you have the Spring Boot actuator dependency on the classpath. + +``` +curl -d '{"state":"STARTED"}' -H "Content-Type: application/json" -X POST http://localhost:8080/actuator/bindings/process3-in-0 +``` + +or + +``` +@Autowired +BindingsEndpoint endpoint; + +@Bean +public ApplicationRunner runner() { + return args -> { + endpoint.changeState("process3-in-0", State.STARTED); + }; +} +``` + +See https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/spring-cloud-stream.html#binding_visualization_control[this section] from the reference docs for more details on this mechanism. + +NOTE: When controlling the bindings by disabling `auto-startup` as described in this section, please note that this is only available for consumer bindings. +In other words, if you use the producer binding, `process3-out-0`, that does not have any effect in terms of disabling the auto starting of the processor, although this producer binding uses the same `StreamsBuilderFactoryBean` as the consumer bindings. + diff --git a/docs/modules/ROOT/pages/kafka/streams/manually-starting-processors.adoc b/docs/modules/ROOT/pages/kafka/streams/manually-starting-processors.adoc new file mode 100644 index 000000000..a0406f57b --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/manually-starting-processors.adoc @@ -0,0 +1,25 @@ +[[manually-starting-kafka-streams-processors]] += Manually starting Kafka Streams processors + +Spring Cloud Stream Kafka Streams binder offers an abstraction called `StreamsBuilderFactoryManager` on top of the `StreamsBuilderFactoryBean` from Spring for Apache Kafka. +This manager API is used for controlling the multiple `StreamsBuilderFactoryBean` per processor in a binder based application. +Therefore, when using the binder, if you manually want to control the auto starting of the various `StreamsBuilderFactoryBean` objects in the application, you need to use `StreamsBuilderFactoryManager`. +You can use the property `spring.kafka.streams.auto-startup` and set this to `false` in order to turn off auto starting of the processors. +Then, in the application, you can use something as below to start the processors using `StreamsBuilderFactoryManager`. + +``` +@Bean +public ApplicationRunner runner(StreamsBuilderFactoryManager sbfm) { + return args -> { + sbfm.start(); + }; +} +``` + +This feature is handy, when you want your application to start in the main thread and let Kafka Streams processors start separately. +For example, when you have a large state store that needs to be restored, if the processors are started normally as is the default case, this may block your application to start. +If you are using some sort of liveness probe mechanism (for example on Kubernetes), it may think that the application is down and attempt a restart. +In order to correct this, you can set `spring.kafka.streams.auto-startup` to `false` and follow the approach above. + +Keep in mind that, when using the Spring Cloud Stream binder, you are not directly dealing with `StreamsBuilderFactoryBean` from Spring for Apache Kafka, rather `StreamsBuilderFactoryManager`, as the `StreamsBuilderFactoryBean` objects are internally managed by the binder. + diff --git a/docs/modules/ROOT/pages/kafka/streams/mixing-high-level-dsl-and-low-level-processor-api.adoc b/docs/modules/ROOT/pages/kafka/streams/mixing-high-level-dsl-and-low-level-processor-api.adoc new file mode 100644 index 000000000..69670c5dd --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/mixing-high-level-dsl-and-low-level-processor-api.adoc @@ -0,0 +1,63 @@ +[[mixing-high-level-dsl-and-low-level-processor-api]] += Mixing high level DSL and low level Processor API + +Kafka Streams provides two variants of APIs. +It has a higher level DSL like API where you can chain various operations that maybe familiar to a lot of functional programmers. +Kafka Streams also gives access to a low level Processor API. +The processor API, although very powerful and gives the ability to control things in a much lower level, is imperative in nature. +Kafka Streams binder for Spring Cloud Stream, allows you to use either the high level DSL or mixing both the DSL and the processor API. +Mixing both of these variants give you a lot of options to control various use cases in an application. +Applications can use the `transform` or `process` method API calls to get access to the processor API. + +Here is a look at how one may combine both the DSL and the processor API in a Spring Cloud Stream application using the `process` API. + +``` +@Bean +public Consumer> process() { + return input -> + input.process(() -> new Processor() { + @Override + @SuppressWarnings("unchecked") + public void init(ProcessorContext context) { + this.context = context; + } + + @Override + public void process(Object key, String value) { + //business logic + } + + @Override + public void close() { + + }); +} +``` + +Here is an example using the `transform` API. + +``` +@Bean +public Consumer> process() { + return (input, a) -> + input.transform(() -> new Transformer>() { + @Override + public void init(ProcessorContext context) { + + } + + @Override + public void close() { + + } + + @Override + public KeyValue transform(Object key, String value) { + // business logic - return transformed KStream; + } + }); +} +``` + +The `process` API method call is a terminal operation while the `transform` API is non terminal and gives you a potentially transformed `KStream` using which you can continue further processing using either the DSL or the processor API. + diff --git a/docs/modules/ROOT/pages/kafka/streams/multi-binders-with-based-binders-and-regular-binder.adoc b/docs/modules/ROOT/pages/kafka/streams/multi-binders-with-based-binders-and-regular-binder.adoc new file mode 100644 index 000000000..cd6d2411f --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/multi-binders-with-based-binders-and-regular-binder.adoc @@ -0,0 +1,101 @@ +[[multi-binders-with-kafka-streams-based-binders-and-regular-kafka-binder]] += Multi binders with Kafka Streams based binders and regular Kafka Binder + +You can have an application where you have both a function/consumer/supplier that is based on the regular Kafka binder and a Kafka Streams based processor. +However, you cannot mix both of them within a single function or consumer. + +Here is an example, where you have both binder based components within the same application. + +``` +@Bean +public Function process() { + return s -> s; +} + +@Bean +public Function, KStream> kstreamProcess() { + + return input -> input; +} + +``` + +This is the relevant parts from the configuration: + +``` +spring.cloud.function.definition=process;kstreamProcess +spring.cloud.stream.bindings.process-in-0.destination=foo +spring.cloud.stream.bindings.process-out-0.destination=bar +spring.cloud.stream.bindings.kstreamProcess-in-0.destination=bar +spring.cloud.stream.bindings.kstreamProcess-out-0.destination=foobar +``` + +Things become a bit more complex if you have the same application as above, but is dealing with two different Kafka clusters, for e.g. the regular `process` is acting upon both Kafka cluster 1 and cluster 2 (receiving data from cluster-1 and sending to cluster-2) and the Kafka Streams processor is acting upon Kafka cluster 2. +Then you have to use the https://cloud.spring.io/spring-cloud-stream/reference/html/spring-cloud-stream.html#multiple-binders[multi binder] facilities provided by Spring Cloud Stream. + +Here is how your configuration may change in that scenario. + +``` +# multi binder configuration +spring.cloud.stream.binders.kafka1.type: kafka +spring.cloud.stream.binders.kafka1.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-1} #Replace kafkaCluster-1 with the approprate IP of the cluster +spring.cloud.stream.binders.kafka2.type: kafka +spring.cloud.stream.binders.kafka2.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} #Replace kafkaCluster-2 with the approprate IP of the cluster +spring.cloud.stream.binders.kafka3.type: kstream +spring.cloud.stream.binders.kafka3.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} #Replace kafkaCluster-2 with the approprate IP of the cluster + +spring.cloud.function.definition=process;kstreamProcess + +# From cluster 1 to cluster 2 with regular process function +spring.cloud.stream.bindings.process-in-0.destination=foo +spring.cloud.stream.bindings.process-in-0.binder=kafka1 # source from cluster 1 +spring.cloud.stream.bindings.process-out-0.destination=bar +spring.cloud.stream.bindings.process-out-0.binder=kafka2 # send to cluster 2 + +# Kafka Streams processor on cluster 2 +spring.cloud.stream.bindings.kstreamProcess-in-0.destination=bar +spring.cloud.stream.bindings.kstreamProcess-in-0.binder=kafka3 +spring.cloud.stream.bindings.kstreamProcess-out-0.destination=foobar +spring.cloud.stream.bindings.kstreamProcess-out-0.binder=kafka3 +``` + +Pay attention to the above configuration. +We have two kinds of binders, but 3 binders all in all, first one is the regular Kafka binder based on cluster 1 (`kafka1`), then another Kafka binder based on cluster 2 (`kafka2`) and finally the `kstream` one (`kafka3`). +The first processor in the application receives data from `kafka1` and publishes to `kafka2` where both binders are based on regular Kafka binder but differnt clusters. +The second processor, which is a Kafka Streams processor consumes data from `kafka3` which is the same cluster as `kafka2`, but a different binder type. + +Since there are three different binder types available in the Kafka Streams family of binders - `kstream`, `ktable` and `globalktable` - if your application has multiple bindings based on any of these binders, that needs to be explicitly provided as the binder type. + +For e.g if you have a processor as below, + +``` +@Bean +public Function, + Function, + Function, KStream>>> enrichOrder() { + + ... +} +``` + +then, this has to be configured in a multi binder scenario as the following. +Please note that this is only needed if you have a true multi-binder scenario where there are multiple processors dealing with multiple clusters within a single application. +In that case, the binders need to be explicitly provided with the bindings to distinguish from other processor's binder types and clusters. + +``` +spring.cloud.stream.binders.kafka1.type: kstream +spring.cloud.stream.binders.kafka1.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} +spring.cloud.stream.binders.kafka2.type: ktable +spring.cloud.stream.binders.kafka2.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} +spring.cloud.stream.binders.kafka3.type: globalktable +spring.cloud.stream.binders.kafka3.environment.spring.cloud.stream.kafka.streams.binder.brokers=${kafkaCluster-2} + +spring.cloud.stream.bindings.enrichOrder-in-0.binder=kafka1 #kstream +spring.cloud.stream.bindings.enrichOrder-in-1.binder=kafka2 #ktablr +spring.cloud.stream.bindings.enrichOrder-in-2.binder=kafka3 #globalktable +spring.cloud.stream.bindings.enrichOrder-out-0.binder=kafka1 #kstream + +# rest of the configuration is omitted. + +``` + diff --git a/docs/modules/ROOT/pages/kafka/streams/overview.adoc b/docs/modules/ROOT/pages/kafka/streams/overview.adoc new file mode 100644 index 000000000..f7b8b73a9 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/overview.adoc @@ -0,0 +1,16 @@ +[[overview]] += Overview + +Spring Cloud Stream includes a binder implementation designed explicitly for https://kafka.apache.org/documentation/streams/[Apache Kafka Streams] binding. +With this native integration, a Spring Cloud Stream "processor" application can directly use the +https://kafka.apache.org/documentation/streams/developer-guide[Apache Kafka Streams] APIs in the core business logic. + +Kafka Streams binder implementation builds on the foundations provided by the https://docs.spring.io/spring-kafka/reference/html/#kafka-streams[Spring for Apache Kafka] project. + +Kafka Streams binder provides binding capabilities for the three major types in Kafka Streams - `KStream`, `KTable` and `GlobalKTable`. + +Kafka Streams applications typically follow a model in which the records are read from an inbound topic, apply business logic, and then write the transformed records to an outbound topic. +Alternatively, a Processor application with no outbound destination can be defined as well. + +In the following sections, we are going to look at the details of Spring Cloud Stream's integration with Kafka Streams. + diff --git a/docs/modules/ROOT/pages/kafka/streams/partition-support-on-the-outbound.adoc b/docs/modules/ROOT/pages/kafka/streams/partition-support-on-the-outbound.adoc new file mode 100644 index 000000000..8b3d50aeb --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/partition-support-on-the-outbound.adoc @@ -0,0 +1,59 @@ +[[partition-support-on-the-outbound]] += Partition support on the outbound + +A Kafka Streams processor usually sends the processed output into an outbound Kafka topic. +If the outbound topic is partitioned and the processor needs to send the outgoing data into particular partitions, the applications needs to provide a bean of type `StreamPartitioner`. +See https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/processor/StreamPartitioner.html[StreamPartitioner] for more details. +Let's see some examples. + +This is the same processor we already saw multiple times, + +``` +@Bean +public Function, KStream> process() { + + ... +} +``` + +Here is the output binding destination: + +``` +spring.cloud.stream.bindings.process-out-0.destination: outputTopic +``` + +If the topic `outputTopic` has 4 partitions, if you don't provide a partitioning strategy, Kafka Streams will use default partitioning strategy which may not be the outcome you want depending on the particular use case. +Let's say, you want to send any key that matches to `spring` to partition 0, `cloud` to partition 1, `stream` to partition 2, and everything else to partition 3. +This is what you need to do in the application. + +``` +@Bean +public StreamPartitioner streamPartitioner() { + return (t, k, v, n) -> { + if (k.equals("spring")) { + return 0; + } + else if (k.equals("cloud")) { + return 1; + } + else if (k.equals("stream")) { + return 2; + } + else { + return 3; + } + }; +} +``` + +This is a rudimentary implementation, however, you have access to the key/value of the record, the topic name and the total number of partitions. +Therefore, you can implement complex partitioning strategies if need be. + +You also need to provide this bean name along with the application configuration. + +``` +spring.cloud.stream.kafka.streams.bindings.process-out-0.producer.streamPartitionerBeanName: streamPartitioner +``` + +Each output topic in the application needs to be configured separately like this. + diff --git a/docs/modules/ROOT/pages/kafka/streams/programming-model.adoc b/docs/modules/ROOT/pages/kafka/streams/programming-model.adoc new file mode 100644 index 000000000..8059c189c --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/programming-model.adoc @@ -0,0 +1,410 @@ +[[programming-model]] += Programming Model + +When using the programming model provided by Kafka Streams binder, both the high-level https://docs.confluent.io/current/streams/developer-guide/dsl-api.html[Streams DSL] and a mix of both the higher level and the lower level https://docs.confluent.io/current/streams/developer-guide/processor-api.html[Processor-API] can be used as options. +When mixing both higher and lower level API's, this is usually achieved by invoking `transform` or `process` API methods on `KStream`. + +[[functional-style]] +== Functional Style + +Starting with Spring Cloud Stream `3.0.0`, Kafka Streams binder allows the applications to be designed and developed using the functional programming style that is available in Java 8. +This means that the applications can be concisely represented as a lambda expression of types `java.util.function.Function` or `java.util.function.Consumer`. + +Let's take a very basic example. + +[source] +---- +@SpringBootApplication +public class SimpleConsumerApplication { + + @Bean + public java.util.function.Consumer> process() { + + return input -> + input.foreach((key, value) -> { + System.out.println("Key: " + key + " Value: " + value); + }); + } +} +---- + +Albeit simple, this is a complete standalone Spring Boot application that is leveraging Kafka Streams for stream processing. +This is a consumer application with no outbound binding and only a single inbound binding. +The application consumes data and it simply logs the information from the `KStream` key and value on the standard output. +The application contains the `SpringBootApplication` annotation and a method that is marked as `Bean`. +The bean method is of type `java.util.function.Consumer` which is parameterized with `KStream`. +Then in the implementation, we are returning a Consumer object that is essentially a lambda expression. +Inside the lambda expression, the code for processing the data is provided. + +In this application, there is a single input binding that is of type `KStream`. +The binder creates this binding for the application with a name `process-in-0`, i.e. the name of the function bean name followed by a dash character (`-`) and the literal `in` followed by another dash and then the ordinal position of the parameter. +You use this binding name to set other properties such as destination. +For example, `spring.cloud.stream.bindings.process-in-0.destination=my-topic`. + +NOTE: If the destination property is not set on the binding, a topic is created with the same name as the binding (if there are sufficient privileges for the application) or that topic is expected to be already available. + +Once built as a uber-jar (e.g., `kstream-consumer-app.jar`), you can run the above example like the following. + +If the applications choose to define the functional beans using Spring's `Component` annotation, the binder also supports that model. +The above functional bean could be rewritten as below. + +``` +@Component(name = "process") +public class SimpleConsumer implements java.util.function.Consumer> { + + @Override + public void accept(KStream input) { + input.foreach((key, value) -> { + System.out.println("Key: " + key + " Value: " + value); + }); + } +} +``` + +[source] +---- +java -jar kstream-consumer-app.jar --spring.cloud.stream.bindings.process-in-0.destination=my-topic +---- + +Here is another example, where it is a full processor with both input and output bindings. +This is the classic word-count example in which the application receives data from a topic, the number of occurrences for each word is then computed in a tumbling time-window. + +[source] +---- +@SpringBootApplication +public class WordCountProcessorApplication { + + @Bean + public Function, KStream> process() { + + return input -> input + .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+"))) + .map((key, value) -> new KeyValue<>(value, value)) + .groupByKey(Serialized.with(Serdes.String(), Serdes.String())) + .windowedBy(TimeWindows.of(5000)) + .count(Materialized.as("word-counts-state-store")) + .toStream() + .map((key, value) -> new KeyValue<>(key.key(), new WordCount(key.key(), value, + new Date(key.window().start()), new Date(key.window().end())))); + } + + public static void main(String[] args) { + SpringApplication.run(WordCountProcessorApplication.class, args); + } +} +---- + +Here again, this is a complete Spring Boot application. The difference here from the first application is that the bean method is of type `java.util.function.Function`. +The first parameterized type for the `Function` is for the input `KStream` and the second one is for the output. +In the method body, a lambda expression is provided that is of type `Function` and as implementation, the actual business logic is given. +Similar to the previously discussed Consumer based application, the input binding here is named as `process-in-0` by default. For the output, the binding name is automatically also set to `process-out-0`. + +Once built as an uber-jar (e.g., `wordcount-processor.jar`), you can run the above example like the following. + +[source] +---- +java -jar wordcount-processor.jar --spring.cloud.stream.bindings.process-in-0.destination=words --spring.cloud.stream.bindings.process-out-0.destination=counts +---- + +This application will consume messages from the Kafka topic `words` and the computed results are published to an output +topic `counts`. + +Spring Cloud Stream will ensure that the messages from both the incoming and outgoing topics are automatically bound as +KStream objects. As a developer, you can exclusively focus on the business aspects of the code, i.e. writing the logic +required in the processor. Setting up Kafka Streams specific configuration required by the Kafka Streams infrastructure +is automatically handled by the framework. + +The two examples we saw above have a single `KStream` input binding. In both cases, the bindings received the records from a single topic. +If you want to multiplex multiple topics into a single `KStream` binding, you can provide comma separated Kafka topics as destinations below. + +`spring.cloud.stream.bindings.process-in-0.destination=topic-1,topic-2,topic-3` + +In addition, you can also provide topic patterns as destinations if you want to match topics against a regular exression. + +`spring.cloud.stream.bindings.process-in-0.destination=input.*` + +[[multiple-input-bindings]] +=== Multiple Input Bindings + +Many non-trivial Kafka Streams applications often consume data from more than one topic through multiple bindings. +For instance, one topic is consumed as `Kstream` and another as `KTable` or `GlobalKTable`. +There are many reasons why an application might want to receive data as a table type. +Think of a use-case where the underlying topic is populated through a change data capture (CDC) mechanism from a database or perhaps the application only cares about the latest updates for downstream processing. +If the application specifies that the data needs to be bound as `KTable` or `GlobalKTable`, then Kafka Streams binder will properly bind the destination to a `KTable` or `GlobalKTable` and make them available for the application to operate upon. +We will look at a few different scenarios how multiple input bindings are handled in the Kafka Streams binder. + +[[bifunction-in-kafka-streams-binder]] +==== BiFunction in Kafka Streams Binder + +Here is an example where we have two inputs and an output. In this case, the application can leverage on `java.util.function.BiFunction`. + +[source] +---- +@Bean +public BiFunction, KTable, KStream> process() { + return (userClicksStream, userRegionsTable) -> (userClicksStream + .leftJoin(userRegionsTable, (clicks, region) -> new RegionWithClicks(region == null ? + "UNKNOWN" : region, clicks), + Joined.with(Serdes.String(), Serdes.Long(), null)) + .map((user, regionWithClicks) -> new KeyValue<>(regionWithClicks.getRegion(), + regionWithClicks.getClicks())) + .groupByKey(Grouped.with(Serdes.String(), Serdes.Long())) + .reduce(Long::sum) + .toStream()); +} +---- + +Here again, the basic theme is the same as in the previous examples, but here we have two inputs. +Java's `BiFunction` support is used to bind the inputs to the desired destinations. +The default binding names generated by the binder for the inputs are `process-in-0` and `process-in-1` respectively. The default output binding is `process-out-0`. +In this example, the first parameter of `BiFunction` is bound as a `KStream` for the first input and the second parameter is bound as a `KTable` for the second input. + +[[biconsumer-in-kafka-streams-binder]] +==== BiConsumer in Kafka Streams Binder + +If there are two inputs, but no outputs, in that case we can use `java.util.function.BiConsumer` as shown below. + +[source] +---- +@Bean +public BiConsumer, KTable> process() { + return (userClicksStream, userRegionsTable) -> {} +} +---- + +[[beyond-two-inputs]] +==== Beyond two inputs + +What if you have more than two inputs? +There are situations in which you need more than two inputs. In that case, the binder allows you to chain partial functions. +In functional programming jargon, this technique is generally known as currying. +With the functional programming support added as part of Java 8, Java now enables you to write curried functions. +Spring Cloud Stream Kafka Streams binder can make use of this feature to enable multiple input bindings. + +Let's see an example. + +[source] +---- +@Bean +public Function, + Function, + Function, KStream>>> enrichOrder() { + + return orders -> ( + customers -> ( + products -> ( + orders.join(customers, + (orderId, order) -> order.getCustomerId(), + (order, customer) -> new CustomerOrder(customer, order)) + .join(products, + (orderId, customerOrder) -> customerOrder + .productId(), + (customerOrder, product) -> { + EnrichedOrder enrichedOrder = new EnrichedOrder(); + enrichedOrder.setProduct(product); + enrichedOrder.setCustomer(customerOrder.customer); + enrichedOrder.setOrder(customerOrder.order); + return enrichedOrder; + }) + ) + ) + ); +} +---- + +Let's look at the details of the binding model presented above. +In this model, we have 3 partially applied functions on the inbound. Let's call them as `f(x)`, `f(y)` and `f(z)`. +If we expand these functions in the sense of true mathematical functions, it will look like these: `f(x) -> (fy) -> f(z) -> KStream`. +The `x` variable stands for `KStream`, the `y` variable stands for `GlobalKTable` and the `z` variable stands for `GlobalKTable`. +The first function `f(x)` has the first input binding of the application (`KStream`) and its output is the function, f(y). +The function `f(y)` has the second input binding for the application (`GlobalKTable`) and its output is yet another function, `f(z)`. +The input for the function `f(z)` is the third input for the application (`GlobalKTable`) and its output is `KStream` which is the final output binding for the application. +The input from the three partial functions which are `KStream`, `GlobalKTable`, `GlobalKTable` respectively are available for you in the method body for implementing the business logic as part of the lambda expression. + +Input bindings are named as `enrichOrder-in-0`, `enrichOrder-in-1` and `enrichOrder-in-2` respectively. Output binding is named as `enrichOrder-out-0`. + +With curried functions, you can virtually have any number of inputs. However, keep in mind that, anything more than a smaller number of inputs and partially applied functions for them as above in Java might lead to unreadable code. +Therefore if your Kafka Streams application requires more than a reasonably smaller number of input bindings, and you want to use this functional model, then you may want to rethink your design and decompose the application appropriately. + +[[output-bindings]] +=== Output Bindings + +Kafka Streams binder allows types of either `KStream` or `KTable` as output bindings. +Behind the scenes, the binder uses the `to` method on `KStream` to send the resultant records to the output topic. +If the application provides a `KTable` as output in the function, the binder still uses this technique by delegating to the `to` method of `KStream`. + +For example both functions below will work: + +``` +@Bean +public Function, KTable> foo() { + return KStream::toTable; + }; +} + +@Bean +public Function, KStream> bar() { + return KTable::toStream; +} +``` + +[[multiple-output-bindings]] +=== Multiple Output Bindings + +Kafka Streams allows writing outbound data into multiple topics. This feature is known as branching in Kafka Streams. +When using multiple output bindings, you need to provide an array of KStream (`KStream[]`) as the outbound return type. + +Here is an example: + +[source] +---- +@Bean +public Function, KStream[]> process() { + + Predicate isEnglish = (k, v) -> v.word.equals("english"); + Predicate isFrench = (k, v) -> v.word.equals("french"); + Predicate isSpanish = (k, v) -> v.word.equals("spanish"); + + return input -> { + final Map> stringKStreamMap = input + .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+"))) + .groupBy((key, value) -> value) + .windowedBy(TimeWindows.of(Duration.ofSeconds(5))) + .count(Materialized.as("WordCounts-branch")) + .toStream() + .map((key, value) -> new KeyValue<>(null, new WordCount(key.key(), value, + new Date(key.window().start()), new Date(key.window().end())))) + .split() + .branch(isEnglish) + .branch(isFrench) + .branch(isSpanish) + .noDefaultBranch(); + + return stringKStreamMap.values().toArray(new KStream[0]); + }; +} +---- + +The programming model remains the same, however the outbound parameterized type is `KStream[]`. +The default output binding names are `process-out-0`, `process-out-1`, `process-out-2` respectively for the function above. +The reason why the binder generates three output bindings is because it detects the length of the returned `KStream` array as three. +Note that in this example, we provide a `noDefaultBranch()`; if we have used `defaultBranch()` instead, that would have required an extra output binding, essentially returning a `KStream` array of length four. + +[[summary-of-function-based-programming-styles-for-kafka-streams]] +=== Summary of Function based Programming Styles for Kafka Streams + +In summary, the following table shows the various options that can be used in the functional paradigm. + +|=== +|Number of Inputs |Number of Outputs |Component to use + +|1|0|java.util.function.Consumer +|2|0|java.util.function.BiConsumer +|1|1..n |java.util.function.Function +|2|1..n |java.util.function.BiFunction +|>= 3 |0..n |Use curried functions + +|=== + +* In the case of more than one output in this table, the type simply becomes `KStream[]`. + +[[function-composition-in-kafka-streams-binder]] +=== Function composition in Kafka Streams binder + +Kafka Streams binder supports minimal forms of functional composition for linear topologies. +Using the Java functional API support, you can write multiple functions and then compose them on your own using the `andThen` method. +For example, assume that you have the following two functions. + +``` +@Bean +public Function, KStream> foo() { + return input -> input.peek((s, s2) -> {}); +} + +@Bean +public Function, KStream> bar() { + return input -> input.peek((s, s2) -> {}); +} +``` + +Even without the functional composition support in the binder, you can compose these two functions as below. + +``` +@Bean +public Function, KStream> composed() { + foo().andThen(bar()); +} +``` + +Then you can provide definitions of the form `spring.cloud.function.definition=foo;bar;composed`. +With the functional composition support in the binder, you don't need to write this third function in which you are doing explicit function composition. + +You can simply do this instead: + +``` +spring.cloud.function.definition=foo|bar +``` + +You can even do this: + +``` +spring.cloud.function.definition=foo|bar;foo;bar +``` + +The composed function's default binding names in this example becomes `foobar-in-0` and `foobar-out-0`. + +[[limitations-of-functional-composition-in-kafka-streams-bincer]] +==== Limitations of functional composition in Kafka Streams bincer + +When you have `java.util.function.Function` bean, that can be composed with another function or multiple functions. +The same function bean can be composed with a `java.util.function.Consumer` as well. In this case, consumer is the last component composed. +A function can be composed with multiple functions, then end with a `java.util.function.Consumer` bean as well. + +When composing the beans of type `java.util.function.BiFunction`, the `BiFunction` must be the first function in the definition. +The composed entities must be either of type `java.util.function.Function` or `java.util.funciton.Consumer`. +In other words, you cannot take a `BiFunction` bean and then compose with another `BiFunction`. + +You cannot compose with types of `BiConsumer` or definitions where `Consumer` is the first component. +You cannot also compose with functions where the output is an array (`KStream[]` for branching) unless this is the last component in the definition. + +The very first `Function` of `BiFunction` in the function definition may use a curried form also. +For example, the following is possible. + +``` +@Bean +public Function, Function, KStream>> curriedFoo() { + return a -> b -> + a.join(b, (value1, value2) -> value1 + value2); +} + +@Bean +public Function, KStream> bar() { + return input -> input.mapValues(value -> value + "From-anotherFooFunc"); +} +``` + +and the function definition could be `curriedFoo|bar`. +Behind the scenes, the binder will create two input bindings for the curried function, and an output binding based on the final function in the definition. +The default input bindings in this case are going to be `curriedFoobar-in-0` and `curriedFoobar-in-1`. +The default output binding for this example becomes `curriedFoobar-out-0`. + +[[special-note-on-using-ktable-as-output-in-function-composition]] +==== Special note on using `KTable` as output in function composition + +Lets say you have the following two functions. + +``` +@Bean +public Function, KTable> foo() { + return KStream::toTable; + }; +} + +@Bean +public Function, KStream> bar() { + return KTable::toStream; +} +``` + +You can compose them as `foo|bar`, but keep in mind that the second function (`bar` in this case) must have a `KTable` as input since the first function (`foo`) has `KTable` as output. + diff --git a/docs/modules/ROOT/pages/kafka/streams/record-serialization-and-deserialization.adoc b/docs/modules/ROOT/pages/kafka/streams/record-serialization-and-deserialization.adoc new file mode 100644 index 000000000..9533b913f --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/record-serialization-and-deserialization.adoc @@ -0,0 +1,140 @@ +[[record-serialization-and-deserialization]] += Record serialization and deserialization + +Kafka Streams binder allows you to serialize and deserialize records in two ways. +One is the native serialization and deserialization facilities provided by Kafka and the other one is the message conversion capabilities of Spring Cloud Stream framework. +Lets look at some details. + +[[inbound-deserialization]] +== Inbound deserialization + +Keys are always deserialized using native Serdes. + +For values, by default, deserialization on the inbound is natively performed by Kafka. +Please note that this is a major change on default behavior from previous versions of Kafka Streams binder where the deserialization was done by the framework. + +Kafka Streams binder will try to infer matching `Serde` types by looking at the type signature of `java.util.function.Function|Consumer`. +Here is the order that it matches the Serdes. + +* If the application provides a bean of type `Serde` and if the return type is parameterized with the actual type of the incoming key or value type, then it will use that `Serde` for inbound deserialization. +For e.g. if you have the following in the application, the binder detects that the incoming value type for the `KStream` matches with a type that is parameterized on a `Serde` bean. +It will use that for inbound deserialization. + + +``` +@Bean +public Serde customSerde() { + ... +} + +@Bean +public Function, KStream> process() { +} +``` + +* Next, it looks at the types and see if they are one of the types exposed by Kafka Streams. If so, use them. +Here are the Serde types that the binder will try to match from Kafka Streams. + + Integer, Long, Short, Double, Float, byte[], UUID and String. + +* If none of the Serdes provided by Kafka Streams don't match the types, then it will use JsonSerde provided by Spring Kafka. In this case, the binder assumes that the types are JSON friendly. +This is useful if you have multiple value objects as inputs since the binder will internally infer them to correct Java types. +Before falling back to the `JsonSerde` though, the binder checks at the default `Serde` s set in the Kafka Streams configuration to see if it is a `Serde` that it can match with the incoming KStream's types. + +If none of the above strategies worked, then the applications must provide the `Serde` s through configuration. +This can be configured in two ways - binding or default. + +First the binder will look if a `Serde` is provided at the binding level. +For e.g. if you have the following processor, + +``` +@Bean +public BiFunction, KTable, KStream> process() {...} +``` + +then, you can provide a binding level `Serde` using the following: + +``` +spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.keySerde=CustomKeySerde +spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde + +spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.keySerde=CustomKeySerde +spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde +``` + +NOTE: If you provide `Serde` as abover per input binding, then that will takes higher precedence and the binder will stay away from any `Serde` inference. + +If you want the default key/value Serdes to be used for inbound deserialization, you can do so at the binder level. + +``` +spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde +spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde +``` + +If you don't want the native decoding provided by Kafka, you can rely on the message conversion features that Spring Cloud Stream provides. +Since native decoding is the default, in order to let Spring Cloud Stream deserialize the inbound value object, you need to explicitly disable native decoding. + +For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process-in-0.consumer.nativeDecoding: false` +You need to disable native decoding for all the inputs individually. Otherwise, native decoding will still be applied for those you do not disable. + +By default, Spring Cloud Stream will use `application/json` as the content type and use an appropriate json message converter. +You can use custom message converters by using the following property and an appropriate `MessageConverter` bean. +``` +spring.cloud.stream.bindings.process-in-0.contentType +``` + +[[outbound-serialization]] +== Outbound serialization + +Outbound serialization pretty much follows the same rules as above for inbound deserialization. +As with the inbound deserialization, one major change from the previous versions of Spring Cloud Stream is that the serialization on the outbound is handled by Kafka natively. +Before 3.0 versions of the binder, this was done by the framework itself. + +Keys on the outbound are always serialized by Kafka using a matching `Serde` that is inferred by the binder. +If it can't infer the type of the key, then that needs to be specified using configuration. + +Value serdes are inferred using the same rules used for inbound deserialization. +First it matches to see if the outbound type is from a provided bean in the application. +If not, it checks to see if it matches with a `Serde` exposed by Kafka such as - `Integer`, `Long`, `Short`, `Double`, `Float`, `byte[]`, `UUID` and `String`. +If that doesnt't work, then it falls back to `JsonSerde` provided by the Spring Kafka project, but first look at the default `Serde` configuration to see if there is a match. +Keep in mind that all these happen transparently to the application. +If none of these work, then the user has to provide the `Serde` to use by configuration. + +Lets say you are using the same `BiFunction` processor as above. Then you can configure outbound key/value Serdes as following. + +``` +spring.cloud.stream.kafka.streams.bindings.process-out-0.producer.keySerde=CustomKeySerde +spring.cloud.stream.kafka.streams.bindings.process-out-0.producer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde +``` + +If Serde inference fails, and no binding level Serdes are provided, then the binder falls back to the `JsonSerde`, but look at the default Serdes for a match. + +Default serdes are configured in the same way as above where it is described under deserialization. + +`spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde` +`spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde` + +If your application uses the branching feature and has multiple output bindings, then these have to be configured per binding. +Once again, if the binder is capable of inferring the `Serde` types, you don't need to do this configuration. + +If you don't want the native encoding provided by Kafka, but want to use the framework provided message conversion, then you need to explicitly disable native encoding since since native encoding is the default. +For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process-out-0.producer.nativeEncoding: false` +You need to disable native encoding for all the output individually in the case of branching. Otherwise, native encoding will still be applied for those you don't disable. + +When conversion is done by Spring Cloud Stream, by default, it will use `application/json` as the content type and use an appropriate json message converter. +You can use custom message converters by using the following property and a corresponding `MessageConverter` bean. +``` +spring.cloud.stream.bindings.process-out-0.contentType +``` + +When native encoding/decoding is disabled, binder will not do any inference as in the case of native Serdes. +Applications need to explicitly provide all the configuration options. +For that reason, it is generally advised to stay with the default options for de/serialization and stick with native de/serialization provided by Kafka Streams when you write Spring Cloud Stream Kafka Streams applications. +The one scenario in which you must use message conversion capabilities provided by the framework is when your upstream producer is using a specific serialization strategy. +In that case, you want to use a matching deserialization strategy as native mechanisms may fail. +When relying on the default `Serde` mechanism, the applications must ensure that the binder has a way forward with correctly map the inbound and outbound with a proper `Serde`, as otherwise things might fail. + +It is worth to mention that the data de/serialization approaches outlined above are only applicable on the edges of your processors, i.e. - inbound and outbound. +Your business logic might still need to call Kafka Streams API's that explicitly need `Serde` objects. +Those are still the responsibility of the application and must be handled accordingly by the developer. + diff --git a/docs/modules/ROOT/pages/kafka/streams/retrying-critical-business-logic.adoc b/docs/modules/ROOT/pages/kafka/streams/retrying-critical-business-logic.adoc new file mode 100644 index 000000000..4083957f4 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/retrying-critical-business-logic.adoc @@ -0,0 +1,105 @@ +[[retrying-critical-business-logic]] += Retrying critical business logic + +There are scenarios in which you might want to retry parts of your business logic that are critical to the application. +There maybe an external call to a relational database or invoking a REST endpoint from the Kafka Streams processor. +These calls can fail for various reasons such as network issues or remote service unavailability. +More often, these failures may self resolve if you can try them again. +By default, Kafka Streams binder creates `RetryTemplate` beans for all the input bindings. + +If the function has the following signature, +``` +@Bean +public java.util.function.Consumer> process() +``` +and with default binding name, the `RetryTemplate` will be registered as `process-in-0-RetryTemplate`. +This is following the convention of binding name (`process-in-0`) followed by the literal `-RetryTemplate`. +In the case of multiple input bindings, there will be a separate `RetryTemplate` bean available per binding. +If there is a custom `RetryTemplate` bean available in the application and provided through `spring.cloud.stream.bindings..consumer.retryTemplateName`, then that takes precedence over any input binding level retry template configuration properties. + +Once the `RetryTemplate` from the binding is injected into the application, it can be used to retry any critical sections of the application. +Here is an example: + +``` +@Bean +public java.util.function.Consumer> process(@Lazy @Qualifier("process-in-0-RetryTemplate") RetryTemplate retryTemplate) { + + return input -> input + .process(() -> new Processor() { + @Override + public void init(ProcessorContext processorContext) { + } + + @Override + public void process(Object o, String s) { + retryTemplate.execute(context -> { + //Critical business logic goes here. + }); + } + + @Override + public void close() { + } + }); +} +``` + +Or you can use a custom `RetryTemplate` as below. + +``` +@EnableAutoConfiguration +public static class CustomRetryTemplateApp { + + @Bean + @StreamRetryTemplate + RetryTemplate fooRetryTemplate() { + RetryTemplate retryTemplate = new RetryTemplate(); + + RetryPolicy retryPolicy = new SimpleRetryPolicy(4); + FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy(); + backOffPolicy.setBackOffPeriod(1); + + retryTemplate.setBackOffPolicy(backOffPolicy); + retryTemplate.setRetryPolicy(retryPolicy); + + return retryTemplate; + } + + @Bean + public java.util.function.Consumer> process() { + + return input -> input + .process(() -> new Processor() { + @Override + public void init(ProcessorContext processorContext) { + } + + @Override + public void process(Object o, String s) { + fooRetryTemplate().execute(context -> { + //Critical business logic goes here. + }); + + } + + @Override + public void close() { + } + }); + } +} +``` + +Note that when retries are exhausted, by default, the last exception will be thrown, causing the processor to terminate. +If you wish to handle the exception and continue processing, you can add a RecoveryCallback to the `execute` method: +Here is an example. +``` +retryTemplate.execute(context -> { + //Critical business logic goes here. + }, context -> { + //Recovery logic goes here. + return null; + )); +``` +Refer to the https://github.com/spring-projects/spring-retry[Spring Retry] project for more information about the RetryTemplate, retry policies, backoff policies and more. + diff --git a/docs/modules/ROOT/pages/kafka/streams/state-cleanup.adoc b/docs/modules/ROOT/pages/kafka/streams/state-cleanup.adoc new file mode 100644 index 000000000..25fca8011 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/state-cleanup.adoc @@ -0,0 +1,8 @@ +[[state-cleanup]] += State Cleanup + +By default, no local state is cleaned up when the binding is stopped. +This is the same behavior effective from Spring Kafka version 2.7. +See https://docs.spring.io/spring-kafka/reference/html/#streams-config[Spring Kafka documentation] for more details. +To modify this behavior simply add a single `CleanupConfig` `@Bean` (configured to clean up on start, stop, or neither) to the application context; the bean will be detected and wired into the factory bean. + diff --git a/docs/modules/ROOT/pages/kafka/streams/state-store.adoc b/docs/modules/ROOT/pages/kafka/streams/state-store.adoc new file mode 100644 index 000000000..0e4b96149 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/state-store.adoc @@ -0,0 +1,69 @@ +[[state-store]] += State Store + +State stores are created automatically by Kafka Streams when the high level DSL is used and appropriate calls are made those trigger a state store. + +If you want to materialize an incoming `KTable` binding as a named state store, then you can do so by using the following strategy. + +Lets say you have the following function. + +[source] +---- +@Bean +public BiFunction, KTable, KStream> process() { + ... +} +---- + +Then by setting the following property, the incoming `KTable` data will be materialized in to the named state store. + +[source] +---- +spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.materializedAs: incoming-store +---- + +You can define custom state stores as beans in your application and those will be detected and added to the Kafka Streams builder by the binder. +Especially when the processor API is used, you need to register a state store manually. +In order to do so, you can create the StateStore as a bean in the application. +Here are examples of defining such beans. + +[source] +---- +@Bean +public StoreBuilder myStore() { + return Stores.keyValueStoreBuilder( + Stores.persistentKeyValueStore("my-store"), Serdes.Long(), + Serdes.Long()); +} + +@Bean +public StoreBuilder otherStore() { + return Stores.windowStoreBuilder( + Stores.persistentWindowStore("other-store", + 1L, 3, 3L, false), Serdes.Long(), + Serdes.Long()); +} +---- + +These state stores can be then accessed by the applications directly. + +During the bootstrap, the above beans will be processed by the binder and passed on to the Streams builder object. + +Accessing the state store: +[source] +---- +Processor() { + + WindowStore state; + + @Override + public void init(ProcessorContext processorContext) { + state = (WindowStore)processorContext.getStateStore("mystate"); + } + ... +} +---- + +This will not work when it comes to registering global state stores. +In order to register a global state store, please see the section below on customizing `StreamsBuilderFactoryBean`. + diff --git a/docs/modules/ROOT/pages/kafka/streams/streamsbuilderfactorybean-customizer.adoc b/docs/modules/ROOT/pages/kafka/streams/streamsbuilderfactorybean-customizer.adoc new file mode 100644 index 000000000..83b7fe869 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/streamsbuilderfactorybean-customizer.adoc @@ -0,0 +1,113 @@ +[[streamsbuilderfactorybean-customizer]] += StreamsBuilderFactoryBean customizer + +It is often required to customize the `StreamsBuilderFactoryBean` that creates the `KafkaStreams` objects. +Based on the underlying support provided by Spring Kafka, the binder allows you to customize the `StreamsBuilderFactoryBean`. +You can use the `StreamsBuilderFactoryBeanCustomizer` to customize the `StreamsBuilderFactoryBean` itself. +Then, once you get access to the `StreamsBuilderFactoryBean` through this customizer, you can customize the corresponding `KafkaStreams` using `KafkaStreamsCustomzier`. +Both of these customizers are part of the Spring for Apache Kafka project. + +Here is an example of using the `StreamsBuilderFactoryBeanCustomizer`. + +``` +@Bean +public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() { + return sfb -> sfb.setStateListener((newState, oldState) -> { + //Do some action here! + }); +} +``` + +The above is shown as an illustration of the things you can do to customize the `StreamsBuilderFactoryBean`. +You can essentially call any available mutation operations from `StreamsBuilderFactoryBean` to customize it. +This customizer will be invoked by the binder right before the factory bean is started. + +Once you get access to the `StreamsBuilderFactoryBean`, you can also customize the underlying `KafkaStreams` object. +Here is a blueprint for doing so. + +``` +@Bean +public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() { + return factoryBean -> { + factoryBean.setKafkaStreamsCustomizer(new KafkaStreamsCustomizer() { + @Override + public void customize(KafkaStreams kafkaStreams) { + kafkaStreams.setUncaughtExceptionHandler((t, e) -> { + + }); + } + }); + }; +} +``` + +`KafkaStreamsCustomizer` will be called by the `StreamsBuilderFactoryBeabn` right before the underlying `KafkaStreams` gets started. + +There can only be one `StreamsBuilderFactoryBeanCustomizer` in the entire application. +Then how do we account for multiple Kafka Streams processors as each of them are backed up by individual `StreamsBuilderFactoryBean` objects? +In that case, if the customization needs to be different for those processors, then the application needs to apply some filter based on the application ID. + +For e.g, + +``` +@Bean +public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() { + + return factoryBean -> { + if (factoryBean.getStreamsConfiguration().getProperty(StreamsConfig.APPLICATION_ID_CONFIG) + .equals("processor1-application-id")) { + factoryBean.setKafkaStreamsCustomizer(new KafkaStreamsCustomizer() { + @Override + public void customize(KafkaStreams kafkaStreams) { + kafkaStreams.setUncaughtExceptionHandler((t, e) -> { + + }); + } + }); + } + }; +``` + +[[using-customizer-to-register-a-global-state-store]] +== Using Customizer to register a global state store + +As mentioned above, the binder does not provide a first class way to register global state stores as a feature. +For that, you need to use the customizer. +Here is how that can be done. + +``` +@Bean +public StreamsBuilderFactoryBeanCustomizer customizer() { + return fb -> { + try { + final StreamsBuilder streamsBuilder = fb.getObject(); + streamsBuilder.addGlobalStore(...); + } + catch (Exception e) { + + } + }; +} +``` + +Again, if you have multiple processors, you want to attach the global state store to the right `StreamsBuilder` by filtering out the other `StreamsBuilderFactoryBean` objects using the application id as outlined above. + +[[using-customizer-to-register-a-production-exception-handler]] +== Using customizer to register a production exception handler + +In the error handling section, we indicated that the binder does not provide a first class way to deal with production exceptions. +Though that is the case, you can still use the `StreamsBuilderFacotryBean` customizer to register production exception handlers. See below. + +``` +@Bean +public StreamsBuilderFactoryBeanCustomizer customizer() { + return fb -> { + fb.getStreamsConfiguration().put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG, + CustomProductionExceptionHandler.class); + }; +} +``` + +Once again, if you have multiple processors, you may want to set it appropriately against the correct `StreamsBuilderFactoryBean`. +You may also add such production exception handlers using the configuration property (See below for more on that), but this is an option if you choose to go with a programmatic approach. + diff --git a/docs/modules/ROOT/pages/kafka/streams/timestamp-extractor.adoc b/docs/modules/ROOT/pages/kafka/streams/timestamp-extractor.adoc new file mode 100644 index 000000000..8372ddcce --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/timestamp-extractor.adoc @@ -0,0 +1,34 @@ +[[timestamp-extractor]] += Timestamp extractor + +Kafka Streams allows you to control the processing of the consumer records based on various notions of timestamp. +By default, Kafka Streams extracts the timestamp metadata embedded in the consumer record. +You can change this default behavior by providing a different `TimestampExtractor` implementation per input binding. +Here are some details on how that can be done. + +``` +@Bean +public Function, + Function, + Function, KStream>>> process() { + return orderStream -> + customers -> + products -> orderStream; +} + +@Bean +public TimestampExtractor timestampExtractor() { + return new WallclockTimestampExtractor(); +} +``` + +Then you set the above `TimestampExtractor` bean name per consumer binding. + +``` +spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.timestampExtractorBeanName=timestampExtractor +spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.timestampExtractorBeanName=timestampExtractor +spring.cloud.stream.kafka.streams.bindings.process-in-2.consumer.timestampExtractorBeanName=timestampExtractor" +``` + +If you skip an input consumer binding for setting a custom timestamp extractor, that consumer will use the default settings. + diff --git a/docs/modules/ROOT/pages/kafka/streams/topology-visualization.adoc b/docs/modules/ROOT/pages/kafka/streams/topology-visualization.adoc new file mode 100644 index 000000000..f127deb1b --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/topology-visualization.adoc @@ -0,0 +1,13 @@ +[[kafka-streams-topology-visualization]] += Kafka Streams topology visualization + +Kafka Streams binder provides the following actuator endpoints for retrieving the topology description using which you can visualize the topology using external tools. + +`/actuator/kafkastreamstopology` + +`/actuator/kafkastreamstopology/` + +You need to include the actuator and web dependencies from Spring Boot to access these endpoints. +Further, you also need to add `kafkastreamstopology` to `management.endpoints.web.exposure.include` property. +By default, the `kafkastreamstopology` endpoint is disabled. + diff --git a/docs/modules/ROOT/pages/kafka/streams/tracing-using-spring-cloud-sleuth.adoc b/docs/modules/ROOT/pages/kafka/streams/tracing-using-spring-cloud-sleuth.adoc new file mode 100644 index 000000000..78f23c267 --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/tracing-using-spring-cloud-sleuth.adoc @@ -0,0 +1,64 @@ +[[tracing-using-spring-cloud-sleuth]] += Tracing using Spring Cloud Sleuth + +When Spring Cloud Sleuth is on the classpath of a Spring Cloud Stream Kafka Streams binder based application, both its consumer and producer are automatically instrumented with tracing information. +However, in order to trace any application specific operations, those need to be explicitly instrumented by the user code. +This can be done by injecting the `KafkaStreamsTracing` bean from Spring Cloud Sleuth in the application and then invoke various Kafka Streams operations through this injected bean. +Here are some examples of using it. + +``` +@Bean +public BiFunction, KTable, KStream> clicks(KafkaStreamsTracing kafkaStreamsTracing) { + return (userClicksStream, userRegionsTable) -> (userClicksStream + .transformValues(kafkaStreamsTracing.peek("span-1", (key, value) -> LOG.info("key/value: " + key + "/" + value))) + .leftJoin(userRegionsTable, (clicks, region) -> new RegionWithClicks(region == null ? + "UNKNOWN" : region, clicks), + Joined.with(Serdes.String(), Serdes.Long(), null)) + .transform(kafkaStreamsTracing.map("span-2", (key, value) -> { + LOG.info("Click Info: " + value.getRegion() + "/" + value.getClicks()); + return new KeyValue<>(value.getRegion(), + value.getClicks()); + })) + .groupByKey(Grouped.with(Serdes.String(), Serdes.Long())) + .reduce(Long::sum, Materialized.as(CLICK_UPDATES)) + .toStream()); +} +``` + +In the example above, there are two places where it adds explicit tracing instrumentation. +First, we are logging the key/value information from the incoming `KStream`. +When this information is logged, the associated span and trace IDs get logged as well so that a monitoring system can track them and correlate with the same span id. +Second, when we call a `map` operation, instead of calling it directly on the `KStream` class, we wrap it inside a `transform` operation and then call `map` from `KafkaStreamsTracing`. +In this case also, the logged message will contain the span ID and trace ID. + +Here is another example, where we use the low-level transformer API for accessing the various Kafka Streams headers. +When spring-cloud-sleuth is on the classpath, all the tracing headers can also be accessed like this. + +``` +@Bean +public Function, KStream> process(KafkaStreamsTracing kafkaStreamsTracing) { + return input -> input.transform(kafkaStreamsTracing.transformer( + "transformer-1", + () -> new Transformer>() { + ProcessorContext context; + + @Override + public void init(ProcessorContext context) { + this.context = context; + } + + @Override + public KeyValue transform(String key, String value) { + LOG.info("Headers: " + this.context.headers()); + LOG.info("K/V:" + key + "/" + value); + // More transformations, business logic execution, etc. go here. + return KeyValue.pair(key, value); + } + + @Override + public void close() { + } + })); +} +``` + diff --git a/docs/modules/ROOT/pages/kafka/streams/usage.adoc b/docs/modules/ROOT/pages/kafka/streams/usage.adoc new file mode 100644 index 000000000..33ca7549a --- /dev/null +++ b/docs/modules/ROOT/pages/kafka/streams/usage.adoc @@ -0,0 +1,17 @@ +[[usage]] += Usage + +For using the Kafka Streams binder, you just need to add it to your Spring Cloud Stream application, using the following maven coordinates: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-kafka-streams + +---- + +A quick way to bootstrap a new project for Kafka Streams binder is to use http://start.spring.io[Spring Initializr] and then select "Cloud Streams" and "Spring for Kafka Streams" as shown below + +image::{github-raw}/docs/src/main/asciidoc/images/spring-initializr-kafka-streams.png[width=800,scaledwidth="75%",align="center"] + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview.adoc index 453d9b419..1c4665226 100644 --- a/docs/modules/ROOT/pages/rabbit/rabbit_overview.adoc +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview.adoc @@ -83,1290 +83,3 @@ This section contains settings specific to the RabbitMQ Binder and bound channel For general binding configuration options and properties, see the https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/current/reference/html/spring-cloud-stream.html#_configuration_options[Spring Cloud Stream core documentation]. -[[rabbit-binder-properties]] -== RabbitMQ Binder Properties - -By default, the RabbitMQ binder uses Spring Boot's `ConnectionFactory`. -Conseuqently, it supports all Spring Boot configuration options for RabbitMQ. -(For reference, see the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#common-application-properties[Spring Boot documentation]). -RabbitMQ configuration options use the `spring.rabbitmq` prefix. - -In addition to Spring Boot options, the RabbitMQ binder supports the following properties: - -spring.cloud.stream.rabbit.binder.adminAddresses:: -A comma-separated list of RabbitMQ management plugin URLs. -Only used when `nodes` contains more than one entry. -Each entry in this list must have a corresponding entry in `spring.rabbitmq.addresses`. -Only needed if you use a RabbitMQ cluster and wish to consume from the node that hosts the queue. -See https://docs.spring.io/spring-amqp/reference/html/_reference.html#queue-affinity[Queue Affinity and the LocalizedQueueConnectionFactory] for more information. -+ -Default: empty. -spring.cloud.stream.rabbit.binder.nodes:: -A comma-separated list of RabbitMQ node names. -When more than one entry, used to locate the server address where a queue is located. -Each entry in this list must have a corresponding entry in `spring.rabbitmq.addresses`. -Only needed if you use a RabbitMQ cluster and wish to consume from the node that hosts the queue. -See https://docs.spring.io/spring-amqp/reference/html/_reference.html#queue-affinity[Queue Affinity and the LocalizedQueueConnectionFactory] for more information. -+ -Default: empty. -spring.cloud.stream.rabbit.binder.compressionLevel:: -The compression level for compressed bindings. -See `java.util.zip.Deflater`. -+ -Default: `1` (BEST_LEVEL). -spring.cloud.stream.binder.connection-name-prefix:: -A connection name prefix used to name the connection(s) created by this binder. -The name is this prefix followed by `#n`, where `n` increments each time a new connection is opened. -+ -Default: none (Spring AMQP default). - -[[rabbitmq-consumer-properties]] -== RabbitMQ Consumer Properties - -The following properties are available for Rabbit consumers only and must be prefixed with `spring.cloud.stream.rabbit.bindings..consumer.`. - -However if the same set of properties needs to be applied to most bindings, to -avoid repetition, Spring Cloud Stream supports setting values for all channels, -in the format of `spring.cloud.stream.rabbit.default.=`. - -Also, keep in mind that binding specific property will override its equivalent in the default. - -acknowledgeMode:: -The acknowledge mode. -+ -Default: `AUTO`. -anonymousGroupPrefix:: -When the binding has no `group` property, an anonymous, auto-delete queue is bound to the destination exchange. -The default naming stragegy for such queues results in a queue named `anonymous.`. -Set this property to change the prefix to something other than the default. -+ -Default: `anonymous.`. -autoBindDlq:: -Whether to automatically declare the DLQ and bind it to the binder DLX. -+ -Default: `false`. -bindingRoutingKey:: -The routing key with which to bind the queue to the exchange (if `bindQueue` is `true`). -Can be multiple keys - see `bindingRoutingKeyDelimiter`. -For partitioned destinations, `-` is appended to each key. -+ -Default: `#`. -bindingRoutingKeyDelimiter:: -When this is not null, 'bindingRoutingKey' is considered to be a list of keys delimited by this value; often a comma is used. -+ -Default: `null`. -bindQueue:: -Whether to declare the queue and bind it to the destination exchange. -Set it to `false` if you have set up your own infrastructure and have previously created and bound the queue. -+ -Default: `true`. -consumerTagPrefix:: -Used to create the consumer tag(s); will be appended by `#n` where `n` increments for each consumer created. -Example: `${spring.application.name}-${spring.cloud.stream.bindings.input.group}-${spring.cloud.stream.instance-index}`. -+ -Default: none - the broker will generate random consumer tags. -containerType:: -Select the type of listener container to be used. -See https://docs.spring.io/spring-amqp/reference/html/_reference.html#choose-container[Choosing a Container] in the Spring AMQP documentation for more information. -Also see <>. -+ -Default: `simple` -deadLetterQueueName:: -The name of the DLQ -+ -Default: `prefix+destination.dlq` -deadLetterExchange:: -A DLX to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -+ -Default: 'prefix+DLX' -deadLetterExchangeType:: -The type of the DLX to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -+ -Default: 'direct' -deadLetterRoutingKey:: -A dead letter routing key to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -+ -Default: `destination` -declareDlx:: -Whether to declare the dead letter exchange for the destination. -Relevant only if `autoBindDlq` is `true`. -Set to `false` if you have a pre-configured DLX. -+ -Default: `true`. -declareExchange:: -Whether to declare the exchange for the destination. -+ -Default: `true`. -delayedExchange:: -Whether to declare the exchange as a `Delayed Message Exchange`. -Requires the delayed message exchange plugin on the broker. -The `x-delayed-type` argument is set to the `exchangeType`. -+ -Default: `false`. -dlqBindingArguments:: -Arguments applied when binding the dlq to the dead letter exchange; used with `headers` `deadLetterExchangeType` to specify headers to match on. -For example `...dlqBindingArguments.x-match=any`, `...dlqBindingArguments.someHeader=someValue`. -+ -Default: empty -dlqDeadLetterExchange:: -If a DLQ is declared, a DLX to assign to that queue. -+ -Default: `none` -dlqDeadLetterRoutingKey:: -If a DLQ is declared, a dead letter routing key to assign to that queue. -+ -Default: `none` -dlqExpires:: -How long before an unused dead letter queue is deleted (in milliseconds). -+ -Default: `no expiration` -dlqLazy:: -Declare the dead letter queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -+ -Default: `false`. -dlqMaxLength:: -Maximum number of messages in the dead letter queue. -+ -Default: `no limit` -dlqMaxLengthBytes:: -Maximum number of total bytes in the dead letter queue from all messages. -+ -Default: `no limit` -dlqMaxPriority:: -Maximum priority of messages in the dead letter queue (0-255). -+ -Default: `none` -dlqOverflowBehavior:: -Action to take when `dlqMaxLength` or `dlqMaxLengthBytes` is exceeded; currently `drop-head` or `reject-publish` but refer to the RabbitMQ documentation. -+ -Default: `none` -dlqQuorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -+ -Default: none - broker default will apply. -dlqQuorum.enabled:: -When true, create a quorum dead letter queue instead of a classic queue. -+ -Default: false -dlqQuorum.initialGroupSize:: -When `quorum.enabled=true`, set the initial quorum size. -+ -Default: none - broker default will apply. -dlqSingleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -+ -Default: `false` -dlqTtl:: -Default time to live to apply to the dead letter queue when declared (in milliseconds). -+ -Default: `no limit` -durableSubscription:: -Whether the subscription should be durable. -Only effective if `group` is also set. -+ -Default: `true`. -exchangeAutoDelete:: -If `declareExchange` is true, whether the exchange should be auto-deleted (that is, removed after the last queue is removed). -+ -Default: `true`. -exchangeDurable:: -If `declareExchange` is true, whether the exchange should be durable (that is, it survives broker restart). -+ -Default: `true`. -exchangeType:: -The exchange type: `direct`, `fanout`, `headers` or `topic` for non-partitioned destinations and `direct`, headers or `topic` for partitioned destinations. -+ -Default: `topic`. -exclusive:: -Whether to create an exclusive consumer. -Concurrency should be 1 when this is `true`. -Often used when strict ordering is required but enabling a hot standby instance to take over after a failure. -See `recoveryInterval`, which controls how often a standby instance attempts to consume. -Consider using `singleActiveConsumer` instead when using RabbitMQ 3.8 or later. -+ -Default: `false`. -expires:: -How long before an unused queue is deleted (in milliseconds). -+ -Default: `no expiration` -failedDeclarationRetryInterval:: -The interval (in milliseconds) between attempts to consume from a queue if it is missing. -+ -Default: 5000 -[[spring-cloud-stream-rabbit-frame-max-headroom]] -frameMaxHeadroom:: -The number of bytes to reserve for other headers when adding the stack trace to a DLQ message header. -All headers must fit within the `frame_max` size configured on the broker. -Stack traces can be large; if the size plus this property exceeds `frame_max` then the stack trace will be truncated. -A WARN log will be written; consider increasing the `frame_max` or reducing the stack trace by catching the exception and throwing one with a smaller stack trace. -+ -Default: 20000 -headerPatterns:: -Patterns for headers to be mapped from inbound messages. -+ -Default: `['*']` (all headers). -lazy:: -Declare the queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -+ -Default: `false`. -maxConcurrency:: -The maximum number of consumers. -Not supported when the `containerType` is `direct`. -+ -Default: `1`. -maxLength:: -The maximum number of messages in the queue. -+ -Default: `no limit` -maxLengthBytes:: -The maximum number of total bytes in the queue from all messages. -+ -Default: `no limit` -maxPriority:: -The maximum priority of messages in the queue (0-255). -+ -Default: `none` -missingQueuesFatal:: -When the queue cannot be found, whether to treat the condition as fatal and stop the listener container. -Defaults to `false` so that the container keeps trying to consume from the queue -- for example, when using a cluster and the node hosting a non-HA queue is down. -+ -Default: `false` -overflowBehavior:: -Action to take when `maxLength` or `maxLengthBytes` is exceeded; currently `drop-head` or `reject-publish` but refer to the RabbitMQ documentation. -+ -Default: `none` -prefetch:: -Prefetch count. -+ -Default: `1`. -prefix:: -A prefix to be added to the name of the `destination` and queues. -+ -Default: "". -queueBindingArguments:: -Arguments applied when binding the queue to the exchange; used with `headers` `exchangeType` to specify headers to match on. -For example `...queueBindingArguments.x-match=any`, `...queueBindingArguments.someHeader=someValue`. -+ -Default: empty -queueDeclarationRetries:: -The number of times to retry consuming from a queue if it is missing. -Relevant only when `missingQueuesFatal` is `true`. -Otherwise, the container keeps retrying indefinitely. -Not supported when the `containerType` is `direct`. -+ -Default: `3` -queueNameGroupOnly:: -When true, consume from a queue with a name equal to the `group`. -Otherwise the queue name is `destination.group`. -This is useful, for example, when using Spring Cloud Stream to consume from an existing RabbitMQ queue. -+ -Default: false. -quorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -+ -Default: none - broker default will apply. -quorum.enabled:: -When true, create a quorum queue instead of a classic queue. -+ -Default: false -quorum.initialGroupSize:: -When `quorum.enabled=true`, set the initial quorum size. -+ -Default: none - broker default will apply. -recoveryInterval:: -The interval between connection recovery attempts, in milliseconds. -+ -Default: `5000`. -requeueRejected:: -Whether delivery failures should be re-queued when retry is disabled or `republishToDlq` is `false`. -+ -Default: `false`. -[[spring-cloud-stream-rabbit-republish-delivery-mode]] -republishDeliveryMode:: -When `republishToDlq` is `true`, specifies the delivery mode of the republished message. -+ -Default: `DeliveryMode.PERSISTENT` -republishToDlq:: -By default, messages that fail after retries are exhausted are rejected. -If a dead-letter queue (DLQ) is configured, RabbitMQ routes the failed message (unchanged) to the DLQ. -If set to `true`, the binder republishs failed messages to the DLQ with additional headers, including the exception message and stack trace from the cause of the final failure. -Also see the <>. -+ -Default: `true` -singleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -+ -Default: `false` -transacted:: -Whether to use transacted channels. -+ -Default: `false`. -ttl:: -Default time to live to apply to the queue when declared (in milliseconds). -+ -Default: `no limit` -txSize:: -The number of deliveries between acks. -Not supported when the `containerType` is `direct`. -+ -Default: `1`. - -[[rabbitmq-stream-consumer]] -== Initial Consumer Support for the RabbitMQ Stream Plugin - -Basic support for the https://rabbitmq.com/stream.html[RabbitMQ Stream Plugin] is now provided. -To enable this feature, you must add the `spring-rabbit-stream` jar to the class path - it must be the same version as `spring-amqp` and `spring-rabbit`. - -IMPORTANT: The consumer properties described above are not supported when you set the `containerType` property to `stream`; `concurrency` is supported for super streams only. -Only a single stream queue can be consumed by each binding. - -To configure the binder to use `containerType=stream`, Spring Boot will automatically configure an `Environment` `@Bean` from the application properties. -You can, optionally, add a customizer to customize the listener container. - -==== -[source, java] ----- -@Bean -ListenerContainerCustomizer customizer() { - return (cont, dest, group) -> { - StreamListenerContainer container = (StreamListenerContainer) cont; - container.setConsumerCustomizer((name, builder) -> { - builder.offset(OffsetSpecification.first()); - }); - // ... - }; -} ----- -==== - -The `name` argument passed to the customizer is `destination + '.' + group + '.container'`. - -The stream `name()` (for the purpose of offset tracking) is set to the binding `destination + '.' + group`. -It can be changed using a `ConsumerCustomizer` shown above. -If you decide to use manual offset tracking, the `Context` is available as a message header: - -==== -[source, java] ----- -int count; - -@Bean -public Consumer> input() { - return msg -> { - System.out.println(msg); - if (++count % 1000 == 0) { - Context context = msg.getHeaders().get("rabbitmq_streamContext", Context.class); - context.consumer().store(context.offset()); - } - }; -} ----- -==== - -Refer to the https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/[RabbitMQ Stream Java Client documentation] for information about configuring the environment and consumer builder. - -[[rabbitmq-super-stream-consumer]] -=== Consumer Support for the RabbitMQ Super Streams - -See https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-super-streams[Super Streams] for information about super streams. - -Use of super streams allows for automatic scale-up scale-down with a single active consumer on each partition of a super stream. - -Configuration example: - -==== -[source, java] ----- -@Bean -public Consumer input() { - ... -} ----- -==== - -==== -[source, properties] ----- -spring.cloud.stream.bindings.input-in-0.destination=super -spring.cloud.stream.bindings.input-in-0.group=test -spring.cloud.stream.bindings.input-in-0.consumer.instance-count=3 -spring.cloud.stream.bindings.input-in-0.consumer.concurrency=3 -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.container-type=STREAM -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.super-stream=true ----- -==== - -The framework will create a super stream named `super`, with 9 partitions. -Up to 3 instances of this application can be deployed. - -[[advanced-listener-container-configuration]] -== Advanced Listener Container Configuration - -To set listener container properties that are not exposed as binder or binding properties, add a single bean of type `ListenerContainerCustomizer` to the application context. -The binder and binding properties will be set and then the customizer will be called. -The customizer (`configure()` method) is provided with the queue name as well as the consumer group as arguments. - -[[advanced-queue/exchange/binding-configuration]] -== Advanced Queue/Exchange/Binding Configuration - -From time to time, the RabbitMQ team add new features that are enabled by setting some argument when declaring, for example, a queue. -Generally, such features are enabled in the binder by adding appropriate properties, but this may not be immediately available in a current version. -Starting with version 3.0.1, you can now add `DeclarableCustomizer` bean(s) to the application context to modify a `Declarable` (`Queue`, `Exchange` or `Binding`) just before the declaration is performed. -This allows you to add arguments that are not currently directly supported by the binder. - -[[rabbit-receiving-batch]] -== Receiving Batched Messages - -With the RabbitMQ binder, there are two types of batches handled by consumer bindings: - -[[batches-created-by-producers]] -=== Batches Created by Producers - -Normally, if a producer binding has `batch-enabled=true` (see <>), or a message is created by a `BatchingRabbitTemplate`, elements of the batch are returned as individual calls to the listener method. -Starting with version 3.0, any such batch can be presented as a `List` to the listener method if `spring.cloud.stream.bindings..consumer.batch-mode` is set to `true`. - -[[consumer-side-batching]] -=== Consumer-side Batching - -Starting with version 3.1, the consumer can be configured to assemble multiple inbound messages into a batch which is presented to the application as a `List` of converted payloads. -The following simple application demonstrates how to use this technique: - -==== -[source, properties] ----- -spring.cloud.stream.bindings.input-in-0.group=someGroup - -spring.cloud.stream.bindings.input-in-0.consumer.batch-mode=true - -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.enable-batching=true -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.batch-size=10 -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.receive-timeout=200 ----- -==== - -==== -[source, java] ----- -@SpringBootApplication -public class Application { - - public static void main(String[] args) { - SpringApplication.run(Application.class, args); - } - - @Bean - Consumer> input() { - return list -> { - System.out.println("Received " + list.size()); - list.forEach(thing -> { - System.out.println(thing); - - // ... - - }); - }; - } - - @Bean - public ApplicationRunner runner(RabbitTemplate template) { - return args -> { - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value1\"}"); - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value2\"}"); - }; - } - - public static class Thing { - - private String field; - - public Thing() { - } - - public Thing(String field) { - this.field = field; - } - - public String getField() { - return this.field; - } - - public void setField(String field) { - this.field = field; - } - - @Override - public String toString() { - return "Thing [field=" + this.field + "]"; - } - - } - -} ----- -==== - -==== -[source] ----- -Received 2 -Thing [field=value1] -Thing [field=value2] ----- -==== - -The number of messages in a batch is specified by the `batch-size` and `receive-timeout` properties; if the `receive-timeout` elapses with no new messages, a "short" batch is delivered. - -IMPORTANT: Consumer-side batching is only supported with `container-type=simple` (the default). - -If you wish to examine headers of consumer-side batched messages, you should consume `Message>`; the headers are a `List>` in a header `AmqpInboundChannelAdapter.CONSOLIDATED_HEADERS`, with the headers for each payload element in the corresponding index. -Again, here is a simple example: - -==== -[source, java] ----- -@SpringBootApplication -public class Application { - - public static void main(String[] args) { - SpringApplication.run(Application.class, args); - } - - @Bean - Consumer>> input() { - return msg -> { - List things = msg.getPayload(); - System.out.println("Received " + things.size()); - @SuppressWarnings("unchecked") - List> headers = - (List>) msg.getHeaders().get(AmqpInboundChannelAdapter.CONSOLIDATED_HEADERS); - for (int i = 0; i < things.size(); i++) { - System.out.println(things.get(i) + " myHeader=" + headers.get(i).get("myHeader")); - - // ... - - } - }; - } - - @Bean - public ApplicationRunner runner(RabbitTemplate template) { - return args -> { - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value1\"}", msg -> { - msg.getMessageProperties().setHeader("myHeader", "headerValue1"); - return msg; - }); - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value2\"}", msg -> { - msg.getMessageProperties().setHeader("myHeader", "headerValue2"); - return msg; - }); - }; - } - - public static class Thing { - - private String field; - - public Thing() { - } - - public Thing(String field) { - this.field = field; - } - - public String getfield() { - return this.field; - } - - public void setfield(String field) { - this.field = field; - } - - @Override - public String toString() { - return "Thing [field=" + this.field + "]"; - } - - } - -} ----- -==== - -==== -[source] ----- -Received 2 -Thing [field=value1] myHeader=headerValue1 -Thing [field=value2] myHeader=headerValue2 ----- -==== - -[[rabbit-prod-props]] -== Rabbit Producer Properties - -The following properties are available for Rabbit producers only and must be prefixed with `spring.cloud.stream.rabbit.bindings..producer.`. - -However if the same set of properties needs to be applied to most bindings, to -avoid repetition, Spring Cloud Stream supports setting values for all channels, -in the format of `spring.cloud.stream.rabbit.default.=`. - -Also, keep in mind that binding specific property will override its equivalent in the default. - -altermateExchange.binding.queue:: -If the exchange does not already exist, and a `name` is provided, bind this queue to the alternate exhange. -A simple durable queue with no arguments is provisioned; if more sophisticated configuration is required, you must configure and bind the queue yourself. -+ -Default: `null` -alternateExchange.binding.routingKey -If the exchange does not already exist, and a `name` and `queue` is provided, bind the queue to the alternate exhange using this routing key. -+ -Default: `#` (for the default `topic` alternate exchange) -alternateExchange.exists:: -Whether the alternate exchange exists, or needs to be provisioned. -+ -Default: `false` -alternateExchange.type:: -If the alternate exchange does not already exist, the type of exchange to provision. -+ -Default: `topic` -alternateExchange.name:: -Configure an alternate exchange on the destination exchange. -+ -Default: `null` -autoBindDlq:: -Whether to automatically declare the DLQ and bind it to the binder DLX. -+ -Default: `false`. -batchingEnabled:: -Whether to enable message batching by producers. -Messages are batched into one message according to the following properties (described in the next three entries in this list): 'batchSize', `batchBufferLimit`, and `batchTimeout`. -See https://docs.spring.io/spring-amqp//reference/html/_reference.html#template-batching[Batching] for more information. -Also see <>. -+ -Default: `false`. -batchSize:: -The number of messages to buffer when batching is enabled. -+ -Default: `100`. -batchBufferLimit:: -The maximum buffer size when batching is enabled. -+ -Default: `10000`. -batchTimeout:: -The batch timeout when batching is enabled. -+ -Default: `5000`. -bindingRoutingKey:: -The routing key with which to bind the queue to the exchange (if `bindQueue` is `true`). -Can be multiple keys - see `bindingRoutingKeyDelimiter`. -For partitioned destinations, `-n` is appended to each key. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `#`. -bindingRoutingKeyDelimiter:: -When this is not null, 'bindingRoutingKey' is considered to be a list of keys delimited by this value; often a comma is used. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `null`. -bindQueue:: -Whether to declare the queue and bind it to the destination exchange. -Set it to `false` if you have set up your own infrastructure and have previously created and bound the queue. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `true`. -compress:: -Whether data should be compressed when sent. -+ -Default: `false`. -confirmAckChannel:: -When `errorChannelEnabled` is true, a channel to which to send positive delivery acknowledgments (aka publisher confirms). -If the channel does not exist, a `DirectChannel` is registered with this name. -The connection factory must be configured to enable publisher confirms. -Mutually exclusive with `useConfirmHeader`. -+ -Default: `nullChannel` (acks are discarded). -deadLetterQueueName:: -The name of the DLQ -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `prefix+destination.dlq` -deadLetterExchange:: -A DLX to assign to the queue. -Relevant only when `autoBindDlq` is `true`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: 'prefix+DLX' -deadLetterExchangeType:: -The type of the DLX to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: 'direct' -deadLetterRoutingKey:: -A dead letter routing key to assign to the queue. -Relevant only when `autoBindDlq` is `true`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `destination` -declareDlx:: -Whether to declare the dead letter exchange for the destination. -Relevant only if `autoBindDlq` is `true`. -Set to `false` if you have a pre-configured DLX. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `true`. -declareExchange:: -Whether to declare the exchange for the destination. -+ -Default: `true`. -delayExpression:: -A SpEL expression to evaluate the delay to apply to the message (`x-delay` header). -It has no effect if the exchange is not a delayed message exchange. -+ -Default: No `x-delay` header is set. -delayedExchange:: -Whether to declare the exchange as a `Delayed Message Exchange`. -Requires the delayed message exchange plugin on the broker. -The `x-delayed-type` argument is set to the `exchangeType`. -+ -Default: `false`. -deliveryMode:: -The delivery mode. -+ -Default: `PERSISTENT`. -dlqBindingArguments:: -Arguments applied when binding the dlq to the dead letter exchange; used with `headers` `deadLetterExchangeType` to specify headers to match on. -For example `...dlqBindingArguments.x-match=any`, `...dlqBindingArguments.someHeader=someValue`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: empty -dlqDeadLetterExchange:: -When a DLQ is declared, a DLX to assign to that queue. -Applies only if `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -dlqDeadLetterRoutingKey:: -When a DLQ is declared, a dead letter routing key to assign to that queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -dlqExpires:: -How long (in milliseconds) before an unused dead letter queue is deleted. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no expiration` -dlqLazy:: -Declare the dead letter queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -dlqMaxLength:: -Maximum number of messages in the dead letter queue. -Applies only if `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -dlqMaxLengthBytes:: -Maximum number of total bytes in the dead letter queue from all messages. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -dlqMaxPriority:: -Maximum priority of messages in the dead letter queue (0-255) -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -dlqQuorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -dlqQuorum.enabled:: -When true, create a quorum dead letter queue instead of a classic queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: false -dlqQuorum.initialGroupSize:: -When `quorum.enabled=true`, set the initial quorum size. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -dlqSingleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `false` -dlqTtl:: -Default time (in milliseconds) to live to apply to the dead letter queue when declared. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -exchangeAutoDelete:: -If `declareExchange` is `true`, whether the exchange should be auto-delete (it is removed after the last queue is removed). -+ -Default: `true`. -exchangeDurable:: -If `declareExchange` is `true`, whether the exchange should be durable (survives broker restart). -+ -Default: `true`. -exchangeType:: -The exchange type: `direct`, `fanout`, `headers` or `topic` for non-partitioned destinations and `direct`, `headers` or `topic` for partitioned destinations. -+ -Default: `topic`. -expires:: -How long (in milliseconds) before an unused queue is deleted. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no expiration` -headerPatterns:: -Patterns for headers to be mapped to outbound messages. -+ -Default: `['*']` (all headers). -lazy:: -Declare the queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `false`. -maxLength:: -Maximum number of messages in the queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -maxLengthBytes:: -Maximum number of total bytes in the queue from all messages. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -maxPriority:: -Maximum priority of messages in the queue (0-255). -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -prefix:: -A prefix to be added to the name of the `destination` exchange. -+ -Default: "". -producerType:: -The type of producer. -* `AMQP` AMQP Client for classic and quorum queues -* `STREAM_SYNC` RabbitMQ Streams Plugin client, blocks until confirm is received -* `STREAM_ASYNC` RabbitMQ Streams Plugin client, does not block -+ -Default: "". -queueBindingArguments:: -Arguments applied when binding the queue to the exchange; used with `headers` `exchangeType` to specify headers to match on. -For example `...queueBindingArguments.x-match=any`, `...queueBindingArguments.someHeader=someValue`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: empty -queueNameGroupOnly:: -When `true`, consume from a queue with a name equal to the `group`. -Otherwise the queue name is `destination.group`. -This is useful, for example, when using Spring Cloud Stream to consume from an existing RabbitMQ queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: false. -quorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -quorum.enabled:: -When true, create a quorum queue instead of a classic queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: false -quorum.initialGroupSize:: -When `quorum.enabled=true`, set the initial quorum size. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -routingKeyExpression:: -A SpEL expression to determine the routing key to use when publishing messages. -For a fixed routing key, use `routingKey`. -+ -Default: `destination` or `destination-` for partitioned destinations. -routingKey:: -A string defining a fixed routing key to use when publishing messages. -+ -Default: see `routingKeyExpression` -singleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `false` -transacted:: -Whether to use transacted channels. -+ -Default: `false`. -ttl:: -Default time (in milliseconds) to live to apply to the queue when declared. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -useConfirmHeader:: -See <>. -Mutually exclusive with `confirmAckChannel`. -+ - -NOTE: In the case of RabbitMQ, content type headers can be set by external applications. -Spring Cloud Stream supports them as part of an extended internal protocol used for any type of transport -- including transports, such as Kafka (prior to 0.11), that do not natively support headers. - -[[publisher-confirms]] -== Publisher Confirms - -There are two mechanisms to get the result of publishing a message; in each case, the connection factory must have `publisherConfirmType` set `ConfirmType.CORRELATED`. -The "legacy" mechanism is to set the `confirmAckChannel` to the bean name of a message channel from which you can retrieve the confirmations asynchronously; negative acks are sent to the error channel (if enabled) - see <>. - -The preferred mechanism, added in version 3.1 is to use a correlation data header and wait for the result via its `Future` property. -This is particularly useful with a batch listener because you can send multiple messages before waiting for the result. -To use this technique, set the `useConfirmHeader` property to true -The following simple application is an example of using this technique: - -==== -[source, properties] ----- -spring.cloud.stream.bindings.input-in-0.group=someGroup -spring.cloud.stream.bindings.input-in-0.consumer.batch-mode=true - -spring.cloud.stream.source=output -spring.cloud.stream.bindings.output-out-0.producer.error-channel-enabled=true - -spring.cloud.stream.rabbit.bindings.output-out-0.producer.useConfirmHeader=true -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.auto-bind-dlq=true -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.batch-size=10 - -spring.rabbitmq.publisher-confirm-type=correlated -spring.rabbitmq.publisher-returns=true ----- -==== - -==== -[source, java] ----- -@SpringBootApplication -public class Application { - - private static final Logger log = LoggerFactory.getLogger(Application.class); - - public static void main(String[] args) { - SpringApplication.run(Application.class, args); - } - - @Autowired - private StreamBridge bridge; - - @Bean - Consumer> input() { - return list -> { - List results = new ArrayList<>(); - list.forEach(str -> { - log.info("Received: " + str); - MyCorrelationData corr = new MyCorrelationData(UUID.randomUUID().toString(), str); - results.add(corr); - this.bridge.send("output-out-0", MessageBuilder.withPayload(str.toUpperCase()) - .setHeader(AmqpHeaders.PUBLISH_CONFIRM_CORRELATION, corr) - .build()); - }); - results.forEach(correlation -> { - try { - Confirm confirm = correlation.getFuture().get(10, TimeUnit.SECONDS); - log.info(confirm + " for " + correlation.getPayload()); - if (correlation.getReturnedMessage() != null) { - log.error("Message for " + correlation.getPayload() + " was returned "); - - // throw some exception to invoke binder retry/error handling - - } - } - catch (InterruptedException e) { - Thread.currentThread().interrupt(); - throw new IllegalStateException(e); - } - catch (ExecutionException | TimeoutException e) { - throw new IllegalStateException(e); - } - }); - }; - } - - - @Bean - public ApplicationRunner runner(BatchingRabbitTemplate template) { - return args -> IntStream.range(0, 10).forEach(i -> - template.convertAndSend("input-in-0", "input-in-0.rbgh303", "foo" + i)); - } - - @Bean - public BatchingRabbitTemplate template(CachingConnectionFactory cf, TaskScheduler taskScheduler) { - BatchingStrategy batchingStrategy = new SimpleBatchingStrategy(10, 1000000, 1000); - return new BatchingRabbitTemplate(cf, batchingStrategy, taskScheduler); - } - -} - -class MyCorrelationData extends CorrelationData { - - private final String payload; - - MyCorrelationData(String id, String payload) { - super(id); - this.payload = payload; - } - - public String getPayload() { - return this.payload; - } - -} ----- -==== - -As you can see, we send each message and then await for the publication results. -If the messages can't be routed, then correlation data is populated with the returned message before the future is completed. - -IMPORTANT: The correlation data must be provided with a unique `id` so that the framework can perform the correlation. - -You cannot set both `useConfirmHeader` and `confirmAckChannel` but you can still receive returned messages in the error channel when `useConfirmHeader` is true, but using the correlation header is more convenient. - -[[rabbitmq-stream-producer]] -== Initial Producer Support for the RabbitMQ Stream Plugin - -Basic support for the https://rabbitmq.com/stream.html[RabbitMQ Stream Plugin] is now provided. -To enable this feature, you must add the `spring-rabbit-stream` jar to the class path - it must be the same version as `spring-amqp` and `spring-rabbit`. - -IMPORTANT: The producer properties described above are not supported when you set the `producerType` property to `STREAM_SYNC` or `STREAM_ASYNC`. - -To configure the binder to use a stream `ProducerType`, Spring Boot will configure an `Environment` `@Bean` from the applicaation properties. -You can, optionally, add a customizer to customize the message handler. - -==== -[source, java] ----- -@Bean -ProducerMessageHandlerCustomizer handlerCustomizer() { - return (hand, dest) -> { - RabbitStreamMessageHandler handler = (RabbitStreamMessageHandler) hand; - handler.setConfirmTimeout(5000); - ((RabbitStreamTemplate) handler.getStreamOperations()).setProducerCustomizer( - (name, builder) -> { - ... - }); - }; -} ----- -==== - -Refer to the https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/[RabbitMQ Stream Java Client documentation] for information about configuring the environment and producer builder. - -[[rabbitmq-super-stream-producer]] -=== Producer Support for the RabbitMQ Super Streams - -See https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-super-streams[Super Streams] for information about super streams. - -Use of super streams allows for automatic scale-up scale-down with a single active consumer on each partition of a super stream. -Using Spring Cloud Stream, you can publish to a super stream either over AMQP, or using the stream client. - -IMPORTANT: The super stream must already exist; creating a super stream is not supported by producer bindings. - -Publishing to a super stream over AMQP: - -==== -[source, properties] ----- -spring.cloud.stream.bindings.output.destination=super -spring.cloud.stream.bindings.output.producer.partition-count=3 -spring.cloud.stream.bindings.output.producer.partition-key-expression=headers['cust-no'] -spring.cloud.stream.rabbit.bindings.output.producer.declare-exchange=false ----- -==== - -Publishing to a super stream using the stream client: - -==== -[source, properties] ----- -spring.cloud.stream.bindings.output.destination=super -spring.cloud.stream.bindings.output.producer.partition-count=3 -spring.cloud.stream.bindings.output.producer.partition-key-expression=headers['cust-no'] -spring.cloud.stream.rabbit.bindings.output.producer.producer-type=stream-async -spring.cloud.stream.rabbit.bindings.output.producer.super-stream=true -spring.cloud.stream.rabbit.bindings.output.producer.declare-exchange=false ----- -==== - -When using the stream client, if you set a `confirmAckChannel`, a copy of a successfully sent message will be sent to that channel. - -[[using-existing-queues/exchanges]] -= Using Existing Queues/Exchanges - -By default, the binder will automatically provision a topic exchange with the name being derived from the value of the destination binding property ``. -The destination defaults to the binding name, if not provided. -When binding a consumer, a queue will automatically be provisioned with the name `.` (if a `group` binding property is specified), or an anonymous, auto-delete queue when there is no `group`. -The queue will be bound to the exchange with the "match-all" wildcard routing key (`#`) for a non-partitioned binding or `-` for a partitioned binding. -The prefix is an empty `String` by default. -If an output binding is specified with `requiredGroups`, a queue/binding will be provisioned for each group. - -There are a number of rabbit-specific binding properties that allow you to modify this default behavior. - -If you have an existing exchange/queue that you wish to use, you can completely disable automatic provisioning as follows, assuming the exchange is named `myExchange` and the queue is named `myQueue`: - -* `spring.cloud.stream.bindings..destination=myExchange` -* `spring.cloud.stream.bindings..group=myQueue` -* `spring.cloud.stream.rabbit.bindings..consumer.bindQueue=false` -* `spring.cloud.stream.rabbit.bindings..consumer.declareExchange=false` -* `spring.cloud.stream.rabbit.bindings..consumer.queueNameGroupOnly=true` - -If you want the binder to provision the queue/exchange, but you want to do it using something other than the defaults discussed here, use the following properties. -Refer to the property documentation above for more information. - -* `spring.cloud.stream.rabbit.bindings..consumer.bindingRoutingKey=myRoutingKey` -* `spring.cloud.stream.rabbit.bindings..consumer.exchangeType=` - -* `spring.cloud.stream.rabbit.bindings..producer.routingKeyExpression='myRoutingKey'` - -There are similar properties used when declaring a dead-letter exchange/queue, when `autoBindDlq` is `true`. - -[[retry-with-the-rabbitmq-binder]] -= Retry With the RabbitMQ Binder - -When retry is enabled within the binder, the listener container thread is suspended for any back off periods that are configured. -This might be important when strict ordering is required with a single consumer. However, for other use cases, it prevents other messages from being processed on that thread. -An alternative to using binder retry is to set up dead lettering with time to live on the dead-letter queue (DLQ) as well as dead-letter configuration on the DLQ itself. -See "`<>`" for more information about the properties discussed here. -You can use the following example configuration to enable this feature: - -* Set `autoBindDlq` to `true`. -The binder create a DLQ. -Optionally, you can specify a name in `deadLetterQueueName`. -* Set `dlqTtl` to the back off time you want to wait between redeliveries. -* Set the `dlqDeadLetterExchange` to the default exchange. -Expired messages from the DLQ are routed to the original queue, because the default `deadLetterRoutingKey` is the queue name (`destination.group`). -Setting to the default exchange is achieved by setting the property with no value, as shown in the next example. - -To force a message to be dead-lettered, either throw an `AmqpRejectAndDontRequeueException` or set `requeueRejected` to `false` (the default) and throw any exception. - -The loop continue without end, which is fine for transient problems, but you may want to give up after some number of attempts. -Fortunately, RabbitMQ provides the `x-death` header, which lets you determine how many cycles have occurred. - -To acknowledge a message after giving up, throw an `ImmediateAcknowledgeAmqpException`. - -[[putting-it-all-together]] -== Putting it All Together - -The following configuration creates an exchange `myDestination` with queue `myDestination.consumerGroup` bound to a topic exchange with a wildcard routing key `#`: - -[source] ---- -spring.cloud.stream.bindings.input.destination=myDestination -spring.cloud.stream.bindings.input.group=consumerGroup -#disable binder retries -spring.cloud.stream.bindings.input.consumer.max-attempts=1 -#dlx/dlq setup -spring.cloud.stream.rabbit.bindings.input.consumer.auto-bind-dlq=true -spring.cloud.stream.rabbit.bindings.input.consumer.dlq-ttl=5000 -spring.cloud.stream.rabbit.bindings.input.consumer.dlq-dead-letter-exchange= ---- - -This configuration creates a DLQ bound to a direct exchange (`DLX`) with a routing key of `myDestination.consumerGroup`. -When messages are rejected, they are routed to the DLQ. -After 5 seconds, the message expires and is routed to the original queue by using the queue name as the routing key, as shown in the following example: - -.Spring Boot application -[source,java] ----- -@SpringBootApplication -public class XDeathApplication { - - public static void main(String[] args) { - SpringApplication.run(XDeathApplication.class, args); - } - - @Bean - public Consumer> listen() { - return message -> { - Map death = message.getHeaders().get("x-death"); - if (death != null && death.get("count").equals(3L)) { - // giving up - don't send to DLX - throw new ImmediateAcknowledgeAmqpException("Failed after 4 attempts"); - } - throw new AmqpRejectAndDontRequeueException("failed"); - }; - } - -} ----- - -Notice that the count property in the `x-death` header is a `Long`. - -[[rabbit-error-channels]] -= Error Channels - -Starting with version 1.3, the binder unconditionally sends exceptions to an error channel for each consumer destination and can also be configured to send async producer send failures to an error channel. -See "`<>`" for more information. - -RabbitMQ has two types of send failures: - -* Returned messages, -* Negatively acknowledged https://www.rabbitmq.com/confirms.html[Publisher Confirms]. - -The latter is rare. -According to the RabbitMQ documentation "[A nack] will only be delivered if an internal error occurs in the Erlang process responsible for a queue.". -You can also get a negative acknowledgment if you publish to a bounded queue with `reject-publish` queue overflow behavior. - -As well as enabling producer error channels (as described in "`<>`"), the RabbitMQ binder only sends messages to the channels if the connection factory is appropriately configured, as follows: - -* `ccf.setPublisherConfirms(true);` -* `ccf.setPublisherReturns(true);` - -When using Spring Boot configuration for the connection factory, set the following properties: - -* `spring.rabbitmq.publisher-confirms` -* `spring.rabbitmq.publisher-returns` - -The payload of the `ErrorMessage` for a returned message is a `ReturnedAmqpMessageException` with the following properties: - -* `failedMessage`: The spring-messaging `Message` that failed to be sent. -* `amqpMessage`: The raw spring-amqp `Message`. -* `replyCode`: An integer value indicating the reason for the failure (for example, 312 - No route). -* `replyText`: A text value indicating the reason for the failure (for example, `NO_ROUTE`). -* `exchange`: The exchange to which the message was published. -* `routingKey`: The routing key used when the message was published. - -Also see <> for an alternative mechanism to receive returned messages. - -For negatively acknowledged confirmations, the payload is a `NackedAmqpMessageException` with the following properties: - -* `failedMessage`: The spring-messaging `Message` that failed to be sent. -* `nackReason`: A reason (if available -- you may need to examine the broker logs for more information). - -There is no automatic handling of these exceptions (such as sending to a <>). -You can consume these exceptions with your own Spring Integration flow. - -[[rabbit-binder-health-indicator]] -= Rabbit Binder Health Indicator - -The health indicator for Rabbit binder delegates to the one provided from Spring Boot. -For more information on this, see https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.endpoints.health.auto-configured-health-indicators[this]. - -You can disable this health indicator at the binder level by using the property - `management.health.binders.enabled` and set this to `false`. -In the case of multi-binder environments, this has to be set on the binder's environment properties. - -When the health indicator is disabled, you should see something like the below in the health actuator endpoint: - -``` -"rabbit": { - "status": "UNKNOWN" -} -``` - -At the Spring Boot level, if you want to disable the Rabbit health indicator, you need to use the property `management.health.rabbit.enabled` and set to `false`. diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/advanced-listener-container-configuration.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/advanced-listener-container-configuration.adoc new file mode 100644 index 000000000..0c5fecdc3 --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/advanced-listener-container-configuration.adoc @@ -0,0 +1,7 @@ +[[advanced-listener-container-configuration]] += Advanced Listener Container Configuration + +To set listener container properties that are not exposed as binder or binding properties, add a single bean of type `ListenerContainerCustomizer` to the application context. +The binder and binding properties will be set and then the customizer will be called. +The customizer (`configure()` method) is provided with the queue name as well as the consumer group as arguments. + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/advanced-queue/exchange/binding-configuration.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/advanced-queue/exchange/binding-configuration.adoc new file mode 100644 index 000000000..336c16df4 --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/advanced-queue/exchange/binding-configuration.adoc @@ -0,0 +1,8 @@ +[[advanced-queue/exchange/binding-configuration]] += Advanced Queue/Exchange/Binding Configuration + +From time to time, the RabbitMQ team add new features that are enabled by setting some argument when declaring, for example, a queue. +Generally, such features are enabled in the binder by adding appropriate properties, but this may not be immediately available in a current version. +Starting with version 3.0.1, you can now add `DeclarableCustomizer` bean(s) to the application context to modify a `Declarable` (`Queue`, `Exchange` or `Binding`) just before the declaration is performed. +This allows you to add arguments that are not currently directly supported by the binder. + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/binder-properties.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/binder-properties.adoc new file mode 100644 index 000000000..c6c25d50f --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/binder-properties.adoc @@ -0,0 +1,37 @@ +[[rabbit-binder-properties]] += RabbitMQ Binder Properties + +By default, the RabbitMQ binder uses Spring Boot's `ConnectionFactory`. +Conseuqently, it supports all Spring Boot configuration options for RabbitMQ. +(For reference, see the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#common-application-properties[Spring Boot documentation]). +RabbitMQ configuration options use the `spring.rabbitmq` prefix. + +In addition to Spring Boot options, the RabbitMQ binder supports the following properties: + +spring.cloud.stream.rabbit.binder.adminAddresses:: +A comma-separated list of RabbitMQ management plugin URLs. +Only used when `nodes` contains more than one entry. +Each entry in this list must have a corresponding entry in `spring.rabbitmq.addresses`. +Only needed if you use a RabbitMQ cluster and wish to consume from the node that hosts the queue. +See https://docs.spring.io/spring-amqp/reference/html/_reference.html#queue-affinity[Queue Affinity and the LocalizedQueueConnectionFactory] for more information. ++ +Default: empty. +spring.cloud.stream.rabbit.binder.nodes:: +A comma-separated list of RabbitMQ node names. +When more than one entry, used to locate the server address where a queue is located. +Each entry in this list must have a corresponding entry in `spring.rabbitmq.addresses`. +Only needed if you use a RabbitMQ cluster and wish to consume from the node that hosts the queue. +See https://docs.spring.io/spring-amqp/reference/html/_reference.html#queue-affinity[Queue Affinity and the LocalizedQueueConnectionFactory] for more information. ++ +Default: empty. +spring.cloud.stream.rabbit.binder.compressionLevel:: +The compression level for compressed bindings. +See `java.util.zip.Deflater`. ++ +Default: `1` (BEST_LEVEL). +spring.cloud.stream.binder.connection-name-prefix:: +A connection name prefix used to name the connection(s) created by this binder. +The name is this prefix followed by `#n`, where `n` increments each time a new connection is opened. ++ +Default: none (Spring AMQP default). + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/prod-props.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/prod-props.adoc new file mode 100644 index 000000000..efc497707 --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/prod-props.adoc @@ -0,0 +1,313 @@ +[[rabbit-prod-props]] += Rabbit Producer Properties + +The following properties are available for Rabbit producers only and must be prefixed with `spring.cloud.stream.rabbit.bindings..producer.`. + +However if the same set of properties needs to be applied to most bindings, to +avoid repetition, Spring Cloud Stream supports setting values for all channels, +in the format of `spring.cloud.stream.rabbit.default.=`. + +Also, keep in mind that binding specific property will override its equivalent in the default. + +altermateExchange.binding.queue:: +If the exchange does not already exist, and a `name` is provided, bind this queue to the alternate exhange. +A simple durable queue with no arguments is provisioned; if more sophisticated configuration is required, you must configure and bind the queue yourself. ++ +Default: `null` +alternateExchange.binding.routingKey +If the exchange does not already exist, and a `name` and `queue` is provided, bind the queue to the alternate exhange using this routing key. ++ +Default: `#` (for the default `topic` alternate exchange) +alternateExchange.exists:: +Whether the alternate exchange exists, or needs to be provisioned. ++ +Default: `false` +alternateExchange.type:: +If the alternate exchange does not already exist, the type of exchange to provision. ++ +Default: `topic` +alternateExchange.name:: +Configure an alternate exchange on the destination exchange. ++ +Default: `null` +autoBindDlq:: +Whether to automatically declare the DLQ and bind it to the binder DLX. ++ +Default: `false`. +batchingEnabled:: +Whether to enable message batching by producers. +Messages are batched into one message according to the following properties (described in the next three entries in this list): 'batchSize', `batchBufferLimit`, and `batchTimeout`. +See https://docs.spring.io/spring-amqp//reference/html/_reference.html#template-batching[Batching] for more information. +Also see <>. ++ +Default: `false`. +batchSize:: +The number of messages to buffer when batching is enabled. ++ +Default: `100`. +batchBufferLimit:: +The maximum buffer size when batching is enabled. ++ +Default: `10000`. +batchTimeout:: +The batch timeout when batching is enabled. ++ +Default: `5000`. +bindingRoutingKey:: +The routing key with which to bind the queue to the exchange (if `bindQueue` is `true`). +Can be multiple keys - see `bindingRoutingKeyDelimiter`. +For partitioned destinations, `-n` is appended to each key. +Only applies if `requiredGroups` are provided and then only to those groups. ++ +Default: `#`. +bindingRoutingKeyDelimiter:: +When this is not null, 'bindingRoutingKey' is considered to be a list of keys delimited by this value; often a comma is used. +Only applies if `requiredGroups` are provided and then only to those groups. ++ +Default: `null`. +bindQueue:: +Whether to declare the queue and bind it to the destination exchange. +Set it to `false` if you have set up your own infrastructure and have previously created and bound the queue. +Only applies if `requiredGroups` are provided and then only to those groups. ++ +Default: `true`. +compress:: +Whether data should be compressed when sent. ++ +Default: `false`. +confirmAckChannel:: +When `errorChannelEnabled` is true, a channel to which to send positive delivery acknowledgments (aka publisher confirms). +If the channel does not exist, a `DirectChannel` is registered with this name. +The connection factory must be configured to enable publisher confirms. +Mutually exclusive with `useConfirmHeader`. ++ +Default: `nullChannel` (acks are discarded). +deadLetterQueueName:: +The name of the DLQ +Only applies if `requiredGroups` are provided and then only to those groups. ++ +Default: `prefix+destination.dlq` +deadLetterExchange:: +A DLX to assign to the queue. +Relevant only when `autoBindDlq` is `true`. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: 'prefix+DLX' +deadLetterExchangeType:: +The type of the DLX to assign to the queue. +Relevant only if `autoBindDlq` is `true`. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: 'direct' +deadLetterRoutingKey:: +A dead letter routing key to assign to the queue. +Relevant only when `autoBindDlq` is `true`. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `destination` +declareDlx:: +Whether to declare the dead letter exchange for the destination. +Relevant only if `autoBindDlq` is `true`. +Set to `false` if you have a pre-configured DLX. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `true`. +declareExchange:: +Whether to declare the exchange for the destination. ++ +Default: `true`. +delayExpression:: +A SpEL expression to evaluate the delay to apply to the message (`x-delay` header). +It has no effect if the exchange is not a delayed message exchange. ++ +Default: No `x-delay` header is set. +delayedExchange:: +Whether to declare the exchange as a `Delayed Message Exchange`. +Requires the delayed message exchange plugin on the broker. +The `x-delayed-type` argument is set to the `exchangeType`. ++ +Default: `false`. +deliveryMode:: +The delivery mode. ++ +Default: `PERSISTENT`. +dlqBindingArguments:: +Arguments applied when binding the dlq to the dead letter exchange; used with `headers` `deadLetterExchangeType` to specify headers to match on. +For example `...dlqBindingArguments.x-match=any`, `...dlqBindingArguments.someHeader=someValue`. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: empty +dlqDeadLetterExchange:: +When a DLQ is declared, a DLX to assign to that queue. +Applies only if `requiredGroups` are provided and then only to those groups. ++ +Default: `none` +dlqDeadLetterRoutingKey:: +When a DLQ is declared, a dead letter routing key to assign to that queue. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `none` +dlqExpires:: +How long (in milliseconds) before an unused dead letter queue is deleted. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `no expiration` +dlqLazy:: +Declare the dead letter queue with the `x-queue-mode=lazy` argument. +See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. +Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +dlqMaxLength:: +Maximum number of messages in the dead letter queue. +Applies only if `requiredGroups` are provided and then only to those groups. ++ +Default: `no limit` +dlqMaxLengthBytes:: +Maximum number of total bytes in the dead letter queue from all messages. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `no limit` +dlqMaxPriority:: +Maximum priority of messages in the dead letter queue (0-255) +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `none` +dlqQuorum.deliveryLimit:: +When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: none - broker default will apply. +dlqQuorum.enabled:: +When true, create a quorum dead letter queue instead of a classic queue. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: false +dlqQuorum.initialGroupSize:: +When `quorum.enabled=true`, set the initial quorum size. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: none - broker default will apply. +dlqSingleActiveConsumer:: +Set to true to set the `x-single-active-consumer` queue property to true. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `false` +dlqTtl:: +Default time (in milliseconds) to live to apply to the dead letter queue when declared. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `no limit` +exchangeAutoDelete:: +If `declareExchange` is `true`, whether the exchange should be auto-delete (it is removed after the last queue is removed). ++ +Default: `true`. +exchangeDurable:: +If `declareExchange` is `true`, whether the exchange should be durable (survives broker restart). ++ +Default: `true`. +exchangeType:: +The exchange type: `direct`, `fanout`, `headers` or `topic` for non-partitioned destinations and `direct`, `headers` or `topic` for partitioned destinations. ++ +Default: `topic`. +expires:: +How long (in milliseconds) before an unused queue is deleted. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `no expiration` +headerPatterns:: +Patterns for headers to be mapped to outbound messages. ++ +Default: `['*']` (all headers). +lazy:: +Declare the queue with the `x-queue-mode=lazy` argument. +See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. +Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `false`. +maxLength:: +Maximum number of messages in the queue. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `no limit` +maxLengthBytes:: +Maximum number of total bytes in the queue from all messages. +Only applies if `requiredGroups` are provided and then only to those groups. ++ +Default: `no limit` +maxPriority:: +Maximum priority of messages in the queue (0-255). +Only applies if `requiredGroups` are provided and then only to those groups. ++ +Default: `none` +prefix:: +A prefix to be added to the name of the `destination` exchange. ++ +Default: "". +producerType:: +The type of producer. +* `AMQP` AMQP Client for classic and quorum queues +* `STREAM_SYNC` RabbitMQ Streams Plugin client, blocks until confirm is received +* `STREAM_ASYNC` RabbitMQ Streams Plugin client, does not block ++ +Default: "". +queueBindingArguments:: +Arguments applied when binding the queue to the exchange; used with `headers` `exchangeType` to specify headers to match on. +For example `...queueBindingArguments.x-match=any`, `...queueBindingArguments.someHeader=someValue`. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: empty +queueNameGroupOnly:: +When `true`, consume from a queue with a name equal to the `group`. +Otherwise the queue name is `destination.group`. +This is useful, for example, when using Spring Cloud Stream to consume from an existing RabbitMQ queue. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: false. +quorum.deliveryLimit:: +When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: none - broker default will apply. +quorum.enabled:: +When true, create a quorum queue instead of a classic queue. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: false +quorum.initialGroupSize:: +When `quorum.enabled=true`, set the initial quorum size. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: none - broker default will apply. +routingKeyExpression:: +A SpEL expression to determine the routing key to use when publishing messages. +For a fixed routing key, use `routingKey`. ++ +Default: `destination` or `destination-` for partitioned destinations. +routingKey:: +A string defining a fixed routing key to use when publishing messages. ++ +Default: see `routingKeyExpression` +singleActiveConsumer:: +Set to true to set the `x-single-active-consumer` queue property to true. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `false` +transacted:: +Whether to use transacted channels. ++ +Default: `false`. +ttl:: +Default time (in milliseconds) to live to apply to the queue when declared. +Applies only when `requiredGroups` are provided and then only to those groups. ++ +Default: `no limit` +useConfirmHeader:: +See <>. +Mutually exclusive with `confirmAckChannel`. ++ + +NOTE: In the case of RabbitMQ, content type headers can be set by external applications. +Spring Cloud Stream supports them as part of an extended internal protocol used for any type of transport -- including transports, such as Kafka (prior to 0.11), that do not natively support headers. + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/publisher-confirms.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/publisher-confirms.adoc new file mode 100644 index 000000000..f5d897ed1 --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/publisher-confirms.adoc @@ -0,0 +1,117 @@ +[[publisher-confirms]] += Publisher Confirms + +There are two mechanisms to get the result of publishing a message; in each case, the connection factory must have `publisherConfirmType` set `ConfirmType.CORRELATED`. +The "legacy" mechanism is to set the `confirmAckChannel` to the bean name of a message channel from which you can retrieve the confirmations asynchronously; negative acks are sent to the error channel (if enabled) - see <>. + +The preferred mechanism, added in version 3.1 is to use a correlation data header and wait for the result via its `Future` property. +This is particularly useful with a batch listener because you can send multiple messages before waiting for the result. +To use this technique, set the `useConfirmHeader` property to true +The following simple application is an example of using this technique: + +==== +[source, properties] +---- +spring.cloud.stream.bindings.input-in-0.group=someGroup +spring.cloud.stream.bindings.input-in-0.consumer.batch-mode=true + +spring.cloud.stream.source=output +spring.cloud.stream.bindings.output-out-0.producer.error-channel-enabled=true + +spring.cloud.stream.rabbit.bindings.output-out-0.producer.useConfirmHeader=true +spring.cloud.stream.rabbit.bindings.input-in-0.consumer.auto-bind-dlq=true +spring.cloud.stream.rabbit.bindings.input-in-0.consumer.batch-size=10 + +spring.rabbitmq.publisher-confirm-type=correlated +spring.rabbitmq.publisher-returns=true +---- +==== + +==== +[source, java] +---- +@SpringBootApplication +public class Application { + + private static final Logger log = LoggerFactory.getLogger(Application.class); + + public static void main(String[] args) { + SpringApplication.run(Application.class, args); + } + + @Autowired + private StreamBridge bridge; + + @Bean + Consumer> input() { + return list -> { + List results = new ArrayList<>(); + list.forEach(str -> { + log.info("Received: " + str); + MyCorrelationData corr = new MyCorrelationData(UUID.randomUUID().toString(), str); + results.add(corr); + this.bridge.send("output-out-0", MessageBuilder.withPayload(str.toUpperCase()) + .setHeader(AmqpHeaders.PUBLISH_CONFIRM_CORRELATION, corr) + .build()); + }); + results.forEach(correlation -> { + try { + Confirm confirm = correlation.getFuture().get(10, TimeUnit.SECONDS); + log.info(confirm + " for " + correlation.getPayload()); + if (correlation.getReturnedMessage() != null) { + log.error("Message for " + correlation.getPayload() + " was returned "); + + // throw some exception to invoke binder retry/error handling + + } + } + catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException(e); + } + catch (ExecutionException | TimeoutException e) { + throw new IllegalStateException(e); + } + }); + }; + } + + + @Bean + public ApplicationRunner runner(BatchingRabbitTemplate template) { + return args -> IntStream.range(0, 10).forEach(i -> + template.convertAndSend("input-in-0", "input-in-0.rbgh303", "foo" + i)); + } + + @Bean + public BatchingRabbitTemplate template(CachingConnectionFactory cf, TaskScheduler taskScheduler) { + BatchingStrategy batchingStrategy = new SimpleBatchingStrategy(10, 1000000, 1000); + return new BatchingRabbitTemplate(cf, batchingStrategy, taskScheduler); + } + +} + +class MyCorrelationData extends CorrelationData { + + private final String payload; + + MyCorrelationData(String id, String payload) { + super(id); + this.payload = payload; + } + + public String getPayload() { + return this.payload; + } + +} +---- +==== + +As you can see, we send each message and then await for the publication results. +If the messages can't be routed, then correlation data is populated with the returned message before the future is completed. + +IMPORTANT: The correlation data must be provided with a unique `id` so that the framework can perform the correlation. + +You cannot set both `useConfirmHeader` and `confirmAckChannel` but you can still receive returned messages in the error channel when `useConfirmHeader` is true, but using the correlation header is more convenient. + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/putting-it-all-together.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/putting-it-all-together.adoc new file mode 100644 index 000000000..e1749d085 --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/putting-it-all-together.adoc @@ -0,0 +1,110 @@ +[[putting-it-all-together]] += Putting it All Together + +The following configuration creates an exchange `myDestination` with queue `myDestination.consumerGroup` bound to a topic exchange with a wildcard routing key `#`: + +[source] +--- +spring.cloud.stream.bindings.input.destination=myDestination +spring.cloud.stream.bindings.input.group=consumerGroup +#disable binder retries +spring.cloud.stream.bindings.input.consumer.max-attempts=1 +#dlx/dlq setup +spring.cloud.stream.rabbit.bindings.input.consumer.auto-bind-dlq=true +spring.cloud.stream.rabbit.bindings.input.consumer.dlq-ttl=5000 +spring.cloud.stream.rabbit.bindings.input.consumer.dlq-dead-letter-exchange= +--- + +This configuration creates a DLQ bound to a direct exchange (`DLX`) with a routing key of `myDestination.consumerGroup`. +When messages are rejected, they are routed to the DLQ. +After 5 seconds, the message expires and is routed to the original queue by using the queue name as the routing key, as shown in the following example: + +.Spring Boot application +[source,java] +---- +@SpringBootApplication +public class XDeathApplication { + + public static void main(String[] args) { + SpringApplication.run(XDeathApplication.class, args); + } + + @Bean + public Consumer> listen() { + return message -> { + Map death = message.getHeaders().get("x-death"); + if (death != null && death.get("count").equals(3L)) { + // giving up - don't send to DLX + throw new ImmediateAcknowledgeAmqpException("Failed after 4 attempts"); + } + throw new AmqpRejectAndDontRequeueException("failed"); + }; + } + +} +---- + +Notice that the count property in the `x-death` header is a `Long`. + +[[rabbit-error-channels]] += Error Channels + +Starting with version 1.3, the binder unconditionally sends exceptions to an error channel for each consumer destination and can also be configured to send async producer send failures to an error channel. +See "`<>`" for more information. + +RabbitMQ has two types of send failures: + +* Returned messages, +* Negatively acknowledged https://www.rabbitmq.com/confirms.html[Publisher Confirms]. + +The latter is rare. +According to the RabbitMQ documentation "[A nack] will only be delivered if an internal error occurs in the Erlang process responsible for a queue.". +You can also get a negative acknowledgment if you publish to a bounded queue with `reject-publish` queue overflow behavior. + +As well as enabling producer error channels (as described in "`<>`"), the RabbitMQ binder only sends messages to the channels if the connection factory is appropriately configured, as follows: + +* `ccf.setPublisherConfirms(true);` +* `ccf.setPublisherReturns(true);` + +When using Spring Boot configuration for the connection factory, set the following properties: + +* `spring.rabbitmq.publisher-confirms` +* `spring.rabbitmq.publisher-returns` + +The payload of the `ErrorMessage` for a returned message is a `ReturnedAmqpMessageException` with the following properties: + +* `failedMessage`: The spring-messaging `Message` that failed to be sent. +* `amqpMessage`: The raw spring-amqp `Message`. +* `replyCode`: An integer value indicating the reason for the failure (for example, 312 - No route). +* `replyText`: A text value indicating the reason for the failure (for example, `NO_ROUTE`). +* `exchange`: The exchange to which the message was published. +* `routingKey`: The routing key used when the message was published. + +Also see <> for an alternative mechanism to receive returned messages. + +For negatively acknowledged confirmations, the payload is a `NackedAmqpMessageException` with the following properties: + +* `failedMessage`: The spring-messaging `Message` that failed to be sent. +* `nackReason`: A reason (if available -- you may need to examine the broker logs for more information). + +There is no automatic handling of these exceptions (such as sending to a <>). +You can consume these exceptions with your own Spring Integration flow. + +[[rabbit-binder-health-indicator]] += Rabbit Binder Health Indicator + +The health indicator for Rabbit binder delegates to the one provided from Spring Boot. +For more information on this, see https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.endpoints.health.auto-configured-health-indicators[this]. + +You can disable this health indicator at the binder level by using the property - `management.health.binders.enabled` and set this to `false`. +In the case of multi-binder environments, this has to be set on the binder's environment properties. + +When the health indicator is disabled, you should see something like the below in the health actuator endpoint: + +``` +"rabbit": { + "status": "UNKNOWN" +} +``` + +At the Spring Boot level, if you want to disable the Rabbit health indicator, you need to use the property `management.health.rabbit.enabled` and set to `false`. diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-consumer-properties.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-consumer-properties.adoc new file mode 100644 index 000000000..288d516de --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-consumer-properties.adoc @@ -0,0 +1,298 @@ +[[rabbitmq-consumer-properties]] += RabbitMQ Consumer Properties + +The following properties are available for Rabbit consumers only and must be prefixed with `spring.cloud.stream.rabbit.bindings..consumer.`. + +However if the same set of properties needs to be applied to most bindings, to +avoid repetition, Spring Cloud Stream supports setting values for all channels, +in the format of `spring.cloud.stream.rabbit.default.=`. + +Also, keep in mind that binding specific property will override its equivalent in the default. + +acknowledgeMode:: +The acknowledge mode. ++ +Default: `AUTO`. +anonymousGroupPrefix:: +When the binding has no `group` property, an anonymous, auto-delete queue is bound to the destination exchange. +The default naming stragegy for such queues results in a queue named `anonymous.`. +Set this property to change the prefix to something other than the default. ++ +Default: `anonymous.`. +autoBindDlq:: +Whether to automatically declare the DLQ and bind it to the binder DLX. ++ +Default: `false`. +bindingRoutingKey:: +The routing key with which to bind the queue to the exchange (if `bindQueue` is `true`). +Can be multiple keys - see `bindingRoutingKeyDelimiter`. +For partitioned destinations, `-` is appended to each key. ++ +Default: `#`. +bindingRoutingKeyDelimiter:: +When this is not null, 'bindingRoutingKey' is considered to be a list of keys delimited by this value; often a comma is used. ++ +Default: `null`. +bindQueue:: +Whether to declare the queue and bind it to the destination exchange. +Set it to `false` if you have set up your own infrastructure and have previously created and bound the queue. ++ +Default: `true`. +consumerTagPrefix:: +Used to create the consumer tag(s); will be appended by `#n` where `n` increments for each consumer created. +Example: `${spring.application.name}-${spring.cloud.stream.bindings.input.group}-${spring.cloud.stream.instance-index}`. ++ +Default: none - the broker will generate random consumer tags. +containerType:: +Select the type of listener container to be used. +See https://docs.spring.io/spring-amqp/reference/html/_reference.html#choose-container[Choosing a Container] in the Spring AMQP documentation for more information. +Also see <>. ++ +Default: `simple` +deadLetterQueueName:: +The name of the DLQ ++ +Default: `prefix+destination.dlq` +deadLetterExchange:: +A DLX to assign to the queue. +Relevant only if `autoBindDlq` is `true`. ++ +Default: 'prefix+DLX' +deadLetterExchangeType:: +The type of the DLX to assign to the queue. +Relevant only if `autoBindDlq` is `true`. ++ +Default: 'direct' +deadLetterRoutingKey:: +A dead letter routing key to assign to the queue. +Relevant only if `autoBindDlq` is `true`. ++ +Default: `destination` +declareDlx:: +Whether to declare the dead letter exchange for the destination. +Relevant only if `autoBindDlq` is `true`. +Set to `false` if you have a pre-configured DLX. ++ +Default: `true`. +declareExchange:: +Whether to declare the exchange for the destination. ++ +Default: `true`. +delayedExchange:: +Whether to declare the exchange as a `Delayed Message Exchange`. +Requires the delayed message exchange plugin on the broker. +The `x-delayed-type` argument is set to the `exchangeType`. ++ +Default: `false`. +dlqBindingArguments:: +Arguments applied when binding the dlq to the dead letter exchange; used with `headers` `deadLetterExchangeType` to specify headers to match on. +For example `...dlqBindingArguments.x-match=any`, `...dlqBindingArguments.someHeader=someValue`. ++ +Default: empty +dlqDeadLetterExchange:: +If a DLQ is declared, a DLX to assign to that queue. ++ +Default: `none` +dlqDeadLetterRoutingKey:: +If a DLQ is declared, a dead letter routing key to assign to that queue. ++ +Default: `none` +dlqExpires:: +How long before an unused dead letter queue is deleted (in milliseconds). ++ +Default: `no expiration` +dlqLazy:: +Declare the dead letter queue with the `x-queue-mode=lazy` argument. +See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. +Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. ++ +Default: `false`. +dlqMaxLength:: +Maximum number of messages in the dead letter queue. ++ +Default: `no limit` +dlqMaxLengthBytes:: +Maximum number of total bytes in the dead letter queue from all messages. ++ +Default: `no limit` +dlqMaxPriority:: +Maximum priority of messages in the dead letter queue (0-255). ++ +Default: `none` +dlqOverflowBehavior:: +Action to take when `dlqMaxLength` or `dlqMaxLengthBytes` is exceeded; currently `drop-head` or `reject-publish` but refer to the RabbitMQ documentation. ++ +Default: `none` +dlqQuorum.deliveryLimit:: +When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. ++ +Default: none - broker default will apply. +dlqQuorum.enabled:: +When true, create a quorum dead letter queue instead of a classic queue. ++ +Default: false +dlqQuorum.initialGroupSize:: +When `quorum.enabled=true`, set the initial quorum size. ++ +Default: none - broker default will apply. +dlqSingleActiveConsumer:: +Set to true to set the `x-single-active-consumer` queue property to true. ++ +Default: `false` +dlqTtl:: +Default time to live to apply to the dead letter queue when declared (in milliseconds). ++ +Default: `no limit` +durableSubscription:: +Whether the subscription should be durable. +Only effective if `group` is also set. ++ +Default: `true`. +exchangeAutoDelete:: +If `declareExchange` is true, whether the exchange should be auto-deleted (that is, removed after the last queue is removed). ++ +Default: `true`. +exchangeDurable:: +If `declareExchange` is true, whether the exchange should be durable (that is, it survives broker restart). ++ +Default: `true`. +exchangeType:: +The exchange type: `direct`, `fanout`, `headers` or `topic` for non-partitioned destinations and `direct`, headers or `topic` for partitioned destinations. ++ +Default: `topic`. +exclusive:: +Whether to create an exclusive consumer. +Concurrency should be 1 when this is `true`. +Often used when strict ordering is required but enabling a hot standby instance to take over after a failure. +See `recoveryInterval`, which controls how often a standby instance attempts to consume. +Consider using `singleActiveConsumer` instead when using RabbitMQ 3.8 or later. ++ +Default: `false`. +expires:: +How long before an unused queue is deleted (in milliseconds). ++ +Default: `no expiration` +failedDeclarationRetryInterval:: +The interval (in milliseconds) between attempts to consume from a queue if it is missing. ++ +Default: 5000 +[[spring-cloud-stream-rabbit-frame-max-headroom]] +frameMaxHeadroom:: +The number of bytes to reserve for other headers when adding the stack trace to a DLQ message header. +All headers must fit within the `frame_max` size configured on the broker. +Stack traces can be large; if the size plus this property exceeds `frame_max` then the stack trace will be truncated. +A WARN log will be written; consider increasing the `frame_max` or reducing the stack trace by catching the exception and throwing one with a smaller stack trace. ++ +Default: 20000 +headerPatterns:: +Patterns for headers to be mapped from inbound messages. ++ +Default: `['*']` (all headers). +lazy:: +Declare the queue with the `x-queue-mode=lazy` argument. +See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. +Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. ++ +Default: `false`. +maxConcurrency:: +The maximum number of consumers. +Not supported when the `containerType` is `direct`. ++ +Default: `1`. +maxLength:: +The maximum number of messages in the queue. ++ +Default: `no limit` +maxLengthBytes:: +The maximum number of total bytes in the queue from all messages. ++ +Default: `no limit` +maxPriority:: +The maximum priority of messages in the queue (0-255). ++ +Default: `none` +missingQueuesFatal:: +When the queue cannot be found, whether to treat the condition as fatal and stop the listener container. +Defaults to `false` so that the container keeps trying to consume from the queue -- for example, when using a cluster and the node hosting a non-HA queue is down. ++ +Default: `false` +overflowBehavior:: +Action to take when `maxLength` or `maxLengthBytes` is exceeded; currently `drop-head` or `reject-publish` but refer to the RabbitMQ documentation. ++ +Default: `none` +prefetch:: +Prefetch count. ++ +Default: `1`. +prefix:: +A prefix to be added to the name of the `destination` and queues. ++ +Default: "". +queueBindingArguments:: +Arguments applied when binding the queue to the exchange; used with `headers` `exchangeType` to specify headers to match on. +For example `...queueBindingArguments.x-match=any`, `...queueBindingArguments.someHeader=someValue`. ++ +Default: empty +queueDeclarationRetries:: +The number of times to retry consuming from a queue if it is missing. +Relevant only when `missingQueuesFatal` is `true`. +Otherwise, the container keeps retrying indefinitely. +Not supported when the `containerType` is `direct`. ++ +Default: `3` +queueNameGroupOnly:: +When true, consume from a queue with a name equal to the `group`. +Otherwise the queue name is `destination.group`. +This is useful, for example, when using Spring Cloud Stream to consume from an existing RabbitMQ queue. ++ +Default: false. +quorum.deliveryLimit:: +When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. ++ +Default: none - broker default will apply. +quorum.enabled:: +When true, create a quorum queue instead of a classic queue. ++ +Default: false +quorum.initialGroupSize:: +When `quorum.enabled=true`, set the initial quorum size. ++ +Default: none - broker default will apply. +recoveryInterval:: +The interval between connection recovery attempts, in milliseconds. ++ +Default: `5000`. +requeueRejected:: +Whether delivery failures should be re-queued when retry is disabled or `republishToDlq` is `false`. ++ +Default: `false`. +[[spring-cloud-stream-rabbit-republish-delivery-mode]] +republishDeliveryMode:: +When `republishToDlq` is `true`, specifies the delivery mode of the republished message. ++ +Default: `DeliveryMode.PERSISTENT` +republishToDlq:: +By default, messages that fail after retries are exhausted are rejected. +If a dead-letter queue (DLQ) is configured, RabbitMQ routes the failed message (unchanged) to the DLQ. +If set to `true`, the binder republishs failed messages to the DLQ with additional headers, including the exception message and stack trace from the cause of the final failure. +Also see the <>. ++ +Default: `true` +singleActiveConsumer:: +Set to true to set the `x-single-active-consumer` queue property to true. ++ +Default: `false` +transacted:: +Whether to use transacted channels. ++ +Default: `false`. +ttl:: +Default time to live to apply to the queue when declared (in milliseconds). ++ +Default: `no limit` +txSize:: +The number of deliveries between acks. +Not supported when the `containerType` is `direct`. ++ +Default: `1`. + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-stream-consumer.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-stream-consumer.adoc new file mode 100644 index 000000000..86ce76dc6 --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-stream-consumer.adoc @@ -0,0 +1,88 @@ +[[rabbitmq-stream-consumer]] += Initial Consumer Support for the RabbitMQ Stream Plugin + +Basic support for the https://rabbitmq.com/stream.html[RabbitMQ Stream Plugin] is now provided. +To enable this feature, you must add the `spring-rabbit-stream` jar to the class path - it must be the same version as `spring-amqp` and `spring-rabbit`. + +IMPORTANT: The consumer properties described above are not supported when you set the `containerType` property to `stream`; `concurrency` is supported for super streams only. +Only a single stream queue can be consumed by each binding. + +To configure the binder to use `containerType=stream`, Spring Boot will automatically configure an `Environment` `@Bean` from the application properties. +You can, optionally, add a customizer to customize the listener container. + +==== +[source, java] +---- +@Bean +ListenerContainerCustomizer customizer() { + return (cont, dest, group) -> { + StreamListenerContainer container = (StreamListenerContainer) cont; + container.setConsumerCustomizer((name, builder) -> { + builder.offset(OffsetSpecification.first()); + }); + // ... + }; +} +---- +==== + +The `name` argument passed to the customizer is `destination + '.' + group + '.container'`. + +The stream `name()` (for the purpose of offset tracking) is set to the binding `destination + '.' + group`. +It can be changed using a `ConsumerCustomizer` shown above. +If you decide to use manual offset tracking, the `Context` is available as a message header: + +==== +[source, java] +---- +int count; + +@Bean +public Consumer> input() { + return msg -> { + System.out.println(msg); + if (++count % 1000 == 0) { + Context context = msg.getHeaders().get("rabbitmq_streamContext", Context.class); + context.consumer().store(context.offset()); + } + }; +} +---- +==== + +Refer to the https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/[RabbitMQ Stream Java Client documentation] for information about configuring the environment and consumer builder. + +[[rabbitmq-super-stream-consumer]] +== Consumer Support for the RabbitMQ Super Streams + +See https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-super-streams[Super Streams] for information about super streams. + +Use of super streams allows for automatic scale-up scale-down with a single active consumer on each partition of a super stream. + +Configuration example: + +==== +[source, java] +---- +@Bean +public Consumer input() { + ... +} +---- +==== + +==== +[source, properties] +---- +spring.cloud.stream.bindings.input-in-0.destination=super +spring.cloud.stream.bindings.input-in-0.group=test +spring.cloud.stream.bindings.input-in-0.consumer.instance-count=3 +spring.cloud.stream.bindings.input-in-0.consumer.concurrency=3 +spring.cloud.stream.rabbit.bindings.input-in-0.consumer.container-type=STREAM +spring.cloud.stream.rabbit.bindings.input-in-0.consumer.super-stream=true +---- +==== + +The framework will create a super stream named `super`, with 9 partitions. +Up to 3 instances of this application can be deployed. + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-stream-producer.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-stream-producer.adoc new file mode 100644 index 000000000..cd8a59b8d --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/rabbitmq-stream-producer.adoc @@ -0,0 +1,122 @@ +[[rabbitmq-stream-producer]] += Initial Producer Support for the RabbitMQ Stream Plugin + +Basic support for the https://rabbitmq.com/stream.html[RabbitMQ Stream Plugin] is now provided. +To enable this feature, you must add the `spring-rabbit-stream` jar to the class path - it must be the same version as `spring-amqp` and `spring-rabbit`. + +IMPORTANT: The producer properties described above are not supported when you set the `producerType` property to `STREAM_SYNC` or `STREAM_ASYNC`. + +To configure the binder to use a stream `ProducerType`, Spring Boot will configure an `Environment` `@Bean` from the applicaation properties. +You can, optionally, add a customizer to customize the message handler. + +==== +[source, java] +---- +@Bean +ProducerMessageHandlerCustomizer handlerCustomizer() { + return (hand, dest) -> { + RabbitStreamMessageHandler handler = (RabbitStreamMessageHandler) hand; + handler.setConfirmTimeout(5000); + ((RabbitStreamTemplate) handler.getStreamOperations()).setProducerCustomizer( + (name, builder) -> { + ... + }); + }; +} +---- +==== + +Refer to the https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/[RabbitMQ Stream Java Client documentation] for information about configuring the environment and producer builder. + +[[rabbitmq-super-stream-producer]] +== Producer Support for the RabbitMQ Super Streams + +See https://blog.rabbitmq.com/posts/2022/07/rabbitmq-3-11-feature-preview-super-streams[Super Streams] for information about super streams. + +Use of super streams allows for automatic scale-up scale-down with a single active consumer on each partition of a super stream. +Using Spring Cloud Stream, you can publish to a super stream either over AMQP, or using the stream client. + +IMPORTANT: The super stream must already exist; creating a super stream is not supported by producer bindings. + +Publishing to a super stream over AMQP: + +==== +[source, properties] +---- +spring.cloud.stream.bindings.output.destination=super +spring.cloud.stream.bindings.output.producer.partition-count=3 +spring.cloud.stream.bindings.output.producer.partition-key-expression=headers['cust-no'] +spring.cloud.stream.rabbit.bindings.output.producer.declare-exchange=false +---- +==== + +Publishing to a super stream using the stream client: + +==== +[source, properties] +---- +spring.cloud.stream.bindings.output.destination=super +spring.cloud.stream.bindings.output.producer.partition-count=3 +spring.cloud.stream.bindings.output.producer.partition-key-expression=headers['cust-no'] +spring.cloud.stream.rabbit.bindings.output.producer.producer-type=stream-async +spring.cloud.stream.rabbit.bindings.output.producer.super-stream=true +spring.cloud.stream.rabbit.bindings.output.producer.declare-exchange=false +---- +==== + +When using the stream client, if you set a `confirmAckChannel`, a copy of a successfully sent message will be sent to that channel. + +[[using-existing-queues/exchanges]] += Using Existing Queues/Exchanges + +By default, the binder will automatically provision a topic exchange with the name being derived from the value of the destination binding property ``. +The destination defaults to the binding name, if not provided. +When binding a consumer, a queue will automatically be provisioned with the name `.` (if a `group` binding property is specified), or an anonymous, auto-delete queue when there is no `group`. +The queue will be bound to the exchange with the "match-all" wildcard routing key (`#`) for a non-partitioned binding or `-` for a partitioned binding. +The prefix is an empty `String` by default. +If an output binding is specified with `requiredGroups`, a queue/binding will be provisioned for each group. + +There are a number of rabbit-specific binding properties that allow you to modify this default behavior. + +If you have an existing exchange/queue that you wish to use, you can completely disable automatic provisioning as follows, assuming the exchange is named `myExchange` and the queue is named `myQueue`: + +* `spring.cloud.stream.bindings..destination=myExchange` +* `spring.cloud.stream.bindings..group=myQueue` +* `spring.cloud.stream.rabbit.bindings..consumer.bindQueue=false` +* `spring.cloud.stream.rabbit.bindings..consumer.declareExchange=false` +* `spring.cloud.stream.rabbit.bindings..consumer.queueNameGroupOnly=true` + +If you want the binder to provision the queue/exchange, but you want to do it using something other than the defaults discussed here, use the following properties. +Refer to the property documentation above for more information. + +* `spring.cloud.stream.rabbit.bindings..consumer.bindingRoutingKey=myRoutingKey` +* `spring.cloud.stream.rabbit.bindings..consumer.exchangeType=` + +* `spring.cloud.stream.rabbit.bindings..producer.routingKeyExpression='myRoutingKey'` + +There are similar properties used when declaring a dead-letter exchange/queue, when `autoBindDlq` is `true`. + +[[retry-with-the-rabbitmq-binder]] += Retry With the RabbitMQ Binder + +When retry is enabled within the binder, the listener container thread is suspended for any back off periods that are configured. +This might be important when strict ordering is required with a single consumer. However, for other use cases, it prevents other messages from being processed on that thread. +An alternative to using binder retry is to set up dead lettering with time to live on the dead-letter queue (DLQ) as well as dead-letter configuration on the DLQ itself. +See "`<>`" for more information about the properties discussed here. +You can use the following example configuration to enable this feature: + +* Set `autoBindDlq` to `true`. +The binder create a DLQ. +Optionally, you can specify a name in `deadLetterQueueName`. +* Set `dlqTtl` to the back off time you want to wait between redeliveries. +* Set the `dlqDeadLetterExchange` to the default exchange. +Expired messages from the DLQ are routed to the original queue, because the default `deadLetterRoutingKey` is the queue name (`destination.group`). +Setting to the default exchange is achieved by setting the property with no value, as shown in the next example. + +To force a message to be dead-lettered, either throw an `AmqpRejectAndDontRequeueException` or set `requeueRejected` to `false` (the default) and throw any exception. + +The loop continue without end, which is fine for transient problems, but you may want to give up after some number of attempts. +Fortunately, RabbitMQ provides the `x-death` header, which lets you determine how many cycles have occurred. + +To acknowledge a message after giving up, throw an `ImmediateAcknowledgeAmqpException`. + diff --git a/docs/modules/ROOT/pages/rabbit/rabbit_overview/receiving-batch.adoc b/docs/modules/ROOT/pages/rabbit/rabbit_overview/receiving-batch.adoc new file mode 100644 index 000000000..80aa7c03c --- /dev/null +++ b/docs/modules/ROOT/pages/rabbit/rabbit_overview/receiving-batch.adoc @@ -0,0 +1,187 @@ +[[rabbit-receiving-batch]] += Receiving Batched Messages + +With the RabbitMQ binder, there are two types of batches handled by consumer bindings: + +[[batches-created-by-producers]] +== Batches Created by Producers + +Normally, if a producer binding has `batch-enabled=true` (see <>), or a message is created by a `BatchingRabbitTemplate`, elements of the batch are returned as individual calls to the listener method. +Starting with version 3.0, any such batch can be presented as a `List` to the listener method if `spring.cloud.stream.bindings..consumer.batch-mode` is set to `true`. + +[[consumer-side-batching]] +== Consumer-side Batching + +Starting with version 3.1, the consumer can be configured to assemble multiple inbound messages into a batch which is presented to the application as a `List` of converted payloads. +The following simple application demonstrates how to use this technique: + +==== +[source, properties] +---- +spring.cloud.stream.bindings.input-in-0.group=someGroup + +spring.cloud.stream.bindings.input-in-0.consumer.batch-mode=true + +spring.cloud.stream.rabbit.bindings.input-in-0.consumer.enable-batching=true +spring.cloud.stream.rabbit.bindings.input-in-0.consumer.batch-size=10 +spring.cloud.stream.rabbit.bindings.input-in-0.consumer.receive-timeout=200 +---- +==== + +==== +[source, java] +---- +@SpringBootApplication +public class Application { + + public static void main(String[] args) { + SpringApplication.run(Application.class, args); + } + + @Bean + Consumer> input() { + return list -> { + System.out.println("Received " + list.size()); + list.forEach(thing -> { + System.out.println(thing); + + // ... + + }); + }; + } + + @Bean + public ApplicationRunner runner(RabbitTemplate template) { + return args -> { + template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value1\"}"); + template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value2\"}"); + }; + } + + public static class Thing { + + private String field; + + public Thing() { + } + + public Thing(String field) { + this.field = field; + } + + public String getField() { + return this.field; + } + + public void setField(String field) { + this.field = field; + } + + @Override + public String toString() { + return "Thing [field=" + this.field + "]"; + } + + } + +} +---- +==== + +==== +[source] +---- +Received 2 +Thing [field=value1] +Thing [field=value2] +---- +==== + +The number of messages in a batch is specified by the `batch-size` and `receive-timeout` properties; if the `receive-timeout` elapses with no new messages, a "short" batch is delivered. + +IMPORTANT: Consumer-side batching is only supported with `container-type=simple` (the default). + +If you wish to examine headers of consumer-side batched messages, you should consume `Message>`; the headers are a `List>` in a header `AmqpInboundChannelAdapter.CONSOLIDATED_HEADERS`, with the headers for each payload element in the corresponding index. +Again, here is a simple example: + +==== +[source, java] +---- +@SpringBootApplication +public class Application { + + public static void main(String[] args) { + SpringApplication.run(Application.class, args); + } + + @Bean + Consumer>> input() { + return msg -> { + List things = msg.getPayload(); + System.out.println("Received " + things.size()); + @SuppressWarnings("unchecked") + List> headers = + (List>) msg.getHeaders().get(AmqpInboundChannelAdapter.CONSOLIDATED_HEADERS); + for (int i = 0; i < things.size(); i++) { + System.out.println(things.get(i) + " myHeader=" + headers.get(i).get("myHeader")); + + // ... + + } + }; + } + + @Bean + public ApplicationRunner runner(RabbitTemplate template) { + return args -> { + template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value1\"}", msg -> { + msg.getMessageProperties().setHeader("myHeader", "headerValue1"); + return msg; + }); + template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value2\"}", msg -> { + msg.getMessageProperties().setHeader("myHeader", "headerValue2"); + return msg; + }); + }; + } + + public static class Thing { + + private String field; + + public Thing() { + } + + public Thing(String field) { + this.field = field; + } + + public String getfield() { + return this.field; + } + + public void setfield(String field) { + this.field = field; + } + + @Override + public String toString() { + return "Thing [field=" + this.field + "]"; + } + + } + +} +---- +==== + +==== +[source] +---- +Received 2 +Thing [field=value1] myHeader=headerValue1 +Thing [field=value2] myHeader=headerValue2 +---- +==== + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream.adoc b/docs/modules/ROOT/pages/spring-cloud-stream.adoc index 83e61840c..953f9fb16 100644 --- a/docs/modules/ROOT/pages/spring-cloud-stream.adoc +++ b/docs/modules/ROOT/pages/spring-cloud-stream.adoc @@ -85,3241 +85,3 @@ This section gives an overview of the following: * <> * <> -[[spring-cloud-stream-overview-application-model]] -== Application Model - -A Spring Cloud Stream application consists of a middleware-neutral core. -The application communicates with the outside world by establishing _bindings_ between destinations -exposed by the external brokers and input/output arguments in your code. Broker specific details -necessary to establish bindings are handled by middleware-specific _Binder_ implementations. - -.Spring Cloud Stream Application -image::SCSt-with-binder.png[width=800,scaledwidth="75%",align="center"] - -[[fat-jar]] -=== Fat JAR - -Spring Cloud Stream applications can be run in stand-alone mode from your IDE for testing. -To run a Spring Cloud Stream application in production, you can create an executable (or "`fat`") JAR by using the standard Spring Boot tooling provided for Maven or Gradle. See the https://docs.spring.io/spring-boot/docs/current/reference/html/howto-build.html#howto-create-an-executable-jar-with-maven[Spring Boot Reference Guide] for more details. - -[[spring-cloud-stream-overview-binder-abstraction]] -== The Binder Abstraction - -Spring Cloud Stream provides Binder implementations for https://github.com/spring-cloud/spring-cloud-stream-binder-kafka[Kafka] and https://github.com/spring-cloud/spring-cloud-stream-binder-rabbit[Rabbit MQ]. -The framework also includes a test binder for integration testing of your applications as spring-cloud-stream application. See <> section for more details. - -Binder abstraction is also one of the extension points of the framework, which means you can implement your own binder on top of Spring Cloud Stream. -In the https://medium.com/@domenicosibilio/how-to-create-a-spring-cloud-stream-binder-from-scratch-ab8b29ee931b[How to create a Spring Cloud Stream Binder from scratch] post a community member documents -in details, with an example, a set of steps necessary to implement a custom binder. -The steps are also highlighted in the `<>` section. - -Spring Cloud Stream uses Spring Boot for configuration, and the Binder abstraction makes it possible for a Spring Cloud Stream application to be flexible in how it connects to middleware. -For example, deployers can dynamically choose, at runtime, the mapping between the external destinations (such as the Kafka topics or RabbitMQ exchanges) and inputs -and outputs of the message handler (such as input parameter of the function and its return argument). -Such configuration can be provided through external configuration properties and in any form supported by Spring Boot (including application arguments, environment variables, and `application.yml` or `application.properties` files). -In the sink example from the <> section, setting the `spring.cloud.stream.bindings.input.destination` application property to `raw-sensor-data` causes it to read from the `raw-sensor-data` Kafka topic or from a queue bound to the `raw-sensor-data` RabbitMQ exchange. - -Spring Cloud Stream automatically detects and uses a binder found on the classpath. -You can use different types of middleware with the same code. -To do so, include a different binder at build time. -For more complex use cases, you can also package multiple binders with your application and have it choose the binder( and even whether to use different binders for different bindings) at runtime. - - -[[spring-cloud-stream-overview-persistent-publish-subscribe-support]] -== Persistent Publish-Subscribe Support - -Communication between applications follows a publish-subscribe model, where data is broadcast through shared topics. -This can be seen in the following figure, which shows a typical deployment for a set of interacting Spring Cloud Stream applications. - -.Spring Cloud Stream Publish-Subscribe -image::SCSt-sensors.png[width=800,scaledwidth="75%",align="center"] - -Data reported by sensors to an HTTP endpoint is sent to a common destination named `raw-sensor-data`. -From the destination, it is independently processed by a microservice application that computes time-windowed averages and by another microservice application that ingests the raw data into HDFS (Hadoop Distributed File System). -In order to process the data, both applications declare the topic as their input at runtime. - -The publish-subscribe communication model reduces the complexity of both the producer and the consumer and lets new applications be added to the topology without disruption of the existing flow. -For example, downstream from the average-calculating application, you can add an application that calculates the highest temperature values for display and monitoring. -You can then add another application that interprets the same flow of averages for fault detection. -Doing all communication through shared topics rather than point-to-point queues reduces coupling between microservices. - -While the concept of publish-subscribe messaging is not new, Spring Cloud Stream takes the extra step of making it an opinionated choice for its application model. -By using native middleware support, Spring Cloud Stream also simplifies use of the publish-subscribe model across different platforms. - -[[consumer-groups]] -== Consumer Groups -While the publish-subscribe model makes it easy to connect applications through shared topics, the ability to scale up by creating multiple instances of a given application is equally important. -When doing so, different instances of an application are placed in a competing consumer relationship, where only one of the instances is expected to handle a given message. - -Spring Cloud Stream models this behavior through the concept of a consumer group. -(Spring Cloud Stream consumer groups are similar to and inspired by Kafka consumer groups.) -Each consumer binding can use the `spring.cloud.stream.bindings..group` property to specify a group name. -For the consumers shown in the following figure, this property would be set as `spring.cloud.stream.bindings..group=hdfsWrite` or `spring.cloud.stream.bindings..group=average`. - -.Spring Cloud Stream Consumer Groups -image::SCSt-groups.png[width=800,scaledwidth="75%",align="center"] - -All groups that subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination. -By default, when a group is not specified, Spring Cloud Stream assigns the application to an anonymous and independent single-member consumer group that is in a publish-subscribe relationship with all other consumer groups. - -[[consumer-types]] -== Consumer Types - -Two types of consumer are supported: - -* Message-driven (sometimes referred to as Asynchronous) -* Polled (sometimes referred to as Synchronous) - -Prior to version 2.0, only asynchronous consumers were supported. A message is delivered as soon as it is available and a thread is available to process it. - -When you wish to control the rate at which messages are processed, you might want to use a synchronous consumer. -// TODO This needs more description. A sentence parallel to the last sentence of the preceding paragraph would help. - -[[durability]] -=== Durability - -Consistent with the opinionated application model of Spring Cloud Stream, consumer group subscriptions are durable. -That is, a binder implementation ensures that group subscriptions are persistent and that, once at least one subscription for a group has been created, the group receives messages, even if they are sent while all applications in the group are stopped. - -[NOTE] -==== -Anonymous subscriptions are non-durable by nature. -For some binder implementations (such as RabbitMQ), it is possible to have non-durable group subscriptions. -==== - -In general, it is preferable to always specify a consumer group when binding an application to a given destination. -When scaling up a Spring Cloud Stream application, you must specify a consumer group for each of its input bindings. -Doing so prevents the application's instances from receiving duplicate messages (unless that behavior is desired, which is unusual). - -[[partitioning]] -== Partitioning Support - -Spring Cloud Stream provides support for partitioning data between multiple instances of a given application. -In a partitioned scenario, the physical communication medium (such as the broker topic) is viewed as being structured into multiple partitions. -One or more producer application instances send data to multiple consumer application instances and ensure that data identified by common characteristics are processed by the same consumer instance. - -Spring Cloud Stream provides a common abstraction for implementing partitioned processing use cases in a uniform fashion. -Partitioning can thus be used whether the broker itself is naturally partitioned (for example, Kafka) or not (for example, RabbitMQ). - -.Spring Cloud Stream Partitioning -image::SCSt-partitioning.png[width=800,scaledwidth="75%",align="center"] - -Partitioning is a critical concept in stateful processing, where it is critical (for either performance or consistency reasons) to ensure that all related data is processed together. -For example, in the time-windowed average calculation example, it is important that all measurements from any given sensor are processed by the same application instance. - -NOTE: To set up a partitioned processing scenario, you must configure both the data-producing and the data-consuming ends. - -[[programming-model]] -= Programming Model - -To understand the programming model, you should be familiar with the following core concepts: - -* *Destination Binders:* Components responsible to provide integration with the external messaging systems. -* *Bindings:* Bridge between the external messaging systems and application provided _Producers_ and _Consumers_ of messages (created by the Destination Binders). -* *Message:* The canonical data structure used by producers and consumers to communicate with Destination Binders (and thus other applications via external messaging systems). - -image::SCSt-overview.png[width=800,scaledwidth="75%",align="center"] - -[[destination-binders]] -== Destination Binders - -Destination Binders are extension components of Spring Cloud Stream responsible for providing the necessary configuration and implementation to facilitate -integration with external messaging systems. -This integration is responsible for connectivity, delegation, and routing of messages to and from producers and consumers, data type conversion, -invocation of the user code, and more. - -Binders handle a lot of the boiler plate responsibilities that would otherwise fall on your shoulders. However, to accomplish that, the binder still needs -some help in the form of minimalistic yet required set of instructions from the user, which typically come in the form of some type of _binding_ configuration. - -While it is out of scope of this section to discuss all of the available binder and binding configuration options (the rest of the manual covers them extensively), -_Binding_ as a concept, does require special attention. The next section discusses it in detail. - -[[bindings]] -== Bindings - -As stated earlier, _Bindings_ provide a bridge between the external messaging system (e.g., queue, topic etc.) and application-provided _Producers_ and _Consumers_. - -The following example shows a fully configured and functioning Spring Cloud Stream application that receives the payload of the message -as a `String` type (see <> section), logs it to the console and sends it down stream after converting it to upper case. - -[source, java] ----- -@SpringBootApplication -public class SampleApplication { - - public static void main(String[] args) { - SpringApplication.run(SampleApplication.class, args); - } - - @Bean - public Function uppercase() { - return value -> { - System.out.println("Received: " + value); - return value.toUpperCase(); - }; - } -} ----- -The above example looks no different then any vanilla spring-boot application. It defines a single bean of type `Function` -and that is it. So, how does it become a spring-cloud-stream application? -It becomes a spring-cloud-stream application simply because of the presence of spring-cloud-stream and binder dependencies -and auto-configuration classes on the classpath, effectively setting the context for your boot application as a spring-cloud-stream application. -And in this context beans of type `Supplier`, `Function` or `Consumer` are treated as defacto message handlers triggering -binding of to destinations exposed by the provided binder following certain naming conventions and -rules to avoid extra configuration. - -[[binding-and-binding-names]] -=== Binding and Binding names - -Binding is an abstraction that represents a bridge between sources and targets exposed by the binder and user code, -This abstraction has a name and while we try to do our best to limit configuration required to run spring-cloud-stream applications, -being aware of such name(s) is necessary for cases where additional per-binding configuration is required. - -Throughout this manual you will see examples of configuration properties such as `spring.cloud.stream.bindings.input.destination=myQueue`. -The `input` segment in this property name is what we refer to as _binding name_ and it could derive via several mechanisms. -The following sub-sections will describe the naming conventions and configuration elements used by spring-cloud-stream to control binding names. - -NOTE: If your binding name has special characters, such as the `.` character, you need to surround the binding key with brackets (`[]`) and then wrap it in qoutes. -For example `spring.cloud.stream.bindings."[my.output.binding.key]".destination`. - -[[functional-binding-names]] -==== Functional binding names - -Unlike the explicit naming required by annotation-based support (legacy) used in the previous versions of spring-cloud-stream, the functional -programming model defaults to a simple convention when it comes to binding names, thus greatly simplifying application configuration. -Let's look at the first example: - -[source, java] ----- -@SpringBootApplication -public class SampleApplication { - - @Bean - public Function uppercase() { - return value -> value.toUpperCase(); - } -} ----- - -In the preceding example we have an application with a single function which acts as message handler. As a `Function` it has an -input and output. -The naming convention used to name input and output bindings is as follows: - -* input - ` + -in- + ` -* output - ` + -out- + ` - -The `in` and `out` corresponds to the type of binding (such as _input_ or _output_). -The `index` is the index of the input or output binding. It is always 0 for typical single input/output function, -so it's only relevant for <>. - -So if for example you would want to map the input of this function to a remote -destination (e.g., topic, queue etc) called "my-topic" you would do so with the following property: ----- ---spring.cloud.stream.bindings.uppercase-in-0.destination=my-topic ----- -Note how `uppercase-in-0` is used as a segment in property name. The same goes for `uppercase-out-0`. - -***Descriptive Binding Names*** - -Some times to improve readability you may want to give your binding a more descriptive name (such as 'account', 'orders' etc). -Another way of looking at it is you can map an _implicit binding name_ to an _explicit binding name_. And you can do it with -`spring.cloud.stream.function.bindings.` property. -This property also provides a migration path for existing applications that rely on custom interface-based -bindings that require explicit names. - -For example, ----- ---spring.cloud.stream.function.bindings.uppercase-in-0=input ----- - -In the preceding example you mapped and effectively renamed `uppercase-in-0` binding name to `input`. Now all configuration -properties can refer to `input` binding name instead (e.g., `--spring.cloud.stream.bindings.input.destination=my-topic`). - -NOTE: While descriptive binding names may enhance the readability aspect of the configuration, they also create -another level of misdirection by mapping an implicit binding name to an explicit binding name. And since all subsequent -configuration properties will use the explicit binding name you must always refer to this 'bindings' property to -correlate which function it actually corresponds to. We believe that for most cases (with the exception of <>) -it may be an overkill, so, it is our recommendation to avoid using it altogether, especially -since not using it provides a clear path between binder destination and binding name, such as `spring.cloud.stream.bindings.uppercase-in-0.destination=sample-topic`, -where you are clearly correlating the input of `uppercase` function to `sample-topic` destination. - -For more on properties and other configuration options please see <> section. - -[[explicit-binding-creation]] -==== Explicit binding creation - -In the previous section we explained how bindings are created implicitly driven by the names of `Function`, `Supplier` or `Consumer` beans provided by your application. -However, there are times when you may need to create binding explicitly where bindings are not tied to any function. This is typically done to -support integrations with other frameworks via `StreamBridge`. - -Spring Cloud Stream allows you to define input and output bindings explicitly via `spring.cloud.stream.input-bindings` and `spring.cloud.stream.output-bindings` -properties. Noticed the plural in the property names allowing you to define multiple bindings by simply using `;` as a delimiter. -Just look at the following test case as an example: - ----- -@Test -public void testExplicitBindings() { - try (ConfigurableApplicationContext context = new SpringApplicationBuilder( - TestChannelBinderConfiguration.getCompleteConfiguration(EmptyConfiguration.class)) - .web(WebApplicationType.NONE) - .run("--spring.jmx.enabled=false", - "--spring.cloud.stream.input-bindings=fooin;barin", - "--spring.cloud.stream.output-bindings=fooout;barout")) { - - - . . . - } -} - -@EnableAutoConfiguration -@Configuration -public static class EmptyConfiguration { -} ----- - -As you can see we have declared two input bindings and two output bindings while our configuration had no functions defined, yet we were able to successfully create these bindings and access their corresponding channels. - - -[[producing-and-consuming-messages]] -== Producing and Consuming Messages - -You can write a Spring Cloud Stream application by simply writing functions and exposing them as `@Bean` s. -You can also use Spring Integration annotations based configuration or -Spring Cloud Stream annotation based configuration, although starting with spring-cloud-stream 3.x -we recommend using functional implementations. - -[[spring_cloud_function]] -=== Spring Cloud Function support - -[[overview]] -==== Overview - -Since Spring Cloud Stream v2.1, another alternative for defining _stream handlers_ and _sources_ is to use build-in -support for https://cloud.spring.io/spring-cloud-function/[Spring Cloud Function] where they can be expressed as beans of - type `java.util.function.[Supplier/Function/Consumer]`. - -To specify which functional bean to bind to the external destination(s) exposed by the bindings, -you must provide `spring.cloud.function.definition` property. - -NOTE: In the event you only have single bean of type `java.util.function.[Supplier/Function/Consumer]`, you can -skip the `spring.cloud.function.definition` property, since such functional bean will be auto-discovered. However, -it is considered best practice to use such property to avoid any confusion. -Some time this auto-discovery can get in the way, since single bean of type `java.util.function.[Supplier/Function/Consumer]` -could be there for purposes other then handling messages, yet being single it is auto-discovered and auto-bound. -For these rare scenarios you can disable auto-discovery by providing `spring.cloud.stream.function.autodetect` property with value set to `false`. - -Here is the example of the application exposing message handler as `java.util.function.Function` effectively supporting -_pass-thru_ semantics by acting as consumer and producer of data. -[source,java] ----- -@SpringBootApplication -public class MyFunctionBootApp { - - public static void main(String[] args) { - SpringApplication.run(MyFunctionBootApp.class); - } - - @Bean - public Function toUpperCase() { - return s -> s.toUpperCase(); - } -} ----- -In the preceding example, we define a bean of type `java.util.function.Function` called _toUpperCase_ to be acting as message handler -whose 'input' and 'output' must be bound to the external destinations exposed by the provided destination binder. -By default the 'input' and 'output' binding names will be `toUpperCase-in-0` and `toUpperCase-out-0`. -Please see <> section for details on naming convention used to establish binding names. - -Below are the examples of simple functional applications to support other semantics: - -Here is the example of a _source_ semantics exposed as `java.util.function.Supplier` -[source,java] ----- -@SpringBootApplication -public static class SourceFromSupplier { - - @Bean - public Supplier date() { - return () -> new Date(12345L); - } -} ----- - -Here is the example of a _sink semantics_ exposed as `java.util.function.Consumer` -[source,java] ----- -@SpringBootApplication -public static class SinkFromConsumer { - - @Bean - public Consumer sink() { - return System.out::println; - } -} ----- - -[[suppliers-sources]] -==== Suppliers (Sources) - -`Function` and `Consumer` are pretty straightforward when it comes to how their invocation is triggered. They are triggered based -on data (events) sent to the destination they are bound to. In other words, they are classic event-driven components. - -However, `Supplier` is in its own category when it comes to triggering. Since it is, by definition, the source (the origin) of the data, it does not -subscribe to any in-bound destination and, therefore, has to be triggered by some other mechanism(s). -There is also a question of `Supplier` implementation, which could be _imperative_ or _reactive_ and which directly relates to the triggering of such suppliers. - -Consider the following sample: - -[source,java] ----- -@SpringBootApplication -public static class SupplierConfiguration { - - @Bean - public Supplier stringSupplier() { - return () -> "Hello from Supplier"; - } -} ----- - -The preceding `Supplier` bean produces a string whenever its `get()` method is invoked. However, who invokes this method and how often? -The framework provides a default polling mechanism (answering the question of "Who?") that will trigger the invocation of the supplier and by default it will do so -every second (answering the question of "How often?"). -In other words, the above configuration produces a single message every second and each message is sent to an `output` destination that is exposed by the binder. -To learn how to customize the polling mechanism, see <> section. - -Consider a different example: - -[source,java] ----- -@SpringBootApplication -public static class SupplierConfiguration { - - @Bean - public Supplier> stringSupplier() { - return () -> Flux.fromStream(Stream.generate(new Supplier() { - @Override - public String get() { - try { - Thread.sleep(1000); - return "Hello from Supplier"; - } catch (Exception e) { - // ignore - } - } - })).subscribeOn(Schedulers.elastic()).share(); - } -} ----- - -The preceding `Supplier` bean adopts the reactive programming style. Typically, and unlike the imperative supplier, -it should be triggered only once, given that the invocation of its `get()` method produces (supplies) the continuous stream of messages and not an -individual message. - -The framework recognizes the difference in the programming style and guarantees that such a supplier is triggered only once. - -However, imagine the use case where you want to poll some data source and return a finite stream of data representing the result set. -The reactive programming style is a perfect mechanism for such a Supplier. However, given the finite nature of the produced stream, -such Supplier still needs to be invoked periodically. - -Consider the following sample, which emulates such use case by producing a finite stream of data: - -[source,java] ----- -@SpringBootApplication -public static class SupplierConfiguration { - - @PollableBean - public Supplier> stringSupplier() { - return () -> Flux.just("hello", "bye"); - } -} ----- - -The bean itself is annotated with `PollableBean` annotation (sub-set of `@Bean`), thus signaling to the framework that although the implementation -of such a supplier is reactive, it still needs to be polled. - -NOTE: There is a `splittable` attribute defined in `PollableBean` which signals to the post processors of this annotation -that the result produced by the annotated component has to be split and is set to `true` by default. It means that -the framework will split the returning sending out each item as an individual message. If this is not -he desired behavior you can set it to `false` at which point such supplier will simply return -the produced Flux without splitting it. - -[[supplier-threading]] -===== Supplier & threading - -NOTE: As you have learned by now, unlike `Function` and `Consumer`, which are triggered by an event (they have input data), `Supplier` does not have -any input and thus triggered by a different mechanism - _poller_, which may have an unpredictable threading mechanism. And while the details of the -threading mechanism most of the time are not relevant to the downstream execution of the function it may present an issue in certain cases -especially with integrated frameworks that may have certain expectations to thread affinity. For example, https://spring.io/projects/spring-cloud-sleuth[Spring Cloud Sleuth] which relies -on tracing data stored in thread local. -For those cases we have another mechanism via `StreamBridge`, where user has more control over threading mechanism. You can get more details -in <> section. - -[[consumer-reactive]] -==== Consumer (Reactive) - -Reactive `Consumer` is a little bit special because it has a void return type, leaving framework with no reference to subscribe to. -Most likely you will not need to write `Consumer>`, and instead write it as a `Function, Mono>` invoking `then` -operator as the last operator on your stream. - -For example: - -[source,java] ----- -public Function, Mono> consumer() { - return flux -> flux.map(..).filter(..).then(); -} ----- - -But if you do need to write an explicit `Consumer>`, remember to subscribe to the incoming Flux. - - -Also, keep in mind that the same rule applies for function composition when mixing reactive and imperative functions. -Spring Cloud Function indeed supports composing reactive functions with imperative, however you must be aware of certain limitations. -For example, assume you have composed reactive function with imperative consumer. -The result of such composition is a reactive `Consumer`. However, there is no way to subscribe to such consumer as discussed earlier in this section, -so this limitation can only be addressed by either making your consumer reactive and subscribing manually (as discussed earlier), or changing your function to be imperative. - - -[[polling-configuration-properties]] -===== Polling Configuration Properties - -The following properties are exposed by Spring Cloud Stream and are prefixed with the `spring.integration.poller.`: - - -fixedDelay:: -Fixed delay for default poller in milliseconds. -+ -Default: 1000L. - -maxMessagesPerPoll:: -Maximum messages for each polling event of the default poller. -+ -Default: 1L. - -cron:: -Cron expression value for the Cron Trigger. -+ -Default: none. - -initialDelay:: -Initial delay for periodic triggers. -+ -Default: 0. - -timeUnit:: -The TimeUnit to apply to delay values. -+ -Default: MILLISECONDS. - -For example `--spring.integration.poller.fixed-delay=2000` sets the poller interval to poll every two seconds. - -[[per-binding-polling-configuration]] -===== Per-binding polling configuration - -The previous section shows how to configure a single default poller that will be applied to all bindings. While it fits well with the model of microservices spring-cloud-stream designed for where each microservice represents a single component (e.g., Supplier) and thus default poller configuration is enough, there are edge cases where -you may have several components that require different polling configurations - -For such cases please use per-binding way of configuring poller. For example, assume you have an output binding `supply-out-0`. In this case you can configure poller for such -binding using `spring.cloud.stream.bindings.supply-out-0.producer.poller..` prefix (e.g., `spring.cloud.stream.bindings.supply-out-0.producer.poller.fixed-delay=2000`). - -[[sending-arbitrary-data-to-an-output-e-g-foreign-event-driven-sources]] -==== Sending arbitrary data to an output (e.g. Foreign event-driven sources) - -There are cases where the actual source of data may be coming from the external (foreign) system that is not a binder. For example, the -source of the data may be a classic REST endpoint. How do we bridge such source with the functional mechanism used by spring-cloud-stream? - -Spring Cloud Stream provides two mechanisms, so let's look at them in more details - -Here, for both samples we'll use a standard MVC endpoint method called `delegateToSupplier` bound to the root web context, -delegating incoming requests to stream via StreamBridge mechanism. - - -[source, java] ----- -@SpringBootApplication -@Controller -public class WebSourceApplication { - - public static void main(String[] args) { - SpringApplication.run(WebSourceApplication.class, "--spring.cloud.stream.output-bindings=toStream"); - } - - @Autowired - private StreamBridge streamBridge; - - @RequestMapping - @ResponseStatus(HttpStatus.ACCEPTED) - public void delegateToSupplier(@RequestBody String body) { - System.out.println("Sending " + body); - streamBridge.send("toStream", body); - } -} ----- - -Here we autowire a `StreamBridge` bean which allows us to send data to an output binding effectively -bridging non-stream application with spring-cloud-stream. Note that preceding example does not have any -source functions defined (e.g., Supplier bean) leaving the framework with no trigger to create source bindings in advance, which would be typical for cases where configuration contains function beans. And that is fine, since `StreamBridge` will initiate creation of output bindings (as well as -destination auto-provisioning if necessary) for non existing bindings on the first call to its `send(..)` operation caching it for -subsequent reuse (see <> for more details). - -However, if you want to pre-create an output binding at the initialization (startup) time you can benefit from `spring.cloud.stream.output-bindings` property where you can declare the name of your sources. -The provided name will be used as a trigger to create a source binding. -You can use `;` to signify multiple sources (multiple output bindings) -(e.g., `--spring.cloud.stream.output-bindings=foo;bar`) - -Also, note that `streamBridge.send(..)` method takes an `Object` for data. This means you can send POJO or `Message` to it and it -will go through the same routine when sending output as if it was from any Function or Supplier providing the same level -of consistency as with functions. This means the output type conversion, partitioning etc are honored as if it was from the output produced by functions. - -[[streambridge-and-dynamic-destinations]] -===== StreamBridge and Dynamic Destinations - -`StreamBridge` can also be used for cases when output destination(s) are not known ahead of time similar to the use cases -described in <> section. - -Let's look at the example - -[source, java] ----- -@SpringBootApplication -@Controller -public class WebSourceApplication { - - public static void main(String[] args) { - SpringApplication.run(WebSourceApplication.class, args); - } - - @Autowired - private StreamBridge streamBridge; - - @RequestMapping - @ResponseStatus(HttpStatus.ACCEPTED) - public void delegateToSupplier(@RequestBody String body) { - System.out.println("Sending " + body); - streamBridge.send("myDestination", body); - } -} ----- - -As you can see the preceding example is very similar to the previous one with the exception of explicit binding instruction provided via -`spring.cloud.stream.output-bindings` property (which is not provided). -Here we're sending data to `myDestination` name which does not exist as a binding. Therefore such name will be treated as dynamic destination -as described in <> section. - - -In the preceding example, we are using `ApplicationRunner` as a _foreign source_ to feed the stream. - - -A more practical example, where the foreign source is REST endpoint. - -[source, java] ----- -@SpringBootApplication -@Controller -public class WebSourceApplication { - - public static void main(String[] args) { - SpringApplication.run(WebSourceApplication.class); - } - - @Autowired - private StreamBridge streamBridge; - - @RequestMapping - @ResponseStatus(HttpStatus.ACCEPTED) - public void delegateToSupplier(@RequestBody String body) { - streamBridge.send("myBinding", body); - } -} ----- - -As you can see inside of `delegateToSupplier` method we're using StreamBridge to send data to `myBinding` binding. And here you're also benefiting from -the dynamic features of `StreamBridge` where if `myBinding` doesn't exist it will be created automatically and cached, otherwise existing binding will be used. - -NOTE: Caching dynamic destinations (bindings) could result in memory leaks in the event there are many dynamic destinations. To have some level of control -we provide a self-evicting caching mechanism for output bindings with default cache size of 10. This means that if your dynamic destination size goes above that number, there is a possibility that an existing binding will be evicted and thus would need to be recreated which could cause minor performance degradation. You can increase the cache size via `spring.cloud.stream.dynamic-destination-cache-size` property setting it to the desired value. - - ----- -curl -H "Content-Type: text/plain" -X POST -d "hello from the other side" http://localhost:8080/ ----- - -By showing two examples we want to emphasize the approach will work with any type of foreign sources. - -NOTE: If you are using the Solace PubSub+ binder, Spring Cloud Stream has reserved the `scst_targetDestination` header (retrievable via BinderHeaders.TARGET_DESTINATION), which allows for messages to be redirected from their bindings' configured destination to the target destination specified by this header. This allows for the binder to manage the resources necessary to publish to dynamic destinations, relieving the framework from having to do so, and avoids the caching issues mentioned in the previous Note. More info https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter#dynamic-producer-destinations[here]. - -[[output-content-type-with-streambridge]] -===== Output Content Type with StreamBridge - -You can also provide specific content type if necessary with the following method signature `public boolean send(String bindingName, Object data, MimeType outputContentType)`. -Or if you send data as a `Message`, its content type will be honored. - -[[using-specific-binder-type-with-streambridge]] -===== Using specific binder type with StreamBridge - -Spring Cloud Stream supports multiple binder scenarios. For example you may be receiving data from Kafka and sending it to RabbitMQ. - -For more information on multiple binders scenarios, please see <> section and specifically <> - -In the event you are planning to use StreamBridge and have more then one binder configured in your application you must also tell StreamBridge -which binder to use. And for that there are two more variations of `send` method: - -[source, java] ----- -public boolean send(String bindingName, @Nullable String binderType, Object data) - -public boolean send(String bindingName, @Nullable String binderType, Object data, MimeType outputContentType) ----- - -As you can see there is one additional argument that you can provide - `binderType`, telling BindingService which binder to use when creating dynamic binding. - -NOTE: For cases where `spring.cloud.stream.output-bindings` property is used or the binding was already created under different binder, the `binderType` -argument will have no effect. - -[[using-channel-interceptors-with-streambridge]] -===== Using channel interceptors with StreamBridge - -Since `StreamBridge` uses a `MessageChannel` to establish the output binding, you can activate channel interceptors when sending data through `StreamBridge`. -It is up to the application to decide which channel interceptors to apply on `StreamBridge`. -Spring Cloud Stream does not inject all the channel interceptors detected into `StreamBridge` unless they are annoatated with `@GlobalChannelInterceptor(patterns = "*")`. - -Let us assume that you have the following two different `StreamBridge` bindings in the application. - -`streamBridge.send("foo-out-0", message);` - -and - -`streamBridge.send("bar-out-0", message);` - -Now, if you want a channel interceptor applied on both the `StreamBridge` bindings, then you can declare the following `GlobalChannelInterceptor` bean. - -``` -@Bean -@GlobalChannelInterceptor(patterns = "*") -public ChannelInterceptor customInterceptor() { - return new ChannelInterceptor() { - @Override - public Message preSend(Message message, MessageChannel channel) { - ... - } - }; -} -``` - -However, if you don't like the global approach above and want to have a dedicated interceptor for each binding, then you can do the following. - -``` -@Bean -@GlobalChannelInterceptor(patterns = "foo-*") -public ChannelInterceptor fooInterceptor() { - return new ChannelInterceptor() { - @Override - public Message preSend(Message message, MessageChannel channel) { - ... - } - }; -} -``` - -and - -``` -@Bean -@GlobalChannelInterceptor(patterns = "bar-*") -public ChannelInterceptor barInterceptor() { - return new ChannelInterceptor() { - @Override - public Message preSend(Message message, MessageChannel channel) { - ... - } - }; -} -``` - -You have the flexibility to make the patterns more strict or customized to your business needs. - -With this approach, the application gets the ability to decide which interceptors to inject in `StreamBridge` rather than applying all the available interceptors. - -NOTE: `StreamBridge` provides a contract through the `StreamOperations` interface that contains all the `send` methods of `StreamBridge`. Therefore, applications may choose to autowire using `StreamOperations`. This is handy when it comes to unit testing code that uses `StreamBridge` by providing a mock or similar mechanisms for the `StreamOperations` interface. - -[[reactive-functions-support]] -==== Reactive Functions support - - -Since _Spring Cloud Function_ is build on top of https://projectreactor.io/[Project Reactor] there isn't much you need to do -to benefit from reactive programming model while implementing `Supplier`, `Function` or `Consumer`. - -For example: - -[source,java] ----- -@SpringBootApplication -public static class SinkFromConsumer { - - @Bean - public Function, Flux> reactiveUpperCase() { - return flux -> flux.map(val -> val.toUpperCase()); - } -} ----- - -[IMPORTANT] -==== -Few important things must be understood when choosing reactive or imperative programming model. - -_Fully reactive or just API?_ - -Using reactive API does not necessarily imply that you can benefit from all of the reactive features of such API. In other words things like back-pressure and other advanced features will only work when they are working with compatible system - such as Reactive Kafka binder. In the event you are using regular Kafka or Rabbit or any other non-reactive binder, you can only benefit from the conveniences of the reactive API itself and not its advanced features, since the actual sources or targets of the stream are not reactive. - - -_Error handling and retries_ - -Throughout this manual you will see several reference on the framework-based error handling, retries and other features as well as configuration properties associated with them. It is important to understand that they only effect the imperative functions and you should NOT have the same expectations when it comes to reactive functions. And here is why. . . -There is a fundamental difference between reactive and imperative functions. -Imperative function is a _message handler_ that is invoked by the framework on each message it receives. So for N messages there will be N invocations of such function and because of that we can wrap such function and add additional functionality such as error handling, retries etc. -Reactive function is _initialization function_. It is invoked only once to get a reference to a Flux/Mono provided by the user to be connected with the one provided by the framework. After that we (the framework) have absolutely no visibility nor control of the stream. -Therefore, with reactive functions you must rely on the richness of the reactive API when it comes to error handling and retries (i.e., `doOnError()`, `.onError*()` etc). -==== - -[[functional-composition]] -==== Functional Composition - -Using functional programming model you can also benefit from functional composition where you can dynamically compose complex handlers from a set of simple functions. -As an example let's add the following function bean to the application defined above -[source,java] ----- -@Bean -public Function wrapInQuotes() { - return s -> "\"" + s + "\""; -} ----- -and modify the `spring.cloud.function.definition` property to reflect your intention to compose a new function from both ‘toUpperCase’ and ‘wrapInQuotes’. -To do so Spring Cloud Function relies on `|` (pipe) symbol. So, to finish our example our property will now look like this: - -[source,java] ----- ---spring.cloud.function.definition=toUpperCase|wrapInQuotes ----- - -NOTE: One of the great benefits of functional composition support provided by _Spring Cloud Function_ is -the fact that you can compose _reactive_ and _imperative_ functions. - -The result of a composition is a single function which, as you may guess, could have a very long and rather cryptic name (e.g., `foo|bar|baz|xyz. . .`) -presenting a great deal of inconvenience when it comes to other configuration properties. This is where _descriptive binding names_ -feature described in <> section can help. - -For example, if we want to give our `toUpperCase|wrapInQuotes` a more descriptive name we can do so -with the following property `spring.cloud.stream.function.bindings.toUpperCase|wrapInQuotes-in-0=quotedUpperCaseInput` allowing -other configuration properties to refer to that binding name (e.g., `spring.cloud.stream.bindings.quotedUpperCaseInput.destination=myDestination`). - -[[functional-composition-and-cross-cutting-concerns]] -===== Functional Composition and Cross-cutting Concerns - -Function composition effectively allows you to address complexity by breaking it down -to a set of simple and individually manageable/testable components that could still be -represented as one at runtime. But that is not the only benefit. - -You can also use composition to address certain cross-cutting non-functional concerns, -such as content enrichment. For example, assume you have an incoming message that may -be lacking certain headers, or some headers are not in the exact state your business -function would expect. You can now implement a separate function that addresses those -concerns and then compose it with the main business function. - -Let’s look at the example - -[source,java] ----- -@SpringBootApplication -public class DemoStreamApplication { - - public static void main(String[] args) { - SpringApplication.run(DemoStreamApplication.class, - "--spring.cloud.function.definition=enrich|echo", - "--spring.cloud.stream.function.bindings.enrich|echo-in-0=input", - "--spring.cloud.stream.bindings.input.destination=myDestination", - "--spring.cloud.stream.bindings.input.group=myGroup"); - - } - - @Bean - public Function, Message> enrich() { - return message -> { - Assert.isTrue(!message.getHeaders().containsKey("foo"), "Should NOT contain 'foo' header"); - return MessageBuilder.fromMessage(message).setHeader("foo", "bar").build(); - }; - } - - @Bean - public Function, Message> echo() { - return message -> { - Assert.isTrue(message.getHeaders().containsKey("foo"), "Should contain 'foo' header"); - System.out.println("Incoming message " + message); - return message; - }; - } -} ----- - -While trivial, this example demonstrates how one function enriches the incoming Message with the additional header(s) (non-functional concern), -so the other function - `echo` - can benefit form it. The `echo` function stays clean and focused on business logic only. -You can also see the usage of `spring.cloud.stream.function.bindings` property to simplify composed binding name. - -[[functions-with-multiple-input-and-output-arguments]] -==== Functions with multiple input and output arguments - -Starting with version 3.0 spring-cloud-stream provides support for functions that -have multiple inputs and/or multiple outputs (return values). What does this actually mean and -what type of use cases it is targeting? - -* _Big Data: Imagine the source of data you're dealing with is highly un-organized and contains various types of data elements -(e.g., orders, transactions etc) and you effectively need to sort it out._ -* _Data aggregation: Another use case may require you to merge data elements from 2+ incoming _streams_. - -The above describes just a few use cases where you may need to use a single function to accept and/or produce -multiple _streams_ of data. And that is the type of use cases we are targeting here. - -Also, note a slightly different emphasis on the concept of _streams_ here. The assumption is that such functions are only valuable -if they are given access to the actual streams of data (not the individual elements). So for that we are relying on -abstractions provided by https://projectreactor.io/[Project Reactor] (i.e., `Flux` and `Mono`) which is already available on the -classpath as part of the dependencies brought in by spring-cloud-functions. - -Another important aspect is representation of multiple input and outputs. While java provides -variety of different abstractions to represent _multiple of something_ those abstractions -are _a) unbounded_, _b) lack arity_ and _c) lack type information_ which are all important in this context. -As an example, let's look at `Collection` or an array which only allows us to -describe _multiple_ of a single type or up-cast everything to an `Object`, affecting the transparent type conversion feature of -spring-cloud-stream and so on. - -So to accommodate all these requirements the initial support is relying on the signature which utilizes another abstraction -provided by _Project Reactor_ - Tuples. However, we are working on allowing a more flexible signatures. - -IMPORTANT: Please refer to <> section to understand the naming convention used to establish _binding names_ -used by such application. - -Let's look at the few samples: - -[source,java] ----- -@SpringBootApplication -public class SampleApplication { - - @Bean - public Function, Flux>, Flux> gather() { - return tuple -> { - Flux stringStream = tuple.getT1(); - Flux intStream = tuple.getT2().map(i -> String.valueOf(i)); - return Flux.merge(stringStream, intStream); - }; - } -} ----- - -The above example demonstrates function which takes two inputs (first of type `String` and second of type `Integer`) -and produces a single output of type `String`. - -So, for the above example the two input bindings will be `gather-in-0` and `gather-in-1` and for consistency the -output binding also follows the same convention and is named `gather-out-0`. - - -Knowing that will allow you to set binding specific properties. -For example, the following will override content-type for `gather-in-0` binding: - ----- ---spring.cloud.stream.bindings.gather-in-0.content-type=text/plain ----- - - -[source,java] ----- -@SpringBootApplication -public class SampleApplication { - - @Bean - public static Function, Tuple2, Flux>> scatter() { - return flux -> { - Flux connectedFlux = flux.publish().autoConnect(2); - UnicastProcessor even = UnicastProcessor.create(); - UnicastProcessor odd = UnicastProcessor.create(); - Flux evenFlux = connectedFlux.filter(number -> number % 2 == 0).doOnNext(number -> even.onNext("EVEN: " + number)); - Flux oddFlux = connectedFlux.filter(number -> number % 2 != 0).doOnNext(number -> odd.onNext("ODD: " + number)); - - return Tuples.of(Flux.from(even).doOnSubscribe(x -> evenFlux.subscribe()), Flux.from(odd).doOnSubscribe(x -> oddFlux.subscribe())); - }; - } -} ----- - -The above example is somewhat of a the opposite from the previous sample and demonstrates function which -takes single input of type `Integer` and produces two outputs (both of type `String`). - -So, for the above example the input binding is `scatter-in-0` and the -output bindings are `scatter-out-0` and `scatter-out-1`. - -And you test it with the following code: -[source,java] ----- -@Test -public void testSingleInputMultiOutput() { - try (ConfigurableApplicationContext context = new SpringApplicationBuilder( - TestChannelBinderConfiguration.getCompleteConfiguration( - SampleApplication.class)) - .run("--spring.cloud.function.definition=scatter")) { - - InputDestination inputDestination = context.getBean(InputDestination.class); - OutputDestination outputDestination = context.getBean(OutputDestination.class); - - for (int i = 0; i < 10; i++) { - inputDestination.send(MessageBuilder.withPayload(String.valueOf(i).getBytes()).build()); - } - - int counter = 0; - for (int i = 0; i < 5; i++) { - Message even = outputDestination.receive(0, 0); - assertThat(even.getPayload()).isEqualTo(("EVEN: " + String.valueOf(counter++)).getBytes()); - Message odd = outputDestination.receive(0, 1); - assertThat(odd.getPayload()).isEqualTo(("ODD: " + String.valueOf(counter++)).getBytes()); - } - } -} ----- - -[[multiple-functions-in-a-single-application]] -==== Multiple functions in a single application - -There may also be a need for grouping several message handlers in a single application. You would do so by -defining several functions. - -[source,java] ----- -@SpringBootApplication -public class SampleApplication { - - @Bean - public Function uppercase() { - return value -> value.toUpperCase(); - } - - @Bean - public Function reverse() { - return value -> new StringBuilder(value).reverse().toString(); - } -} ----- - -In the above example we have configuration which defines two functions `uppercase` and `reverse`. -So first, as mentioned before, we need to notice that there is a a conflict (more then one function) and therefore -we need to resolve it by providing `spring.cloud.function.definition` property pointing to the actual function -we want to bind. Except here we will use `;` delimiter to point to both functions (see test case below). - - -IMPORTANT: As with functions with multiple inputs/outputs, please refer to <> section to understand the naming -convention used to establish _binding names_ used by such application. - -And you test it with the following code: -[source,java] ----- -@Test -public void testMultipleFunctions() { - try (ConfigurableApplicationContext context = new SpringApplicationBuilder( - TestChannelBinderConfiguration.getCompleteConfiguration( - ReactiveFunctionConfiguration.class)) - .run("--spring.cloud.function.definition=uppercase;reverse")) { - - InputDestination inputDestination = context.getBean(InputDestination.class); - OutputDestination outputDestination = context.getBean(OutputDestination.class); - - Message inputMessage = MessageBuilder.withPayload("Hello".getBytes()).build(); - inputDestination.send(inputMessage, "uppercase-in-0"); - inputDestination.send(inputMessage, "reverse-in-0"); - - Message outputMessage = outputDestination.receive(0, "uppercase-out-0"); - assertThat(outputMessage.getPayload()).isEqualTo("HELLO".getBytes()); - - outputMessage = outputDestination.receive(0, "reverse-out-1"); - assertThat(outputMessage.getPayload()).isEqualTo("olleH".getBytes()); - } -} ----- - -[[batch-consumers]] -==== Batch Consumers - -When using a `MessageChannelBinder` that supports batch listeners, and the feature is enabled for the consumer binding, you can set `spring.cloud.stream.bindings..consumer.batch-mode` to `true` to enable the -entire batch of messages to be passed to the function in a `List`. - -[source, java] ----- -@Bean -public Function, Person> findFirstPerson() { - return persons -> persons.get(0); -} ----- - -[[batch-producers]] -==== Batch Producers - -You can also use the concept of batching on the producer side by returning a collection of Messages which effectively provides an -inverse effect where each message in the collection will be sent individually by the binder. - -Consider the following function: - -[source, java] ----- -@Bean -public Function>> batch() { - return p -> { - List> list = new ArrayList<>(); - list.add(MessageBuilder.withPayload(p + ":1").build()); - list.add(MessageBuilder.withPayload(p + ":2").build()); - list.add(MessageBuilder.withPayload(p + ":3").build()); - list.add(MessageBuilder.withPayload(p + ":4").build()); - return list; - }; -} ----- - -Each message in the returned list will be sent individually resulting in four messages sent to output destination. - -[[spring-integration-flow-as-functions]] -==== Spring Integration flow as functions -When you implement a function, you may have complex requirements that fit the category -of https://www.enterpriseintegrationpatterns.com[Enterprise Integration Patterns] (EIP). These are best handled by using a -framework such as https://spring.io/projects/spring-integration[Spring Integration] (SI), which is a reference implementation of EIP. - -Thankfully SI already provides support for exposing integration flows as functions via -https://docs.spring.io/spring-integration/docs/current/reference/html/#java-dsl-gateway[Integration flow as gateway] -Consider the following sample: - -[source, java] ----- -@SpringBootApplication -public class FunctionSampleSpringIntegrationApplication { - - public static void main(String[] args) { - SpringApplication.run(FunctionSampleSpringIntegrationApplication.class, args); - } - - @Bean - public IntegrationFlow uppercaseFlow() { - return IntegrationFlows.from(MessageFunction.class, "uppercase") - .transform(String::toUpperCase) - .logAndReply(LoggingHandler.Level.WARN); - } - - public interface MessageFunction extends Function, Message> { - - } -} ----- - -For those who are familiar with SI you can see we define a bean of type `IntegrationFlow` where we -declare an integration flow that we want to expose as a `Function` (using SI DSL) called `uppercase`. -The `MessageFunction` interface lets us explicitly declare the type of the inputs and outputs for proper type conversion. -See <> section for more on type conversion. - -To receive raw input you can use `from(Function.class, ...)`. - -The resulting function is bound to the input and output destinations exposed by the target binder. - -IMPORTANT: Please refer to <> section to understand the naming -convention used to establish _binding names_ used by such application. - -For more details on interoperability of Spring Integration and Spring Cloud Stream specifically around functional programming model -you may find https://spring.io/blog/2019/10/25/spring-cloud-stream-and-spring-integration[this post] very interesting, as it dives a bit deeper -into various patterns you can apply by merging the best of Spring Integration and Spring Cloud Stream/Functions. - - -[[spring-cloud-streams-overview-using-polled-consumers]] -=== Using Polled Consumers - -[[overview]] -==== Overview - -When using polled consumers, you poll the `PollableMessageSource` on demand. -To define binding for polled consumer you need to provide `spring.cloud.stream.pollable-source` property. - -Consider the following example of a polled consumer binding: - -[source,text] ----- ---spring.cloud.stream.pollable-source=myDestination ----- - -The pollable-source name `myDestination` in the preceding example will result in `myDestination-in-0` binding name to stay -consistent with functional programming model. - -Given the polled consumer in the preceding example, you might use it as follows: - -[source,java] ----- -@Bean -public ApplicationRunner poller(PollableMessageSource destIn, MessageChannel destOut) { - return args -> { - while (someCondition()) { - try { - if (!destIn.poll(m -> { - String newPayload = ((String) m.getPayload()).toUpperCase(); - destOut.send(new GenericMessage<>(newPayload)); - })) { - Thread.sleep(1000); - } - } - catch (Exception e) { - // handle failure - } - } - }; -} ----- - -A less manual and more Spring-like alternative would be to configure a scheduled task bean. For example, - -[source,java] ----- - -@Scheduled(fixedDelay = 5_000) -public void poll() { - System.out.println("Polling..."); - this.source.poll(m -> { - System.out.println(m.getPayload()); - - }, new ParameterizedTypeReference() { }); -} ----- - - -The `PollableMessageSource.poll()` method takes a `MessageHandler` argument (often a lambda expression, as shown here). -It returns `true` if the message was received and successfully processed. - -As with message-driven consumers, if the `MessageHandler` throws an exception, messages are published to error channels, -as discussed in `<>`. - -Normally, the `poll()` method acknowledges the message when the `MessageHandler` exits. -If the method exits abnormally, the message is rejected (not re-queued), but see <>. -You can override that behavior by taking responsibility for the acknowledgment, as shown in the following example: - -[source,java] ----- -@Bean -public ApplicationRunner poller(PollableMessageSource dest1In, MessageChannel dest2Out) { - return args -> { - while (someCondition()) { - if (!dest1In.poll(m -> { - StaticMessageHeaderAccessor.getAcknowledgmentCallback(m).noAutoAck(); - // e.g. hand off to another thread which can perform the ack - // or acknowledge(Status.REQUEUE) - - })) { - Thread.sleep(1000); - } - } - }; -} ----- - -IMPORTANT: You must `ack` (or `nack`) the message at some point, to avoid resource leaks. - -IMPORTANT: Some messaging systems (such as Apache Kafka) maintain a simple offset in a log. If a delivery fails and is re-queued with `StaticMessageHeaderAccessor.getAcknowledgmentCallback(m).acknowledge(Status.REQUEUE);`, any later successfully ack'd messages are redelivered. - -There is also an overloaded `poll` method, for which the definition is as follows: - -[source,java] ----- -poll(MessageHandler handler, ParameterizedTypeReference type) ----- - -The `type` is a conversion hint that allows the incoming message payload to be converted, as shown in the following example: - -[source,java] ----- -boolean result = pollableSource.poll(received -> { - Map payload = (Map) received.getPayload(); - ... - - }, new ParameterizedTypeReference>() {}); ----- - - -[[polled-errors]] -==== Handling Errors - -By default, an error channel is configured for the pollable source; if the callback throws an exception, an `ErrorMessage` is sent to the error channel (`..errors`); this error channel is also bridged to the global Spring Integration `errorChannel`. - -You can subscribe to either error channel with a `@ServiceActivator` to handle errors; without a subscription, the error will simply be logged and the message will be acknowledged as successful. -If the error channel service activator throws an exception, the message will be rejected (by default) and won't be redelivered. -If the service activator throws a `RequeueCurrentMessageException`, the message will be requeued at the broker and will be again retrieved on a subsequent poll. - -If the listener throws a `RequeueCurrentMessageException` directly, the message will be requeued, as discussed above, and will not be sent to the error channels. - -[[event-routing]] -== Event Routing - -Event Routing, in the context of Spring Cloud Stream, is the ability to either -_a) route events to a particular event subscriber_ or -_b) route events produced by an event subscriber to a particular destination_. -Here we'll refer to it as route ‘TO’ and route ‘FROM’. - -[[routing-to-consumer]] -=== Routing TO Consumer -Routing can be achieved by relying on `RoutingFunction` available in Spring Cloud Function 3.0. All you need to do is enable it via -`--spring.cloud.stream.function.routing.enabled=true` application property or provide `spring.cloud.function.routing-expression` property. -Once enabled `RoutingFunction` will be bound to input destination -receiving all the messages and route them to other functions based on the provided instruction. - -IMPORTANT: For the purposes of binding the name of the routing destination is `functionRouter-in-0` -(see RoutingFunction.FUNCTION_NAME and binding naming convention <>). - -Instruction could be provided with individual messages as well as application properties. - -Here are couple of samples: - -[[using-message-headers]] -==== Using message headers -[source,java] ----- -@SpringBootApplication -public class SampleApplication { - - public static void main(String[] args) { - SpringApplication.run(SampleApplication.class, - "--spring.cloud.stream.function.routing.enabled=true"); - } - - @Bean - public Consumer even() { - return value -> { - System.out.println("EVEN: " + value); - }; - } - - @Bean - public Consumer odd() { - return value -> { - System.out.println("ODD: " + value); - }; - } -} ----- -By sending a message to the `functionRouter-in-0` destination exposed by the binder (i.e., rabbit, kafka), -such message will be routed to the appropriate (‘even’ or ‘odd’) Consumer. - -By default `RoutingFunction` will look for a `spring.cloud.function.definition` or `spring.cloud.function.routing-expression` (for more dynamic scenarios with SpEL) -header and if it is found, its value will be treated as the routing instruction. - -For example, -setting `spring.cloud.function.routing-expression` header to value `T(java.lang.System).currentTimeMillis() % 2 == 0 ? 'even' : 'odd'` will end up semi-randomly routing request to either `odd` or `even` functions. -Also, for SpEL, the _root object_ of the evaluation context is `Message` so you can do evaluation on individual headers (or message) as well `....routing-expression=headers['type']` - -[[using-application-properties]] -==== Using application properties - -The `spring.cloud.function.routing-expression` and/or `spring.cloud.function.definition` -can be passed as application properties (e.g., `spring.cloud.function.routing-expression=headers['type']`. -[source,java] ----- -@SpringBootApplication -public class RoutingStreamApplication { - - public static void main(String[] args) { - SpringApplication.run(RoutingStreamApplication.class, - "--spring.cloud.function.routing-expression=" - + "T(java.lang.System).nanoTime() % 2 == 0 ? 'even' : 'odd'"); - } - @Bean - public Consumer even() { - return value -> System.out.println("EVEN: " + value); - } - - @Bean - public Consumer odd() { - return value -> System.out.println("ODD: " + value); - } -} ----- -IMPORTANT: Passing instructions via application properties is especially important for reactive functions given that a reactive -function is only invoked once to pass the Publisher, so access to the individual items is limited. - -[[routing-function-and-output-binding]] -==== Routing Function and output binding - -`RoutingFunction` is a `Function` and as such treated no differently than any other function. Well. . . almost. - -When `RoutingFunction` routes to another `Function`, its output is sent to the output binding of the `RoutingFunction` which -is `functionRouter-in-0` as expected. But what if `RoutingFunction` routes to a `Consumer`? In other words the result of invocation -of the `RoutingFunction` may not produce anything to be sent to the output binding, thus making it necessary to even have one. -So, we do treat `RoutingFunction` a little bit differently when we create bindings. And even though it is transparent to you as a user -(there is really nothing for you to do), being aware of some of the mechanics would help you understand its inner workings. - -So, the rule is; -We never create output binding for the `RoutingFunction`, only input. So when you routing to `Consumer`, the `RoutingFunction` effectively -becomes as a `Consumer` by not having any output bindings. However, if `RoutingFunction` happen to route to another `Function` which produces -the output, the output binding for the `RoutingFunction` will be create dynamically at which point `RoutingFunction` will act as a regular `Function` -with regards to bindings (having both input and output bindings). - -[[routing-from-consumer]] -=== Routing FROM Consumer - -Aside from static destinations, Spring Cloud Stream lets applications send messages to dynamically bound destinations. -This is useful, for example, when the target destination needs to be determined at runtime. -Applications can do so in one of two ways. - -[[spring-cloud-stream-sendto-destination]] -==== spring.cloud.stream.sendto.destination - -You can also delegate to the framework to dynamically resolve the output destination by specifying `spring.cloud.stream.sendto.destination` header -set to the name of the destination to be resolved. - -Consider the following example: - -[source,java] ----- -@SpringBootApplication -@Controller -public class SourceWithDynamicDestination { - - @Bean - public Function> destinationAsPayload() { - return value -> { - return MessageBuilder.withPayload(value) - .setHeader("spring.cloud.stream.sendto.destination", value).build();}; - } -} ----- - -Albeit trivial you can clearly see in this example, our output is a Message with `spring.cloud.stream.sendto.destination` header -set to the value of he input argument. The framework will consult this header and will attempt to create or discover -a destination with that name and send output to it. - - -If destination names are known in advance, you can configure the producer properties as with any other destination. -Alternatively, if you register a `NewDestinationBindingCallback<>` bean, it is invoked just before the binding is created. -The callback takes the generic type of the extended producer properties used by the binder. -It has one method: - -[source, java] ----- -void configure(String destinationName, MessageChannel channel, ProducerProperties producerProperties, - T extendedProducerProperties); ----- - -The following example shows how to use the RabbitMQ binder: - -[source, java] ----- -@Bean -public NewDestinationBindingCallback dynamicConfigurer() { - return (name, channel, props, extended) -> { - props.setRequiredGroups("bindThisQueue"); - extended.setQueueNameGroupOnly(true); - extended.setAutoBindDlq(true); - extended.setDeadLetterQueueName("myDLQ"); - }; -} ----- - -NOTE: If you need to support dynamic destinations with multiple binder types, use `Object` for the generic type and cast the `extended` argument as needed. - -Also, please see <> section to see how yet another option (StreamBridge) can be utilized for similar cases. - -[[post-processing-after-sending-message]] -== Post processing (after sending message) - -Once function is invoked, its result is sent by the framework to a target destination which effectively completes function invocation cycle. - -However such cycle may not be fully complete from the business standpoint until some additional tasks are performed *after* completion of this cycle. -While this could be accomplished with a simple combination of `Consumer` and `StreamBridge` as described in this -https://stackoverflow.com/questions/75917883/post-processing-after-spring-cloud-stream-function[Stack Overflow post], since version 4.0.3 the framework -provides a more idiomatic approach to solve this issue via `PostProcessingFunction` provided by Spring Cloud Function project. -The `PostProcessingFunction` is a special semi-marker function which contains one additional method `postProcess(Message>)` designed -to provide a place for implementing such post processing task. - ----- -package org.springframework.cloud.function.context -. . . -public interface PostProcessingFunction extends Function { - default void postProcess(Message result) { - } -} ----- - -So, now you have two options. - -Option 1: You can implement your function as `PostProcessingFunction` and also include the additional post processing behavior by implementing its `postProcess(Message>)` method. - ----- -private static class Uppercase implements PostProcessingFunction { - - @Override - public String apply(String input) { - return input.toUpperCase(); - } - - @Override - public void postProcess(Message result) { - System.out.println("Function Uppercase has been successfully invoked and its result successfully sent to target destination"); - } -} -. . . -@Bean -public Function uppercase() { - return new Uppercase(); -} ----- - -Option 2: If you already have an existing function and don't want to change its implementation or want to keep your function as POJO, you can simply implement only `postProcess(Message>)` method and compose this new post processing function with your other function. ----- -private static class Logger implements PostProcessingFunction { - - @Override - public void postProcess(Message result) { - System.out.println("Function has been successfully invoked and its result successfully sent to target destination"); - } -} -. . . -@Bean -public Function uppercase() { - return v -> v.toUpperCase(); -} -@Bean -public Function logger() { - return new Logger(); -} -. . . -// and then have your function definition as such `uppercase|logger` ----- - -NOTE: -In case of function composition only the last instance of `PostProcessingFunction` (if present) will take effect. For example, let's say you have the -following function definition - `foo|bar|baz` and both `foo` and `baz` are instances of `PostProcessingFunction`. Only `baz.postProcess(Message>)` will be invoked. -If `baz` is not an instance of `PostProcessingFunction`, then no post processing functionality will be performed. - -One may argue that you can easily do that via function composition by simply composing a post-processor as just another `Function`. That is indeed a possibility however -the post processing functionality in this case will be invoked right after invocation of the previous function and before the message is sent to a target destination -which is before the function invocation cycle is complete. - -[[spring-cloud-stream-overview-error-handling]] -== Error Handling - -In this section we'll explain the general idea behind error handling mechanisms provided by the framework. -We'll be using Rabbit binder as an example, since individual binders define different set -of properties for certain supported mechanisms specific to underlying broker capabilities (such as Kafka binder). - -Errors happen, and Spring Cloud Stream provides several flexible mechanisms to deal with them. Note, the techniques are dependent on binder implementation and the -capability of the underlying messaging middleware as well as programming model (more on this later). - -Whenever Message handler (function) throws an exception, it is propagated back to the binder, at which point binder will make several attempts at re-trying -the same message (3 by default) using `RetryTemplate` provided by the https://github.com/spring-projects/spring-retry[Spring Retry] library. -If retries are unsuccessful it is up to the error handling mechanism which may _drop_ the message, _re-queue_ the message for re-processing or _send the failed message to DLQ_. - -Both Rabbit and Kafka support these concepts (especially DLQ). However, other binders may not, so refer to your individual binder’s documentation for details on supported -error-handling options. - -Keep in mind however, the _reactive function_ does NOT qualify as a Message handler, since it does not handle individual messages and -instead provides a way to connect stream (i.e., Flux) provided by the framework with the one provided by the user. -***Why is this important?*** -That is because anything you read later in this section with regard to Retry Template, dropping failed messages, retrying, -DLQ and configuration properties that assist with all of it ***only*** applies to Message handlers (i.e., imperative functions). - -Reactive API provides a very rich library of its own operators and mechanisms to assist you with error handling specific to -variety of reactive uses cases which are far more complex then simple Message handler cases, So use them, such -as `public final Flux retryWhen(Retry retrySpec);` that you can find in `reactor.core.publisher.Flux`. - -[source,java] ----- -@Bean -public Function, Flux> uppercase() { - return flux -> flux - .retryWhen(Retry.backoff(3, Duration.ofMillis(1000))) - .map(v -> v.toUpperCase()); -} ----- - -[[drop-failed-messages]] -=== Drop Failed Messages - -By default, the system provides error handlers. The first error handler will simply log error message. The second error handler is binder specific error handler -which is responsible for handling error message in the context of a specific messaging system (e.g., send to DLQ). But since no additional error handling configuration was provided (in this current scenario) this handler will not do anything. So essentially after being logged, the message will be dropped. - -While acceptable in some cases, for most cases, it is not, and we need some recovery mechanism to avoid message loss. - -[[handle-error-messages]] -=== Handle Error Messages - -In the previous section we mentioned that by default messages that resulted in error are effectively logged and dropped. The framework also exposes mechanism for you -to provide custom error handler (i.e., to send notification or write to database, etc). You can do so by adding `Consumer` that is specifically designed to accept `ErrorMessage` which aside form all the information about the error (e.g., stack trace etc) contains the original message (the one that triggered the error). -NOTE: Custom error handler is mutually exclusive with framework provided error handlers (i.e., logging and binder error handler - see previous section) to ensure that they do not interfere. - -[source,java] ----- -@Bean -public Consumer myErrorHandler() { - return v -> { - // send SMS notification code - }; -} ----- - -To identify such consumer as an error handler all you need is to provide `error-handler-definition` property pointing to the function name - `spring.cloud.stream.bindings..error-handler-definition=myErrorHandler`. - -For example, for binding name `uppercase-in-0` the property would look like this: - -[source,text] ----- -spring.cloud.stream.bindings.uppercase-in-0.error-handler-definition=myErrorHandler ----- -And if you used special mapping instruction to map binding to a more readable name - `spring.cloud.stream.function.bindings.uppercase-in-0=upper`, then this property would look like this: -[source,text] ----- -spring.cloud.stream.bindings.upper.error-handler-definition=myErrorHandler. ----- - -NOTE: If by accident you declare such handler as a `Function`, it will still work with the exception that nothing is going to be done with its output. However, given that such handler is still relying on functionality provided by Spring Cloud Function, you can also benefit from function composition in the event your handler has some complexity which you would like to address through function composition (however unlikely). - - -***Default Error Handler*** - -If you want to have a single error handler for all function beans, you can use the standard spring-cloud-stream mechanism for defining default properties `spring.cloud.stream.default.error-handler-definition=myErrorHandler` - - -[[dlq-dead-letter-queue]] -=== DLQ - Dead Letter Queue - -Perhaps the most common mechanism, DLQ allows failed messages to be sent to a special destination: the _Dead Letter Queue_. - -When configured, failed messages are sent to this destination for subsequent re-processing or auditing and reconciliation. - -Consider the following example: - -[source,java] ----- -@SpringBootApplication -public class SimpleStreamApplication { - - public static void main(String[] args) throws Exception { - SpringApplication.run(SimpleStreamApplication.class, - "--spring.cloud.function.definition=uppercase", - "--spring.cloud.stream.bindings.uppercase-in-0.destination=uppercase", - "--spring.cloud.stream.bindings.uppercase-in-0.group=myGroup", - "--spring.cloud.stream.rabbit.bindings.uppercase-in-0.consumer.auto-bind-dlq=true" - ); - } - - @Bean - public Function uppercase() { - return personIn -> { - throw new RuntimeException("intentional"); - }); - }; - } -} ----- - -As a reminder, in this example `uppercase-in-0` segment of the property corresponds to the name of the input destination binding. -The `consumer` segment indicates that it is a consumer property. - -NOTE: When using DLQ, at least the `group` property must be provided for proper naming of the DLQ destination. However `group` is often used together -with `destination` property, as in our example. - - -Aside from some standard properties we also set the `auto-bind-dlq` to instruct the binder to create and configure DLQ destination for -`uppercase-in-0` binding which corresponds to `uppercase` destination (see corresponding property), which results in an additional Rabbit queue named `uppercase.myGroup.dlq` (see Kafka documentation for Kafka specific DLQ properties). - -Once configured, all failed messages are routed to this destination preserving the original message for further actions. - -And you can see that the error message contains more information relevant to the original error, as follows: - -[source,text] ----- -. . . . -x-exception-stacktrace: org.springframework.messaging.MessageHandlingException: nested exception is - org.springframework.messaging.MessagingException: has an error, failedMessage=GenericMessage [payload=byte[15], - headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_receivedRoutingKey=input.hello, amqp_deliveryTag=1, - deliveryAttempt=3, amqp_consumerQueue=input.hello, amqp_redelivered=false, id=a15231e6-3f80-677b-5ad7-d4b1e61e486e, - amqp_consumerTag=amq.ctag-skBFapilvtZhDsn0k3ZmQg, contentType=application/json, timestamp=1522327846136}] - at org.spring...integ...han...MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:107) - at. . . . . -Payload: blah ----- - -You can also facilitate immediate dispatch to DLQ (without re-tries) by setting `max-attempts` to '1'. For example, -[source,text] ----- ---spring.cloud.stream.bindings.uppercase-in-0.consumer.max-attempts=1 ----- - -[[retry-template]] -=== Retry Template - -In this section we cover configuration properties relevant to configuration of retry capabilities. - -The `RetryTemplate` is part of the https://github.com/spring-projects/spring-retry[Spring Retry] library. -While it is out of scope of this document to cover all of the capabilities of the `RetryTemplate`, we -will mention the following consumer properties that are specifically related to -the `RetryTemplate`: - -maxAttempts:: -The number of attempts to process the message. -+ -Default: 3. -backOffInitialInterval:: -The backoff initial interval on retry. -+ -Default 1000 milliseconds. -backOffMaxInterval:: -The maximum backoff interval. -+ -Default 10000 milliseconds. -backOffMultiplier:: -The backoff multiplier. -+ -Default 2.0. -defaultRetryable:: -Whether exceptions thrown by the listener that are not listed in the `retryableExceptions` are retryable. -+ -Default: `true`. -retryableExceptions:: -A map of Throwable class names in the key and a boolean in the value. -Specify those exceptions (and subclasses) that will or won't be retried. -Also see `defaultRetriable`. -Example: `spring.cloud.stream.bindings.input.consumer.retryable-exceptions.java.lang.IllegalStateException=false`. -+ -Default: empty. - -While the preceding settings are sufficient for the majority of the customization requirements, they may not satisfy certain complex requirements, at which -point you may want to provide your own instance of the `RetryTemplate`. To do so configure it as a bean in your application configuration. The application provided -instance will override the one provided by the framework. Also, to avoid conflicts you must qualify the instance of the `RetryTemplate` you want to be used by the binder -as `@StreamRetryTemplate`. For example, - -[source,java] ----- -@StreamRetryTemplate -public RetryTemplate myRetryTemplate() { - return new RetryTemplate(); -} ----- -As you can see from the above example you don't need to annotate it with `@Bean` since `@StreamRetryTemplate` is a qualified `@Bean`. - -If you need to be more precise with your `RetryTemplate`, you can specify the bean by name in your `ConsumerProperties` to associate -the specific retry bean per binding. - -[source] ----- -spring.cloud.stream.bindings..consumer.retry-template-name= ----- - - -[[spring-cloud-stream-overview-binders]] -= Binders - -Spring Cloud Stream provides a Binder abstraction for use in connecting to physical destinations at the external middleware. -This section provides information about the main concepts behind the Binder SPI, its main components, and implementation-specific details. - -[[producers-and-consumers]] -== Producers and Consumers - -The following image shows the general relationship of producers and consumers: - -.Producers and Consumers -image::producers-consumers.png[width=800,scaledwidth="75%",align="center"] - -A producer is any component that sends messages to a binding destination. -The binding destination can be bound to an external message broker with a `Binder` implementation for that broker. -When invoking the `bindProducer()` method, the first parameter is the name of the destination within the broker, the second parameter is the instance if local destination to which the producer sends messages, and the third parameter contains properties (such as a partition key expression) to be used within the adapter that is created for that binding destination. - -A consumer is any component that receives messages from the binding destination. -As with a producer, the consumer can be bound to an external message broker. -When invoking the `bindConsumer()` method, the first parameter is the destination name, and a second parameter provides the name of a logical group of consumers. -Each group that is represented by consumer bindings for a given destination receives a copy of each message that a producer sends to that destination (that is, it follows normal publish-subscribe semantics). -If there are multiple consumer instances bound with the same group name, then messages are load-balanced across those consumer instances so that each message sent by a producer is consumed by only a single consumer instance within each group (that is, it follows normal queueing semantics). - -[[spring-cloud-stream-overview-binder-api]] -== Binder SPI - -The Binder SPI consists of a number of interfaces, out-of-the box utility classes, and discovery strategies that provide a pluggable mechanism for connecting to external middleware. - -The key point of the SPI is the `Binder` interface, which is a strategy for connecting inputs and outputs to external middleware. The following listing shows the definition of the `Binder` interface: - -[source,java] ----- -public interface Binder { - Binding bindConsumer(String bindingName, String group, T inboundBindTarget, C consumerProperties); - - Binding bindProducer(String bindingName, T outboundBindTarget, P producerProperties); -} ----- - -The interface is parameterized, offering a number of extension points: - -* Input and output bind targets. -* Extended consumer and producer properties, allowing specific Binder implementations to add supplemental properties that can be supported in a type-safe manner. - -A typical binder implementation consists of the following: - -* A class that implements the `Binder` interface; -* A Spring `@Configuration` class that creates a bean of type `Binder` along with the middleware connection infrastructure. -* A `META-INF/spring.binders` file found on the classpath containing one or more binder definitions, as shown in the following example: -+ -[source] ----- -kafka:\ -org.springframework.cloud.stream.binder.kafka.config.KafkaBinderConfiguration ----- - -NOTE: As it was mentioned earlier Binder abstraction is also one of the extension points of the framework. So if you can't find a suitable binder in the preceding list you can implement your own binder on top of Spring Cloud Stream. -In the https://medium.com/@domenicosibilio/how-to-create-a-spring-cloud-stream-binder-from-scratch-ab8b29ee931b[How to create a Spring Cloud Stream Binder from scratch] post a community member documents -in details, with an example, a set of steps necessary to implement a custom binder. -The steps are also highlighted in the `<>` section. - -[[binder-detection]] -== Binder Detection - -Spring Cloud Stream relies on implementations of the Binder SPI to perform the task of connecting (binding) user code to message brokers. -Each Binder implementation typically connects to one type of messaging system. - -[[classpath-detection]] -=== Classpath Detection - -By default, Spring Cloud Stream relies on Spring Boot's auto-configuration to configure the binding process. -If a single Binder implementation is found on the classpath, Spring Cloud Stream automatically uses it. -For example, a Spring Cloud Stream project that aims to bind only to RabbitMQ can add the following dependency: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-rabbit - ----- - -For the specific Maven coordinates of other binder dependencies, see the documentation of that binder implementation. - -[[multiple-binders]] -== Multiple Binders on the Classpath - -When multiple binders are present on the classpath, the application must indicate which binder is to be used for each destination binding. -Each binder configuration contains a `META-INF/spring.binders` file, which is a simple properties file, as shown in the following example: - -[source] ----- -rabbit:\ -org.springframework.cloud.stream.binder.rabbit.config.RabbitServiceAutoConfiguration ----- - -Similar files exist for the other provided binder implementations (such as Kafka), and custom binder implementations are expected to provide them as well. -The key represents an identifying name for the binder implementation, whereas the value is a comma-separated list of configuration classes that each contain one and only one bean definition of type `org.springframework.cloud.stream.binder.Binder`. - -Binder selection can either be performed globally, using the `spring.cloud.stream.defaultBinder` property (for example, `spring.cloud.stream.defaultBinder=rabbit`) or individually, by configuring the binder on each binding. -For instance, a processor application (that has bindings named `input` and `output` for read and write respectively) that reads from Kafka and writes to RabbitMQ can specify the following configuration: - -[source] ----- -spring.cloud.stream.bindings.input.binder=kafka -spring.cloud.stream.bindings.output.binder=rabbit ----- - -[[multiple-systems]] -== Connecting to Multiple Systems - -By default, binders share the application's Spring Boot auto-configuration, so that one instance of each binder found on the classpath is created. -If your application should connect to more than one broker of the same type, you can specify multiple binder configurations, each with different environment settings. - -NOTE: Turning on explicit binder configuration disables the default binder configuration process altogether. -If you do so, all binders in use must be included in the configuration. -Frameworks that intend to use Spring Cloud Stream transparently may create binder configurations that can be referenced by name, but they do not affect the default binder configuration. -In order to do so, a binder configuration may have its `defaultCandidate` flag set to false (for example, `spring.cloud.stream.binders..defaultCandidate=false`). -This denotes a configuration that exists independently of the default binder configuration process. - -The following example shows a typical configuration for a processor application that connects to two RabbitMQ broker instances: - -[source,yml] ----- -spring: - cloud: - stream: - bindings: - input: - destination: thing1 - binder: rabbit1 - output: - destination: thing2 - binder: rabbit2 - binders: - rabbit1: - type: rabbit - environment: - spring: - rabbitmq: - host: - rabbit2: - type: rabbit - environment: - spring: - rabbitmq: - host: ----- -NOTE: The `environment` property of the particular binder can also be used for any Spring Boot property, -including this `spring.main.sources` which can be useful for adding additional configurations for the -particular binders, e.g. overriding auto-configured beans. - -For example; -[source, yaml] ----- -environment: - spring: - main: - sources: com.acme.config.MyCustomBinderConfiguration ----- - -To activate a specific profile for the particular binder environment, you should use a `spring.profiles.active` property: - -[source, yaml] ----- -environment: - spring: - profiles: - active: myBinderProfile ----- - -[[binder-customizer]] -== Customizing binders in multi binder applications - -When an application has multiple binders in it and wants to customize the binders, then that can be achieved by providing a `BinderCustomizer` implementation. -In the case of applications with a single binder, this special customizer is not necessary since the binder context can access the customization beans directly. -However, this is not the case in a multi-binder scenario, since various binders live in different application contexts. -By providing an implementation of `BinderCustomizer` interface, the binders, although reside in different application contexts, will receive the customization. -Spring Cloud Stream ensures that the customizations take place before the applications start using the binders. -The user must check for the binder type and then apply the necessary customizations. - -Here is an example of providing a `BinderCustomizer` bean. - -[source, java] ----- -@Bean -public BinderCustomizer binderCustomizer() { - return (binder, binderName) -> { - if (binder instanceof KafkaMessageChannelBinder kafkaMessageChannelBinder) { - kafkaMessageChannelBinder.setRebalanceListener(...); - } - else if (binder instanceof KStreamBinder) { - ... - } - else if (binder instanceof RabbitMessageChannelBinder) { - ... - } - }; -} ----- - -Note that, when there are more than one instance of the same type of the binder, the binder name can be used to filter customization. - -[[binding_visualization_control]] -== Binding visualization and control -Spring Cloud Stream supports visualization and control of the Bindings through Actuator endpoints as well as programmatic way. - -[[programmatic-way]] -=== Programmatic way - -Since version 3.1 we expose `org.springframework.cloud.stream.binding.BindingsLifecycleController` which is registered as bean and once -injected could be used to control the lifecycle of individual bindings - -For example, looks at the fragment from one of the test cases. As you can see we retrieve `BindingsLifecycleController` -from spring application context and execute individual methods to control the lifecycle of `echo-in-0` binding.. - -[source,java] ----- -BindingsLifecycleController bindingsController = context.getBean(BindingsLifecycleController.class); -Binding binding = bindingsController.queryState("echo-in-0"); -assertThat(binding.isRunning()).isTrue(); -bindingsController.changeState("echo-in-0", State.STOPPED); -//Alternative way of changing state. For convenience we expose start/stop and pause/resume operations. -//bindingsController.stop("echo-in-0") -assertThat(binding.isRunning()).isFalse(); ----- - -[[actuator]] -=== Actuator -Since actuator and web are optional, you must first add one of the web dependencies as well as add the actuator dependency manually. -The following example shows how to add the dependency for the Web framework: - -[source,xml] ----- - - org.springframework.boot - spring-boot-starter-web - ----- - -The following example shows how to add the dependency for the WebFlux framework: - -[source,xml] ----- - - org.springframework.boot - spring-boot-starter-webflux - ----- - -You can add the Actuator dependency as follows: -[source,xml] ----- - - org.springframework.boot - spring-boot-starter-actuator - ----- - -NOTE: To run Spring Cloud Stream 2.0 apps in Cloud Foundry, you must add `spring-boot-starter-web` and `spring-boot-starter-actuator` to the classpath. Otherwise, the -application will not start due to health check failures. - -You must also enable the `bindings` actuator endpoints by setting the following property: `--management.endpoints.web.exposure.include=bindings`. - -Once those prerequisites are satisfied. you should see the following in the logs when application start: - - : Mapped "{[/actuator/bindings/{name}],methods=[POST]. . . - : Mapped "{[/actuator/bindings],methods=[GET]. . . - : Mapped "{[/actuator/bindings/{name}],methods=[GET]. . . - -To visualize the current bindings, access the following URL: -`http://:/actuator/bindings` - -Alternative, to see a single binding, access one of the URLs similar to the following: -`http://:/actuator/bindings/` - -You can also stop, start, pause, and resume individual bindings by posting to the same URL while providing a `state` argument as JSON, as shown in the following examples: - - curl -d '{"state":"STOPPED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName - curl -d '{"state":"STARTED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName - curl -d '{"state":"PAUSED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName - curl -d '{"state":"RESUMED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName - -NOTE: `PAUSED` and `RESUMED` work only when the corresponding binder and its underlying technology supports it. Otherwise, you see the warning message in the logs. -Currently, only Kafka and [Solace](https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter#consumer-bindings-pauseresume) binders supports the `PAUSED` and `RESUMED` states. - -[[binder-configuration-properties]] -== Binder Configuration Properties - -The following properties are available when customizing binder configurations. These properties exposed via `org.springframework.cloud.stream.config.BinderProperties` - -They must be prefixed with `spring.cloud.stream.binders.`. - -type:: -The binder type. -It typically references one of the binders found on the classpath -- in particular, a key in a `META-INF/spring.binders` file. -+ -By default, it has the same value as the configuration name. -inheritEnvironment:: -Whether the configuration inherits the environment of the application itself. -+ -Default: `true`. -environment:: -Root for a set of properties that can be used to customize the environment of the binder. -When this property is set, the context in which the binder is being created is not a child of the application context. -This setting allows for complete separation between the binder components and the application components. -+ -Default: `empty`. -defaultCandidate:: -Whether the binder configuration is a candidate for being considered a default binder or can be used only when explicitly referenced. -This setting allows adding binder configurations without interfering with the default processing. -+ -Default: `true`. - -[[spring-cloud-stream-overview-custom-binder-impl]] -== Implementing Custom Binders - -In order to implement a custom `Binder`, all you need is to: - -- Add the required dependencies -- Provide a ProvisioningProvider implementation -- Provide a MessageProducer implementation -- Provide a MessageHandler implementation -- Provide a Binder implementation -- Create a Binder Configuration -- Define your binder in META-INF/spring.binders - -***Add the required dependencies*** - -Add the `spring-cloud-stream` dependency to your project _(eg. for Maven)_: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream - ${spring.cloud.stream.version} - ----- - -***Provide a ProvisioningProvider implementation*** - -The `ProvisioningProvider` is responsible for the provisioning of consumer and producer destinations, and is required to convert the logical destinations included in the application.yml or application.properties file in physical destination references. - -Below an example of ProvisioningProvider implementation that simply trims the destinations provided via input/output bindings configuration: - -[source,java] ----- -public class FileMessageBinderProvisioner implements ProvisioningProvider { - - @Override - public ProducerDestination provisionProducerDestination( - final String name, - final ProducerProperties properties) { - - return new FileMessageDestination(name); - } - - @Override - public ConsumerDestination provisionConsumerDestination( - final String name, - final String group, - final ConsumerProperties properties) { - - return new FileMessageDestination(name); - } - - private class FileMessageDestination implements ProducerDestination, ConsumerDestination { - - private final String destination; - - private FileMessageDestination(final String destination) { - this.destination = destination; - } - - @Override - public String getName() { - return destination.trim(); - } - - @Override - public String getNameForPartition(int partition) { - throw new UnsupportedOperationException("Partitioning is not implemented for file messaging."); - } - - } - -} ----- - -***Provide a MessageProducer implementation*** - -The `MessageProducer` is responsible for consuming events and handling them as messages to the client application that is configured to consume such events. - -Here is an example of MessageProducer implementation that extends the `MessageProducerSupport` abstraction in order to poll on a file that matches the trimmed destination name and is located in the project path, while also archiving read messages and discarding consequent identical messages: - -[source,java] ----- -public class FileMessageProducer extends MessageProducerSupport { - - public static final String ARCHIVE = "archive.txt"; - private final ConsumerDestination destination; - private String previousPayload; - - public FileMessageProducer(ConsumerDestination destination) { - this.destination = destination; - } - - @Override - public void doStart() { - receive(); - } - - private void receive() { - ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1); - - executorService.scheduleWithFixedDelay(() -> { - String payload = getPayload(); - - if(payload != null) { - Message receivedMessage = MessageBuilder.withPayload(payload).build(); - archiveMessage(payload); - sendMessage(receivedMessage); - } - - }, 0, 50, MILLISECONDS); - } - - private String getPayload() { - try { - List allLines = Files.readAllLines(Paths.get(destination.getName())); - String currentPayload = allLines.get(allLines.size() - 1); - - if(!currentPayload.equals(previousPayload)) { - previousPayload = currentPayload; - return currentPayload; - } - } catch (IOException e) { - throw new RuntimeException(e); - } - - return null; - } - - private void archiveMessage(String payload) { - try { - Files.write(Paths.get(ARCHIVE), (payload + "\n").getBytes(), CREATE, APPEND); - } catch (IOException e) { - throw new RuntimeException(e); - } - } - -} ----- - -NOTE: When implementing a custom binder, this step is not strictly mandatory as you could always resort to using an already existing MessageProducer implementation! - -***Provide a MessageHandler implementation*** - -The `MessageHandler` provides the logic required to produce an event. - -Here is an example of MessageHandler implementation: - -[source,java] ----- -public class FileMessageHandler implements MessageHandler{ - - @Override - public void handleMessage(Message message) throws MessagingException { - //write message to file - } - -} ----- - -NOTE: When implementing a custom binder, this step is not strictly mandatory as you could always resort to using an already existing MessageHandler implementation! - -***Provide a Binder implementation*** - -You are now able to provide your own implementation of the `Binder` abstraction. This can be easily done by: - -- extending the `AbstractMessageChannelBinder` class -- specifying your ProvisioningProvider as a generic argument of the AbstractMessageChannelBinder -- overriding the `createProducerMessageHandler` and `createConsumerEndpoint` methods - -_eg.:_ -[source,java] ----- -public class FileMessageBinder extends AbstractMessageChannelBinder { - - public FileMessageBinder( - String[] headersToEmbed, - FileMessageBinderProvisioner provisioningProvider) { - - super(headersToEmbed, provisioningProvider); - } - - @Override - protected MessageHandler createProducerMessageHandler( - final ProducerDestination destination, - final ProducerProperties producerProperties, - final MessageChannel errorChannel) throws Exception { - - return message -> { - String fileName = destination.getName(); - String payload = new String((byte[])message.getPayload()) + "\n"; - - try { - Files.write(Paths.get(fileName), payload.getBytes(), CREATE, APPEND); - } catch (IOException e) { - throw new RuntimeException(e); - } - }; - } - - @Override - protected MessageProducer createConsumerEndpoint( - final ConsumerDestination destination, - final String group, - final ConsumerProperties properties) throws Exception { - - return new FileMessageProducer(destination); - } - -} ----- - -***Create a Binder Configuration*** - -It is strictly required that you create a Spring Configuration to initialize the bean for your binder implementation _(and all other beans that you might need)_: - -[source,java] ----- -@Configuration -public class FileMessageBinderConfiguration { - - @Bean - @ConditionalOnMissingBean - public FileMessageBinderProvisioner fileMessageBinderProvisioner() { - return new FileMessageBinderProvisioner(); - } - - @Bean - @ConditionalOnMissingBean - public FileMessageBinder fileMessageBinder(FileMessageBinderProvisioner fileMessageBinderProvisioner) { - return new FileMessageBinder(null, fileMessageBinderProvisioner); - } - -} ----- - -***Define your binder in META-INF/spring.binders*** - -Finally, you must define your binder in a `META-INF/spring.binders` file on the classpath, specifying both the name of the binder and the full qualified name of your Binder Configuration class: - -[source] ----- -myFileBinder:\ -com.example.springcloudstreamcustombinder.config.FileMessageBinderConfiguration ----- - -[[configuration-options]] -= Configuration Options - -Spring Cloud Stream supports general configuration options as well as configuration for bindings and binders. -Some binders let additional binding properties support middleware-specific features. - -Configuration options can be provided to Spring Cloud Stream applications through any mechanism supported by Spring Boot. -This includes application arguments, environment variables, and YAML or .properties files. - -[[binding-service-properties]] -== Binding Service Properties - -These properties are exposed via `org.springframework.cloud.stream.config.BindingServiceProperties` - -spring.cloud.stream.instanceCount:: -The number of deployed instances of an application. -Must be set for partitioning on the producer side. Must be set on the consumer side when using RabbitMQ and with Kafka if `autoRebalanceEnabled=false`. -+ -Default: `1`. - -spring.cloud.stream.instanceIndex:: -The instance index of the application: A number from `0` to `instanceCount - 1`. -Used for partitioning with RabbitMQ and with Kafka if `autoRebalanceEnabled=false`. -Automatically set in Cloud Foundry to match the application's instance index. - -spring.cloud.stream.dynamicDestinations:: -A list of destinations that can be bound dynamically (for example, in a dynamic routing scenario). -If set, only listed destinations can be bound. -+ -Default: empty (letting any destination be bound). - -spring.cloud.stream.defaultBinder:: -The default binder to use, if multiple binders are configured. -See <>. -+ -Default: empty. - -spring.cloud.stream.overrideCloudConnectors:: -This property is only applicable when the `cloud` profile is active and Spring Cloud Connectors are provided with the application. -If the property is `false` (the default), the binder detects a suitable bound service (for example, a RabbitMQ service bound in Cloud Foundry for the RabbitMQ binder) and uses it for creating connections (usually through Spring Cloud Connectors). -When set to `true`, this property instructs binders to completely ignore the bound services and rely on Spring Boot properties (for example, relying on the `spring.rabbitmq.*` properties provided in the environment for the RabbitMQ binder). -The typical usage of this property is to be nested in a customized environment <>. -+ -Default: `false`. - -spring.cloud.stream.bindingRetryInterval:: -The interval (in seconds) between retrying binding creation when, for example, the binder does not support late binding and the broker (for example, Apache Kafka) is down. -Set it to zero to treat such conditions as fatal, preventing the application from starting. -+ -Default: `30` - -[[binding-properties]] -== Binding Properties - -Binding properties are supplied by using the format of `spring.cloud.stream.bindings..=`. -The `` represents the name of the binding being configured. - -For example, for the following function -[source,java] ----- -@Bean -public Function uppercase() { - return v -> v.toUpperCase(); -} ----- -there are two bindings named `uppercase-in-0` for input and `uppercase-out-0` for output. See <> for more details. - -To avoid repetition, Spring Cloud Stream supports setting values for all bindings, in the format of `spring.cloud.stream.default.=` -and `spring.cloud.stream.default..=` for common binding properties. - -When it comes to avoiding repetitions for extended binding properties, this format should be used - `spring.cloud.stream..default..=`. - -[[common-binding-properties]] -=== Common Binding Properties - -These properties are exposed via `org.springframework.cloud.stream.config.BindingProperties` - -The following binding properties are available for both input and output bindings and must be prefixed with `spring.cloud.stream.bindings..` -(for example, `spring.cloud.stream.bindings.uppercase-in-0.destination=ticktock`). - -Default values can be set by using the `spring.cloud.stream.default` prefix (for example `spring.cloud.stream.default.contentType=application/json`). - -destination:: -The target destination of a binding on the bound middleware (for example, the RabbitMQ exchange or Kafka topic). -If binding represents a consumer binding (input), it could be bound to multiple destinations, and the destination names can be specified as comma-separated `String` values. -If not, the actual binding name is used instead. -The default value of this property cannot be overridden. -group:: -The consumer group of the binding. -Applies only to inbound bindings. -See <>. -+ -Default: `null` (indicating an anonymous consumer). -contentType:: -The content type of this binding. -See `<>`. -+ -Default: `application/json`. -binder:: -The binder used by this binding. -See `<>` for details. -+ -Default: `null` (the default binder is used, if it exists). - -[[consumer-properties]] -=== Consumer Properties - -These properties are exposed via `org.springframework.cloud.stream.binder.ConsumerProperties` - -The following binding properties are available for input bindings only and must be prefixed with `spring.cloud.stream.bindings..consumer.` (for example, `spring.cloud.stream.bindings.input.consumer.concurrency=3`). - -Default values can be set by using the `spring.cloud.stream.default.consumer` prefix (for example, `spring.cloud.stream.default.consumer.headerMode=none`). - -autoStartup:: -Signals if this consumer needs to be started automatically -+ -Default: `true`. -concurrency:: -The concurrency of the inbound consumer. -+ -Default: `1`. -partitioned:: -Whether the consumer receives data from a partitioned producer. -+ -Default: `false`. -headerMode:: -When set to `none`, disables header parsing on input. -Effective only for messaging middleware that does not support message headers natively and requires header embedding. -This option is useful when consuming data from non-Spring Cloud Stream applications when native headers are not supported. -When set to `headers`, it uses the middleware's native header mechanism. -When set to `embeddedHeaders`, it embeds headers into the message payload. -+ -Default: depends on the binder implementation. -maxAttempts:: -If processing fails, the number of attempts to process the message (including the first). -Set to `1` to disable retry. -+ -Default: `3`. -backOffInitialInterval:: -The backoff initial interval on retry. -+ -Default: `1000`. -backOffMaxInterval:: -The maximum backoff interval. -+ -Default: `10000`. -backOffMultiplier:: -The backoff multiplier. -+ -Default: `2.0`. -defaultRetryable:: -Whether exceptions thrown by the listener that are not listed in the `retryableExceptions` are retryable. -+ -Default: `true`. -instanceCount:: -When set to a value greater than equal to zero, it allows customizing the instance count of this consumer (if different from `spring.cloud.stream.instanceCount`). -When set to a negative value, it defaults to `spring.cloud.stream.instanceCount`. -See `<>` for more information. -+ -Default: `-1`. -instanceIndex:: -When set to a value greater than equal to zero, it allows customizing the instance index of this consumer (if different from `spring.cloud.stream.instanceIndex`). -When set to a negative value, it defaults to `spring.cloud.stream.instanceIndex`. -Ignored if `instanceIndexList` is provided. -See `<>` for more information. -+ -Default: `-1`. -instanceIndexList:: -Used with binders that do not support native partitioning (such as RabbitMQ); allows an application instance to consume from more than one partition. -+ -Default: empty. -retryableExceptions:: -A map of Throwable class names in the key and a boolean in the value. -Specify those exceptions (and subclasses) that will or won't be retried. -Also see `defaultRetriable`. -Example: `spring.cloud.stream.bindings.input.consumer.retryable-exceptions.java.lang.IllegalStateException=false`. -+ -Default: empty. -useNativeDecoding:: -When set to `true`, the inbound message is deserialized directly by the client library, which must be configured correspondingly (for example, setting an appropriate Kafka producer value deserializer). -When this configuration is being used, the inbound message unmarshalling is not based on the `contentType` of the binding. -When native decoding is used, it is the responsibility of the producer to use an appropriate encoder (for example, the Kafka producer value serializer) to serialize the outbound message. -Also, when native encoding and decoding is used, the `headerMode=embeddedHeaders` property is ignored and headers are not embedded in the message. -See the producer property `useNativeEncoding`. -+ -Default: `false`. -multiplex:: -When set to true, the underlying binder will natively multiplex destinations on the same input binding. -+ -Default: `false`. - -[[advanced-consumer-configuration]] -=== Advanced Consumer Configuration - -For advanced configuration of the underlying message listener container for message-driven consumers, add a single `ListenerContainerCustomizer` bean to the application context. -It will be invoked after the above properties have been applied and can be used to set additional properties. -Similarly, for polled consumers, add a `MessageSourceCustomizer` bean. - -The following is an example for the RabbitMQ binder: - -==== -[source, java] ----- -@Bean -public ListenerContainerCustomizer containerCustomizer() { - return (container, dest, group) -> container.setAdviceChain(advice1, advice2); -} - -@Bean -public MessageSourceCustomizer sourceCustomizer() { - return (source, dest, group) -> source.setPropertiesConverter(customPropertiesConverter); -} ----- -==== - -[[producer-properties]] -=== Producer Properties - -These properties are exposed via `org.springframework.cloud.stream.binder.ProducerProperties` - -The following binding properties are available for output bindings only and must be prefixed with `spring.cloud.stream.bindings..producer.` -(for example, `spring.cloud.stream.bindings.func-out-0.producer.partitionKeyExpression=headers.id`). - -Default values can be set by using the prefix `spring.cloud.stream.default.producer` (for example, `spring.cloud.stream.default.producer.partitionKeyExpression=headers.id`). - -autoStartup:: -Signals if this consumer needs to be started automatically -+ -Default: `true`. -partitionKeyExpression:: -A SpEL expression that determines how to partition outbound data. -If set, outbound data on this binding is partitioned. `partitionCount` must be set to a value greater than 1 to be effective. -See `<>`. -+ -Default: null. - -partitionKeyExtractorName:: -The name of the bean that implements `PartitionKeyExtractorStrategy`. Used to extract a key used to compute -the partition id (see 'partitionSelector*'). Mutually exclusive with 'partitionKeyExpression'. -+ -Default: null. - -partitionSelectorName:: -The name of the bean that implements `PartitionSelectorStrategy`. Used to determine partition id based -on partition key (see 'partitionKeyExtractor*'). Mutually exclusive with 'partitionSelectorExpression'. -+ -Default: null. - - -partitionSelectorExpression:: -A SpEL expression for customizing partition selection. -If neither is set, the partition is selected as the `hashCode(key) % partitionCount`, where `key` is computed through either `partitionKeyExpression`. -+ -Default: `null`. -partitionCount:: -The number of target partitions for the data, if partitioning is enabled. -Must be set to a value greater than 1 if the producer is partitioned. -On Kafka, it is interpreted as a hint. The larger of this and the partition count of the target topic is used instead. -+ -Default: `1`. -requiredGroups:: -A comma-separated list of groups to which the producer must ensure message delivery even if they start after it has been created (for example, by pre-creating durable queues in RabbitMQ). -headerMode:: -When set to `none`, it disables header embedding on output. -It is effective only for messaging middleware that does not support message headers natively and requires header embedding. -This option is useful when producing data for non-Spring Cloud Stream applications when native headers are not supported. -When set to `headers`, it uses the middleware's native header mechanism. -When set to `embeddedHeaders`, it embeds headers into the message payload. -+ -Default: Depends on the binder implementation. - -useNativeEncoding:: -When set to `true`, the outbound message is serialized directly by the client library, which must be configured correspondingly (for example, setting an appropriate Kafka producer value serializer). -When this configuration is being used, the outbound message marshalling is not based on the `contentType` of the binding. -When native encoding is used, it is the responsibility of the consumer to use an appropriate decoder (for example, the Kafka consumer value de-serializer) to deserialize the inbound message. -Also, when native encoding and decoding is used, the `headerMode=embeddedHeaders` property is ignored and headers are not embedded in the message. -See the consumer property `useNativeDecoding`. -+ -Default: `false`. - -errorChannelEnabled:: -When set to true, if the binder supports asynchroous send results, send failures are sent to an error channel for the destination. See Error Handling for more information. -+ -Default: false. - - -[[advanced-producer-configuration]] -=== Advanced Producer Configuration - -In some cases Producer Properties are not enough to properly configure a producing MessageHandler in the binder, or may be you prefer a programmatic approach -while configuring such producing MessageHandler. Regardless of the reason, spring-cloud-stream provides `ProducerMessageHandlerCustomizer` to accomplish it. - - -==== -[source, java] ----- -@FunctionalInterface -public interface ProducerMessageHandlerCustomizer { - - /** - * Configure a {@link MessageHandler} that is being created by the binder for the - * provided destination name. - * @param handler the {@link MessageHandler} from the binder. - * @param destinationName the bound destination name. - */ - void configure(H handler, String destinationName); - -} ----- - -As you can see it gives you access to an actual instance of producing `MessageHandler` which you can configure as you wish. -All you need to do is provide implementation of this strategy and configure it as a `@Bean`. - -==== - -[[content-type-management]] -= Content Type Negotiation - -Data transformation is one of the core features of any message-driven microservice architecture. Given that, in Spring Cloud Stream, such data -is represented as a Spring `Message`, a message may have to be transformed to a desired shape or size before reaching its destination. This is required for two reasons: - -. To convert the contents of the incoming message to match the signature of the application-provided handler. - -. To convert the contents of the outgoing message to the wire format. - -The wire format is typically `byte[]` (that is true for the Kafka and Rabbit binders), but it is governed by the binder implementation. - -In Spring Cloud Stream, message transformation is accomplished with an `org.springframework.messaging.converter.MessageConverter`. - -NOTE: As a supplement to the details to follow, you may also want to read the following https://spring.io/blog/2018/02/26/spring-cloud-stream-2-0-content-type-negotiation-and-transformation[blog post]. - -[[mechanics]] -== Mechanics - -To better understand the mechanics and the necessity behind content-type negotiation, we take a look at a very simple use case by using the following message handler as an example: - -[source, java] ----- - -public Function personFunction {..} ----- - -NOTE: For simplicity, we assume that this is the only handler function in the application (we assume there is no internal pipeline). - -The handler shown in the preceding example expects a `Person` object as an argument and produces a `String` type as an output. -In order for the framework to succeed in passing the incoming `Message` as an argument to this handler, it has to somehow transform the payload of the `Message` type from the wire format to a `Person` type. -In other words, the framework must locate and apply the appropriate `MessageConverter`. -To accomplish that, the framework needs some instructions from the user. -One of these instructions is already provided by the signature of the handler method itself (`Person` type). -Consequently, in theory, that should be (and, in some cases, is) enough. -However, for the majority of use cases, in order to select the appropriate `MessageConverter`, the framework needs an additional piece of information. -That missing piece is `contentType`. - -Spring Cloud Stream provides three mechanisms to define `contentType` (in order of precedence): - -. *HEADER*: The `contentType` can be communicated through the Message itself. By providing a `contentType` header, you declare the content type to use to locate and apply the appropriate `MessageConverter`. - -. *BINDING*: The `contentType` can be set per destination binding by setting the `spring.cloud.stream.bindings.input.content-type` property. -+ -NOTE: The `input` segment in the property name corresponds to the actual name of the destination (which is “input” in our case). This approach lets you declare, on a per-binding basis, the content type to use to locate and apply the appropriate `MessageConverter`. - -. *DEFAULT*: If `contentType` is not present in the `Message` header or the binding, the default `application/json` content type is used to -locate and apply the appropriate `MessageConverter`. - -As mentioned earlier, the preceding list also demonstrates the order of precedence in case of a tie. For example, a header-provided content type takes precedence over any other content type. -The same applies for a content type set on a per-binding basis, which essentially lets you override the default content type. -However, it also provides a sensible default (which was determined from community feedback). - -Another reason for making `application/json` the default stems from the interoperability requirements driven by distributed microservices architectures, where producer and consumer not only run in different JVMs but can also run on different non-JVM platforms. - -When the non-void handler method returns, if the return value is already a `Message`, that `Message` becomes the payload. However, when the return value is not a `Message`, the new `Message` is constructed with the return value as the payload while inheriting -headers from the input `Message` minus the headers defined or filtered by `SpringIntegrationProperties.messageHandlerNotPropagatedHeaders`. -By default, there is only one header set there: `contentType`. This means that the new `Message` does not have `contentType` header set, thus ensuring that the `contentType` can evolve. -You can always opt out of returning a `Message` from the handler method where you can inject any header you wish. - -If there is an internal pipeline, the `Message` is sent to the next handler by going through the same process of conversion. However, if there is no internal pipeline or you have reached the end of it, the `Message` is sent back to the output destination. - -[[content-type-versus-argument-type]] -=== Content Type versus Argument Type - -As mentioned earlier, for the framework to select the appropriate `MessageConverter`, it requires argument type and, optionally, content type information. -The logic for selecting the appropriate `MessageConverter` resides with the argument resolvers (`HandlerMethodArgumentResolvers`), which trigger right before the invocation of the user-defined handler method (which is when the actual argument type is known to the framework). -If the argument type does not match the type of the current payload, the framework delegates to the stack of the -pre-configured `MessageConverters` to see if any one of them can convert the payload. -As you can see, the `Object fromMessage(Message message, Class targetClass);` -operation of the MessageConverter takes `targetClass` as one of its arguments. -The framework also ensures that the provided `Message` always contains a `contentType` header. -When no contentType header was already present, it injects either the per-binding `contentType` header or the default `contentType` header. -The combination of `contentType` argument type is the mechanism by which framework determines if message can be converted to a target type. -If no appropriate `MessageConverter` is found, an exception is thrown, which you can handle by adding a custom `MessageConverter` (see `<>`). - -But what if the payload type matches the target type declared by the handler method? In this case, there is nothing to convert, and the -payload is passed unmodified. While this sounds pretty straightforward and logical, keep in mind handler methods that take a `Message` or `Object` as an argument. -By declaring the target type to be `Object` (which is an `instanceof` everything in Java), you essentially forfeit the conversion process. - -NOTE: Do not expect `Message` to be converted into some other type based only on the `contentType`. -Remember that the `contentType` is complementary to the target type. -If you wish, you can provide a hint, which `MessageConverter` may or may not take into consideration. - -[[message-converters]] -=== Message Converters - -`MessageConverters` define two methods: - -[source, java] ----- -Object fromMessage(Message message, Class targetClass); - -Message toMessage(Object payload, @Nullable MessageHeaders headers); ----- - -It is important to understand the contract of these methods and their usage, specifically in the context of Spring Cloud Stream. - -The `fromMessage` method converts an incoming `Message` to an argument type. -The payload of the `Message` could be any type, and it is -up to the actual implementation of the `MessageConverter` to support multiple types. -For example, some JSON converter may support the payload type as `byte[]`, `String`, and others. -This is important when the application contains an internal pipeline (that is, input -> handler1 -> handler2 ->. . . -> output) and the output of the upstream handler results in a `Message` which may not be in the initial wire format. - -However, the `toMessage` method has a more strict contract and must always convert `Message` to the wire format: `byte[]`. - -So, for all intents and purposes (and especially when implementing your own converter) you regard the two methods as having the following signatures: - -[source, java] ----- -Object fromMessage(Message message, Class targetClass); - -Message toMessage(Object payload, @Nullable MessageHeaders headers); ----- - -[[provided-messageconverters]] -== Provided MessageConverters - -As mentioned earlier, the framework already provides a stack of `MessageConverters` to handle most common use cases. -The following list describes the provided `MessageConverters`, in order of precedence (the first `MessageConverter` that works is used): - -. `JsonMessageConverter`: As the name suggests it supports conversion of the payload of the `Message` to/from POJO for cases when `contentType` is `application/json` (DEFAULT). -. `ByteArrayMessageConverter`: Supports conversion of the payload of the `Message` from `byte[]` to `byte[]` for cases when `contentType` is `application/octet-stream`. It is essentially a pass through and exists primarily for backward compatibility. -. `ObjectStringMessageConverter`: Supports conversion of any type to a `String` when `contentType` is `text/plain`. -It invokes Object’s `toString()` method or, if the payload is `byte[]`, a new `String(byte[])`. - -When no appropriate converter is found, the framework throws an exception. When that happens, you should check your code and configuration and ensure you did not miss anything (that is, ensure that you provided a `contentType` by using a binding or a header). -However, most likely, you found some uncommon case (such as a custom `contentType` perhaps) and the current stack of provided `MessageConverters` -does not know how to convert. If that is the case, you can add custom `MessageConverter`. See <>. - -[[spring-cloud-stream-overview-user-defined-message-converters]] -== User-defined Message Converters - -Spring Cloud Stream exposes a mechanism to define and register additional `MessageConverter` s. -To use it, implement `org.springframework.messaging.converter.MessageConverter`, configure it as a `@Bean`. -It is then appended to the existing stack of `MessageConverter` s. - -NOTE: It is important to understand that custom `MessageConverter` implementations are added to the head of the existing stack. -Consequently, custom `MessageConverter` implementations take precedence over the existing ones, which lets you override as well as add to the existing converters. - -The following example shows how to create a message converter bean to support a new content type called `application/bar`: - -[source,java] ----- -@SpringBootApplication -public static class SinkApplication { - - ... - - @Bean - public MessageConverter customMessageConverter() { - return new MyCustomMessageConverter(); - } -} - -public class MyCustomMessageConverter extends AbstractMessageConverter { - - public MyCustomMessageConverter() { - super(new MimeType("application", "bar")); - } - - @Override - protected boolean supports(Class clazz) { - return (Bar.class.equals(clazz)); - } - - @Override - protected Object convertFromInternal(Message message, Class targetClass, Object conversionHint) { - Object payload = message.getPayload(); - return (payload instanceof Bar ? payload : new Bar((byte[]) payload)); - } -} ----- - -[[inter-application-communication]] -= Inter-Application Communication - -Spring Cloud Stream enables communication between applications. Inter-application communication is a complex issue spanning several concerns, as described in the following topics: - -* `<>` -* `<>` -* `<>` - -[[spring-cloud-stream-overview-connecting-multiple-application-instances]] -== Connecting Multiple Application Instances - -While Spring Cloud Stream makes it easy for individual Spring Boot applications to connect to messaging systems, the typical scenario for Spring Cloud Stream is the creation of multi-application pipelines, where microservice applications send data to each other. -You can achieve this scenario by correlating the input and output destinations of "`adjacent`" applications. - -Suppose a design calls for the Time Source application to send data to the Log Sink application. You could use a common destination named `ticktock` for bindings within both applications. - -Time Source (that has the binding named `output`) would set the following property: - ----- -spring.cloud.stream.bindings.output.destination=ticktock ----- - -Log Sink (that has the binding named `input`) would set the following property: - ----- -spring.cloud.stream.bindings.input.destination=ticktock ----- - -[[spring-cloud-stream-overview-instance-index-instance-count]] -== Instance Index and Instance Count - -When scaling up Spring Cloud Stream applications, each instance can receive information about how many other instances of the same application exist and what its own instance index is. -Spring Cloud Stream does this through the `spring.cloud.stream.instanceCount` and `spring.cloud.stream.instanceIndex` properties. -For example, if there are three instances of a HDFS sink application, all three instances have `spring.cloud.stream.instanceCount` set to `3`, and the individual applications have `spring.cloud.stream.instanceIndex` set to `0`, `1`, and `2`, respectively. - -When Spring Cloud Stream applications are deployed through Spring Cloud Data Flow, these properties are configured automatically; when Spring Cloud Stream applications are launched independently, these properties must be set correctly. -By default, `spring.cloud.stream.instanceCount` is `1`, and `spring.cloud.stream.instanceIndex` is `0`. - -In a scaled-up scenario, correct configuration of these two properties is important for addressing partitioning behavior (see below) in general, and the two properties are always required by certain binders (for example, the Kafka binder) in order to ensure that data are split correctly across multiple consumer instances. - -[[spring-cloud-stream-overview-partitioning]] -== Partitioning - -Partitioning in Spring Cloud Stream consists of two tasks: - -* `<>` -* `<>` - -[[spring-cloud-stream-overview-configuring-output-bindings-partitioning]] -=== Configuring Output Bindings for Partitioning - -You can configure an output binding to send partitioned data by setting one and only one of its `partitionKeyExpression` or `partitionKeyExtractorName` properties, as well as its `partitionCount` property. - -For example, the following is a valid and typical configuration: - ----- -spring.cloud.stream.bindings.func-out-0.producer.partitionKeyExpression=headers.id -spring.cloud.stream.bindings.func-out-0.producer.partitionCount=5 ----- - -Based on that example configuration, data is sent to the target partition by using the following logic. - -A partition key's value is calculated for each message sent to a partitioned output binding based on the `partitionKeyExpression`. -The `partitionKeyExpression` is a SpEL expression that is evaluated against the outbound message (in the preceding example it's the value of the `id` from message headers) for extracting the partitioning key. - -If a SpEL expression is not sufficient for your needs, you can instead calculate the partition key value by providing an implementation of `org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy` and configuring it as a bean (by using the `@Bean` annotation). -If you have more then one bean of type `org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy` available in the Application Context, you can further filter it by specifying its name with the `partitionKeyExtractorName` property, as shown in the following example: - -[source] ----- ---spring.cloud.stream.bindings.func-out-0.producer.partitionKeyExtractorName=customPartitionKeyExtractor ---spring.cloud.stream.bindings.func-out-0.producer.partitionCount=5 -. . . -@Bean -public CustomPartitionKeyExtractorClass customPartitionKeyExtractor() { - return new CustomPartitionKeyExtractorClass(); -} ----- - -NOTE: In previous versions of Spring Cloud Stream, you could specify the implementation of `org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy` by setting the `spring.cloud.stream.bindings.output.producer.partitionKeyExtractorClass` property. -Since version 3.0, this property is removed. - -Once the message key is calculated, the partition selection process determines the target partition as a value between `0` and `partitionCount - 1`. -The default calculation, applicable in most scenarios, is based on the following formula: `key.hashCode() % partitionCount`. -This can be customized on the binding, either by setting a SpEL expression to be evaluated against the 'key' (through the `partitionSelectorExpression` property) or by configuring an implementation of `org.springframework.cloud.stream.binder.PartitionSelectorStrategy` as a bean (by using the @Bean annotation). -Similar to the `PartitionKeyExtractorStrategy`, you can further filter it by using the `spring.cloud.stream.bindings.output.producer.partitionSelectorName` property when more than one bean of this type is available in the Application Context, as shown in the following example: - -[source] ----- ---spring.cloud.stream.bindings.func-out-0.producer.partitionSelectorName=customPartitionSelector -. . . -@Bean -public CustomPartitionSelectorClass customPartitionSelector() { - return new CustomPartitionSelectorClass(); -} ----- - -NOTE: In previous versions of Spring Cloud Stream you could specify the implementation of `org.springframework.cloud.stream.binder.PartitionSelectorStrategy` by setting the `spring.cloud.stream.bindings.output.producer.partitionSelectorClass` property. -Since version 3.0, this property is removed. - -[[spring-cloud-stream-overview-configuring-input-bindings-partitioning]] -=== Configuring Input Bindings for Partitioning - -An input binding (with the binding name `uppercase-in-0`) is configured to receive partitioned data by setting its `partitioned` -property, as well as the `instanceIndex` and `instanceCount` properties on the application itself, as shown in the following example: - ----- -spring.cloud.stream.bindings.uppercase-in-0.consumer.partitioned=true -spring.cloud.stream.instanceIndex=3 -spring.cloud.stream.instanceCount=5 ----- - -The `instanceCount` value represents the total number of application instances between which the data should be partitioned. -The `instanceIndex` must be a unique value across the multiple instances, with a value between `0` and `instanceCount - 1`. -The instance index helps each application instance to identify the unique partition(s) from which it receives data. -It is required by binders using technology that does not support partitioning natively. -For example, with RabbitMQ, there is a queue for each partition, with the queue name containing the instance index. -With Kafka, if `autoRebalanceEnabled` is `true` (default), Kafka takes care of distributing partitions across instances, and these properties are not required. -If `autoRebalanceEnabled` is set to false, the `instanceCount` and `instanceIndex` are used by the binder to determine which partition(s) the instance subscribes to (you must have at least as many partitions as there are instances). -The binder allocates the partitions instead of Kafka. -This might be useful if you want messages for a particular partition to always go to the same instance. -When a binder configuration requires them, it is important to set both values correctly in order to ensure that all of the data is consumed and that the application instances receive mutually exclusive datasets. - -While a scenario in which using multiple instances for partitioned data processing may be complex to set up in a standalone case, Spring Cloud Dataflow can simplify the process significantly by populating both the input and output values correctly and by letting you rely on the runtime infrastructure to provide information about the instance index and instance count. - -[[testing]] -= Testing - -Spring Cloud Stream provides support for testing your microservice applications without connecting to a messaging system. - - -[[spring_integration_test_binder]] -== Spring Integration Test Binder - -Spring Cloud Stream comes with a test binder which you can use for testing the various application components without requiring an actual real-world binder implementation or a message broker. - -This test binder acts as a bridge between _unit_ and _integration_ testing and is based on https://spring.io/projects/spring-integration[Spring Integration] framework as an in-JVM message broker essentially giving you the best of both worlds - a real binder without the networking. - -[[test-binder-configuration]] -=== Test Binder configuration -To enable Spring Integration test binder, all you need is to add it as a dependency. - -***Add required dependencies*** - -Below is the example of the required Maven POM entries. - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-test-binder - test - ----- - -Or for build.gradle.kts - -[source,kotlin] ----- -testImplementation("org.springframework.cloud:spring-cloud-stream-test-binder") ----- - -[[test-binder-usage]] -=== Test Binder usage - -Now you can test your microservice as a simple unit test - -[source,java] ----- -@SpringBootTest -public class SampleStreamTests { - - @Autowired - private InputDestination input; - - @Autowired - private OutputDestination output; - - @Test - public void testEmptyConfiguration() { - this.input.send(new GenericMessage("hello".getBytes())); - assertThat(output.receive().getPayload()).isEqualTo("HELLO".getBytes()); - } - - @SpringBootApplication - @Import(TestChannelBinderConfiguration.class) - public static class SampleConfiguration { - @Bean - public Function uppercase() { - return v -> v.toUpperCase(); - } - } -} ----- - -And if you need more control or want to test several configurations in the same test suite -you can also do the following: - -[source,java] ----- -@EnableAutoConfiguration -public static class MyTestConfiguration { - @Bean - public Function uppercase() { - return v -> v.toUpperCase(); - } -} - -. . . - -@Test -public void sampleTest() { - try (ConfigurableApplicationContext context = new SpringApplicationBuilder( - TestChannelBinderConfiguration.getCompleteConfiguration( - MyTestConfiguration.class)) - .run("--spring.cloud.function.definition=uppercase")) { - InputDestination source = context.getBean(InputDestination.class); - OutputDestination target = context.getBean(OutputDestination.class); - source.send(new GenericMessage("hello".getBytes())); - assertThat(target.receive().getPayload()).isEqualTo("HELLO".getBytes()); - } -} ----- - -For cases where you have multiple bindings and/or multiple inputs and outputs, or simply want to be explicit about names of -the destination you are sending to or receiving from, the `send()` and `receive()` -methods of `InputDestination` and `OutputDestination` are overridden to allow you to provide the name of the input and output destination. - -Consider the following sample: -[source,java] ----- -@EnableAutoConfiguration -public static class SampleFunctionConfiguration { - - @Bean - public Function uppercase() { - return value -> value.toUpperCase(); - } - - @Bean - public Function reverse() { - return value -> new StringBuilder(value).reverse().toString(); - } -} ----- - -and the actual test - -[source,java] ----- -@Test -public void testMultipleFunctions() { - try (ConfigurableApplicationContext context = new SpringApplicationBuilder( - TestChannelBinderConfiguration.getCompleteConfiguration( - SampleFunctionConfiguration.class)) - .run("--spring.cloud.function.definition=uppercase;reverse")) { - - InputDestination inputDestination = context.getBean(InputDestination.class); - OutputDestination outputDestination = context.getBean(OutputDestination.class); - - Message inputMessage = MessageBuilder.withPayload("Hello".getBytes()).build(); - inputDestination.send(inputMessage, "uppercase-in-0"); - inputDestination.send(inputMessage, "reverse-in-0"); - - Message outputMessage = outputDestination.receive(0, "uppercase-out-0"); - assertThat(outputMessage.getPayload()).isEqualTo("HELLO".getBytes()); - - outputMessage = outputDestination.receive(0, "reverse-out-0"); - assertThat(outputMessage.getPayload()).isEqualTo("olleH".getBytes()); - } -} ----- - -For cases where you have additional mapping properties such as `destination` you should use those names. For example, consider a different version of the -preceding test where we explicitly map inputs and outputs of the `uppercase` function to `myInput` and `myOutput` binding names: -[source,java] ----- -@Test -public void testMultipleFunctions() { - try (ConfigurableApplicationContext context = new SpringApplicationBuilder( - TestChannelBinderConfiguration.getCompleteConfiguration( - SampleFunctionConfiguration.class)) - .run( - "--spring.cloud.function.definition=uppercase;reverse", - "--spring.cloud.stream.bindings.uppercase-in-0.destination=myInput", - "--spring.cloud.stream.bindings.uppercase-out-0.destination=myOutput" - )) { - - InputDestination inputDestination = context.getBean(InputDestination.class); - OutputDestination outputDestination = context.getBean(OutputDestination.class); - - Message inputMessage = MessageBuilder.withPayload("Hello".getBytes()).build(); - inputDestination.send(inputMessage, "myInput"); - inputDestination.send(inputMessage, "reverse-in-0"); - - Message outputMessage = outputDestination.receive(0, "myOutput"); - assertThat(outputMessage.getPayload()).isEqualTo("HELLO".getBytes()); - - outputMessage = outputDestination.receive(0, "reverse-out-0"); - assertThat(outputMessage.getPayload()).isEqualTo("olleH".getBytes()); - } -} ----- - - -[[test-binder-and-pollablemessagesource]] -=== Test Binder and PollableMessageSource -Spring Integration Test Binder also allows you to write tests when working with `PollableMessageSource` (see <> for more details). - -The important thing that needs to be understood though is that polling is not event-driven, and that `PollableMessageSource` is a strategy which exposes operation to produce (poll for) a Message (singular). -How often you poll or how many threads you use or where you're polling from (message queue or file system) is entirely up to you; -In other words it is your responsibility to configure Poller or Threads or the actual source of Message. Luckily Spring has plenty of abstractions to configure exactly that. - -Let's look at the example: - -[source, java] ----- -@Test -public void samplePollingTest() { - ApplicationContext context = new SpringApplicationBuilder(SamplePolledConfiguration.class) - .web(WebApplicationType.NONE) - .run("--spring.jmx.enabled=false", "--spring.cloud.stream.pollable-source=myDestination"); - OutputDestination destination = context.getBean(OutputDestination.class); - System.out.println("Message 1: " + new String(destination.receive().getPayload())); - System.out.println("Message 2: " + new String(destination.receive().getPayload())); - System.out.println("Message 3: " + new String(destination.receive().getPayload())); -} - -@Import(TestChannelBinderConfiguration.class) -@EnableAutoConfiguration -public static class SamplePolledConfiguration { - @Bean - public ApplicationRunner poller(PollableMessageSource polledMessageSource, StreamBridge output, TaskExecutor taskScheduler) { - return args -> { - taskScheduler.execute(() -> { - for (int i = 0; i < 3; i++) { - try { - if (!polledMessageSource.poll(m -> { - String newPayload = ((String) m.getPayload()).toUpperCase(); - output.send("myOutput", newPayload); - })) { - Thread.sleep(2000); - } - } - catch (Exception e) { - // handle failure - } - } - }); - }; - } -} ----- - -The above (very rudimentary) example will produce 3 messages in 2 second intervals sending them to the output destination of `Source` -which this binder sends to `OutputDestination` where we retrieve them (for any assertions). -Currently, it prints the following: -[source, text] ----- -Message 1: POLLED DATA -Message 2: POLLED DATA -Message 3: POLLED DATA ----- -As you can see the data is the same. That is because this binder defines a default implementation of the actual `MessageSource` - the source -from which the Messages are polled using `poll()` operation. While sufficient for most testing scenarios, there are cases where you may want -to define your own `MessageSource`. To do so simply configure a bean of type `MessageSource` in your test configuration providing your own -implementation of Message sourcing. - -Here is the example: - -[source, java] ----- -@Bean -public MessageSource source() { - return () -> new GenericMessage<>("My Own Data " + UUID.randomUUID()); -} ----- -rendering the following output; -[source, text] ----- -Message 1: MY OWN DATA 1C180A91-E79F-494F-ABF4-BA3F993710DA -Message 2: MY OWN DATA D8F3A477-5547-41B4-9434-E69DA7616FEE -Message 3: MY OWN DATA 20BF2E64-7FF4-4CB6-A823-4053D30B5C74 ----- - -NOTE: DO NOT name this bean `messageSource` as it is going to be in conflict with the bean of the same name (different type) -provided by Spring Boot for unrelated reasons. - -[[special-note-on-mixing-test-binder-and-regular-middleware-binder-for-testing]] -=== Special Note on Mixing Test Binder and Regular Middleware Binder for Testing - -The Spring Integration based test binder is provided for testing the application without involving an actual middleware based binder such as the Kafka or RabbitMQ binder. -As described in the sections above, the test binder helps you to verify the application behavior quickly by relying on the in-memory Spring Integration channels. -When the test binder is present on the test classpath, Spring Cloud Stream will try to use this binder for all testing purposes wherever it needs a binder for communication. -In other words, you cannot mix both the test binder and a regular middleware binder for testing purposes in the same module. -After testing the application with the test binder, if you want to continue doing further integration tests using the actual middleware binder, it is recommended to add those tests that use the actual binder in a separate module so that those tests can make the proper connection to the actual middleware rather than relying on the in-memory channels provided by the test binder. - -[[health-indicator]] -= Health Indicator - -Spring Cloud Stream provides a health indicator for binders. -It is registered under the name `binders` and can be enabled or disabled by setting the `management.health.binders.enabled` property. - -To enable health check you first need to enable both "web" and "actuator" by including its dependencies (see <>) - -If `management.health.binders.enabled` is not set explicitly by the application, then `management.health.defaults.enabled` is matched as `true` and the binder health indicators are enabled. -If you want to disable health indicator completely, then you have to set `management.health.binders.enabled` to `false`. - -You can use Spring Boot actuator health endpoint to access the health indicator - `/actuator/health`. -By default, you will only receive the top level application status when you hit the above endpoint. -In order to receive the full details from the binder specific health indicators, you need to include the property `management.endpoint.health.show-details` with the value `ALWAYS` in your application. - -Health indicators are binder-specific and certain binder implementations may not necessarily provide a health indicator. - -If you want to completely disable all health indicators available out of the box and instead provide your own health indicators, -you can do so by setting property `management.health.binders.enabled` to `false` and then provide your own `HealthIndicator` beans in your application. -In this case, the health indicator infrastructure from Spring Boot will still pick up these custom beans. -Even if you are not disabling the binder health indicators, you can still enhance the health checks by providing your own `HealthIndicator` beans in addition to the out of the box health checks. - -When you have multiple binders in the same application, health indicators are enabled by default unless the application turns them off by setting `management.health.binders.enabled` to `false`. -In this case, if the user wants to disable health check for a subset of the binders, then that should be done by setting `management.health.binders.enabled` to `false` in the multi binder configurations's environment. -See <> for details on how environment specific properties can be provided. - -If there are multiple binders present in the classpath but not all of them are used in the application, this may cause some issues in the context of health indicators. -There may be implementation specific details as to how the health checks are performed. For example, a Kafka binder may decide the status as `DOWN` if there are no destinations registered by the binder. - -Lets take a concrete situation. Imagine you have both Kafka and Kafka Streams binders present in the classpath, but only use the Kafka Streams binder in the application code, i.e. only provide bindings using the Kafka Streams binder. -Since Kafka binder is not used and it has specific checks to see if any destinations are registered, the binder health check will fail. -The top level application health check status will be reported as `DOWN`. -In this situation, you can simply remove the dependency for kafka binder from your application since you are not using it. - -[[samples]] -= Samples - -For Spring Cloud Stream samples, see the https://github.com/spring-cloud/spring-cloud-stream-samples[spring-cloud-stream-samples] repository on GitHub. - -[[deploying-stream-applications-on-cloudfoundry]] -== Deploying Stream Applications on CloudFoundry - -On CloudFoundry, services are usually exposed through a special environment variable called https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES[VCAP_SERVICES]. - -When configuring your binder connections, you can use the values from an environment variable as explained on the http://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current-SNAPSHOT/reference/htmlsingle/#getting-started-ups[dataflow Cloud Foundry Server] docs. - -[[binder-implementations]] -= Binder Implementations - -The following is the list of available binder implementations - -* https://cloud.spring.io/spring-cloud-stream-binder-rabbit/[RabbitMQ] -* https://cloud.spring.io/spring-cloud-stream-binder-kafka/[Apache Kafka] -* https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis[Amazon Kinesis] -* https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-pubsub-stream-binder[Google PubSub _(partner maintained)_] -* https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter#spring-cloud-stream-binder-for-solace-pubsub[Solace PubSub+ _(partner maintained)_] -* https://aka.ms/spring/docs#spring-cloud-stream-binder-for-azure-event-hubs[Azure Event Hubs _(partner maintained)_] -* https://aka.ms/spring/docs#spring-cloud-stream-binder-for-azure-service-bus[Azure Service Bus _(partner maintained)_] -* https://github.com/alibaba/spring-cloud-alibaba/wiki/RocketMQ-en[Apache RocketMQ _(partner maintained)_] - -As it was mentioned earlier Binder abstraction is also one of the extension points of the framework. So if you can't find a suitable binder in the preceding list you can implement your own binder on top of Spring Cloud Stream. -In the https://medium.com/@domenicosibilio/how-to-create-a-spring-cloud-stream-binder-from-scratch-ab8b29ee931b[How to create a Spring Cloud Stream Binder from scratch] post a community member documents -in details, with an example, a set of steps necessary to implement a custom binder. -The steps are also highlighted in the `<>` section. diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/binder-configuration-properties.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/binder-configuration-properties.adoc new file mode 100644 index 000000000..6d547af98 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/binder-configuration-properties.adoc @@ -0,0 +1,28 @@ +[[binder-configuration-properties]] += Binder Configuration Properties + +The following properties are available when customizing binder configurations. These properties exposed via `org.springframework.cloud.stream.config.BinderProperties` + +They must be prefixed with `spring.cloud.stream.binders.`. + +type:: +The binder type. +It typically references one of the binders found on the classpath -- in particular, a key in a `META-INF/spring.binders` file. ++ +By default, it has the same value as the configuration name. +inheritEnvironment:: +Whether the configuration inherits the environment of the application itself. ++ +Default: `true`. +environment:: +Root for a set of properties that can be used to customize the environment of the binder. +When this property is set, the context in which the binder is being created is not a child of the application context. +This setting allows for complete separation between the binder components and the application components. ++ +Default: `empty`. +defaultCandidate:: +Whether the binder configuration is a candidate for being considered a default binder or can be used only when explicitly referenced. +This setting allows adding binder configurations without interfering with the default processing. ++ +Default: `true`. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/binder-customizer.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/binder-customizer.adoc new file mode 100644 index 000000000..e641313c7 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/binder-customizer.adoc @@ -0,0 +1,32 @@ +[[binder-customizer]] += Customizing binders in multi binder applications + +When an application has multiple binders in it and wants to customize the binders, then that can be achieved by providing a `BinderCustomizer` implementation. +In the case of applications with a single binder, this special customizer is not necessary since the binder context can access the customization beans directly. +However, this is not the case in a multi-binder scenario, since various binders live in different application contexts. +By providing an implementation of `BinderCustomizer` interface, the binders, although reside in different application contexts, will receive the customization. +Spring Cloud Stream ensures that the customizations take place before the applications start using the binders. +The user must check for the binder type and then apply the necessary customizations. + +Here is an example of providing a `BinderCustomizer` bean. + +[source, java] +---- +@Bean +public BinderCustomizer binderCustomizer() { + return (binder, binderName) -> { + if (binder instanceof KafkaMessageChannelBinder kafkaMessageChannelBinder) { + kafkaMessageChannelBinder.setRebalanceListener(...); + } + else if (binder instanceof KStreamBinder) { + ... + } + else if (binder instanceof RabbitMessageChannelBinder) { + ... + } + }; +} +---- + +Note that, when there are more than one instance of the same type of the binder, the binder name can be used to filter customization. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/binder-detection.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/binder-detection.adoc new file mode 100644 index 000000000..0eab20bcd --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/binder-detection.adoc @@ -0,0 +1,23 @@ +[[binder-detection]] += Binder Detection + +Spring Cloud Stream relies on implementations of the Binder SPI to perform the task of connecting (binding) user code to message brokers. +Each Binder implementation typically connects to one type of messaging system. + +[[classpath-detection]] +== Classpath Detection + +By default, Spring Cloud Stream relies on Spring Boot's auto-configuration to configure the binding process. +If a single Binder implementation is found on the classpath, Spring Cloud Stream automatically uses it. +For example, a Spring Cloud Stream project that aims to bind only to RabbitMQ can add the following dependency: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + +---- + +For the specific Maven coordinates of other binder dependencies, see the documentation of that binder implementation. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/binding-properties.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/binding-properties.adoc new file mode 100644 index 000000000..05b1d4e2c --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/binding-properties.adoc @@ -0,0 +1,280 @@ +[[binding-properties]] += Binding Properties + +Binding properties are supplied by using the format of `spring.cloud.stream.bindings..=`. +The `` represents the name of the binding being configured. + +For example, for the following function +[source,java] +---- +@Bean +public Function uppercase() { + return v -> v.toUpperCase(); +} +---- +there are two bindings named `uppercase-in-0` for input and `uppercase-out-0` for output. See <> for more details. + +To avoid repetition, Spring Cloud Stream supports setting values for all bindings, in the format of `spring.cloud.stream.default.=` +and `spring.cloud.stream.default..=` for common binding properties. + +When it comes to avoiding repetitions for extended binding properties, this format should be used - `spring.cloud.stream..default..=`. + +[[common-binding-properties]] +== Common Binding Properties + +These properties are exposed via `org.springframework.cloud.stream.config.BindingProperties` + +The following binding properties are available for both input and output bindings and must be prefixed with `spring.cloud.stream.bindings..` +(for example, `spring.cloud.stream.bindings.uppercase-in-0.destination=ticktock`). + +Default values can be set by using the `spring.cloud.stream.default` prefix (for example `spring.cloud.stream.default.contentType=application/json`). + +destination:: +The target destination of a binding on the bound middleware (for example, the RabbitMQ exchange or Kafka topic). +If binding represents a consumer binding (input), it could be bound to multiple destinations, and the destination names can be specified as comma-separated `String` values. +If not, the actual binding name is used instead. +The default value of this property cannot be overridden. +group:: +The consumer group of the binding. +Applies only to inbound bindings. +See <>. ++ +Default: `null` (indicating an anonymous consumer). +contentType:: +The content type of this binding. +See `<>`. ++ +Default: `application/json`. +binder:: +The binder used by this binding. +See `<>` for details. ++ +Default: `null` (the default binder is used, if it exists). + +[[consumer-properties]] +== Consumer Properties + +These properties are exposed via `org.springframework.cloud.stream.binder.ConsumerProperties` + +The following binding properties are available for input bindings only and must be prefixed with `spring.cloud.stream.bindings..consumer.` (for example, `spring.cloud.stream.bindings.input.consumer.concurrency=3`). + +Default values can be set by using the `spring.cloud.stream.default.consumer` prefix (for example, `spring.cloud.stream.default.consumer.headerMode=none`). + +autoStartup:: +Signals if this consumer needs to be started automatically ++ +Default: `true`. +concurrency:: +The concurrency of the inbound consumer. ++ +Default: `1`. +partitioned:: +Whether the consumer receives data from a partitioned producer. ++ +Default: `false`. +headerMode:: +When set to `none`, disables header parsing on input. +Effective only for messaging middleware that does not support message headers natively and requires header embedding. +This option is useful when consuming data from non-Spring Cloud Stream applications when native headers are not supported. +When set to `headers`, it uses the middleware's native header mechanism. +When set to `embeddedHeaders`, it embeds headers into the message payload. ++ +Default: depends on the binder implementation. +maxAttempts:: +If processing fails, the number of attempts to process the message (including the first). +Set to `1` to disable retry. ++ +Default: `3`. +backOffInitialInterval:: +The backoff initial interval on retry. ++ +Default: `1000`. +backOffMaxInterval:: +The maximum backoff interval. ++ +Default: `10000`. +backOffMultiplier:: +The backoff multiplier. ++ +Default: `2.0`. +defaultRetryable:: +Whether exceptions thrown by the listener that are not listed in the `retryableExceptions` are retryable. ++ +Default: `true`. +instanceCount:: +When set to a value greater than equal to zero, it allows customizing the instance count of this consumer (if different from `spring.cloud.stream.instanceCount`). +When set to a negative value, it defaults to `spring.cloud.stream.instanceCount`. +See `<>` for more information. ++ +Default: `-1`. +instanceIndex:: +When set to a value greater than equal to zero, it allows customizing the instance index of this consumer (if different from `spring.cloud.stream.instanceIndex`). +When set to a negative value, it defaults to `spring.cloud.stream.instanceIndex`. +Ignored if `instanceIndexList` is provided. +See `<>` for more information. ++ +Default: `-1`. +instanceIndexList:: +Used with binders that do not support native partitioning (such as RabbitMQ); allows an application instance to consume from more than one partition. ++ +Default: empty. +retryableExceptions:: +A map of Throwable class names in the key and a boolean in the value. +Specify those exceptions (and subclasses) that will or won't be retried. +Also see `defaultRetriable`. +Example: `spring.cloud.stream.bindings.input.consumer.retryable-exceptions.java.lang.IllegalStateException=false`. ++ +Default: empty. +useNativeDecoding:: +When set to `true`, the inbound message is deserialized directly by the client library, which must be configured correspondingly (for example, setting an appropriate Kafka producer value deserializer). +When this configuration is being used, the inbound message unmarshalling is not based on the `contentType` of the binding. +When native decoding is used, it is the responsibility of the producer to use an appropriate encoder (for example, the Kafka producer value serializer) to serialize the outbound message. +Also, when native encoding and decoding is used, the `headerMode=embeddedHeaders` property is ignored and headers are not embedded in the message. +See the producer property `useNativeEncoding`. ++ +Default: `false`. +multiplex:: +When set to true, the underlying binder will natively multiplex destinations on the same input binding. ++ +Default: `false`. + +[[advanced-consumer-configuration]] +== Advanced Consumer Configuration + +For advanced configuration of the underlying message listener container for message-driven consumers, add a single `ListenerContainerCustomizer` bean to the application context. +It will be invoked after the above properties have been applied and can be used to set additional properties. +Similarly, for polled consumers, add a `MessageSourceCustomizer` bean. + +The following is an example for the RabbitMQ binder: + +==== +[source, java] +---- +@Bean +public ListenerContainerCustomizer containerCustomizer() { + return (container, dest, group) -> container.setAdviceChain(advice1, advice2); +} + +@Bean +public MessageSourceCustomizer sourceCustomizer() { + return (source, dest, group) -> source.setPropertiesConverter(customPropertiesConverter); +} +---- +==== + +[[producer-properties]] +== Producer Properties + +These properties are exposed via `org.springframework.cloud.stream.binder.ProducerProperties` + +The following binding properties are available for output bindings only and must be prefixed with `spring.cloud.stream.bindings..producer.` +(for example, `spring.cloud.stream.bindings.func-out-0.producer.partitionKeyExpression=headers.id`). + +Default values can be set by using the prefix `spring.cloud.stream.default.producer` (for example, `spring.cloud.stream.default.producer.partitionKeyExpression=headers.id`). + +autoStartup:: +Signals if this consumer needs to be started automatically ++ +Default: `true`. +partitionKeyExpression:: +A SpEL expression that determines how to partition outbound data. +If set, outbound data on this binding is partitioned. `partitionCount` must be set to a value greater than 1 to be effective. +See `<>`. ++ +Default: null. + +partitionKeyExtractorName:: +The name of the bean that implements `PartitionKeyExtractorStrategy`. Used to extract a key used to compute +the partition id (see 'partitionSelector*'). Mutually exclusive with 'partitionKeyExpression'. ++ +Default: null. + +partitionSelectorName:: +The name of the bean that implements `PartitionSelectorStrategy`. Used to determine partition id based +on partition key (see 'partitionKeyExtractor*'). Mutually exclusive with 'partitionSelectorExpression'. ++ +Default: null. + + +partitionSelectorExpression:: +A SpEL expression for customizing partition selection. +If neither is set, the partition is selected as the `hashCode(key) % partitionCount`, where `key` is computed through either `partitionKeyExpression`. ++ +Default: `null`. +partitionCount:: +The number of target partitions for the data, if partitioning is enabled. +Must be set to a value greater than 1 if the producer is partitioned. +On Kafka, it is interpreted as a hint. The larger of this and the partition count of the target topic is used instead. ++ +Default: `1`. +requiredGroups:: +A comma-separated list of groups to which the producer must ensure message delivery even if they start after it has been created (for example, by pre-creating durable queues in RabbitMQ). +headerMode:: +When set to `none`, it disables header embedding on output. +It is effective only for messaging middleware that does not support message headers natively and requires header embedding. +This option is useful when producing data for non-Spring Cloud Stream applications when native headers are not supported. +When set to `headers`, it uses the middleware's native header mechanism. +When set to `embeddedHeaders`, it embeds headers into the message payload. ++ +Default: Depends on the binder implementation. + +useNativeEncoding:: +When set to `true`, the outbound message is serialized directly by the client library, which must be configured correspondingly (for example, setting an appropriate Kafka producer value serializer). +When this configuration is being used, the outbound message marshalling is not based on the `contentType` of the binding. +When native encoding is used, it is the responsibility of the consumer to use an appropriate decoder (for example, the Kafka consumer value de-serializer) to deserialize the inbound message. +Also, when native encoding and decoding is used, the `headerMode=embeddedHeaders` property is ignored and headers are not embedded in the message. +See the consumer property `useNativeDecoding`. ++ +Default: `false`. + +errorChannelEnabled:: +When set to true, if the binder supports asynchroous send results, send failures are sent to an error channel for the destination. See Error Handling for more information. ++ +Default: false. + + +[[advanced-producer-configuration]] +== Advanced Producer Configuration + +In some cases Producer Properties are not enough to properly configure a producing MessageHandler in the binder, or may be you prefer a programmatic approach +while configuring such producing MessageHandler. Regardless of the reason, spring-cloud-stream provides `ProducerMessageHandlerCustomizer` to accomplish it. + + +==== +[source, java] +---- +@FunctionalInterface +public interface ProducerMessageHandlerCustomizer { + + /** + * Configure a {@link MessageHandler} that is being created by the binder for the + * provided destination name. + * @param handler the {@link MessageHandler} from the binder. + * @param destinationName the bound destination name. + */ + void configure(H handler, String destinationName); + +} +---- + +As you can see it gives you access to an actual instance of producing `MessageHandler` which you can configure as you wish. +All you need to do is provide implementation of this strategy and configure it as a `@Bean`. + +==== + +[[content-type-management]] += Content Type Negotiation + +Data transformation is one of the core features of any message-driven microservice architecture. Given that, in Spring Cloud Stream, such data +is represented as a Spring `Message`, a message may have to be transformed to a desired shape or size before reaching its destination. This is required for two reasons: + +. To convert the contents of the incoming message to match the signature of the application-provided handler. + +. To convert the contents of the outgoing message to the wire format. + +The wire format is typically `byte[]` (that is true for the Kafka and Rabbit binders), but it is governed by the binder implementation. + +In Spring Cloud Stream, message transformation is accomplished with an `org.springframework.messaging.converter.MessageConverter`. + +NOTE: As a supplement to the details to follow, you may also want to read the following https://spring.io/blog/2018/02/26/spring-cloud-stream-2-0-content-type-negotiation-and-transformation[blog post]. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/binding-service-properties.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/binding-service-properties.adoc new file mode 100644 index 000000000..e062cceff --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/binding-service-properties.adoc @@ -0,0 +1,42 @@ +[[binding-service-properties]] += Binding Service Properties + +These properties are exposed via `org.springframework.cloud.stream.config.BindingServiceProperties` + +spring.cloud.stream.instanceCount:: +The number of deployed instances of an application. +Must be set for partitioning on the producer side. Must be set on the consumer side when using RabbitMQ and with Kafka if `autoRebalanceEnabled=false`. ++ +Default: `1`. + +spring.cloud.stream.instanceIndex:: +The instance index of the application: A number from `0` to `instanceCount - 1`. +Used for partitioning with RabbitMQ and with Kafka if `autoRebalanceEnabled=false`. +Automatically set in Cloud Foundry to match the application's instance index. + +spring.cloud.stream.dynamicDestinations:: +A list of destinations that can be bound dynamically (for example, in a dynamic routing scenario). +If set, only listed destinations can be bound. ++ +Default: empty (letting any destination be bound). + +spring.cloud.stream.defaultBinder:: +The default binder to use, if multiple binders are configured. +See <>. ++ +Default: empty. + +spring.cloud.stream.overrideCloudConnectors:: +This property is only applicable when the `cloud` profile is active and Spring Cloud Connectors are provided with the application. +If the property is `false` (the default), the binder detects a suitable bound service (for example, a RabbitMQ service bound in Cloud Foundry for the RabbitMQ binder) and uses it for creating connections (usually through Spring Cloud Connectors). +When set to `true`, this property instructs binders to completely ignore the bound services and rely on Spring Boot properties (for example, relying on the `spring.rabbitmq.*` properties provided in the environment for the RabbitMQ binder). +The typical usage of this property is to be nested in a customized environment <>. ++ +Default: `false`. + +spring.cloud.stream.bindingRetryInterval:: +The interval (in seconds) between retrying binding creation when, for example, the binder does not support late binding and the broker (for example, Apache Kafka) is down. +Set it to zero to treat such conditions as fatal, preventing the application from starting. ++ +Default: `30` + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/binding_visualization_control.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/binding_visualization_control.adoc new file mode 100644 index 000000000..451b2ae5e --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/binding_visualization_control.adoc @@ -0,0 +1,84 @@ +[[binding_visualization_control]] += Binding visualization and control + +Spring Cloud Stream supports visualization and control of the Bindings through Actuator endpoints as well as programmatic way. + +[[programmatic-way]] +== Programmatic way + +Since version 3.1 we expose `org.springframework.cloud.stream.binding.BindingsLifecycleController` which is registered as bean and once +injected could be used to control the lifecycle of individual bindings + +For example, looks at the fragment from one of the test cases. As you can see we retrieve `BindingsLifecycleController` +from spring application context and execute individual methods to control the lifecycle of `echo-in-0` binding.. + +[source,java] +---- +BindingsLifecycleController bindingsController = context.getBean(BindingsLifecycleController.class); +Binding binding = bindingsController.queryState("echo-in-0"); +assertThat(binding.isRunning()).isTrue(); +bindingsController.changeState("echo-in-0", State.STOPPED); +//Alternative way of changing state. For convenience we expose start/stop and pause/resume operations. +//bindingsController.stop("echo-in-0") +assertThat(binding.isRunning()).isFalse(); +---- + +[[actuator]] +== Actuator +Since actuator and web are optional, you must first add one of the web dependencies as well as add the actuator dependency manually. +The following example shows how to add the dependency for the Web framework: + +[source,xml] +---- + + org.springframework.boot + spring-boot-starter-web + +---- + +The following example shows how to add the dependency for the WebFlux framework: + +[source,xml] +---- + + org.springframework.boot + spring-boot-starter-webflux + +---- + +You can add the Actuator dependency as follows: +[source,xml] +---- + + org.springframework.boot + spring-boot-starter-actuator + +---- + +NOTE: To run Spring Cloud Stream 2.0 apps in Cloud Foundry, you must add `spring-boot-starter-web` and `spring-boot-starter-actuator` to the classpath. Otherwise, the +application will not start due to health check failures. + +You must also enable the `bindings` actuator endpoints by setting the following property: `--management.endpoints.web.exposure.include=bindings`. + +Once those prerequisites are satisfied. you should see the following in the logs when application start: + + : Mapped "{[/actuator/bindings/{name}],methods=[POST]. . . + : Mapped "{[/actuator/bindings],methods=[GET]. . . + : Mapped "{[/actuator/bindings/{name}],methods=[GET]. . . + +To visualize the current bindings, access the following URL: +`http://:/actuator/bindings` + +Alternative, to see a single binding, access one of the URLs similar to the following: +`http://:/actuator/bindings/` + +You can also stop, start, pause, and resume individual bindings by posting to the same URL while providing a `state` argument as JSON, as shown in the following examples: + + curl -d '{"state":"STOPPED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName + curl -d '{"state":"STARTED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName + curl -d '{"state":"PAUSED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName + curl -d '{"state":"RESUMED"}' -H "Content-Type: application/json" -X POST http://:/actuator/bindings/myBindingName + +NOTE: `PAUSED` and `RESUMED` work only when the corresponding binder and its underlying technology supports it. Otherwise, you see the warning message in the logs. +Currently, only Kafka and [Solace](https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter#consumer-bindings-pauseresume) binders supports the `PAUSED` and `RESUMED` states. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/bindings.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/bindings.adoc new file mode 100644 index 000000000..94807b423 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/bindings.adoc @@ -0,0 +1,146 @@ +[[bindings]] += Bindings + +As stated earlier, _Bindings_ provide a bridge between the external messaging system (e.g., queue, topic etc.) and application-provided _Producers_ and _Consumers_. + +The following example shows a fully configured and functioning Spring Cloud Stream application that receives the payload of the message +as a `String` type (see <> section), logs it to the console and sends it down stream after converting it to upper case. + +[source, java] +---- +@SpringBootApplication +public class SampleApplication { + + public static void main(String[] args) { + SpringApplication.run(SampleApplication.class, args); + } + + @Bean + public Function uppercase() { + return value -> { + System.out.println("Received: " + value); + return value.toUpperCase(); + }; + } +} +---- +The above example looks no different then any vanilla spring-boot application. It defines a single bean of type `Function` +and that is it. So, how does it become a spring-cloud-stream application? +It becomes a spring-cloud-stream application simply because of the presence of spring-cloud-stream and binder dependencies +and auto-configuration classes on the classpath, effectively setting the context for your boot application as a spring-cloud-stream application. +And in this context beans of type `Supplier`, `Function` or `Consumer` are treated as defacto message handlers triggering +binding of to destinations exposed by the provided binder following certain naming conventions and +rules to avoid extra configuration. + +[[binding-and-binding-names]] +== Binding and Binding names + +Binding is an abstraction that represents a bridge between sources and targets exposed by the binder and user code, +This abstraction has a name and while we try to do our best to limit configuration required to run spring-cloud-stream applications, +being aware of such name(s) is necessary for cases where additional per-binding configuration is required. + +Throughout this manual you will see examples of configuration properties such as `spring.cloud.stream.bindings.input.destination=myQueue`. +The `input` segment in this property name is what we refer to as _binding name_ and it could derive via several mechanisms. +The following sub-sections will describe the naming conventions and configuration elements used by spring-cloud-stream to control binding names. + +NOTE: If your binding name has special characters, such as the `.` character, you need to surround the binding key with brackets (`[]`) and then wrap it in qoutes. +For example `spring.cloud.stream.bindings."[my.output.binding.key]".destination`. + +[[functional-binding-names]] +=== Functional binding names + +Unlike the explicit naming required by annotation-based support (legacy) used in the previous versions of spring-cloud-stream, the functional +programming model defaults to a simple convention when it comes to binding names, thus greatly simplifying application configuration. +Let's look at the first example: + +[source, java] +---- +@SpringBootApplication +public class SampleApplication { + + @Bean + public Function uppercase() { + return value -> value.toUpperCase(); + } +} +---- + +In the preceding example we have an application with a single function which acts as message handler. As a `Function` it has an +input and output. +The naming convention used to name input and output bindings is as follows: + +* input - ` + -in- + ` +* output - ` + -out- + ` + +The `in` and `out` corresponds to the type of binding (such as _input_ or _output_). +The `index` is the index of the input or output binding. It is always 0 for typical single input/output function, +so it's only relevant for <>. + +So if for example you would want to map the input of this function to a remote +destination (e.g., topic, queue etc) called "my-topic" you would do so with the following property: +---- +--spring.cloud.stream.bindings.uppercase-in-0.destination=my-topic +---- +Note how `uppercase-in-0` is used as a segment in property name. The same goes for `uppercase-out-0`. + +***Descriptive Binding Names*** + +Some times to improve readability you may want to give your binding a more descriptive name (such as 'account', 'orders' etc). +Another way of looking at it is you can map an _implicit binding name_ to an _explicit binding name_. And you can do it with +`spring.cloud.stream.function.bindings.` property. +This property also provides a migration path for existing applications that rely on custom interface-based +bindings that require explicit names. + +For example, +---- +--spring.cloud.stream.function.bindings.uppercase-in-0=input +---- + +In the preceding example you mapped and effectively renamed `uppercase-in-0` binding name to `input`. Now all configuration +properties can refer to `input` binding name instead (e.g., `--spring.cloud.stream.bindings.input.destination=my-topic`). + +NOTE: While descriptive binding names may enhance the readability aspect of the configuration, they also create +another level of misdirection by mapping an implicit binding name to an explicit binding name. And since all subsequent +configuration properties will use the explicit binding name you must always refer to this 'bindings' property to +correlate which function it actually corresponds to. We believe that for most cases (with the exception of <>) +it may be an overkill, so, it is our recommendation to avoid using it altogether, especially +since not using it provides a clear path between binder destination and binding name, such as `spring.cloud.stream.bindings.uppercase-in-0.destination=sample-topic`, +where you are clearly correlating the input of `uppercase` function to `sample-topic` destination. + +For more on properties and other configuration options please see <> section. + +[[explicit-binding-creation]] +=== Explicit binding creation + +In the previous section we explained how bindings are created implicitly driven by the names of `Function`, `Supplier` or `Consumer` beans provided by your application. +However, there are times when you may need to create binding explicitly where bindings are not tied to any function. This is typically done to +support integrations with other frameworks via `StreamBridge`. + +Spring Cloud Stream allows you to define input and output bindings explicitly via `spring.cloud.stream.input-bindings` and `spring.cloud.stream.output-bindings` +properties. Noticed the plural in the property names allowing you to define multiple bindings by simply using `;` as a delimiter. +Just look at the following test case as an example: + +---- +@Test +public void testExplicitBindings() { + try (ConfigurableApplicationContext context = new SpringApplicationBuilder( + TestChannelBinderConfiguration.getCompleteConfiguration(EmptyConfiguration.class)) + .web(WebApplicationType.NONE) + .run("--spring.jmx.enabled=false", + "--spring.cloud.stream.input-bindings=fooin;barin", + "--spring.cloud.stream.output-bindings=fooout;barout")) { + + + . . . + } +} + +@EnableAutoConfiguration +@Configuration +public static class EmptyConfiguration { +} +---- + +As you can see we have declared two input bindings and two output bindings while our configuration had no functions defined, yet we were able to successfully create these bindings and access their corresponding channels. + + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/consumer-groups.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/consumer-groups.adoc new file mode 100644 index 000000000..1c38545ed --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/consumer-groups.adoc @@ -0,0 +1,17 @@ +[[consumer-groups]] += Consumer Groups + +While the publish-subscribe model makes it easy to connect applications through shared topics, the ability to scale up by creating multiple instances of a given application is equally important. +When doing so, different instances of an application are placed in a competing consumer relationship, where only one of the instances is expected to handle a given message. + +Spring Cloud Stream models this behavior through the concept of a consumer group. +(Spring Cloud Stream consumer groups are similar to and inspired by Kafka consumer groups.) +Each consumer binding can use the `spring.cloud.stream.bindings..group` property to specify a group name. +For the consumers shown in the following figure, this property would be set as `spring.cloud.stream.bindings..group=hdfsWrite` or `spring.cloud.stream.bindings..group=average`. + +.Spring Cloud Stream Consumer Groups +image::SCSt-groups.png[width=800,scaledwidth="75%",align="center"] + +All groups that subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination. +By default, when a group is not specified, Spring Cloud Stream assigns the application to an anonymous and independent single-member consumer group that is in a publish-subscribe relationship with all other consumer groups. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/consumer-types.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/consumer-types.adoc new file mode 100644 index 000000000..7303339d8 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/consumer-types.adoc @@ -0,0 +1,29 @@ +[[consumer-types]] += Consumer Types + +Two types of consumer are supported: + +* Message-driven (sometimes referred to as Asynchronous) +* Polled (sometimes referred to as Synchronous) + +Prior to version 2.0, only asynchronous consumers were supported. A message is delivered as soon as it is available and a thread is available to process it. + +When you wish to control the rate at which messages are processed, you might want to use a synchronous consumer. +// TODO This needs more description. A sentence parallel to the last sentence of the preceding paragraph would help. + +[[durability]] +== Durability + +Consistent with the opinionated application model of Spring Cloud Stream, consumer group subscriptions are durable. +That is, a binder implementation ensures that group subscriptions are persistent and that, once at least one subscription for a group has been created, the group receives messages, even if they are sent while all applications in the group are stopped. + +[NOTE] +==== +Anonymous subscriptions are non-durable by nature. +For some binder implementations (such as RabbitMQ), it is possible to have non-durable group subscriptions. +==== + +In general, it is preferable to always specify a consumer group when binding an application to a given destination. +When scaling up a Spring Cloud Stream application, you must specify a consumer group for each of its input bindings. +Doing so prevents the application's instances from receiving duplicate messages (unless that behavior is desired, which is unusual). + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/deploying-applications-on-cloudfoundry.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/deploying-applications-on-cloudfoundry.adoc new file mode 100644 index 000000000..ce041abf1 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/deploying-applications-on-cloudfoundry.adoc @@ -0,0 +1,25 @@ +[[deploying-stream-applications-on-cloudfoundry]] += Deploying Stream Applications on CloudFoundry + +On CloudFoundry, services are usually exposed through a special environment variable called https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES[VCAP_SERVICES]. + +When configuring your binder connections, you can use the values from an environment variable as explained on the http://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current-SNAPSHOT/reference/htmlsingle/#getting-started-ups[dataflow Cloud Foundry Server] docs. + +[[binder-implementations]] += Binder Implementations + +The following is the list of available binder implementations + +* https://cloud.spring.io/spring-cloud-stream-binder-rabbit/[RabbitMQ] +* https://cloud.spring.io/spring-cloud-stream-binder-kafka/[Apache Kafka] +* https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis[Amazon Kinesis] +* https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-pubsub-stream-binder[Google PubSub _(partner maintained)_] +* https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter#spring-cloud-stream-binder-for-solace-pubsub[Solace PubSub+ _(partner maintained)_] +* https://aka.ms/spring/docs#spring-cloud-stream-binder-for-azure-event-hubs[Azure Event Hubs _(partner maintained)_] +* https://aka.ms/spring/docs#spring-cloud-stream-binder-for-azure-service-bus[Azure Service Bus _(partner maintained)_] +* https://github.com/alibaba/spring-cloud-alibaba/wiki/RocketMQ-en[Apache RocketMQ _(partner maintained)_] + +As it was mentioned earlier Binder abstraction is also one of the extension points of the framework. So if you can't find a suitable binder in the preceding list you can implement your own binder on top of Spring Cloud Stream. +In the https://medium.com/@domenicosibilio/how-to-create-a-spring-cloud-stream-binder-from-scratch-ab8b29ee931b[How to create a Spring Cloud Stream Binder from scratch] post a community member documents +in details, with an example, a set of steps necessary to implement a custom binder. +The steps are also highlighted in the `<>` section. diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/destination-binders.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/destination-binders.adoc new file mode 100644 index 000000000..42a08bc82 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/destination-binders.adoc @@ -0,0 +1,14 @@ +[[destination-binders]] += Destination Binders + +Destination Binders are extension components of Spring Cloud Stream responsible for providing the necessary configuration and implementation to facilitate +integration with external messaging systems. +This integration is responsible for connectivity, delegation, and routing of messages to and from producers and consumers, data type conversion, +invocation of the user code, and more. + +Binders handle a lot of the boiler plate responsibilities that would otherwise fall on your shoulders. However, to accomplish that, the binder still needs +some help in the form of minimalistic yet required set of instructions from the user, which typically come in the form of some type of _binding_ configuration. + +While it is out of scope of this section to discuss all of the available binder and binding configuration options (the rest of the manual covers them extensively), +_Binding_ as a concept, does require special attention. The next section discusses it in detail. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/event-routing.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/event-routing.adoc new file mode 100644 index 000000000..dfa17a2fa --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/event-routing.adoc @@ -0,0 +1,170 @@ +[[event-routing]] += Event Routing + +Event Routing, in the context of Spring Cloud Stream, is the ability to either +_a) route events to a particular event subscriber_ or +_b) route events produced by an event subscriber to a particular destination_. +Here we'll refer to it as route ‘TO’ and route ‘FROM’. + +[[routing-to-consumer]] +== Routing TO Consumer +Routing can be achieved by relying on `RoutingFunction` available in Spring Cloud Function 3.0. All you need to do is enable it via +`--spring.cloud.stream.function.routing.enabled=true` application property or provide `spring.cloud.function.routing-expression` property. +Once enabled `RoutingFunction` will be bound to input destination +receiving all the messages and route them to other functions based on the provided instruction. + +IMPORTANT: For the purposes of binding the name of the routing destination is `functionRouter-in-0` +(see RoutingFunction.FUNCTION_NAME and binding naming convention <>). + +Instruction could be provided with individual messages as well as application properties. + +Here are couple of samples: + +[[using-message-headers]] +=== Using message headers +[source,java] +---- +@SpringBootApplication +public class SampleApplication { + + public static void main(String[] args) { + SpringApplication.run(SampleApplication.class, + "--spring.cloud.stream.function.routing.enabled=true"); + } + + @Bean + public Consumer even() { + return value -> { + System.out.println("EVEN: " + value); + }; + } + + @Bean + public Consumer odd() { + return value -> { + System.out.println("ODD: " + value); + }; + } +} +---- +By sending a message to the `functionRouter-in-0` destination exposed by the binder (i.e., rabbit, kafka), +such message will be routed to the appropriate (‘even’ or ‘odd’) Consumer. + +By default `RoutingFunction` will look for a `spring.cloud.function.definition` or `spring.cloud.function.routing-expression` (for more dynamic scenarios with SpEL) +header and if it is found, its value will be treated as the routing instruction. + +For example, +setting `spring.cloud.function.routing-expression` header to value `T(java.lang.System).currentTimeMillis() % 2 == 0 ? 'even' : 'odd'` will end up semi-randomly routing request to either `odd` or `even` functions. +Also, for SpEL, the _root object_ of the evaluation context is `Message` so you can do evaluation on individual headers (or message) as well `....routing-expression=headers['type']` + +[[using-application-properties]] +=== Using application properties + +The `spring.cloud.function.routing-expression` and/or `spring.cloud.function.definition` +can be passed as application properties (e.g., `spring.cloud.function.routing-expression=headers['type']`. +[source,java] +---- +@SpringBootApplication +public class RoutingStreamApplication { + + public static void main(String[] args) { + SpringApplication.run(RoutingStreamApplication.class, + "--spring.cloud.function.routing-expression=" + + "T(java.lang.System).nanoTime() % 2 == 0 ? 'even' : 'odd'"); + } + @Bean + public Consumer even() { + return value -> System.out.println("EVEN: " + value); + } + + @Bean + public Consumer odd() { + return value -> System.out.println("ODD: " + value); + } +} +---- +IMPORTANT: Passing instructions via application properties is especially important for reactive functions given that a reactive +function is only invoked once to pass the Publisher, so access to the individual items is limited. + +[[routing-function-and-output-binding]] +=== Routing Function and output binding + +`RoutingFunction` is a `Function` and as such treated no differently than any other function. Well. . . almost. + +When `RoutingFunction` routes to another `Function`, its output is sent to the output binding of the `RoutingFunction` which +is `functionRouter-in-0` as expected. But what if `RoutingFunction` routes to a `Consumer`? In other words the result of invocation +of the `RoutingFunction` may not produce anything to be sent to the output binding, thus making it necessary to even have one. +So, we do treat `RoutingFunction` a little bit differently when we create bindings. And even though it is transparent to you as a user +(there is really nothing for you to do), being aware of some of the mechanics would help you understand its inner workings. + +So, the rule is; +We never create output binding for the `RoutingFunction`, only input. So when you routing to `Consumer`, the `RoutingFunction` effectively +becomes as a `Consumer` by not having any output bindings. However, if `RoutingFunction` happen to route to another `Function` which produces +the output, the output binding for the `RoutingFunction` will be create dynamically at which point `RoutingFunction` will act as a regular `Function` +with regards to bindings (having both input and output bindings). + +[[routing-from-consumer]] +== Routing FROM Consumer + +Aside from static destinations, Spring Cloud Stream lets applications send messages to dynamically bound destinations. +This is useful, for example, when the target destination needs to be determined at runtime. +Applications can do so in one of two ways. + +[[spring-cloud-stream-sendto-destination]] +=== spring.cloud.stream.sendto.destination + +You can also delegate to the framework to dynamically resolve the output destination by specifying `spring.cloud.stream.sendto.destination` header +set to the name of the destination to be resolved. + +Consider the following example: + +[source,java] +---- +@SpringBootApplication +@Controller +public class SourceWithDynamicDestination { + + @Bean + public Function> destinationAsPayload() { + return value -> { + return MessageBuilder.withPayload(value) + .setHeader("spring.cloud.stream.sendto.destination", value).build();}; + } +} +---- + +Albeit trivial you can clearly see in this example, our output is a Message with `spring.cloud.stream.sendto.destination` header +set to the value of he input argument. The framework will consult this header and will attempt to create or discover +a destination with that name and send output to it. + + +If destination names are known in advance, you can configure the producer properties as with any other destination. +Alternatively, if you register a `NewDestinationBindingCallback<>` bean, it is invoked just before the binding is created. +The callback takes the generic type of the extended producer properties used by the binder. +It has one method: + +[source, java] +---- +void configure(String destinationName, MessageChannel channel, ProducerProperties producerProperties, + T extendedProducerProperties); +---- + +The following example shows how to use the RabbitMQ binder: + +[source, java] +---- +@Bean +public NewDestinationBindingCallback dynamicConfigurer() { + return (name, channel, props, extended) -> { + props.setRequiredGroups("bindThisQueue"); + extended.setQueueNameGroupOnly(true); + extended.setAutoBindDlq(true); + extended.setDeadLetterQueueName("myDLQ"); + }; +} +---- + +NOTE: If you need to support dynamic destinations with multiple binder types, use `Object` for the generic type and cast the `extended` argument as needed. + +Also, please see <> section to see how yet another option (StreamBridge) can be utilized for similar cases. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/mechanics.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/mechanics.adoc new file mode 100644 index 000000000..234f516f0 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/mechanics.adoc @@ -0,0 +1,99 @@ +[[mechanics]] += Mechanics + +To better understand the mechanics and the necessity behind content-type negotiation, we take a look at a very simple use case by using the following message handler as an example: + +[source, java] +---- + +public Function personFunction {..} +---- + +NOTE: For simplicity, we assume that this is the only handler function in the application (we assume there is no internal pipeline). + +The handler shown in the preceding example expects a `Person` object as an argument and produces a `String` type as an output. +In order for the framework to succeed in passing the incoming `Message` as an argument to this handler, it has to somehow transform the payload of the `Message` type from the wire format to a `Person` type. +In other words, the framework must locate and apply the appropriate `MessageConverter`. +To accomplish that, the framework needs some instructions from the user. +One of these instructions is already provided by the signature of the handler method itself (`Person` type). +Consequently, in theory, that should be (and, in some cases, is) enough. +However, for the majority of use cases, in order to select the appropriate `MessageConverter`, the framework needs an additional piece of information. +That missing piece is `contentType`. + +Spring Cloud Stream provides three mechanisms to define `contentType` (in order of precedence): + +. *HEADER*: The `contentType` can be communicated through the Message itself. By providing a `contentType` header, you declare the content type to use to locate and apply the appropriate `MessageConverter`. + +. *BINDING*: The `contentType` can be set per destination binding by setting the `spring.cloud.stream.bindings.input.content-type` property. ++ +NOTE: The `input` segment in the property name corresponds to the actual name of the destination (which is “input” in our case). This approach lets you declare, on a per-binding basis, the content type to use to locate and apply the appropriate `MessageConverter`. + +. *DEFAULT*: If `contentType` is not present in the `Message` header or the binding, the default `application/json` content type is used to +locate and apply the appropriate `MessageConverter`. + +As mentioned earlier, the preceding list also demonstrates the order of precedence in case of a tie. For example, a header-provided content type takes precedence over any other content type. +The same applies for a content type set on a per-binding basis, which essentially lets you override the default content type. +However, it also provides a sensible default (which was determined from community feedback). + +Another reason for making `application/json` the default stems from the interoperability requirements driven by distributed microservices architectures, where producer and consumer not only run in different JVMs but can also run on different non-JVM platforms. + +When the non-void handler method returns, if the return value is already a `Message`, that `Message` becomes the payload. However, when the return value is not a `Message`, the new `Message` is constructed with the return value as the payload while inheriting +headers from the input `Message` minus the headers defined or filtered by `SpringIntegrationProperties.messageHandlerNotPropagatedHeaders`. +By default, there is only one header set there: `contentType`. This means that the new `Message` does not have `contentType` header set, thus ensuring that the `contentType` can evolve. +You can always opt out of returning a `Message` from the handler method where you can inject any header you wish. + +If there is an internal pipeline, the `Message` is sent to the next handler by going through the same process of conversion. However, if there is no internal pipeline or you have reached the end of it, the `Message` is sent back to the output destination. + +[[content-type-versus-argument-type]] +== Content Type versus Argument Type + +As mentioned earlier, for the framework to select the appropriate `MessageConverter`, it requires argument type and, optionally, content type information. +The logic for selecting the appropriate `MessageConverter` resides with the argument resolvers (`HandlerMethodArgumentResolvers`), which trigger right before the invocation of the user-defined handler method (which is when the actual argument type is known to the framework). +If the argument type does not match the type of the current payload, the framework delegates to the stack of the +pre-configured `MessageConverters` to see if any one of them can convert the payload. +As you can see, the `Object fromMessage(Message message, Class targetClass);` +operation of the MessageConverter takes `targetClass` as one of its arguments. +The framework also ensures that the provided `Message` always contains a `contentType` header. +When no contentType header was already present, it injects either the per-binding `contentType` header or the default `contentType` header. +The combination of `contentType` argument type is the mechanism by which framework determines if message can be converted to a target type. +If no appropriate `MessageConverter` is found, an exception is thrown, which you can handle by adding a custom `MessageConverter` (see `<>`). + +But what if the payload type matches the target type declared by the handler method? In this case, there is nothing to convert, and the +payload is passed unmodified. While this sounds pretty straightforward and logical, keep in mind handler methods that take a `Message` or `Object` as an argument. +By declaring the target type to be `Object` (which is an `instanceof` everything in Java), you essentially forfeit the conversion process. + +NOTE: Do not expect `Message` to be converted into some other type based only on the `contentType`. +Remember that the `contentType` is complementary to the target type. +If you wish, you can provide a hint, which `MessageConverter` may or may not take into consideration. + +[[message-converters]] +== Message Converters + +`MessageConverters` define two methods: + +[source, java] +---- +Object fromMessage(Message message, Class targetClass); + +Message toMessage(Object payload, @Nullable MessageHeaders headers); +---- + +It is important to understand the contract of these methods and their usage, specifically in the context of Spring Cloud Stream. + +The `fromMessage` method converts an incoming `Message` to an argument type. +The payload of the `Message` could be any type, and it is +up to the actual implementation of the `MessageConverter` to support multiple types. +For example, some JSON converter may support the payload type as `byte[]`, `String`, and others. +This is important when the application contains an internal pipeline (that is, input -> handler1 -> handler2 ->. . . -> output) and the output of the upstream handler results in a `Message` which may not be in the initial wire format. + +However, the `toMessage` method has a more strict contract and must always convert `Message` to the wire format: `byte[]`. + +So, for all intents and purposes (and especially when implementing your own converter) you regard the two methods as having the following signatures: + +[source, java] +---- +Object fromMessage(Message message, Class targetClass); + +Message toMessage(Object payload, @Nullable MessageHeaders headers); +---- + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/multiple-binders.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/multiple-binders.adoc new file mode 100644 index 000000000..19ec3d7a1 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/multiple-binders.adoc @@ -0,0 +1,24 @@ +[[multiple-binders]] += Multiple Binders on the Classpath + +When multiple binders are present on the classpath, the application must indicate which binder is to be used for each destination binding. +Each binder configuration contains a `META-INF/spring.binders` file, which is a simple properties file, as shown in the following example: + +[source] +---- +rabbit:\ +org.springframework.cloud.stream.binder.rabbit.config.RabbitServiceAutoConfiguration +---- + +Similar files exist for the other provided binder implementations (such as Kafka), and custom binder implementations are expected to provide them as well. +The key represents an identifying name for the binder implementation, whereas the value is a comma-separated list of configuration classes that each contain one and only one bean definition of type `org.springframework.cloud.stream.binder.Binder`. + +Binder selection can either be performed globally, using the `spring.cloud.stream.defaultBinder` property (for example, `spring.cloud.stream.defaultBinder=rabbit`) or individually, by configuring the binder on each binding. +For instance, a processor application (that has bindings named `input` and `output` for read and write respectively) that reads from Kafka and writes to RabbitMQ can specify the following configuration: + +[source] +---- +spring.cloud.stream.bindings.input.binder=kafka +spring.cloud.stream.bindings.output.binder=rabbit +---- + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/multiple-systems.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/multiple-systems.adoc new file mode 100644 index 000000000..a628c3a15 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/multiple-systems.adoc @@ -0,0 +1,63 @@ +[[multiple-systems]] += Connecting to Multiple Systems + +By default, binders share the application's Spring Boot auto-configuration, so that one instance of each binder found on the classpath is created. +If your application should connect to more than one broker of the same type, you can specify multiple binder configurations, each with different environment settings. + +NOTE: Turning on explicit binder configuration disables the default binder configuration process altogether. +If you do so, all binders in use must be included in the configuration. +Frameworks that intend to use Spring Cloud Stream transparently may create binder configurations that can be referenced by name, but they do not affect the default binder configuration. +In order to do so, a binder configuration may have its `defaultCandidate` flag set to false (for example, `spring.cloud.stream.binders..defaultCandidate=false`). +This denotes a configuration that exists independently of the default binder configuration process. + +The following example shows a typical configuration for a processor application that connects to two RabbitMQ broker instances: + +[source,yml] +---- +spring: + cloud: + stream: + bindings: + input: + destination: thing1 + binder: rabbit1 + output: + destination: thing2 + binder: rabbit2 + binders: + rabbit1: + type: rabbit + environment: + spring: + rabbitmq: + host: + rabbit2: + type: rabbit + environment: + spring: + rabbitmq: + host: +---- +NOTE: The `environment` property of the particular binder can also be used for any Spring Boot property, +including this `spring.main.sources` which can be useful for adding additional configurations for the +particular binders, e.g. overriding auto-configured beans. + +For example; +[source, yaml] +---- +environment: + spring: + main: + sources: com.acme.config.MyCustomBinderConfiguration +---- + +To activate a specific profile for the particular binder environment, you should use a `spring.profiles.active` property: + +[source, yaml] +---- +environment: + spring: + profiles: + active: myBinderProfile +---- + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-application-model.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-application-model.adoc new file mode 100644 index 000000000..1a3644246 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-application-model.adoc @@ -0,0 +1,17 @@ +[[spring-cloud-stream-overview-application-model]] += Application Model + +A Spring Cloud Stream application consists of a middleware-neutral core. +The application communicates with the outside world by establishing _bindings_ between destinations +exposed by the external brokers and input/output arguments in your code. Broker specific details +necessary to establish bindings are handled by middleware-specific _Binder_ implementations. + +.Spring Cloud Stream Application +image::SCSt-with-binder.png[width=800,scaledwidth="75%",align="center"] + +[[fat-jar]] +== Fat JAR + +Spring Cloud Stream applications can be run in stand-alone mode from your IDE for testing. +To run a Spring Cloud Stream application in production, you can create an executable (or "`fat`") JAR by using the standard Spring Boot tooling provided for Maven or Gradle. See the https://docs.spring.io/spring-boot/docs/current/reference/html/howto-build.html#howto-create-an-executable-jar-with-maven[Spring Boot Reference Guide] for more details. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-binder-abstraction.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-binder-abstraction.adoc new file mode 100644 index 000000000..f8fbc5a2e --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-binder-abstraction.adoc @@ -0,0 +1,23 @@ +[[spring-cloud-stream-overview-binder-abstraction]] += The Binder Abstraction + +Spring Cloud Stream provides Binder implementations for https://github.com/spring-cloud/spring-cloud-stream-binder-kafka[Kafka] and https://github.com/spring-cloud/spring-cloud-stream-binder-rabbit[Rabbit MQ]. +The framework also includes a test binder for integration testing of your applications as spring-cloud-stream application. See <> section for more details. + +Binder abstraction is also one of the extension points of the framework, which means you can implement your own binder on top of Spring Cloud Stream. +In the https://medium.com/@domenicosibilio/how-to-create-a-spring-cloud-stream-binder-from-scratch-ab8b29ee931b[How to create a Spring Cloud Stream Binder from scratch] post a community member documents +in details, with an example, a set of steps necessary to implement a custom binder. +The steps are also highlighted in the `<>` section. + +Spring Cloud Stream uses Spring Boot for configuration, and the Binder abstraction makes it possible for a Spring Cloud Stream application to be flexible in how it connects to middleware. +For example, deployers can dynamically choose, at runtime, the mapping between the external destinations (such as the Kafka topics or RabbitMQ exchanges) and inputs +and outputs of the message handler (such as input parameter of the function and its return argument). +Such configuration can be provided through external configuration properties and in any form supported by Spring Boot (including application arguments, environment variables, and `application.yml` or `application.properties` files). +In the sink example from the <> section, setting the `spring.cloud.stream.bindings.input.destination` application property to `raw-sensor-data` causes it to read from the `raw-sensor-data` Kafka topic or from a queue bound to the `raw-sensor-data` RabbitMQ exchange. + +Spring Cloud Stream automatically detects and uses a binder found on the classpath. +You can use different types of middleware with the same code. +To do so, include a different binder at build time. +For more complex use cases, you can also package multiple binders with your application and have it choose the binder( and even whether to use different binders for different bindings) at runtime. + + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-binder-api.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-binder-api.adoc new file mode 100644 index 000000000..ec363f179 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-binder-api.adoc @@ -0,0 +1,38 @@ +[[spring-cloud-stream-overview-binder-api]] += Binder SPI + +The Binder SPI consists of a number of interfaces, out-of-the box utility classes, and discovery strategies that provide a pluggable mechanism for connecting to external middleware. + +The key point of the SPI is the `Binder` interface, which is a strategy for connecting inputs and outputs to external middleware. The following listing shows the definition of the `Binder` interface: + +[source,java] +---- +public interface Binder { + Binding bindConsumer(String bindingName, String group, T inboundBindTarget, C consumerProperties); + + Binding bindProducer(String bindingName, T outboundBindTarget, P producerProperties); +} +---- + +The interface is parameterized, offering a number of extension points: + +* Input and output bind targets. +* Extended consumer and producer properties, allowing specific Binder implementations to add supplemental properties that can be supported in a type-safe manner. + +A typical binder implementation consists of the following: + +* A class that implements the `Binder` interface; +* A Spring `@Configuration` class that creates a bean of type `Binder` along with the middleware connection infrastructure. +* A `META-INF/spring.binders` file found on the classpath containing one or more binder definitions, as shown in the following example: ++ +[source] +---- +kafka:\ +org.springframework.cloud.stream.binder.kafka.config.KafkaBinderConfiguration +---- + +NOTE: As it was mentioned earlier Binder abstraction is also one of the extension points of the framework. So if you can't find a suitable binder in the preceding list you can implement your own binder on top of Spring Cloud Stream. +In the https://medium.com/@domenicosibilio/how-to-create-a-spring-cloud-stream-binder-from-scratch-ab8b29ee931b[How to create a Spring Cloud Stream Binder from scratch] post a community member documents +in details, with an example, a set of steps necessary to implement a custom binder. +The steps are also highlighted in the `<>` section. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-connecting-multiple-application-instances.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-connecting-multiple-application-instances.adoc new file mode 100644 index 000000000..3b84026fb --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-connecting-multiple-application-instances.adoc @@ -0,0 +1,20 @@ +[[spring-cloud-stream-overview-connecting-multiple-application-instances]] += Connecting Multiple Application Instances + +While Spring Cloud Stream makes it easy for individual Spring Boot applications to connect to messaging systems, the typical scenario for Spring Cloud Stream is the creation of multi-application pipelines, where microservice applications send data to each other. +You can achieve this scenario by correlating the input and output destinations of "`adjacent`" applications. + +Suppose a design calls for the Time Source application to send data to the Log Sink application. You could use a common destination named `ticktock` for bindings within both applications. + +Time Source (that has the binding named `output`) would set the following property: + +---- +spring.cloud.stream.bindings.output.destination=ticktock +---- + +Log Sink (that has the binding named `input`) would set the following property: + +---- +spring.cloud.stream.bindings.input.destination=ticktock +---- + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-custom-binder-impl.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-custom-binder-impl.adoc new file mode 100644 index 000000000..e7494d2b8 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-custom-binder-impl.adoc @@ -0,0 +1,256 @@ +[[spring-cloud-stream-overview-custom-binder-impl]] += Implementing Custom Binders + +In order to implement a custom `Binder`, all you need is to: + +- Add the required dependencies +- Provide a ProvisioningProvider implementation +- Provide a MessageProducer implementation +- Provide a MessageHandler implementation +- Provide a Binder implementation +- Create a Binder Configuration +- Define your binder in META-INF/spring.binders + +***Add the required dependencies*** + +Add the `spring-cloud-stream` dependency to your project _(eg. for Maven)_: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream + ${spring.cloud.stream.version} + +---- + +***Provide a ProvisioningProvider implementation*** + +The `ProvisioningProvider` is responsible for the provisioning of consumer and producer destinations, and is required to convert the logical destinations included in the application.yml or application.properties file in physical destination references. + +Below an example of ProvisioningProvider implementation that simply trims the destinations provided via input/output bindings configuration: + +[source,java] +---- +public class FileMessageBinderProvisioner implements ProvisioningProvider { + + @Override + public ProducerDestination provisionProducerDestination( + final String name, + final ProducerProperties properties) { + + return new FileMessageDestination(name); + } + + @Override + public ConsumerDestination provisionConsumerDestination( + final String name, + final String group, + final ConsumerProperties properties) { + + return new FileMessageDestination(name); + } + + private class FileMessageDestination implements ProducerDestination, ConsumerDestination { + + private final String destination; + + private FileMessageDestination(final String destination) { + this.destination = destination; + } + + @Override + public String getName() { + return destination.trim(); + } + + @Override + public String getNameForPartition(int partition) { + throw new UnsupportedOperationException("Partitioning is not implemented for file messaging."); + } + + } + +} +---- + +***Provide a MessageProducer implementation*** + +The `MessageProducer` is responsible for consuming events and handling them as messages to the client application that is configured to consume such events. + +Here is an example of MessageProducer implementation that extends the `MessageProducerSupport` abstraction in order to poll on a file that matches the trimmed destination name and is located in the project path, while also archiving read messages and discarding consequent identical messages: + +[source,java] +---- +public class FileMessageProducer extends MessageProducerSupport { + + public static final String ARCHIVE = "archive.txt"; + private final ConsumerDestination destination; + private String previousPayload; + + public FileMessageProducer(ConsumerDestination destination) { + this.destination = destination; + } + + @Override + public void doStart() { + receive(); + } + + private void receive() { + ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1); + + executorService.scheduleWithFixedDelay(() -> { + String payload = getPayload(); + + if(payload != null) { + Message receivedMessage = MessageBuilder.withPayload(payload).build(); + archiveMessage(payload); + sendMessage(receivedMessage); + } + + }, 0, 50, MILLISECONDS); + } + + private String getPayload() { + try { + List allLines = Files.readAllLines(Paths.get(destination.getName())); + String currentPayload = allLines.get(allLines.size() - 1); + + if(!currentPayload.equals(previousPayload)) { + previousPayload = currentPayload; + return currentPayload; + } + } catch (IOException e) { + throw new RuntimeException(e); + } + + return null; + } + + private void archiveMessage(String payload) { + try { + Files.write(Paths.get(ARCHIVE), (payload + "\n").getBytes(), CREATE, APPEND); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + +} +---- + +NOTE: When implementing a custom binder, this step is not strictly mandatory as you could always resort to using an already existing MessageProducer implementation! + +***Provide a MessageHandler implementation*** + +The `MessageHandler` provides the logic required to produce an event. + +Here is an example of MessageHandler implementation: + +[source,java] +---- +public class FileMessageHandler implements MessageHandler{ + + @Override + public void handleMessage(Message message) throws MessagingException { + //write message to file + } + +} +---- + +NOTE: When implementing a custom binder, this step is not strictly mandatory as you could always resort to using an already existing MessageHandler implementation! + +***Provide a Binder implementation*** + +You are now able to provide your own implementation of the `Binder` abstraction. This can be easily done by: + +- extending the `AbstractMessageChannelBinder` class +- specifying your ProvisioningProvider as a generic argument of the AbstractMessageChannelBinder +- overriding the `createProducerMessageHandler` and `createConsumerEndpoint` methods + +_eg.:_ +[source,java] +---- +public class FileMessageBinder extends AbstractMessageChannelBinder { + + public FileMessageBinder( + String[] headersToEmbed, + FileMessageBinderProvisioner provisioningProvider) { + + super(headersToEmbed, provisioningProvider); + } + + @Override + protected MessageHandler createProducerMessageHandler( + final ProducerDestination destination, + final ProducerProperties producerProperties, + final MessageChannel errorChannel) throws Exception { + + return message -> { + String fileName = destination.getName(); + String payload = new String((byte[])message.getPayload()) + "\n"; + + try { + Files.write(Paths.get(fileName), payload.getBytes(), CREATE, APPEND); + } catch (IOException e) { + throw new RuntimeException(e); + } + }; + } + + @Override + protected MessageProducer createConsumerEndpoint( + final ConsumerDestination destination, + final String group, + final ConsumerProperties properties) throws Exception { + + return new FileMessageProducer(destination); + } + +} +---- + +***Create a Binder Configuration*** + +It is strictly required that you create a Spring Configuration to initialize the bean for your binder implementation _(and all other beans that you might need)_: + +[source,java] +---- +@Configuration +public class FileMessageBinderConfiguration { + + @Bean + @ConditionalOnMissingBean + public FileMessageBinderProvisioner fileMessageBinderProvisioner() { + return new FileMessageBinderProvisioner(); + } + + @Bean + @ConditionalOnMissingBean + public FileMessageBinder fileMessageBinder(FileMessageBinderProvisioner fileMessageBinderProvisioner) { + return new FileMessageBinder(null, fileMessageBinderProvisioner); + } + +} +---- + +***Define your binder in META-INF/spring.binders*** + +Finally, you must define your binder in a `META-INF/spring.binders` file on the classpath, specifying both the name of the binder and the full qualified name of your Binder Configuration class: + +[source] +---- +myFileBinder:\ +com.example.springcloudstreamcustombinder.config.FileMessageBinderConfiguration +---- + +[[configuration-options]] += Configuration Options + +Spring Cloud Stream supports general configuration options as well as configuration for bindings and binders. +Some binders let additional binding properties support middleware-specific features. + +Configuration options can be provided to Spring Cloud Stream applications through any mechanism supported by Spring Boot. +This includes application arguments, environment variables, and YAML or .properties files. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-error-handling.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-error-handling.adoc new file mode 100644 index 000000000..f89c65fe6 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-error-handling.adoc @@ -0,0 +1,217 @@ +[[spring-cloud-stream-overview-error-handling]] += Error Handling + +In this section we'll explain the general idea behind error handling mechanisms provided by the framework. +We'll be using Rabbit binder as an example, since individual binders define different set +of properties for certain supported mechanisms specific to underlying broker capabilities (such as Kafka binder). + +Errors happen, and Spring Cloud Stream provides several flexible mechanisms to deal with them. Note, the techniques are dependent on binder implementation and the +capability of the underlying messaging middleware as well as programming model (more on this later). + +Whenever Message handler (function) throws an exception, it is propagated back to the binder, at which point binder will make several attempts at re-trying +the same message (3 by default) using `RetryTemplate` provided by the https://github.com/spring-projects/spring-retry[Spring Retry] library. +If retries are unsuccessful it is up to the error handling mechanism which may _drop_ the message, _re-queue_ the message for re-processing or _send the failed message to DLQ_. + +Both Rabbit and Kafka support these concepts (especially DLQ). However, other binders may not, so refer to your individual binder’s documentation for details on supported +error-handling options. + +Keep in mind however, the _reactive function_ does NOT qualify as a Message handler, since it does not handle individual messages and +instead provides a way to connect stream (i.e., Flux) provided by the framework with the one provided by the user. +***Why is this important?*** +That is because anything you read later in this section with regard to Retry Template, dropping failed messages, retrying, +DLQ and configuration properties that assist with all of it ***only*** applies to Message handlers (i.e., imperative functions). + +Reactive API provides a very rich library of its own operators and mechanisms to assist you with error handling specific to +variety of reactive uses cases which are far more complex then simple Message handler cases, So use them, such +as `public final Flux retryWhen(Retry retrySpec);` that you can find in `reactor.core.publisher.Flux`. + +[source,java] +---- +@Bean +public Function, Flux> uppercase() { + return flux -> flux + .retryWhen(Retry.backoff(3, Duration.ofMillis(1000))) + .map(v -> v.toUpperCase()); +} +---- + +[[drop-failed-messages]] +== Drop Failed Messages + +By default, the system provides error handlers. The first error handler will simply log error message. The second error handler is binder specific error handler +which is responsible for handling error message in the context of a specific messaging system (e.g., send to DLQ). But since no additional error handling configuration was provided (in this current scenario) this handler will not do anything. So essentially after being logged, the message will be dropped. + +While acceptable in some cases, for most cases, it is not, and we need some recovery mechanism to avoid message loss. + +[[handle-error-messages]] +== Handle Error Messages + +In the previous section we mentioned that by default messages that resulted in error are effectively logged and dropped. The framework also exposes mechanism for you +to provide custom error handler (i.e., to send notification or write to database, etc). You can do so by adding `Consumer` that is specifically designed to accept `ErrorMessage` which aside form all the information about the error (e.g., stack trace etc) contains the original message (the one that triggered the error). +NOTE: Custom error handler is mutually exclusive with framework provided error handlers (i.e., logging and binder error handler - see previous section) to ensure that they do not interfere. + +[source,java] +---- +@Bean +public Consumer myErrorHandler() { + return v -> { + // send SMS notification code + }; +} +---- + +To identify such consumer as an error handler all you need is to provide `error-handler-definition` property pointing to the function name - `spring.cloud.stream.bindings..error-handler-definition=myErrorHandler`. + +For example, for binding name `uppercase-in-0` the property would look like this: + +[source,text] +---- +spring.cloud.stream.bindings.uppercase-in-0.error-handler-definition=myErrorHandler +---- +And if you used special mapping instruction to map binding to a more readable name - `spring.cloud.stream.function.bindings.uppercase-in-0=upper`, then this property would look like this: +[source,text] +---- +spring.cloud.stream.bindings.upper.error-handler-definition=myErrorHandler. +---- + +NOTE: If by accident you declare such handler as a `Function`, it will still work with the exception that nothing is going to be done with its output. However, given that such handler is still relying on functionality provided by Spring Cloud Function, you can also benefit from function composition in the event your handler has some complexity which you would like to address through function composition (however unlikely). + + +***Default Error Handler*** + +If you want to have a single error handler for all function beans, you can use the standard spring-cloud-stream mechanism for defining default properties `spring.cloud.stream.default.error-handler-definition=myErrorHandler` + + +[[dlq-dead-letter-queue]] +== DLQ - Dead Letter Queue + +Perhaps the most common mechanism, DLQ allows failed messages to be sent to a special destination: the _Dead Letter Queue_. + +When configured, failed messages are sent to this destination for subsequent re-processing or auditing and reconciliation. + +Consider the following example: + +[source,java] +---- +@SpringBootApplication +public class SimpleStreamApplication { + + public static void main(String[] args) throws Exception { + SpringApplication.run(SimpleStreamApplication.class, + "--spring.cloud.function.definition=uppercase", + "--spring.cloud.stream.bindings.uppercase-in-0.destination=uppercase", + "--spring.cloud.stream.bindings.uppercase-in-0.group=myGroup", + "--spring.cloud.stream.rabbit.bindings.uppercase-in-0.consumer.auto-bind-dlq=true" + ); + } + + @Bean + public Function uppercase() { + return personIn -> { + throw new RuntimeException("intentional"); + }); + }; + } +} +---- + +As a reminder, in this example `uppercase-in-0` segment of the property corresponds to the name of the input destination binding. +The `consumer` segment indicates that it is a consumer property. + +NOTE: When using DLQ, at least the `group` property must be provided for proper naming of the DLQ destination. However `group` is often used together +with `destination` property, as in our example. + + +Aside from some standard properties we also set the `auto-bind-dlq` to instruct the binder to create and configure DLQ destination for +`uppercase-in-0` binding which corresponds to `uppercase` destination (see corresponding property), which results in an additional Rabbit queue named `uppercase.myGroup.dlq` (see Kafka documentation for Kafka specific DLQ properties). + +Once configured, all failed messages are routed to this destination preserving the original message for further actions. + +And you can see that the error message contains more information relevant to the original error, as follows: + +[source,text] +---- +. . . . +x-exception-stacktrace: org.springframework.messaging.MessageHandlingException: nested exception is + org.springframework.messaging.MessagingException: has an error, failedMessage=GenericMessage [payload=byte[15], + headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_receivedRoutingKey=input.hello, amqp_deliveryTag=1, + deliveryAttempt=3, amqp_consumerQueue=input.hello, amqp_redelivered=false, id=a15231e6-3f80-677b-5ad7-d4b1e61e486e, + amqp_consumerTag=amq.ctag-skBFapilvtZhDsn0k3ZmQg, contentType=application/json, timestamp=1522327846136}] + at org.spring...integ...han...MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:107) + at. . . . . +Payload: blah +---- + +You can also facilitate immediate dispatch to DLQ (without re-tries) by setting `max-attempts` to '1'. For example, +[source,text] +---- +--spring.cloud.stream.bindings.uppercase-in-0.consumer.max-attempts=1 +---- + +[[retry-template]] +== Retry Template + +In this section we cover configuration properties relevant to configuration of retry capabilities. + +The `RetryTemplate` is part of the https://github.com/spring-projects/spring-retry[Spring Retry] library. +While it is out of scope of this document to cover all of the capabilities of the `RetryTemplate`, we +will mention the following consumer properties that are specifically related to +the `RetryTemplate`: + +maxAttempts:: +The number of attempts to process the message. ++ +Default: 3. +backOffInitialInterval:: +The backoff initial interval on retry. ++ +Default 1000 milliseconds. +backOffMaxInterval:: +The maximum backoff interval. ++ +Default 10000 milliseconds. +backOffMultiplier:: +The backoff multiplier. ++ +Default 2.0. +defaultRetryable:: +Whether exceptions thrown by the listener that are not listed in the `retryableExceptions` are retryable. ++ +Default: `true`. +retryableExceptions:: +A map of Throwable class names in the key and a boolean in the value. +Specify those exceptions (and subclasses) that will or won't be retried. +Also see `defaultRetriable`. +Example: `spring.cloud.stream.bindings.input.consumer.retryable-exceptions.java.lang.IllegalStateException=false`. ++ +Default: empty. + +While the preceding settings are sufficient for the majority of the customization requirements, they may not satisfy certain complex requirements, at which +point you may want to provide your own instance of the `RetryTemplate`. To do so configure it as a bean in your application configuration. The application provided +instance will override the one provided by the framework. Also, to avoid conflicts you must qualify the instance of the `RetryTemplate` you want to be used by the binder +as `@StreamRetryTemplate`. For example, + +[source,java] +---- +@StreamRetryTemplate +public RetryTemplate myRetryTemplate() { + return new RetryTemplate(); +} +---- +As you can see from the above example you don't need to annotate it with `@Bean` since `@StreamRetryTemplate` is a qualified `@Bean`. + +If you need to be more precise with your `RetryTemplate`, you can specify the bean by name in your `ConsumerProperties` to associate +the specific retry bean per binding. + +[source] +---- +spring.cloud.stream.bindings..consumer.retry-template-name= +---- + + +[[spring-cloud-stream-overview-binders]] += Binders + +Spring Cloud Stream provides a Binder abstraction for use in connecting to physical destinations at the external middleware. +This section provides information about the main concepts behind the Binder SPI, its main components, and implementation-specific details. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-instance-index-instance-count.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-instance-index-instance-count.adoc new file mode 100644 index 000000000..686735be5 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-instance-index-instance-count.adoc @@ -0,0 +1,12 @@ +[[spring-cloud-stream-overview-instance-index-instance-count]] += Instance Index and Instance Count + +When scaling up Spring Cloud Stream applications, each instance can receive information about how many other instances of the same application exist and what its own instance index is. +Spring Cloud Stream does this through the `spring.cloud.stream.instanceCount` and `spring.cloud.stream.instanceIndex` properties. +For example, if there are three instances of a HDFS sink application, all three instances have `spring.cloud.stream.instanceCount` set to `3`, and the individual applications have `spring.cloud.stream.instanceIndex` set to `0`, `1`, and `2`, respectively. + +When Spring Cloud Stream applications are deployed through Spring Cloud Data Flow, these properties are configured automatically; when Spring Cloud Stream applications are launched independently, these properties must be set correctly. +By default, `spring.cloud.stream.instanceCount` is `1`, and `spring.cloud.stream.instanceIndex` is `0`. + +In a scaled-up scenario, correct configuration of these two properties is important for addressing partitioning behavior (see below) in general, and the two properties are always required by certain binders (for example, the Kafka binder) in order to ensure that data are split correctly across multiple consumer instances. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-partitioning.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-partitioning.adoc new file mode 100644 index 000000000..553a9c7e0 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-partitioning.adoc @@ -0,0 +1,91 @@ +[[spring-cloud-stream-overview-partitioning]] += Partitioning + +Partitioning in Spring Cloud Stream consists of two tasks: + +* `<>` +* `<>` + +[[spring-cloud-stream-overview-configuring-output-bindings-partitioning]] +== Configuring Output Bindings for Partitioning + +You can configure an output binding to send partitioned data by setting one and only one of its `partitionKeyExpression` or `partitionKeyExtractorName` properties, as well as its `partitionCount` property. + +For example, the following is a valid and typical configuration: + +---- +spring.cloud.stream.bindings.func-out-0.producer.partitionKeyExpression=headers.id +spring.cloud.stream.bindings.func-out-0.producer.partitionCount=5 +---- + +Based on that example configuration, data is sent to the target partition by using the following logic. + +A partition key's value is calculated for each message sent to a partitioned output binding based on the `partitionKeyExpression`. +The `partitionKeyExpression` is a SpEL expression that is evaluated against the outbound message (in the preceding example it's the value of the `id` from message headers) for extracting the partitioning key. + +If a SpEL expression is not sufficient for your needs, you can instead calculate the partition key value by providing an implementation of `org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy` and configuring it as a bean (by using the `@Bean` annotation). +If you have more then one bean of type `org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy` available in the Application Context, you can further filter it by specifying its name with the `partitionKeyExtractorName` property, as shown in the following example: + +[source] +---- +--spring.cloud.stream.bindings.func-out-0.producer.partitionKeyExtractorName=customPartitionKeyExtractor +--spring.cloud.stream.bindings.func-out-0.producer.partitionCount=5 +. . . +@Bean +public CustomPartitionKeyExtractorClass customPartitionKeyExtractor() { + return new CustomPartitionKeyExtractorClass(); +} +---- + +NOTE: In previous versions of Spring Cloud Stream, you could specify the implementation of `org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy` by setting the `spring.cloud.stream.bindings.output.producer.partitionKeyExtractorClass` property. +Since version 3.0, this property is removed. + +Once the message key is calculated, the partition selection process determines the target partition as a value between `0` and `partitionCount - 1`. +The default calculation, applicable in most scenarios, is based on the following formula: `key.hashCode() % partitionCount`. +This can be customized on the binding, either by setting a SpEL expression to be evaluated against the 'key' (through the `partitionSelectorExpression` property) or by configuring an implementation of `org.springframework.cloud.stream.binder.PartitionSelectorStrategy` as a bean (by using the @Bean annotation). +Similar to the `PartitionKeyExtractorStrategy`, you can further filter it by using the `spring.cloud.stream.bindings.output.producer.partitionSelectorName` property when more than one bean of this type is available in the Application Context, as shown in the following example: + +[source] +---- +--spring.cloud.stream.bindings.func-out-0.producer.partitionSelectorName=customPartitionSelector +. . . +@Bean +public CustomPartitionSelectorClass customPartitionSelector() { + return new CustomPartitionSelectorClass(); +} +---- + +NOTE: In previous versions of Spring Cloud Stream you could specify the implementation of `org.springframework.cloud.stream.binder.PartitionSelectorStrategy` by setting the `spring.cloud.stream.bindings.output.producer.partitionSelectorClass` property. +Since version 3.0, this property is removed. + +[[spring-cloud-stream-overview-configuring-input-bindings-partitioning]] +== Configuring Input Bindings for Partitioning + +An input binding (with the binding name `uppercase-in-0`) is configured to receive partitioned data by setting its `partitioned` +property, as well as the `instanceIndex` and `instanceCount` properties on the application itself, as shown in the following example: + +---- +spring.cloud.stream.bindings.uppercase-in-0.consumer.partitioned=true +spring.cloud.stream.instanceIndex=3 +spring.cloud.stream.instanceCount=5 +---- + +The `instanceCount` value represents the total number of application instances between which the data should be partitioned. +The `instanceIndex` must be a unique value across the multiple instances, with a value between `0` and `instanceCount - 1`. +The instance index helps each application instance to identify the unique partition(s) from which it receives data. +It is required by binders using technology that does not support partitioning natively. +For example, with RabbitMQ, there is a queue for each partition, with the queue name containing the instance index. +With Kafka, if `autoRebalanceEnabled` is `true` (default), Kafka takes care of distributing partitions across instances, and these properties are not required. +If `autoRebalanceEnabled` is set to false, the `instanceCount` and `instanceIndex` are used by the binder to determine which partition(s) the instance subscribes to (you must have at least as many partitions as there are instances). +The binder allocates the partitions instead of Kafka. +This might be useful if you want messages for a particular partition to always go to the same instance. +When a binder configuration requires them, it is important to set both values correctly in order to ensure that all of the data is consumed and that the application instances receive mutually exclusive datasets. + +While a scenario in which using multiple instances for partitioned data processing may be complex to set up in a standalone case, Spring Cloud Dataflow can simplify the process significantly by populating both the input and output values correctly and by letting you rely on the runtime infrastructure to provide information about the instance index and instance count. + +[[testing]] += Testing + +Spring Cloud Stream provides support for testing your microservice applications without connecting to a messaging system. + + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-persistent-publish-subscribe-support.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-persistent-publish-subscribe-support.adoc new file mode 100644 index 000000000..2ba45b0fc --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-persistent-publish-subscribe-support.adoc @@ -0,0 +1,21 @@ +[[spring-cloud-stream-overview-persistent-publish-subscribe-support]] += Persistent Publish-Subscribe Support + +Communication between applications follows a publish-subscribe model, where data is broadcast through shared topics. +This can be seen in the following figure, which shows a typical deployment for a set of interacting Spring Cloud Stream applications. + +.Spring Cloud Stream Publish-Subscribe +image::SCSt-sensors.png[width=800,scaledwidth="75%",align="center"] + +Data reported by sensors to an HTTP endpoint is sent to a common destination named `raw-sensor-data`. +From the destination, it is independently processed by a microservice application that computes time-windowed averages and by another microservice application that ingests the raw data into HDFS (Hadoop Distributed File System). +In order to process the data, both applications declare the topic as their input at runtime. + +The publish-subscribe communication model reduces the complexity of both the producer and the consumer and lets new applications be added to the topology without disruption of the existing flow. +For example, downstream from the average-calculating application, you can add an application that calculates the highest temperature values for display and monitoring. +You can then add another application that interprets the same flow of averages for fault detection. +Doing all communication through shared topics rather than point-to-point queues reduces coupling between microservices. + +While the concept of publish-subscribe messaging is not new, Spring Cloud Stream takes the extra step of making it an opinionated choice for its application model. +By using native middleware support, Spring Cloud Stream also simplifies use of the publish-subscribe model across different platforms. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/overview-user-defined-message-converters.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/overview-user-defined-message-converters.adoc new file mode 100644 index 000000000..c7470f03d --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/overview-user-defined-message-converters.adoc @@ -0,0 +1,53 @@ +[[spring-cloud-stream-overview-user-defined-message-converters]] += User-defined Message Converters + +Spring Cloud Stream exposes a mechanism to define and register additional `MessageConverter` s. +To use it, implement `org.springframework.messaging.converter.MessageConverter`, configure it as a `@Bean`. +It is then appended to the existing stack of `MessageConverter` s. + +NOTE: It is important to understand that custom `MessageConverter` implementations are added to the head of the existing stack. +Consequently, custom `MessageConverter` implementations take precedence over the existing ones, which lets you override as well as add to the existing converters. + +The following example shows how to create a message converter bean to support a new content type called `application/bar`: + +[source,java] +---- +@SpringBootApplication +public static class SinkApplication { + + ... + + @Bean + public MessageConverter customMessageConverter() { + return new MyCustomMessageConverter(); + } +} + +public class MyCustomMessageConverter extends AbstractMessageConverter { + + public MyCustomMessageConverter() { + super(new MimeType("application", "bar")); + } + + @Override + protected boolean supports(Class clazz) { + return (Bar.class.equals(clazz)); + } + + @Override + protected Object convertFromInternal(Message message, Class targetClass, Object conversionHint) { + Object payload = message.getPayload(); + return (payload instanceof Bar ? payload : new Bar((byte[]) payload)); + } +} +---- + +[[inter-application-communication]] += Inter-Application Communication + +Spring Cloud Stream enables communication between applications. Inter-application communication is a complex issue spanning several concerns, as described in the following topics: + +* `<>` +* `<>` +* `<>` + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/partitioning.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/partitioning.adoc new file mode 100644 index 000000000..333187b0b --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/partitioning.adoc @@ -0,0 +1,29 @@ +[[partitioning]] += Partitioning Support + +Spring Cloud Stream provides support for partitioning data between multiple instances of a given application. +In a partitioned scenario, the physical communication medium (such as the broker topic) is viewed as being structured into multiple partitions. +One or more producer application instances send data to multiple consumer application instances and ensure that data identified by common characteristics are processed by the same consumer instance. + +Spring Cloud Stream provides a common abstraction for implementing partitioned processing use cases in a uniform fashion. +Partitioning can thus be used whether the broker itself is naturally partitioned (for example, Kafka) or not (for example, RabbitMQ). + +.Spring Cloud Stream Partitioning +image::SCSt-partitioning.png[width=800,scaledwidth="75%",align="center"] + +Partitioning is a critical concept in stateful processing, where it is critical (for either performance or consistency reasons) to ensure that all related data is processed together. +For example, in the time-windowed average calculation example, it is important that all measurements from any given sensor are processed by the same application instance. + +NOTE: To set up a partitioned processing scenario, you must configure both the data-producing and the data-consuming ends. + +[[programming-model]] += Programming Model + +To understand the programming model, you should be familiar with the following core concepts: + +* *Destination Binders:* Components responsible to provide integration with the external messaging systems. +* *Bindings:* Bridge between the external messaging systems and application provided _Producers_ and _Consumers_ of messages (created by the Destination Binders). +* *Message:* The canonical data structure used by producers and consumers to communicate with Destination Binders (and thus other applications via external messaging systems). + +image::SCSt-overview.png[width=800,scaledwidth="75%",align="center"] + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/post-processing-after-sending-message.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/post-processing-after-sending-message.adoc new file mode 100644 index 000000000..0b1930efa --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/post-processing-after-sending-message.adoc @@ -0,0 +1,76 @@ +[[post-processing-after-sending-message]] += Post processing (after sending message) + +Once function is invoked, its result is sent by the framework to a target destination which effectively completes function invocation cycle. + +However such cycle may not be fully complete from the business standpoint until some additional tasks are performed *after* completion of this cycle. +While this could be accomplished with a simple combination of `Consumer` and `StreamBridge` as described in this +https://stackoverflow.com/questions/75917883/post-processing-after-spring-cloud-stream-function[Stack Overflow post], since version 4.0.3 the framework +provides a more idiomatic approach to solve this issue via `PostProcessingFunction` provided by Spring Cloud Function project. +The `PostProcessingFunction` is a special semi-marker function which contains one additional method `postProcess(Message>)` designed +to provide a place for implementing such post processing task. + +---- +package org.springframework.cloud.function.context +. . . +public interface PostProcessingFunction extends Function { + default void postProcess(Message result) { + } +} +---- + +So, now you have two options. + +Option 1: You can implement your function as `PostProcessingFunction` and also include the additional post processing behavior by implementing its `postProcess(Message>)` method. + +---- +private static class Uppercase implements PostProcessingFunction { + + @Override + public String apply(String input) { + return input.toUpperCase(); + } + + @Override + public void postProcess(Message result) { + System.out.println("Function Uppercase has been successfully invoked and its result successfully sent to target destination"); + } +} +. . . +@Bean +public Function uppercase() { + return new Uppercase(); +} +---- + +Option 2: If you already have an existing function and don't want to change its implementation or want to keep your function as POJO, you can simply implement only `postProcess(Message>)` method and compose this new post processing function with your other function. +---- +private static class Logger implements PostProcessingFunction { + + @Override + public void postProcess(Message result) { + System.out.println("Function has been successfully invoked and its result successfully sent to target destination"); + } +} +. . . +@Bean +public Function uppercase() { + return v -> v.toUpperCase(); +} +@Bean +public Function logger() { + return new Logger(); +} +. . . +// and then have your function definition as such `uppercase|logger` +---- + +NOTE: +In case of function composition only the last instance of `PostProcessingFunction` (if present) will take effect. For example, let's say you have the +following function definition - `foo|bar|baz` and both `foo` and `baz` are instances of `PostProcessingFunction`. Only `baz.postProcess(Message>)` will be invoked. +If `baz` is not an instance of `PostProcessingFunction`, then no post processing functionality will be performed. + +One may argue that you can easily do that via function composition by simply composing a post-processor as just another `Function`. That is indeed a possibility however +the post processing functionality in this case will be invoked right after invocation of the previous function and before the message is sent to a target destination +which is before the function invocation cycle is complete. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/producers-and-consumers.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/producers-and-consumers.adoc new file mode 100644 index 000000000..b600d55cf --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/producers-and-consumers.adoc @@ -0,0 +1,18 @@ +[[producers-and-consumers]] += Producers and Consumers + +The following image shows the general relationship of producers and consumers: + +.Producers and Consumers +image::producers-consumers.png[width=800,scaledwidth="75%",align="center"] + +A producer is any component that sends messages to a binding destination. +The binding destination can be bound to an external message broker with a `Binder` implementation for that broker. +When invoking the `bindProducer()` method, the first parameter is the name of the destination within the broker, the second parameter is the instance if local destination to which the producer sends messages, and the third parameter contains properties (such as a partition key expression) to be used within the adapter that is created for that binding destination. + +A consumer is any component that receives messages from the binding destination. +As with a producer, the consumer can be bound to an external message broker. +When invoking the `bindConsumer()` method, the first parameter is the destination name, and a second parameter provides the name of a logical group of consumers. +Each group that is represented by consumer bindings for a given destination receives a copy of each message that a producer sends to that destination (that is, it follows normal publish-subscribe semantics). +If there are multiple consumer instances bound with the same group name, then messages are load-balanced across those consumer instances so that each message sent by a producer is consumed by only a single consumer instance within each group (that is, it follows normal queueing semantics). + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/producing-and-consuming-messages.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/producing-and-consuming-messages.adoc new file mode 100644 index 000000000..32ec9825b --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/producing-and-consuming-messages.adoc @@ -0,0 +1,973 @@ +[[producing-and-consuming-messages]] += Producing and Consuming Messages + +You can write a Spring Cloud Stream application by simply writing functions and exposing them as `@Bean` s. +You can also use Spring Integration annotations based configuration or +Spring Cloud Stream annotation based configuration, although starting with spring-cloud-stream 3.x +we recommend using functional implementations. + +[[spring_cloud_function]] +== Spring Cloud Function support + +[[overview]] +=== Overview + +Since Spring Cloud Stream v2.1, another alternative for defining _stream handlers_ and _sources_ is to use build-in +support for https://cloud.spring.io/spring-cloud-function/[Spring Cloud Function] where they can be expressed as beans of + type `java.util.function.[Supplier/Function/Consumer]`. + +To specify which functional bean to bind to the external destination(s) exposed by the bindings, +you must provide `spring.cloud.function.definition` property. + +NOTE: In the event you only have single bean of type `java.util.function.[Supplier/Function/Consumer]`, you can +skip the `spring.cloud.function.definition` property, since such functional bean will be auto-discovered. However, +it is considered best practice to use such property to avoid any confusion. +Some time this auto-discovery can get in the way, since single bean of type `java.util.function.[Supplier/Function/Consumer]` +could be there for purposes other then handling messages, yet being single it is auto-discovered and auto-bound. +For these rare scenarios you can disable auto-discovery by providing `spring.cloud.stream.function.autodetect` property with value set to `false`. + +Here is the example of the application exposing message handler as `java.util.function.Function` effectively supporting +_pass-thru_ semantics by acting as consumer and producer of data. +[source,java] +---- +@SpringBootApplication +public class MyFunctionBootApp { + + public static void main(String[] args) { + SpringApplication.run(MyFunctionBootApp.class); + } + + @Bean + public Function toUpperCase() { + return s -> s.toUpperCase(); + } +} +---- +In the preceding example, we define a bean of type `java.util.function.Function` called _toUpperCase_ to be acting as message handler +whose 'input' and 'output' must be bound to the external destinations exposed by the provided destination binder. +By default the 'input' and 'output' binding names will be `toUpperCase-in-0` and `toUpperCase-out-0`. +Please see <> section for details on naming convention used to establish binding names. + +Below are the examples of simple functional applications to support other semantics: + +Here is the example of a _source_ semantics exposed as `java.util.function.Supplier` +[source,java] +---- +@SpringBootApplication +public static class SourceFromSupplier { + + @Bean + public Supplier date() { + return () -> new Date(12345L); + } +} +---- + +Here is the example of a _sink semantics_ exposed as `java.util.function.Consumer` +[source,java] +---- +@SpringBootApplication +public static class SinkFromConsumer { + + @Bean + public Consumer sink() { + return System.out::println; + } +} +---- + +[[suppliers-sources]] +=== Suppliers (Sources) + +`Function` and `Consumer` are pretty straightforward when it comes to how their invocation is triggered. They are triggered based +on data (events) sent to the destination they are bound to. In other words, they are classic event-driven components. + +However, `Supplier` is in its own category when it comes to triggering. Since it is, by definition, the source (the origin) of the data, it does not +subscribe to any in-bound destination and, therefore, has to be triggered by some other mechanism(s). +There is also a question of `Supplier` implementation, which could be _imperative_ or _reactive_ and which directly relates to the triggering of such suppliers. + +Consider the following sample: + +[source,java] +---- +@SpringBootApplication +public static class SupplierConfiguration { + + @Bean + public Supplier stringSupplier() { + return () -> "Hello from Supplier"; + } +} +---- + +The preceding `Supplier` bean produces a string whenever its `get()` method is invoked. However, who invokes this method and how often? +The framework provides a default polling mechanism (answering the question of "Who?") that will trigger the invocation of the supplier and by default it will do so +every second (answering the question of "How often?"). +In other words, the above configuration produces a single message every second and each message is sent to an `output` destination that is exposed by the binder. +To learn how to customize the polling mechanism, see <> section. + +Consider a different example: + +[source,java] +---- +@SpringBootApplication +public static class SupplierConfiguration { + + @Bean + public Supplier> stringSupplier() { + return () -> Flux.fromStream(Stream.generate(new Supplier() { + @Override + public String get() { + try { + Thread.sleep(1000); + return "Hello from Supplier"; + } catch (Exception e) { + // ignore + } + } + })).subscribeOn(Schedulers.elastic()).share(); + } +} +---- + +The preceding `Supplier` bean adopts the reactive programming style. Typically, and unlike the imperative supplier, +it should be triggered only once, given that the invocation of its `get()` method produces (supplies) the continuous stream of messages and not an +individual message. + +The framework recognizes the difference in the programming style and guarantees that such a supplier is triggered only once. + +However, imagine the use case where you want to poll some data source and return a finite stream of data representing the result set. +The reactive programming style is a perfect mechanism for such a Supplier. However, given the finite nature of the produced stream, +such Supplier still needs to be invoked periodically. + +Consider the following sample, which emulates such use case by producing a finite stream of data: + +[source,java] +---- +@SpringBootApplication +public static class SupplierConfiguration { + + @PollableBean + public Supplier> stringSupplier() { + return () -> Flux.just("hello", "bye"); + } +} +---- + +The bean itself is annotated with `PollableBean` annotation (sub-set of `@Bean`), thus signaling to the framework that although the implementation +of such a supplier is reactive, it still needs to be polled. + +NOTE: There is a `splittable` attribute defined in `PollableBean` which signals to the post processors of this annotation +that the result produced by the annotated component has to be split and is set to `true` by default. It means that +the framework will split the returning sending out each item as an individual message. If this is not +he desired behavior you can set it to `false` at which point such supplier will simply return +the produced Flux without splitting it. + +[[supplier-threading]] +==== Supplier & threading + +NOTE: As you have learned by now, unlike `Function` and `Consumer`, which are triggered by an event (they have input data), `Supplier` does not have +any input and thus triggered by a different mechanism - _poller_, which may have an unpredictable threading mechanism. And while the details of the +threading mechanism most of the time are not relevant to the downstream execution of the function it may present an issue in certain cases +especially with integrated frameworks that may have certain expectations to thread affinity. For example, https://spring.io/projects/spring-cloud-sleuth[Spring Cloud Sleuth] which relies +on tracing data stored in thread local. +For those cases we have another mechanism via `StreamBridge`, where user has more control over threading mechanism. You can get more details +in <> section. + +[[consumer-reactive]] +=== Consumer (Reactive) + +Reactive `Consumer` is a little bit special because it has a void return type, leaving framework with no reference to subscribe to. +Most likely you will not need to write `Consumer>`, and instead write it as a `Function, Mono>` invoking `then` +operator as the last operator on your stream. + +For example: + +[source,java] +---- +public Function, Mono> consumer() { + return flux -> flux.map(..).filter(..).then(); +} +---- + +But if you do need to write an explicit `Consumer>`, remember to subscribe to the incoming Flux. + + +Also, keep in mind that the same rule applies for function composition when mixing reactive and imperative functions. +Spring Cloud Function indeed supports composing reactive functions with imperative, however you must be aware of certain limitations. +For example, assume you have composed reactive function with imperative consumer. +The result of such composition is a reactive `Consumer`. However, there is no way to subscribe to such consumer as discussed earlier in this section, +so this limitation can only be addressed by either making your consumer reactive and subscribing manually (as discussed earlier), or changing your function to be imperative. + + +[[polling-configuration-properties]] +==== Polling Configuration Properties + +The following properties are exposed by Spring Cloud Stream and are prefixed with the `spring.integration.poller.`: + + +fixedDelay:: +Fixed delay for default poller in milliseconds. ++ +Default: 1000L. + +maxMessagesPerPoll:: +Maximum messages for each polling event of the default poller. ++ +Default: 1L. + +cron:: +Cron expression value for the Cron Trigger. ++ +Default: none. + +initialDelay:: +Initial delay for periodic triggers. ++ +Default: 0. + +timeUnit:: +The TimeUnit to apply to delay values. ++ +Default: MILLISECONDS. + +For example `--spring.integration.poller.fixed-delay=2000` sets the poller interval to poll every two seconds. + +[[per-binding-polling-configuration]] +==== Per-binding polling configuration + +The previous section shows how to configure a single default poller that will be applied to all bindings. While it fits well with the model of microservices spring-cloud-stream designed for where each microservice represents a single component (e.g., Supplier) and thus default poller configuration is enough, there are edge cases where +you may have several components that require different polling configurations + +For such cases please use per-binding way of configuring poller. For example, assume you have an output binding `supply-out-0`. In this case you can configure poller for such +binding using `spring.cloud.stream.bindings.supply-out-0.producer.poller..` prefix (e.g., `spring.cloud.stream.bindings.supply-out-0.producer.poller.fixed-delay=2000`). + +[[sending-arbitrary-data-to-an-output-e-g-foreign-event-driven-sources]] +=== Sending arbitrary data to an output (e.g. Foreign event-driven sources) + +There are cases where the actual source of data may be coming from the external (foreign) system that is not a binder. For example, the +source of the data may be a classic REST endpoint. How do we bridge such source with the functional mechanism used by spring-cloud-stream? + +Spring Cloud Stream provides two mechanisms, so let's look at them in more details + +Here, for both samples we'll use a standard MVC endpoint method called `delegateToSupplier` bound to the root web context, +delegating incoming requests to stream via StreamBridge mechanism. + + +[source, java] +---- +@SpringBootApplication +@Controller +public class WebSourceApplication { + + public static void main(String[] args) { + SpringApplication.run(WebSourceApplication.class, "--spring.cloud.stream.output-bindings=toStream"); + } + + @Autowired + private StreamBridge streamBridge; + + @RequestMapping + @ResponseStatus(HttpStatus.ACCEPTED) + public void delegateToSupplier(@RequestBody String body) { + System.out.println("Sending " + body); + streamBridge.send("toStream", body); + } +} +---- + +Here we autowire a `StreamBridge` bean which allows us to send data to an output binding effectively +bridging non-stream application with spring-cloud-stream. Note that preceding example does not have any +source functions defined (e.g., Supplier bean) leaving the framework with no trigger to create source bindings in advance, which would be typical for cases where configuration contains function beans. And that is fine, since `StreamBridge` will initiate creation of output bindings (as well as +destination auto-provisioning if necessary) for non existing bindings on the first call to its `send(..)` operation caching it for +subsequent reuse (see <> for more details). + +However, if you want to pre-create an output binding at the initialization (startup) time you can benefit from `spring.cloud.stream.output-bindings` property where you can declare the name of your sources. +The provided name will be used as a trigger to create a source binding. +You can use `;` to signify multiple sources (multiple output bindings) +(e.g., `--spring.cloud.stream.output-bindings=foo;bar`) + +Also, note that `streamBridge.send(..)` method takes an `Object` for data. This means you can send POJO or `Message` to it and it +will go through the same routine when sending output as if it was from any Function or Supplier providing the same level +of consistency as with functions. This means the output type conversion, partitioning etc are honored as if it was from the output produced by functions. + +[[streambridge-and-dynamic-destinations]] +==== StreamBridge and Dynamic Destinations + +`StreamBridge` can also be used for cases when output destination(s) are not known ahead of time similar to the use cases +described in <> section. + +Let's look at the example + +[source, java] +---- +@SpringBootApplication +@Controller +public class WebSourceApplication { + + public static void main(String[] args) { + SpringApplication.run(WebSourceApplication.class, args); + } + + @Autowired + private StreamBridge streamBridge; + + @RequestMapping + @ResponseStatus(HttpStatus.ACCEPTED) + public void delegateToSupplier(@RequestBody String body) { + System.out.println("Sending " + body); + streamBridge.send("myDestination", body); + } +} +---- + +As you can see the preceding example is very similar to the previous one with the exception of explicit binding instruction provided via +`spring.cloud.stream.output-bindings` property (which is not provided). +Here we're sending data to `myDestination` name which does not exist as a binding. Therefore such name will be treated as dynamic destination +as described in <> section. + + +In the preceding example, we are using `ApplicationRunner` as a _foreign source_ to feed the stream. + + +A more practical example, where the foreign source is REST endpoint. + +[source, java] +---- +@SpringBootApplication +@Controller +public class WebSourceApplication { + + public static void main(String[] args) { + SpringApplication.run(WebSourceApplication.class); + } + + @Autowired + private StreamBridge streamBridge; + + @RequestMapping + @ResponseStatus(HttpStatus.ACCEPTED) + public void delegateToSupplier(@RequestBody String body) { + streamBridge.send("myBinding", body); + } +} +---- + +As you can see inside of `delegateToSupplier` method we're using StreamBridge to send data to `myBinding` binding. And here you're also benefiting from +the dynamic features of `StreamBridge` where if `myBinding` doesn't exist it will be created automatically and cached, otherwise existing binding will be used. + +NOTE: Caching dynamic destinations (bindings) could result in memory leaks in the event there are many dynamic destinations. To have some level of control +we provide a self-evicting caching mechanism for output bindings with default cache size of 10. This means that if your dynamic destination size goes above that number, there is a possibility that an existing binding will be evicted and thus would need to be recreated which could cause minor performance degradation. You can increase the cache size via `spring.cloud.stream.dynamic-destination-cache-size` property setting it to the desired value. + + +---- +curl -H "Content-Type: text/plain" -X POST -d "hello from the other side" http://localhost:8080/ +---- + +By showing two examples we want to emphasize the approach will work with any type of foreign sources. + +NOTE: If you are using the Solace PubSub+ binder, Spring Cloud Stream has reserved the `scst_targetDestination` header (retrievable via BinderHeaders.TARGET_DESTINATION), which allows for messages to be redirected from their bindings' configured destination to the target destination specified by this header. This allows for the binder to manage the resources necessary to publish to dynamic destinations, relieving the framework from having to do so, and avoids the caching issues mentioned in the previous Note. More info https://github.com/SolaceProducts/solace-spring-cloud/tree/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter#dynamic-producer-destinations[here]. + +[[output-content-type-with-streambridge]] +==== Output Content Type with StreamBridge + +You can also provide specific content type if necessary with the following method signature `public boolean send(String bindingName, Object data, MimeType outputContentType)`. +Or if you send data as a `Message`, its content type will be honored. + +[[using-specific-binder-type-with-streambridge]] +==== Using specific binder type with StreamBridge + +Spring Cloud Stream supports multiple binder scenarios. For example you may be receiving data from Kafka and sending it to RabbitMQ. + +For more information on multiple binders scenarios, please see <> section and specifically <> + +In the event you are planning to use StreamBridge and have more then one binder configured in your application you must also tell StreamBridge +which binder to use. And for that there are two more variations of `send` method: + +[source, java] +---- +public boolean send(String bindingName, @Nullable String binderType, Object data) + +public boolean send(String bindingName, @Nullable String binderType, Object data, MimeType outputContentType) +---- + +As you can see there is one additional argument that you can provide - `binderType`, telling BindingService which binder to use when creating dynamic binding. + +NOTE: For cases where `spring.cloud.stream.output-bindings` property is used or the binding was already created under different binder, the `binderType` +argument will have no effect. + +[[using-channel-interceptors-with-streambridge]] +==== Using channel interceptors with StreamBridge + +Since `StreamBridge` uses a `MessageChannel` to establish the output binding, you can activate channel interceptors when sending data through `StreamBridge`. +It is up to the application to decide which channel interceptors to apply on `StreamBridge`. +Spring Cloud Stream does not inject all the channel interceptors detected into `StreamBridge` unless they are annoatated with `@GlobalChannelInterceptor(patterns = "*")`. + +Let us assume that you have the following two different `StreamBridge` bindings in the application. + +`streamBridge.send("foo-out-0", message);` + +and + +`streamBridge.send("bar-out-0", message);` + +Now, if you want a channel interceptor applied on both the `StreamBridge` bindings, then you can declare the following `GlobalChannelInterceptor` bean. + +``` +@Bean +@GlobalChannelInterceptor(patterns = "*") +public ChannelInterceptor customInterceptor() { + return new ChannelInterceptor() { + @Override + public Message preSend(Message message, MessageChannel channel) { + ... + } + }; +} +``` + +However, if you don't like the global approach above and want to have a dedicated interceptor for each binding, then you can do the following. + +``` +@Bean +@GlobalChannelInterceptor(patterns = "foo-*") +public ChannelInterceptor fooInterceptor() { + return new ChannelInterceptor() { + @Override + public Message preSend(Message message, MessageChannel channel) { + ... + } + }; +} +``` + +and + +``` +@Bean +@GlobalChannelInterceptor(patterns = "bar-*") +public ChannelInterceptor barInterceptor() { + return new ChannelInterceptor() { + @Override + public Message preSend(Message message, MessageChannel channel) { + ... + } + }; +} +``` + +You have the flexibility to make the patterns more strict or customized to your business needs. + +With this approach, the application gets the ability to decide which interceptors to inject in `StreamBridge` rather than applying all the available interceptors. + +NOTE: `StreamBridge` provides a contract through the `StreamOperations` interface that contains all the `send` methods of `StreamBridge`. Therefore, applications may choose to autowire using `StreamOperations`. This is handy when it comes to unit testing code that uses `StreamBridge` by providing a mock or similar mechanisms for the `StreamOperations` interface. + +[[reactive-functions-support]] +=== Reactive Functions support + + +Since _Spring Cloud Function_ is build on top of https://projectreactor.io/[Project Reactor] there isn't much you need to do +to benefit from reactive programming model while implementing `Supplier`, `Function` or `Consumer`. + +For example: + +[source,java] +---- +@SpringBootApplication +public static class SinkFromConsumer { + + @Bean + public Function, Flux> reactiveUpperCase() { + return flux -> flux.map(val -> val.toUpperCase()); + } +} +---- + +[IMPORTANT] +==== +Few important things must be understood when choosing reactive or imperative programming model. + +_Fully reactive or just API?_ + +Using reactive API does not necessarily imply that you can benefit from all of the reactive features of such API. In other words things like back-pressure and other advanced features will only work when they are working with compatible system - such as Reactive Kafka binder. In the event you are using regular Kafka or Rabbit or any other non-reactive binder, you can only benefit from the conveniences of the reactive API itself and not its advanced features, since the actual sources or targets of the stream are not reactive. + + +_Error handling and retries_ + +Throughout this manual you will see several reference on the framework-based error handling, retries and other features as well as configuration properties associated with them. It is important to understand that they only effect the imperative functions and you should NOT have the same expectations when it comes to reactive functions. And here is why. . . +There is a fundamental difference between reactive and imperative functions. +Imperative function is a _message handler_ that is invoked by the framework on each message it receives. So for N messages there will be N invocations of such function and because of that we can wrap such function and add additional functionality such as error handling, retries etc. +Reactive function is _initialization function_. It is invoked only once to get a reference to a Flux/Mono provided by the user to be connected with the one provided by the framework. After that we (the framework) have absolutely no visibility nor control of the stream. +Therefore, with reactive functions you must rely on the richness of the reactive API when it comes to error handling and retries (i.e., `doOnError()`, `.onError*()` etc). +==== + +[[functional-composition]] +=== Functional Composition + +Using functional programming model you can also benefit from functional composition where you can dynamically compose complex handlers from a set of simple functions. +As an example let's add the following function bean to the application defined above +[source,java] +---- +@Bean +public Function wrapInQuotes() { + return s -> "\"" + s + "\""; +} +---- +and modify the `spring.cloud.function.definition` property to reflect your intention to compose a new function from both ‘toUpperCase’ and ‘wrapInQuotes’. +To do so Spring Cloud Function relies on `|` (pipe) symbol. So, to finish our example our property will now look like this: + +[source,java] +---- +--spring.cloud.function.definition=toUpperCase|wrapInQuotes +---- + +NOTE: One of the great benefits of functional composition support provided by _Spring Cloud Function_ is +the fact that you can compose _reactive_ and _imperative_ functions. + +The result of a composition is a single function which, as you may guess, could have a very long and rather cryptic name (e.g., `foo|bar|baz|xyz. . .`) +presenting a great deal of inconvenience when it comes to other configuration properties. This is where _descriptive binding names_ +feature described in <> section can help. + +For example, if we want to give our `toUpperCase|wrapInQuotes` a more descriptive name we can do so +with the following property `spring.cloud.stream.function.bindings.toUpperCase|wrapInQuotes-in-0=quotedUpperCaseInput` allowing +other configuration properties to refer to that binding name (e.g., `spring.cloud.stream.bindings.quotedUpperCaseInput.destination=myDestination`). + +[[functional-composition-and-cross-cutting-concerns]] +==== Functional Composition and Cross-cutting Concerns + +Function composition effectively allows you to address complexity by breaking it down +to a set of simple and individually manageable/testable components that could still be +represented as one at runtime. But that is not the only benefit. + +You can also use composition to address certain cross-cutting non-functional concerns, +such as content enrichment. For example, assume you have an incoming message that may +be lacking certain headers, or some headers are not in the exact state your business +function would expect. You can now implement a separate function that addresses those +concerns and then compose it with the main business function. + +Let’s look at the example + +[source,java] +---- +@SpringBootApplication +public class DemoStreamApplication { + + public static void main(String[] args) { + SpringApplication.run(DemoStreamApplication.class, + "--spring.cloud.function.definition=enrich|echo", + "--spring.cloud.stream.function.bindings.enrich|echo-in-0=input", + "--spring.cloud.stream.bindings.input.destination=myDestination", + "--spring.cloud.stream.bindings.input.group=myGroup"); + + } + + @Bean + public Function, Message> enrich() { + return message -> { + Assert.isTrue(!message.getHeaders().containsKey("foo"), "Should NOT contain 'foo' header"); + return MessageBuilder.fromMessage(message).setHeader("foo", "bar").build(); + }; + } + + @Bean + public Function, Message> echo() { + return message -> { + Assert.isTrue(message.getHeaders().containsKey("foo"), "Should contain 'foo' header"); + System.out.println("Incoming message " + message); + return message; + }; + } +} +---- + +While trivial, this example demonstrates how one function enriches the incoming Message with the additional header(s) (non-functional concern), +so the other function - `echo` - can benefit form it. The `echo` function stays clean and focused on business logic only. +You can also see the usage of `spring.cloud.stream.function.bindings` property to simplify composed binding name. + +[[functions-with-multiple-input-and-output-arguments]] +=== Functions with multiple input and output arguments + +Starting with version 3.0 spring-cloud-stream provides support for functions that +have multiple inputs and/or multiple outputs (return values). What does this actually mean and +what type of use cases it is targeting? + +* _Big Data: Imagine the source of data you're dealing with is highly un-organized and contains various types of data elements +(e.g., orders, transactions etc) and you effectively need to sort it out._ +* _Data aggregation: Another use case may require you to merge data elements from 2+ incoming _streams_. + +The above describes just a few use cases where you may need to use a single function to accept and/or produce +multiple _streams_ of data. And that is the type of use cases we are targeting here. + +Also, note a slightly different emphasis on the concept of _streams_ here. The assumption is that such functions are only valuable +if they are given access to the actual streams of data (not the individual elements). So for that we are relying on +abstractions provided by https://projectreactor.io/[Project Reactor] (i.e., `Flux` and `Mono`) which is already available on the +classpath as part of the dependencies brought in by spring-cloud-functions. + +Another important aspect is representation of multiple input and outputs. While java provides +variety of different abstractions to represent _multiple of something_ those abstractions +are _a) unbounded_, _b) lack arity_ and _c) lack type information_ which are all important in this context. +As an example, let's look at `Collection` or an array which only allows us to +describe _multiple_ of a single type or up-cast everything to an `Object`, affecting the transparent type conversion feature of +spring-cloud-stream and so on. + +So to accommodate all these requirements the initial support is relying on the signature which utilizes another abstraction +provided by _Project Reactor_ - Tuples. However, we are working on allowing a more flexible signatures. + +IMPORTANT: Please refer to <> section to understand the naming convention used to establish _binding names_ +used by such application. + +Let's look at the few samples: + +[source,java] +---- +@SpringBootApplication +public class SampleApplication { + + @Bean + public Function, Flux>, Flux> gather() { + return tuple -> { + Flux stringStream = tuple.getT1(); + Flux intStream = tuple.getT2().map(i -> String.valueOf(i)); + return Flux.merge(stringStream, intStream); + }; + } +} +---- + +The above example demonstrates function which takes two inputs (first of type `String` and second of type `Integer`) +and produces a single output of type `String`. + +So, for the above example the two input bindings will be `gather-in-0` and `gather-in-1` and for consistency the +output binding also follows the same convention and is named `gather-out-0`. + + +Knowing that will allow you to set binding specific properties. +For example, the following will override content-type for `gather-in-0` binding: + +---- +--spring.cloud.stream.bindings.gather-in-0.content-type=text/plain +---- + + +[source,java] +---- +@SpringBootApplication +public class SampleApplication { + + @Bean + public static Function, Tuple2, Flux>> scatter() { + return flux -> { + Flux connectedFlux = flux.publish().autoConnect(2); + UnicastProcessor even = UnicastProcessor.create(); + UnicastProcessor odd = UnicastProcessor.create(); + Flux evenFlux = connectedFlux.filter(number -> number % 2 == 0).doOnNext(number -> even.onNext("EVEN: " + number)); + Flux oddFlux = connectedFlux.filter(number -> number % 2 != 0).doOnNext(number -> odd.onNext("ODD: " + number)); + + return Tuples.of(Flux.from(even).doOnSubscribe(x -> evenFlux.subscribe()), Flux.from(odd).doOnSubscribe(x -> oddFlux.subscribe())); + }; + } +} +---- + +The above example is somewhat of a the opposite from the previous sample and demonstrates function which +takes single input of type `Integer` and produces two outputs (both of type `String`). + +So, for the above example the input binding is `scatter-in-0` and the +output bindings are `scatter-out-0` and `scatter-out-1`. + +And you test it with the following code: +[source,java] +---- +@Test +public void testSingleInputMultiOutput() { + try (ConfigurableApplicationContext context = new SpringApplicationBuilder( + TestChannelBinderConfiguration.getCompleteConfiguration( + SampleApplication.class)) + .run("--spring.cloud.function.definition=scatter")) { + + InputDestination inputDestination = context.getBean(InputDestination.class); + OutputDestination outputDestination = context.getBean(OutputDestination.class); + + for (int i = 0; i < 10; i++) { + inputDestination.send(MessageBuilder.withPayload(String.valueOf(i).getBytes()).build()); + } + + int counter = 0; + for (int i = 0; i < 5; i++) { + Message even = outputDestination.receive(0, 0); + assertThat(even.getPayload()).isEqualTo(("EVEN: " + String.valueOf(counter++)).getBytes()); + Message odd = outputDestination.receive(0, 1); + assertThat(odd.getPayload()).isEqualTo(("ODD: " + String.valueOf(counter++)).getBytes()); + } + } +} +---- + +[[multiple-functions-in-a-single-application]] +=== Multiple functions in a single application + +There may also be a need for grouping several message handlers in a single application. You would do so by +defining several functions. + +[source,java] +---- +@SpringBootApplication +public class SampleApplication { + + @Bean + public Function uppercase() { + return value -> value.toUpperCase(); + } + + @Bean + public Function reverse() { + return value -> new StringBuilder(value).reverse().toString(); + } +} +---- + +In the above example we have configuration which defines two functions `uppercase` and `reverse`. +So first, as mentioned before, we need to notice that there is a a conflict (more then one function) and therefore +we need to resolve it by providing `spring.cloud.function.definition` property pointing to the actual function +we want to bind. Except here we will use `;` delimiter to point to both functions (see test case below). + + +IMPORTANT: As with functions with multiple inputs/outputs, please refer to <> section to understand the naming +convention used to establish _binding names_ used by such application. + +And you test it with the following code: +[source,java] +---- +@Test +public void testMultipleFunctions() { + try (ConfigurableApplicationContext context = new SpringApplicationBuilder( + TestChannelBinderConfiguration.getCompleteConfiguration( + ReactiveFunctionConfiguration.class)) + .run("--spring.cloud.function.definition=uppercase;reverse")) { + + InputDestination inputDestination = context.getBean(InputDestination.class); + OutputDestination outputDestination = context.getBean(OutputDestination.class); + + Message inputMessage = MessageBuilder.withPayload("Hello".getBytes()).build(); + inputDestination.send(inputMessage, "uppercase-in-0"); + inputDestination.send(inputMessage, "reverse-in-0"); + + Message outputMessage = outputDestination.receive(0, "uppercase-out-0"); + assertThat(outputMessage.getPayload()).isEqualTo("HELLO".getBytes()); + + outputMessage = outputDestination.receive(0, "reverse-out-1"); + assertThat(outputMessage.getPayload()).isEqualTo("olleH".getBytes()); + } +} +---- + +[[batch-consumers]] +=== Batch Consumers + +When using a `MessageChannelBinder` that supports batch listeners, and the feature is enabled for the consumer binding, you can set `spring.cloud.stream.bindings..consumer.batch-mode` to `true` to enable the +entire batch of messages to be passed to the function in a `List`. + +[source, java] +---- +@Bean +public Function, Person> findFirstPerson() { + return persons -> persons.get(0); +} +---- + +[[batch-producers]] +=== Batch Producers + +You can also use the concept of batching on the producer side by returning a collection of Messages which effectively provides an +inverse effect where each message in the collection will be sent individually by the binder. + +Consider the following function: + +[source, java] +---- +@Bean +public Function>> batch() { + return p -> { + List> list = new ArrayList<>(); + list.add(MessageBuilder.withPayload(p + ":1").build()); + list.add(MessageBuilder.withPayload(p + ":2").build()); + list.add(MessageBuilder.withPayload(p + ":3").build()); + list.add(MessageBuilder.withPayload(p + ":4").build()); + return list; + }; +} +---- + +Each message in the returned list will be sent individually resulting in four messages sent to output destination. + +[[spring-integration-flow-as-functions]] +=== Spring Integration flow as functions +When you implement a function, you may have complex requirements that fit the category +of https://www.enterpriseintegrationpatterns.com[Enterprise Integration Patterns] (EIP). These are best handled by using a +framework such as https://spring.io/projects/spring-integration[Spring Integration] (SI), which is a reference implementation of EIP. + +Thankfully SI already provides support for exposing integration flows as functions via +https://docs.spring.io/spring-integration/docs/current/reference/html/#java-dsl-gateway[Integration flow as gateway] +Consider the following sample: + +[source, java] +---- +@SpringBootApplication +public class FunctionSampleSpringIntegrationApplication { + + public static void main(String[] args) { + SpringApplication.run(FunctionSampleSpringIntegrationApplication.class, args); + } + + @Bean + public IntegrationFlow uppercaseFlow() { + return IntegrationFlows.from(MessageFunction.class, "uppercase") + .transform(String::toUpperCase) + .logAndReply(LoggingHandler.Level.WARN); + } + + public interface MessageFunction extends Function, Message> { + + } +} +---- + +For those who are familiar with SI you can see we define a bean of type `IntegrationFlow` where we +declare an integration flow that we want to expose as a `Function` (using SI DSL) called `uppercase`. +The `MessageFunction` interface lets us explicitly declare the type of the inputs and outputs for proper type conversion. +See <> section for more on type conversion. + +To receive raw input you can use `from(Function.class, ...)`. + +The resulting function is bound to the input and output destinations exposed by the target binder. + +IMPORTANT: Please refer to <> section to understand the naming +convention used to establish _binding names_ used by such application. + +For more details on interoperability of Spring Integration and Spring Cloud Stream specifically around functional programming model +you may find https://spring.io/blog/2019/10/25/spring-cloud-stream-and-spring-integration[this post] very interesting, as it dives a bit deeper +into various patterns you can apply by merging the best of Spring Integration and Spring Cloud Stream/Functions. + + +[[spring-cloud-streams-overview-using-polled-consumers]] +== Using Polled Consumers + +[[overview]] +=== Overview + +When using polled consumers, you poll the `PollableMessageSource` on demand. +To define binding for polled consumer you need to provide `spring.cloud.stream.pollable-source` property. + +Consider the following example of a polled consumer binding: + +[source,text] +---- +--spring.cloud.stream.pollable-source=myDestination +---- + +The pollable-source name `myDestination` in the preceding example will result in `myDestination-in-0` binding name to stay +consistent with functional programming model. + +Given the polled consumer in the preceding example, you might use it as follows: + +[source,java] +---- +@Bean +public ApplicationRunner poller(PollableMessageSource destIn, MessageChannel destOut) { + return args -> { + while (someCondition()) { + try { + if (!destIn.poll(m -> { + String newPayload = ((String) m.getPayload()).toUpperCase(); + destOut.send(new GenericMessage<>(newPayload)); + })) { + Thread.sleep(1000); + } + } + catch (Exception e) { + // handle failure + } + } + }; +} +---- + +A less manual and more Spring-like alternative would be to configure a scheduled task bean. For example, + +[source,java] +---- + +@Scheduled(fixedDelay = 5_000) +public void poll() { + System.out.println("Polling..."); + this.source.poll(m -> { + System.out.println(m.getPayload()); + + }, new ParameterizedTypeReference() { }); +} +---- + + +The `PollableMessageSource.poll()` method takes a `MessageHandler` argument (often a lambda expression, as shown here). +It returns `true` if the message was received and successfully processed. + +As with message-driven consumers, if the `MessageHandler` throws an exception, messages are published to error channels, +as discussed in `<>`. + +Normally, the `poll()` method acknowledges the message when the `MessageHandler` exits. +If the method exits abnormally, the message is rejected (not re-queued), but see <>. +You can override that behavior by taking responsibility for the acknowledgment, as shown in the following example: + +[source,java] +---- +@Bean +public ApplicationRunner poller(PollableMessageSource dest1In, MessageChannel dest2Out) { + return args -> { + while (someCondition()) { + if (!dest1In.poll(m -> { + StaticMessageHeaderAccessor.getAcknowledgmentCallback(m).noAutoAck(); + // e.g. hand off to another thread which can perform the ack + // or acknowledge(Status.REQUEUE) + + })) { + Thread.sleep(1000); + } + } + }; +} +---- + +IMPORTANT: You must `ack` (or `nack`) the message at some point, to avoid resource leaks. + +IMPORTANT: Some messaging systems (such as Apache Kafka) maintain a simple offset in a log. If a delivery fails and is re-queued with `StaticMessageHeaderAccessor.getAcknowledgmentCallback(m).acknowledge(Status.REQUEUE);`, any later successfully ack'd messages are redelivered. + +There is also an overloaded `poll` method, for which the definition is as follows: + +[source,java] +---- +poll(MessageHandler handler, ParameterizedTypeReference type) +---- + +The `type` is a conversion hint that allows the incoming message payload to be converted, as shown in the following example: + +[source,java] +---- +boolean result = pollableSource.poll(received -> { + Map payload = (Map) received.getPayload(); + ... + + }, new ParameterizedTypeReference>() {}); +---- + + +[[polled-errors]] +=== Handling Errors + +By default, an error channel is configured for the pollable source; if the callback throws an exception, an `ErrorMessage` is sent to the error channel (`..errors`); this error channel is also bridged to the global Spring Integration `errorChannel`. + +You can subscribe to either error channel with a `@ServiceActivator` to handle errors; without a subscription, the error will simply be logged and the message will be acknowledged as successful. +If the error channel service activator throws an exception, the message will be rejected (by default) and won't be redelivered. +If the service activator throws a `RequeueCurrentMessageException`, the message will be requeued at the broker and will be again retrieved on a subsequent poll. + +If the listener throws a `RequeueCurrentMessageException` directly, the message will be requeued, as discussed above, and will not be sent to the error channels. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/provided-messageconverters.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/provided-messageconverters.adoc new file mode 100644 index 000000000..ef1f87740 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/provided-messageconverters.adoc @@ -0,0 +1,15 @@ +[[provided-messageconverters]] += Provided MessageConverters + +As mentioned earlier, the framework already provides a stack of `MessageConverters` to handle most common use cases. +The following list describes the provided `MessageConverters`, in order of precedence (the first `MessageConverter` that works is used): + +. `JsonMessageConverter`: As the name suggests it supports conversion of the payload of the `Message` to/from POJO for cases when `contentType` is `application/json` (DEFAULT). +. `ByteArrayMessageConverter`: Supports conversion of the payload of the `Message` from `byte[]` to `byte[]` for cases when `contentType` is `application/octet-stream`. It is essentially a pass through and exists primarily for backward compatibility. +. `ObjectStringMessageConverter`: Supports conversion of any type to a `String` when `contentType` is `text/plain`. +It invokes Object’s `toString()` method or, if the payload is `byte[]`, a new `String(byte[])`. + +When no appropriate converter is found, the framework throws an exception. When that happens, you should check your code and configuration and ensure you did not miss anything (that is, ensure that you provided a `contentType` by using a binding or a header). +However, most likely, you found some uncommon case (such as a custom `contentType` perhaps) and the current stack of provided `MessageConverters` +does not know how to convert. If that is the case, you can add custom `MessageConverter`. See <>. + diff --git a/docs/modules/ROOT/pages/spring-cloud-stream/spring_integration_test_binder.adoc b/docs/modules/ROOT/pages/spring-cloud-stream/spring_integration_test_binder.adoc new file mode 100644 index 000000000..c38db3801 --- /dev/null +++ b/docs/modules/ROOT/pages/spring-cloud-stream/spring_integration_test_binder.adoc @@ -0,0 +1,305 @@ +[[spring_integration_test_binder]] += Spring Integration Test Binder + +Spring Cloud Stream comes with a test binder which you can use for testing the various application components without requiring an actual real-world binder implementation or a message broker. + +This test binder acts as a bridge between _unit_ and _integration_ testing and is based on https://spring.io/projects/spring-integration[Spring Integration] framework as an in-JVM message broker essentially giving you the best of both worlds - a real binder without the networking. + +[[test-binder-configuration]] +== Test Binder configuration +To enable Spring Integration test binder, all you need is to add it as a dependency. + +***Add required dependencies*** + +Below is the example of the required Maven POM entries. + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-test-binder + test + +---- + +Or for build.gradle.kts + +[source,kotlin] +---- +testImplementation("org.springframework.cloud:spring-cloud-stream-test-binder") +---- + +[[test-binder-usage]] +== Test Binder usage + +Now you can test your microservice as a simple unit test + +[source,java] +---- +@SpringBootTest +public class SampleStreamTests { + + @Autowired + private InputDestination input; + + @Autowired + private OutputDestination output; + + @Test + public void testEmptyConfiguration() { + this.input.send(new GenericMessage("hello".getBytes())); + assertThat(output.receive().getPayload()).isEqualTo("HELLO".getBytes()); + } + + @SpringBootApplication + @Import(TestChannelBinderConfiguration.class) + public static class SampleConfiguration { + @Bean + public Function uppercase() { + return v -> v.toUpperCase(); + } + } +} +---- + +And if you need more control or want to test several configurations in the same test suite +you can also do the following: + +[source,java] +---- +@EnableAutoConfiguration +public static class MyTestConfiguration { + @Bean + public Function uppercase() { + return v -> v.toUpperCase(); + } +} + +. . . + +@Test +public void sampleTest() { + try (ConfigurableApplicationContext context = new SpringApplicationBuilder( + TestChannelBinderConfiguration.getCompleteConfiguration( + MyTestConfiguration.class)) + .run("--spring.cloud.function.definition=uppercase")) { + InputDestination source = context.getBean(InputDestination.class); + OutputDestination target = context.getBean(OutputDestination.class); + source.send(new GenericMessage("hello".getBytes())); + assertThat(target.receive().getPayload()).isEqualTo("HELLO".getBytes()); + } +} +---- + +For cases where you have multiple bindings and/or multiple inputs and outputs, or simply want to be explicit about names of +the destination you are sending to or receiving from, the `send()` and `receive()` +methods of `InputDestination` and `OutputDestination` are overridden to allow you to provide the name of the input and output destination. + +Consider the following sample: +[source,java] +---- +@EnableAutoConfiguration +public static class SampleFunctionConfiguration { + + @Bean + public Function uppercase() { + return value -> value.toUpperCase(); + } + + @Bean + public Function reverse() { + return value -> new StringBuilder(value).reverse().toString(); + } +} +---- + +and the actual test + +[source,java] +---- +@Test +public void testMultipleFunctions() { + try (ConfigurableApplicationContext context = new SpringApplicationBuilder( + TestChannelBinderConfiguration.getCompleteConfiguration( + SampleFunctionConfiguration.class)) + .run("--spring.cloud.function.definition=uppercase;reverse")) { + + InputDestination inputDestination = context.getBean(InputDestination.class); + OutputDestination outputDestination = context.getBean(OutputDestination.class); + + Message inputMessage = MessageBuilder.withPayload("Hello".getBytes()).build(); + inputDestination.send(inputMessage, "uppercase-in-0"); + inputDestination.send(inputMessage, "reverse-in-0"); + + Message outputMessage = outputDestination.receive(0, "uppercase-out-0"); + assertThat(outputMessage.getPayload()).isEqualTo("HELLO".getBytes()); + + outputMessage = outputDestination.receive(0, "reverse-out-0"); + assertThat(outputMessage.getPayload()).isEqualTo("olleH".getBytes()); + } +} +---- + +For cases where you have additional mapping properties such as `destination` you should use those names. For example, consider a different version of the +preceding test where we explicitly map inputs and outputs of the `uppercase` function to `myInput` and `myOutput` binding names: +[source,java] +---- +@Test +public void testMultipleFunctions() { + try (ConfigurableApplicationContext context = new SpringApplicationBuilder( + TestChannelBinderConfiguration.getCompleteConfiguration( + SampleFunctionConfiguration.class)) + .run( + "--spring.cloud.function.definition=uppercase;reverse", + "--spring.cloud.stream.bindings.uppercase-in-0.destination=myInput", + "--spring.cloud.stream.bindings.uppercase-out-0.destination=myOutput" + )) { + + InputDestination inputDestination = context.getBean(InputDestination.class); + OutputDestination outputDestination = context.getBean(OutputDestination.class); + + Message inputMessage = MessageBuilder.withPayload("Hello".getBytes()).build(); + inputDestination.send(inputMessage, "myInput"); + inputDestination.send(inputMessage, "reverse-in-0"); + + Message outputMessage = outputDestination.receive(0, "myOutput"); + assertThat(outputMessage.getPayload()).isEqualTo("HELLO".getBytes()); + + outputMessage = outputDestination.receive(0, "reverse-out-0"); + assertThat(outputMessage.getPayload()).isEqualTo("olleH".getBytes()); + } +} +---- + + +[[test-binder-and-pollablemessagesource]] +== Test Binder and PollableMessageSource +Spring Integration Test Binder also allows you to write tests when working with `PollableMessageSource` (see <> for more details). + +The important thing that needs to be understood though is that polling is not event-driven, and that `PollableMessageSource` is a strategy which exposes operation to produce (poll for) a Message (singular). +How often you poll or how many threads you use or where you're polling from (message queue or file system) is entirely up to you; +In other words it is your responsibility to configure Poller or Threads or the actual source of Message. Luckily Spring has plenty of abstractions to configure exactly that. + +Let's look at the example: + +[source, java] +---- +@Test +public void samplePollingTest() { + ApplicationContext context = new SpringApplicationBuilder(SamplePolledConfiguration.class) + .web(WebApplicationType.NONE) + .run("--spring.jmx.enabled=false", "--spring.cloud.stream.pollable-source=myDestination"); + OutputDestination destination = context.getBean(OutputDestination.class); + System.out.println("Message 1: " + new String(destination.receive().getPayload())); + System.out.println("Message 2: " + new String(destination.receive().getPayload())); + System.out.println("Message 3: " + new String(destination.receive().getPayload())); +} + +@Import(TestChannelBinderConfiguration.class) +@EnableAutoConfiguration +public static class SamplePolledConfiguration { + @Bean + public ApplicationRunner poller(PollableMessageSource polledMessageSource, StreamBridge output, TaskExecutor taskScheduler) { + return args -> { + taskScheduler.execute(() -> { + for (int i = 0; i < 3; i++) { + try { + if (!polledMessageSource.poll(m -> { + String newPayload = ((String) m.getPayload()).toUpperCase(); + output.send("myOutput", newPayload); + })) { + Thread.sleep(2000); + } + } + catch (Exception e) { + // handle failure + } + } + }); + }; + } +} +---- + +The above (very rudimentary) example will produce 3 messages in 2 second intervals sending them to the output destination of `Source` +which this binder sends to `OutputDestination` where we retrieve them (for any assertions). +Currently, it prints the following: +[source, text] +---- +Message 1: POLLED DATA +Message 2: POLLED DATA +Message 3: POLLED DATA +---- +As you can see the data is the same. That is because this binder defines a default implementation of the actual `MessageSource` - the source +from which the Messages are polled using `poll()` operation. While sufficient for most testing scenarios, there are cases where you may want +to define your own `MessageSource`. To do so simply configure a bean of type `MessageSource` in your test configuration providing your own +implementation of Message sourcing. + +Here is the example: + +[source, java] +---- +@Bean +public MessageSource source() { + return () -> new GenericMessage<>("My Own Data " + UUID.randomUUID()); +} +---- +rendering the following output; +[source, text] +---- +Message 1: MY OWN DATA 1C180A91-E79F-494F-ABF4-BA3F993710DA +Message 2: MY OWN DATA D8F3A477-5547-41B4-9434-E69DA7616FEE +Message 3: MY OWN DATA 20BF2E64-7FF4-4CB6-A823-4053D30B5C74 +---- + +NOTE: DO NOT name this bean `messageSource` as it is going to be in conflict with the bean of the same name (different type) +provided by Spring Boot for unrelated reasons. + +[[special-note-on-mixing-test-binder-and-regular-middleware-binder-for-testing]] +== Special Note on Mixing Test Binder and Regular Middleware Binder for Testing + +The Spring Integration based test binder is provided for testing the application without involving an actual middleware based binder such as the Kafka or RabbitMQ binder. +As described in the sections above, the test binder helps you to verify the application behavior quickly by relying on the in-memory Spring Integration channels. +When the test binder is present on the test classpath, Spring Cloud Stream will try to use this binder for all testing purposes wherever it needs a binder for communication. +In other words, you cannot mix both the test binder and a regular middleware binder for testing purposes in the same module. +After testing the application with the test binder, if you want to continue doing further integration tests using the actual middleware binder, it is recommended to add those tests that use the actual binder in a separate module so that those tests can make the proper connection to the actual middleware rather than relying on the in-memory channels provided by the test binder. + +[[health-indicator]] += Health Indicator + +Spring Cloud Stream provides a health indicator for binders. +It is registered under the name `binders` and can be enabled or disabled by setting the `management.health.binders.enabled` property. + +To enable health check you first need to enable both "web" and "actuator" by including its dependencies (see <>) + +If `management.health.binders.enabled` is not set explicitly by the application, then `management.health.defaults.enabled` is matched as `true` and the binder health indicators are enabled. +If you want to disable health indicator completely, then you have to set `management.health.binders.enabled` to `false`. + +You can use Spring Boot actuator health endpoint to access the health indicator - `/actuator/health`. +By default, you will only receive the top level application status when you hit the above endpoint. +In order to receive the full details from the binder specific health indicators, you need to include the property `management.endpoint.health.show-details` with the value `ALWAYS` in your application. + +Health indicators are binder-specific and certain binder implementations may not necessarily provide a health indicator. + +If you want to completely disable all health indicators available out of the box and instead provide your own health indicators, +you can do so by setting property `management.health.binders.enabled` to `false` and then provide your own `HealthIndicator` beans in your application. +In this case, the health indicator infrastructure from Spring Boot will still pick up these custom beans. +Even if you are not disabling the binder health indicators, you can still enhance the health checks by providing your own `HealthIndicator` beans in addition to the out of the box health checks. + +When you have multiple binders in the same application, health indicators are enabled by default unless the application turns them off by setting `management.health.binders.enabled` to `false`. +In this case, if the user wants to disable health check for a subset of the binders, then that should be done by setting `management.health.binders.enabled` to `false` in the multi binder configurations's environment. +See <> for details on how environment specific properties can be provided. + +If there are multiple binders present in the classpath but not all of them are used in the application, this may cause some issues in the context of health indicators. +There may be implementation specific details as to how the health checks are performed. For example, a Kafka binder may decide the status as `DOWN` if there are no destinations registered by the binder. + +Lets take a concrete situation. Imagine you have both Kafka and Kafka Streams binders present in the classpath, but only use the Kafka Streams binder in the application code, i.e. only provide bindings using the Kafka Streams binder. +Since Kafka binder is not used and it has specific checks to see if any destinations are registered, the binder health check will fail. +The top level application health check status will be reported as `DOWN`. +In this situation, you can simply remove the dependency for kafka binder from your application since you are not using it. + +[[samples]] += Samples + +For Spring Cloud Stream samples, see the https://github.com/spring-cloud/spring-cloud-stream-samples[spring-cloud-stream-samples] repository on GitHub. +