diff --git a/src/reference/asciidoc/appendix.adoc b/src/reference/asciidoc/appendix.adoc index f2eca95e..0f67ea0d 100644 --- a/src/reference/asciidoc/appendix.adoc +++ b/src/reference/asciidoc/appendix.adoc @@ -1,8 +1,9 @@ [[deps-for-21x]] == Override Dependencies to use the 2.1.x kafka-clients with an Embedded Broker -When using `spring-kafka-test` (version 2.2.x) with the 2.1.x `kafka-clients` jar, you will need to override certain transitive dependencies as follows: +When you use `spring-kafka-test` (version 2.2.x) with the 2.1.x `kafka-clients` jar, you need to override certain transitive dependencies, as follows: +==== [source, xml] ---- @@ -46,6 +47,7 @@ When using `spring-kafka-test` (version 2.2.x) with the 2.1.x `kafka-clients` ja test ---- +==== [appendix] [[history]] diff --git a/src/reference/asciidoc/changes-since-1.0.adoc b/src/reference/asciidoc/changes-since-1.0.adoc index 9f934aeb..c828de9a 100644 --- a/src/reference/asciidoc/changes-since-1.0.adoc +++ b/src/reference/asciidoc/changes-since-1.0.adoc @@ -5,59 +5,65 @@ This version requires the 1.0.0 `kafka-clients` or higher. -NOTE: The 1.1.x client is supported, with _version 2.1.5_, but you will need to override dependencies as described in <>. -The 1.1.x client will be supported natively in _version 2.2_. +NOTE: The 1.1.x client is supported with version 2.1.5, but you need to override dependencies as described in <>. + +// TODO: No topic marked deps-for-11x exists in any of the files in the directory that contains this file. + +The 1.1.x client is supported natively in version 2.2. ==== JSON Improvements -The `StringJsonMessageConverter` and `JsonSerializer` now add type information in `Headers`, allowing the converter and `JsonDeserializer` to create specific types on reception, based on the message itself rather than a fixed configured type. +The `StringJsonMessageConverter` and `JsonSerializer` now add type information in `Headers`, letting the converter and `JsonDeserializer` create specific types on reception, based on the message itself rather than a fixed configured type. See <> for more information. ==== Container Stopping Error Handlers -Container Error handlers are now provided for both record and batch listeners that treat any exceptions thrown by the listener as fatal; they stop the container. +Container error handlers are now provided for both record and batch listeners that treat any exceptions thrown by the listener as fatal/ +They stop the container. See <> for more information. -==== Pausing/Resuming Containers +==== Pausing and Resuming Containers -The listener containers now have `pause()` and `resume()` methods (since _version 2.1.3_). +The listener containers now have `pause()` and `resume()` methods (since version 2.1.3). See <> for more information. ==== Stateful Retry -Starting with _version 2.1.3_, stateful retry can be configured; see <> for more information. +Starting with version 2.1.3, you can configure stateful retry. +See <> for more information. ==== Client ID -Starting with _version 2.1.1_, it is now possible to set the `client.id` prefix on `@KafkaListener`. -Previously, to customize the client id, you would need a separate consumer factory (and container factory) per listener. -The prefix is suffixed with `-n` to provide unique client ids when using concurrency. +Starting with version 2.1.1, you can now set the `client.id` prefix on `@KafkaListener`. +Previously, to customize the client ID, you needed a separate consumer factory (and container factory) per listener. +The prefix is suffixed with `-n` to provide unique client IDs when you use concurrency. ==== Logging Offset Commits -By default, logging of topic offset commits is performed with the DEBUG logging level. -Starting with _version 2.1.2_, there is a new property in `ContainerProperties` called `commitLogLevel` which allows you to specify the log level for these messages. +By default, logging of topic offset commits is performed with the `DEBUG` logging level. +Starting with version 2.1.2, a new property in `ContainerProperties` called `commitLogLevel` lets you specify the log level for these messages. See <> for more information. ==== Default @KafkaHandler -Starting with _version 2.1.3_, one of the `@KafkaHandler` s on a class-level `@KafkaListener` can be designated as the default. +Starting with version 2.1.3, you can designate one of the `@KafkaHandler` annotations on a class-level `@KafkaListener` as the default. See <> for more information. ==== ReplyingKafkaTemplate -Starting with _version 2.1.3_, a subclass of `KafkaTemplate` is provided to support request/reply semantics. +Starting with version 2.1.3, a subclass of `KafkaTemplate` is provided to support request/reply semantics. See <> for more information. ==== ChainedKafkaTransactionManager -_version 2.1.3_ introduced the `ChainedKafkaTransactionManager` see <> for more information. +Version 2.1.3 introduced the `ChainedKafkaTransactionManager`. +See <> for more information. ==== Migration Guide from 2.0 -https://github.com/spring-projects/spring-kafka/wiki/Spring-for-Apache-Kafka-2.0-to-2.1-Migration-Guide[2.0 to 2.1 Migration]. +See the https://github.com/spring-projects/spring-kafka/wiki/Spring-for-Apache-Kafka-2.0-to-2.1-Migration-Guide[2.0 to 2.1 Migration] guide. === Changes Between 1.3 and 2.0 @@ -65,9 +71,9 @@ https://github.com/spring-projects/spring-kafka/wiki/Spring-for-Apache-Kafka-2.0 The Spring for Apache Kafka project now requires Spring Framework 5.0 and Java 8. -==== @KafkaListener Changes +==== `@KafkaListener` Changes -You can now annotate `@KafkaListener` methods (and classes, and `@KafkaHandler` methods) with `@SendTo`. +You can now annotate `@KafkaListener` methods (and classes and `@KafkaHandler` methods) with `@SendTo`. If the method returns a result, it is forwarded to the specified topic. See <> for more information. @@ -76,7 +82,7 @@ See <> for more information. Message listeners can now be aware of the `Consumer` object. See <> for more information. -==== ConsumerAwareRebalanceListener +==== Using `ConsumerAwareRebalanceListener` Rebalance listeners can now access the `Consumer` object during rebalance notifications. See <> for more information. @@ -85,40 +91,42 @@ See <> for more information. ==== Support for Transactions -The 0.11.0.0 client library added support for transactions; the `KafkaTransactionManager` and other support for transactions has been added. +The 0.11.0.0 client library added support for transactions. +The `KafkaTransactionManager` and other support for transactions have been added. See <> for more information. ==== Support for Headers -The 0.11.0.0 client library added support for message headers; these can now be mapped to/from `spring-messaging` `MessageHeaders`. +The 0.11.0.0 client library added support for message headers. +These can now be mapped to and from `spring-messaging` `MessageHeaders`. See <> for more information. ==== Creating Topics -The 0.11.0.0 client library provides an `AdminClient` which can be used to create topics. -The `KafkaAdmin` uses this client to automatically add topics defined as `@Bean` s. +The 0.11.0.0 client library provides an `AdminClient`, which you can use to create topics. +The `KafkaAdmin` uses this client to automatically add topics defined as `@Bean` instances. -==== Support for Kafka timestamps +==== Support for Kafka Timestamps -`KafkaTemplate` now supports API to add records with timestamps. +`KafkaTemplate` now supports an API to add records with timestamps. New `KafkaHeaders` have been introduced regarding `timestamp` support. -Also new `KafkaConditions.timestamp()` and `KafkaMatchers.hasTimestamp()` testing utilities have been added. -See <>, <> and <> for more details. +Also, new `KafkaConditions.timestamp()` and `KafkaMatchers.hasTimestamp()` testing utilities have been added. +See <>, <>, and <> for more details. -==== @KafkaListener Changes +==== `@KafkaListener` Changes You can now configure a `KafkaListenerErrorHandler` to handle exceptions. See <> for more information. By default, the `@KafkaListener` `id` property is now used as the `group.id` property, overriding the property configured in the consumer factory (if present). Further, you can explicitly configure the `groupId` on the annotation. -Previously, you would have needed a separate container factory (and consumer factory) to use different `group.id` s for listeners. +Previously, you would have needed a separate container factory (and consumer factory) to use different `group.id` values for listeners. To restore the previous behavior of using the factory configured `group.id`, set the `idIsGroup` property on the annotation to `false`. -==== @EmbeddedKafka Annotation +==== `@EmbeddedKafka` Annotation -For convenience a test class level `@EmbeddedKafka` annotation is provided with the purpose to register `KafkaEmbedded` as a bean. +For convenience, a test class-level `@EmbeddedKafka` annotation is provided, to register `KafkaEmbedded` as a bean. See <> for more information. ==== Kerberos Configuration @@ -143,7 +151,7 @@ Listeners can be configured to receive the entire batch of messages returned by ==== Null Payloads -Null payloads are used to "delete" keys when using log compaction. +Null payloads are used to "`delete`" keys when you use log compaction. ==== Initial Offset @@ -151,7 +159,7 @@ When explicitly assigning partitions, you can now configure the initial offset r ==== Seek -You can now seek the position of each topic/partition. -This can be used to set the initial position during initialization when group management is in use and Kafka assigns the partitions. -You can also seek when an idle container is detected, or at any arbitrary point in your application's execution. +You can now seek the position of each topic or partition. +You can use this to set the initial position during initialization when group management is in use and Kafka assigns the partitions. +You can also seek when an idle container is detected or at any arbitrary point in your application's execution. See <> for more information. diff --git a/src/reference/asciidoc/index.adoc b/src/reference/asciidoc/index.adoc index af52f916..7731a534 100644 --- a/src/reference/asciidoc/index.adoc +++ b/src/reference/asciidoc/index.adoc @@ -1,7 +1,5 @@ [[spring-kafka-reference]] - = Spring for Apache Kafka - :toc: == Preface @@ -18,12 +16,10 @@ include::./whats-new.adoc[] == Introduction This first part of the reference documentation is a high-level overview of Spring for Apache Kafka and the underlying -concepts and some code snippets that will get you up and running as quickly as possible. +concepts and some code snippets that can help you get up and running as quickly as possible. include::quick-tour.adoc[] -// include::whats-new.adoc[] - == Reference This part of the reference documentation details the various components that comprise Spring for Apache Kafka. @@ -37,7 +33,7 @@ include::testing.adoc[] == Spring Integration -This part of the reference shows how to use the `spring-integration-kafka` module of Spring Integration. +This part of the reference guide shows how to use the `spring-integration-kafka` module of Spring Integration. include::si-kafka.adoc[] @@ -45,7 +41,7 @@ include::si-kafka.adoc[] == Other Resources -In addition to this reference documentation, there exist a number of other resources that may help you learn about +In addition to this reference documentation, we recommend a number of other resources that may help you learn about Spring and Apache Kafka. - https://kafka.apache.org/[Apache Kafka Project Home Page] diff --git a/src/reference/asciidoc/kafka.adoc b/src/reference/asciidoc/kafka.adoc index 04f36f91..0454f64a 100644 --- a/src/reference/asciidoc/kafka.adoc +++ b/src/reference/asciidoc/kafka.adoc @@ -1,11 +1,16 @@ [[kafka]] === Using Spring for Apache Kafka +This section offers detailed explanations of the various concerns that impact using Spring for Apache Kafka. +For a quick but less detailed introduction, see <>. + ==== Configuring Topics If you define a `KafkaAdmin` bean in your application context, it can automatically add topics to the broker. -Simply add a `NewTopic` `@Bean` for each topic to the application context. +To do so, you can add a `NewTopic` `@Bean` for each topic to the application context. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -18,23 +23,27 @@ public KafkaAdmin admin() { @Bean public NewTopic topic1() { - return new NewTopic("foo", 10, (short) 2); + return new NewTopic("thing1", 10, (short) 2); } @Bean public NewTopic topic2() { - return new NewTopic("bar", 10, (short) 2); + return new NewTopic("thing2", 10, (short) 2); } ---- +==== -By default, if the broker is not available, a message will be logged, but the context will continue to load. +By default, if the broker is not available, a message is logged, but the context continues to load. You can programmatically invoke the admin's `initialize()` method to try again later. -If you wish this condition to be considered fatal, set the admin's `fatalIfBrokerNotAvailable` property to `true` and the context will fail to initialize. +If you wish this condition to be considered fatal, set the admin's `fatalIfBrokerNotAvailable` property to `true`. +The context then fails to initialize. -NOTE: If the broker supports it (1.0.0 or higher), the admin will increase the number of partitions if it is found that an existing topic has fewer partitions than the `NewTopic.numPartitions`. +NOTE: If the broker supports it (1.0.0 or higher), the admin increases the number of partitions if it is found that an existing topic has fewer partitions than the `NewTopic.numPartitions`. -For more advanced features, such as assigning partitions to replicas, you can use the `AdminClient` directly: +For more advanced features, such as assigning partitions to replicas, you can use the `AdminClient` directly. +The following example shows how to do so: +==== [source, java] ---- @Autowired @@ -46,16 +55,23 @@ private KafkaAdmin admin; ... client.close(); ---- +==== ==== Sending Messages +This section covers how to send messages. + [[kafka-template]] -===== KafkaTemplate +===== Using `KafkaTemplate` + +This section covers how to use `KafkaTemplate` to send messages. ====== Overview -The `KafkaTemplate` wraps a producer and provides convenience methods to send data to kafka topics. +The `KafkaTemplate` wraps a producer and provides convenience methods to send data to Kafka topics. +The following listing shows the relevant methods from `KafkaTemplate`: +==== [source, java] ---- ListenableFuture> sendDefault(V data); @@ -93,21 +109,25 @@ interface ProducerCallback { T doInKafka(Producer producer); } - ---- +==== + +See the https://docs.spring.io/spring-kafka/api/org/springframework/kafka/core/KafkaTemplate.html[Javadoc] for more detail. The `sendDefault` API requires that a default topic has been provided to the template. -The API which take in a `timestamp` as a parameter will store this timestamp in the record. -The behavior of the user provided timestamp is stored is dependent on the timestamp type configured on the Kafka topic. -If the topic is configured to use `CREATE_TIME` then the user specified timestamp will be recorded or generated if not specified. -If the topic is configured to use `LOG_APPEND_TIME` then the user specified timestamp will be ignored and broker will add in the local broker time. +The API takes in a `timestamp` as a parameter and stores this timestamp in the record. +How the user-provided timestamp is stored depends on the timestamp type configured on the Kafka topic. +If the topic is configured to use `CREATE_TIME`, the user specified timestamp is recorded (or generated if not specified). +If the topic is configured to use `LOG_APPEND_TIME`, the user-specified timestamp is ignored and the broker adds in the local broker time. -The `metrics` and `partitionsFor` methods simply delegate to the same methods on the underlying https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/producer/Producer.html[`Producer`]. +The `metrics` and `partitionsFor` methods delegate to the same methods on the underlying https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/producer/Producer.html[`Producer`]. The `execute` method provides direct access to the underlying https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/producer/Producer.html[`Producer`]. -To use the template, configure a producer factory and provide it in the template's constructor: +To use the template, you can configure a producer factory and provide it in the template's constructor. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -130,24 +150,27 @@ public KafkaTemplate kafkaTemplate() { return new KafkaTemplate(producerFactory()); } ---- +==== -The template can also be configured using standard `` definitions. +You can also configure the template by using standard `` definitions. -Then, to use the template, simply invoke one of its methods. +Then, to use the template, you can invoke one of its methods. -When using the methods with a `Message` parameter, topic, partition and key information is provided in a message -header: +When you use the methods with a `Message` parameter, the topic, partition, and key information is provided in a message +header that includes the following items: -- `KafkaHeaders.TOPIC` -- `KafkaHeaders.PARTITION_ID` -- `KafkaHeaders.MESSAGE_KEY` -- `KafkaHeaders.TIMESTAMP` +* `KafkaHeaders.TOPIC` +* `KafkaHeaders.PARTITION_ID` +* `KafkaHeaders.MESSAGE_KEY` +* `KafkaHeaders.TIMESTAMP` -with the message payload being the data. +The message payload is the data. -Optionally, you can configure the `KafkaTemplate` with a `ProducerListener` to get an async callback with the +Optionally, you can configure the `KafkaTemplate` with a `ProducerListener` to get an asynchronous callback with the results of the send (success or failure) instead of waiting for the `Future` to complete. +The following listing shows the definition of the `ProducerListener` interface: +==== [source, java] ---- public interface ProducerListener { @@ -160,22 +183,25 @@ public interface ProducerListener { } ---- +==== -By default, the template is configured with a `LoggingProducerListener` which logs errors and does nothing when the +By default, the template is configured with a `LoggingProducerListener`, which logs errors and does nothing when the send is successful. -`onSuccess` is only called if `isInterestedInSuccess` returns `true`. +`onSuccess` is called only if `isInterestedInSuccess` returns `true`. -For convenience, the abstract `ProducerListenerAdapter` is provided in case you only want to implement one of the +For convenience, the abstract `ProducerListenerAdapter` is provided in case you want to implement only one of the methods. It returns `false` for `isInterestedInSuccess`. Notice that the send methods return a `ListenableFuture`. You can register a callback with the listener to receive the result of the send asynchronously. +The following examlpe shows how to do so: +==== [source, java] ---- -ListenableFuture> future = template.send("foo"); +ListenableFuture> future = template.send("something"); future.addCallback(new ListenableFutureCallback>() { @Override @@ -190,18 +216,22 @@ future.addCallback(new ListenableFutureCallback>() { }); ---- +==== -The `SendResult` has two properties, a `ProducerRecord` and `RecordMetadata`; refer to the Kafka API documentation -for information about those objects. +`SendResult` has two properties, a `ProducerRecord` and `RecordMetadata`. +See the Kafka API documentation for information about those objects. -If you wish to block the sending thread, to await the result, you can invoke the future's `get()` method. +If you wish to block the sending thread to await the result, you can invoke the future's `get()` method. You may wish to invoke `flush()` before waiting or, for convenience, the template has a constructor with an `autoFlush` -parameter which will cause the template to `flush()` on each send. -Note, however that flushing will likely significantly reduce performance. +parameter that causes the template to `flush()` on each send. +Note, however, that flushing likely significantly reduces performance. ====== Examples +This section shows examples of sending messages to Kafka: + .Non Blocking (Async) +==== [source, java] ---- public void sendToKafka(final MyOutputData data) { @@ -242,64 +272,75 @@ public void sendToKafka(final MyOutputData data) { } } ---- +==== [[transactions]] ===== Transactions +This section describes how Spring for Apache Kafka supports transactions. + ====== Overview The 0.11.0.0 client library added support for transactions. -Spring for Apache Kafka adds support in several ways. +Spring for Apache Kafka adds support in the following ways: -- `KafkaTransactionManager` - used with normal Spring transaction support (`@Transactional`, `TransactionTemplate` etc). -- Transactional `KafkaMessageListenerContainer` -- Local transactions with `KafkaTemplate` +* `KafkaTransactionManager`: Used with normal Spring transaction support (`@Transactional`, `TransactionTemplate` etc). +* Transactional `KafkaMessageListenerContainer` +* Local transactions with `KafkaTemplate` Transactions are enabled by providing the `DefaultKafkaProducerFactory` with a `transactionIdPrefix`. In that case, instead of managing a single shared `Producer`, the factory maintains a cache of transactional producers. -When the user `close()` s a producer, it is returned to the cache for reuse instead of actually being closed. +When the user calls `close()` on a producer, it is returned to the cache for reuse instead of actually being closed. The `transactional.id` property of each producer is `transactionIdPrefix` + `n`, where `n` starts with `0` and is incremented for each new producer, unless the transaction is started by a listener container with a record-based listener. -In that case, the `transactional.id` is `...`; this is to properly support fencing zombies https://www.confluent.io/blog/transactions-apache-kafka/[as described here]. +In that case, the `transactional.id` is `...`. +This is to properly support fencing zombies, https://www.confluent.io/blog/transactions-apache-kafka/[as described here]. This new behavior was added in versions 1.3.7, 2.0.6, 2.1.10, and 2.2.0. -If you wish to revert to the previous behavior, set the `producerPerConsumerPartition` property on the `DefaultKafkaProducerFactory` to `false`. +If you wish to revert to the previous behavior, you can set the `producerPerConsumerPartition` property on the `DefaultKafkaProducerFactory` to `false`. -NOTE: While transactions are supported with batch listeners, zombie fencing cannot be supported because a batch may contain records from multiple topics/partitions. +NOTE: While transactions are supported with batch listeners, zombie fencing cannot be supported because a batch may contain records from multiple topics or partitions. -====== KafkaTransactionManager +====== Using `KafkaTransactionManager` -The `KafkaTransactionManager` is an implementation of Spring Framework's `PlatformTransactionManager`; it is provided with a reference to the producer factory in its constructor. -If you provide a custom producer factory, it must support transactions - see `ProducerFactory.transactionCapable()`. +The `KafkaTransactionManager` is an implementation of Spring Framework's `PlatformTransactionManager`. +It is provided with a reference to the producer factory in its constructor. +If you provide a custom producer factory, it must support transactions. +See `ProducerFactory.transactionCapable()`. -You can use the `KafkaTransactionManager` with normal Spring transaction support (`@Transactional`, `TransactionTemplate` etc). -If a transaction is active, any `KafkaTemplate` operations performed within the scope of the transaction will use the transaction's `Producer`. -The manager will commit or rollback the transaction depending on success or failure. -The `KafkaTemplate` must be configured to use the same `ProducerFactory` as the transaction manager. +You can use the `KafkaTransactionManager` with normal Spring transaction support (`@Transactional`, `TransactionTemplate`, and others). +If a transaction is active, any `KafkaTemplate` operations performed within the scope of the transaction use the transaction's `Producer`. +The manager commits or rolls back the transaction, depending on success or failure. +You must configure the `KafkaTemplate` to use the same `ProducerFactory` as the transaction manager. ====== Transactional Listener Container and Exactly Once Processing -You can provide a listener container with a `KafkaAwareTransactionManager` instance; when so configured, the container will start a transaction before invoking the listener. -Any `KafkaTemplate` operations performed by the listener will participate in the transaction. -If the listener successfully processes the record (or records when using a `BatchMessageListener`), the container will send the offset(s) to the transaction using `producer.sendOffsetsToTransaction()`), before the transaction manager commits the transaction. -If the listener throws an exception, the transaction is rolled back and the consumer is repositioned so that the rolled-back record(s) will be retrieved on the next poll. +You can provide a listener container with a `KafkaAwareTransactionManager` instance. +When so configured, the container starts a transaction before invoking the listener. +Any `KafkaTemplate` operations performed by the listener participate in the transaction. +If the listener successfully processes the record (or multiple records, when using a `BatchMessageListener`), the container sends the offsets to the transaction by using `producer.sendOffsetsToTransaction()`), before the transaction manager commits the transaction. +If the listener throws an exception, the transaction is rolled back and the consumer is repositioned so that the rolled-back record(s) can be retrieved on the next poll. See <> for more information and for handling records that repeatedly fail. ====== Transaction Synchronization -If you need to synchronize a Kafka transaction with some other transaction; simply configure the listener container with the appropriate transaction manager (one that supports synchronization, such as the `DataSourceTransactionManager`). -Any operations performed on a **transactional** `KafkaTemplate` from the listener will participate in a single transaction. -The Kafka transaction will be committed (or rolled back) immediately after the controlling transaction. -Before exiting the listener, you should invoke one of the template's `sendOffsetsToTransaction` methods (unless you use a <>). -For convenience, the listener container binds its consumer group id to the thread so, generally, you can use the first method: +If you need to synchronize a Kafka transaction with some other transaction, configure the listener container with the appropriate transaction manager (one that supports synchronization, such as the `DataSourceTransactionManager`). +Any operations performed on a transactional `KafkaTemplate` from the listener participate in a single transaction. +The Kafka transaction is committed (or rolled back) immediately after the controlling transaction. +Before exiting the listener, you should invoke one of the template's `sendOffsetsToTransaction` methods (unless you use a <>). +For convenience, the listener container binds its consumer group ID to the thread, so, generally, you can use the first method. +The following listing shows the two method signatures: +==== [source, java] ---- void sendOffsetsToTransaction(Map offsets); void sendOffsetsToTransaction(Map offsets, String consumerGroupId); ---- +==== -For example: +The following example shows how to use the first signature of the `sendOffsetsToTransaction` method: +==== [source, java] ---- @Bean @@ -319,54 +360,66 @@ KafkaMessageListenerContainer container(ConsumerFactory cf, return new KafkaMessageListenerContainer<>(cf, props); } ---- +==== -NOTE: The offset to be committed is one greater than the offset of the record(s) processed by the listener. +NOTE: The offset to be committed is one greater than the offset of the records processed by the listener. -IMPORTANT: This should only be called when using transaction synchronization. -When a listener container is configured to use a `KafkaTransactionManager`, it will take care of sending the offsets to the transaction. +IMPORTANT: You should call this should only when you use transaction synchronization. +When a listener container is configured to use a `KafkaTransactionManager`, it takes care of sending the offsets to the transaction. [[chained-transaction-manager]] -====== ChainedKafkaTransactionManager +====== Using `ChainedKafkaTransactionManager` -The `ChainedKafkaTransactionManager` was introduced in _version 2.1.3_. +The `ChainedKafkaTransactionManager` was introduced in version 2.1.3. This is a subclass of `ChainedTransactionManager` that can have exactly one `KafkaTransactionManager`. Since it is a `KafkaAwareTransactionManager`, the container can send the offsets to the transaction in the same way as when the container is configured with a simple `KafkaTransactionManager`. This provides another mechanism for synchronizing transactions without having to send the offsets to the transaction in the listener code. -Chain your transaction managers in the desired order and provide the `ChainedTransactionManager` in the `ContainerProperties`. +You should chain your transaction managers in the desired order and provide the `ChainedTransactionManager` in the `ContainerProperties`. -====== KafkaTemplate Local Transactions +====== `KafkaTemplate` Local Transactions You can use the `KafkaTemplate` to execute a series of operations within a local transaction. +The following example shows how to do so: +==== [source, java] ---- boolean result = template.executeInTransaction(t -> { - t.sendDefault("foo", "bar"); - t.sendDefault("baz", "qux"); + t.sendDefault("thing1", "thing2"); + t.sendDefault("cat", "hat"); return true; }); ---- +==== The argument in the callback is the template itself (`this`). -If the callback exits normally, the transaction is committed; if an exception is thrown, the transaction is rolled-back. +If the callback exits normally, the transaction is committed. +If an exception is thrown, the transaction is rolled back. -NOTE: If there is a `KafkaTransactionManager` (or synchronized) transaction in process, it will not be used; a new "nested" transaction is used. +NOTE: If there is a `KafkaTransactionManager` (or synchronized) transaction in process, it is not used. +Instead, a new "nested" transaction is used. [[replying-template]] -===== ReplyingKafkaTemplate +===== Using `ReplyingKafkaTemplate` -_Version 2.1.3_ introduced a subclass of `KafkaTemplate` to provide request/reply semantics; the class is named `ReplyingKafkaTemplate` and has one method (in addition to those in the superclass): +Version 2.1.3 introduced a subclass of `KafkaTemplate` to provide request/reply semantics. +The class is named `ReplyingKafkaTemplate` and has one method (in addition to those in the superclass). +The following listing shows the method's signature: +==== [source, java] ---- RequestReplyFuture sendAndReceive(ProducerRecord record); ---- +==== -The result is a `ListenableFuture` that will asynchronously be populated with the result (or an exception, for a timeout). -The result also has a property `sendFuture` which is the result of calling `KafkaTemplate.send()`; you can use this future to determine the result of the send operation. +The result is a `ListenableFuture` that is asynchronously populated with the result (or an exception, for a timeout). +The result also has a `sendFuture` property, which is the result of calling `KafkaTemplate.send()`. +You can use this future to determine the result of the send operation. -The following Spring Boot application is an example of how to use the feature: +The following Spring Boot application shows an example of how to use the feature: +==== [source, java] ---- @SpringBootApplication @@ -419,13 +472,15 @@ public class KRequestingApplication { } ---- +==== -Note that we can use Boot's auto configured container factory to create the reply container. +Note that we can use Boot's auto-configured container factory to create the reply container. -The template sets a header `KafkaHeaders.CORRELATION_ID` which must be echoed back by the server side. +The template sets a header called `KafkaHeaders.CORRELATION_ID`, which must be echoed back by the server side. -In this case, simple `@KafkaListener` application responds: +In this case, the following `@KafkaListener` application responds: +==== [source, java] ---- @SpringBootApplication @@ -456,44 +511,54 @@ public class KReplyingApplication { } ---- +==== -The `@KafkaListener` infrastructure echoes the correlation id and determines the reply topic. +The `@KafkaListener` infrastructure echoes the correlation ID and determines the reply topic. -See <> for more information about sending replies; the template uses the default header `KafKaHeaders.REPLY_TOPIC` to indicate which topic the reply goes to. +See <> for more information about sending replies. +The template uses the default header `KafKaHeaders.REPLY_TOPIC` to indicate the topic to which the reply goes. -Starting with version 2.2, the template will attempt to detect the reply topic/partition from the configured reply container. -If the container is configured to listen to a single topic or a single `TopicPartitionInitialOffset`, it will be used to set the reply headers. -If the container is configured otherwise, the user must set up the reply header(s); in this case, an INFO log is written during initialization. +Starting with version 2.2, the template tries to detect the reply topic or partition from the configured reply container. +If the container is configured to listen to a single topic or a single `TopicPartitionInitialOffset`, it is used to set the reply headers. +If the container is configured otherwise, the user must set up the reply headers. +In this case, an `INFO` log message is written during initialization. +The following example uses `KafkaHeaders.REPLY_TOPIC`: +==== [source, java] ---- record.headers().add(new RecordHeader(KafkaHeaders.REPLY_TOPIC, "kReplies".getBytes())); ---- +==== -When configuring with a single reply `TopicPartitionInitialOffset`, you can use the same reply topic for multiple templates, as long as each instance listens on a different partition. -When configuring with a single reply topic, each instance must use a different `group.id` - in this case, all instances will receive each reply, but only the instance that sent the request will find the correlation id. +When you configure with a single reply `TopicPartitionInitialOffset`, you can use the same reply topic for multiple templates, as long as each instance listens on a different partition. +When configuring with a single reply topic, each instance must use a different `group.id`. +In this case, all instances receive each reply, but only the instance that sent the request finds the correlation ID. This may be useful for auto-scaling, but with the overhead of additional network traffic and the small cost of discarding each unwanted reply. -When using this setting, it is recommended that you set the template's `sharedReplyTopic` to true, which will reduce the logging level of unexpected replies to DEBUG instead of the default ERROR. +When you use this setting, we recommend that you set the template's `sharedReplyTopic` to `true`, which reduces the logging level of unexpected replies to DEBUG instead of the default ERROR. -IMPORTANT: If you have multiple client instances, and you don't configure them as discussed in the paragraphe above, each instance will need a dedicated reply topic. -An alternative is to set the `KafkaHeaders.REPLY_PARTITION` and use a dedicated partition for each instance; the `Header` contains a 4 byte int (Big-endian). +IMPORTANT: If you have multiple client instances and you do not configure them as discussed in the preceding paragraph, each instance needs a dedicated reply topic. +An alternative is to set the `KafkaHeaders.REPLY_PARTITION` and use a dedicated partition for each instance. +The `Header` contains a four-byte int (big-endian). The server must use this header to route the reply to the correct topic (`@KafkaListener` does this). -In this case, though, the reply container must not use Kafka's group management feature and must be configured to listen on a fixed partition (using a `TopicPartitionInitialOffset` in its `ContainerProperties` constructor). +In this case, though, the reply container must not use Kafka's group management feature and must be configured to listen on a fixed partition (by using a `TopicPartitionInitialOffset` in its `ContainerProperties` constructor). NOTE: The `DefaultKafkaHeaderMapper` requires Jackson to be on the classpath (for the `@KafkaListener`). -If it is not available, the message converter has no header mapper, so you must configure a `MessagingMessageConverter` with a `SimpleKafkaHeaderMapper` as shown above. +If it is not available, the message converter has no header mapper, so you must configure a `MessagingMessageConverter` with a `SimpleKafkaHeaderMapper`, as shown earlier. ==== Receiving Messages -Messages can be received by configuring a `MessageListenerContainer` and providing a Message Listener, or by +You can receive messages by configuring a `MessageListenerContainer` and providing a message listener or by using the `@KafkaListener` annotation. [[message-listeners]] ===== Message Listeners -When using a <> you must provide a listener to receive data. -There are currently eight supported interfaces for message listeners: +When you use a <>, you must provide a listener to receive data. +There are currently eight supported interfaces for message listeners. +The following listing shows these interfaces: +==== [source, java] ---- public interface MessageListener { <1> @@ -545,55 +610,58 @@ public interface BatchAcknowledgingConsumerAwareMessageListener extends Ba } ---- -<1> Use this for processing individual `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using auto-commit, or one of the container-managed <>. +<1> Use this interface for processing individual `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using auto-commit or one of the container-managed <>. -<2> Use this for processing individual `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using one of the manual <>. +<2> Use this interface for processing individual `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using one of the manual <>. -<3> Use this for processing individual `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using auto-commit, or one of the container-managed <>. +<3> Use this interface for processing individual `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using auto-commit or one of the container-managed <>. Access to the `Consumer` object is provided. -<4> Use this for processing individual `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using one of the manual <>. +<4> Use this interface for processing individual `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using one of the manual <>. Access to the `Consumer` object is provided. -<5> Use this for processing all `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using auto-commit, or one of the container-managed <>. -`AckMode.RECORD` is not supported when using this interface since the listener is given the complete batch. +<5> Use this interface for processing all `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using auto-commit or one of the container-managed <>. +`AckMode.RECORD` is not supported when you use this interface, since the listener is given the complete batch. -<6> Use this for processing all `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using one of the manual <>. +<6> Use this interface for processing all `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using one of the manual <>. -<7> Use this for processing all `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using auto-commit, or one of the container-managed <>. -`AckMode.RECORD` is not supported when using this interface since the listener is given the complete batch. +<7> Use this interface for processing all `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using auto-commit or one of the container-managed <>. +`AckMode.RECORD` is not supported when you use this interface, since the listener is given the complete batch. Access to the `Consumer` object is provided. -<8> Use this for processing all `ConsumerRecord` s received from the kafka consumer `poll()` operation when -using one of the manual <>. +<8> Use this interface for processing all `ConsumerRecord` instances received from the Kafka consumer `poll()` operation when +using one of the manual <>. Access to the `Consumer` object is provided. +==== -IMPORTANT: The `Consumer` object is not thread-safe; you must only invoke its methods on the thread that calls the listener. +IMPORTANT: The `Consumer` object is not thread-safe. +You must only invoke its methods on the thread that calls the listener. [[message-listener-container]] ===== Message Listener Containers Two `MessageListenerContainer` implementations are provided: -- `KafkaMessageListenerContainer` -- `ConcurrentMessageListenerContainer` +* `KafkaMessageListenerContainer` +* `ConcurrentMessageListenerContainer` -The `KafkaMessageListenerContainer` receives all message from all topics/partitions on a single thread. -The `ConcurrentMessageListenerContainer` delegates to 1 or more `KafkaMessageListenerContainer` s to provide +The `KafkaMessageListenerContainer` receives all message from all topics or partitions on a single thread. +The `ConcurrentMessageListenerContainer` delegates to one or more `KafkaMessageListenerContainer` instances to provide multi-threaded consumption. [[kafka-container]] -====== KafkaMessageListenerContainer +====== Using `KafkaMessageListenerContainer` -The following constructors are available. +The following constructors are available: +==== [source, java] ---- public KafkaMessageListenerContainer(ConsumerFactory consumerFactory, @@ -602,14 +670,15 @@ public KafkaMessageListenerContainer(ConsumerFactory consumerFactory, public KafkaMessageListenerContainer(ConsumerFactory consumerFactory, ContainerProperties containerProperties, TopicPartitionInitialOffset... topicPartitions) - ---- +==== Each takes a `ConsumerFactory` and information about topics and partitions, as well as other configuration in a `ContainerProperties` object. -The second constructor is used by the `ConcurrentMessageListenerContainer` (see below) to distribute `TopicPartitionInitialOffset` across the consumer instances. +The second constructor is used by the `ConcurrentMessageListenerContainer` (<>) to distribute `TopicPartitionInitialOffset` across the consumer instances. `ContainerProperties` has the following constructors: +==== [source, java] ---- public ContainerProperties(TopicPartitionInitialOffset... topicPartitions) @@ -618,18 +687,23 @@ public ContainerProperties(String... topics) public ContainerProperties(Pattern topicPattern) ---- +==== -The first takes an array of `TopicPartitionInitialOffset` arguments to explicitly instruct the container which partitions to use -(using the consumer `assign()` method), and with an optional initial offset: a positive value is an absolute offset by default; a negative value is relative to the current last offset within a partition by default. -A constructor for `TopicPartitionInitialOffset` is provided that takes an additional `boolean` argument. +The first constructor takes an array of `TopicPartitionInitialOffset` arguments to explicitly instruct the container about which partitions to use +(using the consumer `assign()` method) and with an optional initial offset. +A positive value is an absolute offset by default. +A negative value is relative to the current last offset within a partition by default. +A constructor for `TopicPartitionInitialOffset` that takes an additional `boolean` argument is provided. If this is `true`, the initial offsets (positive or negative) are relative to the current position for this consumer. The offsets are applied when the container is started. -The second takes an array of topics and Kafka allocates the partitions based on the `group.id` property - distributing +The second takes an array of topics, and Kafka allocates the partitions based on the `group.id` property -- distributing partitions across the group. The third uses a regex `Pattern` to select the topics. -To assign a `MessageListener` to a container, use the `ContainerProps.setMessageListener` method when creating the Container: +To assign a `MessageListener` to a container, you can use the `ContainerProps.setMessageListener` method when creating the Container. +The following example shows how to do so: +==== [source, java] ---- ContainerProperties containerProps = new ContainerProperties("topic1", "topic2"); @@ -642,97 +716,111 @@ KafkaMessageListenerContainer container = new KafkaMessageListenerContainer<>(cf, containerProps); return container; ---- +==== -Refer to the JavaDocs for `ContainerProperties` for more information about the various properties that can be set. +Refer to the https://docs.spring.io/spring-kafka/api/org/springframework/kafka/listener/ContainerProperties.html[Javadoc] for `ContainerProperties` for more information about the various properties that you can set. -Since _version 2.1.1_, a new property `logContainerConfig` is available; when true, and INFO logging is enabled, each listener container will write a log message summarizing its configuration properties. +Since version 2.1.1, a new property called `logContainerConfig` is available. +When `true` and `INFO` logging is enabled each listener container writes a log message summarizing its configuration properties. -By default, logging of topic offset commits is performed with the DEBUG logging level. -Starting with _version 2.1.2_, there is a new property in `ContainerProperties` called `commitLogLevel` which allows you to specify the log level for these messages. -For example, to change the log level to INFO, use `containerProperties.setCommitLogLevel(LogIfLevelEnabled.Level.INFO);`. +By default, logging of topic offset commits is performed at the `DEBUG` logging level. +Starting with version 2.1.2, a property in `ContainerProperties` called `commitLogLevel` lets you specify the log level for these messages. +For example, to change the log level to `INFO`, you can use `containerProperties.setCommitLogLevel(LogIfLevelEnabled.Level.INFO);`. -Starting with _version 2.2_, a new container property `missingTopicsFatal` has been added (default `true`). -This prevents the container from starting if any of the configured topics are not present on the broker; it does not apply if the container is configured to listen to a topic pattern (regex). -Previously, the container threads looped within the `consumer.poll()` method waiting for the topic to appear, while logging many messages; aside from the logs, there was no indication that there was a problem. -To restore the previous behavior, set the property to `false`. +Starting with version 2.2, a new container property called `missingTopicsFatal` has been added (default: `true`). +This prevents the container from starting if any of the configured topics are not present on the broker. +It does not apply if the container is configured to listen to a topic pattern (regex). +Previously, the container threads looped within the `consumer.poll()` method waiting for the topic to appear while logging many messages. +Aside from the logs, there was no indication that there was a problem. +To restore the previous behavior, you canset the property to `false`. -====== ConcurrentMessageListenerContainer +[[using-ConcurrentMessageListenerContainer]] +====== Using`ConcurrentMessageListenerContainer` -The single constructor is similar to the first `KafkaListenerContainer` constructor: +The single constructor is similar to the first `KafkaListenerContainer` constructor. +The following listing shows the constructor's signature: +==== [source, java] ---- public ConcurrentMessageListenerContainer(ConsumerFactory consumerFactory, ContainerProperties containerProperties) ---- +==== -It also has a property `concurrency`, e.g. `container.setConcurrency(3)` will create 3 `KafkaMessageListenerContainer` s. +It also has a `concurrency` property. +For example, `container.setConcurrency(3)` creates three `KafkaMessageListenerContainer` instances. -For the first constructor, kafka will distribute the partitions across the consumers using its group management capabilities. +For the first constructor, Kafka distributes the partitions across the consumers using its group management capabilities. [IMPORTANT] ==== When listening to multiple topics, the default partition distribution may not be what you expect. -For example, if you have 3 topics with 5 partitions each and you want to use `concurrency=15` you will only see 5 active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. -This is because the default Kafka `PartitionAssignor` is the `RangeAssignor` (see its javadocs). -For this scenario, you may want to consider using the `RoundRobinAssignor` instead, which will distribute the partitions across all of the consumers. -Then, each consumer will be assigned one topic/partition. -To change the `PartitionAssignor`, set the `partition.assignment.strategy` consumer property (`ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG`) in the properties provided to the `DefaultKafkaConsumerFactory`. +For example, if you have three topics with five partitions each and you want to use `concurrency=15`, you see only five active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. +This is because the default Kafka `PartitionAssignor` is the `RangeAssignor` (see its Javadoc). +For this scenario, you may want to consider using the `RoundRobinAssignor` instead, which distributes the partitions across all of the consumers. +Then, each consumer is assigned one topic or partition. +To change the `PartitionAssignor`, you can set the `partition.assignment.strategy` consumer property (`ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG`) in the properties provided to the `DefaultKafkaConsumerFactory`. -When using Spring Boot: +When using Spring Boot, you can assign set the strategy as follows: +===== [source] ---- spring.kafka.consumer.properties.partition.assignment.strategy=\ org.apache.kafka.clients.consumer.RoundRobinAssignor ---- +===== ==== -For the second constructor, the `ConcurrentMessageListenerContainer` distributes the `TopicPartition` s across the -delegate `KafkaMessageListenerContainer` s. +For the second constructor, the `ConcurrentMessageListenerContainer` distributes the `TopicPartition` instances across the +delegate `KafkaMessageListenerContainer` instances. -If, say, 6 `TopicPartition` s are provided and the `concurrency` is 3; each container will get 2 partitions. -For 5 `TopicPartition` s, 2 containers will get 2 partitions and the third will get 1. -If the `concurrency` is greater than the number of `TopicPartitions`, the `concurrency` will be adjusted down such that -each container will get one partition. +If, say, six `TopicPartition` instances are provided and the `concurrency` is `3`; each container gets two partitions. +For five `TopicPartition` instances, two containers get two partitions, and the third gets one. +If the `concurrency` is greater than the number of `TopicPartitions`, the `concurrency` is adjusted down such that +each container gets one partition. -NOTE: The `client.id` property (if set) will be appended with `-n` where `n` is the consumer instance according to the concurrency. +NOTE: The `client.id` property (if set) is appended with `-n` where `n` is the consumer instance that corresponds to the concurrency. This is required to provide unique names for MBeans when JMX is enabled. -Starting with _version 1.3_, the `MessageListenerContainer` provides an access to the metrics of the underlying `KafkaConsumer`. -In case of `ConcurrentMessageListenerContainer` the `metrics()` method returns the metrics for all the target `KafkaMessageListenerContainer` instances. +Starting with version 1.3, the `MessageListenerContainer` provides access to the metrics of the underlying `KafkaConsumer`. +In the case of `ConcurrentMessageListenerContainer`, the `metrics()` method returns the metrics for all the target `KafkaMessageListenerContainer` instances. The metrics are grouped into the `Map` by the `client-id` provided for the underlying `KafkaConsumer`. [[committing-offsets]] ====== Committing Offsets Several options are provided for committing offsets. -If the `enable.auto.commit` consumer property is true, kafka will auto-commit the offsets according to its +If the `enable.auto.commit` consumer property is `true`, Kafka auto-commits the offsets according to its configuration. -If it is false, the containers support the following `AckMode` s. +If it is `false`, the containers support several `AckMode` settings (described in the next list). -The consumer `poll()` method will return one or more `ConsumerRecords`; the `MessageListener` is called for each record; -the following describes the action taken by the container for each `AckMode` : +The consumer `poll()` method returns one or more `ConsumerRecords`. +The `MessageListener` is called for each record. +The following lists describes the action taken by the container for each `AckMode`: -- RECORD - commit the offset when the listener returns after processing the record. -- BATCH - commit the offset when all the records returned by the `poll()` have been processed. -- TIME - commit the offset when all the records returned by the `poll()` have been processed as long as the `ackTime` +* `RECORD`: Commit the offset when the listener returns after processing the record. +* `BATCH`: Commit the offset when all the records returned by the `poll()` have been processed. +* `TIME`: Commit the offset when all the records returned by the `poll()` have been processed, as long as the `ackTime` since the last commit has been exceeded. -- COUNT - commit the offset when all the records returned by the `poll()` have been processed as long as `ackCount` +* `COUNT`: Commit the offset when all the records returned by the `poll()` have been processed, as long as `ackCount` records have been received since the last commit. -- COUNT_TIME - similar to TIME and COUNT but the commit is performed if either condition is true. -- MANUAL - the message listener is responsible to `acknowledge()` the `Acknowledgment`; -after which, the same semantics as `BATCH` are applied. -- MANUAL_IMMEDIATE - commit the offset immediately when the `Acknowledgment.acknowledge()` method is called by the +* `COUNT_TIME`: Similar to `TIME` and `COUNT`, but the commit is performed if either condition is `true`. +* `MANUAL`: The message listener is responsible to `acknowledge()` the `Acknowledgment`. +After that, the same semantics as `BATCH` are applied. +* `MANUAL_IMMEDIATE`: Commit the offset immediately when the `Acknowledgment.acknowledge()` method is called by the listener. -NOTE: `MANUAL`, and `MANUAL_IMMEDIATE` require the listener to be an `AcknowledgingMessageListener` or a `BatchAcknowledgingMessageListener`; see <>. +NOTE: `MANUAL`, and `MANUAL_IMMEDIATE` require the listener to be an `AcknowledgingMessageListener` or a `BatchAcknowledgingMessageListener`. +See <>. -The `commitSync()` or `commitAsync()` method on the consumer is used, depending on the `syncCommits` container property. +Depending on the `syncCommits` container property, the `commitSync()` or `commitAsync()` method on the consumer is used. -The `Acknowledgment` has this method: +The `Acknowledgment` has the following method: +==== [source, java] ---- public interface Acknowledgment { @@ -741,32 +829,35 @@ public interface Acknowledgment { } ---- +==== -This gives the listener control over when offsets are committed. +This method gives the listener control over when offsets are committed. [[container-auto-startup]] ====== Listener Container Auto Startup -The listener containers implement `SmartLifecycle` and `autoStartup` is `true` by default; the containers are started in a late phase (`Integer.MAX-VALUE - 100`). -Other components that implement `SmartLifecycle`, that handle data from listeners, should be started in an earlier phase. +The listener containers implement `SmartLifecycle`, and `autoStartup` is `true` by default. +The containers are started in a late phase (`Integer.MAX-VALUE - 100`). +Other components that implement `SmartLifecycle`, to handle data from listeners, should be started in an earlier phase. The `- 100` leaves room for later phases to enable components to be auto-started after the containers. [[kafka-listener-annotation]] -===== @KafkaListener Annotation +===== `@KafkaListener` Annotation -====== Introduction +The `@KafkaListener` annotation is used to designate a bean method as a listener for a listener container. +The bean is wrapped in a `MessagingMessageListenerAdapter` configured with various features, such as converters to convert the data, if necessary, to match the method parameters. -The `@KafkaListener` annotation is used to designate a bean method as a listener for a listener container; the bean is wrapped in a `MessagingMessageListenerAdapter` configured with various features, such as converters to convert the data, if necessary, to match the method paramters. - -Most attributes on the annotation can be configured with SpEL using `#{...}` and/or property placeholders `${...}`. -Refer to the javadocs for more information. +You can configure most attributes on the annotation with SpEL by using `#{...}` or property placeholders (`${...}`). +See the https://docs.spring.io/spring-kafka/api/org/springframework/kafka/annotation/KafkaListener.html[Javadoc] for more information. [[record-listener]] ====== Record Listeners -The `@KafkaListener` annotation provides a mechanism for simple POJO listeners: +The `@KafkaListener` annotation provides a mechanism for simple POJO listeners. +The following example shows how to use it: +==== [source, java] ---- public class Listener { @@ -778,10 +869,13 @@ public class Listener { } ---- +==== -This mechanism requires an `@EnableKafka` annotation on one of your `@Configuration` classes and a listener container factory, which is used to configure the underlying -`ConcurrentMessageListenerContainer`: by default, a bean with name `kafkaListenerContainerFactory` is expected. +This mechanism requires an `@EnableKafka` annotation on one of your `@Configuration` classes and a listener container factory, which is used to configure the underlying `ConcurrentMessageListenerContainer`. +By default, a bean with name `kafkaListenerContainerFactory` is expected. +The following example shows how to use `ConcurrentMessageListenerContainer`: +==== [source, java] ---- @Configuration @@ -813,15 +907,17 @@ public class KafkaConfig { } } ---- +==== -Notice that to set container properties, you must use the `getContainerProperties()` method on the factory. +Notice that, to set container properties, you must use the `getContainerProperties()` method on the factory. It is used as a template for the actual properties injected into the container. -Starting with version 2.1.1, it is now possible to set the `client.id` property for consumers created by the annotation. -The `clientIdPrefix` is suffixed with `-n` where `n` is an integer representing the container number when using concurrency. +Starting with version 2.1.1, you can now set the `client.id` property for consumers created by the annotation. +The `clientIdPrefix` is suffixed with `-n`, where `n` is an integer representing the container number when using concurrency. -Starting with version 2.2, you can now override the container factory's `concurrency` and `autoStartup` properties using properties on the annotation itself. -The properties can be simple values, property placeholders or SpEL expressions. +Starting with version 2.2, you can now override the container factory's `concurrency` and `autoStartup` properties by using properties on the annotation itself. +The properties can be simple values, property placeholders, or SpEL expressions. +The following example shows how to do so: ==== [source, java] @@ -834,11 +930,13 @@ public void listen(String data) { ---- ==== -You can also configure POJO listeners with explicit topics and partitions (and, optionally, their initial offsets): +You can also configure POJO listeners with explicit topics and partitions (and, optionally, their initial offsets). +The following example shows how to do so: +==== [source, java] ---- -@KafkaListener(id = "bar", topicPartitions = +@KafkaListener(id = "thing2", topicPartitions = { @TopicPartition(topic = "topic1", partitions = { "0", "1" }), @TopicPartition(topic = "topic2", partitions = "0", partitionOffsets = @PartitionOffset(partition = "1", initialOffset = "100")) @@ -847,31 +945,37 @@ public void listen(ConsumerRecord record) { ... } ---- +==== -Each partition can be specified in the `partitions` or `partitionOffsets` attribute, but not both. +You can specify each partition in the `partitions` or `partitionOffsets` attribute but not both. -When using manual `AckMode`, the listener can also be provided with the `Acknowledgment`; this example also shows -how to use a different container factory. +When using manual `AckMode`, you can also provide the listener with the `Acknowledgment`. +The following example also shows how to use a different container factory. +==== [source, java] ---- -@KafkaListener(id = "baz", topics = "myTopic", +@KafkaListener(id = "cat", topics = "myTopic", containerFactory = "kafkaManualAckListenerContainerFactory") public void listen(String data, Acknowledgment ack) { ... ack.acknowledge(); } ---- +==== -Finally, metadata about the message is available from message headers, the following header names can be used for retrieving the headers of the message: +Finally, metadata about the message is available from message headers. +You can use the following header names to retrieve the headers of the message: -- `KafkaHeaders.RECEIVED_MESSAGE_KEY` -- `KafkaHeaders.RECEIVED_TOPIC` -- `KafkaHeaders.RECEIVED_PARTITION_ID` -- `KafkaHeaders.RECEIVED_TIMESTAMP` -- `KafkaHeaders.TIMESTAMP_TYPE` +* `KafkaHeaders.RECEIVED_MESSAGE_KEY` +* `KafkaHeaders.RECEIVED_TOPIC` +* `KafkaHeaders.RECEIVED_PARTITION_ID` +* `KafkaHeaders.RECEIVED_TIMESTAMP` +* `KafkaHeaders.TIMESTAMP_TYPE` +The following example shows how to use the headers: +==== [source, java] ---- @KafkaListener(id = "qux", topicPattern = "myTopic1") @@ -884,13 +988,16 @@ public void listen(@Payload String foo, ... } ---- +==== [[batch-listeners]] ====== Batch listeners -Starting with _version 1.1_, `@KafkaListener` methods can be configured to receive the entire batch of consumer records received from the consumer poll. -To configure the listener container factory to create batch listeners, set the `batchListener` property: +Starting with version 1.1, you can configure `@KafkaListener` methods to receive the entire batch of consumer records received from the consumer poll. +To configure the listener container factory to create batch listeners, you can set the `batchListener` property. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -902,9 +1009,11 @@ public KafkaListenerContainerFactory batchFactory() { return factory; } ---- +==== -To receive a simple list of payloads: +The following example shows how to receive a list of payloads: +==== [source, java] ---- @KafkaListener(id = "list", topics = "myTopic", containerFactory = "batchFactory") @@ -912,9 +1021,12 @@ public void listen(List list) { ... } ---- +==== -The topic, partition, offset etc are available in headers which parallel the payloads: +The topic, partition, offset, and so on are available in headers that parallel the payloads. +The following example shows how to use the headers: +==== [source, java] ---- @KafkaListener(id = "list", topics = "myTopic", containerFactory = "batchFactory") @@ -926,9 +1038,12 @@ public void listen(List list, ... } ---- +==== -Alternatively you can receive a List of `Message` objects with each offset, etc in each message, but it must be the only parameter (aside from optional `Acknowledgment`, when using manual commits, and/or `Consumer` parameters) defined on the method: +Alternatively, you can receive a `List` of `Message` objects with each offset and other details in each message, but it must be the only parameter (aside from optional `Acknowledgment`, when using manual commits, and/or `Consumer` parameters) defined on the method. +The following example shows how to do so: +==== [source, java] ---- @KafkaListener(id = "listMsg", topics = "myTopic", containerFactory = "batchFactory") @@ -946,14 +1061,17 @@ public void listen16(List> list, Acknowledgment ack, Consumer c ... } ---- +==== No conversion is performed on the payloads in this case. -If the `BatchMessagingMessageConverter` is configured with a `RecordMessageConverter`, you can also add a generic type to the `Message` parameter and the payloads will be converted. +If the `BatchMessagingMessageConverter` is configured with a `RecordMessageConverter`, you can also add a generic type to the `Message` parameter and the payloads are converted. See <> for more information. -You can also receive a list of `ConsumerRecord` objects but it must be the only parameter (aside from optional `Acknowledgment`, when using manual commits, and/or `Consumer` parameters) defined on the method: +You can also receive a list of `ConsumerRecord` objects, but it must be the only parameter (aside from optional `Acknowledgment`, when using manual commits and `Consumer` parameters) defined on the method. +The following example shows how to do so: +==== [source, java] ---- @KafkaListener(id = "listCRs", topics = "myTopic", containerFactory = "batchFactory") @@ -966,10 +1084,13 @@ public void listen(List> list, Acknowledgment ac ... } ---- +==== -Starting with _version 2.2_, the listener can receive the complete `ConsumerRecords` object returned by the `poll()` method, allowing the listener to access additional methods such as `partitions()` which returns the `TopicPartition` s in the list and `records(TopicPartition)` to get selective records. -Again, this must be the only parameter (aside from optional `Acknowledgment`, when using manual commits, and/or `Consumer` parameters) on the method: +Starting with version 2.2, the listener can receive the complete `ConsumerRecords` object returned by the `poll()` method, letting the listener access additional methods, such as `partitions()` (which returns the `TopicPartition` instances in the list) and `records(TopicPartition)` (which gets selective records). +Again, this must be the only parameter (aside from optional `Acknowledgment`, when using manual commits or `Consumer` parameters) on the method. +The following example shows how to do so: +==== [source, java] ---- @KafkaListener(id = "pollResults", topics = "myTopic", containerFactory = "batchFactory") @@ -977,17 +1098,19 @@ public void pollResults(ConsumerRecords records) { ... } ---- +==== -IMPORTANT: If the container factory has a `RecordFilterStrategy` configured, it will be ignored for `ConsumerRecords` listeners, with a WARNing log emitted. +IMPORTANT: If the container factory has a `RecordFilterStrategy` configured, it is ignored for `ConsumerRecords` listeners, with a `WARN` log message emitted. Records can only be filtered with a batch listener if the `>` form of listener is used. ====== Annotation Properties -Starting with _version 2.0_, the `id` property (if present) is used as the Kafka consumer `group.id` property, overriding the configured property in the consumer factory, if present. -You can also set `groupId` explicitly, or set `idIsGroup` to false, to restore the previous behavior of using the consumer factory `group.id`. +Starting with version 2.0, the `id` property (if present) is used as the Kafka consumer `group.id` property, overriding the configured property in the consumer factory, if present. +You can also set `groupId` explicitly or set `idIsGroup` to false to restore the previous behavior of using the consumer factory `group.id`. -You can use property placeholders or SpEL expressions within most annotation properties, for example... +You can use property placeholders or SpEL expressions within most annotation properties, as the following example shows: +==== [source, java] ---- @KafkaListener(topics = "${some.property}") @@ -995,11 +1118,14 @@ You can use property placeholders or SpEL expressions within most annotation pro @KafkaListener(topics = "#{someBean.someProperty}", groupId = "#{someBean.someProperty}.group") ---- +==== -Starting with _version 2.1.2_, the SpEL expressions support a special token `__listener` which is a pseudo bean name which represents the current bean instance within which this annotation exists. +Starting with version 2.1.2, the SpEL expressions support a special token: `__listener`. +It is a pseudo bean name that represents the current bean instance within which this annotation exists. -For example, given... +Consider the following example: +==== [source, java] ---- @Bean @@ -1012,9 +1138,11 @@ public Listener listener2() { return new Listener("topic2"); } ---- +==== -...we can use... +Given the beans in the previous example, we can then use the following: +==== [source, java] ---- public class Listener { @@ -1037,32 +1165,37 @@ public class Listener { } ---- +==== -If, in the unlikely event that you have an actual bean called `__listener`, you can change the expression token using the `beanRef` attribute... +If, in the unlikely event that you have an actual bean called `__listener`, you can change the expression token byusing the `beanRef` attribute. +The following example shows how to do so: +==== [source, java] ---- @KafkaListener(beanRef = "__x", topics = "#{__x.topic}", groupId = "#{__x.topic}.group") ---- +==== ===== Container Thread Naming -Listener containers currently use two task executors, one to invoke the consumer and another which will be used to invoke the listener, when the kafka consumer property `enable.auto.commit` is `false`. +Listener containers currently use two task executors, one to invoke the consumer and another that is used to invoke the listener when the kafka consumer property `enable.auto.commit` is `false`. You can provide custom executors by setting the `consumerExecutor` and `listenerExecutor` properties of the container's `ContainerProperties`. When using pooled executors, be sure that enough threads are available to handle the concurrency across all the containers in which they are used. When using the `ConcurrentMessageListenerContainer`, a thread from each is used for each consumer (`concurrency`). -If you don't provide a consumer executor, a `SimpleAsyncTaskExecutor` is used; this executor creates threads with names `-C-1` (consumer thread). +If you do not provide a consumer executor, a `SimpleAsyncTaskExecutor` is used. +This executor creates threads with names similar to `-C-1` (consumer thread). For the `ConcurrentMessageListenerContainer`, the `` part of the thread name becomes `-m`, where `m` represents the consumer instance. `n` increments each time the container is started. -So, with a bean name of `container`, threads in this container will be named `container-0-C-1`, `container-1-C-1` etc., after the container is started the first time; `container-0-C-2`, `container-1-C-2` etc., after a stop/start. +So, with a bean name of `container`, threads in this container will be named `container-0-C-1`, `container-1-C-1` etc., after the container is started the first time; `container-0-C-2`, `container-1-C-2` etc., after a stop and subsequent start. [[kafka-listener-meta]] -===== @KafkaListener as a Meta Annotation +===== `@KafkaListener` as a Meta Annotation Starting with version 2.2, you can now use `@KafkaListener` as a meta annotation. -For example: +The following example shows how to do so: ==== [source, java] @@ -1086,6 +1219,7 @@ public @interface MyThreeConsumersListener { ==== You must alias at least one of `topics`, `topicPattern`, or `topicPartitions` (and, usually, `id` or `groupId` unless you have specified a `group.id` in the consumer factory configuration). +The following example shows how to do so: ==== [source, java] @@ -1098,11 +1232,13 @@ public void listen1(String in) { ==== [[class-level-kafkalistener]] -===== @KafkaListener on a Class +===== `@KafkaListener` on a Class -When using `@KafkaListener` at the class-level, you specify `@KafkaHandler` at the method level. +When you use `@KafkaListener` at the class-level, you must specify `@KafkaHandler` at the method level. When messages are delivered, the converted message payload type is used to determine which method to call. +The following example shows how to do so: +==== [source, java] ---- @KafkaListener(id = "multi", topics = "myTopic") @@ -1125,22 +1261,28 @@ static class MultiListenerBean { } ---- +==== -Starting with _version 2.1.3_, a `@KafkaHandler` method can be designated as the default method which is invoked if there is no match on other methods. -At most one method can be so designated. +Starting with version 2.1.3, you can designate a `@KafkaHandler` method as the default method that is invoked if there is no match on other methods. +At most, one method can be so designated. When using `@KafkaHandler` methods, the payload must have already been converted to the domain object (so the match can be performed). -Use a custom deserializer, the `JsonDeserializer` or the `(String|Bytes)JsonMessageConverter` with its `TypePrecedence` set to `TYPE_ID` - see <> for more information. +Use a custom deserializer, the `JsonDeserializer`, or the `(String|Bytes)JsonMessageConverter` with its `TypePrecedence` set to `TYPE_ID`. +See <> for more information. [[kafkalistener-lifecycle]] -===== @KafkaListener Lifecycle Management +===== `@KafkaListener` Lifecycle Management The listener containers created for `@KafkaListener` annotations are not beans in the application context. Instead, they are registered with an infrastructure bean of type `KafkaListenerEndpointRegistry`. This bean is automatically declared by the framework and manages the containers' lifecycles; it will auto-start any containers that have `autoStartup` set to `true`. -All containers created by all container factories must be in the same `phase` - see <> for more information. -You can manage the lifecycle programmatically using the registry; starting/stopping the registry will start/stop all the registered containers. -Or, you can get a reference to an individual container using its `id` attribute; you can set `autoStartup` on the annotation, which will override the default setting configured into the container factory. -Simply get a reference to the bean from the application context, such as auto wiring, to manage its registered containers: +All containers created by all container factories must be in the same `phase`. +See <> for more information. +You can manage the lifecycle programmatically by using the registry. +Starting or stopping the registry will start or stop all the registered containers. +Alternatively, you can get a reference to an individual container by using its `id` attribute. +You can set `autoStartup` on the annotation, which overrides the default setting configured into the container factory. +You can get a reference to the bean from the application context, such as auto-wiring, to manage its registered containers. +The following examples show how to do so: ==== [source, java] @@ -1149,9 +1291,7 @@ Simply get a reference to the bean from the application context, such as auto wi public void listen(...) { ... } ---- -==== -==== [source, java] ---- @Autowired @@ -1166,11 +1306,12 @@ private KafkaListenerEndpointRegistry registry; ==== [[kafka-validation]] -===== @KafkaListener @Payload Validation +===== `@KafkaListener` `@Payload` Validation Starting with version 2.2, it is now easier to add a `Validator` to validate `@KafkaListener` `@Payload` arguments. Previously, you had to configure a custom `DefaultMessageHandlerMethodFactory` and add it to the registrar. -Now, you can simply add the validator to the registrar itself. +Now, you can add the validator to the registrar itself. +The following code shows how to do so: ==== [source, java] @@ -1189,7 +1330,7 @@ public class Config implements KafkaListenerConfigurer { ---- ==== -NOTE: When using Spring Boot with the validation starter, a `LocalValidatorFactoryBean` is auto-configured: +NOTE: When you use Spring Boot with the validation starter, a `LocalValidatorFactoryBean` is auto-configured, as the following example shows: ==== [source, java] @@ -1210,7 +1351,7 @@ public class Config implements KafkaListenerConfigurer { ---- ==== -To validate: +The follwing examples show how to validate: ==== [source, java] @@ -1230,11 +1371,7 @@ public static class ValidatedClass { } ---- -==== -and - -==== [source, java] ---- @KafkaListener(id="validated", topics = "annotated35", errorHandler = "validationErrorHandler", @@ -1253,12 +1390,14 @@ public KafkaListenerErrorHandler validationErrorHandler() { ==== [[rebalance-listeners]] -===== Rebalance Listeners +===== Rebalancing Listeners -`ContainerProperties` has a property `consumerRebalanceListener` which takes an implementation of the Kafka client's `ConsumerRebalanceListener` interface. -If this property is not provided, the container will configure a simple logging listener that logs rebalance events under the `INFO` level. -The framework also adds a sub-interface `ConsumerAwareRebalanceListener`: +`ContainerProperties` has a property called `consumerRebalanceListener`, which takes an implementation of the Kafka client's `ConsumerRebalanceListener` interface. +If this property is not provided, the container configures a logging listener that logs rebalance events at the `INFO` level. +The framework also adds a sub-interface `ConsumerAwareRebalanceListener`. +The following listing shows the `ConsumerAwareRebalanceListener` interface definition: +==== [source, java] ---- public interface ConsumerAwareRebalanceListener extends ConsumerRebalanceListener { @@ -1271,10 +1410,14 @@ public interface ConsumerAwareRebalanceListener extends ConsumerRebalanceListene } ---- +==== -Notice that there are two callbacks when partitions are revoked: the first is called immediately; the second is called after any pending offsets are committed. -This is useful if you wish to maintain offsets in some external repository; for example: +Notice that there are two callbacks when partitions are revoked. +The first is called immediately. +The second is called after any pending offsets are committed. +This is useful if you wish to maintain offsets in some external repository, as the following example shows: +==== [source, java] ---- containerProperties.setConsumerRebalanceListener(new ConsumerAwareRebalanceListener() { @@ -1299,27 +1442,30 @@ containerProperties.setConsumerRebalanceListener(new ConsumerAwareRebalanceListe } }); ---- +==== [[annotation-send-to]] -===== Forwarding Listener Results using @SendTo +===== Forwarding Listener Results using `@SendTo` -Starting with _version 2.0_, if you also annotate a `@KafkaListener` with a `@SendTo` annotation and the method invocation returns a result, the result will be forwarded to the topic specified by the `@SendTo`. +Starting with version 2.0, if you also annotate a `@KafkaListener` with a `@SendTo` annotation and the method invocation returns a result, the result is forwarded to the topic specified by the `@SendTo`. The `@SendTo` value can have several forms: -- `@SendTo("someTopic")` routes to the literal topic -- `@SendTo("#{someExpression}")` routes to the topic determined by evaluating the expression once during application context initialization. -- `@SendTo("!{someExpression}")` routes to the topic determined by evaluating the expression at runtime. -The `#root` object for the evaluation has 3 properties: - - request - the inbound `ConsumerRecord` (or `ConsumerRecords` object for a batch listener)) - - source - the `org.springframework.messaging.Message` converted from the `request`. - - result - the method return result. -- `@SendTo` (no properties) - this is treated as `!{source.headers['kafka_replyTopic']}` (since version 2.1.3). +* `@SendTo("someTopic")` routes to the literal topic +* `@SendTo("#{someExpression}")` routes to the topic determined by evaluating the expression once during application context initialization. +* `@SendTo("!{someExpression}")` routes to the topic determined by evaluating the expression at runtime. +The `#root` object for the evaluation has three properties: +** `request`: The inbound `ConsumerRecord` (or `ConsumerRecords` object for a batch listener)) +** `source`: The `org.springframework.messaging.Message` converted from the `request`. +** `result`: The method return result. +* `@SendTo` (no properties): This is treated as `!{source.headers['kafka_replyTopic']}` (since version 2.1.3). -Starting with versions 2.1.11, 2.2.1, property placeholders are resolved within `@SendTo` values. +Starting with versions 2.1.11 and 2.2.1, property placeholders are resolved within `@SendTo` values. -The result of the expression evaluation must be a `String` representing the topic name. +The result of the expression evaluation must be a `String` that represents the topic name. +The following examples show the various ways to use `@SendTo`: +==== [source, java] ---- @KafkaListener(topics = "annotated21") @@ -1358,9 +1504,11 @@ public class MultiListenerSendTo { } ---- +==== Starting with version 2.2, you can add a `ReplyHeadersConfigurer` to the listener container factory. This is consulted to determine which headers you want to set in the reply message. +The following example shows how to add a `ReplyHeadersConfigurer`: ==== [source, java] @@ -1371,13 +1519,14 @@ public ConcurrentKafkaListenerContainerFactory kafkaListenerCon new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory(cf()); factory.setReplyTemplate(template()); - factory.setReplyHeadersConfigurer((k, v) -> k.equals("baz")); + factory.setReplyHeadersConfigurer((k, v) -> k.equals("cat")); return factory; } ---- ==== -You can also add more headers if you wish +You can also add more headers if you wish. +The following example shows how to do so: ==== [source, java] @@ -1406,9 +1555,12 @@ public ConcurrentKafkaListenerContainerFactory kafkaListenerCon ---- ==== -When using `@SendTo`, the `ConcurrentKafkaListenerContainerFactory` must be configured with a `KafkaTemplate` in its `replyTemplate` property, to perform the send. -NOTE: unless you are using <> only the simple `send(topic, value)` method is used, so you may wish to create a subclass to generate the partition and/or key: +When you use `@SendTo`, you must configure the `ConcurrentKafkaListenerContainerFactory` with a `KafkaTemplate` in its `replyTemplate` property to perform the send. +NOTE: Unless you use <> only the simple `send(topic, value)` method is used, so you may wish to create a subclass to generate the partition or key. +The following example shows how to do so: + +==== [source, java] ---- @Bean @@ -1425,9 +1577,14 @@ public KafkaTemplate myReplyingTemplate() { }; } ---- +==== -IMPORTANT: If the listener method returns `Message` or `Collection>`, the listener method is responsible for setting up the message headers for the reply; for example, when handling a request from a `ReplyingKafkaTemplate`, you might do the following: +[IMPORTANT] +==== +If the listener method returns `Message` or `Collection>`, the listener method is responsible for setting up the message headers for the reply. +For example, when handling a request from a `ReplyingKafkaTemplate`, you might do the following: +===== [source, java] ---- @KafkaListener(id = "messageReturned", topics = "someTopic") @@ -1441,12 +1598,18 @@ public Message listen(String in, @Header(KafkaHeaders.REPLY_TOPIC) byte[] rep .build(); } ---- +===== +==== When using request/reply semantics, the target partition can be requested by the sender. -NOTE: You can annotate a `@KafkaListener` method with `@SendTo` even if no result is returned. +[NOTE] +==== +You can annotate a `@KafkaListener` method with `@SendTo` even if no result is returned. This is to allow the configuration of an `errorHandler` that can forward information about a failed message delivery to some topic. +The following example shows how to do so: +===== [source, java] ---- @KafkaListener(id = "voidListenerWithReplyingErrorHandler", topics = "someTopic", @@ -1463,83 +1626,88 @@ public KafkaListenerErrorHandler voidSendToErrorHandler() { }; } ---- +===== See <> for more information. +==== ===== Filtering Messages -In certain scenarios, such as rebalancing, a message may be redelivered that has already been processed. -The framework cannot know whether such a message has been processed or not, that is an application-level -function. -This is known as the http://www.enterpriseintegrationpatterns.com/patterns/messaging/IdempotentReceiver.html[Idempotent -Receiver] pattern and Spring Integration provides an -http://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#idempotent-receiver[implementation thereof]. +In certain scenarios, such as rebalancing, a message that has already been processed may be redelivered. +The framework cannot know whether such a message has been processed or not. +That is an application-level function. +This is known as the http://www.enterpriseintegrationpatterns.com/patterns/messaging/IdempotentReceiver.html[Idempotent Receiver] pattern and Spring Integration provides an http://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#idempotent-receiver[implementation of it]. -The Spring for Apache Kafka project also provides some assistance by means of the `FilteringMessageListenerAdapter` -class, which can wrap your `MessageListener`. -This class takes an implementation of `RecordFilterStrategy` where you implement the `filter` method to signal -that a message is a duplicate and should be discarded. This has an additional property `ackDiscarded` which indicates -whether the adapter should acknowledge the discarded record; it is `false` by default. +The Spring for Apache Kafka project also provides some assistance by means of the `FilteringMessageListenerAdapter` class, which can wrap your `MessageListener`. +This class takes an implementation of `RecordFilterStrategy` in which you implement the `filter` method to signal that a message is a duplicate and should be discarded. +This has an additional property called `ackDiscarded`, which indicates whether the adapter should acknowledge the discarded record. +It is `false` by default. -When using `@KafkaListener`, set the `RecordFilterStrategy` (and optionally `ackDiscarded`) on the container factory and the listener will be wrapped in the appropriate filtering adapter. +When you use `@KafkaListener`, set the `RecordFilterStrategy` (and optionally `ackDiscarded`) on the container factory so that the listener is wrapped in the appropriate filtering adapter. -In addition, a `FilteringBatchMessageListenerAdapter` is provided, for when using a batch <>. +In addition, a `FilteringBatchMessageListenerAdapter` is provided, for when you use a batch <>. -IMPORTANT: The `FilteringBatchMessageListenerAdapter` is ignored if your `@KafkaListener` receives a `ConsumerRecords` instead of `List>` because `ConsumerRecords` is immutable. +IMPORTANT: The `FilteringBatchMessageListenerAdapter` is ignored if your `@KafkaListener` receives a `ConsumerRecords` instead of `List>`, because `ConsumerRecords` is immutable. [[retrying-deliveries]] ===== Retrying Deliveries If your listener throws an exception, the default behavior is to invoke the `ErrorHandler`, if configured, or logged otherwise. -NOTE: Two error handler interfaces are provided `ErrorHandler` and `BatchErrorHandler`; the appropriate type must be configured to match the <>. +NOTE: Two error handler interfaces (`ErrorHandler` and `BatchErrorHandler`) are provided. +You must configure the appropriate type to match the <>. To retry deliveries, a convenient listener adapter `RetryingMessageListenerAdapter` is provided. -It can be configured with a `RetryTemplate` and `RecoveryCallback` - see the https://github.com/spring-projects/spring-retry[spring-retry] +You can configure it with a `RetryTemplate` and `RecoveryCallback` - see the https://github.com/spring-projects/spring-retry[spring-retry] project for information about these components. If a recovery callback is not provided, the exception is thrown to the container after retries are exhausted. -In that case, the `ErrorHandler` will be invoked, if configured, or logged otherwise. +In that case, the `ErrorHandler` is invoked, if configured, or logged otherwise. -When using `@KafkaListener`, set the `RetryTemplate` (and optionally `recoveryCallback`) on the container factory and the listener will be wrapped in the appropriate retrying adapter. +When you use `@KafkaListener`, you can set the `RetryTemplate` (and optionally `recoveryCallback`) on the container factory. +When you do so, the listener is wrapped in the appropriate retrying adapter. -The contents of the `RetryContext` passed into the `RecoveryCallback` will depend on the type of listener. -The context will always have an attribute `record` which is the record for which the failure occurred. -If your listener is acknowledging and/or consumer aware, additional attributes `acknowledgment` and/or `consumer` will be available. +The contents of the `RetryContext` passed into the `RecoveryCallback` depend on the type of listener. +The context always has a `record` attribute, which is the record for which the failure occurred. +If your listener is acknowledging or consumer aware, additional `acknowledgment` or `consumer` attributes are available. For convenience, the `RetryingMessageListenerAdapter` provides static constants for these keys. -See its javadocs for more information. +See its https://docs.spring.io/spring-kafka/api/org/springframework/kafka/listener/adapter/AbstractRetryingMessageListenerAdapter.html[Javadoc] for more information. -A retry adapter is not provided for any of the batch <> because the framework has no knowledge of where, in a batch, the failure occurred. -Users wishing retry capabilities, when using a batch listener, are advised to use a `RetryTemplate` within the listener itself. +A retry adapter is not provided for any of the batch <>, because the framework has no knowledge of where in a batch the failure occurred. +If you need retry capabilities when you use a batch listener, we recommend that you use a `RetryTemplate` within the listener itself. [[stateful-retry]] ===== Stateful Retry -It is important to understand that the retry discussed above suspends the consumer thread (if a `BackOffPolicy` is used); there are no calls to `Consumer.poll()` during the retries. -Kafka has two properties to determine consumer health; the `session.timeout.ms` is used to determine if the consumer is active. -Since version `0.10.1.0` heartbeats are sent on a background thread so a slow consumer no longer affects that. -`max.poll.interval.ms` (default 5 minutes) is used to determine if a consumer appears to be hung (taking too long to process records from the last poll). -If the time between `poll()` s exceeds this, the broker will revoke the assigned partitions and perform a rebalance. +You should understand that the retry discussed in the <> suspends the consumer thread (if a `BackOffPolicy` is used). +There are no calls to `Consumer.poll()` during the retries. +Kafka has two properties to determine consumer health. +The `session.timeout.ms` is used to determine if the consumer is active. +Since version `0.10.1.0`, heartbeats are sent on a background thread, so a slow consumer no longer affects that. +`max.poll.interval.ms` (default: five minutes) is used to determine if a consumer appears to be hung (taking too long to process records from the last poll). +If the time between `poll()` calls exceeds this, the broker revokes the assigned partitions and performs a rebalance. For lengthy retry sequences, with back off, this can easily happen. -Since _version 2.1.3_, you can avoid this problem by using stateful retry in conjunction with a `SeekToCurrentErrorHandler`. -In this case, each delivery attempt will throw the exception back to the container and the error handler will re-seek the unprocessed offsets and the same message will be redelivered by the next `poll()`. +Since version 2.1.3, you can avoid this problem by using stateful retry in conjunction with a `SeekToCurrentErrorHandler`. +In this case, each delivery attempt throws the exception back to the container, the error handler re-seeks the unprocessed offsets, and the same message is redelivered by the next `poll()`. This avoids the problem of exceeding the `max.poll.interval.ms` property (as long as an individual delay between attempts does not exceed it). -So, when using an `ExponentialBackOffPolicy`, it's important to ensure that the `maxInterval` is rather less than the `max.poll.interval.ms` property. -To enable stateful retry, use the `RetryingMessageListenerAdapter` constructor that takes a `stateful` `boolean` argument (set it to `true`). -When configuring using the listener container factory (for `@KafkaListener` s), set the factory's `statefulRetry` property to `true`. +So, when you use an `ExponentialBackOffPolicy`, you must ensure that the `maxInterval` is less than the `max.poll.interval.ms` property. +To enable stateful retry, you can use the `RetryingMessageListenerAdapter` constructor that takes a `stateful` `boolean` argument (set it to `true`). +When you configure the listener container factory (for `@KafkaListener`), set the factory's `statefulRetry` property to `true`. [[idle-containers]] ===== Detecting Idle and Non-Responsive Consumers -While efficient, one problem with asynchronous consumers is detecting when they are idle - users might want to take -some action if no messages arrive for some period of time. +While efficient, one problem with asynchronous consumers is detecting when they are idle. +You might want to take some action if no messages arrive for some period of time. You can configure the listener container to publish a `ListenerContainerIdleEvent` when some time passes with no message delivery. -While the container is idle, an event will be published every `idleEventInterval` milliseconds. +While the container is idle, an event is published every `idleEventInterval` milliseconds. -To configure this feature, set the `idleEventInterval` on the container: +To configure this feature, set the `idleEventInterval` on the container. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -1552,9 +1720,11 @@ public KafkaMessageListenerContainer(ConsumerFactory consumerFac return container; } ---- +==== -Or, for a `@KafkaListener`... +The following example shows how to set the `idleEventInterval` for a `@KafkaListener`: +==== [source, java] ---- @Bean @@ -1567,29 +1737,33 @@ public ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory() { return factory; } ---- +==== -In each of these cases, an event will be published once per minute while the container is idle. +In each of these cases, an event is published once per minute while the container is idle. -In addition, if the broker is unreachable (at the time of writing), the consumer `poll()` method does not exit, so no messages are received, and idle events can't be generated. -To solve this issue, the container will publish a `NonResponsiveConsumerEvent` if a poll does not return within 3x the `pollInterval` property. +In addition, if the broker is unreachable, the consumer `poll()` method does not exit, so no messages are received and idle events cannot be generated. +To solve this issue, the container publishes a `NonResponsiveConsumerEvent` if a poll does not return within 3x the `pollInterval` property. By default, this check is performed once every 30 seconds in each container. -You can modify the behavior by setting the `monitorInterval` and `noPollThreshold` properties in the `ContainerProperties` when configuring the listener container. -Receiving such an event will allow you to stop the container(s), thus waking the consumer so it can terminate. +You can modify this behavior by setting the `monitorInterval` and `noPollThreshold` properties in the `ContainerProperties` when configuring the listener container. +Receiving such an event lets you stop the containers, thus waking the consumer so that it can terminate. ====== Event Consumption -You can capture these events by implementing `ApplicationListener` - either a general listener, or one narrowed to only receive this specific event. +You can capture these events by implementing `ApplicationListener` -- either a general listener or one narrowed to only receive this specific event. You can also use `@EventListener`, introduced in Spring Framework 4.2. -The following example combines the `@KafkaListener` and `@EventListener` into a single class. -It's important to understand that the application listener will get events for all containers so you may need to -check the listener id if you want to take specific action based on which container is idle. +The next example combines `@KafkaListener` and `@EventListener` into a single class. +You should understand that the application listener gets events for all containers, so you may need to +check the listener ID if you want to take specific action based on which container is idle. You can also use the `@EventListener` `condition` for this purpose. See <> for information about event properties. The event is normally published on the consumer thread, so it is safe to interact with the `Consumer` object. +The following example uses both `@KafkaListener` and `@EventListener`: + +==== [source, xml] ---- public class Listener { @@ -1606,37 +1780,43 @@ public class Listener { } ---- +==== -IMPORTANT: Event listeners will see events for all containers; so, in the example above, we narrow the events received based on the listener ID. +IMPORTANT: Event listeners see events for all containers. +Consequently, in the preceding example, we narrow the events received based on the listener ID. Since containers created for the `@KafkaListener` support concurrency, the actual containers are named `id-n` where the `n` is a unique value for each instance to support the concurrency. -Hence we use `startsWith` in the condition. +That is why we use `startsWith` in the condition. -CAUTION: If you wish to use the idle event to stop the lister container, you should not call `container.stop()` on the thread that calls the listener - it will cause delays and unnecessary log messages. +CAUTION: If you wish to use the idle event to stop the lister container, you should not call `container.stop()` on the thread that calls the listener. +Doing so causes delays and unnecessary log messages. Instead, you should hand off the event to a different thread that can then stop the container. -Also, you should not `stop()` the container instance in the event if it is a child container, you should stop the concurrent container instead. +Also, you should not `stop()` the container instance if it is a child container. +You should stop the concurrent container instead. ====== Current Positions when Idle -Note that you can obtain the current positions when idle is detected by implementing `ConsumerSeekAware` in your listener; see `onIdleContainer()` in `<>. +Note that you can obtain the current positions when idle is detected by implementing `ConsumerSeekAware` in your listener. +See `onIdleContainer()` in `<>. ===== Topic/Partition Initial Offset There are several ways to set the initial offset for a partition. -When manually assigning partitions, simply set the initial offset (if desired) in the configured `TopicPartitionInitialOffset` arguments (see <>). +When manually assigning partitions, you can set the initial offset (if desired) in the configured `TopicPartitionInitialOffset` arguments (see <>). You can also seek to a specific offset at any time. -When using group management where the broker assigns partitions: +When you use group management where the broker assigns partitions: -- For a new `group.id`, the initial offset is determined by the `auto.offset.reset` consumer property (`earliest` or `latest`). -- For an existing group id, the initial offset is the current offset for that group id. +* For a new `group.id`, the initial offset is determined by the `auto.offset.reset` consumer property (`earliest` or `latest`). +* For an existing group ID, the initial offset is the current offset for that group ID. You can, however, seek to a specific offset during initialization (or at any time thereafter). [[seek]] ===== Seeking to a Specific Offset -In order to seek, your listener must implement `ConsumerSeekAware` which has the following methods: +In order to seek, your listener must implement `ConsumerSeekAware`, which has the following methods: +==== [source, java] ---- void registerSeekCallback(ConsumerSeekCallback callback); @@ -1645,16 +1825,22 @@ void onPartitionsAssigned(Map assignments, ConsumerSeekCal void onIdleContainer(Map assignments, ConsumerSeekCallback callback); ---- +==== -The first is called when the container is started; this callback should be used when seeking at some arbitrary time after initialization. -You should save a reference to the callback; if you are using the same listener in multiple containers (or in a `ConcurrentMessageListenerContainer`) you should store the callback in a `ThreadLocal` or some other structure keyed by the listener `Thread`. +The first method is called when the container is started. +You should use this callback when seeking at some arbitrary time after initialization. +You should save a reference to the callback. +If you use the same listener in multiple containers (or in a `ConcurrentMessageListenerContainer`), you should store the callback in a `ThreadLocal` or some other structure keyed by the listener `Thread`. When using group management, the second method is called when assignments change. -You can use this method, for example, for setting initial offsets for the partitions, by calling the callback; you must use the callback argument, not the one passed into `registerSeekCallback`. -This method will never be called if you explicitly assign partitions yourself; use the `TopicPartitionInitialOffset` in that case. +You can use this method, for example, for setting initial offsets for the partitions, by calling the callback. +You must use the callback argument, not the one passed into `registerSeekCallback`. +This method is never called if you explicitly assign partitions yourself. +Use the `TopicPartitionInitialOffset` in that case. -The callback has these methods: +The callback has the following methods: +==== [source, java] ---- void seek(String topic, int partition, long offset); @@ -1663,20 +1849,24 @@ void seekToBeginning(String topic, int partition); void seekToEnd(String topic, int partition); ---- +==== -You can also perform seek operations from `onIdleContainer()` when an idle container is detected; see <> for how to enable idle container detection. +You can also perform seek operations from `onIdleContainer()` when an idle container is detected. +See <> for how to enable idle container detection. To arbitrarily seek at runtime, use the callback reference from the `registerSeekCallback` for the appropriate thread. [[container-factory]] ===== Container factory -As discussed in <> a `ConcurrentKafkaListenerContainerFactory` is used to create containers for annotated methods. +As discussed in <>, a `ConcurrentKafkaListenerContainerFactory` is used to create containers for annotated methods. -Starting with _version 2.2_, the same factory can be used to create any `ConcurrentMessageListenerContainer`. -This might be useful if you want to create several containers with similar properties, or you wish to use some externally configured factory, such as the one provided by Spring Boot auto configuration. +Starting with version 2.2, you can use the same factory to create any `ConcurrentMessageListenerContainer`. +This might be useful if you want to create several containers with similar properties or you wish to use some externally configured factory, such as the one provided by Spring Boot auto-configuration. Once the container is created, you can further modify its properties, many of which are set by using `container.getContainerProperties()`. +The following example configures a `ConcurrentMessageListenerContainer`: +==== [source, java] ---- @Bean @@ -1684,50 +1874,54 @@ public ConcurrentMessageListenerContainer( ConcurrentKafkaListenerContainerFactory factory) { ConcurrentMessageListenerContainer container = - factory.createContainer("topic1", "topci2"); + factory.createContainer("topic1", "topic2"); container.setMessageListener(m -> { ... } ); return container; } ---- +==== IMPORTANT: Containers created this way are not added to the endpoint registry. -They should be created as `@Bean` s so that they will be registered with the application context. +They should be created as `@Bean` definitions so that they are registered with the application context. [[thread-safety]] ===== Thread Safety When using a concurrent message listener container, a single listener instance is invoked on all consumer threads. -Listeners, therefore, need to be thread-safe; and it is preferable to use stateless listeners. -If it is not possible to make your listener thread-safe, or adding synchronization would significantly reduce the benefit of adding concurreny, there are several techniques you can use. +Listeners, therefore, need to be thread-safe, and it is preferable to use stateless listeners. +If it is not possible to make your listener thread-safe or adding synchronization would significantly reduce the benefit of adding concurrency, you can use one of a few techniques: -. Use `n` containers with `concurrency=1` with a prototype scoped `MessageListener` bean so each container gets its own instance (this is not possible when using `@KafkaListener`). -. Keep the state in `ThreadLocal` s. -. Have the singleton listener delegate to a bean that is declared in `SimpleThreadScope` or similar. +* Use `n` containers with `concurrency=1` with a prototype scoped `MessageListener` bean so that each container gets its own instance (this is not possible when using `@KafkaListener`). +* Keep the state in `ThreadLocal` instances. +* Have the singleton listener delegate to a bean that is declared in `SimpleThreadScope` (or a similar scope). -To facilitate cleaning up thread state (for 2 and 3), starting with version 2.2, the listener container will publish `ConsumerStoppedEvent` s when each thread exits. -Consume these events with an `ApplicationListener` or `@EventListener` method to remove `ThreadLocal` s, or `remove()` thread-scoped beans from the scope. -Note that `SimpleThreadScope` does not destroy beans that have a destruction interface (e.g. `DisposableBean`) so you should `destroy()` the instance yourself. +To facilitate cleaning up thread state (for the second and third items in the preceding list), starting with version 2.2, the listener container publishes a `ConsumerStoppedEvent` when each thread exits. +You can consume these events with an `ApplicationListener` or `@EventListener` method to remove `ThreadLocal` instances or `remove()` thread-scoped beans from the scope. +Note that `SimpleThreadScope` does not destroy beans that have a destruction interface (such as `DisposableBean`), so you should `destroy()` the instance yourself. IMPORTANT: By default, the application context's event multicaster invokes event listeners on the calling thread. -If you change the multicaster to use an async executor, thread cleanup will not be effective. +If you change the multicaster to use an async executor, thread cleanup is not effective. [[pause-resume]] -==== Pausing/Resuming Listener Containers +==== Pausing and Resuming Listener Containers -_Version 2.1.3_ added `pause()` and `resume()` methods to listener containers. -Previously, you could pause a consumer within a `ConsumerAwareMessageListener` and resume it by listening for `ListenerContainerIdleEvent` s, which provide access to the `Consumer` object. -While you could pause a consumer in an idle container via an event listener, in some cases this was not thread-safe since there is no guarantee that the event listener is invoked on the consumer thread. -To safely pause/resume consumers, you should use the methods on the listener containers. -`pause()` takes effect just before the next `poll()`; `resume` takes effect, just after the current `poll()` returns. -When a container is paused, it continues to `poll()` the consumer, avoiding a rebalance if group management is being used, but will not retrieve any records; refer to the Kafka documentation for more information. +Version 2.1.3 added `pause()` and `resume()` methods to listener containers. +Previously, you could pause a consumer within a `ConsumerAwareMessageListener` and resume it by listening for a `ListenerContainerIdleEvent`, which provides access to the `Consumer` object. +While you could pause a consumer in an idle container byi using an event listener, in some cases, this was not thread-safe, since there is no guarantee that the event listener is invoked on the consumer thread. +To safely pause and resume consumers, you should use the `pause` and `resume` methods on the listener containers. +A `pause()` takes effect just before the next `poll()`; a `resume()` takes effect just after the current `poll()` returns. +When a container is paused, it continues to `poll()` the consumer, avoiding a rebalance if group management is being used, but it does not retrieve any records. +See the Kafka documentation for more information. -Starting with _version 2.1.5_, you can call `isPauseRequested()` to see if `pause()` has been called. -However, the consumers might not have actually paused yet; `isConsumerPaused()` will return true if all `Consumer` s have actually paused. +Starting with version 2.1.5, you can call `isPauseRequested()` to see if `pause()` has been called. +However, the consumers might not have actually paused yet. +`isConsumerPaused()` returns true if all `Consumer` instances have actually paused. -In addition, also since _2.1.5_, `ConsumerPausedEvent` s and `ConsumerResumedEvent` s are published with the container as the `source` property and the `TopicPartition` s involved in the `partitions` s property. +In addition (also since 2.1.5), `ConsumerPausedEvent` and `ConsumerResumedEvent` instances are published with the container as the `source` property and the `TopicPartition` instances involved in the `partitions` property. -This simple Spring Boot application demonstrates using the container registry to get a reference to a `@KafkaListener` method's container and pausing/resuming its consumers, as well as receiving the corresponding events. +The following simple Spring Boot application demonstrates by using the container registry to get a reference to a `@KafkaListener` method's container and pausing or resuming its consumers as well as receiving the corresponding events: +==== [source, java] ---- @SpringBootApplication @@ -1746,12 +1940,12 @@ public class Application implements ApplicationListener { public ApplicationRunner runner(KafkaListenerEndpointRegistry registry, KafkaTemplate template) { return args -> { - template.send("pause.resume.topic", "foo"); + template.send("pause.resume.topic", "thing1"); Thread.sleep(10_000); System.out.println("pausing"); registry.getListenerContainer("pause.resume").pause(); Thread.sleep(10_000); - template.send("pause.resume.topic", "bar"); + template.send("pause.resume.topic", "thing2"); Thread.sleep(10_000); System.out.println("resuming"); registry.getListenerContainer("pause.resume").resume(); @@ -1771,80 +1965,88 @@ public class Application implements ApplicationListener { } ---- +==== -With results: +The following listing shows the results of the preceding example: +==== [source] ---- partitions assigned: [pause.resume.topic-1, pause.resume.topic-0] -foo +thing1 pausing ConsumerPausedEvent [partitions=[pause.resume.topic-1, pause.resume.topic-0]] resuming ConsumerResumedEvent [partitions=[pause.resume.topic-1, pause.resume.topic-0]] -bar +thing2 ---- +==== [[events]] ==== Events The following events are published by listener containers and their consumers: -* `ContainerIdleEvent` - when no messages have been received in `idleInterval` (if configured) -* `NonResponsiveConsumerEvent` - when the consumer appears to be blocked in the `poll` method -* `ConsumerPausedEvent` - issued by each consumer when the container is paused -* `ConsumerResumedEvent` - issued by each consumer when the container is resumed -* `ConsumerStoppingEvent` - issued by each consumer just before stopping -* `ConsumerStoppedEvent` - issued after the consumer is closed; see <> -* `ContainerStoppedEvent` - when all consumers have terminated +* `ContainerIdleEvent`: Issued when no messages have been received in `idleInterval` (if configured). +* `NonResponsiveConsumerEvent`: Issued when the consumer appears to be blocked in the `poll` method. +* `ConsumerPausedEvent`: Issued by each consumer when the container is paused. +* `ConsumerResumedEvent`: Issued by each consumer when the container is resumed. +* `ConsumerStoppingEvent`: Issued by each consumer just before stopping. +* `ConsumerStoppedEvent`: Issued after the consumer is closed. +See <>. +* `ContainerStoppedEvent`: Issued when all consumers have terminated. IMPORTANT: By default, the application context's event multicaster invokes event listeners on the calling thread. If you change the multicaster to use an async executor, you must not invoke any `Consumer` methods when the event contains a reference to the consumer. -The `ContainerIdleEvent` has 6 properties: +The `ContainerIdleEvent` has the following properties: -- `source` - the listener container instance that published the event. -- `container` - the listener container or the parent listener container if the source container is a child. -- `id` - the listener id (or container bean name). -- `idleTime` - the time the container had been idle when the event was published. -- `topicPartitions` - the topics/partitions that the container was assigned at the time the event was generated. -- `consumer` - a reference to the kafka `Consumer` object; for example, if the consumer was previously `pause()` d, it can be `resume()` d when the event is received. -- `paused` if the container is currently paused; see <> for more information. +* `source`: The listener container instance that published the event. +* `container`: The listener container or the parent listener container, if the source container is a child. +* `id`: The listener ID (or container bean name). +* `idleTime`: The time the container had been idle when the event was published. +* `topicPartitions`: The topics and partitions that the container was assigned at the time the event was generated. +* `consumer`: A reference to the Kafka `Consumer` object. +For example, if the consumer's `pause()` method was previously called, it can `resume()` when the event is received. +* `paused`: Whether the container is currently paused. +See <> for more information. The `NonResponsiveConsumerEvent` has the following properties: -- `source` - the listener container instance that published the event. -- `container` - the listener container or the parent listener container if the source container is a child. -- `id` - the listener id (or container bean name). -- `timeSinceLastPoll` - the time just before the container last called `poll()`. -- `topicPartitions` - the topics/partitions that the container was assigned at the time the event was generated. -- `consumer` - a reference to the kafka `Consumer` object; for example, if the consumer was previously `pause()` d, it can be `resume()` d when the event is received. -- `paused` if the container is currently paused; see <> for more information. +* `source`: The listener container instance that published the event. +* `container`: The listener container or the parent listener container, if the source container is a child. +* `id`: The listener ID (or container bean name). +* `timeSinceLastPoll`: The time just before the container last called `poll()`. +* `topicPartitions`: The topics and partitions that the container was assigned at the time the event was generated. +* `consumer`: A reference to the Kafka `Consumer` object. +For example, if the consumer's `pause()` method was previously called, it can `resume()` when the event is received. +* `paused`: Whether the container is currently paused. +See <> for more information. -The `ConsumerPausedEvent`, `ConsumerResumedEvent` and `ConsumerStopping` events have the following properties: +The `ConsumerPausedEvent`, `ConsumerResumedEvent`, and `ConsumerStopping` events have the following properties: -- `source` - the listener container instance that published the event. -- `container` - the listener container or the parent listener container if the source container is a child. -- `partitions` - the `TopicPartition` s involved. +* `source`: The listener container instance that published the event. +* `container`: The listener container or the parent listener container, if the source container is a child. +* `partitions`: The `TopicPartition` instances involved. -The `ConsumerStoppedEvent` and `ContainerStoppedEvent` have the following properties: +The `ConsumerStoppedEvent` and `ContainerStoppedEvent` events have the following properties: -- `source` - the listener container instance that published the event. -- `container` - the listener container or the parent listener container if the source container is a child. +* `source`: The listener container instance that published the event. +* `container`: The listener container or the parent listener container, if the source container is a child. -The `ContainerStoppedEvent` is published by all containers (regardless of whether it is a child or parent). +All containers (whether a child or a parent) publish `ContainerStoppedEvent`. For a parent container, the source and container properties are identical. [[serdes]] -==== Serialization/Deserialization and Message Conversion +==== Serialization, Deserialization, and Message Conversion -===== Overview - -Apache Kafka provides a high-level API for serializing/deserializing record values as well as their keys. +Apache Kafka provides a high-level API for serializing and deserializing record values as well as their keys. It is present with the `org.apache.kafka.common.serialization.Serializer` and `org.apache.kafka.common.serialization.Deserializer` abstractions with some built-in implementations. -Meanwhile we can specify simple (de)serializer classes using Producer and/or Consumer configuration properties, e.g.: +Meanwhile, we can specify serializer and deserializer classes by using `Producer` or `Consumer` configuration properties. +The following example shows how to do so: +==== [source, java] ---- props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class); @@ -1853,75 +2055,91 @@ props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.cla props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); ---- +==== -For more complex or particular cases, the `KafkaConsumer`, and therefore `KafkaProducer`, provides overloaded -constructors to accept `(De)Serializer` instances for `keys` and/or `values`, respectively. +For more complex or particular cases, the `KafkaConsumer` (and, therefore, `KafkaProducer`) provides overloaded +constructors to accept `Serializer` and `Deserializer` instances for `keys` and `values`, respectively. -Using this API, the `DefaultKafkaProducerFactory` and `DefaultKafkaConsumerFactory` also provide properties (via constructors or setter methods) to inject custom `(De)Serializer` s to the target `Producer`/`Consumer`. +When you use this API, the `DefaultKafkaProducerFactory` and `DefaultKafkaConsumerFactory` also provide properties (through constructors or setter methods) to inject custom `Serializer` and `Deserializer` instances into the target `Producer` or `Consumer`. -Spring for Apache Kafka also provides `JsonSerializer`/`JsonDeserializer` implementations based on the +Spring for Apache Kafka also provides `JsonSerializer` and `JsonDeserializer` implementations that are based on the Jackson JSON object mapper. -The `JsonSerializer` is quite simple and just allows writing any Java object as a JSON `byte[]`, the `JsonDeserializer` +The `JsonSerializer` allows writing any Java object as a JSON `byte[]`. +The `JsonDeserializer` requires an additional `Class targetType` argument to allow the deserialization of a consumed `byte[]` to the proper target object. +The following example shows how to create a `JsonDeserializer`: +==== [source, java] ---- -JsonDeserializer barDeserializer = new JsonDeserializer<>(Bar.class); +JsonDeserializer thingDeserializer = new JsonDeserializer<>(Thing.class); ---- +==== -Both `JsonSerializer` and `JsonDeserializer` can be customized with an `ObjectMapper`. +You can customize both `JsonSerializer` and `JsonDeserializer` with an `ObjectMapper`. You can also extend them to implement some particular configuration logic in the `configure(Map configs, boolean isKey)` method. -Starting with _version 2.1_, type information can be conveyed in record `Headers`, allowing the handling of multiple types. -In addition, the serializer/deserializer can be configured using Kafka properties. +Starting with version 2.1, you can convey type information in record `Headers`, allowing the handling of multiple types. +In addition, you can configure the serializer and deserializer by using the following Kafka properties: -- `JsonSerializer.ADD_TYPE_INFO_HEADERS` (default `true`); set to `false` to disable this feature on the `JsonSerializer` (sets the `addTypeInfo` property). -- `JsonSerializer.TYPE_MAPPINGS` (default `empty`); see below. -- `JsonDeserializer.USE_TYPE_INFO_HEADERS` (default `true`); set to `false` to ignore headers set by the serializer. -- `JsonDeserializer.REMOVE_TYPE_INFO_HEADERS` (default `true`); set to `false` to retain headers set by the serializer. -- `JsonDeserializer.KEY_DEFAULT_TYPE`; fallback type for deserialization of keys if no header information is present. -- `JsonDeserializer.VALUE_DEFAULT_TYPE`; fallback type for deserialization of values if no header information is present. -- `JsonDeserializer.TRUSTED_PACKAGES` (default `java.util`, `java.lang`); comma-delimited list of package patterns allowed for deserialization; `*` means deserialize all. -- `JsonDeserializer.TYPE_MAPPINGS` (default `empty`); see below. +* `JsonSerializer.ADD_TYPE_INFO_HEADERS` (default `true`): You can set it to `false` to disable this feature on the `JsonSerializer` (sets the `addTypeInfo` property). +* `JsonSerializer.TYPE_MAPPINGS` (default `empty`): See <>. +* `JsonDeserializer.USE_TYPE_INFO_HEADERS` (default `true`): You can set it to `false` to ignore headers set by the serializer. +* `JsonDeserializer.REMOVE_TYPE_INFO_HEADERS` (default `true`): You can set it to `false` to retain headers set by the serializer. +* `JsonDeserializer.KEY_DEFAULT_TYPE`: Fallback type for deserialization of keys if no header information is present. +* `JsonDeserializer.VALUE_DEFAULT_TYPE`: Fallback type for deserialization of values if no header information is present. +* `JsonDeserializer.TRUSTED_PACKAGES` (default `java.util`, `java.lang`): Comma-delimited list of package patterns allowed for deserialization. +`*` means deserialize all. +* `JsonDeserializer.TYPE_MAPPINGS` (default `empty`): See <>. -Starting with version 2.2, the type information headers (if added by the serializer) will be removed by the deserializer. -You can revert to the previous behavior by setting the `removeTypeHeaders` property to false, either directly on the deserializer, or with the configuration property described above. +Starting with version 2.2, the type information headers (if added by the serializer) are removed by the deserializer. +You can revert to the previous behavior by setting the `removeTypeHeaders` property to `false`, either directly on the deserializer or with the configuration property described earlier. -**Mapping Types** +[[serdes-mapping-types]] +===== Mapping Types -Starting with version 2.2, you can now provide type mappings using the properties in the above list; previously you had to customize the type mapper within the serializer, deserializer. -Mappings consist of a comma-delimited list of `token:className` pairs; on outbound, the payload's class name is mapped to the corresponding token and, on inbound, the token in the type header is mapped to the corresponding class name. +Starting with version 2.2, you can now provide type mappings by using the properties in the preceding list. +Previously, you had to customize the type mapper within the serializer and deserializer. +Mappings consist of a comma-delimited list of `token:className` pairs. +On outbound, the payload's class name is mapped to the corresponding token. +On inbound, the token in the type header is mapped to the corresponding class name. -For example: +The following example creates a set of mappings: ==== [source, java] ---- senderProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class); -senderProps.put(JsonSerializer.TYPE_MAPPINGS, "foo:com.myfoo.Foo, bar:com.mybar.bar"); +senderProps.put(JsonSerializer.TYPE_MAPPINGS, "cat:com.mycat.Cat, hat:com.myhat.hat"); ... consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class); -consumerProps.put(JsonDeSerializer.TYPE_MAPPINGS, "foo:com.yourfoo.Foo, bar:com.yourbar.bar"); +consumerProps.put(JsonDeSerializer.TYPE_MAPPINGS, "cat:com.yourcat.Cat, hat:com.yourhat.hat"); ---- ==== -Of course, the corresponding objects must be compatible. +IMPORTANT: The corresponding objects must be compatible. -https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-messaging.html#boot-features-kafka[Spring Boot], these properties can be provided in the `application.properties` (or yaml) file: +If you use https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-messaging.html#boot-features-kafka[Spring Boot], you can provide these properties in the `application.properties` (or yaml) file. +The following example shows how to do so: ==== [source] ---- spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer -spring.kafka.producer.properties.spring.json.type.mapping=foo:com.myfoo.Foo,bar:com.mybar.Bar +spring.kafka.producer.properties.spring.json.type.mapping=cat:com.mycat.Cat,hat:com.myhat.Hat ---- ==== -IMPORTANT: Only simple configuration can be performed with properties; for more advanced configuration (such as using a custom `ObjectMapper` in the serializer/deserializer), you should use the producer/consumer factory constructors that accept a pre-built serializer and deserializer. For example, with Spring Boot, to override the default factories: +[IMPORTANT] +==== +You can perform only simple configuration with properties. +For more advanced configuration (such as using a custom `ObjectMapper` in the serializer and deserializer), you should use the producer and consumer factory constructors that accept a pre-built serializer and deserializer. +The following Spring Boot example overrides the default factories: +===== [source, java] ---- @Bean @@ -1940,28 +2158,33 @@ public ProducererFactory kafkaProducerFactory(KafkaProperties properti customSerializer, customSerializer); } ---- +===== Setters are also provided, as an alternative to using these constructors. +==== -Starting with version 2.2, you can explicitly configure the deserializer to use the supplied target type and ignore type information in headers, using one of the overloaded constructors that have a boolean `useHeadersIfPresent` (which is `true` by default): +Starting with version 2.2, you can explicitly configure the deserializer to use the supplied target type and ignore type information in headers by using one of the overloaded constructors that have a boolean `useHeadersIfPresent` (which is `true` by default). +The following example shows how to do so: ==== [source, java] ---- -DefaultKafkaConsumerFactory cf = new DefaultKafkaConsumerFactory<>(props, - new IntegerDeserializer(), new JsonDeserializer<>(Foo1.class, false)); +DefaultKafkaConsumerFactory cf = new DefaultKafkaConsumerFactory<>(props, + new IntegerDeserializer(), new JsonDeserializer<>(Cat1.class, false)); ---- ==== ===== Spring Messaging Message Conversion -Although the `Serializer`/`Deserializer` API is quite simple and flexible from the low-level Kafka `Consumer` and -`Producer` perspective, you might need more flexibility at the Spring Messaging level, either when using `@KafkaListener` or <>. -To easily convert to/from `org.springframework.messaging.Message`, Spring for Apache Kafka provides a `MessageConverter` +Although the `Serializer` and `Deserializer` API is quite simple and flexible from the low-level Kafka `Consumer` and +`Producer` perspective, you might need more flexibility at the Spring Messaging level, when using either `@KafkaListener` or <>. +To let you easily convert to and from `org.springframework.messaging.Message`, Spring for Apache Kafka provides a `MessageConverter` abstraction with the `MessagingMessageConverter` implementation and its `StringJsonMessageConverter` and `BytesJsonMessageConverter` customization. -The `MessageConverter` can be injected into `KafkaTemplate` instance directly and via -`AbstractKafkaListenerContainerFactory` bean definition for the `@KafkaListener.containerFactory()` property: +You can inject the `MessageConverter` into a `KafkaTemplate` instance directly and by using +`AbstractKafkaListenerContainerFactory` bean definition for the `@KafkaListener.containerFactory()` property. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -1975,44 +2198,50 @@ public KafkaListenerContainerFactory kafkaJsonListenerContainerFactory() { ... @KafkaListener(topics = "jsonData", containerFactory = "kafkaJsonListenerContainerFactory") -public void jsonListener(Foo foo) { +public void jsonListener(Cat cat) { ... } ---- +==== -When using a `@KafkaListener`, the parameter type is provided to the message converter to assist with the conversion. +When you use a `@KafkaListener`, the parameter type is provided to the message converter to assist with the conversion. [NOTE] ==== -This type inference can only be achieved when the `@KafkaListener` annotation is declared at the method level. -With a class-level `@KafkaListener`, the payload type is used to select which `@KafkaHandler` method to invoke so it must already have been converted before the method can be chosen. +This type inference can be achieved only when the `@KafkaListener` annotation is declared at the method level. +With a class-level `@KafkaListener`, the payload type is used to select which `@KafkaHandler` method to invoke, so it must already have been converted before the method can be chosen. ==== -NOTE: When using the `StringJsonMessageConverter`, you should use a `StringDeserializer` in the kafka consumer configuration and `StringSerializer` in the kafka producer configuration, when using Spring Integration or the `KafkaTemplate.send(Message message)` method. -When using the `BytesJsonMessageConverter`, you should use a `BytesDeserializer` in the kafka consumer configuration and `BytesSerializer` in the kafka producer configuration, when using Spring Integration or the `KafkaTemplate.send(Message message)` method (see <>). -Generally, the `BytesJsonMessageConverter` is more efficient because it avoids a `String` to/from `byte[]` conversion. +NOTE: When you use the `StringJsonMessageConverter`, you should use a `StringDeserializer` in the Kafka consumer configuration and a `StringSerializer` in the Kafka producer configuration when you use Spring Integration or the `KafkaTemplate.send(Message message)` method. +When you use the `BytesJsonMessageConverter`, you should use a `BytesDeserializer` in the Kafka consumer configuration and `BytesSerializer` in the Kafka producer configuration when you use Spring Integration or the `KafkaTemplate.send(Message message)` method (see <>). +Generally, the `BytesJsonMessageConverter` is more efficient because it avoids a `String` to and from `byte[]` conversion. [[error-handling-deserializer]] -===== ErrorHandlingDeserializer +===== Using `ErrorHandlingDeserializer` -When a deserializer fails to deserialize a message, Spring has no way to handle the problem because it occurs before the `poll()` returns. +When a deserializer fails to deserialize a message, Spring has no way to handle the problem, because it occurs before the `poll()` returns. To solve this problem, version 2.2 introduced the `ErrorHandlingDeserializer2`. This deserializer delegates to a real deserializer (key or value). -If the delegate fails to deserialize the record content, the `ErrorHandlingDeserializer2` returns a `null` value and a `DeserializationException` in a header, containing the cause and raw bytes. -When using a record-level `MessageListener`, if the `ConsumerRecord` contains a `DeserializationException` header for either the key or value, the container's `ErrorHandler` is called with the failed `ConsumerRecord`; the record is not passed to the listener. +If the delegate fails to deserialize the record content, the `ErrorHandlingDeserializer2` returns a `null` value and a `DeserializationException` in a header that contains the cause and the raw bytes. +When you use a record-level `MessageListener`, if the `ConsumerRecord` contains a `DeserializationException` header for either the key or value, the container's `ErrorHandler` is called with the failed `ConsumerRecord`. +The record is not passed to the listener. -Alternatively, you can configure the `ErrorHandlingDeserializer2` to create a custom value by providing a `failedDeserializationFunction` which is a `BiConsumer`. -This function is invoked to create an instance of `T` which is passed to the listener, as normal. +Alternatively, you can configure the `ErrorHandlingDeserializer2` to create a custom value by providing a `failedDeserializationFunction`, which is a `BiConsumer`. +This function is invoked to create an instance of `T`, which is passed to the listener in the usual fashion. The raw record value and headers are provided to the function. -The `DeserializationException` can be found (as a serialized Java object) in headers; see the javadocs for the `ErrorHandlingDeserializer2` for more information. +You can find the `DeserializationException` (as a serialized Java object) in headers. +See the https://docs.spring.io/spring-kafka/api/org/springframework/kafka/support/serializer/ErrorHandlingDeserializer2.html[Javadoc] for the `ErrorHandlingDeserializer2` for more information. -When using a `BatchMessageListener`, you **must** provide a `failedDeserializationFunction`, otherwise, the batch of records will not be type safe. +CAUTION: When you use a `BatchMessageListener`, you must provide a `failedDeserializationFunction`. +Otherwise, the batch of records are not type safe. -You can use the `DefaultKafkaConsumerFactory` constructor that takes key and value `Deserializer` objects and wire in appropriate `ErrorHandlingDeserializer2` configured with the proper delegates. -Alternatively, you can use consumer configuration properties which are used by the `ErrorHandlingDeserializer` to instantiate the delegates. -The property names are `ErrorHandlingDeserializer2.KEY_DESERIALIZER_CLASS` and `ErrorHandlingDeserializer2.VALUE_DESERIALIZER_CLASS`; the property value can be a class or class name. -For example: +You can use the `DefaultKafkaConsumerFactory` constructor that takes key and value `Deserializer` objects and wire in appropriate `ErrorHandlingDeserializer2` instances that you have configured with the proper delegates. +Alternatively, you can use consumer configuration properties (which are used by the `ErrorHandlingDeserializer`) to instantiate the delegates. +The property names are `ErrorHandlingDeserializer2.KEY_DESERIALIZER_CLASS` and `ErrorHandlingDeserializer2.VALUE_DESERIALIZER_CLASS`. +The property value can be a class or class name. +The following example shows how to set these properties: +==== [source, java] ---- ... // other props @@ -2025,9 +2254,11 @@ props.put(JsonDeserializer.VALUE_DEFAULT_TYPE, "com.example.MyValue") props.put(JsonDeserializer.TRUSTED_PACKAGES, "com.example") return new DefaultKafkaConsumerFactory<>(props); ---- +==== -The following is an example of using a `failedDeserializationFunction`. +The following example uses a `failedDeserializationFunction`. +==== [source, java] ---- public class BadFoo extends Foo { @@ -2053,9 +2284,11 @@ public class FailedFooProvider implements BiFunction { } ---- +==== -and config +The preceding example uses the following configuration: +==== [source, java] ---- ... @@ -2064,19 +2297,22 @@ consumerProps.put(ErrorHandlingDeserializer2.VALUE_DESERIALIZER_CLASS, JsonDeser consumerProps.put(ErrorHandlingDeserializer2.VALUE_FUNCTION, FailedFooProvider.class); ... ---- +==== [[payload-conversion-with-batch]] ===== Payload Conversion with Batch Listeners -Starting with _version 1.3.2_, you can also use a `StringJsonMessageConverter` or `BytesJsonMessageConverter` within a `BatchMessagingMessageConverter` for converting batch messages, when using a batch listener container factory. +Starting with version 1.3.2, you can also use a `StringJsonMessageConverter` or `BytesJsonMessageConverter` within a `BatchMessagingMessageConverter` to convert batch messages when you use a batch listener container factory. See <> for more information. By default, the type for the conversion is inferred from the listener argument. -If you configure the `(Bytes|String)JsonMessageConverter` with a `DefaultJackson2TypeMapper` that has its `TypePrecedence` set to `TYPE_ID` (instead of the default `INFERRED`), then the converter will use type information in headers (if present) instead. +If you configure the `(Bytes|String)JsonMessageConverter` with a `DefaultJackson2TypeMapper` that has its `TypePrecedence` set to `TYPE_ID` (instead of the default `INFERRED`), the converter uses the type information in headers (if present) instead. This allows, for example, listener methods to be declared with interfaces instead of concrete classes. -Also, the type converter supports mapping so the deserialization can be to a different type than the source (as long as the data is compatible). -This is also useful when using <> where the payload must have already been converted, to determine which method to invoke. +Also, the type converter supports mapping, so the deserialization can be to a different type than the source (as long as the data is compatible). +This is also useful when you use <> where the payload must have already been converted to determine which method to invoke. +The following example creates beans that use this method: +==== [source, java] ---- @Bean @@ -2094,9 +2330,11 @@ public StringJsonMessageConverter converter() { return new StringJsonMessageConverter(); } ---- +==== -Note that for this to work, the method signature for the conversion target must be a container object with a single generic parameter type, such as: +Note that, for this to work, the method signature for the conversion target must be a container object with a single generic parameter type, such as the following: +==== [source, java] ---- @KafkaListener(topics = "blc1") @@ -2104,11 +2342,14 @@ public void listen(List foos, @Header(KafkaHeaders.OFFSET) List offse ... } ---- +==== -Notice that you can still access the batch headers too. +Note that you can still access the batch headers. -If the batch converter has a record converter that supports it, you can also receive a list of messages where the payloads are converted according to the generic type: +If the batch converter has a record converter that supports it, you can also receive a list of messages where the payloads are converted according to the generic type. +The following example shows how to do so: +==== [source, java] ---- @KafkaListener(topics = "blc3", groupId = "blc3") @@ -2116,34 +2357,36 @@ public void listen1(List> fooMessages) { ... } ---- +==== -===== ConversionService Customization +===== `ConversionService` Customization -Starting with _version 2.1.1_, the `org.springframework.core.convert.ConversionService` used by the default +Starting with version 2.1.1, the `org.springframework.core.convert.ConversionService` used by the default `o.s.messaging.handler.annotation.support.MessageHandlerMethodFactory` to resolve parameters for the invocation -of a listener method is supplied with all beans implementing any of the following interfaces: +of a listener method is supplied with all beans that implement any of the following interfaces: - - `org.springframework.core.convert.converter.Converter` - - `org.springframework.core.convert.converter.GenericConverter` - - `org.springframework.format.Formatter` +* `org.springframework.core.convert.converter.Converter` +* `org.springframework.core.convert.converter.GenericConverter` +* `org.springframework.format.Formatter` -This allows you to further customize listener deserialization without changing the default configuration for +This lets you further customize listener deserialization without changing the default configuration for `ConsumerFactory` and `KafkaListenerContainerFactory`. IMPORTANT: Setting a custom `MessageHandlerMethodFactory` on the `KafkaListenerEndpointRegistrar` through a -`KafkaListenerConfigurer` bean will disable this feature. +`KafkaListenerConfigurer` bean disables this feature. [[headers]] ==== Message Headers The 0.11.0.0 client introduced support for headers in messages. -Spring for Apache Kafka _version 2.0_ now supports mapping these headers to/from `spring-messaging` `MessageHeaders`. +As of version 2.0, Spring for Apache Kafka now supports mapping these headers to and from `spring-messaging` `MessageHeaders`. -NOTE: Previous versions mapped `ConsumerRecord` and `ProducerRecord` to spring-messaging `Message` where the value property is mapped to/from the `payload` and other properties (`topic`, `partition`, etc) were mapped to headers. -This is still the case but additional, arbitrary, headers can now be mapped. +NOTE: Previous versions mapped `ConsumerRecord` and `ProducerRecord` to spring-messaging `Message`, where the value property is mapped to and from the `payload` and other properties (`topic`, `partition`, and so on) were mapped to headers. +This is still the case, but additional (arbitrary) headers can now be mapped. -Apache Kafka headers have a simple API: +Apache Kafka headers have a simple API, shown in the following interface definition: +==== [source, java] ---- public interface Header { @@ -2154,9 +2397,12 @@ public interface Header { } ---- +==== -The `KafkaHeaderMapper` strategy is provided to map header entries between Kafka `Headers` and `MessageHeaders`: +The `KafkaHeaderMapper` strategy is provided to map header entries between Kafka `Headers` and `MessageHeaders`. +Its interface definition is as follows: +==== [source, java] ---- public interface KafkaHeaderMapper { @@ -2167,78 +2413,86 @@ public interface KafkaHeaderMapper { } ---- +==== -The `DefaultKafkaHeaderMapper` maps the key to the `MessageHeaders` header name and, in order to support rich header types, for outbound messages, JSON conversion is performed. -A "special" header, with key, `spring_json_header_types` contains a JSON map of `:`. +The `DefaultKafkaHeaderMapper` maps the key to the `MessageHeaders` header name and, in order to support rich header types for outbound messages, JSON conversion is performed. +A "`special`" header (with a key of `spring_json_header_types`) contains a JSON map of `:`. This header is used on the inbound side to provide appropriate conversion of each header value to the original type. -On the inbound side, all Kafka `Header` s are mapped to `MessageHeaders`. -On the outbound side, by default, all `MessageHeaders` are mapped except `id`, `timestamp`, and the headers that map to `ConsumerRecord` properties. +On the inbound side, all Kafka `Header` instances are mapped to `MessageHeaders`. +On the outbound side, by default, all `MessageHeaders` are mapped, except `id`, `timestamp`, and the headers that map to `ConsumerRecord` properties. You can specify which headers are to be mapped for outbound messages, by providing patterns to the mapper. +The following listing shows a number of example mappings: +==== [source, java] ---- -public DefaultKafkaHeaderMapper() { +public DefaultKafkaHeaderMapper() { <1> ... } -public DefaultKafkaHeaderMapper(ObjectMapper objectMapper) { +public DefaultKafkaHeaderMapper(ObjectMapper objectMapper) { <2> ... } -public DefaultKafkaHeaderMapper(String... patterns) { +public DefaultKafkaHeaderMapper(String... patterns) { <3> ... } -public DefaultKafkaHeaderMapper(ObjectMapper objectMapper, String... patterns) { +public DefaultKafkaHeaderMapper(ObjectMapper objectMapper, String... patterns) { <4> ... } ---- -The first constructor will use a default Jackson `ObjectMapper` and map most headers, as discussed above. -The second constructor will use the provided Jackson `ObjectMapper` and map most headers, as discussed above. -The third constructor will use a default Jackson `ObjectMapper` and map headers according to the provided patterns. -The third constructor will use the provided Jackson `ObjectMapper` and map headers according to the provided patterns. +<1> Uses a default Jackson `ObjectMapper` and maps most headers, as discussed before the example. +<2> Uses the provided Jackson `ObjectMapper` and maps most headers, as discussed before the example. +<3> Uses a default Jackson `ObjectMapper` and maps headers according to the provided patterns. +<4> Uses the provided Jackson `ObjectMapper` and maps headers according to the provided patterns. +==== -Patterns are rather simple and can contain either a leading or trailing wildcard `*`, or both, e.g. `*.foo.*`. -Patterns can be negated with a leading `!`. -The first pattern that matches a header name wins (positive or negative). +Patterns are rather simple and can contain a leading wildcard (`*`), a trailing wildcard, or both (for example, `*.cat.*`). +You can negate patterns with a leading `!`. +The first pattern that matches a header name (whether positive or negative) wins. -When providing your own patterns, it is recommended to include `!id` and `!timestamp` since these headers are read-only on the inbound side. +When you provide your own patterns, we recommend including `!id` and `!timestamp`, since these headers are read-only on the inbound side. -IMPORTANT: By default, the mapper will only deserialize classes in `java.lang` and `java.util`. -You can trust other (or all) packages by adding trusted packages using the `addTrustedPackages` method. -If you are receiving messages from untrusted sources, you may wish to add just those packages that you trust. -To trust all packages use `mapper.addTrustedPackages("*")`. +IMPORTANT: By default, the mapper deserializes only classes in `java.lang` and `java.util`. +You can trust other (or all) packages by adding trusted packages with the `addTrustedPackages` method. +If you receive messages from untrusted sources, you may wish to add only those packages you trust. +To trust all packages, you can use `mapper.addTrustedPackages("*")`. -The `DefaultKafkaHeaderMapper` is used in the `MessagingMessageConverter` and `BatchMessagingMessageConverter` by default, as long as Jackson is on the class path. +By default, the `DefaultKafkaHeaderMapper` is used in the `MessagingMessageConverter` and `BatchMessagingMessageConverter`, as long as Jackson is on the class path. With the batch converter, the converted headers are available in the `KafkaHeaders.BATCH_CONVERTED_HEADERS` as a `List>` where the map in a position of the list corresponds to the data position in the payload. -If the converter has no converter (either because Jackson is not present, or it is explicitly set to `null`), the headers from the consumer record are provided unconverted in the `KafkaHeaders.NATIVE_HEADERS` header (a `Headers` object, or a `List` in the case of the batch converter, where the position in the list corresponds to the data position in the payload). +If there is no converter (either because Jackson is not present or it is explicitly set to `null`), the headers from the consumer record are provided unconverted in the `KafkaHeaders.NATIVE_HEADERS` header. +This header is a `Headers` object (or a `List` in the case of the batch converter), where the position in the list corresponds to the data position in the payload). -IMPORTANT: Certain types are not suitable for JSON serialization and a simple `toString()` serialization might be preferred for these types. -The `DefaultKafkaHeaderMapper` has a method `addToStringClasses()` where you can supply names of classes that should be treated this way for outbound mapping. -During inbound mapping, they will be mapped as `String`. -By default, just `org.springframework.util.MimeType` and `org.springframework.http.MediaType` are mapped this way. +IMPORTANT: Certain types are not suitable for JSON serialization, and a simple `toString()` serialization might be preferred for these types. +The `DefaultKafkaHeaderMapper` has a method called `addToStringClasses()` that lets you supply the names of classes that should be treated this way for outbound mapping. +During inbound mapping, they are mapped as `String`. +By default, only `org.springframework.util.MimeType` and `org.springframework.http.MediaType` are mapped this way. [[tombstones]] -==== Null Payloads and Log Compaction 'Tombstone' Records +==== Null Payloads and Log Compaction of 'Tombstone' Records -When using https://kafka.apache.org/documentation/#compaction[Log Compaction], it is possible to send and receive messages with `null` payloads which identifies the deletion of a key. +When you use https://kafka.apache.org/documentation/#compaction[Log Compaction], you can send and receive messages with `null` payloads to identify the deletion of a key. -It is also possible to receive `null` values for other reasons - such as a `Deserializer` that might return `null` when it can't deserialize a value. +You can also receive `null` values for other reasons, such as a `Deserializer` that might return `null` when it cannot deserialize a value. -To send a `null` payload using the `KafkaTemplate` simply pass null into the value argument of the `send()` methods. +To send a `null` payload by using the `KafkaTemplate`, you can pass null into the value argument of the `send()` methods. One exception to this is the `send(Message message)` variant. -Since `spring-messaging` `Message` cannot have a `null` payload, a special payload type `KafkaNull` is used and the framework will send `null`. +Since `spring-messaging` `Message` cannot have a `null` payload, you can use a special payload type called `KafkaNull`, and the framework sends `null`. For convenience, the static `KafkaNull.INSTANCE` is provided. -When using a message listener container, the received `ConsumerRecord` will have a `null` `value()`. +When you use a message listener container, the received `ConsumerRecord` has a `null` `value()`. -To configure the `@KafkaListener` to handle `null` payloads, you must use the `@Payload` annotation with `required = false`; if it's a tombstone message for a compacted log, you will usually also need the key so your application can determine which key was "deleted": +To configure the `@KafkaListener` to handle `null` payloads, you must use the `@Payload` annotation with `required = false`. +If itis a tombstone message for a compacted log, you usually also need the key so that your application can determine which key was "`deleted`". +The following example shows such a configuration: +==== [source, java] ---- @KafkaListener(id = "deletableListener", topics = "myTopic") @@ -2246,21 +2500,25 @@ public void listen(@Payload(required = false) String value, @Header(KafkaHeaders // value == null represents key deletion } ---- +==== -When using a class-level `@KafkaListener` with multiple `@KafkaHandler` methods, some additional configuration is needed - a `@KafkaHandler` method with a `KafkaNull` payload: +When you use a class-level `@KafkaListener` with multiple `@KafkaHandler` methods, some additional configuration is needed. +Specifically, you need a `@KafkaHandler` method with a `KafkaNull` payload. +The following example shows how to configure one: +==== [source, java] ---- @KafkaListener(id = "multi", topics = "myTopic") static class MultiListenerBean { @KafkaHandler - public void listen(String foo) { + public void listen(String cat) { ... } @KafkaHandler - public void listen(Integer bar) { + public void listen(Integer hat) { ... } @@ -2271,21 +2529,25 @@ static class MultiListenerBean { } ---- +==== -Note that the argument will be `null` not a `KafkaNull`. +Note that the argument is `null`, not `KafkaNull`. [[annotation-error-handling]] ==== Handling Exceptions +This section describes how to handle various exceptions that may arise when you use Spring for Apache Kafka. + ===== Listener Error Handlers -Starting with _version 2.0_, the `@KafkaListener` annotation has a new attribute: `errorHandler`. +Starting with version 2.0, the `@KafkaListener` annotation has a new attribute: `errorHandler`. -This attribute is not configured by default. +By default, this attribute is not configured. -Use the `errorHandler` to provide the bean name of a `KafkaListenerErrorHandler` implementation. -This functional interface has one method: +You can use the `errorHandler` to provide the bean name of a `KafkaListenerErrorHandler` implementation. +This functional interface has one method, as the following listing shows: +==== [source, java] ---- @FunctionalInterface @@ -2295,20 +2557,25 @@ public interface KafkaListenerErrorHandler { } ---- +==== -As you can see, you have access to the spring-messaging `Message` object produced by the message converter and the exception that was thrown by the listener, wrapped in a `ListenerExecutionFailedException`. -The error handler can throw the original or a new exception which will be thrown to the container. Anything returned by the error handler is ignored. +You have access to the spring-messaging `Message` object produced by the message converter and the exception that was thrown by the listener, which is wrapped in a `ListenerExecutionFailedException`. +The error handler can throw the original or a new exception, which is thrown to the container. +Anything returned by the error handler is ignored. -It has a sub-interface `ConsumerAwareListenerErrorHandler` that has access to the consumer object, via the method: +It has a sub-interface (`ConsumerAwareListenerErrorHandler`) that has access to the consumer object, through the following method: +==== [source, java] ---- Object handleError(Message message, ListenerExecutionFailedException exception, Consumer consumer); ---- +==== -If your error handler implements this interface you can, for example, adjust the offsets accordingly. -For example, to reset the offset to replay the failed message, you could do something like the following; note however, these are simplistic implementations and you would probably want more checking in the error handler. +If your error handler implements this interface, you can, for example, adjust the offsets accordingly. +For example, to reset the offset to replay the failed message, you could do something like the following: +==== [source, java] ---- @Bean @@ -2324,9 +2591,11 @@ public ConsumerAwareListenerErrorHandler listen3ErrorHandler() { }; } ---- +==== -And for a batch listener: +Similarly, you could do something like the following for a batch listener: +==== [source, java] ---- @Bean @@ -2348,13 +2617,18 @@ public ConsumerAwareListenerErrorHandler listen10ErrorHandler() { }; } ---- +==== This resets each topic/partition in the batch to the lowest offset in the batch. +NOTE: The preceding two examples are simplistic implementations, and you would probably want more checking in the error handler. + ===== Container Error Handlers -You can specify a global error handler used for all listeners in the container factory. +You can specify a global error handler to be used for all listeners in the container factory. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -2368,9 +2642,11 @@ public KafkaListenerContainerFactory data, Consumer consumer); +---- +==== +The `handle` method of the `ConsumerAwareBatchErrorHandler` has the following signature: + +==== +[source, java] +---- void handle(Exception thrownException, ConsumerRecords data, Consumer consumer); ---- +==== -respectively. +Similar to the `@KafkaListener` error handlers, you can reset the offsets as needed, based on the data that failed. -Similar to the `@KafkaListener` error handlers, you can reset the offsets as needed based on the data that failed. - -NOTE: Unlike the listener-level error handlers, however, you should set the container property `ackOnError` to false when making adjustments; otherwise any pending acks will be applied after your repositioning. +NOTE: Unlike the listener-level error handlers, however, you should set the `ackOnError` container property to false when making adjustments. +Otherwise, any pending acks are applied after your repositioning. [[seek-to-current]] ===== Seek To Current Container Error Handlers If an `ErrorHandler` implements `RemainingRecordsErrorHandler`, the error handler is provided with the failed record and any unprocessed records retrieved by the previous `poll()`. -Those records will not be passed to the listener after the handler exits. +Those records are not passed to the listener after the handler exits. +The following listing shows the `RemainingRecordsErrorHandler` interface definition: +==== [source, java] ---- @FunctionalInterface @@ -2419,16 +2707,18 @@ public interface RemainingRecordsErrorHandler extends ConsumerAwareErrorHandler } ---- +==== -This allows implementations to seek all unprocessed topic/partitions so the current record (and the others remaining) will be retrieved by the next poll. -The `SeekToCurrentErrorHandler` does exactly this. +This interface lets implementations seek all unprocessed topics and partitions so that the current record (and the others remaining) are retrieved by the next poll. +`SeekToCurrentErrorHandler` does exactly this. -The container will commit any pending offset commits before calling the error handler. +The container commits any pending offset commits before calling the error handler. To configure the listener container with this handler, add it to the `ContainerProperties`. -For example, with the `@KafkaListener` container factory: +For example, with the `@KafkaListener` container factory, you can add `SeekToCurrentErrorHandler` as follows: +==== [source, java] ---- @Bean @@ -2441,17 +2731,19 @@ public ConcurrentKafkaListenerContainerFactory kafkaListenerCont return factory; } ---- +==== -As an example; if the `poll` returns 6 records (2 from each partition 0, 1, 2) and the listener throws an exception on the fourth record, the container will have acknowledged the first 3 by committing their offsets. -The `SeekToCurrentErrorHandler` will seek to offset 1 for partition 1 and offset 0 for partition 2. -The next `poll()` will return the 3 unprocessed records. +As an example; if the `poll` returns six records (two from each partition 0, 1, 2) and the listener throws an exception on the fourth record, the container acknowledges the first three messages by committing their offsets. +The `SeekToCurrentErrorHandler` seeks to offset 1 for partition 1 and offset 0 for partition 2. +The next `poll()` returns the three unprocessed records. -If the `AckMode` was `BATCH`, the container commits the offsets for the first 2 partitions before calling the error handler. +If the `AckMode` was `BATCH`, the container commits the offsets for the first two partitions before calling the error handler. Starting with version 2.2, the `SeekToCurrentErrorHandler` can now recover (skip) a record that keeps failing. -By default, after 10 failures, the failed record will be logged (ERROR). -You can configure the handler with a custom recoverer (`BiConsumer`) and/or max failures. -Setting the `maxFailures` property to a negative number will cause infinite retries. +By default, after ten failures, the failed record is logged (at the `ERROR` level). +You can configure the handler with a custom recoverer (`BiConsumer`) and maximum failures. +Setting the `maxFailures` property to a negative number causes infinite retries. +The following example configures recovery after three tries: ==== [source, java] @@ -2463,42 +2755,44 @@ SeekToCurrentErrorHandler errorHandler = ---- ==== -Also see <>. +See also <>. -When using transactions, similar functionality is provided by the `DefaultAfterRollbackProcessor`; see <>. +When using transactions, similar functionality is provided by the `DefaultAfterRollbackProcessor`. +See <>. -The `SeekToCurrentBatchErrorHandler` seeks each partition to the first record in each partition in the batch so the whole batch is replayed. -This error handler does not support recovery because the framework cannot know which message in the batch is failing. +The `SeekToCurrentBatchErrorHandler` seeks each partition to the first record in each partition in the batch, so the whole batch is replayed. +This error handler does not support recovery, because the framework cannot know which message in the batch is failing. -After seeking, an exception wrapping the `ListenerExecutionFailedException` is thrown. +After seeking, an exception that wraps the `ListenerExecutionFailedException` is thrown. This is to cause the transaction to roll back (if transactions are enabled). ===== Container Stopping Error Handlers -The `ContainerStoppingErrorHandler` (used with record listeners) will stop the container if the listener throws an exception. -When the `AckMode` is `RECORD`, offsets for already processed records will be committed. -When the `AckMode` is any manual, offsets for already acknowledged records will be committed. -When the `AckMode` is `BATCH`, the entire batch will be replayed when the container is restarted, unless transactions are enabled in which case only the unprocessed records will be re-fetched. +The `ContainerStoppingErrorHandler` (used with record listeners) stops the container if the listener throws an exception. +When the `AckMode` is `RECORD`, offsets for already processed records are committed. +When the `AckMode` is any manual value, offsets for already acknowledged records are committed. +When the `AckMode` is `BATCH`, the entire batch is replayed when the container is restarted (unless transactions are enabled -- in which case, only the unprocessed records are re-fetched). -The `ContainerStoppingBatchErrorHandler` (used with batch listeners) will stop the container and the entire batch will be replayed when the container is restarted. +The `ContainerStoppingBatchErrorHandler` (used with batch listeners) stops the container, and the entire batch is replayed when the container is restarted. -After the container stops, an exception wrapping the `ListenerExecutionFailedException` is thrown. +After the container stops, an exception that wraps the `ListenerExecutionFailedException` is thrown. This is to cause the transaction to roll back (if transactions are enabled). [[after-rollback]] -===== After Rollback Processor +===== After-rollback Processor When using transactions, if the listener throws an exception (and an error handler, if present, throws an exception), the transaction is rolled back. -By default, any unprocessed records (including the failed record) will be re-fetched on the next poll. +By default, any unprocessed records (including the failed record) are re-fetched on the next poll. This is achieved by performing `seek` operations in the `DefaultAfterRollbackProcessor`. -With a batch listener, the entire batch of records will be reprocessed (the container has no knowledge of which record in the batch failed). -To modify this behavior, configure the listener container with a custom `AfterRollbackProcessor`. -For example, with a record-based listener, you might want to keep track of the failed record and give up after some number of attempts - perhaps by publishing it to a dead-letter topic. +With a batch listener, the entire batch of records is reprocessed (the container has no knowledge of which record in the batch failed). +To modify this behavior, you can configure the listener container with a custom `AfterRollbackProcessor`. +For example, with a record-based listener, you might want to keep track of the failed record and give up after some number of attempts, perhaps by publishing it to a dead-letter topic. Starting with version 2.2, the `DefaultAfterRollbackProcessor` can now recover (skip) a record that keeps failing. -By default, after 10 failures, the failed record will be logged (ERROR). -You can configure the processor with a custom recoverer (`BiConsumer`) and/or max failures. -Setting the `maxFailures` property to a negative number will cause infinite retries. +By default, after ten failures, the failed record is logged (at the `ERROR` level). +You can configure the processor with a custom recoverer (`BiConsumer`) and maximum failures. +Setting the `maxFailures` property to a negative number causes infinite retries. +The following example configures recovery after three tries: ==== [source, java] @@ -2510,27 +2804,28 @@ AfterRollbackProcessor processor = ---- ==== -When not using transactions, similar functionality can be achieved by configuring a `SeekToCurrentErrorHandler`; see <>. +When you do not use transactions, you can achieve similar functionality by configuring a `SeekToCurrentErrorHandler`. +See <>. -IMPORTANT: Recovery is not possible with a batch listener since the framework has no knowledge about which record in the batch keeps failing. +IMPORTANT: Recovery is not possible with a batch listener, since the framework has no knowledge about which record in the batch keeps failing. In such cases, the application listener must handle a record that keeps failing. -Also see <>. +See also <>. [[dead-letters]] -===== Publishing Dead-Letter Records +===== Publishing Dead-letter Records -As discussed above, the `SeekToCurrentErrorHandler` and `DefaultAfterRollbackProcessor` can be configured with a record recoverer when the maximum number of failures is reached for a record. -The framework provides the `DeadLetterPublishingRecoverer` which will publish the failed message to another topic. -The recoverer requires a `KafkaTemplate` which is used to send the record. -It also, optionally, can be configured with a `BiFunction, Exception, TopicPartition>` which is called to resolve the destination topic and partition. +As <>, you can configure the `SeekToCurrentErrorHandler` and `DefaultAfterRollbackProcessor` with a record recoverer when the maximum number of failures is reached for a record. +The framework provides the `DeadLetterPublishingRecoverer`, which publishes the failed message to another topic. +The recoverer requires a `KafkaTemplate`, which is used to send the record. +You can also, optionally, configure it with a `BiFunction, Exception, TopicPartition>`, which is called to resolve the destination topic and partition. By default, the dead-letter record is sent to a topic named `.DLT` (the original topic name suffixed with `.DLT`) and to the same partition as the original record. -Therefore, when using the default resolver, the dead-letter topic must have at least as many partitions as the original topic. -If the returned `TopicPartition` has a negative partition, the partition is not set in the `ProducerRecord` and so the partition will be selected by Kafka. -Starting with version 2.2.4, any `ListenerExcutionFailedException` (e.g. thrown when an exception is detected in a `@KafkaListener` method) will be enhanced with the `groupId` property. -This will allow the destination resolver to use this in addition to the information in the `ConsumerRecord` to select the dead letter topic. +Therefore, when you use the default resolver, the dead-letter topic must have at least as many partitions as the original topic. +If the returned `TopicPartition` has a negative partition, the partition is not set in the `ProducerRecord`, so the partition is selected by Kafka. +Starting with version 2.2.4, any `ListenerExecutionFailedException` (thrown, for example, when an exception is detected in a `@KafkaListener` method) is enhanced with the `groupId` property. +This allows the destination resolver to use this, in addition to the information in the `ConsumerRecord` to select the dead letter topic. -The following is an example of wiring a custom destination resolver. +The following example shows how to wire a custom destination resolver: ==== [source, java] @@ -2550,23 +2845,25 @@ ErrorHandler errorHandler = new SeekToCurrentErrorHandler(recoverer, 3); The record sent to the dead-letter topic is enhanced with the following headers: -* `KafkaHeaders.DLT_EXCEPTION_FQCN` - Exception class name. -* `KafkaHeaders.DLT_EXCEPTION_STACKTRACE` - Exception stack trace. -* `KafkaHeaders.DLT_EXCEPTION_MESSAGE` - Exception message. -* `KafkaHeaders.DLT_ORIGINAL_TOPIC` - Original topic. -* `KafkaHeaders.DLT_ORIGINAL_PARTITION` Original partition. -* `KafkaHeaders.DLT_ORIGINAL_OFFSET` - Original offset. -* `KafkaHeaders.DLT_ORIGINAL_TIMESTAMP` - Original timestamp. -* `KafkaHeaders.DLT_ORIGINAL_TIMESTAMP_TYPE` - Original timestamp type. +* `KafkaHeaders.DLT_EXCEPTION_FQCN`: The Exception class name. +* `KafkaHeaders.DLT_EXCEPTION_STACKTRACE`: The Exception stack trace. +* `KafkaHeaders.DLT_EXCEPTION_MESSAGE`: The Exception message. +* `KafkaHeaders.DLT_ORIGINAL_TOPIC`: The original topic. +* `KafkaHeaders.DLT_ORIGINAL_PARTITION`: The original partition. +* `KafkaHeaders.DLT_ORIGINAL_OFFSET`: The original offset. +* `KafkaHeaders.DLT_ORIGINAL_TIMESTAMP`: The original timestamp. +* `KafkaHeaders.DLT_ORIGINAL_TIMESTAMP_TYPE`: The original timestamp type. [[kerberos]] ==== Kerberos -Starting with version 2.0 a `KafkaJaasLoginModuleInitializer` class has been added to assist with Kerberos configuration. -Simply add this bean, with the desired configuration, to your application context. +Starting with version 2.0, a `KafkaJaasLoginModuleInitializer` class has been added to assist with Kerberos configuration. +You can add this bean, with the desired configuration, to your application context. +The following example configures such a bean: +==== [source, java] ---- @Bean @@ -2582,3 +2879,4 @@ public KafkaJaasLoginModuleInitializer jaasConfig() throws IOException { return jaasConfig; } ---- +==== diff --git a/src/reference/asciidoc/preface.adoc b/src/reference/asciidoc/preface.adoc index 284244e6..b0c044f5 100644 --- a/src/reference/asciidoc/preface.adoc +++ b/src/reference/asciidoc/preface.adoc @@ -1,4 +1,4 @@ [[preface]] The Spring for Apache Kafka project applies core Spring concepts to the development of Kafka-based messaging solutions. -We provide a "template" as a high-level abstraction for sending messages. +We provide a "`template`" as a high-level abstraction for sending messages. We also provide support for Message-driven POJOs. diff --git a/src/reference/asciidoc/quick-tour.adoc b/src/reference/asciidoc/quick-tour.adoc index 95c0ca19..ef888080 100644 --- a/src/reference/asciidoc/quick-tour.adoc +++ b/src/reference/asciidoc/quick-tour.adoc @@ -1,14 +1,14 @@ [[quick-tour]] === Quick Tour for the Impatient -==== Introduction +This is the five-minute tour to get started with Spring Kafka. -This is the 5 minute tour to get started with Spring Kafka. - -Prerequisites: install and run Apache Kafka -Then grab the spring-kafka JAR and all of its dependencies - the easiest way to do that is to declare a dependency in -your build tool, e.g. for Maven: +Prerequisites: You must install and run Apache Kafka. +Then you must grab the spring-kafka JAR and all of its dependencies. +The easiest way to do that is to declare a dependency in your build tool. +The following example shows how to do so with Maven: +==== [source,xml,subs="+attributes"] ---- @@ -17,25 +17,31 @@ your build tool, e.g. for Maven: {spring-kafka-version} ---- +==== -And for Gradle: +The following example shows how to do so with Gradle: +==== [source,groovy,subs="+attributes"] ---- compile 'org.springframework.kafka:spring-kafka:{spring-kafka-version}' ---- +==== [[compatibility]] ===== Compatibility -- Apache Kafka Clients 2.0.0 -- Spring Framework 5.1.x -- Minimum Java version: 8 +This quick tour works with the following versions: -===== Very, Very Quick +* Apache Kafka Clients 2.0.0 +* Spring Framework 5.1.x +* Minimum Java version: 8 -Using plain Java to send and receive a message: +===== A Very, Very Quick Example +As the following example shows, you can use plain Java to send and receive a message: + +==== [source,java] ---- @Test @@ -114,11 +120,14 @@ private Map senderProps() { return props; } ---- +==== ===== With Java Configuration -A similar example but with Spring configuration in Java: +You can do the same work as appears in the previous example with Spring configuration in Java. +The following example shows how to do so: +==== [source,java] ---- @Autowired @@ -199,12 +208,14 @@ public class Listener { } ---- +==== ===== Even Quicker, with Spring Boot -The following Spring Boot application sends 3 messages to a topic, receives them, and stops. +Spring Boot can make things even simpler. +The following Spring Boot application sends three messages to a topic, receives them, and stops: -.Application +==== [source, java] ---- @SpringBootApplication @@ -238,14 +249,19 @@ public class Application implements CommandLineRunner { } ---- +==== -Boot takes care of most of the configuration; when using a local broker, the only properties we need are: +Boot takes care of most of the configuration. +When we use a local broker, the only properties we need are the following: .application.properties +==== [source] ---- spring.kafka.consumer.group-id=foo spring.kafka.consumer.auto-offset-reset=earliest ---- +==== -The first because we are using group management to assign topic partitions to consumers so we need a group, the second to ensure the new consumer group will get the messages we just sent, because the container might start after the sends have completed. +We need the first property because we are using group management to assign topic partitions to consumers, so we need a group. +The second property ensures the new consumer group gets the messages we sent, because the container might start after the sends have completed. diff --git a/src/reference/asciidoc/si-kafka.adoc b/src/reference/asciidoc/si-kafka.adoc index 272bd3d6..030122be 100644 --- a/src/reference/asciidoc/si-kafka.adoc +++ b/src/reference/asciidoc/si-kafka.adoc @@ -1,57 +1,63 @@ [[si-kafka]] === Spring Integration for Apache Kafka -==== Introduction - -This documentation pertains to versions 2.0.0 and above; for documentation for earlier releases, see the https://github.com/spring-projects/spring-integration-kafka/blob/1.3.x/README.md[1.3.x README]. +This documentation pertains to versions 2.0.0 and above. +For documentation for earlier releases, see the https://github.com/spring-projects/spring-integration-kafka/blob/1.3.x/README.md[1.3.x README]. Spring Integration Kafka is now based on the http://projects.spring.io/spring-kafka/[Spring for Apache Kafka project]. It provides the following components: -- Outbound Channel Adapter -- Message-Driven Channel Adapter - -These are discussed in the following sections. +* <> +* <> [[si-outbound]] ==== Outbound Channel Adapter The Outbound channel adapter is used to publish messages from a Spring Integration channel to Kafka topics. The channel is defined in the application context and then wired into the application that sends messages to Kafka. -Sender applications can publish to Kafka via Spring Integration messages, which are internally converted -to Kafka messages by the outbound channel adapter, as follows: the payload of the Spring Integration message will be -used to populate the payload of the Kafka message, and (by default) the `kafka_messageKey` header of the Spring -Integration message will be used to populate the key of the Kafka message. +Sender applications can publish to Kafka by using Spring Integration messages, which are internally converted to Kafka messages by the outbound channel adapter, as follows: -The target topic and partition for publishing the message can be customized through the `kafka_topic` +* The payload of the Spring Integration message is used to populate the payload of the Kafka message. +* By default, the `kafka_messageKey` header of the Spring Integration message is used to populate the key of the Kafka message. + +You can customize the target topic and partition for publishing the message through the `kafka_topic` and `kafka_partitionId` headers, respectively. -In addition, the `` provides the ability to extract the key, target topic, and -target partition by applying SpEL expressions on the outbound message. To that end, it supports the mutually exclusive -pairs of attributes `topic`/`topic-expression`, `message-key`/`message-key-expression`, and -`partition-id`/`partition-id-expression`, to allow the specification of `topic`,`message-key` and `partition-id` -respectively as static values on the adapter, or to dynamically evaluate their values at runtime against -the request message. +In addition, the `` provides the ability to extract the key, target topic, and target partition by applying SpEL expressions on the outbound message. +To that end, it supports three mutually exclusive pairs of attributes: + +* `topic` and `topic-expression` +* `message-key` and `message-key-expression` +* `partition-id` and `partition-id-expression` + +These let you specify `topic`, `message-key`, and `partition-id`, respectively, as static values on the adapter or to dynamically evaluate their values at runtime against the request message. IMPORTANT: The `KafkaHeaders` interface (provided by `spring-kafka`) contains constants used for interacting with headers. The `messageKey` and `topic` default headers now require a `kafka_` prefix. When migrating from an earlier version that used the old headers, you need to specify `message-key-expression="headers['messageKey']"` and `topic-expression="headers['topic']"` on the -``, or simply change the headers upstream to -the new headers from `KafkaHeaders` using a `` or `MessageBuilder`. -Or, of course, configure them on the adapter using `topic` and `message-key` if you are using constant values. +``. +Alternatively, you can change the headers upstream to +the new headers from `KafkaHeaders` by using a `` or a `MessageBuilder`. +If you use constant values, you can also configure them on the adapter by using `topic` and `message-key`. NOTE : If the adapter is configured with a topic or message key (either with a constant or expression), those are used and the corresponding header is ignored. -If you wish the header to override the configuration, you need to configure it in an expression, such as: +If you wish the header to override the configuration, you need to configure it in an expression, such as the following: -`topic-expression="headers['topic'] != null ? headers['topic'] : 'myTopic'"`. +==== +[source,java] +---- +topic-expression="headers['topic'] != null ? headers['topic'] : 'myTopic'" +---- +==== The adapter requires a `KafkaTemplate`. -Here is an example of how the Kafka outbound channel adapter is configured with XML: +The following example shows how to configure the Kafka outbound channel adapter with XML: +==== [source, xml] ---- ---- +==== -As you can see, the adapter requires a `KafkaTemplate` which, in turn, requires a suitably configured `KafkaProducerFactory`. +The adapter requires a `KafkaTemplate`, which, in turn, requires a suitably configured `KafkaProducerFactory`. -When using Java Configuration: +The following example shows how to configure the Kafka outbound channel adapter with Java: +==== [source, java] ---- @Bean @@ -112,9 +120,11 @@ public ProducerFactory producerFactory() { return new DefaultKafkaProducerFactory<>(props); } ---- +==== -When using Spring Integration Java DSL: +The following example shows how to configure the Kafka outbound channel adapter Spring Integration Java DSL: +==== [source, java] ---- @Bean @@ -156,24 +166,28 @@ private KafkaProducerMessageHandlerSpec kafkaMessageHandler( .configureKafkaTemplate(t -> t.id("kafkaTemplate:" + topic)); } ---- +==== -If a `send-failure-channel` (`sendFailureChannel`) is provided and a send failure is received (sync or async), an `ErrorMessage` is sent to the channel. -The payload is a `KafkaSendFailureException` with properties `failedMessage`, `record` (the `ProducerRecord`) and `cause`. -The `DefaultErrorMessageStrategy` can be overridden via the `error-message-strategy` property. +If a `send-failure-channel` (`sendFailureChannel`) is provided and a send failure (sync or async) is received, an `ErrorMessage` is sent to the channel. +The payload is a `KafkaSendFailureException` with `failedMessage`, `record` (the `ProducerRecord`) and `cause` properties. +You can override the `DefaultErrorMessageStrategy` by setting the `error-message-strategy` property. -If a `send-success-channel` (`sendSuccessChannel`) is provided, a message with a payload of type `org.apache.kafka.clients.producer.RecordMetadata` will be sent after a successful send. +If a `send-success-channel` (`sendSuccessChannel`) is provided, a message with a payload of type `org.apache.kafka.clients.producer.RecordMetadata` is sent after a successful send. [[si-inbound]] -==== Message Driven Channel Adapter +==== Message-driven Channel Adapter The `KafkaMessageDrivenChannelAdapter` (``) uses a `spring-kafka` `KafkaMessageListenerContainer` or `ConcurrentListenerContainer`. -Starting with _spring-integration-kafka version 2.1_, the `mode` attribute is available (`record` or `batch`, default `record`). -For `record` mode, each message payload is converted from a single `ConsumerRecord`; for mode `batch` the payload is a list of objects which are converted from all the `ConsumerRecord` s returned by the consumer poll. -As with the batched `@KafkaListener`, the `KafkaHeaders.RECEIVED_MESSAGE_KEY`, `KafkaHeaders.RECEIVED_PARTITION_ID`, `KafkaHeaders.RECEIVED_TOPIC` and `KafkaHeaders.OFFSET` headers are also lists with, positions corresponding to the position in the payload. +Starting with spring-integration-kafka version 2.1, the `mode` attribute is available. +It can accept values of `record` or `batch` (default: `record`). +For `record` mode, each message payload is converted from a single `ConsumerRecord`. +For `batch` mode, the payload is a list of objects that are converted from all the `ConsumerRecord` instances returned by the consumer poll. +As with the batched `@KafkaListener`, the `KafkaHeaders.RECEIVED_MESSAGE_KEY`, `KafkaHeaders.RECEIVED_PARTITION_ID`, `KafkaHeaders.RECEIVED_TOPIC`, and `KafkaHeaders.OFFSET` headers are also lists, with positions corresponding to the position in the payload. -An example of xml configuration variant is shown here: +The following example shows how to configure a message-driven channel adapter with XML: +==== [source, xml] ---- ---- +==== -When using Java Configuration: +The following example shows how to configure a message-driven channel adapter with Java: +==== [source, java] ---- @Bean @@ -237,9 +253,11 @@ public ConsumerFactory consumerFactory() { return new DefaultKafkaConsumerFactory<>(props); } ---- +==== -When using Spring Integration Java DSL: +The following example shows how to configure a message-driven channel adapter with the Spring Integration Java DSL: +==== [source, java] ---- @Bean @@ -262,25 +280,30 @@ public IntegrationFlow topic1ListenerFromKafkaFlow() { .get(); } ---- +==== -Received messages will have certain headers populated. -Refer to the `KafkaHeaders` class for more information. +Received messages have certain headers populated. +See the https://docs.spring.io/spring-kafka/api/org/springframework/kafka/support/KafkaHeaders.html[`KafkaHeaders` class] for more information. -IMPORTANT: The `Consumer` object (in the `kafka_consumer` header) is not thread-safe; you must only invoke its methods on the thread that calls the listener within the adapter; if you hand off the message to another thread, you must not call its methods. +IMPORTANT: The `Consumer` object (in the `kafka_consumer` header) is not thread-safe. +You must invoke its methods only on the thread that calls the listener within the adapter. +If you hand off the message to another thread, you must not call its methods. -When a `retry-template` is provided, delivery failures will be retried according to its retry policy. +When a `retry-template` is provided, delivery failures are retried according to its retry policy. An `error-channel` is not allowed in this case. -The `recovery-callback` can be used to handle the error when retries are exhausted. -In most cases, this will be an `ErrorMessageSendingRecoverer` which will send the `ErrorMessage` to a channel. +You can use the `recovery-callback` to handle the error when retries are exhausted. +In most cases, this is an `ErrorMessageSendingRecoverer` that sends the `ErrorMessage` to a channel. -When building `ErrorMessage` (for use in the `error-channel` or `recovery-callback`), you can customize the error message using the `error-message-strategy` property. -By default, a `RawRecordHeaderErrorMessageStrategy` is used; providing access to the converted message as well as the raw `ConsumerRecord`. +When building an `ErrorMessage` (for use in the `error-channel` or `recovery-callback`), you can customize the error message by setting the `error-message-strategy` property. +By default, a `RawRecordHeaderErrorMessageStrategy` is used, to provide access to the converted message as well as the raw `ConsumerRecord`. -Starting with _Spring for Apache Kafka version 2.2_ (_Spring Integration Kafka 3.1_), the container factory used for `@KafkaListener` annotations can also be used to create `ConcurrentMessageListenerContainer` s for other purposes. +Starting with Spring for Apache Kafka version 2.2 (Spring Integration Kafka 3.1), you can also use the container factory that is used for `@KafkaListener` annotations to create `ConcurrentMessageListenerContainer` instances for other purposes. See <> for an example. -With the Java DSL, the container does not have to be configured as a `@Bean` because the DSL will register the container as a bean. +With the Java DSL, the container does not have to be configured as a `@Bean`, because the DSL registers the container as a bean. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -293,21 +316,25 @@ public IntegrationFlow topic2ListenerFromKafkaFlow() { get(); } ---- +==== -Notice that, in this case, the adapter is given an `id` ("topic2Adapter"); the container will be registered in the application context with the name `topic2Adapter.container`. -If the adapter does not have an `id` property, the container's bean name will be the container's fully qualified class name + `#n` where `n` is incremented for each container. +Notice that, in this case, the adapter is given an `id` (`topic2Adapter`). +The container is registered in the application context with a name of `topic2Adapter.container`. +If the adapter does not have an `id` property, the container's bean name is the container's fully qualified class name plus `#n`, where `n` is incremented for each container. [[si-outbound-gateway]] ==== Outbound Gateway -The outbound gateway is for request/reply operations; it is different to most Spring Integration gateways in that the sending thread does not block in the gateway, the reply is processed on the reply listener container thread. -Of course, if user code invokes the gateway behind a synchronous https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#gateway[Messaging Gateway], the user thread will block there until the reply is received (or a timeout occurs). +The outbound gateway is for request/reply operations. +It differs from most Spring Integration gateways in that the sending thread does not block in the gateway and the reply is processed on the reply listener container thread. +If your code invokes the gateway behind a synchronous https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#gateway[Messaging Gateway], the user thread blocks there until the reply is received (or a timeout occurs). -IMPORTANT: the gateway will not accept requests until the reply container has been assigned its topics and partitions. +IMPORTANT: The gateway does not accept requests until the reply container has been assigned its topics and partitions. It is suggested that you add a `ConsumerRebalanceListener` to the template's reply container properties and wait for the `onPartitionsAssigned` call before sending messages to the gateway. -Here is an example of configuring a gateway, with Java Configuration: +The following example shows how to configure a gateway with Java: +==== [source, java] ---- @Bean @@ -317,20 +344,23 @@ public KafkaProducerMessageHandler outGateway( return new KafkaProducerMessageHandler<>(kafkaTemplate); } ---- +==== -Notice that the same class as the <> is used, the only difference being that the kafka template passed into the constructor is a `ReplyingKafkaTemplate` - see <> for more information. +Notice that the same class as the <> is used, the only difference being that the Kafka template passed into the constructor is a `ReplyingKafkaTemplate`. +See <> for more information. -The outbound topic, partition, key etc, are determined the same way as the outbound adapter. +The outbound topic, partition, key, and so on are determined in the same way as the outbound adapter. The reply topic is determined as follows: -1. A message header `KafkaHeaders.REPLY_TOPIC`, if present (must have a `String` or `byte[]` value) - validated against the template's reply container subscribed topics. -2. If the template's `replyContainer` is subscribed to just one topic, it will be used. +. A message header named `KafkaHeaders.REPLY_TOPIC` (if present, it must have a `String` or `byte[]` value) is validated against the template's reply container's subscribed topics. +. If the template's `replyContainer` is subscribed to only one topic, it is used. You can also specify a `KafkaHeaders.REPLY_PARTITION` header to determine a specific partition to be used for replies. -Again, this is validated against the template's reply container subscriptions. +Again, this is validated against the template's reply container's subscriptions. -Configuring with the Java DSL: +The following example shows how to configure an outbound gateway with the Java DSL: +==== [source, java] ---- @Bean @@ -342,9 +372,11 @@ public IntegrationFlow outboundGateFlow( .get(); } ---- +==== -Or: +Alternatively, you can also use a configuration similar to the following bean: +==== [source, java] ---- @Bean @@ -356,16 +388,18 @@ public IntegrationFlow outboundGateFlow() { .get(); } ---- +==== -XML configuration is not currently available for this component. +NOTE: XML configuration is not currently available for this component. [[si-inbound-gateway]] ==== Inbound Gateway The inbound gateway is for request/reply operations. -Configuring an inbound gateway with Java Configuration: +The following example shows how to configure an inbound gateway with Java: +==== [source, java] ---- @Bean @@ -381,9 +415,11 @@ public KafkaInboundGateway inboundGateway( return gateway; } ---- +==== -Configuring a simple upper case converter with the Java DSL: +The following example shows how to configure a simple upper case converter with the Java DSL: +==== [source, java] ---- @Bean @@ -397,9 +433,11 @@ public IntegrationFlow serverGateway( .get(); } ---- +==== -Or: +Alternatively, you could configure an upper-case converter by using code similar to the following: +==== [source, java] ---- @Bean @@ -412,20 +450,24 @@ public IntegrationFlow serverGateway() { .get(); } ---- +==== -XML configuration is not currently available for this component. +NOTE: XML configuration is not currently available for this component. -Starting with _Spring for Apache Kafka version 2.2_ (_Spring Integration Kafka 3.1_), the container factory used for `@KafkaListener` annotations can also be used to create `ConcurrentMessageListenerContainer` s for other purposes. +Starting with Spring for Apache Kafka version 2.2 (Spring Integration Kafka 3.1), you can also use the container factory that is used for `@KafkaListener` annotations to create `ConcurrentMessageListenerContainer` instances for other purposes. See <> and <> for examples. [[message-conversion]] ==== Message Conversion -A `StringJsonMessageConverter` is provided, see <> for more information. +A `StringJsonMessageConverter` is provided. +See <> for more information. When using this converter with a message-driven channel adapter, you can specify the type to which you want the incoming payload to be converted. This is achieved by setting the `payload-type` attribute (`payloadType` property) on the adapter. +The following example shows how to do so in XML configuration: +==== [source, xml] ---- ---- +==== +The following example shows how to set the `payload-type` attribute (`payloadType` property) on the adapter in Java configuration: + +==== [source, java] ---- @Bean @@ -457,15 +503,18 @@ public KafkaMessageDrivenChannelAdapter return kafkaMessageDrivenChannelAdapter; } ---- +==== [[si-tombstones]] ==== Null Payloads and Log Compaction 'Tombstone' Records -Spring Messaging `Message` objects cannot have `null` payloads; when using the Kafka endpoints, `null` payloads (also known as tombstone records) are represented by a payload of type `KafkaNull`. +Spring Messaging `Message` objects cannot have `null` payloads. +When you use the Kafka endpoints, `null` payloads (also known as tombstone records) are represented by a payload of type `KafkaNull`. See <> for more information. Starting with version 3.1 of Spring Integration Kafka, such records can now be received by Spring Integration POJO methods with a true `null` value instead. -Simply mark the parameter with `@Payload(required = false)`. +To do so, mark the parameter with `@Payload(required = false)`. +The following example shows how to do so: ==== [source, java] @@ -487,11 +536,11 @@ See the http://projects.spring.io/spring-kafka/[Spring for Apache Kafka Project The 2.1.x branch introduced the following changes: -* Update to `spring-kafka` 1.1.x; including support of batch payloads -* Support `sync` outbound requests via XML configuration +* Update to `spring-kafka` 1.1.x, including support of batch payloads +* Support `sync` outbound requests in XML configuration * Support `payload-type` for inbound channel adapters -* Support for Enhanced Error handling for the inbound channel adapter (2.1.1) -* Support for send success/failure messages (2.1.2) +* Support for enhanced error handling for the inbound channel adapter (2.1.1) +* Support for send success and failure messages (2.1.2) ===== 2.2.x @@ -503,7 +552,7 @@ The 2.2.x branch introduced the following changes: The 2.3.x branch introduced the following changes: -* Update to `spring-kafka` 1.3.x; including support for transactions and header mapping provided by `kafka-clients` 0.11.0.0 +* Update to `spring-kafka` 1.3.x, including support for transactions and header mapping provided by `kafka-clients` 0.11.0.0 * Support for record timestamps ===== 3.0.x @@ -511,7 +560,7 @@ The 2.3.x branch introduced the following changes: * Update to `spring-kafka` 2.1.x and `kafka-clients` 1.0.0 * Support `ConsumerAwareMessageListener` (`Consumer` is available in a message header) * Update to Spring Integration 5.0 and Java 8 -* Moved Java DSL to main project +* Moved Java DSL to the main project * Added inbound and outbound gateways (3.0.2) ===== 3.1.x diff --git a/src/reference/asciidoc/streams.adoc b/src/reference/asciidoc/streams.adoc index 14029fe5..349499f6 100644 --- a/src/reference/asciidoc/streams.adoc +++ b/src/reference/asciidoc/streams.adoc @@ -1,16 +1,15 @@ [[kafka-streams]] === Kafka Streams Support -==== Introduction - -Starting with _version 1.1.4_, Spring for Apache Kafka provides first class support for https://kafka.apache.org/documentation/streams[Kafka Streams]. -For using it from a Spring application, the `kafka-streams` jar must be present on classpath. -It is an optional dependency of the `spring-kafka` project and isn't downloaded transitively. +Starting with version 1.1.4, Spring for Apache Kafka provides first-class support for https://kafka.apache.org/documentation/streams[Kafka Streams]. +To use it from a Spring application, the `kafka-streams` jar must be present on classpath. +It is an optional dependency of the `spring-kafka` project and is not downloaded transitively. ==== Basics -The reference Apache Kafka Streams documentation suggests this way of using the API: +The reference Apache Kafka Streams documentation suggests the following way of using the API: +==== [source, java] ---- // Use the builders to define the actual processing topology, e.g. to specify @@ -32,17 +31,25 @@ streams.start(); // Stop the Kafka Streams instance streams.close(); ---- +==== -So, we have two main components: `StreamsBuilder` with an API to build `KStream` (or `KTable`) instances and `KafkaStreams` to manage their lifecycle. -Note: all `KStream` instances exposed to a `KafkaStreams` instance by a single `StreamsBuilder` will be started and stopped at the same time, even if they have a fully different logic. -In other words all our streams defined by a `StreamsBuilder` are tied with a single lifecycle control. -Once a `KafkaStreams` instance has been closed via `streams.close()` it cannot be restarted, and a new `KafkaStreams` instance to restart stream processing must be created instead. +So, we have two main components: + +* `StreamsBuilder`: With an API to build `KStream` (or `KTable`) instances. +* `KafkaStreams`: To manage the lifecycle of those instances. + +NOTE: All `KStream` instances exposed to a `KafkaStreams` instance by a single `StreamsBuilder` are started and stopped at the same time, even if they have different logic. +In other words, all streams defined by a `StreamsBuilder` are tied with a single lifecycle control. +Once a `KafkaStreams` instance has been closed by `streams.close()`, it cannot be restarted. +Instead, a new `KafkaStreams` instance to restart stream processing must be created. ==== Spring Management -To simplify the usage of Kafka Streams from the Spring application context perspective and utilize the lifecycle management via container, the Spring for Apache Kafka introduces `StreamsBuilderFactoryBean`. -This is an `AbstractFactoryBean` implementation to expose a `StreamsBuilder` singleton instance as a bean: +To simplify using Kafka Streams from the Spring application context perspective and use the lifecycle management through a container, the Spring for Apache Kafka introduces `StreamsBuilderFactoryBean`. +This is an `AbstractFactoryBean` implementation to expose a `StreamsBuilder` singleton instance as a bean. +The following example creates such a bean: +==== [source, java] ---- @Bean @@ -50,14 +57,18 @@ public FactoryBean myKStreamBuilder(KafkaStreamsConfi return new StreamsBuilderFactoryBean(streamsConfig); } ---- +==== -IMPORTANT: Starting with version 2.2, the stream configuration is now provided as a `KafkaStreamsConfiguration` object, rather than a `StreamsConfig`. +IMPORTANT: Starting with version 2.2, the stream configuration is now provided as a `KafkaStreamsConfiguration` object rather than a `StreamsConfig`. -The `StreamsBuilderFactoryBean` also implements `SmartLifecycle` to manage lifecycle of an internal `KafkaStreams` instance. -Similar to the Kafka Streams API, the `KStream` instances must be defined before starting the `KafkaStreams`, and that also applies for the Spring API for Kafka Streams. -Therefore we have to declare `KStream` s on the `StreamsBuilder` before the application context is refreshed, when we use default `autoStartup = true` on the `StreamsBuilderFactoryBean`. -For example, `KStream` can be just as a regular bean definition, meanwhile the Kafka Streams API is used without any impacts: +The `StreamsBuilderFactoryBean` also implements `SmartLifecycle` to manage the lifecycle of an internal `KafkaStreams` instance. +Similar to the Kafka Streams API, you must define the `KStream` instances before you start the `KafkaStreams`. +That also applies for the Spring API for Kafka Streams. +Therefore, when you use default `autoStartup = true` on the `StreamsBuilderFactoryBean`, you must declare `KStream` instances on the `StreamsBuilder` before the application context is refreshed. +For example, `KStream` can be a regular bean definition, while the Kafka Streams API is used without any impacts. +The following example shows how to do so: +==== [source, java] ---- @Bean @@ -67,17 +78,20 @@ public KStream kStream(StreamsBuilder kStreamBuilder) { return stream; } ---- +==== -If you would like to control lifecycle manually (e.g. stop and start by some condition), you can reference the `StreamsBuilderFactoryBean` bean directly using factory bean (`&`) http://docs.spring.io/spring/docs/current/spring-framework-reference/html/beans.html#beans-factory-extension-factorybean[prefix]. -Since `StreamsBuilderFactoryBean` utilize its internal `KafkaStreams` instance, it is safe to stop and restart it again - a new `KafkaStreams` is created on each `start()`. -Also consider using different `StreamsBuilderFactoryBean` s, if you would like to control lifecycles for `KStream` instances separately. +If you would like to control the lifecycle manually (for example, stopping and starting by some condition), you can reference the `StreamsBuilderFactoryBean` bean directly by using the factory bean (`&`) http://docs.spring.io/spring/docs/current/spring-framework-reference/html/beans.html#beans-factory-extension-factorybean[prefix]. +Since `StreamsBuilderFactoryBean` use its internal `KafkaStreams` instance, it is safe to stop and restart it again. +A new `KafkaStreams` is created on each `start()`. +You might also consider using different `StreamsBuilderFactoryBean` instances, if you would like to control the lifecycles for `KStream` instances separately. -You also can specify `KafkaStreams.StateListener`, `Thread.UncaughtExceptionHandler` and `StateRestoreListener` options on the `StreamsBuilderFactoryBean` which are delegated to the internal `KafkaStreams` instance. -Also apart from setting those options indirectly on `StreamsBuilderFactoryBean`, starting with _version 2.1.5_, a `KafkaStreamsCustomizer` callback interface can be used to configure inner `KafkaStreams` instance. -Note that `KafkaStreamsCustomizer` will override the options which are given via `StreamsBuilderFactoryBean`. -That internal `KafkaStreams` instance can be accessed via `StreamsBuilderFactoryBean.getKafkaStreams()` if you need to perform some `KafkaStreams` operations directly. -You can autowire `StreamsBuilderFactoryBean` bean by type, but you should be sure that you use full type in the bean definition, for example: +You also can specify `KafkaStreams.StateListener`, `Thread.UncaughtExceptionHandler`, and `StateRestoreListener` options on the `StreamsBuilderFactoryBean`, which are delegated to the internal `KafkaStreams` instance. +Also, apart from setting those options indirectly on `StreamsBuilderFactoryBean`, starting with _version 2.1.5_, you can use a `KafkaStreamsCustomizer` callback interface to configure an inner `KafkaStreams` instance. +Note that `KafkaStreamsCustomizer` overrides the options provided by `StreamsBuilderFactoryBean`. +If you need to perform some `KafkaStreams` operations directly, you can access that internal `KafkaStreams` instance by using `StreamsBuilderFactoryBean.getKafkaStreams()`. +You can autowire `StreamsBuilderFactoryBean` bean by type, but you should be sure to use the full type in the bean definition, as the following example shows: +==== [source,java] ---- @Bean @@ -88,8 +102,12 @@ public StreamsBuilderFactoryBean myKStreamBuilder(KafkaStreamsConfiguration stre @Autowired private StreamsBuilderFactoryBean myKStreamBuilderFactoryBean; ---- +==== -Or add `@Qualifier` for injection by name if you use interface bean definition: +Alternatively, you can add `@Qualifier` for injection by name if you use interface bean definition. +The following example shows how to do so: + +==== [source,java] ---- @Bean @@ -101,29 +119,33 @@ public FactoryBean myKStreamBuilder(KafkaStreamsConfiguration st @Qualifier("&myKStreamBuilder") private StreamsBuilderFactoryBean myKStreamBuilderFactoryBean; ---- +==== -==== JSON Serdes +==== JSON Serialization and Deserialization -For serializing and deserializing data when reading or writing to topics or state stores in JSON format, Spring Kafka provides a `JsonSerde` implementation using JSON, delegating to the `JsonSerializer` and `JsonDeserializer` described in <>. -The `JsonSerde` provides the same configuration options via its constructor (target type and/or `ObjectMapper`). -In the following example we use the `JsonSerde` to serialize and deserialize the `Foo` payload of a Kafka stream - the `JsonSerde` can be used in a similar fashion wherever an instance is required. +For serializing and deserializing data when reading or writing to topics or state stores in JSON format, Spring Kafka provides a `JsonSerde` implementation that uses JSON, delegating to the `JsonSerializer` and `JsonDeserializer` described in <>. +The `JsonSerde` implementation provides the same configuration options through its constructor (target type or `ObjectMapper`). +In the following example, we use the `JsonSerde` to serialize and deserialize the `Cat` payload of a Kafka stream (the `JsonSerde` can be used in a similar fashion wherever an instance is required): +==== [source,java] ---- -stream.through(Serdes.Integer(), new JsonSerde<>(Foo.class), "foos"); +stream.through(Serdes.Integer(), new JsonSerde<>(Cat.class), "cats"); ---- +==== IMPORTANT: Since Kafka Streams do not support headers, the `addTypeInfo` property on the `JsonSerializer` is ignored. -==== KafkaStreamsBrancher +==== Using `KafkaStreamsBrancher` -`KafkaStreamBrancher` class introduces more convenient way to build conditional branches on top of `KStream`. +The `KafkaStreamBrancher` class introduces a more convenient way to build conditional branches on top of `KStream`. -Instead of +Consider the following example that does not use `KafkaStreamBrancher`: +==== [source,java] ---- -KStream[] branches = builder.stream("source").branch( +KStream[] branches = builder.stream("source").branch( (key, value) -> value.contains("A"), (key, value) -> value.contains("B"), (key, value) -> true @@ -132,9 +154,11 @@ branches[0].to("A"); branches[1].to("B"); branches[2].to("C"); ---- +==== -we can use +The following example uses `KafkaStreamBrancher`: +==== [source,java] ---- new KafkaStreamsBrancher() @@ -145,27 +169,29 @@ new KafkaStreamsBrancher() .onTopOf(builder.stream("source")); //onTopOf method returns the provided stream so we can continue with method chaining ---- +==== [[streams-config]] ==== Configuration To configure the Kafka Streams environment, the `StreamsBuilderFactoryBean` requires a `KafkaStreamsConfiguration` instance. -See Apache Kafka https://kafka.apache.org/0102/documentation/#streamsconfigs[documentation] for all possible options. +See the Apache Kafka https://kafka.apache.org/0102/documentation/#streamsconfigs[documentation] for all possible options. -IMPORTANT: Starting with version 2.2, the stream configuration is now provided as a `KafkaStreamsConfiguration` object, rather than a `StreamsConfig`. +IMPORTANT: Starting with version 2.2, the stream configuration is now provided as a `KafkaStreamsConfiguration` object, rather than as a `StreamsConfig`. -To avoid boilerplate code for most cases, especially when you develop micro services, Spring for Apache Kafka provides the `@EnableKafkaStreams` annotation, which should be placed on a `@Configuration` class. -All you need is to declare a `KafkaStreamsConfiguration` bean with the name `defaultKafkaStreamsConfig`. -A `StreamsBuilder` bean, with the name `defaultKafkaStreamsBuilder`, will be declared in the application context automatically. -Any additional `StreamsBuilderFactoryBean` beans can be declared and used as well. +To avoid boilerplate code for most cases, especially when you develop microservices, Spring for Apache Kafka provides the `@EnableKafkaStreams` annotation, which you should placed on a `@Configuration` class. +All you need is to declare a `KafkaStreamsConfiguration` bean named `defaultKafkaStreamsConfig`. +A `StreamsBuilder` bean, named `defaultKafkaStreamsBuilder`, is automatically declared in the application context. +You can declare and use any additional `StreamsBuilderFactoryBean` beans as well. By default, when the factory bean is stopped, the `KafkaStreams.cleanUp()` method is called. -Starting with _version 2.1.2_, the factory bean has additional constructors, taking a `CleanupConfig` object that has properties to allow you to control whether the `cleanUp()` method is called during `start()`, `stop()`, or neither. +Starting with version 2.1.2, the factory bean has additional constructors, taking a `CleanupConfig` object that has properties to let you control whether the `cleanUp()` method is called during `start()` or `stop()` or neither. ==== Kafka Streams Example -Putting it all together: +The following example combines all the topics we have covered in this chapter: +==== [source, java] ---- @Configuration @@ -204,3 +230,4 @@ public static class KafkaStreamsConfig { } ---- +==== diff --git a/src/reference/asciidoc/testing.adoc b/src/reference/asciidoc/testing.adoc index 982eea4f..b12e3fc4 100644 --- a/src/reference/asciidoc/testing.adoc +++ b/src/reference/asciidoc/testing.adoc @@ -1,14 +1,14 @@ [[testing]] === Testing Applications -==== Introduction - The `spring-kafka-test` jar contains some useful utilities to assist with testing your applications. ==== JUnit -`o.s.kafka.test.utils.KafkaTestUtils` provides some static methods to set up producer and consumer properties: +`o.s.kafka.test.utils.KafkaTestUtils` provides some static methods to set up producer and consumer properties. +The following listing shows those method signatures: +==== [source, java] ---- /** @@ -28,10 +28,13 @@ public static Map consumerProps(String group, String autoCommit, */ public static Map senderProps(EmbeddedKafkaBroker embeddedKafka) { ... } ---- +==== -A JUnit 4 `@Rule` wrapper for the `EmbeddedKafkaBroker` is provided that creates an embedded Kafka and an embedded Zookeeper server. -(See <> about using `@EmbeddedKafka` with JUnit 5). +A JUnit 4 `@Rule` wrapper for the `EmbeddedKafkaBroker` is provided to create an embedded Kafka and an embedded Zookeeper server. +(See <> for information about using `@EmbeddedKafka` with JUnit 5). +The following listing shows the signatures of those methods: +==== [source, java] ---- /** @@ -52,9 +55,12 @@ public EmbeddedKafkaRule(int count, boolean controlledShutdown, String... topics */ public EmbeddedKafkaRule(int count, boolean controlledShutdown, int partitions, String... topics) { ... } ---- +==== -The `EmbeddedKafkaBroker` class has a utility method allowing you to consume for all the topics it created: +The `EmbeddedKafkaBroker` class has a utility method that lets you consume for all the topics it created. +The following example shows how to use it: +==== [source, java] ---- Map consumerProps = KafkaTestUtils.consumerProps("testT", "false", embeddedKafka); @@ -63,9 +69,12 @@ DefaultKafkaConsumerFactory cf = new DefaultKafkaConsumerFactor Consumer consumer = cf.createConsumer(); embeddedKafka.consumeFromAllEmbeddedTopics(consumer); ---- +==== -The `KafkaTestUtils` has some utility methods to fetch results from the consumer: +The `KafkaTestUtils` has some utility methods to fetch results from the consumer. +The following listing shows those method signatures: +==== [source, java] ---- /** @@ -84,9 +93,11 @@ public static ConsumerRecord getSingleRecord(Consumer consume */ public static ConsumerRecords getRecords(Consumer consumer) { ... } ---- +==== -Usage: +The following example shows how to use `KafkaTestUtils`: +==== [source, java] ---- ... @@ -94,27 +105,30 @@ template.sendDefault(0, 2, "bar"); ConsumerRecord received = KafkaTestUtils.getSingleRecord(consumer, "topic"); ... ---- +==== -When the embedded Kafka and embedded Zookeeper server are started by by the `EmbeddedKafkaBroker`, a system property `spring.embedded.kafka.brokers` is set to the address of the Kafka broker(s) and a system property `spring.embedded.zookeeper.connect` is set to the address of Zookeeper. -Convenient constants `EmbeddedKafkaBroker.SPRING_EMBEDDED_KAFKA_BROKERS` and `EmbeddedKafkaBroker.SPRING_EMBEDDED_ZOOKEEPER_CONNECT` are provided for this property. +When the embedded Kafka and embedded Zookeeper server are started by the `EmbeddedKafkaBroker`, a system property named `spring.embedded.kafka.brokers` is set to the address of the Kafka brokers and a system property named `spring.embedded.zookeeper.connect` is set to the address of Zookeeper. +Convenient constants (`EmbeddedKafkaBroker.SPRING_EMBEDDED_KAFKA_BROKERS` and `EmbeddedKafkaBroker.SPRING_EMBEDDED_ZOOKEEPER_CONNECT`) are provided for this property. -With the `EmbeddedKafkaBroker.brokerProperties(Map)` you can provide additional properties for the Kafka server(s). +With the `EmbeddedKafkaBroker.brokerProperties(Map)`, you can provide additional properties for the Kafka servers. See https://kafka.apache.org/documentation/#brokerconfigs[Kafka Config] for more information about possible broker properties. ==== Configuring Topics +The following example configuration creates topics called `cat` and `hat` with five partitions, a topic called `thing1` with 10 partitions, and a topic called `thing2` with 15 partitions: + ==== [source, java] ---- public class MyTests { @ClassRule - private static EmbeddedKafkaRule embeddedKafka = new EmbeddedKafkaRule(1, false, 5, "foo", "bar"); + private static EmbeddedKafkaRule embeddedKafka = new EmbeddedKafkaRule(1, false, 5, "cat", "hat"); @Test public void test() { embeddedKafkaRule.getEmbeddedKafka() - .addTopics(new NewTopic("baz", 10, (short) 1), new NewTopic("qux", 15, (short) 1)); + .addTopics(new NewTopic("thing1", 10, (short) 1), new NewTopic("thing2", 15, (short) 1)); ... } @@ -122,11 +136,9 @@ public class MyTests { ---- ==== -The above configuration will create topics `foo` and `bar` with 5 partitions, `baz` with 10 and `qux` with 15. +==== Using the Same Brokers for Multiple Test Classes -==== Using the Same Broker(s) for Multiple Test Classes - -There is no built-in support for this, but it can be achieved with something similar to the following: +There is no built-in support for doing so, but you can use the same broker for multiple test classes with something similar to the following: ==== [source, java] @@ -158,7 +170,7 @@ public final class EmbeddedKafkaHolder { ---- ==== -And then, in each test class: +Then, in each test class, you can use something similar to the following: ==== [source, java] @@ -171,16 +183,18 @@ private static EmbeddedKafkaRule embeddedKafka = EmbeddedKafkaHolder.getEmbedded ---- ==== -IMPORTANT: This example provides no mechanism for shutting down the broker(s) when all tests are complete. +IMPORTANT: The preceding example provides no mechanism for shutting down the brokers when all tests are complete. This could be a problem if, say, you run your tests in a Gradle daemon. -You should not use this technique in such a situation, or use something to call `destroy()` on the `EmbeddedKafkaBroker` when your tests are complete. +You should not use this technique in such a situation, or you should use something to call `destroy()` on the `EmbeddedKafkaBroker` when your tests are complete. [[embedded-kafka-annotation]] ==== @EmbeddedKafka Annotation -It is generally recommended to use the rule as a `@ClassRule` to avoid starting/stopping the broker between tests (and use a different topic for each test). -Starting with _version 2.0_, if you are using Spring's test application context caching, you can also declare a `EmbeddedKafkaBroker` bean, so a single broker can be used across multiple test classes. -For convenience a test class level `@EmbeddedKafka` annotation is provided with the purpose to register `EmbeddedKafkaBroker` bean: +We generally recommend that you use the rule as a `@ClassRule` to avoid starting and stopping the broker between tests (and use a different topic for each test). +Starting with version 2.0, if you use Spring's test application context caching, you can also declare a `EmbeddedKafkaBroker` bean, so a single broker can be used across multiple test classes. +For convenience, we provide a test class-level annotation called `@EmbeddedKafka` to register the `EmbeddedKafkaBroker` bean. +The following example shows how to use it: +==== [source, java] ---- @RunWith(SpringRunner.class) @@ -224,9 +238,13 @@ public class KafkaStreamsTests { } ---- -Starting with version 2.2.4, the `@EmbeddedKafka` annotation can also be used to specify the kafka ports property. +==== -The `topics`, `brokerProperties` and `brokerPropertiesLocation` attributes of `@EmbeddedKafka` support property placeholder resolutions: +Starting with version 2.2.4, you can also use the `@EmbeddedKafka` annotation to specify the Kafka ports property. + +The following example sets the `topics`, `brokerProperties`, and `brokerPropertiesLocation` attributes of `@EmbeddedKafka` support property placeholder resolutions: + +==== [source, java] ---- @TestPropertySource(locations = "classpath:/test.properties") @@ -236,21 +254,26 @@ The `topics`, `brokerProperties` and `brokerPropertiesLocation` attributes of `@ "auto.create.topics.enable=${kafka.broker.topics-enable:true}" } brokerPropertiesLocation = "classpath:/broker.properties") ---- -In the example above, the property placeholders `${kafka.topics.another-topic}`, `${kafka.broker.logs-dir}`, and `${kafka.broker.port}` are resolved from the Spring `Environment`. -In addition the broker properties are loaded from the `broker.properties` classpath resource specified by the `brokerPropertiesLocation`. +==== + +In the preceding example, the property placeholders `${kafka.topics.another-topic}`, `${kafka.broker.logs-dir}`, and `${kafka.broker.port}` are resolved from the Spring `Environment`. +In addition, the broker properties are loaded from the `broker.properties` classpath resource specified by the `brokerPropertiesLocation`. Property placeholders are resolved for the `brokerPropertiesLocation` URL and for any property placeholders found in the resource. Properties defined by `brokerProperties` override properties found in `brokerPropertiesLocation`. -The `@EmbeddedKafka` annotation can be used with JUnit 4 or JUnit 5. +You can use the `@EmbeddedKafka` annotation with JUnit 4 or JUnit 5. -==== Embedded Broker in `@SpringBootTest` s +==== Embedded Broker in `@SpringBootTest` Annotations https://start.spring.io/[Spring Initializr] now automatically adds the `spring-kafka-test` dependency in test scope to the project configuration. -IMPORTANT: If your application is using the Kafka binder in `spring-cloud-stream`, if you want to use an embedded broker for tests, you must remove the `spring-cloud-stream-test-support` dependency because it replaces the real binder with a test binder for test casess. -If you wish some tests to use the test binder and some to use the embedded broker, tests using the real binder need to disable the test binder by excluding the binder auto configuration in the test class. - +[IMPORTANT] ==== +If your application uses the Kafka binder in `spring-cloud-stream` and if you want to use an embedded broker for tests, you must remove the `spring-cloud-stream-test-support` dependency, because it replaces the real binder with a test binder for test cases. +If you wish some tests to use the test binder and some to use the embedded broker, tests that use the real binder need to disable the test binder by excluding the binder auto configuration in the test class. +The following example shows how to do so: + +===== [source, java] ---- @RunWith(SpringRunner.class) @@ -260,12 +283,21 @@ public class MyApplicationTests { ... } ---- +===== ==== There are several ways to use an embedded broker in a Spring Boot application test. +They include: + +* <> +* <> + +[[kafka-testing-junit4-class-rule]] ===== JUnit4 Class Rule +The following example shows how to use a JUnit4 class rule to create an embedded broker: + ==== [source, java] ---- @@ -295,8 +327,11 @@ public class MyApplicationTests { ---- ==== +[[kafka-testing-embeddedkafka-annotation]] ===== `@EmbeddedKafka` Annotation or `EmbeddedKafkaBroker` Bean +The following example shows how to use an `@EmbeddedKafka` Annotation to create an embedded broker: + ==== [source, java] ---- @@ -326,6 +361,7 @@ public class MyApplicationTests { The `o.s.kafka.test.hamcrest.KafkaMatchers` provides the following matchers: +==== [source, java] ---- /** @@ -369,9 +405,13 @@ public static Matcher> hasTimestamp(TimestampType type, lon return new ConsumerRecordTimestampMatcher(type, ts); } ---- +==== ==== AssertJ Conditions +You can use the following AssertJ conditions: + +==== [source, java] ---- /** @@ -411,11 +451,13 @@ public static Condition> timestamp(TimestampType type, long return new ConsumerRecordTimestampCondition(type, value); } ---- +==== ==== Example -Putting it all together: +The following example brings together most of the topics covered in this chapter: +==== [source, java] ---- public class KafkaTemplateTests { @@ -469,9 +511,12 @@ public class KafkaTemplateTests { } ---- +==== -The above uses the hamcrest matchers; with `AssertJ`, the final part looks like this... +The preceding example uses the Hamcrest matchers. +With `AssertJ`, the final part looks like the following code: +==== [source, java] ---- assertThat(records.poll(10, TimeUnit.SECONDS)).has(value("foo")); @@ -486,3 +531,4 @@ assertThat(received).has(key(2)); assertThat(received).has(partition(0)); assertThat(received).has(value("baz")); ---- +==== diff --git a/src/reference/asciidoc/whats-new.adoc b/src/reference/asciidoc/whats-new.adoc index 79e981bb..59ce2f2a 100644 --- a/src/reference/asciidoc/whats-new.adoc +++ b/src/reference/asciidoc/whats-new.adoc @@ -1,46 +1,46 @@ -=== What's new in 2.2 Since 2.1 +=== What's New in 2.2 Since 2.1 + +This section covers the changes made from version 2.1 to version 2.2. ==== Kafka Client Version This version requires the 2.0.0 `kafka-clients` or higher. -==== Class/Package Changes +==== Class and Package Changes -The class `ContainerProperties` has been moved from `org.springframework.kafka.listener.config` to `org.springframework.kafka.listener`. +The `ContainerProperties` class has been moved from `org.springframework.kafka.listener.config` to `org.springframework.kafka.listener`. -The enum `AckMode` has been moved from `AbstractMessageListenerContainer` to `ContainerProperties`. +The `AckMode` enum has been moved from `AbstractMessageListenerContainer` to `ContainerProperties`. -`setBatchErrorHandler()` and `setErrorHandler()` methods have been moved from `ContainerProperties` to `AbstractMessageListenerContainer` (and `AbstractKafkaListenerContainerFactory`). +The `setBatchErrorHandler()` and `setErrorHandler()` methods have been moved from `ContainerProperties` to both `AbstractMessageListenerContainer` and `AbstractKafkaListenerContainerFactory`. -==== After rollback processing +==== After Rollback Processing -A new `AfterRollbackProcessor` strategy is provided - see <> for more information. +A new `AfterRollbackProcessor` strategy is provided. +See <> for more information. -==== ConcurrentKafkaListenerContainerFactory changes +==== `ConcurrentKafkaListenerContainerFactory` Changes -The `ConcurrentKafkaListenerContainerFactory` can now be used to create/configure any `ConcurrentMessageListenerContainer`, not just those for `@KafkaListener` annotations. +You can now use the `ConcurrentKafkaListenerContainerFactory` to create and configure any `ConcurrentMessageListenerContainer`, not only those for `@KafkaListener` annotations. See <> for more information. ==== Listener Container Changes -A new container property `missingTopicsFatal` has been added. - +A new container property (`missingTopicsFatal`) has been added. See <> for more information. A `ConsumerStoppedEvent` is now emitted when a consumer terminates. - See <> for more information. Batch listeners can optionally receive the complete `ConsumerRecords` object instead of a `List`. - See <> for more information. -The `DefaultAfterRollbackProcessor` and `SeekToCurrentErrorHandler` can now recover (skip) records that keep failing, and will do so after 10 failures, by default. +The `DefaultAfterRollbackProcessor` and `SeekToCurrentErrorHandler` can now recover (skip) records that keep failing, and, by default, does so after 10 failures. They can be configured to publish failed records to a dead-letter topic. -Starting with version 2.2.4, the consumer's group id can be used while selecting the dead letter topic name. +Starting with version 2.2.4, the consumer's group ID can be used while selecting the dead letter topic name. -See <>, <> and <> for more information. +See <>, <>, and <> for more information. The `ConsumerStoppingEvent` has been added. See <> for more information. @@ -49,11 +49,9 @@ See <> for more information. You can now override the `concurrency` and `autoStartup` properties of the listener container factory by setting properties on the annotation. You can now add configuration to determine which headers (if any) are copied to a reply message. - See <> for more information. You can now use `@KafkaListener` as a meta-annotation on your own annotations. - See <> for more information. It is now easier to configure a `Validator` for `@Payload` validation. @@ -62,37 +60,36 @@ See <> for more information. ==== Header Mapping Changes Headers of type `MimeType` and `MediaType` are now mapped as simple strings in the `RecordHeader` value. -Previously, they were mapped as JSON and only `MimeType` was decoded, `MediaType` could not be decoded. +Previously, they were mapped as JSON and only `MimeType` was decoded. +`MediaType` could not be decoded. They are now simple strings for interoperability. -Also, the `DefaultKafkaHeaderMapper` has a new method `addToStringClasses` allowing the specification of types that should be mapped using `toString()` instead of JSON. - +Also, the `DefaultKafkaHeaderMapper` has a new `addToStringClasses` method, allowing the specification of types that should be mapped by using `toString()` instead of JSON. See <> for more information. ==== Embedded Kafka Changes -The `KafkaEmbedded` class and its `KafkaRule` interface have need deprecated in favor of the `EmbeddedKafkaBroker` and its JUnit 4 `EmbeddedKafkaRule` wrapper. +The `KafkaEmbedded` class and its `KafkaRule` interface have been deprecated in favor of the `EmbeddedKafkaBroker` and its JUnit 4 `EmbeddedKafkaRule` wrapper. The `@EmbeddedKafka` annotation now populates an `EmbeddedKafkaBroker` bean instead of the deprecated `KafkaEmbedded`. -This allows the use of `@EmbeddedKafka` in JUnit 5 tests. -The `@EmbeddedKafka` annotation now has the attribute `ports` to specify the port which will populate the `EmbeddedKafkaBroker`. - +This change allows the use of `@EmbeddedKafka` in JUnit 5 tests. +The `@EmbeddedKafka` annotation now has the attribute `ports` to specify the port that populates the `EmbeddedKafkaBroker`. See <> for more information. ==== JsonSerializer/Deserializer Enhancements -You can now provide type mapping information using producer/consumer properties. +You can now provide type mapping information by using producer and consumer properties. New constructors are available on the deserializer to allow overriding the type header information with the supplied target type. -The `JsonDeserializer` will now remove any type information headers by default. +The `JsonDeserializer` now removes any type information headers by default. -The `JsonDeserializer` can now be configured to ignore type information headers using a kafka property (since 2.2.3). +You can now configure the `JsonDeserializer` to ignore type information headers by using a Kafka property (since 2.2.3). See <> for more information. ==== Kafka Streams Changes -The streams configuration bean must now be a `KafkaStreamsConfiguration` object instead of a `StreamsConfig`. +The streams configuration bean must now be a `KafkaStreamsConfiguration` object instead of a `StreamsConfig` object. The `StreamsBuilderFactoryBean` has been moved from package `...core` to `...config`. @@ -101,7 +98,7 @@ The `KafkaStreamBrancher` has been introduced for better end-user experience whe See <> and <> for more information. -==== Transactional Id +==== Transactional ID When a transaction is started by the listener container, the `transactional.id` is now the `transactionIdPrefix` appended with `..`. -This is to allow proper fencing of zombies https://www.confluent.io/blog/transactions-apache-kafka/[as described here]. +This change allows proper fencing of zombies, https://www.confluent.io/blog/transactions-apache-kafka/[as described here].