From a2458f30290d915018b01e4c8fa20a875b8d2578 Mon Sep 17 00:00:00 2001 From: Soby Chacko Date: Thu, 14 Mar 2024 10:54:08 -0400 Subject: [PATCH] Clarify batch-consuming docs for Kafka binder Based on this issue: https://github.com/spring-cloud/spring-cloud-stream/issues/2917 --- .../kafka/kafka-binder/consume-batches.adoc | 23 ++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/modules/ROOT/pages/kafka/kafka-binder/consume-batches.adoc b/docs/modules/ROOT/pages/kafka/kafka-binder/consume-batches.adoc index 9ad5856bb..3d1e12bbd 100644 --- a/docs/modules/ROOT/pages/kafka/kafka-binder/consume-batches.adoc +++ b/docs/modules/ROOT/pages/kafka/kafka-binder/consume-batches.adoc @@ -4,10 +4,31 @@ Starting with version 3.0, when `spring.cloud.stream.bindings..consumer.ba Otherwise, the method will be called with one record at a time. The size of the batch is controlled by Kafka consumer properties `max.poll.records`, `fetch.min.bytes`, `fetch.max.wait.ms`; refer to the Kafka documentation for more information. +When receiving the batches, the following type signatures are allowed: + +``` +List +Message +``` + +In the first option of `List`, the listener will not get any message headers. +If the second type signature (`Message>`) is used, then the headers can be accessed; however, all the headers are still be in the form of a `Collection`. +Let's take the following example. + +Assume that the `Message` contains a list with ten `Person` objects. +The `MessageHeaders` of the `Message` contains a map of headers with key as the header name and value as a list. +This list contains the header value for that header in the same order as the payload list. +Therefore, it is up to the application to correctly access the header from the `MessageHeaders` map based on the iteration of the payload list. + +Note that, type signatures in the form of `List>` is not allowed when consuming in batch-mode. + Starting with version `4.0.2`, the binder supports DLQ capabilities when consuming in batch mode. Keep in mind that, when using DLQ on a consumer binding that is in batch mode, all the records received from the previous poll will be delivered to the DLQ topic. IMPORTANT: Retry within the binder is not supported when using batch mode, so `maxAttempts` will be overridden to 1. You can configure a `DefaultErrorHandler` (using a `ListenerContainerCustomizer`) to achieve similar functionality to retry in the binder. You can also use a manual `AckMode` and call `Ackowledgment.nack(index, sleep)` to commit the offsets for a partial batch and have the remaining records redelivered. -Refer to the https://docs.spring.io/spring-kafka/docs/2.3.15.BUILD-SNAPSHOT/reference/html/#committing-offsets[Spring for Apache Kafka documentation] for more information about these techniques. +Refer to the https://docs.spring.io/spring-kafka/reference/kafka/receiving-messages/message-listener-container.html#committing-offsets[Spring for Apache Kafka documentation] for more information about these techniques. + +NOTE: When receiving `KafkaNull` objects in the batch-mode, the received list will contain a null element for the corresponding `KafkaNull` object. +This is true for both `List` style type signatures.