SpringCloud/spring-cloud-stream
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Kafka binder partitioning doc improvements (#2735)

Browse Source 
* Kafka binder partitioning doc improvements

* Update docs/src/main/asciidoc/kafka/kafka_partitions.adoc

Co-authored-by: Gary Russell <grussell@vmware.com>

* Update docs/src/main/asciidoc/kafka/kafka_partitions.adoc

Co-authored-by: Gary Russell <grussell@vmware.com>

* Update docs/src/main/asciidoc/kafka/kafka_partitions.adoc

Co-authored-by: Gary Russell <grussell@vmware.com>

---------

Co-authored-by: Gary Russell <grussell@vmware.com>
This commit is contained in:
Soby Chacko
2023-05-15 16:48:25 -04:00
committed by GitHub
parent 1b3fc7074b
commit 4d65ccd235
1 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/src/main/asciidoc/kafka/kafka_partitions.adoc
View File

@@ -54,7 +54,14 @@ spring:
partition-count: 12
----
IMPORTANT: The topic must be provisioned to have enough partitions to achieve the desired concurrency for all consumer groups.
IMPORTANT: It is important to keep in mind that, since Apache Kafka supports partitioning natively, there is no need to rely on binder partitioning as described above unless you are using custom partition keys as in the example or an expression that involves the payload itself.
The binder-provided partitioning selection is otherwise intended for middleware technologies that do not support native partitioning.
Note that we are using a custom key called `partitionKey` in the above example, that will be the determining factor for the partition, thus in this case it is appropriate to use binder partitioning.
When using native Kafka partitioning, i.e, when you do not provide the `partition-key-expression`, then Apache Kafka will select a partition, which by default will be the hash value of the record key over the available number of partitions.
To add a key to an outbound record, set the `KafkaHeaders.KEY` header to the desired key value in a spring-messaging `Message<?>`.
By default, when no record key is provided, Apache Kafka will choose a partition based on the logic described in the https://kafka.apache.org/documentation/#producerconfigs_partitioner.class[Apache Kafka Documentation].
NOTE: The topic must be provisioned to have enough partitions to achieve the desired concurrency for all consumer groups.
The above configuration supports up to 12 consumer instances (6 if their `concurrency` is 2, 4 if their concurrency is 3, and so on).
It is generally best to "`over-provision`" the partitions to allow for future increases in consumers or concurrency.