From db84b14e32d6bcefda8e6ff17c8a17c46ad03c0a Mon Sep 17 00:00:00 2001 From: Gary Russell Date: Wed, 29 Aug 2018 10:30:46 -0400 Subject: [PATCH] Doc: Partition Assignment See: https://stackoverflow.com/questions/52077027/kafkalistener-concurrency-multiple-topics/52078148#52078148 **cherry-pick to 2.1.x, 1.3.x** (cherry picked from commit 9f828a8557834a999a3fc73069cda9c56ce29fc9) --- src/reference/asciidoc/kafka.adoc | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/reference/asciidoc/kafka.adoc b/src/reference/asciidoc/kafka.adoc index 71277af4..d5390824 100644 --- a/src/reference/asciidoc/kafka.adoc +++ b/src/reference/asciidoc/kafka.adoc @@ -635,7 +635,25 @@ public ConcurrentMessageListenerContainer(ConsumerFactory consumerFactory, It also has a property `concurrency`, e.g. `container.setConcurrency(3)` will create 3 `KafkaMessageListenerContainer` s. -For the first constructor, kafka will distribute the partitions across the consumers. +For the first constructor, kafka will distribute the partitions across the consumers using its group management capabilities. + +[IMPORTANT] +==== +When listening to multiple topics, the default partition distribution may not be what you expect. +For example, if you have 3 topics with 5 partitions each and you want to use `concurrency=15` you will only see 5 active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. +This is because the default Kafka `PartitionAssignor` is the `RangeAssignor` (see its javadocs). +For this scenario, you may want to consider using the `RoundRobinAssignor` instead, which will distribute the partitions across all of the consumers. +Then, each consumer will be assigned one topic/partition. +To change the `PartitionAssignor`, set the `partition.assignment.strategy` consumer property (`ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG`) in the properties provided to the `DefaultKafkaConsumerFactory`. + +When using Spring Boot: + +[source] +---- +spring.kafka.consumer.properties.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor +---- +==== + For the second constructor, the `ConcurrentMessageListenerContainer` distributes the `TopicPartition` s across the delegate `KafkaMessageListenerContainer` s.