SpringCloud/spring-cloud-stream
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Dynamic partition improvements in Kafka binder

Browse Source 
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2706

  - Update partition count changes in Kafka dynamically during runtime
  - Checkstyle fixes
  - PartitionHandler changes
  - Improved handling of expression with 'payload' in the logic
  - Removed getter/setter for PartitionHandler and use ReflectionUtils within Test
  - Renamed property to 'dynamicPartitionUpdatesEnabled' and improved documentation (also one line per sentence)
  - Improved/renamed test to use embeddedkafka and really test update behaviour
  - Javadoc
This commit is contained in:
oliverfuehrer
2023-05-09 10:07:57 +02:00
committed by Soby Chacko
parent e19bdd4381
commit 29c3cd7cdd
4 changed files with 141 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/src/main/asciidoc/kafka/kafka_partitions.adoc
View File

@@ -65,6 +65,14 @@ You can override this default by using the `partitionSelectorExpression` or `par
Since partitions are natively handled by Kafka, no special configuration is needed on the consumer side.
Kafka allocates partitions across the instances.
NOTE: The partitionCount for a kafka topic may change during runtime (e.g. due to an adminstration task).
The calculated partitions will be different after that (e.g. new partitions will be used then).
Since 4.0.3 of Spring Cloud Stream runtime changes of partition count will be supported.
See also parameter 'spring.kafka.producer.properties.metadata.max.age.ms' to configure update interval.
Due to some limitations it is not possible to use a 'partition-key-expression' which references the 'payload' of a message, the mechanism will be disabled in that case.
The overall behavior is disabled by default and can be enabled using configuration parameter 'producer.dynamicPartitionUpdatesEnabled=true'.
The following Spring Boot application listens to a Kafka stream and prints (to the console) the partition ID to which each message goes:
[source,java]