diff --git a/README.adoc b/README.adoc index 4f9b410d0..484a8f64a 100644 --- a/README.adoc +++ b/README.adoc @@ -337,6 +337,11 @@ Usually needed if you want to synchronize another transaction with the Kafka tra To achieve exactly once consumption and production of records, the consumer and producer bindings must all be configured with the same transaction manager. + Default: none. +txCommitRecovered:: +When using a transactional binder, the offset of a recovered record (e.g. when retries are exhausted and the record is sent to a dead letter topic) will be committed via a new transaction, by default. +Setting this property to `false` suppresses committing the offset of recovered record. ++ +Default: true. [[reset-offsets]] ==== Resetting Offsets @@ -363,7 +368,7 @@ Set `resetOffsets` to `true` and `startOffset` to `latest`; the binding will per IMPORTANT: If a rebalance occurs after the initial assignment, the seeks will only be performed on any newly assigned partitions that were not assigned during the initial assignment. -For more control over topic offsets, see <>; when a listener is provided, `resetOffsets: true` is ignored. +For more control over topic offsets, see <>; when a listener is provided, `resetOffsets` should not be set to `true`, otherwise, that will cause an error. ==== Consuming Batches @@ -622,6 +627,47 @@ Usually, applications may use principals that do not have administrative rights Consequently, relying on Spring Cloud Stream to create/modify topics may fail. In secure environments, we strongly recommend creating topics and managing ACLs administratively by using Kafka tooling. +====== Multi-binder configuration and JAAS + +When connecting to multiple clusters in which each one requires separate JAAS configuration, then set the JAAS configuration using the property `sasl.jaas.config`. +When this property is present in the applicaiton, it takes precedence over the other strategies mentioned above. +See this https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients[KIP-85] for more details. + +For example, if you have two clusters in your application with separate JAAS configuration, then the following is a template that you can use: + +``` +spring.cloud.stream: + binders: + kafka1: + type: kafka + environment: + spring: + cloud: + stream: + kafka: + binder: + brokers: localhost:9092 + configuration.sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"admin-secret\";" + kafka2: + type: kafka + environment: + spring: + cloud: + stream: + kafka: + binder: + brokers: localhost:9093 + configuration.sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user1\" password=\"user1-secret\";" + kafka.binder: + configuration: + security.protocol: SASL_PLAINTEXT + sasl.mechanism: PLAIN +``` + +Note that both the Kafka clusters, and the `sasl.jaas.config` values for each of them are different in the above configuration. + +See this https://github.com/spring-cloud/spring-cloud-stream-samples/tree/main/multi-binder-samples/kafka-multi-binder-jaas[sample application] for more details on how to setup and run such an application. + [[pause-resume]] ===== Example: Pausing and Resuming the Consumer diff --git a/docs/pom.xml b/docs/pom.xml index 1778c636d..2e7e68d50 100644 --- a/docs/pom.xml +++ b/docs/pom.xml @@ -7,7 +7,7 @@ org.springframework.cloud spring-cloud-stream-binder-kafka-parent - 3.2.0-SNAPSHOT + 3.2.0-M1 jar spring-cloud-stream-binder-kafka-docs diff --git a/docs/src/main/asciidoc/_configprops.adoc b/docs/src/main/asciidoc/_configprops.adoc index 7ff5a9f02..9cf8dad6a 100644 --- a/docs/src/main/asciidoc/_configprops.adoc +++ b/docs/src/main/asciidoc/_configprops.adoc @@ -9,7 +9,6 @@ |spring.cloud.stream.dynamic-destinations | `[]` | A list of destinations that can be bound dynamically. If set, only listed destinations can be bound. |spring.cloud.stream.function.batch-mode | `false` | |spring.cloud.stream.function.bindings | | -|spring.cloud.stream.function.definition | | Definition of functions to bind. If several functions need to be composed into one, use pipes (e.g., 'fooFunc\|barFunc') |spring.cloud.stream.instance-count | `1` | The number of deployed instances of an application. Default: 1. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-count" where 'foo' is the name of the binding. |spring.cloud.stream.instance-index | `0` | The instance id of the application: a number from 0 to instanceCount-1. Used for partitioning and with Kafka. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-index" where 'foo' is the name of the binding. |spring.cloud.stream.instance-index-list | | A list of instance id's from 0 to instanceCount-1. Used for partitioning and with Kafka. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-index-list" where 'foo' is the name of the binding. This setting will override the one set in 'spring.cloud.stream.instance-index' diff --git a/pom.xml b/pom.xml index bb756df80..797f5e965 100644 --- a/pom.xml +++ b/pom.xml @@ -2,12 +2,12 @@ 4.0.0 spring-cloud-stream-binder-kafka-parent - 3.2.0-SNAPSHOT + 3.2.0-M1 pom org.springframework.cloud spring-cloud-build - 3.1.0-SNAPSHOT + 3.1.0-M1 @@ -15,8 +15,8 @@ 2.8.0-M1 5.5.2 2.8.0 - 1.2.0-SNAPSHOT - 3.2.0-SNAPSHOT + 1.2.0-M1 + 3.2.0-M1 true true true diff --git a/spring-cloud-starter-stream-kafka/pom.xml b/spring-cloud-starter-stream-kafka/pom.xml index 42f3042b9..d3c7bb904 100644 --- a/spring-cloud-starter-stream-kafka/pom.xml +++ b/spring-cloud-starter-stream-kafka/pom.xml @@ -4,7 +4,7 @@ org.springframework.cloud spring-cloud-stream-binder-kafka-parent - 3.2.0-SNAPSHOT + 3.2.0-M1 spring-cloud-starter-stream-kafka Spring Cloud Starter Stream Kafka diff --git a/spring-cloud-stream-binder-kafka-core/pom.xml b/spring-cloud-stream-binder-kafka-core/pom.xml index d550a1495..73921a0e1 100644 --- a/spring-cloud-stream-binder-kafka-core/pom.xml +++ b/spring-cloud-stream-binder-kafka-core/pom.xml @@ -5,7 +5,7 @@ org.springframework.cloud spring-cloud-stream-binder-kafka-parent - 3.2.0-SNAPSHOT + 3.2.0-M1 spring-cloud-stream-binder-kafka-core Spring Cloud Stream Kafka Binder Core diff --git a/spring-cloud-stream-binder-kafka-streams/pom.xml b/spring-cloud-stream-binder-kafka-streams/pom.xml index a75033024..e80b2d29c 100644 --- a/spring-cloud-stream-binder-kafka-streams/pom.xml +++ b/spring-cloud-stream-binder-kafka-streams/pom.xml @@ -10,7 +10,7 @@ org.springframework.cloud spring-cloud-stream-binder-kafka-parent - 3.2.0-SNAPSHOT + 3.2.0-M1 diff --git a/spring-cloud-stream-binder-kafka/pom.xml b/spring-cloud-stream-binder-kafka/pom.xml index df027c2af..8a8eb8978 100644 --- a/spring-cloud-stream-binder-kafka/pom.xml +++ b/spring-cloud-stream-binder-kafka/pom.xml @@ -10,7 +10,7 @@ org.springframework.cloud spring-cloud-stream-binder-kafka-parent - 3.2.0-SNAPSHOT + 3.2.0-M1