SpringCloud/spring-cloud-stream
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Addressing Kafka Streams multiple functions issues

Browse Source 
Fixing an issue that causes a race condition when multiple functions
are present in a Kafka Streams application by isolating the responsible
proxy factory per function and not shared.

When multiple Kafka Streams functions are present in an application,
it should be possible to set the application id per function.

When an application provides a bean of type Serde, then the binder
should try to introspect that bean to see if it can be matched for
any inbound or outbound serialization.

Adding tests to verify the changes.

Adding docs.

Resolves #734, #735, #736
This commit is contained in:
Soby Chacko
2019-09-03 19:44:13 -04:00
parent ca8e086d4c
commit 309f588325
12 changed files with 519 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

88
docs/src/main/asciidoc/kafka-streams.adoc
View File

@@ -494,9 +494,9 @@ If you only have one single processor in the application, then you can set this
As a convenience, if you only have a single processor, you can also use `spring.application.name` as the property to delegate the application id.
If you have multiple Kafka Streams processors in the application, then you need to set the application id per processor.
You can set this on the input binding on each processor.
In the case of the functional model, you can attach it to each function as a property.
For e.g. imagine that you have to two following functions.
For e.g. imagine that you have the following functions.
```
@Bean
@@ -514,13 +514,32 @@ public java.util.function.Consumer<KStream<Object, String>> anotherProcess() {
}
```
Then you must set the application id for each, using the following binding properties.
Then you can set the application id for each, using the following binder level properties.
`spring.cloud.stream.kafka.streams.bindings.process_in.applicationId`
`spring.cloud.stream.kafka.streams.binder.process.applicationId`
and
`spring.cloud.stream.kafka.streams.bindings.anotherProcess_in.applicationId`
`spring.cloud.stream.kafka.streams.binder.anotherProcess.applicationId`
In the case of `StreamListener`, you need to set this on the first input binding on the processor.
For e.g. imagine that you have to two following `StreamListener` based processors.
```
@StreamListener
public KStream<String, String> process(@Input("input") <KStream<Object, String>> input) {
...
}
```
Then you must set the application id for this using the following binding property.
`spring.cloud.stream.kafka.streams.bindings.input.applicationId`
Fof function based model also, this approach of setting application id at the binding level will work.
However, setting per function at the binder level as we have seen above is much easier if you are using the functional model.
For production deployments, it is highly recommended to explicitly specify the application ID through configuration.
This is especially going to be very critical if you are auto scaling your application in which case you need to make sure that you are deploying each instance with the same application ID.
@@ -533,9 +552,9 @@ In the case of `StreamListener`, instead of using the function bean name, the ge
====== Summary of setting Application ID
* Auto generated by the binder per processor in the application. This can be overridden by setting at the binding level such as `spring.cloud.stream.kafka.streams.bindings.process_in.applicationId`.
When you have more than one processor, then you have to choose one of these options - either fall back to the defaults or override per input binding.
* If you have a single processor, then you can use `spring.kafka.streams.applicationId`, `spring.application.name` or `spring.cloud.stream.binder.kafka.streams.applicationId`.
* Auto generated by the binder per processor in the application. This can be overridden by setting at the binding level such as `spring.cloud.stream.kafka.streams.bindings.process_in.applicationId` (or binder level per function in the case of functional model).
When you have more than one processor, then you have to choose one of these options - either fall back to the defaults or override.
* If you have a single processor, then you can use `spring.kafka.streams.applicationId`, `spring.application.name` or `spring.cloud.stream.kafka.streams.binder.applicationId`.
==== Custom bindings in the functional style
@@ -596,13 +615,30 @@ Please note that this is a major change on default behavior from previous versio
Kafka Streams binder will try to infer matching Serde types by looking at the type signature of `java.util.function.Function|Consumer` or `StreamListener`.
Here is the order that it matches Serdes.
* First it looks at the types and see if they are one of the types exposed by Kafka Streams. If so, use them.
* If the application provides a bean of type `Serde` and if the return type is parameterized with the actual type of incoming key or value type, then it will use that `Serde` for inbound deserialization.
For e.g. if you have the following in the application, the binder detects that the incoming value type for the `KStream` matches with a type that is parameterized on a `Serde` bean.
It will use that for inbound deserialization.
```
@Bean
public Serde<Foo() customSerde{
...
}
@Bean
public Function<KStream<String, Foo>, KStream<String, Foo>> process() {
}
```
* Next, it looks at the types and see if they are one of the types exposed by Kafka Streams. If so, use them.
Here are the Serde types that the binder will try to match from Kafka Streams.
Integer, Long, Short, Double, Float, byte[] and String.
Integer, Long, Short, Double, Float, byte[], UUID and String.
* If none of the Serdes provided by Kafka Streams don't match the types, then it will use JsonSerde provided by Spring Kafka. In this case, the binder assumes that the types are JSON friendly.
This is useful if you have multiple value objects as inputs since the binder will internally infer them to correct json Serde objects. Otherwise, you have to configure Serde and target types on them individually.
Before falling back to the `JsonSerde` though, the binder checks at the default Serdes's set at the Kafka Streams level to see if it is a Serde that it can match with the incoming KStream's types.
If none of the above strategies worked, then the applications must provide the Serdes through configuration.
This can be configured in two ways - binding or default.
@@ -618,11 +654,11 @@ public BiFunction<KStream<CustomKey, AvroIn1>, KTable<CustomKey, AvroIn2>, KStre
then, you can provide a binding level Serde using the following:
```
spring.cloud.stream.kafka.streams.bindings.process_in_0.keySerde=CustomKeySerde
spring.cloud.stream.kafka.streams.bindings.process_in_0.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
spring.cloud.stream.kafka.streams.bindings.process_in_0.consumer.keySerde=CustomKeySerde
spring.cloud.stream.kafka.streams.bindings.process_in_0.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
spring.cloud.stream.kafka.streams.bindings.process_in_1.keySerde=CustomKeySerde
spring.cloud.stream.kafka.streams.bindings.process_in_1.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
spring.cloud.stream.kafka.streams.bindings.process_in_1.consumer.keySerde=CustomKeySerde
spring.cloud.stream.kafka.streams.bindings.process_in_1.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
```
If you want the default key/value Serdes to be used for inbound deserialization, you can do so at the binder level.
@@ -635,7 +671,7 @@ spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde
If you don't want the native decoding provided by Kafka, you can rely on the message conversion features that Spring Cloud Stream provides.
Since native decoding is the default, in order to let Spring Cloud Stream deserialze the inbound value object, you need to explicitly disable native decoding.
For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process_in_0.nativeDecoding: false`
For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process_in_0.consumer.nativeDecoding: false`
You need to disable native decoding for all the inputs individually. Otherwise, native decoding will still be applied for those you don't disable.
By default, Spring Cloud Stream will use `application/json` as the content type and use an appropriate json message converter.
@@ -650,34 +686,36 @@ Outbound serialization pretty much follows the same rules as above for inbound d
As with the inbound deserialization, one major change from the previous versions of Spring Cloud Stream is that the serialization on the outbound is handled by Kafka natively.
Before 3.0 versions of the binder, this was done by the framework itself.
Keys on the outbound are always serialized by Kafka using a matching Serde that is inferred by the binder.
Keys on the outbound are always serialized by Kafka using a matching `Serde` that is inferred by the binder.
If it can't infer the type of the key, then that needs to be specified using configuration.
Value serdes are inferred using the same rules used for inbound deserialization.
First it matches to see if the outbound type is of a Serde exposed by Kafka such as - Long, Short, Double, Float, byte[] and String.
If that doesnt't work, then fall back to JsonSerde provided by the Spring Kafka project.
First it matches to see if the outbound type is from a provided bean in the application.
If not, it checks to see if it matches with a `Serde` exposed by Kafka such as - Long, Short, Double, Float, byte[] and String.
If that doesnt't work, then fall back to JsonSerde provided by the Spring Kafka project, but first look at the default `Serde` configuration to see if there is a match.
Keep in mind that all these happen transparently to the application.
If none of these work, then the user has to provide the Serde to use by configuration.
If none of these work, then the user has to provide the `Serde` to use by configuration.
Lets say you are using the same `BiFunction` processor as above. Then you can configure outbound key/value Serdes as following.
```
spring.cloud.stream.kafka.streams.bindings.process_out.keySerde=CustomKeySerde
spring.cloud.stream.kafka.streams.bindings.process_out.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
spring.cloud.stream.kafka.streams.bindings.process_out.producer.keySerde=CustomKeySerde
spring.cloud.stream.kafka.streams.bindings.process_out.producer.valueSerde=io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
```
However, falling back to default Serdes for both input deserialization and output serialization is the last resort.
This may or may not work. Therefore, you need to ensure that you have a path forward for the application to correctly retrive the Serde.
If Serde inference fails, no binding level Serdes are provided, then the binder falls back to the default Serdes.
If Serde inference fails, and no binding level Serdes are provided, then the binder falls back to the default Serdes.
`spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde`
`spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde`
However, falling back to default Serdes for both input deserialization and output serialization is the last resort.
This may or may not work. Therefore, you need to ensure that you have a path forward for the application to correctly retrieve the Serde.
If your application uses the branching feature and has multiple output bindings, then these have to be configured per binding.
Once again, if the binder is capable of inferring the Serde types, you don't need to do this configuration.
If you don't want the native encoding provided by Kafka, but want to use the framework provided message conversion, then you need to explicitly disable native decoding since since native decoding is the default.
For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process_out.nativeEncoding: false`
For e.g. if you have the same BiFunction processor as above, then `spring.cloud.stream.bindings.process_out.producer.nativeEncoding: false`
You need to disable native encoding for all the output individually in the case of branching. Otherwise, native encoding will still be applied for those you don't disable.
By default, Spring Cloud Stream will use `application/json` as the content type and use an appropriate json message converter.