Spring/spring-boot
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add Kafka Streams auto-configuration

Browse Source 
See gh-14021
This commit is contained in:
Gary Russell
2018-08-08 14:42:15 -04:00
committed by Stephane Nicoll
parent f9207dd946
commit a7acbbd625
6 changed files with 349 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
spring-boot-project/spring-boot-docs/src/main/asciidoc/appendix-application-properties.adoc
View File

@@ -1039,11 +1039,11 @@ content into your application. Rather, pick only the properties that you need.
spring.kafka.admin.ssl.trust-store-location= # Location of the trust store file.
spring.kafka.admin.ssl.trust-store-password= # Store password for the trust store file.
spring.kafka.admin.ssl.trust-store-type= # Type of the trust store.
spring.kafka.bootstrap-servers= # Comma-delimited list of host:port pairs to use for establishing the initial connection to the Kafka cluster.
spring.kafka.bootstrap-servers= # Comma-delimited list of host:port pairs to use for establishing the initial connection to the Kafka cluster. Applies to all components unless overridden.
spring.kafka.client-id= # ID to pass to the server when making requests. Used for server-side logging.
spring.kafka.consumer.auto-commit-interval= # Frequency with which the consumer offsets are auto-committed to Kafka if 'enable.auto.commit' is set to true.
spring.kafka.consumer.auto-offset-reset= # What to do when there is no initial offset in Kafka or if the current offset no longer exists on the server.
spring.kafka.consumer.bootstrap-servers= # Comma-delimited list of host:port pairs to use for establishing the initial connection to the Kafka cluster.
spring.kafka.consumer.bootstrap-servers= # Comma-delimited list of host:port pairs to use for establishing the initial connection to the Kafka cluster. Overrides the global property, for consumers.
spring.kafka.consumer.client-id= # ID to pass to the server when making requests. Used for server-side logging.
spring.kafka.consumer.enable-auto-commit= # Whether the consumer's offset is periodically committed in the background.
spring.kafka.consumer.fetch-max-wait= # Maximum amount of time the server blocks before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by "fetch.min.bytes".
@@ -1079,7 +1079,7 @@ content into your application. Rather, pick only the properties that you need.
spring.kafka.listener.type=single # Listener type.
spring.kafka.producer.acks= # Number of acknowledgments the producer requires the leader to have received before considering a request complete.
spring.kafka.producer.batch-size= # Default batch size in bytes.
spring.kafka.producer.bootstrap-servers= # Comma-delimited list of host:port pairs to use for establishing the initial connection to the Kafka cluster.
spring.kafka.producer.bootstrap-servers= # Comma-delimited list of host:port pairs to use for establishing the initial connection to the Kafka cluster. Overrides the global property, for producers.
spring.kafka.producer.buffer-memory= # Total bytes of memory the producer can use to buffer records waiting to be sent to the server.
spring.kafka.producer.client-id= # ID to pass to the server when making requests. Used for server-side logging.
spring.kafka.producer.compression-type= # Compression type for all data generated by the producer.
@@ -1105,6 +1105,21 @@ content into your application. Rather, pick only the properties that you need.
spring.kafka.ssl.trust-store-location= # Location of the trust store file.
spring.kafka.ssl.trust-store-password= # Store password for the trust store file.
spring.kafka.ssl.trust-store-type= # Type of the trust store.
spring.kafka.streams.auto-startup= # Whether or not to auto-start the streams factory bean.
spring.kafka.streams.bootstrap-servers= # Comma-delimited list of host:port pairs to use for establishing the initial connection to the Kafka cluster. Overrides the global property, for streams.
spring.kafka.streams.cache-max-bytes-buffering= # Maximum number of memory bytes to be used for buffering across all threads.
spring.kafka.streams.client-id= # ID to pass to the server when making requests. Used for server-side logging.
spring.kafka.streams.properties.*= # Additional Kafka properties used to configure the streams.
spring.kafka.streams.replication-factor= # The replication factor for change log topics and repartition topics created by the stream processing application.
spring.kafka.streams.ssl.key-password= # Password of the private key in the key store file.
spring.kafka.streams.ssl.key-store-location= # Location of the key store file.
spring.kafka.streams.ssl.key-store-password= # Store password for the key store file.
spring.kafka.streams.ssl.key-store-type= # Type of the key store.
spring.kafka.streams.ssl.protocol= # SSL protocol to use.
spring.kafka.streams.ssl.trust-store-location= # Location of the trust store file.
spring.kafka.streams.ssl.trust-store-password= # Store password for the trust store file.
spring.kafka.streams.ssl.trust-store-type= # Type of the trust store.
spring.kafka.streams.state-dir= # Directory location for the state store.
spring.kafka.template.default-topic= # Default topic to which messages are sent.
# RABBIT ({sc-spring-boot-autoconfigure}/amqp/RabbitProperties.{sc-ext}[RabbitProperties])

49
spring-boot-project/spring-boot-docs/src/main/asciidoc/spring-boot-features.adoc
View File

@@ -5634,6 +5634,44 @@ The following component creates a listener endpoint on the `someTopic` topic:
}
----
[[boot-deatures-kafka-streams]]
==== Kafka Streams
Spring for Apache Kafka provides a factory bean to create a `StreamsBuilder` object and
manage the lifecycle of its streams; the factory bean is created when
`@EnableKafkaStreams` is present on a `@Configuration` class.
The factory bean requires a `KafkaStreamsConfiguration` object for streams configuration.
If Spring Boot detects the `kafka-streams` jar on the classpath, it will auto-configure
the `KafkaStreamsConfiguration` bean from the `KafkaProperties` object as well as enabling
the creation of the factory bean by spring-kafka.
There are two required Kafka properties for streams (`bootstrap.servers` and
`application.id`); by default, the `application.id` is set to the `spring.application.name`
property, if present.
The `bootstrap.servers` can be set globally or specifically overridden just for streams.
Several other properties are specifically available as boot properties; other arbitrary
Kafka properties can be set using the `spring.kafka.streams.properties` property.
See <<boot-features-kafka-extra-props>> for more information.
To use the factory bean, simply wire its `StreamsBuilder` into your `@Bean` s.
====
[source, java]
----
@Bean
public KStream<Integer, String> kStream(StreamsBuilder streamsBuilder) {
KStream<Integer, String> stream = streamsBuilder.stream("ks1In");
stream.map((k, v) -> new KeyValue(k, v.toUpperCase()))
.to("ks1Out", Produced.with(Serdes.Integer(), new JsonSerde<>()));
return stream;
}
----
====
By default, the factory bean `autoStartup` property is false; to automatically start the
streams managed by the `StreamsBuilder` object it creates, set property
`spting.kafka.streams.auto-startup=true`.
[[boot-features-kafka-extra-props]]
@@ -5643,13 +5681,14 @@ The properties supported by auto configuration are shown in
(hyphenated or camelCase) map directly to the Apache Kafka dotted properties. Refer to the
Apache Kafka documentation for details.
The first few of these properties apply to both producers and consumers but can be
specified at the producer or consumer level if you wish to use different values for each.
The first few of these properties apply to all components (producers, consumers, admins,
and streams) but can be
specified at the component level if you wish to use different values.
Apache Kafka designates properties with an importance of HIGH, MEDIUM, or LOW. Spring Boot
auto-configuration supports all HIGH importance properties, some selected MEDIUM and LOW
properties, and any properties that do not have a default value.
Only a subset of the properties supported by Kafka are available through the
Only a subset of the properties supported by Kafka are available directly through the
`KafkaProperties` class. If you wish to configure the producer or consumer with additional
properties that are not directly supported, use the following properties:
@@ -5659,11 +5698,13 @@ properties that are not directly supported, use the following properties:
spring.kafka.admin.properties.prop.two=second
spring.kafka.consumer.properties.prop.three=third
spring.kafka.producer.properties.prop.four=fourth
spring.kafka.streams.properties.prop.five=fifth
----
This sets the common `prop.one` Kafka property to `first` (applies to producers,
consumers and admins), the `prop.two` admin property to `second`, the `prop.three`
consumer property to `third` and the `prop.four` producer property to `fourth`.
consumer property to `third`, the `prop.four` producer property to `fourth` and the
`prop.five` streams property to `fifth`.
You can also configure the Spring Kafka `JsonDeserializer` as follows: