Commit Graph

238 Commits

Author SHA1 Message Date
Oleg Zhurakousky
def2c3d0ed SCST-GH-1259 Polishing
- added '@DeprecatedConfigurationProperty'
- minor doc polishing

Resolves #335
2018-03-07 16:04:44 -05:00
Gary Russell
10a44d1e44 SCST-GH-1259: Kafka Binder Doc Polishing
Fixes https://github.com/spring-cloud/spring-cloud-stream/issues/1259

Also deprecate properties that are no longer used.

Missed a save.
2018-03-07 16:03:36 -05:00
Oleg Zhurakousky
0de078ca48 GH-326 added KafkaAutoConfiguration to KafkaBinderConfiguration
- added KafkaAutoConfiguration to the @Import of KafkaBinderConfiguration
- removed 'optional' flag for KafkaProperties from KafkaBinderConfigurationProperties
- fixed KafkaBinderAutoConfigurationPropertiesTest

Resolves #326
Resolves #333
2018-03-06 13:30:41 -05:00
Gary Russell
8035e25359 GH-67: Workaround for SK GH-599
Fixes #67

Spring Kafka currently doesn't support `TPIO.SeekPosition` for initial offsets.
Instead, use 0 and `Long.MAX_VALUE` for `BEGINNING` and `END` respectively.

Resolves #331
2018-03-06 13:16:53 -05:00
Artem Bilan
bcf15ed3be GH-328: Make KafkaBinderMetrics bean conditional
Fixes: spring-cloud/spring-cloud-stream-binder-kafka#328

Since we consider a Micrometer dependency as an optional, it would be
better do not expose beans which depends of that library

* Move `KafkaBinderMetrics` to its own `@Configuration` class with
appropriate conditions on the classpath and beans presence
* Add an `ApplicationContextRunner`-based test-case to achieve a
condition when Micrometer is not in classpath via `FilteredClassLoader`
hook

Resolves #328
Resolves #332
2018-03-06 12:40:16 -05:00
Gary Russell
d37ef750ad GH-67: Reinstate resetOffsets property
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/67

Currently only supported with group management (`autoRebalanceEnabled` - default true).

See https://github.com/spring-projects/spring-kafka/issues/599

Resolves #67
Resolves #329
2018-03-05 20:50:17 -05:00
Oleg Zhurakousky
d8baca5a66 Polishing
Resolves #325
2018-03-05 20:03:24 -05:00
Jon Schneider
b9d7f1f537 Change KafkaBinderMetrics to a tagged TimeGauge 2018-03-05 19:56:12 -05:00
Oleg Zhurakousky
ad819ece92 GH-322 added @Conditional for 'KafkaBinderHealthIndicator' bean 2018-03-02 09:08:39 -05:00
Soby Chacko
e254968eaf Next version: 2.0.0.BUILD-SNAPSHOT 2018-03-01 09:30:11 -05:00
Soby Chacko
25a64e1d85 2.0.0.RC2 Release 2018-03-01 09:15:56 -05:00
Oleg Zhurakousky
353c89ab63 GH-250 Reworked binding event propagation
- Hooked up to the new `BindingCreatedEvent`

Simple polishing

Fixes spring-cloud/spring-cloud-stream-binder-kafka#250
2018-02-28 15:47:47 -05:00
Artem Bilan
227d8f39f6 GH-318: Fix KafkaBinderMetrics for Micrometer
Fixes spring-cloud/spring-cloud-stream-binder-kafka#318

* Use `ToDoubleFunction`-based `MeterRegistry.gauge()` variant to really
calculate a `lag` at runtime instead of statically defined before
* Add `KafkaBinderActuatorTests` integration test to demonstrate how
`Binder` is declared in the separate child context and how
`KafkaBinderMetrics` is not visible from the parent context.
This test also verify the real `gauge` value for the `consumer lag`
and should be used in the future to verify the `KafkaBinderMetrics`
exposure removing the code after TODO in the `KafkaMetricsTestConfig`
2018-02-27 18:51:24 -05:00
Oleg Zhurakousky
555d3ccbd8 GH-319 bumped up SIK and SK versions
Resolves #319
2018-02-27 11:30:36 -05:00
Oleg Zhurakousky
d7c5c9c13b GH-1211 made Actuator optional 2018-02-26 23:03:39 -05:00
Sabby Anandan
e23af0ccc1 Revise Kafka Streams docs (#317)
* Revise Kafka Streams docs

* Remove KStream starter-pom reference

* Remove KStreams from aggregate docs
2018-02-26 19:14:54 -05:00
Soby Chacko
3fd93e7de8 Kafka Streams binder autoconfiguration changes
Make KafkaStreamsBinderSupportAutoConfiguration conditional
on BindingSerive being present in the BeanFactory.
2018-02-23 16:09:58 -05:00
Soby Chacko
90a778da6a Next version: 2.0.0.BUILD-SNAPSHOT 2018-02-23 12:53:17 -05:00
Soby Chacko
99ff426b92 2.0.0.RC1 Release 2018-02-23 12:26:44 -05:00
Rafal Zukowski
0a59b4c628 requestedAcks propoerty changed to string to allow setting it to "all"
polishing
2018-02-22 17:26:42 -05:00
Artem Bilan
a654382e99 Upgrade to SK-2.1.3 and SIK-3.0.2 2018-02-22 16:40:56 -05:00
Soby Chacko
9a4e86a750 KafkaBinderConfigurationProperties duplicate bean
ConfigurationProperties bean provided by Kafka Streams binder
extends from `KafkaBinderConfigurationProperties` used by Kafka binder.
It creates a conflict when autowiring this bean from Kafka binder configuration.
This prevents an application to have both binders in the classpath.
Change the creation of this ConfigurationProperties bean so that it
avoids creating bean using EnbaleConfigurationProperties and then autowiring,
but directly create the Bean using `@Bean`. This prevents the conflict.

Resolves #244
Resolves #315
2018-02-22 08:54:00 -05:00
Nikem
addb40bab5 Added spring-boot-configuration-processor to generate @ConfigurationProperties metadata
Resolves #310
2018-02-16 08:46:45 -05:00
Soby Chacko
cdffcd844c Kafka Streams binder docs updates
Refactoring kafka streams docs into a separate module

Resolves #293
2018-02-16 08:35:52 -05:00
Soby Chacko
a5344655cb Kafka Streams binder name changes
- Rename spring-cloud-stream-binder-kstream to spring-cloud-stream-binder-kafka-streams
 - Corresponding changes in maven pom.xml files
 - Rename relevant classes to prefix with KafkaStreams instead of KStream
 - Corresponding package changes from org.springframework.cloud.stream.kstream to
   org.springframework.cloud.stream.kafka.streams
 - Organize all the configuration property classes in a properties package
 - Remove kstream from all the properties exposed by Apache Kafka Streams binder
 - Classes that need not be public are now moved to package access level
 - Test changes
 - More javadocs to classes

Resolves #246
2018-02-14 14:47:34 -05:00
Gary Russell
72e2aeec2a GH-305: Fix producer initiated transactions
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/305
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/308

Requires https://github.com/spring-projects/spring-integration-kafka/pull/196

`ProducerConfigurationMessageHandler` overrode `handleMessageInternal` to support
producer-initiated transactions. Now that the superclass extends `ARPMH`, this
method is no longer overridable.

Spring Integration Kafka's `KafkaProducerMessageHandler` now detects a transactional
`KafkaTemplate` and will automatically start a transaction if needed, so there is no
longer any need to override that method.

No additional tests needed; see `KafkaTransactionTests`.
2018-02-14 10:17:35 -05:00
Soby Chacko
090125fa71 Multiple Input bindings on same StreamListener method
- In kafka streams applications, it is essential to have multiple bindings for
    various target types such as KStream, KTable etc. These changes allow to have
    more than one type of target type bindings on a single StreamListenerMethod.
  - Currently support KStream and KTable target types
  - Refactoring the KStream listener orchestrator strategy
  - Input bindings are initiated through a proxy and later on wrapped with the
    real target created from a common Kafka Streams StreamsBuilder
  - Adding tests to verify multiple input bindings for KStream and KTable

  Resolves #298
  Resolves #303
Polishing

Materializing KTables as state stores
Void return types on kafka streams StreamListener methods
Polishing
2018-02-10 12:16:23 -05:00
Oleg Zhurakousky
0fcb972cb6 GH-301 polished POM
Resolves #302
2018-02-10 12:05:28 -05:00
Gary Russell
dbe19776f5 GH-301: Fix Binder Kafka Properties Overrides
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/301

For the `AdminClient`, arbitrary Kafka properties (set via the binder `configuration` property) should
supersede any boot properties. There was already special handling for the bootstrap servers, but
other arbitrary properties were ignored.

Add tests, including a test to verify the proper override of boot's broker list if appropriate.
2018-02-08 16:54:32 -05:00
Soby Chacko
f44923e480 Introducing error handling for Kafka Streams binder
- Add support for KIP-161: streams deserialization exception handlers
 - Provide out of the box LogAndContinue and LogAndFail exception handlers
 - Introduce a new exception handler that sends records in error on
   deserialization to a DLQ
 - Ensure that the exception handlers work in both when native decoding
   is enabled (Kafka is doing deserialization) and when native decoding
   is disabled (Binder is doing deserialization)
 - Enhancements on the logic of native decoding in binder.
 - General refactoring and cleanup
 - Adding more tests

 Resolves #275, #281

Introducing multiple StreamListener methods in Kafka Streams apps.

 - Multiple stream listener methods now have individual backing
   StreamBuilder and configuration.
 - Register StreamsBuilderFactoryBean programmatically

 Resolves #299, #300
2018-02-07 05:29:47 -06:00
Soby Chacko
b31f902905 Back to 2.0.0.BUILD-SNAPSHOT 2018-02-05 13:56:31 -05:00
Soby Chacko
13cc4e0155 Update for 2.0.0.M4 Release 2018-02-05 13:41:45 -05:00
Soby Chacko
5e0688df72 Doc structure changes
Introduce a top level doc that aggregates various sub docs
2018-02-05 12:59:18 -05:00
Oleg Zhurakousky
abd43cfffa fixed KafkaBinderTest to accomodate DefaultPollableMessageSource's signature change 2018-02-04 13:30:41 -05:00
Oleg Zhurakousky
8057b1f764 GH-295 Fixed tests to comply with double-conversion related changes in core
See https://github.com/spring-cloud/spring-cloud-stream/issues/1130
See https://github.com/spring-cloud/spring-cloud-stream/issues/1071

Resolves #295
2018-02-03 08:06:37 -05:00
Soby Chacko
78ae3d1867 Update dependencies
Spring Kafka to 2.1.2.RELEASE
Spring Integration Kafka to 3.0.1.RELEASE
Test compile fixes from micrometer class changes in KafkaMetrics tests
2018-02-02 13:32:02 -05:00
Soby Chacko
a7378c9132 Addressing PR review comments
Resolves #274
2018-01-24 10:51:37 -05:00
Soby Chacko
9500ccfe46 KStream binder - nativeEncoding and Serde changes 2018-01-24 10:07:31 -05:00
Soby Chacko
3a5aed61c9 Redesigning the branching support in KStream binder
Instead of relying on a property based approach, use proper
multiple output bindings to support the branching feature.
This is accomplished through a combination of using `SendTo`
annotation with multiple outputs and overriding the default
StreamListener method setup orchestration in the binder.
2018-01-24 10:07:31 -05:00
Soby Chacko
86146bfe81 Support for KStream branch and split
Resolves #265

- Ensure that branching can be done and any message conversions
  done on all outbound data to multiple topics
- Refine MessageConversion logic in KStream binder
- Refactor message conversion code into a specific class
- Cleanup and refactoring
- Test to verify branching support
- More refinements in the way nativeEncoding/nativeDecoding logic works in KStream binder
- Doc changes for KStream binder
2018-01-24 10:07:31 -05:00
Gary Russell
f2abb59b19 Fix Polling Consumer Tests
- Add a sleep to the while loop
- Unbind at the end of the test

Resolves #289
2018-01-24 10:01:02 -05:00
Gary Russell
36c39974ad Fix default partition header expression 2018-01-22 13:35:24 -05:00
Gary Russell
16a195a887 GH-277: Add Polled Consumer
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/277

Resolves #282
2018-01-18 12:59:12 -05:00
Oleg Zhurakousky
bc562e3a77 Polished KafkaBinderTests
polished KafkaBinderTests to account for changes in core related to partitioning - 00748985d6

Resolves #272
2018-01-17 15:52:23 -05:00
Gary Russell
39dd68a8ad SCST-GH-1166: Support Producer-initiated tx
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/1166

Previously, transactions were only supported if initiated by a consumer.

Also fix races in `KafkaBinderTests.testResume()` (messages sent before subscribing).
2018-01-17 14:27:32 -05:00
Oleg Zhurakousky
8daaed43a9 upgraded to spring-kafka 2.1.1.BUILD-SNAPSHOT 2018-01-12 09:17:29 -05:00
Oleg Zhurakousky
d13d92131c GH-279 added Log4J-1 support 2018-01-09 15:46:26 -05:00
Oleg Zhurakousky
0b4ccdefce Added test dependency on log4j:log4j since it is still required by Kafka
Resolves #262
2018-01-03 19:22:18 -05:00
Gary Russell
ae269e729b GH-261: Support idle container events
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/261

This facilitates pause/resume.
2018-01-03 19:22:18 -05:00
Soby Chacko
9182adcf56 KStream binder outbound keys changes
The serializer can fall back to the default common one if there is not
a more specific one provided.

Resolves #271
2018-01-03 14:42:18 -05:00