Commit Graph

313 Commits

Author SHA1 Message Date
Soby Chacko
e2fc45c4ce Next update version: 2.1.0.BUILD-SNAPSHOT 2018-09-21 11:37:51 -04:00
Soby Chacko
74a044b0c0 2.1.0.M3 Release 2018-09-21 11:00:17 -04:00
Soby Chacko
70f385553a Application ID resolution for Kafka streams binder (#450)
* Application ID resolution for kafka streams binder

Avoid the need to rely on group property for application id.
First, check binding specific application id, if not look at defaults.
If nothing works, fall back to the default application id set by the
boot auto configuration.

Modify tests.
Update docs.

Resolves #448

* Addressing PR review comments

* Minor polishing

* Addressing PR review comments
2018-09-21 10:17:34 -04:00
Soby Chacko
f5739a9c7f Missing beans registration in Kafka Streams binder
There are duplicate code in various binder configurations where we register missing beans.
Consolidate them into a common class that implements ImportBeanDefinitionRegistrar and then
import this class in the binder configurations.

Resolves #445
2018-09-20 11:28:32 -04:00
Soby Chacko
39f5490488 Refactor bootstrap server configuration
There is a common piece of code repeated in both producer and consumer
configuration where it is populating the bootstrap server configuration.
Refactoring into a common method.

Resolves #208
2018-09-19 16:15:19 -04:00
Soby Chacko
d445a2bff9 Docs for GlobalKTable binding
Resolves #443
2018-09-19 15:05:48 -04:00
Oleg Zhurakousky
5446485cbf Updating with changes related to core GH-1484
Resolves #447
2018-09-19 10:45:04 +02:00
Soby Chacko
cc61d916b8 Extended default properties
Allow applications to configure default values for extended prducer and
consumer properties across multiple bindings in order to avoid repetition.

Address the changes in both Kafka and Kafka Streams binders.

Add integration test to verify both binding specific and default extended properties.

Resolves #444
Requires https://github.com/spring-cloud/spring-cloud-stream/pull/1477
2018-09-17 20:09:44 -04:00
Soby Chacko
41d9136812 GH-384: Binding support for GlobalKTable
Resolves spring-cloud/spring-cloud-stream-binder-kafka#384

* New binding targets and binder implementation for GlobalKTable within kafka-streams binder
* Refactoring existing structure to accommodate the new binder
* Adding integration test to verify the GlobalKTable behavior

Resolves #384

* Addressing PR review comments

* Update spring-kafka to 2.2.0.M3
Addressing PR review comments
Polishing

* Addressing PR review comments

* Addressing PR review comments
2018-09-14 10:51:15 -04:00
Soby Chacko
71a6a6cf28 Package structure changes for StreamBuilderFactoryBean 2018-09-12 12:05:52 -04:00
Gary Russell
36361d19bf GH-1459: Pollable Consumer and Requeue
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/1459

Requires https://github.com/spring-cloud/spring-cloud-stream/pull/1467
2018-09-10 10:22:04 -04:00
Soby Chacko
e869516e3d Handling tombstones in kafka streams binder
Handle tombstones gracefully in the kafka streams binder
Modify tests to verify

Resolves spring-cloud/spring-cloud-stream-binder-kafka#294

* Addressing PR review comments

* Addressing PR review comments
2018-09-05 12:17:54 -04:00
Gary Russell
ea288c62eb GH-435: useNativeEncoding and Transactions
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/435

All common properties should be settable on the transactional producer.
2018-09-04 15:56:26 -04:00
Soby Chacko
afa337f718 Remove deprecations in tests
Resolves #433
2018-09-04 15:28:36 -04:00
Soby Chacko
e4b08e888e Next update version: 2.1.0.BUILD-SNAPSHOT 2018-08-28 10:15:08 -04:00
Soby Chacko
de4b5a9443 2.1.0.M2 Release 2018-08-28 10:04:25 -04:00
Soby Chacko
a090614709 kafka Streams binder configuration fix
(to address core changes made in regards to Boot 2.1)

Ensure that KafkaStreamsBinderSupportAutoConfiguration runs
after BindingServiceConfiguration from core.
2018-08-14 10:33:51 -04:00
Gary Russell
a3f7ca9756 Fix mock to use poll(Duration) 2018-08-07 14:45:33 -04:00
Soby Chacko
82b07a0120 Fixing checkstyle issues 2018-08-07 14:23:47 -04:00
Soby Chacko
0d0cf8dcb7 Update to spring-cloud-build 2.1.0 snapshot
Spring Boot 2.1.0 snapshot
Spring Kafka 2.2.0/3.1.0 snapshots
Apache Kafka client 2.0.0
Fixing tests
Removing deprecations and removals
Polishing

Resolves #424
2018-08-07 14:02:52 -04:00
Soby Chacko
cbfe03be2f Kafka Streams binder test disabling
Temporarily disabling KafkaBinderBootstrapTest in Kafka Streams binder
due to builds taking much longer times due to this.
2018-08-03 08:11:52 -04:00
Soby Chacko
1f0c6cabc6 Revert "Kafka Streams binder test disabling"
This reverts commit 0c61cedc85.
2018-08-03 08:10:59 -04:00
Soby Chacko
0c61cedc85 Kafka Streams binder test disabling
Temporarily disabling KafkaBinderBootstrapTest in Kafka Streams binder
due to builds taking much longer times due to this.
2018-08-02 12:59:02 -04:00
Soby Chacko
44210f1b72 Update Kafka binder metrics docs
Fix the wrong metric name used in the Kafka binder metrics for consumer offset lag.
Update the description.

Resolves #422
2018-08-02 09:57:27 -04:00
Soby Chacko
7007f9494a JAAS initializer regression
Fix JAAS initializer with setting the missing properties.

Resoves #419

Polishing
2018-07-27 14:34:17 -04:00
Gary Russell
da268bb6dd GH-309: Use actual partition count
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/309

If more partitions exist than those configured, use the actual.

Resolves #416
2018-07-25 20:31:27 +02:00
Soby Chacko
e03baefbaf Kafka streams binder issues in custom environment
When kafka streams binder is used in the context of a custom environment, possibly
for a multi binder use case, it is unable to query the outer context as the normal
parent context is absent. This change will ensure that the binder context has access
to the outer context so that it can use any beans it needs from it in KStream or KTable
binder configuration.

Resolves #411
2018-07-25 13:14:47 -04:00
Gary Russell
01396b6573 GH-413: Configure Kafka Streams Cleanup Behavior
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/413
2018-07-24 16:53:57 -04:00
Soby Chacko
3f009a8267 Autoconfigure optimization
Add spring-boot-autoconfigure-processor to the kafka-streams binder for
auto configuration optimization.

Resolves #406
2018-07-24 15:56:48 -04:00
Gary Russell
31bb86b002 GH-404: Synchronize shared consumer
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/404

The fix for issue https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/231
added a shared consumer but the consumer is not thread safe. Add synchronization.

Also, a timeout was added to the `KafkaBinderHealthIndicator` but not to the
`KafkaBinderMetrics` which has a similar shared consumer; add a timeout there.
2018-07-11 13:30:24 -04:00
Soby Chacko
cc2dfd1d08 Upgrade kafka client to 1.1.0 (#405)
* Upgrade kafka client to 1.1.0

Upgrade kafka client to 1.1.0 for both kafka and kafka-streams binders

Resolves #370

* Address review comments
2018-07-09 17:37:58 -04:00
jmaxwell
fc768ba695 GH-402 Add additional data to DLQ message headers 2018-07-02 11:34:46 -04:00
Soby Chacko
38d6deb4d5 Provide programmatic access to KafkaStreams object
Providing access to the underlying StreamBuilderFactoryBean by making the bean name
deterministic. Eariler, the binder was using UUID to make the stream builder factory
bean names unique in the event of multiple StreamListeners. Switching to use the
method name instead to keep the StreamBuilder factory beans unique while providing
a deterministic way to giving it programmatic access.

Polishing docs

Fixes #396
2018-06-29 16:45:55 -04:00
Soby Chacko
2a0b9015de Next update version: 2.1.0.BUILD-SNAPSHOT 2018-06-27 14:11:40 -04:00
Soby Chacko
321919abc9 2.1.0.M1 2018-06-27 13:55:58 -04:00
Soby Chacko
b09def9ccc Polishing Kafka Streams binder docs
Resolves #390
2018-06-27 11:25:22 -04:00
Soby Chacko
54d7c333d3 Interactive query - polishing
Renaming InteractiveQueryServices to InteractiveQueryService
2018-06-27 11:03:46 -04:00
Soby Chacko
015b1a7fa1 Fix bad link in docs
Resolves #359
2018-06-27 10:14:17 -04:00
Soby Chacko
020821f471 Fix typo in kafka streams docs
Resolves #400
2018-06-27 09:08:21 -04:00
UltimaPhoenix
2e14ba99e3 Fix unit test
Remove unnecessary semicolon

Replace deprecated method with the new one

Test refactoring
2018-06-27 08:56:48 -04:00
Oleg Zhurakousky
bd5cc2f89d GH-398 added support for container customization
PLease see https://github.com/spring-cloud/spring-cloud-stream-binder-rabbit/issues/139 for more details

Resolves #398

Added test and polishing
2018-06-27 08:45:31 -04:00
Gary Russell
ca2983c881 GH-373: Support multiplexed consumers
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/373

When a consumer is multiplexed, configure the container to listen to multiple topics.
Also for the polled consumer.

When using a DLQ, determine the queue name from the topic in the failed record (unless
an explicit DLQ name has been provisioned - in which case, the same DLQ will be used
for all topics.

Resolves #383
2018-06-12 14:52:51 -04:00
Soby Chacko
1b00179f25 Kafka Streams interactive query enhancements
* When running interactive queries against multiple instances under the same application id,
  ensure that the application can retrieve the proper instance that is hosting the queried state store
* Introduce a new API level service called InteractiveQueryServices
* Perform refactoring to support this enhancement
* Deprecate QueryableStoreRegistry in 2.1.0 in favor of InteractiveQueryServices

Resolves #369
2018-06-08 11:22:31 -04:00
Artem Bilan
f533177d21 GH-381: Remove duplicated SCSt-binder-test dep
Fixes spring-cloud/spring-cloud-stream-binder-kafka#381
2018-05-14 15:25:36 -04:00
Gary Russell
990a5142ce GH-339: Support topic patterns
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/339

- support regex patterns for consumer destinations to consume from multiple topics
- `enableDlq` is not available in this mode

As well as the test case, tested with a boot app...

```
spring.cloud.stream.bindings.input.destination=kbgh339.*
spring.cloud.stream.kafka.bindings.input.consumer.destination-is-pattern=true
```

and

```
2018-04-30 15:12:49.718  : partitions assigned: [kbgh339a-0]
2018-04-30 15:17:46.585  : partitions revoked: [kbgh339a-0]
2018-04-30 15:17:46.655  : partitions assigned: [kbgh339a-0, kbgh339b-0]
```

after adding a new topic matching the pattern.

Doc polishing.
2018-05-10 11:10:49 -04:00
Soby Chacko
13693e8e66 Kafka Streams DLQ related changes
DLQ handling needs to be adjusted in kafka streams binder due to the multiplexing of input topics.
This commit changes it accordingly in KStream and KTable binders.
Add tests to verify.
2018-05-01 12:02:53 -04:00
Soby Chacko
70cd7dc2f9 Upgrade spring cloud stream version to 2.1.0 sanpshot
Always enable multiplex to true in kafka streams binder
2018-05-01 10:10:42 -04:00
Sarath Shyam
894597309b Fix kaka-streams binder to consume messages from multiple input topics #361
Modified KafkaStreamsStreamListenerSetupMethodOrchestrator#getkStream
so that the KStream object is built from list of topic names
2018-04-30 15:05:47 -04:00
Thomas Cheyney
74689bf007 Reuse Kafka consumer Metrics
Polishing
2018-04-30 14:19:53 -04:00
Gary Russell
d2012c287a GH-360: Improve Binder Producer/Consumer Config
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/360

`producer-configuration` and `consumer-configuration` improperly appear in content-assist.

These are methods used by the binders to get merged configuration data (boot and binder).

Rename the methods and add `producerProperties` and `consumerProperties` to allow
configuration.
2018-04-27 12:37:48 -04:00