Commit Graph

287 Commits

Author SHA1 Message Date
Soby Chacko
e03baefbaf Kafka streams binder issues in custom environment
When kafka streams binder is used in the context of a custom environment, possibly
for a multi binder use case, it is unable to query the outer context as the normal
parent context is absent. This change will ensure that the binder context has access
to the outer context so that it can use any beans it needs from it in KStream or KTable
binder configuration.

Resolves #411
2018-07-25 13:14:47 -04:00
Gary Russell
01396b6573 GH-413: Configure Kafka Streams Cleanup Behavior
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/413
2018-07-24 16:53:57 -04:00
Soby Chacko
3f009a8267 Autoconfigure optimization
Add spring-boot-autoconfigure-processor to the kafka-streams binder for
auto configuration optimization.

Resolves #406
2018-07-24 15:56:48 -04:00
Gary Russell
31bb86b002 GH-404: Synchronize shared consumer
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/404

The fix for issue https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/231
added a shared consumer but the consumer is not thread safe. Add synchronization.

Also, a timeout was added to the `KafkaBinderHealthIndicator` but not to the
`KafkaBinderMetrics` which has a similar shared consumer; add a timeout there.
2018-07-11 13:30:24 -04:00
Soby Chacko
cc2dfd1d08 Upgrade kafka client to 1.1.0 (#405)
* Upgrade kafka client to 1.1.0

Upgrade kafka client to 1.1.0 for both kafka and kafka-streams binders

Resolves #370

* Address review comments
2018-07-09 17:37:58 -04:00
jmaxwell
fc768ba695 GH-402 Add additional data to DLQ message headers 2018-07-02 11:34:46 -04:00
Soby Chacko
38d6deb4d5 Provide programmatic access to KafkaStreams object
Providing access to the underlying StreamBuilderFactoryBean by making the bean name
deterministic. Eariler, the binder was using UUID to make the stream builder factory
bean names unique in the event of multiple StreamListeners. Switching to use the
method name instead to keep the StreamBuilder factory beans unique while providing
a deterministic way to giving it programmatic access.

Polishing docs

Fixes #396
2018-06-29 16:45:55 -04:00
Soby Chacko
2a0b9015de Next update version: 2.1.0.BUILD-SNAPSHOT 2018-06-27 14:11:40 -04:00
Soby Chacko
321919abc9 2.1.0.M1 2018-06-27 13:55:58 -04:00
Soby Chacko
b09def9ccc Polishing Kafka Streams binder docs
Resolves #390
2018-06-27 11:25:22 -04:00
Soby Chacko
54d7c333d3 Interactive query - polishing
Renaming InteractiveQueryServices to InteractiveQueryService
2018-06-27 11:03:46 -04:00
Soby Chacko
015b1a7fa1 Fix bad link in docs
Resolves #359
2018-06-27 10:14:17 -04:00
Soby Chacko
020821f471 Fix typo in kafka streams docs
Resolves #400
2018-06-27 09:08:21 -04:00
UltimaPhoenix
2e14ba99e3 Fix unit test
Remove unnecessary semicolon

Replace deprecated method with the new one

Test refactoring
2018-06-27 08:56:48 -04:00
Oleg Zhurakousky
bd5cc2f89d GH-398 added support for container customization
PLease see https://github.com/spring-cloud/spring-cloud-stream-binder-rabbit/issues/139 for more details

Resolves #398

Added test and polishing
2018-06-27 08:45:31 -04:00
Gary Russell
ca2983c881 GH-373: Support multiplexed consumers
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/373

When a consumer is multiplexed, configure the container to listen to multiple topics.
Also for the polled consumer.

When using a DLQ, determine the queue name from the topic in the failed record (unless
an explicit DLQ name has been provisioned - in which case, the same DLQ will be used
for all topics.

Resolves #383
2018-06-12 14:52:51 -04:00
Soby Chacko
1b00179f25 Kafka Streams interactive query enhancements
* When running interactive queries against multiple instances under the same application id,
  ensure that the application can retrieve the proper instance that is hosting the queried state store
* Introduce a new API level service called InteractiveQueryServices
* Perform refactoring to support this enhancement
* Deprecate QueryableStoreRegistry in 2.1.0 in favor of InteractiveQueryServices

Resolves #369
2018-06-08 11:22:31 -04:00
Artem Bilan
f533177d21 GH-381: Remove duplicated SCSt-binder-test dep
Fixes spring-cloud/spring-cloud-stream-binder-kafka#381
2018-05-14 15:25:36 -04:00
Gary Russell
990a5142ce GH-339: Support topic patterns
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/339

- support regex patterns for consumer destinations to consume from multiple topics
- `enableDlq` is not available in this mode

As well as the test case, tested with a boot app...

```
spring.cloud.stream.bindings.input.destination=kbgh339.*
spring.cloud.stream.kafka.bindings.input.consumer.destination-is-pattern=true
```

and

```
2018-04-30 15:12:49.718  : partitions assigned: [kbgh339a-0]
2018-04-30 15:17:46.585  : partitions revoked: [kbgh339a-0]
2018-04-30 15:17:46.655  : partitions assigned: [kbgh339a-0, kbgh339b-0]
```

after adding a new topic matching the pattern.

Doc polishing.
2018-05-10 11:10:49 -04:00
Soby Chacko
13693e8e66 Kafka Streams DLQ related changes
DLQ handling needs to be adjusted in kafka streams binder due to the multiplexing of input topics.
This commit changes it accordingly in KStream and KTable binders.
Add tests to verify.
2018-05-01 12:02:53 -04:00
Soby Chacko
70cd7dc2f9 Upgrade spring cloud stream version to 2.1.0 sanpshot
Always enable multiplex to true in kafka streams binder
2018-05-01 10:10:42 -04:00
Sarath Shyam
894597309b Fix kaka-streams binder to consume messages from multiple input topics #361
Modified KafkaStreamsStreamListenerSetupMethodOrchestrator#getkStream
so that the KStream object is built from list of topic names
2018-04-30 15:05:47 -04:00
Thomas Cheyney
74689bf007 Reuse Kafka consumer Metrics
Polishing
2018-04-30 14:19:53 -04:00
Gary Russell
d2012c287a GH-360: Improve Binder Producer/Consumer Config
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/360

`producer-configuration` and `consumer-configuration` improperly appear in content-assist.

These are methods used by the binders to get merged configuration data (boot and binder).

Rename the methods and add `producerProperties` and `consumerProperties` to allow
configuration.
2018-04-27 12:37:48 -04:00
slamhan
109295464f QueryableStore retrieval stops at InvalidStateStoreException
If there are multiple streams, there is a code path that throws
a premature InvalidStateStoreException. Fixing that issue.

Fixes #366

Polishing.
2018-04-24 11:13:25 -04:00
Lei Chen
cf41a8a4eb Allow Kafka Streams state store creation
* Allow Kafka Streams state store creation when using process/transform method in DSL
 * Add unit test for state store
 * Address code review comments
 * Add author and javadocs
 * Integration test fixing for state store
 * Polishing
2018-04-20 15:53:43 -04:00
Soby Chacko
77540a2027 Kafka Streams initializr image for docs 2018-04-12 17:51:40 -04:00
Danish Garg
a2592698c9 Changed occurances of map calls on kafka streams to mapValues
Resolves #357
2018-04-11 15:19:21 -04:00
Oleg Zhurakousky
5543b4ed1e Post-release update to 2.1.0.BUILD-SNAPSHOT 2018-04-06 15:17:12 -04:00
Oleg Zhurakousky
acb8eef43b 2.0.0.RELEASE 2018-04-06 14:17:24 -04:00
Soby Chacko
b3f8cf41ef 2.0 release updates 2018-04-06 13:00:21 -04:00
Gary Russell
cbf693f14e Fix Stub in mock tests to return the group id
Needed for SK 2.1.5.
2018-04-06 12:44:24 -04:00
Gary Russell
39dd048ee5 Fix DLQ and raw/embedded headers
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/351

- DLQ should support embedded headers for backwards compatibility with 1.x apps
- DLQ should support `HeaderMode.none` for when using older brokers with raw data

Forward port of https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/pull/350

Resolves #352
2018-04-06 10:56:15 -04:00
Soby Chacko
0689e87489 Polishing kafka streams docs 2018-04-04 11:25:18 -04:00
Soby Chacko
2c3787faa1 Kafka streams outbound converter changes
Use getMessageConverterForAllRegistered() from CompositeMessageConverterFactory

Resolves #353
2018-04-02 17:19:19 -04:00
Soby Chacko
84f0fb28ae Address IllegalAccessException in Kafka Streams binder
When StreamListener methods are contained in a top level non-public class, Kafka
Streams binder throws an IllegalAccessException. Fixing it by making it accessible.

Resolves #348
2018-04-02 15:32:08 -04:00
Soby Chacko
710ff2c292 Fix NPE in Kafka Streams binder
* Fix NPE in Kafka Streams binder

Fix NPE when user provided consumer properties are missing in kafka streams binder

Resolves #343

* Addressing PR review comments
2018-03-28 13:32:37 -04:00
Oleg Zhurakousky
11a275a299 Merge pull request #342 from bewithvk/patch-1
Fix the broken image link of kafka-binider.
2018-03-24 08:00:28 -04:00
bewithvk
3526a298c8 Fix the broken image link of kafka-binider. 2018-03-21 10:45:43 -05:00
Jay Bryant
e152d0c073 Correcting my own error
I inadvertently added a word (I had two sentences in mind and got parts of both of them).

Resolves #341
2018-03-21 11:01:36 -04:00
Jay Bryant
976b903352 Incorporated feedback from Gary Russell
Gary caught a grammatical goof and pointed out some content that can be removed, because Kafka now has a feature it didn't use to have. That prompted me to rewrite the leader paragraph above that content, too.

Thanks, Gary.
2018-03-20 12:20:15 -05:00
Jay Bryant
9861c80355 Full editing pass for Spring Cloud Stream Kafka Binder
I corrected grammar and spelling and edited for a corporate voice. I also add a link or two.
2018-03-20 09:19:29 -05:00
Soby Chacko
7057e225df Back to 2.0.0.BUILD-SNAPSHOT 2018-03-12 16:31:48 -04:00
Soby Chacko
b8267ea81e 2.0.0.RC3 Release 2018-03-12 15:23:09 -04:00
Gary Russell
2b595b004f GH-337: Add ackEachRecord property
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/337
Resolves #338
2018-03-10 11:02:50 -05:00
Gary Russell
de45edc962 Add missing binder headerMapperBeanName to doc. 2018-03-08 11:46:03 -05:00
Gary Russell
2406fe5237 Event Publisher Polishing
Now that the abstract binder makes its event publisher available to subclasses,
use it, if present, instead of the application context.
In most cases, they will be the same object, but the user might override the
publisher.

Resolves #336
2018-03-08 09:23:14 -05:00
Oleg Zhurakousky
b5a0013e1e GH-330 Polishing
Resolves #330
Resolves #334
2018-03-08 09:11:11 -05:00
Gary Russell
c814ad5595 GH-330: Kafka Topic Provisioning Improvements
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/330

- allow override of binder-wide `replicationFactor` for each binding
- allow specific partition/replica configuration
- allow setting `NewTopic.configs()` properties, similar to the consumer and producer
- use a new `AdminClient` for provisioning (and `close()` it) instead of keeping a long-lived connection open.
2018-03-08 09:10:27 -05:00
Oleg Zhurakousky
def2c3d0ed SCST-GH-1259 Polishing
- added '@DeprecatedConfigurationProperty'
- minor doc polishing

Resolves #335
2018-03-07 16:04:44 -05:00