Commit Graph

793 Commits

Author SHA1 Message Date
Soby Chacko
0bedc606ce Temporarily disabling KafkaBinderTests 2021-07-13 15:03:40 -04:00
Oleg Zhurakousky
b5cb32767b Fix POMs for 3.2 2021-07-13 16:50:01 +02:00
Oleg Zhurakousky
1a45ff797a JUnit 5 migration related to boot 2.6 2021-07-13 13:00:55 +02:00
Soby Chacko
f7537e795e Adding docs for multi binder JAAS configuration
Adding documentation for connecting to multiple Kafka clusters
with separate JAAS configuraiton from within a single application.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/944
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/874
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/924
2021-07-07 15:37:37 -04:00
Soby Chacko
7cae3aa54f GH-1096: Named components in Kafka Streams
Support KIP-307 in Kafka Streams binder where the input (source)
and output (sink) bindings are customized with user-provided names.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1096

Resolves #1098
2021-07-05 14:48:09 +02:00
Soby Chacko
5454d54faf Support Kafka Streams binder function composition
Composed function defintions can start with a `java.util.function.Function`
or `java.util.function.BiFunction` and compose with other functions or consumers.
In the case of a consumer, this needs to be the last unit in the function definition.

`java.util.function.BiConumer` is not eligibe for function composition.

The first component in the definition can be a curried function as well.

Adding tests and docs.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1088

Resolves #1091
2021-07-05 14:29:25 +02:00
Soby Chacko
4d68282937 Refactor JAAS initializer tests 2021-06-23 21:33:11 -04:00
Soby Chacko
9cf4cdc7c1 KafkaStreamsBinderBootstrapTest updates
Migrate tests from StreamListener to the functional model.
2021-06-23 18:30:56 -04:00
Soby Chacko
69a3c36db9 GH-1094: Fix JAAS initializer tests on CI
Before running the JAAS initilizer security tests, remove reference to
JAAS related  config files created previously in other tests. This is done
through clearing a system property (java.security.auth.login.config) in
the tests. Without clearing this property, these tests run into a race condition.

Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1094
2021-06-23 17:56:38 -04:00
Soby Chacko
b8be381828 Temporarily disabling Kafka Streams jaas test 2021-06-22 21:31:36 -04:00
Soby Chacko
162993dd46 Test fixes for Kafka Streams jass config tests
Provide different application id's.
2021-06-22 21:02:10 -04:00
Soby Chacko
ee3096658a GH-1092: Fix JAAS config issues (Kafka Streams)
Currently, Kafka Streams binder does not honor JAAS configuration properties.
Address this by adding the same `KafkaJaasLoginModuleInitializer` bean used in
regular Kafka binder.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1092
2021-06-22 17:16:33 -04:00
Soby Chacko
6e46054d7b Address deprecations for StreamsBuilderFactoryBeanCustomizer
StreamsBuilderFactoryBeanCustomizer is now renamed to StreamsBuilderFactoryBeanConfigurer
2021-06-08 16:56:47 -04:00
Gary Russell
7bc90c10a2 GH-1084: Add txCommitRecovered Property
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1084

Update copyrights
2021-06-02 18:01:32 -04:00
buildmaster
5adeea2acb Bumping versions to 3.1.4-SNAPSHOT after release 2021-05-26 15:50:17 +00:00
buildmaster
65664a2d4c Going back to snapshots 2021-05-26 15:50:17 +00:00
buildmaster
f5e12a29c1 Update SNAPSHOT to 3.1.3 2021-05-26 15:48:31 +00:00
Soby Chacko
416580e607 Version updates 2021-05-26 10:38:33 -04:00
aleksevi
a81093734e GH-1081 Suspicious multiplication of ScheduledExecutorService in the method "bindTo" of class "KafkaBinderMetrics"
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1081

The wrong consequence from the absense of scheduler.shutdown():
1) First of all we created the pool with 1 thread.
2) After we lost the reference on it and created the pool with 2 threads.
3) But, the first pool is not yet collected by GC and now you have 3 threads together. And so on.

Each thread does nothing, but it takes system memory and takes part in the scheduling process.
After 30 topics, for example, we potentially have (30+1)*15=465 threads.
It is already serious additional load on the switching contexts and the native memory.

Removed waiting after scheduler stop request.

checkstyle fixes
2021-05-26 10:37:42 -04:00
Soby Chacko
5f4e3eebc5 Removing deprecated setAckOnError method call (#1082)
Going forward, binder will use the default ack on error settings
in Spring Kafka, which is true by default. If the applicaitons
need to change this behavior, then a custom error handler must
be provided through ListenerContainerCustomizer.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1079
2021-05-24 16:46:26 -04:00
Taras Danylchuk
cb42e80dac gh-1059 : Added health indicator for kafka messages listener containers created via kafka binder
renamed variables, return unknown on empty listener list

added tests, fixed PR comments

checkstyle fix
2021-05-17 15:20:23 -04:00
Soby Chacko
4fb5037fd7 Use dlqProducerProperties for DLQ topics
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1077
2021-05-14 11:54:01 -04:00
Soby Chacko
d8b8c8d9fd Resetting offsets docs clarification 2021-05-06 18:16:47 -04:00
Soby Chacko
9e4a1075d4 Native compilation changes for Kafka Streams binder
This comnmit effectively removes changes introduced by a4ad9e2c0b.
Removing spring.factories mechanism of registering binders in preference to using spring.binders.
2021-04-29 19:35:15 -04:00
Soby Chacko
79497ad264 Native compilation changes
This commit effectively reverts 4e6881830a

Remove spring.factories changes introduced in the commit mentiond above.
Revert the binder to use spring.binders mechanism to register binder configurations.
2021-04-29 19:08:01 -04:00
Oleg Zhurakousky
7e48afa005 Merge pull request #1067 from sobychacko/kafka-streams-native-1
Native changes required by Kafka Streams binder
2021-04-28 19:13:52 +02:00
Oleg Zhurakousky
87ac491230 Merge pull request #1066 from sobychacko/kafka-binder-native-1
More changes required for native compilation
2021-04-28 19:13:43 +02:00
Soby Chacko
a4ad9e2c0b Native changes required by Kafka Streams binder 2021-04-28 12:26:53 -04:00
Soby Chacko
4e6881830a More changes required for native compilation
These are Kafka binder changes required for native applications.
2021-04-28 12:22:51 -04:00
Soby Chacko
829ce1cf7e Native compilation changes
This commit is a counter part to the commit below in core:

ae017e222f
2021-04-22 11:35:08 -04:00
buildmaster
c5289589c1 Bumping versions 2021-03-27 08:52:52 +00:00
Soby Chacko
dd607627ed Fix destination as pattern issues in Kafka Streams (#1054)
When destination-is-pattern property is enabled and native deserialization
is used, then Kafka Streams binder does not use the correct Serde types.
Fixing this issue by providing the proper Serdes to the StreamsBuilder.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1051
2021-03-25 16:20:33 -04:00
Gary Russell
0ea4315af8 GH-1043: Support Custom BatchMessageConverter
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1043
2021-03-24 13:23:53 -04:00
Soby Chacko
f25dbff2b7 Kafka Streams component bean scanning enhancements
When functional beans are discovered from libraries in the classpath,
it causes issues when Kafka Streams functions are scanned and bootstrapped.
Binder expects the users to provide function definition property although the
application does not directly include these functional beans or is aware of it.
Fixing this issue by excluding non kafka streams function from scanning.

Null check around adding micrometer listener to the StreamsBuilder.

Addressing issues raised by the comments here:
https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1030#issuecomment-804039087
2021-03-22 19:06:07 -04:00
Gary Russell
f8b290844b GH-1046: Fix Out of Order Offset Commit with DLQ
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1046

When using ack mode `MANUAL_IMMEDIATE`, we must acknowledge the delivery on
the container thread, otherwise the commit will be queued for later processing,
possibly causing out of order commits, incorrectly reducing the committed offset.

Wait for the send to complete before acknowledging; use a timeout slightly larger
than the configured producer delivery timeout to avoid premature timeouts.

Tested with user-provided test case.

Fix missing timeout buffer.

More context for this issue: https://gitter.im/spring-cloud/spring-cloud-stream?at=6050f43595e23446e43cacd1
2021-03-19 15:36:29 -04:00
Gary Russell
a7e025794c Fix typo in doc 2021-03-18 13:29:21 -04:00
Gary Russell
634a73c9ff Doc Polishing for resetOffsets (#1045)
* Doc Polishing for resetOffsets

* Fix typos.
2021-03-18 13:13:18 -04:00
buildmaster
bc2f692964 Bumping versions to 3.1.3-SNAPSHOT after release 2021-03-16 13:53:15 +00:00
buildmaster
7eefe6c567 Going back to snapshots 2021-03-16 13:53:15 +00:00
buildmaster
c5d108ef89 Update SNAPSHOT to 3.1.2 2021-03-16 13:51:50 +00:00
Soby Chacko
47e7ca07a8 KTable event type routing
Introduce event type based routing for KTable types. This is
already available for KStream types.
See this commit: 386a361a66

Adding a new DeserialiationExceptionHander to address event type routing
use case for GlobalKTable.
See this comment: https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1003#issuecomment-799847819

Upgrade Spring Kafka to 2.6.6
Upgrade Spring Integration to 5.4.4
2021-03-15 20:23:16 -04:00
Soby Chacko
7cc001ac4c Support KStream lifecycle through binding endpoint (#1042)
* Support KStream lifecycle through binding endpoint

Introduce the ability for Kafka Streams application's lifecycle
management through actuator binding endpoints. Kafka Streams
only supports STOP and START operations. PAUSE/RESUME operations
that is available in regular message channel based binders
are not available in Kafka Streams binder.

Adding tests and docs.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1038
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/850

https://stackoverflow.com/questions/60282225/why-doesnt-kstreambinder-have-a-lifecycle-for-defaultbinding

* Addressing PR review comments

* Addressing PR review

* cleanup unused code
2021-03-15 16:14:19 -04:00
Soby Chacko
a7299df63f Cleanup Kafka Streams metrics support
StreamsListener (for Micrometer) is now directly available
in Spring Kafka starting from 2.5.3 (as KafkaStreamsMicrometerListener).
Removing the temporory interface added in the binder.

Addressing PR review comments.

Modifying tests to verify.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1040
2021-03-12 15:14:19 -05:00
Soby Chacko
33aa926940 Fix checkstyle 2021-03-09 17:22:23 -05:00
Soby Chacko
a1fb7f0a2d Kafka Streams function detection improvements (#1033)
* Kafka Streams function detection improvements

Allow Kafka Streams functions defined as Component beans
to be candidates for establishing bindings. Currently, Kafka Streams
functions need to be written as functional beans using @Bean.
Adding this improvement so that if applications prefer to write
the business logic using @Component, then it is possible to do so.

Adding test cases to verify the behavior.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1030

* Kafka Streams functions and bean name overriding

Whn Kafka Streams function bean names are overridden,
there is an issue with scanning it properly for binding.
Addressing this issue.

* Adding docs for Component based model

* Addressing PR review comments
2021-03-09 15:50:10 -05:00
Soby Chacko
e2eca34e4b Cleaning up Kafka producer factories
Calling DisposableBean.destroy() on manually created
Kafka producer factories. This affects both Kafka and
Kafka Streams binders (in the case of Kafka Streams binder,
it only matters when using the DLQ support).

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1034
2021-03-02 17:26:37 -05:00
Soby Chacko
bc02da2900 Docs cleanup
Apply changes from a previous commit to the proper docs file.

See the prior relevant commit at the URL below:

af5778d157

Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1032
2021-03-02 16:42:21 -05:00
Soby Chacko
1f52ec5020 Fix minor issue processing ackMode 2021-02-22 12:22:28 -05:00
Soby Chacko
394b8a6685 Restore autoCommitOffset behavior
Until autoCommitOffset is fully removed (it is deprecated already in 3.1),
honor the old behavior if the application uses the property.

Restore autoCommitOnError from deprecated status to active for polled consumers.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1026
2021-02-09 17:48:56 -05:00
Soby Chacko
32939e30fe AdminClient customizer (#1023)
* AdminClient customizer

Provide the ability for applications to customize AdminClient by
introducing a new interface AdminClientConfigCustomizer.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1014

* Addressing PR review comments
2021-02-03 10:25:03 -05:00