Commit Graph

204 Commits

Author SHA1 Message Date
Soby Chacko
bec40fba89 KafkaBinderConfigurationProperties native changes
- Earlier, we had to add Autowired on KafkaBinderConfigurationProperties constructor
   as there were some issues with runtime hints generation which caused issues when
   running an app in native mode. Spring Boot fixed these issues and we can remove
   this unnecessary Autowired from the constructor.

See the following issues from Spring Boot for more details.

https://github.com/spring-projects/spring-boot/issues/34507
https://github.com/spring-projects/spring-boot/issues/35564

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2640
2023-09-07 16:54:09 -04:00
Soby Chacko
d822129d67 DltAwareProcessor enhancements
Cleaning up the code for the custom DltAwareProcessor
2023-09-06 20:36:35 -04:00
Gyurian Norbert
0949d4e145 GH-2761: Support non-CLASSPATH based certificates
If kafka truststore and keystore locations are not local files, then they
are converted to org.springframework.core.io.Resource resources, then copied
to local file system. This means that, paths can be defined as HTTP resources too.
Currently, the Kafka binder only supports CLASSPATH based resources. It would be
useful if we can support non-CLASSPATH like resources such as HTTP, so that if an
application uses config server for example to store certificates, then those will
be copied from it's HTTP endpoint to the local filesystem as Resources.

Checkstyle, documentation fixes.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2761
2023-09-05 12:58:42 -04:00
Soby Chacko
acc8b1cb92 KafkaNull Test Changes
There was a regression introduced in Spring Cloud Function where consumers
of type Consumer<Message<?>> receive null values when tombstone records
are given as KafkaNull.

See this issue for more details: https://github.com/spring-cloud/spring-cloud-function/issues/1060

Regression is addressed in Spring Cloud Function and
making the corresponding test changes in Spring Cloud Stream Kafka binder.
2023-08-30 12:38:39 -04:00
Soby Chacko
8b7accac2a Renable Pulsar binder 2023-08-29 16:37:25 -04:00
Soby Chacko
3b4e1795c3 KafkaJaasLoginModuleInitializer init changes
- Insted of unconditionally initializing KafkaJaasLoginModuleInitializer,
   only create it, if jaas properties are set properly. By creating it always,
   we unncessarily expose Java security related class requirements to the binder
   apps. For e.g. if the underlying JVM security classes change and require
   reflection, that could break AOT apps that don't use a secure Kafka cluster.
   Fixing these type of issues by creating KafkaJaasLoginModuleInitializer only if required.
2023-08-10 12:18:16 -04:00
buildmaster
6c28820e8e Going back to snapshots 2023-08-10 01:30:10 +00:00
buildmaster
4743859cd7 Update SNAPSHOT to 4.1.0-M1 2023-08-10 01:26:38 +00:00
spencergibb
76332e1e42 jenkins/releaser hacking 2023-08-09 16:08:30 -04:00
spencergibb
2380afbe1f fixes checkstyle issues 2023-08-09 15:48:56 -04:00
spencergibb
7505144fcb Ignores pulsar-binder snapshot temporarily
The module is disabled
2023-08-09 15:26:10 -04:00
spencergibb
a45a2811e0 Disables pulsar-binder temporarily 2023-08-09 15:22:10 -04:00
Soby Chacko
7905601e51 DltAwareProcessor constructor changes 2023-07-24 15:14:38 -04:00
Soby Chacko
6bb4da89ce DLTAwareProcessor (Kafka Streams binder) changes
- Optional BiConsumer for processor record recoverer
 - Optional Supplier for downstream record timestamp
2023-07-24 15:04:57 -04:00
Soby Chacko
02567b1dae Kafka Streams binder error handling
- Initial support for Kafka Streams binder runtime error handling
 - This feature is very early in it's implementation and experimental right now
2023-07-21 18:23:58 -04:00
Soby Chacko
919d3c85fc Kafka Streams binder security test changes
- Invalidate security configuration in Kafka Streams binder
   security related tests. See the previous commit for more details.
2023-07-10 15:18:01 -04:00
Soby Chacko
0f0f29cfed Kafka binder security test changes
- Before running a test, invalidate any existing JVM-wide static
   security configuration so that tests are forced to create/use
   fresh security configuration. Without this, test suites with
   multiple security tests might fail as they might overlap with
   security confugiruation from other tests.
2023-07-10 14:23:09 -04:00
Soby Chacko
2a63301efe Reactor Kafka Binder Health Indicator (#2755)
* Reactor Kafka Binder Health Indicator

 - Provide a new abstraction for general Kafka binder related HealthIndicators.
 - Refactor Kafka binder to use the new abstraction
 - Add HealthIndicator implementation for the ReactorKafkaBinder

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2752

* Addressing PR review

* Addressing PR review

* Addressing PR review
2023-06-22 13:29:43 -04:00
Soby Chacko
0dd986d0ad Initial migration of Pulsar binder
- Moving Pulsar binder from the old 4.1.x branch
2023-06-16 16:05:46 +02:00
Soby Chacko
9ac24da343 Minor refactoring to KafkaBinderHealthIndicator
- Instead of implementing HealthIndicator directlry,
   extend from AbstractHealthIndicator provided by Spring Boot.
2023-06-14 16:52:45 -04:00
Soby Chacko
63d2bd3256 Updating to 4.1.0-SNAPSHOT 2023-06-14 14:22:10 -04:00
Soby Chacko
2093cb6a99 Multi output partition issues
- When using reactive functions, partition selector strategy does not
   use the configured partition count for multiple outbounds. This is
   because we take the first configured output binding and apply it's
   partition counts on all the outbound reactive streams (Tuples).
   Addressing this issue by properly applying the correct partition handling
   per output binding.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2750
2023-06-14 17:41:22 +02:00
buildmaster
0b7d773b30 Bumping versions to 4.0.4-SNAPSHOT after release 2023-05-24 18:47:37 +00:00
buildmaster
16cf46d74f Going back to snapshots 2023-05-24 18:47:37 +00:00
buildmaster
1550366414 Update SNAPSHOT to 4.0.3 2023-05-24 18:43:23 +00:00
jamesforward
14a444885f Multiple functions in Kafka Streams binder
* Trim functionUnits so we can declare them in a multi-line fashion,
   as currently whitespace is still considered.

 * Polishing

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2742
2023-05-22 15:58:57 -04:00
Soby Chacko
0ab5dfc05a KafkaStreams Functions Detection Logic (#2738)
* KafkaStreams Functions Detection Logic

Kafka Streams functions declared in super classes are no longer
detected by the binder discovery algorithms. Fixing this issue
by properly scanning the methods from the super classes.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2737

* Addressing PR review
2023-05-22 09:34:48 -04:00
Gary Russell
ac6dd61630 GH-2729: Reactor Kafka Binder SenderResult Support (#2730)
* GH-2729: Reactor Kafka Binder SenderResult Support

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2729

Allow configuration of a `FluxMessageChannel` to receive `SenderResult`s.
Add `SenderResultMessageHandler` to consume from that channel.

Remove undocumented `sendResult` header, which has no value without the
sender result correlation metadata.

* Add integration test and polish property docs.

- result channel must be FMC for reactive binder.

* Add documentation.

* Remove SenderResultMessageHandler.
2023-05-17 16:10:20 -04:00
oliverfuehrer
29c3cd7cdd Dynamic partition improvements in Kafka binder
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2706

  - Update partition count changes in Kafka dynamically during runtime
  - Checkstyle fixes
  - PartitionHandler changes
  - Improved handling of expression with 'payload' in the logic
  - Removed getter/setter for PartitionHandler and use ReflectionUtils within Test
  - Renamed property to 'dynamicPartitionUpdatesEnabled' and improved documentation (also one line per sentence)
  - Improved/renamed test to use embeddedkafka and really test update behaviour
  - Javadoc
2023-05-09 10:14:12 -04:00
Soby Chacko
7ee3125bc7 Fixing a Kafka binder unit test failure
Latest version of Mockito (5.3.0) used in Spring Boot 3.1.0,
apparently needs the specific Collection type when capturing
values via the ArgumentCaptor. Earlier versisons didn't mandate this.
2023-05-01 18:24:15 -04:00
Gary Russell
c21da59294 GH-2709: Reactive Kafka Binder and Topic Patterns
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2709

Add support for `...destinationIsPattern` as is supported by the message channel
binder.
2023-04-18 10:16:54 -04:00
조찬형[Plasma]
d5e5699f3b Replace duplicate code with existing method
Resolves #2702
2023-04-11 14:36:54 +02:00
Gary Russell
57733739e5 GH-2673: Use Binder Admin Config with Observation
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2673

Also configure observation (if enabled) on DLQ template.
Resolves #2703
2023-04-11 14:33:42 +02:00
Gary Russell
f48523f774 GH-2691: Kafka Reactive Binder Support Multiplex (#2698)
* GH-2691: Kafka Reactive Binder Support Multiplex

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2691

* Add docs.
2023-04-05 15:50:07 -04:00
Gary Russell
92279db2a1 GH-2686: Reactive Kafka Fix Auto Commit
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2686

Due to `.concatMap()`, offsets were committed before passing the records to
the pipeline.
2023-04-05 15:46:54 -04:00
Marcin Zajączkowski
43d3d9c415 Include also topic name is warnings and errors
Here the messages related to unexpected number of partitions.
Prior to that, it could be not trivial to be sure what topic is affected.

checkstyle fix
2023-04-04 10:44:19 -04:00
Gary Russell
64835ef1d7 GH-2685: Reactive Kafka Customizers - Add Generics
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2685

Required to allow customization of serializers/deserializers.
2023-04-03 13:11:34 -04:00
Soby Chacko
b814c0f395 Rely on Boot for reactor-kafka dependency 2023-04-03 12:04:56 -04:00
Soby Chacko
4c78d862eb Update reactor-kafka to 1.3.17
Update reactor-kafka to 1.3.17 in reactive-kafka binder

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2678
2023-03-30 14:27:21 -04:00
buildmaster
ea37eed9be Bumping versions 2023-03-29 17:33:29 +00:00
buildmaster
b2f2b51b90 Bumping versions 2023-03-28 17:27:48 +00:00
buildmaster
58c028f339 Bumping versions to 4.0.3-SNAPSHOT after release 2023-03-28 14:51:23 +00:00
buildmaster
c45f2e3ac7 Going back to snapshots 2023-03-28 14:51:22 +00:00
buildmaster
d932cf990f Update SNAPSHOT to 4.0.2 2023-03-28 14:47:17 +00:00
Oleg Zhurakousky
98051a0a1e GH-2674 Cleanup deprecated and obsolete properties in StreamFunctionProperties
Resolves #2674
2023-03-21 13:41:36 +01:00
Steven PG
e5d4916a4d Stream batch produce consume (#2654)
Kafka Binder-based Batch Producer and Consumer

 - Provide a new sample for batch producer and consumer with the Kafka binder
2023-03-09 20:15:53 -05:00
Soby Chacko
ce371f0bfb KafkaBinderConfigurationProperties metadata
Remove @ConfigurationProperties from KafkaBinderConfigurationProperties
and move it to the respective @Bean methods in regular Kafka binder
and it's reactive counter part.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2640
2023-03-07 17:31:02 -05:00
Fernando Blanch
807f51f175 Support pause/resume for consumer bindings with multiple destinations from BindingsLifecycleController
add test queryng a binding that not exists return empty list

remove unnecessary formatting changes

Resolves #2660
Resolves #2658
2023-03-01 13:33:22 +01:00
Soby Chacko
c875670b73 Test cleanup - Kafka Streams binder 2023-02-27 17:02:31 -05:00
Soby Chacko
47c8adcf99 Remove lingering usage of javax.annotation.PostConstruct
Use jakarta.annotation instead.
2023-02-27 16:04:47 -05:00