Commit Graph

196 Commits

Author SHA1 Message Date
Soby Chacko
5f266ce580 Kafka binder RuntimeHints
- Adding missing runtime hints for Kafka binder properties

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2604
2023-09-08 18:28:44 -04:00
Gyurian Norbert
d3e85a527b GH-2761: Support non-CLASSPATH based certificates
If kafka truststore and keystore locations are not local files, then they
are converted to org.springframework.core.io.Resource resources, then copied
to local file system. This means that, paths can be defined as HTTP resources too.
Currently, the Kafka binder only supports CLASSPATH based resources. It would be
useful if we can support non-CLASSPATH like resources such as HTTP, so that if an
application uses config server for example to store certificates, then those will
be copied from it's HTTP endpoint to the local filesystem as Resources.

Checkstyle, documentation fixes.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2761
2023-09-05 13:15:53 -04:00
Soby Chacko
e933100244 KafkaNull Test Changes
There was a regression introduced in Spring Cloud Function where consumers
of type Consumer<Message<?>> receive null values when tombstone records
are given as KafkaNull.

See this issue for more details: https://github.com/spring-cloud/spring-cloud-function/issues/1060

Regression is addressed in Spring Cloud Function and
making the corresponding test changes in Spring Cloud Stream Kafka binder.
2023-08-30 12:37:53 -04:00
Soby Chacko
048041bfdb KafkaJaasLoginModuleInitializer init changes
- Insted of unconditionally initializing KafkaJaasLoginModuleInitializer,
   only create it, if jaas properties are set properly. By creating it always,
   we unncessarily expose Java security related class requirements to the binder
   apps. For e.g. if the underlying JVM security classes change and require
   reflection, that could break AOT apps that don't use a secure Kafka cluster.
   Fixing these type of issues by creating KafkaJaasLoginModuleInitializer only if required.
2023-08-10 12:19:03 -04:00
buildmaster
afe3666962 Bumping versions to 4.0.5-SNAPSHOT after release 2023-07-27 20:02:52 +00:00
buildmaster
69bffeb7a6 Going back to snapshots 2023-07-27 20:02:52 +00:00
buildmaster
f0d66cf8b6 Update SNAPSHOT to 4.0.4 2023-07-27 19:58:30 +00:00
Soby Chacko
203a82e046 Kafka Streams binder security test changes
- Invalidate security configuration in Kafka Streams binder
   security related tests. See the previous commit for more details.
2023-07-26 10:12:27 -04:00
Soby Chacko
d230b62bcc Kafka binder security test changes
- Before running a test, invalidate any existing JVM-wide static
   security configuration so that tests are forced to create/use
   fresh security configuration. Without this, test suites with
   multiple security tests might fail as they might overlap with
   security confugiruation from other tests.
2023-07-26 10:12:20 -04:00
buildmaster
db695b1f02 Bumping versions 2023-07-14 02:32:51 +00:00
Soby Chacko
a2842e7edd Test fix in ReactorKafkaBinderHealthIndicator 2023-06-22 16:26:02 -04:00
Soby Chacko
806e6ea047 Reactor Kafka Binder Health Indicator (#2755)
* Reactor Kafka Binder Health Indicator

 - Provide a new abstraction for general Kafka binder related HealthIndicators.
 - Refactor Kafka binder to use the new abstraction
 - Add HealthIndicator implementation for the ReactorKafkaBinder

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2752

* Addressing PR review

* Addressing PR review

* Addressing PR review
2023-06-22 14:25:04 -04:00
Soby Chacko
3059a8af6c Minor refactoring to KafkaBinderHealthIndicator
- Instead of implementing HealthIndicator directlry,
   extend from AbstractHealthIndicator provided by Spring Boot.
2023-06-14 16:54:30 -04:00
Soby Chacko
2093cb6a99 Multi output partition issues
- When using reactive functions, partition selector strategy does not
   use the configured partition count for multiple outbounds. This is
   because we take the first configured output binding and apply it's
   partition counts on all the outbound reactive streams (Tuples).
   Addressing this issue by properly applying the correct partition handling
   per output binding.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2750
2023-06-14 17:41:22 +02:00
buildmaster
0b7d773b30 Bumping versions to 4.0.4-SNAPSHOT after release 2023-05-24 18:47:37 +00:00
buildmaster
16cf46d74f Going back to snapshots 2023-05-24 18:47:37 +00:00
buildmaster
1550366414 Update SNAPSHOT to 4.0.3 2023-05-24 18:43:23 +00:00
jamesforward
14a444885f Multiple functions in Kafka Streams binder
* Trim functionUnits so we can declare them in a multi-line fashion,
   as currently whitespace is still considered.

 * Polishing

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2742
2023-05-22 15:58:57 -04:00
Soby Chacko
0ab5dfc05a KafkaStreams Functions Detection Logic (#2738)
* KafkaStreams Functions Detection Logic

Kafka Streams functions declared in super classes are no longer
detected by the binder discovery algorithms. Fixing this issue
by properly scanning the methods from the super classes.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2737

* Addressing PR review
2023-05-22 09:34:48 -04:00
Gary Russell
ac6dd61630 GH-2729: Reactor Kafka Binder SenderResult Support (#2730)
* GH-2729: Reactor Kafka Binder SenderResult Support

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2729

Allow configuration of a `FluxMessageChannel` to receive `SenderResult`s.
Add `SenderResultMessageHandler` to consume from that channel.

Remove undocumented `sendResult` header, which has no value without the
sender result correlation metadata.

* Add integration test and polish property docs.

- result channel must be FMC for reactive binder.

* Add documentation.

* Remove SenderResultMessageHandler.
2023-05-17 16:10:20 -04:00
oliverfuehrer
29c3cd7cdd Dynamic partition improvements in Kafka binder
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2706

  - Update partition count changes in Kafka dynamically during runtime
  - Checkstyle fixes
  - PartitionHandler changes
  - Improved handling of expression with 'payload' in the logic
  - Removed getter/setter for PartitionHandler and use ReflectionUtils within Test
  - Renamed property to 'dynamicPartitionUpdatesEnabled' and improved documentation (also one line per sentence)
  - Improved/renamed test to use embeddedkafka and really test update behaviour
  - Javadoc
2023-05-09 10:14:12 -04:00
Soby Chacko
7ee3125bc7 Fixing a Kafka binder unit test failure
Latest version of Mockito (5.3.0) used in Spring Boot 3.1.0,
apparently needs the specific Collection type when capturing
values via the ArgumentCaptor. Earlier versisons didn't mandate this.
2023-05-01 18:24:15 -04:00
Gary Russell
c21da59294 GH-2709: Reactive Kafka Binder and Topic Patterns
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2709

Add support for `...destinationIsPattern` as is supported by the message channel
binder.
2023-04-18 10:16:54 -04:00
조찬형[Plasma]
d5e5699f3b Replace duplicate code with existing method
Resolves #2702
2023-04-11 14:36:54 +02:00
Gary Russell
57733739e5 GH-2673: Use Binder Admin Config with Observation
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2673

Also configure observation (if enabled) on DLQ template.
Resolves #2703
2023-04-11 14:33:42 +02:00
Gary Russell
f48523f774 GH-2691: Kafka Reactive Binder Support Multiplex (#2698)
* GH-2691: Kafka Reactive Binder Support Multiplex

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2691

* Add docs.
2023-04-05 15:50:07 -04:00
Gary Russell
92279db2a1 GH-2686: Reactive Kafka Fix Auto Commit
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2686

Due to `.concatMap()`, offsets were committed before passing the records to
the pipeline.
2023-04-05 15:46:54 -04:00
Marcin Zajączkowski
43d3d9c415 Include also topic name is warnings and errors
Here the messages related to unexpected number of partitions.
Prior to that, it could be not trivial to be sure what topic is affected.

checkstyle fix
2023-04-04 10:44:19 -04:00
Gary Russell
64835ef1d7 GH-2685: Reactive Kafka Customizers - Add Generics
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2685

Required to allow customization of serializers/deserializers.
2023-04-03 13:11:34 -04:00
Soby Chacko
b814c0f395 Rely on Boot for reactor-kafka dependency 2023-04-03 12:04:56 -04:00
Soby Chacko
4c78d862eb Update reactor-kafka to 1.3.17
Update reactor-kafka to 1.3.17 in reactive-kafka binder

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2678
2023-03-30 14:27:21 -04:00
buildmaster
ea37eed9be Bumping versions 2023-03-29 17:33:29 +00:00
buildmaster
b2f2b51b90 Bumping versions 2023-03-28 17:27:48 +00:00
buildmaster
58c028f339 Bumping versions to 4.0.3-SNAPSHOT after release 2023-03-28 14:51:23 +00:00
buildmaster
c45f2e3ac7 Going back to snapshots 2023-03-28 14:51:22 +00:00
buildmaster
d932cf990f Update SNAPSHOT to 4.0.2 2023-03-28 14:47:17 +00:00
Oleg Zhurakousky
98051a0a1e GH-2674 Cleanup deprecated and obsolete properties in StreamFunctionProperties
Resolves #2674
2023-03-21 13:41:36 +01:00
Steven PG
e5d4916a4d Stream batch produce consume (#2654)
Kafka Binder-based Batch Producer and Consumer

 - Provide a new sample for batch producer and consumer with the Kafka binder
2023-03-09 20:15:53 -05:00
Soby Chacko
ce371f0bfb KafkaBinderConfigurationProperties metadata
Remove @ConfigurationProperties from KafkaBinderConfigurationProperties
and move it to the respective @Bean methods in regular Kafka binder
and it's reactive counter part.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2640
2023-03-07 17:31:02 -05:00
Fernando Blanch
807f51f175 Support pause/resume for consumer bindings with multiple destinations from BindingsLifecycleController
add test queryng a binding that not exists return empty list

remove unnecessary formatting changes

Resolves #2660
Resolves #2658
2023-03-01 13:33:22 +01:00
Soby Chacko
c875670b73 Test cleanup - Kafka Streams binder 2023-02-27 17:02:31 -05:00
Soby Chacko
47c8adcf99 Remove lingering usage of javax.annotation.PostConstruct
Use jakarta.annotation instead.
2023-02-27 16:04:47 -05:00
Gary Russell
82aa2cf41a GH-2653: Reactor Kafka Binder Fix Offset Commits
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2653

Add support for auto and manual offset commits.
Resolves #2656
2023-02-23 15:24:37 +01:00
spencergibb
cf7c6fc18c Adds httpclient version
apache httpclient 4 dep mgmt was removed in boot 3.1
2023-02-15 15:42:42 -05:00
Soby Chacko
371815806b Supporting DLQ in batch mode (#2649)
* Supporting DLQ in batch mode

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2317

* PR review

Co-authored-by: Gary Russell <grussell@vmware.com>

---------

Co-authored-by: Gary Russell <grussell@vmware.com>
2023-02-13 12:05:28 -05:00
Soby Chacko
4100f27791 KafkaBinderConfigurationProperties metadata
When generating configuration metadata (spring-configuration-metadata.json),
KafkaBinderConfigurationProperties class is missing all the information except
for the constructor bound KafkaProperties. We need to add the @Autowired on the
constructor so that Boot will not do a constructor binding and thus include all
the JavaBeans style properties in the metadata. See this Boot issue for more details.

https://github.com/spring-projects/spring-boot/issues/34031

In addition, this @Autowired is necessary for the properties to be correctly
bound when running in native mode.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2640
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2644
2023-02-07 18:16:45 -05:00
Gary Russell
8411d9c548 GH-2634: Javadoc Polishing 2023-01-31 10:37:43 -05:00
Gary Russell
a325b8b9fd GH-2634: Reactor Kafka Binder Customization
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2634

Allow customization of `ReceiverOptions` and `ProducerOptions`.
2023-01-27 09:51:03 -05:00
Byungjun You
7895025c47 Use 'instanceof' pattern variable
Update authors and copyright years

Fixing checkstyle issue
2023-01-26 11:30:04 -05:00
buildmaster
b49fde97c8 Bumping versions 2023-01-26 16:01:13 +00:00