Commit Graph

194 Commits

Author SHA1 Message Date
spencergibb
7505144fcb Ignores pulsar-binder snapshot temporarily
The module is disabled
2023-08-09 15:26:10 -04:00
spencergibb
a45a2811e0 Disables pulsar-binder temporarily 2023-08-09 15:22:10 -04:00
Soby Chacko
7905601e51 DltAwareProcessor constructor changes 2023-07-24 15:14:38 -04:00
Soby Chacko
6bb4da89ce DLTAwareProcessor (Kafka Streams binder) changes
- Optional BiConsumer for processor record recoverer
 - Optional Supplier for downstream record timestamp
2023-07-24 15:04:57 -04:00
Soby Chacko
02567b1dae Kafka Streams binder error handling
- Initial support for Kafka Streams binder runtime error handling
 - This feature is very early in it's implementation and experimental right now
2023-07-21 18:23:58 -04:00
Soby Chacko
919d3c85fc Kafka Streams binder security test changes
- Invalidate security configuration in Kafka Streams binder
   security related tests. See the previous commit for more details.
2023-07-10 15:18:01 -04:00
Soby Chacko
0f0f29cfed Kafka binder security test changes
- Before running a test, invalidate any existing JVM-wide static
   security configuration so that tests are forced to create/use
   fresh security configuration. Without this, test suites with
   multiple security tests might fail as they might overlap with
   security confugiruation from other tests.
2023-07-10 14:23:09 -04:00
Soby Chacko
2a63301efe Reactor Kafka Binder Health Indicator (#2755)
* Reactor Kafka Binder Health Indicator

 - Provide a new abstraction for general Kafka binder related HealthIndicators.
 - Refactor Kafka binder to use the new abstraction
 - Add HealthIndicator implementation for the ReactorKafkaBinder

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2752

* Addressing PR review

* Addressing PR review

* Addressing PR review
2023-06-22 13:29:43 -04:00
Soby Chacko
0dd986d0ad Initial migration of Pulsar binder
- Moving Pulsar binder from the old 4.1.x branch
2023-06-16 16:05:46 +02:00
Soby Chacko
9ac24da343 Minor refactoring to KafkaBinderHealthIndicator
- Instead of implementing HealthIndicator directlry,
   extend from AbstractHealthIndicator provided by Spring Boot.
2023-06-14 16:52:45 -04:00
Soby Chacko
63d2bd3256 Updating to 4.1.0-SNAPSHOT 2023-06-14 14:22:10 -04:00
Soby Chacko
2093cb6a99 Multi output partition issues
- When using reactive functions, partition selector strategy does not
   use the configured partition count for multiple outbounds. This is
   because we take the first configured output binding and apply it's
   partition counts on all the outbound reactive streams (Tuples).
   Addressing this issue by properly applying the correct partition handling
   per output binding.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2750
2023-06-14 17:41:22 +02:00
buildmaster
0b7d773b30 Bumping versions to 4.0.4-SNAPSHOT after release 2023-05-24 18:47:37 +00:00
buildmaster
16cf46d74f Going back to snapshots 2023-05-24 18:47:37 +00:00
buildmaster
1550366414 Update SNAPSHOT to 4.0.3 2023-05-24 18:43:23 +00:00
jamesforward
14a444885f Multiple functions in Kafka Streams binder
* Trim functionUnits so we can declare them in a multi-line fashion,
   as currently whitespace is still considered.

 * Polishing

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2742
2023-05-22 15:58:57 -04:00
Soby Chacko
0ab5dfc05a KafkaStreams Functions Detection Logic (#2738)
* KafkaStreams Functions Detection Logic

Kafka Streams functions declared in super classes are no longer
detected by the binder discovery algorithms. Fixing this issue
by properly scanning the methods from the super classes.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2737

* Addressing PR review
2023-05-22 09:34:48 -04:00
Gary Russell
ac6dd61630 GH-2729: Reactor Kafka Binder SenderResult Support (#2730)
* GH-2729: Reactor Kafka Binder SenderResult Support

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2729

Allow configuration of a `FluxMessageChannel` to receive `SenderResult`s.
Add `SenderResultMessageHandler` to consume from that channel.

Remove undocumented `sendResult` header, which has no value without the
sender result correlation metadata.

* Add integration test and polish property docs.

- result channel must be FMC for reactive binder.

* Add documentation.

* Remove SenderResultMessageHandler.
2023-05-17 16:10:20 -04:00
oliverfuehrer
29c3cd7cdd Dynamic partition improvements in Kafka binder
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2706

  - Update partition count changes in Kafka dynamically during runtime
  - Checkstyle fixes
  - PartitionHandler changes
  - Improved handling of expression with 'payload' in the logic
  - Removed getter/setter for PartitionHandler and use ReflectionUtils within Test
  - Renamed property to 'dynamicPartitionUpdatesEnabled' and improved documentation (also one line per sentence)
  - Improved/renamed test to use embeddedkafka and really test update behaviour
  - Javadoc
2023-05-09 10:14:12 -04:00
Soby Chacko
7ee3125bc7 Fixing a Kafka binder unit test failure
Latest version of Mockito (5.3.0) used in Spring Boot 3.1.0,
apparently needs the specific Collection type when capturing
values via the ArgumentCaptor. Earlier versisons didn't mandate this.
2023-05-01 18:24:15 -04:00
Gary Russell
c21da59294 GH-2709: Reactive Kafka Binder and Topic Patterns
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2709

Add support for `...destinationIsPattern` as is supported by the message channel
binder.
2023-04-18 10:16:54 -04:00
조찬형[Plasma]
d5e5699f3b Replace duplicate code with existing method
Resolves #2702
2023-04-11 14:36:54 +02:00
Gary Russell
57733739e5 GH-2673: Use Binder Admin Config with Observation
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2673

Also configure observation (if enabled) on DLQ template.
Resolves #2703
2023-04-11 14:33:42 +02:00
Gary Russell
f48523f774 GH-2691: Kafka Reactive Binder Support Multiplex (#2698)
* GH-2691: Kafka Reactive Binder Support Multiplex

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2691

* Add docs.
2023-04-05 15:50:07 -04:00
Gary Russell
92279db2a1 GH-2686: Reactive Kafka Fix Auto Commit
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2686

Due to `.concatMap()`, offsets were committed before passing the records to
the pipeline.
2023-04-05 15:46:54 -04:00
Marcin Zajączkowski
43d3d9c415 Include also topic name is warnings and errors
Here the messages related to unexpected number of partitions.
Prior to that, it could be not trivial to be sure what topic is affected.

checkstyle fix
2023-04-04 10:44:19 -04:00
Gary Russell
64835ef1d7 GH-2685: Reactive Kafka Customizers - Add Generics
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2685

Required to allow customization of serializers/deserializers.
2023-04-03 13:11:34 -04:00
Soby Chacko
b814c0f395 Rely on Boot for reactor-kafka dependency 2023-04-03 12:04:56 -04:00
Soby Chacko
4c78d862eb Update reactor-kafka to 1.3.17
Update reactor-kafka to 1.3.17 in reactive-kafka binder

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2678
2023-03-30 14:27:21 -04:00
buildmaster
ea37eed9be Bumping versions 2023-03-29 17:33:29 +00:00
buildmaster
b2f2b51b90 Bumping versions 2023-03-28 17:27:48 +00:00
buildmaster
58c028f339 Bumping versions to 4.0.3-SNAPSHOT after release 2023-03-28 14:51:23 +00:00
buildmaster
c45f2e3ac7 Going back to snapshots 2023-03-28 14:51:22 +00:00
buildmaster
d932cf990f Update SNAPSHOT to 4.0.2 2023-03-28 14:47:17 +00:00
Oleg Zhurakousky
98051a0a1e GH-2674 Cleanup deprecated and obsolete properties in StreamFunctionProperties
Resolves #2674
2023-03-21 13:41:36 +01:00
Steven PG
e5d4916a4d Stream batch produce consume (#2654)
Kafka Binder-based Batch Producer and Consumer

 - Provide a new sample for batch producer and consumer with the Kafka binder
2023-03-09 20:15:53 -05:00
Soby Chacko
ce371f0bfb KafkaBinderConfigurationProperties metadata
Remove @ConfigurationProperties from KafkaBinderConfigurationProperties
and move it to the respective @Bean methods in regular Kafka binder
and it's reactive counter part.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2640
2023-03-07 17:31:02 -05:00
Fernando Blanch
807f51f175 Support pause/resume for consumer bindings with multiple destinations from BindingsLifecycleController
add test queryng a binding that not exists return empty list

remove unnecessary formatting changes

Resolves #2660
Resolves #2658
2023-03-01 13:33:22 +01:00
Soby Chacko
c875670b73 Test cleanup - Kafka Streams binder 2023-02-27 17:02:31 -05:00
Soby Chacko
47c8adcf99 Remove lingering usage of javax.annotation.PostConstruct
Use jakarta.annotation instead.
2023-02-27 16:04:47 -05:00
Gary Russell
82aa2cf41a GH-2653: Reactor Kafka Binder Fix Offset Commits
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2653

Add support for auto and manual offset commits.
Resolves #2656
2023-02-23 15:24:37 +01:00
spencergibb
cf7c6fc18c Adds httpclient version
apache httpclient 4 dep mgmt was removed in boot 3.1
2023-02-15 15:42:42 -05:00
Soby Chacko
371815806b Supporting DLQ in batch mode (#2649)
* Supporting DLQ in batch mode

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2317

* PR review

Co-authored-by: Gary Russell <grussell@vmware.com>

---------

Co-authored-by: Gary Russell <grussell@vmware.com>
2023-02-13 12:05:28 -05:00
Soby Chacko
4100f27791 KafkaBinderConfigurationProperties metadata
When generating configuration metadata (spring-configuration-metadata.json),
KafkaBinderConfigurationProperties class is missing all the information except
for the constructor bound KafkaProperties. We need to add the @Autowired on the
constructor so that Boot will not do a constructor binding and thus include all
the JavaBeans style properties in the metadata. See this Boot issue for more details.

https://github.com/spring-projects/spring-boot/issues/34031

In addition, this @Autowired is necessary for the properties to be correctly
bound when running in native mode.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2640
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2644
2023-02-07 18:16:45 -05:00
Gary Russell
8411d9c548 GH-2634: Javadoc Polishing 2023-01-31 10:37:43 -05:00
Gary Russell
a325b8b9fd GH-2634: Reactor Kafka Binder Customization
Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2634

Allow customization of `ReceiverOptions` and `ProducerOptions`.
2023-01-27 09:51:03 -05:00
Byungjun You
7895025c47 Use 'instanceof' pattern variable
Update authors and copyright years

Fixing checkstyle issue
2023-01-26 11:30:04 -05:00
buildmaster
b49fde97c8 Bumping versions 2023-01-26 16:01:13 +00:00
Soby Chacko
3b1b29e86d Health indicator fixes when topic is pattern (#2635)
When destination is pattern, even when the cluster is down,
health indicator is erroneously reporting the status as UP.
Addressing this issue by calling a simple listTopics on the
consumer when the destination is pattern, and if it throws an
exception, report the status as DOWN. If the call succeeds, we
assume that the cluster is accessible.

Resolves https://github.com/spring-cloud/spring-cloud-stream/issues/2628
2023-01-25 13:23:38 -05:00
buildmaster
7bff24af68 Bumping versions 2023-01-25 17:26:18 +00:00