215 Commits

Author SHA1 Message Date
Gary Russell
8a5c0377ef GH-3637: Fix KafkaMessageSource First Poll
Resolves https://github.com/spring-projects/spring-integration/issues/3637

If there are no records to receive, the poll blocked for `20*pollTimeout`.

Wake the consumer during partition assignment; however, this changes the behavior
when there are records present - the first poll always returns no records.

Detect that the wakeup was due to assignment and perform another poll.

Also add some debug logging, and don't create a new consumer if the source was
stopped.

**cherry-pick to 5.4.x**
2021-09-30 16:35:52 -04:00
Gary Russell
fba4b20451 GH-3600: Fix Kafka Tx Synchronization
Resolves https://github.com/spring-projects/spring-integration/issues/3600

Previously, the `KafkaProducerMessageHandler` did not synchronize a transaction
with some other transaction (e.g. JDBC); it published in a local transaction instead.

Also configure the Gradle Kotlin Daemon to work with JDK 16.

**cherry-pick to 5.4.x**
2021-07-28 11:56:05 -04:00
Artem Bilan
0331933c16 Rely on MProducerSupport.active for Flux (#3423)
* Rely on `MProducerSupport.active` for `Flux`

* Fix `MessageProducerSupport` to extract an `active` flag and set it before
`isRunning` - the `Flux` subscription relies on the `takeWhile()`
where in case of `autoStartup = false` we will never start consume because
it is set to `true` already after `doStart()`
* Refactor all the `MessageProducerSupport` implementation with similar
`active` state to use already one from the super class

**Cherry-pick to 5.3.x**

* * Remove `MessageProducerSupport.setActive()`
to not let to mutate it from the implementations
* Set `active` to `false` in the `destroy()`
* Clean up and fix typos in the affected `JmsMessageDrivenEndpoint`

* * Pull `active` flag down to the `AbstractEndpoint`
* Set `active = true` in the `start()` before calling `doStart()`
* Do same for `active = false` in the `stop()`
* Clean up `AbstractEndpoint` impls to not call `doStart/doStop` for nothing
* Refactor endpoints to rely on the `active` state from the `AbstractEndpoint`
not their own
2020-11-06 13:51:03 -05:00
Gary Russell
3cf66cc539 Fix previous commit 2020-10-19 11:48:18 -04:00
Gary Russell
b6d60bfd69 KPMH - Remove unused parameter 2020-10-19 11:32:33 -04:00
Gary Russell
75d14d32e7 Further Improve Kafka Test Run Time
Configurable `timeoutBuffer`.
2020-10-16 16:28:34 -04:00
Gary Russell
9c6000e453 Kafka Outbound Endpoints - Add futuresChannel
Currently, the only way to block is to set `sync=true` which waits for the future.
The problem with this is it only supports one-at-a-time publication.

Add an option to send the send futures to a channel, allowing the application to
send multiple records and then wait on the futures later.

Also fix long-running test.

* Add docs.
2020-10-16 14:37:37 -04:00
Artem Bilan
c7ff99a4e8 Use LogAccessor from SF
* Change main classes to use a `LogAccessor` API to simplify code flow
* Fix tests according `LogAccessor` property
* Fix some Sonar smells
2020-10-06 13:56:50 -04:00
Artem Bilan
f45206a013 Fix race condition in the MessageDrivenAdapterTests
The `pause` variable is set in the container already after a `consumer.pause()` call.
Therefore it's state might not be changed after acquiring the latch from the mock answer
2020-08-18 11:17:41 -04:00
Artem Bilan
9d557426b5 * Fix new Sonar smells
* Remove redundant `@SuppressWarnings("deprecation")`
* Add `Duration.ofSeconds(10)` to `StepVerifier.verify()`
to avoid infinite wait and lose failing text context on the hang CI build
2020-08-08 12:29:24 -04:00
Gary Russell
3de0445aaa GH-3358: Kafka IB Gateway - implement Pausable
Resolves https://github.com/spring-projects/spring-integration/issues/3358

Also add `isPaused()` to `Pausable`.

* Fix `@since` to the proper version
2020-08-07 15:26:57 -04:00
Gary Russell
1beb854fb4 Remove Legacy Metrics
- Simplify MBeans - instead of wrapping to expose lifecycle methods,
  implement `ManageableLifecycle`. Register an additional MBean for
  polled endpoints to control the lifecycle.

* Polishing

- Move `QueueChannel` `@ManagedAttribute`s to `QueueChannelOperations`
- Make all `AbstractEndpoints` `IntegrationManagedResource`s and remove `ManagedEndpoint`
  to allow exposure of any `@Managed*` methods (including those on `Pausable`)
- Revert to `Lifecycle` for classes that are not related to endpoints
- Remove legacy metrics from docs
2020-08-07 12:56:57 -04:00
Artem Bilan
094eb2e021 Fix new Sonar smells 2020-08-06 10:45:44 -04:00
Gary Russell
acdc25a172 Upgrade to spring-kafka 2.6.0 2020-08-04 12:14:16 -04:00
Artem Bilan
8b16aede89 Fix new Sonar smells 2020-07-16 11:36:04 -04:00
Gary Russell
9c252be028 Fix Kafka Send Timeout
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/928

Kafka has a longer default send timeout; this means a send could be successful long
after Spring has timed out the send.

**I will backport to the 3.3.x extension after merge**

* Use setSendTimeout; make it final.
2020-07-15 10:35:33 -04:00
Artem Bilan
a19e37237b Fix new and some old Sonar smells 2020-07-14 10:49:28 -04:00
Artem Bilan
6e11d4cdf9 Fix new and some old Sonar smells 2020-07-13 14:07:07 -04:00
Gary Russell
9223613ee1 ConsumerRecord Logging Metadata Option 2020-07-10 15:14:49 -04:00
Gary Russell
b860a2e818 Merge Spring Integration Kafka into Main Repo
- merge master branch from old repo (`git-filter-repo`) for src branch
- update build.gradle
- copy docs from Spring for Apache Kafka Chapter

Doc polishing.

Fix http: references in tests.

* Add `What's New` note and fix `polling-consumer.adoc`
not linking to external project any more
2020-06-25 11:46:38 -04:00
Gary Russell
3ec74f380f KMSource - fix RebalanceListener (incremental)
Support cooperative rebalancing
 - incremental assignment/revocation
 - re-pause if a rebalance occurs while paused

**cheerry-pick to 3.2.x**
2020-06-25 11:29:09 -04:00
Artem Bilan
a28647259c Upgrade dependencies; prepare for release 2020-06-25 11:29:09 -04:00
Gary Russell
8771f8e18c Fix tests for latest SK snapshot 2020-06-25 11:29:08 -04:00
Artem Bilan
04019dbb96 Fix checkstyle.xml for Windows new line 2020-06-25 11:29:08 -04:00
Artem Bilan
bfef5a49cf Fix Kotlin tests according upstream changes
* Upgrade dependencies
2020-06-25 11:29:08 -04:00
Gary Russell
213772dcaf Fix S-K Deprecations 2020-06-25 11:29:08 -04:00
Gary Russell
1e484d0fad Support flush after send
* Add `flushExpression`.

* Add DSL, XML configuration.
2020-06-25 11:29:08 -04:00
Artem Bilan
e0be544a40 Add more XSD version into spring.schemas
* Fix some deprecation in `build.gradle`
2020-06-25 11:29:07 -04:00
Gary Russell
a58169f4e1 Fix Gradle Deprecations
* Rename schema

* Upgrade to Gradle 6.3-rc-3 for transitive dependency fix.

* Remove version from schema; polish `build.gradle` to remove unneeded features.
2020-06-25 11:29:07 -04:00
Gary Russell
81b9ae4a7a Use core Kotlin DSL 2020-06-25 11:29:07 -04:00
Gary Russell
0b7ea6d3c2 Remove deprecated methods 2020-06-25 11:29:07 -04:00
Gary Russell
bdd236052b Upgrade to SK 2.5.0
- add deliveryAttempt header when retry is not configured
- temporary work around for gradle bug, jar with test classifier missing from CP
2020-06-25 11:29:07 -04:00
Gary Russell
aaaaf86c07 GH-296: DSL: Support inline pub/sub subscriptions
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/296

* * Add generic type to specsto avoid cast to `BroadcastCapableChannel`.

* * Remove unneeded fields in the abstract spec

* Restore log4j config
2020-06-25 11:29:06 -04:00
Gary Russell
2331eee642 GH-296: Add Kafka-backed MessageChannels
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/296

* * Add DSL support

* * Polishing and XML support
2020-06-25 11:29:06 -04:00
Gary Russell
905a5155bc Master to 3.3; support non-Tx if template allows
* SI to 5.3.0.B-S
2020-06-25 11:29:06 -04:00
Artem Bilan
f596e49242 Upgrade to latest releases
* Upgrade to Gradle 6.1.1
* Fix for latest Checkstyle
* Fix for latest SI-Kotlin-DSL
2020-06-25 11:29:06 -04:00
Artem Bilan
27f0775fb8 Apply SI-Kotlin-DSL in tests (#292)
* Apply SI-Kotlin-DSL in tests

* Upgrade dependencies

* Upgrade to latest releases
2020-06-25 11:29:05 -04:00
Gary Russell
386c64c617 Support ProducerRecord outbound payloads 2020-06-25 11:29:05 -04:00
Gary Russell
cf3aa29a23 GH-290: Add ProducerRecordCreator
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/290
2020-06-25 11:29:05 -04:00
Gary Russell
6cddb2c169 Revert "Apply SI-Kotlin-DSL in tests"
This reverts commit b03ddcb953c18469564994921efb84a072a0e358.
2020-06-25 11:29:05 -04:00
Artem Bilan
9b0b87d37e Apply SI-Kotlin-DSL in tests 2020-06-25 11:29:05 -04:00
Artem Bilan
bf0134cb52 Upgrade to latest GAs
* Move all the test to JUnit 5; remove redundant JUnit 4 dependencies
2020-06-25 11:29:04 -04:00
tomvandenberge
0e74cd1cee Add XML attribute for header-mapper
* Added support for header-mapper to outbound-channel-adapter and
outbound-gateway XML

* Follow up on review

* Corrected mistake

* Corrected indentation
2020-06-25 11:29:04 -04:00
Gary Russell
8cc2b2cc00 KMessageSource getter for consumer properties
- allow customization after creation - e.g. SCSt `MessageSourceCustomizer`
- also fix deprecation in `ReplyingKafkaTemplate`
2020-06-25 11:29:04 -04:00
Gary Russell
adf91edf05 KMSource: Consumer Property Overrides
- support property overrides in the message source
2020-06-25 11:29:04 -04:00
Artem Bilan
f6d858c7aa Polishing JavaDocs and possible NPEs
https://build.spring.io/browse/INTEXT-KAFKA-310/
2020-06-25 11:29:04 -04:00
Gary Russell
3831919cbc GH-282: KMSource - support async commits etc
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/282

Use sync-related properties from `ConsumerProperties`.
2020-06-25 11:29:03 -04:00
Anshul Mehra
c20ef85168 GH-280: Add topicPattern support to KafkaMessageSource
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/280

* Refactored to topics/topicPattern to be a property to
prevent constructor telescoping

* Add support for manual partition assignment

- Cleanup after rebasing from #283
- Add additional tests

* Fix checkstyle

* Remove redundant factory methods

* Add better test for static assignment

* Use ConsumerProperties 

Fix typo where ContainerProperties was used 
instead of ConsumerProperties

* Simplify construction

* Remove unused import

* Use ObjectUtils

* Use component name for clientId

* Collapse static assignment for loop

* Fix lint warning
2020-06-25 11:29:03 -04:00
Anshul Mehra
e52c535f7c Use ConsumerProperties in KafkaMessageSource
* Preserve existing constructors

* Add @deprecated in java docs
2020-06-25 11:29:03 -04:00
Gary Russell
e6cdb8818f GH-279: Fix tests for latest SK Snapshots
- add tests for tx.id overrides
2020-06-25 11:29:03 -04:00