163 Commits

Author SHA1 Message Date
abilan
1093430b78 Fix import in KafkaProducerMessageHandlerTests 2023-06-08 14:59:40 -04:00
Gary Russell
f9f16310af GH-8638: Kafka: Send All Fails to Failure Channel
Resolves https://github.com/spring-projects/spring-integration/issues/8638

Previously, immediate failures (e.g. timeout getting metadata) were
only thrown as exceptions, and not sent to the failure channel, if present.

**cherry-pick to all supported branches**
# Conflicts:
#	spring-integration-kafka/src/main/java/org/springframework/integration/kafka/outbound/KafkaProducerMessageHandler.java
2023-06-08 13:52:04 -04:00
abilan
0cce72a5bc Fix previous imports optimization for XML configs 2023-01-09 17:34:36 -05:00
abilan
aab52048f5 Realign imports code style with main 2023-01-09 17:11:07 -05:00
abilan
e63edde7cb GH-3966: Kafka XML config: Expose more attributes
Fixes https://github.com/spring-projects/spring-integration/issues/3966

Exposed setters in the `KafkaInboundGateway` and `KafkaMessageDrivenChannelAdapter`
as an XML attributes for `kafka` namespace

**Cherry-pick to `5.5.x`**
2022-12-12 16:05:31 -05:00
Artem Bilan
b6b649396b Fix KafkaOutAdapterParserTests race condition
The `Future` for `Producer.send()` can be fulfilled before
we return it (`isDone()`), so `KafkaTemplate` short-circuits
with a `KafkaException`.

* Modify `KafkaOutboundAdapterParserTests.testSyncMode()`
to assert against `KafkaException` which may be thrown earlier
then previously expected `KafkaProducerException`

**Cherry-pick to `5.5.x`**
2022-07-18 13:06:20 -04:00
Artem Bilan
eec3e956fe GH-3734: Support MessageHistory on Kafka deser
Fixes https://github.com/spring-projects/spring-integration/issues/3734

The `MessageHistory` is not in the trusted packages of the `DefaultKafkaHeaderMapper`
therefore it fails when `MessageHistory.read()` is performed.

* Configure default `DefaultKafkaHeaderMapper` in the `SubscribableKafkaChannel`,
`KafkaInboundGateway`, `KafkaMessageDrivenChannelAdapter` and `KafkaMessageSource`
to also trust packages exposed via `JacksonJsonUtils.DEFAULT_TRUSTED_PACKAGES`
which include a `MessageHistory`, too.

* Verify in some integration Kafka tests to be sure that `MessageHistory` is
deserialized properly in the transferred headers
* Rework some tests to use `@EmbeddedKafka` instead of `@BeforeAll/@AfterAll`

**Cherry-pick to `5.5.x`**
2022-03-22 16:02:08 -04:00
Gary Russell
e2557a42c2 GH-3617: KPMH - Option To Use Template's Converter
Resolves https://github.com/spring-projects/spring-integration/issues/3617

Optionally use the `KafkaTemplate` message converter instead of the default
(or supplied) `ProducerRecordCreator`.
2021-10-06 16:10:57 -04:00
Artem Bilan
0fb64f1762 Upgrade to Kotlin 1.5
* To align with the rest of Spring portfolio, based on Spring Framework 5.3.x generation,
it is better to upgrade to the Kotlin version supported by SF
* Fix  `ClassUtils.isLambda()` to treat `$inlined$` classes as Kotlin lambdas and let
Java DSL to delegate to the `LambdaMessageProcessor` for better expected type handling.
Looks like Kotlin does not carry type info in its lambdas anymore (at least for functions).
* Add `-parameters` to the compiler args for possible method argument names discovery.
Essentially realign with the rest of portfolio
* Fix Kotlin tests for deprecated API
2021-09-13 16:46:05 -04:00
Gary Russell
1b7d622777 GH-3600: Fix Kafka Tx Synchronization
Resolves https://github.com/spring-projects/spring-integration/issues/3600

Previously, the `KafkaProducerMessageHandler` did not synchronize a transaction
with some other transaction (e.g. JDBC); it published in a local transaction instead.

Also configure the Gradle Kotlin Daemon to work with JDK 16.

**cherry-pick to 5.4.x**
2021-07-28 11:54:31 -04:00
Gary Russell
95fce11b0a GH-3573: Add ConsumerRecord(s) to ConversionExcep.
Resolves https://github.com/spring-projects/spring-integration/issues/3573
2021-06-03 12:47:19 -04:00
Artem Bilan
eef31d4b34 Fix new Sonar smells; optimize some tests 2021-02-24 11:34:29 -05:00
Artem Bilan
9c686ddcb7 More JVM memory for Gradle
* Fix race condition in the `KafkaOutboundAdapterParserTests`
2021-02-23 11:28:04 -05:00
Artem Bilan
ad57c6187a Fix MessageSourceTest for the latest spring-kafka
Related to: https://github.com/spring-projects/spring-kafka/issues/1659
2020-12-21 17:31:45 -05:00
Gary Russell
75d14d32e7 Further Improve Kafka Test Run Time
Configurable `timeoutBuffer`.
2020-10-16 16:28:34 -04:00
Gary Russell
9c6000e453 Kafka Outbound Endpoints - Add futuresChannel
Currently, the only way to block is to set `sync=true` which waits for the future.
The problem with this is it only supports one-at-a-time publication.

Add an option to send the send futures to a channel, allowing the application to
send multiple records and then wait on the futures later.

Also fix long-running test.

* Add docs.
2020-10-16 14:37:37 -04:00
Artem Bilan
c7ff99a4e8 Use LogAccessor from SF
* Change main classes to use a `LogAccessor` API to simplify code flow
* Fix tests according `LogAccessor` property
* Fix some Sonar smells
2020-10-06 13:56:50 -04:00
Artem Bilan
f45206a013 Fix race condition in the MessageDrivenAdapterTests
The `pause` variable is set in the container already after a `consumer.pause()` call.
Therefore it's state might not be changed after acquiring the latch from the mock answer
2020-08-18 11:17:41 -04:00
Gary Russell
3de0445aaa GH-3358: Kafka IB Gateway - implement Pausable
Resolves https://github.com/spring-projects/spring-integration/issues/3358

Also add `isPaused()` to `Pausable`.

* Fix `@since` to the proper version
2020-08-07 15:26:57 -04:00
Gary Russell
acdc25a172 Upgrade to spring-kafka 2.6.0 2020-08-04 12:14:16 -04:00
Gary Russell
9c252be028 Fix Kafka Send Timeout
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/928

Kafka has a longer default send timeout; this means a send could be successful long
after Spring has timed out the send.

**I will backport to the 3.3.x extension after merge**

* Use setSendTimeout; make it final.
2020-07-15 10:35:33 -04:00
Artem Bilan
6e11d4cdf9 Fix new and some old Sonar smells 2020-07-13 14:07:07 -04:00
Gary Russell
b860a2e818 Merge Spring Integration Kafka into Main Repo
- merge master branch from old repo (`git-filter-repo`) for src branch
- update build.gradle
- copy docs from Spring for Apache Kafka Chapter

Doc polishing.

Fix http: references in tests.

* Add `What's New` note and fix `polling-consumer.adoc`
not linking to external project any more
2020-06-25 11:46:38 -04:00
Gary Russell
3ec74f380f KMSource - fix RebalanceListener (incremental)
Support cooperative rebalancing
 - incremental assignment/revocation
 - re-pause if a rebalance occurs while paused

**cheerry-pick to 3.2.x**
2020-06-25 11:29:09 -04:00
Gary Russell
8771f8e18c Fix tests for latest SK snapshot 2020-06-25 11:29:08 -04:00
Artem Bilan
bfef5a49cf Fix Kotlin tests according upstream changes
* Upgrade dependencies
2020-06-25 11:29:08 -04:00
Gary Russell
1e484d0fad Support flush after send
* Add `flushExpression`.

* Add DSL, XML configuration.
2020-06-25 11:29:08 -04:00
Gary Russell
81b9ae4a7a Use core Kotlin DSL 2020-06-25 11:29:07 -04:00
Gary Russell
bdd236052b Upgrade to SK 2.5.0
- add deliveryAttempt header when retry is not configured
- temporary work around for gradle bug, jar with test classifier missing from CP
2020-06-25 11:29:07 -04:00
Gary Russell
aaaaf86c07 GH-296: DSL: Support inline pub/sub subscriptions
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/296

* * Add generic type to specsto avoid cast to `BroadcastCapableChannel`.

* * Remove unneeded fields in the abstract spec

* Restore log4j config
2020-06-25 11:29:06 -04:00
Gary Russell
2331eee642 GH-296: Add Kafka-backed MessageChannels
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/296

* * Add DSL support

* * Polishing and XML support
2020-06-25 11:29:06 -04:00
Gary Russell
905a5155bc Master to 3.3; support non-Tx if template allows
* SI to 5.3.0.B-S
2020-06-25 11:29:06 -04:00
Artem Bilan
f596e49242 Upgrade to latest releases
* Upgrade to Gradle 6.1.1
* Fix for latest Checkstyle
* Fix for latest SI-Kotlin-DSL
2020-06-25 11:29:06 -04:00
Artem Bilan
27f0775fb8 Apply SI-Kotlin-DSL in tests (#292)
* Apply SI-Kotlin-DSL in tests

* Upgrade dependencies

* Upgrade to latest releases
2020-06-25 11:29:05 -04:00
Gary Russell
386c64c617 Support ProducerRecord outbound payloads 2020-06-25 11:29:05 -04:00
Gary Russell
6cddb2c169 Revert "Apply SI-Kotlin-DSL in tests"
This reverts commit b03ddcb953c18469564994921efb84a072a0e358.
2020-06-25 11:29:05 -04:00
Artem Bilan
9b0b87d37e Apply SI-Kotlin-DSL in tests 2020-06-25 11:29:05 -04:00
Artem Bilan
bf0134cb52 Upgrade to latest GAs
* Move all the test to JUnit 5; remove redundant JUnit 4 dependencies
2020-06-25 11:29:04 -04:00
tomvandenberge
0e74cd1cee Add XML attribute for header-mapper
* Added support for header-mapper to outbound-channel-adapter and
outbound-gateway XML

* Follow up on review

* Corrected mistake

* Corrected indentation
2020-06-25 11:29:04 -04:00
Gary Russell
8cc2b2cc00 KMessageSource getter for consumer properties
- allow customization after creation - e.g. SCSt `MessageSourceCustomizer`
- also fix deprecation in `ReplyingKafkaTemplate`
2020-06-25 11:29:04 -04:00
Gary Russell
adf91edf05 KMSource: Consumer Property Overrides
- support property overrides in the message source
2020-06-25 11:29:04 -04:00
Gary Russell
3831919cbc GH-282: KMSource - support async commits etc
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/282

Use sync-related properties from `ConsumerProperties`.
2020-06-25 11:29:03 -04:00
Anshul Mehra
c20ef85168 GH-280: Add topicPattern support to KafkaMessageSource
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/280

* Refactored to topics/topicPattern to be a property to
prevent constructor telescoping

* Add support for manual partition assignment

- Cleanup after rebasing from #283
- Add additional tests

* Fix checkstyle

* Remove redundant factory methods

* Add better test for static assignment

* Use ConsumerProperties 

Fix typo where ContainerProperties was used 
instead of ConsumerProperties

* Simplify construction

* Remove unused import

* Use ObjectUtils

* Use component name for clientId

* Collapse static assignment for loop

* Fix lint warning
2020-06-25 11:29:03 -04:00
Anshul Mehra
e52c535f7c Use ConsumerProperties in KafkaMessageSource
* Preserve existing constructors

* Add @deprecated in java docs
2020-06-25 11:29:03 -04:00
Gary Russell
e6cdb8818f GH-279: Fix tests for latest SK Snapshots
- add tests for tx.id overrides
2020-06-25 11:29:03 -04:00
Urs Keller
4cd81fda78 Fix NPE for retryTemp!=null but recoveryCall==null
Starting with version `3.0.x`, the `KafkaInboundGateway` and
`KafkaMessageDrivenChannelAdapter` rely on the
`RetryContext.getRetryCount()` for the
`IntegrationMessageHeaderAccessor.DELIVERY_ATTEMPT` message header, but
the `RetryContext` is obtained from the `TheadLocal` `attributesHolder`,
which is populated from the `RetryListener.open()` only if
`recoveryCallback != null`

* Fix `KafkaInboundGateway` and `KafkaMessageDrivenChannelAdapter` to
populate `TheadLocal` `attributesHolder` with the `RetryContext`
from the `RetryListener.open()` when only `retryTemplate != null`

**Cherry-pick to `master` & 3.0.x**

# Conflicts:
#	src/test/java/org/springframework/integration/kafka/inbound/MessageDrivenAdapterTests.java
2020-06-25 11:29:02 -04:00
Gary Russell
41de7495e6 Resolve deprecations and other compiler warnings 2020-06-25 11:29:02 -04:00
Cameron Mayfield
600e88ec56 GH-272: Add KafkaMDrivenChAdapterSpec.payloadType
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/272

Add `payloadType` option into `KafkaMessageDrivenChannelAdapterSpec`
2020-06-25 11:29:02 -04:00
Gary Russell
a1b51c9f47 Fix tests for SK 2.3.0.M3 2020-06-25 11:29:02 -04:00
Gary Russell
14302088c5 SIGH-2958: Add source record to common SI Header
Resolves https://github.com/spring-projects/spring-integration/issues/2958

Enables applications to be agnostic regarding the source of
a message. e.g Kafka Vs. AMQP.
2020-06-25 11:29:01 -04:00