Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,342 Commits 25 Branches 0 Tags
d96a2d930c500f05abda87df68870bc147b85256
Commit Graph

10342 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Artem Bilan
d96a2d930c Add HTTPS entries for XSD into spring.schemas 
Related to spring-projects/spring-integration#2987

**Cherry-pick to 3.1.x, 3.0.x & 2.3.x**
 2020-06-25 11:29:02 -04:00
Gary Russell
41de7495e6 Resolve deprecations and other compiler warnings 2020-06-25 11:29:02 -04:00
Cameron Mayfield
600e88ec56 GH-272: Add KafkaMDrivenChAdapterSpec.payloadType 
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/272

Add `payloadType` option into `KafkaMessageDrivenChannelAdapterSpec`
 2020-06-25 11:29:02 -04:00
Gary Russell
a1b51c9f47 Fix tests for SK 2.3.0.M3 2020-06-25 11:29:02 -04:00
Gary Russell
bc443474cb Pausable package change 2020-06-25 11:29:01 -04:00
Gary Russell
14302088c5 SIGH-2958: Add source record to common SI Header 
Resolves https://github.com/spring-projects/spring-integration/issues/2958

Enables applications to be agnostic regarding the source of
a message. e.g Kafka Vs. AMQP.
 2020-06-25 11:29:01 -04:00
Gary Russell
c25b3b45f4 Allow fetching multiple records 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/267

- add an option to allow fetching multiple records per poll

* DSL for new constructor arg
 2020-06-25 11:29:01 -04:00
Anshul Mehra
cd6d17e4b0 GH-265: Ensure assignTimeout is used when no partitions are assigned 
* Refactored to rely on assignedPartitions for determining whether to
use pollTimeout or assignTimeout
* Removed redundant assigned flag

fixes #265
 2020-06-25 11:29:01 -04:00
Gary Russell
28fc819fe9 Change MessageSource.payloadType to Class<?> 
- consistency with other endpoints
- avoid the need for SpEL in XML config
 2020-06-25 11:29:01 -04:00
Gary Russell
22b1e70e13 GH-195: Add XML Schema support for new components 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/195

- gateways
- message source

Polishing

Don't start adapters in parser tests

Fix XML filename for test
 2020-06-25 11:29:00 -04:00
Gary Russell
f59a3687a9 Fix race in KafkaProducerMessageHandlerTests. 2020-06-25 11:29:00 -04:00
Gary Russell
e9833af732 Fix javadoFix javadoc 2020-06-25 11:29:00 -04:00
Gary Russell
e76fd2bea2 Remove Lifecyle since it's the super of Pausable 2020-06-25 11:29:00 -04:00
Gary Russell
ca125a8a6f Polishing - PR comments. 2020-06-25 11:29:00 -04:00
Gary Russell
087247d96f GH-259: Add pause/resume to KafkaMessageSource 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/259

Enable pause/resume on the polled consumer.

**cherry-pick to 3.1.x**
 2020-06-25 11:28:59 -04:00
Gary Russell
ceb68b40ef GH-260: Fix outbound tx synchronization 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/260

The outbound endpoint used
`TransactionSynchronizationManager.isActualTransactionActive()`
to detect a global transaction instead of
`TransactionSynchronizationManager.getResource(this.kafkaTemplate.getProducerFactory()) == null`.

The KTM uses `SYNCHRONIZATION_NEVER` by default so the first will
return false unless the synchronization is explicitly set. This
causes a new transaction to be started.

Use the presence of the resource holder instead.

**cherry-pick to 3.1.x**
 2020-06-25 11:28:59 -04:00
Artem Bilan
cc3895f40d Fix XSD for the current version 2020-06-25 11:28:59 -04:00
Artem Bilan
eb7f94ae0d URL Cleanup 
This commit updates URLs to prefer the https protocol.
Redirects are not followed to avoid accidentally expanding intentionally
shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed Success
These URLs were switched to an https URL with a 2xx status.
While the status was successful, your review is still recommended.

* [ ] http://www.apache.org/licenses/ with 1 occurrences migrated to:
  https://www.apache.org/licenses/ ([https](https://www.apache.org/licenses/) result 200).
* [ ] http://www.apache.org/licenses/LICENSE-2.0 with 28 occurrences migrated to:
  https://www.apache.org/licenses/LICENSE-2.0 ([https](https://www.apache.org/licenses/LICENSE-2.0) result 200).
 2020-06-25 11:28:59 -04:00
Gary Russell
71cdeee0b3 GH-235: KPMH: Flush the template in stop() 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/235

**cherry-pick to 3.1.x**
 2020-06-25 11:28:58 -04:00
Spring Operator
4f54c4cab3 URL Cleanup 
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed But Review Recommended
These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended.

* [ ] http://help.github.com/send-pull-requests (404) with 1 occurrences migrated to:
  https://help.github.com/send-pull-requests ([https](https://help.github.com/send-pull-requests) result 404).

## Fixed Success
These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended.

* [ ] http://kafka.apache.org/ with 1 occurrences migrated to:
  https://kafka.apache.org/ ([https](https://kafka.apache.org/) result 200).
* [ ] http://www.apache.org with 1 occurrences migrated to:
  https://www.apache.org ([https](https://www.apache.org) result 200).
* [ ] http://www.springframework.org/schema/integration/spring-integration.xsd with 1 occurrences migrated to:
  https://www.springframework.org/schema/integration/spring-integration.xsd ([https](https://www.springframework.org/schema/integration/spring-integration.xsd) result 200).
* [ ] http://static.springsource.org/spring-integration/reference (301) with 1 occurrences migrated to:
  https://docs.spring.io/spring-integration/reference ([https](https://static.springsource.org/spring-integration/reference) result 301).
* [ ] http://www.springframework.org with 1 occurrences migrated to:
  https://www.springframework.org ([https](https://www.springframework.org) result 301).
* [ ] http://www.springsource.com with 2 occurrences migrated to:
  https://www.springsource.com ([https](https://www.springsource.com) result 301).

# Ignored
These URLs were intentionally ignored.

* http://www.springframework.org/schema/beans with 1 occurrences
* http://www.springframework.org/schema/integration with 2 occurrences
* http://www.springframework.org/schema/integration/kafka with 2 occurrences
* http://www.springframework.org/schema/tool with 2 occurrences
* http://www.w3.org/2001/XMLSchema with 1 occurrences
 2020-06-25 11:28:58 -04:00
Spring Operator
9b670950a5 URL Cleanup 
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed But Review Recommended
These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended.

* http://www.puppycrawl.com/dtds/configuration_1_2.dtd (404) with 1 occurrences migrated to:
  https://www.puppycrawl.com/dtds/configuration_1_2.dtd ([https](https://www.puppycrawl.com/dtds/configuration_1_2.dtd) result 404).
* http://www.puppycrawl.com/dtds/suppressions_1_1.dtd (404) with 1 occurrences migrated to:
  https://www.puppycrawl.com/dtds/suppressions_1_1.dtd ([https](https://www.puppycrawl.com/dtds/suppressions_1_1.dtd) result 404).

## Fixed Success
These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended.

* http://www.springframework.org/schema/beans/spring-beans.xsd with 2 occurrences migrated to:
  https://www.springframework.org/schema/beans/spring-beans.xsd ([https](https://www.springframework.org/schema/beans/spring-beans.xsd) result 200).
* http://www.springframework.org/schema/context/spring-context.xsd with 1 occurrences migrated to:
  https://www.springframework.org/schema/context/spring-context.xsd ([https](https://www.springframework.org/schema/context/spring-context.xsd) result 200).
* http://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd with 2 occurrences migrated to:
  https://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd ([https](https://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd) result 200).
* http://www.springframework.org/schema/integration/spring-integration.xsd with 1 occurrences migrated to:
  https://www.springframework.org/schema/integration/spring-integration.xsd ([https](https://www.springframework.org/schema/integration/spring-integration.xsd) result 200).

# Ignored
These URLs were intentionally ignored.

* http://www.springframework.org/schema/beans with 4 occurrences
* http://www.springframework.org/schema/context with 2 occurrences
* http://www.springframework.org/schema/integration with 2 occurrences
* http://www.springframework.org/schema/integration/kafka with 4 occurrences
* http://www.w3.org/2001/XMLSchema-instance with 2 occurrences

* Upgrade Checkstyle and its DTDs for HTTPS
 2020-06-25 11:28:58 -04:00
Gary Russell
7fae571c2d Changes for compatibility with Spring Kafka 2.3 
Also remove test for bad partition in DSL tests - the send now gives an error
about the topic not being present in metadata instead of "not in range".

Use sync for publishing in DSL tests, to fail fast.
 2020-06-25 11:28:58 -04:00
Artem Bilan
daedfb9210 Upgrade dependencies and fix issues 2020-06-25 11:28:58 -04:00
Gary Russell
e5b861dcea Remove unnecessary throws Exception clause 2020-06-25 11:28:57 -04:00
Gary Russell
1873e9ae41 GH-211: MessageSource acks; fix concurrency 
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/211

Use `ConcurrentHashMap` with synchronized set values to avoid concurrency issues
when acks are processed on multiple threads.
 2020-06-25 11:28:57 -04:00
Gary Russell
759e0e236d Nullable fixes 2020-06-25 11:28:57 -04:00
Gary Russell
19c3c4607f Fix TODOs in outbound channel adapter 
- always use `successChannel` for metadata
- unwrap `ExecutionException`
 2020-06-25 11:28:57 -04:00
Artem Bilan
f29bd3e322 GH-212: Add ConsumerSeekAware impl to Inbounds (#213) 
* GH-212: Add ConsumerSeekAware impl to Inbounds

Fixes spring-projects/spring-integration-kafka#212

**Cherry-pick to 3.0.x**

* GH-212: Add ConsumerSeekAware impl to Inbounds

Fixes spring-projects/spring-integration-kafka#212

* Introduce a new `IntegrationKafkaHeaders.CONSUMER_SEEK_CALLBACK`
header to be populated to messages for sending to the channel
* Populate that header from the `KafkaInboundGateway` and
`KafkaMessageDrivenChannelAdapter` into the message from the
`seekCallBack` property if `ListenerContainer` is single-threaded or
from the `ThreadLocal<ConsumerSeekAware.ConsumerSeekCallback>` otherwise;
and only if newly introduced `setAdditionalHeaders` is `true`
* Populate `seekCallBack` property or `ThreadLocal<?>` from the
`registerSeekCallback()` implementation from the internal listeners
* Add `setOnPartitionsAssignedSeekCallback(BiConsumer)` and
`setOnIdleSeekCallback(BiConsumer)` options to react for the appropriate
event from the underlying container and perform appropriate seek management
* Add new options to the DSL classes and cover them with tests, including
check for new `IntegrationKafkaHeaders.CONSUMER_SEEK_CALLBACK` header

**Cherry-pick to 3.0.x**

* Address PR comments: remove unnecessary API

* *Polishing `setOnPartitionsAssignedSeekCallback()` JavaDocs
*Close producers in the `KafkaProducerMessageHandlerTests`
 2020-06-25 11:28:57 -04:00
Gary Russell
bedc0c8fdc GH-209: Upgrade to kafka-clients 2.0.0 
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/209

* README Polishing
 2020-06-25 11:28:56 -04:00
Artem Bilan
0c1cfbc977 Upgrade to SI-5.1 M1, SK-2.2 M1 
* Fixes according the upgrades
* Polishing some code style
 2020-06-25 11:28:56 -04:00
Gary Russell
8decbe7810 GH-204: DSL Register container as bean if needed 
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/204

When an external container is provided to the DSL, register it as a bean
if it is not already a bean.

Polishing id from PR comments; add gateway support too.

* Polishing JavaDocs and omissions in the test-case
 2020-06-25 11:28:56 -04:00
Artem Bilan
e296d66a4b Kotlin compatibility: hide recursive generics 
Related to https://jira.spring.io/browse/INT-4458

**Cherry-pick to 3.0.x**
 2020-06-25 11:28:56 -04:00
Gary Russell
1e006c9de7 Fix imports 2020-06-25 11:28:55 -04:00
Gary Russell
c807ed5335 Fix typo in warning log. 2020-06-25 11:28:55 -04:00
Artem Bilan
1be6a6a3fa Start version 3.1 for compatibility with SK-2.2 2020-06-25 11:28:55 -04:00
Gary Russell
fc3f33671b GH-198: Support pause/resume on inbound adapter 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/198

You can now pause/resume the consumer.

Any records previously fetched will be processed before the pause takes effect.
The listener container requires an `idleEventInterval`, a resume will take effect
on the next `ListenerContainerIdleEvent`.

* Delegate Pausable methods to listener container

Requires https://github.com/spring-projects/spring-kafka/pull/584

* Polishing
 2020-06-25 11:28:55 -04:00
Gary Russell
d13551e6c3 Revert "GH-198: Support pause/resume on inbound adapter" 
This reverts commit 7befa83ba87058540f624ec1bf33fcabf054f3b5.

Temporary revert; pushed to 3.0.3
 2020-06-25 11:28:55 -04:00
Gary Russell
17ea7dace7 GH-198: Support pause/resume on inbound adapter 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/198

You can now pause/resume the consumer.

Any records previously fetched will be processed before the pause takes effect.
The listener container requires an `idleEventInterval`, a resume will take effect
on the next `ListenerContainerIdleEvent`.

* Delegate Pausable methods to listener container

Requires https://github.com/spring-projects/spring-kafka/pull/584

* Polishing
 2020-06-25 11:28:54 -04:00
Gary Russell
7b7bcf1014 Fix Race in InboundGatewayTests 
The debug log indicates that the reply was successfully sent but never
received. My working theory is that the reply consumer got the reply
during the `poll()` to force the subscription.

Start the consumers before the replies are sent.

I will also add code to `KafkaEmbedded` to seek to beginning all topics
if necessary.
 2020-06-25 11:28:54 -04:00
Gary Russell
98a7c4241d Trace logging for InboundGatewayTests 2020-06-25 11:28:54 -04:00
Gary Russell
accd2d9088 Debug logging for InboundGatewayTests 2020-06-25 11:28:54 -04:00
Gary Russell
bd774de5c0 Support transaction initiation in ob endpoints 
* Use template.isTransactional()
 2020-06-25 11:28:54 -04:00
Gary Russell
2b292f2a7d GH-161: Implement Gateways 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/161

Initial commit - outbound

TODO: inbound, DSL, XML, Docs

Fix reply topic.

Polishing - don't leak internal headers; don't propagate outbound headers (topic etc.).

Polishing

Validate reply topic and partition against replying template.

DSL

Checkstyle fixes.

Add inbound gateway.

Inbound DSL

Polishing - PR comments.

Polishing - PR comments

More polishing

Increase test timeouts.

Increase test timeouts.

* Code style polishing
* Fix `KafkaInboundGatewaySpec.getComponentsToRegister()` to `get()`
on Specs
* Remove `getContainer()` in the `KafkaMessageListenerContainerSpec` in
favor of `get()` in the `IntegrationComponentSpec`
 2020-06-25 11:28:53 -04:00
Mark
86d8373084 Polish synchronization in KafkaMessageSource 
* Remove volatile from running flag

As running is always accessed through `KafkaMessageSource` monitor use existing `synchronized` to guard access to it.
Also move setting `running=true` from consumerMonitor in `createConsumer()` method to `doReceive()` as it's guarded by needed lock

* Extract common stopping consumer logic
 2020-06-25 11:28:53 -04:00
Gary Russell
a997740c29 More Lifecycle Polishing 
- `null` the consumer on `stop()`
- set `running` to true on first receive if needed
 2020-06-25 11:28:53 -04:00
Gary Russell
28b4dd814c Remove pause/resumer; add Lifecycle 
Pause/resume was too chatty - user can avoid a rebalance with `max.poll.interval.ms`.

Implement `Lifecyle` and `close()` the consumer on `stop()`.
 2020-06-25 11:28:53 -04:00
Gary Russell
d738447005 GH-189: MessageSource - remove seek after poll 
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/189

If the DKCFactory is being used, force the `max.poll.records` property to 1.
Otherwise, throw an `IllegalArgumentException`.
 2020-06-25 11:28:53 -04:00
Gary Russell
23bc29958c GH-184: Polishing - user rebalance listener 
Needed by the SCST Kafka Binder

* GH-184: Polishing - add raw header option
 2020-06-25 11:28:52 -04:00
Gary Russell
b44dab40fa GH-184: Add Polled MessageSource 
Resolves https://github.com/spring-projects/spring-integration-kafka/issues/184

Polishing - changes in core

Pause/Resume; Add integration tests

Synchronize all consumer operations on the consumer, to support noAutoAck()

Polishing - PR Comments

More polishing

Major rework - see commit comment

- only fetch one record at a time (warn log and seek if max.poll.records incorrect)
- when an application has multiple outstanding records,
 - track records; apply commits in the right order, deferring if necessary
 - when requeued, mark later offsets as rolled back to prevent commits
- Support transactions
 - start/end transactions
 - provide access to the producer

Fix transaction synchronization

- if there is an existing Kafka transaction, participate in it
- if not, bind a new transactional resource to the thread - allows a KafkaTemplate to
  participate by utilizing the producer factory message header
- if there is an existing non-kafka transaction, sync the kafka transaction with it

Check for KafkaMessageHeaders when building the message.

Polishing - PR Comments

Simplify AckInfo

Remove internal transaction support - the user can start a transacion before calling receive
and send offsets to the transaction using a KafkaTemplate.

Polishing

* Simple polishing according IDEA warnings
* Upgrade to Gradle 4.4.1
* Some upgrades and dependencies polishing
 2020-06-25 11:28:52 -04:00
Gary Russell
29973e525d GH-185: Add deliveryAttempts header with retry 
Resolves: https://github.com/spring-projects/spring-integration-kafka/issues/185

When a `RetryTemplate` is wired into the message-driven adapter, add
and increment an `AtomicInteger`-valued header.

See https://jira.spring.io/browse/INT-4369

* Actualize Copyright in the affected classes
 2020-06-25 11:28:52 -04:00