Since the latest Spring Kafka introduces `RECEIVED_` headers on the receiving adapter,
we don't have clashes any more with the `kafka_topic`, `kafka_partition` headers on the producing adapter
Since we are in the `2.0` code line it is safe to just remove the `enableHeaderRouting` even if it is a breaking change.
Doc Polishing
JIRA: https://jira.spring.io/browse/INTEXT-220
Add `WindowingOffsetManagerTests`
Rework `WindowingOffsetManager` to be "resettable" on `flush()`.
* `flush()` `WindowingOffsetManager` on each `getOffset()`, `resetOffsets()` and `deleteOffset()`.
Exactly for the `RingBufferProcessor` processor completion to allow to have the access to "real" offset with those operations over `delegate`.
* Since Reactor's `Stream` isn't restartable create an new instance recursively from the `onComplete()`
JIRA: https://jira.spring.io/browse/INTEXT-216
Having the default value for the `NamedComponent` contract (`beanName`)
doesn't allow us to override (generate) the unique bean name for the component.
Clean up `KafkaProducerContext.beanName` to `null` by default.
**Cherry-pick to 1.3.x**
Fixes https://github.com/spring-projects/spring-integration-kafka/issues/70
* Refactor the message listener container to finish tasks on broker connection failure
and spawn new ones whenever a new broker is discovered;
* Add an explicit disconnect operation on the ConnectionFactory to release the connection
on failure;
* Other improvements and test refactorings;
Optimize the fetch logic
Add test for recognizing new brokers
Formatting
Formatting and refactoring
Test improvements, polishing
Prevent the KafkaMessageListenerContainer from blocking during shutdown
If the RingBufferProcessor used by the KafkaMessageListenerContainer blocks during
processing, it may be prevented from handling the onConsume() notfication which is
the shutdown indicator. Thus, the stopping of the component may be effectively
delayed.
To solve this, the processing task of the RingBufferProcessor is wrapped in a
cancelable task that is managed by one of the container components, and canceled
on shutdown, allowing for the interruption of the processing thread. Subsequently,
all messages cached by the ringbuffer are discarded, without being processed or
acknowdledged.
Additionally, an option for acknowledging processed messages on success only is
added. This allows the component to run continuously and advance even in the case
of sporadic errors, but to resume from the last successfully processed message.
Polishing
JIRA: https://jira.spring.io/browse/INTEXT-206
The previous logic in the `ConcurrentMessageListenerDispatcher` create exactly `consumers`
number of `QueueingMessageListenerInvoker`, although we use only the `partitions.size()`.
Fix the logic to the `this.consumers = Math.min(partitions.size(), consumers);` to create maximum
consumers according to the partitions count.
Introduces the concept of ProducerListener, that can be injected in a ProducerContext and receive notifications when a message has been acknowledged or rejected.
While the Kafka API currently supports providing a Callback, it requires the creation of a distinct instance for receiving contextual information about the message that has been sent or rejected. This allows the use of the Callback mechanism with an injectable ProducerListener strategy.
Polishing and Callback Integration Test
Fixes#79 (https://github.com/spring-projects/spring-integration-kafka/pull/79)
Adds an implementation for offset management based on Kafka's own offset management support
Update copyright and use fill name in author tag.
Refactoring the native offset manager
Remove unused imports
Polishing
Adds two fields to the producer-configuration that allows to set the send mode (async or sync) and an optional timeout in sync mode. Default behavior is async.
This allows for an upstream application to verify that the message have been handed off to Kafka following the delivery options specified, and not just residing in memory at the producer.
JIRA: https://jira.spring.io/browse/INTEXT-175
Since `IntegrationEvaluationContextAware` infrastructure had had wrong design and has been deprecated in the SI-4.2, change its usage to the proper way according to the SI-core recipes.
Tested against Spring IO-2.0.
- revert to using a custom KafkaMessageHeaders class directly;
- avoid the use of MutableMessageHeaders and the subsequent assignment of ids and timestamps within a GenericMessageTemplate
JIRA: https://jira.spring.io/browse/INTEXT-168
- Replace producers with the new producer API
- Change KafkaProducerContext and ProducerConfiguration to match the new Producer's behaviour (i.e. providing partition as argument)
- support using the `partitionId` message header for targetting specific partitions;
- allow using a SpEL expression for partitioning;
- remove configurations that do not apply anymore, i.e. async (new producer is always async), batch size as message count, etc.
- add new configuration options wherever necessary;
- tweak schema - allow reuse of Encoders and Partitioners if available;
TODO: Add missing unit tests;
Fixes and polishing
JIRA: https://jira.spring.io/browse/INTEXT-165
- retry fetching metadata until either successful or container stopped;
- do not stop the fetch task on connection errors, and handle it as a case when all partitions have moved;
- test broker stopped case + test enhancements for supporting broker restart;
Unused imports
Removed sysout calls from framework code
Reduced socket timeout, to improve timing on failover tests
JIRA: https://jira.spring.io/browse/INTEXT-156https://jira.spring.io/browse/INTEXT-157
- `KafkaMessageListenerContainer#stop()` returns only after the container has been effectively stopped, i.e. no more messages are dispatched to the listeners;
- add a configurable timeout property to prevent the previous operation from blocking indefinitely;
- create separate XSDs for 1.0 and 1.1 schemas;
Corrections after review
- removed 1.0 schema
- removed time-sensitive tests
Corrections after review
- removed 1.0 schema
- removed time-sensitive tests
- removed unused logger in QueueingMessageListenerInvoker
Clear messages after stopping
Clear partitionsByBrokersMap
Polishing imports and properties order
Using a regular HashMap instead of GS Collections' regular map improves performance, as the latter is more expensive to create.
Cleanup - put references back
JIRA: https://jira.spring.io/browse/INTEXT-152
- upgrade to Spring Integration 4.1 and Spring 4.1 for MessageHeaders subclassing support;
- by default, the KafkaMessageDrivenChannelAdapter will create an optimized version of KafkaMessage;
- add settings for avoiding expensive ID and timestamp generation by default, with the option of turning them on;
- add a setting for reverting to superclass MessageBuilderFactory where that is necessary;
* JavaDocs polishing
