Fixes#79 (https://github.com/spring-projects/spring-integration-kafka/pull/79)
Adds an implementation for offset management based on Kafka's own offset management support
Update copyright and use fill name in author tag.
Refactoring the native offset manager
Remove unused imports
Polishing
Adds two fields to the producer-configuration that allows to set the send mode (async or sync) and an optional timeout in sync mode. Default behavior is async.
This allows for an upstream application to verify that the message have been handed off to Kafka following the delivery options specified, and not just residing in memory at the producer.
JIRA: https://jira.spring.io/browse/INTEXT-175
Since `IntegrationEvaluationContextAware` infrastructure had had wrong design and has been deprecated in the SI-4.2, change its usage to the proper way according to the SI-core recipes.
Tested against Spring IO-2.0.
- revert to using a custom KafkaMessageHeaders class directly;
- avoid the use of MutableMessageHeaders and the subsequent assignment of ids and timestamps within a GenericMessageTemplate
JIRA: https://jira.spring.io/browse/INTEXT-168
- Replace producers with the new producer API
- Change KafkaProducerContext and ProducerConfiguration to match the new Producer's behaviour (i.e. providing partition as argument)
- support using the `partitionId` message header for targetting specific partitions;
- allow using a SpEL expression for partitioning;
- remove configurations that do not apply anymore, i.e. async (new producer is always async), batch size as message count, etc.
- add new configuration options wherever necessary;
- tweak schema - allow reuse of Encoders and Partitioners if available;
TODO: Add missing unit tests;
Fixes and polishing
JIRA: https://jira.spring.io/browse/INTEXT-165
- retry fetching metadata until either successful or container stopped;
- do not stop the fetch task on connection errors, and handle it as a case when all partitions have moved;
- test broker stopped case + test enhancements for supporting broker restart;
Unused imports
Removed sysout calls from framework code
Reduced socket timeout, to improve timing on failover tests
JIRA: https://jira.spring.io/browse/INTEXT-156https://jira.spring.io/browse/INTEXT-157
- `KafkaMessageListenerContainer#stop()` returns only after the container has been effectively stopped, i.e. no more messages are dispatched to the listeners;
- add a configurable timeout property to prevent the previous operation from blocking indefinitely;
- create separate XSDs for 1.0 and 1.1 schemas;
Corrections after review
- removed 1.0 schema
- removed time-sensitive tests
Corrections after review
- removed 1.0 schema
- removed time-sensitive tests
- removed unused logger in QueueingMessageListenerInvoker
Clear messages after stopping
Clear partitionsByBrokersMap
Polishing imports and properties order
Using a regular HashMap instead of GS Collections' regular map improves performance, as the latter is more expensive to create.
Cleanup - put references back
JIRA: https://jira.spring.io/browse/INTEXT-152
- upgrade to Spring Integration 4.1 and Spring 4.1 for MessageHeaders subclassing support;
- by default, the KafkaMessageDrivenChannelAdapter will create an optimized version of KafkaMessage;
- add settings for avoiding expensive ID and timestamp generation by default, with the option of turning them on;
- add a setting for reverting to superclass MessageBuilderFactory where that is necessary;
* JavaDocs polishing
JIRA: https://jira.spring.io/browse/INTEXT-153
- ensure that the fetch loop is exited immediately after the listener container is stopped;
- do not dispatch messages for processing once components are stopped;
- Introduce an Acknowledgment object that message processors can invoke;
- Introduce an AcknowedgingMessageListener variation of the MessageListener that receives an Acknowledgment reference for the processed message;
- Add 'autoCommitOffset' settings to the KafkaMessageListenerContainer and KafkaMessageDrivenChannelAdapter, and the ability to inject a MessageListener or an AcknowledgingMessageListener in either (the allowed type depending on the offset management strategy)
- Prepopulate a message header for SI messages created by the KafkaMessageDrivenChannelAdapter if autoCommit is enabled or disabled;
- OffsetManager only sets values that are higher than the ones already set in a session (barring reset) - this is to prevent asynchrous acks to mistakenly revert checkpoints
Addressed PR comments
- removed autoOffsetCommit flag for the KafkaMessageListenerContainer, relying only on the messageListener type to detect one versus the other;
- added since tags
- added copyright
@since tags
Polishing
JIRA: https://jira.spring.io/browse/INTEXT-131, https://jira.spring.io/browse/INTEXT-134
Add deletion support to OffsetManagers
- Separate MetadataStore-related functionality and extract an AbstractOffsetManager to handle resetting/retrieving default offsets from Kafka
- Move connection parameters to the Configuration class
- Add deletion support to OffsetManagers
- Introduce KafkaOffsetManager
Renamed to TopicUtils
Use an embedded broker for outbound tests.
INTEXT-135 Dynamic producing to topics in the KafkaProducingMessageHandler
Rework how DefaultConnectionFactory manages cached metadata
-Renamed PartitionBrokerMap to a more apt MetadataCache
-Removed bidirectional mapping
-MetadataCache is still immutable but now allows to create new instances with new TopicMetadata
-As a result, DefaultConnectionFactory can load topic data as required, without requiring explicit resets
- DefaultConnectionFactory will refresh internally if data about a topic is not available, transparently retrieving data from the server
Corrections after PR comments:
- use a readLock for getLeaders as well
- renamed `refreshLeaders` to `refreshMetadata`
- various minor fixes
Double lock check
Make AbstractOffsetManager Disposable an close on that
Corrections, logs, etc.
INTEXT-131-134: Polishing
* Add `initialOffsets` "free" `AbstractOffsetManager`
* Get rid of `connectionFactory` ctor argument for the `KafkaTopicOffsetManager`
* `start/stop` checks to the `KafkaMessageDrivenChannelAdapterWithKafkaOffsetManagerTests`
* Add note to the `README.md` about `KafkaTopicOffsetManager`
JIRA: https://jira.spring.io/browse/INTEXT-123https://jira.spring.io/browse/INTEXT-124https://jira.spring.io/browse/INTEXT-125
The implementation has the following components:
- Configuration - allows retrieving a seed broker set either from a preconfigured list or from Zookeeper
- ConnectionFactory - manages and caches connections to a set of brokers
- Connection - handles low-level SimpleConsumer API calls and converts them to internal objects such as KafkaMessage and KafkaMessageBus
- KafkaTemplate - implements higher-level read operations on a Kafka broker
- OffsetManager - stores (potentially in a persistent fashion) offsets for a group of consumers, configures/resets initial offsets as necessary - can be configured to start at an arbitrary offset in a partition (e.g. replay since offset 100), or relative to a given timestamp (e.g. replay since Monday)
- KafkaMessageListenerContainer - retrieves messages from a given broker for an arbitrary set of partitions or topics (i.e. all partitions in the topics), invoking a MessageListener. Concurrency is adjustable, and allows processing multiple sets of partitions in parallel (while preserving ordering within a partition). Can poll multiple brokers (each on a parallel thread)
- KafkaInboundChannelAdapter - channel adapter using KMLC as underlying implementation
- AbstractDecodingMessageListener - utility base class for a MessageListener implementation that decodes the payload and key
Tests:
- Single and multi-broker configurations
- Partition subset retrieval
- Replicated sets
- Compression
- Starting offsets (with reset on wrong offset)
- Error handling (servers dropping and leader changes)
TO DO:
- namespace support
Remove warnings
Cleanup
Use ZookeeperConnect for configuring Zookeeper
Fixes, including:
- Extracted Connection and ConnectionFactory interfaces
- Made fields final where necessary
- Fixed Javadoc
- Added assertions
- Corrected headers
Updated copyright to 2015
Make connectivity parameters: timeouts, buffer size, minimum fetch size, fetch timeout configurable via ConnectionFactory. Remove sleep in KafkaMessageListenerContainer Add javadocs.
Addressing PR comments:
- renamed KafkaInboundChannelAdapter to KafkaMessageDrivenChannelAdapter
- removed unnecessary setters
- close ZkClient properly
- fix version
Must use TopicAndPartition as key
Log messages too via LoggingErrorHandler
Newline
- Make `MessageStoreOffsetManager` implement `Closeable` and `Flushable`
- Fix the configuration of `ErrorHandler`
We don't really need this initialization - it's premature optimization
Use ':' instead of ' ' as separator for the key
Fixes:
- Use 'getPhase' from superclass
- more cleanup
Defer to the superclass MessageBuilder factory and keep SI_FATAL_WHEN_NO_BEANFACTORY happy
Fix warnings in tests
Changes to support testing against external brokers
- refactored the rules so that an abstraction can be used
- topic name is configurable
- added (Ignored) TestSingleBrokerExternal that tests against an external broker
- refactored OutboundTest to use the new consumer
Properly catch TopicExistsException
Do not hold state for MetadataStoreOffsetManager
In KafkaMessageListenerContainer FetcherTasks will block if they have no partitions to listen to (to do - use separate monitors per task)
Updated test for offset reset to latest to wait until an actual refresh event takes place, thus preventing the test from not completing in case of a race condition
KafkaMessageDrivenChannelAdapter and KafkaMessageListenerContainer now throw new TopicNotFoundException if the topic does not exist on the broker
Renamed test classes to use the Test suffix.
Cleanup
Test if the component is still running when exiting the wait block
Serialization warnings
Addressing the last round of PR comments
Test configuration cleanup
The final polishing: code style, JavaDocs, renaming for test classes to the finish with `*Tests` suffix
Fixes GH10 https://github.com/spring-projects/spring-integration-kafka/issues/10
Previously the `ConsumerConfiguration` used an internal `ExecutorService`.
In the management environment (App Server) it causes an issue with unmanaged Threads.
* Add `setExecutor` option and expose it to the Namespace to inject a managed `Executor`
* Stop an internal `ExecutorService` manually from the `ConsumerConfiguration.shutdown()`
Polishing
JIRA: https://jira.spring.io/browse/INTEXT-100
Add `topic(-expression)` and `message-key(-expression)` attributes to avoid upstream configuration
to specify them in the `MessageHeaders`
Polishing for XSD
Conflicts:
spring-integration-kafka/src/main/java/org/springframework/integration/kafka/support/KafkaProducerContext.java
spring-integration-kafka/src/main/java/org/springframework/integration/kafka/support/ProducerConfiguration.java
spring-integration-kafka/src/main/resources/org/springframework/integration/config/xml/spring-integration-kafka-1.0.xsd
spring-integration-kafka/src/test/java/org/springframework/integration/kafka/config/xml/KafkaOutboundAdapterParserTests.java
INTEXT-100: Add `Assert.notNull(this.evaluationContext);`
INTEXT-100: Fix parser potential NPE
INTEXT-100: Add `KafkaHeaders`
Merge branch 'INTEXT-100-1' of ..\spring-integration-extensions into INTEXT-100
Conflicts:
src/main/java/org/springframework/integration/kafka/outbound/KafkaProducerMessageHandler.java
src/main/java/org/springframework/integration/kafka/support/KafkaProducerContext.java
src/main/java/org/springframework/integration/kafka/support/ProducerConfiguration.java
src/test/java/org/springframework/integration/kafka/config/xml/KafkaOutboundAdapterParserTests.java
* Polishing according the rebase to `master`
* Polishing for the `KafkaRunning` Rule to use `ZkClient` and its `getAllBrokersInCluster`
* Add a note to the `README.md`
Addressing PR comments
Polishing
- Minor doc polish + change tabs to spaces in code
- Enhance test to include expressions
JIRA: https://jira.spring.io/browse/INTEXT-121
When using `async` producers, the buffered messages need to be flushed.
Previously, when the context was stopped, such messages were lost.
- Implement `SmartLifecyle` in the producer context and propagate the `stop()` to the underlying producer(s).
- Remove unused attributes from the consumer parser.
- Add a test case to show the buffered message is received after closing the producer.
- Add KafkaRunning JUnit `@Rule`
Add KafkaRunning Rule to Parser Tests
Polishing
JIRA: https://jira.spring.io/browse/INT-3570
In 4.1 we removed the 'old' schemas to reduce clutter. However certain
extensions such as kafka import the 4.0 schema which is no longer on
the classpath when using Spring Integration 4.1 when there is no internet
connection.
INT-3570 states that we should restore the 4.0 core schema and mapping
to resolve this issue. In the meantime, this work around adds the schema
to the kafka jar.
It should be reverted when INT-3570 is released in Spring Integration 4.1.1.
