JIRA: https://jira.spring.io/browse/INTEXT-123https://jira.spring.io/browse/INTEXT-124https://jira.spring.io/browse/INTEXT-125
The implementation has the following components:
- Configuration - allows retrieving a seed broker set either from a preconfigured list or from Zookeeper
- ConnectionFactory - manages and caches connections to a set of brokers
- Connection - handles low-level SimpleConsumer API calls and converts them to internal objects such as KafkaMessage and KafkaMessageBus
- KafkaTemplate - implements higher-level read operations on a Kafka broker
- OffsetManager - stores (potentially in a persistent fashion) offsets for a group of consumers, configures/resets initial offsets as necessary - can be configured to start at an arbitrary offset in a partition (e.g. replay since offset 100), or relative to a given timestamp (e.g. replay since Monday)
- KafkaMessageListenerContainer - retrieves messages from a given broker for an arbitrary set of partitions or topics (i.e. all partitions in the topics), invoking a MessageListener. Concurrency is adjustable, and allows processing multiple sets of partitions in parallel (while preserving ordering within a partition). Can poll multiple brokers (each on a parallel thread)
- KafkaInboundChannelAdapter - channel adapter using KMLC as underlying implementation
- AbstractDecodingMessageListener - utility base class for a MessageListener implementation that decodes the payload and key
Tests:
- Single and multi-broker configurations
- Partition subset retrieval
- Replicated sets
- Compression
- Starting offsets (with reset on wrong offset)
- Error handling (servers dropping and leader changes)
TO DO:
- namespace support
Remove warnings
Cleanup
Use ZookeeperConnect for configuring Zookeeper
Fixes, including:
- Extracted Connection and ConnectionFactory interfaces
- Made fields final where necessary
- Fixed Javadoc
- Added assertions
- Corrected headers
Updated copyright to 2015
Make connectivity parameters: timeouts, buffer size, minimum fetch size, fetch timeout configurable via ConnectionFactory. Remove sleep in KafkaMessageListenerContainer Add javadocs.
Addressing PR comments:
- renamed KafkaInboundChannelAdapter to KafkaMessageDrivenChannelAdapter
- removed unnecessary setters
- close ZkClient properly
- fix version
Must use TopicAndPartition as key
Log messages too via LoggingErrorHandler
Newline
- Make `MessageStoreOffsetManager` implement `Closeable` and `Flushable`
- Fix the configuration of `ErrorHandler`
We don't really need this initialization - it's premature optimization
Use ':' instead of ' ' as separator for the key
Fixes:
- Use 'getPhase' from superclass
- more cleanup
Defer to the superclass MessageBuilder factory and keep SI_FATAL_WHEN_NO_BEANFACTORY happy
Fix warnings in tests
Changes to support testing against external brokers
- refactored the rules so that an abstraction can be used
- topic name is configurable
- added (Ignored) TestSingleBrokerExternal that tests against an external broker
- refactored OutboundTest to use the new consumer
Properly catch TopicExistsException
Do not hold state for MetadataStoreOffsetManager
In KafkaMessageListenerContainer FetcherTasks will block if they have no partitions to listen to (to do - use separate monitors per task)
Updated test for offset reset to latest to wait until an actual refresh event takes place, thus preventing the test from not completing in case of a race condition
KafkaMessageDrivenChannelAdapter and KafkaMessageListenerContainer now throw new TopicNotFoundException if the topic does not exist on the broker
Renamed test classes to use the Test suffix.
Cleanup
Test if the component is still running when exiting the wait block
Serialization warnings
Addressing the last round of PR comments
Test configuration cleanup
The final polishing: code style, JavaDocs, renaming for test classes to the finish with `*Tests` suffix
Fixes GH10 https://github.com/spring-projects/spring-integration-kafka/issues/10
Previously the `ConsumerConfiguration` used an internal `ExecutorService`.
In the management environment (App Server) it causes an issue with unmanaged Threads.
* Add `setExecutor` option and expose it to the Namespace to inject a managed `Executor`
* Stop an internal `ExecutorService` manually from the `ConsumerConfiguration.shutdown()`
Polishing
JIRA: https://jira.spring.io/browse/INTEXT-100
Add `topic(-expression)` and `message-key(-expression)` attributes to avoid upstream configuration
to specify them in the `MessageHeaders`
Polishing for XSD
Conflicts:
spring-integration-kafka/src/main/java/org/springframework/integration/kafka/support/KafkaProducerContext.java
spring-integration-kafka/src/main/java/org/springframework/integration/kafka/support/ProducerConfiguration.java
spring-integration-kafka/src/main/resources/org/springframework/integration/config/xml/spring-integration-kafka-1.0.xsd
spring-integration-kafka/src/test/java/org/springframework/integration/kafka/config/xml/KafkaOutboundAdapterParserTests.java
INTEXT-100: Add `Assert.notNull(this.evaluationContext);`
INTEXT-100: Fix parser potential NPE
INTEXT-100: Add `KafkaHeaders`
Merge branch 'INTEXT-100-1' of ..\spring-integration-extensions into INTEXT-100
Conflicts:
src/main/java/org/springframework/integration/kafka/outbound/KafkaProducerMessageHandler.java
src/main/java/org/springframework/integration/kafka/support/KafkaProducerContext.java
src/main/java/org/springframework/integration/kafka/support/ProducerConfiguration.java
src/test/java/org/springframework/integration/kafka/config/xml/KafkaOutboundAdapterParserTests.java
* Polishing according the rebase to `master`
* Polishing for the `KafkaRunning` Rule to use `ZkClient` and its `getAllBrokersInCluster`
* Add a note to the `README.md`
Addressing PR comments
Polishing
- Minor doc polish + change tabs to spaces in code
- Enhance test to include expressions
JIRA: https://jira.spring.io/browse/INTEXT-121
When using `async` producers, the buffered messages need to be flushed.
Previously, when the context was stopped, such messages were lost.
- Implement `SmartLifecyle` in the producer context and propagate the `stop()` to the underlying producer(s).
- Remove unused attributes from the consumer parser.
- Add a test case to show the buffered message is received after closing the producer.
- Add KafkaRunning JUnit `@Rule`
Add KafkaRunning Rule to Parser Tests
Polishing
JIRA: https://jira.spring.io/browse/INT-3570
In 4.1 we removed the 'old' schemas to reduce clutter. However certain
extensions such as kafka import the 4.0 schema which is no longer on
the classpath when using Spring Integration 4.1 when there is no internet
connection.
INT-3570 states that we should restore the 4.0 core schema and mapping
to resolve this issue. In the meantime, this work around adds the schema
to the kafka jar.
It should be reverted when INT-3570 is released in Spring Integration 4.1.1.
JIRA: https://jira.spring.io/browse/INTEXT-111
- Remove hard coded names for bean names at the producer context parser
- Fix and update tests
- Remove the use of `BeanFactory` to get the producerConfigurations in
KakfaProducerContext and use setter to set the producerConfigurations
- Add logic to send the message from producer configuration when there
is no header specified for the topic and single producer configuration
is used.
Add tests to verify placeholder values
Fix code review comment and formatting
Polishing code style.
Add `KafkaProducerContext#theProducerConfiguration` for single `producerConfigurations` entry to avoid iterators
JIRA: https://jira.spring.io/browse/INTEXT-112https://jira.spring.io/browse/INTEXT-111
- Avoid using hardcoded bean names for consumer context
- Remove Integer value topic streams count, instead use string value
- This will allow placeholder values to be used for `streams` attribute
- Set `group-id` as a bean name for consumer configuration bean only if
the group id is specified explicitly. This will allow placeholder values
not being used as a bean name.
- Add test cases to verify multiple consumer configurations within the
same consumer context.
Set consumerConfigurations as propertyValue
Better handling of consumerConfigurations
- Add getConsumerConfiguration(String groupId) in KafkaConsumerContext
- Fix tests
- Add additional test case to check multi consumer contexts
Fix getConsumerConfiguration(groupId)
Polishing
JIRA: https://jira.spring.io/browse/INTEXT-104
`order` attribute now honored if kafka outbound adapter is connected to a subscribable channel
unit tests and samples are updated
Polishing: use `<xsd:attributeGroup ref="integration:channelAdapterAttributes"/>` for adapter tags to cover `SmartLifecycle` options.
JIRA: https://jira.spring.io/browse/INTEXT-110
Perform the necessary API Changes (Partioner interface in Kafka is now non-genericized for Keys - concrete Object is the new type for Key)
Unit test updates for an internal Kafka class (MessageMetadata)
Updates for the samples Kafka project to use the new M2 snapshot, Spring 4.0 Messaging changes, other build related minor changes, etc.
JIRA: https://jira.spring.io/browse/INTEXT-106
* Upgrade to SI 4.0 and other libs
* Upgrade to Gradle 1.12
* Polisihng for `build.gradle`
* Fix compile warnings
* Use new `joinGroup(SocketAddress, NetworkInterface)` API
* Resolve `NetworkInterface` in UDP components as
`getByInetAddress(InetAddress.getByName(localAddress))`
* join group as `socket.joinGroup(new InetSocketAddress(this.group, 0), null);`
* Fix `SocketTestUtils.chooseANic()` to select a real `NetworkInterface`
bound to a real IP address
* Fix `MulticastRule` to populate a `multicast.local.address` against a
host name for the network interface not an interface logical name
* Support Java 14
* Provide changes to avoid deprecated Java API
and have a compatibility back to Java 8
* Change affected test classes to JUnit 5 whenever it is possible
* Ignore/Disable some TCP/IP tests which don't pass on Java 14
* Fix (some) TCP tests on JRE 14
* Fix SSL Handshake test - client side handshake is successful with java 14
- change the badClient cert to a badServer cert to force an error on the client side
Co-authored-by: artembilan <raven666>
Co-authored-by: Gary Russell <grussell@pivotal.io>
Fixes https://github.com/spring-projects/spring-integration/issues/3307
The `DefaultLockRepository.acquire()` is transactional method and
can fail with the `TransactionTimedOutException`.
When we call `JdbcLock.lock()`, we expect an attempt until we really obtain a lock.
* Treat a `TransactionTimedOutException` as a `TransientDataAccessException` and
therefore retry a locking attempt logic
**Cherry-pick to 5.3.x, 5.2.x & 4.3.x**
Resolves https://github.com/spring-projects/spring-integration/issues/3299
Connect before creating the `TcpNioConnection` object and publishing the
`TcpConnectionOpenEvent`.
This was a regression caused by supporting connect timout; which moved
the connect to after the object was created and event published, causing
the `connectionId` to start with `unknown`.
**cherry-pick to 5.3.x, 5.2.x**
* GH-3283: HTTP Inbound handle SpEL errors
Fixes https://github.com/spring-projects/spring-integration/issues/3283
* Process all the request message preparation exceptions
in the provided error channel to let target application
to make a decision about an appropriate HTTP status instead of
default 500 Server Error
* * Rephrase `ResponseStatusException` doc in the http.adoc
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
* GH-3280: NullChannel as reply for void gateways
Fixes https://github.com/spring-projects/spring-integration/issues/3280
To properly support a one-way gateway (`void` return type), it is
better to ignore any possible replies from downstream instead of
unexpected `no output-channel or replyChannel header available`
error
* Populate a `nullChannel` into a `replyChannel` header
for `void` gateway methods when `replyChannel` is not set explicitly
* Remove redundant `GatewayProxyFactoryBean.PARSER` in favor of
similar `EXPRESSION_PARSER` in the super class
* Test and document the feature
* * Fix language in the `whats-new.adoc`
Co-authored-by: Gary Russell <grussell@vmware.com>
* * Fix language in gateway.adoc
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3288
* For better end-user experience with Kotlin DSL and get a gain from
existing `Transformer` implementations add a `transform(Transformer)`
EI-method into the `KotlinIntegrationFlowDefinition`
* Also add `filter(MessageSelector)` for any out-of-the-box `MessageSelector`
**Cherry-pick to 5.3.x**
