Fixes: #8998
Apparently some databases don't throw a proper code to identify a `DuplicateKeyException`, so better to catch `DataIntegrityViolationException` with the same meaning.
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #8983
Since collections in MongoDB cannot be deserialized to custom types anymore,
add a new `DocumentToMessageHistoryConverter` to convert the `Document` with `components`
into a `MessageHistory` instance back
**Auto-cherry-pick to `6.2.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8773
The https://github.com/spring-projects/spring-integration/issues/8732 introduced a filtering for messages in group.
So, plain `removeMessage()` doesn't work any more if message is connected to some group yet.
Therefore, `DelayHandler` is failing.
* Introduce `getMessageFromGroup()` and `removeMessageFromGroupById()` into `MessageGroupStore` API
and implement it respectively in all the stores
* Remove `@LongRunningTest` from delayer integration tests and adjust its config to delay not for a long
**Cherry-pick to `6.1.x`**
* Fix KV Stores for same message in multiple groups
If same message is stored into different groups with the same KV store,
the removal of one group would lead to removal the message for the other one.
* Improve KV Store to save message with the key including a group id
* Respectively, refine the removal API to include group id into keys
* Also change the `MESSAGE_GROUP_KEY_PREFIX` for group records to `GROUP_OF_MESSAGES_`
since the `MESSAGE_` prefix includes group records as well for various operations
based on key pattern
* * Use `[^GROUP_]` in the pattern for messages count
**Cherry-pick to `6.1.x`**
* * Fix MongoDB MS for message removal logic
* * Bring back `GROUP_OF_MESSAGES_` prefix
to avoid complex regexp and don't bother for edge cases,
where even that regexp may fail
Fixes https://github.com/spring-projects/spring-integration/issues/8692
* Added `createIndexes` in `AbstractConfigurableMongoDbMessageStore`
* Added Javadoc for `setCreateIndex()` method
* Removed `afterPropertiesSet()` in `MongoDbChannelMessageStore` and update `whats-new.adoc` and `mongodb.adoc` files
**Cherry-pick to `6.1.x` & `6.0.x`**
* GH-8685: Re-fetch group after setting condition
Fixes https://github.com/spring-projects/spring-integration/issues/8685
The `AbstractCorrelatingMessageHandler` updates the group metadata in DB
not only for provided `condition`, but also a `lastModified` field.
A subsequent scheduling for group timeout takes the `lastModified`
to compare with the value in the store after re-fetching group in task.
This does not reflect reality since adding `condition` modifies the data in DB,
but in-memory state remains the same.
* Re-fetch a group from the store in the `AbstractCorrelatingMessageHandler.setGroupConditionIfAny()`.
* Verify expected behavior via new `ConfigurableMongoDbMessageGroupStoreTests.groupIsForceReleaseAfterTimeoutWhenGroupConditionIsSet()`
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* * Fix Checkstyle violation in the test
* GH-3763: Add `handleReactive()` for Java DSL
Fixes https://github.com/spring-projects/spring-integration/issues/3763
Add a convenient terminal operator to `BaseIntegrationFlowDefinition`
based on a `ReactiveMessageHandler`.
Also add an overload like `handleReactive(ReactiveMessageHandlerSpec)`
to let end-user to choose a protocol-specific channel adapter
* * Fix `Namespace Factory` wording in the `BaseIntegrationFlowDefinition` Javadocs
* Fix language in Docs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
* GH-8586: Deprecate IntegrationComponentSpec.get()
Fixes https://github.com/spring-projects/spring-integration/issues/8586
The `IntegrationComponentSpec` is not a plain wrapper around single component.
Sometimes it comes with several components where all of them must be registered
as beans.
If `IntegrationComponentSpec.get()` is called from end-user code, we may lose
other related components, for example filters in the `FileInboundChannelAdapterSpec`.
* Deprecate `IntegrationComponentSpec.get()` with no-op for end-user,
rather encourage to leave it as is and let the framework take care about its lifecycle
and related components registration
* Fix `IntegrationComponentSpec` logic to deal as a simple `FactoryBean` instead of
extra overhead via `AbstractFactoryBean`
* Use `IntegrationComponentSpec.getObject()` in the framework code where `get()` was called
* Fix tests to expose `IntegrationComponentSpec` as beans instead of previously called `get()`
* Some other clean up and typos fixes in the affected classes
* Document the change
* * Revert `ObjectStringMapBuilder` in the `KafkaInboundGatewaySpec.getComponentsToRegister()`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
* * Remove trailing whitespace in the `ScriptMessageSourceSpec`
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
Looks like sometime Spring Framework XML parser can properly determine the target factory method to chose,
but sometimes it does that in a wrong order.
* Add `type="java.lang.Class"` to the ctor args everywhere for `class="org.mockito.Mockito" factory-method="mock"`
`<bean>` definitions
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
Fixes https://github.com/spring-projects/spring-integration/issues/3839
* Create a new base interface for Mongo container-based tests
* Migrate all tests to JUnit5
* Remove obsolete classes related to JUnit4 Mongo rule
* Fix code style & readability here and there, a few Sonar issues
* Fix failing test `validateWithConfiguredPollerFlow`
* * This test was failing with Mongo error
`$and/$or/$nor must be a nonempty array`. The cause was too small
polling ratio of the update query, the consecutive read queries
failed as no 'Oleg' document was in collection after ~100 ms.
Couple of changes after the pull request review:
* Get back the blank lines in test classes
* Get back the blank lines for some inner classes
* Eliminate bad renaming consequences
* Remove a couple of garbage todos
* Couple of places with code cleanup
Fixes https://github.com/spring-projects/spring-integration/issues/3623
* `IntegrationFlow` refactoring
* Apply several code style improvements and good practices
* Code style: no empty lines for methods javadocs
* make deprecated implementation reuse actual one instead of the copy-paste approach
* add whats-new comments
* Fix whats-new page according to standards
* Upgrade Spring dependencies to the latest SNAPSHOTs
* Fix tests to verify against stack traces: the message
of the `NestedRuntimeException` does not include the nested exception information.
Related to https://github.com/spring-projects/spring-framework/issues/25162
* Fix `JdbcMessageStore` and `DefaultLockRepository` to rely on the `DataIntegrityViolationException`
instead of only its `DuplicateKeyException` extension.
This is the current behavior of the SQL errors translation
* Disable `WebFluxDslTests.testValidation()` - doesn't subscribe to the reply somehow...
* Refine `SimplePool.PoolSemaphore.reducePermits()`
Currently, the `MongoDbAvailableRule` logs a warning about missed availability
of the MongoDb server and returns from the `Statement.evaluate()` making the
test not ignored but as passed
* Fix the exception handling in the `MongoDbAvailableRule` for the `Statement`
to call `Assume.assumeTrue()` instead of the plain logging to mark the test as
ignored.
This will make a JUnit test report looking correctly and also will let Sonar Cube
to report test coverage as adequate
**Cherry-pick to `5.5.x`**
The `MappingMongoConverter` now used different path when it
iterates documents form conversion from a DB cursor
* Fix `MongoDbMessageStore.MessageReadingMongoConverter` to override
a `read(TypeInformation<S>, Bson)` method to convert a `MessageWrapper` properly
* Upgrade to the latest MongoDb driver
* Upgrade to Java 17, SF-6.0, Gradle 7.2
* Upgrade to Jakarta dependencies and respective namespaces
* Fix some tests for Java 17 compatibility
* Fix wrong Javadocs
* Add some missed Javadocs
* Fix more `jakarta` namespace
* Fix WS & XML modules to use Jakarta EE
* `--add-opens` in some modules for their reflection-based tests
* Disable Kafka tests which does not work on Windows; see Apache Kafka `3.0.1`
* Upgrade to JUnit `5.8.1`
* Migrate JMS tests to Artemis
* Remove RMI module as it was deprecated before
* Fix `pr-build-workflow.yml` for Java 17
* Fix JavaDocs warnings using `Xdoclint:syntax` per module, not in the top-level `api` task
* Move docs for version `6.0`
* Fix `MiscellaneousTests` to extend `ActiveMQMultiContextTests`
to let it to close ActiveMQ connection factory in the end of test
* Make `RequestReplyScenariosWithTempReplyQueuesTests` more robust
closing `DefaultMessageListenerContainer` and `ExecutorService`
in the end of tests
* Fix new Sonar smells
* Rework some SQL calls in the `JdbcMessageStore` to note expose a
`PreparedStatement` API
* Fix JavaDoc in the `CorrelationHandlerSpec`
* Introduce a `groupConditionSupplier` for MGS
* Add a `MessageGroup.condition` option
* Add a `MessageGroupStore.conditionSupplier` option
* Use it from the `SimpleMessageStore.addMessagesToGroup()` API
to populate `condition` (if any) into a `MessageGroup`
* Introduce a `GroupConditionProvider` contract to be implemented
on those `ReleaseStrategy` contracts which could be aware of group condition
* Populated a `GroupConditionProvider.getGroupConditionSupplier()`
into a `MessageGroupStore` from the `AbstractCorrelatingMessageHandler`
for end-user convenience
* Rework a `FileMarkerReleaseStrategy` to implement a `GroupConditionProvider`
to provide a function which produces a condition from `file_lineCount` header
of the `END` marker message
* Make the `FileMarkerReleaseStrategy` logic already based on the condition from a group
* Delegate `GroupConditionProvider` from the `FileAggregator`
* Add test for empty file aggregation
* * Implement `condition` in the `AbstractKeyValueMessageStore` and `MongoDbMessageStore`
* Test `condition` for `mongo-aggregator-config.xml` and `FileAggregatorTests` against GemFire
* * Implement `condition` in the `ConfigurableMongoDbMessageStore`
* * Implement `condition` in the `JdbcMessageStore`
* `FileAggregatorTests` against `JdbcMessageStore`
* Refactor `JdbcMessageStore` for better handling of message group metadata
* Remove unused `MARKED` column in the DDL in favor of newly introduced `CONDITION`
* * Add docs for message group condition
* * Move `conditionSupplier` option from MGS to AbstractCorrelatingMH
* Make it as a `BiFunction` to propagate existing condition alongside with the
message to consult
* Expose `groupConditionSupplier` in Java & XML DSLs
* * Fix language in docs
The latest changes in the `MappingMongoConverter` resolves the target type from the document.
That type is a `GenericMessage`.
The `GenericConversionService` doesn't match such a target type for the
`MessageReadConverter implements Converter<Document, Message<?>>` and fail.
* Change the `MessageReadConverter` generic type into the `GenericMessage` for matching data in
the collection
* Add MongoDbMessageSource UPDATE option
* Extract `AbstractMongoDbMessageSource` with common options and methods
for both `MongoDbMessageSource` and `ReactiveMongoDbMessageSource`
* Add an `updateExpression` option into MongoDb source implementations
* Implement respective `update` logic after fetching the data from the collection
* Cover both reactive and blocking updates with tests
* Add `MongoDbMessageSourceSpec` into Java DSL for MongoDb channel adapters
* Expose an `update` XML attribute for the `<int-mongo:inbound-channel-adapter>`
* Upgrade MongoDb driver for latest Spring Data compatibility
* Document a new feature
* Upgrade `mongodb.adoc` for code block switch whenever it is appropriate
* * Add a Kotlin sample for `MongoDb.outboundGateway()` DSL
* Apply suggestions from code review
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3446
* For better resources utilization provide a `Stream<Message<?>>` API
on the `MessageGroupStore`, `MessageGroup` and `MessageGroupQueue`
* Use this API in the `DelayHandler` when it reschedules persisted messages
* INT-3045: Add in & out ZeroMq channel adapters
JIRA: https://jira.spring.io/browse/INT-3045
* Add `ZeroMqMessageHandler` to produce messages into one-way ZeroMq sockets
* Add `ZeroMqMessageProducer` to consumer messages from one-way ZeroMq sockets
* Add `ConvertingBytesMessageMapper` impl for the `BytesMessageMapper` to
delegate an actual conversion into the provided `MessageConverter`
* Add `ZeroMqHeaders` for message headers constants representing ZeroMq message attributes
* Fix `ZeroMqChannel` for the proper deferred `zeroMqProxy` evaluation
* Add more JavaDocs
* Fix `ZeroMqChannelTests.testPubSubBind()` to be sure that really all the
subscribed channels get the same message from the `PUB` socket
* * Fix typo in the `ConvertingBytesMessageMapper`
* Add `this` for `doOnError()` in the `ZeroMqChannel` & `ZeroMqMessageProducer`
* Change the bind logic in the `ZeroMqMessageProducer` to `port` and let it to
bind to random port.
The actual port is available later via `getBoundPort()`
* Introduce a `ZeroMqMessageProducer.receiveRaw()` to let received `ZMsg` to
be produce as a `payload`
* Add a logic into `ZeroMqMessageHandler` to treat `ZMsg` in the payload of
request message as is without any conversion
* Fix race condition in the `ZeroMqMessageProducer` to destroy `consumerScheduler`
when the main `Flux` is complete
* * Add Java DSL for ZeroMq components
* Extract `ReactiveMessageHandlerSpec` for `ReactiveMessageHandler` impls
* Add debug message into `EmbeddedJsonHeadersMessageMapper` when cannot `decodeNativeFormat()`
* Make `ReactiveMongoDbMessageHandlerSpec` extending `ReactiveMessageHandlerSpec`
* Make `ZeroMqProxy` `autoStartup` by default
* Add `ZeroMqDslTests` to cover all the Java DSL for ZeroMq
* Introduce a `MimeTypeSerializer` to serialize a `MimeType` into JSON as a plain string;
use it as extra serializer in the `JacksonJsonUtils.messagingAwareMapper()`
* Fix typo for the `AllowListTypeResolverBuilder` inner class
* * Add some docs
* Fix Checkstyle violations
* * More docs
* Fix language in Docs
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
* GH-3336: Change MongoDb Store sequence to long
Fixes https://github.com/spring-projects/spring-integration/issues/3336
Turns out there are some scenarios where too many messages
are transferred through the message store, so `int` for
sequence is not enough as a type
* Change sequence to `long` to widen a sequence lifespan
* * Change MongoDb store to deal with `Number.longValue()`
instead of casting which doesn't work from `Integer` to `Long`.
This way we can keep an old sequence document with an `int`
type for value
* Documents with new `long` type for their sequence field are OK.
The `NumberToNumberConverter` has an effect converting `int` to `long`
properly.
* The `into()` query can't infer the type for entity if we provide `Object.class`
* The `updateFirst()` does not support sort queries any more - replace with `findAndModify()`
* Add `getMessageGroupCount()` into tests
**Cherry-pick to 5.3.x & 5.2.x**
* Remove redundant `@SuppressWarnings("deprecation")`
* Add `Duration.ofSeconds(10)` to `StepVerifier.verify()`
to avoid infinite wait and lose failing text context on the hang CI build
* Introduce a `ReceiveMessageAdvice`
* Deprecate an `AbstractMessageSourceAdvice` in favor of
`default` method in the `MessageSourceMutator`
* Move a `applyReceiveOnlyAdviceChain()` logic into the `AbstractPollingEndpoint`:
now both `PollingConsumer` and `SourcePollingChannelAdapter` can use
`ReceiveMessageAdvice`
* Introduce a `SimpleActiveIdleReceiveMessageAdvice` based already
on the `ReceiveMessageAdvice` and deprecate a `SimpleActiveIdleMessageSourceAdvice`
which is fully replaceable with newly introduced `SimpleActiveIdleReceiveMessageAdvice`
* Add `@SuppressWarnings("deprecation")` for those out-of-the-box `ReceiveMessageAdvice`
implementation which still use an `AbstractMessageSourceAdvice` for
backward compatibility
* Document a new feature and give the `MessageSourceMutator` a new meaning
* * Fix language in the `polling-consumer.adoc`
Fixes https://github.com/spring-projects/spring-integration/issues/2788
* Introduce a `MessageProducerSupport.subscribeToPublisher(Publisher<Message<?>>)`
for components which produces `Flux` for data from their source
* Such a component is auto-stopped when subscription to that `Publisher` is canceled
* Implement a `MongoDbChangeStreamMessageProducer` based on the reactive support for
in Spring Data MongoDb
* Implement a Java DSL for `MongoDbChangeStreamMessageProducer`
* Disable a test for change stream since it requires server of version 4.x started with 'replSet' option
* Add `MongoHeaders` for change stream events
* Change `MessageProducerSupport` to use a `takeWhile((message) -> isRunning())`
instead of storing a `subscription` from a callback
* Document new features
* Remove trailing whitespaces
* Doc Polishing.
* GH-3155: Add support for Java DSL extensions
Fixes https://github.com/spring-projects/spring-integration/issues/3155
Provide an `IntegrationFlowExtension` for possible custom EI-operators
in the target project use-cases.
* * Move `IntegrationFlowExtension` tests ot its own test class
* Make all the `IntegrationComponentSpec` ctors as `protected` for possible custom extensions
* Make some `BaseIntegrationFlowDefinition` methods and properties as `protected` to get them
access from the `IntegrationFlowExtension` implementations
* Document the feature
* * Fix language and typos in docs
* * Add `protected` to one more `GatewayEndpointSpec` ctor
* Add JavaDocs to `GatewayEndpointSpec` methods
* * Add `protected` to one more `JmsPollableMessageChannelSpec` ctor
* Gradle 6.1.1
* MongoDb 4.0
* Other Spring dependencies to `BUILD-SNAPSHOT`
* Fix MongoDb tests according latest Spring Data
* State in the docs that both MongoDb driver are `optional` now in the dependencies
* Implement `ReactiveMongoDbMessageSourceSpec` and factories for it
* Rework `ReactiveMongoDbMessageSourceTests` to use new Java DSL for `ReactiveMongoDbMessageSource`
* Document changes
