Fixes: #9481
Issue link: https://github.com/spring-projects/spring-integration/issues/9481
MySQL throws `CannotAcquireLockException` in case of duplicate key failure.
PostgreSQL just rollbacks a transaction not letting us move on with a `SELECT`
* Include `TransientDataAccessException` to the catch block of the `INSERT`
to ignore it for the subsequent `SELECT`
* Add logic to determine `PostgreSQL` database vendor and include `ON CONFLICT DO NOTHING`
hint into the `INSERT` to not fail in case of duplicate key found on `putIfAbsent` operation
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9381
Currently, there is no way to know in one place what Control Bus commands are available and with what arguments
* Add `ControlBusCommandRegistry` infrastructure bean to gather control bus commands from beans and expose them for invocation
* Add `ControlBusController` to expose a `/control-bus` REST service against the mentioned `ControlBusCommandRegistry`
* Add `@EnableIntegrationManagement(loadControlBusCommands)` to be able to load all the Control Bus commands from the application context instead of on demand by default
* Deprecated existing SpEL(and Groovy)-based Control Bus functionality in favor of new, more manageable, logic
Fixes: #9291
* Modify `unlock()` method of `JdbcLock`: if the lock ownership can not be removed due to data expiration, a `ConcurrentModificationException` should be thrown.
* Modify `unlock()` method of `RedisLock`: if the lock ownership can not be removed due to data expiration, a `ConcurrentModificationException` should be thrown.
* Maintain test cases
* Document the new behavior
Fixes: #9192
With modern drivers we don't need BLOB-specific handling anymore.
The regular `PreparedStatement.setBytes()` and `ResultSet.getBytes()`
are enough for our serialized messages
Fixes: #9061
`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover.
With this change the connection is renewed when notifications are not received for a certain time.
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9050
Turns out not all JDBC drivers support a `LocalDateTime` type conversion.
* Use `Timestamp.valueOf(LocalDateTime)` for `TIMESTAMP` params of the queries
in the `DefaultLockRepository`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #8998
Apparently some databases don't throw a proper code to identify a `DuplicateKeyException`, so better to catch `DataIntegrityViolationException` with the same meaning.
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8773
The https://github.com/spring-projects/spring-integration/issues/8732 introduced a filtering for messages in group.
So, plain `removeMessage()` doesn't work any more if message is connected to some group yet.
Therefore, `DelayHandler` is failing.
* Introduce `getMessageFromGroup()` and `removeMessageFromGroupById()` into `MessageGroupStore` API
and implement it respectively in all the stores
* Remove `@LongRunningTest` from delayer integration tests and adjust its config to delay not for a long
**Cherry-pick to `6.1.x`**
* GH-8770: Add `PostgresSubsChannel.errorHandler`
Fixes https://github.com/spring-projects/spring-integration/issues/8770
The problem with the `PostgresSubscribableChannel.notifyUpdate()` is that the try-catch block is outside the loop,
so the loop will die on an exception, leaving further messages unprocessed.
* Add ``PostgresSubscribableChannel.errorHandler` option to be invoked
after a `RetryTemplate` and for every failed message.
* The `askForMessage()` new logic is to catch an exception on a message and call `errorHandler`
returning a `FALLBACK_STUB` to continue an outer loop in the `notifyUpdate()`
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Rename private `PostgresSubscribableChannel.askForMessage()` method to more specific `pollAndDispatchMessage()`
The `PostgresSubscribableChannel` uses a task executor
for dispatching messages.
Even if we stop `PostgresChannelMessageTableSubscriber` and
unsubscribe from the channel, the task might be ongoing.
* Use explicit `ThreadPoolTaskExecutor` in the test
to shout it down and wait for tasks to be completed before
verifying DB status
* Optimize `PostgresSubscribableChannel` to mark TX for rollback
when we got a message from DB, but no handlers subscribed
Fixes https://github.com/spring-projects/spring-integration/issues/8760
* Make `PostgresChannelMessageStoreQueryProvider` to use single `DELETE ... RETURNING` for polling statements
* Add `isUsingSingleStatementForPoll` and use it from `JdbcChannelMessageStore`
* Execute Postgres init scripts to `PostgresContainerTest`
* Code clean up
* Document the new feature
Fixes https://github.com/spring-projects/spring-integration/issues/8748
The Oracle DB throws `ORA-08177: can't serialize access for this transaction`
when other transaction on the row has begun
* Change the isolation for `DefaultLockRepository.acquire()` transaction
to the `READ_COMMITTED` for what database automatically and silently
restarts the entire SQL statement, and no error occurs.
* Add `oracle` dependencies to JDBC module
* Introduce `OracleContainerTest` and implement it for `OracleLockRegistryTests`
**Cherry-pick to `6.1.x` & `6.0.x`**
* Fix Checkstyle violations for `private` inner classes where they have to be also `final`
* Fix deprecations after upgrades
* Remove manual coroutines handling in the `GatewayProxyFactoryBean`
in favor of the same functionality in the `JdkDynamicProxy`
The `ExpirableLockRegistry.expireUnusedOlderThan()` uses a query to the target store by mistake.
The logic of this API is indeed about the local cache for those lock instances.
We cannot effect any other processes with this cache.
And even if we remove our local instance while it is locked in other process,
that doesn't mean that on the next `obtain()` call, when we got a fresh local instance, we will be able to acquire a lock in target store.
* Fix `isAcquiredInThisProcess()` to check only `localLock.isLocked()`
**Cherry-pick to `6.1.x` & `6.0.x`**
When transaction is configured for the `PostgresSubscribableChannel.notifyUpdate()`
and it is rolled back, the next poll in that loop will return the same message.
Again and again if transaction is always rolled back.
This leads to the condition when we never leave this loop even if
we fully unsubscribed from this channel.
The issue has need spotted after introducing `SKIP LOCKED`
for `PostgresChannelMessageStoreQueryProvider` which leads to
the locked record in DB in the mentioned above transaction.
* Introduce `PostgresSubscribableChannel.hasHandlers` flag
to check in the `notifyUpdate()` before performing poll query in DB.
**Cherry-pick to `6.1.x` & `6.0.x`**
* GH-8732 Don't remove JDBC message if other groups
Fixes https://github.com/spring-projects/spring-integration/issues/8732
When same message is added into different groups,
its record in the `INT_MESSAGE` must remain until the last group is removed
* Improve `JdbcMessageStore.DELETE_MESSAGES_FROM_GROUP` SQL
to ignore those messages for removal which has other group records in the `INT_GROUP_TO_MESSAGE` table
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* * Fix `JdbcMessageStore.removeMessage()` and `removeMessagesFromGroup()`
to remove from `INT_MESSAGE` only if there is no other records in `INT_GROUP_TO_MESSAGE`
* * Improve `MessageStore.removeMessage()` Javadoc
mentioning `MessageGroupStore` specifics
* Add a version for deprecation on `AbstractChannelMessageStoreQueryProvider`
* Remove dead links from `ChannelMessageStoreQueryProvider` impl JavaDocs
* Use text blocks for long query strings
* Move `SELECT_COMMON` constant to the `ChannelMessageStoreQueryProvider` interface
* Move methods from `AbstractChannelMessageStoreQueryProvider` to `default` in the `ChannelMessageStoreQueryProvider` interface.
* Remove usage of this deprecated method.
* Fix `message-store.adoc` to mention `ChannelMessageStoreQueryProvider` instead of deprecated abstract class.
Fixes https://github.com/spring-projects/spring-integration/issues/8711
* Add `SKIP LOCKED` into `MySqlChannelMessageStoreQueryProvider` and `PostgresChannelMessageStoreQueryProvider`
* Stop the `postgresChannelMessageTableSubscriber` after the latch waiting 10s has been passed in `testMessagesDispatchedInTransaction` test case
* GH-8680: Check DB for table on start
Fixes https://github.com/spring-projects/spring-integration/issues/8680
If database is not initialized properly before application start, we may lose messages
at runtime when we fail to insert data into DB
* Implement `SmartLifecycle` on `JdbcMessageStore`, `JdbcChannelMessageStore`,
`JdbcMetadataStore`, and `DefaultLockRepository` to perform `SELECT COUNT` query in `start()`
to fail fast if no required table is present.
* Refactor `AbstractJdbcChannelMessageStoreTests` into JUnit 5 and use `MySqlContainerTest`
for more coverage
* Fix newly failed tests which had DB not initialized before
* Exclude `commons-logging` from `commons-dbcp2` dependency to avoid
classpath conflict
* Document the new feature
* * Fix HTTP URL in the `DataSource-mysql-context.xml`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
* * Add `setCheckDatabaseOnStart(false)` to disable the check query for all the SI JDBC components
* Fix language in Javadocs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
* GH-8642: Revise executors in the project
Fixes https://github.com/spring-projects/spring-integration/issues/8642
* Rework some `Executors.newSingleThreadExecutor()` to `ExecutorServiceAdapter(new SimpleAsyncTaskExecutor())`
* Expose `TaskExecutor` setters; deprecate `ExecutorService`-based
* Some other code clean up in the effected classes: `LogAccessor`, no `synchronized` in critical blocks
* Give a meaningful prefix for default threads in the context of components, e.g. `SubscribableRedisChannel` - `getBeanName() + "-"`
* * Fix `PostgresChannelMessageTableSubscriberTests` for
`PostgresSubscribableChannel` initialization to let it create
its internal `Executor`
* Use an `AsyncTaskExecutor` injection instead of `ExecutorServiceAdapter` wrapping
* Fix `LockRegistryLeaderInitiatorTests` for `taskExecutor` injection
* Bring back `LockRegistryLeaderInitiator.setExecutorService()`
as an accident after property auto-renaming
* Fix race condition in the `JdbcPollingChannelAdapterParserTests`
making SELECT and UPDATE as a part of the same transaction
* Remove `inProcess = JAVA_EXEC` from ASCIIDoc Gradle tasks in attempt
to make them working in parallel
Fixes https://github.com/spring-projects/spring-integration/issues/8623
Turns out not all JDBC drivers (or RDBMS vendors) support `java.time.Instant`
mapping to their `TIMESTAMP` type.
For example the PostgreSQL fails like:
```
org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of java.time.Instant.
```
* Use `LocalDateTime.now(ZoneOffset.UTC)` instead `Instant.now()`.
Essentially bringing back the behavior from the previous version
Fixes https://github.com/spring-projects/spring-integration/issues/3866
Some RDBMS vendors (or their JDBC drivers) may just log the problem
without throwing an exception.
* Expose setters for UPDATE and INSERTS queries in the `DefaultLockRepository`
to let end-user to modify them respectively, e.g. be able to add a PostgreSQL
`ON CONFLICT DO NOTHING` hint.
* Refactor `DefaultLockRepository` to `Instant` instead of `LocalDateTime`
with zone offset.
* Refactor `ttl` property to `Duration` type
* Fix `dead-lock` typo to `deadlock`
Fixes https://github.com/spring-projects/spring-integration/issues/8582
* Introduce a `PostgresSubscribableChannel.setTransactionManager()`
to wrap a message polling and dispatching operation into a transaction
* In addition add a `RetryTemplate` support around transaction attempts
**Cherry-pick to `6.0.x`**
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
Looks like sometime Spring Framework XML parser can properly determine the target factory method to chose,
but sometimes it does that in a wrong order.
* Add `type="java.lang.Class"` to the ctor args everywhere for `class="org.mockito.Mockito" factory-method="mock"`
`<bean>` definitions
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
* Removed overridden `getMethodValue()` in the `CookieTests` which is removed from SF already
* Re-enable JDBC tests related to complex named params because of the fix in SF
* `--add-opens` for Tomcat warnings
* Upgrade to `log4j-slf4j2-impl`
* Add `cassandra.yaml` resource with increased timeouts to wait more in busy environment from Cassandra cluster
* INT-3333: Return empty list as is from DB gateway
Fixes https://jira.spring.io/browse/INT-3333
In `JdbcOutboundGateway` and `JpaOutboundGateway` the empty result list
is treated as "no reply" and therefore `null` is returned cause
the flow to stop at this point.
It is better to return such a result as is and the target application to
decided what to do with it, e.g. a `discardChannel` on a downstream splitter
configuration.
NOTE: the `MongoDbOutboundGateway` doesn't treat an empty result as a `null`
* * Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3872
Adds an implementation for a Postres-compatible notification listener for a `JdbcChannelMessageStore`.
* Introduce `PostgresChannelMessageTableSubscriber.Subscription` contract
* Implement a `PostgresSubscribableChannel`
* Add Javadoc and fix code formatting issues.
* Handle temporary connection loss.
* Add tests for `PostgresChannelMessageTableSubscriber`
* Replace NOTIFY command with pg_notify function call.
* Code style clean up
* Fix some Javadocs
* Use `DataSourceInitializer` in the `PostgresChannelMessageTableSubscriberTests` to populate scripts
* Use Java text block for DB scripts
