Fixes: #3444
* Add `DistributedLock` interface which is implemented by `RedisLock` and `JdbcLock`.
* Modify `LockRegistry`, `ExpirableLockRegistry`, `RenewableLockRegistry` interfaces.
* Modify implementation of `DefaultLockRepository, `JdbcLockRegistry`, `RedisLockRegistry`
* Modify ddl of `INT_LOCK` table.
* Maintain test cases and documents.
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Modify `LockRegistry`, `DistributedLock` interfaces.
* Modify implementation of `DefaultLockRepository, `JdbcLockRegistry`, `RedisLockRegistry`
* Modify ddl of `INT_LOCK` table.
* Maintain test cases and documents.
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Modify DistributedLock` interfaces.
* Modify implementation of `JdbcLockRegistry`, `RedisLockRegistry`.
* Modify JavaDoc.
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Maintain document
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Add @Deprecated
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
Follow up for: https://github.com/spring-projects/spring-integration/pull/10081
* Rename `StoredProcExecutorConfigurer` to `StoredProcExecutorSpec`
* Add `Jdbc.storedProcExecutorSpec()` factory
* Improve logic of `StoredProcExecutor`-based Java DSL classes to handle an external `StoredProcExecutor` as well
* Add `package-info.java` into the `jdbc/dsl` package
* Add `/jdbc/dsl.adoc` and respective `What's New` entry
* Rearrange `JavaCompile` and `KotlinCompilationTask` Gradle options
* Comment out `Werror` for `Javadoc` tasks to avoid build failure
* Disable `UdpChannelAdapterTests.testMulticastReceiver()` since it fails somehow on Java `23` & `24`
* Disable `MySqlTxTimeoutMessageStoreTests.testInt3181ConcurrentPolling()` since it is not stable
Fixes: https://github.com/spring-projects/spring-integration/issues/6380
The `MapSqlParameterSource` is much faster, then reflection or SpEL,
so, it would be great to have such an interaction when we evaluate values for SQL queries
* Enhance `BeanPropertySqlParameterSourceFactory` to use `MapSqlParameterSource` if `input` is a `Map`
* Expose `JdbcMessageHandler.usePayloadAsParameterSource` for convenience with `SqlParameterSourceFactory`,
especially when the payload is a map
Adding two messages into a group may end up with wrong insert order.
Or better to say the timestamp of first messages inserted might be behind in the future after the second one.
This leads to a wrong order return for polling operation and, therefore, verification in the test.
* Fix `PostgresChannelMessageTableSubscriberTests` inserting `Thread.sleep(100);`
between `messageStore.addMessageToGroup()` operations
Fixes: https://github.com/spring-projects/spring-integration/issues/5123
When `RedisMessageStore`, for example, adds and removes messages, it operates on two keys separately, which may cause problems in multi-threading due to non-atomic operations.
Although using Redis to delay messages is not a good idea, the abnormal loss of messages in the logs alerted me when the number of requests was not large.
By comparing the logs, the problem that the message group representing the metadata is not consistent with the actual message.
A simple solution is to add lock like in the `SimpleMessageStore`, which is also the approach taken in this pull request.
* Add `LockRegistry` to `AbstractMessageGroupStore`
* Normalize access levels and method name about the lock of `MessageGroupStore`
* Add document about the lock of `AbstractMessageGroupStore`
Fixes: #9481
Issue link: https://github.com/spring-projects/spring-integration/issues/9481
MySQL throws `CannotAcquireLockException` in case of duplicate key failure.
PostgreSQL just rollbacks a transaction not letting us move on with a `SELECT`
* Include `TransientDataAccessException` to the catch block of the `INSERT`
to ignore it for the subsequent `SELECT`
* Add logic to determine `PostgreSQL` database vendor and include `ON CONFLICT DO NOTHING`
hint into the `INSERT` to not fail in case of duplicate key found on `putIfAbsent` operation
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9381
Currently, there is no way to know in one place what Control Bus commands are available and with what arguments
* Add `ControlBusCommandRegistry` infrastructure bean to gather control bus commands from beans and expose them for invocation
* Add `ControlBusController` to expose a `/control-bus` REST service against the mentioned `ControlBusCommandRegistry`
* Add `@EnableIntegrationManagement(loadControlBusCommands)` to be able to load all the Control Bus commands from the application context instead of on demand by default
* Deprecated existing SpEL(and Groovy)-based Control Bus functionality in favor of new, more manageable, logic
Fixes: #9291
* Modify `unlock()` method of `JdbcLock`: if the lock ownership can not be removed due to data expiration, a `ConcurrentModificationException` should be thrown.
* Modify `unlock()` method of `RedisLock`: if the lock ownership can not be removed due to data expiration, a `ConcurrentModificationException` should be thrown.
* Maintain test cases
* Document the new behavior
Fixes: #9192
With modern drivers we don't need BLOB-specific handling anymore.
The regular `PreparedStatement.setBytes()` and `ResultSet.getBytes()`
are enough for our serialized messages
Fixes: #9061
`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover.
With this change the connection is renewed when notifications are not received for a certain time.
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9050
Turns out not all JDBC drivers support a `LocalDateTime` type conversion.
* Use `Timestamp.valueOf(LocalDateTime)` for `TIMESTAMP` params of the queries
in the `DefaultLockRepository`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #8998
Apparently some databases don't throw a proper code to identify a `DuplicateKeyException`, so better to catch `DataIntegrityViolationException` with the same meaning.
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8773
The https://github.com/spring-projects/spring-integration/issues/8732 introduced a filtering for messages in group.
So, plain `removeMessage()` doesn't work any more if message is connected to some group yet.
Therefore, `DelayHandler` is failing.
* Introduce `getMessageFromGroup()` and `removeMessageFromGroupById()` into `MessageGroupStore` API
and implement it respectively in all the stores
* Remove `@LongRunningTest` from delayer integration tests and adjust its config to delay not for a long
**Cherry-pick to `6.1.x`**
* GH-8770: Add `PostgresSubsChannel.errorHandler`
Fixes https://github.com/spring-projects/spring-integration/issues/8770
The problem with the `PostgresSubscribableChannel.notifyUpdate()` is that the try-catch block is outside the loop,
so the loop will die on an exception, leaving further messages unprocessed.
* Add ``PostgresSubscribableChannel.errorHandler` option to be invoked
after a `RetryTemplate` and for every failed message.
* The `askForMessage()` new logic is to catch an exception on a message and call `errorHandler`
returning a `FALLBACK_STUB` to continue an outer loop in the `notifyUpdate()`
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Rename private `PostgresSubscribableChannel.askForMessage()` method to more specific `pollAndDispatchMessage()`
The `PostgresSubscribableChannel` uses a task executor
for dispatching messages.
Even if we stop `PostgresChannelMessageTableSubscriber` and
unsubscribe from the channel, the task might be ongoing.
* Use explicit `ThreadPoolTaskExecutor` in the test
to shout it down and wait for tasks to be completed before
verifying DB status
* Optimize `PostgresSubscribableChannel` to mark TX for rollback
when we got a message from DB, but no handlers subscribed
Fixes https://github.com/spring-projects/spring-integration/issues/8760
* Make `PostgresChannelMessageStoreQueryProvider` to use single `DELETE ... RETURNING` for polling statements
* Add `isUsingSingleStatementForPoll` and use it from `JdbcChannelMessageStore`
* Execute Postgres init scripts to `PostgresContainerTest`
* Code clean up
* Document the new feature
Fixes https://github.com/spring-projects/spring-integration/issues/8748
The Oracle DB throws `ORA-08177: can't serialize access for this transaction`
when other transaction on the row has begun
* Change the isolation for `DefaultLockRepository.acquire()` transaction
to the `READ_COMMITTED` for what database automatically and silently
restarts the entire SQL statement, and no error occurs.
* Add `oracle` dependencies to JDBC module
* Introduce `OracleContainerTest` and implement it for `OracleLockRegistryTests`
**Cherry-pick to `6.1.x` & `6.0.x`**
* Fix Checkstyle violations for `private` inner classes where they have to be also `final`
* Fix deprecations after upgrades
* Remove manual coroutines handling in the `GatewayProxyFactoryBean`
in favor of the same functionality in the `JdkDynamicProxy`
The `ExpirableLockRegistry.expireUnusedOlderThan()` uses a query to the target store by mistake.
The logic of this API is indeed about the local cache for those lock instances.
We cannot effect any other processes with this cache.
And even if we remove our local instance while it is locked in other process,
that doesn't mean that on the next `obtain()` call, when we got a fresh local instance, we will be able to acquire a lock in target store.
* Fix `isAcquiredInThisProcess()` to check only `localLock.isLocked()`
**Cherry-pick to `6.1.x` & `6.0.x`**
When transaction is configured for the `PostgresSubscribableChannel.notifyUpdate()`
and it is rolled back, the next poll in that loop will return the same message.
Again and again if transaction is always rolled back.
This leads to the condition when we never leave this loop even if
we fully unsubscribed from this channel.
The issue has need spotted after introducing `SKIP LOCKED`
for `PostgresChannelMessageStoreQueryProvider` which leads to
the locked record in DB in the mentioned above transaction.
* Introduce `PostgresSubscribableChannel.hasHandlers` flag
to check in the `notifyUpdate()` before performing poll query in DB.
**Cherry-pick to `6.1.x` & `6.0.x`**
* GH-8732 Don't remove JDBC message if other groups
Fixes https://github.com/spring-projects/spring-integration/issues/8732
When same message is added into different groups,
its record in the `INT_MESSAGE` must remain until the last group is removed
* Improve `JdbcMessageStore.DELETE_MESSAGES_FROM_GROUP` SQL
to ignore those messages for removal which has other group records in the `INT_GROUP_TO_MESSAGE` table
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* * Fix `JdbcMessageStore.removeMessage()` and `removeMessagesFromGroup()`
to remove from `INT_MESSAGE` only if there is no other records in `INT_GROUP_TO_MESSAGE`
* * Improve `MessageStore.removeMessage()` Javadoc
mentioning `MessageGroupStore` specifics
* Add a version for deprecation on `AbstractChannelMessageStoreQueryProvider`
* Remove dead links from `ChannelMessageStoreQueryProvider` impl JavaDocs
* Use text blocks for long query strings
* Move `SELECT_COMMON` constant to the `ChannelMessageStoreQueryProvider` interface
* Move methods from `AbstractChannelMessageStoreQueryProvider` to `default` in the `ChannelMessageStoreQueryProvider` interface.
* Remove usage of this deprecated method.
* Fix `message-store.adoc` to mention `ChannelMessageStoreQueryProvider` instead of deprecated abstract class.
Fixes https://github.com/spring-projects/spring-integration/issues/8711
* Add `SKIP LOCKED` into `MySqlChannelMessageStoreQueryProvider` and `PostgresChannelMessageStoreQueryProvider`
* Stop the `postgresChannelMessageTableSubscriber` after the latch waiting 10s has been passed in `testMessagesDispatchedInTransaction` test case
* GH-8680: Check DB for table on start
Fixes https://github.com/spring-projects/spring-integration/issues/8680
If database is not initialized properly before application start, we may lose messages
at runtime when we fail to insert data into DB
* Implement `SmartLifecycle` on `JdbcMessageStore`, `JdbcChannelMessageStore`,
`JdbcMetadataStore`, and `DefaultLockRepository` to perform `SELECT COUNT` query in `start()`
to fail fast if no required table is present.
* Refactor `AbstractJdbcChannelMessageStoreTests` into JUnit 5 and use `MySqlContainerTest`
for more coverage
* Fix newly failed tests which had DB not initialized before
* Exclude `commons-logging` from `commons-dbcp2` dependency to avoid
classpath conflict
* Document the new feature
* * Fix HTTP URL in the `DataSource-mysql-context.xml`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
* * Add `setCheckDatabaseOnStart(false)` to disable the check query for all the SI JDBC components
* Fix language in Javadocs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
* GH-8642: Revise executors in the project
Fixes https://github.com/spring-projects/spring-integration/issues/8642
* Rework some `Executors.newSingleThreadExecutor()` to `ExecutorServiceAdapter(new SimpleAsyncTaskExecutor())`
* Expose `TaskExecutor` setters; deprecate `ExecutorService`-based
* Some other code clean up in the effected classes: `LogAccessor`, no `synchronized` in critical blocks
* Give a meaningful prefix for default threads in the context of components, e.g. `SubscribableRedisChannel` - `getBeanName() + "-"`
* * Fix `PostgresChannelMessageTableSubscriberTests` for
`PostgresSubscribableChannel` initialization to let it create
its internal `Executor`
* Use an `AsyncTaskExecutor` injection instead of `ExecutorServiceAdapter` wrapping
* Fix `LockRegistryLeaderInitiatorTests` for `taskExecutor` injection
* Bring back `LockRegistryLeaderInitiator.setExecutorService()`
as an accident after property auto-renaming
* Fix race condition in the `JdbcPollingChannelAdapterParserTests`
making SELECT and UPDATE as a part of the same transaction
* Remove `inProcess = JAVA_EXEC` from ASCIIDoc Gradle tasks in attempt
to make them working in parallel
