Fixes: #9481
Issue link: https://github.com/spring-projects/spring-integration/issues/9481
MySQL throws `CannotAcquireLockException` in case of duplicate key failure.
PostgreSQL just rollbacks a transaction not letting us move on with a `SELECT`
* Include `TransientDataAccessException` to the catch block of the `INSERT`
to ignore it for the subsequent `SELECT`
* Add logic to determine `PostgreSQL` database vendor and include `ON CONFLICT DO NOTHING`
hint into the `INSERT` to not fail in case of duplicate key found on `putIfAbsent` operation
(cherry picked from commit 98d0266ba0)
Fixes: #9061
`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover.
With this change the connection is renewed when notifications are not received for a certain time.
(cherry picked from commit 642278dad1)
Fixes: #9050
Turns out not all JDBC drivers support a `LocalDateTime` type conversion.
* Use `Timestamp.valueOf(LocalDateTime)` for `TIMESTAMP` params of the queries
in the `DefaultLockRepository`
(cherry picked from commit f71a2234d8)
Fixes: #8998
Apparently some databases don't throw a proper code to identify a `DuplicateKeyException`, so better to catch `DataIntegrityViolationException` with the same meaning.
(cherry picked from commit 676733ccf7)
Fixes https://github.com/spring-projects/spring-integration/issues/8773
The https://github.com/spring-projects/spring-integration/issues/8732 introduced a filtering for messages in group.
So, plain `removeMessage()` doesn't work any more if message is connected to some group yet.
Therefore, `DelayHandler` is failing.
* Introduce `getMessageFromGroup()` and `removeMessageFromGroupById()` into `MessageGroupStore` API
and implement it respectively in all the stores
* Remove `@LongRunningTest` from delayer integration tests and adjust its config to delay not for a long
**Cherry-pick to `6.1.x`**
* GH-8770: Add `PostgresSubsChannel.errorHandler`
Fixes https://github.com/spring-projects/spring-integration/issues/8770
The problem with the `PostgresSubscribableChannel.notifyUpdate()` is that the try-catch block is outside the loop,
so the loop will die on an exception, leaving further messages unprocessed.
* Add ``PostgresSubscribableChannel.errorHandler` option to be invoked
after a `RetryTemplate` and for every failed message.
* The `askForMessage()` new logic is to catch an exception on a message and call `errorHandler`
returning a `FALLBACK_STUB` to continue an outer loop in the `notifyUpdate()`
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Rename private `PostgresSubscribableChannel.askForMessage()` method to more specific `pollAndDispatchMessage()`
The `PostgresSubscribableChannel` uses a task executor
for dispatching messages.
Even if we stop `PostgresChannelMessageTableSubscriber` and
unsubscribe from the channel, the task might be ongoing.
* Use explicit `ThreadPoolTaskExecutor` in the test
to shout it down and wait for tasks to be completed before
verifying DB status
* Optimize `PostgresSubscribableChannel` to mark TX for rollback
when we got a message from DB, but no handlers subscribed
Fixes https://github.com/spring-projects/spring-integration/issues/8760
* Make `PostgresChannelMessageStoreQueryProvider` to use single `DELETE ... RETURNING` for polling statements
* Add `isUsingSingleStatementForPoll` and use it from `JdbcChannelMessageStore`
* Execute Postgres init scripts to `PostgresContainerTest`
* Code clean up
* Document the new feature
Fixes https://github.com/spring-projects/spring-integration/issues/8748
The Oracle DB throws `ORA-08177: can't serialize access for this transaction`
when other transaction on the row has begun
* Change the isolation for `DefaultLockRepository.acquire()` transaction
to the `READ_COMMITTED` for what database automatically and silently
restarts the entire SQL statement, and no error occurs.
* Add `oracle` dependencies to JDBC module
* Introduce `OracleContainerTest` and implement it for `OracleLockRegistryTests`
**Cherry-pick to `6.1.x` & `6.0.x`**
* Fix Checkstyle violations for `private` inner classes where they have to be also `final`
* Fix deprecations after upgrades
* Remove manual coroutines handling in the `GatewayProxyFactoryBean`
in favor of the same functionality in the `JdkDynamicProxy`
The `ExpirableLockRegistry.expireUnusedOlderThan()` uses a query to the target store by mistake.
The logic of this API is indeed about the local cache for those lock instances.
We cannot effect any other processes with this cache.
And even if we remove our local instance while it is locked in other process,
that doesn't mean that on the next `obtain()` call, when we got a fresh local instance, we will be able to acquire a lock in target store.
* Fix `isAcquiredInThisProcess()` to check only `localLock.isLocked()`
**Cherry-pick to `6.1.x` & `6.0.x`**
When transaction is configured for the `PostgresSubscribableChannel.notifyUpdate()`
and it is rolled back, the next poll in that loop will return the same message.
Again and again if transaction is always rolled back.
This leads to the condition when we never leave this loop even if
we fully unsubscribed from this channel.
The issue has need spotted after introducing `SKIP LOCKED`
for `PostgresChannelMessageStoreQueryProvider` which leads to
the locked record in DB in the mentioned above transaction.
* Introduce `PostgresSubscribableChannel.hasHandlers` flag
to check in the `notifyUpdate()` before performing poll query in DB.
**Cherry-pick to `6.1.x` & `6.0.x`**
* GH-8732 Don't remove JDBC message if other groups
Fixes https://github.com/spring-projects/spring-integration/issues/8732
When same message is added into different groups,
its record in the `INT_MESSAGE` must remain until the last group is removed
* Improve `JdbcMessageStore.DELETE_MESSAGES_FROM_GROUP` SQL
to ignore those messages for removal which has other group records in the `INT_GROUP_TO_MESSAGE` table
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* * Fix `JdbcMessageStore.removeMessage()` and `removeMessagesFromGroup()`
to remove from `INT_MESSAGE` only if there is no other records in `INT_GROUP_TO_MESSAGE`
* * Improve `MessageStore.removeMessage()` Javadoc
mentioning `MessageGroupStore` specifics
* Add a version for deprecation on `AbstractChannelMessageStoreQueryProvider`
* Remove dead links from `ChannelMessageStoreQueryProvider` impl JavaDocs
* Use text blocks for long query strings
* Move `SELECT_COMMON` constant to the `ChannelMessageStoreQueryProvider` interface
* Move methods from `AbstractChannelMessageStoreQueryProvider` to `default` in the `ChannelMessageStoreQueryProvider` interface.
* Remove usage of this deprecated method.
* Fix `message-store.adoc` to mention `ChannelMessageStoreQueryProvider` instead of deprecated abstract class.
Fixes https://github.com/spring-projects/spring-integration/issues/8711
* Add `SKIP LOCKED` into `MySqlChannelMessageStoreQueryProvider` and `PostgresChannelMessageStoreQueryProvider`
* Stop the `postgresChannelMessageTableSubscriber` after the latch waiting 10s has been passed in `testMessagesDispatchedInTransaction` test case
* GH-8680: Check DB for table on start
Fixes https://github.com/spring-projects/spring-integration/issues/8680
If database is not initialized properly before application start, we may lose messages
at runtime when we fail to insert data into DB
* Implement `SmartLifecycle` on `JdbcMessageStore`, `JdbcChannelMessageStore`,
`JdbcMetadataStore`, and `DefaultLockRepository` to perform `SELECT COUNT` query in `start()`
to fail fast if no required table is present.
* Refactor `AbstractJdbcChannelMessageStoreTests` into JUnit 5 and use `MySqlContainerTest`
for more coverage
* Fix newly failed tests which had DB not initialized before
* Exclude `commons-logging` from `commons-dbcp2` dependency to avoid
classpath conflict
* Document the new feature
* * Fix HTTP URL in the `DataSource-mysql-context.xml`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
* * Add `setCheckDatabaseOnStart(false)` to disable the check query for all the SI JDBC components
* Fix language in Javadocs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
* GH-8642: Revise executors in the project
Fixes https://github.com/spring-projects/spring-integration/issues/8642
* Rework some `Executors.newSingleThreadExecutor()` to `ExecutorServiceAdapter(new SimpleAsyncTaskExecutor())`
* Expose `TaskExecutor` setters; deprecate `ExecutorService`-based
* Some other code clean up in the effected classes: `LogAccessor`, no `synchronized` in critical blocks
* Give a meaningful prefix for default threads in the context of components, e.g. `SubscribableRedisChannel` - `getBeanName() + "-"`
* * Fix `PostgresChannelMessageTableSubscriberTests` for
`PostgresSubscribableChannel` initialization to let it create
its internal `Executor`
* Use an `AsyncTaskExecutor` injection instead of `ExecutorServiceAdapter` wrapping
* Fix `LockRegistryLeaderInitiatorTests` for `taskExecutor` injection
* Bring back `LockRegistryLeaderInitiator.setExecutorService()`
as an accident after property auto-renaming
* Fix race condition in the `JdbcPollingChannelAdapterParserTests`
making SELECT and UPDATE as a part of the same transaction
* Remove `inProcess = JAVA_EXEC` from ASCIIDoc Gradle tasks in attempt
to make them working in parallel
Fixes https://github.com/spring-projects/spring-integration/issues/8623
Turns out not all JDBC drivers (or RDBMS vendors) support `java.time.Instant`
mapping to their `TIMESTAMP` type.
For example the PostgreSQL fails like:
```
org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of java.time.Instant.
```
* Use `LocalDateTime.now(ZoneOffset.UTC)` instead `Instant.now()`.
Essentially bringing back the behavior from the previous version
Fixes https://github.com/spring-projects/spring-integration/issues/3866
Some RDBMS vendors (or their JDBC drivers) may just log the problem
without throwing an exception.
* Expose setters for UPDATE and INSERTS queries in the `DefaultLockRepository`
to let end-user to modify them respectively, e.g. be able to add a PostgreSQL
`ON CONFLICT DO NOTHING` hint.
* Refactor `DefaultLockRepository` to `Instant` instead of `LocalDateTime`
with zone offset.
* Refactor `ttl` property to `Duration` type
* Fix `dead-lock` typo to `deadlock`
Fixes https://github.com/spring-projects/spring-integration/issues/8582
* Introduce a `PostgresSubscribableChannel.setTransactionManager()`
to wrap a message polling and dispatching operation into a transaction
* In addition add a `RetryTemplate` support around transaction attempts
**Cherry-pick to `6.0.x`**
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
Looks like sometime Spring Framework XML parser can properly determine the target factory method to chose,
but sometimes it does that in a wrong order.
* Add `type="java.lang.Class"` to the ctor args everywhere for `class="org.mockito.Mockito" factory-method="mock"`
`<bean>` definitions
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
* Removed overridden `getMethodValue()` in the `CookieTests` which is removed from SF already
* Re-enable JDBC tests related to complex named params because of the fix in SF
* `--add-opens` for Tomcat warnings
* Upgrade to `log4j-slf4j2-impl`
* Add `cassandra.yaml` resource with increased timeouts to wait more in busy environment from Cassandra cluster
* INT-3333: Return empty list as is from DB gateway
Fixes https://jira.spring.io/browse/INT-3333
In `JdbcOutboundGateway` and `JpaOutboundGateway` the empty result list
is treated as "no reply" and therefore `null` is returned cause
the flow to stop at this point.
It is better to return such a result as is and the target application to
decided what to do with it, e.g. a `discardChannel` on a downstream splitter
configuration.
NOTE: the `MongoDbOutboundGateway` doesn't treat an empty result as a `null`
* * Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3872
Adds an implementation for a Postres-compatible notification listener for a `JdbcChannelMessageStore`.
* Introduce `PostgresChannelMessageTableSubscriber.Subscription` contract
* Implement a `PostgresSubscribableChannel`
* Add Javadoc and fix code formatting issues.
* Handle temporary connection loss.
* Add tests for `PostgresChannelMessageTableSubscriber`
* Replace NOTIFY command with pg_notify function call.
* Code style clean up
* Fix some Javadocs
* Use `DataSourceInitializer` in the `PostgresChannelMessageTableSubscriberTests` to populate scripts
* Use Java text block for DB scripts
Fixed https://github.com/spring-projects/spring-integration/issues/3885
Turns out the `Date` is stored into DB without time zone, so instances
in different zones may not react properly for updates
* Change the logic in the `DefaultLockRepository` to store time via
`LocalDateTime.now(ZoneOffset.UTC)` to ensure that all instance deals with the same
time representation without time zone offsets
The `CONDITION` word is reserved one in some RDBMS vendors
* Rename the `CONDITION` column to `GROUP_CONDITION` to avoid possible failures
and get rid of the conditional code on the `JdbcMessageStore`
* GH-3828: Initial Spring AOT support
Fixes https://github.com/spring-projects/spring-integration/issues/3828
* Provide an infrastructure based on a new Spring AOT engine in the latest Spring Framework
* Introduce `RuntimeHintsRegistrar` impls into modules which require some reflection,
resources or proxies and serialization available in the native image
* Mark some framework method with the `@Reflective` to make their reflection
info available in the native image, for example for SpEL or JMX invocations
* Add a `GatewayProxyBeanRegistrationAotProcessor` to register proxy interfaces
info for messaging gateways (either instance of the `GatewayProxyFactoryBean`)
* Rework `ConverterRegistrar` to not use a `beanFactory.getBeansWithAnnotation()`
since it is not available after AOT phase.
Instead, register an intermediate `IntegrationConverterRegistration` bean definition
from the `IntegrationConverterInitializer`
* Refactor `GlobalChannelInterceptorInitializer` a bit according to a new logic in the
`IntegrationConverterInitializer`
* Remove `JsonNodeWrapperToJsonNodeConverter` bean registration from the
`DefaultConfiguringBeanFactoryPostProcessor` - it is added by the `ConverterRegistrar`
into the target `ConversionService`
* Fix `ParentContextTests` respectively a `JsonNodeWrapperToJsonNodeConverter` bean removal
* Refactor `XsltPayloadTransformer` to not load a `ServletContextResource`, but just use its
name for the `xslResource` condition
* * Rework AOT support according latest changes and requirements
* Remove `@Bean` reflection since it is not needed any more
* Add `AotDetector.useGeneratedArtifacts()` condition to not register beans
one more time at runtime after AOT build phase
* Fix deprecation in the WebFlux test from the latest SF
* GH-3844: Rework messaging annotation with @Bean
Fixes https://github.com/spring-projects/spring-integration/issues/3844
* Make `MessagingAnnotationPostProcessor` as a `BeanDefinitionRegistryPostProcessor`
to process bean definitions as early as possible and register respective messaging
components at that early phase
* Make bean definitions parsing logic optional for AOT and native mode since beans
have bean parsed during AOT building phase
* Introduce a `BeanDefinitionPropertiesMapper` for easier mapping
of the annotation attributes to the target `BeanDefinition`
* Remove `@Bean`-related logic from method parsing process
* Change the logic for `@Bean`-based endpoint bean names:
since we don't deal with methods on the bean definition phase, then method name
does not make sense.
It even may mislead if we `@Bean` name is based on a method by default, so we end up
with duplicated word in the target endpoint bean name.
Now we don't
* Fix `configuration.adoc` respectively for a new endpoint bean name logic
* In the end the new logic in the `AbstractMethodAnnotationPostProcessor`
is similar to XML parsers: we feed annotation attributes to the
`AbstractStandardMessageHandlerFactoryBean` impls
* * Fix language in docs and exception message
* Upgrade Spring dependencies to the latest SNAPSHOTs
* Fix tests to verify against stack traces: the message
of the `NestedRuntimeException` does not include the nested exception information.
Related to https://github.com/spring-projects/spring-framework/issues/25162
* Fix `JdbcMessageStore` and `DefaultLockRepository` to rely on the `DataIntegrityViolationException`
instead of only its `DuplicateKeyException` extension.
This is the current behavior of the SQL errors translation
* Disable `WebFluxDslTests.testValidation()` - doesn't subscribe to the reply somehow...
* Refine `SimplePool.PoolSemaphore.reducePermits()`
* GH-3797: Improve batch processing in the framework
Fixes https://github.com/spring-projects/spring-integration/issues/3797
* Handle `Message` items of the `Iterable` payload properly in the `JdbcMessageHandler`.
Otherwise, they've been wrapped into an extra `Message`
* Produce a single message with a `Collection<Message<?>>` payload in the `AggregatingMessageHandler`
when the `getOutputProcessor()` is not an instance of `SimpleMessageGroupProcessor`
* Mention these changes in docs
* Point to the error handling sample from docs
* * Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
Related to https://stackoverflow.com/questions/72305387/spring-integration-delayer-starts-sending-null-message-payloads-when-switched
In some async use-cases (e.g. `DelayHandler`), the context classloader
might be different for the data to be deserialized from message store.
* Fix `JdbcMessageStore` to populate a bean `ClassLoader` into default
`AllowListDeserializingConverter` from the application context.
The provided `Deserializer` must ensure such a `ClassLoader` itself
* Add warning message to the `LambdaMessageProcessor` when converter
returns `null` for the payload it cannot convert to expected type.
Cannot be raised as error since some applications may already rely
on the `null` conversion result in their method arguments
**Cherry-pick to `5.5.x`**
The `withReuse(true)` and `testcontainers.reuse.enable=true` don't work together with `@Container`.
The JUnit extension gathers those containers and stop them in the end of test class unconditionally.
* Remove `@Container` annotation usage
* Use `@BeforeAll` and `GenericContainer.start()` manually
This way the container ensures to reuse existing running container and don't start a fresh one.
Since the container instance is stored in a `static` property, it is really started only once.
The rest tests in a suite just reuse that existing container.
Ryuk container will take care about their stopping and removal eventually afte JVM exit.
**Cherry-pick to `5.5.x`**
* GH-3785: Close stream for persistent collection
Fixes https://github.com/spring-projects/spring-integration/issues/3785
* Fix `CollectionArgumentResolver` and `PayloadsArgumentResolver` to
close the `Stream` of message after its usage
* Rework `AbstractKeyValueMessageStore.removeMessagesFromGroup()`
to iterate input collection of messages not its stream to avoid
the mentioned problem
**Cherry-pick to `5.5.x`**
* * Add `JdbcMessageStoreTests.testMessageGroupStreamNoConnectionPoolLeak()`
to ensure that no leaks in the connection pool anymore.
* Improve `MessageGroupStore.streamMessagesForGroup()` JavaDocs about
requirements to close the `Stream` from persistent message store impls
