Fixes: #9061
`PostgresChannelMessageTableSubscriber` never renews the connection.
This causes problems on DB failover.
With this change the connection is renewed when notifications are not received for a certain time.
(cherry picked from commit 642278dad1)
# Conflicts:
# spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/channel/PostgresChannelMessageTableSubscriber.java
# spring-integration-jdbc/src/test/java/org/springframework/integration/jdbc/channel/PostgresChannelMessageTableSubscriberTests.java
Fixes: #9050
Turns out not all JDBC drivers support a `LocalDateTime` type conversion.
* Use `Timestamp.valueOf(LocalDateTime)` for `TIMESTAMP` params of the queries
in the `DefaultLockRepository`
(cherry picked from commit f71a2234d8)
Fixes: #8998
Apparently some databases don't throw a proper code to identify a `DuplicateKeyException`, so better to catch `DataIntegrityViolationException` with the same meaning.
(cherry picked from commit 676733ccf7)
# Conflicts:
# spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/metadata/JdbcMetadataStore.java
# spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/store/JdbcChannelMessageStore.java
Fixes https://github.com/spring-projects/spring-integration/issues/8773
The https://github.com/spring-projects/spring-integration/issues/8732 introduced a filtering for messages in group.
So, plain `removeMessage()` doesn't work any more if message is connected to some group yet.
Therefore, `DelayHandler` is failing.
* Introduce `getMessageFromGroup()` and `removeMessageFromGroupById()` into `MessageGroupStore` API
and implement it respectively in all the stores
* Remove `@LongRunningTest` from delayer integration tests and adjust its config to delay not for a long
**Cherry-pick to `6.1.x`**
# Conflicts:
# spring-integration-core/src/main/java/org/springframework/integration/handler/DelayHandler.java
* GH-8770: Add `PostgresSubsChannel.errorHandler`
Fixes https://github.com/spring-projects/spring-integration/issues/8770
The problem with the `PostgresSubscribableChannel.notifyUpdate()` is that the try-catch block is outside the loop,
so the loop will die on an exception, leaving further messages unprocessed.
* Add ``PostgresSubscribableChannel.errorHandler` option to be invoked
after a `RetryTemplate` and for every failed message.
* The `askForMessage()` new logic is to catch an exception on a message and call `errorHandler`
returning a `FALLBACK_STUB` to continue an outer loop in the `notifyUpdate()`
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Rename private `PostgresSubscribableChannel.askForMessage()` method to more specific `pollAndDispatchMessage()`
The `PostgresSubscribableChannel` uses a task executor
for dispatching messages.
Even if we stop `PostgresChannelMessageTableSubscriber` and
unsubscribe from the channel, the task might be ongoing.
* Use explicit `ThreadPoolTaskExecutor` in the test
to shout it down and wait for tasks to be completed before
verifying DB status
* Optimize `PostgresSubscribableChannel` to mark TX for rollback
when we got a message from DB, but no handlers subscribed
# Conflicts:
# spring-integration-jdbc/src/test/java/org/springframework/integration/jdbc/channel/PostgresChannelMessageTableSubscriberTests.java
Fixes https://github.com/spring-projects/spring-integration/issues/8748
The Oracle DB throws `ORA-08177: can't serialize access for this transaction`
when other transaction on the row has begun
* Change the isolation for `DefaultLockRepository.acquire()` transaction
to the `READ_COMMITTED` for what database automatically and silently
restarts the entire SQL statement, and no error occurs.
* Add `oracle` dependencies to JDBC module
* Introduce `OracleContainerTest` and implement it for `OracleLockRegistryTests`
**Cherry-pick to `6.1.x` & `6.0.x`**
The `ExpirableLockRegistry.expireUnusedOlderThan()` uses a query to the target store by mistake.
The logic of this API is indeed about the local cache for those lock instances.
We cannot effect any other processes with this cache.
And even if we remove our local instance while it is locked in other process,
that doesn't mean that on the next `obtain()` call, when we got a fresh local instance, we will be able to acquire a lock in target store.
* Fix `isAcquiredInThisProcess()` to check only `localLock.isLocked()`
**Cherry-pick to `6.1.x` & `6.0.x`**
# Conflicts:
# spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/lock/JdbcLockRegistry.java
When transaction is configured for the `PostgresSubscribableChannel.notifyUpdate()`
and it is rolled back, the next poll in that loop will return the same message.
Again and again if transaction is always rolled back.
This leads to the condition when we never leave this loop even if
we fully unsubscribed from this channel.
The issue has need spotted after introducing `SKIP LOCKED`
for `PostgresChannelMessageStoreQueryProvider` which leads to
the locked record in DB in the mentioned above transaction.
* Introduce `PostgresSubscribableChannel.hasHandlers` flag
to check in the `notifyUpdate()` before performing poll query in DB.
**Cherry-pick to `6.1.x` & `6.0.x`**
* GH-8732 Don't remove JDBC message if other groups
Fixes https://github.com/spring-projects/spring-integration/issues/8732
When same message is added into different groups,
its record in the `INT_MESSAGE` must remain until the last group is removed
* Improve `JdbcMessageStore.DELETE_MESSAGES_FROM_GROUP` SQL
to ignore those messages for removal which has other group records in the `INT_GROUP_TO_MESSAGE` table
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* * Fix `JdbcMessageStore.removeMessage()` and `removeMessagesFromGroup()`
to remove from `INT_MESSAGE` only if there is no other records in `INT_GROUP_TO_MESSAGE`
* * Improve `MessageStore.removeMessage()` Javadoc
mentioning `MessageGroupStore` specifics
Fixes https://github.com/spring-projects/spring-integration/issues/8623
Turns out not all JDBC drivers (or RDBMS vendors) support `java.time.Instant`
mapping to their `TIMESTAMP` type.
For example the PostgreSQL fails like:
```
org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of java.time.Instant.
```
* Use `LocalDateTime.now(ZoneOffset.UTC)` instead `Instant.now()`.
Essentially bringing back the behavior from the previous version
Fixes https://github.com/spring-projects/spring-integration/issues/3866
Some RDBMS vendors (or their JDBC drivers) may just log the problem
without throwing an exception.
* Expose setters for UPDATE and INSERTS queries in the `DefaultLockRepository`
to let end-user to modify them respectively, e.g. be able to add a PostgreSQL
`ON CONFLICT DO NOTHING` hint.
* Refactor `DefaultLockRepository` to `Instant` instead of `LocalDateTime`
with zone offset.
* Refactor `ttl` property to `Duration` type
* Fix `dead-lock` typo to `deadlock`
Fixes https://github.com/spring-projects/spring-integration/issues/8582
* Introduce a `PostgresSubscribableChannel.setTransactionManager()`
to wrap a message polling and dispatching operation into a transaction
* In addition add a `RetryTemplate` support around transaction attempts
**Cherry-pick to `6.0.x`**
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
Looks like sometime Spring Framework XML parser can properly determine the target factory method to chose,
but sometimes it does that in a wrong order.
* Add `type="java.lang.Class"` to the ctor args everywhere for `class="org.mockito.Mockito" factory-method="mock"`
`<bean>` definitions
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
* Removed overridden `getMethodValue()` in the `CookieTests` which is removed from SF already
* Re-enable JDBC tests related to complex named params because of the fix in SF
* `--add-opens` for Tomcat warnings
* Upgrade to `log4j-slf4j2-impl`
* Add `cassandra.yaml` resource with increased timeouts to wait more in busy environment from Cassandra cluster
* INT-3333: Return empty list as is from DB gateway
Fixes https://jira.spring.io/browse/INT-3333
In `JdbcOutboundGateway` and `JpaOutboundGateway` the empty result list
is treated as "no reply" and therefore `null` is returned cause
the flow to stop at this point.
It is better to return such a result as is and the target application to
decided what to do with it, e.g. a `discardChannel` on a downstream splitter
configuration.
NOTE: the `MongoDbOutboundGateway` doesn't treat an empty result as a `null`
* * Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3872
Adds an implementation for a Postres-compatible notification listener for a `JdbcChannelMessageStore`.
* Introduce `PostgresChannelMessageTableSubscriber.Subscription` contract
* Implement a `PostgresSubscribableChannel`
* Add Javadoc and fix code formatting issues.
* Handle temporary connection loss.
* Add tests for `PostgresChannelMessageTableSubscriber`
* Replace NOTIFY command with pg_notify function call.
* Code style clean up
* Fix some Javadocs
* Use `DataSourceInitializer` in the `PostgresChannelMessageTableSubscriberTests` to populate scripts
* Use Java text block for DB scripts
Fixed https://github.com/spring-projects/spring-integration/issues/3885
Turns out the `Date` is stored into DB without time zone, so instances
in different zones may not react properly for updates
* Change the logic in the `DefaultLockRepository` to store time via
`LocalDateTime.now(ZoneOffset.UTC)` to ensure that all instance deals with the same
time representation without time zone offsets
The `CONDITION` word is reserved one in some RDBMS vendors
* Rename the `CONDITION` column to `GROUP_CONDITION` to avoid possible failures
and get rid of the conditional code on the `JdbcMessageStore`
* GH-3828: Initial Spring AOT support
Fixes https://github.com/spring-projects/spring-integration/issues/3828
* Provide an infrastructure based on a new Spring AOT engine in the latest Spring Framework
* Introduce `RuntimeHintsRegistrar` impls into modules which require some reflection,
resources or proxies and serialization available in the native image
* Mark some framework method with the `@Reflective` to make their reflection
info available in the native image, for example for SpEL or JMX invocations
* Add a `GatewayProxyBeanRegistrationAotProcessor` to register proxy interfaces
info for messaging gateways (either instance of the `GatewayProxyFactoryBean`)
* Rework `ConverterRegistrar` to not use a `beanFactory.getBeansWithAnnotation()`
since it is not available after AOT phase.
Instead, register an intermediate `IntegrationConverterRegistration` bean definition
from the `IntegrationConverterInitializer`
* Refactor `GlobalChannelInterceptorInitializer` a bit according to a new logic in the
`IntegrationConverterInitializer`
* Remove `JsonNodeWrapperToJsonNodeConverter` bean registration from the
`DefaultConfiguringBeanFactoryPostProcessor` - it is added by the `ConverterRegistrar`
into the target `ConversionService`
* Fix `ParentContextTests` respectively a `JsonNodeWrapperToJsonNodeConverter` bean removal
* Refactor `XsltPayloadTransformer` to not load a `ServletContextResource`, but just use its
name for the `xslResource` condition
* * Rework AOT support according latest changes and requirements
* Remove `@Bean` reflection since it is not needed any more
* Add `AotDetector.useGeneratedArtifacts()` condition to not register beans
one more time at runtime after AOT build phase
* Fix deprecation in the WebFlux test from the latest SF
* GH-3844: Rework messaging annotation with @Bean
Fixes https://github.com/spring-projects/spring-integration/issues/3844
* Make `MessagingAnnotationPostProcessor` as a `BeanDefinitionRegistryPostProcessor`
to process bean definitions as early as possible and register respective messaging
components at that early phase
* Make bean definitions parsing logic optional for AOT and native mode since beans
have bean parsed during AOT building phase
* Introduce a `BeanDefinitionPropertiesMapper` for easier mapping
of the annotation attributes to the target `BeanDefinition`
* Remove `@Bean`-related logic from method parsing process
* Change the logic for `@Bean`-based endpoint bean names:
since we don't deal with methods on the bean definition phase, then method name
does not make sense.
It even may mislead if we `@Bean` name is based on a method by default, so we end up
with duplicated word in the target endpoint bean name.
Now we don't
* Fix `configuration.adoc` respectively for a new endpoint bean name logic
* In the end the new logic in the `AbstractMethodAnnotationPostProcessor`
is similar to XML parsers: we feed annotation attributes to the
`AbstractStandardMessageHandlerFactoryBean` impls
* * Fix language in docs and exception message
* Upgrade Spring dependencies to the latest SNAPSHOTs
* Fix tests to verify against stack traces: the message
of the `NestedRuntimeException` does not include the nested exception information.
Related to https://github.com/spring-projects/spring-framework/issues/25162
* Fix `JdbcMessageStore` and `DefaultLockRepository` to rely on the `DataIntegrityViolationException`
instead of only its `DuplicateKeyException` extension.
This is the current behavior of the SQL errors translation
* Disable `WebFluxDslTests.testValidation()` - doesn't subscribe to the reply somehow...
* Refine `SimplePool.PoolSemaphore.reducePermits()`
* GH-3797: Improve batch processing in the framework
Fixes https://github.com/spring-projects/spring-integration/issues/3797
* Handle `Message` items of the `Iterable` payload properly in the `JdbcMessageHandler`.
Otherwise, they've been wrapped into an extra `Message`
* Produce a single message with a `Collection<Message<?>>` payload in the `AggregatingMessageHandler`
when the `getOutputProcessor()` is not an instance of `SimpleMessageGroupProcessor`
* Mention these changes in docs
* Point to the error handling sample from docs
* * Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
Related to https://stackoverflow.com/questions/72305387/spring-integration-delayer-starts-sending-null-message-payloads-when-switched
In some async use-cases (e.g. `DelayHandler`), the context classloader
might be different for the data to be deserialized from message store.
* Fix `JdbcMessageStore` to populate a bean `ClassLoader` into default
`AllowListDeserializingConverter` from the application context.
The provided `Deserializer` must ensure such a `ClassLoader` itself
* Add warning message to the `LambdaMessageProcessor` when converter
returns `null` for the payload it cannot convert to expected type.
Cannot be raised as error since some applications may already rely
on the `null` conversion result in their method arguments
**Cherry-pick to `5.5.x`**
The `withReuse(true)` and `testcontainers.reuse.enable=true` don't work together with `@Container`.
The JUnit extension gathers those containers and stop them in the end of test class unconditionally.
* Remove `@Container` annotation usage
* Use `@BeforeAll` and `GenericContainer.start()` manually
This way the container ensures to reuse existing running container and don't start a fresh one.
Since the container instance is stored in a `static` property, it is really started only once.
The rest tests in a suite just reuse that existing container.
Ryuk container will take care about their stopping and removal eventually afte JVM exit.
**Cherry-pick to `5.5.x`**
* GH-3785: Close stream for persistent collection
Fixes https://github.com/spring-projects/spring-integration/issues/3785
* Fix `CollectionArgumentResolver` and `PayloadsArgumentResolver` to
close the `Stream` of message after its usage
* Rework `AbstractKeyValueMessageStore.removeMessagesFromGroup()`
to iterate input collection of messages not its stream to avoid
the mentioned problem
**Cherry-pick to `5.5.x`**
* * Add `JdbcMessageStoreTests.testMessageGroupStreamNoConnectionPoolLeak()`
to ensure that no leaks in the connection pool anymore.
* Improve `MessageGroupStore.streamMessagesForGroup()` JavaDocs about
requirements to close the `Stream` from persistent message store impls
* GH-3733 Configure TxManager for DefLockRepository
Fixes https://github.com/spring-projects/spring-integration/issues/3733
The `@Transactional` resolves a primary `TransactionManager` bean
from the application context which might not be sufficient for all
the use-case.
To make it work with the custom (or specific) `TransactionManager`
we have to extend a `DefaultLockRepository` and override all those
`@Transactional` method
* Change the logic of the `DefaultLockRepository` from `@Transactional`
to the `TransactionTemplate` and use provided `TransactionManager`
or resolve one from the application context
* Adjust tests to use explicit `TransactionManager` and call
`afterSingletonsInstantiated()` to initialize a default `TransactionTemplate`
* Mention the change in the docs
* * Extracted all the `TransactionTemplate`s to the properties for caching
* Add `BeanInitializationException` for no-unique `PlatformTransactionManager`
bean in the `afterSingletonsInstantiated()`
* Fix language in the exception message
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
The default `TTL` in the `DefaultLockRepository` is 10 seconds which mislead us
when other client instance has just 100 millis.
So, not-adjusted client spins for expired lock wastefully leading to the
synchronization barriers to fail
**Cherry-pick to `5.4.x`**
* Increase a `default-delay="1000"` for `GatewayInterfaceTests`
to test the late reply scenario
* Use a `ConcurrentHashMap` for `JdbcMessageStore.queryCache`
to avoid a possible `ConcurrentModificationException`
* Upgrade to Gradle `7.3.3`
* Upgrade to the latest milestones for releasing
* Upgrade to Kryo-5.2.1 and update source code according breaking changes
* Upgrade to H2-2.1.210 and re-enable respective stored procedure tests
Turns out something is broken in H2 `Parser` for stored procedures.
So, parameters are now doubled which is definitely not expected.
* Disable those H2 tests where parametrized stored procedures are used
* Add `--add-opens` to `asciidoctor` Gradle task to avoid compilation warning
* Upgrade to Kotlin `1.6.10` and modify its Gradle task respective
* Add empty impl for new `AnnotationMetadataAdapter.getDeclaredMethods()`
* Parse HTTP methods to new `String methodNames` property of the `RequestMapping`
since `HttpMethod` is not an `enum` anymore.
* Ignore JDBC tests which rely on a map property resolution: the `[]` placeholder
is not handled in the latest SF anymore.
* Fix `WebFluxInboundEndpoint` for deprecations
* Upgrade to Java 17, SF-6.0, Gradle 7.2
* Upgrade to Jakarta dependencies and respective namespaces
* Fix some tests for Java 17 compatibility
* Fix wrong Javadocs
* Add some missed Javadocs
* Fix more `jakarta` namespace
* Fix WS & XML modules to use Jakarta EE
* `--add-opens` in some modules for their reflection-based tests
* Disable Kafka tests which does not work on Windows; see Apache Kafka `3.0.1`
* Upgrade to JUnit `5.8.1`
* Migrate JMS tests to Artemis
* Remove RMI module as it was deprecated before
* Fix `pr-build-workflow.yml` for Java 17
* Fix JavaDocs warnings using `Xdoclint:syntax` per module, not in the top-level `api` task
* Move docs for version `6.0`
Fixes https://github.com/spring-projects/spring-integration/issues/3683
If a transaction is already active while `JdbcLockRegistry` uses
`DefaultLockRepository` to acquire/release a lock, the repository
must execute SQL queries in a separate transaction to prevent
problems with blocking, deadlocking etc.
It also allows to properly follow transaction isolation level that
is set on some methods of `DefaultLockRepository`.
Previously all methods of DefaultLockRepository supported the current
transaction if there are any. Now all methods will always create new
transaction on each call.
* Change tests to be more unit-ish
* Change `@since` for the `DefaultLockRepositoryTests` to `5.3.10`
**Cherry-pick to `5.4.x` & `5.3.x`**
