* GH-3554: Eval remote dir on each synchToLocal
Fixes https://github.com/spring-projects/spring-integration/issues/3554
The `remoteDirectoryExpression` was introduced into an
`AbstractInboundFileSynchronizer` to let end-user to evaluate a remote directory
on each poll (essentially on each `synchronizeToLocalDirectory()` call).
The fix for the https://jira.spring.io/browse/INT-4491 introduced a regression
when such an expression was evaluated only once when we call a `setRemoteDirectory()`.
So, an original purpose of this option was lost and we don't get an actual
remote dir on each poll
* Remove `evaluateRemoteDirectory()` method and its usage since it doesn't
reflect expectation of the remote dir expression property
* Reinstate the `remoteDirectoryExpression` evaluation in the
`synchronizeToLocalDirectory()`
* Propagate the result of that expression into further methods for copying
files
* Remove setting of the `remoteDirectory` variable into a global `EvaluationContext`
of the `AbstractInboundFileSynchronizer` instance since it is not thread-safe
* Instead create an `EvaluationContext` locally for each `synchronizeToLocalDirectory()`
call and set the `remoteDirectory` variable into this scoped instances
* Generate a local file name from the `localFilenameGeneratorExpression`
against locally created `EvaluationContext` with the mentioned `remoteDirectory` variable
* Cover the expected functionality with a unit-test
**Cherry-pick to `5.4.x` & `5.3.x`**
* * Fix `testRemoteDirectoryRefreshedOnEachSynchronization` according PR review
SO: https://stackoverflow.com/questions/67214907/zeromq-with-spring-spring-integration-zeromq
The `Mono` is created in several places in ZeroMQ components from their constructors.
That leads to the reactive stream to be configured just after ctor, which will ignore
any changes to the options which are used from that `Mono` definition.
For example this code `.doOnNext(this.sendSocketConfigurer)` is done once during
reactive stream definition.
* Fix all the ZeroMQ components to defer usage of the options which could be changed
after ctor initialization
* Cover affected option changes in the tests
**Cherry-pick to `5.4.x`**
The `AbstractPersistentAcceptOnceFileListFilter`
extends now an `AbstractDirectoryAwareFileListFilter`
and makes all the implementors to implement its `isDirectory()` method.
This is a breaking change in the point release which makes implementations
outside of the core project not compilable
The regression after: https://github.com/spring-projects/spring-integration/issues/3488
Related fix in the Spring Cloud GCP: https://github.com/spring-cloud/spring-cloud-gcp/pull/2654
* Implement an `isDirectory()` in the `AbstractPersistentAcceptOnceFileListFilter`
to let other implementations do not worry during upgrade
**Cherry-pick to `5.4.x` & `5.3.x`**
* Make `@IntegrationConverter` Native compatible
* Add `BASE_PACKAGE` into an `IntegrationContextUtils`;
deprecate similar in the `IntegrationConfigUtils`.
This fixes a package tangle between `config` and `context`
* Move `ConverterRegistrar` and `CustomConversionServiceFactoryBean`
into a `config` package since they are package protected and
created their instances in the `IntegrationConverterInitializer`
functional way instead of reflection
* Use new `IntegrationContextUtils.BASE_PACKAGE` constant instead of
deprecated one
* Make `DefaultConfiguringBeanFactoryPostProcessor` `public` to
make it available for Spring Native `trigger` option in the `@NativeHint`
declaration
* Simplify logic around `JsonPath` to just a `ClassUtils.isPresent()`
* Move the `@IntegrationConverter` processing logic into the `ConverterRegistrar`
to avoid reflection via `BeanDefinition` ctor arg manipulation
* Move the reflection logic into a `ConverterParser` which, being a part of XML
configuration, is not going to be compatible with native any way
* Mark `JsonNodeWrapperToJsonNodeConverter` with an `@IntegrationConverter`
since it is not registered via reflection any more
* Expose `MicrometerMetricsCaptorRegistrar.METER_REGISTRY_PRESENT` and use
it in the `IntegrationGraphServer`
* Extract `UnmarshallingTransformer.MIME_MESSAGE_PRESENT` for less
reflection at runtime
* Use `null` for a `ClassLoader` arg in the `ClassUtils.isPresent()`
relying on the default one internally
* * Fix Checkstyle violations
Currently the `MessageHistoryRegistrar` can parse several sources for message history -
`@EnableMessageHistory` and/or `<message-history>`.
Since its logic relies on the reflection it is not compatible with Spring Native.
Plus it causes confusion when several sources are declared so when time comes to
change something in that configuration, we may miss some place to re-align with
our new requirements.
Better to reject extra configurations and enforce end-users to use only one
`@EnableMessageHistory` or `<message-history>`.
This is actually a preferences for many other `@Enable...` in Spring portfolio.
Plus we got a benefit with a Spring Native compatibility
* Fix `MessageHistoryRegistrar` to parse only one `@EnableMessageHistory`.
Register `MessageHistoryConfigurer` function way for Spring Native compatibility
* Clean up `PublisherRegistrar` for better readability
* Fix message history tests which exposed several configurations
* Add JavaDoc into `EnableMessageHistory`
* Refactor `MessageHistoryConfigurer` to let to override patterns configuration
at runtime
* Clean up `message-history.adoc`
* Optimize `InboundOneWayErrorTests` to not close ctx after each test.
Probably will affect ActiveMQ connection to avoid a race condition
when it is not started yet, but we produce a message
Related to https://build.spring.io/browse/INT-SONAR-3564
When we reset shared Redis connection between tests,
it may lead to a dead lock on some async command in process
when the next test initiate a new connection
* Remove `resetConnection()` from the `ReactiveRedisStreamMessageProducerTests` `@After` -
looks like it doesn't effect anything in a whole test suite
* GH-3533: Register WebSocket endpoints at runtime
Fixes https://github.com/spring-projects/spring-integration/issues/3533
* Rework `WebSocketIntegrationConfigurationInitializer` to register beans functional way
to avoid reflection for Spring Native support
* Move `IntegrationServletWebSocketHandlerRegistry` into a separate file for better readability
* Implement `DestructionAwareBeanPostProcessor` for `IntegrationServletWebSocketHandlerRegistry`
to track runtime bean registrations and removals
* Introduce an `IntegrationDynamicWebSocketHandlerMapping` to manage runtime mapping
registrations and removals
* Add `servlet-api` dependency into `websocket` to be able to compile an
`IntegrationDynamicWebSocketHandlerMapping`
* Fix typo in the exception message of the `StandardIntegrationFlowRegistration`
* Start dynamically added beans together with associated `IntegrationFlow` in the
`StandardIntegrationFlowContext`
* Document new feature
* * Fix language in docs
* Don't start those `SmartLifecycle`s together with a dynamic flow
which are not `isAutoStartup()`
* * Fix `TomcatWebSocketTestServer` to configure servlet for `loadOnStartup = 1`
* Fix `WebSocketDslTests` to make `clientWebSocketContainer.setAutoStartup(true)`
* Fix complexity in the `DefaultJmsHeaderMapper` according Sonar report
* Optimize all JMS tests to rely on a shared `ActiveMQConnectionFactory` resource
and disable JMX & statistics for embedded ActiveMQ broker
* Increase timeout for some sporadically failing tests
When we build with Java 9+ and target for Java 8,
we get an incompatible bytecode around these method for `ByteBuffer`:
```
position(int)
limit(int)
mark()
reset()
clear()
flip()
rewind()
```
The recommendation is to cast to `Buffer` when we call these methods
* Fix all the production code using `ByteBuffer` for recommended cast to `Buffer`
Related to https://build.spring.io/browse/INTEXT-AWS-306
See more info in this Jetty issue: https://github.com/eclipse/jetty.project/issues/3244
This forces the race condition that was fixed in #3535 consistently.
Before the `IdleCanceler` was used to cancel the `idle()` call in addition to the normal flow.
That should not be necessary any more.
Related to: https://github.com/spring-cloud/stream-applications/issues/156
The `IntegrationReactiveUtils` uses a general `Flux.retry()`
operator to always retry for all the errors.
On the other hand it has only a `.doOnError(MessagingException.class)`
which leads swallowing all the other exceptions from logs and handling.
* Replace with `retryWhen()` for the `MessagingException` predicate
failing for all other exceptions.
The end-user may add their own `retry` or error handling mechanism to
the returned `Flux` from the `IntegrationReactiveUtils`
**Cherry-pick to 5.4.x**
Fixes https://github.com/spring-projects/spring-integration/issues/3541
* Fix `DefaultConfiguringBeanFactoryPostProcessor` to check for `jsonPath` bean
a `com.jayway.jsonpath.JsonPath` on CP instead of `JsonPathUtils`
which is always there since it is a part of `spring-integration-core`
* Remove `json-path` and `jackson-databind` from tests classpath
to be sure that we have coverage for the mentioned above option classpath entries
* Add `json-path` and `jackson-databind` into test classpath whenever it is necessary
`IMAPFolder.idle()` by default keeps idle-ing after each response.
We don't need that, because we want to fetch the new mails immediately (over the same connection).
To make `idle()` not keep on going, we called
`folder.isOpen()` which in most cases makes `idle()` stop by calling `noop()`
internally (to keep the connection alive) which leads to the current
`idle()`-call being canceled.
The problem this commits addresses is that the call to `noop()` only
happens if there were no calls in the last second.
There is a check for that in `com.sun.mail.imap.IMAPFolder.keepConnectionAlive`.
So if a new message appears less then a second after idle started, we will miss it.
This new way interrupts `idle()` more often than before, e.g. when there
is an expunge-message (i.e. a message was deleted), which we don't care about.
But the subsequent `receive()` in IdleTask will simply get no
messages and then turn around and start idle-ing again.
**Cherry-pick to `5.3.x` & `master`**
* GH-3502: More refactoring to avoid reflection
Fixes https://github.com/spring-projects/spring-integration/issues/3502
* Move `ChannelInitializer` bean registration into an `AbstractIntegrationNamespaceHandler` -
it was never used for annotations and Java DSL...
* Rework `IntegrationFlows.fromSupplier()` to call a provided `Supplier` directly -
not via reflection in the `MethodInvokingMessageSource`
* Resolve new Sonar smells
* Rework `EndpointSpec` to accept an expected factory bean instance via ctor arg
instead of reflection
* Rework `Jackson2JsonObjectMapper` to use well-known module instances directly -
not via reflection from their class names
* * Revert `DefaultMethodInvokingMethodInterceptor.methodHandleCache` property definition wrap
* GH-3529: Add HTTP & WebFlux extractResponseBody
Fixes https://github.com/spring-projects/spring-integration/issues/3529
* Expose a convenient `extractResponseBody` option on the HTTP client components
to let end-user to decide if the body of `ResponseEntity` must be extracted (default)
or the whole `ResponseEntity` should be produced as a reply message payload
* Remove a deprecated since `5.3` `encode-uri` option
* Document the new feature
* Rework `webflux.adoc` chapter for the code snippet switcher
* * Fix language in Docs
* Mention a default value in JavaDocs
* Fix `MiscellaneousTests` to extend `ActiveMQMultiContextTests`
to let it to close ActiveMQ connection factory in the end of test
* Make `RequestReplyScenariosWithTempReplyQueuesTests` more robust
closing `DefaultMessageListenerContainer` and `ExecutorService`
in the end of tests
* Fix new Sonar smells
* Rework some SQL calls in the `JdbcMessageStore` to note expose a
`PreparedStatement` API
* Fix JavaDoc in the `CorrelationHandlerSpec`
Resolves https://github.com/spring-projects/spring-integration/issues/3526
`FailoverClientConnectionFactory`
The logic to detect we had iterated over all factories and including
the one from which the previous connection was established was incorrect,
causing an infite loop until one of the factory connections was successful.
Change the logic to detect we have reset the iterator and the current failure
is from the same factory as the one from which the previous connection was
established.
**cherry-pick to 5.4.x, 5.3.x**
* Add diagnostics.
* Fix race in test.
* More race fixes and diagnostics.
* Remove diagnostics.
Related to https://build.spring.io/browse/INT-MASTERSPRING40-1393
The connection factory must be closed independently of test outcome.
Otherwise the ActiveMQ resource is kept in the memory and the rest
of test in JMS module fails with misleading message like:
`javax.management.InstanceAlreadyExistsException: org.apache.activemq:type=Broker,brokerName=localhost`
* GH-3521: Delayer: schedule release task with TX
Fixes https://github.com/spring-projects/spring-integration/issues/3521
There is a race condition when transactional `MessageStore` is used
for `DelayHandler`, so the message is not visible for reads until after TX is
committed, but a scheduled release task may be already ready after delay
* Register a `TransactionSynchronization` with scheduling a releasing task
when TX is committed
**Cherry-pick to `5.4.x` & `5.3.x`**
* Fix language in delayer.adoc
Co-authored-by: Gary Russell <grussell@vmware.com>
Co-authored-by: Gary Russell <grussell@vmware.com>
* Introduce a `groupConditionSupplier` for MGS
* Add a `MessageGroup.condition` option
* Add a `MessageGroupStore.conditionSupplier` option
* Use it from the `SimpleMessageStore.addMessagesToGroup()` API
to populate `condition` (if any) into a `MessageGroup`
* Introduce a `GroupConditionProvider` contract to be implemented
on those `ReleaseStrategy` contracts which could be aware of group condition
* Populated a `GroupConditionProvider.getGroupConditionSupplier()`
into a `MessageGroupStore` from the `AbstractCorrelatingMessageHandler`
for end-user convenience
* Rework a `FileMarkerReleaseStrategy` to implement a `GroupConditionProvider`
to provide a function which produces a condition from `file_lineCount` header
of the `END` marker message
* Make the `FileMarkerReleaseStrategy` logic already based on the condition from a group
* Delegate `GroupConditionProvider` from the `FileAggregator`
* Add test for empty file aggregation
* * Implement `condition` in the `AbstractKeyValueMessageStore` and `MongoDbMessageStore`
* Test `condition` for `mongo-aggregator-config.xml` and `FileAggregatorTests` against GemFire
* * Implement `condition` in the `ConfigurableMongoDbMessageStore`
* * Implement `condition` in the `JdbcMessageStore`
* `FileAggregatorTests` against `JdbcMessageStore`
* Refactor `JdbcMessageStore` for better handling of message group metadata
* Remove unused `MARKED` column in the DDL in favor of newly introduced `CONDITION`
* * Add docs for message group condition
* * Move `conditionSupplier` option from MGS to AbstractCorrelatingMH
* Make it as a `BiFunction` to propagate existing condition alongside with the
message to consult
* Expose `groupConditionSupplier` in Java & XML DSLs
* * Fix language in docs
Resolves https://github.com/spring-projects/spring-integration/issues/3523
- `TcpNetConnection` - publish open event on reader thread to avoid race with first read.
- Intercepted connections - Ensure that the event source is always the outermost interceptor.
- Also reduce delays during NIO client connect.
**cherry-pick to 5.4.x, 5.3.x**
5.3.x will require cherry picking GH-3509 commits.
Resolves https://github.com/spring-projects/spring-integration/issues/3509
When a connection was intercepted, the `TcpSender.add...` was called with the
interceptor, but the `removeDead...` was called with the actual connection,
causing a memory leak.
Always call the `TcpSender` with the outer-most interceptor.
# Conflicts:
# spring-integration-ip/src/main/java/org/springframework/integration/ip/tcp/connection/TcpNetServerConnectionFactory.java
# spring-integration-ip/src/test/java/org/springframework/integration/ip/tcp/TcpSendingMessageHandlerTests.java
* INT-4116: Introduce FileAggregator
JIRA: https://jira.spring.io/browse/INT-4116
* Implement a `FileSplitter.FileMarker`-based aggregation strategies
and utilize them in a general `FileAggregator` component
* Make `HeaderAttributeCorrelationStrategy.attributeName` as `final`; add `Assert.notEmpty()`
* Fix `AggregatorFactoryBean` and `AggregatorSpec` to parse the provided processor
for possible `CorrelationStrategy` and/or `ReleaseStrategy`
* Introduce short-cut methods into Java & Kotlin DSL for an `aggregate()` configuration
* Introduce a `FileHeaders.LINE_COUNT` for header to be populated in the `FileSplitter`.
We need this info in the `FileAggregator` to avoid possible overhead with JSON deserialization
of the `FileSplitter.FileMarker` messages
* Test and document the feature
* Improve `FileSplitter` doc for code block switch (tabs)
* * Rework FileAggregator do not use Java Streams
