Even if `Executors.newSingleThreadExecutor()` returns a `FinalizableDelegatedExecutorService`,
an instance is kept in the memory until JVM exists.
That may lead to memory leak since we have a lot of threads in memory.
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9381
Currently, there is no way to know in one place what Control Bus commands are available and with what arguments
* Add `ControlBusCommandRegistry` infrastructure bean to gather control bus commands from beans and expose them for invocation
* Add `ControlBusController` to expose a `/control-bus` REST service against the mentioned `ControlBusCommandRegistry`
* Add `@EnableIntegrationManagement(loadControlBusCommands)` to be able to load all the Control Bus commands from the application context instead of on demand by default
* Deprecated existing SpEL(and Groovy)-based Control Bus functionality in favor of new, more manageable, logic
* GH-9294: Set permissions to target file on rename
Fixes: #9294
When the `deleteSourceFiles` property of `FileWritingMessageHandler` is `true`, the `chmod` is not set on the target file.
* Fix `FileWritingMessageHandler.handleFileMessage()` logic to `setPermissions(resultFile)` after `move` operation
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
* * Set only `OWNER_READ` permission for file in unit test
Fixes: #8898
In some cases not all retched remote files are processed, and after changing the `SessionFactory` (e.g. `RotatingServerAdvice`)
thy might not be processed on the next polling cycle
Related to: #9129
The `remoteDirectoryPath` might be `null`, so it is not correct to attempt `remoteDirectoryPath.charAt(0)`.
Plus `/null` is not correct path.
* Use `/` for empty remote dir.
* Check for `remoteDirectoryPath != null`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9129
If `.remoteDirectory("/sftpSource")`, then `FileHeaders.REMOTE_DIRECTORY` is `//sftpSource`
* Fix `AbstractInboundFileSynchronizer` to not add `/` if one is already present in the beginning of the `remoteDirectoryPath`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #8967
In the `fair` mode the `StandardRotationPolicy` re-configures an `AbstractFetchLimitingMessageSource`
for a new directory (and possible new `ConnectionFactory`) in the `beforeReceive()`.
However, with default `maxFetchSize` (or bigger than `1`), the `receive()`` would poll `toBeReceived` internal queue
for files cached from the previous polling cycle.
Since we rotate the source immediately to a new set of options, all those cached files don't make sense
or even can cause the problem on fetching their content in case of `AbstractRemoteFileStreamingMessageSource`
when we rotate to a new `ConnectionFactory`.
* Call `fetchLimitingMessageSource.setMaxFetchSize(1);` in the `StandardRotationPolicy.beforeReceive()`
when `fair && !this.initialized`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Related to: #8856
Many tests create their own `ThreadPoolTaskScheduler` beans.
Therefore, its default phase might affect the memory and performance.
* Use `phase = SmartLifecycle.DEFAULT_PHASE / 2` for manual
`ThreadPoolTaskScheduler` beans
* Migrate affected tests classes to JUnit 5
* Make some other configuration adjustments for better performance
**Cherry-pick to `6.2.x`**
The Checkstyle violation for unused imports in the `FileTailingMessageProducerTests`
Leftover from the previous fix for this class
**Cherry-pick to `6.1.x`**
The `OSDelegatingFileTailingMessageProducer` publishes event from scheduled task
* Modify `FileTailingMessageProducerTests.testGuts()` to use `CountDownLatch`
to wait for those asynchronous events
**Cherry-pick to `6.1.x`**
Fixes: gh-8800
The `SmbFile.getName()` returns plain file name, not with a
sub-directory as it is expected by the `AbstractRemoteFileOutboundGateway`
for (S)FTP protocols
* Introduce `AbstractRemoteFileOutboundGateway.getFullFileName()`
with dir and file name concatenation by default (as it was before)
and override it in the `SmbOutboundGateway` to return the full path according this protocol.
* Fix `SmbOutboundGateway.enhanceNameWithSubDirectory()` to return an `SmbFile` instance
since it has all the info about remote file, include requested directory
* Fix `SmbOutboundGateway.getFilename()` to remove the trailing `/` in the dir name
since this is not what expected by the `AbstractRemoteFileOutboundGateway` logic
**Cherry-pick to `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8792
1. Use a `SftpStreamingMessageSource` with a `maxFetchSize = 5` and a `ChainFileListFilter` filter composed with
`SftpSystemMarkerFilePresentFileListFilter` which `supportsSingleFileFiltering == false`
2. Put 2 files in the folder and invoke `SftpStreamingMessageSource.receive()` method twice.
3. Put 5 files in the folder and invoke `SftpStreamingMessageSource.receive()`` method five times.
4. The last two files won't be received.
When you set max fetch size to a number bigger than one (for example 5) and at a certain point
it is necessary to `this.toBeReceived.clear()` inside `AbstractRemoteFileStreamingMessageSource.doReceive()`,
those removed elements from toBeReceived are not rolled back.
* Fix `AbstractRemoteFileStreamingMessageSource.listFiles()` to calculate `maxFetchSize` as `getMaxFetchSize() - this.fetched.get()`
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
Related to: https://stackoverflow.com/questions/77268009/how-to-use-sftp-spring-integration-mput-with-sftpoutboundgateway-when-file-objec
In some scenarios when the flow starts with a file inbound channel adapter
and then an MPUT operation is performed for remote file outbound gateway,
the populated in the beginning `FileHeaders.FILENAME` is used from
the `DefaultFileNameGenerator` for all the files from local directory to
upload.
Such a behaviour leads only to the last file in the target remote
directory and only with the name from that header.
* Fix the `AbstractRemoteFileOutboundGateway` to remove a `FileHeaders.FILENAME` header
when message is build for specific item from MPUT request.
This way an original local file is used when we upload directory.
* GH-8745: Add RFT.shouldMarkSessionAsDirty()
Fixes https://github.com/spring-projects/spring-integration/issues/8745
Not all errors caught in the `RemoteFileTemplate.execute()`
are fatal to mark session as dirty and physically close the target session
in the cache
* Introduce a `RemoteFileTemplate.shouldMarkSessionAsDirty()`
to consult with an exception if it is really a fatal error to close
the session in the end.
* Override `shouldMarkSessionAsDirty()` in the `RemoteFileTemplate`
implementations to check statuses of respective protocol errors
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Fix tests for pool interaction
* * Fix language in Javadocs
* Add more `not dirty` statuses to `SftpRemoteFileTemplate` & `SmbRemoteFileTemplate`
* Fix Checkstyle violations for `private` inner classes where they have to be also `final`
* Fix deprecations after upgrades
* Remove manual coroutines handling in the `GatewayProxyFactoryBean`
in favor of the same functionality in the `JdkDynamicProxy`
Fixes https://github.com/spring-projects/spring-integration/issues/8691
* Remove setAge with TimeUnit
Turned out we don't need anymore since we're using Duration for age in FtpLastModifiedFileListFilter and SftpLastModifiedFileListFilter.
* Add changes to the docs
* Introduce AbstractLastModifiedFileListFilter
* Some code readability improvements
* Make language in the docs more official
* GH-2971: Add `LockRegistry.executeLocked()` API
Fixes https://github.com/spring-projects/spring-integration/issues/2971
* Following best practice and well-known patterns with `Jdbc`, `Rest` or `Jms` templates,
introduce `default` methods into `LockRegistry` interface to make it easier to perform
tasks when within a lock.
* Since all the required logic is now covered by those `LockRegistry.executeLocked()` methods,
there is no need in the dedicated abstract `WhileLockedProcessor` class.
Deprecated it for removal in the next version
* Use a new `LockRegistry.executeLocked()` API in the `FileWritingMessageHandler`
instead of just deprecated `WhileLockedProcessor`
* To satisfy Java limitations for checked lambdas, introduce `CheckedCallable` and `CheckedRunnable` utilities
similar to interfaces in the `io.micrometer.observation.Observation`
* Change existing `CheckedFunction` to expose extra generic argument for `Throwable`
* Add dedicated chapter for distributed lock into docs
* Fix some links and typos in the docs
* * Fix Javadoc for `CheckedFunction`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
* Introduce a `SplitterSpec` which can accept possible
splitter variants: `expression`, `function`, `ref` etc.
This way we are going to have a complex endpoint configuration
only with a single method argument.
* Use `SplitterSpec` in a newly introduced `splitWith()`
* Deprecate those `split()` methods which use `SplitterEndpointSpec`.
This method are complex enough because of their several arguments
* Refactor some common `MessageHandler` options initialization
into the `ConsumerEndpointSpec.doGet()`
* Disable failing now `MethodInvokingMessageProcessorTests.testCollectionArgument()`:
or Jackson problem, or fresh SF
* Groovy DSL will be fixed in the separate PR:
https://stackoverflow.com/questions/76595843/groovy-selects-a-defaultgroovymethods-split-instead-of-mine-one
* GH-8626: Provide cleaner `transform()` DSL
Fixes https://github.com/spring-projects/spring-integration/issues/8626
* Add missed `transform(String beanName, @Nullable String methodName)` API
* Introduce a `TransformerSpec` to expose a strict API to configure
transformer variants.
* Introduce `transformWith(Consumer<TransformerSpec>)` as a single point
of all possible transformer and its endpoint options
* Deprecate those `IntegrationFlowDefinition.transform()` variants
which are harder to configure as several lambda arguments
This change will make Kotlin & Groovy DSL more readable and straightforward
* * Use new `transformWith()` in tests where a deprecated API still used
* Add JavaDocs to `TransformerSpec`
* Fix generic types for `BaseIntegrationFlowDefinition.transformWith()` -
they make sense exactly on the `TransformerSpec.transformer()` only
* Apply `transformWith()` for Groovy DSL
* Introduce a new `ClassUtils.isLambda(Object candidate)`
and add a check for Groovy `Closure`
* Fix `GroovyIntegrationFlowDefinition.createConfigurerIfAny()` to propagate a `Consumer` argument
down to the `Closure`
* * Rename `TransformerSpec -> TransformerEndpointSpec` for better context
meaning of the class
* Introduce `KotlinTransformerEndpointSpec` as an extension of the `TransformerEndpointSpec`
to have an `inline fun <reified P> transformer(crossinline function: (P) -> Any)`
for Kotlin style
* Add `KotlinIntegrationFlowDefinition.transformWith(KotlinTransformerEndpointSpec)`
* Deprecate Kotlin methods which are covered by the mentioned `transformWith()`
* Fix tests to use new API
* Mentioned the change in the doc
* The latest `sshd-sftp-2.10.0` has a bug not removing trailing `.`
in the "unrooted" path when we ask to create a remote directory.
It works on Windows well, but fails on UNIX file systems.
* The `commons-io-2.13.0` has `Tailer` ctors deprecated
and exposes a builder API.
It starts a tailer process though unconditionally in
its own thread.
Use `setStartThread(false)` to have the tailer process
managed by our own `TaskExecutor`
* Add `ApacheCommonsFileTailingMessageProducer.setPollingDelayDuration(Duration)`
* Deprecate a `TailerListener` impl on the `ApacheCommonsFileTailingMessageProducer`
in favor of an internal instance
* Fix NPEs in DSL Specs
The `BaseWsInboundGatewaySpec` and `TailAdapterSpec` don't override super methods
and when we call them we fail with NPE since `target` was not populated.
* Fix `BaseWsInboundGatewaySpec` and its inheritors to populate the `target`
from their ctors.
* Remove redundant methods from `BaseWsInboundGatewaySpec` hierarchy
* Propagate `AbstractWebServiceInboundGateway` properties directly to the target
from the `BaseWsInboundGatewaySpec` inheritors
* Override `MessageProducerSpec` methods in the `TailAdapterSpec`
to populate respective option into the `FileTailInboundChannelAdapterFactoryBean`
* Expose more missed options for producer endpoint in the `FileTailInboundChannelAdapterFactoryBean`
**Cherry-pick to `6.0.x` & `5.5.x`**
* * `acceptIfNotNull()` for `errorMessageStrategy` in the `FileTailInboundChannelAdapterFactoryBean`
* * Fix `Unmarshaller` population logic in the `MarshallingWsInboundGatewaySpec`
* GH-3557: Add maxDepth, dirPredicate to FileReadMS
Fixes https://github.com/spring-projects/spring-integration/issues/3557
* Expose a `watchMaxDepth` on the `FileReadingMessageSource` for its `Files.walkFileTree()` API usage
* Add `watchDirPredicate` option ot the `FileReadingMessageSource` to skip sub-tree for `Files.walkFileTree()`
scanning according to some condition against directory `Path`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3869
* Move `ContextHolderRequestHandlerAdvice` to the `core` module for more general purposes
* Add `ContextHolderRequestHandlerAdviceTests`
* Rework `DelegatingSessionFactoryTests` to rely on the `ContextHolderRequestHandlerAdvice`.
This allows us to remove unnecessary XML configuration for this test class
* Document the feature
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
* GH-8562: Fix streaming source for remote calls
Fixes https://github.com/spring-projects/spring-integration/issues/8562
The `AbstractRemoteFileStreamingMessageSource.doReceive()` takes files first from a `toBeReceived` queue.
When `AbstractRemoteFileStreamingMessageSource.remoteFileToMessage()` fails to fetch the file content
because of interim connection issue, we reset this file from a filter and rethrow an exception.
The next `receive()` call will just go ahead to the next entry in the `toBeReceived` queue, but the
file we have just failed for will be retried only on the next list call to the remove directory.
This essentially breaks a possible in-order target application logic.
* Introduce `AbstractRemoteFileStreamingMessageSource.strictOrder` option to clear the `toBeReceived` queue
when we fail in the `remoteFileToMessage()`, so the next `receive()`
call would re-fetch files from remote dir, because the filter has been reset for those files.
* Fix `AbstractFileInfo.toString()` to not perform remote calls when we just log this file.
For example, we reset the file for connection failure and log the message about it,
but it fails again because we request `size` of the file which may require a remote connection.
**Cherry-pick to `6.0.x` & `5.5.x`**
* * Revert `AbstractFileInfo` changes
* Override `toString()` in `SmbFileInfo` instead -
exactly the place where connection is used to obtain
file attributes like `size` or `lastModified`
The default expression in the `DefaultFileNameGenerator`
is based on a message header which simply can be retrieved
with much faster `FunctionExpression`
* Some other code style refactoring in the `DefaultFileNameGenerator`
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
Starting with Spring Data 2022.0.0 there is not going to be GemFire (Geode) support
directly from Spring team
A source code of the `spring-integration-gemfire` is moving to Spring Integration Extensions
from where community may consider to pull it and support in their own manner
* GH-3572: Migrate SFTP from `jsch` to `sshd-sftp`
Fixes https://github.com/spring-projects/spring-integration/issues/3572
* Rework SFTP module from the JSch API to more modern `sshd-sftp`
* Migrate generics of most API from `ChannelSftp.LsEntry` to the `SftpClient.DirEntry`
* Rework `DefaultSftpSessionFactory` to deal with an `SshClient` and create `SftpClient`
wrapped to the `SftpSession`
* Implement a `ResourceKnownHostsServerKeyVerifier` to load `known-hosts` from any possible resource
* Implement an expected `SftpSession.list()` with just file name to take or pattern matching
* Remove some unused tests and their config
* Remove tests for custom `UserInfo` since we don't provide any custom out-of-the-box
* Test a new `ResourceKnownHostsServerKeyVerifier` against default `known-hosts` file
* * Some tests improvements
* * Improve generics handling for `FileUtils`
* GH-3828: Initial Spring AOT support
Fixes https://github.com/spring-projects/spring-integration/issues/3828
* Provide an infrastructure based on a new Spring AOT engine in the latest Spring Framework
* Introduce `RuntimeHintsRegistrar` impls into modules which require some reflection,
resources or proxies and serialization available in the native image
* Mark some framework method with the `@Reflective` to make their reflection
info available in the native image, for example for SpEL or JMX invocations
* Add a `GatewayProxyBeanRegistrationAotProcessor` to register proxy interfaces
info for messaging gateways (either instance of the `GatewayProxyFactoryBean`)
* Rework `ConverterRegistrar` to not use a `beanFactory.getBeansWithAnnotation()`
since it is not available after AOT phase.
Instead, register an intermediate `IntegrationConverterRegistration` bean definition
from the `IntegrationConverterInitializer`
* Refactor `GlobalChannelInterceptorInitializer` a bit according to a new logic in the
`IntegrationConverterInitializer`
* Remove `JsonNodeWrapperToJsonNodeConverter` bean registration from the
`DefaultConfiguringBeanFactoryPostProcessor` - it is added by the `ConverterRegistrar`
into the target `ConversionService`
* Fix `ParentContextTests` respectively a `JsonNodeWrapperToJsonNodeConverter` bean removal
* Refactor `XsltPayloadTransformer` to not load a `ServletContextResource`, but just use its
name for the `xslResource` condition
* * Rework AOT support according latest changes and requirements
* Remove `@Bean` reflection since it is not needed any more
* Add `AotDetector.useGeneratedArtifacts()` condition to not register beans
one more time at runtime after AOT build phase
* Fix deprecation in the WebFlux test from the latest SF
Fixes https://github.com/spring-projects/spring-integration/issues/3827
The `AbstractRemoteFileOutboundGateway.doGet()` for a `STREAM` option
does not close the `session` in case of error.
This may lead to some leaks or exhausted caches
* Close `session` in the `catch()` of the `AbstractRemoteFileOutboundGateway.doGet()`
* Adjust the `SftpServerOutboundTests` to configure a `CachingSessionFactory`
for the `testStream()` to verify there is no leaks attempting to
`GET STREAM` non-existing remote file twice
**Cherry-pick to `5.5.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/3840
* Create a new abstraction `RedisTest` for Testcontainers-based tests
* Move existing common utility methods to the new interface
* Migrate all existing Redis tests to use this new `RedisTest` interface
* Migrate all existing Redis tests to JUnit5
* Make all existing Redis tests confirming to JUnit5 standards, such as
default methods and classes visibility instead of public
* Add a dependency for parametrized tests in JUnit5
* Improve assertions readability across tests, for instance:
assertThat(thing.size()).isEqualTo(2) -> assertThat(thing).hasSize(2)
* Add real assertions to
`RedisLockRegistryTests.twoRedisLockRegistryTest` (earlier it did not
have assertions at all)
* Reformat, rearrange and cleanup the code
* Fix a couple of small changes after the code review:
* Change base interface name to be consistent with other similar places
* Typo in javadocs
* Small tests readability improvement regarding assertions
* Add `opens java.util` to `spring-integration-redis`
to satisfy a `ReactiveRedisStreamMessageHandlerTests.testMessageWithListPayload()`
requirements
Fixes https://github.com/spring-projects/spring-integration/issues/3623
* `IntegrationFlow` refactoring
* Apply several code style improvements and good practices
* Code style: no empty lines for methods javadocs
* make deprecated implementation reuse actual one instead of the copy-paste approach
* add whats-new comments
* Fix whats-new page according to standards
