Fixes: #3444
* Add `DistributedLock` interface which is implemented by `RedisLock` and `JdbcLock`.
* Modify `LockRegistry`, `ExpirableLockRegistry`, `RenewableLockRegistry` interfaces.
* Modify implementation of `DefaultLockRepository, `JdbcLockRegistry`, `RedisLockRegistry`
* Modify ddl of `INT_LOCK` table.
* Maintain test cases and documents.
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Modify `LockRegistry`, `DistributedLock` interfaces.
* Modify implementation of `DefaultLockRepository, `JdbcLockRegistry`, `RedisLockRegistry`
* Modify ddl of `INT_LOCK` table.
* Maintain test cases and documents.
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Modify DistributedLock` interfaces.
* Modify implementation of `JdbcLockRegistry`, `RedisLockRegistry`.
* Modify JavaDoc.
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Maintain document
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
GH-3444: Add Custom TTL support for RedisLock, and JdbcLock
Fixes: #3444
* Add @Deprecated
Signed-off-by: Eddie Cho <chochoforwork@gmail.com>
Fixes: #9988
Issue link: https://github.com/spring-projects/spring-integration/issues/9988
This change allows dynamic determination of `FileExistsMode` using SpEL expressions,
making the component more flexible when handling file existence conflicts.
* Add `fileExistsModeExpression` field and setter methods
* Use `resolveFileExistsMode()` in put and get operations
* Add changes to the docs
* Ignore `temporaryFileName` when `FileExistsMode.APPEND`
Improve runtime behavior by ignoring temporary filename settings when file exists mode is `APPEND`.
Now, in `FileExistsMode.APPEND` mode, content is always appended directly to the original file regardless of `useTemporaryFileName` setting.
In `RemoteFileTemplate`:
- Remove exception validation when `APPEND` mode is used with temporary filenames
- Modify logic to skip applying `temporaryFileSuffix` in `APPEND` mode
In `AbstractRemoteFileOutboundGateway`:
- Remove logic that disabled temporary filenames when setting `APPEND` mode
* Apply review feedback on `FileExistsMode` expression
- Optimize `EvaluationContext` usage by creating it once in `doInit()`
- Enhance expression evaluation to support String representation of `FileExistsMode`
- Optimize temporary filename handling logic in `RemoteFileTemplate`
- Add warning message for incompatible `APPEND` mode with temporary filenames
- Rename method to `setFileExistsModeExpressionString` for consistency
- Update Java DSL support in `RemoteFileOutboundGatewaySpec`
- Update reference documentation and release notes
* Apply additional review feedback on `FileExistsMode` expression
- Add `Function` variant to `RemoteFileOutboundGatewaySpec`
- Update documentations to use one-sentence-per-line style
- Improve code flow in `resolveFileExistsMode()` method
* Fix additional review feedback on `FileExistsMode` expression
- Use `Object` instead of `String` in `fileExistsModeFunction`
- Fix return method call in `fileExistsModeFunction` (`remoteDirectoryExpression` -> `fileExistsModeExpression`)
- Fix `standardEvaluationContext` initialization in `doInit()`
- Ensure proper `EvaluationContext` usage in other methods
Signed-off-by: Jooyoung Pyoung <pyoungjy@gmail.com>
Fixes: https://github.com/spring-projects/spring-integration/issues/9869
Currently, FileListFilters on the last modified attribute of a file are limited to the use case of discarding files that are too young.
It would be great to have a filter implementation which would accept files which are not old enough.
* Implement `AbstractRecentFileListFilter` for all the supported file protocols
Fixes: https://github.com/spring-projects/spring-integration/issues/9792
The `StreamTransformer` closes `IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE` header value,
so this becomes unusable afterward.
In addition, it may even cause some problems downstream when the message could be serialized for
subsequent network interaction.
* Add logic to the `StreamTransformer` to build a new message, but remove `IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE` header
* Verify that header is removed in the `StreamingInboundTests`.
Addition `StreamingInboundTests` clean up for better code style.
Fixes: https://github.com/spring-projects/spring-integration/issues/9683
Since we don't support SpEL-based Control Bus functionality anymore,
there is no need to keep separate `controlBusOnRegistry()`
* In all DSLs Deprecate `controlBusOnRegistry()` and restore `controlBus()` which is now fully based
on the `ControlBusCommandRegistry`
* Deprecate now out of use `<control-bus use-registry="">` attribute
* Fix `ControlBusParser` to not deal with `use-registry` attribute anymore
* Remove deprecated before `ExpressionControlBusFactoryBean` and `ExpressionCommandMessageProcessor`
* Remove `use-registry` from test configs
* Fix `ControlBusChainTests` to rely on a new Control Bus functionality
Fixes: #9684
Issue link: https://github.com/spring-projects/spring-integration/issues/9684
The `RemoteFileTemplate` is missing to create a temporary remote directory
* Fix `RemoteFileTemplate.sendFileToRemoteDirectory()` to create a `temporaryRemoteDirectory`
as well if it is different from already created `remoteDirectory`
* Add `SftpServerOutboundTests.autoCreateTemporaryDirectory()` to verify that `temporaryRemoteDirectory` is created and in-use
* Some other `SftpServerOutboundTests` refactoring for better code style
**Auto-cherry-pick to `6.3.x`**
Fixes: #9594
Issue link: https://github.com/spring-projects/spring-integration/issues/9594
The `String.replaceFirst()` for directory with `[]` or `()` leads to a regex execution
which does not really replace the root path because of mismatch between regex and file path.
Essentially, the `Matcher.quoteReplacement()` does not do the trick we would expect from it.
* Use `Path.relativize()` API instead which works in canonical paths and proper file separators
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9427
Issue link: https://github.com/spring-projects/spring-integration/issues/9427
The `expression` of the `AbstractRemoteFileOutboundGateway` could be `null` and
ignored for some expressions making a DSL factories inconsistent
* Improve `AbstractRemoteFileOutboundGateway` constructor JavaDocs explaining
`expression` argument in more details
* Add `@Nullable` for this `expression` arg in all the `AbstractRemoteFileOutboundGateway` implementations
* Add new DSL factory method for `outboundGateway()` without this `expression` for those commands when it is not needed
Even if `Executors.newSingleThreadExecutor()` returns a `FinalizableDelegatedExecutorService`,
an instance is kept in the memory until JVM exists.
That may lead to memory leak since we have a lot of threads in memory.
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9381
Currently, there is no way to know in one place what Control Bus commands are available and with what arguments
* Add `ControlBusCommandRegistry` infrastructure bean to gather control bus commands from beans and expose them for invocation
* Add `ControlBusController` to expose a `/control-bus` REST service against the mentioned `ControlBusCommandRegistry`
* Add `@EnableIntegrationManagement(loadControlBusCommands)` to be able to load all the Control Bus commands from the application context instead of on demand by default
* Deprecated existing SpEL(and Groovy)-based Control Bus functionality in favor of new, more manageable, logic
* GH-9294: Set permissions to target file on rename
Fixes: #9294
When the `deleteSourceFiles` property of `FileWritingMessageHandler` is `true`, the `chmod` is not set on the target file.
* Fix `FileWritingMessageHandler.handleFileMessage()` logic to `setPermissions(resultFile)` after `move` operation
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
* * Set only `OWNER_READ` permission for file in unit test
Fixes: #8898
In some cases not all retched remote files are processed, and after changing the `SessionFactory` (e.g. `RotatingServerAdvice`)
thy might not be processed on the next polling cycle
Related to: #9129
The `remoteDirectoryPath` might be `null`, so it is not correct to attempt `remoteDirectoryPath.charAt(0)`.
Plus `/null` is not correct path.
* Use `/` for empty remote dir.
* Check for `remoteDirectoryPath != null`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9129
If `.remoteDirectory("/sftpSource")`, then `FileHeaders.REMOTE_DIRECTORY` is `//sftpSource`
* Fix `AbstractInboundFileSynchronizer` to not add `/` if one is already present in the beginning of the `remoteDirectoryPath`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #8967
In the `fair` mode the `StandardRotationPolicy` re-configures an `AbstractFetchLimitingMessageSource`
for a new directory (and possible new `ConnectionFactory`) in the `beforeReceive()`.
However, with default `maxFetchSize` (or bigger than `1`), the `receive()`` would poll `toBeReceived` internal queue
for files cached from the previous polling cycle.
Since we rotate the source immediately to a new set of options, all those cached files don't make sense
or even can cause the problem on fetching their content in case of `AbstractRemoteFileStreamingMessageSource`
when we rotate to a new `ConnectionFactory`.
* Call `fetchLimitingMessageSource.setMaxFetchSize(1);` in the `StandardRotationPolicy.beforeReceive()`
when `fair && !this.initialized`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Related to: #8856
Many tests create their own `ThreadPoolTaskScheduler` beans.
Therefore, its default phase might affect the memory and performance.
* Use `phase = SmartLifecycle.DEFAULT_PHASE / 2` for manual
`ThreadPoolTaskScheduler` beans
* Migrate affected tests classes to JUnit 5
* Make some other configuration adjustments for better performance
**Cherry-pick to `6.2.x`**
The Checkstyle violation for unused imports in the `FileTailingMessageProducerTests`
Leftover from the previous fix for this class
**Cherry-pick to `6.1.x`**
The `OSDelegatingFileTailingMessageProducer` publishes event from scheduled task
* Modify `FileTailingMessageProducerTests.testGuts()` to use `CountDownLatch`
to wait for those asynchronous events
**Cherry-pick to `6.1.x`**
Fixes: gh-8800
The `SmbFile.getName()` returns plain file name, not with a
sub-directory as it is expected by the `AbstractRemoteFileOutboundGateway`
for (S)FTP protocols
* Introduce `AbstractRemoteFileOutboundGateway.getFullFileName()`
with dir and file name concatenation by default (as it was before)
and override it in the `SmbOutboundGateway` to return the full path according this protocol.
* Fix `SmbOutboundGateway.enhanceNameWithSubDirectory()` to return an `SmbFile` instance
since it has all the info about remote file, include requested directory
* Fix `SmbOutboundGateway.getFilename()` to remove the trailing `/` in the dir name
since this is not what expected by the `AbstractRemoteFileOutboundGateway` logic
**Cherry-pick to `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8792
1. Use a `SftpStreamingMessageSource` with a `maxFetchSize = 5` and a `ChainFileListFilter` filter composed with
`SftpSystemMarkerFilePresentFileListFilter` which `supportsSingleFileFiltering == false`
2. Put 2 files in the folder and invoke `SftpStreamingMessageSource.receive()` method twice.
3. Put 5 files in the folder and invoke `SftpStreamingMessageSource.receive()`` method five times.
4. The last two files won't be received.
When you set max fetch size to a number bigger than one (for example 5) and at a certain point
it is necessary to `this.toBeReceived.clear()` inside `AbstractRemoteFileStreamingMessageSource.doReceive()`,
those removed elements from toBeReceived are not rolled back.
* Fix `AbstractRemoteFileStreamingMessageSource.listFiles()` to calculate `maxFetchSize` as `getMaxFetchSize() - this.fetched.get()`
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
Related to: https://stackoverflow.com/questions/77268009/how-to-use-sftp-spring-integration-mput-with-sftpoutboundgateway-when-file-objec
In some scenarios when the flow starts with a file inbound channel adapter
and then an MPUT operation is performed for remote file outbound gateway,
the populated in the beginning `FileHeaders.FILENAME` is used from
the `DefaultFileNameGenerator` for all the files from local directory to
upload.
Such a behaviour leads only to the last file in the target remote
directory and only with the name from that header.
* Fix the `AbstractRemoteFileOutboundGateway` to remove a `FileHeaders.FILENAME` header
when message is build for specific item from MPUT request.
This way an original local file is used when we upload directory.
* GH-8745: Add RFT.shouldMarkSessionAsDirty()
Fixes https://github.com/spring-projects/spring-integration/issues/8745
Not all errors caught in the `RemoteFileTemplate.execute()`
are fatal to mark session as dirty and physically close the target session
in the cache
* Introduce a `RemoteFileTemplate.shouldMarkSessionAsDirty()`
to consult with an exception if it is really a fatal error to close
the session in the end.
* Override `shouldMarkSessionAsDirty()` in the `RemoteFileTemplate`
implementations to check statuses of respective protocol errors
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Fix tests for pool interaction
* * Fix language in Javadocs
* Add more `not dirty` statuses to `SftpRemoteFileTemplate` & `SmbRemoteFileTemplate`
* Fix Checkstyle violations for `private` inner classes where they have to be also `final`
* Fix deprecations after upgrades
* Remove manual coroutines handling in the `GatewayProxyFactoryBean`
in favor of the same functionality in the `JdkDynamicProxy`
Fixes https://github.com/spring-projects/spring-integration/issues/8691
* Remove setAge with TimeUnit
Turned out we don't need anymore since we're using Duration for age in FtpLastModifiedFileListFilter and SftpLastModifiedFileListFilter.
* Add changes to the docs
* Introduce AbstractLastModifiedFileListFilter
* Some code readability improvements
* Make language in the docs more official
* GH-2971: Add `LockRegistry.executeLocked()` API
Fixes https://github.com/spring-projects/spring-integration/issues/2971
* Following best practice and well-known patterns with `Jdbc`, `Rest` or `Jms` templates,
introduce `default` methods into `LockRegistry` interface to make it easier to perform
tasks when within a lock.
* Since all the required logic is now covered by those `LockRegistry.executeLocked()` methods,
there is no need in the dedicated abstract `WhileLockedProcessor` class.
Deprecated it for removal in the next version
* Use a new `LockRegistry.executeLocked()` API in the `FileWritingMessageHandler`
instead of just deprecated `WhileLockedProcessor`
* To satisfy Java limitations for checked lambdas, introduce `CheckedCallable` and `CheckedRunnable` utilities
similar to interfaces in the `io.micrometer.observation.Observation`
* Change existing `CheckedFunction` to expose extra generic argument for `Throwable`
* Add dedicated chapter for distributed lock into docs
* Fix some links and typos in the docs
* * Fix Javadoc for `CheckedFunction`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
* Introduce a `SplitterSpec` which can accept possible
splitter variants: `expression`, `function`, `ref` etc.
This way we are going to have a complex endpoint configuration
only with a single method argument.
* Use `SplitterSpec` in a newly introduced `splitWith()`
* Deprecate those `split()` methods which use `SplitterEndpointSpec`.
This method are complex enough because of their several arguments
* Refactor some common `MessageHandler` options initialization
into the `ConsumerEndpointSpec.doGet()`
* Disable failing now `MethodInvokingMessageProcessorTests.testCollectionArgument()`:
or Jackson problem, or fresh SF
* Groovy DSL will be fixed in the separate PR:
https://stackoverflow.com/questions/76595843/groovy-selects-a-defaultgroovymethods-split-instead-of-mine-one
* GH-8626: Provide cleaner `transform()` DSL
Fixes https://github.com/spring-projects/spring-integration/issues/8626
* Add missed `transform(String beanName, @Nullable String methodName)` API
* Introduce a `TransformerSpec` to expose a strict API to configure
transformer variants.
* Introduce `transformWith(Consumer<TransformerSpec>)` as a single point
of all possible transformer and its endpoint options
* Deprecate those `IntegrationFlowDefinition.transform()` variants
which are harder to configure as several lambda arguments
This change will make Kotlin & Groovy DSL more readable and straightforward
* * Use new `transformWith()` in tests where a deprecated API still used
* Add JavaDocs to `TransformerSpec`
* Fix generic types for `BaseIntegrationFlowDefinition.transformWith()` -
they make sense exactly on the `TransformerSpec.transformer()` only
* Apply `transformWith()` for Groovy DSL
* Introduce a new `ClassUtils.isLambda(Object candidate)`
and add a check for Groovy `Closure`
* Fix `GroovyIntegrationFlowDefinition.createConfigurerIfAny()` to propagate a `Consumer` argument
down to the `Closure`
* * Rename `TransformerSpec -> TransformerEndpointSpec` for better context
meaning of the class
* Introduce `KotlinTransformerEndpointSpec` as an extension of the `TransformerEndpointSpec`
to have an `inline fun <reified P> transformer(crossinline function: (P) -> Any)`
for Kotlin style
* Add `KotlinIntegrationFlowDefinition.transformWith(KotlinTransformerEndpointSpec)`
* Deprecate Kotlin methods which are covered by the mentioned `transformWith()`
* Fix tests to use new API
* Mentioned the change in the doc
* The latest `sshd-sftp-2.10.0` has a bug not removing trailing `.`
in the "unrooted" path when we ask to create a remote directory.
It works on Windows well, but fails on UNIX file systems.
* The `commons-io-2.13.0` has `Tailer` ctors deprecated
and exposes a builder API.
It starts a tailer process though unconditionally in
its own thread.
Use `setStartThread(false)` to have the tailer process
managed by our own `TaskExecutor`
* Add `ApacheCommonsFileTailingMessageProducer.setPollingDelayDuration(Duration)`
* Deprecate a `TailerListener` impl on the `ApacheCommonsFileTailingMessageProducer`
in favor of an internal instance
* Fix NPEs in DSL Specs
The `BaseWsInboundGatewaySpec` and `TailAdapterSpec` don't override super methods
and when we call them we fail with NPE since `target` was not populated.
* Fix `BaseWsInboundGatewaySpec` and its inheritors to populate the `target`
from their ctors.
* Remove redundant methods from `BaseWsInboundGatewaySpec` hierarchy
* Propagate `AbstractWebServiceInboundGateway` properties directly to the target
from the `BaseWsInboundGatewaySpec` inheritors
* Override `MessageProducerSpec` methods in the `TailAdapterSpec`
to populate respective option into the `FileTailInboundChannelAdapterFactoryBean`
* Expose more missed options for producer endpoint in the `FileTailInboundChannelAdapterFactoryBean`
**Cherry-pick to `6.0.x` & `5.5.x`**
* * `acceptIfNotNull()` for `errorMessageStrategy` in the `FileTailInboundChannelAdapterFactoryBean`
* * Fix `Unmarshaller` population logic in the `MarshallingWsInboundGatewaySpec`
* GH-3557: Add maxDepth, dirPredicate to FileReadMS
Fixes https://github.com/spring-projects/spring-integration/issues/3557
* Expose a `watchMaxDepth` on the `FileReadingMessageSource` for its `Files.walkFileTree()` API usage
* Add `watchDirPredicate` option ot the `FileReadingMessageSource` to skip sub-tree for `Files.walkFileTree()`
scanning according to some condition against directory `Path`
* Fix language in docs
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3869
* Move `ContextHolderRequestHandlerAdvice` to the `core` module for more general purposes
* Add `ContextHolderRequestHandlerAdviceTests`
* Rework `DelegatingSessionFactoryTests` to rely on the `ContextHolderRequestHandlerAdvice`.
This allows us to remove unnecessary XML configuration for this test class
* Document the feature
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
