Fixes: #9427
Issue link: https://github.com/spring-projects/spring-integration/issues/9427
The `expression` of the `AbstractRemoteFileOutboundGateway` could be `null` and
ignored for some expressions making a DSL factories inconsistent
* Improve `AbstractRemoteFileOutboundGateway` constructor JavaDocs explaining
`expression` argument in more details
* Add `@Nullable` for this `expression` arg in all the `AbstractRemoteFileOutboundGateway` implementations
* Add new DSL factory method for `outboundGateway()` without this `expression` for those commands when it is not needed
Fixes: #9380
Expose a `Consumer<SshClient> sshClientConfigurer` option for the `DefaultSftpSessionFactory`
to further customize an internal `SshClient` instance.
Even if `Executors.newSingleThreadExecutor()` returns a `FinalizableDelegatedExecutorService`,
an instance is kept in the memory until JVM exists.
That may lead to memory leak since we have a lot of threads in memory.
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9272
* close ClientSession when closing SftpSession
* fix whitespace issues
* stop the SshClient on bean destruction
* use convenient assertions
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9129
If `.remoteDirectory("/sftpSource")`, then `FileHeaders.REMOTE_DIRECTORY` is `//sftpSource`
* Fix `AbstractInboundFileSynchronizer` to not add `/` if one is already present in the beginning of the `remoteDirectoryPath`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9114
If path is not in normalized presentation, the SFTP operation might fail like:
```
Caused by: SFTP error (SSH_FX_NO_SUCH_PATH): The file path does not exist or is invalid.
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.throwStatusException(AbstractSftpClient.java:277)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributesResponse(AbstractSftpClient.java:333)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributes(AbstractSftpClient.java:325)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.lstat(AbstractSftpClient.java:1010)
at org.springframework.integration.sftp.session.SftpSession.exists(SftpSession.java:191)
```
* Use it now like this `this.sftpClient.lstat(normalizePath(path))`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9123
The `/` at the beginning of the remote dir path is not necessary when listing files, although it is necessary to download them
The `SftpTemplate.get()` should work also with `remote-dir/MyFile.csv` as input.
* Fix `SftpSession.readRaw()` to call `sftpClient.canonicalPath(source)` if the path does not start with a `/`.
Something similar what is does
* Delegate to `SftpSession.readRaw()` from the `SftpSession.read()`
* Reuse `normalizePath()` for `doList()`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8797
After migration to Apache MINA we have missed to fix `DefaultSftpSessionFactory.timeout`
to be `0` by default as it states in its Javadocs and reference manual
It is `null` by default which really means an infinite wait.
* Fix `DefaultSftpSessionFactory.timeout` to be a reasonable 30 seconds by default
* Fix `setTimeout()` Javadocs and respective `session-factory.adoc`
* Propagate this `timeout` down to the `SftpClient` for its commands interactions
Fixes https://github.com/spring-projects/spring-integration/issues/8792
1. Use a `SftpStreamingMessageSource` with a `maxFetchSize = 5` and a `ChainFileListFilter` filter composed with
`SftpSystemMarkerFilePresentFileListFilter` which `supportsSingleFileFiltering == false`
2. Put 2 files in the folder and invoke `SftpStreamingMessageSource.receive()` method twice.
3. Put 5 files in the folder and invoke `SftpStreamingMessageSource.receive()`` method five times.
4. The last two files won't be received.
When you set max fetch size to a number bigger than one (for example 5) and at a certain point
it is necessary to `this.toBeReceived.clear()` inside `AbstractRemoteFileStreamingMessageSource.doReceive()`,
those removed elements from toBeReceived are not rolled back.
* Fix `AbstractRemoteFileStreamingMessageSource.listFiles()` to calculate `maxFetchSize` as `getMaxFetchSize() - this.fetched.get()`
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* GH-8745: Add RFT.shouldMarkSessionAsDirty()
Fixes https://github.com/spring-projects/spring-integration/issues/8745
Not all errors caught in the `RemoteFileTemplate.execute()`
are fatal to mark session as dirty and physically close the target session
in the cache
* Introduce a `RemoteFileTemplate.shouldMarkSessionAsDirty()`
to consult with an exception if it is really a fatal error to close
the session in the end.
* Override `shouldMarkSessionAsDirty()` in the `RemoteFileTemplate`
implementations to check statuses of respective protocol errors
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Fix tests for pool interaction
* * Fix language in Javadocs
* Add more `not dirty` statuses to `SftpRemoteFileTemplate` & `SmbRemoteFileTemplate`
Fixes https://github.com/spring-projects/spring-integration/issues/8713
* Introduce `DefaultSftpSessionFactory.createSftpClient()` factory method
which can be overridden for any custom `SftpClient` use-case
* Add changes to the docs
* Some code clean up
**Cherry-pick to `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8691
* Remove setAge with TimeUnit
Turned out we don't need anymore since we're using Duration for age in FtpLastModifiedFileListFilter and SftpLastModifiedFileListFilter.
* Add changes to the docs
* Introduce AbstractLastModifiedFileListFilter
* Some code readability improvements
* Make language in the docs more official
Fixes https://github.com/spring-projects/spring-integration/issues/8708
According to the `org.apache.sshd.common.channel.ChannelAsyncOutputStream.writeBuffer()` JavaDocs cannot be used concurrently.
* Introduce internal `DefaultSftpSessionFactory.ConcurrentSftpClient` extension
of the `DefaultSftpClient` to set a `Lock` around `super.send(cmd, buffer);`
* Remove lock from the `SftpSession` since it now is managed by the mentioned `ConcurrentSftpClient`
**Cherry-pick to `6.1.x` & `6.0.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8674
The `ResourceKnownHostsServerKeyVerifier` does not take into account that several different
keys can be present in the known hosts resource for the same host/port
* Fix `ResourceKnownHostsServerKeyVerifier` to find a list of knows host for the requested session.
Then iterate of this result to match the key type first and then compare keys and their `revoked` marker
**Cherry-pick to `6.1.x` & `6.0.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8563
* Rework the `SftpSession.rename()` logic to be similar to what is there in the `FtpSession`, for example:
try to remove a remote target file before renaming.
* Fix `SftpSession.remove()` to check for `SftpConstants.SSH_FX_NO_SUCH_FILE` response status
to return `false`
Not clear why `SftpRemoteFileTemplateTests.testINT3412AppendStatRmdir()`
is failing on CI Linux, so add a tracing diagnostics to see how Apache SSHD behaves over there
Fixes https://github.com/spring-projects/spring-integration/issues/8581
Do not overwrite configuration of externally provided `SshClient` in the `DefaultSftpSessionFactory`
* Replace JUnit `assertDoesNotThrow` by AssertJ `assertThatNoException` in test
**Cherry-pick to `6.0.x`**
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
* GH-8559: Document how to enable SOCKS for SFTP
Fixes https://github.com/spring-projects/spring-integration/issues/8559
An out-of-the-box `SshClient` does not provide a smooth HTTP/SOCKS proxy configuration.
* Mention in the `sftp.adoc` that `JGitSshClient`, configured with SOCKS,
can be injected into a `DefaultSftpSessionFactory`
* Fix Javadocs for `DefaultSftpSessionFactory`, respectively
* Fix language in doc
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3980
When not `isSharedSession`, the `initClient()` is called for every session
we request from the factory and in concurrent calls we end up with not initialized SSH client
in some threads.
* Add `synchronized` double check logic to the `initClient()` to block other threads while the first one
initialize the client
* Use `volatile boolean` instead of `AtomicBoolean` and change its state when `SshClient` is created and started
* GH-3962: Support SFTP < v5 for rename
Fixes https://github.com/spring-projects/spring-integration/issues/3962
Turns out the `SftpClient.CopyMode` options are supported only starting SFTP v5
* Check for the version from the client and fallback to the logic we had before with JSsch:
try to rename, delete existing file if such an exception status is thrown and retry to rename
* * Fix `SftpOutboundTests` with answering an explicit SFTP version for an `SftpClient` mock
Fixes https://github.com/spring-projects/spring-integration/issues/3974
* Treat a leading `/` as an indicator of absolute path request in the `SftpSession.doList()`
* Call `sftpClient.canonicalPath()` for path without a leading `/` to resolve it as relative path in the user home
* Add a note in docs for `LS` command
Fixes https://github.com/spring-projects/spring-integration/issues/3969
In previous version for SFTP client (Jsch), the empty path for `LS` command
has meant a `user home`.
Turns out the MINA `SftpClient` does not support automatic user home resolution
from the empty path.
* Fix `SftpSession` to resolve an empty path into a user home via `canicalPath()` operation
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
The plain `SocketAddress.toString()` may resolve
to `hostName/IP` pair which is not parsed properly via `URI.getHost()`
downstream leaving `/IP` part for `uri.getPath()` request.
In the end this may lead to wrong file name to be used in the SFTP logic
* Fix `SftpSession.getHostPort()` to use `SshdSocketAddress` utility
to resolve host and port properly from the provided `SocketAddress`
The `192.168.1.61` is Private Use IP.
It is an intranet IP address, which is usually allocated to mobile phones, desktops, laptops, TVs, smart speakers and other devices.
The logic is taken from a similar test class in the Apache MINA project
* GH-3572: Migrate SFTP from `jsch` to `sshd-sftp`
Fixes https://github.com/spring-projects/spring-integration/issues/3572
* Rework SFTP module from the JSch API to more modern `sshd-sftp`
* Migrate generics of most API from `ChannelSftp.LsEntry` to the `SftpClient.DirEntry`
* Rework `DefaultSftpSessionFactory` to deal with an `SshClient` and create `SftpClient`
wrapped to the `SftpSession`
* Implement a `ResourceKnownHostsServerKeyVerifier` to load `known-hosts` from any possible resource
* Implement an expected `SftpSession.list()` with just file name to take or pattern matching
* Remove some unused tests and their config
* Remove tests for custom `UserInfo` since we don't provide any custom out-of-the-box
* Test a new `ResourceKnownHostsServerKeyVerifier` against default `known-hosts` file
* * Some tests improvements
* * Improve generics handling for `FileUtils`
Fixes https://github.com/spring-projects/spring-integration/issues/3827
The `AbstractRemoteFileOutboundGateway.doGet()` for a `STREAM` option
does not close the `session` in case of error.
This may lead to some leaks or exhausted caches
* Close `session` in the `catch()` of the `AbstractRemoteFileOutboundGateway.doGet()`
* Adjust the `SftpServerOutboundTests` to configure a `CachingSessionFactory`
for the `testStream()` to verify there is no leaks attempting to
`GET STREAM` non-existing remote file twice
**Cherry-pick to `5.5.x`**
* Mostly changes are related to the `TaskScheduler` and `Trigger` APIs
* Migrate to `micrometer-tracing` dependency
* Rework `SocketTestUtils` to use a `InetAddress.getLocalHost()`
for more stability and performance on Windows
* Fix docs for new `PeriodicTrigger` API
Fixes https://github.com/spring-projects/spring-integration/issues/3623
* `IntegrationFlow` refactoring
* Apply several code style improvements and good practices
* Code style: no empty lines for methods javadocs
* make deprecated implementation reuse actual one instead of the copy-paste approach
* add whats-new comments
* Fix whats-new page according to standards
* Upgrade Spring dependencies to the latest SNAPSHOTs
* Fix tests to verify against stack traces: the message
of the `NestedRuntimeException` does not include the nested exception information.
Related to https://github.com/spring-projects/spring-framework/issues/25162
* Fix `JdbcMessageStore` and `DefaultLockRepository` to rely on the `DataIntegrityViolationException`
instead of only its `DuplicateKeyException` extension.
This is the current behavior of the SQL errors translation
* Disable `WebFluxDslTests.testValidation()` - doesn't subscribe to the reply somehow...
* Refine `SimplePool.PoolSemaphore.reducePermits()`
