* `SftpSessionFactoryTests.concurrentGetSessionDoesntCauseFailure()`
may not report properly into an `ArrayList` from another thread.
Use `asyncTaskExecutor.submitCompletable()` instead and deal with their result afterward.
* Use `CompletableFuture` for the `TcpListener` logic in the `FailoverClientConnectionFactoryTests.failoverAllDeadAfterSuccess()`
Fixes: #9909
Issue link: https://github.com/spring-projects/spring-integration/issues/9909
The `SftpSession.close()` closes `sftpClient` and its `clientSession` unconditionally.
At the same time the `DefaultSftpSessionFactory.isSharedSession` is expected to expose only a single shared client.
When this `DefaultSftpSessionFactory` is used concurrently, there is a chance that one thread would close
that shared client and another won't be able to interact due to `clientSession` is closed.
* Fix `SftpSession` accepting an `isSharedClient` flag and doing nothing in the `close()` if client is shared.
* Propagate `isSharedSession` state down to the `SftpSession` from the `DefaultSftpSessionFactory`
* Closed shared client and its `clientSession` in the `DefaultSftpSessionFactory.destroy()`
**Auto-cherry-pick to `6.4.x` & `6.3.x`**
Fixes: https://github.com/spring-projects/spring-integration/issues/9869
Currently, FileListFilters on the last modified attribute of a file are limited to the use case of discarding files that are too young.
It would be great to have a filter implementation which would accept files which are not old enough.
* Implement `AbstractRecentFileListFilter` for all the supported file protocols
Fixes: #9821
Issue link: https://github.com/spring-projects/spring-integration/issues/9821
Previously, Spring Integration used JSch library for SFTP protocol,
and that one was able to follow symlinks.
Currently, the `SftpSession` uses `LSTAT` command which does not follow symlinks
* Fix `SftpSession` to call `sftpClient.stat()` instead of `lstat()`
to mitigate migration pain from JSch
**Auto-cherry-pick to `6.4.x` & `6.3.x`**
Fixes: #9796
Issue link: https://github.com/spring-projects/spring-integration/issues/9796
The `org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync` is shared object for the `DefaultSftpClient`
and it cannot be used concurrently.
The guarded `send()` operation in the `ConcurrentSftpClient` is not enough
since `DefaultSftpClient.write()` is called directly from the `SftpOutputStreamAsync.internalFlush()`.
And this in the end is called from the `SftpSession.write()`
* Fix `DefaultSftpSessionFactory.ConcurrentSftpClient` to override the `write()` method instead.
Guard it with a `Lock` and call `sftpMessage.waitUntilSent();` to ensure that no concurrent access
to the underlying `SftpOutputStreamAsync`.
**Auto-cherry-pick to `6.4.x` & `6.3.x`**
Fixes: #9684
Issue link: https://github.com/spring-projects/spring-integration/issues/9684
The `RemoteFileTemplate` is missing to create a temporary remote directory
* Fix `RemoteFileTemplate.sendFileToRemoteDirectory()` to create a `temporaryRemoteDirectory`
as well if it is different from already created `remoteDirectory`
* Add `SftpServerOutboundTests.autoCreateTemporaryDirectory()` to verify that `temporaryRemoteDirectory` is created and in-use
* Some other `SftpServerOutboundTests` refactoring for better code style
**Auto-cherry-pick to `6.3.x`**
Fixes: #9427
Issue link: https://github.com/spring-projects/spring-integration/issues/9427
The `expression` of the `AbstractRemoteFileOutboundGateway` could be `null` and
ignored for some expressions making a DSL factories inconsistent
* Improve `AbstractRemoteFileOutboundGateway` constructor JavaDocs explaining
`expression` argument in more details
* Add `@Nullable` for this `expression` arg in all the `AbstractRemoteFileOutboundGateway` implementations
* Add new DSL factory method for `outboundGateway()` without this `expression` for those commands when it is not needed
Fixes: #9380
Expose a `Consumer<SshClient> sshClientConfigurer` option for the `DefaultSftpSessionFactory`
to further customize an internal `SshClient` instance.
Even if `Executors.newSingleThreadExecutor()` returns a `FinalizableDelegatedExecutorService`,
an instance is kept in the memory until JVM exists.
That may lead to memory leak since we have a lot of threads in memory.
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9272
* close ClientSession when closing SftpSession
* fix whitespace issues
* stop the SshClient on bean destruction
* use convenient assertions
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9129
If `.remoteDirectory("/sftpSource")`, then `FileHeaders.REMOTE_DIRECTORY` is `//sftpSource`
* Fix `AbstractInboundFileSynchronizer` to not add `/` if one is already present in the beginning of the `remoteDirectoryPath`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9114
If path is not in normalized presentation, the SFTP operation might fail like:
```
Caused by: SFTP error (SSH_FX_NO_SUCH_PATH): The file path does not exist or is invalid.
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.throwStatusException(AbstractSftpClient.java:277)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributesResponse(AbstractSftpClient.java:333)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributes(AbstractSftpClient.java:325)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.lstat(AbstractSftpClient.java:1010)
at org.springframework.integration.sftp.session.SftpSession.exists(SftpSession.java:191)
```
* Use it now like this `this.sftpClient.lstat(normalizePath(path))`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9123
The `/` at the beginning of the remote dir path is not necessary when listing files, although it is necessary to download them
The `SftpTemplate.get()` should work also with `remote-dir/MyFile.csv` as input.
* Fix `SftpSession.readRaw()` to call `sftpClient.canonicalPath(source)` if the path does not start with a `/`.
Something similar what is does
* Delegate to `SftpSession.readRaw()` from the `SftpSession.read()`
* Reuse `normalizePath()` for `doList()`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8797
After migration to Apache MINA we have missed to fix `DefaultSftpSessionFactory.timeout`
to be `0` by default as it states in its Javadocs and reference manual
It is `null` by default which really means an infinite wait.
* Fix `DefaultSftpSessionFactory.timeout` to be a reasonable 30 seconds by default
* Fix `setTimeout()` Javadocs and respective `session-factory.adoc`
* Propagate this `timeout` down to the `SftpClient` for its commands interactions
Fixes https://github.com/spring-projects/spring-integration/issues/8792
1. Use a `SftpStreamingMessageSource` with a `maxFetchSize = 5` and a `ChainFileListFilter` filter composed with
`SftpSystemMarkerFilePresentFileListFilter` which `supportsSingleFileFiltering == false`
2. Put 2 files in the folder and invoke `SftpStreamingMessageSource.receive()` method twice.
3. Put 5 files in the folder and invoke `SftpStreamingMessageSource.receive()`` method five times.
4. The last two files won't be received.
When you set max fetch size to a number bigger than one (for example 5) and at a certain point
it is necessary to `this.toBeReceived.clear()` inside `AbstractRemoteFileStreamingMessageSource.doReceive()`,
those removed elements from toBeReceived are not rolled back.
* Fix `AbstractRemoteFileStreamingMessageSource.listFiles()` to calculate `maxFetchSize` as `getMaxFetchSize() - this.fetched.get()`
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* GH-8745: Add RFT.shouldMarkSessionAsDirty()
Fixes https://github.com/spring-projects/spring-integration/issues/8745
Not all errors caught in the `RemoteFileTemplate.execute()`
are fatal to mark session as dirty and physically close the target session
in the cache
* Introduce a `RemoteFileTemplate.shouldMarkSessionAsDirty()`
to consult with an exception if it is really a fatal error to close
the session in the end.
* Override `shouldMarkSessionAsDirty()` in the `RemoteFileTemplate`
implementations to check statuses of respective protocol errors
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Fix tests for pool interaction
* * Fix language in Javadocs
* Add more `not dirty` statuses to `SftpRemoteFileTemplate` & `SmbRemoteFileTemplate`
Fixes https://github.com/spring-projects/spring-integration/issues/8713
* Introduce `DefaultSftpSessionFactory.createSftpClient()` factory method
which can be overridden for any custom `SftpClient` use-case
* Add changes to the docs
* Some code clean up
**Cherry-pick to `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8691
* Remove setAge with TimeUnit
Turned out we don't need anymore since we're using Duration for age in FtpLastModifiedFileListFilter and SftpLastModifiedFileListFilter.
* Add changes to the docs
* Introduce AbstractLastModifiedFileListFilter
* Some code readability improvements
* Make language in the docs more official
Fixes https://github.com/spring-projects/spring-integration/issues/8708
According to the `org.apache.sshd.common.channel.ChannelAsyncOutputStream.writeBuffer()` JavaDocs cannot be used concurrently.
* Introduce internal `DefaultSftpSessionFactory.ConcurrentSftpClient` extension
of the `DefaultSftpClient` to set a `Lock` around `super.send(cmd, buffer);`
* Remove lock from the `SftpSession` since it now is managed by the mentioned `ConcurrentSftpClient`
**Cherry-pick to `6.1.x` & `6.0.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8674
The `ResourceKnownHostsServerKeyVerifier` does not take into account that several different
keys can be present in the known hosts resource for the same host/port
* Fix `ResourceKnownHostsServerKeyVerifier` to find a list of knows host for the requested session.
Then iterate of this result to match the key type first and then compare keys and their `revoked` marker
**Cherry-pick to `6.1.x` & `6.0.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8563
* Rework the `SftpSession.rename()` logic to be similar to what is there in the `FtpSession`, for example:
try to remove a remote target file before renaming.
* Fix `SftpSession.remove()` to check for `SftpConstants.SSH_FX_NO_SUCH_FILE` response status
to return `false`
Not clear why `SftpRemoteFileTemplateTests.testINT3412AppendStatRmdir()`
is failing on CI Linux, so add a tracing diagnostics to see how Apache SSHD behaves over there
Fixes https://github.com/spring-projects/spring-integration/issues/8581
Do not overwrite configuration of externally provided `SshClient` in the `DefaultSftpSessionFactory`
* Replace JUnit `assertDoesNotThrow` by AssertJ `assertThatNoException` in test
**Cherry-pick to `6.0.x`**
Currently, many timeouts in the project are like `-1` or other negative value
with a meaning to wait indefinitely.
According to distributed systems design and bad demo developing experience
it is not OK to block forever.
* Rework most of the timeouts in the framework to be `30` seconds.
Only one remained as `1` seconds is a `PollingConsumer` where it is
better to not block even for those 30 seconds when no messages in the queue,
but let the polling task be rescheduled.
* Remove the `MessagingGatewaySupport.replyTimeout` propagation down to the
`PollingConsumer` correlator where it was a `-1` before and blocked
the polling thread on the `Queue.poll()`.
This fixed the problem with a single thread in a pool for auto-configured `TaskScheduler`.
Now with 1 seconds wait time we are able to switch to other scheduled tasks
even with only 1 thread in the pool
* GH-8559: Document how to enable SOCKS for SFTP
Fixes https://github.com/spring-projects/spring-integration/issues/8559
An out-of-the-box `SshClient` does not provide a smooth HTTP/SOCKS proxy configuration.
* Mention in the `sftp.adoc` that `JGitSshClient`, configured with SOCKS,
can be injected into a `DefaultSftpSessionFactory`
* Fix Javadocs for `DefaultSftpSessionFactory`, respectively
* Fix language in doc
Co-authored-by: Gary Russell <grussell@vmware.com>
---------
Co-authored-by: Gary Russell <grussell@vmware.com>
Fixes https://github.com/spring-projects/spring-integration/issues/3980
When not `isSharedSession`, the `initClient()` is called for every session
we request from the factory and in concurrent calls we end up with not initialized SSH client
in some threads.
* Add `synchronized` double check logic to the `initClient()` to block other threads while the first one
initialize the client
* Use `volatile boolean` instead of `AtomicBoolean` and change its state when `SshClient` is created and started
* GH-3962: Support SFTP < v5 for rename
Fixes https://github.com/spring-projects/spring-integration/issues/3962
Turns out the `SftpClient.CopyMode` options are supported only starting SFTP v5
* Check for the version from the client and fallback to the logic we had before with JSsch:
try to rename, delete existing file if such an exception status is thrown and retry to rename
* * Fix `SftpOutboundTests` with answering an explicit SFTP version for an `SftpClient` mock
Fixes https://github.com/spring-projects/spring-integration/issues/3974
* Treat a leading `/` as an indicator of absolute path request in the `SftpSession.doList()`
* Call `sftpClient.canonicalPath()` for path without a leading `/` to resolve it as relative path in the user home
* Add a note in docs for `LS` command
Fixes https://github.com/spring-projects/spring-integration/issues/3969
In previous version for SFTP client (Jsch), the empty path for `LS` command
has meant a `user home`.
Turns out the MINA `SftpClient` does not support automatic user home resolution
from the empty path.
* Fix `SftpSession` to resolve an empty path into a user home via `canicalPath()` operation
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
The plain `SocketAddress.toString()` may resolve
to `hostName/IP` pair which is not parsed properly via `URI.getHost()`
downstream leaving `/IP` part for `uri.getPath()` request.
In the end this may lead to wrong file name to be used in the SFTP logic
* Fix `SftpSession.getHostPort()` to use `SshdSocketAddress` utility
to resolve host and port properly from the provided `SocketAddress`
