* `SftpSessionFactoryTests.concurrentGetSessionDoesntCauseFailure()`
may not report properly into an `ArrayList` from another thread.
Use `asyncTaskExecutor.submitCompletable()` instead and deal with their result afterward.
* Use `CompletableFuture` for the `TcpListener` logic in the `FailoverClientConnectionFactoryTests.failoverAllDeadAfterSuccess()`
Fixes: #9909
Issue link: https://github.com/spring-projects/spring-integration/issues/9909
The `SftpSession.close()` closes `sftpClient` and its `clientSession` unconditionally.
At the same time the `DefaultSftpSessionFactory.isSharedSession` is expected to expose only a single shared client.
When this `DefaultSftpSessionFactory` is used concurrently, there is a chance that one thread would close
that shared client and another won't be able to interact due to `clientSession` is closed.
* Fix `SftpSession` accepting an `isSharedClient` flag and doing nothing in the `close()` if client is shared.
* Propagate `isSharedSession` state down to the `SftpSession` from the `DefaultSftpSessionFactory`
* Closed shared client and its `clientSession` in the `DefaultSftpSessionFactory.destroy()`
**Auto-cherry-pick to `6.4.x` & `6.3.x`**
Fixes: https://github.com/spring-projects/spring-integration/issues/9869
Currently, FileListFilters on the last modified attribute of a file are limited to the use case of discarding files that are too young.
It would be great to have a filter implementation which would accept files which are not old enough.
* Implement `AbstractRecentFileListFilter` for all the supported file protocols
Fixes: #9821
Issue link: https://github.com/spring-projects/spring-integration/issues/9821
Previously, Spring Integration used JSch library for SFTP protocol,
and that one was able to follow symlinks.
Currently, the `SftpSession` uses `LSTAT` command which does not follow symlinks
* Fix `SftpSession` to call `sftpClient.stat()` instead of `lstat()`
to mitigate migration pain from JSch
**Auto-cherry-pick to `6.4.x` & `6.3.x`**
Fixes: #9684
Issue link: https://github.com/spring-projects/spring-integration/issues/9684
The `RemoteFileTemplate` is missing to create a temporary remote directory
* Fix `RemoteFileTemplate.sendFileToRemoteDirectory()` to create a `temporaryRemoteDirectory`
as well if it is different from already created `remoteDirectory`
* Add `SftpServerOutboundTests.autoCreateTemporaryDirectory()` to verify that `temporaryRemoteDirectory` is created and in-use
* Some other `SftpServerOutboundTests` refactoring for better code style
**Auto-cherry-pick to `6.3.x`**
Fixes: #9380
Expose a `Consumer<SshClient> sshClientConfigurer` option for the `DefaultSftpSessionFactory`
to further customize an internal `SshClient` instance.
Even if `Executors.newSingleThreadExecutor()` returns a `FinalizableDelegatedExecutorService`,
an instance is kept in the memory until JVM exists.
That may lead to memory leak since we have a lot of threads in memory.
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9272
* close ClientSession when closing SftpSession
* fix whitespace issues
* stop the SshClient on bean destruction
* use convenient assertions
**Auto-cherry-pick to `6.3.x` & `6.2.x`**
Fixes: #9129
If `.remoteDirectory("/sftpSource")`, then `FileHeaders.REMOTE_DIRECTORY` is `//sftpSource`
* Fix `AbstractInboundFileSynchronizer` to not add `/` if one is already present in the beginning of the `remoteDirectoryPath`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9114
If path is not in normalized presentation, the SFTP operation might fail like:
```
Caused by: SFTP error (SSH_FX_NO_SUCH_PATH): The file path does not exist or is invalid.
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.throwStatusException(AbstractSftpClient.java:277)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributesResponse(AbstractSftpClient.java:333)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributes(AbstractSftpClient.java:325)
at org.apache.sshd.sftp.client.impl.AbstractSftpClient.lstat(AbstractSftpClient.java:1010)
at org.springframework.integration.sftp.session.SftpSession.exists(SftpSession.java:191)
```
* Use it now like this `this.sftpClient.lstat(normalizePath(path))`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes: #9123
The `/` at the beginning of the remote dir path is not necessary when listing files, although it is necessary to download them
The `SftpTemplate.get()` should work also with `remote-dir/MyFile.csv` as input.
* Fix `SftpSession.readRaw()` to call `sftpClient.canonicalPath(source)` if the path does not start with a `/`.
Something similar what is does
* Delegate to `SftpSession.readRaw()` from the `SftpSession.read()`
* Reuse `normalizePath()` for `doList()`
**Auto-cherry-pick to `6.2.x` & `6.1.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8797
After migration to Apache MINA we have missed to fix `DefaultSftpSessionFactory.timeout`
to be `0` by default as it states in its Javadocs and reference manual
It is `null` by default which really means an infinite wait.
* Fix `DefaultSftpSessionFactory.timeout` to be a reasonable 30 seconds by default
* Fix `setTimeout()` Javadocs and respective `session-factory.adoc`
* Propagate this `timeout` down to the `SftpClient` for its commands interactions
Fixes https://github.com/spring-projects/spring-integration/issues/8792
1. Use a `SftpStreamingMessageSource` with a `maxFetchSize = 5` and a `ChainFileListFilter` filter composed with
`SftpSystemMarkerFilePresentFileListFilter` which `supportsSingleFileFiltering == false`
2. Put 2 files in the folder and invoke `SftpStreamingMessageSource.receive()` method twice.
3. Put 5 files in the folder and invoke `SftpStreamingMessageSource.receive()`` method five times.
4. The last two files won't be received.
When you set max fetch size to a number bigger than one (for example 5) and at a certain point
it is necessary to `this.toBeReceived.clear()` inside `AbstractRemoteFileStreamingMessageSource.doReceive()`,
those removed elements from toBeReceived are not rolled back.
* Fix `AbstractRemoteFileStreamingMessageSource.listFiles()` to calculate `maxFetchSize` as `getMaxFetchSize() - this.fetched.get()`
**Cherry-pick to `6.1.x`, `6.0.x` & `5.5.x`**
* GH-8745: Add RFT.shouldMarkSessionAsDirty()
Fixes https://github.com/spring-projects/spring-integration/issues/8745
Not all errors caught in the `RemoteFileTemplate.execute()`
are fatal to mark session as dirty and physically close the target session
in the cache
* Introduce a `RemoteFileTemplate.shouldMarkSessionAsDirty()`
to consult with an exception if it is really a fatal error to close
the session in the end.
* Override `shouldMarkSessionAsDirty()` in the `RemoteFileTemplate`
implementations to check statuses of respective protocol errors
**Cherry-pick to `6.1.x` & `6.0.x`**
* * Fix tests for pool interaction
* * Fix language in Javadocs
* Add more `not dirty` statuses to `SftpRemoteFileTemplate` & `SmbRemoteFileTemplate`
Fixes https://github.com/spring-projects/spring-integration/issues/8691
* Remove setAge with TimeUnit
Turned out we don't need anymore since we're using Duration for age in FtpLastModifiedFileListFilter and SftpLastModifiedFileListFilter.
* Add changes to the docs
* Introduce AbstractLastModifiedFileListFilter
* Some code readability improvements
* Make language in the docs more official
Fixes https://github.com/spring-projects/spring-integration/issues/8708
According to the `org.apache.sshd.common.channel.ChannelAsyncOutputStream.writeBuffer()` JavaDocs cannot be used concurrently.
* Introduce internal `DefaultSftpSessionFactory.ConcurrentSftpClient` extension
of the `DefaultSftpClient` to set a `Lock` around `super.send(cmd, buffer);`
* Remove lock from the `SftpSession` since it now is managed by the mentioned `ConcurrentSftpClient`
**Cherry-pick to `6.1.x` & `6.0.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/8674
The `ResourceKnownHostsServerKeyVerifier` does not take into account that several different
keys can be present in the known hosts resource for the same host/port
* Fix `ResourceKnownHostsServerKeyVerifier` to find a list of knows host for the requested session.
Then iterate of this result to match the key type first and then compare keys and their `revoked` marker
**Cherry-pick to `6.1.x` & `6.0.x`**
Not clear why `SftpRemoteFileTemplateTests.testINT3412AppendStatRmdir()`
is failing on CI Linux, so add a tracing diagnostics to see how Apache SSHD behaves over there
Fixes https://github.com/spring-projects/spring-integration/issues/8581
Do not overwrite configuration of externally provided `SshClient` in the `DefaultSftpSessionFactory`
* Replace JUnit `assertDoesNotThrow` by AssertJ `assertThatNoException` in test
**Cherry-pick to `6.0.x`**
Fixes https://github.com/spring-projects/spring-integration/issues/3980
When not `isSharedSession`, the `initClient()` is called for every session
we request from the factory and in concurrent calls we end up with not initialized SSH client
in some threads.
* Add `synchronized` double check logic to the `initClient()` to block other threads while the first one
initialize the client
* Use `volatile boolean` instead of `AtomicBoolean` and change its state when `SshClient` is created and started
* GH-3962: Support SFTP < v5 for rename
Fixes https://github.com/spring-projects/spring-integration/issues/3962
Turns out the `SftpClient.CopyMode` options are supported only starting SFTP v5
* Check for the version from the client and fallback to the logic we had before with JSsch:
try to rename, delete existing file if such an exception status is thrown and retry to rename
* * Fix `SftpOutboundTests` with answering an explicit SFTP version for an `SftpClient` mock
Fixes https://github.com/spring-projects/spring-integration/issues/3974
* Treat a leading `/` as an indicator of absolute path request in the `SftpSession.doList()`
* Call `sftpClient.canonicalPath()` for path without a leading `/` to resolve it as relative path in the user home
* Add a note in docs for `LS` command
Fixes https://github.com/spring-projects/spring-integration/issues/3969
In previous version for SFTP client (Jsch), the empty path for `LS` command
has meant a `user home`.
Turns out the MINA `SftpClient` does not support automatic user home resolution
from the empty path.
* Fix `SftpSession` to resolve an empty path into a user home via `canicalPath()` operation
Fixes https://github.com/spring-projects/spring-integration/issues/3686
* Add `src/idea` with respective editor config for IntelliJ IDEA.
Must be imported into an IDE
* Add `src/eclipse` with respective editor config for Eclipse/STS
* Reformat imports in source code according a new editor config
The plain `SocketAddress.toString()` may resolve
to `hostName/IP` pair which is not parsed properly via `URI.getHost()`
downstream leaving `/IP` part for `uri.getPath()` request.
In the end this may lead to wrong file name to be used in the SFTP logic
* Fix `SftpSession.getHostPort()` to use `SshdSocketAddress` utility
to resolve host and port properly from the provided `SocketAddress`
The `192.168.1.61` is Private Use IP.
It is an intranet IP address, which is usually allocated to mobile phones, desktops, laptops, TVs, smart speakers and other devices.
The logic is taken from a similar test class in the Apache MINA project
* GH-3572: Migrate SFTP from `jsch` to `sshd-sftp`
Fixes https://github.com/spring-projects/spring-integration/issues/3572
* Rework SFTP module from the JSch API to more modern `sshd-sftp`
* Migrate generics of most API from `ChannelSftp.LsEntry` to the `SftpClient.DirEntry`
* Rework `DefaultSftpSessionFactory` to deal with an `SshClient` and create `SftpClient`
wrapped to the `SftpSession`
* Implement a `ResourceKnownHostsServerKeyVerifier` to load `known-hosts` from any possible resource
* Implement an expected `SftpSession.list()` with just file name to take or pattern matching
* Remove some unused tests and their config
* Remove tests for custom `UserInfo` since we don't provide any custom out-of-the-box
* Test a new `ResourceKnownHostsServerKeyVerifier` against default `known-hosts` file
* * Some tests improvements
* * Improve generics handling for `FileUtils`
Fixes https://github.com/spring-projects/spring-integration/issues/3827
The `AbstractRemoteFileOutboundGateway.doGet()` for a `STREAM` option
does not close the `session` in case of error.
This may lead to some leaks or exhausted caches
* Close `session` in the `catch()` of the `AbstractRemoteFileOutboundGateway.doGet()`
* Adjust the `SftpServerOutboundTests` to configure a `CachingSessionFactory`
for the `testStream()` to verify there is no leaks attempting to
`GET STREAM` non-existing remote file twice
**Cherry-pick to `5.5.x`**
* Mostly changes are related to the `TaskScheduler` and `Trigger` APIs
* Migrate to `micrometer-tracing` dependency
* Rework `SocketTestUtils` to use a `InetAddress.getLocalHost()`
for more stability and performance on Windows
* Fix docs for new `PeriodicTrigger` API
Fixes https://github.com/spring-projects/spring-integration/issues/3623
* `IntegrationFlow` refactoring
* Apply several code style improvements and good practices
* Code style: no empty lines for methods javadocs
* make deprecated implementation reuse actual one instead of the copy-paste approach
* add whats-new comments
* Fix whats-new page according to standards
* Upgrade Spring dependencies to the latest SNAPSHOTs
* Fix tests to verify against stack traces: the message
of the `NestedRuntimeException` does not include the nested exception information.
Related to https://github.com/spring-projects/spring-framework/issues/25162
* Fix `JdbcMessageStore` and `DefaultLockRepository` to rely on the `DataIntegrityViolationException`
instead of only its `DuplicateKeyException` extension.
This is the current behavior of the SQL errors translation
* Disable `WebFluxDslTests.testValidation()` - doesn't subscribe to the reply somehow...
* Refine `SimplePool.PoolSemaphore.reducePermits()`
* GH-3647: Use remoteDirExpression in MV command
Fixes https://github.com/spring-projects/spring-integration/issues/3647
To simplify a source and renameTo remote file expressions, the `remoteDirectoryExpression`
is consulted now, when they are not full paths.
This is useful when we want just to rename a remote file in some dir
* * Add JavaDoc for `getDirectoryExpressionProcessor()`
* Fix language in docs
Resolves https://github.com/spring-projects/spring-integration/issues/3488
Resolves two problems:
- When changes are made deep in the directory tree, they were not detected because
the directory is in the metadata store and only passes the filter if a file
immediately under it is changed, changing the directory's timestamp.
This is solved by subclassing `AbstractDirectoryAwareFileListFilter`, allowing its
`alwaysAcceptDirectories` property to be set.
- Only the filename was used as a metadata key; causing problems if a file with the
same name appears multiple times in the tree.
This is solved with a new property on `AbstractDirectoryAwareFileListFilter` used by
the gateways to determine whether to filter the raw file names returned by the session
(previous behavior) or the full path relative to the root directory.
**cherry-pick to 5.4.x, 5.3.x**
* Some code style clean up
* More GH Actions goodies
* Add RabbitMQ, MongoDB & Redis services into GH actions
* Remove Travis
* * Add `distributionSha256Sum` into `gradle-wrapper.properties`
* Try `TERM: dumb` to see difference for Gradle output in GH action
* Add `-S` to see the reason of test failure
* * Try MQTT plugin for RabbitMQ image
* * Try `-i` for test failure in Gradle on Actions
* * Use `eclipse-mosquitto` image for MQTT on actions
* Fix `FtpServerOutboundTests` for the proper FTP file list
* Output file list for the failed session in the `RotatingServersTests`
* * Adjust SOUT in the test for Checkstyle
* * Make some change to `RotatingServersTests`.
Looks like the order for `@BeforeAll` methods is not determined
and we may start to interaction with FTP server which is not
started yet
* Fix SSL Handshake Test
- we already made this change for the NIO test, but the NET test is failing the same
way - the cert failure switched to the server so the expected exception was not
thrown on the client side. Due to JVM changes.
* * Fix SFTP tests for wrong host and auto-startup state
* Remove `-i` for Gradle in the GH Actions
* * Try `--no-parallel` for Gradle in the GH Actions
* * Try GH actions without Mosquitto
* * Try `cyrilix/rabbitmq-mqtt` for GH Actions
Co-authored-by: Gary Russell <grussell@pivotal.io>
