Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

GH-3043: Add FileHeaders.REMOTE_HOST header (#3044)

Browse Source 
* GH-3043: Add FileHeaders.REMOTE_HOST header

Fixes https://github.com/spring-projects/spring-integration/issues/3043

* Populate a `FileHeaders.REMOTE_HOST` from the
`AbstractRemoteFileStreamingMessageSource` and "get"-based commands
in the `AbstractRemoteFileOutboundGateway`
* Extract the value from the a `Session.getHost()` contract
* The `AbstractInboundFileSynchronizingMessageSource` cannot be
addressed with this because the real message is already based on the
locally stored file
* Adjust some affected tests according our code style requirements

* * Add remote file info support into `AbstractInboundFileSynchronizingMessageSource`
* Introduce a `MetadataStore` functionality into the `AbstractInboundFileSynchronizer`
to gather a remote file info an save it in the URI style against local file
* Retrieve such an info in the `AbstractInboundFileSynchronizingMessageSource`
during local file polling
* Introduce `protocol()` contract for the `AbstractInboundFileSynchronizer`
to build a proper URI in the metadata for external readers to distinguish
remote files properly
* Document the feature

* * Fix some typos in Docs

* * Rename property and header constant to the `HOST_PORT` pair
* Fix typos in Docs
* Add  `remote-file-metadata-store` and `metadata-store-prefix` into XSD
of (S)FTP Inbound Channel Adapters
* Add `remoteFileMetadataStore` and `metadataStorePrefix` options
into `RemoteFileInboundChannelAdapterSpec` for Java DSL
This commit is contained in:
Artem Bilan
2019-08-28 08:58:14 -04:00
committed by Gary Russell
parent ff15d5265d
commit a756e6334d
33 changed files with 605 additions and 297 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
src/reference/asciidoc/ftp.adoc
View File

@@ -444,7 +444,7 @@ the removal of the failed file from the filter:
<int-ftp:inbound-channel-adapter id="ftpAdapter"
session-factory="ftpSessionFactory"
channel="requestChannel"
remote-directory-expression="'/sftpSource'"
remote-directory-expression="'/ftpSource'"
local-directory="file:myLocalDir"
auto-create-local-directory="true"
filename-pattern="*.txt">
@@ -636,6 +636,7 @@ Starting with version 5.0, the `FileHeaders.REMOTE_FILE_INFO` header provides ad
If you set the `fileInfoJson` property on the `FtpStreamingMessageSource` to `false`, the header contains an `FtpFileInfo` object.
The `FTPFile` object provided by the underlying Apache Net library can be accessed by using the `FtpFileInfo.getFileInfo()` method.
The `fileInfoJson` property is not available when you use XML configuration, but you can set it by injecting the `FtpStreamingMessageSource` into one of your configuration classes.
See also <<ftp-remote-file-info>>.
Starting with version 5.1, the generic type of the `comparator` is `FTPFile`.
Previously, it was `AbstractFileInfo<FTPFile>`.
@@ -1575,3 +1576,19 @@ public ApplicationEventListeningMessageProducer eventsAdapter() {
}
----
====
[[ftp-remote-file-info]]
=== Remote File Information
Starting with version 5.2, the `FtpStreamingMessageSource` (<<ftp-streaming>>), `FtpInboundFileSynchronizingMessageSource` (<<ftp-inbound>>) and "read"-commands of the `FtpOutboundGateway` (<<ftp-outbound-gateway>>) provide additional headers in the message to produce with an information about the remote file:
* `FileHeaders.REMOTE_HOST_PORT` - the host:port pair the remote session has been connected to during file transfer operation;
* `FileHeaders.REMOTE_DIRECTORY` - the remote directory the operation has been performed;
* `FileHeaders.REMOTE_FILE` - the remote file name; applicable only for single file operations.
Since the `FtpInboundFileSynchronizingMessageSource` doesn't produce messages against remote files, but using a local copy, the `AbstractInboundFileSynchronizer` stores an information about remote file in the `MetadataStore` (which can be configured externally) in the URI style (`protocol://host:port/remoteDirectory#remoteFileName`) during synchronization operation.
This metadata is retrieved by the `FtpInboundFileSynchronizingMessageSource` when local file is polled.
When local file is deleted, it is recommended to remove its metadata entry.
The `AbstractInboundFileSynchronizer` provides a `removeRemoteFileMetadata()` callback for this purpose.
In addition there is a `setMetadataStorePrefix()` to be used in the metadata keys.
It is recommended to have this prefix be different from the one used in the `MetadataStore`-based `FileListFilter` implementations, when the same `MetadataStore` instance is shared between these components, to avoid entry overriding because both filter and `AbstractInboundFileSynchronizer` use the same local file name for the metadata entry key.

34
src/reference/asciidoc/sftp.adoc
View File

@@ -200,7 +200,7 @@ injection, as the following example shows:
Version 4.2 introduced the `DelegatingSessionFactory`, which allows the selection of the actual session factory at
runtime.
Prior to invoking the FTP endpoint, you can call `setThreadKey()` on the factory to associate a key with the current thread.
Prior to invoking the SFTP endpoint, you can call `setThreadKey()` on the factory to associate a key with the current thread.
That key is then used to look up the actual session factory to be used.
You can clear the key by calling `clearThreadKey()` after use.
@@ -334,7 +334,7 @@ Unlike outbound gateways and adapters, where the root object of the SpEL evaluat
Consequently, the root object of the SpEL evaluation context is the original name of the remote file (a `String`).
The inbound channel adapter first retrieves the file to a local directory and then emits each file according to the poller configuration.
Starting with version 5.0, you can limit the number of files fetched from the FTP server when new file retrievals are needed.
Starting with version 5.0, you can limit the number of files fetched from the SFTP server when new file retrievals are needed.
This can be beneficial when the target files are large or when running in a clustered system with a persistent file list filter, discussed later in this section.
Use `max-fetch-size` for this purpose.
A negative value (the default) means no limit and all matching files are retrieved.
@@ -363,7 +363,7 @@ Since version 4.0, this filter requires a `ConcurrentMetadataStore`.
When used with a shared data store (such as `Redis` with the `RedisMetadataStore`), this lets filter keys be shared across multiple application or server instances.
Starting with version 5.0, the `SftpPersistentAcceptOnceFileListFilter` with an in-memory `SimpleMetadataStore` is applied by default for the `SftpInboundFileSynchronizer`.
This filter is also applied, together with the `regex` or `pattern` option in the XML configuration, as well as through `FtpInboundChannelAdapterSpec` in Java DSL.
This filter is also applied, together with the `regex` or `pattern` option in the XML configuration, as well as through `SftpInboundChannelAdapterSpec` in Java DSL.
You can handle any other use-cases by using `CompositeFileListFilter` (or `ChainFileListFilter`).
The above discussion refers to filtering the files before retrieving them.
@@ -635,6 +635,7 @@ Starting with version 5.0, the `FileHeaders.REMOTE_FILE_INFO` header provides ad
If you set the `fileInfoJson` property on the `SftpStreamingMessageSource` to `false`, the header contains an `SftpFileInfo` object.
You can access the `LsEntry` object provided by the underlying Jsch library by using the `SftpFileInfo.getFileInfo()` method.
The `fileInfoJson` property is not available when you use XML configuration, but you can set it by injecting the `SftpStreamingMessageSource` into one of your configuration classes.
See also <<sftp-remote-file-info>>.
Starting with version 5.1, the generic type of the `comparator` is `LsEntry`.
Previously, it was `AbstractFileInfo<LsEntry>`.
@@ -771,8 +772,8 @@ This allows files retrieved from different directories to be downloaded to simil
----
@Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(Ftp.inboundAdapter(sf())
.filter(new FtpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "rotate"))
return IntegrationFlows.from(Sftp.inboundAdapter(sf())
.filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "rotate"))
.localDirectory(new File(tmpDir))
.localFilenameExpression("#remoteDirectory + T(java.io.File).separator + #root")
.remoteDirectory("."),
@@ -1283,7 +1284,7 @@ public class SftpJavaApplication {
@Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost("localhost");
sf.setPort(port);
sf.setUsername("foo");
@@ -1319,8 +1320,7 @@ public class SftpJavaApplication {
When performing operations on multiple files (by using `mget` and `mput`) an exception can occur some time after one or more files have been transferred.
In this case (starting with version 4.2), a `PartialSuccessException` is thrown.
As well as the usual `MessagingException` properties (`failedMessage` and `cause`), this exception has two additional
properties:
As well as the usual `MessagingException` properties (`failedMessage` and `cause`), this exception has two additional properties:
* `partialResults`: The successful transfer results.
* `derivedInput`: The list of files generated from the request message (such as local files to transfer for an `mput`).
@@ -1366,7 +1366,7 @@ log4j.category.com.jcraft.jsch=DEBUG
=== MessageSessionCallback
Starting with Spring Integration version 4.2, you can use a `MessageSessionCallback<F, T>` implementation with the `<int-sftp:outbound-gateway/>` (`SftpOutboundGateway`) to perform any operation on the `Session<LsEntry>` with the `requestMessage` context.
You can use it for any non-standard or low-level FTP operation (or several), such as allowing access from an integration flow definition, or functional interface (lambda) implementation injection.
You can use it for any non-standard or low-level SFTP operation (or several), such as allowing access from an integration flow definition, or functional interface (lambda) implementation injection.
The following example uses a lambda:
====
@@ -1433,3 +1433,19 @@ public ApplicationEventListeningMessageProducer eventsAdapter() {
}
----
====
[[sftp-remote-file-info]]
=== Remote File Information
Starting with version 5.2, the `SftpStreamingMessageSource` (<<sftp-streaming>>), `SftpInboundFileSynchronizingMessageSource` (<<sftp-inbound>>) and "read"-commands of the `SftpOutboundGateway` (<<sftp-outbound-gateway>>) provide additional headers in the message to produce with an information about the remote file:
* `FileHeaders.REMOTE_HOST_PORT` - the host:port pair the remote session has been connected to during file transfer operation;
* `FileHeaders.REMOTE_DIRECTORY` - the remote directory the operation has been performed;
* `FileHeaders.REMOTE_FILE` - the remote file name; applicable only for single file operations.
Since the `SftpInboundFileSynchronizingMessageSource` doesn't produce messages against remote files, but using a local copy, the `AbstractInboundFileSynchronizer` stores an information about remote file in the `MetadataStore` (which can be configured externally) in the URI style (`protocol://host:port/remoteDirectory#remoteFileName`) during synchronization operation.
This metadata is retrieved by the `SftpInboundFileSynchronizingMessageSource` when local file is polled.
When local file is deleted, it is recommended to remove its metadata entry.
The `AbstractInboundFileSynchronizer` provides a `removeRemoteFileMetadata()` callback for this purpose.
In addition there is a `setMetadataStorePrefix()` to be used in the metadata keys.
It is recommended to have this prefix be different from the one used in the `MetadataStore`-based `FileListFilter` implementations, when the same `MetadataStore` instance is shared between these components, to avoid entry overriding because both filter and `AbstractInboundFileSynchronizer` use the same local file name for the metadata entry key.

6
src/reference/asciidoc/whats-new.adoc
View File

@@ -157,4 +157,8 @@ See <<./router.adoc#dynamic-routers, Dynamic Routers>> for more information.
==== FTP/SFTP Changes
The `RotatingServerAdvice` is decoupled now from the `RotationPolicy` and its `StandardRotationPolicy`.
See <<./ftp.adoc#ftp-rotating-server-advice, Polling Multiple Servers and Directories>> for more information.
The remote file information, including host/port and directory are included now into message headers in the `AbstractInboundFileSynchronizingMessageSource` and `AbstractRemoteFileStreamingMessageSource` implementations.
Also this information is included into headers in the read operations results of the `AbstractRemoteFileOutboundGateway` implementations.
See <<./ftp.adoc#ftp, FTP(S) Support>> and <<./sftp.adoc#sftp, SFTP Support>> for more information.