Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-4115: (S)FtpPersistentFileFilter by Default

Browse Source 
JIRA: https://jira.spring.io/browse/INT-4115

Apply `(S)FtpPersistentAcceptOnceFileListFilter` for the `(S)FtpInboundFileSynchronizer` by default to avoid cases to sync the same remote files to the local directory again.
Especially when `localFileName` strategy is applied and we end up with new local files, but with the same remote content

Accept `(S)FtpPersistentAcceptOnceFileListFilter` for streaming adapters

Make `doSetFilter()` as `protected final`

Fix tests after rebase

Fix "What's New" after rebase

Compose `PersistentAcceptOnceFileListFilter` together with the regex or pattern filters
Document such a behavior

Address PR comments for formatting and typo
This commit is contained in:
Artem Bilan
2017-02-06 16:04:47 -05:00
committed by Gary Russell
parent d3fb8b8f9e
commit 10ce68d3e3
29 changed files with 328 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
src/reference/asciidoc/ftp.adoc
View File

@@ -256,7 +256,7 @@ Unlike outbound gateways and adapters where the root object of the SpEL Evaluati
So, the root object of the SpEL Evaluation Context is the original name of the remote file (String).
The inbound channel adapter first retrieves the file to a local directory and then emits each file according to the poller configuration.
Starting with _version 5.0_ you can now limit the number of files fetched from the FTP server when new file retrievals are needed.
Starting with _version 5.0_, you can now limit the number of files fetched from the FTP server when new file retrievals are needed.
This can be beneficial when the target files are very large and/or when running in a clustered system with a persistent file list filter discussed below.
Use `max-fetch-size` for this purpose; a negative value (default) means no limit and all matching files will be retrieved.
@@ -283,6 +283,10 @@ This filter matches on the filename and the remote modified time.
Since _version 4.0_, this filter requires a `ConcurrentMetadataStore`.
When used with a shared data store (such as `Redis` with the `RedisMetadataStore`) this allows filter keys to be shared across multiple application or server instances.
Starting with _version 5.0_, the `FtpPersistentAcceptOnceFileListFilter` with in-memory `SimpleMetadataStore` is applied by default for the `FtpInboundFileSynchronizer`.
This filter is also applied together with the `regex` or `pattern` option in the XML configuration as well as via `FtpInboundChannelAdapterSpec` in Java DSL.
Any other use-cases can be reached via `CompositeFileListFilter` (or `ChainFileListFilter`).
The above discussion refers to filtering the files before retrieving them.
Once the files have been retrieved, an additional filter is applied to the files on the file system.
By default, this is an `AcceptOnceFileListFilter` which, as discussed, retains state in memory and does not consider the file's modified time.
@@ -291,7 +295,7 @@ Unless your application removes files after processing, the adapter will re-proc
Also, if you configure the `filter` to use a `FtpPersistentAcceptOnceFileListFilter`, and the remote file timestamp changes (causing it to be re-fetched), the default local filter will not allow this new file to be processed.
Use the `local-filter` attribute to configure the behavior of the local file system filter.
Starting with _verion 4.3.8_, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
Starting with _version 4.3.8_, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
This filter stores the accepted file names and modified timestamp in an instance of the `MetadataStore` strategy (<<metadata-store>>), and will detect changes to the local file modified time.
The default `MetadataStore` is a `SimpleMetadataStore` which stores state in memory.
@@ -528,12 +532,11 @@ See <<file-splitter>> and <<stream-transformer>> for more information about thes
Only one of `filename-pattern`, `filename-regex`, `filter` or `filter-expression` is allowed.
IMPORTANT: Unlike the non-streaming inbound channel adapter, this adapter does not prevent duplicates by default.
If you do not delete the remote file (e.g. using an outbound gateway with an rm command) and you wish to prevent the
file being processed again, you can configure an `FtpPersistentFileListFilter` in the `filter` attribute.
If you don't actually want to persist the state, an in-memory `SimpleMetadataStore` can be used with the filter.
If you wish to use a filename pattern (or regex) as well, use a `CompositeFileListFilter`.
The java configuration below shows one technique to remove the remote file after processing.
IMPORTANT: Starting with _version 5.0_, by default, the `FtpStreamingMessageSource` adapter prevents duplicates for remote files via `FtpPersistentAcceptOnceFileListFilter` based on the in-memory `SimpleMetadataStore`.
This filter is also applied by default together with the filename pattern (or regex) as well.
If there is a requirement to allow duplicates, the `AcceptAllFileListFilter` can be used.
Any other use-cases can be reached via `CompositeFileListFilter` (or `ChainFileListFilter`).
The java configuration below shows one technique to remove the remote file after processing, avoiding duplicates.
Use the `max-fetch-size` attribute to limit the number of files fetched on each poll when a fetch is necessary; set to 1 and use a persistent filter when running in a clustered environment.
@@ -560,10 +563,9 @@ public class FtpJavaApplication {
@Bean
@InboundChannelAdapter(channel = "stream")
public MessageSource<InputStream> ftpMessageSource() {
FtpStreamingMessageSource messageSource = new FtpStreamingMessageSource(template(), null);
FtpStreamingMessageSource messageSource = new FtpStreamingMessageSource(template());
messageSource.setRemoteDirectory("ftpSource/");
messageSource.setFilter(new FtpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(),
"streaming"));
messageSource.setFilter(new AcceptAllFileListFilter<>());
messageSource.setMaxFetchSize(1);
return messageSource;
}

22
src/reference/asciidoc/sftp.adoc
View File

@@ -345,6 +345,10 @@ This filter matches on the filename and the remote modified time.
Since _version 4.0_, this filter requires a `ConcurrentMetadataStore`.
When used with a shared data store (such as `Redis` with the `RedisMetadataStore`) this allows filter keys to be shared across multiple application or server instances.
Starting with _version 5.0_, the `SftpPersistentAcceptOnceFileListFilter` with in-memory `SimpleMetadataStore` is applied by default for the `SftpInboundFileSynchronizer`.
This filter is also applied together with the `regex` or `pattern` option in the XML configuration as well as via `FtpInboundChannelAdapterSpec` in Java DSL.
Any other use-cases can be reached via `CompositeFileListFilter` (or `ChainFileListFilter`).
The above discussion refers to filtering the files before retrieving them.
Once the files have been retrieved, an additional filter is applied to the files on the file system.
By default, this is an`AcceptOnceFileListFilter` which, as discussed, retains state in memory and does not consider the file's modified time.
@@ -353,7 +357,7 @@ Unless your application removes files after processing, the adapter will re-proc
Also, if you configure the `filter` to use a `FtpPersistentAcceptOnceFileListFilter`, and the remote file timestamp changes (causing it to be re-fetched), the default local filter will not allow this new file to be processed.
Use the `local-filter` attribute to configure the behavior of the local file system filter.
Starting with _verion 4.3.8_, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
Starting with _version 4.3.8_, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
This filter stores the accepted file names and modified timestamp in an instance of the `MetadataStore` strategy (<<metadata-store>>), and will detect changes to the local file modified time.
The default `MetadataStore` is a `SimpleMetadataStore` which stores state in memory.
@@ -567,12 +571,11 @@ See <<file-splitter>> and <<stream-transformer>> for more information about thes
Only one of `filename-pattern`, `filename-regex`, `filter` or `filter-expression` is allowed.
IMPORTANT: Unlike the non-streaming inbound channel adapter, this adapter does not prevent duplicates by default.
If you do not delete the remote file (e.g. using an outbound gateway with an rm command) and you wish to prevent the
file being processed again, you can configure an `SftpPersistentFileListFilter` in the `filter` attribute.
If you don't actually want to persist the state, an in-memory `SimpleMetadataStore` can be used with the filter.
If you wish to use a filename pattern (or regex) as well, use a `CompositeFileListFilter`.
The java configuration below shows one technique to remove the remote file after processing.
IMPORTANT: Starting with _version 5.0_, by default, the `SftpStreamingMessageSource` adapter prevents duplicates for remote files via `SftpPersistentAcceptOnceFileListFilter` based on the in-memory `SimpleMetadataStore`.
This filter is also applied by default together with the filename pattern (or regex) as well.
If there is a requirement to allow duplicates, the `AcceptAllFileListFilter` can be used.
Any other use-cases can be reached via `CompositeFileListFilter` (or `ChainFileListFilter`).
The java configuration below shows one technique to remove the remote file after processing, avoiding duplicates.
Use the `max-fetch-size` attribute to limit the number of files fetched on each poll when a fetch is necessary; set to 1 and use a persistent filter when running in a clustered environment.
@@ -599,10 +602,9 @@ public class SftpJavaApplication {
@Bean
@InboundChannelAdapter(channel = "stream")
public MessageSource<InputStream> ftpMessageSource() {
SftpStreamingMessageSource messageSource = new SftpStreamingMessageSource(template(), null);
SftpStreamingMessageSource messageSource = new SftpStreamingMessageSource(template());
messageSource.setRemoteDirectory("sftpSource/");
messageSource.setFilter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(),
"streaming"));
messageSource.setFilter(new AcceptAllFileListFilter<>());
messageSource.setMaxFetchSize(1);
return messageSource;
}

8
src/reference/asciidoc/whats-new.adoc
View File

@@ -97,17 +97,17 @@ The file outbound channel adapter (`FileWritingMessageHandler`) now supports the
The inbound channel adapters now have a property `max-fetch-size` which is used to limit the number of files fetched during a poll when there are no files currently in the local directory.
They also are configured with a `FileSystemPersistentAcceptOnceFileListFilter` in the `local-filter` by default.
See <<ftp-inbound>> and <<sftp-inbound>> for more information.
The regex and pattern filters can now be configured to always pass directories.
This can be useful when using recursion in the outbound gateways.
See <<ftp-outbound-gateway>> and <<sftp-outbound-gateway>> for more information.
All the Inbound Channel Adapters (streaming and synchronization-based) now use an appropriate `AbstractPersistentAcceptOnceFileListFilter` implementation by default to prevent remote files duplicate downloads.
The FTP and SFTP outbound gateways now support the `REPLACE_IF_MODIFIED` `FileExistsMode` when fetching remote files.
See <<ftp-outbound-gateway>> and <<sftp-outbound-gateway>> for more information.
The (S)FTP streaming inbound channel adapters now add remote file information in a message header.
See <<ftp-streaming>> and <<sftp-streaming>> for more information.
See <<ftp>> and <<sftp>> for more information.
The FTP and SFTP outbound channel adapters, as well as `PUT` command of the outbound gateways, now support `InputStream` as `payload`, too.