Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-4095: Support Limiting (S)FTP Files Fetched

Browse Source 
JIRA: https://jira.spring.io/browse/INT-4095

Limit the number of remote files fetched on each poll (when it is necessary to fetch files).

Polishing - PR Comments

Polishing - Decouple MaxFetchSize from Poller

Polishing - PR Comments

Schemas and Docs

More Polishing

* Polishing according PR comments
This commit is contained in:
Gary Russell
2016-08-18 18:15:04 -04:00
committed by Artem Bilan
parent 53237aa833
commit ea4763faa9
25 changed files with 534 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
src/reference/asciidoc/ftp.adoc
View File

@@ -172,6 +172,7 @@ The _FTP Inbound Channel Adapter_ is a special listener that will connect to the
local-filename-generator-expression="#this.toUpperCase() + '.a'"
local-filter="myFilter"
temporary-file-suffix=".writing"
max-fetch-size="-1"
local-directory=".">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
@@ -184,6 +185,11 @@ If you want to override this behavior you can set the `local-filename-generator-
Unlike outbound gateways and adapters where the root object of the SpEL Evaluation Context is a `Message`, this inbound adapter does not yet have the Message at the time of evaluation since that's what it ultimately generates with the transferred file as its payload.
So, the root object of the SpEL Evaluation Context is the original name of the remote file (String).
The inbound channel adapter first retrieves the file to a local directory and then emits each file according to the poller configuration.
Starting with _version 5.0_ you can now limit the number of files fetched from the FTP server when new file retrievals are needed.
This can be beneficial when the target files are very large and/or when running in a clustered system with a persistent file list filter discussed below.
Use `max-fetch-size` for this purpose; a negative value (default) means no limit and all matching files will be retrieved.
Starting with _Spring Integration 3.0_, you can specify the `preserve-timestamp` attribute (default `false`); when `true`, the local file's modified timestamp will be set to the value retrieved from the server; otherwise it will be set to the current time.
Starting with _version 4.2_, you can specify `remote-directory-expression` instead of `remote-directory`, allowing
@@ -209,7 +215,7 @@ When used with a shared data store (such as `Redis` with the `RedisMetadataStore
The above discussion refers to filtering the files before retrieving them.
Once the files have been retrieved, an additional filter is applied to the files on the file system.
By default, this is an`AcceptOnceFileListFilter` which, as discussed, retains state in memory and does not consider the file's modified time.
By default, this is an `AcceptOnceFileListFilter` which, as discussed, retains state in memory and does not consider the file's modified time.
Unless your application removes files after processing, the adapter will re-process the files on disk by default after an application restart.
Also, if you configure the `filter` to use a `FtpPersistentAcceptOnceFileListFilter`, and the remote file timestamp changes (causing it to be re-fetched), the default local filter will not allow this new file to be processed.
@@ -367,6 +373,7 @@ public class FtpJavaApplication {
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
source.setMaxFetchSize(1);
return source;
}

7
src/reference/asciidoc/sftp.adoc
View File

@@ -306,6 +306,7 @@ The _SFTP Inbound Channel Adapter_ is a special listener that will connect to th
local-filename-generator-expression="#this.toUpperCase() + '.a'"
local-filter="myFilter"
temporary-file-suffix=".writing"
max-fetch-size="-1"
delete-remote-files="false">
<int:poller fixed-rate="1000"/>
</int-sftp:inbound-channel-adapter>
@@ -318,6 +319,11 @@ If you want to override this behavior you can set the `local-filename-generator-
Unlike outbound gateways and adapters where the root object of the SpEL Evaluation Context is a `Message`, this inbound adapter does not yet have the Message at the time of evaluation since that's what it ultimately generates with the transferred file as its payload.
So, the root object of the SpEL Evaluation Context is the original name of the remote file (String).
The inbound channel adapter first retrieves the file to a local directory and then emits each file according to the poller configuration.
Starting with _version 5.0_ you can now limit the number of files fetched from the FTP server when new file retrievals are needed.
This can be beneficial when the target files are very large and/or when running in a clustered system with a persistent file list filter discussed below.
Use `max-fetch-size` for this purpose; a negative value (default) means no limit and all matching files will be retrieved.
Starting with _Spring Integration 3.0_, you can specify the `preserve-timestamp` attribute (default `false`); when `true`, the local file's modified timestamp will be set to the value retrieved from the server; otherwise it will be set to the current time.
Starting with _version 4.2_, you can specify `remote-directory-expression` instead of `remote-directory`, allowing
@@ -476,6 +482,7 @@ public class SftpJavaApplication {
source.setLocalDirectory(new File("ftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
source.setMaxFetchSize(1);
return source;
}

4
src/reference/asciidoc/whats-new.adoc
View File

@@ -33,3 +33,7 @@ See <<gateway-error-handling>> for more information.
Some inconsistencies with rendering IMAP mail content have been resolved.
See <<imap-format-important, the note in the Mail-Receiving Channel Adapter Section>> for more information.
==== (S)FTP Changes
The inbound channel adapters now have a property `max-fetch-size` which is used to limit the number of files fetched during a poll when there are no files currently in the local directory.