diff --git a/src/reference/docbook/ftp.xml b/src/reference/docbook/ftp.xml index 83fec62b0a..5279b3bc09 100644 --- a/src/reference/docbook/ftp.xml +++ b/src/reference/docbook/ftp.xml @@ -157,6 +157,7 @@ protected void postProcessClientBeforeConnect(T client) throws IOException { preserve-timestamp="true" local-filename-generator-expression="#this.toUpperCase() + '.a'" local-filter="myFilter" + temporary-file-suffix=".writing" local-directory="."> ]]> @@ -187,7 +188,7 @@ protected void postProcessClientBeforeConnect(T client) throws IOException { list of files. This filter determines which remote files are retrieved. You can also combine a pattern based filter with other filters, such as an AcceptOnceFileListFilter to avoid synchronizing files that have previously been fetched, by using a CompositeFileListFilter. - + The AcceptOnceFileListFilter stores its state in memory. If you wish the state to survive a system restart, consider using the @@ -202,28 +203,42 @@ protected void postProcessClientBeforeConnect(T client) throws IOException { (such as Redis with the RedisMetadataStore) this allows filter keys to be shared across multiple application or server instances. - - - Beginning with version 3.0, you can also specify a filter used to filter the files locally, once they have - been retrieved. The default filter is an AcceptOnceFileListFilter which prevents processing - files with the same name multiple times in the same JVM execution; this can now be overridden - (for example with an AcceptAllFileListFilter), using the local-filter attribute. - Previously, the default AcceptOnceFileListFilter could not be overridden. - - - The AcceptOnceFileListFilter stores its state in memory. If you wish the - state to survive a system restart, consider using the - FileSystemPersistentAcceptOnceFileListFilter as a local filter instead. This filter stores - the accepted file names in an instance of the - MetadataStore strategy (). - - This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a - re-synchronized file from being processed, you should use the preserve-timestamp attribute discussed above. - - - - The 'remote-file-separator' attribute allows you to configure a + The above discussion refers to filtering the files before retrieving them. Once the files have been + retrieved, an additional filter is applied to the files on the file system. By default, this is an + AcceptOnceFileListFilter which, as discussed, retains state in memory and does + not consider the file's modified time. Unless your application removes files after processing, the + adapter will re-process the files on disk by default after an application restart. + + + Also, if you configure the filter to use a + FtpPersistentAcceptOnceFileListFilter, and the remote file + timestamp changes (causing it to be re-fetched), the default local filter will not allow this + new file to be processed. + + + Use the local-filter attribute to configure the behavior of the local file system + filter. To solve these particular use cases, you can use a + FileSystemPersistentAcceptOnceFileListFilter as a local filter instead. + This filter also stores the accepted file names and modified timestamp in an instance of the + MetadataStore strategy (), and + will detect the change in the local file modified time. + + + Further, if you use a distributed MetadataStore (such as + or ) you can + have multiple instances of the same adapter/application and be sure that one and only one will + process a file. + + + The actual local filter is a CompositeFileListFilter containing the supplied + filter and a pattern filter that prevents processing files that are in the process of + being downloaded (based on the temporary-file-suffix); files are downloaded with + this suffix (default: .writing) and the file is renamed to its final name when the + transfer is complete, making it 'visible' to the filter. + + + The remote-file-separator attribute allows you to configure a file separator character to use if the default '/' is not applicable for your particular environment. diff --git a/src/reference/docbook/sftp.xml b/src/reference/docbook/sftp.xml index 4630533847..1ae435469d 100644 --- a/src/reference/docbook/sftp.xml +++ b/src/reference/docbook/sftp.xml @@ -280,6 +280,7 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/sftp auto-create-local-directory="true" local-filename-generator-expression="#this.toUpperCase() + '.a'" local-filter="myFilter" + temporary-file-suffix=".writing" delete-remote-files="false"> ]]> @@ -327,26 +328,40 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/sftp (such as Redis with the RedisMetadataStore) this allows filter keys to be shared across multiple application or server instances. - - - Beginning with version 3.0, you can also specify a filter used to filter the files locally, once they have - been retrieved. The default filter is an AcceptOnceFileListFilter which prevents processing - files with the same name multiple times in the same JVM execution; this can now be overridden - (for example with an AcceptAllFileListFilter), using the local-filter attribute. - Previously, the default AcceptOnceFileListFilter could not be overridden. - - - The AcceptOnceFileListFilter stores its state in memory. If you wish the - state to survive a system restart, consider using the - FileSystemPersistentAcceptOnceFileListFilter as a local filter instead. This filter stores - the accepted file names in an instance of the - MetadataStore strategy (). - - This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a - re-synchronized file from being processed, you should use the preserve-timestamp attribute discussed above. - - - + + The above discussion refers to filtering the files before retrieving them. Once the files have been + retrieved, an additional filter is applied to the files on the file system. By default, this is an + AcceptOnceFileListFilter which, as discussed, retains state in memory and does + not consider the file's modified time. Unless your application removes files after processing, the + adapter will re-process the files on disk by default after an application restart. + + + Also, if you configure the filter to use a + FtpPersistentAcceptOnceFileListFilter, and the remote file + timestamp changes (causing it to be re-fetched), the default local filter will not allow this + new file to be processed. + + + Use the local-filter attribute to configure the behavior of the local file system + filter. To solve these particular use cases, you can use a + FileSystemPersistentAcceptOnceFileListFilter as a local filter instead. + This filter also stores the accepted file names and modified timestamp in an instance of the + MetadataStore strategy (), and + will detect the change in the local file modified time. + + + Further, if you use a distributed MetadataStore (such as + or ) you can + have multiple instances of the same adapter/application and be sure that one and only one will + process a file. + + + The actual local filter is a CompositeFileListFilter containing the supplied + filter and a pattern filter that prevents processing files that are in the process of + being downloaded (based on the temporary-file-suffix); files are downloaded with + this suffix (default: .writing) and the file is renamed to its final name when the + transfer is complete, making it 'visible' to the filter. + Please refer to the schema for more detail on these attributes.