GH-2777: Remote File Filter Improvements

Resolves https://github.com/spring-projects/spring-integration/issues/2777

If the filter supports it, defer filtering until the last possible moment.
Then, the worst case scenario after a catastrophic failure (e.g. power loss),
would be that at most one file will be incorrectly filtered on restart.

Polishing and add more tests.

Polishing Javadocs

More Polishing

Final polishing

More polishing.

Polishing and docPolishing and docs.

* Fix typos in Docs

src/reference/asciidoc/file.adoc

@@ -461,7 +461,7 @@ public class FileReadingJavaApplication {
                   .transform(Files.toStringTransformer())
                   .channel("processFileChannel")
                   .get();
-        }
+    }
 
 }
 ----
@@ -482,15 +482,15 @@ Examples of such events include the following:
 ====
 [source,bash]
 ----
-[message=tail: cannot open `/tmp/somefile' for reading:
+[message=tail: cannot open '/tmp/somefile' for reading:
                No such file or directory, file=/tmp/somefile]
 
-[message=tail: `/tmp/somefile' has become accessible, file=/tmp/somefile]
+[message=tail: '/tmp/somefile' has become accessible, file=/tmp/somefile]
 
-[message=tail: `/tmp/somefile' has become inaccessible:
+[message=tail: '/tmp/somefile' has become inaccessible:
                No such file or directory, file=/tmp/somefile]
 
-[message=tail: `/tmp/somefile' has appeared;
+[message=tail: '/tmp/somefile' has appeared;
                following end of new file, file=/tmp/somefile]
 ----
 ====
@@ -1114,3 +1114,33 @@ public class FileSplitterApplication {
 }
 ----
 ====
+
+[[remote-persistent-flf]]
+=== Remote Persistent File List Filters
+
+Inbound and streaming inbound remote file channel adapters (`FTP`, `SFTP`, and other technologies) are configured with corresponding implementations of `AbstractPersistentFileListFilter` by default, configured with an in-memory `MetadataStore`.
+To run in a cluster, these can be replaced with filters using a shared `MetadataStore` (see <<metadata-store>> for more information).
+These filters are used to prevent fetching the same file multiple times (unless it's modified time changes).
+Starting with version 5.2, a file is added to the filter immediately before the file is fetched (and reversed if the fetch fails).
+
+IMPORTANT: In the event of a catastrophic failure (such as power loss), it is possible that the file currently being fetched will remain in the filter and won't be re-fetched when restarting the application.
+In this case you would need to manually remove this file from the `MetadataStore`.
+
+In previous versions, the files were filtered before any were fetched, meaning that several files could be in this state after a catastrophic failure.
+
+In order to facilitate this new behavior, two new methods have been added to `FileListFilter`.
+
+====
+[source, java]
+----
+boolean accept(F file);
+
+boolean supportsSingleFileFiltering();
+----
+====
+
+If a filter returns `true` in `supportsSingleFileFiltering`, it **must** implement `accept()`.
+
+If a remote filter does not support single file filtering (such as the `AbstractMarkerFilePresentFileListFilter`), the adapters revert to the previous behavior.
+
+If multiple filters are in used (using a `CompositeFileListFilter` or `ChainFileListFilter`), then **all** of the delegate filters must support single file filtering in order for the composite filter to support it.

src/reference/asciidoc/ftp.adoc

@@ -344,6 +344,8 @@ Unless your application removes files after processing, the adapter will re-proc
 
 Also, if you configure the `filter` to use a `FtpPersistentAcceptOnceFileListFilter` and the remote file timestamp changes (causing it to be re-fetched), the default local filter does not let this new file be processed.
 
+For more information about this filter, and how it is used, see <<remote-persistent-flf>>.
+
 You can use the `local-filter` attribute to configure the behavior of the local file system filter.
 Starting with version 4.3.8, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
 This filter stores the accepted file names and modified timestamp in an instance of the `MetadataStore` strategy (see <<metadata-store>>) and detects changes to the local file modified time.
@@ -623,6 +625,8 @@ If you need to allow duplicates, you can use `AcceptAllFileListFilter`.
 Any other use cases can be handled by `CompositeFileListFilter` (or `ChainFileListFilter`).
 The Java configuration (<<ftp-streaming-java,later in the document>>) shows one technique to remove the remote file after processing to avoid duplicates.
 
+For more information about the `FtpPersistentAcceptOnceFileListFilter`, and how it is used, see <<remote-persistent-flf>>.
+
 Use the `max-fetch-size` attribute to limit the number of files fetched on each poll when a fetch is necessary.
 Set it to `1` and use a persistent filter when running in a clustered environment.
 See <<ftp-max-fetch>> for more information.

src/reference/asciidoc/sftp.adoc

@@ -371,7 +371,9 @@ Once the files have been retrieved, an additional filter is applied to the files
 By default, this is an`AcceptOnceFileListFilter`, which, as discussed in this section, retains state in memory and does not consider the file's modified time.
 Unless your application removes files after processing, the adapter re-processes the files on disk by default after an application restart.
 
-Also, if you configure the `filter` to use a `FtpPersistentAcceptOnceFileListFilter` and the remote file timestamp changes (causing it to be re-fetched), the default local filter does not allow this new file to be processed.
+Also, if you configure the `filter` to use a `SftpPersistentAcceptOnceFileListFilter` and the remote file timestamp changes (causing it to be re-fetched), the default local filter does not allow this new file to be processed.
+
+For more information about this filter, and how it is used, see <<remote-persistent-flf>>.
 
 You can use the `local-filter` attribute to configure the behavior of the local file system filter.
 Starting with version 4.3.8, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
@@ -622,6 +624,8 @@ If you need to allow duplicates, you can use the `AcceptAllFileListFilter`.
 You can handle any other use cases by using `CompositeFileListFilter` (or `ChainFileListFilter`).
 The Java configuration <<sftp-streaming-java-config,shown later>> shows one technique to remove the remote file after processing, avoiding duplicates.
 
+For more information about the `SftpPersistentAcceptOnceFileListFilter`, and how it is used, see <<remote-persistent-flf>>.
+
 You can use the `max-fetch-size` attribute to limit the number of files fetched on each poll when a fetch is necessary.
 Set it to `1` and use a persistent filter when running in a clustered environment.
 See <<sftp-max-fetch>> for more information.

src/reference/asciidoc/whats-new.adoc

@@ -10,7 +10,13 @@ If you are interested in more details, see the Issue Tracker tickets that were r
 [[x5.2-general]]
 === General Changes
 
-[[5.2-tcp]]
+[[x5.2-file]]
+==== File Changes
+
+Some improvements to filtering remote files have been made.
+See <<remote-persistent-flf>> for more information.
+
+[[x5.2-tcp]]
 ==== TCP Changes
 
 The length header used by the `ByteArrayLengthHeaderSerializer` can now include the length of the header in addition to the payload.