INT-4283: MarkerFilePresentFileListFilters

JIRA: https://jira.spring.io/browse/INT-4283

Filter that only passes files where a corresponding "marker" file is also present.

Implementations for file and(S)FTP.

* Polishing - PR Comments

* Missed one

src/reference/asciidoc/file.adoc

@@ -523,6 +523,21 @@ The file will be reopened for each chunk (the default is to keep the file open).
 
 IMPORTANT: Specifying the `delay`, `end` or `reopen` attributes, forces the use of the Apache commons-io adapter and the `native-options` attribute is not allowed.
 
+[[file-incomplete]]
+==== Dealing With Incomplete Data
+
+A common problem in file transfer scenarios is how to determine that the transfer is complete, so you don't start reading an incomplete file.
+A common technique to solve this problem is to write the file with a temporary name and then atomically rename it to the final name.
+This, together with a filter that masks the temporary file from being picked up by the consumer provides a robust solution.
+This technique is used by Spring Integration components that write files (locally or remotely); by default, they append `.writing` to the file name and remove it when the transfer is complete.
+
+Another common technique is to write a second "marker" file to indicate the file transfer is complete.
+In this scenario, say, you should not consider `foo.txt` to be available for use until `foo.txt.complete` is also present.
+Spring Integration _version 5.0_ introduces new filters to support this mechanism.
+Implementations are provided for the file system (`FileSystemMarkerFilePresentFileListFilter`), <<ftp-incomplete, FTP>> and <<sftp-incomplete, SFTP>>.
+They are configurable such that the marker file can have any name, although it will usually be related to the file being transferred.
+See the javadocs for more information.
+
 [[file-writing]]
 === Writing files
 

src/reference/asciidoc/ftp.adoc

@@ -509,6 +509,14 @@ public class FtpJavaApplication {
 }
 ----
 
+[[ftp-incomplete]]
+==== Dealing With Incomplete Data
+
+See <<file-incomplete>>.
+
+The `FtpSystemMarkerFilePresentFileListFilter` is provided to filter remote files that don't have a corresponding marker file on the remote system.
+See the javadocs for configuration information.
+
 [[ftp-streaming]]
 === FTP Streaming Inbound Channel Adapter
 

src/reference/asciidoc/sftp.adoc

@@ -554,6 +554,14 @@ public class SftpJavaApplication {
 }
 ----
 
+[[sftp-incomplete]]
+==== Dealing With Incomplete Data
+
+See <<file-incomplete>>.
+
+The `SftpSystemMarkerFilePresentFileListFilter` is provided to filter remote files that don't have the corresponding marker file on the remote system.
+See the javadocs for configuration information.
+
 [[sftp-streaming]]
 === SFTP Streaming Inbound Channel Adapter
 

src/reference/asciidoc/whats-new.adoc

@@ -132,6 +132,8 @@ See <<file-writing-destination-exists>> for more information.
 They also now support setting file permissions on the newly written file.
 See <<file-permissions>> for more information.
 
+A new `FileSystemMarkerFilePresentFileListFilter` is now available; see <<file-incomplete>> for more information.
+
 ==== (S)FTP Changes
 
 The inbound channel adapters now have a property `max-fetch-size` which is used to limit the number of files fetched during a poll when there are no files currently in the local directory.
@@ -159,6 +161,7 @@ The `RemoteFileTemplate` is supplied now with the `invoke(OperationsCallback<F,
 
 See <<ftp>> and <<sftp>> for more information.
 
+New filters for detecting incomplete remote files are now provided; see <<ftp-incomplete>> and <<sftp-incomplete>> for more information.
 
 ==== Integration Properties