INT-2898 Add Persistent FileListFilters

- Abstract implementation
- Implementation for the file system
- Implementation for remote files ((S)FTP)

JIRA: https://jira.springsource.org/browse/INT-2898

INT-2889 Polishing

- Add (S)FTP test cases
- Docs

src/reference/docbook/file.xml

@@ -35,6 +35,14 @@
       <interfacename>FileListFilter</interfacename>. By default, an
       <classname>AcceptOnceFileListFilter</classname> is used. This filter
       ensures files are picked up only once from the directory.
+      <note>
+        The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
+        state to survive a system restart, consider using the
+        <classname>FileSystemPersistentAcceptOnceFileListFilter</classname> instead. This filter stores
+        the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
+        several store implementations (such as Redis), or you can provide your own. This filter matches on
+        the filename and modified time.
+      </note>
       <programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
     class="org.springframework.integration.file.FileReadingMessageSource"
     p:inputDirectory="${input.directory}"

src/reference/docbook/ftp.xml

@@ -184,14 +184,37 @@ protected void postProcessClientBeforeConnect(T client) throws IOException {
 	(e.g. <code>filename-regex=".*\.test$"</code>). And of course if you need complete control you can use <code>filter</code>
 	attribute and provide a reference to any custom implementation of the
 	<classname>org.springframework.integration.file.filters.FileListFilter</classname>, a strategy interface for filtering a
-	list of files. This filter determines which remote files are retrieved.
+	list of files. This filter determines which remote files are retrieved. You can also combine a pattern based filter
+	with other filters, such as an <classname>AcceptOnceFileListFilter</classname> to avoid synchronizing files that
+	have previously been fetched, by using a <classname>CompositeFileListFilter</classname>.
   	</para>
+	<para>
+        The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
+        state to survive a system restart, consider using the
+        <classname>FtpPersistentAcceptOnceFileListFilter</classname> instead. This filter stores
+        the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
+        several store implementations (such as Redis), or you can provide your own. This filter matches on
+        the filename and the remote modified time.
+	</para>  	
 	<note>
-	Beginning with 3.0, you can also specify a filter used to filter the files locally, once they have
-	been retrieved. The default filter is an <classname>AcceptOnceFileListFilter</classname> which prevents processing
-	files with the same name multiple times in the same JVM execution; this can now be overridden
-	(for example with an <classname>AcceptAllFileListFilter</classname>), using the <code>local-filter</code> attribute.
-	Previously, the default	<classname>AcceptOnceFileListFilter</classname> could not be overridden.
+	  <para>
+		Beginning with <emphasis>version 3.0</emphasis>, you can also specify a filter used to filter the files locally, once they have
+		been retrieved. The default filter is an <classname>AcceptOnceFileListFilter</classname> which prevents processing
+		files with the same name multiple times in the same JVM execution; this can now be overridden
+		(for example with an <classname>AcceptAllFileListFilter</classname>), using the <code>local-filter</code> attribute.
+		Previously, the default	<classname>AcceptOnceFileListFilter</classname> could not be overridden.
+	  </para>
+	  <para>
+        The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
+        state to survive a system restart, consider using the
+        <classname>FileSystemPersistentAcceptOnceFileListFilter</classname> as a local filter instead. This filter stores
+        the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
+        several store implementations (such as Redis), or you can provide your own.
+        <important>
+        	This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a
+        	re-synchronized file from being processed, you should use the <code>preserve-timestamp</code> attribute discussed above.
+        </important>
+	  </para>
 	</note>
 	<para>
 	The 'remote-file-separator' attribute allows you to configure a

src/reference/docbook/sftp.xml

@@ -297,14 +297,37 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/sftp
 	(e.g. <code>filename-regex=".*\.test$"</code>). And of course if you need complete control you can use the <code>filter</code>
 	attribute to provide a reference to a custom implementation of the
 	<classname>org.springframework.integration.file.filters.FileListFilter</classname> - a strategy interface for filtering a
-	list of files. This filter determines which remote files are retrieved.
+	list of files. This filter determines which remote files are retrieved. You can also combine a pattern based filter
+	with other filters, such as an <classname>AcceptOnceFileListFilter</classname> to avoid synchronizing files that
+	have previously been fetched, by using a <classname>CompositeFileListFilter</classname>.
   	</para>
+	<para>
+        The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
+        state to survive a system restart, consider using the
+        <classname>SftpPersistentAcceptOnceFileListFilter</classname> instead. This filter stores
+        the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
+        several store implementations (such as Redis), or you can provide your own. This filter matches on
+        the filename and the remote modified time.
+	</para>  	
 	<note>
-	Beginning with 3.0, you can also specify a filter used to filter the files locally, once they have
-	been retrieved. The default filter is an <classname>AcceptOnceFileListFilter</classname> which prevents processing
-	files with the same name multiple times in the same JVM execution; this can now be overridden
-	(for example with an <classname>AcceptAllFileListFilter</classname>), using the <code>local-filter</code> attribute.
-	Previously, the default	<classname>AcceptOnceFileListFilter</classname> could not be overridden.
+	  <para>
+		Beginning with <emphasis>version 3.0</emphasis>, you can also specify a filter used to filter the files locally, once they have
+		been retrieved. The default filter is an <classname>AcceptOnceFileListFilter</classname> which prevents processing
+		files with the same name multiple times in the same JVM execution; this can now be overridden
+		(for example with an <classname>AcceptAllFileListFilter</classname>), using the <code>local-filter</code> attribute.
+		Previously, the default	<classname>AcceptOnceFileListFilter</classname> could not be overridden.
+	  </para>
+	  <para>
+        The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
+        state to survive a system restart, consider using the
+        <classname>FileSystemPersistentAcceptOnceFileListFilter</classname> as a local filter instead. This filter stores
+        the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
+        several store implementations (such as Redis), or you can provide your own.
+        <important>
+        	This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a
+        	re-synchronized file from being processed, you should use the <code>preserve-timestamp</code> attribute discussed above.
+        </important>
+	  </para>
 	</note>
   	<para>
   	Please refer to the schema for more detail on these attributes.

src/reference/docbook/whats-new.xml

@@ -604,5 +604,13 @@
 				For more information, see <xref linkend="redis"/>.
 			</para>
 		</section>
+		<section id="3.0-filelistfilter">
+			<title>Persistend File List Filters (file, (S)FTP)</title>
+			<para>
+				New <classname>FileListFilter</classname>s that use a persistent <classname>MetadataStore</classname> are
+				now available. These can be used to prevent duplicate files after a system restart. See
+				<xref linkend="file-reading"/>, <xref linkend="ftp-inbound"/>, and <xref linkend="sftp-inbound"/> for more information. 
+			</para>
+		</section>
 	</section>
 </chapter>