spring-integration/src/reference/docbook/file.xml

<?xml version="1.0" encoding="UTF-8"?>
<chapter xmlns="http://docbook.org/ns/docbook" version="5.0" xml:id="files"
    xmlns:xlink="http://www.w3.org/1999/xlink">
  <title>File Support</title>

  <section id="file-intro">
    <title>Introduction</title>
    <para>
      Spring Integration's File support extends the Spring Integration Core with
      a dedicated vocabulary to deal with reading, writing, and transforming files.
      It provides a namespace that enables elements defining Channel Adapters dedicated
      to files and support for Transformers that can read file contents into strings or
      byte arrays.
      </para>
    <para>
      This section will explain the workings of <classname>FileReadingMessageSource</classname>
      and <classname>FileWritingMessageHandler</classname> and how to configure them as
      <emphasis>beans</emphasis>. Also the support for dealing with files through file specific
      implementations of <interfacename>Transformer</interfacename> will be discussed. Finally the
      file specific namespace will be explained.
    </para>
  </section>

  <section id="file-reading">
    <title>Reading Files</title>
    <para>
      A <classname>FileReadingMessageSource</classname> can be used to consume files from the filesystem.
      This is an implementation of <interfacename>MessageSource</interfacename> that creates messages from
      a file system directory. <programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
    class="org.springframework.integration.file.FileReadingMessageSource"
    p:inputDirectory="${input.directory}"/>]]></programlisting>
    </para>
    <para>
      To prevent creating messages for certain files, you may supply a
      <interfacename>FileListFilter</interfacename>. By default, an
      <classname>AcceptOnceFileListFilter</classname> is used. This filter
      ensures files are picked up only once from the directory.
      <note>
        The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
        state to survive a system restart, consider using the
        <classname>FileSystemPersistentAcceptOnceFileListFilter</classname> instead. This filter stores
        the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
        several store implementations (such as Redis), or you can provide your own. This filter matches on
        the filename and modified time.
      </note>
      <programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
    class="org.springframework.integration.file.FileReadingMessageSource"
    p:inputDirectory="${input.directory}"
    p:filter-ref="customFilterBean"/>]]></programlisting>
    </para>
    <para>
      A common problem with reading files is that a file may be detected before
      it is ready. The default <classname>AcceptOnceFileListFilter</classname>
      does not prevent this. In most cases, this can be prevented if the
      file-writing process renames each file as soon as it is ready for
      reading. A filename-pattern or filename-regex filter that accepts only files that are
      ready (e.g. based on a known suffix), composed with the default
      <classname>AcceptOnceFileListFilter</classname> allows for this.
      The <classname>CompositeFileListFilter</classname> enables the
      composition.
      <programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
    class="org.springframework.integration.file.FileReadingMessageSource"
    p:inputDirectory="${input.directory}"
    p:filter-ref="compositeFilter"/>
<bean id="compositeFilter"
    class="org.springframework.integration.file.filters.CompositeFileListFilter">
    <constructor-arg>
        <list>
            <bean class="o.s.i.file.filters.AcceptOnceFileListFilter"/>
            <bean class="o.s.i.file.filters.RegexPatternFileListFilter">
                <constructor-arg value="^test.*$"/>
            </bean>
        </list>
    </constructor-arg>
</bean>]]></programlisting>
    </para>
    <para>
      The configuration can be simplified using the file specific namespace. To do
      this use the following template.
      <programlisting language="xml"><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:int="http://www.springframework.org/schema/integration"
  xmlns:int-file="http://www.springframework.org/schema/integration/file"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
    http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/integration
    http://www.springframework.org/schema/integration/spring-integration.xsd
    http://www.springframework.org/schema/integration/file
    http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">
</beans>]]></programlisting>
    </para>
        <para>
      Within this namespace you can reduce the FileReadingMessageSource and wrap
      it in an inbound Channel Adapter like this:
      <programlisting language="xml"><![CDATA[<int-file:inbound-channel-adapter id="filesIn1"
    directory="file:${input.directory}" prevent-duplicates="true"/>

<int-file:inbound-channel-adapter id="filesIn2"
    directory="file:${input.directory}"
    filter="customFilterBean" />

<int-file:inbound-channel-adapter id="filesIn3"
    directory="file:${input.directory}"
    filename-pattern="test*" />

<int-file:inbound-channel-adapter id="filesIn4"
    directory="file:${input.directory}"
    filename-regex="test[0-9]+\.txt" />]]></programlisting>
    </para>
        <para>
      The first channel adapter is relying on the default filter that just prevents
      duplication, the second is using a custom filter, the third is using the
      <emphasis>filename-pattern</emphasis> attribute to add an <classname>AntPathMatcher</classname>
      based filter, and the fourth is using the <emphasis>filename-regex</emphasis> attribute to add a
      regular expression Pattern based filter to the <classname>FileReadingMessageSource</classname>.
      The <emphasis>filename-pattern</emphasis> and <emphasis>filename-regex</emphasis> attributes are
      each mutually exclusive with the regular <emphasis>filter</emphasis> reference attribute. However,
      you can use the <emphasis>filter</emphasis> attribute to reference an instance of
      <classname>CompositeFileListFilter</classname> that combines any number of filters, including one
      or more pattern based filters to fit your particular needs.
    </para>
        <para>
            When multiple processes are reading from the same directory it can be desirable to lock files to prevent
            them from being picked up concurrently. To do this you can use a <interfacename>FileLocker</interfacename>.
            There is a java.nio based implementation available out of the box, but it is also possible to implement your
            own locking scheme. The nio locker can be injected as follows
            <programlisting  language="xml"><![CDATA[<int-file:inbound-channel-adapter id="filesIn"
    directory="file:${input.directory}" prevent-duplicates="true">
    <int-file:nio-locker/>
</int-file:inbound-channel-adapter>]]></programlisting>
    </para>
        <para>
            A custom locker you can configure like this:
             <programlisting  language="xml"><![CDATA[<int-file:inbound-channel-adapter id="filesIn"
    directory="file:${input.directory}" prevent-duplicates="true">
    <int-file:locker ref="customLocker"/>
</int-file:inbound-channel-adapter>]]></programlisting>
        </para>
      <note>
            When a file inbound adapter is configured with a locker, it will take the responsibility to acquire a
            lock before the file is allowed to be received.
            <emphasis role="bold">It will not assume the responsibility to unlock the file.</emphasis>
            If you have processed the file and keeping the locks hanging around you have a memory leak. If this is
            a problem in your case you should call FileLocker.unlock(File file) yourself at the appropriate time.
      </note>
        <para>
            When filtering and locking files is not enough it might be needed to control the way files are listed entirely. To
            implement this type of requirement you can use an implementation of <interfacename>DirectoryScanner</interfacename>.
            This scanner allows you to determine entirely what files are listed each poll. This is also the interface
            that Spring Integration uses internally to wire FileListFilters FileLocker to the FileReadingMessageSource.
            A custom DirectoryScanner can be injected into the &lt;int-file:inbound-channel-adapter/&gt; on the <code>scanner</code>
            attribute.
            <programlisting language="xml"><![CDATA[<int-file:inbound-channel-adapter id="filesIn" directory="file:${input.directory}"
    prevent-duplicates="true" scanner="customDirectoryScanner"/>]]></programlisting>
        </para>
        <para>
            This gives you full freedom to choose the ordering, listing and locking strategies.
        </para>
        <important>
            It is important to understand that filters (including patterns, regex, prevent-duplicates etc) and lockers,
            are actually used by the scanner. Any of these attributes set on the adapter are subsequently injected into the
            scanner. For this reason, if you need to provide a custom scanner and you have multiple file inbound adapters
            in the same application context, each adapter must be provided with its own instance of the scanner, either
            by declaring separate beans, or declaring <code>scope="prototype"</code> on the scanner bean so that the
            context will create a new instance for each use.
        </important>
        <section id="file-tailing">
          <title>'Tail'ing Files</title>
          <para>
            Another popular use case is to get 'lines' from the end (or tail) of a file. Two implementations are provided;
            the first, <classname>OSDelegatingFileTailingMessageProducer</classname>, uses the native <code>tail</code>
            command (on operating systems that have one). This is likely the most efficient implementation on those
            platforms. For operating systems that do not have a <code>tail</code> command, the second implementation
            <classname>ApacheCommonsFileTailingMessageProducer</classname> which uses the Apache <code>commons-io
            Tailer</code> class.
          </para>
          <para>
            In both cases, file system events, such as files being unavailable etc, are published as
            <interfacename>ApplicationEvent</interfacename>s using the normal Spring event publishing mechanism.
            Examples of such events are:
          </para>
          <para><code>
          [message=tail: cannot open `/tmp/foo' for reading:
               No such file or directory, file=/tmp/foo]
          </code></para>
          <para><code>
          [message=tail: `/tmp/foo' has become accessible, file=/tmp/foo]
          </code></para>
          <para><code>
          [message=tail: `/tmp/foo' has become inaccessible:
               No such file or directory, file=/tmp/foo]
           </code></para>
           <para><code>
          [message=tail: `/tmp/foo' has appeared;
               following end of new file, file=/tmp/foo]
          </code></para>
          <para>
            This sequence of events might occur, for example, when a file is rotated.
          </para>
          <note>
            Not all platforms supporting a <code>tail</code> command provide these status messages.
          </note>
          <para>
            Example configurations:
          </para>
          <programlisting language="xml"><![CDATA[<int-file:tail-inbound-channel-adapter id="native"
	channel="input"
	task-executor="exec"
	file="/tmp/foo"/>]]></programlisting>
          <para>
            This creates a native adapter with default '-F -n 0' options (follow the file name from the current end).
          </para>
          <programlisting language="xml"><![CDATA[<int-file:tail-inbound-channel-adapter id="native"
	channel="input"
	native-options="-F -n 6"
	task-executor="exec"
	file-delay=10000
	file="/tmp/foo"/>]]></programlisting>
          <para>
            This creates a native adapter with '-F -n 6' options (follow the file name, emit up to 6 lines before the current end).
            If the tail command fails (on some platforms, a missing file causes the <code>tail</code> to fail, even with
            <code>-F</code> specified), the command will be retried every 10 seconds.
          </para>
          <programlisting language="xml"><![CDATA[<int-file:tail-inbound-channel-adapter id="apache"
	channel="input"
	task-executor="exec"
	file="/tmp/bar"
	delay="2000"
	end="false"
	reopen="true"
	file-delay="10000"/>]]></programlisting>
          <para>
            This creates a commons-io <classname>Tailer</classname> adapter that examines the file for new lines every
            2 seconds, and checks for existence of a missing file every 10 seconds. The file will be tailed from the
            beginning (<code>end="false"</code>) instead of the end (which is the default). The file will be
            reopened for each chunk (the default is to keep the file open).
          </para>
        </section>
  </section>
	<section id="file-writing">
		<title>Writing files</title>
		<para>
			To write messages to the file system you can use a
			<classname><ulink url="http://static.springsource.org/spring-integration/api/org/springframework/integration/file/FileWritingMessageHandler.html">FileWritingMessageHandler</ulink></classname>.
			This class can deal with <emphasis>File</emphasis>,
			<emphasis>String</emphasis>, or <emphasis>byte array</emphasis>
			payloads.
		</para>
		<para>
			You can configure the encoding and the charset that
			will be used in case of a String payload.
		</para>
		<para>
			To make things easier, you can configure the <classname>FileWritingMessageHandler</classname>
			as part of an <emphasis>Outbound Channel Adapter</emphasis> or
			<emphasis>Outbound Gateway</emphasis> using the provided XML namespace
			support.
		</para>
		<section id="file-writing-file-names">
			<title>Generating Filenames</title>
			<para>
				In its simplest form, the <classname>FileWritingMessageHandler</classname>
				only requires a destination directory for writing the files. The
				name of the file to be written is determined by the handler's
				<classname><ulink url="http://static.springsource.org/spring-integration/api/org/springframework/integration/file/FileNameGenerator.html">FileNameGenerator</ulink></classname>.
				The <ulink url="http://static.springsource.org/spring-integration/api/org/springframework/integration/file/DefaultFileNameGenerator.html">default implementation</ulink>
				looks for a Message header whose key matches the constant defined
				as <code><ulink url="http://static.springsource.org/spring-integration/api/constant-values.html#org.springframework.integration.file.FileHeaders.FILENAME">FileHeaders.FILENAME</ulink></code>.
			</para>
			<para>
				Alternatively, you can specify an expression
				to be evaluated against the Message in order to generate a file name, e.g.:
				<emphasis>headers['myCustomHeader'] + '.foo'</emphasis>. The expression must
				evaluate to a <classname>String</classname>. For convenience,
				the <classname>DefaultFileNameGenerator</classname> also
				provides the <emphasis>setHeaderName</emphasis> method, allowing you
				to explicitly specify the Message header whose value shall be
				used as the filename.
			</para>
			<para>
				Once setup, the <classname>DefaultFileNameGenerator</classname> will
				employ the following resolution steps to determine the filename
				for a given Message payload:
			</para>
			<orderedlist>
				<listitem>
					Evaluate the expression against the Message and, if the result is a
					non-empty <classname>String</classname>, use it as the
					filename.
				</listitem>
				<listitem>
					Otherwise, if the payload is a <classname>java.io.File</classname>, use the
					file's filename.
				</listitem>
				<listitem>
					Otherwise, use the Message ID appended with <quote>.msg</quote>
					as the filename.
				</listitem>
			</orderedlist>
			<para>
				When using the XML namespace support, both, the
				<emphasis>File Oubound Channel Adapter</emphasis> and
				the <emphasis>File Outbound Gateway</emphasis> support the
				following two mutually exclusive configuration attributes:
			</para>
			<itemizedlist>
				<listitem><code>filename-generator</code> (a reference to a <classname>FileNameGenerator</classname>) implementation)</listitem>
				<listitem><code>filename-generator-expression</code> (an expression evaluating to a <classname>String</classname>)</listitem>
			</itemizedlist>
			<para>
				While writing files, a temporary file suffix will be used (default:
				<quote>.writing</quote>). It is appended to the filename while
				the file is being written. To customize the suffix, you
				can set the <emphasis>temporary-file-suffix</emphasis> attribute
				on both, the <emphasis>File Oubound Channel Adapter</emphasis> and
				the <emphasis>File Outbound Gateway</emphasis>.
			</para>
			<note>
				When using the <emphasis>APPEND</emphasis> file <emphasis>mode</emphasis>,
				the <emphasis>temporary-file-suffix</emphasis> attribute is ignored,
				since the data is appended to the file directly.
			</note>
		</section>
		<section id="file-writing-output-directory">
			<title>Specifying the Output Directory</title>
			<para>
				Both, the <emphasis>File Oubound Channel Adapter</emphasis> and
				the <emphasis>File Outbound Gateway</emphasis> provide two
				configuration attributes for specifying the output directory:
			</para>
			<itemizedlist>
				<listitem>
					<para><emphasis>directory</emphasis></para>
				</listitem>
				<listitem>
					<para><emphasis>directory-expression</emphasis></para>
				</listitem>
			</itemizedlist>
			<note>
				The <emphasis>directory-expression</emphasis> attribute is available since
				Spring Integration 2.2.
			</note>
			<para><emphasis role="bold">Using the directory attribute</emphasis></para>
			<para>
				When using the <emphasis>directory</emphasis> attribute, the output
				directory will be set to a fixed value, that is set at
				intialization time of the <classname>FileWritingMessageHandler</classname>.
				If you don't specify this attribute, then you must use the
				<emphasis>directory-expression</emphasis> attribute.
			</para>
			<para><emphasis role="bold">Using the directory-expression attribute</emphasis></para>
			<para>
				If you want to have full SpEL support you would choose the
				<emphasis>directory-expression</emphasis> attribute. This attribute
				accepts a SpEL expression that is evaluated for each message being
				processed. Thus, you have full access to a Message's payload and
				its headers to dynamically specify the output file directory.
			</para>
			<para>
				The SpEL expression must resolve to either a <classname>String</classname>
				or to <classname>java.io.File</classname>. Furthermore the resulting
				<classname>String</classname> or <classname>File</classname> must
				point to a directory. If you don't specify the
				<emphasis>directory-expression</emphasis> attribute, then you
				must set the <emphasis>directory</emphasis> attribute.
			</para>
			<para><emphasis role="bold">Using the auto-create-directory attribute</emphasis></para>
			<para>
				If the destination directory does not exists, yet, by default the
				respective destination directory and any non-existing parent directories
				are being created automatically. You can set the
				<emphasis>auto-create-directory</emphasis> attribute to
				<emphasis>false</emphasis> in order to prevent that. This attribute
				applies to both, the <emphasis>directory</emphasis> and the
				<emphasis>directory-expression</emphasis> attribute.
			</para>
			<note>
				<para>
					When using the <emphasis>directory</emphasis> attribute and
					<emphasis>auto-create-directory</emphasis> is <code>false</code>,
					the following change was made starting with Spring Integration 2.2:
				</para>
				<para>
					Instead of checking for the existence of the destination directory
					at initialization time of the adapter, this check is now
					performed for each message being processed.
				</para>
				<para>
					Furthermore, if <emphasis>auto-create-directory</emphasis> is
					<code>true</code> and the directory was deleted between the
					processing of messages, the directory will be re-created for each
					message being processed.
				</para>
			</note>
		</section>
		<section id="file-writing-destination-exists">
			<title>Dealing with Existing Destination Files</title>
			<para>
				When writing files and the destination file already exists, the
				default behavior is to overwrite that target file. This behavior,
				though, can be changed by setting the <emphasis>mode</emphasis>
				attribute on the respective File Outbound components. The following
				options exist:
			</para>
			<itemizedlist>
				<listitem>REPLACE (Default)</listitem>
				<listitem>APPEND</listitem>
				<listitem>FAIL</listitem>
				<listitem>IGNORE</listitem>
			</itemizedlist>
			<note>
				The <emphasis>mode</emphasis> attribute and the options
				<emphasis>APPEND</emphasis>, <emphasis>FAIL</emphasis> and
				<emphasis>IGNORE</emphasis>, are available since
				<emphasis>Spring Integration 2.2</emphasis>.
			</note>
			<para><emphasis>REPLACE</emphasis></para>
			<para>
				If the target file already exists, it will be overwritten. If the
				<emphasis>mode</emphasis> attribute is not specified, then this
				is the default behavior when writing files.
			</para>
			<para><emphasis>APPEND</emphasis></para>
			<para>
				This mode allows you to append Message content to the existing
				file instead of creating a new file each time. Note that this
				attribute is mutually exclusive with <emphasis>temporary-file-suffix</emphasis>
				attribute since when appending content to the existing file, the
				adapter no longer uses a temporary file.
			</para>
			<para><emphasis>FAIL</emphasis></para>
			<para>
				If the target file exists, a
				<ulink url="http://static.springsource.org/spring-integration/api/org/springframework/integration/MessageHandlingException.html">MessageHandlingException</ulink>
				is thrown.
			</para>
			<para><emphasis>IGNORE</emphasis></para>
			<para>
				If the target file exists, the message payload is silently
				ignored.
				<note>
					When using a temporary file suffix (default: <code>.writing</code>), the <emphasis>IGNORE</emphasis>
					mode will apply if the final file name exists, or the temporary file name exists.
				</note>
			</para>
		</section>
		<section id="file-outbound-channel-adapter">
			<title>File Outbound Channel Adapter</title>
			<programlisting language="xml"><![CDATA[<int-file:outbound-channel-adapter id="filesOut" directory="${input.directory.property}"/>]]></programlisting>
			<para>
				The namespace based configuration also supports a <code>delete-source-files</code> attribute.
				If set to <code>true</code>, it will trigger the deletion of the original source files after writing
				to a destination. The default value for that flag is <code>false</code>.
			</para>
			<programlisting language="xml"><![CDATA[<int-file:outbound-channel-adapter id="filesOut"
    directory="${output.directory}"
    delete-source-files="true"/>]]></programlisting>
			<note>
				The <code>delete-source-files</code> attribute will only have an effect if the inbound
				Message has a File payload or if the <classname>FileHeaders.ORIGINAL_FILE</classname> header
				value contains either the source File instance or a String representing the original file path.
			</note>
		</section>
		<section id="file-writing-output-gateway">
			<title>Outbound Gateway</title>
			<para>
				In cases where you want to continue processing messages based on
				the written file, you can use the <code>outbound-gateway</code>
				instead. It plays a very similar role as the
				<code>outbound-channel-adapter</code>. However, after writing the
				file, it will also send it to the reply channel as the payload of
				a Message.
			</para>
			<programlisting language="xml"><![CDATA[<int-file:outbound-gateway id="mover" request-channel="moveInput"
    reply-channel="output"
    directory="${output.directory}"
    mode="REPLACE" delete-source-files="true"/>]]></programlisting>
			<para>
				As mentioned earlier, you can also specify the <emphasis>mode</emphasis>
				attribute, which defines the behavior of how to deal with situations
				where the destination file already exists. Please see
				<xref linkend="file-writing-destination-exists"/> for further
				details. Generally, when using the
				<emphasis>File Outbound Gateway</emphasis>, the result file is
				returned as the Message payload on the reply channel.
			</para>
			<para>
				This also applies when specifying the <emphasis>IGNORE</emphasis>
				mode. In that case the pre-existing destination file is returned.
				If the payload of the request message was a file, you still have
				access to that original file through the Message Header
				<emphasis><ulink url="http://static.springsource.org/spring-integration/api/org/springframework/integration/file/FileHeaders.html">FileHeaders.ORIGINAL_FILE</ulink></emphasis>.
			</para>
			<note>
				The 'outbound-gateway' works well in cases where you want to first move a file and then send it
				through a processing pipeline. In such cases, you may connect the file namespace's
				'inbound-channel-adapter' element to the 'outbound-gateway' and then connect that gateway's
				reply-channel to the beginning of the pipeline.
			</note>
			<para>
				If you have more elaborate requirements or need to support additional payload types as input
				to be converted to file content you could extend the FileWritingMessageHandler, but a much
				better option is to rely on a <classname>Transformer</classname>.
			</para>
		</section>
	</section>
  <section id="file-transforming">
    <title>File Transformers</title>
    <para>
      To transform data read from the file system to objects and the other way around you need
      to do some work. Contrary to <classname>FileReadingMessageSource</classname> and to a
      lesser extent <classname>FileWritingMessageHandler</classname>, it is very likely that you
      will need your own mechanism to get the job done. For this you can implement the
      <interfacename>Transformer</interfacename> interface. Or extend the
      <classname>AbstractFilePayloadTransformer</classname> for inbound messages. Some obvious
      implementations have been provided.
    </para>
    <para>
      <classname>FileToByteArrayTransformer</classname> transforms Files into byte[]s using
      Spring's <classname>FileCopyUtils</classname>. It is often better to use a sequence of
      transformers than to put all transformations in a single class. In that case the File to
      byte[] conversion might be a logical first step.
    </para>
    <para>
      <classname>FileToStringTransformer</classname> will convert Files to Strings as the name
      suggests. If nothing else, this can be useful for debugging (consider using with a Wire Tap).
    </para>
    <para>
      To configure File specific transformers you can use the appropriate elements from the file namespace.
      <programlisting language="xml"><![CDATA[<int-file:file-to-bytes-transformer  input-channel="input" output-channel="output"
    delete-files="true"/>

<int-file:file-to-string-transformer input-channel="input" output-channel="output"
    delete-files="true" charset="UTF-8"/>]]></programlisting>
    </para>
    <para>
      The <emphasis>delete-files</emphasis> option signals to the transformer that it should delete
      the inbound File after the transformation is complete. This is in no way a replacement for using the
      <classname>AcceptOnceFileListFilter</classname> when the FileReadingMessageSource is being used in a
      multi-threaded environment (e.g. Spring Integration in general).
    </para>
  </section>

</chapter>