Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-2388 Update Gradle build

Browse Source 
This is a significant update to the build system, including the changes
listed below. README.md has been updated with instructions on the most
important day-to-day commands.

 - Eliminate buildSrc submodule

   In favor of using the new bundlor and docbook-reference plugins. The
   net effect is a large reduction in number of lines of build code.
   Common docbook resources, stylesheets, etc are stored directly in the
   docbook plugin.

   This means that --recursive is no longer required when cloning and
   there will never be a need to use `git submodule` commands. README
   files have been updated to reflect.

   Use of the new bundlor plugin also means the removal of template.mf
   files from the source tree in favor of an inline approach. See
   build.gradle for details. Bundlor 'import templates' are built up
   programmatically and kept physically close to gradle dependency
   declarations, leading to more convenience when changing these values
   and hopefully fewer errors / version inconsistencies over time.

   Certain tests depended on the presence of template.mf files, all of
   which have recently been removed from the source tree in favor of the
   new bundlor plugin which allows for inlining bundlor configuration
   within the Gradle build script. These tests now create temp files
   using the java.io.File API instead.

 - Upgrade to Gradle 1.0-milestone-6

   The m6 release is significantly faster when resolving dependencies
   and has a number of valuable new features over the earlier m3
   version. Review the release notes for Gradle 1.0-milestone-6 online
   for full details.

 - Switch to repo.springsource.org repository

   Previously the project build declared as many repositories as
   necessary to resolve all project dependencies.

   Now depending on a single 'virtual repository' defined within the
   SpringSource Artifactory instance at http://repo.springsource.org.
   Currently, the virtual repository in use is 'libs-milestone', which
   allows for the resolution of all "milestone-or-better" versions of
   all S2 and third-party dependencies.

   Should snapshot dependencies become required, this value may be
   changed from 'libs-milestone' to 'libs-snapshot'. To build only
   against GA releases, change the value to 'libs-release'.

 - New build plan(s)

   Spring Integration build plans have been updated to use the
   Artifactory Bamboo plugin and publish to repo.springsource.org.
   Build plans have names like 2.1.x to reflect the version under
   development, not necessarily the name of the branch, as this may
   change over time and across major releases.

 - Improve release process

   As mentioned above, Spring Integration will now use the Artifactory
   Bamboo plugin to publish releases and also use Artifactory's support
   for pushing builds directly into Maven Central via oss.sonatype.org.

   Generate poms that contain all necessary fields for onboarding at
   Maven central (scm, developers, organization, licenses, etc).

   Generate -source and -javadoc poms to comply with Maven Central
   onboarding rules (and for general good practice anyway).

   Generation of PGP signatures, sha1 and md5 checksums are all handled
   automatically by Artifactory. These are also requirements for
   automated entry into Maven Central.

 - Remove source-level pom generation

   Automatic generation of Maven poms suitable for use in building
   Spring Integration is no longer supported. Generation and
   publication of poms for the purpose of dependency management remains
   supported.

   Sonar support has to date depended on these poms, but will be
   switched over to use the Gradle Sonar plugin shortly.

 - Eliminate docs subproject

   Move docs/src to the root of the project and eliminate docs as a
   formal subproject. This simplifies the build in a number of ways,
   including removing the need for distinguishing between 'subprojects'
   and 'javaprojects' as well as allowing users to build both 'api' and
   'reference' docs without qualifying with a ':docs' prefix.

   Also rename the src/info directory to src/dist to better reflect that
   these files are packaged with the distribution. For example, the
   readme.txt there is really the distribution readme, distinct from the
   README.md at the root of the project which is for building from source,
   etc.
This commit is contained in:
Chris Beams
2011-11-08 14:38:53 -08:00
committed by Mark Fisher
parent ae0ecff886
commit f30da932e8
225 changed files with 811 additions and 7510 deletions
Download Patch File Download Diff File Expand all files Collapse all files

248
src/reference/docbook/file.xml Normal file
View File

@@ -0,0 +1,248 @@
<?xml version="1.0" encoding="UTF-8"?>
<chapter xmlns="http://docbook.org/ns/docbook" version="5.0" xml:id="files"
xmlns:xlink="http://www.w3.org/1999/xlink">
<title>File Support</title>
<section id="file-intro">
<title>Introduction</title>
<para>
Spring Integration's File support extends the Spring Integration Core with
a dedicated vocabulary to deal with reading, writing, and transforming files.
It provides a namespace that enables elements defining Channel Adapters dedicated
to files and support for Transformers that can read file contents into strings or
byte arrays.
</para>
<para>
This section will explain the workings of <classname>FileReadingMessageSource</classname>
and <classname>FileWritingMessageHandler</classname> and how to configure them as
<emphasis>beans</emphasis>. Also the support for dealing with files through file specific
implementations of <interfacename>Transformer</interfacename> will be discussed. Finally the
file specific namespace will be explained.
</para>
</section>
<section id="file-reading">
<title>Reading Files</title>
<para>
A <classname>FileReadingMessageSource</classname> can be used to consume files from the filesystem.
This is an implementation of <interfacename>MessageSource</interfacename> that creates messages from
a file system directory. <programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
class="org.springframework.integration.file.FileReadingMessageSource"
p:inputDirectory="${input.directory}"/>]]></programlisting>
</para>
<para>
To prevent creating messages for certain files, you may supply a
<interfacename>FileListFilter</interfacename>. By default, an
<classname>AcceptOnceFileListFilter</classname> is used. This filter
ensures files are picked up only once from the directory.
<programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
class="org.springframework.integration.file.FileReadingMessageSource"
p:inputDirectory="${input.directory}"
p:filter-ref="customFilterBean"/>]]></programlisting>
</para>
<para>
A common problem with reading files is that a file may be detected before
it is ready. The default <classname>AcceptOnceFileListFilter</classname>
does not prevent this. In most cases, this can be prevented if the
file-writing process renames each file as soon as it is ready for
reading. A filename-pattern or filename-regex filter that accepts only files that are
ready (e.g. based on a known suffix), composed with the default
<classname>AcceptOnceFileListFilter</classname> allows for this.
The <classname>CompositeFileListFilter</classname> enables the
composition.
<programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
class="org.springframework.integration.file.FileReadingMessageSource"
p:inputDirectory="${input.directory}"
p:filter-ref="compositeFilter"/>
<bean id="compositeFilter"
class="org.springframework.integration.file.filters.CompositeFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.file.filters.AcceptOnceFileListFilter"/>
<bean class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
<constructor-arg value="^test.*$"/>
</bean>
</list>
</constructor-arg>
</bean>]]></programlisting>
</para>
<para>
The configuration can be simplified using the file specific namespace. To do
this use the following template.
<programlisting language="xml"><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-file="http://www.springframework.org/schema/integration/file"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration
http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/file
http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">
</beans>]]></programlisting>
</para>
<para>
Within this namespace you can reduce the FileReadingMessageSource and wrap
it in an inbound Channel Adapter like this:
<programlisting language="xml"><![CDATA[ <int-file:inbound-channel-adapter id="filesIn1"
directory="file:${input.directory}" prevent-duplicates="true"/>
<int-file:inbound-channel-adapter id="filesIn2"
directory="file:${input.directory}"
filter="customFilterBean" />
<int-file:inbound-channel-adapter id="filesIn3"
directory="file:${input.directory}"
filename-pattern="test*" />
<int-file:inbound-channel-adapter id="filesIn4"
directory="file:${input.directory}"
filename-regex="test[0-9]+\.txt" /> ]]></programlisting>
</para>
<para>
The first channel adapter is relying on the default filter that just prevents
duplication, the second is using a custom filter, the third is using the
<emphasis>filename-pattern</emphasis> attribute to add an <classname>AntPathMatcher</classname>
based filter, and the fourth is using the <emphasis>filename-regex</emphasis> attribute to add a
regular expression Pattern based filter to the <classname>FileReadingMessageSource</classname>.
The <emphasis>filename-pattern</emphasis> and <emphasis>filename-regex</emphasis> attributes are
each mutually exclusive with the regular <emphasis>filter</emphasis> reference attribute. However,
you can use the <emphasis>filter</emphasis> attribute to reference an instance of
<classname>CompositeFileListFilter</classname> that combines any number of filters, including one
or more pattern based filters to fit your particular needs.
</para>
<para>
When multiple processes are reading from the same directory it can be desirable to lock files to prevent
them from being picked up concurrently. To do this you can use a <interfacename>FileLocker</interfacename>.
There is a java.nio based implementation available out of the box, but it is also possible to implement your
own locking scheme. The nio locker can be injected as follows
<programlisting language="xml"><![CDATA[ <int-file:inbound-channel-adapter id="filesIn"
directory="file:${input.directory}" prevent-duplicates="true">
<int-file:nio-locker/>
</int-file:inbound-channel-adapter>]]></programlisting>
</para>
<para>
A custom locker you can configure like this:
<programlisting language="xml"><![CDATA[ <int-file:inbound-channel-adapter id="filesIn"
directory="file:${input.directory}" prevent-duplicates="true">
<int-file:locker ref="customLocker"/>
</int-file:inbound-channel-adapter>]]></programlisting>
</para>
<note>
When a file inbound adapter is configured with a locker, it will take the responsibility to acquire a
lock before the file is allowed to be received.
<emphasis role="bold">It will not assume the responsibility to unlock the file.</emphasis>
If you have processed the file and keeping the locks hanging around you have a memory leak. If this is
a problem in your case you should call FileLocker.unlock(File file) yourself at the appropriate time.
</note>
<para>
When filtering and locking files is not enough it might be needed to control the way files are listed entirely. To
implement this type of requirement you can use an implementation of <interfacename>DirectoryScanner</interfacename>.
This scanner allows you to determine entirely what files are listed each poll. This is also the interface
that Spring Integration uses internally to wire FileListFilters FileLocker to the FileReadingMessageSource.
A custom DirectoryScanner can be injected into the &lt;int-file:inbound-channel-adapter/&gt; on the <code>scanner</code>
attribute.
<programlisting language="xml"><![CDATA[ <int-file:inbound-channel-adapter id="filesIn" directory="file:${input.directory}"
prevent-duplicates="true" scanner="customDirectoryScanner"/>]]></programlisting>
</para>
<para>
This gives you full freedom to choose the ordering, listing and locking strategies.
</para>
</section>
<section id="file-writing">
<title>Writing files</title>
<para>
To write messages to the file system you can use a
<classname>FileWritingMessageHandler</classname>. This class can deal with
File, String, or byte array payloads. In its simplest form the
<classname>FileWritingMessageHandler </classname> only requires a
destination directory for writing the files. The name of the file to be
written is determined by the handler's <classname>FileNameGenerator</classname>.
The default implementation looks for a Message header whose key matches
the constant defined as <code>FileHeaders.FILENAME</code>.
</para>
<para>
Additionally, you can configure the encoding and the charset that
will be used in case of a String payload.
</para>
<para>
To make things easier you can configure the FileWritingMessageHandler as
part of an outbound channel adapter using the namespace.
<programlisting language="xml"><![CDATA[ <int-file:outbound-channel-adapter id="filesOut" directory="${input.directory.property}"/>]]></programlisting>
</para>
<para>
The namespace based configuration also supports a <code>delete-source-files</code> attribute.
If set to <code>true</code>, it will trigger deletion of the original source files after writing
to a destination. The default value for that flag is <code>false</code>.
<programlisting language="xml"><![CDATA[ <int-file:outbound-channel-adapter id="filesOut"
directory="${output.directory}"
delete-source-files="true"/>]]></programlisting>
<note>
The <code>delete-source-files</code> attribute will only have an effect if the inbound
Message has a File payload or if the <classname>FileHeaders.ORIGINAL_FILE</classname> header
value contains either the source File instance or a String representing the original file path.
</note>
</para>
<para>
In cases where you want to continue processing messages based on the written File you can use
the <code>outbound-gateway</code> instead. It plays a very similar role as the
<code>outbound-channel-adapter</code>. However after writing the File, it will also send it
to the reply channel as the payload of a Message.
<programlisting language="xml"><![CDATA[ <int-file:outbound-gateway id="mover" request-channel="moveInput"
reply-channel="output"
directory="${output.directory}"
delete-source-files="true"/>]]></programlisting>
</para>
<note>
The 'outbound-gateway' works well in cases where you want to first move a file and then send it
through a processing pipeline. In such cases, you may connect the file namespace's
'inbound-channel-adapter' element to the 'outbound-gateway' and then connect that gateway's
reply-channel to the beginning of the pipeline.
</note>
<para>
If you have more elaborate requirements or need to support additional payload types as input
to be converted to file content you could extend the FileWritingMessageHandler, but a much
better option is to rely on a <classname>Transformer</classname>.
</para>
</section>
<section id="file-transforming">
<title>File Transformers</title>
<para>
To transform data read from the file system to objects and the other way around you need
to do some work. Contrary to <classname>FileReadingMessageSource</classname> and to a
lesser extent <classname>FileWritingMessageHandler</classname>, it is very likely that you
will need your own mechanism to get the job done. For this you can implement the
<interfacename>Transformer</interfacename> interface. Or extend the
<classname>AbstractFilePayloadTransformer</classname> for inbound messages. Some obvious
implementations have been provided.
</para>
<para>
<classname>FileToByteArrayTransformer</classname> transforms Files into byte[]s using
Spring's <classname>FileCopyUtils</classname>. It is often better to use a sequence of
transformers than to put all transformations in a single class. In that case the File to
byte[] conversion might be a logical first step.
</para>
<para>
<classname>FileToStringTransformer</classname> will convert Files to Strings as the name
suggests. If nothing else, this can be useful for debugging (consider using with a Wire Tap).
</para>
<para>
To configure File specific transformers you can use the appropriate elements from the file namespace.
<programlisting language="xml"><![CDATA[ <int-file:file-to-bytes-transformer input-channel="input" output-channel="output"
delete-files="true"/>
<int-file:file-to-string-transformer input-channel="input" output-channel="output"
delete-files="true" charset="UTF-8"/>]]></programlisting>
</para>
<para>
The <emphasis>delete-files</emphasis> option signals to the transformer that it should delete
the inbound File after the transformation is complete. This is in no way a replacement for using the
<classname>AcceptOnceFileListFilter</classname> when the FileReadingMessageSource is being used in a
multi-threaded environment (e.g. Spring Integration in general).
</para>
</section>
</chapter>