Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-2388 Update Gradle build

Browse Source 
This is a significant update to the build system, including the changes
listed below. README.md has been updated with instructions on the most
important day-to-day commands.

 - Eliminate buildSrc submodule

   In favor of using the new bundlor and docbook-reference plugins. The
   net effect is a large reduction in number of lines of build code.
   Common docbook resources, stylesheets, etc are stored directly in the
   docbook plugin.

   This means that --recursive is no longer required when cloning and
   there will never be a need to use `git submodule` commands. README
   files have been updated to reflect.

   Use of the new bundlor plugin also means the removal of template.mf
   files from the source tree in favor of an inline approach. See
   build.gradle for details. Bundlor 'import templates' are built up
   programmatically and kept physically close to gradle dependency
   declarations, leading to more convenience when changing these values
   and hopefully fewer errors / version inconsistencies over time.

   Certain tests depended on the presence of template.mf files, all of
   which have recently been removed from the source tree in favor of the
   new bundlor plugin which allows for inlining bundlor configuration
   within the Gradle build script. These tests now create temp files
   using the java.io.File API instead.

 - Upgrade to Gradle 1.0-milestone-6

   The m6 release is significantly faster when resolving dependencies
   and has a number of valuable new features over the earlier m3
   version. Review the release notes for Gradle 1.0-milestone-6 online
   for full details.

 - Switch to repo.springsource.org repository

   Previously the project build declared as many repositories as
   necessary to resolve all project dependencies.

   Now depending on a single 'virtual repository' defined within the
   SpringSource Artifactory instance at http://repo.springsource.org.
   Currently, the virtual repository in use is 'libs-milestone', which
   allows for the resolution of all "milestone-or-better" versions of
   all S2 and third-party dependencies.

   Should snapshot dependencies become required, this value may be
   changed from 'libs-milestone' to 'libs-snapshot'. To build only
   against GA releases, change the value to 'libs-release'.

 - New build plan(s)

   Spring Integration build plans have been updated to use the
   Artifactory Bamboo plugin and publish to repo.springsource.org.
   Build plans have names like 2.1.x to reflect the version under
   development, not necessarily the name of the branch, as this may
   change over time and across major releases.

 - Improve release process

   As mentioned above, Spring Integration will now use the Artifactory
   Bamboo plugin to publish releases and also use Artifactory's support
   for pushing builds directly into Maven Central via oss.sonatype.org.

   Generate poms that contain all necessary fields for onboarding at
   Maven central (scm, developers, organization, licenses, etc).

   Generate -source and -javadoc poms to comply with Maven Central
   onboarding rules (and for general good practice anyway).

   Generation of PGP signatures, sha1 and md5 checksums are all handled
   automatically by Artifactory. These are also requirements for
   automated entry into Maven Central.

 - Remove source-level pom generation

   Automatic generation of Maven poms suitable for use in building
   Spring Integration is no longer supported. Generation and
   publication of poms for the purpose of dependency management remains
   supported.

   Sonar support has to date depended on these poms, but will be
   switched over to use the Gradle Sonar plugin shortly.

 - Eliminate docs subproject

   Move docs/src to the root of the project and eliminate docs as a
   formal subproject. This simplifies the build in a number of ways,
   including removing the need for distinguishing between 'subprojects'
   and 'javaprojects' as well as allowing users to build both 'api' and
   'reference' docs without qualifying with a ':docs' prefix.

   Also rename the src/info directory to src/dist to better reflect that
   these files are packaged with the distribution. For example, the
   readme.txt there is really the distribution readme, distinct from the
   README.md at the root of the project which is for building from source,
   etc.
This commit is contained in:
Chris Beams
2011-11-08 14:38:53 -08:00
committed by Mark Fisher
parent ae0ecff886
commit f30da932e8
225 changed files with 811 additions and 7510 deletions
Download Patch File Download Diff File Expand all files Collapse all files

379
src/reference/docbook/ftp.xml Normal file
View File

@@ -0,0 +1,379 @@
<?xml version="1.0" encoding="UTF-8"?>
<chapter xmlns="http://docbook.org/ns/docbook" version="5.0" xml:id="ftp"
xmlns:xlink="http://www.w3.org/1999/xlink">
<title>FTP/FTPS Adapters</title>
<para>
Spring Integration provides support for file transfer operations via FTP and FTPS.
</para>
<section id="ftp-intro">
<title>Introduction</title>
<para>
The File Transfer Protocol (FTP) is a simple network protocol which allows you to transfer files between two computers on the Internet.
</para>
<para>
There are two actors when it comes to FTP communication: <emphasis>client</emphasis> and <emphasis>server</emphasis>.
To transfer files with FTP/FTPS, you use a <emphasis>client</emphasis> which initiates a connection to a remote computer
that is running an FTP <emphasis>server</emphasis>. After the connection is established, the <emphasis>client</emphasis> can choose
to send and/or receive copies of files.
</para>
<para>
Spring Integration supports sending and receiving files over FTP/FTPS by providing three <emphasis>client</emphasis>
side endpoints: <emphasis>Inbound Channel Adapter</emphasis>, <emphasis>Outbound Channel Adapter</emphasis>, and
<emphasis>Outbound Gateway</emphasis>. It also provides
convenient namespace-based configuration options for defining these <emphasis>client</emphasis> components.
</para>
<para>
To use the <emphasis>FTP</emphasis> namespace, add the following to the header of your XML file:
<programlisting language="xml"><![CDATA[xmlns:int-ftp="http://www.springframework.org/schema/integration/ftp"
xsi:schemaLocation="http://www.springframework.org/schema/integration/ftp
http://www.springframework.org/schema/integration/ftp/spring-integration-ftp.xsd"
]]></programlisting>
</para>
</section>
<section id="ftp-session-factory">
<title>FTP Session Factory</title>
<para>
Before configuring FTP adapters you must configure an <emphasis>FTP Session Factory</emphasis>. You can configure
the <emphasis>FTP Session Factory</emphasis> with a regular bean definition where the implementation class is <classname>org.springframework.integration.ftp.session.DefaultFtpSessionFactory</classname>:
Below is a basic configuration:
<programlisting language="xml"><![CDATA[<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="localhost"/>
<property name="port" value="22"/>
<property name="username" value="kermit"/>
<property name="password" value="frog"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="100000"/>
</bean>]]></programlisting>
</para>
<para>
For FTPS connections all you need to do is use <classname>org.springframework.integration.ftp.session.DefaultFtpsSessionFactory</classname> instead.
Below is the complete configuration sample:
<programlisting language="xml"><![CDATA[<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.client.DefaultFtpsClientFactory">
<property name="host" value="localhost"/>
<property name="port" value="22"/>
<property name="username" value="oleg"/>
<property name="password" value="password"/>
<property name="clientMode" value="1"/>
<property name="fileType" value="2"/>
<property name="useClientMode" value="true"/>
<property name="cipherSuites" value="a,b.c"/>
<property name="keyManager" ref="keyManager"/>
<property name="protocol" value="SSL"/>
<property name="trustManager" ref="trustManager"/>
<property name="prot" value="P"/>
<property name="needClientAuth" value="true"/>
<property name="authValue" value="oleg"/>
<property name="sessionCreation" value="true"/>
<property name="protocols" value="SSL, TLS"/>
<property name="implicit" value="true"/>
</bean>]]></programlisting>
</para>
<para>
Every time an adapter requests a session object from its <interfacename>SessionFactory</interfacename> the session is
returned from a session pool maintained by a caching wrapper around the factory. A Session in the session pool might go stale
(if it has been disconnected by the server due to inactivity) so the <interfacename>SessionFactory</interfacename>
will perform validation to make sure that it never returns a stale session to the adapter. If a stale session was encountered,
it will be removed from the pool, and a new one will be created.
<note>
If you experience connectivity problems and would like to trace Session creation as well as see which Sessions are
polled you may enable it by setting the logger to TRACE level (e.g., log4j.category.org.springframework.integration.file=TRACE)
</note>
</para>
<para>
Now all you need to do is inject these session factories into your adapters. Obviously the protocol (FTP or FTPS) that an adapter will
use depends on the type of session factory that has been injected into the adapter.
</para>
<para>
<note>
A more practical way to provide values for <emphasis>FTP/FTPS Session Factories</emphasis> is by using Spring's property
placeholder support (See: http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/beans.html#beans-factory-placeholderconfigurer).
</note>
</para>
</section>
<section id="ftp-inbound">
<title>FTP Inbound Channel Adapter</title>
<para>
The <emphasis>FTP Inbound Channel Adapter</emphasis> is a special listener that will connect to the FTP server and will listen
for the remote directory events (e.g., new file created) at which point it will initiate a file transfer.
<programlisting language="xml"><![CDATA[<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpSessionFactory"
charset="UTF-8"
auto-create-local-directory="true"
delete-remote-files="true"
filename-pattern="*.txt"
remote-directory="some/remote/path"
remote-file-separator="/"
local-filename-generator-expression="#this.toUpperCase() + '.a'"
local-directory=".">
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>]]></programlisting>
As you can see from the configuration above you can configure an <emphasis>FTP Inbound Channel Adapter</emphasis> via the <code>inbound-channel-adapter</code>
element while also providing values for various attributes such as <code>local-directory</code>, <code>filename-pattern</code>
(which is based on simple pattern matching, not regular expressions), and of course the reference to a <code>session-factory</code>.
</para>
<para>
By default the transferred file will carry the same name as the original file. If you want to override this behavior you
can set the <code>local-filename-generator-expression</code> attribute which allows you to provide a SpEL Expression to generate
the name of the local file. Unlike outbound gateways and adapters where the root object of the SpEL Evaluation Context
is a <classname>Message</classname>, this inbound adapter does not yet have the Message at the time of evaluation since
that's what it ultimately generates with the transferred file as its payload. So, the root object of the SpEL Evaluation Context
is the original name of the remote file (String).
</para>
<para>
Some times file filtering based on the simple pattern specified via <code>filename-pattern</code> attribute might not be
sufficient. If this is the case, you can use the <code>filename-regex</code> attribute to specify a Regular Expression
(e.g. <code>filename-regex=".*\.test$"</code>). And of course if you need complete control you can use <code>filter</code>
attribute and provide a reference to any custom implementation of the
<classname>org.springframework.integration.file.filters.FileListFilter</classname>, a strategy interface for filtering a
list of files.
</para>
<note>
As of Spring Integration 2.0.2, we have added a 'remote-file-separator' attribute. That allows you to configure a
file separator character to use if the default '/' is not applicable for your particular environment.
</note>
<para>
Please refer to the schema for more details on these attributes.
</para>
<para>
It is also important to understand that the <emphasis>FTP Inbound Channel Adapter</emphasis> is a <emphasis>Polling Consumer</emphasis> and
therefore you must configure a poller (either via a global default or a local sub-element).
Once a file has been transferred, a Message with a <classname>java.io.File</classname> as its payload will be generated and sent to the channel
identified by the <code>channel</code> attribute.
</para>
<para>
<emphasis>More on File Filtering and Large Files</emphasis>
</para>
<para>
Some times the file that just appeared in the monitored (remote) directory is not complete. Typically such a file
will be written with temporary extension (e.g., foo.txt.writing) and then renamed after the writing process finished.
As a user in most cases you are only interested in files that are complete and would like to filter only files that are complete.
To handle these scenarios you can use the filtering support provided by the <code>filename-pattern</code>, <code>filename-regex</code>
and <code>filter</code> attributes. Here is an example that uses a custom Filter implementation.
<programlisting language="xml"><![CDATA[<int-ftp:inbound-channel-adapter
channel="ftpChannel"
session-factory="ftpSessionFactory"
filter="customFilter"
local-directory="file:/my_transfers">
remote-directory="some/remote/path"
<int:poller fixed-rate="1000"/>
</int-ftp:inbound-channel-adapter>
<bean id="customFilter" class="org.example.CustomFilter"/>
]]></programlisting>
</para>
<para>
<emphasis>Poller configuration notes for the inbound FTP adapter</emphasis>
</para>
<para>
The job of the inbound FTP adapter consists of two tasks:
<emphasis>1) Communicate with a remote server in order to transfer files from a remote directory to a local directory.</emphasis>
<emphasis>2) For each transferred file, generate a Message with that file as a payload and send it to the channel identified by the 'channel' attribute.</emphasis>
That is why they are called 'channel-adapters' rather than just 'adapters'. The main job of such an adapter is to generate a
Message to be sent to a Message Channel. Essentially, the second task mentioned above takes precedence in such a way that
*IF* your local directory already has one or more files it will first generate Messages from those, and *ONLY*
when all local files have been processed, will it initiate the remote communication to retrieve more files.
</para>
<para>
Also, when configuring a trigger on the poller you should pay close attention to the <code>max-messages-per-poll</code>
attribute. Its default value is 1 for all <classname>SourcePollingChannelAdapter</classname> instances (including FTP).
This means that as soon as one file is processed, it will wait for the next execution time as determined by your
trigger configuration. If you happened to have one or more files sitting in the <code>local-directory</code>, it would process
those files before it would initiate communication with the remote FTP server. And, if the <code>max-messages-per-poll</code>
were set to 1 (default), then it would be processing only one file at a time with intervals as defined by your trigger,
essentially working as <emphasis>one-poll = one-file</emphasis>.
</para>
<para>
For typical file-transfer use cases, you most likely want the opposite behavior: to process all the files you can for each
poll and only then wait for the next poll. If that is the case, set <code>max-messages-per-poll</code> to -1. Then, on
each poll, the adapter will attempt to generate as many Messages as it possibly can. In other words, it will process
everything in the local directory, and then it will connect to the remote directory to transfer everything that is available
there to be processed locally. Only then is the poll operation considered complete, and the poller will wait for the next execution time.
</para>
<para>
You can alternatively set the 'max-messages-per-poll' value to a positive value indicating the upward limit of Messages to be created
from files with each poll. For example, a value of 10 means that on each poll it will attempt to process no more than 10 files.
</para>
</section>
<section id="ftp-outbound">
<title>FTP Outbound Channel Adapter</title>
<para>
The <emphasis>FTP Outbound Channel Adapter</emphasis> relies upon a <classname>MessageHandler</classname> implementation that will connect to the
FTP server and initiate an FTP transfer for every file it receives in the payload of incoming Messages. It also supports several
representations of the <emphasis>File</emphasis> so you are not limited only to java.io.File typed payloads.
The <emphasis>FTP Outbound Channel Adapter</emphasis>
supports the following payloads: 1) <classname>java.io.File</classname> - the actual file object;
2) <classname>byte[]</classname> - a byte array that represents the file contents; and 3) <classname>java.lang.String</classname> -
text that represents the file contents.
<programlisting language="xml"><![CDATA[<int-ftp:outbound-channel-adapter id="ftpOutbound"
channel="ftpChannel"
session-factory="ftpSessionFactory"
charset="UTF-8"
remote-file-separator="/"
filename-generator="fileNameGenerator"/>]]></programlisting>
As you can see from the configuration above you can configure an <emphasis>FTP Outbound Channel Adapter</emphasis> via the
<code>outbound-channel-adapter</code> element while also providing values for various attributes such as <code>filename-generator</code>
(an implementation of the <classname>org.springframework.integration.file.FileNameGenerator</classname> strategy interface),
a reference to a <code>client-factory</code>, as well as other attributes. Please refer to the schema for more details on
the available attributes.
<note>
By default Spring Integration will use <classname>org.springframework.integration.file.DefaultFileNameGenerator</classname> if none is specified.
<classname>DefaultFileNameGenerator</classname> will determine the file name based on the value of the <code>file_name</code> header (if it exists)
in the MessageHeaders, or if the payload of the Message is already a <classname>java.io.File</classname>, then it will use the original name of that file.
</note>
</para>
<para>
<important>
Defining certain values (e.g., remote-directory) might be platform/ftp server dependent. For example as it
was reported on this forum http://forum.springsource.org/showthread.php?p=333478&amp;posted=1#post333478 on some
platforms you must add slash to the end of the directory definition (e.g., remote-directory="/foo/bar/"
instead of remote-directory="/foo/bar")
</important>
</para>
</section>
<section id="ftp-outbound-gateway">
<title>FTP Outbound Gateway</title>
<para>
The <emphasis>FTP Outbound Gateway</emphasis> provides a limited set of commands to interact with a remote FTP/FTPS server.
<para>
Commands supported are:
<itemizedlist>
<listitem>ls (list files)</listitem>
<listitem>get (retrieve file(s))</listitem>
<listitem>rm (remove file(s))</listitem>
</itemizedlist>
</para>
<para>
ls supports the following options:
<itemizedlist>
<listitem>-1 - just retrieve a list of filenames, default is to retrieve a
list of <classname>FileInfo</classname> objects.</listitem>
<listitem>-a - include all files (including those starting with '.')</listitem>
<listitem>-f - do not sort the list</listitem>
<listitem>-dirs - include directories (excluded by default)</listitem>
<listitem>-links - include symbolic links (excluded by default)</listitem>
</itemizedlist>
</para>
<para>
In addition, filename filtering is provided, in the same manner as the
<classname>inbound-channel-adapter</classname>.
</para>
<para>
The message payload resulting from an <emphasis>ls</emphasis> operation is a list of file names,
or a list of <classname>FileInfo</classname> objects. These objects provide
information such as modified time, permissions etc.
</para>
<para>
The remote directory that the <emphasis>ls</emphasis> command acted on is provided
in the <classname>file_remoteDirectory</classname> header.
</para>
<para>
<emphasis>get</emphasis> supports the following option:
<itemizedlist>
<listitem>-P - preserve the timestamp of the remote file</listitem>
</itemizedlist>
</para>
<para>
The message payload resulting from a <emphasis>get</emphasis> operation is a
<classname>File</classname> object representing the retrieved file.
</para>
<para>
The remote directory is provided in the <classname>file_remoteDirectory</classname> header, and the filename is
provided in the <classname>file_remoteFile</classname> header.
</para>
<para></para>
<para>
The <emphasis>rm</emphasis> command has no options.
</para>
<para>
<note>
Filters are not supported with the <emphasis>rm</emphasis> command.
</note>
</para>
<para>
The message payload resulting from an <emphasis>rm</emphasis> operation is Boolean.TRUE if the
remove was successful, Boolean.FALSE otherwise.
The remote directory is provided in the <classname>file_remoteDirectory</classname> header, and the filename is
provided in the <classname>file_remoteFile</classname> header.
</para>
<para></para>
<para>
In each case, the PATH that these commands act on is provided by the 'expression'
property of the gateway.
</para>
</para>
<para>
Here is an example of a gateway configured for an ls command...
<programlisting language="xml"><![CDATA[<int-ftp:outbound-gateway id="gateway1"
session-factory="ftpSessionFactory"
request-channel="inbound1"
command="ls"
command-options="-1"
expression="payload"
reply-channel="toSplitter"/>
]]></programlisting>
</para>
<para>
The payload of the message sent to the toSplitter channel is a list of String objects
containing the filename of each file. If the <classname>command-options</classname> was
omitted, it would be a list of <classname>FileInfo</classname> objects. Options are
provided space-delimited, e.g. <classname>command-options="-1 -dirs -links"</classname>.
</para>
</section>
<section id="ftp-session-caching">
<title>FTP Session Caching</title>
<para>
As of version 2.1 we've exposed more flexibility with regard to session management for remote file adapters (e.g., FTP, SFTP etc).
In previous versions the sessions were cached automatically by default. We did expose a <code>cache-sessions</code> attribute for
disabling the auto caching, but that solution did not provide a way to configure other session caching attributes. For example, one
of the requested features was to support a limit on the number of sessions created since a remote server may impose a limit on the
number of client connections. To support that requirement and other configuration options, we decided to promote explicit definition
of the <classname>CachingSessionFactory</classname> instance. That provides the <code>sessionCacheSize</code> and <code>sessionWaitTimeout</code>
properties. As its name suggests, the <code>sessionCacheSize</code> property controls how many active sessions this adapter will
maintain in its cache (the DEFAULT is unbounded). If the <code>sessionCacheSize</code> threshold has been reached, any attempt to
acquire another session will block until either one of the cached sessions becomes available or until the wait time for a Session
expires (the DEFAULT wait time is Integer.MAX_VALUE). The <code>sessionWaitTimeout</code> property enables configuration of that value.
</para>
<para>
If you want your Sessions to be cached, simply configure your default Session Factory as described above and then
wrap it in an instance of <classname>CachingSessionFactory</classname> where you may provide those additional properties.
<programlisting language="xml"><![CDATA[<bean id="ftpSessionFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="localhost"/>
</bean>
<bean id="cachingSessionFactory" class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<constructor-arg ref="ftpSessionFactory"/>
<constructor-arg value="10"/>
<property name="sessionWaitTimeout" value="1000"/>
</bean>]]></programlisting>
In the above example you see a <classname>CachingSessionFactory</classname> created with the
<code>sessionCacheSize</code> set to 10 and the <code>sessionWaitTimeout</code> set to 1 second (its value is in millliseconds).
</para>
</section>
</chapter>