INT-4015: Streaming Remote File Inbound Adapter
JIRA: https://jira.spring.io/browse/INT-4015 https://jira.spring.io/browse/INT-3854 Initial commit. Reworked to emit an input stream and use the file splitter. Add StreamTransformer. Add CLOSABLE_RESOURCE header so we can close the session automatically. Implement INT-3854, FTP, SFTP (S)FTP Namespace Changes Docs - also fixes a PDF overflow Polishing - PR Comments checkstyle fixes Polishing - Add Namespace for StreamParser Polishing - PR Comments
This commit is contained in:
Gary Russell
committed by
Artem Bilan
Artem Bilan
parent
6b6a38f8cb
commit
287d924fc0
@@ -272,7 +272,8 @@ This means that (when this event is enabled), filters such as the `AcceptOnceFil
|
||||
meaning that, if a file with the same name appears, it will pass the filter and be sent as a message.
|
||||
|
||||
For this purpose the `watch-events`
|
||||
(`FileReadingMessageSource.setWatchEvents(FileReadingMessageSource.WatchEventType... watchEvents)`) has been introduced.
|
||||
(`FileReadingMessageSource.setWatchEvents(WatchEventType... watchEvents)`) has been introduced
|
||||
(`WatchEventType` is a public inner enum in `FileReadingMessageSource`).
|
||||
With such an option we can implement some scenarios, when we would like to do one downstream flow logic for new files,
|
||||
and other for modified.
|
||||
We can achieve that with different `<int-file:inbound-channel-adapter>` definitions, but for the same directory:
|
||||
@@ -858,3 +859,10 @@ public MessageHandler fileSplitter() {
|
||||
return splitter;
|
||||
}
|
||||
----
|
||||
|
||||
The `FileSplitter` will also split any text-based `InputStream` into lines.
|
||||
When used in conjunction with an FTP or SFTP streaming inbound channel adapter, or an FTP or SFTP outbound gateway
|
||||
using the `stream` option to retrieve a file, starting with _version 4.3_, the splitter will automatically close
|
||||
the session supporting the stream, when the file is completely consumed.
|
||||
See <<ftp-streaming>> and <<sftp-streaming>> as well as <<ftp-outbound-gateway>> and <<sftp-outbound-gateway>> for more
|
||||
information about these facilities.
|
||||
|
||||
@@ -163,7 +163,6 @@ The _FTP Inbound Channel Adapter_ is a special listener that will connect to the
|
||||
<int-ftp:inbound-channel-adapter id="ftpInbound"
|
||||
channel="ftpChannel"
|
||||
session-factory="ftpSessionFactory"
|
||||
charset="UTF-8"
|
||||
auto-create-local-directory="true"
|
||||
delete-remote-files="true"
|
||||
filename-pattern="*.txt"
|
||||
@@ -328,6 +327,41 @@ This will work for any `ResettableFileListFilter`.
|
||||
class="org.springframework.integration.transaction.PseudoTransactionManager" />
|
||||
----
|
||||
|
||||
[[ftp-streaming]]
|
||||
=== FTP Streaming Inbound Channel Adapter
|
||||
|
||||
The streaming inbound channel adapter was introduced in _version 4.3_.
|
||||
This adapter produces message with payloads of type `InputStream`, allowing files to be fetched without writing to the
|
||||
local file system.
|
||||
Since the session remains open, the consuming application is responsible for closing the session when the file has been
|
||||
consumed.
|
||||
The session is provided in the `closeableResource` header (`IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE`).
|
||||
Standard framework components, such as the `FileSplitter` and `StreamTransformer` will automatically close the session.
|
||||
See <<file-splitter>> and <<stream-transformer>> for more information about these components.
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<int-ftp:inbound-streaming-channel-adapter id="ftpInbound"
|
||||
channel="ftpChannel"
|
||||
session-factory="sessionFactory"
|
||||
filename-pattern="*.txt"
|
||||
filename-regex=".*\.txt"
|
||||
filter="filter"
|
||||
remote-file-separator="/"
|
||||
comparator="comparator"
|
||||
remote-directory-expression="'foo/bar'">
|
||||
<int:poller fixed-rate="1000" />
|
||||
</int-ftp:inbound-streaming-channel-adapter>
|
||||
----
|
||||
|
||||
Only one of `filename-pattern`, `filename-regex` or `filter` is allowed.
|
||||
|
||||
IMPORTANT: Unlike the non-streaming inbound channel adapter, this adapter does not prevent duplicates by default.
|
||||
If you do not delete the remote file (e.g. using an outbound gateway with an rm command) and you wish to prevent the
|
||||
file being processed again, you can configure an `FtpPersistentFileListFilter` in the `filter` attribute.
|
||||
If you don't actually want to persist the state, an in-memory `SimpleMetadataStore` can be used with the filter.
|
||||
If you wish to use a filename pattern (or regex) as well, use a `CompositeFileListFilter`.
|
||||
|
||||
[[ftp-outbound]]
|
||||
=== FTP Outbound Channel Adapter
|
||||
|
||||
@@ -394,7 +428,7 @@ Commands supported are:
|
||||
|
||||
ls lists remote file(s) and supports the following options:
|
||||
|
||||
* -1 - just retrieve a list of filenames, default is to retrieve a list of `FileInfo` objects.
|
||||
* -1 - just retrieve a list of file names, default is to retrieve a list of `FileInfo` objects.
|
||||
* -a - include all files (including those starting with '.')
|
||||
* -f - do not sort the list
|
||||
* -dirs - include directories (excluded by default)
|
||||
@@ -433,10 +467,26 @@ The remote directory is provided in the `file_remoteDirectory` header, and the f
|
||||
The message payload resulting from a _get_ operation is a `File` object representing the retrieved file, or
|
||||
an `InputStream` when the `-stream` option is provided.
|
||||
This option allows retrieving the file as a stream.
|
||||
For text files, a common use case is to combine this operation with a <<file-splitter>>.
|
||||
For text files, a common use case is to combine this operation with a <<file-splitter,File Splitter>> or
|
||||
<<stream-transformer,Stream Transformer>>.
|
||||
When consuming remote files as streams, the user is responsible for closing the `Session` after the stream is
|
||||
consumed.
|
||||
For convenience, the `Session` is provided in the `file_remoteSession` header.
|
||||
For convenience, the `Session` is provided in the `closeableResource` header, a convenience method is provided on the
|
||||
`IntegrationMessageHeaderAccessor`:
|
||||
|
||||
[source, java]
|
||||
----
|
||||
Closeable closeable = new IntegrationMessageHeaderAccessor(message).getCloseableResource();
|
||||
if (closeable != null) {
|
||||
closeable.close();
|
||||
}
|
||||
----
|
||||
|
||||
Note: In previous releases the session was in the `file_remoteSession` header, but this is deprecated - use
|
||||
`closableResource` instead.
|
||||
|
||||
Framework components such as the <<file-splitter,File Splitter>> and <<stream-transformer,Stream Transformer>> will
|
||||
automatically close the session after the data is transferred.
|
||||
|
||||
The following shows an example of consuming a file as a stream:
|
||||
|
||||
@@ -450,19 +500,17 @@ The following shows an example of consuming a file as a stream:
|
||||
remote-directory="ftpTarget"
|
||||
reply-channel="stream" />
|
||||
|
||||
<int:chain input-channel="stream">
|
||||
<int-file:splitter markers="true" />
|
||||
<int:payload-type-router resolution-required="false" default-output-channel="output">
|
||||
<int:mapping type="org.springframework.integration.file.splitter.FileSplitter$FileMarker"
|
||||
channel="markers" />
|
||||
</int:payload-type-router>
|
||||
</int:chain>
|
||||
|
||||
<int:service-activator input-channel="markers"
|
||||
expression="payload.mark.toString().equals('END') ? headers['file_remoteSession'].close() : null"/>
|
||||
<int-file:splitter input-channel="stream" output-channel="lines" />
|
||||
----
|
||||
|
||||
The file lines are sent to the channel `output`.
|
||||
Note: if you consume the input stream in a custom component, you *must* close the `Session`.
|
||||
You can either do that in your custom code, or route a copy of the message to a `service-activator` and use SpEL:
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<int:service-activator input-channel="closeSession"
|
||||
expression="headers['closeableResource'].close()" />
|
||||
----
|
||||
|
||||
*mget*
|
||||
|
||||
@@ -475,7 +523,7 @@ _mget_ retrieves multiple remote files based on a pattern and supports the follo
|
||||
|
||||
The message payload resulting from an _mget_ operation is a `List<File>` object - a List of File objects, each representing a retrieved file.
|
||||
|
||||
The remote directory is provided in the `file_remoteDirectory` header, and the pattern for the filenames is provided in the `file_remoteFile` header.
|
||||
The remote directory is provided in the `file_remoteDirectory` header, and the pattern for the file names is provided in the `file_remoteFile` header.
|
||||
|
||||
[NOTE]
|
||||
.Notes for when using recursion (`-R`)
|
||||
|
||||
@@ -434,6 +434,41 @@ This will work for any `ResettableFileListFilter`.
|
||||
class="org.springframework.integration.transaction.PseudoTransactionManager" />
|
||||
----
|
||||
|
||||
[[sftp-streaming]]
|
||||
=== SFTP Streaming Inbound Channel Adapter
|
||||
|
||||
The streaming inbound channel adapter was introduced in _version 4.3_.
|
||||
This adapter produces message with payloads of type `InputStream`, allowing files to be fetched without writing to the
|
||||
local file system.
|
||||
Since the session remains open, the consuming application is responsible for closing the session when the file has been
|
||||
consumed.
|
||||
The session is provided in the `closeableResource` header (`IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE`).
|
||||
Standard framework components, such as the `FileSplitter` and `StreamTransformer` will automatically close the session.
|
||||
See <<file-splitter>> and <<stream-transformer>> for more information about these components.
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<int-sftp:inbound-streaming-channel-adapter id="ftpInbound"
|
||||
channel="ftpChannel"
|
||||
session-factory="sessionFactory"
|
||||
filename-pattern="*.txt"
|
||||
filename-regex=".*\.txt"
|
||||
filter="filter"
|
||||
remote-file-separator="/"
|
||||
comparator="comparator"
|
||||
remote-directory-expression="'foo/bar'">
|
||||
<int:poller fixed-rate="1000" />
|
||||
</int-sftp:inbound-streaming-channel-adapter>
|
||||
----
|
||||
|
||||
Only one of `filename-pattern`, `filename-regex` or `filter` is allowed.
|
||||
|
||||
IMPORTANT: Unlike the non-streaming inbound channel adapter, this adapter does not prevent duplicates by default.
|
||||
If you do not delete the remote file (e.g. using an outbound gateway with an rm command) and you wish to prevent the
|
||||
file being processed again, you can configure an `SftpPersistentFileListFilter` in the `filter` attribute.
|
||||
If you don't actually want to persist the state, an in-memory `SimpleMetadataStore` can be used with the filter.
|
||||
If you wish to use a filename pattern (or regex) as well, use a `CompositeFileListFilter`.
|
||||
|
||||
[[sftp-outbound]]
|
||||
=== SFTP Outbound Channel Adapter
|
||||
|
||||
@@ -539,36 +574,50 @@ The remote directory is provided in the `file_remoteDirectory` header, and the f
|
||||
The message payload resulting from a _get_ operation is a `File` object representing the retrieved file, or
|
||||
an `InputStream` when the `-stream` option is provided.
|
||||
This option allows retrieving the file as a stream.
|
||||
For text files, a common use case is to combine this operation with a <<file-splitter>>.
|
||||
For text files, a common use case is to combine this operation with a <<file-splitter,File Splitter>> or
|
||||
<<stream-transformer,Stream Transformer>>.
|
||||
When consuming remote files as streams, the user is responsible for closing the `Session` after the stream is
|
||||
consumed.
|
||||
For convenience, the `Session` is provided in the `file_remoteSession` header.
|
||||
For convenience, the `Session` is provided in the `closeableResource` header, a convenience method is provided on the
|
||||
`IntegrationMessageHeaderAccessor`:
|
||||
|
||||
[source, java]
|
||||
----
|
||||
Closeable closeable = new IntegrationMessageHeaderAccessor(message).getCloseableResource();
|
||||
if (closeable != null) {
|
||||
closeable.close();
|
||||
}
|
||||
----
|
||||
|
||||
Note: In previous releases the session was in the `file_remoteSession` header, but this is deprecated - use
|
||||
`closableResource` instead.
|
||||
|
||||
Framework components such as the <<file-splitter,File Splitter>> and <<stream-transformer,Stream Transformer>> will
|
||||
automatically close the session after the data is transferred.
|
||||
|
||||
The following shows an example of consuming a file as a stream:
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<int-sftp:outbound-gateway session-factory="sftpSessionFactory"
|
||||
request-channel="inboundGetStream"
|
||||
command="get"
|
||||
command-options="-stream"
|
||||
expression="payload"
|
||||
remote-directory="ftpTarget"
|
||||
reply-channel="stream" />
|
||||
<int-sftp:outbound-gateway session-factory="ftpSessionFactory"
|
||||
request-channel="inboundGetStream"
|
||||
command="get"
|
||||
command-options="-stream"
|
||||
expression="payload"
|
||||
remote-directory="ftpTarget"
|
||||
reply-channel="stream" />
|
||||
|
||||
<int:chain input-channel="stream">
|
||||
<int-file:splitter markers="true" />
|
||||
<int:payload-type-router resolution-required="false" default-output-channel="output">
|
||||
<int:mapping type="org.springframework.integration.file.splitter.FileSplitter$FileMarker"
|
||||
channel="markers" />
|
||||
</int:payload-type-router>
|
||||
</int:chain>
|
||||
|
||||
<int:service-activator input-channel="markers"
|
||||
expression="payload.mark.toString().equals('END') ? headers['file_remoteSession'].close() : null"/>
|
||||
<int-file:splitter input-channel="stream" output-channel="lines" />
|
||||
----
|
||||
|
||||
The file lines are sent to the channel `output`.
|
||||
Note: if you consume the input stream in a custom component, you *must* close the `Session`.
|
||||
You can either do that in your custom code, or route a copy of the message to a `service-activator` and use SpEL:
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<int:service-activator input-channel="closeSession"
|
||||
expression="headers['closeableResource'].close()" />
|
||||
----
|
||||
|
||||
*mget*
|
||||
|
||||
|
||||
@@ -70,9 +70,12 @@ Just like Routers, Aggregators and other components, as of Spring Integration 2.
|
||||
In the above configuration we are achieving a simple transformation of the _payload_ with a simple SpEL expression and without writing a custom transformer.
|
||||
Our _payload_ (assuming String) will be upper-cased and concatenated with the current timestamp with some simple formatting.
|
||||
|
||||
_Common Transformers_
|
||||
===== Common Transformers
|
||||
|
||||
There are also a few Transformer implementations available out of the box.
|
||||
|
||||
====== Object-to-String Transformer
|
||||
|
||||
Because, it is fairly common to use the `toString()` representation of an Object, Spring Integration provides an `ObjectToStringTransformer` whose output is a Message with a String payload.
|
||||
That String is the result of invoking the toString() operation on the inbound Message's payload.
|
||||
[source,xml]
|
||||
@@ -112,7 +115,7 @@ These will use standard Java serialization by default, but you can provide an im
|
||||
<int:payload-deserializing-transformer input-channel="bytesIn" output-channel="objectsOut"/>
|
||||
----
|
||||
|
||||
_Object-to-Map Transformer_
|
||||
====== Object-to-Map and Map-to-Object Transformers
|
||||
|
||||
Spring Integration also provides _Object-to-Map_ and _Map-to-Object_ transformers which utilize the Spring Expression Language (SpEL) to serialize and de-serialize the object graphs.
|
||||
The object hierarchy is introspected to the most primitive types (String, int, etc.).
|
||||
@@ -210,7 +213,35 @@ NOTE: NOTE: 'ref' and 'type' attributes are mutually exclusive.
|
||||
You can only use one.
|
||||
Also, if using the 'ref' attribute, you must point to a 'prototype' scoped bean, otherwise a BeanCreationException will be thrown.
|
||||
|
||||
*JSON Transformers*
|
||||
[[stream-transformer]]
|
||||
====== Stream Transformer
|
||||
|
||||
The `StreamTransformer` transforms `InputStream` payloads to a `byte[]` or a `String` if a `charset` is provided.
|
||||
|
||||
[source, xml]
|
||||
----
|
||||
<int:stream-transformer input-channel="directInput" output-channel="output"/> <!-- byte[] -->
|
||||
|
||||
<int:stream-transformer id="withCharset" charset="UTF-8"
|
||||
input-channel="charsetChannel" output-channel="output"/> <!-- String -->
|
||||
----
|
||||
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
@Transformer(inputChannel = "stream", outputChannel = "data")
|
||||
public StreamTransformer streamToBytes() {
|
||||
return new StreamTransformer(); // transforms to byte[]
|
||||
}
|
||||
|
||||
@Bean
|
||||
@Transformer(inputChannel = "stream", outputChannel = "data")
|
||||
public StreamTransformer streamToString() {
|
||||
return new StreamTransformer("UTF-8"); // transforms to String
|
||||
}
|
||||
----
|
||||
|
||||
====== JSON Transformers
|
||||
|
||||
_Object to JSON_ and _JSON to Object_ transformers are provided.
|
||||
|
||||
|
||||
@@ -27,6 +27,17 @@ The `PersistentMessageGroup`, - lazy-load proxy, - implementation is provided fo
|
||||
which return this instance for the `getMessageGroup()` when their `lazyLoadMessageGroups` is `true` (defaults).
|
||||
See <<message-store>> for more information.
|
||||
|
||||
==== FTP/SFTP Streaming Inbound Channel Adapters
|
||||
|
||||
New inbound channel adapters are provided that return an `InputStream` for each file allowing you to retrieve remote
|
||||
files without writing them to the local file system
|
||||
See <<ftp-streaming>> and <<sftp-streaming>> for more information.
|
||||
|
||||
==== Stream Transformer
|
||||
|
||||
A new `StreamTransformer` is provided to transform an `InputStream` payload to either a `byte[]` or `String`.
|
||||
See <<stream-transformer>> for more information.
|
||||
|
||||
[[x4.3-general]]
|
||||
=== General Changes
|
||||
|
||||
@@ -122,6 +133,13 @@ See <<file-flushing>> for more information.
|
||||
The outbound channel adapter can now be configured to set the destination file's lastmodified timestamp.
|
||||
See <<file-timestamps>> for more information.
|
||||
|
||||
===== Splitter Changes
|
||||
|
||||
The `FileSplitter` will now automatically close an (S)FTP session when the file is completely read.
|
||||
This applies when the outbound gateway returns an `InputStream` or the new (S)FTP streaming channel adapters are being
|
||||
used.
|
||||
See <<file-splitter>> for more information.
|
||||
|
||||
==== AMQP Changes
|
||||
|
||||
===== Content Type Message Converter
|
||||
@@ -178,6 +196,8 @@ See <<sftp-outbound>> and <<sftp-outbound-gateway>> for more information.
|
||||
|
||||
==== FTP Changes
|
||||
|
||||
===== Session Changes
|
||||
|
||||
The `FtpSession` now supports `null` for the `list()` and `listNames()` method, since it is possible by the
|
||||
underlying FTP Client.
|
||||
With that the `FtpOutboundGateway` can now be configured without `remoteDirectory` expression.
|
||||
|
||||
Reference in New Issue
Block a user