INT-4491: (S)FTP inbound rotate dirs/servers
JIRA: https://jira.spring.io/browse/INT-4491 Add Rotating Server/Directory Polling Advice. **cherry-pick to 5.0.x** * Polishing - PR Comments. * Polishing * Polishing; revert `KeyDirectory`; WARN about `TaskExecutor` and `MessageSoureMutator`(s). * More polishing - PR comments * Apply stashed changes. * Fix WARN log - the `SyncTaskExecutor` is wrapped. # Conflicts: # spring-integration-core/src/main/java/org/springframework/integration/endpoint/SourcePollingChannelAdapter.java # src/reference/asciidoc/whats-new.adoc
This commit is contained in:
Gary Russell
committed by
Artem Bilan
Artem Bilan
parent
170cc37270
commit
9f937f54cd
@@ -224,6 +224,8 @@ Convenience methods have been added so this can easily be done from a message fl
|
||||
IMPORTANT: When using session caching (see <<ftp-session-caching>>), each of the delegates should be cached; you
|
||||
cannot cache the `DelegatingSessionFactory` itself.
|
||||
|
||||
Starting with _version 5.0.7_, the `DelegatingSessionFactory` can be used in conjuction with a `RotatingServerAdvice` to poll multiple servers; see <<ftp-rotating-server-advice>>.
|
||||
|
||||
[[ftp-inbound]]
|
||||
=== FTP Inbound Channel Adapter
|
||||
|
||||
@@ -620,6 +622,91 @@ public class FtpJavaApplication {
|
||||
|
||||
Notice that, in this example, the message handler downstream of the transformer has an advice that removes the remote file after processing.
|
||||
|
||||
[[ftp-rotating-server-advice]]
|
||||
=== Inbound Channel Adapters: Polling Multiple Servers and Directories
|
||||
|
||||
Starting with _version 5.0.7_, the `RotatingServerAdvice` is available; when configured as a poller advice, the inbound adapters can poll multiple servers and directories.
|
||||
Configure the advice and add it to the poller's advice chain as normal.
|
||||
A `DelegatingSessionFactory` is used to select the server see <<ftp-dsf>> for more information.
|
||||
The advice configuration consists of a list of `RotatingServerAdvice.KeyDirectory` objects.
|
||||
|
||||
.Example
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public RotatingServerAdvice advice() {
|
||||
List<KeyDirectory> keyDirectories = new ArrayList<>();
|
||||
keyDirectories.add(new KeyDirectory("one", "foo"));
|
||||
keyDirectories.add(new KeyDirectory("one", "bar"));
|
||||
keyDirectories.add(new KeyDirectory("two", "baz"));
|
||||
keyDirectories.add(new KeyDirectory("two", "qux"));
|
||||
keyDirectories.add(new KeyDirectory("three", "fiz"));
|
||||
keyDirectories.add(new KeyDirectory("three", "buz"));
|
||||
return new RotatingServerAdvice(delegatingSf(), keyDirectories);
|
||||
}
|
||||
----
|
||||
|
||||
This advice will poll directory `foo` on server `one` until no new files exist then move to directory `bar` and then directory `baz` on server `two`, etc.
|
||||
|
||||
This default behavior can be modified with the `fair` constructor arg:
|
||||
|
||||
.fair
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public RotatingServerAdvice advice() {
|
||||
...
|
||||
return new RotatingServerAdvice(delegatingSf(), keyDirectories, true);
|
||||
}
|
||||
----
|
||||
|
||||
In this case, the advice will move to the next server/directory regardless of whether the previous poll returned a file.
|
||||
|
||||
Alternatively, you can provide your own `RotatingServerAdvice.RotationPolicy` to reconfigure the message source as needed:
|
||||
|
||||
.policy
|
||||
[source, java]
|
||||
----
|
||||
public interface RotationPolicy {
|
||||
|
||||
void beforeReceive(MessageSource<?> source);
|
||||
|
||||
void afterReceive(boolean messageReceived, MessageSource<?> source);
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
and
|
||||
|
||||
.custom
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public RotatingServerAdvice advice() {
|
||||
return new RotatingServerAdvice(myRotationPolicy());
|
||||
}
|
||||
----
|
||||
|
||||
The `local-filename-generator-expression` attribute (`localFilenameGeneratorExpression` on the synchronizer) can now contain the `#remoteDirectory` variable.
|
||||
This allows files retrieved from different directories to be downloaded to similar directories locally:
|
||||
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public IntegrationFlow flow() {
|
||||
return IntegrationFlows.from(Ftp.inboundAdapter(sf())
|
||||
.filter(new FtpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "rotate"))
|
||||
.localDirectory(new File(tmpDir))
|
||||
.localFilenameExpression("#remoteDirectory + T(java.io.File).separator + #root")
|
||||
.remoteDirectory("."),
|
||||
e -> e.poller(Pollers.fixedDelay(1).advice(advice())))
|
||||
.channel(MessageChannels.queue("files"))
|
||||
.get();
|
||||
}
|
||||
----
|
||||
|
||||
IMPORTANT: Do not configure a `TaskExecutor` on the poller when using this advice; see <<conditional-pollers>> for more information.
|
||||
|
||||
[[ftp-max-fetch]]
|
||||
=== Inbound Channel Adapters: Controlling Remote File Fetching
|
||||
|
||||
|
||||
@@ -162,7 +162,15 @@ It enables you to examine and or reconfigure the source at this time. Returning
|
||||
Message<?> afterReceive(Message<?> result, MessageSource<?> source)
|
||||
|
||||
This method is called after the `receive()` method; again, you can reconfigure the source, or take any action perhaps depending on the result (which can be `null` if there was no message created by the source).
|
||||
You can even return a different message!
|
||||
You can even return a different message
|
||||
|
||||
.Thread safety
|
||||
[IMPORTANT]
|
||||
====
|
||||
You should not configure the poller with a `TaskExecutor` if an advice mutates the `MessageSource`.
|
||||
If an advice mutates the source, such mutations are not thread safe and could cause unexpected results, especially with high frequency pollers.
|
||||
Consider using a downstream `ExecutorChannel` instead of adding an executor to the poller if you need to process poll results concurrently.
|
||||
====
|
||||
|
||||
.Advice Chain Ordering
|
||||
[IMPORTANT]
|
||||
|
||||
@@ -239,6 +239,8 @@ Convenience methods have been added so this can easily be done from a message fl
|
||||
IMPORTANT: When using session caching (see <<sftp-session-caching>>), each of the delegates should be cached; you
|
||||
cannot cache the `DelegatingSessionFactory` itself.
|
||||
|
||||
Starting with _version 5.0.7_, the `DelegatingSessionFactory` can be used in conjuction with a `RotatingServerAdvice` to poll multiple servers; see <<sftp-rotating-server-advice>>.
|
||||
|
||||
[[sftp-session-caching]]
|
||||
=== SFTP Session Caching
|
||||
|
||||
@@ -665,6 +667,91 @@ public class SftpJavaApplication {
|
||||
|
||||
Notice that, in this example, the message handler downstream of the transformer has an advice that removes the remote file after processing.
|
||||
|
||||
[[sftp-rotating-server-advice]]
|
||||
=== Inbound Channel Adapters: Polling Multiple Servers and Directories
|
||||
|
||||
Starting with _version 5.0.7_, the `RotatingServerAdvice` is available; when configured as a poller advice, the inbound adapters can poll multiple servers and directories.
|
||||
Configure the advice and add it to the poller's advice chain as normal.
|
||||
A `DelegatingSessionFactory` is used to select the server see <<ftp-dsf>> for more information.
|
||||
The advice configuration consists of a list of `RotatingServerAdvice.KeyDirectory` objects.
|
||||
|
||||
.Example
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public RotatingServerAdvice advice() {
|
||||
List<KeyDirectory> keyDirectories = new ArrayList<>();
|
||||
keyDirectories.add(new KeyDirectory("one", "foo"));
|
||||
keyDirectories.add(new KeyDirectory("one", "bar"));
|
||||
keyDirectories.add(new KeyDirectory("two", "baz"));
|
||||
keyDirectories.add(new KeyDirectory("two", "qux"));
|
||||
keyDirectories.add(new KeyDirectory("three", "fiz"));
|
||||
keyDirectories.add(new KeyDirectory("three", "buz"));
|
||||
return new RotatingServerAdvice(delegatingSf(), keyDirectories);
|
||||
}
|
||||
----
|
||||
|
||||
This advice will poll directory `foo` on server `one` until no new files exist then move to directory `bar` and then directory `baz` on server `two`, etc.
|
||||
|
||||
This default behavior can be modified with the `fair` constructor arg:
|
||||
|
||||
.fair
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public RotatingServerAdvice advice() {
|
||||
...
|
||||
return new RotatingServerAdvice(delegatingSf(), keyDirectories, true);
|
||||
}
|
||||
----
|
||||
|
||||
In this case, the advice will move to the next server/directory regardless of whether the previous poll returned a file.
|
||||
|
||||
Alternatively, you can provide your own `RotatingServerAdvice.RotationPolicy` to reconfigure the message source as needed:
|
||||
|
||||
.policy
|
||||
[source, java]
|
||||
----
|
||||
public interface RotationPolicy {
|
||||
|
||||
void beforeReceive(MessageSource<?> source);
|
||||
|
||||
void afterReceive(boolean messageReceived, MessageSource<?> source);
|
||||
|
||||
}
|
||||
----
|
||||
|
||||
and
|
||||
|
||||
.custom
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public RotatingServerAdvice advice() {
|
||||
return new RotatingServerAdvice(myRotationPolicy());
|
||||
}
|
||||
----
|
||||
|
||||
The `local-filename-generator-expression` attribute (`localFilenameGeneratorExpression` on the synchronizer) can now contain the `#remoteDirectory` variable.
|
||||
This allows files retrieved from different directories to be downloaded to similar directories locally:
|
||||
|
||||
[source, java]
|
||||
----
|
||||
@Bean
|
||||
public IntegrationFlow flow() {
|
||||
return IntegrationFlows.from(Ftp.inboundAdapter(sf())
|
||||
.filter(new FtpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "rotate"))
|
||||
.localDirectory(new File(tmpDir))
|
||||
.localFilenameExpression("#remoteDirectory + T(java.io.File).separator + #root")
|
||||
.remoteDirectory("."),
|
||||
e -> e.poller(Pollers.fixedDelay(1).advice(advice())))
|
||||
.channel(MessageChannels.queue("files"))
|
||||
.get();
|
||||
}
|
||||
----
|
||||
|
||||
IMPORTANT: Do not configure a `TaskExecutor` on the poller when using this advice; see <<conditional-pollers>> for more information.
|
||||
|
||||
[[sftp-max-fetch]]
|
||||
=== Inbound Channel Adapters: Controlling Remote File Fetching
|
||||
|
||||
|
||||
@@ -202,6 +202,10 @@ New filters for detecting incomplete remote files are now provided.
|
||||
|
||||
The `FtpOutboundGateway` and `SftpOutboundGateway` now support an option to remove the remote file after a successful transfer using the `GET` or `MGET` commands.
|
||||
|
||||
A `RotatingServerAdvice` is now available to poll multiple servers and/or directories with the inbound channel adapters.
|
||||
|
||||
Also inbound adapter `localFilenameExpression` s can contain the variable `#remoteDirectory` which contains the remote directory being polled.
|
||||
|
||||
See <<ftp>> and <<sftp>> for more information.
|
||||
|
||||
==== Integration Properties
|
||||
|
||||
Reference in New Issue
Block a user