Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-4086: (S)FTP: Add Local File Tree Support

Browse Source 
JIRA: https://jira.spring.io/browse/INT-4086

The remote files may be carried with sub-path, not just file name.
The same sub-dirs logic can be achieve with the `localFilenameGeneratorExpression`

* Add the logic into `AbstractInboundFileSynchronizer` to build sub-dirs for local files
* Switch on `WatchService` for the internal `FileReadingMessageSource` in the `AbstractInboundFileSynchronizingMessageSource` to react for all the changes in the `localDirectory`

Introduce `RecursiveDirectoryScanner` and use it for `AbstractInboundFileSynchronizingMessageSource` by default

* Add `maxDepth` and `fileVisitOptions` options to the `RecursiveDirectoryScanner`
* Polishing JavaDocs, tests and docs
This commit is contained in:
Artem Bilan
2016-08-09 17:52:17 -04:00
committed by Gary Russell
parent 6dbc721d73
commit 96ca9647a4
17 changed files with 289 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
src/reference/asciidoc/file.adoc
View File

@@ -162,6 +162,13 @@ It is used by the internal (`PriorityBlockingQueue`) to reorder its content acco
Therefore, to process files in a specific order, you should provide a comparator to the `FileReadingMessageSource`,
rather than ordering the list produced by a custom `DirectoryScanner`.
Starting with _version 5.0_, a new `RecursiveDirectoryScanner` is presented to perform file tree visiting.
The implementation is based on the `Files.walk(Path start, int maxDepth, FileVisitOption... options)` functionality.
The root directory (`DirectoryScanner.listFiles(File)` argument) is excluded from the result.
All other sub-directories includes/excludes are based on the target `FileListFilter` implementation.
For example the `SimplePatternFileListFilter` filters directories by default.
See `AbstractDirectoryAwareFileListFilter` and its implementations for more information.
[[file-namespace-support]]
==== Namespace Support

14
src/reference/asciidoc/ftp.adoc
View File

@@ -389,8 +389,7 @@ This will work for any `ResettableFileListFilter`.
remote-directory-expression="'/sftpSource'"
local-directory="file:myLocalDir"
auto-create-local-directory="true"
filename-pattern="*.txt"
local-filter="acceptOnceFilter">
filename-pattern="*.txt">
<int:poller fixed-rate="1000">
<int:transactional synchronization-factory="syncFactory" />
</int:poller>
@@ -400,13 +399,22 @@ This will work for any `ResettableFileListFilter`.
class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
<int:transaction-synchronization-factory id="syncFactory">
<int:after-rollback expression="@acceptOnceFilter.remove(payload)" />
<int:after-rollback expression="payload.delete()" />
</int:transaction-synchronization-factory>
<bean id="transactionManager"
class="org.springframework.integration.transaction.PseudoTransactionManager" />
----
Starting with _version 5.0_, the Inbound Channel Adapter can build sub-directories locally according the generated local file name.
That can be a remote sub-path as well.
To be able to read local directory recursively for modification according the hierarchy support, an internal `FileReadingMessageSource` now have been switched to use a new `RecursiveDirectoryScanner` based on the `Files.walk()` algorithm.
Also the `AbstractInboundFileSynchronizingMessageSource` can now be switched to the `WatchService` -based `DirectoryScanner` via `setUseWatchService()` option.
It is also configured for all the `WatchEventType` s to react for any modifications in local directory.
The reprocessing sample above is based on the build-in functionality of the `FileReadingMessageSource.WatchServiceDirectoryScanner` to perform `ResettableFileListFilter.remove()` when the file is deleted (`StandardWatchEventKinds.ENTRY_DELETE`) from the local directory.
See <<watch-service-directory-scanner>> for more information.
==== Configuring with Java Configuration
The following Spring Boot application provides an example of configuring the inbound adapter using Java configuration:

13
src/reference/asciidoc/sftp.adoc
View File

@@ -427,8 +427,7 @@ This will work for any `ResettableFileListFilter`.
remote-directory-expression="'/sftpSource'"
local-directory="file:myLocalDir"
auto-create-local-directory="true"
filename-pattern="*.txt"
local-filter="acceptOnceFilter">
filename-pattern="*.txt">
<int:poller fixed-rate="1000">
<int:transactional synchronization-factory="syncFactory" />
</int:poller>
@@ -438,13 +437,21 @@ This will work for any `ResettableFileListFilter`.
class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
<int:transaction-synchronization-factory id="syncFactory">
<int:after-rollback expression="@acceptOnceFilter.remove(payload)" />
<int:after-rollback expression="payload.delete()" />
</int:transaction-synchronization-factory>
<bean id="transactionManager"
class="org.springframework.integration.transaction.PseudoTransactionManager" />
----
Starting with _version 5.0_, the Inbound Channel Adapter can build sub-directories locally according the generated local file name.
That can be a remote sub-path as well.
To be able to read local directory recursively for modification according the hierarchy support, an internal `FileReadingMessageSource` now have been switched to use a new `RecursiveDirectoryScanner` based on the `Files.walk()` algorithm.
Also the `AbstractInboundFileSynchronizingMessageSource` can now be switched to the `WatchService` -based `DirectoryScanner` via `setUseWatchService()` option.
It is also configured for all the `WatchEventType` s to react for any modifications in local directory.
The reprocessing sample above is based on the build-in functionality of the `FileReadingMessageSource.WatchServiceDirectoryScanner` to perform `ResettableFileListFilter.remove()` when the file is deleted (`StandardWatchEventKinds.ENTRY_DELETE`) from the local directory.
See <<watch-service-directory-scanner>> for more information.
==== Configuring with Java Configuration
The following Spring Boot application provides an example of configuring the inbound adapter using Java configuration:

4
src/reference/asciidoc/whats-new.adoc
View File

@@ -125,8 +125,12 @@ The (S)FTP streaming inbound channel adapters now add remote file information in
The FTP and SFTP outbound channel adapters, as well as `PUT` command of the outbound gateways, now support `InputStream` as `payload`, too.
The inbound channel adapters now can build file tree locally and use a new `RecursiveDirectoryScanner` by default for local directory.
Also these adapters can now be switched to the `WatchService` instead.
See <<ftp>> and <<sftp>> for more information.
==== Integration Properties
Since _version 4.3.2_ a new `spring.integration.readOnly.headers` global property has been added to customize the list of headers which should not be copied to a newly created `Message` by the `MessageBuilder`.