Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-4232: (S)FTP inbound: Fix lastModified Logic

Browse Source 
JIRA: https://jira.spring.io/browse/INT-4232

Currently to implement the synchronization logic when remote file is modified there is only one way: remove local file and provide a `localFilter` which can be clear from the removed file as well.

The Reference Manual claims that `FileSystemPersistentAcceptOnceFileListFilter` can let us pick up a fresh version of the remote file if we configure `preserveTimestamp = true`, but actually `AbstractInboundFileSynchronizer` just bypass that logic because it check a local file for existence.

* Modify `AbstractInboundFileSynchronizer` to compare `lastModified` of the remote file with local version if `preserveTimestamp == true`
* Also replace `AcceptOnceFileListFilter` to the `FileSystemPersistentAcceptOnceFileListFilter` since the first one doesn't care about `lastModified`.
If `preserveTimestamp` isn't in use the `FileSystemPersistentAcceptOnceFileListFilter` works fully similar to the `AcceptOnceFileListFilter`

**Cherry-pick to 4.3.x except test-case**

Make `SftpStreamingMessageSourceTests` compatible with Windows

Looks like SFTP embedded server provides different permissions for files on different OSs

Doc Polishing

Conflicts:
	spring-integration-file/src/test/java/org/springframework/integration/file/remote/synchronizer/AbstractRemoteFileSynchronizerTests.java
	spring-integration-ftp/src/test/java/org/springframework/integration/ftp/dsl/FtpTests.java
	spring-integration-sftp/src/test/java/org/springframework/integration/sftp/inbound/SftpStreamingMessageSourceTests.java
	src/reference/asciidoc/whats-new.adoc
Resolved.
This commit is contained in:
Artem Bilan
2017-02-22 14:36:13 -05:00
committed by Gary Russell
parent 2d4385a871
commit 909060abd5
6 changed files with 34 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
spring-integration-file/src/main/java/org/springframework/integration/file/remote/synchronizer/AbstractInboundFileSynchronizer.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2016 the original author or authors.
* Copyright 2002-2017 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -59,6 +59,7 @@ import org.springframework.util.ObjectUtils;
* @author Oleg Zhurakousky
* @author Gary Russell
* @author Artem Bilan
*
* @since 2.0
*/
public abstract class AbstractInboundFileSynchronizer<F>
@@ -289,8 +290,11 @@ public abstract class AbstractInboundFileSynchronizer<F>
return;
}
long modified = getModified(remoteFile);
File localFile = new File(localDirectory, localFileName);
if (!localFile.exists()) {
if (!localFile.exists() || (this.preserveTimestamp && modified != localFile.lastModified())) {
String tempFileName = localFile.getAbsolutePath() + this.temporaryFileSuffix;
File tempFile = new File(tempFileName);
OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(tempFile));
@@ -322,7 +326,7 @@ public abstract class AbstractInboundFileSynchronizer<F>
}
}
if (this.preserveTimestamp) {
localFile.setLastModified(getModified(remoteFile));
localFile.setLastModified(modified);
}
}
}

11
spring-integration-file/src/main/java/org/springframework/integration/file/remote/synchronizer/AbstractInboundFileSynchronizingMessageSource.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2016 the original author or authors.
* Copyright 2002-2017 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -30,7 +30,9 @@ import org.springframework.integration.file.FileReadingMessageSource;
import org.springframework.integration.file.filters.AcceptOnceFileListFilter;
import org.springframework.integration.file.filters.CompositeFileListFilter;
import org.springframework.integration.file.filters.FileListFilter;
import org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter;
import org.springframework.integration.file.filters.RegexPatternFileListFilter;
import org.springframework.integration.metadata.SimpleMetadataStore;
import org.springframework.messaging.Message;
import org.springframework.util.Assert;
@@ -55,6 +57,7 @@ import org.springframework.util.Assert;
* @author Josh Long
* @author Oleg Zhurakousky
* @author Gary Russell
* @author Artem Bilan
*/
public abstract class AbstractInboundFileSynchronizingMessageSource<F>
extends AbstractMessageSource<File> implements Lifecycle {
@@ -82,7 +85,7 @@ public abstract class AbstractInboundFileSynchronizingMessageSource<F>
*/
private final FileReadingMessageSource fileSource;
private volatile FileListFilter<File> localFileListFilter = new AcceptOnceFileListFilter<File>();
private volatile FileListFilter<File> localFileListFilter;
public AbstractInboundFileSynchronizingMessageSource(AbstractInboundFileSynchronizer<F> synchronizer) {
@@ -142,6 +145,10 @@ public abstract class AbstractInboundFileSynchronizingMessageSource<F>
}
}
this.fileSource.setDirectory(this.localDirectory);
if (this.localFileListFilter == null) {
this.localFileListFilter = new FileSystemPersistentAcceptOnceFileListFilter(
new SimpleMetadataStore(), getComponentName());
}
this.fileSource.setFilter(this.buildFilter());
if (this.getBeanFactory() != null) {
this.fileSource.setBeanFactory(this.getBeanFactory());

4
spring-integration-ftp/src/test/resources/log4j.properties
View File

@@ -5,5 +5,5 @@ log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ABSOLUTE} %5p %t %c{2}:%L - %m%n
log4j.category.org.springframework=WARN
log4j.category.org.springframework.integration=DEBUG
log4j.category.org.springframework.integration.file=DEBUG
log4j.category.org.springframework.integration=WARN
log4j.category.org.springframework.integration.file=WARN

5
src/reference/asciidoc/ftp.adoc
View File

@@ -215,8 +215,9 @@ Unless your application removes files after processing, the adapter will re-proc
Also, if you configure the `filter` to use a `FtpPersistentAcceptOnceFileListFilter`, and the remote file timestamp changes (causing it to be re-fetched), the default local filter will not allow this new file to be processed.
Use the `local-filter` attribute to configure the behavior of the local file system filter.
To solve these particular use cases, you can use a `FileSystemPersistentAcceptOnceFileListFilter` as a local filter instead.
This filter also stores the accepted file names and modified timestamp in an instance of the`MetadataStore` strategy (<<metadata-store>>), and will detect the change in the local file modified time.
Starting with _verion 4.3.8_, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
This filter stores the accepted file names and modified timestamp in an instance of the `MetadataStore` strategy (<<metadata-store>>), and will detect changes to the local file modified time.
The default `MetadataStore` is a `SimpleMetadataStore` which stores state in memory.
Since __version 4.1.5__, these filters have a new property `flushOnUpdate` which will cause them to flush the
metadata store on every update (if the store implements `Flushable`).

5
src/reference/asciidoc/sftp.adoc
View File

@@ -347,8 +347,9 @@ Unless your application removes files after processing, the adapter will re-proc
Also, if you configure the `filter` to use a `FtpPersistentAcceptOnceFileListFilter`, and the remote file timestamp changes (causing it to be re-fetched), the default local filter will not allow this new file to be processed.
Use the `local-filter` attribute to configure the behavior of the local file system filter.
To solve these particular use cases, you can use a `FileSystemPersistentAcceptOnceFileListFilter` as a local filter instead.
This filter also stores the accepted file names and modified timestamp in an instance of the`MetadataStore` strategy (<<metadata-store>>), and will detect the change in the local file modified time.
Starting with _verion 4.3.8_, a `FileSystemPersistentAcceptOnceFileListFilter` is configured by default.
This filter stores the accepted file names and modified timestamp in an instance of the `MetadataStore` strategy (<<metadata-store>>), and will detect changes to the local file modified time.
The default `MetadataStore` is a `SimpleMetadataStore` which stores state in memory.
Since __version 4.1.5__, these filters have a new property `flushOnUpdate` which will cause them to flush the
metadata store on every update (if the store implements `Flushable`).

10
src/reference/asciidoc/whats-new.adoc
View File

@@ -227,6 +227,11 @@ See <<http-inbound>> for more information.
A new factory bean is provided to simplify the configuration of Jsch proxies for SFTP.
See <<sftp-proxy-factory-bean>> for more information.
===== Inbound Channel Adapter
The inbound channel adapter is now configured with a `FileSystemPersistentAcceptOnceFileListFilter` in the `local-filter` by default.
See <<sftp-inbound>> for more information.
===== chmod
The SFTP outbound gateway (for `put` and `mput` commands) and the SFTP outbound channel adapter now support the
@@ -243,6 +248,11 @@ With that the `FtpOutboundGateway` can now be configured without `remoteDirector
And the `<int-ftp:inbound-channel-adapter>` can be configured without `remote-directory`/`remote-directory-expression`.
See <<ftp>> for more information.
===== Inbound Channel Adapter
The inbound channel adapter is now configured with a `FileSystemPersistentAcceptOnceFileListFilter` in the `local-filter` by default.
See <<ftp-inbound>> for more information.
==== Router Changes
The `ErrorMessageExceptionTypeRouter` supports now the `Exception` superclass mappings to avoid duplication