Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-3919: FTP: Allow null for Remote Directory

Browse Source 
JIRA: https://jira.spring.io/browse/INT-3919

Since `FtpClient` supports `null` for the `LS` command, treating it as a current `working directory`,
there is no reason to forbid `null` from the FTP adapters end-user perspective.

* Allow `null` for the `FtpSession` `list()` and `listNames()` methods
* Allow `null` in the `remote-directory` for the `<int-ftp:inbound-channel-adapter>`
* Allow `null` in the `expression` for the `FtpOutboundGateway`

Polishing - send error if error on async output

Cover `onFailure()` from `onSuccess()` with the `errorChannel`

Address PR Comments

* Get rid of `null` population for the `remoteDirectoryExpression` in the `AbstractPollingInboundChannelAdapterParser`
* Populate `new LiteralExpression(null)` from the `FtpInboundFileSynchronizer` ctor
* Introduce `buildRemotePath(parent, child)` function in the `AbstractRemoteFileOutboundGateway` with the `null` logic for `parent`
* Rework `mGetWithoutRecursion()` to use `LS` command and allow `null` for the dir.
* Fix tests according the new `mGetWithoutRecursion()` logic

Polishing
This commit is contained in:
Artem Bilan
2016-02-05 16:06:10 -05:00
committed by Gary Russell
parent 2fad35d9b8
commit 7595e4f142
13 changed files with 297 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
spring-integration-file/src/main/java/org/springframework/integration/file/config/AbstractRemoteFileInboundChannelAdapterParser.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2013 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -33,6 +33,7 @@ import org.springframework.util.StringUtils;
* @author Oleg Zhurakousky
* @author Mark Fisher
* @author Gary Russell
* @author Artem Bilan
* @since 2.0
*/
public abstract class AbstractRemoteFileInboundChannelAdapterParser extends AbstractPollingInboundChannelAdapterParser {
@@ -46,8 +47,10 @@ public abstract class AbstractRemoteFileInboundChannelAdapterParser extends Abst
// configure the InboundFileSynchronizer properties
BeanDefinition expressionDef = IntegrationNamespaceUtils.createExpressionDefinitionFromValueOrExpression(
"remote-directory", "remote-directory-expression", parserContext, element, true);
synchronizerBuilder.addPropertyValue("remoteDirectoryExpression", expressionDef);
"remote-directory", "remote-directory-expression", parserContext, element, false);
if (expressionDef != null) {
synchronizerBuilder.addPropertyValue("remoteDirectoryExpression", expressionDef);
}
IntegrationNamespaceUtils.setValueIfAttributeDefined(synchronizerBuilder, element, "delete-remote-files");
IntegrationNamespaceUtils.setValueIfAttributeDefined(synchronizerBuilder, element, "preserve-timestamp");
@@ -57,7 +60,8 @@ public abstract class AbstractRemoteFileInboundChannelAdapterParser extends Abst
this.configureFilter(synchronizerBuilder, element, parserContext);
// build the MessageSource
BeanDefinitionBuilder messageSourceBuilder = BeanDefinitionBuilder.genericBeanDefinition(this.getMessageSourceClassname());
BeanDefinitionBuilder messageSourceBuilder =
BeanDefinitionBuilder.genericBeanDefinition(getMessageSourceClassname());
messageSourceBuilder.addConstructorArgValue(synchronizerBuilder.getBeanDefinition());
String comparator = element.getAttribute("comparator");
if (StringUtils.hasText(comparator)) {
@@ -65,17 +69,21 @@ public abstract class AbstractRemoteFileInboundChannelAdapterParser extends Abst
}
IntegrationNamespaceUtils.setReferenceIfAttributeDefined(messageSourceBuilder, element, "local-filter");
IntegrationNamespaceUtils.setValueIfAttributeDefined(messageSourceBuilder, element, "local-directory");
IntegrationNamespaceUtils.setValueIfAttributeDefined(messageSourceBuilder, element, "auto-create-local-directory");
IntegrationNamespaceUtils.setValueIfAttributeDefined(messageSourceBuilder, element,
"auto-create-local-directory");
String localFileGeneratorExpression = element.getAttribute("local-filename-generator-expression");
if (StringUtils.hasText(localFileGeneratorExpression)) {
BeanDefinitionBuilder localFileGeneratorExpressionBuilder = BeanDefinitionBuilder.genericBeanDefinition(ExpressionFactoryBean.class);
BeanDefinitionBuilder localFileGeneratorExpressionBuilder =
BeanDefinitionBuilder.genericBeanDefinition(ExpressionFactoryBean.class);
localFileGeneratorExpressionBuilder.addConstructorArgValue(localFileGeneratorExpression);
synchronizerBuilder.addPropertyValue("localFilenameGeneratorExpression", localFileGeneratorExpressionBuilder.getBeanDefinition());
synchronizerBuilder.addPropertyValue("localFilenameGeneratorExpression",
localFileGeneratorExpressionBuilder.getBeanDefinition());
}
return messageSourceBuilder.getBeanDefinition();
}
private void configureFilter(BeanDefinitionBuilder synchronizerBuilder, Element element, ParserContext parserContext) {
private void configureFilter(BeanDefinitionBuilder synchronizerBuilder, Element element,
ParserContext parserContext) {
String filter = element.getAttribute("filter");
String fileNamePattern = element.getAttribute("filename-pattern");
String fileNameRegex = element.getAttribute("filename-regex");

6
spring-integration-file/src/main/java/org/springframework/integration/file/config/AbstractRemoteFileOutboundGatewayParser.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2015 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -58,7 +58,9 @@ public abstract class AbstractRemoteFileOutboundGatewayParser extends AbstractCo
}
else {
builder.addConstructorArgValue(element.getAttribute("command"));
builder.addConstructorArgValue(element.getAttribute(EXPRESSION_ATTRIBUTE));
if (element.hasAttribute(EXPRESSION_ATTRIBUTE)) {
builder.addConstructorArgValue(element.getAttribute(EXPRESSION_ATTRIBUTE));
}
}
IntegrationNamespaceUtils.setValueIfAttributeDefined(builder, element, "command-options", "options");
IntegrationNamespaceUtils.setValueIfAttributeDefined(builder, element, "reply-timeout", "sendTimeout");

162
spring-integration-file/src/main/java/org/springframework/integration/file/remote/gateway/AbstractRemoteFileOutboundGateway.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2015 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -47,7 +47,6 @@ import org.springframework.integration.file.remote.session.SessionFactory;
import org.springframework.integration.file.support.FileExistsMode;
import org.springframework.integration.handler.AbstractReplyProducingMessageHandler;
import org.springframework.integration.handler.ExpressionEvaluatingMessageProcessor;
import org.springframework.integration.support.AbstractIntegrationMessageBuilder;
import org.springframework.integration.support.PartialSuccessException;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandlingException;
@@ -65,7 +64,7 @@ import org.springframework.util.StringUtils;
*/
public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReplyProducingMessageHandler {
private final RemoteFileTemplate<F> remoteFileTemplate;
protected final RemoteFileTemplate<F> remoteFileTemplate;
protected final Command command;
@@ -512,7 +511,7 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
private Object doLs(Message<?> requestMessage) {
String dir = this.fileNameProcessor.processMessage(requestMessage);
if (!dir.endsWith(this.remoteFileTemplate.getRemoteFileSeparator())) {
if (dir != null && !dir.endsWith(this.remoteFileTemplate.getRemoteFileSeparator())) {
dir += this.remoteFileTemplate.getRemoteFileSeparator();
}
final String fullDir = dir;
@@ -529,9 +528,9 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
}
private Object doGet(final Message<?> requestMessage) {
final String remoteFilePath = this.fileNameProcessor.processMessage(requestMessage);
final String remoteFilename = this.getRemoteFilename(remoteFilePath);
final String remoteDir = this.getRemoteDirectory(remoteFilePath, remoteFilename);
final String remoteFilePath = this.fileNameProcessor.processMessage(requestMessage);
final String remoteFilename = getRemoteFilename(remoteFilePath);
final String remoteDir = getRemoteDirectory(remoteFilePath, remoteFilename);
Session<F> session = null;
Object payload;
if (this.options.contains(Option.STREAM)) {
@@ -550,30 +549,27 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
@Override
public File doInSession(Session<F> session) throws IOException {
return AbstractRemoteFileOutboundGateway.this.get(requestMessage, session, remoteDir, remoteFilePath,
remoteFilename, true);
return get(requestMessage, session, remoteDir, remoteFilePath, remoteFilename, true);
}
});
}
AbstractIntegrationMessageBuilder<Object> builder = this.getMessageBuilderFactory().withPayload(payload)
.setHeader(FileHeaders.REMOTE_DIRECTORY, remoteDir)
.setHeader(FileHeaders.REMOTE_FILE, remoteFilename);
if (session != null) {
builder.setHeader(FileHeaders.REMOTE_SESSION, session);
}
return builder.build();
return getMessageBuilderFactory().withPayload(payload)
.setHeader(FileHeaders.REMOTE_DIRECTORY, remoteDir)
.setHeader(FileHeaders.REMOTE_FILE, remoteFilename)
.setHeader(FileHeaders.REMOTE_SESSION, session)
.build();
}
private Object doMget(final Message<?> requestMessage) {
final String remoteFilePath = this.fileNameProcessor.processMessage(requestMessage);
final String remoteFilename = this.getRemoteFilename(remoteFilePath);
final String remoteDir = this.getRemoteDirectory(remoteFilePath, remoteFilename);
final String remoteFilePath = this.fileNameProcessor.processMessage(requestMessage);
final String remoteFilename = getRemoteFilename(remoteFilePath);
final String remoteDir = getRemoteDirectory(remoteFilePath, remoteFilename);
List<File> payload = this.remoteFileTemplate.execute(new SessionCallback<F, List<File>>() {
@Override
public List<File> doInSession(Session<F> session) throws IOException {
return AbstractRemoteFileOutboundGateway.this.mGet(requestMessage, session, remoteDir, remoteFilename);
return mGet(requestMessage, session, remoteDir, remoteFilename);
}
});
return this.getMessageBuilderFactory().withPayload(payload)
@@ -583,10 +579,12 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
}
private Object doRm(Message<?> requestMessage) {
final String remoteFilePath = this.fileNameProcessor.processMessage(requestMessage);
String remoteFilename = this.getRemoteFilename(remoteFilePath);
String remoteDir = this.getRemoteDirectory(remoteFilePath, remoteFilename);
final String remoteFilePath = this.fileNameProcessor.processMessage(requestMessage);
String remoteFilename = getRemoteFilename(remoteFilePath);
String remoteDir = getRemoteDirectory(remoteFilePath, remoteFilename);
boolean payload = this.remoteFileTemplate.remove(remoteFilePath);
return this.getMessageBuilderFactory().withPayload(payload)
.setHeader(FileHeaders.REMOTE_DIRECTORY, remoteDir)
.setHeader(FileHeaders.REMOTE_FILE, remoteFilename)
@@ -595,8 +593,8 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
private Object doMv(Message<?> requestMessage) {
String remoteFilePath = this.fileNameProcessor.processMessage(requestMessage);
String remoteFilename = this.getRemoteFilename(remoteFilePath);
String remoteDir = this.getRemoteDirectory(remoteFilePath, remoteFilename);
String remoteFilename = getRemoteFilename(remoteFilePath);
String remoteDir = getRemoteDirectory(remoteFilePath, remoteFilename);
String remoteFileNewPath = this.renameProcessor.processMessage(requestMessage);
Assert.hasLength(remoteFileNewPath, "New filename cannot be empty");
@@ -635,8 +633,7 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
return this.doPut(requestMessage);
}
else {
List<String> replies = this.putLocalDirectory(requestMessage, file, null);
return replies;
return putLocalDirectory(requestMessage, file, null);
}
}
@@ -717,7 +714,9 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
private List<F> listFilesInRemoteDir(Session<F> session, String directory, String subDirectory) throws IOException {
List<F> lsFiles = new ArrayList<F>();
F[] files = session.list(directory + subDirectory);
String remoteDirectory = buildRemotePath(directory, subDirectory);
F[] files = session.list(remoteDirectory);
boolean recursion = this.options.contains(Option.RECURSIVE);
if (!ObjectUtils.isEmpty(files)) {
Collection<F> filteredFiles = this.filterFiles(files);
@@ -742,6 +741,17 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
return lsFiles;
}
private String buildRemotePath(String parent, String child) {
String remotePath = null;
if (parent != null) {
remotePath = (parent + child);
}
else if (StringUtils.hasText(child)) {
remotePath = "." + this.remoteFileTemplate.getRemoteFileSeparator() + child;
}
return remotePath;
}
protected final List<F> filterFiles(F[] files) {
return (this.filter != null) ? this.filter.filterFiles(files) : Arrays.asList(files);
}
@@ -784,8 +794,8 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
* @return The file.
* @throws IOException Any IOException.
*/
protected File get(Message<?> message, Session<F> session, String remoteDir, String remoteFilePath, String remoteFilename, boolean lsFirst)
throws IOException {
protected File get(Message<?> message, Session<F> session, String remoteDir, String remoteFilePath,
String remoteFilename, boolean lsFirst) throws IOException {
F[] files = null;
if (lsFirst) {
files = session.list(remoteFilePath);
@@ -796,7 +806,8 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
throw new MessagingException(remoteFilePath + " is not a file");
}
}
File localFile = new File(this.generateLocalDirectory(message, remoteDir), this.generateLocalFileName(message, remoteFilename));
File localFile =
new File(generateLocalDirectory(message, remoteDir), generateLocalFileName(message, remoteFilename));
FileExistsMode fileExistsMode = this.fileExistsMode;
boolean appending = FileExistsMode.APPEND.equals(fileExistsMode);
boolean replacing = FileExistsMode.REPLACE.equals(fileExistsMode);
@@ -874,37 +885,40 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
private List<File> mGetWithoutRecursion(Message<?> message, Session<F> session, String remoteDirectory,
String remoteFilename) throws IOException {
String path = this.generateFullPath(remoteDirectory, remoteFilename);
String[] fileNames = session.listNames(path);
if (fileNames == null) {
fileNames = new String[0];
}
if (fileNames.length == 0 && this.options.contains(Option.EXCEPTION_WHEN_EMPTY)) {
throw new MessagingException("No files found at " + remoteDirectory
List<File> files = new ArrayList<File>();
String remotePath = buildRemotePath(remoteDirectory, remoteFilename);
@SuppressWarnings("unchecked")
List<AbstractFileInfo<F>> remoteFiles = (List<AbstractFileInfo<F>>) ls(session, remotePath);
if (remoteFiles.size() == 0 && this.options.contains(Option.EXCEPTION_WHEN_EMPTY)) {
throw new MessagingException("No files found at "
+ (remoteDirectory != null ? remoteDirectory : "Client Working Directory")
+ " with pattern " + remoteFilename);
}
List<File> files = new ArrayList<File>();
String remoteFileSeparator = this.remoteFileTemplate.getRemoteFileSeparator();
try {
for (String fileName : fileNames) {
File file;
if (fileName.contains(remoteFileSeparator) &&
fileName.startsWith(remoteDirectory)) { // the server returned the full path
file = this.get(message, session, remoteDirectory, fileName,
fileName.substring(fileName.lastIndexOf(remoteFileSeparator)), false);
}
else {
file = this.get(message, session, remoteDirectory,
this.generateFullPath(remoteDirectory, fileName), fileName, false);
for (AbstractFileInfo<F> lsEntry : remoteFiles) {
if (lsEntry.isDirectory()) {
continue;
}
String fullFileName = remoteDirectory != null
? remoteDirectory + getFilename(lsEntry)
: getFilename(lsEntry);
/*
* With recursion, the filename might contain subdirectory information
* normalize each file separately.
*/
String fileName = this.getRemoteFilename(fullFileName);
String actualRemoteDirectory = this.getRemoteDirectory(fullFileName, fileName);
File file = this.get(message, session, actualRemoteDirectory,
fullFileName, fileName, false);
files.add(file);
}
}
catch (Exception e) {
if (files.size() > 0) {
throw new PartialSuccessException(message,
"Partially successful 'mget' operation on " + remoteDirectory, e, files,
Arrays.asList(fileNames));
"Partially successful recursive 'mget' operation on "
+ (remoteDirectory != null ? remoteDirectory : "Client Working Directory"),
e, files, remoteFiles);
}
else if (e instanceof MessagingException) {
throw (MessagingException) e;
@@ -920,14 +934,17 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
String remoteFilename) throws IOException {
List<File> files = new ArrayList<File>();
@SuppressWarnings("unchecked")
List<AbstractFileInfo<F>> fileNames = (List<AbstractFileInfo<F>>) this.ls(session, remoteDirectory);
List<AbstractFileInfo<F>> fileNames = (List<AbstractFileInfo<F>>) ls(session, remoteDirectory);
if (fileNames.size() == 0 && this.options.contains(Option.EXCEPTION_WHEN_EMPTY)) {
throw new MessagingException("No files found at " + remoteDirectory
throw new MessagingException("No files found at "
+ (remoteDirectory != null ? remoteDirectory : "Client Working Directory")
+ " with pattern " + remoteFilename);
}
try {
for (AbstractFileInfo<F> lsEntry : fileNames) {
String fullFileName = remoteDirectory + this.getFilename(lsEntry);
String fullFileName = remoteDirectory != null
? remoteDirectory + getFilename(lsEntry)
: getFilename(lsEntry);
/*
* With recursion, the filename might contain subdirectory information
* normalize each file separately.
@@ -942,7 +959,9 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
catch (Exception e) {
if (files.size() > 0) {
throw new PartialSuccessException(message,
"Partially successful recursive 'mget' operation on " + remoteDirectory, e, files, fileNames);
"Partially successful recursive 'mget' operation on "
+ (remoteDirectory != null ? remoteDirectory : "Client Working Directory"),
e, files, fileNames);
}
else if (e instanceof MessagingException) {
throw (MessagingException) e;
@@ -957,45 +976,30 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
private String getRemoteDirectory(String remoteFilePath, String remoteFilename) {
String remoteDir = remoteFilePath.substring(0, remoteFilePath.lastIndexOf(remoteFilename));
if (remoteDir.length() == 0) {
remoteDir = this.remoteFileTemplate.getRemoteFileSeparator();
return null;
}
return remoteDir;
}
private String generateFullPath(String remoteDirectory, String remoteFilename) {
String path;
String remoteFileSeparator = this.remoteFileTemplate.getRemoteFileSeparator();
if (remoteFileSeparator.equals(remoteDirectory)) {
path = remoteFilename;
}
else if (remoteDirectory.endsWith(remoteFileSeparator)) {
path = remoteDirectory + remoteFilename;
}
else {
path = remoteDirectory + remoteFileSeparator + remoteFilename;
}
return path;
}
/**
* @param remoteFilePath The remote file path.
* @return The remote file name.
*/
protected String getRemoteFilename(String remoteFilePath) {
String remoteFileName;
int index = remoteFilePath.lastIndexOf(this.remoteFileTemplate.getRemoteFileSeparator());
if (index < 0) {
remoteFileName = remoteFilePath;
return remoteFilePath;
}
else {
remoteFileName = remoteFilePath.substring(index + 1);
return remoteFilePath.substring(index + 1);
}
return remoteFileName;
}
private File generateLocalDirectory(Message<?> message, String remoteDirectory) {
EvaluationContext evaluationContext = ExpressionUtils.createStandardEvaluationContext(this.getBeanFactory());
evaluationContext.setVariable("remoteDirectory", remoteDirectory);
EvaluationContext evaluationContext = ExpressionUtils.createStandardEvaluationContext(getBeanFactory());
if (remoteDirectory != null) {
evaluationContext.setVariable("remoteDirectory", remoteDirectory);
}
//TODO see org.springframework.integration.context.CustomConversionServiceFactoryBean
// File localDir = this.localDirectoryExpression.getValue(evaluationContext, message, File.class);
String localDirPath = this.localDirectoryExpression.getValue(evaluationContext, message, String.class);
@@ -1008,7 +1012,7 @@ public abstract class AbstractRemoteFileOutboundGateway<F> extends AbstractReply
private String generateLocalFileName(Message<?> message, String remoteFileName){
if (this.localFilenameGeneratorExpression != null){
EvaluationContext evaluationContext = ExpressionUtils.createStandardEvaluationContext(this.getBeanFactory());
EvaluationContext evaluationContext = ExpressionUtils.createStandardEvaluationContext(getBeanFactory());
evaluationContext.setVariable("remoteFileName", remoteFileName);
return this.localFilenameGeneratorExpression.getValue(evaluationContext, message, String.class);
}

14
spring-integration-file/src/main/java/org/springframework/integration/file/remote/synchronizer/AbstractInboundFileSynchronizer.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2015 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -200,6 +200,14 @@ public abstract class AbstractInboundFileSynchronizer<F>
if (this.evaluationContext == null) {
this.evaluationContext = ExpressionUtils.createStandardEvaluationContext(this.beanFactory);
}
doInit();
}
/**
* Subclasses can override to perform initialization - called from
* {@link InitializingBean#afterPropertiesSet()}.
*/
protected void doInit() {
}
protected final List<F> filterFiles(F[] files) {
@@ -271,7 +279,9 @@ public abstract class AbstractInboundFileSynchronizer<F>
Session<F> session) throws IOException {
String remoteFileName = this.getFilename(remoteFile);
String localFileName = this.generateLocalFileName(remoteFileName);
String remoteFilePath = remoteDirectoryPath + remoteFileSeparator + remoteFileName;
String remoteFilePath = remoteDirectoryPath != null
? (remoteDirectoryPath + remoteFileSeparator + remoteFileName)
: remoteFileName;
if (!this.isFile(remoteFile)) {
if (logger.isDebugEnabled()) {
logger.debug("cannot copy, not a file: " + remoteFilePath);

19
spring-integration-file/src/test/java/org/springframework/integration/file/remote/gateway/RemoteFileOutboundGatewayTests.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2015 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -23,6 +23,7 @@ import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.not;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertSame;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
@@ -188,8 +189,10 @@ public class RemoteFileOutboundGatewayTests {
}
@Override
public String[] listNames(String path) throws IOException {
return new String[] { path1, path2 };
public TestLsEntry[] list(String path) throws IOException {
return new TestLsEntry[] {
new TestLsEntry(path1.replaceFirst("testremote/", ""), 123, false, false, 1234, "-r--r--r--"),
new TestLsEntry(path2.replaceFirst("testremote/", ""), 123, false, false, 1234, "-r--r--r--")};
}
});
@@ -220,8 +223,8 @@ public class RemoteFileOutboundGatewayTests {
}
@Override
public String[] listNames(String path) throws IOException {
return new String[]{"f1"};
public TestLsEntry[] list(String path) throws IOException {
return new TestLsEntry[]{new TestLsEntry("f1", 123, false, false, 1234, "-r--r--r--")};
}
});
@@ -621,10 +624,8 @@ public class RemoteFileOutboundGatewayTests {
assertEquals(outFile, out.getPayload());
assertTrue(outFile.exists());
outFile.delete();
assertEquals("/",
out.getHeaders().get(FileHeaders.REMOTE_DIRECTORY));
assertEquals("f1",
out.getHeaders().get(FileHeaders.REMOTE_FILE));
assertNull(out.getHeaders().get(FileHeaders.REMOTE_DIRECTORY));
assertEquals("f1", out.getHeaders().get(FileHeaders.REMOTE_FILE));
}
@SuppressWarnings("unchecked")

28
spring-integration-ftp/src/main/java/org/springframework/integration/ftp/gateway/FtpOutboundGateway.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2015 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -82,6 +82,32 @@ public class FtpOutboundGateway extends AbstractRemoteFileOutboundGateway<FTPFil
super(remoteFileTemplate, command, expression);
}
/**
* Construct an instance with the supplied session factory, a command ('ls', 'get'
* etc).
* <p> The {@code remoteDirectory} expression is {@code null} assuming to use
* the {@code workingDirectory} from the FTP Client.
* @param sessionFactory the session factory.
* @param command the command.
* @since 4.3
*/
public FtpOutboundGateway(SessionFactory<FTPFile> sessionFactory, String command) {
this(sessionFactory, command, null);
}
/**
* Construct an instance with the supplied remote file template, a command ('ls',
* 'get' etc).
* <p> The {@code remoteDirectory} expression is {@code null} assuming to use
* the {@code workingDirectory} from the FTP Client.
* @param remoteFileTemplate the remote file template.
* @param command the command.
* @since 4.3
*/
public FtpOutboundGateway(RemoteFileTemplate<FTPFile> remoteFileTemplate, String command) {
this(remoteFileTemplate, command, null);
}
@Override
public String getComponentType() {
return "ftp:outbound-gateway";

5
spring-integration-ftp/src/main/java/org/springframework/integration/ftp/inbound/FtpInboundFileSynchronizer.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2014 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -18,6 +18,7 @@ package org.springframework.integration.ftp.inbound;
import org.apache.commons.net.ftp.FTPFile;
import org.springframework.expression.common.LiteralExpression;
import org.springframework.integration.file.remote.session.Session;
import org.springframework.integration.file.remote.session.SessionFactory;
import org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer;
@@ -40,9 +41,9 @@ public class FtpInboundFileSynchronizer extends AbstractInboundFileSynchronizer<
*/
public FtpInboundFileSynchronizer(SessionFactory<FTPFile> sessionFactory) {
super(sessionFactory);
setRemoteDirectoryExpression(new LiteralExpression(null));
}
@Override
protected boolean isFile(FTPFile file) {
return file != null && file.isFile();

9
spring-integration-ftp/src/main/java/org/springframework/integration/ftp/session/AbstractFtpSessionFactory.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2014 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -16,7 +16,6 @@
package org.springframework.integration.ftp.session;
import java.io.IOException;
import java.net.SocketException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -40,8 +39,6 @@ import org.springframework.util.Assert;
*/
public abstract class AbstractFtpSessionFactory<T extends FTPClient> implements SessionFactory<FTPFile> {
public static final String DEFAULT_REMOTE_WORKING_DIRECTORY = "/";
private final Log logger = LogFactory.getLog(this.getClass());
protected FTPClientConfig config;
@@ -172,7 +169,7 @@ public abstract class AbstractFtpSessionFactory<T extends FTPClient> implements
}
}
private T createClient() throws SocketException, IOException {
private T createClient() throws IOException {
final T client = this.createClientInstance();
Assert.notNull(client, "client must not be null");
client.configure(this.config);
@@ -200,7 +197,7 @@ public abstract class AbstractFtpSessionFactory<T extends FTPClient> implements
// Login
if (!client.login(username, password)) {
throw new IllegalStateException("Login failed. The respponse from the server is: " +
throw new IllegalStateException("Login failed. The response from the server is: " +
client.getReplyString());
}

22
spring-integration-ftp/src/main/java/org/springframework/integration/ftp/session/FtpSession.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2013 the original author or authors.
* Copyright 2002-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -36,6 +36,7 @@ import org.springframework.util.Assert;
* @author Mark Fisher
* @author Oleg Zhurakousky
* @author Gary Russell
* @author Artem Bilan
* @since 2.0
*/
public class FtpSession implements Session<FTPFile> {
@@ -55,22 +56,21 @@ public class FtpSession implements Session<FTPFile> {
@Override
public boolean remove(String path) throws IOException {
Assert.hasText(path, "path must not be null");
boolean completed = this.client.deleteFile(path);
if (!completed) {
if (!this.client.deleteFile(path)) {
throw new IOException("Failed to delete '" + path + "'. Server replied with: " + client.getReplyString());
}
return completed;
else {
return true;
}
}
@Override
public FTPFile[] list(String path) throws IOException {
Assert.hasText(path, "path must not be null");
return this.client.listFiles(path);
}
@Override
public String[] listNames(String path) throws IOException {
Assert.hasText(path, "path must not be null");
return this.client.listNames(path);
}
@@ -83,7 +83,7 @@ public class FtpSession implements Session<FTPFile> {
throw new IOException("Failed to copy '" + path +
"'. Server replied with: " + this.client.getReplyString());
}
logger.info("File has been successfully transfered from: " + path);
logger.info("File has been successfully transferred from: " + path);
}
@Override
@@ -93,7 +93,8 @@ public class FtpSession implements Session<FTPFile> {
}
InputStream inputStream = this.client.retrieveFileStream(source);
if (inputStream == null) {
throw new IOException("Failed to obtain InputStream for remote file " + source + ": " + this.client.getReplyCode());
throw new IOException("Failed to obtain InputStream for remote file " + source + ": "
+ this.client.getReplyCode());
}
return inputStream;
}
@@ -123,7 +124,7 @@ public class FtpSession implements Session<FTPFile> {
+ "'. Server replied with: " + this.client.getReplyString());
}
if (logger.isInfoEnabled()) {
logger.info("File has been successfully transfered to: " + path);
logger.info("File has been successfully transferred to: " + path);
}
}
@@ -196,7 +197,8 @@ public class FtpSession implements Session<FTPFile> {
Assert.hasText(path, "'path' must not be empty");
String currentWorkingPath = this.client.printWorkingDirectory();
Assert.state(currentWorkingPath != null, "working directory cannot be determined, therefore exists check can not be completed");
Assert.state(currentWorkingPath != null,
"working directory cannot be determined, therefore exists check can not be completed");
boolean exists = false;
try {

20
spring-integration-ftp/src/test/java/org/springframework/integration/ftp/outbound/FtpServerOutboundTests-context.xml
View File

@@ -41,8 +41,9 @@
<int-ftp:outbound-gateway session-factory="ftpSessionFactory"
request-channel="inboundMGet"
command="mget"
command-options="-f"
expression="payload"
local-directory-expression="@ftpServer.targetLocalDirectoryName + #remoteDirectory"
local-directory-expression="@ftpServer.targetLocalDirectoryName + (#remoteDirectory ?: '')"
local-filename-generator-expression="#remoteFileName.replaceFirst('ftpSource', 'localTarget')"
reply-channel="output"/>
@@ -169,5 +170,22 @@
<bean id="messageSessionCallback"
class="org.springframework.integration.ftp.outbound.FtpServerOutboundTests$TestMessageSessionCallback"/>
<int-ftp:outbound-gateway session-factory="ftpSessionFactory"
request-channel="inboundLs"
command="ls"
command-options="-1"
reply-channel="output"/>
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="output"
auto-startup="false"
session-factory="ftpSessionFactory"
auto-create-local-directory="true"
delete-remote-files="false"
filename-pattern="*.txt"
temporary-file-suffix=".foo"
local-directory="#{T (System).getProperty('java.io.tmpdir') + T (java.util.UUID).randomUUID().toString()}">
<int:poller fixed-delay="100"/>
</int-ftp:inbound-channel-adapter>
</beans>

94
spring-integration-ftp/src/test/java/org/springframework/integration/ftp/outbound/FtpServerOutboundTests.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2013-2015 the original author or authors.
* Copyright 2013-2016 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -17,12 +17,17 @@
package org.springframework.integration.ftp.outbound;
import static org.hamcrest.Matchers.anyOf;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.isOneOf;
import static org.hamcrest.Matchers.not;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertSame;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
@@ -44,6 +49,7 @@ import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPFile;
import org.hamcrest.Matchers;
import org.junit.Before;
@@ -57,6 +63,7 @@ import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.expression.spel.standard.SpelExpressionParser;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.endpoint.SourcePollingChannelAdapter;
import org.springframework.integration.file.FileHeaders;
import org.springframework.integration.file.filters.FileListFilter;
import org.springframework.integration.file.remote.InputStreamCallback;
@@ -139,6 +146,12 @@ public class FtpServerOutboundTests {
@Autowired
private DirectChannel inboundCallback;
@Autowired
private DirectChannel inboundLs;
@Autowired
private SourcePollingChannelAdapter ftpInbound;
@Before
public void setup() {
this.ftpServer.recursiveDelete(ftpServer.getTargetLocalDirectory());
@@ -183,7 +196,7 @@ public class FtpServerOutboundTests {
@SuppressWarnings("unchecked")
public void testInt2866LocalDirectoryExpressionMGET() {
String dir = "ftpSource/";
this.inboundMGet.send(new GenericMessage<Object>(dir + "*.txt"));
this.inboundMGet.send(new GenericMessage<Object>("*.txt"));
Message<?> result = this.output.receive(1000);
assertNotNull(result);
List<File> localFiles = (List<File>) result.getPayload();
@@ -205,11 +218,29 @@ public class FtpServerOutboundTests {
}
}
@Test
@SuppressWarnings("unchecked")
public void testMGETOnNullDir() throws IOException {
Session<FTPFile> session = ftpSessionFactory.getSession();
((FTPClient) session.getClientInstance()).changeWorkingDirectory("ftpSource");
session.close();
this.inboundMGet.send(new GenericMessage<Object>(""));
Message<?> result = this.output.receive(1000);
assertNotNull(result);
List<File> localFiles = (List<File>) result.getPayload();
for (File file : localFiles) {
assertThat(file.getName(), isOneOf("localTarget1.txt", "localTarget2.txt"));
assertThat(file.getName(), not(containsString("null")));
}
}
@Test
@SuppressWarnings("unchecked")
public void testInt3172LocalDirectoryExpressionMGETRecursive() {
String dir = "ftpSource/";
this.inboundMGetRecursive.send(new GenericMessage<Object>(dir + "*"));
this.inboundMGetRecursive.send(new GenericMessage<Object>("*"));
Message<?> result = this.output.receive(1000);
assertNotNull(result);
List<File> localFiles = (List<File>) result.getPayload();
@@ -399,17 +430,19 @@ public class FtpServerOutboundTests {
@Test
public void testMgetPartial() throws Exception {
Session<FTPFile> session = spyOnSession();
doAnswer(new Answer<String[]>() {
doAnswer(new Answer<FTPFile[]>() {
@Override
public String[] answer(InvocationOnMock invocation) throws Throwable {
String[] files = (String[]) invocation.callRealMethod();
public FTPFile[] answer(InvocationOnMock invocation) throws Throwable {
FTPFile[] files = (FTPFile[]) invocation.callRealMethod();
// add an extra file where the get will fail
files = Arrays.copyOf(files, files.length + 1);
files[files.length - 1] = "bogus.txt";
FTPFile bogusFile = new FTPFile();
bogusFile.setName("bogus.txt");
files[files.length - 1] = bogusFile;
return files;
}
}).when(session).listNames("ftpSource/subFtpSource/*");
}).when(session).list("ftpSource/subFtpSource/*");
String dir = "ftpSource/subFtpSource/";
try {
this.inboundMGet.send(new GenericMessage<Object>(dir + "*"));
@@ -547,6 +580,51 @@ public class FtpServerOutboundTests {
assertEquals("FOO", receive.getPayload());
}
@Test
@SuppressWarnings("unchecked")
public void testLsForNullDir() throws IOException {
Session<FTPFile> session = ftpSessionFactory.getSession();
((FTPClient) session.getClientInstance()).changeWorkingDirectory("ftpSource");
session.close();
this.inboundLs.send(new GenericMessage<String>("foo"));
Message<?> receive = this.output.receive(10000);
assertNotNull(receive);
assertThat(receive.getPayload(), instanceOf(List.class));
List<String> files = (List<String>) receive.getPayload();
assertEquals(2, files.size());
assertThat(files, containsInAnyOrder("ftpSource1.txt", "ftpSource2.txt"));
FTPFile[] ftpFiles = ftpSessionFactory.getSession().list(null);
for (FTPFile ftpFile : ftpFiles) {
if (!ftpFile.isDirectory()) {
assertTrue(files.contains(ftpFile.getName()));
}
}
}
@Test
public void testInboundChannelAdapterWithNullDir() throws IOException {
Session<FTPFile> session = ftpSessionFactory.getSession();
((FTPClient) session.getClientInstance()).changeWorkingDirectory("ftpSource");
session.close();
this.ftpInbound.start();
Message<?> message = this.output.receive(10000);
assertNotNull(message);
assertThat(message.getPayload(), instanceOf(File.class));
assertEquals("ftpSource1.txt", ((File) message.getPayload()).getName());
message = this.output.receive(10000);
assertNotNull(message);
assertThat(message.getPayload(), instanceOf(File.class));
assertEquals("ftpSource2.txt", ((File) message.getPayload()).getName());
assertNull(this.output.receive(10));
this.ftpInbound.stop();
}
public static class SortingFileListFilter implements FileListFilter<File> {
@Override

15
src/reference/asciidoc/ftp.adoc
View File

@@ -191,6 +191,9 @@ Starting with _version 4.2_, you can specify `remote-directory-expression` inste
you to dynamically determine the directory on each poll.
e.g `remote-directory-expression="@myBean.determineRemoteDir()"`.
Starting with _version 4.3_, the `remote-directory`/`remote-directory-expression` attributes can be omitted assuming `null`.
In this case, according to the FTP protocol, the Client working directory is used as a default remote directory.
Sometimes file filtering based on the simple pattern specified via `filename-pattern` attribute might not be sufficient.
If this is the case, you can use the `filename-regex` attribute to specify a Regular Expression (e.g.
`filename-regex=".*\.test$"`).
@@ -255,7 +258,10 @@ Here is an example that uses a custom Filter implementation.
_Poller configuration notes for the inbound FTP adapter_
The job of the inbound FTP adapter consists of two tasks: _1) Communicate with a remote server in order to transfer files from a remote directory to a local directory.__2) For each transferred file, generate a Message with that file as a payload and send it to the channel identified by the 'channel' attribute._ That is why they are called 'channel-adapters' rather than just 'adapters'.
The job of the inbound FTP adapter consists of two tasks:
_1) Communicate with a remote server in order to transfer files from a remote directory to a local directory._
_2) For each transferred file, generate a Message with that file as a payload and send it to the channel identified by the 'channel' attribute._
That is why they are called 'channel-adapters' rather than just 'adapters'.
The main job of such an adapter is to generate a Message to be sent to a Message Channel.
Essentially, the second task mentioned above takes precedence in such a way that *IF* your local directory already has one or more files it will first generate Messages from those, and *ONLY* when all local files have been processed, will it initiate the remote communication to retrieve more files.
@@ -407,6 +413,13 @@ If the `-dirs` option is included, each recursive directory is also returned as
In this case, it is recommended that the `-1` is not used because you would not be able to determine files Vs.
directories, which is achievable using the `FileInfo` objects.
Starting with _version 4.3_, the `FtpSession` supports `null` for the `list()` and `listNames()` methods,
therefore the `expression` attribute can be omitted.
From Java perspective there are two new constructor without `expression` argument for convenience.
The `null` for `LS` command is treated as an Client working directory according to the FTP protocol.
The working directory can be set via the `FTPClient.changeWorkingDirectory()` function when you extend the
`DefaultFtpSessionFactory` and implement `postProcessClientBeforeConnect()` callback.
*get*
_get_ retrieves a remote file and supports the following option:

8
src/reference/asciidoc/whats-new.adoc
View File

@@ -114,6 +114,14 @@ See <<http-inbound>> for more information.
A new factory bean is provided to simplify the configuration of Jsch proxies for SFTP.
See <<sftp-proxy-factory-bean>> for more information.
==== FTP Changes
The `FtpSession` now supports `null` for the `list()` and `listNames()` method, since it is possible by the
underlying FTP Client.
With that the `FtpOutboundGateway` can now be configured without `remoteDirectory` expression.
And the `<int-ftp:inbound-channel-adapter>` can be configured without `remote-directory`/`remote-directory-expression`.
See <<ftp>> for more information.
==== Router Changes
The `ErrorMessageExceptionTypeRouter` supports now the `Exception` superclass mappings to avoid duplication