[[ftp-outbound-gateway]]
= FTP Outbound Gateway

The FTP outbound gateway provides a limited set of commands to interact with a remote FTP or FTPS server.
The supported commands are:

* `ls` (list files)
* `nlst` (list file names)
* `get` (retrieve file)
* `mget` (retrieve file(s))
* `rm` (remove file(s))
* `mv` (move/rename file)
* `put` (send file)
* `mput` (send multiple files)

[[ftp-using-ls]]
== Using the `ls` Command

`ls` lists remote files and supports the following options:

* `-1`: Retrieve a list of file names.
The default is to retrieve a list of `FileInfo` objects.
* `-a`: Include all files (including those starting with '.')
* `-f`: Do not sort the list
* `-dirs`: Include directories (they are excluded by default)
* `-links`: Include symbolic links (they are excluded by default)
* `-R`: List the remote directory recursively

In addition, filename filtering is provided, in the same manner as the `inbound-channel-adapter`.
See xref:ftp/inbound.adoc[FTP Inbound Channel Adapter].

The message payload resulting from an `ls` operation is a list of file names or a list of `FileInfo` objects.
These objects provide information such as modified time, permissions, and other details.

The remote directory that the `ls` command acted on is provided in the `file_remoteDirectory` header.

When using the recursive option (`-R`), the `fileName` includes any subdirectory elements, representing a relative path to the file (relative to the remote directory).
If the `-dirs` option is included, each recursive directory is also returned as an element in the list.
In this case, it is recommended that you not use the `-1` option, because you would not be able to distinguish files from directories, which you can do with the `FileInfo` objects.

Starting with version 4.3, the `FtpSession` supports `null` for the `list()` and `listNames()` methods.
Therefore, you can omit the `expression` attribute.
For convenience, Java configuration has two constructors that do not have an `expression` argument.
or `LS`, `NLST`, `PUT` and `MPUT` commands, `null` is treated as the client working directory, according to the FTP protocol.
All other commands must be supplied with the `expression` to evaluate the remote path against the request message.
You can set the working directory with the `FTPClient.changeWorkingDirectory()` function when you extend the `DefaultFtpSessionFactory` and implement the `postProcessClientAfterConnect()` callback.

[[using-the-nlst-command]]
== Using the `nlst` Command

Version 5 introduced support for the `nlst` command.

`nlst` lists remote file names and supports only one option:

* `-f`: Do not sort the list

The message payload resulting from an `nlst` operation is a list of file names.

The remote directory that the `nlst` command acted on is provided in the `file_remoteDirectory` header.

Unlike the `-1` option for the xref:ftp/outbound-gateway.adoc#ftp-using-ls[`ls` command], which uses the `LIST` command, the `nlst` command sends an `NLST` command to the target FTP server.
This command is useful when the server does not support `LIST` (due to security restrictions, for example).
The result of the `nlst` operation is the names without other detail.
Therefore, the framework cannot determine if an entity is a directory, to perform filtering or recursive listing, for example.

[[using-the-get-command]]
== Using the `get` Command

`get` retrieves a remote file.
It supports the following option:

* `-P`: Preserve the timestamp of the remote file.
* `-stream`: Retrieve the remote file as a stream.

* `-D`: Delete the remote file after successful transfer.
The remote file is not deleted if the transfer is ignored, because the `FileExistsMode` is `IGNORE` and the local file already exists.

The `file_remoteDirectory` header provides the remote directory name, and the `file_remoteFile` header provides the file name.

The message payload resulting from a `get` operation is a `File` object that represents the retrieved file or an `InputStream` when you use the `-stream` option.
The `-stream` option allows retrieving the file as a stream.
For text files, a common use case is to combine this operation with a xref:file/splitter.adoc[file splitter] or a xref:transformer.adoc#stream-transformer[stream transformer].
When consuming remote files as streams, you are responsible for closing the `Session` after the stream is consumed.
For convenience, the `Session` is provided in the `closeableResource` header, which you can access with a convenience method on `IntegrationMessageHeaderAccessor`
The following example shows how to use the convenience method:

[source, java]
----
Closeable closeable = new IntegrationMessageHeaderAccessor(message).getCloseableResource();
if (closeable != null) {
    closeable.close();
}
----

Framework components such as the xref:file/splitter.adoc[file splitter] and the xref:transformer.adoc#stream-transformer[stream transformer] automatically close the session after the data is transferred.

The following example shows how to consume a file as a stream:

[source, xml]
----
<int-ftp:outbound-gateway session-factory="ftpSessionFactory"
                            request-channel="inboundGetStream"
                            command="get"
                            command-options="-stream"
                            expression="payload"
                            remote-directory="ftpTarget"
                            reply-channel="stream" />

<int-file:splitter input-channel="stream" output-channel="lines" />
----

NOTE: If you consume the input stream in a custom component, you must close the `Session`.
You can do so either in your custom code or by routing a copy of the message to a `service-activator` and using SpEL, as the following example shows:

[source, xml]
----
<int:service-activator input-channel="closeSession"
    expression="headers['closeableResource'].close()" />
----

[[using-the-mget-command]]
== Using the `mget` Command

`mget` retrieves multiple remote files based on a pattern and supports the following options:

* `-P`: Preserve the timestamps of the remote files.

* `-R`: Retrieve the entire directory tree recursively.

* `-x`: Throw an exception if no files match the pattern (otherwise an empty list is returned).

* `-D`: Delete each remote file after successful transfer.
The remote file is not deleted if the transfer is ignored, because the `FileExistsMode` is `IGNORE` and the local file already exists.

The message payload resulting from an `mget` operation is a `List<File>` object (that is, a `List` of `File` objects, each representing a retrieved file).

IMPORTANT: Starting with version 5.0, if the `FileExistsMode` is `IGNORE`, the payload of the output message no longer contains files that were not fetched due to the file already existing.
Previously, the list contained all files, including those that already existed.

The expression used to determine the remote path should produce a result that ends with `*` - e.g. `somedir/*` will fetch the complete tree under `somedir`.

Starting with version 5.0, a recursive `mget`, combined with the new `FileExistsMode.REPLACE_IF_MODIFIED` mode, can be used to periodically synchronize an entire remote directory tree locally.
This mode replaces the local file's last modified timestamp with the remote timestamp, regardless of the `-P` (preserve timestamp) option.

.Using recursion (`-R`)
[IMPORTANT]
=====
The pattern is ignored, and `*` is assumed.
By default, the entire remote tree is retrieved.
However, files in the tree can be filtered, by providing a `FileListFilter`.
Directories in the tree can also be filtered this way.
A `FileListFilter` can be provided by reference, by `filename-pattern`, or by `filename-regex` attributes.
For example, `filename-regex="(subDir|.*1.txt)"` retrieves all files ending with `1.txt` in the remote directory and the `subDir` child directory.
However, the next example shows an alternative, which version 5.0 made available.

If a subdirectory is filtered, no additional traversal of that subdirectory is performed.

The `-dirs` option is not allowed (the recursive `mget` uses the recursive `ls` to obtain the directory tree, so the directories themselves cannot be included in the list).

Typically, you would use the `#remoteDirectory` variable in the `local-directory-expression` so that the remote directory structure is retained locally.
=====

The persistent file list filters now have a boolean property `forRecursion`.
Setting this property to `true`, also sets `alwaysAcceptDirectories`, which means that the recursive operation on the outbound gateways (`ls` and `mget`) will now always traverse the full directory tree each time.
This is to solve a problem where changes deep in the directory tree were not detected.
In addition, `forRecursion=true` causes the full path to files to be used as the metadata store keys; this solves a problem where the filter did not work properly if a file with the same name appears multiple times in different directories.
IMPORTANT: This means that existing keys in a persistent metadata store will not be found for files beneath the top level directory.
For this reason, the property is `false` by default; this may change in a future release.

Starting with version 5.0, the `FtpSimplePatternFileListFilter` and `FtpRegexPatternFileListFilter` can be configured to always pass directories by setting the `alwaysAcceptDirectories` property to `true`.
Doing so allows recursion for a simple pattern, as the following examples show:

[source, xml]
----
<bean id="starDotTxtFilter"
        class="org.springframework.integration.ftp.filters.FtpSimplePatternFileListFilter">
    <constructor-arg value="*.txt" />
    <property name="alwaysAcceptDirectories" value="true" />
</bean>

<bean id="dotStarDotTxtFilter"
            class="org.springframework.integration.ftp.filters.FtpRegexPatternFileListFilter">
    <constructor-arg value="^.*\.txt$" />
    <property name="alwaysAcceptDirectories" value="true" />
</bean>
----

Once you have defined filters such as those in the preceding example, you can use one by setting the `filter` property on the gateway.

See also xref:ftp/outbound-gateway.adoc#ftp-partial[Outbound Gateway Partial Success (`mget` and `mput`)].

[[ftp-put-command]]
== Using the `put` Command

The `put` command sends a file to the remote server.
The payload of the message can be a `java.io.File`, a `byte[]`, or a `String`.
A `remote-filename-generator` (or expression) is used to name the remote file.
Other available attributes include `remote-directory`, `temporary-remote-directory`, and their `*-expression` equivalents: `use-temporary-file-name` and `auto-create-directory`.
See the https://github.com/spring-projects/spring-integration/tree/main/spring-integration-core/src/main/resources/org/springframework/integration/config[schema] documentation for more information.

The message payload resulting from a `put` operation is a `String` that represents the full path of the file on the server after transfer.

Version 5.2 introduced the `chmod` attribute, which changes the remote file permissions after upload.
You can use the conventional Unix octal format (for example, `600` allows read-write for the file owner only).
When configuring the adapter using java, you can use `setChmod(0600)`.
Only applies if your FTP server supports the `SITE CHMOD` subcommand.

[[using-the-mput-command]]
== Using the `mput` Command

The `mput` sends multiple files to the server and supports only one option:

* `-R`: Recursive.
Send all files (possibly filtered) in the directory and its subdirectories.

The message payload must be a `java.io.File` (or `String`) that represents a local directory.
Since version 5.1, a collection of `File` or `String` is also supported.

This command supports the same attributes as the xref:ftp/outbound-gateway.adoc#ftp-put-command[`put` command].
In addition, files in the local directory can be filtered with one of `mput-pattern`, `mput-regex`, `mput-filter`, or `mput-filter-expression`.
The filter works with recursion, as long as the subdirectories themselves pass the filter.
Subdirectories that do not pass the filter are not recursed.

The message payload resulting from an `mput` operation is a `List<String>` object (that is, a `List` of remote file paths that result from the transfer).

See also xref:ftp/outbound-gateway.adoc#ftp-partial[Outbound Gateway Partial Success (`mget` and `mput`)].

Version 5.2 introduced the `chmod` attribute, which lets you change the remote file permissions after upload.
You can use the conventional Unix octal format (for example, `600` allows read-write for the file owner only).
When configuring the adapter with Java, you can use `setChmodOctal("600")` or `setChmod(0600)`.
Only applies if your FTP server supports the `SITE CHMOD` subcommand.

[[using-the-rm-command]]
== Using the `rm` Command

The `rm` command removes files.

The `rm` command has no options.

The message payload resulting from an `rm` operation is `Boolean.TRUE` if the remove was successful or `Boolean.FALSE` otherwise.
The `file_remoteDirectory` header provides the remote directory, and the `file_remoteFile` header provides the file name.

[[using-the-mv-command]]
== Using the `mv` Command

The `mv` command moves files.

The `mv` command has no options.

The `expression` attribute defines the "`from`" path and the `rename-expression` attribute defines the "`to`" path.
By default, the `rename-expression` is `headers['file_renameTo']`.
This expression must not evaluate to null or an empty `String`.
If necessary, any necessary remote directories are created.
The payload of the result message is `Boolean.TRUE`.
The `file_remoteDirectory` header provides the original remote directory, and `file_remoteFile` header provides the file name.
The new path is in the `file_renameTo` header.

Starting with version 5.5.6, the `remoteDirectoryExpression` can be used in the `mv` command for convenience.
If the "`from`" file is not a full file path, the result of `remoteDirectoryExpression` is used as the remote directory.
The same applies for the "`to`" file, for example, if the task is just to rename a remote file in some directory.

[[additional-information-about-ftp-outbound-gateway-commands]]
== Additional Information about FTP Outbound Gateway Commands

The `get` and `mget` commands support the `local-filename-generator-expression` attribute.
It defines a SpEL expression to generate the name of local files during the transfer.
The root object of the evaluation context is the request message.
 The `remoteFileName` variable, which is particularly useful for `mget`, is also available -- for example, `local-filename-generator-expression="#remoteFileName.toUpperCase() + headers.something"`.

The `get` and `mget` commands support the `local-directory-expression` attribute.
It defines a SpEL expression to generate the name of local directories during the transfer.
The root object of the evaluation context is the request message but.
The `remoteDirectory` variable, which is particularly useful for `mget`, is also available -- for example: `local-directory-expression="'/tmp/local/' + #remoteDirectory.toUpperCase() + headers.something"`.
This attribute is mutually exclusive with the `local-directory` attribute.

For all commands, the 'expression' property of the gateway provides the path on which the command acts.
For the `mget` command, the expression might evaluate to '*', meaning to retrieve all files, or 'somedirectory/*', and so on.

The following example shows a gateway configured for an `ls` command:

[source,xml]
----
<int-ftp:outbound-gateway id="gateway1"
    session-factory="ftpSessionFactory"
    request-channel="inbound1"
    command="ls"
    command-options="-1"
    expression="payload"
    reply-channel="toSplitter"/>
----

The payload of the message sent to the `toSplitter` channel is a list of `String` objects that each contain the name of a file.
If the `command-options` attribute was omitted, it holds `FileInfo` objects.
It uses space-delimited options -- for example, `command-options="-1 -dirs -links"`.

Starting with version 4.2, the `GET`, `MGET`, `PUT` and `MPUT` commands support a `FileExistsMode` property (`mode` when using the namespace support).
This affects the behavior when the local file exists (`GET` and `MGET`) or the remote file exists (`PUT` and `MPUT`).
Supported modes are `REPLACE`, `APPEND`, `FAIL`, and `IGNORE`.
For backwards compatibility, the default mode for `PUT` and `MPUT` operations is `REPLACE`.
For `GET` and `MGET` operations, the default is `FAIL`.

Starting with version 5.0, the `setWorkingDirExpression()` (`working-dir-expression` in XML) option is provided on the `FtpOutboundGateway` (`<int-ftp:outbound-gateway>` in XML).
It lets you change the client working directory at runtime.
The expression is evaluated against the request message.
The previous working directory is restored after each gateway operation.

[[configuring-with-java-configuration]]
== Configuring with Java Configuration

The following Spring Boot application shows an example of how to configure the outbound gateway with Java configuration:

[source, java]
----
@SpringBootApplication
public class FtpJavaApplication {

    public static void main(String[] args) {
        new SpringApplicationBuilder(FtpJavaApplication.class)
            .web(false)
            .run(args);
    }

    @Bean
    public SessionFactory<FTPFile> ftpSessionFactory() {
        DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
        sf.setHost("localhost");
        sf.setPort(port);
        sf.setUsername("foo");
        sf.setPassword("foo");
        sf.setTestSession(true);
        return new CachingSessionFactory<FTPFile>(sf);
    }

    @Bean
    @ServiceActivator(inputChannel = "ftpChannel")
    public MessageHandler handler() {
        FtpOutboundGateway ftpOutboundGateway =
                          new FtpOutboundGateway(ftpSessionFactory(), "ls", "'my_remote_dir/'");
        ftpOutboundGateway.setOutputChannelName("lsReplyChannel");
        return ftpOutboundGateway;
    }

}
----

[[configuring-with-the-java-dsl]]
== Configuring with the Java DSL

The following Spring Boot application shows an example of how to configure the outbound gateway with the Java DSL:

[source, java]
----
@SpringBootApplication
public class FtpJavaApplication {

    public static void main(String[] args) {
        new SpringApplicationBuilder(FtpJavaApplication.class)
            .web(false)
            .run(args);
    }

    @Bean
    public SessionFactory<FTPFile> ftpSessionFactory() {
        DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
        sf.setHost("localhost");
        sf.setPort(port);
        sf.setUsername("foo");
        sf.setPassword("foo");
        sf.setTestSession(true);
        return new CachingSessionFactory<FTPFile>(sf);
    }

    @Bean
    public FtpOutboundGatewaySpec ftpOutboundGateway() {
        return Ftp.outboundGateway(ftpSessionFactory(),
            AbstractRemoteFileOutboundGateway.Command.MGET, "payload")
            .options(AbstractRemoteFileOutboundGateway.Option.RECURSIVE)
            .regexFileNameFilter("(subFtpSource|.*1.txt)")
            .localDirectoryExpression("'localDirectory/' + #remoteDirectory")
            .localFilenameExpression("#remoteFileName.replaceFirst('ftpSource', 'localTarget')");
    }

    @Bean
    public IntegrationFlow ftpMGetFlow(AbstractRemoteFileOutboundGateway<FTPFile> ftpOutboundGateway) {
        return f -> f
            .handle(ftpOutboundGateway)
            .channel(c -> c.queue("remoteFileOutputChannel"));
    }

}
----

[[ftp-partial]]
== Outbound Gateway Partial Success (`mget` and `mput`)

When you perform operations on multiple files (by using `mget` and `mput`), an exception can occur some time after one or more files have been transferred.
In this case (starting with version 4.2), a `PartialSuccessException` is thrown.
As well as the usual `MessagingException` properties (`failedMessage` and `cause`), this exception has two additional properties:

* `partialResults`: The successful transfer results.
* `derivedInput`: The list of files generated from the request message (for example, local files to transfer for an `mput`).

These attributes let you determine which files were successfully transferred and which were not.

In the case of a recursive `mput`, the `PartialSuccessException` may have nested `PartialSuccessException` occurrences.

Consider the following directory structure:

[source]
----
root/
|- file1.txt
|- subdir/
   | - file2.txt
   | - file3.txt
|- zoo.txt
----

If the exception occurs on `file3.txt`, the `PartialSuccessException` thrown by the gateway has `derivedInput` of `file1.txt`, `subdir`, and `zoo.txt` and `partialResults` of `file1.txt`.
Its `cause` is another `PartialSuccessException` with `derivedInput` of `file2.txt` and `file3.txt` and `partialResults` of `file2.txt`.

