//tag::ref-doc[]
= SFTP Source
This source application supports transfer of files using the SFTP protocol.
Files are transferred from the `remote` directory to the `local` directory where the app is deployed.
Messages emitted by the source are provided as a byte array by default. However, this can be
customized using the `--mode` option:
- *ref* Provides a `java.io.File` reference
- *lines* Will split files line-by-line and emit a new message for each line
- *contents* The default. Provides the contents of a file as a byte array
When using `--mode=lines`, you can also provide the additional option `--withMarkers=true`.
If set to `true`, the underlying `FileSplitter` will emit additional _start-of-file_ and _end-of-file_ marker messages before and after the actual data.
The payload of these 2 additional marker messages is of type `FileSplitter.FileMarker`. The option `withMarkers` defaults to `false` if not explicitly set.
See link:../../../functions/supplier/sftp-supplier/README.adoc[`sftp-supplier`] for advanced configuration options.
See also link:../../../functions/common/metadata-store-common/README.adoc[MetadataStore] options for possible shared persistent store configuration used to prevent duplicate messages on restart.
== Input
N/A (Fetches files from an SFTP server).
== Output
=== mode = contents
==== Headers:
* `Content-Type: application/octet-stream`
* `file_name: <file name>`
* `file_remoteFileInfo <file metadata>`
* `file_remoteHostPort: <host:port>`
* `file_remoteDirectory: <relative-path>`
* `file_remoteFile: <file-name>`
* `sftp_selectedServer: <server-key>` (if multi-source)
==== Payload:
A `byte[]` filled with the file contents.
=== mode = lines
==== Headers:
* `Content-Type: text/plain`
* `file_name: <file name>`
* `correlationId: <UUID>` (same for each line)
* `sequenceNumber: <n>`
* `sequenceSize: 0` (number of lines is not know until the file is read)
* `file_marker : <file marker>` (if with-markers is enabled)
==== Payload:
A `String` for each line.
The first line is optionally preceded by a message with a `START` marker payload.
The last line is optionally followed by a message with an `END` marker payload.
Marker presence and format are determined by the `with-markers` and `markers-json` properties.
=== mode = ref
==== Headers:
* `file_remoteHostPort: <host:port>`
* `file_remoteDirectory: <relative-path>`
* `file_remoteFile: <file-name>`
* `file_originalFile: <absolute-path-of-local-file>`
* `file_name <local-file-name>`
* `file_relativePath`
* `file_remoteFile: <remote-file-name>`
* `sftp_selectedServer: <server-key>` (if multi-source)
==== Payload:
A `java.io.File` object.
== Options
The **$$ftp$$** $$source$$ has the following options:
//tag::configuration-properties[]
Properties grouped by prefix:
=== file.consumer
$$markers-json$$:: $$When 'fileMarkers == true', specify if they should be produced as FileSplitter.FileMarker objects or JSON.$$ *($$Boolean$$, default: `$$true$$`)*
$$mode$$:: $$The FileReadingMode to use for file reading sources. Values are 'ref' - The File object, 'lines' - a message per line, or 'contents' - the contents as bytes.$$ *($$FileReadingMode$$, default: `$$<none>$$`, possible values: `ref`,`lines`,`contents`)*
$$with-markers$$:: $$Set to true to emit start of file/end of file marker messages before/after the data. Only valid with FileReadingMode 'lines'.$$ *($$Boolean$$, default: `$$<none>$$`)*
=== metadata.store.dynamo-db
$$create-delay$$:: $$Delay between create table retries.$$ *($$Integer$$, default: `$$1$$`)*
$$create-retries$$:: $$Retry number for create table request.$$ *($$Integer$$, default: `$$25$$`)*
$$read-capacity$$:: $$Read capacity on the table.$$ *($$Long$$, default: `$$1$$`)*
$$table$$:: $$Table name for metadata.$$ *($$String$$, default: `$$<none>$$`)*
$$time-to-live$$:: $$TTL for table entries.$$ *($$Integer$$, default: `$$<none>$$`)*
$$write-capacity$$:: $$Write capacity on the table.$$ *($$Long$$, default: `$$1$$`)*
=== metadata.store.gemfire
$$region$$:: $$Gemfire region name for metadata.$$ *($$String$$, default: `$$<none>$$`)*
=== metadata.store.jdbc
$$region$$:: $$Unique grouping identifier for messages persisted with this store.$$ *($$String$$, default: `$$DEFAULT$$`)*
$$table-prefix$$:: $$Prefix for the custom table name.$$ *($$String$$, default: `$$<none>$$`)*
=== metadata.store.mongo-db
$$collection$$:: $$MongoDB collection name for metadata.$$ *($$String$$, default: `$$metadataStore$$`)*
=== metadata.store.redis
$$key$$:: $$Redis key for metadata.$$ *($$String$$, default: `$$<none>$$`)*
=== metadata.store
$$type$$:: $$Indicates the type of metadata store to configure (default is 'memory'). You must include the corresponding Spring Integration dependency to use a persistent store.$$ *($$StoreType$$, default: `$$<none>$$`, possible values: `mongodb`,`gemfire`,`redis`,`dynamodb`,`jdbc`,`zookeeper`,`hazelcast`,`memory`)*
=== metadata.store.zookeeper
$$connect-string$$:: $$Zookeeper connect string in form HOST:PORT.$$ *($$String$$, default: `$$127.0.0.1:2181$$`)*
$$encoding$$:: $$Encoding to use when storing data in Zookeeper.$$ *($$Charset$$, default: `$$UTF-8$$`)*
$$retry-interval$$:: $$Retry interval for Zookeeper operations in milliseconds.$$ *($$Integer$$, default: `$$1000$$`)*
$$root$$:: $$Root node - store entries are children of this node.$$ *($$String$$, default: `$$/SpringIntegration-MetadataStore$$`)*
=== sftp.supplier
$$auto-create-local-dir$$:: $$Set to true to create the local directory if it does not exist.$$ *($$Boolean$$, default: `$$true$$`)*
$$delay-when-empty$$:: $$Duration of delay when no new files are detected.$$ *($$Duration$$, default: `$$1s$$`)*
$$delete-remote-files$$:: $$Set to true to delete remote files after successful transfer.$$ *($$Boolean$$, default: `$$false$$`)*
$$directories$$:: $$A list of factory "name.directory" pairs.$$ *($$String[]$$, default: `$$<none>$$`)*
$$factories$$:: $$A map of factory names to factories.$$ *($$Map<String, Factory>$$, default: `$$<none>$$`)*
$$fair$$:: $$True for fair rotation of multiple servers/directories. This is false by default so if a source has more than one entry, these will be received before the other sources are visited.$$ *($$Boolean$$, default: `$$false$$`)*
$$filename-pattern$$:: $$A filter pattern to match the names of files to transfer.$$ *($$String$$, default: `$$<none>$$`)*
$$filename-regex$$:: $$A filter regex pattern to match the names of files to transfer.$$ *($$Pattern$$, default: `$$<none>$$`)*
$$list-only$$:: $$Set to true to return file metadata without the entire payload.$$ *($$Boolean$$, default: `$$false$$`)*
$$local-dir$$:: $$The local directory to use for file transfers.$$ *($$File$$, default: `$$<none>$$`)*
$$max-fetch$$:: $$The maximum number of remote files to fetch per poll; default unlimited. Does not apply when listing files or building task launch requests.$$ *($$Integer$$, default: `$$<none>$$`)*
$$preserve-timestamp$$:: $$Set to true to preserve the original timestamp.$$ *($$Boolean$$, default: `$$true$$`)*
$$remote-dir$$:: $$The remote FTP directory.$$ *($$String$$, default: `$$/$$`)*
$$remote-file-separator$$:: $$The remote file separator.$$ *($$String$$, default: `$$/$$`)*
$$rename-remote-files-to$$:: $$A SpEL expression resolving to the new name remote files must be renamed to after successful transfer.$$ *($$Expression$$, default: `$$<none>$$`)*
$$stream$$:: $$Set to true to stream the file rather than copy to a local directory.$$ *($$Boolean$$, default: `$$false$$`)*
$$tmp-file-suffix$$:: $$The suffix to use while the transfer is in progress.$$ *($$String$$, default: `$$.tmp$$`)*
=== sftp.supplier.factory
$$allow-unknown-keys$$:: $$True to allow an unknown or changed key.$$ *($$Boolean$$, default: `$$false$$`)*
$$host$$:: $$The host name of the server.$$ *($$String$$, default: `$$localhost$$`)*
$$known-hosts-expression$$:: $$A SpEL expression resolving to the location of the known hosts file.$$ *($$Expression$$, default: `$$<none>$$`)*
$$pass-phrase$$:: $$Passphrase for user's private key.$$ *($$String$$, default: `$$<empty string>$$`)*
$$password$$:: $$The password to use to connect to the server.$$ *($$String$$, default: `$$<none>$$`)*
$$port$$:: $$The port of the server.$$ *($$Integer$$, default: `$$22$$`)*
$$private-key$$:: $$Resource location of user's private key.$$ *($$Resource$$, default: `$$<none>$$`)*
$$username$$:: $$The username to use to connect to the server.$$ *($$String$$, default: `$$<none>$$`)*
=== sftp.supplier.sort-by
$$attribute$$:: $$Attribute of the file listing entry to sort by (FILENAME, ATIME: last access time, MTIME: last modified time).$$ *($$Attribute$$, default: `$$<none>$$`)*
$$dir$$:: $$Sorting direction (ASC or DESC).$$ *($$Dir$$, default: `$$<none>$$`)*
//end::configuration-properties[]
== Examples
```
java -jar sftp_source.jar --sftp.supplier.remote-dir=foo --file.mode=lines --sftp.supplier.factory.host=sftpserver \
--sftp.supplier.factory.username=user --ftp.supplier.factory.password=pw --sftp.supplier.local-dir=/foo
```
//end::ref-doc[]
5ba0269a0b