Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-4297: Add FileSplitter.setFirstLineAsHeader()

Browse Source 
JIRA: https://jira.spring.io/browse/INT-4297

* Add `FileSplitter.setFirstLineAsHeader()` and extract the first line from
the context before `iterator`.
* Populate the extracted header to each subsequent line

* Don't use `Streams` to read first line
* Add XSD and Java DSL option for the `firstLineAsHeader`
* Document the feature
* Add Java DSL sample for the `Files.slitter()`

Add a note about more complex header enrichment

Doc Polishing and fix field name.
This commit is contained in:
Artem Bilan
2017-06-26 16:53:25 -04:00
committed by Gary Russell
parent 5216dc2b02
commit d8382e0bd0
9 changed files with 243 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
src/reference/asciidoc/file.adoc
View File

@@ -911,12 +911,13 @@ Other payload types will be emitted unchanged.
apply-sequence="" <5>
requires-reply="" <6>
charset="" <7>
input-channel="" <8>
output-channel="" <9>
send-timeout="" <10>
auto-startup="" <11>
order="" <12>
phase="" /> <13>
first-line-as-header="" <8>
input-channel="" <9>
output-channel="" <10>
send-timeout="" <11>
auto-startup="" <12>
order="" <13>
phase="" /> <14>
----
<1> The bean name of the splitter.
@@ -948,32 +949,21 @@ Default: `false`.
<7> Set the charset name to be used when reading the text data into `String` payloads.
Default: platform charset.
<8> Set the input channel used to send messages to the splitter.
<8> The header name for the first line to be carried as a header in the messages emitted for the remaining lines.
Since _version 5.0_.
<9> Set the output channel to which messages will be sent.
<9> Set the input channel used to send messages to the splitter.
<10> Set the send timeout - only applies if the `output-channel` can block - such as a full `QueueChannel`.
<10> Set the output channel to which messages will be sent.
<11> Set to `false` to disable automatically starting the splitter when the context is refreshed.
<11> Set the send timeout - only applies if the `output-channel` can block - such as a full `QueueChannel`.
<12> Set to `false` to disable automatically starting the splitter when the context is refreshed.
Default: `true`.
<12> Set the order of this endpoint if the `input-channel` is a `<publish-subscribe-channel/>`.
<13> Set the order of this endpoint if the `input-channel` is a `<publish-subscribe-channel/>`.
<13> Set the startup phase for the splitter (used when `auto-startup` is `true`).
*Java Configuration*
[source, java]
----
@Splitter(inputChannel="toSplitter")
@Bean
public MessageHandler fileSplitter() {
FileSplitter splitter = new FileSplitter(true, true);
splitter.setApplySequence(true);
splitter.setOutputChannel(outputChannel);
return splitter;
}
----
<14> Set the startup phase for the splitter (used when `auto-startup` is `true`).
The `FileSplitter` will also split any text-based `InputStream` into lines.
When used in conjunction with an FTP or SFTP streaming inbound channel adapter, or an FTP or SFTP outbound gateway
@@ -990,3 +980,61 @@ public FileSplitter(boolean iterator, boolean markers, boolean markersJson)
----
When `markersJson` is true, the markers will be represented as a JSON string, as long as a suitable JSON processor library, such as Jackson or Boon, is on the classpath.
Starting with _version 5.0_, the `firstLineAsHeader` option is introduced to specify that the first line of content is a header (such as column names in a CSV file).
The argument passed to this property is the header name under which the first line will be carried as a header in the messages emitted for the remaining lines.
This line is not included in the sequence header (if `applySequence` is true) nor in the `FileMarker.END` `lineCount`.
If file contains only the header line, the file is treated as empty and therefore only `FileMarker` s are emitted during splitting (if markers are enabled, otherwise no messages are emitted).
By default (if no header name is set), the first line is considered data and will be the payload of the first emitted message.
If you need more complex logic about headers extraction from the file content (not first line, not the whole content of the line, not one header etc.), consider to use <<header-enricher, Header Enricher>> upfront of the `FileSplitter`.
The lines which have been moved to the headers might be filtered downstream from the normal content process.
==== Configuring with Java Configuration
[source, java]
----
@Splitter(inputChannel="toSplitter")
@Bean
public MessageHandler fileSplitter() {
FileSplitter splitter = new FileSplitter(true, true);
splitter.setApplySequence(true);
splitter.setOutputChannel(outputChannel);
return splitter;
}
----
==== Configuring with the Java DSL
The following Spring Boot application provides an example of configuring the inbound adapter using the Java DSL:
[source, java]
----
@SpringBootApplication
public class FileSplitterApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(FileSplitterApplication.class)
.web(false)
.run(args);
}
@Bean
public IntegrationFlow fileSplitterFlow() {
return IntegrationFlows
.from(Files.inboundAdapter(tmpDir.getRoot())
.filter(new ChainFileListFilter<File>()
.addFilter(new AcceptOnceFileListFilter<>())
.addFilter(new ExpressionFileListFilter<>(
new FunctionExpression<File>(f -> "foo.tmp".equals(f.getName()))))))
.split(Files.splitter()
.markers()
.charset(StandardCharsets.US_ASCII)
.firstLineAsHeader("fileHeader")
.applySequence(true))
.channel(c -> c.queue("fileSplittingResultChannel"))
.get();
}
}
----

2
src/reference/asciidoc/whats-new.adoc
View File

@@ -147,6 +147,8 @@ They also now support setting file permissions on the newly written file.
A new `FileSystemMarkerFilePresentFileListFilter` is now available; see <<file-incomplete>> for more information.
The `FileSplitter` now provides a `firstLineAsHeader` option to carry the first line of content as a header in the messages emitted for the remaining lines.
See <<files>> for more information.
==== (S)FTP Changes