Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-4116: Introduce FileAggregator (#3511)

Browse Source 
* INT-4116: Introduce FileAggregator

JIRA: https://jira.spring.io/browse/INT-4116

* Implement a `FileSplitter.FileMarker`-based aggregation strategies
and utilize them in a general `FileAggregator` component
* Make `HeaderAttributeCorrelationStrategy.attributeName` as `final`; add `Assert.notEmpty()`
* Fix `AggregatorFactoryBean` and `AggregatorSpec` to parse the provided processor
for possible `CorrelationStrategy` and/or `ReleaseStrategy`
* Introduce short-cut methods into Java & Kotlin DSL for an `aggregate()` configuration
* Introduce a `FileHeaders.LINE_COUNT` for header to be populated in the `FileSplitter`.
We need this info in the `FileAggregator` to avoid possible overhead with JSON deserialization
of the `FileSplitter.FileMarker` messages
* Test and document the feature
* Improve `FileSplitter` doc for code block switch (tabs)

* * Rework FileAggregator do not use Java Streams
This commit is contained in:
Artem Bilan
2021-03-16 15:06:36 -04:00
committed by GitHub
parent be2a698618
commit 4342586361
18 changed files with 719 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/reference/asciidoc/aggregator.adoc
View File

@@ -969,3 +969,5 @@ Flux<Message<?>> window =
.convertSendAndReceive(new Integer[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }, Flux.class);
----
====
See also <<./file.adoc#file-aggregator, File Aggregator>>.

208
src/reference/asciidoc/file.adoc
View File

@@ -363,7 +363,7 @@ Otherwise, the files from those events are treated the same way.
The `ResettableFileListFilter` implementations pick up the `ENTRY_DELETE` events.
Consequently, their files are provided for the `remove()` operation.
When this event is enabled, filters such as the `AcceptOnceFileListFilter` have the file removed
When this event is enabled, filters such as the `AcceptOnceFileListFilter` have the file removed.
As a result, if a file with the same name appears, it passes the filter and is sent as a message.
For this purpose, the `watch-events` property (`FileReadingMessageSource.setWatchEvents(WatchEventType... watchEvents)`) has been introduced.
@@ -990,10 +990,77 @@ However, it is only practical for relatively short files.
Inbound payloads can be `File`, `String` (a `File` path), `InputStream`, or `Reader`.
Other payload types are emitted unchanged.
The following listing shows all the possible attributes for `<int-file:splitter>`:
The following listing shows possible ways to configure a `FileSplitter`:
====
[source, xml]
[source, java, role="primary"]
.Java DSL
----
@SpringBootApplication
public class FileSplitterApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(FileSplitterApplication.class)
.web(false)
.run(args);
}
@Bean
public IntegrationFlow fileSplitterFlow() {
return IntegrationFlows
.from(Files.inboundAdapter(tmpDir.getRoot())
.filter(new ChainFileListFilter<File>()
.addFilter(new AcceptOnceFileListFilter<>())
.addFilter(new ExpressionFileListFilter<>(
new FunctionExpression<File>(f -> "foo.tmp".equals(f.getName()))))))
.split(Files.splitter()
.markers()
.charset(StandardCharsets.US_ASCII)
.firstLineAsHeader("fileHeader")
.applySequence(true))
.channel(c -> c.queue("fileSplittingResultChannel"))
.get();
}
}
----
[source, kotlin, role="secondary"]
.Kotlin DSL
----
@Bean
fun fileSplitterFlow() =
integrationFlow(
Files.inboundAdapter(tmpDir.getRoot())
.filter(
ChainFileListFilter<File?>()
.addFilter(AcceptOnceFileListFilter())
.addFilter(ExpressionFileListFilter(FunctionExpression { f: File? -> "foo.tmp" == f!!.name }))
)
) {
split(
Files.splitter()
.markers()
.charset(StandardCharsets.US_ASCII)
.firstLineAsHeader("fileHeader")
.applySequence(true)
)
channel { queue("fileSplittingResultChannel") }
}
----
[source, java, role="secondary"]
.Java
----
@Splitter(inputChannel="toSplitter")
@Bean
public MessageHandler fileSplitter() {
FileSplitter splitter = new FileSplitter(true, true);
splitter.setApplySequence(true);
splitter.setOutputChannel(outputChannel);
return splitter;
}
----
[source, xml, role="secondary"]
.XML
----
<int-file:splitter id="splitter" <1>
iterator="" <2>
@@ -1062,68 +1129,14 @@ When `markersJson` is true, the markers are represented as a JSON string (using
Version 5.0 introduced the `firstLineAsHeader` option to specify that the first line of content is a header (such as column names in a CSV file).
The argument passed to this property is the header name under which the first line is carried as a header in the messages emitted for the remaining lines.
This line is not included in the sequence header (if `applySequence` is true) nor in the `lineCount` associated with `FileMarker.END` .
This line is not included in the sequence header (if `applySequence` is true) nor in the `lineCount` associated with `FileMarker.END`.
NOTE: Starting with version 5.5, the lineCount` is also included as a `FileHeaders.LINE_COUNT` into headers of the `FileMarker.END` message, since the `FileMarker` could be serialized into JSON.
If a file contains only the header line, the file is treated as empty and, therefore, only `FileMarker` instances are emitted during splitting (if markers are enabled -- otherwise, no messages are emitted).
By default (if no header name is set), the first line is considered to be data and becomes the payload of the first emitted message.
If you need more complex logic about header extraction from the file content (not first line, not the whole content of the line, not one particular header, and so on), consider using <<./content-enrichment.adoc#header-enricher,header enricher>> ahead of the `FileSplitter`.
Note that the lines that have been moved to the headers might be filtered downstream from the normal content process.
==== Configuring with Java Configuration
The following Spring Boot application shows an example of how to configure a file splitter with Java configuration:
====
[source, java]
----
@Splitter(inputChannel="toSplitter")
@Bean
public MessageHandler fileSplitter() {
FileSplitter splitter = new FileSplitter(true, true);
splitter.setApplySequence(true);
splitter.setOutputChannel(outputChannel);
return splitter;
}
----
====
==== Configuring with the Java DSL
The following Spring Boot application shows an example of how to configure a file splitter with the Java DSL:
====
[source, java]
----
@SpringBootApplication
public class FileSplitterApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(FileSplitterApplication.class)
.web(false)
.run(args);
}
@Bean
public IntegrationFlow fileSplitterFlow() {
return IntegrationFlows
.from(Files.inboundAdapter(tmpDir.getRoot())
.filter(new ChainFileListFilter<File>()
.addFilter(new AcceptOnceFileListFilter<>())
.addFilter(new ExpressionFileListFilter<>(
new FunctionExpression<File>(f -> "foo.tmp".equals(f.getName()))))))
.split(Files.splitter()
.markers()
.charset(StandardCharsets.US_ASCII)
.firstLineAsHeader("fileHeader")
.applySequence(true))
.channel(c -> c.queue("fileSplittingResultChannel"))
.get();
}
}
----
====
[[idempotent-file-splitter]]
==== Idempotent Downstream Processing a Split File
@@ -1168,6 +1181,83 @@ public IntegrationFlow flow() {
----
====
[[file-aggregator]]
=== File Aggregator
Starting with version 5.5, a `FileAggregator` is introduced to cover other side of `FileSplitter` use-case when START/END markers are enabled.
For convenience the `FileAggregator` implements all three sequence details strategies:
- The `HeaderAttributeCorrelationStrategy` with the `FileHeaders.FILENAME` attribute is used for correlation key calculation.
When markers are enabled on the `FileSplitter`, it does not populate sequence details headers, since START/END marker messages are also included into the sequence size.
The `FileHeaders.FILENAME` is still populated for each line emitted, including START/END marker messages.
- The `FileMarkerReleaseStrategy` - checks for `FileSplitter.FileMarker.Mark.END` message in the group and then compare a `FileHeaders.LINE_COUNT` header value with the group size minus `2` - `FileSplitter.FileMarker` instances.
- The `FileAggregatingMessageGroupProcessor` just removes `FileSplitter.FileMarker` messages from the group and collect the rest of messages into a list payload to produce.
The following listing shows possible ways to configure a `FileAggregator`:
====
[source, java, role="primary"]
.Java DSL
----
@Bean
public IntegrationFlow fileSplitterAggregatorFlow(TaskExecutor taskExecutor) {
return f -> f
.split(Files.splitter()
.markers()
.firstLineAsHeader("firstLine"))
.channel(c -> c.executor(taskExecutor))
.filter(payload -> !(payload instanceof FileSplitter.FileMarker),
e -> e.discardChannel("aggregatorChannel"))
.<String, String>transform(String::toUpperCase)
.channel("aggregatorChannel")
.aggregate(new FileAggregator())
.channel(c -> c.queue("resultChannel"));
}
----
[source, kotlin, role="secondary"]
.Kotlin DSL
----
@Bean
fun fileSplitterAggregatorFlow(taskExecutor: TaskExecutor?) =
integrationFlow {
split(Files.splitter().markers().firstLineAsHeader("firstLine"))
channel { executor(taskExecutor) }
filter<Any>({ it !is FileMarker }) { discardChannel("aggregatorChannel") }
transform(String::toUpperCase)
channel("aggregatorChannel")
aggregate(FileAggregator())
channel { queue("resultChannel") }
}
----
[source, java, role="secondary"]
.Java
----
@serviceActivator(inputChannel="toAggregateFile")
@Bean
public AggregatorFactoryBean fileAggregator() {
AggregatorFactoryBean aggregator = new AggregatorFactoryBean();
aggregator.setProcessorBean(new FileAggregator());
aggregator.setOutputChannel(outputChannel);
return aggregator;
}
----
[source, xml, role="secondary"]
.XML
----
<int:chain input-channel="input" output-channel="output">
<int-file:splitter markers="true"/>
<int:aggregator>
<bean class="org.springframework.integration.file.aggregator.FileAggregator"/>
</int:aggregator>
</int:chain>
----
====
If default behavior of the `FileAggregator` does not satisfy the target logic, it is recommended to configure an aggregator endpoint with individual strategies.
See `FileAggregator` JavaDocs for more information.
[[remote-persistent-flf]]
=== Remote Persistent File List Filters

4
src/reference/asciidoc/splitter.adoc
View File

@@ -142,6 +142,4 @@ List<LineItem> extractItems(Order order) {
----
====
See also <<./handler-advice.adoc#advising-with-annotations,Advising Endpoints Using Annotations>>.
See also <<./dsl.adoc#java-dsl-splitters,Splitters>> in the Java DSL chapter.
See also <<./handler-advice.adoc#advising-with-annotations,Advising Endpoints Using Annotations>>, <<./dsl.adoc#java-dsl-splitters,Splitters>> and <<./file.adoc#file-splitter, File Splitter>>.

6
src/reference/asciidoc/whats-new.adoc
View File

@@ -15,6 +15,12 @@ If you are interested in more details, see the Issue Tracker tickets that were r
[[x5.5-new-components]]
=== New Components
[[x5.5-file-aggregator]]
==== File Aggregator
A `FileSplitter.FileMaker`-based implementation of `CorrelationStrategy`, `ReleaseStrategy` and `MessageGroupProcessor` as a `FileAggregator` component was introduced.
See <<./file.adoc#file-aggregator, File Aggregator>> for more information.
[[x5.5-general]]
=== General Changes