Issue #30 : Add S3 inbound streaming channel adapter.

Fixes GH-30 (https://github.com/spring-projects/spring-integration-aws/issues/30)

* Add S3InboundStreamingMessageSource and related S3FileInfo
* Extend spring-integration-aws-1.1.xsd with s3-inbound-streaming-channel-adapter tag.
* Add S3StreamingInboundChannelAdapterParser and s3-inbound-streaming-channel-adapter handler.
* Add S3 streaming spring tests.
* Add S3 streaming documentation to README.md.

Issue #30: Resolve code style and formatting issues

Polishing:
* Revert some `build.gradle` changes
* Add JavaDoc to the `S3FileInfo.getPermissions()`
This commit is contained in:
Christian Tzolov
2016-08-31 16:26:54 +02:00
committed by Artem Bilan
parent de4dfed908
commit da77e4b89e
10 changed files with 767 additions and 2 deletions

View File

@@ -115,6 +115,82 @@ An XML variant may look like:
</int-aws:s3-inbound-channel-adapter>
````
###Streaming Inbound Channel Adapter
This adapter produces message with payloads of type `InputStream`, allowing S3 objects to be fetched without writing to the local file system.
Since the session remains open, the consuming application is responsible for closing the session when the file has been consumed.
The session is provided in the closeableResource header (`IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE`). Standard framework components, such as the `FileSplitter` and `StreamTransformer` will automatically close the session.
The following Spring Boot application provides an example of configuring the S3 inbound streaming adapter using Java configuration:
````java
@SpringBootApplication
public class S3JavaApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(S3JavaApplication.class)
.web(false)
.run(args);
}
@Autowired
private AmazonS3 amazonS3;
@Bean
@InboundChannelAdapter(value = "s3Channel", poller = @Poller(fixedDelay = "100"))
public MessageSource<InputStream> s3InboundStreamingMessageSource() {
S3StreamingMessageSource messageSource = new S3StreamingMessageSource(template());
messageSource.setRemoteDirectory(S3_BUCKET);
messageSource.setFilter(new S3PersistentAcceptOnceFileListFilter(new SimpleMetadataStore(),
"streaming"));
return messageSource;
}
@Bean
@Transformer(inputChannel = "s3Channel", outputChannel = "data")
public org.springframework.integration.transformer.Transformer transformer() {
return new StreamTransformer();
}
@Bean
public S3RemoteFileTemplate template() {
return new S3RemoteFileTemplate(new S3SessionFactory(amazonS3));
}
@Bean
public PollableChannel s3Channel() {
return new QueueChannel();
}
}
````
An XML variant may look like:
````xml
<bean id="metadataStore" class="org.springframework.integration.metadata.SimpleMetadataStore"/>
<bean id="acceptOnceFilter" class="org.springframework.integration.aws.support.filters.S3PersistentAcceptOnceFileListFilter">
<constructor-arg index="0" ref="metadataStore"/>
<constructor-arg index="1" value="streaming"/>
</bean>
<bean id="s3SessionFactory" class="org.springframework.integration.aws.support.S3SessionFactory"/>
<int-aws:s3-inbound-streaming-channel-adapter channel="s3Channel"
session-factory="s3SessionFactory"
filter="acceptOnceFilter"
remote-directory-expression="'my_bucket'">
<int:poller fixed-rate="1000"/>
</int-aws:s3-inbound-streaming-channel-adapter>
````
Only one of `filename-pattern`, `filename-regex` or `filter` is allowed.
> NOTE: Unlike the non-streaming inbound channel adapter, this adapter does not prevent duplicates by default.
> If you do not delete the remote file and you wish to prevent the file being processed again, you can configure an `S3PersistentFileListFilter` in the `filter` attribute.
> If you dont actually want to persist the state, an in-memory `SimpleMetadataStore` can be used with the filter.
> If you wish to use a filename pattern (or regex) as well, use a `CompositeFileListFilter`.
###Outbound Channel Adapter
The S3 Outbound Channel Adapter is represented by the `S3MessageHandler` (`<int-aws:s3-outbound-channel-adapter>`