Fixes: spring-projects/spring-integration-aws#62
* Make `SnsMessageHandler extends AbstractAwsMessageHandler`
* Remove SNS Outbound Gateway variant since `SnsMessageHandler` covers
that part via `successChannel` and `failureChannel`
* Fix XSD for SQS and SNS
* Fix SQS and SNS tests according their logic changes
* Fix README for new changes
Fixes: spring-cloud/spring-cloud-stream-binder-aws-kinesis#29
Previously the same header name has been used for sending and receiving
operations (e.g. Kinesis `stream`).
This causes collisions in streaming processes when we receive message
from the AWS and send it downstream to AWS.
The header presence has a precedence over configured property/expression.
Therefore we send message to the AWS (e.g. Kinesis) using wrong
destination or other correlation properties
* Add `AwsHeaders.RECEIVED_*` headers to avoid collisions
* Remove Jackson dependency since it is managed now properly by SC-AWS
* Upgrade to SI-5.0, SC-AWS-2.0, Gradle-4.4.1 and some Gradle plugins
* Add `Jackson` dependency for compatibility with SF
* Implement new API of super classes
* Fix deprecations
* Fix tests for new state of classes under test
* Rename XSD to version 2.0
* Always call provided `AsyncHandler` from the internal instance
in the `KinesisMessageHandler`
* Make the `KinesisMessageHandler.obtainAsyncHandler()` as generic method
* Rename `sendFailureChannel` property just to the `failureChannel`
since the real operation is `put` not send
* Add `AwsHeaders.SERVICE_RESULT` to represent the service result, e.g.
in case of `PutRecordsRequest` in the `KinesisMessageHandler` to
send on success the whole `PutRecordsResult`
* Fix README to reflect the current reality of the code
* Fix `KinesisMessageHandlerTests` for provided `AsyncHandler` verification
This is groundwork to allow usage of a failure channel within the Kinesis binder per.
This implementation is intended to be backward-compatible with respect to the current handling
of `AsyncHandler`. Client code can still provide an `AsyncHandler`,
but doing so precludes the usage of channels for successful or unsuccessful sends.
Renaming to AwsRequestFailureException
generic getasynchandler method
always delegate or build handler
Added readme docs and using channel for tests
Since the listener may take some time for records processing,
there is a possibility that checkpoint will be stored too late
after the process and thus we are able to get the same records
in different channel adapter for the same shard, even if they are
in the same consumer group and use shared `MetadataStore`
This solution is some compromise for the current state of things and
has to be reconsidered in the future in favor of proper rebalance and
shard leader election solution
As a workaround for the duplicate records an additional
`@IdempotentReceiver` approach can be used
* Upgrade to Gradle 4.2.1, Checkstyle 8.3, AssertJ 3.8.0
* Fix race condition in the `KinesisMessageDrivenChannelAdapterTests`
Fixesspring-projects/spring-integration-aws#72
* Document Kinesis Channel Adapters
* Fix some inconsistency in the `KinesisMessageHandler`
* Add integration test against `KinesisLocalRunning` `@Rule`
* Document testing against Kinesalite
Fixesspring-projects/spring-integration-aws#64
* Add `DynamoDbRunning` for testing against locally ran DynamoDB
* Upgrade to Gradle 4.0.1, SI-4.3.11
* Switch on some Checkstyle rules for tests
Resolvesspring-projects/spring-integration-aws#75
Since we can build S3 entity key any deep path,
e.g. `my_bucket/foo/bar/baz/file.name` and S3 Object `key` representation
is exactly the whole path without bucket name, we should treat only bucket
as a remote dir; the key should be as a file name
* Change `S3InboundFileSynchronizer` logic to extract bucket name before
performing `copyFileToLocalDirectory()`
* Change `S3StreamingMessageSource` to override the `S3FileInfo.remoteDirectory`
to the only bucket name after `poll()`.
* Use for both cases a new `S3Session.normalizeBucketName()` method
* Upgrade to Gradle 4.0 and some other `build.gradle` polishing
Fixesspring-projects/spring-integration-aws#70
* Following the AWS Kinesis recommendation about some sleep interval in
between request to the Kinesis service, add `idleBetweenPolls` option
to the `KinesisMessageDrivenChannelAdapter` to sleep in the
`ConsumerDispatcher` main loop after each execution cycle
* Make default `recordLimit` as `10000` - to fetch as much as possible
data per one GetRecords request
* Change `Lock` and `Condition` in the `ConsumerInvoker` to the
`Semaphore`.
We really may release permit from the `ShardConsumer` to perform
the `ConsumerInvoker` cycle before stepping in that cycle
Fixes: spring-projects/spring-integration-aws#67
* Use `org.springframework.util.StringUtils#trimLeadingCharacter()`
to remove any useless leading `/` in the bucket name
When shard iterator is expired (`ExpiredIteratorException`),
we have to restart from the stored sequence number
When shard iterator is throttled (`ProvisionedThroughputExceededException`),
we have to "sleep" for backoff period
* Catch `ExpiredIteratorException` on the `.amazonKinesis.getRecords()` request
and move `ShardConsumer` to the `ConsumerState.EXPIRED`.
This state is treated as `NEW`, therefore a fresh `amazonKinesis.getShardIterator()`
is performed
* Catch `ProvisionedThroughputExceededException` on the `.amazonKinesis.getRecords()`
request and move `ShardConsumer` to the `ConsumerState.SLEEP`
for the `KinesisMessageDrivenChannelAdapter.this.consumerBackoff` period
Fixes GH-22 (https://github.com/spring-projects/spring-integration-aws/issues/22)
Rework logic to the `dispatching` and tasks
* Add `LimitExceededException` and configurable retry logic for the `describeStream`
* Skip `CLOSED` shards which has been read before according stored `checkpoint`. Will be useful for `after resharding` algorithm
* Add `adapt to resharding` logic
* Add `KinesisMessageDrivenChannelAdapterTests` based on mocks
* Add Thread Affinity for `ShardConsumer`s via `concurrency` option
* Introduce `concurrency` option and `ConsumerInvoker`.
`ShardConsumer`s are now distributed between `ConsumerInvoker`s if `concurrency > 0`
`ConsumerInvoker`s are scheduled to the `ConsumerExecutor` as a `isLongLived` tasks
The concurrency is adjusted if there are no more `ShardConsumer`s to process (`STOP` state because of closed shard).
At the same time newly populated `ShardConsumer`s (e.g. after resharding) is distributed evenly between existing `ConsumerExecutor`s if `maxConcurrency` is exceeded.
Otherwise fresh `ConsumerExecutor` is started for new `ShardConsumer`
Fixes GH-54 (https://github.com/spring-projects/spring-integration-aws/issues/54)
For consistency with other `AbstractRemoteFileStreamingMessageSource` implementations (e.g. `FtpStreamingMessageSource`) rename `S3InboundStreamingMessageSource` to the `S3StreamingMessageSource`
Since there was no any Milestones for the `1.1` version yet, it is safe to do this renaming breaking change
Fixes GH-22 (https://github.com/spring-projects/spring-integration-aws/issues/22)
* Add `KinesisMessageHandler` based on the `AmazonKinesisAsync.putRecordAsync()` operation.
The logic is fully similar to the `KafkaProducerMessageHandler` since both protocols pursue the same behavior
* Add some Kinesis specific `AwsHeaders`
* Upgrade to Gradle 2.14.1
Provide delegates for the SC-AWS `SimpleMessageListenerContainer` `start/stop/isRunning` method for individual queues.
Mark `SqsMessageDrivenChannelAdapter` with `@ManagedResource` and `@IntegrationManagedResource` to make those operation accessible from JMX/ControlBus
Cover the change with test-case
Provide some upgrades
Fix test for NPE from AWS SDK around `lastModified`: https://github.com/aws/aws-sdk-java/issues/822
Fixes GH-45 (https://github.com/spring-projects/spring-integration-aws/issues/45)
S3InboundFileSynchronizer failing when processing through S3 directories - modified approach for splitPathToBucketAndKey to account for the fact that there will be more than one forward slash in the path when subdirectories are used in an S3 bucket. Set up split operation to use the limit parameter to force the bucket identifier into the first element and the remainder of the text (the key) into the second element.
* modified unit test to include subdirectories.
*fixed additional usages of the split operation without the match limit parameter. DRY'd up some code. Cleaned up unit tests.
* added @author annotation at project lead's request.
Add some polishing
Fixes GH-39 (https://github.com/spring-projects/spring-integration-aws/issues/39)
Amazon SNS HTTP notifications are sent with the `content-type` header as `text/plain`, not `application/json` as it is expected by the `MappingJackson2HttpMessageConverter` by default.
*Added `text/plain` as supported content type for SNS inbound notification
* Corrections after review
Fixes GH-36 (https://github.com/spring-projects/spring-integration-aws/issues/36)
* Add ability to set `queueStopTimeout` for `SimpleMessageListenerContainer` in `SqsMessageDrivenChannelAdapter`
* Update unit tests to test `queueStopTimeout`
* Ensure that we don't override the default `queueStopTimeout` with `0` `long`
* Upgrade to `SI-4.2.8`
Update tests in SqsMessageDrivenChannelAdapterParserTests to invoke setAutoStartup
Change setAutoStartUp to invoke super in SqsMessageDrivenChannelAdapter
Add override to SqsMessageDrivenChannelAdapter
Remove call to setAutoStartup in SqsMessageDrivenChannelAdapterParserTests
I many cases we deal in application just with simple logical name for the target AWS entities, e.g. `myQueue`, `testBucket`.
Actually they must be resolved into the physical resources against the current environment.
E.g. the same S3 `testBucket` ca be fully different in different regions.
The SQS queue must be resolved into the resources with the current `Stack` context.
JIRA: https://jira.spring.io/browse/INTEXT-221
Also apply Checkstyle for the project
Move testing to the `AssertJ`
Adjust `.travis.yml` for Gradle 2.13
Add `CODE_OF_CONDUCT.adoc`
JIRA: https://jira.spring.io/browse/INTEXT-51,
https://jira.spring.io/browse/INTEXT-194,
https://jira.spring.io/browse/INTEXT-199
* Add S3 Inbound Channel Adapter fully based on the `AbstractInboundFileSynchronizer`,
`AbstractInboundFileSynchronizingMessageSource` implementation.
* Provide the standard SI `RemoteFileTemplate` and `SessionFactory` abstractions implementations.
* Upgrade to Gradle-2.12
* Provide Namespace support
* Ensure everything with tests
* Remove legacy, redundant implementation, infrastructure code for it and test-cases.
Everything in favor of the new implementation.
* Add `<int-aws:s3-outbound-channel-adapter>` and `<int-aws:s3-outbound-gateway>`
components. Their parsers and tests for them.
* Remove the old S3 Outbound Channel Adapter implementation and its tests
Rename `AwsNamespaceHandler` properly
`git mv -f File file` does the trick
Add `.travis.yml` to enable Travis CI on PRs
Implement the `S3MessageHandler` based on the `TransferManager`.
Provide `upload`, `download` and `copy` operations.
Support single file upload/download as well as directory structure.
The `S3MessageHandler` may behave as `one-way`, as well as `request-reply` component.
The former is blocked, the second is async.
See test-cases and JavaDocs for more info.
JIRA: https://jira.spring.io/browse/INTEXT-197
* Modified the `org/springframework/integration/aws/config/xml/spring-integration-aws-1.0.xsd` to add `channelAdapterAttributes` to expose
`phase` attribute in `s3 outbound adapter`.
* Modified `AmazonS3OutboundChannelAdapterParserTests` - Added Unit Test Class to test the `channelAdapterAttributes`.
* Modified `s3-valid-outbound-cases.xml` - Added separate definition `withChannelAdapterAttributes` for phase attribute.
Polishing according PR comments
JIRA: https://jira.spring.io/browse/INTEXT-187
* Modified the `spring-integration-aws-1.0.xsd` to accommodate `request-handler-advice-chain`
in s3 outbound adapter after feedback.
* Modified `AmazonS3OutboundChannelAdapterParserTests` - Added Unit Test Class with Mock Object to test the request handler chain.
* Modified `s3-valid-outbound-cases.xml` - Added separate definition withHandlerChain with mock Operations.
JIRA: https://jira.spring.io/browse/INTEXT-141
Adding in a constructor to the `SqsMessageHandler` so that `QueueMessagingTemplate` can be configured independently.
Updated with comments
- Updated the `spring-integration-aws-1.0.xsd` to include the `queue-messaging-template` param.
- Updated the `SqsOutboundChannelAdapterParserTest` to look for the `queue-messaging-template` param, and validate that `sqs` and `resource-resolver-id` params are mutually exclusive to `queue-messaging-template`.
- Added tests for the parser.
- Added test for the new constructor.
Updated with feedback
- Validation for `sqs` parameter missing in xml config
- Made `SqsMessageHandlerWithQueueMessagingTemplateTests` use the same tests as `SqsMessageHandler` tests by moving the tests to a common parent class: `AbstractSqsMessageHandlerTest`
- Other minor fixes.
Whitespace and error message fixes.
Polishing
