Fixesspring-projects/spring-integration-aws#70
* The `ConsumerDispatcher` is infinite, non-blocking loop:
add short `Thread.sleep(1)` to the loop to yield the CPU
* The `ConsumerInvoker` is infinite loop, too, but can be blocked:
add `Condition` to wait to the particular notification from the
`ShardConsumer` when it changes its state
Fixesspring-projects/spring-integration-aws#71
It isn't correct to call `converter.convert()` in the
`KinesisMessageHandler` if `payload` is already `ByteBuffer` or `byte[]`
On the other hand an application might be interested in the `byte[]`
payload on the consumer side.
The `KinesisMessageDrivenChannelAdapter` must be able to produce
`byte[]` `payload` without any conversion.
* Add logic into `KinesisMessageHandler` to check the `payload` type
before calling `converter.convert()`
* Allow to configure `converter` to `null` for the
`KinesisMessageDrivenChannelAdapter`.
The body of the consumed record is presented in the `payload` as is -
`byte[]`
* Add JavaDoc to the `KinesisMessageHandler.setConverter()`
Fixes: spring-projects/spring-integration-aws#67
* Use `org.springframework.util.StringUtils#trimLeadingCharacter()`
to remove any useless leading `/` in the bucket name
Fixesspring-projects/spring-integration-aws#63
The `CLOSED` shards have `endingSequenceNumber` value and they can't be
considered for consuming independently of the value in the `checkpoint`
* Introduce local `shardsToConsume` variable in the
`KinesisMessageDrivenChannelAdapter#populateShardsForStream()` to
store only `OPEN` shards for future consumption
* Fix the logic to determine the last `shardId` for a subsequent
`describeStream()` request based on the shards result exactly from
the `describeStreamResult.getStreamDescription().getShards()`, not
already filtered `shardsToConsume`
Fixes: spring-projects/spring-integration-aws#56
The upgrade to SC-AWS-1.2 RC1 resolves US_East_2 Region concern
since AWS SDK is polled in version `1.1.98`
Also fix deprecation warnings in S3 support components
When shard iterator is expired (`ExpiredIteratorException`),
we have to restart from the stored sequence number
When shard iterator is throttled (`ProvisionedThroughputExceededException`),
we have to "sleep" for backoff period
* Catch `ExpiredIteratorException` on the `.amazonKinesis.getRecords()` request
and move `ShardConsumer` to the `ConsumerState.EXPIRED`.
This state is treated as `NEW`, therefore a fresh `amazonKinesis.getShardIterator()`
is performed
* Catch `ProvisionedThroughputExceededException` on the `.amazonKinesis.getRecords()`
request and move `ShardConsumer` to the `ConsumerState.SLEEP`
for the `KinesisMessageDrivenChannelAdapter.this.consumerBackoff` period
Fixes: spring-projects/spring-integration-aws#61
With the `shutdownNow()` the currently ran task are marked with the
`Thread.interrupt()`.
The AWS SDK catch this situation and throws `AbortedException`
* To let currently ran task finish properly switch from the
`shutdownNow()` to the `shutdown()` for internal executors
The external Spring-based executors (`ExecutorConfigurationSupport`)
must be supplied with the `waitForTasksToCompleteOnShutdown` option
Fixes: spring-projects/spring-integration-aws#60
The attempts to extract `this.consumerInvokers.get(0)` when the `concurrency == 0` lead to the NPE
* Wrap `this.consumerInvokers.get(0)` block to the `if(concurrency != 0)`
`ConsumerInvoker` uses iterator over `consumers` property, which can be
concurrently modified in the `populateConsumer()`` method, thus triggering
`ConcurrentModificationException`:
```
Exception in thread "kinesisChannel-kinesis-consumer-1" java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ConsumerInvoker.run(KinesisMessageDrivenChannelAdapter.java:866)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
* Use a `ConcurrentLinkedQueue` instead of `ArrayList` to fix a concurrent Modification problem.
Fixes GH-22 (https://github.com/spring-projects/spring-integration-aws/issues/22)
Rework logic to the `dispatching` and tasks
* Add `LimitExceededException` and configurable retry logic for the `describeStream`
* Skip `CLOSED` shards which has been read before according stored `checkpoint`. Will be useful for `after resharding` algorithm
* Add `adapt to resharding` logic
* Add `KinesisMessageDrivenChannelAdapterTests` based on mocks
* Add Thread Affinity for `ShardConsumer`s via `concurrency` option
* Introduce `concurrency` option and `ConsumerInvoker`.
`ShardConsumer`s are now distributed between `ConsumerInvoker`s if `concurrency > 0`
`ConsumerInvoker`s are scheduled to the `ConsumerExecutor` as a `isLongLived` tasks
The concurrency is adjusted if there are no more `ShardConsumer`s to process (`STOP` state because of closed shard).
At the same time newly populated `ShardConsumer`s (e.g. after resharding) is distributed evenly between existing `ConsumerExecutor`s if `maxConcurrency` is exceeded.
Otherwise fresh `ConsumerExecutor` is started for new `ShardConsumer`
Fixes GH-54 (https://github.com/spring-projects/spring-integration-aws/issues/54)
For consistency with other `AbstractRemoteFileStreamingMessageSource` implementations (e.g. `FtpStreamingMessageSource`) rename `S3InboundStreamingMessageSource` to the `S3StreamingMessageSource`
Since there was no any Milestones for the `1.1` version yet, it is safe to do this renaming breaking change
Fixes GH-22 (https://github.com/spring-projects/spring-integration-aws/issues/22)
* Add `KinesisMessageHandler` based on the `AmazonKinesisAsync.putRecordAsync()` operation.
The logic is fully similar to the `KafkaProducerMessageHandler` since both protocols pursue the same behavior
* Add some Kinesis specific `AwsHeaders`
* Upgrade to Gradle 2.14.1
To avoid some local work with files, add `S3PersistentAcceptOnceFileListFilter` to the `S3InboundFileSynchronizer` as a default one
* Upgrade to Gradle 3.1
* Upgrade to SI-4.3.2
* Some `build.gradle` polishing
* Downgrade to Checkstyle-6.16.1 because of `RequireThis` bug
According changes in the `RequireThis`, the `this.` modifier is required now only in case of overlapping:
https://github.com/checkstyle/checkstyle/issues/3418
* Add `validateOnlyOverlapping = false` to reinstate the previous behavior, but we should think one more time if we really should be so strict with this rule
Provide delegates for the SC-AWS `SimpleMessageListenerContainer` `start/stop/isRunning` method for individual queues.
Mark `SqsMessageDrivenChannelAdapter` with `@ManagedResource` and `@IntegrationManagedResource` to make those operation accessible from JMX/ControlBus
Cover the change with test-case
Provide some upgrades
Fix test for NPE from AWS SDK around `lastModified`: https://github.com/aws/aws-sdk-java/issues/822
Fixes GH-45 (https://github.com/spring-projects/spring-integration-aws/issues/45)
S3InboundFileSynchronizer failing when processing through S3 directories - modified approach for splitPathToBucketAndKey to account for the fact that there will be more than one forward slash in the path when subdirectories are used in an S3 bucket. Set up split operation to use the limit parameter to force the bucket identifier into the first element and the remainder of the text (the key) into the second element.
* modified unit test to include subdirectories.
*fixed additional usages of the split operation without the match limit parameter. DRY'd up some code. Cleaned up unit tests.
* added @author annotation at project lead's request.
Add some polishing
Fixes GH-39 (https://github.com/spring-projects/spring-integration-aws/issues/39)
Amazon SNS HTTP notifications are sent with the `content-type` header as `text/plain`, not `application/json` as it is expected by the `MappingJackson2HttpMessageConverter` by default.
*Added `text/plain` as supported content type for SNS inbound notification
* Corrections after review
Fixes GH-36 (https://github.com/spring-projects/spring-integration-aws/issues/36)
* Add ability to set `queueStopTimeout` for `SimpleMessageListenerContainer` in `SqsMessageDrivenChannelAdapter`
* Update unit tests to test `queueStopTimeout`
* Ensure that we don't override the default `queueStopTimeout` with `0` `long`
* Upgrade to `SI-4.2.8`
Update tests in SqsMessageDrivenChannelAdapterParserTests to invoke setAutoStartup
Change setAutoStartUp to invoke super in SqsMessageDrivenChannelAdapter
Add override to SqsMessageDrivenChannelAdapter
Remove call to setAutoStartup in SqsMessageDrivenChannelAdapterParserTests
