Spring/spring-integration-aws
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

GH-22: Add KinesisMessageDrivenChannelAdapter

Browse Source 
Fixes GH-22 (https://github.com/spring-projects/spring-integration-aws/issues/22)

Rework logic to the `dispatching` and tasks

* Add `LimitExceededException` and configurable retry logic for the `describeStream`
* Skip `CLOSED` shards which has been read before according stored `checkpoint`. Will be useful for `after resharding` algorithm
* Add `adapt to resharding` logic
* Add `KinesisMessageDrivenChannelAdapterTests` based on mocks
* Add Thread Affinity for `ShardConsumer`s via `concurrency` option
* Introduce `concurrency` option and `ConsumerInvoker`.
  `ShardConsumer`s are now distributed between `ConsumerInvoker`s if `concurrency > 0`
  `ConsumerInvoker`s are scheduled to the `ConsumerExecutor` as a `isLongLived` tasks
  The concurrency is adjusted if there are no more `ShardConsumer`s to process (`STOP` state because of closed shard).
  At the same time newly populated `ShardConsumer`s (e.g. after resharding) is distributed evenly between existing `ConsumerExecutor`s if `maxConcurrency` is exceeded.
Otherwise fresh `ConsumerExecutor` is started for new `ShardConsumer`
This commit is contained in:
Artem Bilan
2016-10-19 17:47:55 -04:00
committed by Marius Bogoevici
parent 0e31261545
commit ef2f5f7aef
12 changed files with 1669 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
src/test/resources/log4j.properties
View File

@@ -2,7 +2,8 @@ log4j.rootCategory=WARN, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - <%m>%n
log4j.appender.stdout.layout.ConversionPattern=%d %5p %c{1} [%t] : %m%n
log4j.category.org.springframework=WARN
log4j.category.org.springframework.integration.aws=WARN
log4j.category.org.springframework.integration=WARN