Resolves https://github.com/spring-projects/spring-integration-aws/issues/190
Swallows `ProvisionedThroughputExceededException` while checkpointing
exhausted shards to avoid the `ShardConsumer` from not being marked as
closed and therefore be left in an inconsistent state which will only
throw exceptions as the `shardIterator` would be `null` and the
`ShardConsumer` wouldn't be marked as `CLOSED`.
If we don't have a connection to AWS or our thread is interrupted
for some reason, we should just re-throw that exception as is.
* Only catch a `LockNotGrantedException` which really indicates
that the state of the lock record has been changed.
Therefore we also have to reset local state and try again
Related to https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/148
If a `DynamoDbLock` has been locked before,
it contains a `lockItem` indicating that we can send a heart-beat
the next tine when we would like to re-lock again instead of calling
regular lock and fail because the lock record exists already.
On the other hand the heart-beat can fail by itself for many reasons
including the case when record in DB was removed somehow.
* Change the `DynamoDbLock.doLock()` logic to catch `sendHeartBeat()`
exception and reset local state to let it to try to lock again with
the regular `tryAcquireLock()` API
Related to https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/148
The current implementation has a flaw when it uses a distributed lock
for an exclusive access to shard for consuming only once at start up.
Such a behavior cause the problem when we have a network glitch at
runtime, so the lock is broken, but consumer is still active to retry
consumption attempts
* Add `renewLockIfAny()` logic ot the `ShardConsumer`, so we ensure
that we are still a lock holder and don't consume otherwise
* Add `unlockFuture` logic to block the `ShardConsumer.stop()`
until we really got lock unlocked.
Otherwise we end up with the race condition when we are still
stopping, but already ready to start a new consumer for the same
shard
Resolves: https://github.com/spring-projects/spring-integration-aws/issues/183
Exposes `BillingMode` option on the `DynamoDbMetadataStore` and
`DynamoDbLockRegistry`.
Moves the `LockRegistry` table creation to inside
this project as the DynamoDB Lock client doesn't support pay per request
model and that library seems abandoned.
Fixes https://github.com/spring-projects/spring-integration-aws/issues/184
During creating `KclMessageDrivenChannelAdapter`
if `config == null` a default `KinesisClientLibConfiguration` is created
with parameter `leaseCleanupIntervalMillis == 0` which causes an `IllegalArgumentExcpetoin`
when KPL is initialized
* Use default reasonable values for lease clean up intervals.
* Copy respective constants from the `KinesisClientLibConfiguration` into the `KclMessageDrivenChannelAdapter`
since they are `private` over there
Fixes https://github.com/spring-projects/spring-integration-aws/issues/177
* When the end of a shard is detected, checkpoint with the `endingSequenceNumber`
* Add a test
* code review
* Default to directly checkpointing closed shards except when in manual checkpoint mode
* update tests
* `endingSequenceNumber` should be higher than the last record's sequence number
* Checkpoint when in manual mode if shard was empty
Fixes https://github.com/spring-projects/spring-integration-aws/issues/158
It turns out that closed shards are still candidates for consumption
when we have concurrency
* Remove closed shards from the `shardConsumerManager.shardOffsetsToConsumer`
* Upgrade to Spring Integration 5.2.3, Jackson 2.10.2
Related to https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/117
It can be useful in the target application to have a full control
over KCL options to configure a `KclMessageDrivenChannelAdapter`
* Expose a `KinesisClientLibConfiguration`-based ctor in the
`KclMessageDrivenChannelAdapter`
* Add assert in setters when options must be configured on
the provided `KclMessageDrivenChannelAdapter`
* Extract `stream` and `group` options from the provided
`KclMessageDrivenChannelAdapter`
Fixes https://github.com/spring-projects/spring-integration-aws/issues/157
It turns out that `implementation` provides a `runtime` dependency in
Maven's POM, which is not what we would expect to behave as an `optional`
* Apply Gradle variants for those dependencies which should be as
`optional` in the generate POM
* Some Gradle build config refactoring
