Fixes https://github.com/spring-projects/spring-integration-aws/issues/108
* Add `CheckpointMode.periodic`
* Implement `CheckpointMode.periodic` in the `KinesisMessageDrivenChannelAdapter`
* Support all the `CheckpointMode`s in the `KclMessageDrivenChannelAdapter`
* Add `@author`, polishing and Copyrights
* Since internal `KinesisProxy` instance is fully based on the
`CredentialsProvider` provided in the `KinesisClientLibConfiguration`,
we don't have choice unless expose such an option for the
`KclMessageDrivenChannelAdapter`
* Make it as `DefaultAWSCredentialsProviderChain` for those ctors which
are based on the default client and add one more ctor argument for
externally provided clients
* Move an `executor` option to the setter and rely on the
`SimpleAsyncTaskExecutor` by default
* Do not copy `record.getData()` to a new array since
`record.getData().array()` is fully enough for the current version
relates to spring-cloud/spring-cloud-stream-binder-aws-kinesis#65, spring-cloud/spring-cloud-stream-binder-aws-kinesis#53, #99
Add UUID consumer generation
fixed author tag
upgraded kcl version to 2.0.5
fixed source file encoding
- added setter for checkpoint interval
- change application name (concatenate consumer group and stream name)
fixed javadoc
take into account requested changes from spring-integration
fixed code style issues
take into account requested changes from spring-integration
fixed code style issues
take into account requested changes from spring-integration
Fixesspring-projects/spring-integration-aws#106
It is really OK to have a `null` returned for the
`S3MessageHandler.objectAclExpression` with the meaning do not set any
ACL on the S3 object.
* Allow `null` for `objectAclExpression` evaluation result
* Optimize asserts skipping string concatenation until the `Supplier`
invocation
* Introduce a `S3MessageHandler.MessageS3ProgressListener` to have a
request message context in the `progressChanged` event case
Fixes https://github.com/spring-projects/spring-integration-aws/issues/97
This fixes an issue where `additionalTimeToWait` has a negative value
that is subtracted later by the Amazon DynamoDB client to
`leaseDuration`, making it impossible to acquire the lock if expired.
Also allow to use the existing `refreshPeriod` properties in this case.
* Use default refresh period in tryLock()
The 'refreshPeriod' value might not satisfy both tryLock() and lock()
requirements and should probably be used exclusively in the second case.
We can make the value used for tryLock() configurable as well in another
pull request.
* Rollback previous commit and add author name
There is a race condition when real consumers are started later in
different thread.
* Add `eventually()` matcher to check the `Map<KinesisShardOffset, ?> shardConsumers`
for particular keys to appear eventually
Since we don't have choice and wait at minimum `leaseDuration` to be
able to iterate in the Amazon LockClient at least two times and that is
going to be just a contract of this client, so we need to add up a
`timeout` for the `tryLock` to that `leaseDuration`
https://stackoverflow.com/questions/51428196/spring-aws-kinesis-binder-acquiring-and-releasing-lock-issues-in-dynamo-db-while
According the logic in the `AmazonDynamoDBLockClient.acquireLock()` we
need to let the loop to iterate at least twice to really get access to
the existing lock and check its expiration status
* Fix the `DynamoDbLockRegistry.tryLock()` to set a proper
`.withRefreshPeriod(0L)` and `.withRefreshPeriod(0L)` to let it pull
the DB until we reach a `leaseDuration` limit on the lock or its
expiration status
Fixes https://github.com/spring-projects/spring-integration-aws/issues/90
* Add an internal `ShardConsumerManager` which is responsible to initiate
a locking for the shard key in the provided consumer group and populating
a `ShardConsumer` if `tryLock()` is successful or no `LockRegistry` at all
* Additional logic is added to always iterate over candidate shards if
`tryLock()` on the matter is not successful.
This way the current `KinesisMessageDrivenChannelAdapter` picks up
those shards which have been locked by the consumer which has just
left a cluster and unlocked distributed locks
* Now all the shards are considered as candidates independently of the
`streams` or `shardOffsets` configuration
* Improve Kinesis tests performance
To be able to run Kinesalite and Dynalite npm services locally in
parallel, we need to provide unique ports for them.
Therefore a default `4567` is left for the Kinesalite, meanwhile the
`DynamoDbLocalRunning` is expecting a 4568 now
Since the contract of the `tryLock()` to wait as close to the provided
timeout as possible, then we should not wait for the `leasePeriod` as
minimum.
Also we should sleep in between attempts not more then provided timeout
If `DynamoDbLockRegistry` and `DynamoDbMetaDataStore` are not declared
as beans (or their `afterPropertiesSet()` is not called), they hanging on
the `awaitForActive()`
When we try to lock the DynamoDB item we need to iterate at least
the lease time, but we still don't need to sleep too much - maximum
the time requested for `tryLock()`
Fixes https://github.com/spring-projects/spring-integration-aws/issues/90
* The `KinesisMessageDrivenChannelAdapter` can now be supplied with the
`LockRegistry` (e.g. `DynamoDbLockRegistry`) and when stream-based
configuration is used, the channel adapter performs `tryLock()` for the
shard in the channel adapter consumer group.
Therefor only one listener in the group is able to consume from the shard
Note: there is no yet full support for rebalance functionality.
And such a feature can be implemented using Spring Cloud Bus with the
command to stop and start channel adapters when a new
`KinesisMessageDrivenChannelAdapter` arrives to the cluster
* GH-66: Add DynamoDbLockRegistry implementation
Fixes https://github.com/spring-projects/spring-integration-aws/issues/66
* * Remove `Lifecycle` from `DynamoDbLockRegistry` in favor of a thread
execution in the `afterPropertiesSet()`
* Fix `lock()` interruptibility logic
* * Remove `mavenLocal()` since the upstream PR is merged
* decrease an amount of expectations in the Kinesis test
* * Upgrade to SC-AWS-2.0.0.RC2
* * Add Docs for the `DynamoDbLockRegistry`
Fixesspring-projects/spring-integration-aws#85
* For better component management add
`SqsMessageDrivenChannelAdapter.getQueue()` to return what queues this
channel adapter is subscribed
* Upgrade to Gradle 4.6 and Checkstyle 8.8
Fixesspring-projects/spring-integration-aws#84Fixesspring-cloud/spring-cloud-stream-binder-aws-kinesis#38Fixesspring-cloud/spring-cloud-stream-binder-aws-kinesis#36
Even if AWS Client has some reconnect and retry mechanism, it can be
exhausted and no connection error is rethrown to the `KinesisMessageDrivenChannelAdapter`
anyway.
On the other hand the error can be thrown from the record processor -
the flow on the `outputChannel`.
* log the exception around AWS Client calls and let background process
to restore/retry
* log the exception around message to send to let the processor to move
to the next record or perform the next task
* null the current `task` in the `ShardConsumer` in the `finally` block
to avoid hanging the thread without ability to moving to some other state
without end-user interaction
* When perform the `batch` checkpoint, check the result and if it is
negative, consider such a situation as processed and skip records from
sending downstream
* Upgrade dependencies
* Use Log4J2 for tests logging
Relates spring-cloud-stream-binder-aws-kinesis/#34
* Introduce `KinesisMessageHeaderErrorMessageStrategy` to populate
`AwsHeaders.RAW_RECORD` to the `ErrorMessage` headers
* Improve `KinesisMessageDrivenChannelAdapter` for the `AttributeAccessor`.
Also `try...finally` the `processRecords()` to perform important tasks
independently of the `processRecords()` result
Fixes: spring-projects/spring-integration-aws#62
* Make `SnsMessageHandler extends AbstractAwsMessageHandler`
* Remove SNS Outbound Gateway variant since `SnsMessageHandler` covers
that part via `successChannel` and `failureChannel`
* Fix XSD for SQS and SNS
* Fix SQS and SNS tests according their logic changes
* Fix README for new changes
