Fixes: #244
The `UserRecordResult` has an `attempts()` property which might be used post-put request logic
in the output channel.
* Introduce a `KinesisResponse` extension to adapt a `UserRecordResult`.
This way the further flow logic may consult low-level result from KPL via existing `AwsHeaders.SERVICE_RESULT`
header in the reply message
* GH-241: Support changing metrics-level for `KclMessageDrivenChannelAdapter`
* Add test that `metricsLevel` is set correctly
* Modify test body and add author
* Add reference to instance variable
* Set `metricsFactory` to `NullMetricsFactory` if metrics level is `NONE`
Fixes https://github.com/spring-projects/spring-integration-aws/issues/238
The `shardIterator` from Kinesis reply for records is not a sequence number anymore in AWS SKD v2.
* Fix `rewindIteratorOnError()` logic to extract the highest offset from the current failed `GetRecordsResponse`
to see if we have checkpointed it or we need to re-request the current `shardIterator`
According with AWS SDK doc the nextMarker field is only returned when defining the delimiter query param:
> NextMarker
When the response is truncated (the IsTruncated element value in the response is true), you can use the key name in this field as the marker parameter in the subsequent request to get the next set of objects. Amazon S3 lists objects in alphabetical order.
Note
This element is returned only if you have the delimiter request parameter specified. If the response does not include the NextMarker element and it is truncated, you can use the value of the last Key element in the response as the marker parameter in the subsequent request to get the next set of object keys.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_ResponseElements
After upgrading my project to newest version of `spring-integration-aws` I've started facing this loop issue for buckets with more than `1000` items, where `isTruncated` is true and `nextMarker` is `null`.
* Update the `S3Session` code to use the `key` from last item as `nextMarker` in next the request when response `isTruncated`, avoiding infinite loop.
* Fix Checkstyle violations
* Add `aws-crt-client` to support latest S3 transfer protocol
* Explicitly set respective system properties for AWS HTTP clients
to avoid classpath conflicts
* Upgrade to Testcontainers `1.18.3`
* Simplify `LocalstackContainerTest` to rely on the `LocalStackContainer.getEndpoint()`
for all the AWS services requested
Fixes https://github.com/spring-projects/spring-integration-aws/issues/224
* Fix `SnsMessageBuilder` -> `SnsBodyBuilder` in the `SnsMessageHandler.onInit()` comment
for SpEL import registration
* Upgrade to Localstack `2.1.0`
* Rename `SnsMessageBuilderTests` -> `SnsBodyBuilderTests` for consistency of its purpose
The error in the `KinesisMessageDrivenChannelAdapter.ShardConsumer.processTask()`
might be also thrown just direct from the `amazonKinesis.getRecords(getRecordsRequest)`.
If it happens first time, the `ShardCheckpointer` is not initialized with sequence numbers.
Therefore, a condition `highestSequence.equals(lastCheckpoint)` may lead to NPE.
* Rework the logic in the `KinesisMessageDrivenChannelAdapter.ShardConsumer.rewindIteratorOnError()`
to deal with a `null` for `this.checkpointer.getHighestSequence()` and reuse the current `shardIterator`
in the next request if no any commits happened.
* Remove `ShardCheckpointer.firstSequenceInBatch` since this is exactly a meaning of `shardIterator` representation.
Related to: https://github.com/spring-projects/spring-integration-aws/issues/223
Fixes https://github.com/spring-projects/spring-integration-aws/issues/223
A `amazonKinesis.getRecords(getRecordsRequest).join()` throws `CompletionException`
where the target reason is in a `cause`
* Fix `KinesisMessageDrivenChannelAdapter.getRecords()` to `catch (CompletionException)`
and then process its `cause` respectively
* Upgrade to Spring Cloud AWS `3.0.0` GA
* Upgrade other deps to the latest versions
Fixes https://github.com/spring-projects/spring-integration-aws/issues/221
* Modify the logic of the `KinesisMessageDrivenChannelAdapter` to rewind
a shard iterator to the failed sequence for any errors.
A rewinding sequence is determined from extra properties in the `ShardCheckpointer`
* Remove `RequestShardForSequenceException` since more natural behavior to
react for any record processor error without end-user interaction
The `messageGroupId` and `messageDeduplicationId` properties
must be set into a `PublishRequest` when topic is in FIFO mode.
Otherwise, they must not be set and fully ignoring them is enough
* Simplify some code in the `KclMessageDrivenChannelAdapter`
* Tried to upgrade to LocalStack `2.0` - failed on S3 `createBucket`
with `Length Required` error.
Sounds like there was no LocalStack and AWS SDK integration testing
* Fix config mutation and its propagation down to the `Scheduler`
* Fix conversion from `ByteBuffer`
* Add `KclMessageDrivenChannelAdapterTests` to verify KCL in real action.
The test is slow enough because KCL has a long initialization phase
* Disable `KplKclIntegrationTests` back because KPL native daemon
makes some real calls to EC2 and does not understand credentials
from Testcontainers
The existing `com.amazonaws:dynamodb-lock-client` does not implement
a locking algorithm properly and there is no easy way to determine if
lock has been abandoned according to the current `leaseDuration` behavior.
* Implement `DynamoDbLockRepository` to interact with lock table in this manner:
- `lockKey` as a primary key
- `lockOwner` as a unique owner client for the lock
- `createdAt` just an info when the lock was created by the client
- `expiredAt` the time in epoch seconds how long the lock is treated as valid
- this `expiredAt` can be configured as a DynamoDb `TTL` feature
* The `DynamoDbLockRepository` uses a `leaseDuration` to calculate an `expiredAt`
and compares it with the current epoch seconds to see if lock is not valid anymore
* Rework `DynamoDbLockRegistry` to rely on the `DynamoDbLockRepository`
* Remove `com.amazonaws:dynamodb-lock-client` dependency as we don't need it anymore
* Implement `RenewableLockRegistry` contract for simple `expiredAt` update
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/186
The `DynamoDbLockClient` waits extra `leaseDuration` time in a loop breaking a `tryLock()` contract.
* Fix `DynamoDbLockRegistry.tryLock()` to decrease an actual `additionalTimeToWait` by `leaseDuration`,
so the target `DynamoDbLockClient` when it adds this `leaseDuration` will wait an actual timeout requested
by the `tryLock()` contract.
This way a `tryLock(0)` will definitely return immediately since we really are not interested in blocking
**cherry-pick to 2.5.x**
Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/187
The `ShardConsumer` performs a `getShardIterator()` request with some sequence number from checkpointer.
When the shard is closed, it fails with an `InvalidArgumentException` not letting us mark such a consumer a stopped.
* Catch `InvalidArgumentException` on the `getShardIterator()` and see if error is related to a closed shard.
Log such a message and proceed to a `STOP` state for the current `ShardConsumer`.
**Cherry-pick to `2.5.x`**
