Utilize an existing `consumer.useNativeDecoding` together with the
`listenerMode = batch` instead of artificial `rawRecords` mode.
Currently it requires `useNativeDecoding = true` anyway to get a
proper `List<Record>` payload downstream
Related to https://github.com/spring-projects/spring-integration-aws/issues/90
Since SI-AWS already added a `LockRegistry` support into the
`KinesisMessageDrivenChannelAdapter` that is going to be a good benefit
for the Kinesis Binder to support it as well and also expose a
`DynamoDbLockRegistry` properties for the external configuration
Also upgrade dependencies to be ready for the upcoming release
Update Docs for new features
Fixes: spring-cloud-stream-binder-aws-kinesis#52
The AWS Kinesis uses its own, server-side, algorithm to determine the
target shard for the record, therefore we don't need to use SCSt
partitioning functionality and just propagate a provided
`partitionKeyExpression` down to the `KinesisMessageHandler`
* Remove `starter` module as redundant
* Absorb `core` module to the `binder` directly - there is not going to
be some other Kinesis binder implementation
* Fix poms and Docs on the matter
* Upgrade dependencies to Milestones to be ready for releasing
* Add `@ConditionalOnMissingBean` for the `amazonKinesis` bean to be
able to override it in the tests, e.g. with mocks or against Kinesalite
* Make default `partitionKey` against `payload.hashCode()`
* Fix typo in the `KinesisStreamProvisioner`
* Fix `KinesisBinderTests` for compatibility with the latest SCSt
* Replace `log4j.properties` with the `logback-test.xml` to get a real
logging in tests
* Mention `Object.hashCode()` for partition key in the Docs
Provides the error channel to the underlying `KinesisMessageHandler`.
Resolvesspring-cloud/spring-cloud-stream-binder-aws-kinesis#10
Adding a test for the error channel
Fixing checkstyle errors
Fixing docs
Forgot a word
Add more Consumer and Producer properties:
* `sendTimeout`
* `listenerMode`
* `checkpointMode`
* `recordsLimit`
* `idleBetweenPolls`
* `consumerBackoff`
* Added producer properties and hooked them into the underlying configuration of `KinesisMessageHandler`
* Added consumer properties and hooked them into the underlying configuration of `KinesisMessageDrivenChannelAdapter`
* Added documentation
Resolvesspring-cloud/spring-cloud-stream-binder-aws-kinesis#9
Fixes: spring-cloud/spring-cloud-stream-binder-aws-kinesis#16
* Add missed `this.` prefix
* Rename `createTableDelay` and `createTableRetries` to the `createDelay`
and `createRetries` respectively
* Add `KinesisBinderTests.testPartitionCountIncreasedIfAutoAddPartitionsSet()`
to check the `autoAddShards` functionality
Fixes: spring-cloud/spring-cloud-stream-binder-aws-kinesis#16
Adding the ability for the binder to automatically create shards
on startup if the current stream's count is less than what is desired.
The properties that control this functionality and their defaults are as follows
```
spring.cloud.stream.kinesis.binder.autoAddShards=false
spring.cloud.stream.kinesis.binder.minShardCount=1
```
* Added the `autoAddShards` and `minShardCount` properties
and code to update stream shard count
* Added the `createTableDelay` and `createTableRetries` properties
to the checkpoint configuration
* Added documentation entries for the new properties
Resolvesspring-cloud/spring-cloud-stream-binder-aws-kinesis#18
Also adding in configuration for DynamoDb read and write capacity
Adds some documentation abut checkpoint properties
Extracts Checkpoint properties as a nested class
Fixes bad style formatting
Removes property check so a DynamoDb table is always used with the
Kinesis binder
* Fix Checkstyle plugin configuration to override the one from the
parent `spring-cloud-build`
Fixesspring-cloud/spring-cloud-stream-binder-aws-kinesis#11Fixesspring-cloud/spring-cloud-stream-binder-aws-kinesis#12
* Document how SCSt consumer groups are mapped to the AWS Kinesis support
* Implement partitioning logic on the consumer and producer sides
* Override distribution tests to consume all expected messages,
since there is no guarantee which shard will be used for the partition
key, we consume now from all the listeners until successful result
* Document partition mapping and logic
This pull request adds ability for producers and consumers to create a stream
if it isn't already existing.
I've used the property `spring.cloud.stream.kinesis.binder.autoCreateStreams`
to trigger the process.
I've intentionally kept the scope to auto-creation only and not included shard
manipulation in this PR.
Included are passing unit tests and an integration test that works if a local
Kinesalite instance is running.
I also have a example project at https://github.com/JacobASeverson/stream-example-kinesis
that I've used to verify the functionality within AWS.
* Added ability to auto-create streams to the provisioner
* Added unit testing for the provisioner new stream handler
* Added an integration test for binding
Moving tests to appropriate project
Removing , default to creation, and remove stream handler layer
Cleanup for integration test
Setting shard size appropriately for producer and consumer
Forgot to clean up pom experiment
Removed the `KinesisStream` abstraction and added level enabled checks for logging
Adding static credentials
Adding a configurable consumer property for startup timeouts
Adding documentation for , merging upstream, and new Checkstyle errors
Finishing binder test for auto creation of stream
Fix imports for tests
Getting rid of spring-cloud-stream-binder-test dep
* Polish dependencies
* Polishing code style
* Add Eclipse code formatter configs
