Commit Graph

249 Commits

Author SHA1 Message Date
Artem Bilan
1da944ece8 Use Localstack 0.2.25
* Upgrade some other dependencies including Gradle
* Fix compatibility with the latest Localstack
2020-12-14 12:41:55 -05:00
acm19
f8ca0e4056 GH-183: Expose BillingMode option
Resolves: https://github.com/spring-projects/spring-integration-aws/issues/183

Exposes `BillingMode` option on the `DynamoDbMetadataStore` and
`DynamoDbLockRegistry`. 
Moves the `LockRegistry` table creation to inside
this project as the DynamoDB Lock client doesn't support pay per request
model and that library seems abandoned.
2020-12-11 12:01:00 -05:00
Refal
c99698dfb6 GH-184: Correct default values in KclMDChAdapter
Fixes https://github.com/spring-projects/spring-integration-aws/issues/184

During creating `KclMessageDrivenChannelAdapter`
if `config == null` a default `KinesisClientLibConfiguration` is created 
with parameter `leaseCleanupIntervalMillis == 0` which causes an `IllegalArgumentExcpetoin` 
when KPL is initialized

* Use default reasonable values for lease clean up intervals.
* Copy respective constants from the `KinesisClientLibConfiguration` into the `KclMessageDrivenChannelAdapter`
since they are `private` over there
2020-12-01 10:05:42 -05:00
Spring Buildmaster
a0d0752cdc [artifactory-release] Next development version 2020-10-29 15:53:44 +00:00
Spring Buildmaster
705b56a7a9 [artifactory-release] Release version 2.3.4.RELEASE 2020-10-29 15:53:40 +00:00
Artem Bilan
10d4652c21 Upgrade dependencies; prepare for release
Fixes https://github.com/spring-projects/spring-integration-aws/issues/162

* Fix deprecations
2020-10-29 11:45:34 -04:00
Greg Eales
8a959d0db4 Fix example in README for AdjacentParentShardId 2020-10-21 09:43:29 -04:00
Greg Eales
2b25b7e5d7 GH-177: Checkpoint closed shards for Kinesis
Fixes https://github.com/spring-projects/spring-integration-aws/issues/177

* When the end of a shard is detected, checkpoint with the `endingSequenceNumber`
* Add a test
* code review
* Default to directly checkpointing closed shards except when in manual checkpoint mode
* update tests
* `endingSequenceNumber` should be higher than the last record's sequence number
* Checkpoint when in manual mode if shard was empty
2020-10-20 13:25:00 -04:00
Greg Eales
5a3c6a1a95 GH-179: Kinesis: Add "shards to consume" filter
Fixes https://github.com/spring-projects/spring-integration-aws/issues/179

* code review
* rename to `shardListFilter` and add example to README
2020-10-20 12:59:44 -04:00
Greg Eales
c53bdfbf4c GH-175: Iterate listShards() pages
Fixes https://github.com/spring-projects/spring-integration-aws/issues/175

`KinesisMessageDrivenChannelAdapter` doesn't read from all shards

* Add loop to get all pages of shards from Kinesis
* Upgrade to the latest SC-AWS
2020-10-19 13:08:46 -04:00
sean-madden
ee79a1c632 GH-173: Use DEFAULT_DDB_BILLING_MODE instead of null
Fixes https://github.com/spring-projects/spring-integration-aws/issues/173
2020-09-10 19:42:22 -04:00
Spring Buildmaster
e42f4b62f4 [artifactory-release] Next development version 2020-08-17 16:52:17 +00:00
Spring Buildmaster
b0bc9cc958 [artifactory-release] Release version 2.3.3.RELEASE 2020-08-17 16:52:13 +00:00
Artem Bilan
99af572d68 Upgrade dependencies; prepare for release
* Fix compatibility with the latest KCL
2020-08-17 12:46:36 -04:00
Anton Malinovskiy
e0df55a6c3 GH-167: Fix consumer start for close/open shards
Fixes https://github.com/spring-projects/spring-integration-aws/issues/167

A combo of closing / opening shards leads to consumer not starting
2020-08-10 14:16:59 -04:00
Matthias
10e96e2de2 GH-163: Replace describeStream with listStream
Fixes https://github.com/spring-projects/spring-integration-aws/issues/163
2020-06-15 14:18:06 -04:00
Spring Buildmaster
a4e551d720 [artifactory-release] Next development version 2020-06-01 14:27:25 +00:00
Spring Buildmaster
7f206099ad [artifactory-release] Release version 2.3.2.RELEASE 2020-06-01 14:27:21 +00:00
artembilan
c53b927784 Upgrade dependencies; prepare for release 2020-06-01 10:08:58 -04:00
Artem Bilan
bc4a1c7c59 DynamoDbLockRegistry: Don't use managed threads
https://stackoverflow.com/questions/62025839/dynamodb-unlock-errors-during-application-stop-when-spring-kinesis-binder-tries

It turns out that we may call `unlock()` on the `DynamoDbLock`
when application context is already shouted down.
In this case any managed `TaskExecutor`s are out of use, too.

* Fix `DynamoDbLockRegistry` to use independent single-use
thread for unlock operations when main thread is already
interrupted
2020-05-26 15:37:32 -04:00
xfrancois
94e4481e6f GH-160: Custom endpoint for S3Session.getHostPort
Fixes https://github.com/spring-projects/spring-integration-aws/issues/160

* Set endpoint with setter instead of constructor

* Add unit test

* Edit getHostPortWithEndpoint test

* Add check to verify endpoint when receiving a message
2020-04-06 11:42:43 -04:00
Artem Bilan
6b82fde8a1 GH-158: Remove CLOSED shards from locking
Fixes https://github.com/spring-projects/spring-integration-aws/issues/158

It turns out that closed shards are still candidates for consumption
when we have concurrency
* Remove closed shards from the `shardConsumerManager.shardOffsetsToConsumer`
* Upgrade to Spring Integration 5.2.3, Jackson 2.10.2
2020-02-04 16:00:13 -05:00
Spring Buildmaster
55b12dc4a8 [artifactory-release] Next development version 2020-01-10 15:47:35 +00:00
Spring Buildmaster
9ac5bfdd69 [artifactory-release] Release version 2.3.1.RELEASE 2020-01-10 15:47:31 +00:00
Artem Bilan
3159b4f7d1 Prepare for release 2020-01-10 10:44:06 -05:00
Artem Bilan
5c3537d756 Expose KclMDChannelAdapter.getConsumerGroup() 2019-12-26 14:40:20 -05:00
Artem Bilan
112da0085c Upgrade dependencies; prepare for release 2019-12-26 14:14:30 -05:00
Artem Bilan
3a64f35162 Expose KinesisClientLibConfiguration ctor for KCL
Related to https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues/117

It can be useful in the target application to have a full control
over KCL options to configure a `KclMessageDrivenChannelAdapter`

* Expose a `KinesisClientLibConfiguration`-based ctor in the
`KclMessageDrivenChannelAdapter`
* Add assert in setters when options must be configured on
the provided `KclMessageDrivenChannelAdapter`
* Extract `stream` and `group` options from the provided
`KclMessageDrivenChannelAdapter`
2019-12-26 13:42:05 -05:00
Artem Bilan
b290e54362 GH-157: Fix optional dependencies for pom
Fixes https://github.com/spring-projects/spring-integration-aws/issues/157

It turns out that `implementation` provides a `runtime` dependency in
Maven's POM, which is not what we would expect to behave as an `optional`

* Apply Gradle variants for those dependencies which should be as
`optional` in the generate POM
* Some Gradle build config refactoring
2019-12-02 15:05:51 -05:00
Spring Buildmaster
9aa3ba4d0b [artifactory-release] Next development version 2019-11-27 15:02:37 +00:00
Spring Buildmaster
1d8180915f [artifactory-release] Release version 2.3.0.RELEASE 2019-11-27 15:02:34 +00:00
Artem Bilan
936d1cda00 Upgrade dependencies; prepare for release 2019-11-27 09:57:30 -05:00
Artem Bilan
64d6162d8f Disable KplKclIntegrationTests on Windows
It turns out that KPL doesn't support Windows any more.

See: https://github.com/awslabs/amazon-kinesis-producer/issues/284
2019-11-21 09:59:16 -05:00
Artem Bilan
ad8c47658a Upgrade dependencies 2019-11-20 21:54:26 -05:00
Artem Bilan
d83f63644a Tweak publications for custom artifacts 2019-11-20 21:06:02 -05:00
Artem Bilan
e8b96a5786 Use artifactory plugin directly 2019-11-20 20:51:16 -05:00
Artem Bilan
2f1c1ed4ed Upgrade to Gradle 6; fix all the deprecations 2019-11-20 17:49:55 -05:00
Artem Bilan
496d8e92e2 Remove allowMissingJavadoc for JavadocMethod
It looks like Checkstyle starting with version 8.25 doesn't provide an
`allowMissingJavadoc` for `JavadocMethod` module any more
2019-10-30 10:36:29 -04:00
Artem Bilan
32753c6181 Upgrade some dependencies 2019-10-30 10:21:13 -04:00
Artem Bilan
603c2da9f4 GH-113: Add KPL+KCL tests against Local Stack
Fixes https://github.com/spring-projects/spring-integration-aws/issues/113

* Fix some properties in the `KclMessageDrivenChannelAdapter` for a
reasonable default value
* Introduce `LocalStackSslEnvironmentProvider` to populate a `USE_SSL`
ENV var into a Docker instance for Local Stack
* Add more factory methods into an `ExtendedDockerTestUtils`, especially
variants to obtain an HTTPS-based service clients
2019-10-23 09:16:44 -04:00
Artem Bilan
507f652122 Disable CBOR for AWS; enable Kinesis tests
* Since Local Stack Kinesis support is based on Kinesalite, it turns out
that `System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true");`
does the trick to make Kinesis integration tests to pass with Local Stack
in Docker
* Suppress `serial` warning in the `AwsIntegrationEvent` classes
* Make `junit-jupiter-api` as `testCompile` dependency
* Fix README in regards Local Stack support for testing
2019-10-22 15:04:19 -04:00
Artem Bilan
ab70876923 GH-112: Add KinesisShardEndedEvent
Fixes https://github.com/spring-projects/spring-integration-aws/issues/112

* Introduce an `AwsIntegrationEvent` hierarchy
* Emit a `KinesisShardEndedEvent` from the `KinesisMessageDrivenChannelAdapter`
when current `shardIterator` is returned as `null` indicating the end
of the shard
* NOTE: `KclMessageDrivenChannelAdapter` cannot emit such an event
unit upgrade to KCL v2
2019-10-21 15:08:17 -04:00
Artem Bilan
161b8349d3 Fix Checkstyle for unused imports 2019-10-21 13:59:51 -04:00
Artem Bilan
3db0453417 Rely on EVN variable for Docker host name
* Introduce `EnvironmentHostNameResolver` to resolve Docker host name
from the environment variable
* Make Local Stack tests conditional based on the ENV variable mentioned
above
* Disable `KinesisIntegrationTests` because of `502 Bad Gateway` for
 Kinesis service in Local Stack
* Disable
`DynamoDbLockRegistryLeaderInitiatorTests.testDistributedLeaderElection()` -
looks like two instances cannot interaction with table in Local Stack
concurrently properly
* Fix `DynamoDbLockRegistry` to catch and ignore `ResourceInUseException`
instead of re-throwing it without a reason
* Remove `junit-vintage-engine` - no JUnit 4 tests any more
2019-10-21 13:52:58 -04:00
Artem Bilan
f1800f64e2 randomizePorts = false for Local Stack in Docker 2019-10-21 11:11:29 -04:00
Artem Bilan
ef9267bff5 Add junit-vintage-engine for JUnit 4 tests 2019-10-21 11:05:43 -04:00
Artem Bilan
437cfcfd24 Fix Checkstyle for imports 2019-10-21 10:59:18 -04:00
Artem Bilan
f2fe714853 Try LocalStack with JUnit 4 2019-10-21 10:57:30 -04:00
Artem Bilan
3b2a0f7743 randomizePorts for Local Stack Docker 2019-10-18 21:38:23 -04:00
Artem Bilan
0a5242bc26 Add useJUnitPlatform() into Gradle build 2019-10-18 15:32:12 -04:00