Spring/spring-integration-aws
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
188 Commits 10 Branches 0 Tags
5fb1ec2738fc6e0e7522948e7a42e8bdfcaa67d0
Commit Graph

188 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Spring Buildmaster
5fb1ec2738 [artifactory-release] Next development version 2019-09-12 21:00:24 +00:00
Spring Buildmaster
0eccf48046 [artifactory-release] Release version 2.2.1.RELEASE 2019-09-12 21:00:21 +00:00
Artem Bilan
2d8f7d1d7e Fix race condition in the KinesisMDChannelAdapter 
The `KinesisMessageDrivenChannelAdapter` may exit its `active` state,
but `consumerInvokers` won't be cleared in the main consumer loop

**Cherry-pick to master**
 2019-09-12 16:58:02 -04:00
Artem Bilan
3860df6e74 Upgrade to dependency-management-plugin:1.0.8 2019-09-12 16:32:58 -04:00
Artem Bilan
0d9d2ff052 Destroy() must throws Exception 2019-09-10 13:46:00 -04:00
Artem Bilan
2219d3bc3b GH-151: Recreate KCL Worker on each restart 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/151

**Cherry-pick to 2.2.x**

# Conflicts:
#	src/main/java/org/springframework/integration/aws/inbound/kinesis/KclMessageDrivenChannelAdapter.java

* Upgrade to the latest dependency versions
 2019-09-10 13:44:23 -04:00
Artem Bilan
5e9f76a166 GH-151: Recreate KCL Worker on each restart 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/151

**Cherry-pick to 2.2.x**

# Conflicts:
#	src/main/java/org/springframework/integration/aws/inbound/kinesis/KclMessageDrivenChannelAdapter.java
 2019-09-10 13:40:37 -04:00
Artem Bilan
978e8deae9 Add HTTPS entries for XSD into spring.schemas 
Related to spring-projects/spring-integration#2987

**Cherry-pick to 2.1.x, 2.0.x & 1.1.x**
 2019-07-18 13:20:52 -04:00
Dirk Bonhomme
c4e288f7c8 GH-148: Check for periodic checkpoint mode properly 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/148

Check for configured checkpoint mode  before committing periodic checkpoint in KCL adapter

(cherry picked from commit 213d793369)
 2019-07-10 10:49:58 -04:00
Artem Bilan
60f0004afd Add bindSourceRecord for Kinesis Channel Adapters 2019-06-17 13:37:39 -04:00
Sergiu Pantiru
e6e2465846 GH-146 log sequenceNumber when consumer starts 2019-05-23 10:35:47 -04:00
Spring Buildmaster
83a588c5ab [artifactory-release] Next development version 2019-05-08 21:30:22 +00:00
Spring Buildmaster
e225123f50 [artifactory-release] Release version 2.2.0.RELEASE 2019-05-08 21:30:19 +00:00
Artem Bilan
0ae25ea826 Upgrade dependencies; prepare for release 2019-05-08 16:20:44 -04:00
Sebastian Dusza
9185cd186a Update messageDeletionPolicy section in README.md 
It wasn't up to date.
 2019-04-05 15:56:14 -04:00
javarno
aefb5c2522 GH-142: Configure workerId in KCL adapter 
* Allow to modify the worker identifier used in KCL adapter.

Fixes https://github.com/spring-projects/spring-integration-aws/issues/142
 2019-04-02 17:55:59 -04:00
Artem Bilan
f635861872 URL Cleanup 
This commit updates URLs to prefer the https protocol.
Redirects are not followed to avoid accidentally expanding intentionally
 shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed Success
These URLs were switched to an https URL with a 2xx status.
While the status was successful, your review is still recommended.

* [ ] http://www.apache.org/licenses/ with 1 occurrences migrated to:
  https://www.apache.org/licenses/ ([https](https://www.apache.org/licenses/) result 200).
* [ ] http://www.apache.org/licenses/LICENSE-2.0 with 72 occurrences migrated to:
  https://www.apache.org/licenses/LICENSE-2.0 ([https](https://www.apache.org/licenses/LICENSE-2.0) result 200).
 2019-03-25 13:38:12 -04:00
Artem Bilan
562b20bd4d Fix link for GH PullRequest help 2019-03-20 21:26:08 -04:00
Spring Operator
d38fd18d23 URL Cleanup 
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed But Review Recommended
These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended.

* [ ] http://queue-url.com/ (UnknownHostException) with 2 occurrences migrated to:
  https://queue-url.com/ ([https](https://queue-url.com/) result UnknownHostException).
* [ ] http://queue-url.com/bar (UnknownHostException) with 1 occurrences migrated to:
  https://queue-url.com/bar ([https](https://queue-url.com/bar) result UnknownHostException).
* [ ] http://queue-url.com/baz (UnknownHostException) with 1 occurrences migrated to:
  https://queue-url.com/baz ([https](https://queue-url.com/baz) result UnknownHostException).
* [ ] http://queue-url.com/foo (UnknownHostException) with 2 occurrences migrated to:
  https://queue-url.com/foo ([https](https://queue-url.com/foo) result UnknownHostException).
* [ ] http://help.github.com/send-pull-requests (404) with 1 occurrences migrated to:
  https://help.github.com/send-pull-requests ([https](https://help.github.com/send-pull-requests) result 404).

## Fixed Success
These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended.

* [ ] http://aws.amazon.com/ with 1 occurrences migrated to:
  https://aws.amazon.com/ ([https](https://aws.amazon.com/) result 200).
* [ ] http://aws.amazon.com/products/ with 3 occurrences migrated to:
  https://aws.amazon.com/products/ ([https](https://aws.amazon.com/products/) result 200).
* [ ] http://aws.amazon.com/ses/ with 2 occurrences migrated to:
  https://aws.amazon.com/ses/ ([https](https://aws.amazon.com/ses/) result 200).
* [ ] http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html with 1 occurrences migrated to:
  https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html ([https](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) result 200).
* [ ] http://docs.spring.io/spring-integration/reference/ with 1 occurrences migrated to:
  https://docs.spring.io/spring-integration/reference/ ([https](https://docs.spring.io/spring-integration/reference/) result 200).
* [ ] http://projects.spring.io/spring-integration/ with 1 occurrences migrated to:
  https://projects.spring.io/spring-integration/ ([https](https://projects.spring.io/spring-integration/) result 200).
* [ ] http://www.apache.org with 1 occurrences migrated to:
  https://www.apache.org ([https](https://www.apache.org) result 200).
* [ ] http://www.springframework.org/schema/integration/spring-integration.xsd with 1 occurrences migrated to:
  https://www.springframework.org/schema/integration/spring-integration.xsd ([https](https://www.springframework.org/schema/integration/spring-integration.xsd) result 200).
* [ ] http://aws.amazon.com/documentation/ses/ with 2 occurrences migrated to:
  https://aws.amazon.com/documentation/ses/ ([https](https://aws.amazon.com/documentation/ses/) result 301).
* [ ] http://aws.amazon.com/sdkforjava/ with 1 occurrences migrated to:
  https://aws.amazon.com/sdkforjava/ ([https](https://aws.amazon.com/sdkforjava/) result 301).
* [ ] http://contributor-covenant.org with 1 occurrences migrated to:
  https://contributor-covenant.org ([https](https://contributor-covenant.org) result 301).
* [ ] http://contributor-covenant.org/version/1/3/0/ with 1 occurrences migrated to:
  https://contributor-covenant.org/version/1/3/0/ ([https](https://contributor-covenant.org/version/1/3/0/) result 301).
* [ ] http://www.springframework.org with 1 occurrences migrated to:
  https://www.springframework.org ([https](https://www.springframework.org) result 301).

# Ignored
These URLs were intentionally ignored.

* http://localhost with 4 occurrences
* http://testQueue.amazonaws.com with 2 occurrences
* http://www.springframework.org/schema/beans with 1 occurrences
* http://www.springframework.org/schema/integration with 2 occurrences
* http://www.springframework.org/schema/integration/aws with 2 occurrences
* http://www.springframework.org/schema/tool with 2 occurrences
* http://www.w3.org/2001/XMLSchema with 1 occurrences
 2019-03-20 21:25:00 -04:00
Spring Operator
5d27d7e3da URL Cleanup 
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed But Review Recommended
These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended.

* http://www.puppycrawl.com/dtds/configuration_1_2.dtd (404) with 1 occurrences migrated to:
  https://www.puppycrawl.com/dtds/configuration_1_2.dtd ([https](https://www.puppycrawl.com/dtds/configuration_1_2.dtd) result 404).
* http://www.puppycrawl.com/dtds/suppressions_1_1.dtd (404) with 1 occurrences migrated to:
  https://www.puppycrawl.com/dtds/suppressions_1_1.dtd ([https](https://www.puppycrawl.com/dtds/suppressions_1_1.dtd) result 404).
* http://www.springframework.org/schema/cloud/aws/messaging/spring-cloud-aws-messaging.xsd (404) with 7 occurrences migrated to:
  https://www.springframework.org/schema/cloud/aws/messaging/spring-cloud-aws-messaging.xsd ([https](https://www.springframework.org/schema/cloud/aws/messaging/spring-cloud-aws-messaging.xsd) result 404).
* http://www.springframework.org/schema/integration/aws/spring-integration-aws.xsd (404) with 12 occurrences migrated to:
  https://www.springframework.org/schema/integration/aws/spring-integration-aws.xsd ([https](https://www.springframework.org/schema/integration/aws/spring-integration-aws.xsd) result 404).

## Fixed Success
These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended.

* http://www.springframework.org/schema/beans/spring-beans.xsd with 12 occurrences migrated to:
  https://www.springframework.org/schema/beans/spring-beans.xsd ([https](https://www.springframework.org/schema/beans/spring-beans.xsd) result 200).
* http://www.springframework.org/schema/integration/spring-integration.xsd with 4 occurrences migrated to:
  https://www.springframework.org/schema/integration/spring-integration.xsd ([https](https://www.springframework.org/schema/integration/spring-integration.xsd) result 200).

# Ignored
These URLs were intentionally ignored.

* http://www.springframework.org/schema/beans with 24 occurrences
* http://www.springframework.org/schema/cloud/aws/messaging with 14 occurrences
* http://www.springframework.org/schema/integration with 8 occurrences
* http://www.springframework.org/schema/integration/aws with 24 occurrences
* http://www.w3.org/2001/XMLSchema-instance with 12 occurrences
 2019-03-20 12:58:12 -04:00
Artem Bilan
51714181db Upgrade to SI-5.1.4 2019-03-20 12:09:11 -04:00
Artem Bilan
f1b3d65d7e Rename XSD to 2.2; Fix HTTPS for Checkstyle 2019-03-20 11:35:51 -04:00
Arnaud Lecollaire
79ccf55659 GH-108: Handle all CheckpointMode in KclMDrChAd 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/108

* Add `CheckpointMode.periodic`
* Implement `CheckpointMode.periodic` in the `KinesisMessageDrivenChannelAdapter`
* Support all the `CheckpointMode`s in the `KclMessageDrivenChannelAdapter`

* Add `@author`, polishing and Copyrights
 2019-03-14 15:43:37 -04:00
Artem Bilan
b1cad7e55a Add CredentialsProvider for KCP Channel Adapter 
* Since internal `KinesisProxy` instance is fully based on the
`CredentialsProvider` provided in the `KinesisClientLibConfiguration`,
we don't have choice unless expose such an option for the
`KclMessageDrivenChannelAdapter`
* Make it as `DefaultAWSCredentialsProviderChain` for those ctors which
are based on the default client and add one more ctor argument for
externally provided clients
* Move an `executor` option to the setter and rely on the
`SimpleAsyncTaskExecutor` by default
* Do not copy `record.getData()` to a new array since
`record.getData().array()` is fully enough for the current version
 2019-03-14 10:31:40 -04:00
Artem Bilan
dd9f3a7925 Downgrade to KCL 1.9.3 
* Upgrade to DynamoDB Lock Client 1.1
* Use current releases for SI and SC-AWS
* Fix compatibility with current SI
 2019-03-13 10:30:10 -04:00
Spring Operator
1b671378d7 URL Cleanup 
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener).

# Fixed URLs

## Fixed Success
These URLs were fixed successfully.

* http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/ migrated to:
  https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/ ([https](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/) result 200).
* http://spring.io migrated to:
  https://spring.io ([https](https://spring.io) result 200).
* http://www.apache.org/licenses/LICENSE-2.0.txt migrated to:
  https://www.apache.org/licenses/LICENSE-2.0.txt ([https](https://www.apache.org/licenses/LICENSE-2.0.txt) result 200).
* http://repo.spring.io/libs-milestone migrated to:
  https://repo.spring.io/libs-milestone ([https](https://repo.spring.io/libs-milestone) result 302).
* http://repo.spring.io/libs-staging-local migrated to:
  https://repo.spring.io/libs-staging-local ([https](https://repo.spring.io/libs-staging-local) result 302).
* http://repo.spring.io/plugins-release migrated to:
  https://repo.spring.io/plugins-release ([https](https://repo.spring.io/plugins-release) result 302).
 2019-03-06 09:36:41 -05:00
Artem Bilan
53597170e4 Prepare 2.2 version 
* Upgrade to SI-5.2, Gradle 5.2.1, Checkstyle 8.18
* Make Jackson dependency as mandatory
* Use `awaitility` assertions
 2019-03-01 16:08:49 -05:00
Artem Bilan
82c0135669 GH-116: Document optional dependencies 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/116
 2019-03-01 11:50:28 -05:00
Artem Bilan
e5acf0f4de Upgrade dependencies; fix for compatibility 2019-02-25 13:14:44 -05:00
Sylvain Marcadal
5c27dfade5 Add KCL/KPL message handler and channel adapter 
relates to spring-cloud/spring-cloud-stream-binder-aws-kinesis#65, spring-cloud/spring-cloud-stream-binder-aws-kinesis#53, #99

Add UUID consumer generation

fixed author tag

upgraded kcl version to 2.0.5

fixed source file encoding

- added setter for checkpoint interval
- change application name (concatenate consumer group and stream name)

fixed javadoc

take into account requested changes from spring-integration

fixed code style issues

take into account requested changes from spring-integration

fixed code style issues

take into account requested changes from spring-integration
 2019-02-11 11:20:16 -05:00
Artem Bilan
0995564a66 GH-106: Allow null for S3MH.objectAclExpression 
Fixes spring-projects/spring-integration-aws#106

It is really OK to have a `null` returned for the
`S3MessageHandler.objectAclExpression` with the meaning do not set any
ACL on the S3 object.

* Allow `null` for `objectAclExpression` evaluation result
* Optimize asserts skipping string concatenation until the `Supplier`
invocation
* Introduce a `S3MessageHandler.MessageS3ProgressListener` to have a
request message context in the `progressChanged` event case
 2019-01-15 10:16:01 -05:00
Artem Bilan
432d8b640e Start 2.1.0 version 2018-10-30 16:32:20 -04:00
Artem Bilan
a3dd4763c6 Add note about project version and its deps 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/100
 2018-10-15 14:30:49 -04:00
Artem Bilan
0ea00784f0 Use default LocalStack ports 
According to https://github.com/localstack/localstack,
DynamoDB listens at `4569` by default, and Kinesis listens at `4568`
 by default.

Aligning with that setup means that it is simpler to run these tests
without having to tune the LocalStack ports.

See also: https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/pull/64
 2018-10-08 10:27:09 -04:00
Karl Lessard
ed89f1243f GH-97: wait at least leaseDuration to acquire the lock 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/97

This fixes an issue where `additionalTimeToWait` has a negative value
that is subtracted later by the Amazon DynamoDB client to
`leaseDuration`, making it impossible to acquire the lock if expired.

Also allow to use the existing `refreshPeriod` properties in this case.

* Use default refresh period in tryLock()

The 'refreshPeriod' value might not satisfy both tryLock() and lock()
requirements and should probably be used exclusively in the second case.

We can make the value used for tryLock() configurable as well in another
pull request.

* Rollback previous commit and add author name
 2018-09-21 15:46:04 -04:00
Artem Bilan
857d184cca Fix KinesisMessageDrivenChannelAdapterTests 
There is a race condition when real consumers are started later in
different thread.

* Add `eventually()` matcher to check the `Map<KinesisShardOffset, ?> shardConsumers`
for particular keys to appear eventually
 2018-09-19 13:07:58 -04:00
skelly201
6e5e9f191b Add SqsMessageHandler DestinationResolver ctor 2018-09-19 11:27:27 -04:00
Spring Buildmaster
4476c09f61 [artifactory-release] Next development version 2018-08-21 17:34:32 +00:00
Spring Buildmaster
4043c150d1 [artifactory-release] Release version 2.0.0.RELEASE 2018-08-21 17:34:27 +00:00
Artem Bilan
2cbe87f90d Fix README typos 2018-08-01 16:32:40 -04:00
Spring Buildmaster
7c1e3f83fa [artifactory-release] Next development version 2018-07-30 14:33:53 +00:00
Spring Buildmaster
33048d892e [artifactory-release] Release version 2.0.0.RC1 2018-07-30 14:33:47 +00:00
Artem Bilan
19d11057c2 Upgrade to the latest dependencies 2018-07-27 17:00:16 -04:00
Artem Bilan
1e1e1acc39 Fix DynamoDbLockRegistry.tryLock for "fair" time 
Since we don't have choice and wait at minimum `leaseDuration` to be
able to iterate in the Amazon LockClient at least two times and that is
going to be just a contract of this client, so we need to add up a
`timeout` for the `tryLock` to that `leaseDuration`
 2018-07-20 12:44:51 -04:00
Artem Bilan
dd52e8f304 Fix DynamoDbLockReg for LockClient requirements 
https://stackoverflow.com/questions/51428196/spring-aws-kinesis-binder-acquiring-and-releasing-lock-issues-in-dynamo-db-while

According the logic in the `AmazonDynamoDBLockClient.acquireLock()` we
need to let the loop to iterate at least twice to really get access to
the existing lock and check its expiration status

* Fix the `DynamoDbLockRegistry.tryLock()` to set a proper
`.withRefreshPeriod(0L)` and `.withRefreshPeriod(0L)` to let it pull
the DB until we reach a `leaseDuration` limit on the lock or its
expiration status
 2018-07-19 18:21:23 -04:00
Artem Bilan
0e02b0d470 GH-90: Rework locking logic in the KinesisMDChA 
Fixes https://github.com/spring-projects/spring-integration-aws/issues/90

* Add an internal `ShardConsumerManager` which is responsible to initiate
a locking for the shard key in the provided consumer group and populating
a `ShardConsumer` if `tryLock()` is successful or no `LockRegistry` at all
* Additional logic is added to always iterate over candidate shards if
`tryLock()` on the matter is not successful.
This way the current `KinesisMessageDrivenChannelAdapter` picks up
those shards which have been locked by the consumer which has just
left a cluster and unlocked distributed locks
* Now all the shards are considered as candidates independently of the
`streams` or `shardOffsets` configuration
* Improve Kinesis tests performance
 2018-07-16 13:17:13 -04:00
Artem Bilan
fc34f814e5 Fix DynamoDbMetadataStore.initialized flag logic 
https://stackoverflow.com/questions/51257855/spring-cloud-aws-kinesis-stream-binder-failed-to-start-due-to-improper-bean-init
 2018-07-10 13:46:30 -04:00
Artem Bilan
3a279ca67f Rename DynamoDbMetadataStore class for consistency 2018-07-02 20:52:40 -04:00
Artem Bilan
19b21579dc Change DynamoDbLocalRunning default port to 4568 
To be able to run Kinesalite and Dynalite npm services locally in
parallel, we need to provide unique ports for them.
Therefore a default `4567` is left for the Kinesalite, meanwhile the
`DynamoDbLocalRunning` is expecting a 4568 now
 2018-07-02 13:30:39 -04:00
Artem Bilan
662416cd7b Fix DynamoDbLock.tryLock() for the real timeout 
Since the contract of the `tryLock()` to wait as close to the provided
timeout as possible, then we should not wait for the `leasePeriod` as
minimum.
Also we should sleep in between attempts not more then provided timeout
 2018-06-28 12:43:35 -04:00