diff --git a/binders/kinesis-binder/pom.xml b/binders/kinesis-binder/pom.xml index e4e5f8d68..0046214ab 100644 --- a/binders/kinesis-binder/pom.xml +++ b/binders/kinesis-binder/pom.xml @@ -28,7 +28,6 @@ spring-cloud-stream-binder-kinesis - spring-cloud-stream-binder-kinesis-docs diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/pom.xml b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/pom.xml deleted file mode 100644 index cc33e7043..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/pom.xml +++ /dev/null @@ -1,65 +0,0 @@ - - - 4.0.0 - spring-cloud-stream-binder-kinesis-docs - - org.springframework.cloud - spring-cloud-stream-binder-kinesis-parent - 4.0.0-SNAPSHOT - - jar - spring-cloud-stream-binder-kinesis-docs - Spring Cloud Stream AWS Kinesis Binder Docs - - spring-cloud-stream-binder-kinesis - ${basedir}/.. - 3.4 - .*stream.* - deploy - - - - ${project.groupId} - spring-cloud-stream-binder-kinesis - ${project.version} - - - - src/main/asciidoc - - - - docs - - - - pl.project13.maven - git-commit-id-plugin - - - maven-dependency-plugin - - - maven-resources-plugin - - - org.codehaus.mojo - exec-maven-plugin - - - org.asciidoctor - asciidoctor-maven-plugin - - - maven-antrun-plugin - - - maven-deploy-plugin - - - - - - diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/Guardfile b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/Guardfile deleted file mode 100644 index bdd4d7298..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/Guardfile +++ /dev/null @@ -1,20 +0,0 @@ -require 'asciidoctor' -require 'erb' - -guard 'shell' do - watch(/.*\.adoc$/) {|m| - Asciidoctor.render_file('index.adoc', \ - :in_place => true, \ - :safe => Asciidoctor::SafeMode::UNSAFE, \ - :attributes=> { \ - 'source-highlighter' => 'prettify', \ - 'icons' => 'font', \ - 'linkcss'=> 'true', \ - 'copycss' => 'true', \ - 'doctype' => 'book'}) - } -end - -guard 'livereload' do - watch(%r{^.+\.(css|js|html)$}) -end diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/_configprops.adoc b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/_configprops.adoc deleted file mode 100644 index 557d96957..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/_configprops.adoc +++ /dev/null @@ -1,49 +0,0 @@ -|=== -|Name | Default | Description - -|spring.cloud.stream.binders | | Additional per-binder properties (see {@link BinderProperties}) if more then one binder of the same type is used (i.e., connect to multiple instances of RabbitMq). Here you can specify multiple binder configurations, each with different environment settings. For example; spring.cloud.stream.binders.rabbit1.environment. . . , spring.cloud.stream.binders.rabbit2.environment. . . -|spring.cloud.stream.binding-retry-interval | `30` | Retry interval (in seconds) used to schedule binding attempts. Default: 30 sec. -|spring.cloud.stream.bindings | | Additional binding properties (see {@link BinderProperties}) per binding name (e.g., 'input`). For example; This sets the content-type for the 'input' binding of a Sink application: 'spring.cloud.stream.bindings.input.contentType=text/plain' -|spring.cloud.stream.default-binder | | The name of the binder to use by all bindings in the event multiple binders available (e.g., 'rabbit'). -|spring.cloud.stream.dynamic-destination-cache-size | `10` | The maximum size of Least Recently Used (LRU) cache of dynamic destinations. Once this size is reached, new destinations will trigger the removal of old destinations. Default: 10 -|spring.cloud.stream.dynamic-destinations | `[]` | A list of destinations that can be bound dynamically. If set, only listed destinations can be bound. -|spring.cloud.stream.function.batch-mode | `false` | -|spring.cloud.stream.function.bindings | | -|spring.cloud.stream.function.reactive | | -|spring.cloud.stream.input-bindings | | A semi-colon delimited string to explicitly define input bindings (specifically for cases when there is no implicit trigger to create such bindings such as Function, Supplier or Consumer). -|spring.cloud.stream.instance-count | `1` | The number of deployed instances of an application. Default: 1. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-count" where 'foo' is the name of the binding. -|spring.cloud.stream.instance-index | `0` | The instance id of the application: a number from 0 to instanceCount-1. Used for partitioning and with Kafka. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-index" where 'foo' is the name of the binding. -|spring.cloud.stream.instance-index-list | | A list of instance id's from 0 to instanceCount-1. Used for partitioning and with Kafka. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-index-list" where 'foo' is the name of the binding. This setting will override the one set in 'spring.cloud.stream.instance-index' -|spring.cloud.stream.integration.message-handler-not-propagated-headers | | Message header names that will NOT be copied from the inbound message. -|spring.cloud.stream.kinesis.binder.auto-add-shards | `false` | -|spring.cloud.stream.kinesis.binder.auto-create-stream | `true` | -|spring.cloud.stream.kinesis.binder.checkpoint.billing-mode | | -|spring.cloud.stream.kinesis.binder.checkpoint.create-delay | `1` | -|spring.cloud.stream.kinesis.binder.checkpoint.create-retries | `25` | -|spring.cloud.stream.kinesis.binder.checkpoint.read-capacity | `1` | -|spring.cloud.stream.kinesis.binder.checkpoint.table | | -|spring.cloud.stream.kinesis.binder.checkpoint.time-to-live | | -|spring.cloud.stream.kinesis.binder.checkpoint.write-capacity | `1` | -|spring.cloud.stream.kinesis.binder.describe-stream-backoff | `1000` | -|spring.cloud.stream.kinesis.binder.describe-stream-retries | `50` | -|spring.cloud.stream.kinesis.binder.headers | `[]` | -|spring.cloud.stream.kinesis.binder.kpl-kcl-enabled | `false` | Enables the usage of Amazon KCL/KPL libraries for all message consumption and production. -|spring.cloud.stream.kinesis.binder.locks.billing-mode | | -|spring.cloud.stream.kinesis.binder.locks.heartbeat-period | `5` | -|spring.cloud.stream.kinesis.binder.locks.lease-duration | `20` | -|spring.cloud.stream.kinesis.binder.locks.partition-key | | -|spring.cloud.stream.kinesis.binder.locks.read-capacity | `1` | -|spring.cloud.stream.kinesis.binder.locks.refresh-period | | -|spring.cloud.stream.kinesis.binder.locks.sort-key | | -|spring.cloud.stream.kinesis.binder.locks.sort-key-name | | -|spring.cloud.stream.kinesis.binder.locks.table | | -|spring.cloud.stream.kinesis.binder.locks.write-capacity | `1` | -|spring.cloud.stream.kinesis.binder.min-shard-count | `1` | -|spring.cloud.stream.kinesis.bindings | | -|spring.cloud.stream.output-bindings | | A semi-colon delimited string to explicitly define output bindings (specifically for cases when there is no implicit trigger to create such bindings such as Function, Supplier or Consumer). -|spring.cloud.stream.override-cloud-connectors | `false` | This property is only applicable when the cloud profile is active and Spring Cloud Connectors are provided with the application. If the property is false (the default), the binder detects a suitable bound service (for example, a RabbitMQ service bound in Cloud Foundry for the RabbitMQ binder) and uses it for creating connections (usually through Spring Cloud Connectors). When set to true, this property instructs binders to completely ignore the bound services and rely on Spring Boot properties (for example, relying on the spring.rabbitmq.* properties provided in the environment for the RabbitMQ binder). The typical usage of this property is to be nested in a customized environment when connecting to multiple systems. -|spring.cloud.stream.pollable-source | `none` | A semi-colon delimited list of binding names of pollable sources. Binding names follow the same naming convention as functions. For example, name '...pollable-source=foobar' will be accessible as 'foobar-iin-0'' binding -|spring.cloud.stream.sendto.destination | `none` | The name of the header used to determine the name of the output destination -|spring.cloud.stream.source | | A semi-colon delimited string representing the names of the sources based on which source bindings will be created. This is primarily to support cases where source binding may be required without providing a corresponding Supplier. (e.g., for cases where the actual source of data is outside of scope of spring-cloud-stream - HTTP -> Stream) @deprecated use {@link #outputBindings} - -|=== \ No newline at end of file diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/building.adoc b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/building.adoc deleted file mode 100644 index e8b37bb3b..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/building.adoc +++ /dev/null @@ -1,59 +0,0 @@ -[[building]] -== Building - -:jdkversion: 1.7 - -=== Basic Compile and Test - -To build the source you will need to install JDK {jdkversion}. - -The build uses the Maven wrapper so you don't have to install a specific version of Maven. -To enable the tests, you should have https://github.com/mhart/kinesalite[Kinesalite] server running before building. - -The main build command is - ----- -$ ./mvnw clean install ----- - -You can also add `-DskipTests` if you like, to avoid running the tests. - -NOTE: You can also install Maven (>=3.3.3) yourself and run the `mvn` command in place of `./mvnw` in the examples below. -If you do that you also might need to add `-P spring` if your local Maven settings do not contain repository declarations for spring pre-release artifacts. - -NOTE: Be aware that you might need to increase the amount of memory available to Maven by setting a `MAVEN_OPTS` environment variable with a value like `-Xmx512m -XX:MaxPermSize=128m`. -We try to cover this in the `.mvn` configuration, so if you find you have to do it to make a build succeed, please raise a ticket to get the settings added to source control. - - -The projects that require middleware generally include a `docker-compose.yml`, so consider using https://docs.docker.com/compose[Docker Compose] to run the middleware servers in Docker containers. - -=== Documentation - -There is a "full" profile that will generate documentation. - -=== Working with the code -If you don't have an IDE preference we would recommend that you use https://www.springsource.com/developer/sts[Spring Tools Suite] or https://eclipse.org[Eclipse] when working with the code. -We use the https://eclipse.org/m2e/[m2eclipe] eclipse plugin for maven support. -Other IDEs and tools should also work without issue. - -==== Importing into eclipse with m2eclipse -We recommend the https://eclipse.org/m2e/[m2eclipe] eclipse plugin when working with eclipse. -If you don't already have m2eclipse installed it is available from the "eclipse marketplace". - -Unfortunately m2e does not yet support Maven 3.3, so once the projects are imported into Eclipse you will also need to tell m2eclipse to use the `.settings.xml` file for the projects. -If you do not do this you may see many different errors related to the POMs in the projects. -Open your Eclipse preferences, expand the Maven preferences, and select User Settings. -In the User Settings field click Browse and navigate to the Spring Cloud project you imported selecting the `.settings.xml` file in that project. -Click Apply and then OK to save the preference changes. - -NOTE: Alternatively you can copy the repository settings from https://github.com/spring-cloud/spring-cloud-build/blob/main/.settings.xml[`.settings.xml`] into your own `~/.m2/settings.xml`. - -==== Importing into eclipse without m2eclipse -If you prefer not to use m2eclipse you can generate eclipse project metadata using the following command: - -[indent=0] ----- - $ ./mvnw eclipse:eclipse ----- - -The generated eclipse projects can be imported by selecting `import existing projects` from the `file` menu. diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/contributing.adoc b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/contributing.adoc deleted file mode 100644 index 4a2b460d4..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/contributing.adoc +++ /dev/null @@ -1,26 +0,0 @@ -[[contributing]] -== Contributing - -Spring Cloud is released under the non-restrictive Apache 2.0 license, and follows a very standard Github development process, using Github tracker for issues and merging pull requests into main. -If you want to contribute even something trivial please do not hesitate, but follow the guidelines below. - -=== Sign the Contributor License Agreement - -Before we accept a non-trivial patch or pull request we will need you to sign the https://support.springsource.com/spring_committer_signup[contributor's agreement]. -Signing the contributor's agreement does not grant anyone commit rights to the main repository, but it does mean that we can accept your contributions, and you will get an author credit if we do. -Active contributors might be asked to join the core team, and given the ability to merge pull requests. - -=== Code Conventions and Housekeeping - -None of these is essential for a pull request, but they will all help. -They can also be added after the original pull request but before a merge. - -* Use the Spring Framework code format conventions. If you use Eclipse you can import formatter settings using the `eclipse-code-formatter.xml` file from the https://github.com/spring-cloud/build/tree/main/eclipse-coding-conventions.xml[Spring Cloud Build] project. - If using IntelliJ, you can use the https://plugins.jetbrains.com/plugin/6546[Eclipse Code Formatter Plugin] to import the same file. -* Make sure all new `.java` files to have a simple Javadoc class comment with at least an `@author` tag identifying you, and preferably at least a paragraph on what the class is for. -* Add the ASF license header comment to all new `.java` files (copy from existing files in the project) -* Add yourself as an `@author` to the .java files that you modify substantially (more than cosmetic changes). -* Add some Javadocs and, if you change the namespace, some XSD doc elements. -* A few unit tests would help a lot as well -- someone has to do it. -* If no-one else is using your branch, please rebase it against the current main (or other target branch in the main project). -* When writing a commit message please follow https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html[these conventions], if you are fixing an existing issue please add `Fixes gh-XXXX` at the end of the commit message (where XXXX is the issue number). diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/index-docinfo.xml b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/index-docinfo.xml deleted file mode 100644 index 5604bb46b..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/index-docinfo.xml +++ /dev/null @@ -1,14 +0,0 @@ -Spring Cloud Stream Kinesis Binder -{spring-cloud-stream-binder-kinesis-version} - - 2017-2021 - VMWare, Inc. - - - - Copies of this document may be made for your own use and for distribution to - others, provided that you do not charge any fee for such copies and further - provided that each copy contains this Copyright Notice, whether distributed in - print or electronically. - - diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/index.adoc b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/index.adoc deleted file mode 100644 index c5eb76277..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/index.adoc +++ /dev/null @@ -1,34 +0,0 @@ -[[spring-cloud-stream-binder-kinesis-reference]] -= Spring Cloud Stream AWS Kinesis Binder Reference Guide -Artem Bilan -:doctype: book -:toc: -:toclevels: 4 -:source-highlighter: prettify -:numbered: -:icons: font -:hide-uri-scheme: -:spring-cloud-stream-binder-kinesis-repo: snapshot -:github-tag: main -:spring-cloud-stream-binder-kinesis-docs-version: current -:spring-cloud-stream-binder-kinesis-docs: https://docs.spring.io/spring-cloud-stream-binder-kinesis/docs/{spring-cloud-stream-binder-kinesis-docs-version}/reference -:spring-cloud-stream-binder-kinesis-docs-current: https://docs.spring.io/spring-cloud-stream-binder-kinesis/docs/current-SNAPSHOT/reference/html/ -:github-repo: spring-cloud/spring-cloud-stream-binder-aws-kinesis -:github-raw: https://raw.github.com/{github-repo}/{github-tag} -:github-code: https://github.com/{github-repo}/tree/{github-tag} -:github-wiki: https://github.com/{github-repo}/wiki -:github-main-code: https://github.com/{github-repo}/tree/main -:sc-ext: java -// ====================================================================================== - -= Reference Guide -include::overview.adoc[] - - -= Appendices -[appendix] -include::building.adoc[] -[appendix] -include::contributing.adoc[] - -// ====================================================================================== diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/ruby/generate_readme.sh b/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/ruby/generate_readme.sh deleted file mode 100644 index 6d0ce9dc5..000000000 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/ruby/generate_readme.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env ruby - -base_dir = File.join(File.dirname(__FILE__),'../../..') -src_dir = File.join(base_dir, "/src/main/asciidoc") -require 'asciidoctor' -require 'optparse' - -options = {} -file = "#{src_dir}/README.adoc" - -OptionParser.new do |o| - o.on('-o OUTPUT_FILE', 'Output file (default is stdout)') { |file| options[:to_file] = file unless file=='-' } - o.on('-h', '--help') { puts o; exit } - o.parse! -end - -file = ARGV[0] if ARGV.length>0 - -# Copied from https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/scripts/asciidoc-coalescer.rb -doc = Asciidoctor.load_file file, safe: :unsafe, header_only: true, attributes: options[:attributes] -header_attr_names = (doc.instance_variable_get :@attributes_modified).to_a -header_attr_names.each {|k| doc.attributes[%(#{k}!)] = '' unless doc.attr? k } -attrs = doc.attributes -attrs['allow-uri-read'] = true -puts attrs - -out = "// Do not edit this file (e.g. go instead to src/main/asciidoc)\n\n" -doc = Asciidoctor.load_file file, safe: :unsafe, parse: false, attributes: attrs -out << doc.reader.read - -unless options[:to_file] - puts out -else - File.open(options[:to_file],'w+') do |file| - file.write(out) - end -end diff --git a/docs/src/main/asciidoc/index.adoc b/docs/src/main/asciidoc/index.adoc index ff6a9b6eb..289d1f588 100644 --- a/docs/src/main/asciidoc/index.adoc +++ b/docs/src/main/asciidoc/index.adoc @@ -13,6 +13,7 @@ The reference documentation consists of the following sections: <> :: Spring Cloud Stream binder reference for Rabbit MQ <> :: Spring Cloud Stream binder reference for Apache Kafka <> :: Spring Cloud Stream binder reference for Apache Kafka Streams +<> :: Spring Cloud Stream binder reference for AWS Kinesis <> :: A collection of Partner maintained binder implementations for Spring Cloud Stream (e.g., Azure Event Hubs, Google PubSub, Solace PubSub+) diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/images/kinesis-binder.png b/docs/src/main/asciidoc/kinesis/images/kinesis-binder.png similarity index 100% rename from binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/images/kinesis-binder.png rename to docs/src/main/asciidoc/kinesis/images/kinesis-binder.png diff --git a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/overview.adoc b/docs/src/main/asciidoc/kinesis/spring-cloud-stream-binder-kinesis.adoc similarity index 78% rename from binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/overview.adoc rename to docs/src/main/asciidoc/kinesis/spring-cloud-stream-binder-kinesis.adoc index 6069f854c..63f0e563b 100644 --- a/binders/kinesis-binder/spring-cloud-stream-binder-kinesis-docs/src/main/asciidoc/overview.adoc +++ b/docs/src/main/asciidoc/kinesis/spring-cloud-stream-binder-kinesis.adoc @@ -1,4 +1,6 @@ -[partintro] +[[spring-cloud-stream-binder-kinesis-reference]] += Spring Cloud Stream AWS Kinesis Binder Reference Guide + -- This guide describes the https://aws.amazon.com/kinesis/[AWS Kinesis] implementation of the Spring Cloud Stream Binder. It contains information about its design, usage and configuration options, as well as information on how the Stream Cloud Stream concepts map into AWS Kinesis specific constructs. @@ -23,11 +25,11 @@ This implementation uses Spring Integration AWS Kinesis Channel Adapters at its The following captures how the Kinesis Binder implementation maps each of the configured destination to a AWS Kinesis Streams: .Kinesis Binder -image::images/kinesis-binder.png[width=300,scaledwidth="50%"] +image::{github-raw}/docs/src/main/asciidoc/images/kinesis-binder.png[width=300,scaledwidth="50%"] Unlike https://kafka.apache.org/[Apache Kafka] the AWS Kinesis doesn't provide out-of-the-box support for consumer groups. The support of this feature is implemented as a part of `MetadataStore` key for shard checkpoints in the `KinesisMessageDrivenChannelAdapter` - `[CONSUMER_GROUP]:[STREAM]:[SHARD_ID]`. -In addition the `LockRegistry` is used to ensure exclusive access to each shard. +In addition, the `LockRegistry` is used to ensure exclusive access to each shard. This way only one channel adapter in the same consumer group will consumer messages from a single shard in the stream it is configured for. The partitioning logic in AWS Kinesis is similar to the Apache Kafka support, but with slightly different logic. @@ -45,7 +47,7 @@ The Spring Cloud Stream partition handling logic is excluded in case of AWS Kine On the consumer side the `instanceCount` and `instanceIndex` are used to distribute shards between consumers in group evenly. This has an effect only for regular `KinesisMessageDrivenChannelAdapter` which can assign specific shards for the target Kinesis consumer. -With Kinesis Client Library we can only subscriber to the provided stream and shards distribution is done by that client. +With Kinesis Client Library we can only subscribe to the provided stream and shards distribution is done by that client. See more information in the Kinesis Client Library https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html[documentation]. == Consumer Groups @@ -54,8 +56,8 @@ A `single consumer` for the message is ensured by https://docs.spring.io/spring- To have a highly available consumer group for your kinesis stream: - - Ensure all instances of your consumer applications use a shared `DynamoDbMetadataStore` and `DynamoDbLockRegistry` (See below for configuration options). - - Use same group name for the channel in all application instances by using property `spring.cloud.stream.bindings..group`. +- Ensure all instances of your consumer applications use a shared `DynamoDbMetadataStore` and `DynamoDbLockRegistry` (See below for configuration options). +- Use same group name for the channel in all application instances by using property `spring.cloud.stream.bindings..group`. These configurations alone guarantee HA, message ordering and guaranteed message delivery. However, even distribution across instances is not guaranteed as of now. @@ -69,8 +71,8 @@ It can be configured using `spring.cloud.stream.bindings..consume It is possible to evenly distribute shard across all instances within a single consumer group. This done by configuring: - - `spring.cloud.stream.instanceCount=` to number of instances - - `spring.cloud.stream.instanceIndex=` current instance's index +- `spring.cloud.stream.instanceCount=` to number of instances +- `spring.cloud.stream.instanceIndex=` current instance's index The only way to achieve HA in this case is that, when an instance processing a particular shard goes down, another instance must have `spring.cloud.stream.instanceIndex=` to be the same as the failed instance's index to start processing from those shards. @@ -86,36 +88,36 @@ For general binding configuration options and properties, please refer to the ht The following properties are available for Kinesis Binder configuration, which start with the `spring.cloud.stream.kinesis.binder.` prefix headers:: - The set of custom headers to transfer over AWS Kinesis +The set of custom headers to transfer over AWS Kinesis + Default: "correlationId", "sequenceSize", "sequenceNumber", "contentType", "originalContentType". describeStreamBackoff:: - The amount of time in milliseconds in between retries for the `DescribeStream` operation +The amount of time in milliseconds in between retries for the `DescribeStream` operation + Default: `1000`. describeStreamRetries:: - The amount of times the consumer will retry a `DescribeStream` operation waiting for the stream to be in `ACTIVE` state +The amount of times the consumer will retry a `DescribeStream` operation waiting for the stream to be in `ACTIVE` state + Default: `50`. autoCreateStream:: - If set to `true`, the binder will create the stream automatically. +If set to `true`, the binder will create the stream automatically. If set to `false`, the binder will rely on the stream being already created. + Default: `true` autoAddShards:: - If set to `true`, the binder will create new shards automatically. +If set to `true`, the binder will create new shards automatically. If set to `false`, the binder will rely on the shard size of the stream being already configured. If the shard count of the target stream is smaller than the expected value, the binder will ignore that value + Default: `false` minShardCount:: - Effective only if `autoAddShards` is set to `true`. +Effective only if `autoAddShards` is set to `true`. The minimum number of shards that the binder will configure on the stream from which it produces/consumes data. It can be superseded by the `partitionCount` setting of the producer or by the value of `instanceCount * concurrency` settings of the producer (if either is larger) + Default: `1` kplKclEnabled:: - Enable the usage of https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html[Kinesis Client Library] / https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-kpl.html[Kinesis Producer Library] for all message consumption and production +Enable the usage of https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html[Kinesis Client Library] / https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-kpl.html[Kinesis Producer Library] for all message consumption and production + Default: `false` @@ -127,33 +129,33 @@ This is not configurable. DynamoDB Checkpoint properties are prefixed with `spring.cloud.stream.kinesis.binder.checkpoint.` table:: - The name to give the DynamoDb table +The name to give the DynamoDb table + Default: `SpringIntegrationMetadataStore` createDelay:: - The amount of time in seconds between each polling attempt while waiting for the checkpoint DynamoDB table to be created +The amount of time in seconds between each polling attempt while waiting for the checkpoint DynamoDB table to be created + Default: `1` createRetries:: - The amount of times the consumer will poll DynamoDB while waiting for the checkpoint table to be created +The amount of times the consumer will poll DynamoDB while waiting for the checkpoint table to be created + Default: `25` billingMode:: - The Billing Mode of the DynamoDB table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.OnDemand[DynamoDB On-Demand Mode]. Possible values are `provisioned` and `payPerRequest`. If left empty or set to `payPerRequest` both `readCapacity` and `writeCapacity` are ignored +The Billing Mode of the DynamoDB table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.OnDemand[DynamoDB On-Demand Mode]. Possible values are `provisioned` and `payPerRequest`. If left empty or set to `payPerRequest` both `readCapacity` and `writeCapacity` are ignored + Default: `payPerRequest` readCapacity:: - The Read capacity of the DynamoDb table. +The Read capacity of the DynamoDb table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual[DynamoDB Provisioned Throughput]. This property is used only when `billingMode` is set to `provisioned` + Default: `1` writeCapacity:: - The write capacity of the DynamoDb table. +The write capacity of the DynamoDb table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual[DynamoDB Provisioned Throughput]. This property is used only when `billingMode` is set to `provisioned` + Default: `1` timeToLive:: - A period in seconds for items expiration. +A period in seconds for items expiration. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html[DynamoDB TTL] + No default - means no records expiration. @@ -165,46 +167,46 @@ This is implemented using https://github.com/spring-projects/spring-integration- DynamoDB `LockRegistry` properties are prefixed with `spring.cloud.stream.kinesis.binder.locks.` table:: - The name to give the DynamoDB table +The name to give the DynamoDB table + Default: `SpringIntegrationLockRegistry` billingMode:: - The Billing Mode of the DynamoDB table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.OnDemand[DynamoDB On-Demand Mode]. Possible values are `provisioned` and `payPerRequest`. If left empty or set to `payPerRequest` both `readCapacity` and `writeCapacity` are ignored +The Billing Mode of the DynamoDB table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.OnDemand[DynamoDB On-Demand Mode]. Possible values are `provisioned` and `payPerRequest`. If left empty or set to `payPerRequest` both `readCapacity` and `writeCapacity` are ignored + Default: `payPerRequest` readCapacity:: - The Read capacity of the DynamoDB table. +The Read capacity of the DynamoDB table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual[DynamoDB Provisioned Throughput]. This property is used only when `billingMode` is set to `provisioned` + Default: `1` writeCapacity:: - The write capacity of the DynamoDb table. +The write capacity of the DynamoDb table. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual[DynamoDB Provisioned Throughput]. This property is used only when `billingMode` is set to `provisioned` + Default: `1` leaseDuration:: - The length of time that the lease for the lock will be granted for. - If this is set to, for example, 30 seconds, then the lock will expire if the heartbeat is not sent for at least 30 seconds (which would happen if the box or the heartbeat thread dies, for example.) +The length of time that the lease for the lock will be granted for. +If this is set to, for example, 30 seconds, then the lock will expire if the heartbeat is not sent for at least 30 seconds (which would happen if the box or the heartbeat thread dies, for example.) + Default: `20` heartbeatPeriod:: - How often to update DynamoDB to note that the instance is still running (recommendation is to make this at least 3 times smaller than the `leaseDuration` - for example `heartBeatPeriod=1` second, `leaseDuration=10` seconds could be a reasonable configuration, make sure to include a buffer for network latency.) +How often to update DynamoDB to note that the instance is still running (recommendation is to make this at least 3 times smaller than the `leaseDuration` - for example `heartBeatPeriod=1` second, `leaseDuration=10` seconds could be a reasonable configuration, make sure to include a buffer for network latency.) + Default: `5` refreshPeriod:: - How long to wait before trying to get the lock again (if set to 10 seconds, for example, it would attempt to do so every 10 seconds) +How long to wait before trying to get the lock again (if set to 10 seconds, for example, it would attempt to do so every 10 seconds) + Default: `1000` partitionKey:: - The partition key name of the table. +The partition key name of the table. + Default: `lockKey` sortKeyName:: - The sort key name for DynamoDB table partitioning. +The sort key name for DynamoDB table partitioning. + Default: `sortKey` sortKey:: - The sort key to try and acquire the lock on (specify if and only if the table has sort keys) +The sort key to try and acquire the lock on (specify if and only if the table has sort keys) + Default: `SpringIntegrationLocks` @@ -213,48 +215,48 @@ Default: `SpringIntegrationLocks` The following properties are available for Kinesis consumers only and must be prefixed with `spring.cloud.stream.kinesis.bindings..consumer` startTimeout:: - The amount of time to wait for the consumer to start, in milliseconds. +The amount of time to wait for the consumer to start, in milliseconds. + Default: `60000`. listenerMode:: - The mode in which records are processed. - If `record`, each `Message` will contain `byte[]` from a single `Record.data`. - If `batch`, each `Message` will contain a `List` extracted from the consumed records. - When `useNativeDecoding = true` is used on the consumer together with the `listenerMode = batch`, there is no any out-of-the-box conversion happened and a result message contains a payload like `List`. - It's up to target application to convert those records manually. +The mode in which records are processed. +If `record`, each `Message` will contain `byte[]` from a single `Record.data`. +If `batch`, each `Message` will contain a `List` extracted from the consumed records. +When `useNativeDecoding = true` is used on the consumer together with the `listenerMode = batch`, there is no any out-of-the-box conversion happened and a result message contains a payload like `List`. +It's up to target application to convert those records manually. + Default: `record` checkpointMode:: - The mode in which checkpoints are updated. - If `record`, checkpoints occur after each record is processed (but this option is only effective if `listenerMode` is set to `record`). If `batch`, checkpoints occur after each batch of records is processed. - If `manual`, checkpoints occur on demand via the `Checkpointer` callback. - If `periodic`, checkpoints occurs at specified time interval (from `interval` property in checkpoint configuration) +The mode in which checkpoints are updated. +If `record`, checkpoints occur after each record is processed (but this option is only effective if `listenerMode` is set to `record`). If `batch`, checkpoints occur after each batch of records is processed. +If `manual`, checkpoints occur on demand via the `Checkpointer` callback. +If `periodic`, checkpoints occurs at specified time interval (from `interval` property in checkpoint configuration) + Default: `batch` checkpointInterval:: - The interval, in milliseconds, between two checkpoints when checkpoint mode is `periodic`. +The interval, in milliseconds, between two checkpoints when checkpoint mode is `periodic`. + Default - `5000` workerId:: - The worker identifier used to distinguish different workers/processes (only used when Kinesis Client Library is enabled). +The worker identifier used to distinguish different workers/processes (only used when Kinesis Client Library is enabled). + No default - if not set, default value inside spring-integration-aws will be used (random UUID). recordsLimit:: - The maximum number of records to poll per `GetRecords` request. - Must not be greater than `10000`. +The maximum number of records to poll per `GetRecords` request. +Must not be greater than `10000`. + Default: `10000` idleBetweenPolls:: - The sleep interval used in the main loop between shards polling cycles, in milliseconds. Must not be less than `250`. +The sleep interval used in the main loop between shards polling cycles, in milliseconds. Must not be less than `250`. + Default: `1000` consumerBackoff:: - The amount of time the consumer will wait to attempt another `GetRecords` operation after a read with no results, in milliseconds. +The amount of time the consumer will wait to attempt another `GetRecords` operation after a read with no results, in milliseconds. + Default: `1000` shardIteratorType:: - The `com.amazonaws.services.kinesis.model.ShardIteratorType` name with an optional `sequenceNumber` for the `AT_SEQUENCE_NUMBER/AFTER_SEQUENCE_NUMBER` or milliseconds for the `AT_TIMESTAMP` after `:`. - For example: `AT_TIMESTAMP:1515090166767`. +The `com.amazonaws.services.kinesis.model.ShardIteratorType` name with an optional `sequenceNumber` for the `AT_SEQUENCE_NUMBER/AFTER_SEQUENCE_NUMBER` or milliseconds for the `AT_TIMESTAMP` after `:`. +For example: `AT_TIMESTAMP:1515090166767`. + Default: `LATEST` for anonymous groups and `TRIM_HORIZON` otherwise. @@ -265,7 +267,7 @@ This happens only for new consumer groups. Any subsequent starts of the consumer in the same group are adjusted according the stored checkpoint via `AFTER_SEQUENCE_NUMBER` iterator type. dynamoDbStreams:: - The `boolean` flag indicating that Kinesis consumer channel adapter should adapt DynamoDB Streams functionality instead of regular Kinesis streams. +The `boolean` flag indicating that Kinesis consumer channel adapter should adapt DynamoDB Streams functionality instead of regular Kinesis streams. The `spring.cloud.stream.bindings..destination` value must be a DynamoDB table name. Default: `false`. @@ -287,12 +289,12 @@ The `instanceCount` and `shardId` are considered as mutually exclusive. The following properties are available for Kinesis producers only and must be prefixed with `spring.cloud.stream.kinesis.bindings..producer.`. sync:: - Whether the producer should act in a synchronous manner with respect to writing records into a stream. +Whether the producer should act in a synchronous manner with respect to writing records into a stream. If true, the producer will wait for a response from Kinesis after a `PutRecord` operation. + Default: `false` sendTimeout:: - Effective only if `sync` is set to `true`. The amount of time to wait for a response from Kinesis after a `PutRecord` operation, in milliseconds. +Effective only if `sync` is set to `true`. The amount of time to wait for a response from Kinesis after a `PutRecord` operation, in milliseconds. + Default: `10000`