diff --git a/README.adoc b/README.adoc index f275776ab..d84b97fe9 100644 --- a/README.adoc +++ b/README.adoc @@ -5,7 +5,7 @@ Edit the files in the src/main/asciidoc/ directory instead. //// -:jdkversion: 1.8 +:jdkversion: 17 :github-tag: master :github-repo: spring-cloud/spring-cloud-stream @@ -18,218 +18,95 @@ image::https://badges.gitter.im/spring-cloud/spring-cloud-stream.svg[Gitter, lin // ====================================================================================== -= Preface -=== A Brief History of Spring's Data Integration Journey +== Introuduction +Spring Cloud Stream is a framework for building message-driven microservice applications. +Spring Cloud Stream builds upon Spring Boot to create standalone, production-grade Spring applications and uses Spring Integration to provide connectivity to message brokers. +It provides opinionated configuration of middleware from several vendors, introducing the concepts of persistent publish-subscribe semantics, consumer groups, and partitions. +These are called binder implementations in the parlance of Spring Cloud Stream. +Out of the box, Spring Cloud Stream provides binder implementations for Apache Kafka and RabbitMQ. +While these two binder implementations are based on Message Channels, Spring Cloud Stream also provides another binder implementation for Kafka Streams that does not use message channels, but native Kafka Streams types such as KStream, KTable etc. -Spring's journey on Data Integration started with https://projects.spring.io/spring-integration/[Spring Integration]. With its programming model, it provided a consistent developer experience to build applications that can embrace http://www.enterpriseintegrationpatterns.com/[Enterprise Integration Patterns] to connect with external systems such as, databases, message brokers, and among others. +== Apache Kafka Binder -Fast forward to the cloud-era, where microservices have become prominent in the enterprise setting. https://projects.spring.io/spring-boot/[Spring Boot] transformed the way how developers built Applications. With Spring's programming model and the runtime responsibilities handled by Spring Boot, it became seamless to develop stand-alone, production-grade Spring-based microservices. +=== Usage -To extend this to Data Integration workloads, Spring Integration and Spring Boot were put together into a new project. Spring Cloud Stream was born. +To use Apache Kafka binder, you need to add `spring-cloud-stream-binder-kafka` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: -[%hardbreaks] -With Spring Cloud Stream, developers can: - -- Build, test and deploy data-centric applications in isolation. -- Apply modern microservices architecture patterns, including composition through messaging. -- Decouple application responsibilities with event-centric thinking. An event can represent something that has happened in time, to which the downstream consumer applications can react without knowing where it originated or the producer's identity. -- Port the business logic onto message brokers (such as RabbitMQ, Apache Kafka, Amazon Kinesis). -- Rely on the framework's automatic content-type support for common use-cases. Extending to different data conversion types is possible. -- and many more. . . - -=== Quick Start - -You can try Spring Cloud Stream in less than 5 min even before you jump into any details by following this three-step guide. - -We show you how to create a Spring Cloud Stream application that receives messages coming from the messaging middleware of your choice (more on this later) and logs received messages to the console. -We call it `LoggingConsumer`. -While not very practical, it provides a good introduction to some of the main concepts -and abstractions, making it easier to digest the rest of this user guide. - -The three steps are as follows: - -. <> -. <> -. <> - -[[spring-cloud-stream-preface-creating-sample-application]] -==== Creating a Sample Application by Using Spring Initializr -To get started, visit the https://start.spring.io[Spring Initializr]. From there, you can generate our `LoggingConsumer` application. To do so: - -. In the *Dependencies* section, start typing `stream`. -When the "`Cloud Stream`" option should appears, select it. -. Start typing either 'kafka' or 'rabbit'. -. Select "`Kafka`" or "`RabbitMQ`". -+ -Basically, you choose the messaging middleware to which your application binds. -We recommend using the one you have already installed or feel more comfortable with installing and running. -Also, as you can see from the Initilaizer screen, there are a few other options you can choose. -For example, you can choose Gradle as your build tool instead of Maven (the default). -. In the *Artifact* field, type 'logging-consumer'. -+ -The value of the *Artifact* field becomes the application name. -If you chose RabbitMQ for the middleware, your Spring Initializr should now be as follows: - -[%hardbreaks] -[%hardbreaks] -[%hardbreaks] -image::/docs/src/main/asciidoc/images/spring-initializr.png[align="center"] - -[%hardbreaks] -[%hardbreaks] - -. Click the *Generate Project* button. -+ -Doing so downloads the zipped version of the generated project to your hard drive. -. Unzip the file into the folder you want to use as your project directory. - -TIP: We encourage you to explore the many possibilities available in the Spring Initializr. -It lets you create many different kinds of Spring applications. - -[[spring-cloud-stream-preface-importing-project]] -==== Importing the Project into Your IDE - -Now you can import the project into your IDE. -Keep in mind that, depending on the IDE, you may need to follow a specific import procedure. -For example, depending on how the project was generated (Maven or Gradle), you may need to follow specific import procedure (for example, in Eclipse or STS, you need to use File -> Import -> Maven -> Existing Maven Project). - -Once imported, the project must have no errors of any kind. Also, `src/main/java` should contain `com.example.loggingconsumer.LoggingConsumerApplication`. - -Technically, at this point, you can run the application's main class. -It is already a valid Spring Boot application. -However, it does not do anything, so we want to add some code. - -[[spring-cloud-stream-preface-adding-message-handler]] -==== Adding a Message Handler, Building, and Running - -Modify the `com.example.loggingconsumer.LoggingConsumerApplication` class to look as follows: - -[source, java] +[source,xml] ---- -@SpringBootApplication -public class LoggingConsumerApplication { - - public static void main(String[] args) { - SpringApplication.run(LoggingConsumerApplication.class, args); - } - - @Bean - public Consumer log() { - return person -> { - System.out.println("Received: " + person); - }; - } - - public static class Person { - private String name; - public String getName() { - return name; - } - public void setName(String name) { - this.name = name; - } - public String toString() { - return this.name; - } - } -} + + org.springframework.cloud + spring-cloud-stream-binder-kafka + ---- -As you can see from the preceding listing: +Alternatively, you can also use the Spring Cloud Stream Kafka Starter, as shown in the following example for Maven: -* We are using functional programming model (see <>) to define a single message handler as `Consumer`. -* We are relying on framework conventions to bind such handler to the input destination binding exposed by the binder. - -Doing so also lets you see one of the core features of the framework: It tries to automatically convert incoming message payloads to type `Person`. - -You now have a fully functional Spring Cloud Stream application that does listens for messages. -From here, for simplicity, we assume you selected RabbitMQ in <>. -Assuming you have RabbitMQ installed and running, you can start the application by running its `main` method in your IDE. - -You should see following output: - -[source] +[source,xml] ---- - --- [ main] c.s.b.r.p.RabbitExchangeQueueProvisioner : declaring queue for inbound: input.anonymous.CbMIwdkJSBO1ZoPDOtHtCg, bound to: input - --- [ main] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [localhost:5672] - --- [ main] o.s.a.r.c.CachingConnectionFactory : Created new connection: rabbitConnectionFactory#2a3a299:0/SimpleConnection@66c83fc8. . . - . . . - --- [ main] o.s.i.a.i.AmqpInboundChannelAdapter : started inbound.input.anonymous.CbMIwdkJSBO1ZoPDOtHtCg - . . . - --- [ main] c.e.l.LoggingConsumerApplication : Started LoggingConsumerApplication in 2.531 seconds (JVM running for 2.897) + + org.springframework.cloud + spring-cloud-starter-stream-kafka + ---- -Go to the RabbitMQ management console or any other RabbitMQ client and send a message to `input.anonymous.CbMIwdkJSBO1ZoPDOtHtCg`. -The `anonymous.CbMIwdkJSBO1ZoPDOtHtCg` part represents the group name and is generated, so it is bound to be different in your environment. -For something more predictable, you can use an explicit group name by setting `spring.cloud.stream.bindings.input.group=hello` (or whatever name you like). +== Apache Kafka Streams Binder -The contents of the message should be a JSON representation of the `Person` class, as follows: +=== Usage - {"name":"Sam Spade"} +To use Apache Kafka Streams binder, you need to add `spring-cloud-stream-binder-kafka-streams` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: -Then, in your console, you should see: +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-kafka-streams + +---- -`Received: Sam Spade` +== RabbitMQ Binder +=== Usage -You can also build and package your application into a boot jar (by using `./mvnw clean install`) and run the built JAR by using the `java -jar` command. +To use the RabbitMQ binder, you can add it to your Spring Cloud Stream application, by using the following Maven coordinates: -Now you have a working (albeit very basic) Spring Cloud Stream application. +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + +---- +Alternatively, you can use the Spring Cloud Stream RabbitMQ Starter, as follows: -[[spel-and-streaming-data]] +[source,xml] +---- + + org.springframework.cloud + spring-cloud-starter-stream-rabbit + +---- -== Spring Expression Language (SpEL) in the context of Streaming data +== Resources -Throughout this reference manual you will encounter many features and examples where you can utilize Spring Expression Language (SpEL). It is important to understand certain limitations when it comes to using it. +For more information, please visit the https://spring.io/projects/spring-cloud-stream[project website]: -SpEL gives you access to the current Message as well as the Application Context you are running in. -However it is important to understand what type of data SpEL can see especially in the context of the incoming Message. -From the broker, the message arrives in a form of a byte[]. It is then transformed to a `Message` by the binders where as you can see the payload of the message maintains its raw form. The headers of the message are ``, where values are typically another primitive or a collection/array of primitives, hence Object. -That is because binder does not know the required input type as it has no access to the user code (function). So effectively binder delivered an envelope with the payload and some readable meta-data in the form of message headers, just like the letter delivered by mail. -This means that while accessing payload of the message is possible you will only have access to it as raw data (i.e., byte[]). And while it may be very common for developers to ask for ability to have SpEL access to fields of a payload object as concrete type (e.g., Foo, Bar etc), you can see how difficult or even impossible would it be to achieve. -Here is one example to demonstrate the problem; Imagine you have a routing expression to route to different functions based on payload type. This requirement would imply payload conversion from byte[] to a specific type and then applying the SpEL. However, in order to perform such conversion we would need to know the actual type to pass to converter and that comes from function's signature which we don’t know which one. A better approach to solve this requirement would be to pass the type information as message headers (e.g., `application/json;type=foo.bar.Baz` ). You’ll get a clear readable String value that could be accessed and evaluated in a year and easy to read SpEL expression. - -Additionally it is considered very bad practice to use payload for routing decisions, since the payload is considered to be privileged data - data only to be read by its final recipient. Again, using the mail delivery analogy you would not want the mailman to open your envelope and read the contents of the letter to make some delivery decisions. The same concept applies here, especially when it is relatively easy to include such information when generating a Message. It enforces certain level of discipline related to the design of data to be transmitted over the network and which pieces of such data can be considered as public and which are privileged. - -[[spel-and-streaming-data]] - -== Spring Expression Language (SpEL) in the context of Streaming data - -Throughout this reference manual you will encounter many features and examples where you can utilize Spring Expression Language (SpEL). It is important to understand certain limitations when it comes to using it. - -SpEL gives you access to the current Message as well as the Application Context you are running in. -However it is important to understand what type of data SpEL can see especially in the context of the incoming Message. -From the broker, the message arrives in a form of a byte[]. It is then transformed to a `Message` by the binders where as you can see the payload of the message maintains its raw form. The headers of the message are ``, where values are typically another primitive or a collection/array of primitives, hence Object. -That is because binder does not know the required input type as it has no access to the user code (function). So effectively binder delivered an envelope with the payload and some readable meta-data in the form of message headers, just like the letter delivered by mail. -This means that while accessing payload of the message is possible you will only have access to it as raw data (i.e., byte[]). And while it may be very common for developers to ask for ability to have SpEL access to fields of a payload object as concrete type (e.g., Foo, Bar etc), you can see how difficult or even impossible would it be to achieve. -Here is one example to demonstrate the problem; Imagine you have a routing expression to route to different functions based on payload type. This requirement would imply payload conversion from byte[] to a specific type and then applying the SpEL. However, in order to perform such conversion we would need to know the actual type to pass to converter and that comes from function's signature which we don’t know which one. A better approach to solve this requirement would be to pass the type information as message headers (e.g., `application/json;type=foo.bar.Baz` ). You’ll get a clear readable String value that could be accessed and evaluated in a year and easy to read SpEL expression. - -Additionally it is considered very bad practice to use payload for routing decisions, since the payload is considered to be privileged data - data only to be read by its final recipient. Again, using the mail delivery analogy you would not want the mailman to open your envelope and read the contents of the letter to make some delivery decisions. The same concept applies here, especially when it is relatively easy to include such information when generating a Message. It enforces certain level of discipline related to the design of data to be transmitted over the network and which pieces of such data can be considered as public and which are privileged. - -= Appendices -[appendix] -[[building]] == Building -:jdkversion: 1.8 +:jdkversion: 17 === Basic Compile and Test To build the source you will need to install JDK {jdkversion}. -The build uses the Maven wrapper so you don't have to install a specific -version of Maven. To enable the tests for Redis, Rabbit, and Kafka bindings you -should have those servers running before building. See below for more -information on running the servers. - -The main build command is +Spring Cloud uses Maven for most build-related activities, and you +should be able to get off the ground quite quickly by cloning the +project you are interested in and typing ---- -$ ./mvnw clean install +$ ./mvnw install ---- -You can also add '-DskipTests' if you like, to avoid running the tests. - NOTE: You can also install Maven (>=3.3.3) yourself and run the `mvn` command in place of `./mvnw` in the examples below. If you do that you also might need to add `-P spring` if your local Maven settings do not @@ -242,42 +119,46 @@ the `.mvn` configuration, so if you find you have to do it to make a build succeed, please raise a ticket to get the settings added to source control. +The projects that require middleware (i.e. Redis) for testing generally +require that a local instance of [Docker](https://www.docker.com/get-started) is installed and running. -The projects that require middleware generally include a -`docker-compose.yml`, so consider using -http://compose.docker.io/[Docker Compose] to run the middeware servers -in Docker containers. See the README in the -https://github.com/spring-cloud-samples/scripts[scripts demo -repository] for specific instructions about the common cases of mongo, -rabbit and redis. === Documentation -There is a "full" profile that will generate documentation. +The spring-cloud-build module has a "docs" profile, and if you switch +that on it will try to build asciidoc sources from +`src/main/asciidoc`. As part of that process it will look for a +`README.adoc` and process it by loading all the includes, but not +parsing or rendering it, just copying it to `${main.basedir}` +(defaults to `${basedir}`, i.e. the root of the project). If there are +any changes in the README it will then show up after a Maven build as +a modified file in the correct place. Just commit it and push the change. === Working with the code If you don't have an IDE preference we would recommend that you use -http://www.springsource.com/developer/sts[Spring Tools Suite] or -http://eclipse.org[Eclipse] when working with the code. We use the -http://eclipse.org/m2e/[m2eclipe] eclipse plugin for maven support. Other IDEs and tools -should also work without issue. +https://www.springsource.com/developer/sts[Spring Tools Suite] or +https://eclipse.org[Eclipse] when working with the code. We use the +https://eclipse.org/m2e/[m2eclipse] eclipse plugin for maven support. Other IDEs and tools +should also work without issue as long as they use Maven 3.3.3 or better. + +==== Activate the Spring Maven profile +Spring Cloud projects require the 'spring' Maven profile to be activated to resolve +the spring milestone and snapshot repositories. Use your preferred IDE to set this +profile to be active, or you may experience build errors. ==== Importing into eclipse with m2eclipse -We recommend the http://eclipse.org/m2e/[m2eclipe] eclipse plugin when working with +We recommend the https://eclipse.org/m2e/[m2eclipse] eclipse plugin when working with eclipse. If you don't already have m2eclipse installed it is available from the "eclipse marketplace". -Unfortunately m2e does not yet support Maven 3.3, so once the projects -are imported into Eclipse you will also need to tell m2eclipse to use -the `.settings.xml` file for the projects. If you do not do this you -may see many different errors related to the POMs in the -projects. Open your Eclipse preferences, expand the Maven -preferences, and select User Settings. In the User Settings field -click Browse and navigate to the Spring Cloud project you imported -selecting the `.settings.xml` file in that project. Click Apply and -then OK to save the preference changes. - -NOTE: Alternatively you can copy the repository settings from https://github.com/spring-cloud/spring-cloud-build/blob/master/.settings.xml[`.settings.xml`] into your own `~/.m2/settings.xml`. +NOTE: Older versions of m2e do not support Maven 3.3, so once the +projects are imported into Eclipse you will also need to tell +m2eclipse to use the right profile for the projects. If you +see many different errors related to the POMs in the projects, check +that you have an up to date installation. If you can't upgrade m2e, +add the "spring" profile to your `settings.xml`. Alternatively you can +copy the repository settings from the "spring" profile of the parent +pom into your `settings.xml`. ==== Importing into eclipse without m2eclipse If you prefer not to use m2eclipse you can generate eclipse project metadata using the @@ -291,9 +172,11 @@ following command: The generated eclipse projects can be imported by selecting `import existing projects` from the `file` menu. -[[contributing]] + == Contributing +:spring-cloud-build-branch: master + Spring Cloud is released under the non-restrictive Apache 2.0 license, and follows a very standard Github development process, using Github tracker for issues and merging pull requests into master. If you want @@ -302,12 +185,17 @@ follow the guidelines below. === Sign the Contributor License Agreement Before we accept a non-trivial patch or pull request we will need you to sign the -https://support.springsource.com/spring_committer_signup[contributor's agreement]. +https://cla.pivotal.io/sign/spring[Contributor License Agreement]. Signing the contributor's agreement does not grant anyone commit rights to the main repository, but it does mean that we can accept your contributions, and you will get an author credit if we do. Active contributors might be asked to join the core team, and given the ability to merge pull requests. +=== Code of Conduct +This project adheres to the Contributor Covenant https://github.com/spring-cloud/spring-cloud-build/blob/master/docs/src/main/asciidoc/code-of-conduct.adoc[code of +conduct]. By participating, you are expected to uphold this code. Please report +unacceptable behavior to spring-code-of-conduct@pivotal.io. + === Code Conventions and Housekeeping None of these is essential for a pull request, but they will all help. They can also be added after the original pull request but before a merge. @@ -315,9 +203,9 @@ added after the original pull request but before a merge. * Use the Spring Framework code format conventions. If you use Eclipse you can import formatter settings using the `eclipse-code-formatter.xml` file from the - https://github.com/spring-cloud/build/tree/master/eclipse-coding-conventions.xml[Spring + https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/spring-cloud-dependencies-parent/eclipse-code-formatter.xml[Spring Cloud Build] project. If using IntelliJ, you can use the - http://plugins.jetbrains.com/plugin/6546[Eclipse Code Formatter + https://plugins.jetbrains.com/plugin/6546[Eclipse Code Formatter Plugin] to import the same file. * Make sure all new `.java` files to have a simple Javadoc class comment with at least an `@author` tag identifying you, and preferably at least a paragraph on what the class is @@ -330,9 +218,190 @@ added after the original pull request but before a merge. * A few unit tests would help a lot as well -- someone has to do it. * If no-one else is using your branch, please rebase it against the current master (or other target branch in the main project). -* When writing a commit message please follow http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html[these conventions], +* When writing a commit message please follow https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html[these conventions], if you are fixing an existing issue please add `Fixes gh-XXXX` at the end of the commit message (where XXXX is the issue number). +=== Checkstyle + +Spring Cloud Build comes with a set of checkstyle rules. You can find them in the `spring-cloud-build-tools` module. The most notable files under the module are: + +.spring-cloud-build-tools/ +---- +└── src +    ├── checkstyle +    │   └── checkstyle-suppressions.xml <3> +    └── main +    └── resources +    ├── checkstyle-header.txt <2> +    └── checkstyle.xml <1> +---- +<1> Default Checkstyle rules +<2> File header setup +<3> Default suppression rules + +==== Checkstyle configuration + +Checkstyle rules are *disabled by default*. To add checkstyle to your project just define the following properties and plugins. + +.pom.xml +---- + +true <1> + true + <2> + true + <3> + + + + + <4> + io.spring.javaformat + spring-javaformat-maven-plugin + + <5> + org.apache.maven.plugins + maven-checkstyle-plugin + + + + + + <5> + org.apache.maven.plugins + maven-checkstyle-plugin + + + + +---- +<1> Fails the build upon Checkstyle errors +<2> Fails the build upon Checkstyle violations +<3> Checkstyle analyzes also the test sources +<4> Add the Spring Java Format plugin that will reformat your code to pass most of the Checkstyle formatting rules +<5> Add checkstyle plugin to your build and reporting phases + +If you need to suppress some rules (e.g. line length needs to be longer), then it's enough for you to define a file under `${project.root}/src/checkstyle/checkstyle-suppressions.xml` with your suppressions. Example: + +.projectRoot/src/checkstyle/checkstyle-suppresions.xml +---- + + + + + + +---- + +It's advisable to copy the `${spring-cloud-build.rootFolder}/.editorconfig` and `${spring-cloud-build.rootFolder}/.springformat` to your project. That way, some default formatting rules will be applied. You can do so by running this script: + +```bash +$ curl https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/.editorconfig -o .editorconfig +$ touch .springformat +``` + +=== IDE setup + +==== Intellij IDEA + +In order to setup Intellij you should import our coding conventions, inspection profiles and set up the checkstyle plugin. +The following files can be found in the https://github.com/spring-cloud/spring-cloud-build/tree/master/spring-cloud-build-tools[Spring Cloud Build] project. + +.spring-cloud-build-tools/ +---- +└── src +    ├── checkstyle +    │   └── checkstyle-suppressions.xml <3> +    └── main +    └── resources +    ├── checkstyle-header.txt <2> +    ├── checkstyle.xml <1> +    └── intellij +       ├── Intellij_Project_Defaults.xml <4> +       └── Intellij_Spring_Boot_Java_Conventions.xml <5> +---- +<1> Default Checkstyle rules +<2> File header setup +<3> Default suppression rules +<4> Project defaults for Intellij that apply most of Checkstyle rules +<5> Project style conventions for Intellij that apply most of Checkstyle rules + +.Code style + +image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/{spring-cloud-build-branch}/docs/src/main/asciidoc/images/intellij-code-style.png[Code style] + +Go to `File` -> `Settings` -> `Editor` -> `Code style`. There click on the icon next to the `Scheme` section. There, click on the `Import Scheme` value and pick the `Intellij IDEA code style XML` option. Import the `spring-cloud-build-tools/src/main/resources/intellij/Intellij_Spring_Boot_Java_Conventions.xml` file. + +.Inspection profiles + +image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/{spring-cloud-build-branch}/docs/src/main/asciidoc/images/intellij-inspections.png[Code style] + +Go to `File` -> `Settings` -> `Editor` -> `Inspections`. There click on the icon next to the `Profile` section. There, click on the `Import Profile` and import the `spring-cloud-build-tools/src/main/resources/intellij/Intellij_Project_Defaults.xml` file. + +.Checkstyle + +To have Intellij work with Checkstyle, you have to install the `Checkstyle` plugin. It's advisable to also install the `Assertions2Assertj` to automatically convert the JUnit assertions + +image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/{spring-cloud-build-branch}/docs/src/main/asciidoc/images/intellij-checkstyle.png[Checkstyle] + +Go to `File` -> `Settings` -> `Other settings` -> `Checkstyle`. There click on the `+` icon in the `Configuration file` section. There, you'll have to define where the checkstyle rules should be picked from. In the image above, we've picked the rules from the cloned Spring Cloud Build repository. However, you can point to the Spring Cloud Build's GitHub repository (e.g. for the `checkstyle.xml` : `https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/spring-cloud-build-tools/src/main/resources/checkstyle.xml`). We need to provide the following variables: + +- `checkstyle.header.file` - please point it to the Spring Cloud Build's, `spring-cloud-build-tools/src/main/resources/checkstyle-header.txt` file either in your cloned repo or via the `https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/spring-cloud-build-tools/src/main/resources/checkstyle-header.txt` URL. +- `checkstyle.suppressions.file` - default suppressions. Please point it to the Spring Cloud Build's, `spring-cloud-build-tools/src/checkstyle/checkstyle-suppressions.xml` file either in your cloned repo or via the `https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/spring-cloud-build-tools/src/checkstyle/checkstyle-suppressions.xml` URL. +- `checkstyle.additional.suppressions.file` - this variable corresponds to suppressions in your local project. E.g. you're working on `spring-cloud-contract`. Then point to the `project-root/src/checkstyle/checkstyle-suppressions.xml` folder. Example for `spring-cloud-contract` would be: `/home/username/spring-cloud-contract/src/checkstyle/checkstyle-suppressions.xml`. + +IMPORTANT: Remember to set the `Scan Scope` to `All sources` since we apply checkstyle rules for production and test sources. + +=== Duplicate Finder + +Spring Cloud Build brings along the `basepom:duplicate-finder-maven-plugin`, that enables flagging duplicate and conflicting classes and resources on the java classpath. + +==== Duplicate Finder configuration + +Duplicate finder is *enabled by default* and will run in the `verify` phase of your Maven build, but it will only take effect in your project if you add the `duplicate-finder-maven-plugin` to the `build` section of the projecst's `pom.xml`. + +.pom.xml +[source,xml] +---- + + + + org.basepom.maven + duplicate-finder-maven-plugin + + + +---- + +For other properties, we have set defaults as listed in the https://github.com/basepom/duplicate-finder-maven-plugin/wiki[plugin documentation]. + +You can easily override them but setting the value of the selected property prefixed with `duplicate-finder-maven-plugin`. For example, set `duplicate-finder-maven-plugin.skip` to `true` in order to skip duplicates check in your build. + +If you need to add `ignoredClassPatterns` or `ignoredResourcePatterns` to your setup, make sure to add them in the plugin configuration section of your project: + +[source,xml] +---- + + + + org.basepom.maven + duplicate-finder-maven-plugin + + + org.joda.time.base.BaseDateTime + .*module-info + + + changelog.txt + + + + + + + +---- -// ====================================================================================== diff --git a/binders/kafka-binder/README.adoc b/binders/kafka-binder/README.adoc deleted file mode 100644 index 41f76bb91..000000000 --- a/binders/kafka-binder/README.adoc +++ /dev/null @@ -1,190 +0,0 @@ -//// -DO NOT EDIT THIS FILE. IT WAS GENERATED. -Manual changes to this file will be lost when it is generated again. -Edit the files in the src/main/asciidoc/ directory instead. -//// - - -:jdkversion: 1.8 -:github-tag: master -:github-repo: spring-cloud/spring-cloud-stream-binder-kafka - -:github-raw: https://raw.githubusercontent.com/{github-repo}/{github-tag} -:github-code: https://github.com/{github-repo}/tree/{github-tag} - -image::https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-kafka.svg?style=svg["CircleCI", link="https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-kafka"] -image::https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-kafka/branch/{github-tag}/graph/badge.svg["codecov", link="https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-kafka"] -image::https://badges.gitter.im/spring-cloud/spring-cloud-stream.svg[Gitter, link="https://gitter.im/spring-cloud/spring-cloud-stream?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge"] - -// ====================================================================================== - -//= Overview -[partintro] --- -This guide describes the Apache Kafka implementation of the Spring Cloud Stream Binder. -It contains information about its design, usage, and configuration options, as well as information on how the Stream Cloud Stream concepts map onto Apache Kafka specific constructs. -In addition, this guide explains the Kafka Streams binding capabilities of Spring Cloud Stream. --- - -== Apache Kafka Binder - -=== Usage - -To use Apache Kafka binder, you need to add `spring-cloud-stream-binder-kafka` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-kafka - ----- - -Alternatively, you can also use the Spring Cloud Stream Kafka Starter, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-starter-stream-kafka - ----- - -== Apache Kafka Streams Binder - -=== Usage - -To use Apache Kafka Streams binder, you need to add `spring-cloud-stream-binder-kafka-streams` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-kafka-streams - ----- - -= Appendices -[appendix] -[[building]] -== Building - -:jdkversion: 1.7 - -=== Basic Compile and Test - -To build the source you will need to install JDK {jdkversion}. - -The build uses the Maven wrapper so you don't have to install a specific -version of Maven. To enable the tests, you should have Kafka server 0.9 or above running -before building. See below for more information on running the servers. - -The main build command is - ----- -$ ./mvnw clean install ----- - -You can also add '-DskipTests' if you like, to avoid running the tests. - -NOTE: You can also install Maven (>=3.3.3) yourself and run the `mvn` command -in place of `./mvnw` in the examples below. If you do that you also -might need to add `-P spring` if your local Maven settings do not -contain repository declarations for spring pre-release artifacts. - -NOTE: Be aware that you might need to increase the amount of memory -available to Maven by setting a `MAVEN_OPTS` environment variable with -a value like `-Xmx512m -XX:MaxPermSize=128m`. We try to cover this in -the `.mvn` configuration, so if you find you have to do it to make a -build succeed, please raise a ticket to get the settings added to -source control. - - -The projects that require middleware generally include a -`docker-compose.yml`, so consider using -https://compose.docker.io/[Docker Compose] to run the middeware servers -in Docker containers. - -=== Documentation - -There is a "full" profile that will generate documentation. - -=== Working with the code -If you don't have an IDE preference we would recommend that you use -https://www.springsource.com/developer/sts[Spring Tools Suite] or -https://eclipse.org[Eclipse] when working with the code. We use the -https://eclipse.org/m2e/[m2eclipe] eclipse plugin for maven support. Other IDEs and tools -should also work without issue. - -==== Importing into eclipse with m2eclipse -We recommend the https://eclipse.org/m2e/[m2eclipe] eclipse plugin when working with -eclipse. If you don't already have m2eclipse installed it is available from the "eclipse -marketplace". - -Unfortunately m2e does not yet support Maven 3.3, so once the projects -are imported into Eclipse you will also need to tell m2eclipse to use -the `.settings.xml` file for the projects. If you do not do this you -may see many different errors related to the POMs in the -projects. Open your Eclipse preferences, expand the Maven -preferences, and select User Settings. In the User Settings field -click Browse and navigate to the Spring Cloud project you imported -selecting the `.settings.xml` file in that project. Click Apply and -then OK to save the preference changes. - -NOTE: Alternatively you can copy the repository settings from https://github.com/spring-cloud/spring-cloud-build/blob/master/.settings.xml[`.settings.xml`] into your own `~/.m2/settings.xml`. - -==== Importing into eclipse without m2eclipse -If you prefer not to use m2eclipse you can generate eclipse project metadata using the -following command: - -[indent=0] ----- - $ ./mvnw eclipse:eclipse ----- - -The generated eclipse projects can be imported by selecting `import existing projects` -from the `file` menu. -[[contributing] -== Contributing - -Spring Cloud is released under the non-restrictive Apache 2.0 license, -and follows a very standard Github development process, using Github -tracker for issues and merging pull requests into master. If you want -to contribute even something trivial please do not hesitate, but -follow the guidelines below. - -=== Sign the Contributor License Agreement -Before we accept a non-trivial patch or pull request we will need you to sign the -https://support.springsource.com/spring_committer_signup[contributor's agreement]. -Signing the contributor's agreement does not grant anyone commit rights to the main -repository, but it does mean that we can accept your contributions, and you will get an -author credit if we do. Active contributors might be asked to join the core team, and -given the ability to merge pull requests. - -=== Code Conventions and Housekeeping -None of these is essential for a pull request, but they will all help. They can also be -added after the original pull request but before a merge. - -* Use the Spring Framework code format conventions. If you use Eclipse - you can import formatter settings using the - `eclipse-code-formatter.xml` file from the - https://github.com/spring-cloud/build/tree/master/eclipse-coding-conventions.xml[Spring - Cloud Build] project. If using IntelliJ, you can use the - https://plugins.jetbrains.com/plugin/6546[Eclipse Code Formatter - Plugin] to import the same file. -* Make sure all new `.java` files to have a simple Javadoc class comment with at least an - `@author` tag identifying you, and preferably at least a paragraph on what the class is - for. -* Add the ASF license header comment to all new `.java` files (copy from existing files - in the project) -* Add yourself as an `@author` to the .java files that you modify substantially (more - than cosmetic changes). -* Add some Javadocs and, if you change the namespace, some XSD doc elements. -* A few unit tests would help a lot as well -- someone has to do it. -* If no-one else is using your branch, please rebase it against the current master (or - other target branch in the main project). -* When writing a commit message please follow https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html[these conventions], - if you are fixing an existing issue please add `Fixes gh-XXXX` at the end of the commit - message (where XXXX is the issue number). - -// ====================================================================================== diff --git a/binders/kafka-binder/docs/src/main/asciidoc/README.adoc b/binders/kafka-binder/docs/src/main/asciidoc/README.adoc deleted file mode 100644 index ae836620e..000000000 --- a/binders/kafka-binder/docs/src/main/asciidoc/README.adoc +++ /dev/null @@ -1,57 +0,0 @@ -:jdkversion: 1.8 -:github-tag: master -:github-repo: spring-cloud/spring-cloud-stream-binder-kafka - -:github-raw: https://raw.githubusercontent.com/{github-repo}/{github-tag} -:github-code: https://github.com/{github-repo}/tree/{github-tag} - -image::https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-kafka.svg?style=svg["CircleCI", link="https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-kafka"] -image::https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-kafka/branch/{github-tag}/graph/badge.svg["codecov", link="https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-kafka"] -image::https://badges.gitter.im/spring-cloud/spring-cloud-stream-binder-kafka.svg[Gitter, link="https://gitter.im/spring-cloud/spring-cloud-stream-binder-kafka?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge"] - -// ====================================================================================== - -== Apache Kafka Binder - -=== Usage - -To use Apache Kafka binder, you need to add `spring-cloud-stream-binder-kafka` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-kafka - ----- - -Alternatively, you can also use the Spring Cloud Stream Kafka Starter, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-starter-stream-kafka - ----- - -== Apache Kafka Streams Binder - -=== Usage - -To use Apache Kafka Streams binder, you need to add `spring-cloud-stream-binder-kafka-streams` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-kafka-streams - ----- - -= Appendices -[appendix] -include::building.adoc[] -include::contributing.adoc[] - -// ====================================================================================== diff --git a/binders/rabbit-binder/README.adoc b/binders/rabbit-binder/README.adoc deleted file mode 100644 index 6636bbb2c..000000000 --- a/binders/rabbit-binder/README.adoc +++ /dev/null @@ -1,1305 +0,0 @@ -//// -DO NOT EDIT THIS FILE. IT WAS GENERATED. -Manual changes to this file will be lost when it is generated again. -Edit the files in the src/main/asciidoc/ directory instead. -//// - - -:jdkversion: 1.8 -:github-tag: master -:github-repo: spring-cloud/spring-cloud-stream-binder-rabbit - -:github-raw: https://raw.githubusercontent.com/{github-repo}/{github-tag} -:github-code: https://github.com/{github-repo}/tree/{github-tag} - -image::https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-rabbit.svg?style=svg["CircleCI", link="https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-rabbit"] -image::https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-rabbit/branch/{github-tag}/graph/badge.svg["codecov", link="https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-rabbit"] -image::https://badges.gitter.im/spring-cloud/spring-cloud-stream-binder-rabbit.svg[Gitter, link="https://gitter.im/spring-cloud/spring-cloud-stream-binder-rabbit?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge"] - -// ====================================================================================== - -//= Overview -[partintro] --- -This guide describes the RabbitMQ implementation of the Spring Cloud Stream Binder. -It contains information about its design, usage and configuration options, as well as information on how the Stream Cloud Stream concepts map into RabbitMQ specific constructs. --- - -== Usage - -To use the RabbitMQ binder, you can add it to your Spring Cloud Stream application, by using the following Maven coordinates: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-stream-binder-rabbit - ----- - -Alternatively, you can use the Spring Cloud Stream RabbitMQ Starter, as follows: - -[source,xml] ----- - - org.springframework.cloud - spring-cloud-starter-stream-rabbit - ----- - -== RabbitMQ Binder Overview - -The following simplified diagram shows how the RabbitMQ binder operates: - -.RabbitMQ Binder -image::{github-raw}/docs/src/main/asciidoc/images/rabbit-binder.png[width=300,scaledwidth="50%"] - -By default, the RabbitMQ Binder implementation maps each destination to a `TopicExchange`. -For each consumer group, a `Queue` is bound to that `TopicExchange`. -Each consumer instance has a corresponding RabbitMQ `Consumer` instance for its group's `Queue`. -For partitioned producers and consumers, the queues are suffixed with the partition index and use the partition index as the routing key. -For anonymous consumers (those with no `group` property), an auto-delete queue (with a randomized unique name) is used. - -By using the optional `autoBindDlq` option, you can configure the binder to create and configure dead-letter queues (DLQs) (and a dead-letter exchange `DLX`, as well as routing infrastructure). -By default, the dead letter queue has the name of the destination, appended with `.dlq`. -If retry is enabled (`maxAttempts > 1`), failed messages are delivered to the DLQ after retries are exhausted. -If retry is disabled (`maxAttempts = 1`), you should set `requeueRejected` to `false` (the default) so that failed messages are routed to the DLQ, instead of being re-queued. -In addition, `republishToDlq` causes the binder to publish a failed message to the DLQ (instead of rejecting it). -This feature lets additional information (such as the stack trace in the `x-exception-stacktrace` header) be added to the message in headers. -See the <> for information about truncated stack traces. -This option does not need retry enabled. -You can republish a failed message after just one attempt. -Starting with version 1.2, you can configure the delivery mode of republished messages. -See the <>. - -If the stream listener throws an `ImmediateAcknowledgeAmqpException`, the DLQ is bypassed and the message simply discarded. -Starting with version 2.1, this is true regardless of the setting of `republishToDlq`; previously it was only the case when `republishToDlq` was `false`. - -IMPORTANT: Setting `requeueRejected` to `true` (with `republishToDlq=false` ) causes the message to be re-queued and redelivered continually, which is likely not what you want unless the reason for the failure is transient. -In general, you should enable retry within the binder by setting `maxAttempts` to greater than one or by setting `republishToDlq` to `true`. - -Starting with version 3.1.2, if the consumer is marked as `transacted`, publishing to the DLQ will participate in the transaction. -This allows the transaction to roll back if the publishing fails for some reason (for example, if the user is not authorized to publish to the dead letter exchange). -In addition, if the connection factory is configured for publisher confirms or returns, the publication to the DLQ will wait for the confirmation and check for a returned message. -If a negative acknowledgment or returned message is received, the binder will throw an `AmqpRejectAndDontRequeueException`, allowing the broker to take care of publishing to the DLQ as if the `republishToDlq` property is `false`. - -See <> for more information about these properties. - -The framework does not provide any standard mechanism to consume dead-letter messages (or to re-route them back to the primary queue). -Some options are described in <>. - -NOTE: When multiple RabbitMQ binders are used in a Spring Cloud Stream application, it is important to disable 'RabbitAutoConfiguration' to avoid the same configuration from `RabbitAutoConfiguration` being applied to the two binders. -You can exclude the class by using the `@SpringBootApplication` annotation. - -Starting with version 2.0, the `RabbitMessageChannelBinder` sets the `RabbitTemplate.userPublisherConnection` property to `true` so that the non-transactional producers avoid deadlocks on consumers, which can happen if cached connections are blocked because of a https://www.rabbitmq.com/memory.html[memory alarm] on the broker. - -NOTE: Currently, a `multiplex` consumer (a single consumer listening to multiple queues) is only supported for message-driven consumers; polled consumers can only retrieve messages from a single queue. - -== Configuration Options - -This section contains settings specific to the RabbitMQ Binder and bound channels. - -For general binding configuration options and properties, see the https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/current/reference/html/spring-cloud-stream.html#_configuration_options[Spring Cloud Stream core documentation]. - -[[rabbit-binder-properties]] -=== RabbitMQ Binder Properties - -By default, the RabbitMQ binder uses Spring Boot's `ConnectionFactory`. -Conseuqently, it supports all Spring Boot configuration options for RabbitMQ. -(For reference, see the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#common-application-properties[Spring Boot documentation]). -RabbitMQ configuration options use the `spring.rabbitmq` prefix. - -In addition to Spring Boot options, the RabbitMQ binder supports the following properties: - -spring.cloud.stream.rabbit.binder.adminAddresses:: -A comma-separated list of RabbitMQ management plugin URLs. -Only used when `nodes` contains more than one entry. -Each entry in this list must have a corresponding entry in `spring.rabbitmq.addresses`. -Only needed if you use a RabbitMQ cluster and wish to consume from the node that hosts the queue. -See https://docs.spring.io/spring-amqp/reference/html/#queue-affinity[Queue Affinity and the LocalizedQueueConnectionFactory] for more information. -+ -Default: empty. -spring.cloud.stream.rabbit.binder.nodes:: -A comma-separated list of RabbitMQ node names. -When more than one entry, used to locate the server address where a queue is located. -Each entry in this list must have a corresponding entry in `spring.rabbitmq.addresses`. -Only needed if you use a RabbitMQ cluster and wish to consume from the node that hosts the queue. -See https://docs.spring.io/spring-amqp/reference/html/_reference.html#queue-affinity[Queue Affinity and the LocalizedQueueConnectionFactory] for more information. -+ -Default: empty. -spring.cloud.stream.rabbit.binder.compressionLevel:: -The compression level for compressed bindings. -See `java.util.zip.Deflater`. -+ -Default: `1` (BEST_LEVEL). -spring.cloud.stream.binder.connection-name-prefix:: -A connection name prefix used to name the connection(s) created by this binder. -The name is this prefix followed by `#n`, where `n` increments each time a new connection is opened. -+ -Default: none (Spring AMQP default). - -=== RabbitMQ Consumer Properties - -The following properties are available for Rabbit consumers only and must be prefixed with `spring.cloud.stream.rabbit.bindings..consumer.`. - -However if the same set of properties needs to be applied to most bindings, to -avoid repetition, Spring Cloud Stream supports setting values for all channels, -in the format of `spring.cloud.stream.rabbit.default.=`. - -Also, keep in mind that binding specific property will override its equivalent in the default. - -acknowledgeMode:: -The acknowledge mode. -+ -Default: `AUTO`. -anonymousGroupPrefix:: -When the binding has no `group` property, an anonymous, auto-delete queue is bound to the destination exchange. -The default naming stragegy for such queues results in a queue named `anonymous.`. -Set this property to change the prefix to something other than the default. -+ -Default: `anonymous.`. -autoBindDlq:: -Whether to automatically declare the DLQ and bind it to the binder DLX. -+ -Default: `false`. -bindingRoutingKey:: -The routing key with which to bind the queue to the exchange (if `bindQueue` is `true`). -Can be multiple keys - see `bindingRoutingKeyDelimiter`. -For partitioned destinations, `-` is appended to each key. -+ -Default: `#`. -bindingRoutingKeyDelimiter:: -When this is not null, 'bindingRoutingKey' is considered to be a list of keys delimited by this value; often a comma is used. -+ -Default: `null`. -bindQueue:: -Whether to declare the queue and bind it to the destination exchange. -Set it to `false` if you have set up your own infrastructure and have previously created and bound the queue. -+ -Default: `true`. -consumerTagPrefix:: -Used to create the consumer tag(s); will be appended by `#n` where `n` increments for each consumer created. -Example: `${spring.application.name}-${spring.cloud.stream.bindings.input.group}-${spring.cloud.stream.instance-index}`. -+ -Default: none - the broker will generate random consumer tags. -containerType:: -Select the type of listener container to be used. -See https://docs.spring.io/spring-amqp/reference/html/_reference.html#choose-container[Choosing a Container] in the Spring AMQP documentation for more information. -+ -Default: `simple` -deadLetterQueueName:: -The name of the DLQ -+ -Default: `prefix+destination.dlq` -deadLetterExchange:: -A DLX to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -+ -Default: 'prefix+DLX' -deadLetterExchangeType:: -The type of the DLX to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -+ -Default: 'direct' -deadLetterRoutingKey:: -A dead letter routing key to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -+ -Default: `destination` -declareDlx:: -Whether to declare the dead letter exchange for the destination. -Relevant only if `autoBindDlq` is `true`. -Set to `false` if you have a pre-configured DLX. -+ -Default: `true`. -declareExchange:: -Whether to declare the exchange for the destination. -+ -Default: `true`. -delayedExchange:: -Whether to declare the exchange as a `Delayed Message Exchange`. -Requires the delayed message exchange plugin on the broker. -The `x-delayed-type` argument is set to the `exchangeType`. -+ -Default: `false`. -dlqBindingArguments:: -Arguments applied when binding the dlq to the dead letter exchange; used with `headers` `deadLetterExchangeType` to specify headers to match on. -For example `...dlqBindingArguments.x-match=any`, `...dlqBindingArguments.someHeader=someValue`. -+ -Default: empty -dlqDeadLetterExchange:: -If a DLQ is declared, a DLX to assign to that queue. -+ -Default: `none` -dlqDeadLetterRoutingKey:: -If a DLQ is declared, a dead letter routing key to assign to that queue. -+ -Default: `none` -dlqExpires:: -How long before an unused dead letter queue is deleted (in milliseconds). -+ -Default: `no expiration` -dlqLazy:: -Declare the dead letter queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -+ -Default: `false`. -dlqMaxLength:: -Maximum number of messages in the dead letter queue. -+ -Default: `no limit` -dlqMaxLengthBytes:: -Maximum number of total bytes in the dead letter queue from all messages. -+ -Default: `no limit` -dlqMaxPriority:: -Maximum priority of messages in the dead letter queue (0-255). -+ -Default: `none` -dlqOverflowBehavior:: -Action to take when `dlqMaxLength` or `dlqMaxLengthBytes` is exceeded; currently `drop-head` or `reject-publish` but refer to the RabbitMQ documentation. -+ -Default: `none` -dlqQuorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -+ -Default: none - broker default will apply. -dlqQuorum.enabled:: -When true, create a quorum dead letter queue instead of a classic queue. -+ -Default: false -dlqQuorum.initialQuorumSize:: -When `quorum.enabled=true`, set the initial quorum size. -+ -Default: none - broker default will apply. -dlqSingleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -+ -Default: `false` -dlqTtl:: -Default time to live to apply to the dead letter queue when declared (in milliseconds). -+ -Default: `no limit` -durableSubscription:: -Whether the subscription should be durable. -Only effective if `group` is also set. -+ -Default: `true`. -exchangeAutoDelete:: -If `declareExchange` is true, whether the exchange should be auto-deleted (that is, removed after the last queue is removed). -+ -Default: `true`. -exchangeDurable:: -If `declareExchange` is true, whether the exchange should be durable (that is, it survives broker restart). -+ -Default: `true`. -exchangeType:: -The exchange type: `direct`, `fanout`, `headers` or `topic` for non-partitioned destinations and `direct`, headers or `topic` for partitioned destinations. -+ -Default: `topic`. -exclusive:: -Whether to create an exclusive consumer. -Concurrency should be 1 when this is `true`. -Often used when strict ordering is required but enabling a hot standby instance to take over after a failure. -See `recoveryInterval`, which controls how often a standby instance attempts to consume. -Consider using `singleActiveConsumer` instead when using RabbitMQ 3.8 or later. -+ -Default: `false`. -expires:: -How long before an unused queue is deleted (in milliseconds). -+ -Default: `no expiration` -failedDeclarationRetryInterval:: -The interval (in milliseconds) between attempts to consume from a queue if it is missing. -+ -Default: 5000 -[[spring-cloud-stream-rabbit-frame-max-headroom]] -frameMaxHeadroom:: -The number of bytes to reserve for other headers when adding the stack trace to a DLQ message header. -All headers must fit within the `frame_max` size configured on the broker. -Stack traces can be large; if the size plus this property exceeds `frame_max` then the stack trace will be truncated. -A WARN log will be written; consider increasing the `frame_max` or reducing the stack trace by catching the exception and throwing one with a smaller stack trace. -+ -Default: 20000 -headerPatterns:: -Patterns for headers to be mapped from inbound messages. -+ -Default: `['*']` (all headers). -lazy:: -Declare the queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -+ -Default: `false`. -maxConcurrency:: -The maximum number of consumers. -Not supported when the `containerType` is `direct`. -+ -Default: `1`. -maxLength:: -The maximum number of messages in the queue. -+ -Default: `no limit` -maxLengthBytes:: -The maximum number of total bytes in the queue from all messages. -+ -Default: `no limit` -maxPriority:: -The maximum priority of messages in the queue (0-255). -+ -Default: `none` -missingQueuesFatal:: -When the queue cannot be found, whether to treat the condition as fatal and stop the listener container. -Defaults to `false` so that the container keeps trying to consume from the queue -- for example, when using a cluster and the node hosting a non-HA queue is down. -+ -Default: `false` -overflowBehavior:: -Action to take when `maxLength` or `maxLengthBytes` is exceeded; currently `drop-head` or `reject-publish` but refer to the RabbitMQ documentation. -+ -Default: `none` -prefetch:: -Prefetch count. -+ -Default: `1`. -prefix:: -A prefix to be added to the name of the `destination` and queues. -+ -Default: "". -queueBindingArguments:: -Arguments applied when binding the queue to the exchange; used with `headers` `exchangeType` to specify headers to match on. -For example `...queueBindingArguments.x-match=any`, `...queueBindingArguments.someHeader=someValue`. -+ -Default: empty -queueDeclarationRetries:: -The number of times to retry consuming from a queue if it is missing. -Relevant only when `missingQueuesFatal` is `true`. -Otherwise, the container keeps retrying indefinitely. -Not supported when the `containerType` is `direct`. -+ -Default: `3` -queueNameGroupOnly:: -When true, consume from a queue with a name equal to the `group`. -Otherwise the queue name is `destination.group`. -This is useful, for example, when using Spring Cloud Stream to consume from an existing RabbitMQ queue. -+ -Default: false. -quorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -+ -Default: none - broker default will apply. -quorum.enabled:: -When true, create a quorum queue instead of a classic queue. -+ -Default: false -quorum.initialQuorumSize:: -When `quorum.enabled=true`, set the initial quorum size. -+ -Default: none - broker default will apply. -recoveryInterval:: -The interval between connection recovery attempts, in milliseconds. -+ -Default: `5000`. -requeueRejected:: -Whether delivery failures should be re-queued when retry is disabled or `republishToDlq` is `false`. -+ -Default: `false`. -[[spring-cloud-stream-rabbit-republish-delivery-mode]] -republishDeliveryMode:: -When `republishToDlq` is `true`, specifies the delivery mode of the republished message. -+ -Default: `DeliveryMode.PERSISTENT` -republishToDlq:: -By default, messages that fail after retries are exhausted are rejected. -If a dead-letter queue (DLQ) is configured, RabbitMQ routes the failed message (unchanged) to the DLQ. -If set to `true`, the binder republishs failed messages to the DLQ with additional headers, including the exception message and stack trace from the cause of the final failure. -Also see the <>. -+ -Default: `true` -singleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -+ -Default: `false` -transacted:: -Whether to use transacted channels. -+ -Default: `false`. -ttl:: -Default time to live to apply to the queue when declared (in milliseconds). -+ -Default: `no limit` -txSize:: -The number of deliveries between acks. -Not supported when the `containerType` is `direct`. -+ -Default: `1`. - -=== Advanced Listener Container Configuration - -To set listener container properties that are not exposed as binder or binding properties, add a single bean of type `ListenerContainerCustomizer` to the application context. -The binder and binding properties will be set and then the customizer will be called. -The customizer (`configure()` method) is provided with the queue name as well as the consumer group as arguments. - -=== Advanced Queue/Exchange/Binding Configuration - -From time to time, the RabbitMQ team add new features that are enabled by setting some argument when declaring, for example, a queue. -Generally, such features are enabled in the binder by adding appropriate properties, but this may not be immediately available in a current version. -Starting with version 3.0.1, you can now add `DeclarableCustomizer` bean(s) to the application context to modify a `Declarable` (`Queue`, `Exchange` or `Binding`) just before the declaration is performed. -This allows you to add arguments that are not currently directly supported by the binder. - -[[rabbit-receiving-batch]] -=== Receiving Batched Messages - -With the RabbitMQ binder, there are two types of batches handled by consumer bindings: - -==== Batches Created by Producers - -Normally, if a producer binding has `batch-enabled=true` (see <>), or a message is created by a `BatchingRabbitTemplate`, elements of the batch are returned as individual calls to the listener method. -Starting with version 3.0, any such batch can be presented as a `List` to the listener method if `spring.cloud.stream.bindings..consumer.batch-mode` is set to `true`. - -==== Consumer-side Batching - -Starting with version 3.1, the consumer can be configured to assemble multiple inbound messages into a batch which is presented to the application as a `List` of converted payloads. -The following simple application demonstrates how to use this technique: - -==== -[source, properties] ----- -spring.cloud.stream.bindings.input-in-0.group=someGroup - -spring.cloud.stream.bindings.input-in-0.consumer.batch-mode=true - -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.enable-batching=true -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.batch-size=10 -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.receive-timeout=200 ----- -==== - -==== -[source, java] ----- -@SpringBootApplication -public class Application { - - public static void main(String[] args) { - SpringApplication.run(Application.class, args); - } - - @Bean - Consumer> input() { - return list -> { - System.out.println("Received " + list.size()); - list.forEach(thing -> { - System.out.println(thing); - - // ... - - }); - }; - } - - @Bean - public ApplicationRunner runner(RabbitTemplate template) { - return args -> { - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value1\"}"); - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value2\"}"); - }; - } - - public static class Thing { - - private String field; - - public Thing() { - } - - public Thing(String field) { - this.field = field; - } - - public String getField() { - return this.field; - } - - public void setField(String field) { - this.field = field; - } - - @Override - public String toString() { - return "Thing [field=" + this.field + "]"; - } - - } - -} ----- -==== - -==== -[source] ----- -Received 2 -Thing [field=value1] -Thing [field=value2] ----- -==== - -The number of messages in a batch is specified by the `batch-size` and `receive-timeout` properties; if the `receive-timeout` elapses with no new messages, a "short" batch is delivered. - -IMPORTANT: Consumer-side batching is only supported with `container-type=simple` (the default). - -If you wish to examine headers of consumer-side batched messages, you should consume `Message>`; the headers are a `List>` in a header `AmqpInboundChannelAdapter.CONSOLIDATED_HEADERS`, with the headers for each payload element in the corresponding index. -Again, here is a simple example: - -==== -[source, java] ----- -@SpringBootApplication -public class Application { - - public static void main(String[] args) { - SpringApplication.run(Application.class, args); - } - - @Bean - Consumer>> input() { - return msg -> { - List things = msg.getPayload(); - System.out.println("Received " + things.size()); - @SuppressWarnings("unchecked") - List> headers = - (List>) msg.getHeaders().get(AmqpInboundChannelAdapter.CONSOLIDATED_HEADERS); - for (int i = 0; i < things.size(); i++) { - System.out.println(things.get(i) + " myHeader=" + headers.get(i).get("myHeader")); - - // ... - - } - }; - } - - @Bean - public ApplicationRunner runner(RabbitTemplate template) { - return args -> { - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value1\"}", msg -> { - msg.getMessageProperties().setHeader("myHeader", "headerValue1"); - return msg; - }); - template.convertAndSend("input-in-0.someGroup", "{\"field\":\"value2\"}", msg -> { - msg.getMessageProperties().setHeader("myHeader", "headerValue2"); - return msg; - }); - }; - } - - public static class Thing { - - private String field; - - public Thing() { - } - - public Thing(String field) { - this.field = field; - } - - public String getfield() { - return this.field; - } - - public void setfield(String field) { - this.field = field; - } - - @Override - public String toString() { - return "Thing [field=" + this.field + "]"; - } - - } - -} ----- -==== - -==== -[source] ----- -Received 2 -Thing [field=value1] myHeader=headerValue1 -Thing [field=value2] myHeader=headerValue2 ----- -==== - -[[rabbit-prod-props]] -=== Rabbit Producer Properties - -The following properties are available for Rabbit producers only and must be prefixed with `spring.cloud.stream.rabbit.bindings..producer.`. - -However if the same set of properties needs to be applied to most bindings, to -avoid repetition, Spring Cloud Stream supports setting values for all channels, -in the format of `spring.cloud.stream.rabbit.default.=`. - -Also, keep in mind that binding specific property will override its equivalent in the default. - - -autoBindDlq:: -Whether to automatically declare the DLQ and bind it to the binder DLX. -+ -Default: `false`. -batchingEnabled:: -Whether to enable message batching by producers. -Messages are batched into one message according to the following properties (described in the next three entries in this list): 'batchSize', `batchBufferLimit`, and `batchTimeout`. -See https://docs.spring.io/spring-amqp//reference/html/_reference.html#template-batching[Batching] for more information. -Also see <>. -+ -Default: `false`. -batchSize:: -The number of messages to buffer when batching is enabled. -+ -Default: `100`. -batchBufferLimit:: -The maximum buffer size when batching is enabled. -+ -Default: `10000`. -batchTimeout:: -The batch timeout when batching is enabled. -+ -Default: `5000`. -bindingRoutingKey:: -The routing key with which to bind the queue to the exchange (if `bindQueue` is `true`). -Can be multiple keys - see `bindingRoutingKeyDelimiter`. -For partitioned destinations, `-n` is appended to each key. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `#`. -bindingRoutingKeyDelimiter:: -When this is not null, 'bindingRoutingKey' is considered to be a list of keys delimited by this value; often a comma is used. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `null`. -bindQueue:: -Whether to declare the queue and bind it to the destination exchange. -Set it to `false` if you have set up your own infrastructure and have previously created and bound the queue. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `true`. -compress:: -Whether data should be compressed when sent. -+ -Default: `false`. -confirmAckChannel:: -When `errorChannelEnabled` is true, a channel to which to send positive delivery acknowledgments (aka publisher confirms). -If the channel does not exist, a `DirectChannel` is registered with this name. -The connection factory must be configured to enable publisher confirms. -Mutually exclusive with `useConfirmHeader`. -+ -Default: `nullChannel` (acks are discarded). -deadLetterQueueName:: -The name of the DLQ -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `prefix+destination.dlq` -deadLetterExchange:: -A DLX to assign to the queue. -Relevant only when `autoBindDlq` is `true`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: 'prefix+DLX' -deadLetterExchangeType:: -The type of the DLX to assign to the queue. -Relevant only if `autoBindDlq` is `true`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: 'direct' -deadLetterRoutingKey:: -A dead letter routing key to assign to the queue. -Relevant only when `autoBindDlq` is `true`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `destination` -declareDlx:: -Whether to declare the dead letter exchange for the destination. -Relevant only if `autoBindDlq` is `true`. -Set to `false` if you have a pre-configured DLX. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `true`. -declareExchange:: -Whether to declare the exchange for the destination. -+ -Default: `true`. -delayExpression:: -A SpEL expression to evaluate the delay to apply to the message (`x-delay` header). -It has no effect if the exchange is not a delayed message exchange. -+ -Default: No `x-delay` header is set. -delayedExchange:: -Whether to declare the exchange as a `Delayed Message Exchange`. -Requires the delayed message exchange plugin on the broker. -The `x-delayed-type` argument is set to the `exchangeType`. -+ -Default: `false`. -deliveryMode:: -The delivery mode. -+ -Default: `PERSISTENT`. -dlqBindingArguments:: -Arguments applied when binding the dlq to the dead letter exchange; used with `headers` `deadLetterExchangeType` to specify headers to match on. -For example `...dlqBindingArguments.x-match=any`, `...dlqBindingArguments.someHeader=someValue`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: empty -dlqDeadLetterExchange:: -When a DLQ is declared, a DLX to assign to that queue. -Applies only if `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -dlqDeadLetterRoutingKey:: -When a DLQ is declared, a dead letter routing key to assign to that queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -dlqExpires:: -How long (in milliseconds) before an unused dead letter queue is deleted. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no expiration` -dlqLazy:: -Declare the dead letter queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -dlqMaxLength:: -Maximum number of messages in the dead letter queue. -Applies only if `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -dlqMaxLengthBytes:: -Maximum number of total bytes in the dead letter queue from all messages. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -dlqMaxPriority:: -Maximum priority of messages in the dead letter queue (0-255) -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -dlqQuorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -dlqQuorum.enabled:: -When true, create a quorum dead letter queue instead of a classic queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: false -dlqQuorum.initialQuorumSize:: -When `quorum.enabled=true`, set the initial quorum size. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -dlqSingleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `false` -dlqTtl:: -Default time (in milliseconds) to live to apply to the dead letter queue when declared. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -exchangeAutoDelete:: -If `declareExchange` is `true`, whether the exchange should be auto-delete (it is removed after the last queue is removed). -+ -Default: `true`. -exchangeDurable:: -If `declareExchange` is `true`, whether the exchange should be durable (survives broker restart). -+ -Default: `true`. -exchangeType:: -The exchange type: `direct`, `fanout`, `headers` or `topic` for non-partitioned destinations and `direct`, `headers` or `topic` for partitioned destinations. -+ -Default: `topic`. -expires:: -How long (in milliseconds) before an unused queue is deleted. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no expiration` -headerPatterns:: -Patterns for headers to be mapped to outbound messages. -+ -Default: `['*']` (all headers). -lazy:: -Declare the queue with the `x-queue-mode=lazy` argument. -See https://www.rabbitmq.com/lazy-queues.html["`Lazy Queues`"]. -Consider using a policy instead of this setting, because using a policy allows changing the setting without deleting the queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `false`. -maxLength:: -Maximum number of messages in the queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -maxLengthBytes:: -Maximum number of total bytes in the queue from all messages. -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -maxPriority:: -Maximum priority of messages in the queue (0-255). -Only applies if `requiredGroups` are provided and then only to those groups. -+ -Default: `none` -prefix:: -A prefix to be added to the name of the `destination` exchange. -+ -Default: "". -queueBindingArguments:: -Arguments applied when binding the queue to the exchange; used with `headers` `exchangeType` to specify headers to match on. -For example `...queueBindingArguments.x-match=any`, `...queueBindingArguments.someHeader=someValue`. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: empty -queueNameGroupOnly:: -When `true`, consume from a queue with a name equal to the `group`. -Otherwise the queue name is `destination.group`. -This is useful, for example, when using Spring Cloud Stream to consume from an existing RabbitMQ queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: false. -quorum.deliveryLimit:: -When `quorum.enabled=true`, set a delivery limit after which the message is dropped or dead-lettered. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -quorum.enabled:: -When true, create a quorum queue instead of a classic queue. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: false -quorum.initialQuorumSize:: -When `quorum.enabled=true`, set the initial quorum size. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: none - broker default will apply. -routingKeyExpression:: -A SpEL expression to determine the routing key to use when publishing messages. -For a fixed routing key, use `routingKey`. -+ -Default: `destination` or `destination-` for partitioned destinations. -routingKey:: -A string defining a fixed routing key to use when publishing messages. -+ -Default: see `routingKeyExpression` -singleActiveConsumer:: -Set to true to set the `x-single-active-consumer` queue property to true. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `false` -transacted:: -Whether to use transacted channels. -+ -Default: `false`. -ttl:: -Default time (in milliseconds) to live to apply to the queue when declared. -Applies only when `requiredGroups` are provided and then only to those groups. -+ -Default: `no limit` -useConfirmHeader:: -See <>. -Mutually exclusive with `confirmAckChannel`. -+ - -NOTE: In the case of RabbitMQ, content type headers can be set by external applications. -Spring Cloud Stream supports them as part of an extended internal protocol used for any type of transport -- including transports, such as Kafka (prior to 0.11), that do not natively support headers. - -[[publisher-confirms]] -=== Publisher Confirms - -There are two mechanisms to get the result of publishing a message; in each case, the connection factory must have `publisherConfirmType` set `ConfirmType.CORRELATED`. -The "legacy" mechanism is to set the `confirmAckChannel` to the bean name of a message channel from which you can retrieve the confirmations asynchronously; negative acks are sent to the error channel (if enabled) - see <>. - -The preferred mechanism, added in version 3.1 is to use a correlation data header and wait for the result via its `Future` property. -This is particularly useful with a batch listener because you can send multiple messages before waiting for the result. -To use this technique, set the `useConfirmHeader` property to true -The following simple application is an example of using this technique: - -==== -[source, properties] ----- -spring.cloud.stream.bindings.input-in-0.group=someGroup -spring.cloud.stream.bindings.input-in-0.consumer.batch-mode=true - -spring.cloud.stream.source=output -spring.cloud.stream.bindings.output-out-0.producer.error-channel-enabled=true - -spring.cloud.stream.rabbit.bindings.output-out-0.producer.useConfirmHeader=true -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.auto-bind-dlq=true -spring.cloud.stream.rabbit.bindings.input-in-0.consumer.batch-size=10 - -spring.rabbitmq.publisher-confirm-type=correlated -spring.rabbitmq.publisher-returns=true ----- -==== - -==== -[source, java] ----- -@SpringBootApplication -public class Application { - - private static final Logger log = LoggerFactory.getLogger(Application.class); - - public static void main(String[] args) { - SpringApplication.run(Application.class, args); - } - - @Autowired - private StreamBridge bridge; - - @Bean - Consumer> input() { - return list -> { - List results = new ArrayList<>(); - list.forEach(str -> { - log.info("Received: " + str); - MyCorrelationData corr = new MyCorrelationData(UUID.randomUUID().toString(), str); - results.add(corr); - this.bridge.send("output-out-0", MessageBuilder.withPayload(str.toUpperCase()) - .setHeader(AmqpHeaders.PUBLISH_CONFIRM_CORRELATION, corr) - .build()); - }); - results.forEach(correlation -> { - try { - Confirm confirm = correlation.getFuture().get(10, TimeUnit.SECONDS); - log.info(confirm + " for " + correlation.getPayload()); - if (correlation.getReturnedMessage() != null) { - log.error("Message for " + correlation.getPayload() + " was returned "); - - // try to re-publish, send a DLQ, etc - - } - } - catch (InterruptedException e) { - Thread.currentThread().interrupt(); - e.printStackTrace(); - } - catch (ExecutionException | TimeoutException e) { - e.printStackTrace(); - } - }); - }; - } - - - @Bean - public ApplicationRunner runner(BatchingRabbitTemplate template) { - return args -> IntStream.range(0, 10).forEach(i -> - template.convertAndSend("input-in-0", "input-in-0.rbgh303", "foo" + i)); - } - - @Bean - public BatchingRabbitTemplate template(CachingConnectionFactory cf, TaskScheduler taskScheduler) { - BatchingStrategy batchingStrategy = new SimpleBatchingStrategy(10, 1000000, 1000); - return new BatchingRabbitTemplate(cf, batchingStrategy, taskScheduler); - } - -} - -class MyCorrelationData extends CorrelationData { - - private final String payload; - - MyCorrelationData(String id, String payload) { - super(id); - this.payload = payload; - } - - public String getPayload() { - return this.payload; - } - -} ----- -==== - -As you can see, we send each message and then await for the publication results. -If the messages can't be routed, then correlation data is populated with the returned message before the future is completed. - -IMPORTANT: The correlation data must be provided with a unique `id` so that the framework can perform the correlation. - -You cannot set both `useConfirmHeader` and `confirmAckChannel` but you can still receive returned messages in the error channel when `useConfirmHeader` is true, but using the correlation header is more convenient. - -== Using Existing Queues/Exchanges - -By default, the binder will automatically provision a topic exchange with the name being derived from the value of the destination binding property ``. -The destination defaults to the binding name, if not provided. -When binding a consumer, a queue will automatically be provisioned with the name `.` (if a `group` binding property is specified), or an anonymous, auto-delete queue when there is no `group`. -The queue will be bound to the exchange with the "match-all" wildcard routing key (`#`) for a non-partitioned binding or `-` for a partitioned binding. -The prefix is an empty `String` by default. -If an output binding is specified with `requiredGroups`, a queue/binding will be provisioned for each group. - -There are a number of rabbit-specific binding properties that allow you to modify this default behavior. - -If you have an existing exchange/queue that you wish to use, you can completely disable automatic provisioning as follows, assuming the exchange is named `myExchange` and the queue is named `myQueue`: - -* `spring.cloud.stream.bindings..destination=myExhange` -* `spring.cloud.stream.bindings..group=myQueue` -* `spring.cloud.stream.rabbit.bindings..consumer.bindQueue=false` -* `spring.cloud.stream.rabbit.bindings..consumer.declareExchange=false` -* `spring.cloud.stream.rabbit.bindings..consumer.queueNameGroupOnly=true` - -If you want the binder to provision the queue/exchange, but you want to do it using something other than the defaults discussed here, use the following properties. -Refer to the property documentation above for more information. - -* `spring.cloud.stream.rabbit.bindings..consumer.bindingRoutingKey=myRoutingKey` -* `spring.cloud.stream.rabbit.bindings..consumer.exchangeType=` - -* `spring.cloud.stream.rabbit.bindings..producer.routingKeyExpression='myRoutingKey'` - -There are similar properties used when declaring a dead-letter exchange/queue, when `autoBindDlq` is `true`. - -== Retry With the RabbitMQ Binder - -When retry is enabled within the binder, the listener container thread is suspended for any back off periods that are configured. -This might be important when strict ordering is required with a single consumer. However, for other use cases, it prevents other messages from being processed on that thread. -An alternative to using binder retry is to set up dead lettering with time to live on the dead-letter queue (DLQ) as well as dead-letter configuration on the DLQ itself. -See "`<>`" for more information about the properties discussed here. -You can use the following example configuration to enable this feature: - -* Set `autoBindDlq` to `true`. -The binder create a DLQ. -Optionally, you can specify a name in `deadLetterQueueName`. -* Set `dlqTtl` to the back off time you want to wait between redeliveries. -* Set the `dlqDeadLetterExchange` to the default exchange. -Expired messages from the DLQ are routed to the original queue, because the default `deadLetterRoutingKey` is the queue name (`destination.group`). -Setting to the default exchange is achieved by setting the property with no value, as shown in the next example. - -To force a message to be dead-lettered, either throw an `AmqpRejectAndDontRequeueException` or set `requeueRejected` to `false` (the default) and throw any exception. - -The loop continue without end, which is fine for transient problems, but you may want to give up after some number of attempts. -Fortunately, RabbitMQ provides the `x-death` header, which lets you determine how many cycles have occurred. - -To acknowledge a message after giving up, throw an `ImmediateAcknowledgeAmqpException`. - -=== Putting it All Together - -The following configuration creates an exchange `myDestination` with queue `myDestination.consumerGroup` bound to a topic exchange with a wildcard routing key `#`: - -[source] ---- -spring.cloud.stream.bindings.input.destination=myDestination -spring.cloud.stream.bindings.input.group=consumerGroup -#disable binder retries -spring.cloud.stream.bindings.input.consumer.max-attempts=1 -#dlx/dlq setup -spring.cloud.stream.rabbit.bindings.input.consumer.auto-bind-dlq=true -spring.cloud.stream.rabbit.bindings.input.consumer.dlq-ttl=5000 -spring.cloud.stream.rabbit.bindings.input.consumer.dlq-dead-letter-exchange= ---- - -This configuration creates a DLQ bound to a direct exchange (`DLX`) with a routing key of `myDestination.consumerGroup`. -When messages are rejected, they are routed to the DLQ. -After 5 seconds, the message expires and is routed to the original queue by using the queue name as the routing key, as shown in the following example: - -.Spring Boot application -[source, java] ----- -@SpringBootApplication -@EnableBinding(Sink.class) -public class XDeathApplication { - - public static void main(String[] args) { - SpringApplication.run(XDeathApplication.class, args); - } - - @StreamListener(Sink.INPUT) - public void listen(String in, @Header(name = "x-death", required = false) Map death) { - if (death != null && death.get("count").equals(3L)) { - // giving up - don't send to DLX - throw new ImmediateAcknowledgeAmqpException("Failed after 4 attempts"); - } - throw new AmqpRejectAndDontRequeueException("failed"); - } - -} ----- - -Notice that the count property in the `x-death` header is a `Long`. - -[[rabbit-error-channels]] -== Error Channels - -Starting with version 1.3, the binder unconditionally sends exceptions to an error channel for each consumer destination and can also be configured to send async producer send failures to an error channel. -See "`<>`" for more information. - -RabbitMQ has two types of send failures: - -* Returned messages, -* Negatively acknowledged https://www.rabbitmq.com/confirms.html[Publisher Confirms]. - -The latter is rare. -According to the RabbitMQ documentation "[A nack] will only be delivered if an internal error occurs in the Erlang process responsible for a queue.". -You can also get a negative acknowledgment if you publish to a bounded queue with `reject-publish` queue overflow behavior. - -As well as enabling producer error channels (as described in "`<>`"), the RabbitMQ binder only sends messages to the channels if the connection factory is appropriately configured, as follows: - -* `ccf.setPublisherConfirms(true);` -* `ccf.setPublisherReturns(true);` - -When using Spring Boot configuration for the connection factory, set the following properties: - -* `spring.rabbitmq.publisher-confirms` -* `spring.rabbitmq.publisher-returns` - -The payload of the `ErrorMessage` for a returned message is a `ReturnedAmqpMessageException` with the following properties: - -* `failedMessage`: The spring-messaging `Message` that failed to be sent. -* `amqpMessage`: The raw spring-amqp `Message`. -* `replyCode`: An integer value indicating the reason for the failure (for example, 312 - No route). -* `replyText`: A text value indicating the reason for the failure (for example, `NO_ROUTE`). -* `exchange`: The exchange to which the message was published. -* `routingKey`: The routing key used when the message was published. - -Also see <> for an alternative mechanism to receive returned messages. - -For negatively acknowledged confirmations, the payload is a `NackedAmqpMessageException` with the following properties: - -* `failedMessage`: The spring-messaging `Message` that failed to be sent. -* `nackReason`: A reason (if available -- you may need to examine the broker logs for more information). - -There is no automatic handling of these exceptions (such as sending to a <>). -You can consume these exceptions with your own Spring Integration flow. - -= Appendices -[appendix] -[[building]] -== Building - -:jdkversion: 1.8 - -=== Basic Compile and Test - -Pre-requisites: - -* To compile, JDK {jdkversion} installed. -* To run tests, RabbitMQ server running on `localhost:5672` - - -The build uses the Maven wrapper so you don't have to install a specific -version of Maven. The main build command is - ----- -$ ./mvnw clean install ----- - -NOTE: There are scripts in `./ci-docker-compose` that use https://docs.docker.com/compose//[Docker Compose] to -start/stop a local RabbitMQ server. - -You can also add '-DskipTests' if you like, to avoid running the tests. - -NOTE: You can also install Maven (>=3.3.3) yourself and run the `mvn` command -in place of `./mvnw` in the examples below. If you do that you also -might need to add `-P spring` if your local Maven settings do not -contain repository declarations for spring pre-release artifacts. - -NOTE: Be aware that you might need to increase the amount of memory -available to Maven by setting a `MAVEN_OPTS` environment variable with -a value like `-Xmx512m -XX:MaxPermSize=128m`. We try to cover this in -the `.mvn` configuration, so if you find you have to do it to make a -build succeed, please raise a ticket to get the settings added to -source control. - - -=== Documentation - -There is a "docs" profile that will generate documentation. - -`./mvnw clean package -Pdocs -DskipTests` - -The reference documentation can then be found in `docs/target/contents/reference`. - -=== Working with the code -If you don't have an IDE preference we would recommend that you use -https://www.springsource.com/developer/sts[Spring Tools Suite] or -https://eclipse.org[Eclipse] when working with the code. We use the -https://eclipse.org/m2e/[m2eclipe] eclipse plugin for maven support. Other IDEs and tools -should also work without issue. - -==== Importing into eclipse with m2eclipse -We recommend the https://eclipse.org/m2e/[m2eclipe] eclipse plugin when working with -eclipse. If you don't already have m2eclipse installed it is available from the "eclipse -marketplace". - -Unfortunately m2e does not yet support Maven 3.3, so once the projects -are imported into Eclipse you will also need to tell m2eclipse to use -the `.settings.xml` file for the projects. If you do not do this you -may see many different errors related to the POMs in the -projects. Open your Eclipse preferences, expand the Maven -preferences, and select User Settings. In the User Settings field -click Browse and navigate to the Spring Cloud project you imported -selecting the `.settings.xml` file in that project. Click Apply and -then OK to save the preference changes. - -NOTE: Alternatively you can copy the repository settings from https://github.com/spring-cloud/spring-cloud-build/blob/master/.settings.xml[`.settings.xml`] into your own `~/.m2/settings.xml`. - -==== Importing into eclipse without m2eclipse -If you prefer not to use m2eclipse you can generate eclipse project metadata using the -following command: - -[indent=0] ----- - $ ./mvnw eclipse:eclipse ----- - -The generated eclipse projects can be imported by selecting `import existing projects` -from the `file` menu. -[[contributing]] -== Contributing - -Spring Cloud is released under the non-restrictive Apache 2.0 license, -and follows a very standard Github development process, using Github -tracker for issues and merging pull requests into master. If you want -to contribute even something trivial please do not hesitate, but -follow the guidelines below. - -=== Sign the Contributor License Agreement -Before we accept a non-trivial patch or pull request we will need you to sign the -https://support.springsource.com/spring_committer_signup[contributor's agreement]. -Signing the contributor's agreement does not grant anyone commit rights to the main -repository, but it does mean that we can accept your contributions, and you will get an -author credit if we do. Active contributors might be asked to join the core team, and -given the ability to merge pull requests. - -=== Code Conventions and Housekeeping -None of these is essential for a pull request, but they will all help. They can also be -added after the original pull request but before a merge. - -* Use the Spring Framework code format conventions. If you use Eclipse - you can import formatter settings using the - `eclipse-code-formatter.xml` file from the - https://github.com/spring-cloud/build/tree/master/eclipse-coding-conventions.xml[Spring - Cloud Build] project. If using IntelliJ, you can use the - https://plugins.jetbrains.com/plugin/6546[Eclipse Code Formatter - Plugin] to import the same file. -* Make sure all new `.java` files to have a simple Javadoc class comment with at least an - `@author` tag identifying you, and preferably at least a paragraph on what the class is - for. -* Add the ASF license header comment to all new `.java` files (copy from existing files - in the project) -* Add yourself as an `@author` to the .java files that you modify substantially (more - than cosmetic changes). -* Add some Javadocs and, if you change the namespace, some XSD doc elements. -* A few unit tests would help a lot as well -- someone has to do it. -* If no-one else is using your branch, please rebase it against the current master (or - other target branch in the main project). -* When writing a commit message please follow https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html[these conventions], - if you are fixing an existing issue please add `Fixes gh-XXXX` at the end of the commit - message (where XXXX is the issue number). - -// ====================================================================================== diff --git a/binders/rabbit-binder/README.md b/binders/rabbit-binder/README.md deleted file mode 100644 index ab7f53dc4..000000000 --- a/binders/rabbit-binder/README.md +++ /dev/null @@ -1,2 +0,0 @@ -# spring-cloud-stream-binder-rabbit -Spring Cloud Stream Binder implementation for Rabbit diff --git a/binders/rabbit-binder/docs/src/main/asciidoc/README.adoc b/binders/rabbit-binder/docs/src/main/asciidoc/README.adoc deleted file mode 100644 index 3a6c87de1..000000000 --- a/binders/rabbit-binder/docs/src/main/asciidoc/README.adoc +++ /dev/null @@ -1,22 +0,0 @@ -:jdkversion: 1.8 -:github-tag: master -:github-repo: spring-cloud/spring-cloud-stream-binder-rabbit - -:github-raw: https://raw.githubusercontent.com/{github-repo}/{github-tag} -:github-code: https://github.com/{github-repo}/tree/{github-tag} - -image::https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-rabbit.svg?style=svg["CircleCI", link="https://circleci.com/gh/spring-cloud/spring-cloud-stream-binder-rabbit"] -image::https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-rabbit/branch/{github-tag}/graph/badge.svg["codecov", link="https://codecov.io/gh/spring-cloud/spring-cloud-stream-binder-rabbit"] -image::https://badges.gitter.im/spring-cloud/spring-cloud-stream-binder-rabbit.svg[Gitter, link="https://gitter.im/spring-cloud/spring-cloud-stream-binder-rabbit?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge"] - -// ====================================================================================== - -//= Overview -include::overview.adoc[] - -= Appendices -[appendix] -include::building.adoc[] -include::contributing.adoc[] - -// ====================================================================================== diff --git a/core/docs/pom.xml b/core/docs/pom.xml index a6fe56999..cc7f21778 100644 --- a/core/docs/pom.xml +++ b/core/docs/pom.xml @@ -62,6 +62,29 @@ maven-antrun-plugin + + + readme + prepare-package + + run + + + + + + + + + + + + + + + + maven-deploy-plugin diff --git a/core/docs/src/main/asciidoc/README.adoc b/core/docs/src/main/asciidoc/README.adoc index 235d8e2e3..3ef666aa2 100644 --- a/core/docs/src/main/asciidoc/README.adoc +++ b/core/docs/src/main/asciidoc/README.adoc @@ -1,4 +1,4 @@ -:jdkversion: 1.8 +:jdkversion: 17 :github-tag: master :github-repo: spring-cloud/spring-cloud-stream @@ -11,136 +11,17 @@ image::https://badges.gitter.im/spring-cloud/spring-cloud-stream.svg[Gitter, lin // ====================================================================================== -= Preface -include::preface.adoc[] +== Introuduction +include::intro.adoc[] + +== Resources + +For more information, please visit the https://spring.io/projects/spring-cloud-stream[project website]: -= Appendices -[appendix] -[[building]] == Building -:jdkversion: 1.8 +include::https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/docs/src/main/asciidoc/building.adoc[] -=== Basic Compile and Test - -To build the source you will need to install JDK {jdkversion}. - -The build uses the Maven wrapper so you don't have to install a specific -version of Maven. To enable the tests for Redis, Rabbit, and Kafka bindings you -should have those servers running before building. See below for more -information on running the servers. - -The main build command is - ----- -$ ./mvnw clean install ----- - -You can also add '-DskipTests' if you like, to avoid running the tests. - -NOTE: You can also install Maven (>=3.3.3) yourself and run the `mvn` command -in place of `./mvnw` in the examples below. If you do that you also -might need to add `-P spring` if your local Maven settings do not -contain repository declarations for spring pre-release artifacts. - -NOTE: Be aware that you might need to increase the amount of memory -available to Maven by setting a `MAVEN_OPTS` environment variable with -a value like `-Xmx512m -XX:MaxPermSize=128m`. We try to cover this in -the `.mvn` configuration, so if you find you have to do it to make a -build succeed, please raise a ticket to get the settings added to -source control. - - -The projects that require middleware generally include a -`docker-compose.yml`, so consider using -http://compose.docker.io/[Docker Compose] to run the middeware servers -in Docker containers. See the README in the -https://github.com/spring-cloud-samples/scripts[scripts demo -repository] for specific instructions about the common cases of mongo, -rabbit and redis. - -=== Documentation - -There is a "full" profile that will generate documentation. - -=== Working with the code -If you don't have an IDE preference we would recommend that you use -http://www.springsource.com/developer/sts[Spring Tools Suite] or -http://eclipse.org[Eclipse] when working with the code. We use the -http://eclipse.org/m2e/[m2eclipe] eclipse plugin for maven support. Other IDEs and tools -should also work without issue. - -==== Importing into eclipse with m2eclipse -We recommend the http://eclipse.org/m2e/[m2eclipe] eclipse plugin when working with -eclipse. If you don't already have m2eclipse installed it is available from the "eclipse -marketplace". - -Unfortunately m2e does not yet support Maven 3.3, so once the projects -are imported into Eclipse you will also need to tell m2eclipse to use -the `.settings.xml` file for the projects. If you do not do this you -may see many different errors related to the POMs in the -projects. Open your Eclipse preferences, expand the Maven -preferences, and select User Settings. In the User Settings field -click Browse and navigate to the Spring Cloud project you imported -selecting the `.settings.xml` file in that project. Click Apply and -then OK to save the preference changes. - -NOTE: Alternatively you can copy the repository settings from https://github.com/spring-cloud/spring-cloud-build/blob/master/.settings.xml[`.settings.xml`] into your own `~/.m2/settings.xml`. - -==== Importing into eclipse without m2eclipse -If you prefer not to use m2eclipse you can generate eclipse project metadata using the -following command: - -[indent=0] ----- - $ ./mvnw eclipse:eclipse ----- - -The generated eclipse projects can be imported by selecting `import existing projects` -from the `file` menu. - -[[contributing]] == Contributing -Spring Cloud is released under the non-restrictive Apache 2.0 license, -and follows a very standard Github development process, using Github -tracker for issues and merging pull requests into master. If you want -to contribute even something trivial please do not hesitate, but -follow the guidelines below. - -=== Sign the Contributor License Agreement -Before we accept a non-trivial patch or pull request we will need you to sign the -https://support.springsource.com/spring_committer_signup[contributor's agreement]. -Signing the contributor's agreement does not grant anyone commit rights to the main -repository, but it does mean that we can accept your contributions, and you will get an -author credit if we do. Active contributors might be asked to join the core team, and -given the ability to merge pull requests. - -=== Code Conventions and Housekeeping -None of these is essential for a pull request, but they will all help. They can also be -added after the original pull request but before a merge. - -* Use the Spring Framework code format conventions. If you use Eclipse - you can import formatter settings using the - `eclipse-code-formatter.xml` file from the - https://github.com/spring-cloud/build/tree/master/eclipse-coding-conventions.xml[Spring - Cloud Build] project. If using IntelliJ, you can use the - http://plugins.jetbrains.com/plugin/6546[Eclipse Code Formatter - Plugin] to import the same file. -* Make sure all new `.java` files to have a simple Javadoc class comment with at least an - `@author` tag identifying you, and preferably at least a paragraph on what the class is - for. -* Add the ASF license header comment to all new `.java` files (copy from existing files - in the project) -* Add yourself as an `@author` to the .java files that you modify substantially (more - than cosmetic changes). -* Add some Javadocs and, if you change the namespace, some XSD doc elements. -* A few unit tests would help a lot as well -- someone has to do it. -* If no-one else is using your branch, please rebase it against the current master (or - other target branch in the main project). -* When writing a commit message please follow http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html[these conventions], - if you are fixing an existing issue please add `Fixes gh-XXXX` at the end of the commit - message (where XXXX is the issue number). - - -// ====================================================================================== +include::https://raw.githubusercontent.com/spring-cloud/spring-cloud-build/master/docs/src/main/asciidoc/contributing.adoc[] diff --git a/core/docs/src/main/asciidoc/_configprops.adoc b/core/docs/src/main/asciidoc/_configprops.adoc index 55f3103e3..ca66c7ea8 100644 --- a/core/docs/src/main/asciidoc/_configprops.adoc +++ b/core/docs/src/main/asciidoc/_configprops.adoc @@ -9,6 +9,7 @@ |spring.cloud.stream.dynamic-destinations | `[]` | A list of destinations that can be bound dynamically. If set, only listed destinations can be bound. |spring.cloud.stream.function.batch-mode | `false` | |spring.cloud.stream.function.bindings | | +|spring.cloud.stream.function.reactive | | |spring.cloud.stream.input-bindings | | A semi-colon delimited string to explicitly define input bindings (specifically for cases when there is no implicit trigger to create such bindings such as Function, Supplier or Consumer). |spring.cloud.stream.instance-count | `1` | The number of deployed instances of an application. Default: 1. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-count" where 'foo' is the name of the binding. |spring.cloud.stream.instance-index | `0` | The instance id of the application: a number from 0 to instanceCount-1. Used for partitioning and with Kafka. NOTE: Could also be managed per individual binding "spring.cloud.stream.bindings.foo.consumer.instance-index" where 'foo' is the name of the binding. diff --git a/core/docs/src/main/asciidoc/intro.adoc b/core/docs/src/main/asciidoc/intro.adoc new file mode 100644 index 000000000..712b3da8b --- /dev/null +++ b/core/docs/src/main/asciidoc/intro.adoc @@ -0,0 +1,69 @@ +Spring Cloud Stream is a framework for building message-driven microservice applications. +Spring Cloud Stream builds upon Spring Boot to create standalone, production-grade Spring applications and uses Spring Integration to provide connectivity to message brokers. +It provides opinionated configuration of middleware from several vendors, introducing the concepts of persistent publish-subscribe semantics, consumer groups, and partitions. +These are called binder implementations in the parlance of Spring Cloud Stream. +Out of the box, Spring Cloud Stream provides binder implementations for Apache Kafka and RabbitMQ. +While these two binder implementations are based on Message Channels, Spring Cloud Stream also provides another binder implementation for Kafka Streams that does not use message channels, but native Kafka Streams types such as KStream, KTable etc. + +Below, you can find more information on how to use these various out-of-the-box binder implementations in your applications. + +== Apache Kafka Binder + +=== Usage + +To use Apache Kafka binder, you need to add `spring-cloud-stream-binder-kafka` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-kafka + +---- + +Alternatively, you can also use the Spring Cloud Stream Kafka Starter, as shown in the following example for Maven: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-starter-stream-kafka + +---- + +== Apache Kafka Streams Binder + +=== Usage + +To use Apache Kafka Streams binder, you need to add `spring-cloud-stream-binder-kafka-streams` as a dependency to your Spring Cloud Stream application, as shown in the following example for Maven: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-kafka-streams + +---- + +== RabbitMQ Binder +=== Usage + +To use the RabbitMQ binder, you can add it to your Spring Cloud Stream application, by using the following Maven coordinates: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-stream-binder-rabbit + +---- + +Alternatively, you can use the Spring Cloud Stream RabbitMQ Starter, as follows: + +[source,xml] +---- + + org.springframework.cloud + spring-cloud-starter-stream-rabbit + +----