From 531de2d41e42a81ddf4dec8bd9d8e8e0c97bc62d Mon Sep 17 00:00:00 2001 From: Dave Syer Date: Thu, 9 Jul 2015 09:28:18 +0100 Subject: [PATCH] Tidy up readme --- README.md | 48 ++++++++++++++++++++++++++---------------------- roadmap.md | 37 +++++++++++++++++-------------------- 2 files changed, 43 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 2a079005d..a8188a1d3 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,12 @@ -# Spring Integration: Messaging as a Microservice +# Spring Cloud Streams: Messaging as a Microservice -This is an experimental project allowing a user to develop and run messaging microservices using Spring Integration and run them locally or in the cloud, or even on Spring XD. It also allows a user to develop and run an XD module locally. Just create `MessageChannels` "input" and/or "output" and add `@EnableMessageBus` and run your app as a Spring Boot app (single application context). You just need to connect to the physical broker for the bus, which is automatic if the relevant bus implementation is available on the classpath. The sample uses Redis. +This project allows a user to develop and run messaging microservices using Spring Integration and run them locally, or in the cloud, or even on Spring XD. It also allows a user to develop and run an XD module locally. Just create `MessageChannels` "input" and/or "output" and add `@EnableChannelBinding` and run your app as a Spring Boot app (single application context). You just need to connect to the physical broker for the bus, which is automatic if the relevant bus implementation is available on the classpath. The sample uses Redis. Here's a sample source module (output channel only): ``` @SpringBootApplication -@EnableMessageBus +@EnableChannelBinding @ComponentScan(basePackageClasses=ModuleDefinition.class) public class ModuleApplication { @@ -36,22 +36,21 @@ public class ModuleDefinition { } ``` -The `bootstrap.yml` has the module group (a.k.a. stream name in XD), name and index, e.g. +The `application.yml` has the mapping from channel names to external broker handles (queues, topics, routing keys, etc. depending on the broker), e.g. ``` --- spring: - bus: - group: testtock - name: ${spring.application.name:ticker} - index: 0 # source + cloud: + channels: + outputChannelName: ${spring.application.name:ticker} ``` -To be deployable as an XD module in a "traditional" way you need `/config/*.properties` to point to any available Java config classes (via `base_packages` or `options_class`), or else you can put traditional XML configuration in `/config/*.xml`. You don't need those things to run as a consumer or producer to an existing XD system. There's an XML version of the same sample (a "timer" source). +To be deployable as an XD module in a "traditional" way you need `/config/*.properties` to point to any available Java config classes (via `base_packages` or `options_class`), or else you can put traditional XML configuration in `/config/*.xml`. You don't need those things to run as a consumer or producer to an existing XD system, but you do need to replace the `outputChannelName` with `group`, `module` and `index` (the index is a sequential counter that XD uses to label the modules in a stream from left to right). There's an XML version of the same sample (a "timer" source). ## Multiple Input or Output Channels -A module can have multiple input or output channels. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` beans named `input.*` or `output.*` and the names are converted to external channel names on the bus. The external channel names are the "natural" channel name for the module (i.e. `.` or `spring.bus.[input|output]ChannelName` if supplied) plus the `MessageChannel` bean name, period separated. In addition, the bean name can be `input.[queue|topic|tap]:*` or `output.[queue|topic]:*` (i.e. with a channel type as a colon-separated prefix), and the semantics of the external bus channel changes accordingly (a tap is like a topic). For example, you can have two `MessageChannels` called "output" and "output.topic:foo" in a module deployed with "group=bar" and "index=2", and the result is 2 external channels called "bar.2" and "topic:foo.bar.2". +A module can have multiple input or output channels. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` beans named `input.*` or `output.*` and the names are converted to external channel names on the broker. The external channel names are the `spring.cloud.streams.[input|output]ChannelName` plus the `MessageChannel` bean name, period separated. In addition, the bean name can be `input.[queue|topic|tap]:*` or `output.[queue|topic]:*` (i.e. with a channel type as a colon-separated prefix), and the semantics of the external bus channel changes accordingly (a tap is like a topic). For example, you can have two `MessageChannels` called "output" and "output.topic:foo" in a module with `outputChannelName=bar`, and the result is 2 external channels called "bar" and "topic:foo.bar". ## XD Module Samples @@ -85,29 +84,34 @@ You can run in standalone mode from your IDE for testing. To run in production y ## Making Standalone Modules Talk to Each Other -The "group" and "index" are used to create physical endpoints in the external broker (e.g. `queue..` in Redis), so a source (output only) has `index=0` (the default) and downstream modules have the same group but incremented index, with a sink module (input only) having the highest index. To listen to the output from an existing app, just use the same "group" name and an index 1 larger than the app before it in the chain. The index can be anything, as long as successive modules have consecutive values. +The `[input,output]ChannelName` are used to create physical endpoints in the external broker (e.g. `queue.` in Redis). -> Note: since the same naming conventions are used in XD, you can spy on or send messages to an existing XD stream by copying the stream name (to `spring.bus.group`) and knowing the index of the XD module you want to interact with. +For an XD module the channel names are `.` and a source (output only) has `index=0` (the default) and downstream modules have the same group but incremented index, with a sink module (input only) having the highest index. To listen to the output from a running XD module, just use the same "group" name and an index 1 larger than the app before it in the chain. + +> Note: since the same naming conventions are used in XD, you can steal messages from or send messages to an existing XD stream by copying the stream name (to `spring.cloud.streams.group`) and knowing the index of the XD module you want to interact with. ## Taps -All output channels are also tapped by default so you can also attach a module to a pub-sub endpoint and listen to the tap if you know the module metadata (e.g. `topic.tap:stream:..` in Redis). To tap an existing output channel you just need to know its group, name and index, e.g. +All output channels can be also tapped so you can also attach a module to a pub-sub endpoint and listen to the tap if you know the module metadata. To tap an existing vanilla module you need to know its `outputChannelName` and the tap name is then `tap:`, so you can listen to it on an input channel named `input.topic.tap:`. The tap is only active if you explicitly ask for it: you can do that by POSTing to the HTTP endpoint `/taps/` (where the channel name can be the internal or external name, e.g. "output" or the external name mapped to the output channel). + +To tap an existing output channel in an XD module you just need to know its group, name and index, e.g. ``` spring: - bus: - group: tocktap - name: logger - index: 0 - tap: - group: testtock - name: ticker + cloud: + channels: + group: tocktap + name: logger index: 0 + tap: + group: testtock + name: ticker + index: 0 ``` -The `spring.bus.tap` section tells the module runner which topic you want to subscribe to. It creates a new group (a tap can't be in the same group as the one it is tapping) and starts a new index count, in case anyone wants to listen downstream. +The `spring.cloud.channels.tap` section tells the module runner which topic you want to subscribe to. It creates a new group (a tap can't be in the same group as the one it is tapping) and starts a new index count, in case anyone wants to listen downstream. -## Build Spring Bus +## Build Spring Cloud Streams ### Pre-requisites * Required : diff --git a/roadmap.md b/roadmap.md index bcb251579..0713f5179 100644 --- a/roadmap.md +++ b/roadmap.md @@ -6,18 +6,18 @@ This project was originally motivated by the goal of allowing a developer to bui ## Basic Programming Model -Just create `MessageChannels` "input" and/or "output" and add `@EnableMessageBus` and run your app as a Spring Boot app (single application context). You need to connect to the physical broker for the bus, which is automatic if the relevant bus implementation is available on the classpath. The sample uses Redis. +Just create `MessageChannels` "input" and/or "output" and add `@EnableChannelBinding` and run your app as a Spring Boot app (single application context). You need to connect to the physical broker, which is automatic if the relevant implementation is available on the classpath. The sample uses Redis. Here's a sample source module (output channel only): ``` @SpringBootApplication -@EnableMessageBus +@EnableChannelBinding @ComponentScan(basePackageClasses=ModuleDefinition.class) -public class MessageBusApplication { +public class MessagingApplication { public static void main(String[] args) { - SpringApplication.run(MessageBusApplication.class, args); + SpringApplication.run(MessagingApplication.class, args); } } @@ -42,15 +42,14 @@ public class ModuleDefinition { } ``` -The `bootstrap.yml` has the module group (a.k.a. stream name), name and index, e.g. +The `application.yml` has the external channel names, e.g. ``` --- spring: - bus: - group: testtock - name: ${spring.application.name:ticker} - index: 0 # source + cloud: + channels: + outputChannelName: ${spring.application.name:ticker} ``` ## Richer Input and Output @@ -79,11 +78,7 @@ To be deployable as an XD module in a "traditional" way you need `/config/*.prop - [x] Support for pubsub as "primary" input/output (in addition to the existing queue semantics) -- [x] Endpoint "/channels" for module configuration metadata ("/bus" is taken by Spring Cloud) - -- [x] Discover channel names through Spring Cloud service discovery (via "/channels" endpoint on remote components) - -- [x] Listen for changes in discovery catalog and potentially rebind channels +- [x] Endpoint "/channels" for module configuration metadata - [ ] `@BusClient` like `@FeignClient` where the remote service (and optionally channel) can be specified @@ -93,7 +88,9 @@ To be deployable as an XD module in a "traditional" way you need `/config/*.prop - [ ] Correlation and message tracing -- [ ] Extract `spring-xd-dirt` dependencies into a separate module +- [x] Extract `spring-xd-dirt` dependencies into a separate module + + - [ ] Remove xd package names - [ ] Re-use existing XD modules as libraries @@ -109,17 +106,17 @@ The best plan for making progress, where we keep in sight the goal of eventually - [ ] Ditto configuration properties in `xd.*` should be moved to `spring.bus.*` (or something). -- [ ] We need the XML configuration from `/META-INF/spring-xd/bus/**` and `/META-INF/spring-xd/analytics` but +- [x] We need the XML configuration from `/META-INF/spring-xd/bus/**` and `/META-INF/spring-xd/analytics` but - [x] There are no defaults for several properties in `xd.messagebus.*` so applications have to have a load of boilerplate configuration in `application.yml`. Fixed by adding `@PropertySources` to the default configuration. - - [ ] The "codec.xml" is in `spring-xd-dirt` which we don't want to depend on. Maybe it should be in the messagebus SPI jar? Or we can make a copy and risk it changing in XD. + - [x] The "codec.xml" is in `spring-xd-dirt` which we don't want to depend on. Maybe it should be in the messagebus SPI jar? Or we can make a copy and risk it changing in XD. - [x] Do we need the analytics configuration? It should at least be optional. Answer "no" (but support for analytics would be cool). -- [ ] The `spring-xd-dirt` library contains some of the primitives we might need, especially when building the bridge to create XD modules as apps. It would be best if they could be extracted into another library. +- [x] The `spring-xd-dirt` library contains some of the primitives we might need, especially when building the bridge to create XD modules as apps. It would be best if they could be extracted into another library. - - [ ] `MessageBusAwareChannelResolver` and `MessageBusAwareRouterBeanPostProcessor` should be pulled out of dirt (and dirt should ultimately depend on this project). + - [x] `MessageBusAwareChannelResolver` and `MessageBusAwareRouterBeanPostProcessor` should be pulled out of dirt (and dirt should ultimately depend on this project). - [x] A `ModuleDefinition` is only needed to initialize options for an XD module (so not really needed for the general case). We can split the module options initializer out into a separate module that you only need if you know you want to test an XD module with its native options metadata. @@ -127,7 +124,7 @@ The best plan for making progress, where we keep in sight the goal of eventually - [ ] `BusUtils` (e.g. to construct external channel names) -- [ ] There is a curator dependency in Spring XD that can't be shaken off. +- [x] There is a curator dependency in Spring XD that can't be shaken off. - [ ] Spring XD plugins provide a rich set of lifecycle hooks, but those would not all be needed and are an awkward mismatch with a "pure-play" Spring Boot approach, where the application is either running or not.