From 0a927b67118a5ddd00bbe0f50c970fdb919bb31e Mon Sep 17 00:00:00 2001 From: Ilayaperumal Gopinathan Date: Tue, 25 Aug 2015 18:27:36 -0700 Subject: [PATCH] Update documentation - update steps to run the `s-c-s-module-launcher` on standalone, docker and lattice - update recent changes into readme - Fix docker compose to use the common binding name so that `time | log` works Remove explicit cloud profile for lattice run Remove tap reference Remove explicit server.port for docker-compose Update port mapping for docker run --- README.adoc | 105 +++++++----------- docs/src/main/asciidoc/intro.adoc | 38 +------ spring-cloud-stream-module-launcher/README.md | 58 +++++++--- .../docker-compose.yml | 7 +- 4 files changed, 96 insertions(+), 112 deletions(-) diff --git a/README.adoc b/README.adoc index ffc002d5e..312e1befc 100644 --- a/README.adoc +++ b/README.adoc @@ -12,7 +12,7 @@ Here's a sample source module (output channel only): @ComponentScan(basePackageClasses=TimerSource.class) public class ModuleApplication { - public static void main(String[] args) throws InterruptedException { + public static void main(String[] args) { SpringApplication.run(ModuleApplication.class, args); } @@ -26,7 +26,7 @@ public class TimerSource { private String format; @Bean - @InboundChannelAdapter(value = Source.OUTPUT, autoStartup = "false", poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1")) + @InboundChannelAdapter(value = Source.OUTPUT, poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1")) public MessageSource timerMessageSource() { return () -> new GenericMessage<>(new SimpleDateFormat(format).format(new Date())); } @@ -34,18 +34,6 @@ public class TimerSource { } ---- -The `application.yml` has the mapping from channel names to external broker handles (queues, topics, routing keys, etc. depending on the broker), e.g. - -.application.yml ----- ---- -spring: - cloud: - stream: - bindings: - output: ${spring.application.name:ticker} ----- - `@EnableModule` is parameterized by an interface (in this case `Source`) which declares input and output channels. `Source`, `Sink` and `Processor` are provided off the shelf, but you can define others. Here's the definition of `Source` [source,java] @@ -71,7 +59,7 @@ public class ModuleApplicationTests { @Test public void contextLoads() { - assertNotNull(this.sink.output()); + assertNotNull(this.source.output()); } } @@ -81,7 +69,7 @@ NOTE: In this case there is only one `Source` in the application context so ther == Multiple Input or Output Channels -A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly (a tap is like a topic). For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo". +A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly. For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo". == Samples @@ -120,63 +108,53 @@ For an XD module the channel names are `.` and a source (output on > Note: since the same naming conventions are used in XD, you can steal messages from or send messages to an existing XD stream by copying the stream name (to `spring.cloud.streams.group`) and knowing the index of the XD module you want to interact with. -== Taps - -All output channels can be also tapped so you can also attach a module to a pub-sub endpoint and listen to the tap if you know the module metadata. To tap an existing vanilla module you need to know its `outputChannelName` and the tap name is then `tap:`, so you can listen to it on an input channel named `input.topic.tap:`. The tap is only active if you explicitly ask for it: you can do that by POSTing to the HTTP endpoint `/taps/` (where the channel name can be the internal or external name, e.g. "output" or the external name mapped to the output channel). - -To tap an existing output channel in an XD module you just need to know its group, name and index, e.g. - -``` -spring: - cloud: - stream: - group: tocktap - name: logger - index: 0 - tap: - group: testtock - name: ticker - index: 0 -``` - -The `spring.cloud.stream.tap` section tells the module runner which topic you want to subscribe to. It creates a new group (a tap can't be in the same group as the one it is tapping) and starts a new index count, in case anyone wants to listen downstream. - - == Building :jdkversion: 1.8 === Basic Compile and Test -To build the source you will need to install -http://maven.apache.org/run-maven/index.html[Apache Maven] v3.0.6 or above and JDK {jdkversion}. +To build the source you will need to install JDK {jdkversion}. Spring Cloud uses Maven for most build-related activities, and you should be able to get off the ground quite quickly by cloning the project you are interested in and typing ---- -$ mvn install -s .settings.xml +$ ./mvnw install ---- -NOTE: You may need to increase the amount of memory available to Maven by setting -a `MAVEN_OPTS` environment variable with the value `-Xmx512m -XX:MaxPermSize=128m` +NOTE: You can also install Maven (>=3.3.3) yourself and run the `mvn` command +in place of `./mvnw` in the examples below. If you do that you also +might need to add `-P spring` if your local Maven settings do not +contain repository declarations for spring pre-release artifacts. -The `.settings.xml` is only required the first time (or after updates -to dependencies). It is there to provide repository declarations so -that those do not need to be hard coded in the project poms. +NOTE: Be aware that you might need to increase the amount of memory +available to Maven by setting a `MAVEN_OPTS` environment variable with +a value like `-Xmx512m -XX:MaxPermSize=128m`. We try to cover this in +the `.mvn` configuration, so if you find you have to do it to make a +build succeed, please raise a ticket to get the settings added to +source control. For hints on how to build the project look in `.travis.yml` if there is one. There should be a "script" and maybe "install" command. Also look at the "services" section to see if any services need to be running locally (e.g. mongo or rabbit). Ignore the git-related bits -that you might find in "before_install" since they will be able git +that you might find in "before_install" since they're related to setting git credentials and you already have those. -If you need mongo, rabbit or redis, see the README in the https://github.com/spring-cloud-samples/scripts[scripts -demo repository] for -instructions. For example consider using the "fig.yml" with -http://www.fig.sh/[Fig] to run them in Docker containers. +The projects that require middleware generally include a +`docker-compose.yml`, so consider using +http://compose.docker.io/[Docker Compose] to run the middeware servers +in Docker containers. See the README in the +https://github.com/spring-cloud-samples/scripts[scripts demo +repository] for specific instructions about the common cases of mongo, +rabbit and redis. + +NOTE: migration to the Maven wrapper (`./mvnw`) is underway. If you +find a project that doesn't have it yet, raise an issue to get it +added, and build with the command from `.travis.yml` (usually +`mvn install -s .settings.xml`). === Documentation @@ -201,13 +179,17 @@ We recommend the http://eclipse.org/m2e/[m2eclipe] eclipse plugin when working w eclipse. If you don't already have m2eclipse installed it is available from the "eclipse marketplace". -Once the projects are imported into Eclipse you will also need to tell m2eclipse -to use the `.settings.xml` file for the projects. If you do not do this you may -see errors many different errors related to the POMs in the projects. -Open your Eclipse preferences, expand the Maven preferences, and select User Settings. -In the User Settings field click Browse and navigate to the Spring Cloud project you -imported selecting the `.settings.xml` file in that project. Click Apply and then OK to -save the preference changes. +Unfortunately m2e does not yet support Maven 3.3, so once the projects +are imported into Eclipse you will also need to tell m2eclipse to use +the `.settings.xml` file for the projects. If you do not do this you +may see errors many different errors related to the POMs in the +projects. Open your Eclipse preferences, expand the Maven +preferences, and select User Settings. In the User Settings field +click Browse and navigate to the Spring Cloud project you imported +selecting the `.settings.xml` file in that project. Click Apply and +then OK to save the preference changes. + +NOTE: Alternatively you can copy the repository settings from https://github.com/spring-cloud/spring-cloud-build/blob/master/.settings.xml[`.settings.xml`] into your own `~/.m2/settings.xml`. ==== Importing into eclipse without m2eclipse If you prefer not to use m2eclipse you can generate eclipse project metadata using the @@ -215,7 +197,7 @@ following command: [indent=0] ---- - $ mvn eclipse:eclipse + $ ./mvnw eclipse:eclipse ---- The generated eclipse projects can be imported by selecting `import existing projects` @@ -223,9 +205,8 @@ from the `file` menu. ==== Adding Project Lombok Agent -Spring Cloud uses [Project -Lombok](http://projectlombok.org/features/index.html) to generate -getters and setters etc. Compiling from the command line this +Spring Cloud uses http://projectlombok.org/features/index.html[Project Lombok] +to generate getters and setters etc. Compiling from the command line this shouldn't cause any problems, but in an IDE you need to add an agent to the JVM. Full instructions can be found in the Lombok website. The sign that you need to do this is a lot of compiler errors to do with diff --git a/docs/src/main/asciidoc/intro.adoc b/docs/src/main/asciidoc/intro.adoc index 00326bcc9..90ef7a53d 100644 --- a/docs/src/main/asciidoc/intro.adoc +++ b/docs/src/main/asciidoc/intro.adoc @@ -22,7 +22,7 @@ public class TimerSource { private String format; @Bean - @InboundChannelAdapter(value = Source.OUTPUT, autoStartup = "false", poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1")) + @InboundChannelAdapter(value = Source.OUTPUT, poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1")) public MessageSource timerMessageSource() { return () -> new GenericMessage<>(new SimpleDateFormat(format).format(new Date())); } @@ -30,18 +30,6 @@ public class TimerSource { } ---- -The `application.yml` has the mapping from channel names to external broker handles (queues, topics, routing keys, etc. depending on the broker), e.g. - -.application.yml ----- ---- -spring: - cloud: - stream: - bindings: - output: ${spring.application.name:ticker} ----- - `@EnableModule` is parameterized by an interface (in this case `Source`) which declares input and output channels. `Source`, `Sink` and `Processor` are provided off the shelf, but you can define others. Here's the definition of `Source` [source,java] @@ -77,7 +65,7 @@ NOTE: In this case there is only one `Source` in the application context so ther == Multiple Input or Output Channels -A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly (a tap is like a topic). For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo". +A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly. For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo". == Samples @@ -115,25 +103,3 @@ The `[input,output]ChannelName` are used to create physical endpoints in the ext For an XD module the channel names are `.` and a source (output only) has `index=0` (the default) and downstream modules have the same group but incremented index, with a sink module (input only) having the highest index. To listen to the output from a running XD module, just use the same "group" name and an index 1 larger than the app before it in the chain. > Note: since the same naming conventions are used in XD, you can steal messages from or send messages to an existing XD stream by copying the stream name (to `spring.cloud.streams.group`) and knowing the index of the XD module you want to interact with. - -== Taps - -All output channels can be also tapped so you can also attach a module to a pub-sub endpoint and listen to the tap if you know the module metadata. To tap an existing vanilla module you need to know its `outputChannelName` and the tap name is then `tap:`, so you can listen to it on an input channel named `topic.tap:`. The tap is only active if you explicitly ask for it: you can do that by POSTing to the HTTP endpoint `/taps/` (where the channel name can be the internal or external name, e.g. "output" or the external name mapped to the output channel). - -To tap an existing output channel in an XD module you just need to know its group, name and index, e.g. - -``` -spring: - cloud: - stream: - group: tocktap - name: logger - index: 0 - tap: - group: testtock - name: ticker - index: 0 -``` - -The `spring.cloud.stream.tap` section tells the module runner which topic you want to subscribe to. It creates a new group (a tap can't be in the same group as the one it is tapping) and starts a new index count, in case anyone wants to listen downstream. - diff --git a/spring-cloud-stream-module-launcher/README.md b/spring-cloud-stream-module-launcher/README.md index 5c5587de7..a6aa29ff1 100644 --- a/spring-cloud-stream-module-launcher/README.md +++ b/spring-cloud-stream-module-launcher/README.md @@ -9,7 +9,7 @@ The Module Launcher provides a single entry point that bootstraps module JARs lo ```` git clone https://github.com/spring-cloud/spring-cloud-stream.git cd spring-cloud-stream -mvn -s .settings.xml package +mvn package cd .. ```` @@ -22,39 +22,69 @@ cd .. From the `spring-cloud-stream/spring-cloud-stream-module-launcher` directory: ```` -java -Dmodules=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar -java -Dmodules=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar +java -Dmodules=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -Dspring.cloud.stream.bindings.output=ticktock -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar +java -Dmodules=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -Dserver.port=8081 -Dspring.cloud.stream.bindings.input=ticktock -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar ```` -The time messages will be emitted every 5 seconds. The console for the log module will display each: +Note that `server.port` needs to be specified explicitly for the log sink module as the time source module already uses the default port `8080`. +The binding property is set to use the same name `ticktock` for both the output/input bindings of source/sink modules so that the log sink receives messages from the time source. + +The time messages will be emitted every second. The console for the log module will display each: ```` -2015-07-31 11:51:42.133 INFO 3388 --- [hannel-adapter1] sink.LogSink : Received: 2015-07-31 11:51:36 -2015-07-31 11:51:42.135 INFO 3388 --- [hannel-adapter1] sink.LogSink : Received: 2015-07-31 11:51:41 -2015-07-31 11:51:46.569 INFO 3388 --- [hannel-adapter1] sink.LogSink : Received: 2015-07-31 11:51:46 +2015-08-26 14:21:44.546 INFO 35725 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-26 14:21:44 +2015-08-26 14:21:45.548 INFO 35725 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-26 14:21:45 +2015-08-26 14:21:46.550 INFO 35725 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-26 14:21:46 ```` *NOTE:* the two modules will be launched within a single process if both are provided (comma-delimited) via `-Dmodules` ## Running with Docker -The easiest way to get a demo working is to use `docker-compose` (from this directory): +The easiest way to get a demo working is to use `docker-compose` (From the `spring-cloud-stream/spring-cloud-stream-module-launcher` directory): + +Make sure to set `DOCKER_HOST`. If you are running a `boot2docker` VM, $(boot2docker shellinit) would set that up. ``` $ mvn package docker:build $ docker-compose up ... logsink_1 | 2015-08-11 08:25:49.909 INFO 1 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-11 08:25:49 -logsink_1 | 2015-08-11 08:25:54.909 INFO 1 --- [hannel-adapter1] o.s.cloud.stream.module.log.Log +logsink_1 | 2015-08-11 08:25:50.909 INFO 1 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-11 08:25:50 ... ``` You can also run each module individually as a Docker process by passing environment variables for the module name as well as the host machine's IP address for the redis connection to be established within the container: +To find out redis host IP: +``` +Get the container ID of redis: `docker ps` +Get the IP address by inspecting the container: `docker inspect ` + +``` +To run the modules individually on docker: +```` +docker run -p 8080:8080 -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT \ + -e spring.cloud.stream.bindings.output=ticktock -e SPRING_REDIS_HOST= springcloud/stream-module-launcher + +docker run -p 8081:8080 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT \ + -e spring.cloud.stream.bindings.input=ticktock -e SPRING_REDIS_HOST= springcloud/stream-module-launcher +```` +Note the binding name `ticktock` is specified for the source's output and sink's input. +The port mapping is done so that individual modules' http endpoints can be accessed via the docker VM port. + +To run pub/sub modules individually on docker, the binding name has to start with `topic:`. ```` -docker run -p 8080:8080 -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -e SPRING_REDIS_HOST= springcloud/stream-module-launcher -docker run -p 8081:8081 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -e SPRING_REDIS_HOST= springcloud/stream-module-launcher +docker run -p 8080:8080 -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT \ + -e spring.cloud.stream.bindings.output=topic:foo -e SPRING_REDIS_HOST= springcloud/stream-module-launcher + +docker run -p 8081:8080 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT \ + -e spring.cloud.stream.bindings.input=topic:foo -e SPRING_REDIS_HOST= springcloud/stream-module-launcher + +docker run -p 8082:8080 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT \ + -e spring.cloud.stream.bindings.input=topic:foo -e SPRING_REDIS_HOST= springcloud/stream-module-launcher ```` +In the above scenario, both the sink modules receive the same messages from the time source. ## Running on Lattice @@ -69,6 +99,8 @@ $ ltc create redis redis -r 3: Run the modules as long-running processes (LRPs) on Lattice: ```` -$ ltc create time springcloud/stream-module-launcher -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -e SPRING_PROFILES_ACTIVE=cloud -$ ltc create log springcloud/stream-module-launcher -p 8081 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -e SPRING_PROFILES_ACTIVE=cloud +$ ltc create time springcloud/stream-module-launcher -m 512 \ + -e SPRING_CLOUD_STREAM_BINDINGS_OUTPUT=ticktock -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT +$ ltc create log springcloud/stream-module-launcher -m 512 \ + -e SPRING_CLOUD_STREAM_BINDINGS_INPUT=ticktock -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT ```` diff --git a/spring-cloud-stream-module-launcher/docker-compose.yml b/spring-cloud-stream-module-launcher/docker-compose.yml index bf14d474d..21dd45fe0 100644 --- a/spring-cloud-stream-module-launcher/docker-compose.yml +++ b/spring-cloud-stream-module-launcher/docker-compose.yml @@ -5,15 +5,20 @@ timesource: image: springcloud/stream-module-launcher links: - redis + ports: + - "8080:8080" environment: SPRING_REDIS_HOST: redis MODULES: org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT + SPRING_CLOUD_STREAM_BINDINGS_OUTPUT: ticktock logsink: image: springcloud/stream-module-launcher links: - redis + ports: + - "8081:8080" environment: SPRING_REDIS_HOST: redis MODULES: org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT - \ No newline at end of file + SPRING_CLOUD_STREAM_BINDINGS_INPUT: ticktock