SpringCloud/spring-cloud-stream
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update documentation

Browse Source 
- update steps to run the `s-c-s-module-launcher` on standalone, docker and lattice
 - update recent changes into readme
 - Fix docker compose to use the common binding name so that `time | log` works

Remove explicit cloud profile for lattice run

Remove tap reference

Remove explicit server.port for docker-compose

Update port mapping for docker run
This commit is contained in:
Ilayaperumal Gopinathan
2015-08-25 18:27:36 -07:00
committed by Mark Fisher
parent 7663372062
commit 0a927b6711
4 changed files with 96 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
README.adoc
View File

@@ -12,7 +12,7 @@ Here's a sample source module (output channel only):
@ComponentScan(basePackageClasses=TimerSource.class)
public class ModuleApplication {
public static void main(String[] args) throws InterruptedException {
public static void main(String[] args) {
SpringApplication.run(ModuleApplication.class, args);
}
@@ -26,7 +26,7 @@ public class TimerSource {
private String format;
@Bean
@InboundChannelAdapter(value = Source.OUTPUT, autoStartup = "false", poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1"))
@InboundChannelAdapter(value = Source.OUTPUT, poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1"))
public MessageSource<String> timerMessageSource() {
return () -> new GenericMessage<>(new SimpleDateFormat(format).format(new Date()));
}
@@ -34,18 +34,6 @@ public class TimerSource {
}
----
The `application.yml` has the mapping from channel names to external broker handles (queues, topics, routing keys, etc. depending on the broker), e.g.
.application.yml
----
---
spring:
cloud:
stream:
bindings:
output: ${spring.application.name:ticker}
----
`@EnableModule` is parameterized by an interface (in this case `Source`) which declares input and output channels. `Source`, `Sink` and `Processor` are provided off the shelf, but you can define others. Here's the definition of `Source`
[source,java]
@@ -71,7 +59,7 @@ public class ModuleApplicationTests {
@Test
public void contextLoads() {
assertNotNull(this.sink.output());
assertNotNull(this.source.output());
}
}
@@ -81,7 +69,7 @@ NOTE: In this case there is only one `Source` in the application context so ther
== Multiple Input or Output Channels
A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly (a tap is like a topic). For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo".
A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly. For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo".
== Samples
@@ -120,63 +108,53 @@ For an XD module the channel names are `<group>.<index>` and a source (output on
> Note: since the same naming conventions are used in XD, you can steal messages from or send messages to an existing XD stream by copying the stream name (to `spring.cloud.streams.group`) and knowing the index of the XD module you want to interact with.
== Taps
All output channels can be also tapped so you can also attach a module to a pub-sub endpoint and listen to the tap if you know the module metadata. To tap an existing vanilla module you need to know its `outputChannelName` and the tap name is then `tap:<outputChannelName>`, so you can listen to it on an input channel named `input.topic.tap:<outputChannelName>`. The tap is only active if you explicitly ask for it: you can do that by POSTing to the HTTP endpoint `/taps/<channelName>` (where the channel name can be the internal or external name, e.g. "output" or the external name mapped to the output channel).
To tap an existing output channel in an XD module you just need to know its group, name and index, e.g.
```
spring:
cloud:
stream:
group: tocktap
name: logger
index: 0
tap:
group: testtock
name: ticker
index: 0
```
The `spring.cloud.stream.tap` section tells the module runner which topic you want to subscribe to. It creates a new group (a tap can't be in the same group as the one it is tapping) and starts a new index count, in case anyone wants to listen downstream.
== Building
:jdkversion: 1.8
=== Basic Compile and Test
To build the source you will need to install
http://maven.apache.org/run-maven/index.html[Apache Maven] v3.0.6 or above and JDK {jdkversion}.
To build the source you will need to install JDK {jdkversion}.
Spring Cloud uses Maven for most build-related activities, and you
should be able to get off the ground quite quickly by cloning the
project you are interested in and typing
----
$ mvn install -s .settings.xml
$ ./mvnw install
----
NOTE: You may need to increase the amount of memory available to Maven by setting
a `MAVEN_OPTS` environment variable with the value `-Xmx512m -XX:MaxPermSize=128m`
NOTE: You can also install Maven (>=3.3.3) yourself and run the `mvn` command
in place of `./mvnw` in the examples below. If you do that you also
might need to add `-P spring` if your local Maven settings do not
contain repository declarations for spring pre-release artifacts.
The `.settings.xml` is only required the first time (or after updates
to dependencies). It is there to provide repository declarations so
that those do not need to be hard coded in the project poms.
NOTE: Be aware that you might need to increase the amount of memory
available to Maven by setting a `MAVEN_OPTS` environment variable with
a value like `-Xmx512m -XX:MaxPermSize=128m`. We try to cover this in
the `.mvn` configuration, so if you find you have to do it to make a
build succeed, please raise a ticket to get the settings added to
source control.
For hints on how to build the project look in `.travis.yml` if there
is one. There should be a "script" and maybe "install" command. Also
look at the "services" section to see if any services need to be
running locally (e.g. mongo or rabbit). Ignore the git-related bits
that you might find in "before_install" since they will be able git
that you might find in "before_install" since they're related to setting git
credentials and you already have those.
If you need mongo, rabbit or redis, see the README in the https://github.com/spring-cloud-samples/scripts[scripts
demo repository] for
instructions. For example consider using the "fig.yml" with
http://www.fig.sh/[Fig] to run them in Docker containers.
The projects that require middleware generally include a
`docker-compose.yml`, so consider using
http://compose.docker.io/[Docker Compose] to run the middeware servers
in Docker containers. See the README in the
https://github.com/spring-cloud-samples/scripts[scripts demo
repository] for specific instructions about the common cases of mongo,
rabbit and redis.
NOTE: migration to the Maven wrapper (`./mvnw`) is underway. If you
find a project that doesn't have it yet, raise an issue to get it
added, and build with the command from `.travis.yml` (usually
`mvn install -s .settings.xml`).
=== Documentation
@@ -201,13 +179,17 @@ We recommend the http://eclipse.org/m2e/[m2eclipe] eclipse plugin when working w
eclipse. If you don't already have m2eclipse installed it is available from the "eclipse
marketplace".
Once the projects are imported into Eclipse you will also need to tell m2eclipse
to use the `.settings.xml` file for the projects. If you do not do this you may
see errors many different errors related to the POMs in the projects.
Open your Eclipse preferences, expand the Maven preferences, and select User Settings.
In the User Settings field click Browse and navigate to the Spring Cloud project you
imported selecting the `.settings.xml` file in that project. Click Apply and then OK to
save the preference changes.
Unfortunately m2e does not yet support Maven 3.3, so once the projects
are imported into Eclipse you will also need to tell m2eclipse to use
the `.settings.xml` file for the projects. If you do not do this you
may see errors many different errors related to the POMs in the
projects. Open your Eclipse preferences, expand the Maven
preferences, and select User Settings. In the User Settings field
click Browse and navigate to the Spring Cloud project you imported
selecting the `.settings.xml` file in that project. Click Apply and
then OK to save the preference changes.
NOTE: Alternatively you can copy the repository settings from https://github.com/spring-cloud/spring-cloud-build/blob/master/.settings.xml[`.settings.xml`] into your own `~/.m2/settings.xml`.
==== Importing into eclipse without m2eclipse
If you prefer not to use m2eclipse you can generate eclipse project metadata using the
@@ -215,7 +197,7 @@ following command:
[indent=0]
----
$ mvn eclipse:eclipse
$ ./mvnw eclipse:eclipse
----
The generated eclipse projects can be imported by selecting `import existing projects`
@@ -223,9 +205,8 @@ from the `file` menu.
==== Adding Project Lombok Agent
Spring Cloud uses [Project
Lombok](http://projectlombok.org/features/index.html) to generate
getters and setters etc. Compiling from the command line this
Spring Cloud uses http://projectlombok.org/features/index.html[Project Lombok]
to generate getters and setters etc. Compiling from the command line this
shouldn't cause any problems, but in an IDE you need to add an agent
to the JVM. Full instructions can be found in the Lombok website. The
sign that you need to do this is a lot of compiler errors to do with

38
docs/src/main/asciidoc/intro.adoc
View File

@@ -22,7 +22,7 @@ public class TimerSource {
private String format;
@Bean
@InboundChannelAdapter(value = Source.OUTPUT, autoStartup = "false", poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1"))
@InboundChannelAdapter(value = Source.OUTPUT, poller = @Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1"))
public MessageSource<String> timerMessageSource() {
return () -> new GenericMessage<>(new SimpleDateFormat(format).format(new Date()));
}
@@ -30,18 +30,6 @@ public class TimerSource {
}
----
The `application.yml` has the mapping from channel names to external broker handles (queues, topics, routing keys, etc. depending on the broker), e.g.
.application.yml
----
---
spring:
cloud:
stream:
bindings:
output: ${spring.application.name:ticker}
----
`@EnableModule` is parameterized by an interface (in this case `Source`) which declares input and output channels. `Source`, `Sink` and `Processor` are provided off the shelf, but you can define others. Here's the definition of `Source`
[source,java]
@@ -77,7 +65,7 @@ NOTE: In this case there is only one `Source` in the application context so ther
== Multiple Input or Output Channels
A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly (a tap is like a topic). For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo".
A module can have multiple input or output channels all defined either as `@Input` and `@Output` methods in an interface (preferrable) or as bean definitions. Instead of just one channel named "input" or "output" you can add multiple `MessageChannel` methods annotated `@Input` or `@Output` and the names are converted to external channel names on the broker. The external channel names can be specified as properties that consist of the channel names prefixed with `spring.cloud.stream.bindings` (e.g. `spring.cloud.stream.bindings.input` or `spring.cloud.stream.bindings.output`). External channel names can have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly. For example, you can have two `MessageChannels` called "output" and "foo" in a module with `spring.cloud.stream.bindings.output=bar` and `spring.cloud.stream.bindings.foo=topic:foo`, and the result is 2 external channels called "bar" and "topic:foo".
== Samples
@@ -115,25 +103,3 @@ The `[input,output]ChannelName` are used to create physical endpoints in the ext
For an XD module the channel names are `<group>.<index>` and a source (output only) has `index=0` (the default) and downstream modules have the same group but incremented index, with a sink module (input only) having the highest index. To listen to the output from a running XD module, just use the same "group" name and an index 1 larger than the app before it in the chain.
> Note: since the same naming conventions are used in XD, you can steal messages from or send messages to an existing XD stream by copying the stream name (to `spring.cloud.streams.group`) and knowing the index of the XD module you want to interact with.
== Taps
All output channels can be also tapped so you can also attach a module to a pub-sub endpoint and listen to the tap if you know the module metadata. To tap an existing vanilla module you need to know its `outputChannelName` and the tap name is then `tap:<outputChannelName>`, so you can listen to it on an input channel named `topic.tap:<outputChannelName>`. The tap is only active if you explicitly ask for it: you can do that by POSTing to the HTTP endpoint `/taps/<channelName>` (where the channel name can be the internal or external name, e.g. "output" or the external name mapped to the output channel).
To tap an existing output channel in an XD module you just need to know its group, name and index, e.g.
```
spring:
cloud:
stream:
group: tocktap
name: logger
index: 0
tap:
group: testtock
name: ticker
index: 0
```
The `spring.cloud.stream.tap` section tells the module runner which topic you want to subscribe to. It creates a new group (a tap can't be in the same group as the one it is tapping) and starts a new index count, in case anyone wants to listen downstream.

58
spring-cloud-stream-module-launcher/README.md
View File

@@ -9,7 +9,7 @@ The Module Launcher provides a single entry point that bootstraps module JARs lo
````
git clone https://github.com/spring-cloud/spring-cloud-stream.git
cd spring-cloud-stream
mvn -s .settings.xml package
mvn package
cd ..
````
@@ -22,39 +22,69 @@ cd ..
From the `spring-cloud-stream/spring-cloud-stream-module-launcher` directory:
````
java -Dmodules=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar
java -Dmodules=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar
java -Dmodules=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -Dspring.cloud.stream.bindings.output=ticktock -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar
java -Dmodules=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -Dserver.port=8081 -Dspring.cloud.stream.bindings.input=ticktock -jar target/spring-cloud-stream-module-launcher-1.0.0.BUILD-SNAPSHOT.jar
````
The time messages will be emitted every 5 seconds. The console for the log module will display each:
Note that `server.port` needs to be specified explicitly for the log sink module as the time source module already uses the default port `8080`.
The binding property is set to use the same name `ticktock` for both the output/input bindings of source/sink modules so that the log sink receives messages from the time source.
The time messages will be emitted every second. The console for the log module will display each:
````
2015-07-31 11:51:42.133 INFO 3388 --- [hannel-adapter1] sink.LogSink : Received: 2015-07-31 11:51:36
2015-07-31 11:51:42.135 INFO 3388 --- [hannel-adapter1] sink.LogSink : Received: 2015-07-31 11:51:41
2015-07-31 11:51:46.569 INFO 3388 --- [hannel-adapter1] sink.LogSink : Received: 2015-07-31 11:51:46
2015-08-26 14:21:44.546 INFO 35725 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-26 14:21:44
2015-08-26 14:21:45.548 INFO 35725 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-26 14:21:45
2015-08-26 14:21:46.550 INFO 35725 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-26 14:21:46
````
*NOTE:* the two modules will be launched within a single process if both are provided (comma-delimited) via `-Dmodules`
## Running with Docker
The easiest way to get a demo working is to use `docker-compose` (from this directory):
The easiest way to get a demo working is to use `docker-compose` (From the `spring-cloud-stream/spring-cloud-stream-module-launcher` directory):
Make sure to set `DOCKER_HOST`. If you are running a `boot2docker` VM, $(boot2docker shellinit) would set that up.
```
$ mvn package docker:build
$ docker-compose up
...
logsink_1 | 2015-08-11 08:25:49.909 INFO 1 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-11 08:25:49
logsink_1 | 2015-08-11 08:25:54.909 INFO 1 --- [hannel-adapter1] o.s.cloud.stream.module.log.Log
logsink_1 | 2015-08-11 08:25:50.909 INFO 1 --- [hannel-adapter1] o.s.cloud.stream.module.log.LogSink : Received: 2015-08-11 08:25:50
...
```
You can also run each module individually as a Docker process by passing environment variables for the module name as well as the host machine's IP address for the redis connection to be established within the container:
To find out redis host IP:
```
Get the container ID of redis: `docker ps`
Get the IP address by inspecting the container: `docker inspect <containerID>`
```
To run the modules individually on docker:
````
docker run -p 8080:8080 -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT \
-e spring.cloud.stream.bindings.output=ticktock -e SPRING_REDIS_HOST=<Redis-Host-IP> springcloud/stream-module-launcher
docker run -p 8081:8080 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT \
-e spring.cloud.stream.bindings.input=ticktock -e SPRING_REDIS_HOST=<Redis-Host-IP> springcloud/stream-module-launcher
````
Note the binding name `ticktock` is specified for the source's output and sink's input.
The port mapping is done so that individual modules' http endpoints can be accessed via the docker VM port.
To run pub/sub modules individually on docker, the binding name has to start with `topic:`.
````
docker run -p 8080:8080 -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -e SPRING_REDIS_HOST=<host.ip> springcloud/stream-module-launcher
docker run -p 8081:8081 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -e SPRING_REDIS_HOST=<host.ip> springcloud/stream-module-launcher
docker run -p 8080:8080 -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT \
-e spring.cloud.stream.bindings.output=topic:foo -e SPRING_REDIS_HOST=<Redis-Host-IP> springcloud/stream-module-launcher
docker run -p 8081:8080 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT \
-e spring.cloud.stream.bindings.input=topic:foo -e SPRING_REDIS_HOST=<Redis-Host-IP> springcloud/stream-module-launcher
docker run -p 8082:8080 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT \
-e spring.cloud.stream.bindings.input=topic:foo -e SPRING_REDIS_HOST=<Redis-Host-IP> springcloud/stream-module-launcher
````
In the above scenario, both the sink modules receive the same messages from the time source.
## Running on Lattice
@@ -69,6 +99,8 @@ $ ltc create redis redis -r
3: Run the modules as long-running processes (LRPs) on Lattice:
````
$ ltc create time springcloud/stream-module-launcher -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT -e SPRING_PROFILES_ACTIVE=cloud
$ ltc create log springcloud/stream-module-launcher -p 8081 -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT -e SPRING_PROFILES_ACTIVE=cloud
$ ltc create time springcloud/stream-module-launcher -m 512 \
-e SPRING_CLOUD_STREAM_BINDINGS_OUTPUT=ticktock -e MODULES=org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT
$ ltc create log springcloud/stream-module-launcher -m 512 \
-e SPRING_CLOUD_STREAM_BINDINGS_INPUT=ticktock -e MODULES=org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT
````

7
spring-cloud-stream-module-launcher/docker-compose.yml
View File

@@ -5,15 +5,20 @@ timesource:
image: springcloud/stream-module-launcher
links:
- redis
ports:
- "8080:8080"
environment:
SPRING_REDIS_HOST: redis
MODULES: org.springframework.cloud.stream.module:time-source:1.0.0.BUILD-SNAPSHOT
SPRING_CLOUD_STREAM_BINDINGS_OUTPUT: ticktock
logsink:
image: springcloud/stream-module-launcher
links:
- redis
ports:
- "8081:8080"
environment:
SPRING_REDIS_HOST: redis
MODULES: org.springframework.cloud.stream.module:log-sink:1.0.0.BUILD-SNAPSHOT
SPRING_CLOUD_STREAM_BINDINGS_INPUT: ticktock