diff --git a/ghpages.sh b/ghpages.sh index b96a721b..407f6481 100644 --- a/ghpages.sh +++ b/ghpages.sh @@ -1,8 +1,8 @@ #!/bin/bash -x -git remote add docs https://github.com/spring-cloud/spring-cloud-static +git remote | grep docs || git remote add docs https://github.com/spring-cloud/spring-cloud-static -if ! (git fetch docs && git checkout --track docs/gh-pages); then +if ! (git fetch docs && git checkout --track docs/gh-pages || git checkout gh-pages); then echo "No gh-pages, error" exit 1 fi diff --git a/spring-cloud.html b/spring-cloud.html index 533160ec..084c7fc7 100644 --- a/spring-cloud.html +++ b/spring-cloud.html @@ -550,32 +550,31 @@ body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-b -
The Spring Cloud Stream project allows a user to develop and run messaging microservices using Spring Integration. Just add @EnableBinding and run your app as a Spring Boot app (single application context). You just need to connect to the physical broker for the bindings, which is automatic if the relevant binder implementation is available on the classpath. The sample uses Redis.
The Spring Cloud Stream project allows a user to develop and run messaging microservices using Spring Integration.
+Just add @EnableBinding and run your app as a Spring Boot app (single application context).
+Spring Cloud Stream applications connect to the physical broker through bindings, which link Spring Integration
+channels to physical broker destinations, for either input (consumer bindings) or output (producer bindings).
+The creation of the bindings, and therefore their broker-specific implementation is handled by a binder, which is
+another important abstraction of Spring Cloud Stream. Binders abstract out the broker-specific implementation details.
+In order to connect to a specific type of broker (e.g. Rabbit or Kafka) you just need to have the relevant binder
+implementation on the classpath.
Here’s a sample source module (output channel only):
+Here’s a sample source app (output channel only):
@SpringBootApplication
-@ComponentScan(basePackageClasses=TimerSource.class)
-public class ModuleApplication {
+public class StreamApplication {
public static void main(String[] args) {
- SpringApplication.run(ModuleApplication.class, args);
+ SpringApplication.run(StreamApplication.class, args);
}
-
}
-@Configuration
@EnableBinding(Source.class)
public class TimerSource {
@@ -4939,40 +4900,45 @@ public class TimerSource {
public MessageSource<String> timerMessageSource() {
return () -> new GenericMessage<>(new SimpleDateFormat(format).format(new Date()));
}
-
}
@EnableBinding is parameterized by one or more interfaces (in this case a single Source interface), which declares input and output channels. The interfaces Source, Sink and Processor are provided off the shelf, but you can define others. Here’s the definition of Source:
@EnableBinding is parameterized by one or more interfaces (in this case a single Source interface), which declares
+input and/or output channels. The interfaces Source, Sink and Processor are provided off the shelf, but you can
+define others. Here’s the definition of Source:
public interface Source {
- @Output("output")
+ String OUTPUT = "output";
+
+ @Output(Source.OUTPUT)
MessageChannel output();
}
The @Output annotation is used to identify output channels (messages leaving the module) and @Input is used to identify input channels (messages entering the module). It is optionally parameterized by a channel name - if the name is not provided the method name is used instead. An implementation of the interface is created for you and can be used in the application context by autowiring it, e.g. into a test case:
The @Output annotation is used to identify output channels (messages leaving the app), and @Input is used to
+identify input channels (messages entering the app). It is optionally parameterized by a channel name - if the name is
+not provided the method name is used instead. An implementation of the interface is created for you and can be used in
+the application context by autowiring it, e.g. into a test case:
@RunWith(SpringJUnit4ClassRunner.class)
-@SpringApplicationConfiguration(classes = ModuleApplication.class)
+@SpringApplicationConfiguration(classes = StreamApplication.class)
@WebAppConfiguration
@DirtiesContext
-public class ModuleApplicationTests {
+public class StreamApplicationTests {
- @Autowired
- private Source source
-
- @Test
- public void contextLoads() {
- assertNotNull(this.source.output());
- }
+ @Autowired
+ private Source source
+ @Test
+ public void contextLoads() {
+ assertNotNull(this.source.output());
+ }
}
Source in the application context so there is no need to qualify it when it is autowired. If there is ambiguity, e.g. if you are composing one module from some others, you can use @Bindings qualifier to inject a specific channel set. The @Bindings qualifier takes a parameter which is the class that carries the @EnableBinding annotation (in this case the TimerSource).
+In this case there is only one Source in the application context so there is no need to qualify it when it is
+autowired. If there is ambiguity, e.g. if you are composing one application from some others, you can use the
+@Bindings qualifier to inject a specific channel set. The @Bindings qualifier takes a parameter which is the class
+that carries the @EnableBinding annotation (in this case the TimerSource).
Source in the application context so
A module can have multiple input or output channels defined as @Input and @Output methods in an interface. Instead of just one channel named "input" or "output" you can add multiple MessageChannel methods annotated @Input or @Output and their names will be converted to external channel names on the broker. It is common to specify the channel names at runtime in order to have multiple modules communicate over a well known channel names. Channel names can be specified as properties that consist of the channel names prefixed with spring.cloud.stream.bindings (e.g. spring.cloud.stream.bindings.input or spring.cloud.stream.bindings.output). These properties can be specified though environment variables, the application YAML file or the other mechanism supported by Spring Boot.
A stream app can have multiple input or output channels defined as @Input and @Output methods in an interface.
+Instead of just one channel named "input" or "output", you can add multiple MessageChannel methods annotated with
+@Input or @Output, and their names will be converted to external destination names on the broker. It is common to
+specify the channel names at runtime in order to have multiple applications communicate over well known destination
+names. Channel names can be specified as properties that consist of the channel names prefixed with
+spring.cloud.stream.bindings (e.g. spring.cloud.stream.bindings.input or spring.cloud.stream.bindings.output).
+These properties can be specified though environment variables, the application YAML file, or any of the other
+mechanisms supported by Spring Boot.
Channel names can also have a channel type as a colon-separated prefix, and the semantics of the external bus channel changes accordingly. For example, you can have two MessageChannels called "output" and "foo" in a module with spring.cloud.stream.bindings.output=bar and spring.cloud.stream.bindings.foo=topic:foo, and the result is 2 external channels called "bar" and "topic:foo". The queue prefix for point to point semantics is also supported. Note, that in a future release only topic (pub/sub) semantics will be supported.
For example, you can have two MessageChannels called "default" and "tap" in an application with
+spring.cloud.stream.bindings.default.destination=foo and spring.cloud.stream.bindings.tap.destination=bar,
+and the result is 2 bindings to an external broker with destinations called "foo" and "bar".
While Spring Cloud Stream makes it easy for individual modules to connect to messaging systems, the typical scenario for Spring Cloud Stream is the creation of multi-module pipelines, where modules are sending data to each other. This can be achieved by correlating the input and output destinations of adjacent modules, as in the following example.
+While Spring Cloud Stream makes it easy for individual boot apps to connect to messaging systems, the typical scenario +for Spring Cloud Stream is the creation of multi-app pipelines, where microservice apps are sending data to each other. +This can be achieved by correlating the input and output destinations of adjacent apps, as in the following example.
Supposing that the design calls for the time-source module to send data to the log-sink module, we will use a common destination named foo for both modules. time-source will set spring.cloud.stream.bindings.output=foo and log-sink will set spring.cloud.stream.bindings.input=foo.
Supposing that the design calls for the time-source app to send data to the log-sink app, we will use a
+common destination named ticktock for bindings within both apps. time-source will set
+spring.cloud.stream.bindings.output.destination=ticktock, and log-sink will set
+spring.cloud.stream.bindings.input.destination=ticktock.
The input and output channel names are the common properties to set in order to have Spring Cloud Stream applications communicate with each other as the channels are bound to an external message broker automatically. However, there are a number of scenarios when it is required to configure other attributes besides the channel name. This is done using the following naming scheme: spring.cloud.stream.bindings.<channelName>.<attributeName>=<attributeValue>. The destination attribute can also be used for configuring the external channel, as follows: spring.cloud.stream.bindings.input.destination=foo. This is equivalent to spring.cloud.stream.bindings.input=foo, but the latter can be used only when there are no other attributes to set on the binding. In other words, spring.cloud.stream.bindings.input.destination=foo,spring.cloud.stream.bindings.input.partitioned=true is a valid setup, whereas spring.cloud.stream.bindings.input=foo,spring.cloud.stream.bindings.input.partitioned=true is not valid.
Spring Cloud Stream is a library focusing on building message-driven microservices, and more specifically stream +processing applications. In such scenarios, communication between different logical applications follows a +publish-subscribe pattern, with data being broadcast through a shared topic, but at the same time, it is important to +be able to scale up by creating multiple instances of a given application, which are in a competing consumer +relationship with each other.
+Spring Cloud Stream models this behavior through the concept of a consumer group, which is similar to the notion of
+consumer groups in Kafka. Each consumer binding can specify a group name such as
+spring.cloud.stream.bindings.input.group=foo (the actual name of the binding may vary). Each consumer group bound to
+a given destination will receive a copy of the published data, but within the group, only one application will receive
+each specific message.
If no consumer group is specified for a given binding, then the binding is treated as if belonging to an anonymous, +independent, single-member consumer group. Otherwise said, if no consumer group is specified for a binding, it will be +in a publish-subscribe relationship with any other consumer groups.
+In general, it is preferable to always specify a consumer group when binding an application to a given destination. +When scaling up a Spring Cloud Stream application, a consumer group must be specified for each of its input bindings, +in order to prevent its instances from receiving duplicate messages (unless that behavior is desired, which is a less +common use case).
+|
+ Note
+ |
++This feature has been introduced since version 1.0.0.M4. + | +
When scaling up Spring Cloud Stream applications, each instance can receive information about how many other instances
+of the same application exist and what its own instance index is. This is done through the
+spring.cloud.stream.instanceCount and spring.cloud.stream.instanceIndex properties. For example, if there are 3
+instances of the HDFS sink application, all three will have spring.cloud.stream.instanceCount set to 3, and the
+applications will have spring.cloud.stream.instanceIndex set to 0, 1 and 2, respectively. When Spring Cloud Stream
+applications are deployed via Spring Cloud Data Flow, these properties are configured automatically, but when Spring
+Cloud Stream applications are launched independently, these properties must be set correctly. By default
+spring.cloud.stream.instanceCount is 1, and spring.cloud.stream.instanceIndex is 0.
Setting up the two properties correctly on scale up scenarios is important for addressing partitioning behavior in +general (see below), and they are always required by certain types of binders (e.g. the Kafka binder) in order to +ensure that data is split correctly across multiple consumer instances.
+The input and output destination names are the primary properties to set in order to have Spring Cloud Stream
+applications communicate with each other as their channels are bound to an external message broker automatically.
+However, there are a number of scenarios where it is required to configure other attributes besides the destination
+name. This is done using the following naming scheme:
+spring.cloud.stream.bindings.<channelName>.<attributeName>=<attributeValue>. The destination attribute is one such
+example: spring.cloud.stream.bindings.input.destination=foo. A shorthand equivalent can be used as follows:
+spring.cloud.stream.bindings.input=foo, but that shorthand can only be used only when there are no other attributes
+to set on the binding. In other words,
+spring.cloud.stream.bindings.input.destination=foo,spring.cloud.stream.bindings.input.partitioned=true is a valid
+setup, whereas spring.cloud.stream.bindings.input=foo,spring.cloud.stream.bindings.input.partitioned=true is not.
Spring Cloud Stream provides support for partitioning data between multiple instances of a given application. In a partitioned scenario, one or more producer modules will send data to one or more consumer modules, ensuring that data with common characteristics is processed by the same consumer instance. The physical communication medium (i.e. the broker topic or queue) is viewed as structured into multiple partitions. Regardless whether the broker type is naturally partitioned (e.g. Kafka) or not (e.g. Rabbit or Redis), Spring Cloud Stream provides a common abstraction for implementing partitioned processing use cases in a uniform fashion.
+Spring Cloud Stream provides support for partitioning data between multiple instances of a given application. In a +partitioned scenario, one or more producer apps will send data to one or more consumer apps, ensuring that data with +common characteristics is processed by the same consumer instance. The physical communication medium (i.e. the broker +topic or queue) is viewed as structured into multiple partitions. Regardless of whether the broker type is naturally +partitioned (e.g. Kafka) or not (e.g. Rabbit), Spring Cloud Stream provides a common abstraction for implementing +partitioned processing use cases in a uniform fashion.
Setting up a partitioned processing scenario requires configuring both the data producing and the data consuming end.
An output channel is configured to send partitioned data, by setting one and only one of its partitionKeyExpression or partitionKeyExtractorClass properties, as well as its partitionCount property. For example seting spring.cloud.stream.bindings.output.partitionKeyExpression=payload.id,spring.cloud.stream.bindings.output.partitionCount=5 is a valid and typical configuration.
An output binding is configured to send partitioned data, by setting one and only one of its partitionKeyExpression
+or partitionKeyExtractorClass properties, as well as its partitionCount property. For example, setting
+spring.cloud.stream.bindings.output.partitionKeyExpression=payload.id,spring.cloud.stream.bindings.output.partitionCount=5
+is a valid and typical configuration.
Based on this configuration, the data will be sent to the target partition using the following logic. A partition key’s value is calculated for each message sent to a partitioned output channel based on the partitionKeyExpression. The partitionKeyExpression is a SpEL expression that is evaluated against the outbound message for extracting the partitioning key. If a SpEL expression is not sufficent for your needs, you can instead calculate the partition key value by setting the the property partitionKeyExtractorClass. This class must implement the interface org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy. While, in general, the SpEL expression is enough, more complex cases may use the custom implementation strategy.
Based on this configuration, the data will be sent to the target partition using the following logic. A partition key’s
+value is calculated for each message sent to a partitioned output channel based on the partitionKeyExpression. The
+partitionKeyExpression is a SpEL expression that is evaluated against the outbound message for extracting the
+partitioning key. If a SpEL expression is not sufficient for your needs, you can instead calculate the partition key
+value by setting the property partitionKeyExtractorClass. This class must implement the interface
+org.springframework.cloud.stream.binder.PartitionKeyExtractorStrategy. While, in general, the SpEL expression should
+suffice, more complex cases may use the custom implementation strategy.
Once the message key is calculated, the partition selection process will determine the target partition as a value between 0 and partitionCount. The default calculation, applicable in most scenarios is based on the formula key.hashCode() % partitionCount. This can be customized on the binding, either by setting a SpEL expression to be evaluated against the key via the partitionSelectorExpression property, or by setting a org.springframework.cloud.stream.binder.PartitionSelectorStrategy implementation via the partitionSelectorClass property.
Once the message key is calculated, the partition selection process will determine the target partition as a value
+between 0 and partitionCount - 1. The default calculation, applicable in most scenarios is based on the formula
+key.hashCode() % partitionCount. This can be customized on the binding, either by setting a SpEL expression to be
+evaluated against the key via the partitionSelectorExpression property, or by setting a
+org.springframework.cloud.stream.binder.PartitionSelectorStrategy implementation via the partitionSelectorClass
+property.
Additional properties can be configured for more advanced scenarios, as described in the following section.
An input channel is configured to receive partitioned data by setting its partitioned binding property, as well as the instance index and instance count properties on the module, as follows: spring.cloud.stream.bindings.input.partitioned=true,spring.cloud.stream.instanceIndex=3,spring.cloud.stream.instanceCount=5. The instance count value represents the total number of similar modules between which the data needs to be partitioned, whereas instance index must be value unique across the multiple instances between 0 and instanceCount - 1. The instance index helps each module to identify the unique partition (or in the case of Kafka, the partition set) that they receive data from. It is important that both values are set correctly in order to ensure that all the data is consumed, as well as that the modules receive mutually exclusive datasets.
An input binding is configured to receive partitioned data by setting its partitioned property, as well as the
+instance index and instance count properties on the app itself, as follows:
+spring.cloud.stream.bindings.input.partitioned=true,spring.cloud.stream.instanceIndex=3,spring.cloud.stream.instanceCount=5.
+The instance count value represents the total number of app instances between which the data needs to be partitioned,
+whereas instance index must be a unique value across the multiple instances, between 0 and instanceCount - 1. The
+instance index helps each app instance to identify the unique partition (or in the case of Kafka, the partition set)
+from which it receives data. It is important that both values are set correctly in order to ensure that all the data is
+consumed, and that the app instances receive mutually exclusive datasets.
While setting up multiple instances for partitioned data processing may be complex in the standalone case, Spring Cloud Data Flow can simplify the process significantly, by populating both the input and output values correctly, as well as relying on the runtime infrastructure to provide information about the instance index and instance count.
+While setting up multiple instances for partitioned data processing may be complex in the standalone case, Spring Cloud +Data Flow can simplify the process significantly, by populating both the input and output values correctly, as well as +relying on the runtime infrastructure to provide information about the instance index and instance count.
Spring Cloud Stream relies on implementations of the Binder SPI to perform the task of connecting channels to message brokers. Each binder implementation typically connects to one type of messaging system. Spring Cloud Stream provides out of the box binders for Redis, Rabbit and Kafka.
+Spring Cloud Stream relies on implementations of the Binder SPI to perform the task of connecting channels to message +brokers. Each Binder implementation typically connects to one type of messaging system. Spring Cloud Stream provides +out of the box binders for Kafka, RabbitMQ and Redis.
By default, Spring Cloud Stream relies on Spring Boot’s auto-configuration configure the binding process. If a single binder implementation is found on the classpath, Spring Cloud Stream will use it automatically. So, for example, a Spring Cloud Stream project that aims to connect to Rabbit MQ can simply add the following dependency to their application:
+By default, Spring Cloud Stream relies on Spring Boot’s auto-configuration to configure the binding process. If a +single binder implementation is found on the classpath, Spring Cloud Stream will use it automatically. So, for example, +a Spring Cloud Stream project that aims to bind only to RabbitMQ can simply add the following dependency:
Source in the application context so
When multiple binders are present on the classpath, the application must indicate what binder has to be used for the channel. Each binder configuration contains a META-INF/spring.binders, which is in fact a property file:
When multiple binders are present on the classpath, the application must indicate which binder is to be used for each
+channel binding. Each binder configuration contains a META-INF/spring.binders, which is a simple properties file:
Similar files exist for the other binder implementations (i.e. Kafka and Redis), and it is expected that custom binder implementations will provide them, too. The key represents an identifying name for the binder implementation, whereas the value is a comma-separated list of configuration classes that contain one and only one bean definition of the type org.springframework.cloud.stream.binder.Binder.
Similar files exist for the other binder implementations (i.e. Kafka and Redis), and it is expected that custom binder
+implementations will provide them, too. The key represents an identifying name for the binder implementation, whereas
+the value is a comma-separated list of configuration classes that contain one and only one bean definition of the type
+org.springframework.cloud.stream.binder.Binder.
Selecting the binder can be done globally by either using the spring.cloud.stream.defaultBinder property, e.g. spring.cloud.stream.defaultBinder=redis, or by individually configuring them on each channel.
Selecting the binder can be done globally by either using the spring.cloud.stream.defaultBinder property, e.g.
+spring.cloud.stream.defaultBinder=rabbit, or by individually configuring them on each channel binding.
For instance, a processor module that reads from Rabbit and writes to Redis can specify the following configuration: spring.cloud.stream.bindings.input.binder=rabbit,spring.cloud.stream.bindings.output.binder=redis.
For instance, a processor app that reads from Kafka and writes to Rabbit can specify the following configuration:
+spring.cloud.stream.bindings.input.binder=kafka,spring.cloud.stream.bindings.output.binder=rabbit.
By default, binders share the Spring Boot autoconfiguration of the application module and create one instance of each binder found on the classpath. In scenarios where a module should connect to more than one broker of the same type, Spring Cloud Stream allows you to specify multiple binder configurations, with different environment settings. Please note that turning on explicit binder configuration will disable the default binder configuration process altogether, so all the binders in use must be included in the configuration.
+By default, binders share the Spring Boot auto-configuration of the application and create one instance of each binder +found on the classpath. In scenarios where an application should connect to more than one broker of the same type, +Spring Cloud Stream allows you to specify multiple binder configurations, with different environment settings. Please +note that turning on explicit binder configuration will disable the default binder configuration process altogether, so +all the binders in use must be included in the configuration.
For example, this is the typical configuration for a processor that connects to two rabbit instances:
+For example, this is the typical configuration for a processor that connects to two RabbitMQ broker instances:
Code using the Spring Cloud Stream library can be deployed as a standalone application or be used as a Spring Cloud Data Flow module. In standalone mode your application will run happily as a service or in any PaaS (Cloud Foundry, Lattice, Heroku, Azure, etc.). Spring Cloud Data Flow helps orchestrating the communication between instances, so the aspects of module configuration that deal with module interconnection will be configured transparently.
+Code using the Spring Cloud Stream library can be deployed as a standalone application or be used as a Spring Cloud +Data Flow module. In standalone mode, your application will run happily as a service or in any PaaS (Cloud Foundry, +Heroku, Azure, etc.). Spring Cloud Data Flow helps orchestrate the communication between instances, so the aspects of +configuration that deal with application interconnection will be configured transparently.
You can run in standalone mode from your IDE for testing. To run in production you can create an executable (or "fat") JAR using the standard Spring Boot tooling provided by Maven or Gradle.
+You can run in standalone mode from your IDE for testing. To run in production you can create an executable (or "fat") +JAR using the standard Spring Boot tooling provided for Maven or Gradle.
There are several samples, all running on the redis transport (so you need redis running locally to test them).
+Spring Cloud Stream provides a health indicator for the binders, registered under the name of binders. It can be
+enabled or disabled using the management.health.binders.enabled property.
As described above, Spring Cloud Stream provides a binder abstraction for connecting to physical destinations. This +section will provide more information about the main concepts behind the Binder SPI, its main components, as well as +details specific to different implementations.
+
+A producer is any component that sends messages to a channel. That channel can be bound to an external message broker
+via a Binder implementation for that broker. When invoking the bindProducer method, the first parameter is the name
+of the destination within that broker. The second parameter is the local channel instance to which the producer will be
+sending messages, and the third parameter contains properties to be used within the adapter that is created for that
+channel, such as a partition key expression.
A consumer is any component that receives messages from a channel. As with the producer, the consumer’s channel can be
+bound to an external message broker, and the first parameter for the bindConsumer method is the destination name.
+However, on the consumer side, a second parameter provides the name of a logical group of consumers. Each group
+represented by consumer bindings for a given destination will receive a copy of each message that a producer sends to
+that destination (i.e. pub/sub semantics). If there are multiple consumer instances bound using the same group name,
+then messages will be load balanced across those consumer instances so that each message sent by a producer would only
+be consumed by a single consumer instance within each group (i.e. queue semantics).
+The Kafka Binder implementation maps the destination to a Kafka topic, and the consumer group maps directly to the same +Kafka concept. Spring Cloud Stream does not use the high level consumer, but implements a similar concept for the +simple consumer.
+
+The RabbitMQ Binder implementation maps the destination to a TopicExchange, and for each consumer group, a Queue
+will be bound to that TopicExchange. Each consumer instance that binds will trigger creation of a corresponding
+RabbitMQ Consumer instance for its group’s Queue.
+|
+ Note
+ |
++we recommend only using the Redis Binder for development + | +
The Redis Binder creates a LIST (which performs the role of a queue) for each consumer group. A consumer binding will
+trigger BRPOP operations on its group’s LIST. A producer binding will consult a ZSET to determine what groups
+currently have active consumers, and then for each message being sent, an LPUSH operation will be executed on each of
+those group’s LISTs.
If you’re just getting started with Spring Cloud Task, this is the section +for you! Here we answer the basic “what?”, “how?” and “why?” questions. You’ll +find a gentle introduction to Spring Cloud Task. We’ll then build our first Spring Cloud +Task application, discussing some core principles as we go.
+Spring Cloud Task makes it easy to create short lived microservices. We provide +capabilities that allow short lived JVM processes to be executed on demand in a production +environment.
+You need Java installed (Java 7 or better, we recommend Java 8) and to build you need to have Maven installed as well.
+Spring Cloud Task uses a relational database to store the results of an executed task. +While you can begin developing a task without a database (the status of the task is logged + as part of the task repository’s updates), for production environments, you’ll want to +utilize a supported database. Below is a list of the ones currently supported:
source is a Java config version of the classic "timer" module from Spring XD. It has a "fixedDelay" option (in milliseconds) for the period between emitting messages.
H2
sink is a Java config version of the classic "log" module from Spring XD. It has no options (but some could easily be added), and just logs incoming messages at INFO level.
HSQLDB
transform is a simple pass through logging transformer (just logs the incoming message and passes it on).
MySql
double is a combination of 2 modules defined locally (a source and a sink, so the whole app is self contained).
Oracle
extended is a multi-module mashup of source | transform | transform | sink, where the modules are defined in the other samples and referred to in this app just as dependencies.
Postgres
If you run the source and the sink and point them at the same redis instance (e.g. do nothing to get the one on localhost, or the one they are both bound to as a service on Cloud Foundry) then they will form a "stream" and start talking to each other. All the samples have friendly JMX and Actuator endpoints for inspecting what is going on in the system.
To build the source you will need to install JDK 1.7.
+A good place to start is with a simple "Hello World!" application so we’ll create the +Spring Cloud Task equivalent to highlight the features of the framework. We’ll use Apache +Maven as a build tool for this project since most IDEs have good support for it.
+|
+ Note
+ |
++The spring.io web site contains many “Getting Started” guides that use Spring Boot. +If you’re looking to solve a specific problem; check there first. You can shortcut the +steps below by going to start.spring.io and creating a new project. This will +automatically generate a new project structure so that you can start coding right the way. +Check the documentation for more details. + | +
The build uses the Maven wrapper so you don’t have to install a specific -version of Maven. To enable the tests for Redis, Rabbit, and Kafka bindings you -should have those servers running before building. See below for more -information on running the servers.
-The main build command is
+Before we begin, open a terminal to check that you have valid versions of Java and Maven +installed.
$ ./mvnw clean install+
$ java -version
+java version "1.8.0_31"
+Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
+Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
You can also add '-DskipTests' if you like, to avoid running the tests.
+$ mvn -v
+Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T15:58:10-05:00)
+Maven home: /usr/local/Cellar/maven/3.2.3/libexec
+Java version: 1.8.0_31, vendor: Oracle Corporation
+
-You can also install Maven (>=3.3.3) yourself and run the mvn command
-in place of ./mvnw in the examples below. If you do that you also
-might need to add -P spring if your local Maven settings do not
-contain repository declarations for spring pre-release artifacts.
+This sample needs to be created in its own folder. Subsequent instructions assume
+you have created a suitable folder and that it is your "current directory".
|
|
- Note
- |
-
-Be aware that you might need to increase the amount of memory
-available to Maven by setting a MAVEN_OPTS environment variable with
-a value like -Xmx512m -XX:MaxPermSize=128m. We try to cover this in
-the .mvn configuration, so if you find you have to do it to make a
-build succeed, please raise a ticket to get the settings added to
-source control.
- |
-
The projects that require middleware generally include a
-docker-compose.yml, so consider using
-Docker Compose to run the middeware servers
-in Docker containers. See the README in the
-scripts demo
-repository for specific instructions about the common cases of mongo,
-rabbit and redis.
There is a "full" profile that will generate documentation.
-If you don’t have an IDE preference we would recommend that you use -Spring Tools Suite or -Eclipse when working with the code. We use the -m2eclipe eclipse plugin for maven support. Other IDEs and tools -should also work without issue.
-We recommend the m2eclipe eclipse plugin when working with -eclipse. If you don’t already have m2eclipse installed it is available from the "eclipse -marketplace".
+We need to start by creating a Maven pom.xml file. The pom.xml is the recipe that
+will be used to build your project. Open your favorite text editor and add the following:
<?xml version="1.0" encoding="UTF-8"?> +<project xmlns="http://maven.apache.org/POM/4.0.0" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + + <modelVersion>4.0.0</modelVersion> + + <groupId>com.example</groupId> + <artifactId>myproject</artifactId> + <packaging>jar</packaging> + <version>0.0.1-SNAPSHOT</version> + + <parent> + <groupId>org.springframework.boot</groupId> + <artifactId>spring-boot-starter-parent</artifactId> + <version>1.3.2.RELEASE</version> + </parent> + + <properties> + <start-class>com.example.SampleTask</start-class> + </properties> + + <dependencies> + <dependency> + <groupId>org.springframework.boot</groupId> + <artifactId>spring-boot-starter</artifactId> + </dependency> + </dependencies> + + <build> + <plugins> + <plugin> + <groupId>org.springframework.boot</groupId> + <artifactId>spring-boot-maven-plugin</artifactId> + </plugin> + </plugins> + </build> +</project>+
Unfortunately m2e does not yet support Maven 3.3, so once the projects
-are imported into Eclipse you will also need to tell m2eclipse to use
-the .settings.xml file for the projects. If you do not do this you
-may see many different errors related to the POMs in the
-projects. Open your Eclipse preferences, expand the Maven
-preferences, and select User Settings. In the User Settings field
-click Browse and navigate to the Spring Cloud project you imported
-selecting the .settings.xml file in that project. Click Apply and
-then OK to save the preference changes.
This should give you a working build. You can test it out by running mvn package (you
+can ignore the "jar will be empty - no content was marked for inclusion!" warning for
+now).
-Alternatively you can copy the repository settings from .settings.xml into your own ~/.m2/settings.xml.
+At this point you could import the project into an IDE (most modern Java IDE’s
+include built-in support for Maven). For simplicity we will continue to use a plain text
+editor for this example.
|
If you prefer not to use m2eclipse you can generate eclipse project metadata using the -following command:
+A Spring Cloud Task is made up of a Spring Boot application that is expected to end. In
+our POM above, we created the shell of a Spring Boot application from a dependency
+perspective by setting our parent to use the spring-boot-starter-parent.
Spring Boot provides a number of additional "Starter POMs". Some of which are appropriate
+for use within tasks (spring-boot-starter-batch, spring-boot-starter-jdbc, etc) and
+some may not be ('spring-boot-starter-web` is probably not going to be used in a task).
+The indicator of if a starter makes sense or not comes down to if the resulting
+application will end (batch based applications typically end, the
+spring-boot-starter-web dependency bootstraps a servlet container which probably wont').
For this example, we’ll only need to add a single additional dependency, the one for +Spring Cloud Task itself:
$ ./mvnw eclipse:eclipse+
<dependency>
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-task-core</artifactId>
+ <version>1.0.0.BUILD-SNAPSHOT</version>
+ </dependency>
+To finish our application, we need to create a single Java file. Maven will compile the
+sources from src/main/java by default so you need to create that folder structure. Then
+add a file named src/main/java/com/example/SampleTask.java:
package com.example;
+
+import org.springframework.boot.*;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.cloud.task.configuration.EnableTask;
+import org.springframework.context.annotation.Bean;
+
+@SpringBootApplication
+@EnableTask
+public class SampleTask {
+
+ @Bean
+ public CommandLineRunner commandLineRunner() {
+ return new HelloWorldCommandLineRunner();
+ }
+
+ public static void main(String[] args) {
+ SpringApplication.run(SampleTask.class, args);
+ }
+
+ public static class HelloWorldCommandLineRunner implements CommandLineRunner {
+
+ @Override
+ public void run(String... strings) throws Exception {
+ System.out.println("Hello World!");
+ }
+ }
+}
The generated eclipse projects can be imported by selecting import existing projects
-from the file menu.
-[[contributing]
-== Contributing
While it may not look like much, quite a bit is going on. To read more about the Spring +Boot specifics, take a look at their reference documentation here: +http://docs.spring.io/spring-boot/docs/current/reference/html/
+The first non boot annotation in our example is the @EnableTask annotation. This class
+level annotation tells Spring Cloud Task to bootstrap it’s functionality. This occurs by
+importing an additional configuration class, SimpleTaskConfiguration by default. This
+additional configuration registers the TaskRepository and the infrastructure for its
+use.
Spring Cloud is released under the non-restrictive Apache 2.0 license, -and follows a very standard Github development process, using Github -tracker for issues and merging pull requests into master. If you want -to contribute even something trivial please do not hesitate, but -follow the guidelines below.
+Out of the box, the TaskRepository will use an in memory Map to record the results
+of a task. Obviously this isn’t a practical solution for a production environment since
+the Map goes away once the task ends. However, for a quick getting started
+experience we use this as a default as well as echoing to the logs what is being updated
+in that repository. Later in this documentation we’ll cover how to customize the
+configuration of the pieces provided by Spring Cloud Task.
Before we accept a non-trivial patch or pull request we will need you to sign the -contributor’s agreement. -Signing the contributor’s agreement does not grant anyone commit rights to the main -repository, but it does mean that we can accept your contributions, and you will get an -author credit if we do. Active contributors might be asked to join the core team, and -given the ability to merge pull requests.
+When our sample application is run, Spring Boot will launch our
+HelloWorldCommandLineRunner outputting our "Hello World!" message to standard out. The
+TaskLifecyceListener will record the start of the task and the end of the task in the
+repository.
None of these is essential for a pull request, but they will all help. They can also be -added after the original pull request but before a merge.
+The main method serves as the entry point to any java application. Our main method
+delegates to Spring Boot’s SpringApplication class. You can read more about it in the
+Spring Boot documentation.
In Spring, there are many ways to bootstrap an application’s logic. Spring Boot provides
+a convenient method of doing so in an organized manor via their *Runner interfaces
+(CommandLineRunner or ApplicationRunner). A well behaved task will bootstrap any
+logic via one of these two runners.
The lifecycle of a task is considered from before the *Runner#run methods are executed
+to once they are all complete. Spring Boot allows an application to use multiple
+*Runner implementation and Spring Cloud Task doesn’t attempt to impede on this convention.
|
+ Note
+ |
+
+Any processing bootstrapped from mechanisms other than a CommandLineRunner or
+ApplicationRunner (using InitializingBean#afterPropertiesSet for example) will not be
+ recorded by Spring Cloud Task.
+ |
+
At this point, your application should work. Since this application is Spring Boot based,
+ we can run it from the command line via the command $ mvn spring-boot:run from the root
+ of our applicaiton:
$ mvn clean spring-boot:run
+....... . . .
+....... . . . (Maven log output here)
+....... . . .
+
+
+ . ____ _ __ _ _
+ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
+( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
+ \\/ ___)| |_)| | | | | || (_| | ) ) ) )
+ ' |____| .__|_| |_|_| |_\__, | / / / /
+ =========|_|==============|___/=/_/_/_/
+ :: Spring Boot :: (v1.3.2.RELEASE)
+
+2016-01-25 11:08:10.183 INFO 12943 --- [ main] com.example.SampleTask : Starting SampleTask on Michaels-MacBook-Pro-2.local with PID 12943 (/Users/mminella/Documents/IntelliJWorkspace/spring-cloud-task-example/target/classes started by mminella in /Users/mminella/Documents/IntelliJWorkspace/spring-cloud-task-example)
+2016-01-25 11:08:10.185 INFO 12943 --- [ main] com.example.SampleTask : No active profile set, falling back to default profiles: default
+2016-01-25 11:08:10.226 INFO 12943 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@2a2c3676: startup date [Mon Jan 25 11:08:10 CST 2016]; root of context hierarchy
+2016-01-25 11:08:11.051 INFO 12943 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
+2016-01-25 11:08:11.065 INFO 12943 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Creating: TaskExecution{executionId=0, externalExecutionID='null', exitCode=0, taskName='application', startTime=Mon Jan 25 11:08:11 CST 2016, endTime=null, statusCode='null', exitMessage='null', parameters=[]}
+Hello World!
+2016-01-25 11:08:11.071 INFO 12943 --- [ main] com.example.SampleTask : Started SampleTask in 1.095 seconds (JVM running for 3.826)
+2016-01-25 11:08:11.220 INFO 12943 --- [ Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@2a2c3676: startup date [Mon Jan 25 11:08:10 CST 2016]; root of context hierarchy
+2016-01-25 11:08:11.222 INFO 12943 --- [ Thread-1] o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution{executionId=0, externalExecutionID='null', exitCode=0, taskName='application', startTime=Mon Jan 25 11:08:11 CST 2016, endTime=Mon Jan 25 11:08:11 CST 2016, statusCode='null', exitMessage='null', parameters=[]}
+2016-01-25 11:08:11.222 INFO 12943 --- [ Thread-1] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
+If you notice, there are three lines of interest in the above output:
Use the Spring Framework code format conventions. If you use Eclipse
-you can import formatter settings using the
-eclipse-code-formatter.xml file from the
-Spring
-Cloud Build project. If using IntelliJ, you can use the
-Eclipse Code Formatter
-Plugin to import the same file.
SimpleTaskRepository logged out the creation of the entry in the TaskRepository.
Make sure all new .java files to have a simple Javadoc class comment with at least an
-@author tag identifying you, and preferably at least a paragraph on what the class is
-for.
The execution of our CommandLineRunner, demonstrated by the "Helo World!" output.
Add the ASF license header comment to all new .java files (copy from existing files
-in the project)
Add yourself as an @author to the .java files that you modify substantially (more
-than cosmetic changes).
Add some Javadocs and, if you change the namespace, some XSD doc elements.
-A few unit tests would help a lot as well — someone has to do it.
-If no-one else is using your branch, please rebase it against the current master (or -other target branch in the main project).
-When writing a commit message please follow these conventions,
-if you are fixing an existing issue please add Fixes gh-XXXX at the end of the commit
-message (where XXXX is the issue number).
SimpleTaskREpository logging the completion of the task in the TaskRepository.
Unresolved directive in spring-cloud.adoc - include::../../../task/spring-cloud-task-docs/src/main/asciidoc/index.adoc[]
+This section goes into more detail about Spring Cloud Task. How to use it, how to +configure it, as well as the appropriate extension points are all covered in this section.
+In most cases, the modern cloud environment is designed around the execution of processes +that are not expected to end. If they do, they are typically restarted. While most +platforms do have some method to execute a process that isn’t restarted when it ends, the + results of that execution are typically not maintained in a consumable way. Spring Cloud + Task brings the ability to execute short lived processes in an environment and record the + results. This allows for a microservices architecture around short lived processes as + well as longer running services.
+While this functionality is useful in a cloud environment, the same issues can arise in a +traditional deployment model as well. When executing Spring Boot applications via a +scheduler like cron, it can be useful to be able to monitor the results of the application +after it’s completion.
+A Spring Cloud Task takes the approach that a Spring Boot application can have a start and an +end and still be successful. Batch applications are just one example of where short lived +processes can be helpful. Spring Cloud Task records lifecycle events of a given task.
+The lifecycle consists of a single task execution. This is a physical execution of a
+Spring Boot application configured to be a task (annotated with the @EnableTask
+annotation).
At the beginning of a task, an entry in the TaskRepository is created recording the
+start event. This event is triggered via the ContextRefreshEvent being triggered by
+Spring Framework.
|
+ Note
+ |
+
+As Spring Cloud Task is expected to consist of a single application context. If
+multiple application contexts are used (parent/child relationships for example), the first
+ContextRefreshEvent that is published by Spring will be recorded as the start of the
+task.
+ |
+
|
+ Note
+ |
+
+The recording of a task will only occur upon the successful bootstrapping of an
+ApplicationContext. If the context fails to bootstrap at all, the task’s execution will
+not be recorded.
+ |
+
Upon completion of all of the *Runner#run calls from Spring Boot or the failure of an
+ApplicationContext (indicated via a ApplicationFailedEvent), the task execution is
+updated in the repository with the results.
The information stored in the TaskRepository is modeled in the TaskExecution class and
+consists of the following information:
| Field | +Description | +
|---|---|
|
+The unique id for the task’s execution. |
+
|
+The exit code generated from an |
+
|
+The name for the task as determined by the configured |
+
|
+The time the task was started as indicated by the |
+
|
+The time the task was completed as indicated by the |
+
|
+Any information available at the time of exit. If an exception is the cause of the end
+of the task (as indicated via an |
+
|
+A |
+
When a task completes, it will want to return an exit code to the OS. If we take a look +at our original example, we can see that we are not controlling that aspect of our +application. So if an exception is thrown, the JVM will return a code that may or may not +be of any use to you in the debugging of that.
+As such, Spring Boot provides an interface, ExitCodeExceptionMapper that allows you to
+map uncaught exceptions to exit codes. This allows you to be able to indicate at that
+level what went wrong. Also, by mapping exit codes in this manor, Spring Cloud Task will
+record the exit code returned.
Spring Cloud Task provides an out of the box configuration as defined in the
+ DefaultTaskConfigurer and SimpleTaskConfiguration. This section will walk through
+the defaults as well as how to customize Spring Cloud Task for your needs
Spring Cloud Task utilizes a datasource for storing the results of task executions. By
+default, we provide an in memory instance of H2 to provide a simple method of
+bootstrapping development. However, in a production environment, you’ll want to configure
+your own DataSource.
If your application utilizes only a single DataSource and that will serve as both your
+business schema as well as the task repository, all you need to do is provide any
+DataSource (via Spring Boot’s configuration conventions is the easiest way). This will
+be automatically used by Spring Cloud Task for the repository.
If your application utilizes more than one DataSource, you’ll need to configure the
+task repository with the appropriate DataSource. This customization can be done via an
+implementation of the TaskConfigurer.
The TaskConfigurer is a strategy interface allowing for users to customize the way
+components of Spring Cloud Task are configured. By default, we provide the
+DefaultTaskConfigurer that provides logical defaults (Map based in memory components
+useful for development if no DataSource is provided and JDBC based components if there
+is a DataSource available.
The TaskConfigurer allows the configuration of three main components:
| Component | +Description | +Default (provided by DefaultTaskConfigurer) |
+
|---|---|---|
|
+The implementation of the |
+
|
+
|
+The implementation of the |
+
|
+
|
+A transaction manager to be used when executing updates for tasks. |
+
|
+
In most cases, the name of the task will be the application name as configured via Spring
+Boot. However, there are some cases, where you may want to map the run of a task to a
+different name. Spring Data Flow is an example of this (where you want the task to be run
+with the name of the task definition). Because of this, we offer the ability to customize
+how the task is named via the TaskNameResolver interface.
By default, Spring Cloud Task provides the SimpleTaskNameResolver which will use the
+following options (in order of precedence):
A Spring Boot property (configured any of the ways Spring Boot allows)
+spring.cloud.task.name.
The application name as resolved using Spring Boot’s rules (obtained via
+ApplicationContext#getId).