65f37c170a
Update URLs pointing to springframework/springsource to spring.io. Remove links that have no redirection
Spring Batch Integration
Many users of Spring Batch may encounter requirements that are outside the scope of Spring Batch, yet may be efficiently and concisely implemented using Spring Integration. Conversely, Spring Batch users may encounter Spring Batch requirements and need a way to efficiently integrate both frameworks. In this context several patterns and use-cases emerge and Spring Batch Integration will address those requirements:
- Launching Batch Jobs through Messages
- Providing Feedback with Informational Messages
- Externalizing Batch Process Execution
Downloading artifacts
Using Maven
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-integration</artifactId>
<version>1.3.0.BUILD-SNAPSHOT</version>
</dependency>
Using Gradle
compile 'org.springframework.batch:spring-batch-integration:1.3.0.BUILD-SNAPSHOT'
Documentation
Namespace Support
As of Spring Batch Integration 1.3, we start providing dedicated XML Namespace support, with the aim to provide an easier configuration experience. In order to activate the namespace, add the following namespace declarations to your Spring XML Application Context file:
<beans ...
xmlns:batch-int="http://www.springframework.org/schema/batch-integration"
xsi:schemaLocation="...
http://www.springframework.org/schema/batch-integration http://www.springframework.org/schema/batch-integration/spring-batch-integration.xsd">
...
</beans>
A fully configured Spring XML Application Context file for Spring Batch Integration may look like the following:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:batch-int="http://www.springframework.org/schema/batch-integration"
xmlns:jdbc="http://www.springframework.org/schema/jdbc"
xsi:schemaLocation="
http://www.springframework.org/schema/batch-integration http://www.springframework.org/schema/batch-integration/spring-batch-integration.xsd
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc.xsd">
...
</beans>
You can also append version numbers to the referenced XSD file such as:
xsi:schemaLocation="...
http://www.springframework.org/schema/batch-integration http://www.springframework.org/schema/batch-integration/spring-batch-integration.xsd">
However, as a version-less declaration will always use the latest schema, we generally don't recommend appending the version number to the XSD name. Adding a version number, for instance, would create possibly issues when updating the Spring Batch Integration dependencies as they may require more recent versions of the XML schema.
Launching Batch Jobs through Messages
When starting Batch Jobs using the core Spring Batch API you basically have 2 options:
- Command line via the CommandLineJobRunner
- Programatically via either JobOperator.start() or JobLauncher.run().
For more information please also see the Spring Batch reference documentation: Running a Job
For example, you may want to use the CommandLineJobRunner when invoking Batch Jobs using a shell script. Alternatively, you may use the JobOperator directly, for example when using Spring Batch as part of a web application. However, what about more complex use-cases? Maybe you need to poll a remote (S)FTP server to retrieve the data for the Batch Job. Or your application has to support multiple different data sources simultaneously. For example, you may receive data files not only via the web, but also FTP etc. Maybe additional transformation of the input files is needed before invoking [Spring Batch[].
Therefore, it would be much more powerful to execute the batch job using Spring Integration and its numerous adapters. For example, you can use a File Inbound Channel Adapter to monitor a directory in the file-system and start the Batch Job as soon as the input file arrives. Additionally you can create Spring Integration flows that use multiple different adapters to easily ingest data for your Batch Jobs from multiple sources simultaneously using configuration only. Implementing all these scenarios with Spring Integration is easy as it allow for an decoupled event-driven execution of the JobLauncher.
Spring Batch Integration provides the JobLaunchingMessageHandler that you can use to launch batch jobs. The input for the JobLaunchingMessageHandler is provided by a Spring Integration message, which payload is of type JobLaunchRequest. This class is a wrapper around the Job that needs to be launched as well as the JobParameters necessary to launch the Batch job.
The following image illustrates the typical Spring Integration message flow in order to start a Batch job.
Transforming a file into a JobLaunchRequest
public class FileMessageToJobRequest {
private Job job;
private String fileParameterName;
public void setFileParameterName(String fileParameterName) {
this.fileParameterName = fileParameterName;
}
public void setJob(Job job) {
this.job = job;
}
@Transformer
public JobLaunchRequest toRequest(Message<File> message) {
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
jobParametersBuilder.addString(fileParameterName, message.getPayload().getAbsolutePath());
return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}
}
The JobExecution Response
When a Batch Job is being executed, a JobExecution instance is returned. This instance can be used to determine the status of an execution. If a JobExecution was able to be created successfully, it will always be returned, regardless of whether or not the actual execution was successful.
The exact behavior on how the JobExecution instance is returned depends on the provided TaskExecutor. If a synchronous (single-threaded) TaskExecutor implementation is used, the JobExecution response is only returned after the job completes. When using an asynchronous TaskExecutor, the JobExecution instance is returned immediately. Users can then take the id of JobExecution instance (JobExecution#getJobId()) and query the JobRepository for the job's updated status using the JobExplorer. For more information, please refer to the Spring Batch reference documentation on Querying the Repository.
For more information regarding the returned JobExecution instance, please see:
Java Bean Configuration
<batch:job-repository id="jobRepository"/>
<bean id="batchjobExecutor" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository"/>
</bean>
<int:transformer input-channel="files" output-channel="requests">
<bean class="com.manning.siia.batch.FileMessageToJobRequest">
<property name="job" ref="importPayments"/>
<property name="fileParameterName" value="input.file.name"/>
</bean>
</int:transformer>
<int:service-activator method="launch" input-channel="requests"
output-channel="statuses">
<bean class="org.springframework.batch.integration.launch.JobLaunchingMessageHandler">
<constructor-arg ref="jobLauncher"/>
</bean>
</int:service-activator>
XML Namespace Support
As of Spring Batch Integration 1.3, dedicated XML namespace for launching Spring Batch jobs is available. This allows you to rewrite the above Service Activator with:
<batch-int:job-launching-gateway request-channel="requestChannel" reply-channel="replyChannel" job-launcher="jobLauncher"/>
Available Attributes of the Job-Launching Gateway
id Identifies the underlying Spring bean definition, which is an instance of either:
The exact implementation depends on whether the component's input channel is a
auto-startup Boolean flag to indicate that the endpoint should start automatically on startup. The default is true.
request-channel The input MessageChannel of this endpoint.
reply-channel Message Channel to which the resulting JobExecution payload will be sent.
reply-timeout Allows you to specify how long this gateway will wait for the reply message to be sent successfully to the reply channel before throwing an exception. This attribute only applies when the channel might block, for example when using a bounded queue channel that is currently full.
Also, keep in mind that when sending to a DirectChannel, the invocation will occur in the sender's thread. Therefore, the failing of the send operation may be caused by other components further downstream.
The reply-timeout attribute maps to the sendTimeout property of the underlying MessagingTemplate instance (org.springframework.integration.core.MessagingTemplate). The attribute will default, if not specified, to -1, meaning that by default, the Gateway will wait indefinitely. The value is specified in milliseconds.
job-launcher Pass in a custom JobLauncher bean reference. This attribute is optional. If not specified the adapter will re-use the instance that is registered under the id jobLauncher (E.g. when using the @EnableBatchProcessing annotation via JavaConfig). If no default instance exists an exception is thrown.
order Specifies the order for invocation when this endpoint is connected as a subscriber to a SubscribableChannel.
Sub-Elements
When this Gateway is receiving messages from a PollableChannel, you must either provide a global default Poller or provide a Poller sub-element to the Job-Launching Gateway:
<batch-int:job-launching-gateway request-channel="queueChannel" reply-channel="replyChannel" job-launcher="jobLauncher">
<int:poller fixed-rate="1000"/>
</batch-int:job-launching-gateway>
Providing Feedback with Informational Messages
As Spring Batch jobs can run for long times, providing progress information will be critical. For example, stake-holders may want to be notified if a Batch Job has failed. Spring Batch provides support for this information being gathered through:
- Active polling or
- Event-driven, using listeners.
When starting a Spring Batch job asynchronously, e.g. by using the Job-Launching Gateway, a JobExecution instance is returned. Thus, JobExecution#getJobId can be used to continuously poll for status updates by retrieving updated instances of the JobExecution from the JobRepository using the JobExplorer. However, this is considered sub-optimal and an event-driven approach should be preferred.
Therefore, Spring Batch provides listeners such as:
In the following example, a Spring Batch job was configured with JobExecutionListener. Thus, Spring Integration will receive and process any Batch job status changes. For example, the received JobExecution can be inspected using a Router. Job that are deemed recoverable may cause a restart of a Spring Batch job while a successful job completion may be routed to an Mail Outbound Channel Adapter, so that Email notification are sent out.
Here is a brief example how a listener is configured:
<batch:job id="importPayments">
...
<batch:listeners>
<batch:listener ref="notificationExecutionsListener"/>
</batch:listeners>
</batch:job>
<int:gateway id="notificationExecutionsListener"
service-interface="o.s.batch.core.JobExecutionListener"
default-request-channel="jobExecutions"/>
Externalizing Batch Process Execution
The two integration approaches discussed so far suggest use-cases where Spring Integration wraps Spring Batch like an outer-shell. However, Spring Batch can also use Spring Integration internally. Using this approach, Spring Batch users can delegate the processing of items or even chunks to outside processes. This allows you to offload complex processing. Spring Batch Integration provides dedicated support for:
- Remote Chunking
- Remote Partitioning
Asynchronous Processors
Asynchronous Processors help you to to scale the processing of items. In the asynchronous processor use-case, an ItemProcessor serves as a dispatcher, executing the ItemProcessor's logic for an item on a new thread. The Future is passed to the AsynchItemWriter to be written once the processor completes.
Therefore, you can increase performance by using asynchronous item processing, basically allowing you to implement fork-join scenarios. The AsyncItemWriter will gather the results and write back the chunk as soon as all the results become available.
Remote Chunking
Taking things one step further, one can also externalize the chunk processing using the ChunkMessageChannelItemWriter which is provided by Spring Batch Integration. That way you can send an entire chunk to a Gateway, reducing Spring Batch's responsibilities to read items, group them and send out the chunk. Once sent, Spring Batch will continue the process of reading and grouping items, without waiting for the results. Rather it is the responsibility of the ChunkMessageChannelItemWriter to gather the results and integrate them back into the Spring Batch process.
Using Spring Integration you have full control over the concurrency of your processes, for instance by using QueueChannels instead of DirectChannels. Furthermore, by relying on Spring Integration's rich collection of Channel Adapters (E.g. JMS or AMQP), you can distribute chunks of a Batch job to external systems for processing.
For more information, please also consult the Spring Batch manual, specifically the chapter on Remote Chunking.
Remote Partitioning
Remote Partitioning, on the other hand, is useful when the problem is not the processing of items, but the associated I/O represents the bottleneck. Using Remote Partitioning, work can be farmed out to slaves that execute complete Spring Batch steps. Thus, each slave has its own reader, processor and writer. For this purpose, Spring Batch Integration provides the MessageChannelPartitionHandler.
This implementation of the PartitionHandler interface uses MessageChannel instances to send instructions to remote workers and receive their responses. This provides a nice abstraction from the transports (E.g. JMS or AMQP) being used to communicate with the remote workers.
For more information, please see:
Resources
Further Reading
Books
The book Spring Integration in Action by Mark Fisher, Jonas Partner, Marius Bogoevici and Iwein Fuld provides a dedicated on how to use Spring Integration and Spring Batch together.
Pro Spring Integration by Dr. Mark Lui, Mario Gray, Andy Chan and Josh Long has a chapter titled Spring Integration and Spring Batch.
Pro Spring Batch by Michael Minella, provides detailed information on Spring Batch Integration in Chapter 11, Scaling and Tuning.
Presentations
- Introduction to Spring Integration and Spring Batch (Video)
- Introduction to Spring Integration and Spring Batch (Slides)
- Heavy Lifting in the Cloud with Spring Batch
- https://github.com/mminella/Spring-Batch-Talk-2.0 - Slides and code
- http://www.youtube.com/watch?v=CYTj5YT7CZU - Video
- Deploying and Monitoring Spring Integration and Spring Batch Applications
Articles
- Spring Batch integration module for GridGain
Related GitHub projects
Spring Batch
Spring Integration
- Spring Integration
- Spring Integration Samples
- Spring Integration Templates
- Spring Integration Dsl Groovy
- Spring Integration Dsl Scala
- Spring Integration Pattern Catalog
Spring XD
Spring for Apache Hadoop
Getting support
Check out the Spring forums and the spring-integration and spring-batch tags on Stack Overflow. Commercial support is available too.
Issue Tracking
Report issues via the Spring Batch Admin JIRA.
Building from source
Spring Batch Integration is built using Maven.
Check out sources
git clone git://github.com/SpringSource/spring-batch-admin.git
Go to the Spring Batch Integration project
$ cd spring-batch-admin/spring-batch-integration
Compile and test, build all jars
$ mvn clean package
Install project into your local Maven cache
$ mvn install
Contributing
Pull requests are welcome. Please see the contributor guidelines for details. Additionally, if you are contributing, we recommend following the process for Spring Integration as outlined in the administrator guidelines.
Contributors
- @david_syer - Dave Syer
- @michaelminella - Michael Minella
- @ghillert - Gunnar Hillert
License
The Spring Integration Extensions Framework is released under version 2.0 of the Apache License unless noted differently for individual extension Modules, but this should be the rare exception.