From 4e25560c4da851dd8cd345941000b6ccaf51cbcf Mon Sep 17 00:00:00 2001 From: Chris Schaefer Date: Fri, 28 Mar 2014 11:56:12 -0400 Subject: [PATCH] BATCH-2195: Document Spring Batch Integration * Removed APT files * General formatting and spelling fixes * Converted README.md to docbook * Expanded converted README.md docs to include usable code snippets --- spring-batch-integration/README.md | 394 ----- .../handling-informational-messages.png | Bin .../reference/images/launch-batch-job-svg.svg | 0 .../reference/images/launch-batch-job.png | Bin .../reference/images/remote-chunking-sbi.png | Bin .../reference/images/remote-partitioning.png | Bin src/site/docbook/reference/scalability.xml | 5 - .../reference/spring-batch-integration.xml | 1311 ++++++++++++----- 8 files changed, 905 insertions(+), 805 deletions(-) delete mode 100644 spring-batch-integration/README.md rename {spring-batch-integration/src => src/site/docbook}/reference/images/handling-informational-messages.png (100%) rename {spring-batch-integration/src => src/site/docbook}/reference/images/launch-batch-job-svg.svg (100%) rename {spring-batch-integration/src => src/site/docbook}/reference/images/launch-batch-job.png (100%) rename spring-batch-integration/src/reference/images/remote-chunking.png => src/site/docbook/reference/images/remote-chunking-sbi.png (100%) rename {spring-batch-integration/src => src/site/docbook}/reference/images/remote-partitioning.png (100%) diff --git a/spring-batch-integration/README.md b/spring-batch-integration/README.md deleted file mode 100644 index 10451ff45..000000000 --- a/spring-batch-integration/README.md +++ /dev/null @@ -1,394 +0,0 @@ -Spring Batch Integration -======================== - -Many users of [Spring Batch][] may encounter requirements that are outside the scope of [Spring Batch][], yet may be efficiently and concisely implemented using [Spring Integration][]. Conversely, [Spring Batch][] users may encounter [Spring Batch][] requirements and need a way to efficiently integrate both frameworks. In this context several patterns and use-cases emerge and *Spring Batch Integration* will address those requirements: - -* [Launching Batch Jobs through Messages](#launching-batch-jobs-through-messages) -* [Providing Feedback with Informational Messages](#providing-feedback-with-informational-messages) -* [Externalizing Batch Process Execution](#externalizing-batch-process-execution) - -# Downloading artifacts - -## Using Maven - - - org.springframework.batch - spring-batch-integration - 1.3.0.BUILD-SNAPSHOT - - -## Using Gradle - - compile 'org.springframework.batch:spring-batch-integration:1.3.0.BUILD-SNAPSHOT' - -# Documentation - -## Namespace Support - -As of Spring Batch Integration 1.3, we start providing dedicated XML Namespace support, with the aim to provide an easier configuration experience. In order to activate the namespace, add the following namespace declarations to your Spring XML Application Context file: - -```xml - - ... - -``` -A fully configured Spring XML Application Context file for Spring Batch Integration may look like the following: - -```xml - - ... - -``` - -You can also append version numbers to the referenced XSD file such as: - -```xml -xsi:schemaLocation="... - http://www.springframework.org/schema/batch-integration http://www.springframework.org/schema/batch-integration/spring-batch-integration.xsd"> -``` - -However, as a version-less declaration will always use the latest schema, we generally don't recommend appending the version number to the XSD name. Adding a version number, for instance, would create possibly issues when updating the *Spring Batch Integration* dependencies as they may require more recent versions of the XML schema. - -## Launching Batch Jobs through Messages - -When starting Batch Jobs using the core Spring Batch API you basically have 2 options: - -* Command line via the [CommandLineJobRunner](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/launch/support/CommandLineJobRunner.html) -* Programatically via either *[JobOperator](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/launch/JobOperator.html).start()* or *[JobLauncher](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/launch/JobLauncher.html).run()*. - -> For more information please also see the [Spring Batch][] reference documentation: [Running a Job](http://docs.spring.io/spring-batch/reference/html/configureJob.html#runningAJob) - -For example, you may want to use the *CommandLineJobRunner* when invoking Batch Jobs using a shell script. Alternatively, you may use the *JobOperator* directly, for example when using [Spring Batch][] as part of a web application. However, what about more complex use-cases? Maybe you need to poll a remote (S)FTP server to retrieve the data for the Batch Job. Or your application has to support multiple different data sources simultaneously. For example, you may receive data files not only via the web, but also FTP etc. Maybe additional transformation of the input files is needed before invoking [Spring Batch[]. - -Therefore, it would be much more powerful to execute the batch job using [Spring Integration][] and its numerous adapters. For example, you can use a *File Inbound Channel Adapter* to monitor a directory in the file-system and start the Batch Job as soon as the input file arrives. Additionally you can create [Spring Integration][] flows that use multiple different adapters to easily ingest data for your Batch Jobs from multiple sources simultaneously using configuration only. Implementing all these scenarios with [Spring Integration][] is easy as it allow for an decoupled event-driven execution of the *JobLauncher*. - -*Spring Batch Integration* provides the *[JobLaunchingMessageHandler](http://docs.spring.io/spring-batch-admin/apidocs/org/springframework/batch/integration/launch/JobLaunchingMessageHandler.html)* that you can use to launch batch jobs. The input for the *JobLaunchingMessageHandler* is provided by a [Spring Integration][] message, which payload is of type *[JobLaunchRequest](http://docs.spring.io/spring-batch-admin/apidocs/org/springframework/batch/integration/launch/JobLaunchRequest.html)*. This class is a wrapper around the Job that needs to be launched as well as the [JobParameters](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/JobParameters.html) necessary to launch the Batch job. - -The following image illustrates the typical [Spring Integration][] message flow in order to start a Batch job. - -![Alt text](src/reference/images/launch-batch-job.png "Launch Batch Job") - -### Transforming a file into a JobLaunchRequest - -```java -public class FileMessageToJobRequest { - private Job job; - private String fileParameterName; - - public void setFileParameterName(String fileParameterName) { - this.fileParameterName = fileParameterName; - } - public void setJob(Job job) { - this.job = job; - } - - @Transformer - public JobLaunchRequest toRequest(Message message) { - JobParametersBuilder jobParametersBuilder = new JobParametersBuilder(); - jobParametersBuilder.addString(fileParameterName, message.getPayload().getAbsolutePath()); - return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters()); - } -} -``` - -### The JobExecution Response - -When a Batch Job is being executed, a [JobExecution](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/JobExecution.html) instance is returned. This instance can be used to determine the status of an execution. If a *JobExecution* was able to be created successfully, it will always be returned, regardless of whether or not the actual execution was successful. - -The exact behavior on how the *JobExecution* instance is returned depends on the provided *TaskExecutor*. If a **synchronous** (single-threaded) *TaskExecutor* implementation is used, the *JobExecution* response is only returned *after* the job completes. When using an **asynchronous** *TaskExecutor*, the *JobExecution* instance is returned immediately. Users can then take the *id* of *JobExecution* instance (*JobExecution#getJobId()*) and query the *JobRepository* for the job's updated status using the [JobExplorer](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/explore/JobExplorer.html). For more information, please refer to the *Spring Batch* reference documentation on [Querying the Repository](http://docs.spring.io/spring-batch/reference/html/configureJob.html#queryingRepository). - -For more information regarding the returned *JobExecution* instance, please see: - -* http://docs.spring.io/spring-batch/reference/html/domain.html#domainJobExecution - -### Java Bean Configuration - -```xml - - - - - - - - - - - - - - - - - - -``` - -### XML Namespace Support - -As of *Spring Batch Integration 1.3*, dedicated XML namespace for launching *[Spring Batch][]* jobs is available. This allows you to rewrite the above *Service Activator* with: - -```xml - -``` - -#### Available Attributes of the Job-Launching Gateway - -**id** -Identifies the underlying Spring bean definition, which is an instance of either: - -* [EventDrivenConsumer](http://docs.spring.io/spring-integration/api/org/springframework/integration/endpoint/EventDrivenConsumer.html) or -* [PollingConsumer](http://docs.spring.io/spring-integration/api/org/springframework/integration/endpoint/PollingConsumer.html) - -The exact implementation depends on whether the component's input channel is a - -* [SubscribableChannel](http://docs.spring.io/spring-integration/api/org/springframework/integration/core/SubscribableChannel.html) or -* [PollableChannel](http://docs.spring.io/spring-integration/api/org/springframework/integration/core/PollableChannel.html) - -**auto-startup** -Boolean flag to indicate that the endpoint should start automatically on startup. The default is *true*. - -**request-channel** -The input [MessageChannel](http://docs.spring.io/spring-integration/api/org/springframework/integration/MessageChannel.html) of this endpoint. - -**reply-channel** -*Message Channel* to which the resulting *[JobExecution](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/JobExecution.html)* payload will be sent. - -**reply-timeout** -Allows you to specify how long this gateway will wait for the reply message to be sent successfully to the reply channel before throwing an exception. This attribute only applies when the channel might block, for example when using a bounded queue channel that is currently full. - -Also, keep in mind that when sending to a [DirectChannel](http://docs.spring.io/spring-integration/api/org/springframework/integration/channel/DirectChannel.html), the invocation will occur in the sender's thread. Therefore, the failing of the send operation may be caused by other components further downstream. - -The *reply-timeout* attribute maps to the *sendTimeout* property of the underlying [MessagingTemplate](http://docs.spring.io/spring-integration/api/org/springframework/integration/core/MessagingTemplate.html) instance (org.springframework.integration.core.MessagingTemplate). The attribute will default, if not specified, to *-1*, meaning that by default, the Gateway will wait indefinitely. The value is specified in milliseconds. - -**job-launcher** -Pass in a custom *[JobLauncher](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/launch/JobLauncher.html)* bean reference. This attribute is optional. If not specified the adapter will re-use the instance that is registered under the id *jobLauncher* (E.g. when using the [@EnableBatchProcessing](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/configuration/annotation/EnableBatchProcessing.html) annotation via JavaConfig). If no default instance exists an exception is thrown. - -**order** -Specifies the order for invocation when this endpoint is connected as a subscriber to a [SubscribableChannel](http://docs.spring.io/spring-integration/api/org/springframework/integration/core/SubscribableChannel.html). - -#### Sub-Elements - -When this Gateway is receiving messages from a [PollableChannel](http://docs.spring.io/spring-integration/api/org/springframework/integration/core/PollableChannel.html), you must either provide a global default Poller or provide a Poller sub-element to the -*Job-Launching Gateway*: - -```xml - - - -``` - -## Providing Feedback with Informational Messages - -As [Spring Batch][] jobs can run for long times, providing progress information will be critical. For example, stake-holders may want to be notified if a Batch Job has failed. [Spring Batch][] provides support for this information being gathered through: - -* Active polling or -* Event-driven, using listeners. - -When starting a *Spring Batch* job asynchronously, e.g. by using the *Job-Launching Gateway*, a [JobExecution](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/JobExecution.html) instance is returned. Thus, *JobExecution#getJobId* can be used to continuously poll for status updates by retrieving updated instances of the *JobExecution* from the *JobRepository* using the *JobExplorer*. However, this is considered sub-optimal and an event-driven approach should be preferred. - -Therefore, [Spring Batch][] provides listeners such as: - -* [StepListener](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/StepListener.html) -* [ChunkListener](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/ChunkListener.html) -* [JobExecutionListener](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/JobExecutionListener.html) - -In the following example, a *Spring Batch* job was configured with *JobExecutionListener*. Thus, *Spring Integration* will receive and process any Batch job status changes. For example, the received *JobExecution* can be inspected using a *Router*. Job that are deemed recoverable may cause a restart of a Spring Batch job while a successful job completion may be routed to an Mail Outbound Channel Adapter, so that Email notification are sent out. - -![Alt text](src/reference/images/handling-informational-messages.png "Handling Information Messages") - -Here is a brief example how a listener is configured: - -```xml - - ... - - - - - - -``` - -## Externalizing Batch Process Execution - -The two integration approaches discussed so far suggest use-cases where [Spring Integration][] wraps [Spring Batch][] like an outer-shell. However, [Spring Batch][] can also use [Spring Integration][] internally. Using this approach, [Spring Batch][] users can delegate the processing of items or even chunks to outside processes. This allows you to offload complex processing. *Spring Batch Integration* provides dedicated support for: - -* Remote Chunking -* Remote Partitioning - -### Asynchronous Processors - -Asynchronous Processors help you to to scale the processing of items. In the asynchronous processor use-case, an *ItemProcessor* serves as a dispatcher, executing the *ItemProcessor*'s logic for an item on a new thread. The *Future* is passed to the *AsynchItemWriter* to be written once the processor completes. - -* [AsyncItemWriter](http://docs.spring.io/spring-batch/spring-batch-integration/apidocs/org/springframework/batch/integration/async/AsyncItemWriter.html) -* [AsyncItemProcessor](http://docs.spring.io/spring-batch/spring-batch-integration/apidocs/org/springframework/batch/integration/async/AsyncItemProcessor.html) - -Therefore, you can increase performance by using asynchronous item processing, basically allowing you to implement *fork-join* scenarios. The *AsyncItemWriter* will gather the results and write back the chunk as soon as all the results become available. - -### Remote Chunking - -![Alt text](src/reference/images/remote-chunking.png "Remote Chunking") - -Taking things one step further, one can also externalize the chunk processing using the *[ChunkMessageChannelItemWriter](http://docs.spring.io/spring-batch/spring-batch-integration/apidocs/org/springframework/batch/integration/chunk/ChunkMessageChannelItemWriter.html)* which is provided by *Spring Batch Integration*. That way you can send an entire chunk to a Gateway, reducing *Spring Batch*'s responsibilities to read items, group them and send out the chunk. Once sent, *Spring Batch* will continue the process of reading and grouping items, without waiting for the results. Rather it is the responsibility of the *ChunkMessageChannelItemWriter* to gather the results and integrate them back into the [Spring Batch][] process. - -Using *Spring Integration* you have full control over the concurrency of your processes, for instance by using *QueueChannel*s instead of *DirectChannels*. Furthermore, by relying on *Spring Integration*'s rich collection of Channel Adapters (E.g. JMS or AMQP), you can distribute chunks of a Batch job to external systems for processing. - -For more information, please also consult the *Spring Batch* manual, specifically the chapter on [Remote Chunking](http://docs.spring.io/spring-batch/reference/html/scalability.html#remoteChunking). - -### Remote Partitioning - -![Alt text](src/reference/images/remote-partitioning.png "Remote Chunking") - -Remote Partitioning, on the other hand, is useful when the problem is not the processing of items, but the associated I/O represents the bottleneck. Using Remote Partitioning, work can be farmed out to slaves that execute complete [Spring Batch][] steps. Thus, each slave has its own *reader*, *processor* and *writer*. For this purpose, *Spring Batch Integration* provides the [MessageChannelPartitionHandler](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/integration/partition/MessageChannelPartitionHandler.html). - -This implementation of the [PartitionHandler](http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/partition/PartitionHandler.html) interface uses *MessageChannel* instances to send instructions to remote workers and receive their responses. This provides a nice abstraction from the transports (E.g. JMS or AMQP) being used to communicate with the remote workers. - -For more information, please see: - -* http://docs.spring.io/spring-batch/reference/html/scalability.html#partitioning - -# Resources - -## Further Reading - -### Books - -The book [Spring Integration in Action][] by Mark Fisher, Jonas Partner, Marius Bogoevici and Iwein Fuld provides a dedicated on how to use [Spring Integration][] and [Spring Batch][] together. - -[Pro Spring Integration][] by Dr. Mark Lui, Mario Gray, Andy Chan and Josh Long has a chapter titled *Spring Integration and Spring Batch*. - -[Pro Spring Batch][] by Michael Minella, provides detailed information on *Spring Batch Integration* in Chapter 11, *Scaling and Tuning*. - -### Presentations - -* Introduction to Spring Integration and Spring Batch (Video) - - http://www.infoq.com/presentations/Spring-Integration-Batch -* Introduction to Spring Integration and Spring Batch (Slides) - - http://www.slideshare.net/hillert/introduction-to-spring-integration-and-spring-batch -* Heavy Lifting in the Cloud with Spring Batch - - https://github.com/mminella/Spring-Batch-Talk-2.0 - Slides and code - - http://www.youtube.com/watch?v=CYTj5YT7CZU - Video -* Deploying and Monitoring Spring Integration and Spring Batch Applications - - http://www.slideshare.net/DaveSyer/syer-monitoring-integration-and-batch - -### Articles - -* Spring Batch integration module for GridGain - - http://aloiscochard.blogspot.com/2010/04/spring-batch-integration-module-for.html - -## Related GitHub projects - -### Spring Batch - -* [Spring Batch][] -* [Spring Batch Admin][] - -### Spring Integration - -* [Spring Integration][] -* [Spring Integration Samples][] -* [Spring Integration Templates][] -* [Spring Integration Dsl Groovy][] -* [Spring Integration Dsl Scala][] -* [Spring Integration Pattern Catalog][] - -### Spring XD - -* [Spring XD][] - -### Spring for Apache Hadoop - -* [Spring for Apache Hadoop][] - -## Getting support - -Check out the [Spring forums][] and the [spring-integration][spring-integration tag] and [spring-batch][spring-batch tag] tags on [Stack Overflow][]. [Commercial support][] is available too. - -## Issue Tracking - -Report issues via the [Spring Batch Admin JIRA][]. - -## Building from source - -*Spring Batch Integration* is built using [Maven][]. - -### Check out sources - - git clone git://github.com/SpringSource/spring-batch-admin.git - -### Go to the Spring Batch Integration project - - $ cd spring-batch-admin/spring-batch-integration - -### Compile and test, build all jars - - $ mvn clean package - -### Install project into your local Maven cache - - $ mvn install - -## Contributing - -[Pull requests][] are welcome. Please see the [contributor guidelines][] for details. Additionally, if you are contributing, we recommend following the process for Spring Integration as outlined in the [administrator guidelines][]. - -## Contributors - -* [@david_syer](https://twitter.com/david\_syer) - Dave Syer -* [@michaelminella](https://twitter.com/michaelminella) - Michael Minella -* [@ghillert](https://twitter.com/ghillert) - Gunnar Hillert - -## License - -The Spring Integration Extensions Framework is released under version 2.0 of the [Apache License][] unless noted differently for individual extension Modules, but this should be the rare exception. - -[administrator guidelines]: https://github.com/SpringSource/spring-integration/wiki/Administrator-Guidelines -[Apache License]: http://www.apache.org/licenses/LICENSE-2.0 -[Pull requests]: http://help.github.com/send-pull-requests -[Spring Batch]: http://projects.spring.io/spring-batch/ -[Spring Integration]: http://projects.spring.io/spring-integration/ -[contributor guidelines]: https://github.com/SpringSource/spring-integration/wiki/Contributor-guidelines - -[Maven]: http://maven.apache.org -[Spring Batch Admin JIRA]: https://jira.spring.io/browse/BATCHADM - -[Spring Batch]: https://github.com/spring-projects/spring-batch -[Spring Batch Admin]: https://github.com/SpringSource/spring-batch-admin - -[Spring Integration Samples]: https://github.com/SpringSource/spring-integration-samples -[Spring Integration Templates]: https://github.com/SpringSource/spring-integration-templates/tree/master/si-sts-templates -[Spring Integration Dsl Groovy]: https://github.com/SpringSource/spring-integration-dsl-groovy -[Spring Integration Dsl Scala]: https://github.com/SpringSource/spring-integration-dsl-scala -[Spring Integration Pattern Catalog]: https://github.com/SpringSource/spring-integration-pattern-catalog - -[Spring XD]: https://github.com/SpringSource/spring-xd -[Spring for Apache Hadoop]: https://github.com/SpringSource/spring-hadoop - -[Spring Integration in Action]: http://www.manning.com/fisher/ -[Pro Spring Integration]: http://www.apress.com/9781430233459 -[Pro Spring Batch]: http://www.apress.com/9781430234524 - -[Spring forums]: http://forum.spring.io -[spring-integration tag]: http://stackoverflow.com/questions/tagged/spring-integration -[spring-batch tag]: http://stackoverflow.com/questions/tagged/spring-batch - -[Stack Overflow]: http://stackoverflow.com/faq -[Commercial support]: https://www.vmware.com/support/services/vfabric-developer.html diff --git a/spring-batch-integration/src/reference/images/handling-informational-messages.png b/src/site/docbook/reference/images/handling-informational-messages.png similarity index 100% rename from spring-batch-integration/src/reference/images/handling-informational-messages.png rename to src/site/docbook/reference/images/handling-informational-messages.png diff --git a/spring-batch-integration/src/reference/images/launch-batch-job-svg.svg b/src/site/docbook/reference/images/launch-batch-job-svg.svg similarity index 100% rename from spring-batch-integration/src/reference/images/launch-batch-job-svg.svg rename to src/site/docbook/reference/images/launch-batch-job-svg.svg diff --git a/spring-batch-integration/src/reference/images/launch-batch-job.png b/src/site/docbook/reference/images/launch-batch-job.png similarity index 100% rename from spring-batch-integration/src/reference/images/launch-batch-job.png rename to src/site/docbook/reference/images/launch-batch-job.png diff --git a/spring-batch-integration/src/reference/images/remote-chunking.png b/src/site/docbook/reference/images/remote-chunking-sbi.png similarity index 100% rename from spring-batch-integration/src/reference/images/remote-chunking.png rename to src/site/docbook/reference/images/remote-chunking-sbi.png diff --git a/spring-batch-integration/src/reference/images/remote-partitioning.png b/src/site/docbook/reference/images/remote-partitioning.png similarity index 100% rename from spring-batch-integration/src/reference/images/remote-partitioning.png rename to src/site/docbook/reference/images/remote-partitioning.png diff --git a/src/site/docbook/reference/scalability.xml b/src/site/docbook/reference/scalability.xml index b2eea9a16..6cbc734b0 100644 --- a/src/site/docbook/reference/scalability.xml +++ b/src/site/docbook/reference/scalability.xml @@ -182,11 +182,6 @@ single consumer for each message. JMS is the obvious candidate, but other options exist in the grid computing and shared memory product space (e.g. Java Spaces). - - Spring Batch has a sister project Spring Batch Admin, which - provides(amongst other things) implementations of various patterns - like this one using Spring Integration. These are implemented in a - module called Spring Batch Integration.
diff --git a/src/site/docbook/reference/spring-batch-integration.xml b/src/site/docbook/reference/spring-batch-integration.xml index 0d9dacbcb..ca4e56e8b 100644 --- a/src/site/docbook/reference/spring-batch-integration.xml +++ b/src/site/docbook/reference/spring-batch-integration.xml @@ -4,430 +4,929 @@ Spring Batch Integration -
- Overview of the Spring Integration Batch Module - Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring - Integration. Here is a list. These are features that can extend Spring Batch, or use Spring batch features - in the context of Spring Integration. Work in progress waiting for community feedback. Many issues to do - with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result - of these use cases being prototyped. + + Spring Batch Integration Introduction + + Many users of Spring Batch may encounter requirements that are + outside the scope of Spring Batch, yet may be efficiently and + concisely implemented using Spring Integration. Conversely, Spring + Batch users may encounter Spring Batch requirements and need a way + to efficiently integrate both frameworks. In this context several + patterns and use-cases emerge and Spring Batch Integration will + address those requirements. - - - - - - - - - - - ID - - - Description - - - Status - - - Sub-package - - - Comments - - - - 1 - - Message triggers job - - Complete - launch - Complete. Also lots of opportunities with monitoring progress. - - - 2 - - Chunking and multi-VM job execution - - Complete - chunk - Failures might need some analysis. Use of stateful StepExecutionListener requires use of - step scope. - - - - 3 - - Asynchronous Aggregator - - Unstarted - - - - - 4 - - Stateful and non-linear jobs - -> job = flow - - Complete - job - Simple use cases work well with Spring Batch 2.0 and no Integration features. - - - 5 - - Flexible item processing model - (as message flow) -> step = flow - - Complete - item - Complete (v. simple using MessagingGateway). Unit tests only. - - - 6 - - Automatic repeat / retry - - Complete - retry (unit test) - Unit tests only, since it just uses existing features. - - - 7 - - Restartable file processing - - Complete - file - Seems to hang together. Not tested thoroughly, but apparently someone is using it. - - - - 8 - - Asynchronous item processing - - Complete - async - A general purpose ItemProcesor that returns a Future. - - - - - Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes. If we - implement 1, then we also don't need to do any more scheduling and triggering in Spring Batch. + + The line between Spring Batch and Spring Integration is not always + clear, but there are guidelines that one can follow. Principally, + these are: think about granularity, and apply common patterns. Some + of those common patterns are described in this reference manual + section. - Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one. We - implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration - later (with probably a split of repeat/retry out of Batch at that time). + + Adding messaging to a batch process enables automation of + operations, and also separation and strategizing of key concerns. + For example a message might trigger a job to execute, and then the + sending of the message can be exposed in a variety of ways. Or when + a job completes or fails that might trigger a message to be sent, + and the consumers of those messages might have operational concerns + that have nothing to do with the application itself. Messaging can + also be embedded in a job, for example reading or writing items for + processing via channels. Remote partitioning and remote chunking + provide methods to distribute workloads over an number of workers. -
- Message<anchor id="Triggers"/>Triggers<!-- anchor_end --> Job - - Description: - + + Some key concepts that we will cover are: + + + + + Namespace Support + + + + + Launching + Batch Jobs through Messages + + + + + Providing + Feedback with Informational Messages + + + + + Asynchronous + Processors + + + + + Externalizing + Batch Process Execution + + + + + Namespace Support + + Since Spring Batch Integration 1.3, dedicated XML Namespace + support was added, with the aim to provide an easier configuration + experience. In order to activate the namespace, add the following + namespace declarations to your Spring XML Application Context + file: + + +<beans xmlns="http://www.springframework.org/schema/beans" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xmlns:batch-int="http://www.springframework.org/schema/batch-integration" + xsi:schemaLocation=" + http://www.springframework.org/schema/batch-integration + http://www.springframework.org/schema/batch-integration/spring-batch-integration.xsd"> + + ... + +</beans> + + + A fully configured Spring XML Application Context file for Spring + Batch Integration may look like the following: + + +<beans xmlns="http://www.springframework.org/schema/beans" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xmlns:int="http://www.springframework.org/schema/integration" + xmlns:batch="http://www.springframework.org/schema/batch" + xmlns:batch-int="http://www.springframework.org/schema/batch-integration" + xsi:schemaLocation=" + http://www.springframework.org/schema/batch-integration + http://www.springframework.org/schema/batch-integration/spring-batch-integration.xsd + http://www.springframework.org/schema/batch + http://www.springframework.org/schema/batch/spring-batch.xsd + http://www.springframework.org/schema/beans + http://www.springframework.org/schema/beans/spring-beans.xsd + http://www.springframework.org/schema/integration + http://www.springframework.org/schema/integration/spring-integration.xsd"> + + ... + +</beans> + + + Appending version numbers to the referenced XSD file is also + allowed but, as a version-less declaration will always use the + latest schema, we generally don't recommend appending the version + number to the XSD name. Adding a version number, for instance, + would create possibly issues when updating the Spring Batch + Integration dependencies as they may require more recent versions + of the XML schema. + + + + Launching Batch Jobs through Messages + + When starting batch jobs using the core Spring Batch API you + basically have 2 options: + + - User sends message to channel (maybe through a scheduler) - - - System interprets message payload as parameters for JobLauncher - - - System launches job execution - - - If message had a replyTo, System acknowledges with JobExecution - - - User accepts response and uses it to monitor progress - - - Variation: - - - System waits for job to finish and replies when it is over - - - User polls for replies and gets notification about end of execution - - - Variation: - - - User wants to block on send and only receive response when job is done - - -
-
- <anchor id="Chunking"/>Chunking<!-- anchor_end --> and Multi-VM - - Description: - - - Step flushes chunk as message to outgoing channel (repeat up to throttle limit) - - - Worker thread picks up chunk and processes it - - - Worker thread replies to response channel - - - Step picks up reply and aggregates the counts - - - Step blocks until all the requests are satisfied - - - TODO: failure modes -
-
- Asynchronous<anchor id="Aggregator"/>Aggregator<!-- anchor_end --> - - Job is executed over long period. Many jobs can be executing concurrently. - Description: - - - Input stage for each job: System reads all items and marks with the job instance id in a - durable repository (staging table) + + Command line via the CommandLineJobRunner - System sends each item (or chunks of items that can be processed together as appropriate) to a - channel + + Programatically via either + JobOperator.start() or + JobLauncher.run(). + + + + + For example, you may want to use the + CommandLineJobRunner when invoking Batch Jobs + using a shell script. Alternatively, you may use the + JobOperator directly, for example when using + Spring Batch as part of a web application. However, what about + more complex use-cases? Maybe you need to poll a remote (S)FTP + server to retrieve the data for the Batch Job. Or your application + has to support multiple different data sources simultaneously. For + example, you may receive data files not only via the web, but also + FTP etc. Maybe additional transformation of the input files is + needed before invoking Spring Batch. + + + Therefore, it would be much more powerful to execute the batch job + using Spring Integration and its numerous adapters. For example, + you can use a File Inbound Channel Adapter to + monitor a directory in the file-system and start the Batch Job as + soon as the input file arrives. Additionally you can create Spring + Integration flows that use multiple different adapters to easily + ingest data for your Batch Jobs from multiple sources + simultaneously using configuration only. Implementing all these + scenarios with Spring Integration is easy as it allow for an + decoupled event-driven execution of the + JobLauncher. + + + Spring Batch Integration provides the + JobLaunchingMessageHandler class that you can + use to launch batch jobs. The input for the + JobLaunchingMessageHandler is provided by a + Spring Integration message, which payload is of type + JobLaunchRequest. This class is a wrapper around the Job + that needs to be launched as well as the JobParameters + necessary to launch the Batch job. + + + The following image illustrates the typical Spring Integration + message flow in order to start a Batch job. The + EIP (Enterprise IntegrationPatterns) website + provides a full overview of messaging icons and their descriptions. + + + + + + + + + + + + + Transforming a file into a JobLaunchRequest + +package io.spring.sbi; + +import org.springframework.batch.core.Job; +import org.springframework.batch.core.JobParametersBuilder; +import org.springframework.batch.integration.launch.JobLaunchRequest; +import org.springframework.integration.annotation.Transformer; +import org.springframework.messaging.Message; + +import java.io.File; + +public class FileMessageToJobRequest { + private Job job; + private String fileParameterName; + + public void setFileParameterName(String fileParameterName) { + this.fileParameterName = fileParameterName; + } + + public void setJob(Job job) { + this.job = job; + } + + @Transformer + public JobLaunchRequest toRequest(Message<File> message) { + JobParametersBuilder jobParametersBuilder = + new JobParametersBuilder(); + + jobParametersBuilder.addString(fileParameterName, + message.getPayload().getAbsolutePath()); + + return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters()); + } +} + + + + The JobExecution Response + + When a Batch Job is being executed, a + JobExecution instance is returned. This + instance can be used to determine the status of an execution. If + a JobExecution was able to be created + successfully, it will always be returned, regardless of whether + or not the actual execution was successful. + + + The exact behavior on how the JobExecution + instance is returned depends on the provided + TaskExecutor. If a + synchronous (single-threaded) + TaskExecutor implementation is used, the + JobExecution response is only returned + after the job completes. When using an + asynchronous + TaskExecutor, the + JobExecution instance is returned + immediately. Users can then take the id of + JobExecution instance + (JobExecution.getJobId()) and query the + JobRepository for the job's updated status + using the JobExplorer. For more + information, please refer to the Spring + Batch reference documentation on + Querying + the Repository. + + + The following configuration will create a file + inbound-channel-adapter to listen for CSV + files in the provided directory, hand them off to our + transformer (FileMessageToJobRequest), + launch the job via the Job Launching + Gateway then simply log the output of the + JobExecution via the + logging-channel-adapter. + + + + Spring Batch Integration Configuration + +<int:channel id="inboundFileChannel"/> +<int:channel id="outboundJobRequestChannel"/> +<int:channel id="jobLaunchReplyChannel"/> + +<int-file:inbound-channel-adapter id="filePoller" + channel="inboundFileChannel" + directory="file:/tmp/myfiles/" + filename-pattern="*.csv"> + <int:poller fixed-rate="1000"/> +</int-file:inbound-channel-adapter> + +<int:transformer input-channel="inboundFileChannel" + output-channel="outboundJobRequestChannel"> + <bean class="io.spring.sbi.FileMessageToJobRequest"> + <property name="job" ref="personJob"/> + <property name="fileParameterName" value="input.file.name"/> + </bean> +</int:transformer> + +<batch-int:job-launching-gateway request-channel="outboundJobRequestChannel" + reply-channel="jobLaunchReplyChannel"/> + +<int:logging-channel-adapter channel="jobLaunchReplyChannel"/> + + + Now that we are polling for files and launching jobs, we need to + configure for example our Spring Batch + ItemReader to utilize found file + represented by the job parameter "input.file.name": + + + + Example ItemReader Configuration + +<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" + scope="step"> + <property name="resource" value="file://#{jobParameters['input.file.name']}"/> + ... +</bean> + + + The main points of interest here are injecting the value of + #{jobParameters['input.file.name']} + as the Resource property value and setting the ItemReader bean + to be of Step scope to take advantage of + the late binding support which allows access to the + jobParameters variable. + + + Available Attributes of the Job-Launching Gateway + + + + id Identifies the + underlying Spring bean definition, which is an instance of + either: + + + + + EventDrivenConsumer + + + + + PollingConsumer + + + + + The exact implementation depends on whether the component's + input channel is a: + + + + + SubscribableChannel or + + + + + PollableChannel + + + + + + + + + auto-startup + Boolean flag to indicate that the endpoint should start automatically on + startup. The default istrue. + + + + + request-channel + The input MessageChannel of this endpoint. + + + + + reply-channel Message Channel + to which the resulting JobExecution payload will be sent. + + + + + reply-timeout + Allows you to specify how long this gateway will wait for the reply message + to be sent successfully to the reply channel before throwing + an exception. This attribute only applies when the channel + might block, for example when using a bounded queue channel + that is currently full. Also, keep in mind that when sending to a + DirectChannel, the invocation will occur + in the sender's thread. Therefore, the failing of the send + operation may be caused by other components further downstream. + The reply-timeout attribute maps to the + sendTimeout property of the underlying + MessagingTemplate instance. The attribute + will default, if not specified, to-1, + meaning that by default, the Gateway will wait indefinitely. + The value is specified in milliseconds. + + + + + job-launcher + Pass in a + custom + JobLauncher + bean reference. This + attribute is optional. If not specified the adapter will + re-use the instance that is registered under the id + jobLauncher. If no default instance + exists an exception is thrown. + + + + + order + Specifies the order + for invocation when this endpoint is connected as a subscriber + to a SubscribableChannel. + + + + + + Sub-Elements + + When this Gateway is receiving messages from a + PollableChannel, you must either provide + a global default Poller or provide a Poller sub-element to the + Job Launching Gateway: + + +<batch-int:job-launching-gateway request-channel="queueChannel" + reply-channel="replyChannel" job-launcher="jobLauncher"> + <int:poller fixed-rate="1000"/> +</batch-int:job-launching-gateway> + + + + + + Providing Feedback with Informational Messages + + As Spring Batch jobs can run for long times, providing progress + information will be critical. For example, stake-holders may want + to be notified if a some or all parts of a Batch Job has failed. + Spring Batch provides support for this information being gathered + through: + + + + + Active polling or - Items flow through message pipeline, occasionally pausing until certain conditions are met, - possibly for days at a time + + Event-driven, using listeners. + + + + + When starting a Spring Batch job asynchronously, e.g. by using the + Job Launching Gateway, a + JobExecution instance is returned. Thus, + JobExecution.getJobId() can be used to + continuously poll for status updates by retrieving updated + instances of the JobExecution from the + JobRepository using the + JobExplorer. However, this is considered + sub-optimal and an event-driven approach should be preferred. + + + Therefore, Spring Batch provides listeners such as: + + + + + StepListener - Aggregator sits and waits for all items in a job to be finished and then wraps up - - -
-
- Stateful and non-linear<anchor id="jobs"/>jobs<!-- anchor_end --> - - Dependencies beyween steps and conditional flow between steps. Each handler node in a message flow is - a step execution, with all the robustness guarantees from the Spring Batch meta data. - - Description: - - - User launches job - - - System sends message to channel containing job execution - - - Handler accepts message and executes a step - - - Handler translates result of step execution into the same form that it accepted the original - request + + ChunkListener - System routes message to next handler, possibly dynamically based on data in the message + + JobExecutionListener + + + + + In the following example, a Spring Batch job was configured with a + StepExecutionListener. Thus, Spring + Integration will receive and process any step before/after step + events. For example, the received + StepExecution can be inspected using a + Router. Based on the results of that + inspection, various things can occur for example routing a message + to a Mail Outbound Channel Adapter, so that an Email notification + can be send out based on some condition. + + + + + + + + + + + Below is an example of how a listener is configured to send a + message to a Gateway for + StepExecution events and log its output to a + logging-channel-adapter: + + + First create the notifications integration beans: + + +<int:channel id="stepExecutionsChannel"/> + +<int:gateway id="notificationExecutionsListener" + service-interface="org.springframework.batch.core.StepExecutionListener" + default-request-channel="stepExecutionsChannel"/> + +<int:logging-channel-adapter channel="stepExecutionsChannel"/> + + + Then modify your job to add a step level listener: + + +<job id="importPayments"> + <step id="step1"> + <tasklet ../> + <chunk ../> + <listeners> + <listener ref="notificationExecutionsListener"/> + </listeners> + </tasklet> + ... + </step> +</job> + + + + Asynchronous Processors + + Asynchronous Processors help you to to scale the processing of + items. In the asynchronous processor use-case, an + AsyncItemProcessor serves as a dispatcher, + executing the ItemProcessor's logic for an + item on a new thread. The Future is passed to + the AsynchItemWriter to be written once the + processor completes. + + + Therefore, you can increase performance by using asynchronous item + processing, basically allowing you to implement + fork-join scenarios. The + AsyncItemWriter will gather the results and + write back the chunk as soon as all the results become available. + + + Configuration of both the AsyncItemProcessor + and AsyncItemWriter are simple, first the + AsyncItemProcessor: + + +<bean id="processor" + class="org.springframework.batch.integration.async.AsyncItemProcessor"> + <property name="delegate"> + <bean class="your.ItemProcessor"/> + </property> + <property name="taskExecutor"> + <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor"/> + </property> +</bean> + + + The property "delegate" is actually + a reference to your ItemProcessor bean and + the "taskExecutor" property is a + reference to the TaskExecutor of your choice. + + + Then we configure the AsyncItemWriter: + + +<bean id="itemWriter" + class="org.springframework.batch.integration.async.AsyncItemWriter"> + <property name="delegate"> + <bean id="itemWriter" class="your.ItemWriter"/> + </property> +</bean> + + + Again, the property "delegate" is + actually a reference to your ItemWriter bean. + + + + Externalizing Batch Process Execution + + The integration approaches discussed so far suggest use-cases + where Spring Integration wraps Spring Batch like an outer-shell. + However, Spring Batch can also use Spring Integration internally. + Using this approach, Spring Batch users can delegate the + processing of items or even chunks to outside processes. This + allows you to offload complex processing. Spring Batch Integration + provides dedicated support for: + + + + + Remote Chunking - Next handler does the same... until one of the routing decisions leads to a reply channel + + Remote Partitioning - - System receives reply and transfers information to job execution (e.g. status) as necessary - - - - Variation: failure in one of the handlers - Variation: restart after failure -
-
- <anchor id="Flexible"/>Flexible<!-- anchor_end --> item processing model - - Description: - - - Step hands item to ItemWriter - - - Item is converted to message and sent to synchronous flow - - - Handler accepts message and does something with item - - - System routes result to next handler, possibly dynamically - - - Variation: failure - - - Handler throws exception - - - System propagates exception up to ItemWriter (forces rollback under normal circs - hence - synchronous flow) - - - -
-
- Automatic repeat / retry - Description (repeat): - - - - User sends message to channel - - - System start a transaction and reseives message, then processes it - - - User sends another message - - - System receives and processes it in the same transaction - - - ... repeat ... - - - System determines that batch is complete and commits transaction - - -
-
- Restartable file processing - Largefiles need to be processed, so message payload of file - contents is not practical. One line or XML event per message with failover and restartability from - Spring Batch. - - Description: - - - User triggers file processing (sends message, copies file to directory, etc.) - - - System starts new job - - - System processes file line by line (or even by event), wrapping each one as a message and - sending it to a synchronous flow - - - - System commits periodically (as determined by Spring Batch step configuration) - - - Variation: failure and restart - - - Item processing fails - - - System aborts job and sends message to failure channel (or failure message to normal reply - channel) - - - - Operator fixes problem and triggers restart (another message channel?) - - - System restarts job for same file at point where it left off - - - System completes processing - - - System sends sucess message to reply channel - - - Variation: send to asynchronous flow. Same as main use case but item message is sent to asynchronous - flow. Not as robust because if the lights go out then meesages will be lost, but at least a large file - can be split into smaller chunks. - -
-
- Asynchronous item processing - This is actually a variation onflexible item processing model. - - Description (async): - - - - ItemProcessor executes in background (non-transactionally) - - - ItemWriter collects outputs from futures before phyically writing data - - -
-
-
- Remote Chunking Implementation -
- Basic Use Case - Description: - - - Step flushes chunk as message to outgoing channel (repeat up to throttle limit) - - - Worker thread picks up chunk and processes it - - - Worker thread replies to response channel - - - Step picks up reply if there is one and aggregates the counts - - - Step reapeats until no more input data - - - Step blocks until all the outstanding requests are satisfied - - -
-
- Implementation - A ChunkProcessor acts as a kind of Throttling Asynchronous Messaging Gateway, which isn't a - pattern that is supported out of the box with Spring Integration. There is a SimpleMessagingGateway that - provides programmatic access to send and receive payloads (instead of messages), so the pattern can be - manually implemented in the ChunkProcessor. - - The current implementation is in the form of an ItemWriter (ChunkMessageChannelItemWriter) which is a - StepExecutionListener (blocks and waits for the outstanding responses in the afterStep). The - ChunkProcessor can then simply be a vanilla implementation from Spring Batch. - - The ChunkMessageChannelItemWriter implements the Throttling part of the pattern by keeping track of - the number of outstanding requests (which it has to do anyway) and blocking until a response arrives if - the number is above a configurable limit. It wouldn't be necessary to do this manually in the - writer if the messages were only going over local MessageChannels: the requests would either be - processed serially in a single thread, or else there would be a thread pool with limited size - controlling the workers. But since the messages are going to JMS we need to either explicitly throttle - in the writer (or else rely on vendor features for producer flow control), otherwise the JMS Queue could - easily be overwhelmed and start barfing (which happened in one of the early prototypes on an Accenture - project). - - Throttling MessageChannel.send() might be something Spring Integration could do, but it only makes - sense really in the context of this gateway pattern (because you need something to react against to - decide when to release another send). - - The gateway is used to send requests to the workers, and then to receive responses in the same thread, - but only waiting for a response when the step is complete. To implement this with JMS backed channels we - need a Spring Integration inbound adapter that translates PollableChannel.receive() into - JmsTemplate.receiveAndConvert(). In the unlikely event of a problem in the receiver the JMS message - should roll back. JmsDestinationPollingAdapter actually almost does what we need but there is no support - for configuring it without a scheduled poller. - -
-
- \ No newline at end of file + + + Remote Chunking + + + + + + + + + + Taking things one step further, one can also externalize the + chunk processing using the + ChunkMessageChannelItemWriter which is + provided by Spring Batch Integration which will send items out + and collect the result. Once sent, Spring Batch will continue the + process of reading and grouping items, without waiting for the results. + Rather it is the responsibility of the ChunkMessageChannelItemWriter + to gather the results and integrate them back into the Spring Batch process. + + + Using Spring Integration you have full + control over the concurrency of your processes, for instance by + using a QueueChannel instead of a + DirectChannel. Furthermore, by relying on + Spring Integration's rich collection of Channel Adapters (E.g. + JMS or AMQP), you can distribute chunks of a Batch job to + external systems for processing. + + + A simple job with a step to be remotely chunked would have a + configuration similar to the following: + + +<job id="personJob"> + <step id="step1"> + <tasklet> + <chunk reader="itemReader" writer="itemWriter" commit-interval="200"/> + </tasklet> + ... + </step> +</job> + + + The ItemReader reference would point to the bean you would like + to use for reading data on the master. The ItemWriter reference + points to a special ItemWriter + "ChunkMessageChannelItemWriter" + as described above. The processor (if any) is left off the + master configuration as it is configured on the slave. The + following configuration provides a basic master setup. It's + advised to check any additional component properties such as + throttle limits and so on when implementing your use case. + + +<bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory"> + <property name="brokerURL" value="tcp://localhost:61616"/> +</bean> + +<int-jms:outbound-channel-adapter id="requests" destination-name="requests"/> + +<bean id="messagingTemplate" + class="org.springframework.integration.core.MessagingTemplate"> + <property name="defaultChannel" ref="requests"/> + <property name="receiveTimeout" value="2000"/> +</bean> + +<bean id="itemWriter" + class="org.springframework.batch.integration.chunk.ChunkMessageChannelItemWriter" + scope="step"> + <property name="messagingOperations" ref="messagingTemplate"/> + <property name="replyChannel" ref="replies"/> +</bean> + +<bean id="chunkHandler" + class="org.springframework.batch.integration.chunk.RemoteChunkHandlerFactoryBean"> + <property name="chunkWriter" ref="itemWriter"/> + <property name="step" ref="step1"/> +</bean> + +<int:channel id="replies"> + <int:queue/> +</int:channel> + +<int-jms:message-driven-channel-adapter id="jmsReplies" + destination-name="replies" + channel="replies"/> + + + This configuration provides us with a number of beans. We + configure our messaging middleware using ActiveMQ and + inbound/outbound JMS adapters provided by Spring Integration. As + shown, our itemWriter bean which is + referenced by our job step utilizes the + ChunkMessageChannelItemWriter for writing chunks over the + configured middleware. + + + Now lets move on to the slave configuration: + + +<bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory"> + <property name="brokerURL" value="tcp://localhost:61616"/> +</bean> + +<int:channel id="requests"/> +<int:channel id="replies"/> + +<int-jms:message-driven-channel-adapter id="jmsIn" + destination-name="requests" + channel="requests"/> + +<int-jms:outbound-channel-adapter id="outgoingReplies" + destination-name="replies" + channel="replies"> +</int-jms:outbound-channel-adapter> + +<int:service-activator id="serviceActivator" + input-channel="requests" + output-channel="replies" + ref="chunkProcessorChunkHandler" + method="handleChunk"/> + +<bean id="chunkProcessorChunkHandler" + class="org.springframework.batch.integration.chunk.ChunkProcessorChunkHandler"> + <property name="chunkProcessor"> + <bean class="org.springframework.batch.core.step.item.SimpleChunkProcessor"> + <property name="itemWriter"> + <bean class="io.spring.sbi.PersonItemWriter"/> + </property> + <property name="itemProcessor"> + <bean class="io.spring.sbi.PersonItemProcessor"/> + </property> + </bean> + </property> +</bean> + + + Most of these configuration items should look familiar from the + master configuration. Slaves do not need access to things like + the Spring Batch JobRepository nor access + to the actual job configuration file. The main bean of interest + is the + "chunkProcessorChunkHandler". The + chunkProcessor property of + ChunkProcessorChunkHandler takes a + configured SimpleChunkProcessor which is + where you would provide a reference to your + ItemWriter and optionally your + ItemProcessor that will run on the slave + when it receives chunks from the master. + + + For more information, please also consult the Spring Batch + manual, specifically the chapter on + Remote + Chunking. + + + + Remote Partitioning + + + + + + + + + + Remote Partitioning, on the other hand, is useful when the + problem is not the processing of items, but the associated I/O + represents the bottleneck. Using Remote Partitioning, work can + be farmed out to slaves that execute complete Spring Batch + steps. Thus, each slave has its own + ItemReader, + ItemProcessor and + ItemWriter. For this purpose, Spring Batch + Integration provides the + MessageChannelPartitionHandler. + + + This implementation of the PartitionHandler + interface uses MessageChannel instances to + send instructions to remote workers and receive their responses. + This provides a nice abstraction from the transports (E.g. JMS + or AMQP) being used to communicate with the remote workers. + + + The reference manual section + Remote + Partitioning provides an overview of the concepts and + components needed to configure Remote Partitioning and shows an + example of using the default + TaskExecutorPartitionHandler to partition + in separate local threads of execution. For Remote Partitioning + to multiple JVM's, two additional components are required: + + + + + Remoting fabric or grid environment + + + + + A PartitionHandler implementation that supports the desired + remoting fabric or grid environment + + + + + Similar to Remote Chunking JMS can be used as the "remoting + fabric" and the PartitionHandler implementation to be used + as described above is the + MessageChannelPartitionHandler. The example + shown below assumes an existing partitioned job and focuses on + the MessageChannelPartitionHandler and JMS + configuration: + + +<bean id="partitionHandler" + class="org.springframework.batch.integration.partition.MessageChannelPartitionHandler"> + <property name="stepName" value="step1"/> + <property name="gridSize" value="3"/> + <property name="replyChannel" ref="outbound-replies"/> + <property name="messagingOperations"> + <bean class="org.springframework.integration.core.MessagingTemplate"> + <property name="defaultChannel" ref="outbound-requests"/> + <property name="receiveTimeout" value="100000"/> + </bean> + </property> +</bean> + +<int:channel id="outbound-requests"/> +<int-jms:outbound-channel-adapter destination="requestsQueue" + channel="outbound-requests"/> + +<int:channel id="inbound-requests"/> +<int-jms:message-driven-channel-adapter destination="requestsQueue" + channel="inbound-requests"/> + +<bean id="stepExecutionRequestHandler" + class="org.springframework.batch.integration.partition.StepExecutionRequestHandler"> + <property name="jobExplorer" ref="jobExplorer"/> + <property name="stepLocator" ref="stepLocator"/> +</bean> + +<int:service-activator ref="stepExecutionRequestHandler" input-channel="inbound-requests" + output-channel="outbound-staging"/> + +<int:channel id="outbound-staging"/> +<int-jms:outbound-channel-adapter destination="stagingQueue" + channel="outbound-staging"/> + +<int:channel id="inbound-staging"/> +<int-jms:message-driven-channel-adapter destination="stagingQueue" + channel="inbound-staging"/> + +<int:aggregator ref="partitionHandler" input-channel="inbound-staging" + output-channel="outbound-replies"/> + +<int:channel id="outbound-replies"> + <int:queue/> +</int:channel> + +<bean id="stepLocator" + class="org.springframework.batch.integration.partition.BeanFactoryStepLocator" /> + + + Also ensure the partition handler attribute + maps to the partitionHandler bean: + + +<job id="personJob"> + <step id="step1.master"> + <partition partitioner="partitioner" handler="partitionHandler"/> + ... + </step> +</job> + + + + +