Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add initial 3.0 documentation

Browse Source 
* What's new
  * JSR-352
  * Convert Spring Batch Integration apt files into docbook and use as chapter
  * Remove site dir from Spring Batch Integration where converted apt files used
    to live
This commit is contained in:
Chris Schaefer
2014-03-07 13:21:06 -05:00
committed by Michael Minella
parent de7df9c1f1
commit 1ad8bd803a
10 changed files with 855 additions and 378 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
spring-batch-integration/src/site/apt/chunking.apt
View File

@@ -1,68 +0,0 @@
------
Remote Chunking
------
Dave Syer
------
March 2008
Remote Chunking Implementation
* Basic Use Case
Description:
[[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit)
[[1]] Worker thread picks up chunk and processes it
[[1]] Worker thread replies to response channel
[[1]] Step picks up reply if there is one and aggregates the counts
[[1]] Step reapeats until no more input data
[[1]] Step blocks until all the outstanding requests are satisfied
* Implementation
A ChunkProcessor acts as a kind of Throttling Asynchronous Messaging
Gateway, which isn't a pattern that is supported out of the box with
Spring Integration. There is a SimpleMessagingGateway that provides
programmatic access to send and receive payloads (instead of
messages), so the pattern can be manually implemented in the
ChunkProcessor.
The current implementation is in the form of an ItemWriter
(ChunkMessageChannelItemWriter) which is a StepExecutionListener
(blocks and waits for the outstanding responses in the afterStep).
The ChunkProcessor can then simply be a vanilla implementation from
Spring Batch.
The ChunkMessageChannelItemWriter implements the Throttling part of
the pattern by keeping track of the number of outstanding requests
(which it has to do anyway) and blocking until a response arrives if
the number is above a configurable limit. It wouldn't be necessary
to do this manually in the writer if the messages were only going
over local MessageChannels: the requests would either be processed
serially in a single thread, or else there would be a thread pool
with limited size controlling the workers. But since the messages
are going to JMS we need to either explicitly throttle in the writer
(or else rely on vendor features for producer flow control),
otherwise the JMS Queue could easily be overwhelmed and start
barfing (which happened in one of the early prototypes on an
Accenture project).
Throttling MessageChannel.send() might be something Spring
Integration could do, but it only makes sense really in the context
of this gateway pattern (because you need something to react against
to decide when to release another send).
The gateway is used to send requests to the workers, and then to
receive responses in the same thread, but only waiting for a
response when the step is complete. To implement this with JMS
backed channels we need a Spring Integration inbound adapter that
translates PollableChannel.receive() into
JmsTemplate.receiveAndConvert(). In the unlikely event of a problem
in the receiver the JMS message should roll back.
JmsDestinationPollingAdapter actually almost does what we need but
there is no support for configuring it without a scheduled poller.

187
spring-batch-integration/src/site/apt/index.apt
View File

@@ -1,187 +0,0 @@
------
Spring Integration Batch
------
Dave Syer
------
March 2008
Overview of the Spring Integration Batch Module
Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring Integration. Here is a list. These are features that can extend Spring Batch, or use Spring batch features in the context of Spring Integration. Work in progress waiting for community feedback. Many issues to do with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result of these use cases being prototyped.
*---+---+---+---+---+
|<<ID>>|<<Description>>|<<Status>>|<<Sub-package>>|<<Comments>>|
*----
|1|{{{Triggers}Message triggers job}}|Complete|launch|Complete. Also lots of opportunities with monitoring progress.|
*----
|2|{{{Chunking}Chunking and multi-VM job execution}}|Complete|chunk|Failures might need some analysis. Use of stateful StepExecutionListener requires use of step scope.|
*----
|3|{{{Aggregator}Asynchronous Aggregator}}|Unstarted| | |
*----
|4|{{{jobs}Stateful and non-linear jobs}} -> job = flow|Complete|job|Simple use cases work well with Spring Batch 2.0 and no Integration features.|
*----
|5|{{{Flexible}Flexible item processing model}} (as message flow) -> step = flow|Complete|item|Complete (v. simple using MessagingGateway). Unit tests only.|
*----
|6|{{{repeat}Automatic repeat / retry}}|Complete|retry (unit test)|Unit tests only, since it just uses existing features.|
*----
|7|{{{files}Restartable file processing}}|Complete|file|Seems to hang together. Not tested thoroughly, but apparently someone is using it.|
*----
|8|{{{async}Asynchronous item processing}}|Complete|async|A general purpose ItemProcesor that returns a Future.|
*----
Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes. If we implement 1, then we also don't need to do any more scheduling and triggering in Spring Batch.
Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one. We implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration later (with probably a split of repeat/retry out of Batch at that time).
* Message {Triggers} Job
Description:
[[1]] User sends message to channel (maybe through a scheduler)
[[1]] System interprets message payload as parameters for JobLauncher
[[1]] System launches job execution
[[1]] If message had a replyTo, System acknowledges with JobExecution
[[1]] User accepts response and uses it to monitor progress
Variation:
[[1]] System waits for job to finish and replies when it is over
[[1]] User polls for replies and gets notification about end of execution
Variation:
[[1]] User wants to block on send and only receive response when job is done
* {Chunking} and Multi-VM
Description:
[[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit)
[[1]] Worker thread picks up chunk and processes it
[[1]] Worker thread replies to response channel
[[1]] Step picks up reply and aggregates the counts
[[1]] Step blocks until all the requests are satisfied
TODO: failure modes
* Asynchronous {Aggregator}
Job is executed over long period. Many jobs can be executing concurrently.
Description:
[[1]] Input stage for each job: System reads all items and marks with the job instance id in a durable repository (staging table)
[[1]] System sends each item (or chunks of items that can be processed together as appropriate) to a channel
[[1]] Items flow through message pipeline, occasionally pausing until certain conditions are met, possibly for days at a time
[[1]] Aggregator sits and waits for all items in a job to be finished and then wraps up
* Stateful and non-linear {jobs}
Dependencies beyween steps and conditional flow between steps. Each handler node in a message flow is a step execution, with all the robustness guarantees from the Spring Batch meta data.
Description:
[[1]] User launches job
[[1]] System sends message to channel containing job execution
[[1]] Handler accepts message and executes a step
[[1]] Handler translates result of step execution into the same form that it accepted the original request
[[1]] System routes message to next handler, possibly dynamically based on data in the message
[[1]] Next handler does the same... until one of the routing decisions leads to a reply channel
[[1]] System receives reply and transfers information to job execution (e.g. status) as necessary
Variation: failure in one of the handlers
Variation: restart after failure
* {Flexible} item processing model
Description:
[[1]] Step hands item to ItemWriter
[[1]] Item is converted to message and sent to synchronous flow
[[1]] Handler accepts message and does something with item
[[1]] System routes result to next handler, possibly dynamically
Variation: failure
[[1]] Handler throws exception
[[1]] System propagates exception up to ItemWriter (forces rollback under normal circs - hence synchronous flow)
* Automatic repeat / retry
Description ({repeat}):
[[1]] User sends message to channel
[[1]] System start a transaction and reseives message, then processes it
[[1]] User sends another message
[[1]] System receives and processes it in the same transaction
[[1]] ... repeat ...
[[1]] System determines that batch is complete and commits transaction
* Restartable file processing
Large {files} need to be processed, so message payload of file contents is not practical. One line or XML event per message with failover and restartability from Spring Batch.
Description:
[[1]] User triggers file processing (sends message, copies file to directory, etc.)
[[1]] System starts new job
[[1]] System processes file line by line (or even by event), wrapping each one as a message and sending it to a synchronous flow
[[1]] System commits periodically (as determined by Spring Batch step configuration)
Variation: failure and restart
[[1]] Item processing fails
[[1]] System aborts job and sends message to failure channel (or failure message to normal reply channel)
[[1]] Operator fixes problem and triggers restart (another message channel?)
[[1]] System restarts job for same file at point where it left off
[[1]] System completes processing
[[1]] System sends sucess message to reply channel
Variation: send to asynchronous flow. Same as main use case but item message is sent to asynchronous flow. Not as robust because if the lights go out then meesages will be lost, but at least a large file can be split into smaller chunks.
* Asynchronous item processing
This is actually a variation on {{{Flexible}flexible item processing model}}.
Description ({async}):
[[1]] ItemProcessor executes in background (non-transactionally)
[[1]] ItemWriter collects outputs from futures before phyically writing data

17
spring-batch-integration/src/site/site.xml
View File

@@ -1,17 +0,0 @@
<?xml version="1.0" encoding="ISO-8859-1"?>
<project name="Spring Batch: ${project.name}">
<body>
<links>
<item name="${project.name}" href="index.html"/>
</links>
<menu name="Documentation">
<item name="${project.name}" href="index.html"/>
</menu>
<menu ref="reports"/>
</body>
</project>

7
src/site/docbook/reference/appendix.xml
View File

@@ -78,7 +78,8 @@
<entry align="left">Reads via iBATIS based on a query. Pages
through the rows so that large datasets can be read without
running out of memory. See HOWTO - Read from a Database</entry>
running out of memory. See HOWTO - Read from a Database. This
ItemReader is now deprecated as of Spring Batch 3.0.</entry>
</row>
<row>
@@ -238,8 +239,8 @@
<row>
<entry align="left">IbatisBatchItemWriter</entry>
<entry align="left">Writes items in a batch using the SqlMapClientTemplate
execute() method</entry>
<entry align="left">Writes items in a batch using the iBatis API's
directly. This ItemWriter is deprecated as of Spring Batch 3.0.</entry>
</row>
<row>

13
src/site/docbook/reference/index.xml
View File

@@ -42,11 +42,10 @@
<surname>Minella</surname>
</author>
<author>
<firstname>Chris</firstname>
<surname>Schaefer</surname>
</author>
<author>
<firstname>Chris</firstname>
<surname>Schaefer</surname>
</author>
</authorgroup>
<legalnotice>
@@ -79,6 +78,10 @@
<xi:include href="common-patterns.xml" />
<xi:include href="jsr-352.xml" />
<xi:include href="spring-batch-integration.xml" />
<xi:include href="appendix.xml" />
<xi:include href="schema-appendix.xml" />

368
src/site/docbook/reference/jsr-352.xml Normal file
View File

@@ -0,0 +1,368 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter id="jsr-352" xreflabel="JSR-352 Support">
<title>JSR-352 Support</title>
<para>As of Spring Batch 3.0 support for JSR-352 has been fully implemented. This section will briefly describe high level JSR-352 specific concepts. It is encouraged to read the full JSR-352 spec, located at the JSR-352 home page: https://jcp.org/en/jsr/detail?id=352</para>
<section id="jsrBatchContexts">
<title id="jsrBatchContexts">Batch Contexts</title>
<para>JSR-352 Batch Contexts provide information about the batch job and interaction with the batch runtime. Context objects can be injected into fields of batch artifact classes using the @Inject annotation.</para>
<section id="jsrJobContext">
<title>JobContext</title>
<para>The JobContext represents the entire batch job. There is one JobContext per job execution and exists for the life of the Job.</para>
</section>
<section id="jsrStepContext">
<title>StepContext</title>
<para>The StepContext represents the current executing step. There is one StepContext per step execution and exists for the life of the Step.</para>
</section>
</section>
<section id="jsrConfiguringAndRunningAJob">
<title id="jsrConfiguringAndRunningAJob">Configuring and Running a Job</title>
<section id="jsrConfiguringAJob">
<title>Configuring a Job</title>
<section id="jsrNamespace">
<title>JSR-352 namespace</title>
<para>JSR-352 API classes reside in the package namespace of javax.batch. The XML namespace also known as the Job Specification Language (JSL) in the language in which Jobs are defined in. The JSL is represented by an XML schema. Refer to the JSR-352 spec document or the XSD itself for more details.</para>
</section>
<section id="jsrBatchXMLConfiguration">
<title>batch.xml configuration</title>
<para>The batch.xml file contains a mapping of batch artifacts to be loaded and used by the runtime. Batch artifact definitions consist of a reference identifier and a fully qualified class name. The batch.xml file resides in the META-INF directory.</para>
</section>
<section id="jsrSpringConfiguration">
<title>Spring configuration</title>
<para>In addition to configuring batch artifacts in the batch.xml file, Spring Batch allows you to configure batch artifacts in the typical Spring bean fashion. To utilize this configuration method simply define your JSR-352 job inside a Spring configuration file and reference your Spring beans via the ref tag as shown in the snippet below.</para>
<para><programlisting>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;beans xmlns=&quot;http://www.springframework.org/schema/beans&quot;
xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
xsi:schemaLocation=&quot;http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://xmlns.jcp.org/xml/ns/javaee
http://xmlns.jcp.org/xml/ns/javaee/jobXML_1_0.xsd&quot;&gt;
&lt;job id=&quot;job1&quot; xmlns=&quot;http://xmlns.jcp.org/xml/ns/javaee&quot; version=&quot;1.0&quot;&gt;
&lt;step id=&quot;step1&quot;&gt;
&lt;chunk&gt;
&lt;reader ref=&quot;testReader&quot;/&gt;
&lt;/chunk&gt;
&lt;/step&gt;
...
&lt;/job&gt;
&lt;bean id=&quot;testReader&quot; class=&quot;TestItemReader&quot; scope=&quot;step&quot;/&gt;
&lt;/beans&gt;</programlisting></para>
</section>
</section>
<section id="jsrRunningAJob">
<title>Running a Job</title>
<section id="jsrJobOperator">
<title>JsrJobOperator</title>
<para>The JsrJobOperator is a javax.batch.operations.JobOperator implementation which supports various operations for dealing with batch jobs. Job XML files are expected to reside in the META-INF/batch-jobs directory.</para>
<para>Typically used JobOperator methods for operating a job include:</para>
<para>
<programlisting>start(String jobXMLName, Properties jobParameters)
restart(long executionId, Properties restartParameters)
stop(long executionId)
abandon(long executionId)</programlisting></para>
<note>When providing the jobXMLname to the start method of JobOperator, the META-INF/batch-jobs prefix nor the .xml file extension should be added.</note>
</section>
</section>
<section id="jsrInterceptingJobExecution">
<title>Intercepting Job Execution</title>
<section id="jsrJobListener">
<title>JobListener</title>
<para>As with Spring Batch, JSR-352 allows for Job level listeners. Job listeners are configured as a child element of the Job in the JSL and implement the javax.batch.api.listener.JobListener interface.</para>
</section>
</section>
</section>
<section id="jsrJobProperties">
<title id="jsrJobProperties">Job Properties</title>
<section id="jsrPropertySupport">
<title>Property Support</title>
<para>JSR-352 allows for properties to be defined at the Job, Step and batch artifact level by way of configuration in the JSL. Batch properties are configured at each level in the following way:</para>
<para>
<programlisting>&lt;properties&gt;
&lt;property name=&quot;propertyName1&quot; value=&quot;propertyValue1&quot;/&gt;
&lt;property name=&quot;propertyName2&quot; value=&quot;propertyValue2&quot;/&gt;
&lt;/properties&gt;</programlisting>
Properties may be configured on any batch artifact.</para>
</section>
<section id="jsrBatchPropertyAnnotation">
<title>@BatchProperty annotation</title>
<para>Properties are referenced in batch artifacts by annotating class fields with the @BatchProperty and @Inject annotations. As defined by JSR-352, fields for properties must be String typed. Any type conversion is up to the implementing developer to perform.</para>
<para>An ItemReader artifact could be configured with a properties block such as the one described above and accessed as such:
<programlisting>public class MyItemReader implements ItemReader {
@Inject
@BatchProperty
private String propertyName1;
...
}</programlisting>
The value of the field "propertyName1" will be "propertyValue1"</para>
</section>
<section id="jsrPropertySubstitution">
<title>Property Substitution</title>
<para>Property substitution is provided by way of operators and simple conditional expressions. The general usage is #{operator['key']}.</para>
<para>Supported operators:</para>
<para>
<itemizedlist>
<listitem>
<para>jobParameters - access job parameter values that the job was started/restarted with.</para>
</listitem>
<listitem>
<para>jobProperties - access properties configured at the job level of the JSL.</para>
</listitem>
<listitem>
<para>systemProperties - access named system properties.</para>
</listitem>
<listitem>
<para>partitionPlan - access named property from the parition plan of a partitioned step.</para>
</listitem>
</itemizedlist>
</para>
<para>
<programlisting>#{jobParameters['unresolving.prop']}?:#{systemProperties['file.separator']}</programlisting>
The left hand side of the assignment is the expected value, the right hand side is the default value. In this example, the result will resolve to a value of the system property file.separator as #{jobParameters['unresolving.prop']} is assumed to not be resolvable. If neither expressions can be resolved, an empty String will be returned. Multiple conditions can be used, which are separated by a ';'.
</para>
</section>
</section>
<section id="jsrRetry">
<title id="jsrRetry">Retry</title>
<para>JSR-352 provides support for retry by way of listeners for chunk level artifacts. Listeners are configured at the step level contained in a listeners block.</para>
<section id="jsrRetrylisteners">
<title>Retry Listeners</title>
<para>JSR-352 provides listener interfaces for various cases of retry handling:</para>
<informaltable frame="all" rowsep="1" colsep="1">
<tgroup cols="2">
<colspec align="left"/>
<colspec align="left"/>
<tbody>
<row>
<entry>
<emphasis role="bold">Artifact Interface</emphasis>
</entry>
<entry>
<emphasis role="bold">Listener Interface</emphasis>
</entry>
</row>
<row>
<entry>javax.batch.api.chunk.ItemReader</entry>
<entry>javax.batch.api.chunk.listener.RetryReadListener</entry>
</row>
<row>
<entry>javax.batch.api.chunk.ItemProcessor</entry>
<entry>javax.batch.api.chunk.listener.RetryProcessListener</entry>
</row>
<row>
<entry>javax.batch.api.chunk.ItemWriter</entry>
<entry>javax.batch.api.chunk.listener.RetryWriteListener</entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
</section>
<section id="jsrConfiguringAStep">
<title id="jsrConfiguringAStep">Configuring a Step</title>
<section id="jsrBatchlet">
<title>Batchlet</title>
<para>Task oriented step's in the JSR-352 implementation are backed by implementations of javax.batch.api.Batchlet. A Batchlet is analogous to a Spring Batch Tasklet.</para>
</section>
<section id="jsrControllingStepFlow">
<title>Controlling Step Flow</title>
<section id="jsrDecider">
<title>Decider</title>
<para>JSR-352 decision support for steps, splits and flows are implemented via implementations of the javax.batch.api.Decider interface.</para>
</section>
<section id="jsrConditionalFlow">
<title>Conditional Flow</title>
<para>JSR-352 allows for both the next attribute and a next element to be present as transition elements. If both are present, the next value specified in the Step attribute will be preferred.</para>
</section>
<section id="jsrConfiguringForStop">
<title>Configuring For Stop</title>
<para>JSR-352 transition elements are matched against in the order they are defined in the JSL.</para>
</section>
</section>
<section id="jsrInterceptingStepExecution">
<title>Intercepting Step Execution</title>
<para>JSR-352 provides support for intercepting Step execution by way of listeners. Listeners are configured at the step level contained in a listeners block.</para>
<informaltable frame="all" rowsep="1" colsep="1">
<tgroup cols="2">
<colspec align="left"/>
<colspec align="left"/>
<tbody>
<row>
<entry>
<emphasis role="bold">Interception Point</emphasis>
</entry>
<entry>
<emphasis role="bold">Listener Interface</emphasis>
</entry>
</row>
<row>
<entry>Step</entry>
<entry>javax.batch.api.listener.StepListener</entry>
</row>
<row>
<entry>Chunk</entry>
<entry>javax.batch.api.chunk.listener.ChunkListener</entry>
</row>
<row>
<entry>Item Reading</entry>
<entry>javax.batch.api.chunk.listener.ItemReadListener</entry>
</row>
<row>
<entry>Item Processing</entry>
<entry>javax.batch.api.chunk.listener.ItemProcessListener</entry>
</row>
<row>
<entry>Item Writing</entry>
<entry>javax.batch.api.chunk.listener.ItemWriteListener</entry>
</row>
<row>
<entry>Skip Read</entry>
<entry>javax.batch.api.chunk.listener.SkipReadListener</entry>
</row>
<row>
<entry>Skip Process</entry>
<entry>javax.batch.api.chunk.listener.SkipProcessListener</entry>
</row>
<row>
<entry>Skip Write</entry>
<entry>javax.batch.api.chunk.listener.SkipWriteListener</entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section id="jsrCompletionPolicies">
<title>Completion Policies</title>
<section id="jsrCheckpointAlgorithm">
<title>Checkpoint Algorithm</title>
<para>By default, completion is based on reaching the end of the chunk dictated by its specified (or default) size. A custom algorithm can be created by implementing the javax.batch.api.chunk.CheckpointAlgorithm interface and referencing it via the chunk attribute "checkpoint-policy".</para>
</section>
</section>
</section>
<section id="jsrItemReadersAndWriters">
<title id="jsrItemReadersAndWriters">ItemReaders and ItemWriters</title>
<para>JSR-352 provides interfaces to read, process and write data.</para>
<informaltable frame="all" rowsep="1" colsep="1">
<tgroup cols="2">
<colspec align="left"/>
<colspec align="left"/>
<tbody>
<row>
<entry>
<emphasis role="bold">Role</emphasis>
</entry>
<entry>
<emphasis role="bold">Interface</emphasis>
</entry>
</row>
<row>
<entry>Reading</entry>
<entry>javax.batch.api.chunk.ItemReader</entry>
</row>
<row>
<entry>Processing</entry>
<entry>javax.batch.api.chunk.ItemProcessor</entry>
</row>
<row>
<entry>Writing</entry>
<entry>javax.batch.api.chunk.ItemProcessor</entry>
</row>
</tbody>
</tgroup>
</informaltable>
</section>
<section id="jsrScaling">
<title id="jsrScaling">Scaling</title>
<section id="jsrPartitioning">
<title>Partitioning</title>
<section id="jsrPartitionPlan">
<title>Partition Plan</title>
<para>A partition plan defines the attributes of a partitioned step. Both chunk oriented and task oriented Step's may be partitioned. Some of these items include the number of partitions, threads and the partitions properties. For example consider the following partitioned Step sample:
<programlisting>...
&lt;step&gt;
...
&lt;partition&gt;
&lt;plan partitions=&quot;10&quot; threads=&quot;2&quot;&gt;
&lt;properties&gt;
&lt;property name=&quot;partitionProperty1&quot; value=&quot;value1&quot;/&gt;
&lt;/properties&gt;
&lt;/plan&gt;
&lt;/partition&gt;
...
&lt;/step&gt;
...</programlisting>
</para>
</section>
<section id="jsrPartitionMapper">
<title>Partition Mapper</title>
<para>The Partition Mapper provides a way to programmatically calculate the number of partitions and threads for a partitioned Step. The Partition Mapper is a child element of partition and implementations should implement the javax.batch.api.partition.PartitionMapper interface.</para>
</section>
<section id="jsrPartitionReducer">
<title>Partition Reducer</title>
<para>The Partition Reducer provides a way to programmatically intercept the partitioned Step's lifecycle. The Partition Reducer is a child element of partition and implementations should implement the javax.batch.api.partition.PartitionReducer interface.</para>
</section>
<section id="jsrPartitionCollector">
<title>Partition Collector</title>
<para>The Partition Collector sends results from each partition to the Partition Analyzer at the end of each checkpoint and again at the end of the partition. The Partition Collector is a child element of partition and implementations should implement the javax.batch.api.partition.PartitionCollector interface.</para>
</section>
<section id="jsrPartitionAnalyzer">
<title>Partition Analyzer</title>
<para>The Partition Analyzer is a collection point for data returned by the Partition Collector's and can be used to implement custom exit status handling for the individual partitions. The Partition Analyzer is a child element of a partition and implements should implement the javax.batch.api.partition.PartitionAnalyzer interface.</para>
</section>
</section>
</section>
</chapter>

2
src/site/docbook/reference/readersAndWriters.xml
View File

@@ -2255,6 +2255,8 @@ itemReader.close(executionContext);</programlisting>
<section id="IbatisPagingItemReader">
<title>IbatisPagingItemReader</title>
<note>This reader is deprecated as of Spring Batch 3.0.</note>
<para>If you use IBATIS for your data access then you can use the
<classname>IbatisPagingItemReader</classname> which, as the name
indicates, is an implementation of a paging

433
src/site/docbook/reference/spring-batch-integration.xml Normal file
View File

@@ -0,0 +1,433 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter id="springBatchIntegration" xreflabel="Spring Batch Integration">
<title>Spring Batch Integration</title>
<section>
<title>Overview of the Spring Integration Batch Module</title>
<para>Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring
Integration. Here is a list. These are features that can extend Spring Batch, or use Spring batch features
in the context of Spring Integration. Work in progress waiting for community feedback. Many issues to do
with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result
of these use cases being prototyped.
</para>
<informaltable frame="all" rowsep="1" colsep="1">
<tgroup cols="5">
<colspec align="left"/>
<colspec align="left"/>
<colspec align="left"/>
<colspec align="left"/>
<colspec align="left"/>
<tbody>
<row>
<entry>
<emphasis role="bold">ID</emphasis>
</entry>
<entry>
<emphasis role="bold">Description</emphasis>
</entry>
<entry>
<emphasis role="bold">Status</emphasis>
</entry>
<entry>
<emphasis role="bold">Sub-package</emphasis>
</entry>
<entry>
<emphasis role="bold">Comments</emphasis>
</entry>
</row>
<row>
<entry>1</entry>
<entry>
<link linkend="Triggers">Message triggers job</link>
</entry>
<entry>Complete</entry>
<entry>launch</entry>
<entry>Complete. Also lots of opportunities with monitoring progress.</entry>
</row>
<row>
<entry>2</entry>
<entry>
<link linkend="Chunking">Chunking and multi-VM job execution</link>
</entry>
<entry>Complete</entry>
<entry>chunk</entry>
<entry>Failures might need some analysis. Use of stateful StepExecutionListener requires use of
step scope.
</entry>
</row>
<row>
<entry>3</entry>
<entry>
<link linkend="Aggregator">Asynchronous Aggregator</link>
</entry>
<entry>Unstarted</entry>
<entry></entry>
<entry></entry>
</row>
<row>
<entry>4</entry>
<entry>
<link linkend="jobs">Stateful and non-linear jobs</link>
-&gt; job = flow
</entry>
<entry>Complete</entry>
<entry>job</entry>
<entry>Simple use cases work well with Spring Batch 2.0 and no Integration features.</entry>
</row>
<row>
<entry>5</entry>
<entry>
<link linkend="Flexible">Flexible item processing model</link>
(as message flow) -&gt; step = flow
</entry>
<entry>Complete</entry>
<entry>item</entry>
<entry>Complete (v. simple using MessagingGateway). Unit tests only.</entry>
</row>
<row>
<entry>6</entry>
<entry>
<link linkend="automaticRepeat">Automatic repeat / retry</link>
</entry>
<entry>Complete</entry>
<entry>retry (unit test)</entry>
<entry>Unit tests only, since it just uses existing features.</entry>
</row>
<row>
<entry>7</entry>
<entry>
<link linkend="files">Restartable file processing</link>
</entry>
<entry>Complete</entry>
<entry>file</entry>
<entry>Seems to hang together. Not tested thoroughly, but apparently someone is using it.
</entry>
</row>
<row>
<entry>8</entry>
<entry>
<link linkend="async">Asynchronous item processing</link>
</entry>
<entry>Complete</entry>
<entry>async</entry>
<entry>A general purpose ItemProcesor that returns a Future.</entry>
</row>
</tbody>
</tgroup>
</informaltable>
<para>Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes. If we
implement 1, then we also don&apos;t need to do any more scheduling and triggering in Spring Batch.
</para>
<para>Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one. We
implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration
later (with probably a split of repeat/retry out of Batch at that time).
</para>
<section>
<title>Message<anchor id="Triggers"/>Triggers<!-- anchor_end --> Job
</title>
<para>Description:</para>
<orderedlist numeration="arabic">
<listitem>
<para>User sends message to channel (maybe through a scheduler)</para>
</listitem>
<listitem>
<para>System interprets message payload as parameters for JobLauncher</para>
</listitem>
<listitem>
<para>System launches job execution</para>
</listitem>
<listitem>
<para>If message had a replyTo, System acknowledges with JobExecution</para>
</listitem>
<listitem>
<para>User accepts response and uses it to monitor progress</para>
</listitem>
</orderedlist>
<para>Variation:</para>
<orderedlist numeration="arabic">
<listitem>
<para>System waits for job to finish and replies when it is over</para>
</listitem>
<listitem>
<para>User polls for replies and gets notification about end of execution</para>
</listitem>
</orderedlist>
<para>Variation:</para>
<orderedlist numeration="arabic">
<listitem>
<para>User wants to block on send and only receive response when job is done</para>
</listitem>
</orderedlist>
</section>
<section>
<title><anchor id="Chunking"/>Chunking<!-- anchor_end --> and Multi-VM
</title>
<para>Description:</para>
<orderedlist numeration="arabic">
<listitem>
<para>Step flushes chunk as message to outgoing channel (repeat up to throttle limit)</para>
</listitem>
<listitem>
<para>Worker thread picks up chunk and processes it</para>
</listitem>
<listitem>
<para>Worker thread replies to response channel</para>
</listitem>
<listitem>
<para>Step picks up reply and aggregates the counts</para>
</listitem>
<listitem>
<para>Step blocks until all the requests are satisfied</para>
</listitem>
</orderedlist>
<para>TODO: failure modes</para>
</section>
<section>
<title>Asynchronous<anchor id="Aggregator"/>Aggregator<!-- anchor_end -->
</title>
<para>Job is executed over long period. Many jobs can be executing concurrently.</para>
<para>Description:</para>
<orderedlist numeration="arabic">
<listitem>
<para>Input stage for each job: System reads all items and marks with the job instance id in a
durable repository (staging table)
</para>
</listitem>
<listitem>
<para>System sends each item (or chunks of items that can be processed together as appropriate) to a
channel
</para>
</listitem>
<listitem>
<para>Items flow through message pipeline, occasionally pausing until certain conditions are met,
possibly for days at a time
</para>
</listitem>
<listitem>
<para>Aggregator sits and waits for all items in a job to be finished and then wraps up</para>
</listitem>
</orderedlist>
</section>
<section>
<title>Stateful and non-linear<anchor id="jobs"/>jobs<!-- anchor_end -->
</title>
<para>Dependencies beyween steps and conditional flow between steps. Each handler node in a message flow is
a step execution, with all the robustness guarantees from the Spring Batch meta data.
</para>
<para>Description:</para>
<orderedlist numeration="arabic">
<listitem>
<para>User launches job</para>
</listitem>
<listitem>
<para>System sends message to channel containing job execution</para>
</listitem>
<listitem>
<para>Handler accepts message and executes a step</para>
</listitem>
<listitem>
<para>Handler translates result of step execution into the same form that it accepted the original
request
</para>
</listitem>
<listitem>
<para>System routes message to next handler, possibly dynamically based on data in the message
</para>
</listitem>
<listitem>
<para>Next handler does the same... until one of the routing decisions leads to a reply channel
</para>
</listitem>
<listitem>
<para>System receives reply and transfers information to job execution (e.g. status) as necessary
</para>
</listitem>
</orderedlist>
<para>Variation: failure in one of the handlers</para>
<para>Variation: restart after failure</para>
</section>
<section>
<title><anchor id="Flexible"/>Flexible<!-- anchor_end --> item processing model
</title>
<para>Description:</para>
<orderedlist numeration="arabic">
<listitem>
<para>Step hands item to ItemWriter</para>
</listitem>
<listitem>
<para>Item is converted to message and sent to synchronous flow</para>
</listitem>
<listitem>
<para>Handler accepts message and does something with item</para>
</listitem>
<listitem>
<para>System routes result to next handler, possibly dynamically</para>
</listitem>
</orderedlist>
<para>Variation: failure</para>
<orderedlist numeration="arabic">
<listitem>
<para>Handler throws exception</para>
</listitem>
<listitem>
<para>System propagates exception up to ItemWriter (forces rollback under normal circs - hence
synchronous flow)
</para>
</listitem>
</orderedlist>
</section>
<section>
<title>Automatic repeat / retry</title>
<para>Description (<anchor id="automaticRepeat"/>repeat<!-- anchor_end -->):
</para>
<orderedlist numeration="arabic">
<listitem>
<para>User sends message to channel</para>
</listitem>
<listitem>
<para>System start a transaction and reseives message, then processes it</para>
</listitem>
<listitem>
<para>User sends another message</para>
</listitem>
<listitem>
<para>System receives and processes it in the same transaction</para>
</listitem>
<listitem>
<para>... repeat ...</para>
</listitem>
<listitem>
<para>System determines that batch is complete and commits transaction</para>
</listitem>
</orderedlist>
</section>
<section>
<title>Restartable file processing</title>
<para>Large<anchor id="files"/>files<!-- anchor_end --> need to be processed, so message payload of file
contents is not practical. One line or XML event per message with failover and restartability from
Spring Batch.
</para>
<para>Description:</para>
<orderedlist numeration="arabic">
<listitem>
<para>User triggers file processing (sends message, copies file to directory, etc.)</para>
</listitem>
<listitem>
<para>System starts new job</para>
</listitem>
<listitem>
<para>System processes file line by line (or even by event), wrapping each one as a message and
sending it to a synchronous flow
</para>
</listitem>
<listitem>
<para>System commits periodically (as determined by Spring Batch step configuration)</para>
</listitem>
</orderedlist>
<para>Variation: failure and restart</para>
<orderedlist numeration="arabic">
<listitem>
<para>Item processing fails</para>
</listitem>
<listitem>
<para>System aborts job and sends message to failure channel (or failure message to normal reply
channel)
</para>
</listitem>
<listitem>
<para>Operator fixes problem and triggers restart (another message channel?)</para>
</listitem>
<listitem>
<para>System restarts job for same file at point where it left off</para>
</listitem>
<listitem>
<para>System completes processing</para>
</listitem>
<listitem>
<para>System sends sucess message to reply channel</para>
</listitem>
</orderedlist>
<para>Variation: send to asynchronous flow. Same as main use case but item message is sent to asynchronous
flow. Not as robust because if the lights go out then meesages will be lost, but at least a large file
can be split into smaller chunks.
</para>
</section>
<section>
<title>Asynchronous item processing</title>
<para>This is actually a variation on<link linkend="Flexible">flexible item processing model</link>.
</para>
<para>Description (<anchor id="async"/>async<!-- anchor_end -->):
</para>
<orderedlist numeration="arabic">
<listitem>
<para>ItemProcessor executes in background (non-transactionally)</para>
</listitem>
<listitem>
<para>ItemWriter collects outputs from futures before phyically writing data</para>
</listitem>
</orderedlist>
</section>
</section>
<section>
<title>Remote Chunking Implementation</title>
<section>
<title>Basic Use Case</title>
<para>Description:</para>
<orderedlist numeration="arabic">
<listitem>
<para>Step flushes chunk as message to outgoing channel (repeat up to throttle limit)</para>
</listitem>
<listitem>
<para>Worker thread picks up chunk and processes it</para>
</listitem>
<listitem>
<para>Worker thread replies to response channel</para>
</listitem>
<listitem>
<para>Step picks up reply if there is one and aggregates the counts</para>
</listitem>
<listitem>
<para>Step reapeats until no more input data</para>
</listitem>
<listitem>
<para>Step blocks until all the outstanding requests are satisfied</para>
</listitem>
</orderedlist>
</section>
<section>
<title>Implementation</title>
<para>A ChunkProcessor acts as a kind of Throttling Asynchronous Messaging Gateway, which isn&apos;t a
pattern that is supported out of the box with Spring Integration. There is a SimpleMessagingGateway that
provides programmatic access to send and receive payloads (instead of messages), so the pattern can be
manually implemented in the ChunkProcessor.
</para>
<para>The current implementation is in the form of an ItemWriter (ChunkMessageChannelItemWriter) which is a
StepExecutionListener (blocks and waits for the outstanding responses in the afterStep). The
ChunkProcessor can then simply be a vanilla implementation from Spring Batch.
</para>
<para>The ChunkMessageChannelItemWriter implements the Throttling part of the pattern by keeping track of
the number of outstanding requests (which it has to do anyway) and blocking until a response arrives if
the number is above a configurable limit. It wouldn&apos;t be necessary to do this manually in the
writer if the messages were only going over local MessageChannels: the requests would either be
processed serially in a single thread, or else there would be a thread pool with limited size
controlling the workers. But since the messages are going to JMS we need to either explicitly throttle
in the writer (or else rely on vendor features for producer flow control), otherwise the JMS Queue could
easily be overwhelmed and start barfing (which happened in one of the early prototypes on an Accenture
project).
</para>
<para>Throttling MessageChannel.send() might be something Spring Integration could do, but it only makes
sense really in the context of this gateway pattern (because you need something to react against to
decide when to release another send).
</para>
<para>The gateway is used to send requests to the workers, and then to receive responses in the same thread,
but only waiting for a response when the step is complete. To implement this with JMS backed channels we
need a Spring Integration inbound adapter that translates PollableChannel.receive() into
JmsTemplate.receiveAndConvert(). In the unlikely event of a problem in the receiver the JMS message
should roll back. JmsDestinationPollingAdapter actually almost does what we need but there is no support
for configuring it without a scheduled poller.
</para>
</section>
</section>
</chapter>

12
src/site/docbook/reference/step.xml
View File

@@ -1727,5 +1727,17 @@ itemWriter.write(items);</programlisting>
<programlisting>&lt;bean class="org.springframework.batch.core.scope.StepScope" /&gt;</programlisting>
</section>
<section id="job-scope">
<title>Job Scope</title>
<para>Job scope, introduced in Spring Batch 3.0 is similar to Step scope in configuration but is a scope for the job context and ensures there is only one instance of a particular bean for the entire Job. To Job scope your beans set the scope to "job":
<programlisting>
&lt;bean id=&quot;jobScopedBean&quot; scope=&quot;step&quot; class=&quot;com.test.MyJobScopedBean&quot;/&gt;
</programlisting>
</para>
</section>
</section>
</chapter>

126
src/site/docbook/reference/whatsnew.xml
View File

@@ -1,130 +1,60 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter id="whatsNew">
<title>What's New in Spring Batch 2.2</title>
<title>What's New in Spring Batch 3.0</title>
<para>The Spring Batch 2.2 release has six major themes:</para>
<para>The Spring Batch 3.0 release has five major themes:</para>
<itemizedlist>
<listitem>
<para>Spring Data Integration</para>
<para>JSR-352 Support</para>
</listitem>
<listitem>
<para>Java Configuration</para>
<para>Promote Spring Batch Integration to Spring Batch</para>
</listitem>
<listitem>
<para>Spring Retry</para>
<para>Upgrade to Support Spring 4 and Java 8</para>
</listitem>
<listitem>
<para>Job Parameters</para>
<para>JobScope Support</para>
</listitem>
<listitem>
<para>SQLite Support</para>
</listitem>
</itemizedlist>
<section id="whatsNewSpringData">
<title id="s.2.1.6">Spring Data Integration</title>
<section id="whatsNewJSR-352Support">
<title>JSR-352 Support</title>
<para>Since the 2.0 release of Spring Batch, the Spring Data project has brought
support for the NoSQL movement to Spring. The 2.2 release of Spring Batch has added
support for MongoDB, Neo4j and Gemfire natively through the Spring Data abstractions.</para>
<para>This release has also added support for writing to any custom Spring Data Repository a
user may write. The <classname>RepositoryItemReader</classname> and
<classname>RepositoryItemWriter</classname> each wrap a repository implementation (
<classname>PagingAndSortingRepository</classname> and <classname>CrudRepository</classname>
respectively) to retrieve data from and persist data to.</para>
<para>The JSR-352 (Batch Applications for the Java Platform) specification has been fully implemented. See <xref linkend="jsr-352"/> for more details.</para>
</section>
<section id="whatsNewJavaConfiguration">
<title>Java Configuration</title>
<section id="whatsNewPromoteSpringBatchIntegrationToSpringBatch">
<title>Promote Spring Batch Integration to Spring Batch</title>
<para>Until 2.2.0 the only option for configuring a job was via XML (either through the batch DSL or
by hand). However, in 2.2.0, Java based configuration has been added as a way to define Spring Batch
Jobs. To support this new configuration option, an annotation and builder classes have been added. What
was previously defined as this:</para>
<programlisting>&lt;batch&gt;
&lt;job-repository/&gt;
&lt;job id="myJob"&gt;
&lt;step id="step1".../&gt;
&lt;step id="step2".../&gt;
&lt;/job&gt;
&lt;beans:bean id="transactionManager".../&gt;
&lt;beans:bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"&gt;
&lt;beans:property name="jobRepository" ref="jobRepository"/&gt;
&lt;/beans:bean&gt;
&lt;/batch&gt;
</programlisting>
<para>Can now be configured using the <classname>@EnableBatchProcessing</classname> annotation and the
provided <classname>JobBuilderFactory</classname> and <classname>StepBuilderFactory</classname> as show below:</para>
<programlisting> &#064;Configuration
&#064;EnableBatchProcessing
&#064;Import(DataSourceCnfiguration.class)
public class AppConfig {
&#064;Autowired
private JobBuilderFactory jobs;
&#064;Bean
public Job job() {
return jobs.get(&quot;myJob&quot;).start(step1()).next(step2()).build();
}
&#064;Bean
protected Step step1() {
...
}
&#064;Bean
protected Step step2() {
...
}
}</programlisting>
<para>The <classname>@EnableBatchProcessing</classname> annotation makes a number
of common dependencies available for autowiring by default. This list includes a
<classname>JobRepsitory</classname>, <classname>JobLauncher</classname>,
<classname>JobRegistry</classname>, <classname>PlatformTransactionManager</classname>,
<classname>JobBuilderFactory</classname>, and a <classname>StepBuilderFactory</classname>.
More information on how to configure Jobs and Steps with the new
Java config can be found in <xref linkend="javaConfig" /></para>
<para>The Spring Batch Integration project has been merged into the main Spring Batch code base. Spring Batch Integration provides support for Remote Chunking, Asynchronous handing and so on. See <xref linkend="springBatchIntegration"/> for more details.</para>
</section>
<section id="whatsNewSpringRetry">
<title>Spring Retry</title>
<section id="whatsNewUpgradeToSupportSpring4andJava8">
<title>Upgrade to Support Spring 4 and Java 8</title>
<para>The ability to retry an operation via the <classname>RetryTemplate</classname>
has always been a feature of Spring Batch. That ability has been identified as a
useful feature for other frameworks (Spring Integration for example). With the 2.2.0
release, the retry logic has been extracted from Spring Batch into it's own library
called Spring Retry. With this change, there are two main impacts. The first is
that the majority of the <literal>org.springframework.batch.retry</literal> package
has been moved into this new library. With that move, the package name has also
dropped the batch to become <literal>org.springframework.retry</literal>.</para>
<para>Spring Batch now depends on Spring 4.0 and has been tested for compatibility against Java 8. Java 6 still remains the default source/target version for the Spring Batch distribution.</para>
</section>
<section id="whatsNewJobParameters">
<title>Job Parameters</title>
<section id="whatsNewJobScopeSupport">
<title>JobScope Support</title>
<para>Prior to the 2.2.0 release of Spring Batch, all parameters pass to a job execution
were used as part of the identity of the job. This limited the ability to change job
parameters during a rerun of a job. To accommodate this use case, 2.2.0 introduced the
idea of non-identifying job parameters.</para>
<para>By default, job parameters in 2.2.0 are still identifying. However, Spring Batch
now allows a user to specify a parameter not be used in the identity of a job instance.
In order to support this change, the domain model for batch changed. Before 2.2.0, job
parameters were associated with a <classname>JobInstance</classname>. 2.2.0 and beyond,
they are associated with a <classname>JobExecution</classname>. This also required the
underlying database schema for the job repository to change.</para>
<para>A new "job" scope has been added, see section <xref linkend="job-scope"/> for more details.</para>
</section>
<section id="whatsNewSQLiteSupport">
<title>SQLite Support</title>
<para>SQLite support has been added along with database schema scripts.</para>
</section>
</chapter>