Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Initial move over from i21 repo.

Browse Source
This commit is contained in:
dsyer
2007-08-15 20:04:43 +00:00
parent 3237c34eb5
commit 170c815916
781 changed files with 67769 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
src/.project Normal file
View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>batch-master</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
</buildSpec>
<natures>
</natures>
</projectDescription>

40
src/site/apt/batch-principles-guidelines.apt Normal file
View File

@@ -0,0 +1,40 @@
------
Spring Batch Overview
------
Scott Wintermute
------
May 2007
General Principles and Guidelines for Batch Architectures
The following are a number of key principles, guidelines, and general considerations to take into consideration when building a batch solution.
* A batch architecture typically affects on-line architecture and vice versa. Design with both architectures and environments in mind using common building blocks when possible.
* Simplify as much as possible and avoid building complex logical structures in single batch applications.
* Process data as close to where the data physically resides as possible or vice versa (i.e., keep your data where your processing occurs).
* Minimize system resource use, especially I/O. Perform as many operations as possible in internal memory.
* Review application I/O (analyze SQL statements) to ensure that unnecessary physical I/O is avoided. In particular, the following four common flaws need to be looked for:
** Reading data for every transaction when the data could be read once and kept cached or in the working storage;
** Rereading data for a transaction where the data was read earlier in the same transaction;
** Causing unnecessary table or index scans;
** Not specifying key values in the WHERE clause of an SQL statement.
* Do not do things twice in a batch run. For instance, if you need data summarization for reporting purposes, increment stored totals if possible when data is being initially processed, so your reporting application does not have to reprocess the same data.
* Allocate enough memory at the beginning of a batch application to avoid time-consuming reallocation during the process.
* Always assume the worst with regard to data integrity. Insert adequate checks and record validation to maintain data integrity.
* Implement checksums for internal validation where possible. For example, flat files should have a trailer record telling the total of records in the file and an aggregate of the key fields.
* Plan and execute stress tests as early as possible in a production-like environment with realistic data volumes.
* In large batch systems backups can be challenging, especially if the system is running concurrent with on-line on a 24-7 basis. Database backups are typically well taken care of in the on-line design, but file backups should be considered to be just as important. If the system depends on flat files, file backup procedures should not only be in place and documented, but regularly tested as well.

196
src/site/apt/batch-processing-strategies.apt Normal file
View File

@@ -0,0 +1,196 @@
------
Batch Processing Strategy
------
Scott Wintermute, Wayne Lund
------
May 2007
Batch Processing Strategies
To help design and implement batch systems, basic batch application building blocks and patterns should be provided to the designers and programmers in form of sample structure charts and code shells. When starting to design a batch job, the business logic should be decomposed into a series of steps which can be implemented using the following standard building blocks:
* <<Conversion Applications:>> For each type of file supplied by or generated to an external system, a conversion application will need to be created to convert the transaction records supplied into a standard format required for processing. This type of batch application can partly or entirely consist of translation utility modules (see Basic Batch Services).
* <<Validation Applications:>> Validation applications ensure that all input/output records are correct and consistent. Validation is typically based on file headers and trailers, checksums and validation algorithms as well as record level cross-checks.
* <<Extract Applications:>> An application that reads a set of records from a database or input file, selects records based on predefined rules, and writes the records to an output file.
* <<Extract/Update Applications:>> An application that reads records from a database or an input file, and makes changes to a database or an output file driven by the data found in each input record.
* <<Processing and Updating Applications:>> An application that performs processing on input transactions from an extract or a validation application. The processing will usually involve reading a database to obtain data required for processing, potentially updating the database and creating records for output processing.
* <<Output/Format Applications:>> Applications reading an input file, restructures data from this record according to a standard format, and produces an output file for printing or transmission to another program or system.
Additionally a basic application shell should be provided for business logic that cannot be built using the previously mentioned building blocks.
In addition to the main building blocks, each application may use one or more of standard utility steps, such as:
* Sort - A Program that reads an input file and produces an output file where records have been re-sequenced according to a sort key field in the records. Sorts are usually performed by standard system utilities.
* Split - A program that reads a single input file, and writes each record to one of several output files based on a field value. Splits can be tailored or performed by parameter-driven standard system utilities.
* Merge - A program that reads records from multiple input files and produces one output file with combined data from the input files. Merges can be tailored or performed by parameter-driven standard system utilities.
Batch applications can additionally be categorized by their input source:
* Database-driven applications are driven by rows or values retrieved from the database.
* File-driven applications are driven by records or values retrieved from a file.
* Message-driven applications are driven by messages retrieved from a message queue.
The foundation of any batch system is the processing strategy. Factors affecting the selection of the strategy include: estimated batch system volume, concurrency with on-line or with another batch systems, available batch windows (and with more enterprises wanting to be up and running 24x7, this leaves no obvious batch windows).
Typical processing options for batch are:
* Normal processing in a batch window during off-line
* Concurrent batch / on-line processing
* Parallel processing of many different batch runs or jobs at the same time
* Partitioning (i.e. processing of many instances of the same job at the same time)
* A combination of these
The order in the list above reflects the implementation complexity, processing in a batch window being the easiest and partitioning the most complex to implement.
Some or all of these options may be supported by a commercial scheduler.
In the following section these processing options are discussed in more detail. It is important to notice that the commit and locking strategy adopted by batch processes will be dependent on the type of processing performed, and as a rule of thumb and the on-line locking strategy should also use the same principles. Therefore, the batch architecture cannot be simply an afterthought when designing an overall architecture.
The locking strategy can use only normal database locks, or an additional custom locking service can be implemented in the architecture. The locking service would track database locking (for example by storing the necessary information in a dedicated db-table) and give or deny permissions to the application programs requesting a db operation. Retry logic could also be implemented by this architecture to avoid aborting a batch job in case of a lock situation.
<<1. Normal processing in a batch window>>
For simple batch processes running in a separate batch window, where the data being updated is not required by on-line users or other batch processes, concurrency is not an issue and a single commit can be done at the end of the batch run.
In most cases a more robust approach is more appropriate. A thing to keep in mind is that batch systems have a tendency to grow as time goes by, both in terms of complexity and the data volumes they will handle. If no locking strategy is in place and the system still relies on a single commit point, modifying the batch programs can be painful. Therefore, even with the simplest batch systems, consider the need for commit logic for restart-recovery options as well as the information concerning the more complex cases below.
<<2. Concurrent batch / on-line processing>>
Batch applications processing data that can simultaneously be updated by on-line users, should not lock any data (either in the database or in files) which could be required by on-line users for more than a few seconds. Also updates should be committed to the database at the end of every few transaction. This minimizes the portion of data that is unavailable to other processes and the elapsed time the data is unavailable.
Another option to minimize physical locking is to have a logical row-level locking implemented using either an Optimistic Locking Pattern or a Pessimistic Locking Pattern.
* Optimistic locking assumes a low likelihood of record contention. It typically means inserting a timestamp column in each database table used concurrently by both batch and on-line processing. When an application fetches a row for processing, it also fetches the timestamp. As the application then tries to update the processed row, the update uses the original timestamp in the WHERE clause. If the timestamp matches, the data and the timestamp will be updated successfully. If the timestamp does not match, this indicates that another application has updated the same row between the fetch and the update attempt and therefore the update cannot be performed.
* Pessimistic locking is any locking strategy that assumes there is a high likelihood of record contention and therefore either a physical or logical lock needs to be obtained at retrieval time. One type of pessimistic logical locking uses a dedicated lock-column in the database table. When an application retrieves the row for update, it sets a flag in the lock column. With the flag in place, other applications attempting to retrieve the same row will logically fail. When the application that set the flag updates the row, it also clears the flag, enabling the row to be retrieved by other applications. Please note, that the integrity of data must be maintained also between the initial fetch and the setting of the flag, for example by using db locks (e.g., SELECT FOR UPDATE). Note also that this method suffers from the same downside as physical locking except that it is somewhat easier to manage building a time-out mechanism that will get the lock released if the user goes to lunch while the record is locked.
These patterns are not necessarily suitable for batch processing, but they might be used for concurrent batch and on-line processing (e.g. in cases where the database doesn't support row-level locking). As a general rule, optimistic locking is more suitable for on-line applications, while pessimistic locking is more suitable for batch applications. Whenever logical locking is used, the same scheme must be used for all applications accessing data entities protected by logical locks.
Note that both of these solutions only address locking a single record. Often we may need to lock a logically related group of records. With physical locks, you have to manage these very carefully in order to avoid potential deadlocks. With logical locks, it is usually best to build a logical lock manager that understands the logical record groups you want to protect and can ensure that locks are coherent and non-deadlocking. This logical lock manager usually uses its own tables for lock management, contention reporting, time-out mechanism, etc.
<<3. Parallel Processing>>
Parallel processing allows multiple batch runs / jobs to run in parallel to minimize the total elapsed batch processing time. This is not a problem as long as the jobs are not sharing the same files, db-tables or index spaces. If they do, this service should be implemented using partitioned data. Another option is to build an architecture module for maintaining interdependencies using a control table. A control table should contain a row for each shared resource and whether it is in use by an application or not. The batch architecture or the application in a parallel job would then retrieve information from that table to determine if it can get access to the resource it needs or not.
If the data access is not a problem, parallel processing can be implemented through the use of additional threads to process in parallel. In the mainframe environment, parallel job classes have traditionally been used, in order to ensure adequate CPU time for all the processes. Regardless, the solution has to be robust enough to ensure time slices for all the running processes.
Other key issues in parallel processing include load balancing and the availability of general system resources such as files, database buffer pools etc. Also note that the control table itself can easily become a critical resource.
<<4. Partitioning>>
Using partitioning allows multiple versions of large batch applications to run concurrently. The purpose of this is to reduce the elapsed time required to process long batch jobs. Processes which can be successfully partitioned are those where the input file can be split and/or the main database tables partitioned to allow the application to run against different sets of data.
In addition, processes which are partitioned must be designed to only process their assigned data set. A partitioning architecture has to be closely tied to the database design and the database partitioning strategy. Please note, that the database partitioning doesn't necessarily mean physical partitioning of the database, although in most cases this is advisable. The following picture illustrates the partitioning approach:
[images/partitioned.png]
The architecture should be flexible enough to allow dynamic configuration of the number of partitions. Both automatic and user controlled configuration should be considered. Automatic configuration may be based on parameters such as the input file size and/or the number of input records.
<<4.1 Partitioning Approaches>>
The following lists some of the possible partitioning approaches. Selecting a partitioning approach has to be done on a case-by-case basis.
<1. Fixed and Even Break-Up of Record Set>
This involves breaking the input record set into an even number of portions (e.g. 10, where each portion will have exactly 1/10th of the entire record set). Each portion is then processed by one instance of the batch/extract application.
In order to use this approach, preprocessing will be required to split the recordset up. The result of this split will be a lower and upper bound placement number which can be used as input to the batch/extract application in order to restrict its processing to its portion alone.
Preprocessing could be a large overhead as it has to calculate and determine the bounds of each portion of the record set.
<2. Breakup by a Key Column>
This involves breaking up the input record set by a key column such as a location code, and assigning data from each key to a batch instance. In order to achieve this, column values can either be
<3. Assigned to a batch instance via a partitioning table (see below for details).>
<4. Assigned to a batch instance by a portion of the value (e.g. values 0000-0999, 1000 - 1999, etc.)>
Under option 1, addition of new values will mean a manual reconfiguration of the batch/extract to ensure that the new value is added to a particular instance.
Under option 2, this will ensure that all values are covered via an instance of the batch job. However, the number of values processed by one instance is dependent on the distribution of column values (i.e. there may be a large number of locations in the 0000-0999 range, and few in the 1000-1999 range). Under this option, the data range should be designed with partitioning in mind.
Under both options, the optimal even distribution of records to batch instances cannot be realized. There is no dynamic configuration of the number of batch instances used.
<5. Breakup by Views>
This approach is basically breakup by a key column, but on the database level. It involves breaking up the recordset into views. These views will be used by each instance of the batch application during its processing. The breakup will be done by grouping the data.
With this option, each instance of a batch application will have to be configured to hit a particular view (instead of the master table). Also, with the addition of new data values, this new group of data will have to be included into a view. There is no dynamic configuration capability, as a change in the number of instances will result in a change to the views.
<6. Addition of a Processing Indicator>
This involves the addition of a new column to the input table, which acts as an indicator. As a preprocessing step, all indicators would be marked to non-processed. During the record fetch stage of the batch application, records are read on the condition that that record is marked non-processed, and once they are read (with lock), they are marked processing. When that record is completed, the indicator is updated to either complete or error. Many instances of a batch application can be started without a change, as the additional column ensures that a record is only processed once.
With this option, I/O on the table increases dynamically. In the case of an updating batch application, this impact is reduced, as a write will have to occur anyway.
<7. Extract Table to a Flat File>
This involves the extraction of the table into a file. This file can then be split into multiple segments and used as input to the batch instances.
With this option, the additional overhead of extracting the table into a file, and splitting it, may cancel out the effect of multi-partitioning. Dynamic configuration can be achieved via changing the file splitting script.
<8. Use of a Hashing Column>
This scheme involves the addition of a hash column (key/index) to the database tables used to retrieve the driver record. This hash column will have an indicator to determine which instance of the batch application will process this particular row. For example, if there are three batch instances to be started, then an indicator of 'A' will mark that row for processing by instance 1, an indicator of 'B' will mark that row for processing by instance 2, etc.
The procedure used to retrieve the records would then have an additional WHERE clause to select all rows marked by a particular indicator. The inserts in this table would involve the addition of the marker field, which would be defaulted to one of the instances (e.g. 'A').
A simple batch application would be used to update the indicators such as to redistribute the load between the different instances. When a sufficiently large number of new rows have been added, this batch can be run (anytime, except in the batch window) to redistribute the new rows to other instances.
Additional instances of the batch application only require the running of the batch application as above to redistribute the indicators to cater for a new number of instances.
<<4.2 Database and Application design Principles>>
An architecture that supports multi-partitioned applications which run against partitioned database tables using the key column approach, should include a central partition repository for storing partition parameters. This provides flexibility and ensures maintainability. The repository will generally consist of a single table known as the partition table.
Information stored in the partition table will be static and in general should be maintained by the DBA. The table should consist of one row of information for each partition of a multi-partitioned application. The table should have columns for: Program ID Code, Partition Number (Logical ID of the partition), Low Value of the db key column for this partition, High Value of the db key column for this partition.
On program start-up the program id and partition number should be passed to the application from the architecture (Control Processing Tasklet). These variables are used to read the partition table, to determine what range of data the application is to process (if a key column approach is used). In addition the partition number must be used throughout the processing to:
* Add to the output files/database updates in order for the merge process to work properly
* Report normal processing to the batch log and any errors that occur during execution to the architecture error handler
<<4.3 Minimizing Deadlocks>>
When applications run in parallel or partitioned, contention in database resources and deadlocks may occur. It is critical that the database design team eliminates potential contention situations as far as possible as part of the database design.
Also ensure that the database index tables are designed with deadlock prevention and performance in mind.
Deadlocks or hot spots often occur in administration or architecture tables such as log tables, control tables, and lock tables. The implications of these should be taken into account as well. A realistic stress test is crucial for identifying the possible bottlenecks in the architecture.
To minimize the impact of conflicts on data, the architecture should provide services such as wait-and-retry intervals when attaching to a database or when encountering a deadlock. This means a built-in mechanism to react to certain database return codes and instead of issuing an immediate error handling, waiting a predetermined amount of time and retrying the database operation.
<<4.4 Parameter Passing and Validation>>
The partition architecture should be relatively transparent to application developers. The architecture should perform all tasks associated with running the application in a partitioned mode including:
* Retrieve partition parameters before application start-up
* Validate partition parameters before application start-up
* Pass parameters to application at start-up
The validation should include checks to ensure that:
* the application has sufficient partitions to cover the whole data range
* there are no gaps between partitions
If the database is partitioned, some additional validation may be necessary to ensure that a single partition does not span database partitions.
Also the architecture should take into consideration the consolidation of partitions. Key questions include:
* Must all the partitions be finished before going into the next job step?
* What happens if one of the partitions aborts?

104
src/site/apt/blotter.apt Normal file
View File

@@ -0,0 +1,104 @@
------
Spring Batch-Retry Comments
------
Dave Syer
------
February 2007
Open Comments and Questions
* Batches and Asynchronous JMS
There are a large number of common concerns between
<<<DefaultMessageListenerContainer>>> and <<<RepeatTemplate>>>. In
fact one could imagine <<<DefaultMessageListenerContainer>>> being a
simple example of a batch, something like:
+---
RepeatTemplate template = new RepeatTemplate();
template.setTaskExecutor(new SimpleAsyncTaskExecutor());
template.setTerminationPolicy(new TerminateNeverPolicy());
template.execute(new JmsItemProviderCallback(jmsTemplate));
+---
* Instead of setting the transactionManager in
<<<DefaultMessageListenerContainer>>>, wrap the callback in a
transaction proxy.
* Instead of setting sessionTransacted=true in the
<<<DefaultMessageListenerContainer>>>, set it on the
<<<JmsTemplate>>>.
I think that would give me 80% of the functionality in a
<<<DefaultMessageListenerContainer>>> without any changes to
<<<RepeatTemplate>>>. Maybe the other 20% would be useful additions
to <<<RepeatTemplate>>> anyway (like being able to stop and start).
Maybe there is a case for sharing some code, e.g. a base class.
Maybe the batch project should be an offshoot from the core.task
package. It certainly looks like the
<<<DefaultMessageListenerContainer>>> could be a lot simpler (and
easier to test), if it delegates transactional properties to
something not a lot different from a <<<RepeatTemplate>>>.
Resolved
* Using <<<TaskExecutor>>> in Batches
* What can you do with a <<<RepeatTemplate>>> that you couldn't do with
a <<<TaskExecutor>>>? Maybe <<<RepeatTemplate>>> should be a
<<<TaskExecutor>>>, or use one to execute the batch? Probably the
latter would work best, on the basis of preferring composition to
inheritance generally.
* Asynchronous Batching
* Can you run a batch asynchronously? Would need a thread-safe
<<<RepeatContext>>> that can be shared amongst participating
threads, and used to determine termination conditions.
* If <<<RepeatTemplate>>> used a <<<TaskExecutor>>> to execute its
tasks, asynchronous batch might be as simple as using an
asynchronous <<<TaskExecutor>>> internally - the same
<<<RepeatTemplate>>> would be able to operate in both modes, just
by changing the <<<TaskExecutor>>>.
* Using RetryContext to Stash State for the Policies
E.g. in <<<RetryTemplate>>>:
+---
protected void setupContext(RetryCallback callback,
RetryContext context) {
if (callback instanceof AttributeAccessor) {
AttributeAccessor accessor = (AttributeAccessor) callback;
String[] names = accessor.attributeNames();
for (int i = 0; i < names.length; i++) {
String name = names[i];
context.setAttribute(name, accessor.getAttribute(name));
}
}
}
+---
But this is pants: it makes the callback stateful. We can't store
state in the callback to do with the current item becaue the
callback might (will?) be shared between attempts in a concurrent
system.
So what to do? Nothing - the state for retry is nothing to do with
the callback, and it is natural to store it in the context.
* Asynchronous Batching - Thread Safe Context
* What would happen if several threads were sharing a context object
via a synchronization manager in a thread local (like
<<<TransactionSynchronizationManager>>>)? The context itself had
better be thread safe, otherwise the concurrent peers might assume
that they have the only copy of the context and try and modify it.

311
src/site/apt/building.apt Normal file
View File

@@ -0,0 +1,311 @@
------
Building Spring Batch
------
Dave Syer
------
April 2007
Building Spring Batch
Spring Batch is organised as a reactor build in Maven (m2). To
build from the command line use
+---
$ mvn install
+---
or the goal of your choice (compile, test, etc.). This builds the
artifact (e.g. jar file) from the project in the current directory,
and deploys it to you local m2 repo at
<<<${user.home}/.m2/repository>>>. See below for instructions on how
to build the documentation and web site.
By default the whole project (including subprojects) will be built
using Maven's "reactor" plugin. This can be expensive. To build
only one module, cd to that directory first. Or at the top level
use -N (for non-recursive) to exclude subprojects.
+---
$ mvn -N install
+---
* Skipping Tests
The profile <<fast>> skips all the tests, so
+---
mvn -o install -P fast
+---
is the quickest way to update your local repo (assuming the tests
are OK). It is equivalent of setting <<<-Dmaven.test.skip=true>>>.
* Eclipse IDE
Each of the reactor modules at the top level also builds on its own
if you use the (excellent) Eclipse-plugin for m2
(http://m2eclipse.codehaus.org/update/). Get version 0.0.10 or
better, because it supports dependencies on Eclipse projects that
are themselves Maven parents of the current project.
* Dependencies
If you get multiple versions of the same jar across projects, or a
jar is appearing in the classpath that you don't think is necessary,
look into the dependency structure and try and exclude it from
wherever it is being transitively included. To see the dependencies
for a project look in the site for the dependency report.
Alternatively (very useful for quickly locating a rogue jar) use
+---
$ mvn -P snapshots dependency:tree
+---
We use the "snapshots" profile here so that we get a snapshot of the
dependency plugin (older versions did not have the tree goal, but
newer versions are not stable enough to use in production).
* Documentation
With the exception of reference docs, please put content in the
project that it is most closely associated with. Here is a
{{{sitemap.html}site map}} to help you decide.
** Quotidian Web Content
Maven allows you to choose from a range of source format for
building web content. For Spring Batch we prefer the "almost plain
text" version. See files under <<<src/site/apt>>> in all the projects
for examples, and also refer to the
{{{http://maven.apache.org/guides/mini/guide-apt-format.html}Apt
Format Guide}} on the Maven website.
N.B. you put .apt source files in a subdirectory called <<<apt>>>,
but they are moved to the top level when the site is built. Thus
<<<apt/index.apt>>> becomes <<<index.html>>>.
*** Using emacs to edit .apt files
Because the .apt format relies on indentation in plain text files,
the emacs auto-fill feature in text mode makes editing very
convenient. Put this in your .emacs
+---
(setq auto-mode-alist (cons '("\\.apt\\'" . text-mode) auto-mode-alist))
+---
Then use <<<M-q>>> to auto-fill the current paragraph. Emacs
adjusts the indentation of all the lines to match the first one (or
the first two if the second is different.
If anyone knows how to do this with Eclipse or other editors, let us
know and we'll put a note here.
** Reference Guide
The <<<docs>>> project is reserved for reference guides in the
normal Spring docbook format. Each chapter of the reference guide
is in a separate xml file under <<<src/site/docbook/reference>>>.
The easiest way to work with the reference guide is to cd to the
<<<docs>>> module, and run Maven from there.
Use the DTD with a validating XML editor (e.g. Eclipse) to explore
the docbook format. Also look at existing examples in Spring Batch
and in the Core Spring Framework source code.
N.B. there is no need to explicitly create section numbers in the
XML - this is done for you by the build when everything is stitched
together into a book.
N.B. you put docbook .xml source files in a subdirectory called
<<<docbook>>>, but they are moved to the top level when the site is
built. Thus <<<docbook/reference/index.xml>>> becomes
<<<reference/index.html>>>.
** Adding a new chapter to the Reference Guide
Here is a skeleton chapter including the DTD to get you started on a
new chapter.
+---
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter id="chapter-id">
<title>Chapter Title</title>
<section>
<title>Introduction</title>
<para></para>
</section>
</chapter>
+---
Create a file with the template above, and put it in
<<<docbook/reference>>>. Use lower case, dash separated file names
(XML style), e.g. <<<my-new-chapter.xml>>>.
Add the chapter to the master book in <<<index.xml>>> using
+---
<xi:include href="my-new-chapter.xml"/>
+---
* Adding graphics
Put (e.g.) PNG image content in <<<src/site/resources/images>>>, and
then refer to the file using the <<<images/>>> directory prefix.
** In .apt
With no whitespace add the image name in square brackets (\[\]):
+---
[images/MyFigure.png] Caption content here is not rendered by default
in a browser (it's the ALT content)...
+---
** In docbook
Use the \<mediaobject\> element:
+---
<mediaobject>
<imageobject role="fo">
<imagedata fileref="src/site/resources/reference/images/mypic.png" format="PNG" align="center"/>
</imageobject>
<imageobject role="html">
<imagedata fileref="images/mypic.png" format="PNG" align="center"/>
</imageobject>
<caption>
<para>
Figure 1: the figure caption...
</para>
</caption>
</mediaobject>
+---
* Program Listings in Docbook (Including XML)
Use CDATA to save you from having to use the HTML escapes for all
the special characters. E.g.
+---
<programlisting><![CDATA[
<!-- ... my program listing here -->
]]>
</programlisting>
+---
* Dynamic Editing
To see your changes to web site content as soon as you have typed
it, use
+---
mvn site:run
+---
and go to http://localhost:8080.
In a project with unit tests, you can skip the tests and go straight
to the documentation using
+---
mvn -o site:run -P fast
+---
If you are offline, or want to speed things up a bit, the "-o" stops
Maven from trying to resolve dependencies on the internet.
Use -N to build only the current project, not subprojects, So this
is pretty useful at the top level:
+---
mvn -N -o site:run -P fast
+---
In the <<<docs>>> project the docbook reference guide shows up at
http://localhost:8080/reference/*.html, where * is the name of an
xml file with a chapter in it. There is no link to these pages on
the site because the real docbook generated output is much nicer,
but this is still pretty useful for debugging and dynamic
editing.
Note that the formatting is a bit limited compared to the whole
docbook stylesheet - Maven uses Doxia to squish all of docbook into
some simple wiki-like formatting rules. In particular it can't
generate the index page in the format we need it, so you may see
errors from <<<mvn site:run>>> if you visit that page. One of the
features is that the <<<\<xi:include\>>>> syntax we use to build the
index and table of contents in the docbook-generated pages does not
work. Images are another problem. Use the generated content from
<<<mvn site>>> to view these artifacts.
* Building and deploying the web site
There is a bug in the m2 reactor (MNG-740) which means that we have
to install the parent pom to the local repo first.
So do it this way:
+---
$ mvn install -P fast
$ mvn clean site site:deploy
+---
Add "-P deployment" to deploy to the real website (requires ssh
access to static.springframework.org).
The default without -P is to deploy to <<<target/staging>>>, so we
don't get accidental updates to the site. To test the site contents
navigate with your browser to that directory. The site:stage goal
deos not work properly for this build: all the subprojects are not
integrated into the staging site, so use site:deploy instead.
The static website content is not deleted during the deployment
process - merely replaced. If you need to clean everything up from
scratch you need to delete the contents on the server as well
(using ssh).
Problems?
Make sure your source code is up to date. Delete everything from
your local Spring Batch repo
<<<${user.home}/.m2/repository/org/springframework/batch>>>. If
necessary, delete a project or directory and update from SVN again.
Try
+---
$ mvn install
+---
or
+---
$ mvn clean install
+---
or
+---
$ mvn clean install -P fast
+---
from the top level, and
+---
$ mvn -U ...
+---
from wherever you are (top level or sub-project). The latter will
update any older plugins you have in your local Maven repository.
Some people have had trouble building the web site without this.
If you get <<<OutOfMemoryError>>> e.g. building the site, use
MAVEN_OPTS to boost the heap size (on the command line if you have a
sensible shell):
+---
$ MAVEN_OPTS=-Xmx256m mvn site
+---

159
src/site/apt/cases/async.apt Normal file
View File

@@ -0,0 +1,159 @@
------
Asynchronous Chunk Processing Use Case
------
Dave Syer
------
January 2007
Use Case: Asynchronous Chunk Processing
* Goal
Increased the efficiency of chunk processing by having it execute
asynchronously: each record in a separate thread. Maintain
transactional intergrity of the chunk.
* Scope
* All chunks might conceivably benefit from parallel processing, so
we don't want any unnecessary restrictions on the batch operation,
or its implementation. A should be possible for Client to write a
batch operation without reference to the fact that it might run in
an asynchronous chunk.
* Preconditions
* Input data exists with non-trivial size: chunks contain more than
one record.
* Batch processing of a record is slow, or can be delayed, so that
the asynchronous processing can take longer than launching the
threads.
* A chunk can be made to fail after at least one record is
processed.
* Success
* A chunk is processed and the results inspected to verify that all
records were processed.
* Transactional behaviour is verified by rolling back a chunk and
verifying that no records were processed.
* Description
The vanilla case proceeds as for normal {{{chunks.html}chunk
processing}}, but:
[[1]] Within a chunk, Container processes records in parallel.
[[1]] At the end of a chunk, Container waits for the last record
to be processed (with a timeout if the wait is long).
* Variations
** Rollback on Failure
If there is an exception in one of the record processing threads,
the whole chunk should roll back:
[[1]] Client throws exception in record processing.
[[1]] Container catahes exception and attempts to abort other
running processes.
[[1]] Container waits for running processes to abort (or finish
normally, but preferably to abort).
[[1]] Container propagates the exception and signals transaction to
rollback.
** Timeout
If there is a timeout during a chunk, it might happen before the
chunk has finished, or while waiting for the processes to complete
before exiting.
[[1]] At end of chunk, Container is waiting for all processes to
finish. It times out, according to a parameter set by the
Operator.
[[1]] Container does not start any new processes, and attempts to
abort running processes.
[[1]] Container waits for running processes to abort (or finish
normally, but preferably to abort).
[[1]] Container throws a time out exception and signals chunk
transaction to rollback.
* Implementation
* The implementation of this use case could be tricky in the general
case. In particular, the transactional nature is going to be hard
or impossible to maintain across multiple threads without the
individual processes being aware of the transaction, and (perhaps)
without global (XA) transaction support.
A "normal" local transaction is thread bound - i.e. it only executes
in one thread. If the code inside the transaction creates new
threads, then they might not finish processing before the parent
exits and the transaction wants to finish. The transaction needs to
wait for the sub-processes before committing, or (more difficult)
rolling back. The rollback case basically forces us to a model of
one transaction per thread, and therefore to one transaction per
data item in a concurrent environment.
Otherwise some transactional semantics might be respected in a
parallel process, but others certainly will not be because
synchronizations and resources are managed at the level of the
thread where the transaction started. If the transaction manager is
a local one (not XA) there is little hope even that the datasource
resource would be the same for all the parallel threads and the
parent method.
If we use a global transaction manager to make the parallel
processes transactional, how will they know which transaction to
participate in? There could be many active chunks, and each would
have its own threads - how would each one be able to guide its child
processes to participate in the same transaction?
* Beware a framework that extracts data from an <<<ItemProvider>>>
before executing the business logic (e.g. in a
<<<ItemProcessor>>>). It is not enough to allow concurrent
processing but simply insist that the individual records are
processed transactionally because the <<<ItemProvider>>> will then
not be able to participate in the transaction - its next record has
already been passed to the consumer when the transaction starts, so
if there is a rollback then the record is lost.
This is the origin of the signature:
+---
public interface ItemProvider {
Object next();
}
+---
There is no peeking and no iteratror-style <<<hasNext>>>. If there
is a processing problem, transactional clients of the
<<<ItemProvider>>> throw an exception <after> the provider's
<<<next()>>> has been called, but in the same thread (so that
transactional semantics are preserved and the data provider reverts
to its previous state).
This means that in the callback interface also picks up an
<<<Object>>> return type
+---
public interface RepeatCallback {
Object doInIteration(BatchContext context);
}
+---
so we can return an object, which is null when the processing has
finished.
In the end we decided against the <<<Object>>> retrun type and went
with a boolean flag to signal (false) for no more processing.

208
src/site/apt/cases/chunks.apt Normal file
View File

@@ -0,0 +1,208 @@
------
Commit Periodically Use Case
------
Dave Syer
------
January 2007
Use Case: Commit Batch Process Periodically
* Goal
Read a file line-by-line and process into database inserts, for
example using the Jdbc API. Commit periodically, and if there is a
fault where the database transaction rolls back, then the file
reader is reset to the place it was after the last successful
commit.
To develop a batch process to achieve the goal above should be as
simple a process as possible. The more that can be done with simple
POJOs and Spring configuration the better.
* Scope
To keep things simple for now, assume that:
* All lines in the input file are in the same format and each line
generates a single database insert (or a fixed number).
* The file is read synchronously by a single consumer.
* Preconditions
* A file exists in the right format, with a sufficiently large
number of lines to be realistic.
* A mechanism exists to force a rollback at a non-trivial position
(not during the first commit), but produce a successful operation
on the second try.
* A framework for retry exists, so that the case above can be
tested.
* Success
Integration test confirms that
* All data are processed and records inserted successfully.
* When a rollback occurs and the retry is successful, the complete
dataset is processed (same result as successful run).
* Batch operations can be implemented without framework code (or
with minimal dependencies, e.g. through interfaces). Launching
the batch might require access to framework code.
* Description
The vanilla successful batch use case proceeds as follows:
[[1]] Container starts a transaction.
[[1]] Container makes resources available, e.g. opens file and
creates <<<FileChannel>>> for it.
[[1]] Client reads a line from the file, and converts it to a
database statement, then runs it.
[[1]] Container increments counter.
[[1]] Repeat previous two steps until a counter is equal to chunk
size.
[[1]] Container commits database transaction.
[[1]] Repeat chunk processing until input source is exhausted.
* Variations
** Non-fatal Chunk Failure
If there is an unrecoverable database exception during execution of
client code:
[[1]] Container rolls back current transaction.
[[1]] Container resets input source to the point it was at before
failure.
[[1]] Container retries chunk.
** Fatal Chunk Failure
If there is an error in the input data in the middle of a chunk
(could be manifested as database exception, e.g. uniqueness
exception, or nullable exception):
[[1]] Container rolls back current transaction.
[[1]] Container terminates batch and notifies client of precise
details, including the line number of error, and the last line
that was committed (last of the previous chunk).
There is no need to reset the input source because the error is
fatal.
To restart:
[[1]] Operator truncates the input file so the completed chunks
are not repeated.
[[1]] Operator fixes bad line (if there was one), and starts the
batch process wit hthe same parameters.
Variations on this theme are also necessary, e.g. a tolerance for a
small number of bad records in the input data.
* Implementation
* The concept of a batch iterator seems relevant here (see also the
{{{simple.html}simple}} use case). The iterator could be more than
just a loop that might terminate early: here it could also manage
the file cursor on the input source. In this design there is a
<<<ItemProvider>>> interface that can take care of termination and
iteration (e.g. iterator-like method signatures).
* Another design idea (more encapsulated and more in keeping with
existing Spring practice) is to make the data source transaction
aware, and for the client use it like a database resource, through a
template. In this case there is a <<<FileInputTemplate>>>. The
<<<ItemProvider>>> needs to be aware of the data source template, so
that it can terminate when the data is exhausted.
In this version of events there are two kinds of resource in play.
The transaction itself, and the data sources that are aware of the
transaction. The comparison with <<<DataSourceTransactionManager>>>
and <<<JdbcTemplate>>> is obvious. The client is often completely
unaware of the transaction manager, which is applied through an
interceptor, whereas the data source is used explicitly with its own
API through a template. The Client can concentrate on his domain,
and not be concerned with infrastructure or resource handling.
* The analogy with <<<JmsTemplate>>> is even stronger. If the input
data came from JMS instead of a file, we would hardly have to do
anything to implement very robust chunking. JMS is the obvious best
practice and already provides all the transactional semantics we
need for chunking - simply roll back a transaction and the records
processed return to the message system for delivery to the next
consumer. Bad records can be sent to a bad message queue for
independent processing. JMS might ssem like overkill for a lot of
batch processes, but it is tempting to say that if the robustness is
needed then the we should take that as a sign that installing and
configuring JMS is worth the extra effort.
* Naturally we do not want to insist that the client code is aware
of the transaction that is surrounding it - this would be the normal
practice familiar from the Spring programming model. Should a
client need access to transaction-scoped resources, the usual way to
do that is to wrap the transactional resource (data source etc.) in
a proxy that uses a synchronization, or a more generic thread-bound
resource (using <<<TransactionSynchronizationManager>>>). The aim
is to retain this separation in a batch operation. The batch
framework itself might provide some of these synchronizations.
* The {{{simple.html}Simple Batch Repeat}} is actually a pretty good
model for the chunk processing in this use case. This observation
leads to another: that a batch of chunks is a nested (or composed)
batch - the outer termination policy is dependent only on the data
source having further records to process, the inner one is a simple
iterator (with a check for empty data). A simplified programming
model for this is
+---
RepeatCallback chunkCallback = new RepeatCallback() {
public boolean doInIteration(RepeatContext context) {
int count = 0;
do {
Object result = callback.doWithRepeat(context);
} while (result!=null && count++<chunkSize);
return result!=null;
}
});
batchTemplate.iterate(chunkCallback);
+---
The transaction boundary is demarcated at the chunk level
(<<<chunkCallback.doWithRepeat()>>>). Thw termination policy depends
only on a data source eventually returning null.
* N.B. the chunkSize can be dynamic. E.g., if the chunk is long
during a nightime batch window, and short when the window is over,
in case the batch has to be terminated.
* Chunking can also be implemented simply in an
<<<ExecutionHandler>>>. The handler just buffers records up to a
chunk size, and then executes them all in one step (which might be
transactional). This is easier to implement, and easier to
configure for the clients, but cannot easily be made both concurrent
and transactional.

89
src/site/apt/cases/file-to-file.apt Normal file
View File

@@ -0,0 +1,89 @@
------
Copy File to File
------
Dave Syer
------
January 2007
Use Case: Copy File to File
* Goal
Read a file line-by-line and process into a file in a different
format (possibly different number of lines). Commit periodically
and in the event of an error both data sources (input and output)
rollback to the last known good point.
* Scope
To keep things simple for now, assume that:
* All lines in the file are in the same format and the final
output is an aggregate.
* The files are read and written synchronously by a single
consumer.
* This use case requires two kinds of transactional file source.
One is read-only and the other is write-only. Only one consumer
can use the write-only source at a time.
* Preconditions
* An input file exists in the right format, with a sufficiently
large number of lines to be realistic.
* Success
Integration test confirms that
* All data are processed and output produced successfully.
* Description
Very similar to the use case {{{chunks.html}Copy File to
Database}}, but involving transactional access to an output source
which is a file. Also we are introducing the idea of an aggregate
function for the output.
The vanilla successful case proceeds as in the file to database
version, except that:
[[1]] A successful chunk results in a line in an intermediate file
output source.
[[1]] After all chunks are successfully processed the intermediate
file is itself processed in a single transaction to complete the
aggregate. The output is itself sent to an output channel
(e.g. database or file).
* Variations
* Chunk failure variations proceed as in the use case
{{{chunks.html}Copy File to Database}}. In the case of a
restart after fatal failure, the intermediate output file need does
not need to be reset or re-created.
* Implementation
* The write-only file source is new in this use case. It has a
similar flavour to the read-only version, but also has more serious
implications for implementation and usage. Since a file system is
not inherently transactional, when we create the write-only data
source we are assuming that consumers will play by the rules,
principally that there is only one consumer at a time.
* With some external limitations the write-only file source can be
implemented so that within a single JVM it will behave like a
transactional database datasource. We can provide a
<<<FileOutputTemplate>>> that hides the resource acquisition and
release, and interacts with an existing transaction to provide the
transactional behaviour that is required.
* File-based transactional resources are a lot like messaging
clients. We can send a message (write a line) through a sender
client, and receive a message (read a line) through a consumer
client. In the case of a transaction rollback, all sent messages
are guaranteed not to reach consumers, and all received messages are
returned to the queue. Maybe ActiveMQ has a file transport already?
Mule definitely does, but it isn't transactional.

101
src/site/apt/cases/index.apt Normal file
View File

@@ -0,0 +1,101 @@
------
Use Cases
------
Dave Syer
------
January 2007
Use Cases for Spring Batch
These are more like scenarios or flows than real use cases in formal
UML terms, but they serve a useful purpose as both. We don't want
to be over formal, and probably code is being written and tested at
the same time as these use cases. But there are many stakeholders
in this project, and use cases are a useful resource to make sure
they are all agreed on scope and certain implementation details.
* {{{simple.html}Simple Batch Repeat}}
* {{{retry.html}Automatic Retry After Failure}}
* {{{chunks.html}Commit Batch Process Periodically}}: chunk
processing.
* {{{async.html}Asynchronous Chunk Processing}}: parallel
processing within a chunk.
* {{{file-to-file.html}Copy File to File in a Batch}}
* {{{parallel.html}Massively Parallel Batch Processing}}
* {{{restart.html}Manual Restart After Failure}}
* {{{steps.html}Sequential Processing of Dependent Steps}}
* {{{partial.html}Partial Processing}}: skip records (e.g. on rollback).
* Whole-Batch Transaction - transactional support for the whole
batch, not just chunks. Quite a common requirement, but not
always practical using normal transaction support. May require a
staging area, and a decision after it is full about whether to
copy it in one big batch (e.g. using native database tools) or
chunk it (e.g. if it is now in a form for which chunk failure is
easier to deal with).
* {{{scheduled.html}Scheduled Processing}}: Batch Jobs controlled by scheduler (e.g. start, stop, suspend, kill)
* Actors
The following actors are involved in the use cases (Container and
Client being the most common / important).
** Client or Business Domain
Code written by the batch developer.
One aim us that the client is a POJO - the batch behaviour, boundary
conditions, transactions etc. can be dealt with by the Container in
such as way that the client does not need to know about them. The
client may have access to framework abstractions, like templated
data sources (<<<JdbcTemplate>>> etc.), but these should work the
same whether they are in a batch or not.
** Container
An application that converts user requests for batch jobs into
running processes. Container concerns are robustness, traceability,
manageability.
** Framework
The Framework is the infrastructure code that the Container depends
on, and possibly spi implementations where knowledge of the
non-business logic resides.
The Framework provides two kinds of infrastruture (as per usual
Spring cornerstones <AOP> and <Portable Service Abstractions>):
* For cross-cutting concerns there are interceptors that can be
wrapped around client code without it needing any knowledge of the
Framework at all. An existing parallel is with transaction
support - the client code can use <<<TransactionTemplate>>>
directly, but does not always need to.
* Concrete abstractions that allow access to resources in a
uniform way without needing to know the details of how they are
provided (e.g. partitioned). Client code can use these
abstractions like it would a use a <<<DaoSupport>>>.
** Operator
The batch operator is not a developer. Tools are provided for the
Operator to be able to stop and start a batch, and to monitor the
progress and status of on ongoing or finished batch.
** Business User
The Operator has technical skills, e.g. a member of an application
support team, but may need help with business-related decisions.
For instance if input data are bad, he would not expect to be able
to fix them alone because they might be bad for a business reason.

239
src/site/apt/cases/parallel.apt Normal file
View File

@@ -0,0 +1,239 @@
------
Parallel Processing Use Case
------
Dave Syer
------
January 2007
Use Case: Massively Parallel Batch Processing
* Goal
Support efficient processing of really large batch jobs (100K -
1000K records) through parallel processing, across multiple
processes or physical or virtual machines. The goals of other use
cases should not be compromised, e.g. we need to be able to start
and stop a batch job easily (for non developer), and trace the
progress and failure points of a batch. The client code should not
be aware of whether the processing is parallel or serial.
* Scope
* Any batch operation that reads data item-by-item from an input
source is capable of being scaled up by parallelizing.
* The initial implementation might concentrate on multiple threads
in a single process. Ultimately we need to be able to support
multiple processes each one running in an application server (so
that jobs that require EJBs can be used).
* Preconditions
* A data source with multiple chunks (commitable units) - more chunks
than parallel processes.
* A way for the container to launch parallel processes.
* Success
* A batch completes successfully, and the results are verified.
* A batch fails in one of the nodes, and when restarted processes
the remaining records.
* Description
[[1]] Container splits input data into partitions.
[[1]] Container sends input data (or references to them) to
processing nodes.
[[1]] Processing nodes act independently, converting the input data
and sending it transactionally to output source (as per normal
single process batch).
[[1]] Container collects status data from individual nodes for
reporting and auditing.
[[1]] When all nodes are complete Container decides that batch is
complete finishes processing.
* Variations
Two failure cases can be distinguished, bad input data on a node and
an internal node failure have different implications for how to
proceed. In both cases, however
[[1]] Container catches exception and classifies it. Rolls back
current transaction to preserve state of data (input and output).
[[1]] Container saves state for restart from last known good
point, including a pointer to the next input record.
Then if a processing node detects bad data in the input source, it
cannot be restarted or re-distributed because the data need to be
modified for a successful outcome.
[[1]] Container alerts Operator of the location and nature of the
failure.
[[1]] Operator waits for batch to finish - the overall status will
be a failure, but most of the data might be consumed.
[[1]] Operator fixes problem and restarts batch.
[[1]] Container does not re-process data that has already been
processed successfully. The parallel processing nodes are used as
before.
[[1]] Batch completes normally.
If a processing node fails unrecoverably (e.g. after retry timeout),
but with no indication that the input data were bad, then the data
can be re-used: Container returns unprocessed input data, and
redistributes it to other nodes.
* Implementation
* The hard thing about this use case is the partitioning of input
(and output) sources. This has to be done in such a way that the
individual operations are unaware that they are participating in a
batch farm. Partitioning has to be at least partially deterministic
because restarts have to be able to ignore data that have already
been processed successfully.
Consider two examples: a file input source and a JDBC (SQL query)
based input source. Each provides its own challenges.
** File Data Source Partitioning
* If each node reads the whole file there could be a performance
issue. They would all need to have instructions about which lines
to process.
* If each record of input data is a line, this isn't so bad. Each
node can have a range of line numbers to process. The only problem
is knowing how many lines there are, and how many nodes, so that the
job can be partitionaed efficiently.
* But if each input record can span a variable number of lines (not
that unlikely in practice), then we can't use line numbers
* Maybe the best solution is to have a single process parsing the
file and sending it to a message queue, either formally using a
messaging infrastructure or informally using some sort of
roll-your-own approach. The integration pattern could then be a
simple Eager Consumer, assuming that all records are processed
independently. The messaging semantics would simply have to ensure
that a consumer can roll back and return the input records to a
queue for another consumer to retry.
For large batches a real messaging infrastructure (JMS etc.) with
guaranteed delivery would be a benefit, but might be seen as
overkill for a system that didn't otherwise require it. In this
case we could imagine the partitioning process being one of simply
dividing the input file up into smaller files, which are then
processed by individual nodes independently. The integration
pattern is then different - more like a Router.
* What would parallel processing look like to the client? We can
make it completely transparent if we assume that the client only
ever implements <<<ItemProvider>>> and <<<ItemProcessor>>>. The
client code is unaware of the partitioning of its data source:
+---
batchTemplate.iterate(new ItemProviderRepeatCallback(provider, processor));
+---
* Parallelisation could also take place at the level of the
<<<ItemProvider>>> - we could proxy the data provider and wrap it in
a partitioning proxy:
+---
<bean id="itemProvider"
class="org.springframework.aop.framework.ProxyFactoryBean">
<property name="target">
<bean class="test.input.TradeItemProvider">
...
</bean>
</property>
<property name="interceptorNames" value="partitioner"/>
</bean>
<bean id="partitioner"
class="org.springframework.core.batch.support.provider.PartioningInterceptor">
...
</bean>
+---
** SQL Data Source Partitioning
* If each node is allowed to do its own query or queries to
determine the input data:
* Each node has to be given a way to narrow the query so that they
don't all use the same data. There is no easy universal way to
achieve this, and in the general case we have to know in advance
when we are going to execute in a parallel or as a single process.
Maybe a range of primary keys would work as a special case that we
could support as a strategy.
* Maybe we could assume that all nodes execute precisely the same
query, and then provide a way to add a cursor to the result set,
so it can be treated a bit more like a file.
* We might be forced to use a distributed transaction to ensure
that all the nodes see the same data. This would be unfortunate,
but possibly necessary. It would be up to the client to configure
distributed transactions if that was required, otherwise the
result might be unpredictable if data can be added to an input
source while it is being read.
* If only one query is done by the Container and the results shared
out amongst the nodes we face the issue of how to send the data
between nodes. Performance problems might ensue. Plus (more
seriously) the individual nodes would now need a different
implementation if they were acting in a parallel cluster to the
vanilla serial processing case - a single node would do the query
and work directly with the results, whereas in a parallel
environment it would be one step removed from the actual query.
This breaks our encapsulation design goal.
* When considering the approach to partitioning the data source
we should follow closely the discussion above on partitioning a file
input source. If the client is to remain unaware of the batch
parameters, then an interceptor looks like the best approach.
If each node prefers to do its own query then an interceptor would
have to catch the call to a JDBC template and modify the query
dynamically. This is quite a scary thing to be doing - it might end
up with us needing to parse the SQL and add where clauses. Maybe a
client should be forced to specify (in the case of a parallel batch)
how his query should be partitioned. For example:
+---
<bean id="inputSource"
class="test.input.SqlInputDataProvider">
<property name="query">
<value>SELECT * from T_INPUT</value>
</property>
<property name="partitionQuery">
<value>SELECT * from T_INPUT where ID>=? and ID<?</value>
</property>
</bean>
+---
It would be an error to run a batch in parallel if the partition
query had not been provided.
* What happens if the data source changes between a batch and
restart? We can't legislate for that because it is outside the
realm of what can be controlled through a transaction. A restart
might produce different results than the original failed batch would
have done were it successful.

154
src/site/apt/cases/partial.apt Normal file
View File

@@ -0,0 +1,154 @@
------
Partial Processing Use Case
------
Dave Syer
------
January 2007
Partial Processing
* Goal
Support partial processing of a batch, without having to interrupt
or manually restart, but enabling corrective action to be taken
after the process has finished to complete the processing of failed
records. A batch that is going to fail completely can be be
identified as soon as possible, but one which is substantially
alright can run as far as possible to prevent costly duplication.
Records that are skipped are reported in such a way that they can be
easily identified by the Operator and / or Business User and a new
batch created to finish the original goal. By the same token, in
the case of an aborted batch where a minority of records are
processed successfully first time, it should be possible to identify
the successful records and exclude them from data presented on
restart.
* Scope
Any batch should be configurable to support partial processing.
* Preconditions
* A data source with a small number of bad records exists.
* Success
* A test data set with a small number of bad records is run through
the batch processer and completes normally. Operator confirms
that the good recirds are all processed and then fixes and
resubmits the bad records, and confirms that they are also
correctly processed with no duplicates.
* Description
The vanilla flow proceeds as follows:
[[1]] Batch processing begins as per normal (see for example
{{{chunks.apt}chunk processing use case}}).
[[1]] A record is processed. This step repeats until...
[[1]] Container detects a bad record, e.g. by catching a
classified execption.
[[1]] Container logs the exception in a way that identifies the
bad record easily and immediately to the Operator.
[[1]] Container stores an identifier for the bad record (or the
whole record) in a location designated to the Operator for that
purpose.
[[1]] Container determines that the batch can still succeed
despite the cumulative number or nature of bad records - the bad
record is skipped. Container goes back to normal processing, and
eventually completes the whole batch.
* Variations
** Abort Batch Early
The batch cannot skip all records. After each failure the decision
about whether to coninue has to be made:
[[1]] When a record is processed successfully, Container logs the
event in a form that can be used later to identify successful
records in case the batch is aborted.
[[1]] Container determines that a sufficiently large fraction of
the records processed so far have failed. The faction relevant is
to be specified through configuration meta data (not specified by
business logic).
[[1]] Container aborts the batch with a clear signal to the
Operator that it has aborted owing to an unacceptable number of
errors.
* Implementation
* When the decision to abort is taken, Container may have
successfully processed a small number of records and the
corresponding transactions might have committed. Those records that
were successfully processed on the first attempt are easy to
exclude from the restart, if transactional semantics are respected
by the item processing.
* The decision to abort is based on exception classification. Each
time an item is processed, the framework needs to catch exceptions
and classify them as
* fatal: signals an abort - rethrow.
* transient: nominally fatal, but the operation is retryable.
* non-fatal: signals a skip.
The transient failure is really just a sub-type of fatal case. It
is treated differently by the {{{retry.html}retry framework}} but
not necessarily by the vanilla batch.
* Actually we can't decide what action to take simply on the
evidence of the current exception. What we need to do is decide,
potentially based on the whole history of exceptions in a given
batch, whether the latest one should trigger an abort. E.g. a
simple and sensible policy would be to abort if the total number of
exceptions reaches a threshold, either absolute or relative to the
number of items processed.
* So how does it look? In the template...
+---
public void iterate(RepeatCallback callback) {
...
try {
result = callback.doInIteration(context);
} catch (Exception e) {
handleException(e); // Maybe re-throw, maybe not...
}
...
}
+---
If the callback was transactional it has already rolled back. If
the whole <<<iterate()>>> was transactional we need to rethrow
* If the processing is asynchronous, the template has to execute in
a separate thread (see {{{async.html}asynchronous example}}). In
this case the whole thread (i.e. the <<<iterate()>>>) has to be
transactional. Whoever is counting failed items needs to be
poooling information from multiple threads.
* It may also be the role of the framework to translate exceptions
into a batch-specific hierarchy. This is not the same concern as
exception classification (as done for instance by the Spring Jdbc
and Jms templates). Exception classification might also be of
value, but the argument is not as clear cut as the existing core
templates, where there is an underlying Jave EE API checked
exception to convert. In the absence of a batch-specific exception
hierarchy definition, we could choose to leave exception translation
out of the batch framework.

86
src/site/apt/cases/restart.apt Normal file
View File

@@ -0,0 +1,86 @@
------
Restart Use Case
------
Dave Syer
------
January 2007
Use Case: Manual Restart After Failure
* Goal
Restart a failed or interrupted batch and have it pick up where it
left off (within limits of transaction boundaries) to save time and
resources. A key goal is that the management of the batch process
(locating a job and its input and results, starting, scheduling,
restarting) should be as easy as possible for a non-developer, like
an application support team with some business back up.
* Scope
Any batch should be able to restart gracefully, even if (depending
on chosen container or client implementation) it might have to go
right back to the beginning.
* Preconditions
* It is possible to identify exception conditions under which a
restart will be able to carry on processing a batch from where it
left off.
* There exists a presistent storage mechanism for the initial
conditions.
* Success
* Force a batch to fail, and then fix the problem and restart. See
successful completion with no duplicate results.
* Description
[[1]] A batch operation encounters an exception which forces the
process to stop processing.
[[1]] Container catches exception and classifies it.
[[1]] Container logs event with enough information to identify the
location of the job and the nature of the problem.
[[1]] Container saves initial condition from last commit point, to
enable restart to start from the last known good operation.
[[1]] Operator fixes problem (e.g. makes missing resource available,
edits input file).
[[1]] Operator restarts batch.
[[1]] Container loads initial conditions and continues processing.
* Variations
* Some restarts might lend themsleves to being handled automatically
- see the use case {{{retry.html}Automatic Retry}}.
* Implementation
* The saving of initial conditions needs to be strategised. In some
cases saving a native serialization to a file will suffice. In
others a database might be used, or some custom serialization
(persist / rehydrate).
* The initial condition is naturally under control of the
<<<DataProvider>>>. The client need not know about the persistence
and rehydration. In fact explicit persistence and rehydration might
be overkill - just relying on the transaction semantics might be
adequate in a lot of cases. The <<<DataProvider>>> would have to be
aware of the transactions, which we assume are normally demarcated
in the <<<ExecutionHandler>>>. Since the point at which persistence
is needed is tied to transaction commits, there may have to be some
transaction synchronization.
* The persistence of initial conditions is a cross cutting concern.
It may lend itself (along with the application of an execution
handler generally) to being implemented as an aspect. Compare the
<<<TransactionTemplate>>>, where the most common usage is via an
interceptor, but occasionally the template is used directly by
client code.

279
src/site/apt/cases/retry.apt Normal file
View File

@@ -0,0 +1,279 @@
------
Automatic Retry Use Case
------
Dave Syer
------
January 2007
Use Case: Automatic Retry
* Goal
Support automatic retry of an operation if it fails in certain
pre-determined ways. Client code is not aware of the details of
when and how many times to retry the operation, and various
strategies for those details are available. The decision about
whether to retry or abandon lies with the Framework, but is
parameterisable through some retry meta data.
Retryable operations are usually transactional, but this can be
provided by a normal transaction template or interceptor
(transaction meta data are independent of the retry meta data).
* Scope
Any operation can be retried, but there are restrictions on nesting
transactions (normally an inner transaction needs to be
propagation=NESTED).
* Preconditions
An operation exists that can be forced to fail and is able to
succeed on a retry.
* Success
* Verify that an operation fails and then succeeds on a retry.
* Verify that back off policy (time between retries) can be
strategised without changing client code.
* Verify that the retry policy can be strategised, and can be used
to change the number of retry attempts depending on the type of
exception thrown in the retry block.
* Description
Successful retry proceeds as follows:
[[1]] Framework executes an operation provided by Client.
[[1]] The operation fails and Framework catches an exception,
classified as retryable.
[[1]] Framework waits for a pre-defined back off period. The
period is not be fixed, but is strategised so that different
policies can be applied. The most common and useful policy is an
exponentially increasing back off delay, with a ceiling.
[[1]] Framework repeats the operation.
[[1]] Processing is successful.
[[1]] Framework stores and / or logs statistics about the retry
for management purposes. Details?
* Variations
The following variations are supported.
** Retry Failure
A retry can fail for a number of reasons. E.g. if the number of
retries is too high, or there is a timeout, or an exception of
another sort that cannot be classified as retryable.
[[1]] Last retry attempt fails and Framework determines that
another retry is not permitted by the current policy.
[[1]] Framework records status for management purposes.
[[1]] Framework throws a recognisable exception?
[[1]] Control may return to client (if the exception was caught),
or the processing may end.
** Transient and Non-transient Failures
We may wish to classify exceptions into (at least) three types, and
vary the retry policy based on the classification:
* Transient failures come from resources that are external and may
have independent lifecycles to the client process. Examples are
database deadlock, network connectivity. It is always worth
retrying on a transient failure, and normally we can keep retrying
(if not forever then for a very long time), in the belief that
eventually the resource will become available again.
* Non-transient failures can be retried a few times. This is the
default.
* Non-retryable failures like a configuration or input data error
should not be retried (they will always fail the same way).
** Early Termination
Normally client code is unaware of the Framework, but occasionally
emergency measures might be taken inside client code where all
further retry attempts are vetoed for the current block.
** Stateful Retry
A stateful (or external) retry is used to force a roll back of an
external message (or other data) resource, so that the message will
be re-delivered. The implementation has to be stateful so it can
remember the context for the failed message next time it is
delivered. The additional features of a stateful retry, as opposed
to a normal rollback, are that:
* A message can be retried indefinitely or up to a set number of
times, after which an error processing route is taken.
* A back-off delay is used at the <beginning> of the retry
before any other transactional resources are enlisted.
* Implementation
* The vanilla case and most of the variations can be achieved with a
simple template approach:
+---
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setRetryPolicy(new SimpleRetryPolicy(5));
Object result = retryTemplate.execute(new RetryCallback() {
public Object doWithRetry(RetryContext context) throws Throwable {
// do some processing
return result;
}
});
+---
* Schematically we can represent the implementation of the [retry}
template as follows:
+---
1 | TRY {
1.1 | do something;
2 | } FAIL {
2.1 | if (retry limit reached) {
2.2 | rethrow exception;
| } else {
2.3 | TRY(1) again;
| }
| }
+---
* The template has policies for back off and retry (whether or not
to retry the last exception). The example above shows the retry
policy being set to simply retry all exceptions up to a limit of 5
times.
* The <<<RetryContext>>> has an API that allows clients to override
the retry policy. The context can also be accessed as a thread
local from a static convenience class, in the case that the callback
is implemented as a wrapper around a POJO.
* External retry is the most difficult variation to implement, and
doesn't fit naturally into the template model above. Two things
depend on the retry count - back-off delay and the decision to
follow the recovery path - so it needs to be available at the
beginning of every processing block.
We will discuss the implementation from a JMS-flavoured viewpoint,
where the current item being processed is a message. This can be
generalised to more generic data types, as long as the item can be
rejected transactionally to signal that we require it to be
re-delivered to this or another consumer.
Consider this pattern, which is very typical:
+---
1 | SESSION {
2 | receive;
3 | RETRY {
| remote access;
| }
| }
+---
A <<<RetryTemplate>>> is responsible for the RETRY(3) block. But
we can't put the same wrapper around the whole process:
+---
0 | RETRY { // Do not do this!
1 | SESSION {
2 | receive;
3 | RETRY {
| remote access;
| }
| }
| }
+---
because the receive(2) might not get the same message back on the
second and subsequent attempts (another consumer might get it, or it
might come out of order). So external retry has a different flow -
it might be a different implementation of the same interface, or a
different parameterisation of the normal retry template.
We can break down the implementation of an external retry into steps
as follows:
+---
1 | SESSION {
2 | receive;
3 | TRY {
3.1 | if (already processed) {
3.2 | backoff;
| }
4 | RETRY {
| remote access;
| }
5 | } FAIL {
5.1 | if (retry limit reached) {
5.2 | recover;
| } else {
5.3 | rethrow exception;
| }
| }
| }
+---
Decisions (3.1) and (5.1) require knowledge of the history of
processing the current message. Note that the action on failure is
the opposite to the vanilla case {{{#retry}above}} - if the retry
limit is not reached then we rethrow the exception.
If the retry limit is not reached then the rethrow(5.3) causes the
SESSION(1) to roll back, and the message will be re-delivered.
RETRY(4) is a normal retry with a template.
The retry logic is easy to implement - the hard bit is that the
policies depend on the history of the message. This requires some
special retry and back off policies that are aware of the history:
* When a message arrives, at the beginning of the TRY(3) above, we
need to update our knowledge of its history.
* The backoff policy can decide whether to back off immediately
when it is initialized at step (3.1).
* The retry decision at (5.1) has to be aware of the history as
well as some simple exception classification rules.
* If the retry cannot proceed the retry policy can take steps to
recover (5.2), e.g. send the current message to an error queue.
The exception should not propagate in this case.
* If we fail and rethrow (5.3), then we need to store the
knowledge of the message history somewhere where another consumer
can access it.
There is a small conundrum about what value to return from the
TRY(3) block if it ultimately fails (5.2) - a normal retry never
completes unless it is successful, but an external retry can
complete if it is unsuccessful. The obvious choice is to return
null. It probably won't matter in a messaging application anyway
because the client of the retry block probably isn't expecting
anything. It may matter if the TRY(3) block is part of a batch
because the batch template uses null as a signal that the current
batch is complete. But on the other hand it might be a good
strategy to close the batch if processing a message fails.
With JMS there is no indication in the <<<Message>>> how many times
it has been rejected - only a flag <<<getJMSRedelivered>>> to show
that it has failed at least once. To count the number of retries,
we have to store a global map of messages (ids) to retry counts
(within a single VM - for more than one OS process each one has to
be independent).

34
src/site/apt/cases/scheduled.apt Normal file
View File

@@ -0,0 +1,34 @@
------
Scheduler Managed Use Case
------
Wayne Lund
------
May 2007
Use Case: Scheduler Managed Processing
* Goal
Ensure that an Enterprise Scheduler can interact with the Batch Launcher to start, stop,
suspend and/or kill a batch job.
* Scope
* Batch jobs tends to run within carefully planned job stream schedules. At a minimum this requires
an integration between the Batch Launcher (in the abstract) and the scheduler's control mechanism to
start and stop batch jobs and then to understand the results of the batch job execution
(e.g. COMPLETED, ABENDED, etc.) so that subsequent actions may be taken.
* Preconditions
* A mechanism has been established for the scheduler to launch a batch job. This is often times
a simple unix or dos shell script.
* A mapping of exit codes to the error code numbers that the scheduler is expecting on the exiting
of a batch job.
* Success
* Batch Jobs are launched and managed by scheduler
* Description

290
src/site/apt/cases/simple.apt Normal file
View File

@@ -0,0 +1,290 @@
------
Simple Batch Repeat Use Case
------
Dave Syer
------
January 2007
Use Case: Simple Batch Repeat
* Goal
Repeat a simple operation such as processing a data item, or a
message, up to a fixed number of times, normally with a transaction
scoped to the whole batch. Transaction resources are shared between
the operations in the batch, leading to performance benefits.
* Scope
The operation to be repeated:
* Can expect to use and manage its own I/O or datastore resources,
but not necessarily transactions;
* May need to introspect the batch status (as a variation);
* Executes synchronously or asynchronously (as a variation).
* Is stateless - this is not a framework restriction in principle,
but simplifies the implementation for now. See in the
{{{#store}Implementation}} section below for some notes on
stateful synchronisation;
* Should be implementable as a POJO if desired.
* Preconditions
Client code can locate and acquire all the resources it needs for
the batched operation, and can force transactions to rollback for
testing purposes.
* Success
* Verify that a successful batch executed a fixed number of times.
* Verify that a batch completes early but successfully if an
underlying transaction times out.
* Terminate a batch by failing one of the operations, and verify
that the preceding operations rolled back (subject to batch meta
data).
* Execute a batch asynchronously and verify that the correct number
of operations is performed.
* Description
We are often interested in a specific scenario of this use case
where the batched operation is:
* Read a message or data item from an endpoint like a JMS
Destination.
* Do some business processing involving database reads and writes.
The vanilla successful batch use case proceeds as follows:
[[1]] Framework starts a batch, acquiring resources as needed and
creating a context for the execution.
[[1]] Client provides a batch operation in the form of a source of
data items and a processor acting on the data item.
[[1]] Framework executes batch operation.
[[1]] Repeat the last step until the batch size is reached.
[[1]] Framework commits the batch. All database changes are
committed and received messages removed from the endpoints.
* Variations
** Rollback
If one of the operations rolls back it will throw an exception.
Normal transaction semantics determine what happens next. Usually
(in the scenario described above) there is an outer transaction for
the whole batch, which rolls back as well: all the messages remain
unsent, and all the data remain uncommitted. A retry will receive
exactly the same initial conditions.
** Timeout
The batch size is not fixed. The use case proceeds as above, but in
the middle of a batch operation execution:
[[1]] Framework determines that the batch has timed out operation
(e.g. while it was waiting for an incoming message).
[[1]] Framework commits the batch with all operations so far
complete - possibly a smaller than normal size.
** Asynchronous Processing
Instead of the Framework waiting for each operation to complete it
could spin them off independently into separate threads or a work
queue. The batch still has to have a definite endpoint, so the
Framework waits for all the operations to finish or fail
before cmpleting the batch.
** Introspection of Batch Context
Client may wish to inspect the state of the ongoing batch operation,
and potentially force an early completion.
* {Implementation}
* The completion of the batch loop is handled by a policy delegate
that we can use to strategise the concept of a loop that might
complete early. This can cover both the timeout variation and the
vanilla use case flow.
* What form should the batch template (<<<RepeatOperations>>>)
interface take? We might start with something like this:
+---
batchTemplate.iterate(new RepeatCallback() {
public boolean doInIteration() {
// do stuff
}
});
+---
* A nice tool for a batch operation in a callback is an iterator
through a data set or message endpoint (<<<ItemProvider>>>), coupled
with a handler for processing the item. This adds a potential
implementation of <<<RepeatCallback>>> that knows about the
<<<ItemProvider>>> and adds a processor object. E.g. as an
anonymous inner class:
+---
final ItemProvider provider = new JmsItemProvider();
final ItemProcessor processor = new ItemProcessor() {
public void process(Object data) {
// do something with the data (a record)
}
};
batchTemplate.execute(new RepeatCallback() {
public boolean doInIteration() {
Object data = provider.next();
if (data!=null) {
processor.process(data);
}
return data!=null;
}
});
+---
* Is a batch template with callback the best implementation? Could
we perhaps use or re-use <<<TaskExecutor>>> somehow? Which is
better for the client:
+---
batchTemplate.iterate(new RepeatCallback() {
public boolean doInIteration() {
// do stuff
}
});
+---
where the batch template might itself use a <<<TaskExecutor>>>
internally, or
+---
batchTemplate.iterate(new Runnable() {
public void run() {
// do stuff with data
};
});
+---
where the batch template is a <<<TaskExecutor>>>. Probably the
former because it is more encapsulated: it gives the framework more
freedom to implement the template in any way it needs to, e.g. to
accommodate more complicated use cases.
* To {store} up SQL operations until the end of a batch, and take
advantage of JDBC driver efficiencies, the client needs to store
some state during the batch, and also register a transaction
synchronisation. For this kind of scenario we introduce an
interceptor framework in the template execution. The template calls
back to interceptors, which themselves can strategise clean up and
close-type behaviour:
+---
public class RepeatTemplate implements RepeatOperations {
public void iterate(RepeatCallback callback) {
// set up the batch
interceptors.open();
while (running) {
// allow interceptor to pre-process and veto continuation
interceptor.before();
// continue only if batch is ongoing
if (running = callback.doInIteration()!=null) {
interceptor.after();
}
}
// clean up or commit the whole batch
interceptor.close();
}
}
+---
The <<<RepeatInterceptor>>> can be stateful, and can store up inserts
until the end of the batch. If the <<<RepeatTemplate.iterate>>> is
transactional then they will only happen if the transaction is
successful.
This way the client can even decide to use a batch interceptor
that runs in its own transaction at the end of the batch.
* There is no need for an overall batch timeout because the inner
operations are synchronous and have their own timeout metadata
though transaction definitions. The whole batch (outer transaction)
may still have a timeout attribute, and then there is a corner case
where the batch operations are all successful, but because they all
took a long time the whole batch rolls back because of the timeout.
* The context of the ongoing batch is closely linked with the
completion policy. The completion policy is pluggable into the
batch template, and acts as a factory for context objects which can
then be inspected by Client in the callback. For example:
+---
public class RepeatTemplate implements RepeatOperations {
public void iterate(RepeatCallback callback) {
// set up the batch session
RepeatContext context = completionPolicy.start();
while (!completionPolicy.isComplete(context)) {
// callback gets the context as an argument
callback.doInIteration(context);
completionPolicy.update(context);
}
}
}
+---
* The example above provides Client the opportunity to inspect the
context through the callback interface. If Client is a POJO,
Framework has to create a callback and wrap it, in which case there
needs to be a global accessor for the current context or session.
The template is then responsible for registering the current context
with a <<<RepeatSynchronizationManager>>>. E.g.client code can look
at the session and mark it as complete if desired
(c.f. <<<TransactionStatus>>>):
+---
public Object doMyBatch() {
// do some processing
// something bad happened...
RepeatContext context = RepeatSynchronizationManager.getContext();
context.setCompleteOnly();
}
+---

176
src/site/apt/cases/steps.apt Normal file
View File

@@ -0,0 +1,176 @@
------
Batch: Sequential Steps Use Case
------
Dave Syer
------
January 2007
Use Case: Sequential Processing of Dependent Steps
* Goal
Compose a batch operation from a sequence of dependent steps.
Define and implement the operation only once, and allow restart
after failure without having to change configuration, and without
having to repeat steps that were successful.
A sub-goal is to allow the progress of a batch through the steps to
be traced accurately for reporting and auditing purposes. This
requires the steps to be uniquely identified.
* Scope
* Simple linear sequence of steps. Slightly more complicated
requirements can be handled by putting independent steps in a
sequence (no need for splits and joins).
* Preconditions
* A non-trivial sequence is defined:
* more than one step:
* the effects of each step can be measured.
* The sequence can be interrupted or artificially terminated in the
second or subsequent step.
* Success
* A non-trivial sequence executes successfully. The progress and
success of each step can be verified by the tester.
* The same sequence is forced to fail on second step in such a way
that the first step result is not suspected of being in error,
e.g. by interrupting it. When it is restarted the first step is not
repeated, and the sequence is successful.
* The same sequence is forced to fail on second step in such a way
that the first step result is obviously in error, even though it
completed normally. When the batch is restarted the first step <is>
repeated, and the sequence is successful.
* Description
The vanilla successful case proceeds as follows:
[[1]] Container logs the start of a step, uniquely indentifying
the initial conditions.
[[1]] Container stores internal state so that initial conditions
can be re-created in the event of a restart.
[[1]] Step execution proceeds as per one of the other use cases
(e.g. {{{file-to-database.html}Copy File to Database}}), including
transactional behaviour.
[[1]] Client instructs Container to store internal state needed by
further steps (e.g. cached reference data).
[[1]] Container logs successful completion of step, and stores
[[1]] Repeat for next and subsequent steps. Internal state is
passed from one state to the next.
* Variations
** Internal Failure of Step
If a step fails internally, e.g. because of resource becoming
temporarily unavailable, the sequence can be restarted without
repeating the previous steps.
[[1]] Operator fixes resource problem (e.g. starts web service).
[[1]] Operator restarts batch with no configuration or input data
changes.
[[1]] Container resumes batch from the last commit point of the
failed step.
[[1]] Sequence completes normally.
The process above could be carried out by the container entirely (no
need for operator intervention) if a retry policy is in effect.
** Failure of Step Owing to Bad Initial State
If a step fails because it receives bad data from an earlier step,
the Container cannot recover without intervention.
[[1]] Operator attempts to restart without doing anything to fix
the problem.
[[1]] Container detects bad initial state immediately and fails
fast.
If the original problem can be located and fixed (e.g. input data
for earlier step is revised):
[[1]] Operator restarts batch signalling to container which step
to begin with.
[[1]] Container locates initial state for the first step to be
executed.
[[1]] Container starts execution from the beginning of the desired
state. This time the input data are different, so the sequence
can complete normally.
* Implementation
* The need to save state for subsequent steps leads to the
introduction of a batch context concept. And the need for
initialising restarts leads to the context being serializable,
either natively or by some pluggable strategy (this is covered in
the {{{restart.html}Restart after Failure}} use case).
Unfortunately, the need for {{{parallel.html}parallel processing}}
and automatic {{{restart.html}restart}} also makes it practically
impossible for steps to handle the context at the level of a single
thread of execution, where the client needs to implement business
logic. If a step is executing in parallel, then each node needs to
be able to restart independently, but the context needs to be a
single object that can be passed on to the next step (unless all the
steps are parallelised with the same multiplicity, which might not
be efficient in general).
Thus batch context must be defined and managed by the template or
execution handler.
* The requirement for steps might have implications for the
implementer of the batch operation (the client). Obviously a client
defines the sequence of steps according to the business requirement,
but ideally we would like him to be unaware of the reporting and
restart infrastructure. Maybe an array of callbacks works (the
callback interface is irrelevant, except that it accepts a context
object as an argument):
+---
batchTemplate.iterate(new RepeatCallback[] {
new RepeatCallback() {
public boolean doInIteration(RepeatContext context) {
// do stuff for step one
};
},
new RepeatCallback() {
public boolean doInIteration(RepeatContext context) {
// do stuff for step two - the context
// is the same...
};
}
});
+---
Notice that there is no need for the context to be set explicitly
before executing the callback. The context is handled internally to
the batch template using an analogue of the
<<<TransactionSynchronizationManager>>>.
* If we prefer that clients never need to know about batch
templates, then the code above needs to be automated. This would be
where an additional domain layer might come into play
(c.f. <<<Step>>>).

22
src/site/apt/cases/template.apt Normal file
View File

@@ -0,0 +1,22 @@
------
Template Use Case
------
Dave Syer
------
January 2007
Use Case: Template
* Goal
* Scope
* Preconditions
* Success
* Description
* Variations
* Implementation

7
src/site/apt/changelog.apt Normal file
View File

@@ -0,0 +1,7 @@
Changelog: Spring Batch
See the individual subprojects for their changelogs:
* {{{spring-batch-infrastructure/changelog.html}Infrastructure}}
* {{{spring-batch-container/changelog.html}Container}}

152
src/site/apt/features.apt Normal file
View File

@@ -0,0 +1,152 @@
------
Spring Batch Features
------
Dave Syer
------
July 2007
Spring Batch Features and Roadmap
* 1.0 Features
The following features are supported by Spring Batch 1.0:
** Optimisation and Infrastructure
* RepeatOperations: an abstraction for grouping repeated
operations together and moving the iteration logic into the
framework.
* RetryOperations: an abstraction for automatic retry.
* InputSource abstraction and implementations for flat files, xml
streaming and simple database queries.
* Flat files are supported with fixed length and delimited records
(input and ouput).
* Xml is supported through Xstream mapping between objects and Xml
elements (input and ouput).
* A database input source is provided that maps a row of a ResultSet
identified by a simple (single column) primary key.
* OutputSource abstraction and implementations for flat files and
xml (the Sql case is just a regular Jdbc Dao).
* InputSource and OutputSource implementations are generally
Restartable and Skippable. Skippable means that they can be asked by
clients to mark items as skipped, and not provide or process them
next time they arrive.
* Complementary to InputSource and OutputSource is a higher-level
abstraction layer with ItemProvider and ItemProcessor. Some
specialised concrete retry and repeat strategies have dependencies
on ItemProvider and/or ItemProcessor.
** Core Domain
* JobConfiguration is the root of the core domain - it is what most
developers and operators will be happy to call a "job": a recipe for
how to construct and run a JobInstance.
* A JobConfiguration is composed of a list of StepConfigurations
(sequential step model for job).
* StepConfiguration is a wrapper for a "unit of work", otherwise
known as a Tasklet (formerly Module).
* JobExecutor is the entry point for launching a JobConfiguration.
* StepExecutor is the corresponding point for a StepConfiguration.
StepExecutor is the main strategy for different scaling,
distribution and processing approaches. The 1.0 release contains
implementations for in-process execution (single VM). See below.
** Job Execution and Management
* A simple JobExecutorFacade to launch jobs. Start a new one or
restart one that has previously failed. The facade can be used by a
command-line or JMX launcher to take simple input parameters and
convert them to the form required by the Core.
* Persistence of job meta data for management and reporting
purposes: job and step identifiers, commit counts, task counts,
statistics (a human readable represenation of the state of the job -
can be augmented by developers).
* ItemProviderProcessorTasklet - uses an ItemProvider to obtain the
next record to process, and hands it to an ItemProvider if it is not
null.
Developers are encouraged to use the ItemProviderProcessorTasklet
rather than implementing their own, because this is the
implementation that in future versions of Spring Batch will be able
to adapt to different deployment architectures, and take advantage
of automatic scaling up through distributed processing.
* A StepExecutor (SimpleStepExecutor) that can run a
StepConfiguration in the same process (VM).
* Concurrent execution of chunks (a chunk is a batch of items
processed in the same transaction) through the Spring TaskExecutor
abstraction.
* Additional StepExecutor implementation that is aware of whether
its task is Recoverable - take recovery action on error.
* Automatic retry of a chunk and recovery for items that have
exhausted their retry count.
* Translation of job execution result into an exit code for
schedulers running the job as an OS process.
** Samples
* A range of samples is available as a separate module. They all
use a common simple configuration and extend in various ways to show
the different features of the Execution module.
** Partial Support or Potentially Unstable APIs
A milestone is a milestone, so we are going to continue refining the
APIs until we get to a release candidate. Hopefully most of the
developer touch points are functionally pretty stable, even if the
names and packages might still change. Partial implementations or
areas currently known to be undergoing refactoring as of
1.0-m2-SNAPSHOT are listed in JIRA
(http://opensource.atlassian.com/projects/spring/browse/BATCH) for
items marked as open for versions 1.0-m2 or 1.0.
Documentation is extensive but still incomplete. The recent
refactoring to create the Execution module is not yet reflected, so
the "container" concept is still ubiquitous.
* Roadmap (Beyond 1.0).
* Remote or distributed execution of steps. The step has to be
partitioned and the partition information passed on to the remote
processes to avoid double counting. The remote execution might be
in an EJB or other RPC like a web service.
* Asynchronous pipeline processing - steps execute concurrently and
optionally in separate processes. Feedback loop between consumers
and producers to prevent overflows.
* Issue tracking - a job is not finished until all issues with its
executions are resolved. Spring Batch can provide hooks to
integrate with internal issue tracking systems so that the lifetime
of a job can be properly managed.
* Auditing. Implement hooks to monitor not only what jobs execute
and the result of the execution (as per 1.0 possibly with some
richer options for detailed outcome reports), but also who has
executed the job, what changes they made to runtime parameters.
* OSGi support. Deploy the Spring Batch framework as an OSGi
service. Deploy individual jobs or groups of jobs as additional
bundles that depend on the core.
* No Plans Yet to Support
* Triggering.

98
src/site/apt/index.apt Normal file
View File

@@ -0,0 +1,98 @@
------
Spring Batch
------
Dave Syer, Scott Wintermute
------
March 2007, May 2007
Introduction
Many applications within the enterprise domain require bulk processing to perform business operations in mission critical environments. These business operations include automated, complex processing of large volumes of information that is most efficiently processed without user interaction. These operations typically include time based events (e.g. month-end calculations, notices or correspondence), periodic application of complex business rules processed repetitively across very large data sets (e.g. insurance benefit determination or rate adjustments), or the integration of information that is received from internal and external systems that typically requires formatting, validation and processing in a transactional manner into the system of record. Batch processing is used to process billions of transactions every day for enterprises.
Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. Spring Batch builds upon the productivity, POJO-based development approach, and general ease of use capabilities people have come to know from the Spring Framework, while making it easy for developers to access and leverage more advance enterprise services when necessary.
Spring Batch provides reusable functions that are essential in
processing large volumes of records, including logging/tracing,
transaction management, job processing statistics, job restart,
skip, and resource management. It also provides more advance
technical services and features that will enable extremely
high-volume and high performance batch jobs though optimization and
partitioning techniques. Simple as well as complex, high-volume
batch jobs can leverage the framework in a highly scalable manner to
process significant volumes of information.
Spring Batch is part of the
{{{http://www.springframework.org/sub-projects}Spring Portfolio}}.
* Spring Batch Architecture
Spring Batch is designed with extensibility and a diverse group of
end users in mind. The figure below shows a sketch of the layered
architecture that supports the extensibility and ease of use for
end-user developers.
[images/ContainerLayer.png] Spring Batch Architecture showing
Infrastructure and Container Layers. Potential container
implementations support different platforms and end-user goals from
the same blocks of business logic in the Application Layer.
The initial release provides an Infrastructure layer in the form of
low level tools. There is also a simple Container application,
using the infrastructure in its implementation. The container
provides robust features for traceability and management of the
batch lifecycle. A key goal is that the management of the batch
process (locating a job and its input, starting, scheduling,
restarting, and finally processing to created results) should be as
easy as possible for developers.
The Infrastructure provides the ability to batch operations
together, and to retry an piece of work if there is an exception.
Both requirements have a transactional flavour, and similar concepts
are relevant (propagation, synchronisation). They also both lend
themselves to the template programming model common in Spring,
c.f. <<<TransactionTemplate>>>, <<<JdbcTemplate>>>,
<<<JmsTemplate>>>.
The Simple Batch Execution Container is the first container available. It provides a robust set of integrated features including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management to enable the management of the full lifecycle of traditional batch processing. A number of sample jobs are packaged with this container and are described in detail to more clearly articulate usage and capabilities of the container.
* Roadmap
Once the framework is released it can be used immediately to
simplify batch optimisations and automatic retries. The framework
is oriented around application developers not needing to know any
details of the framework - there are a few application developer
interfaces that can be used for convenient construction of data
processing pipelines, but apart from that we support as close to a
POJO programming model as is practical. This is similar to the
approach taken in Spring Core in the area of DAO implementation.
A Partitioned Batch Execution Container is also being developed that will provide alternate scaling solutions. This container will provide more advance technical services and features to enable extremely high-volume and high performance batch jobs though proven optimization and partitioning techniques. Proven scaling techniques will be provided as partitioned strategies allowing users to spread the load across a pool of clustered J2EE application servers. There are also discussions to leverage grid technologies as an alternate scaling solution.
Matt Welsh's work shows that
{{{http://www.eecs.harvard.edu/~mdw/proj/seda/}SEDA}} has enormous
benefits over more rigid processing architectures, and messaging
containers give us a lot of resilience out of the box. So we also
want to provide a more SEDA flavoured container, or container
support, as well as supporting the more traditional ETL style
approach. There might be a tie in with Mule and/or other ESB tools
here, giving the benefit of a very scalable architecture, where the
choice of transport and distribution strategy can be made as late as
possible. The same application code could be used in principle for
a standalone tool processing a small amount of data, and a massive
enterprise-scale bulk-processing engine.
* Background
While open source software projects and associated communities have focused greater attention on web-based and SOA messaging-based architecture frameworks, there has been a notable lack of focus on reusable architecture frameworks to accommodate Java-based batch processing needs, despite continued needs to handle such processing within enterprise IT environments. The lack of a standard, reusable batch architecture has resulted in the proliferation of many one-off, in-house solutions developed within client enterprise IT functions.
Interface21 and Accenture are collaborating to change this. Accenture's hands-on industry and technical experience in implementing batch architectures, Interface21's depth of technical experience, and Spring's proven programming model together mark a natural and powerful partnership to create high-quality, market relevant software aimed at filling an important gap in enterprise Java. Both companies are also currently working with a number of clients solving similar problems developing Spring-based batch architecture solutions. This has provided some useful additional detail and real-life constraints helping to ensure the solution can be applied to the real-world problems posed by clients. For these reasons and many more, Interface21 and Accenture have teamed to collaborate on the development of Spring Batch.
Accenture is contributing previously proprietary batch processing architecture frameworks -- based upon decades worth of experience in building batch architectures with the last several generations of platforms (i.e., COBOL/Mainframe, C++/Unix, and now Java/anywhere) -- to the Spring Batch project along with committer resources to drive support, enhancements, and the future roadmap.
The collaborative effort between Accenture and Interface21 aims to promote the standardization of software processing approaches, frameworks, and tools that can be consistently leveraged by enterprise users when creating batch applications. Companies and government agencies desiring to deliver standard, proven solutions to their enterprise IT environments will benefit from Spring Batch.
* Links:
* A discussion {{{blotter.html}blotter}}.

58
src/site/apt/scratch.apt Normal file
View File

@@ -0,0 +1,58 @@
------
Spring Batch Scratch
------
Dave Syer
------
March 2007
+---
| restore;
|
1 | BATCH(repeat=until exhausted) {
|
2 | RETRY(outer) {
|
3 | TX(datasource=batch) {
|
4 | TX(datasource=business) {
|
5 | BATCH(repeat=5) {
|
6 | RETRY(inner) {
6.1 | input;
7 | } PROCESS {
| output;
8 | } RECOVER {
| recover;
| }
|
| }
|
4.1 | savepoint;
|
| }
|
| }
|
| }
|
| }
+---
* The order of the transaction nesting might be important, but only
if they are not XA, and only if there is a partial failure (inner
commits and the outer rolls back), and only if that happens on the
last attempt at RETRY(2).
* Batch TX is outside business TX so the worse that can happen is
that we might restart from the same point twice (if the inner TX
commits and the outer rolls back). If they were the other way
round the batch savepoint(4.1) could commit and the business
processing (7) roll back - then we would miss the business
processing if the batch had to restart.
* The savepoint(4.1) needs to be implemented so that the state it
saves is synchronized with the business TX(4). That way if TX(4)
rolls back the savepoint will always be the correct state to
restart if a partial failure is followed by a successful RETRY(2).

101
src/site/apt/sitemap.apt Normal file
View File

@@ -0,0 +1,101 @@
------
Site Map
------
Dave Syer
------
April 2007
Spring Batch Site Map
* Overview
* Main Site - high-level information and links to sub-projects
(called "modules" in Maven speak):
* Docs - reference documentation, user guides
* Infrastructure - CI build and technical information
* Integration Tests - reports on tests of infrastructure
* Container - CI and technical information about container layer
* Main Site
* Splash page - welcome, mission statement, download links
* Whitepaper (JavaOne presentation translated to HTML)
* Occasional Articles (e.g. transactions)
* Use Cases
* Project Information (standdard Maven stuff)
* Developers
* Source Repository
* License
* etc.
* Documentation
* Splash page - welcome, links to rest of reference docs.
* User Guides (docbook, Spring branded reference guides - HTML, HTML
Single Page and PDF). Two choices: one big guide with parts as
listed below, or multiple mini-guides. The former is probably
better.
Maybe we could also break each of these down a bit more...
* Infrastructure - How to use the core API
* Simple Container
* Partitioning Container
* Other Containers?
* Changelog
* Infrastructure
* Splash page explaining the role of infrastructure, and high level
API packaging.
* Changelog
* Project information (duplicated from Main Site - Maven "feature")
* CI Reports
* JUnit test report
* Clover coverage
* JDepend report
* Javadocs
* Integration Tests
* Changelog
* Project information (duplicated from Main Site - Maven "feature" -
TODO: find a way to switch them off in sub-projects)
* CI Reports (same as for infrastructure).
* Container
Should the use case go here (showing which ones are implemented)?
* Changelog
* Project information
* CI Reports (same as for infrastructure).

354
src/site/apt/transactions.apt Normal file
View File

@@ -0,0 +1,354 @@
------
Spring Batch-Retry Transaction Propagation
------
Dave Syer
------
February 2007
Batch Processing and Transactions
* {Simple Batching} with No Retry
Consider the following simple example of a nested batch with no
retries. This is a very common scenario for batch processing, where
an input source is processed until exhausted, but we commit
periodically at the end of a "chunk" of processing.
+---
1 | REPEAT(until=exhausted) {
|
2 | TX {
3 | REPEAT(size=5) {
3.1 | input;
3.2 | output;
| }
| }
|
| }
+---
The input operation (3.1) could be a message-based receive
(e.g. JMS), or a file-based read, but to recover and continue
processing with a chance of completing the whole job, it must be
transactional. The same applies to the operation at (3.2) - it must
be either transactional or idempotent.
If the chunk at REPEAT(3) fails because of a database exception at
(3.2), then TX(2) will roll back the whole chunk.
* Simple Stateless Retry
It is also useful to use a retry for an operation which is not
transactional, like a call to a web-service or other remote
resource. For example:
+---
0 | TX {
1 | input;
1.1 | output;
2 | RETRY {
2.1 | remote access;
| }
| }
+---
This is actually one of the most useful applications of a retry,
since a remote call is much more likely to fail and be retryable
than a database update. As long as the remote access (2.1)
eventually succeeds, the transaction TX(0) will commit. If the
remote access (2.1) eventually fails, then the transaction TX(0) is
guaranteed to roll back.
* {Typical} Repeat-Retry Pattern
The most typical batch processing pattern is to add a retry to the
inner block of the chunk in the {{{#Simple Batching}simple}} example.
Consider this:
+---
1 | REPEAT(until=exhausted, exception=not critical) {
|
2 | TX {
3 | REPEAT(size=5) {
|
4 | RETRY(stateful, exception=deadlock loser) {
4.1 | input;
5 | } PROCESS {
5.1 | output;
6 | } SKIP and RECOVER {
| notify;
| }
|
| }
| }
|
| }
+---
The inner RETRY(4) block is marked as "stateful" - see the
{{{#Typical}typical}} use case for a description of an stateful
retry. This means that if the the retry PROCESS(5) block fails, the
behaviour of the RETRY(4) is as follows.
* Throw an exception, rolling back the transaction TX(2) at the
chunk level, and allowing the item to be re-presented to the input
queue.
* When the item re-appears, it might be retried depending on the
retry policy in place, executing PROCESS(5) again. The second and
subsequent attempts might fail again and rethrow the exception.
* Eventually the item re-appears for the final time: the retry
policy disallows another attempt, so PROCESS(5) is never
executed. In this case we follow a RECOVER(6) path, effectively
"skipping" the item that was received and is being processed.
Notice that the notation used for the RETRY(4) in the plan above
shows explictly that the the input step (4.1) is part of the retry.
It also makes clear that there are two alternate paths for
processing: the normal case is denoted by PROCESS(5), and the
recovery path is a separate block, RECOVER(6). The two alternate
paths are completely distinct: only one is ever taken in normal
circumstances.
In special cases (e.g. a special <<<TranscationValidException>>>
type), the retry policy might be able to determine that the
RECOVER(6) path can be taken on the last attempt after PROCESS(5)
has just failed, instead of waiting for the item to be re-presented.
This is not the default behaviour because it requires detailed
knowledge of what has happened inside the PROCESS(5) block, which is
not usually available - e.g. if the output included write
access before the failure, then the exception should be rethrown to
ensure transactional integrity.
The completion policy in the outer, REPEAT(1) is crucial to the
success of the above plan. If the output(5.1) fails it may throw an
exception (it usually does, as described), in which case the
transaction TX(2) fails and the exception could propagate up through
the outer batch REPEAT(1). We do not want the whole batch to stop
because the RETRY(4) might still be successful if we try again, so
we add the exception=not critical to the outer REPEAT(1).
Note, however, that if the TX(2) fails and we <do> try again, by
virtue of the outer completion policy, the item that is next
processed in the inner REPEAT(3) is not guaranteed to be the one
that just failed. It might well be, but it depends on the
implementation of the input(4.1). Thus the output(5.1) might fail
again, on a new item, or on the old one. The client of the batch
should not assume that each RETRY(4) attempt is going to process the
same items as the last on ethat failed. E.g. if the termination
policy for REPEAT(1) is to fail after 10 attempts, it will fail
after 10 consecutive attempts, but not necessarily at the same item.
This is consistent with the overall retry strategy: it is the inner
RETRY(4) that is aware of the history of each item, and can decide
whether or not to have another attempt at it.
* Asynchronous Chunk Processing
The inner batches or chunks in the {{{#Typical}typical}} example
above can be executed concurrently by configuring the outer batch to
use an <<<AsyncTaskExecutor>>>. The outer batch waits for all the
chunks to complete before completing.
+---
1 | REPEAT(until=exhausted, concurrent, exception=not critical) {
|
2 | TX {
3 | REPEAT(size=5) {
|
4 | RETRY(stateful, exception=deadlock loser) {
4.1 | input;
5 | } PROCESS {
| output;
6 | } RECOVER {
| recover;
| }
|
| }
| }
|
| }
+---
* Asynchronous Item Processing
The individual items in chunks in the {{{#Typical}typical}}
can also in principle be processed concurrently. In this case the
transaction boundary has to move to the level of the individual
item, so that each transaction is on a single thread:
+---
1 | REPEAT(until=exhausted, exception=not critical) {
|
2 | REPEAT(size=5, concurrent) {
|
3 | TX {
4 | RETRY(stateful, exception=deadlock loser) {
4.1 | input;
5 | } PROCESS {
| output;
6 | } RECOVER {
| recover;
| }
| }
|
| }
|
| }
+---
This plan sacrifices the optimisation benefit, that the simple plan
had, of having all the transactional resources chunked together. It
is only useful if the cost of the processing (5) is much higher than
the cost of transaction management (3).
Interactions Between Batching and Transaction Propagation
There is a tighter coupling between batch-retry and TX management
than we would ideally like. In particular an stateless retry cannot
be used to retry database operations with a transaction manager that
doesn't support NESTED propagation.
For a simple example using retry without repeat, consider this:
+---
1 | TX {
|
1.1 | input;
2.2 | database access;
2 | RETRY {
3 | TX {
3.1 | database access;
| }
| }
|
| }
+---
Again, and for the same reason, the inner transaction TX(3) can
cause the outer transaction TX(1) to fail, even if the RETRY(2) is
eventually successful.
Unfortunately the same effect percolates from the retry block up to
the surrounding repeat batch if there is one:
+---
1 | TX {
|
2 | REPEAT(size=5) {
2.1 | input;
2.2 | database access;
3 | RETRY {
4 | TX {
4.1 | database access;
| }
| }
| }
|
| }
+---
Now if TX(3) rolls back it can pollute the whole batch at TX(1) and
force it to roll back at the end.
What about non-default propagation?
* In the last example PROPAGATION_REQUIRES_NEW at TX(3) will
prevent the outer TX(1) from being polluted if both transactions
are eventually successful. But if TX(3) commits and TX(1) rolls
back, then TX(3) stays committed, so we violate the transaction
contract for TX(1).
If TX(3) rolls back, TX(1) does not necessarily (but it probably
will in practice because the retry will throw a roll back
exception).
* PROPAGATION_NESTED at TX(3) works as we require in the retry
case (and for a batch with skips): TX(3) can commit, but
subsequently be rolled back by the outer transaction TX(1). If
TX(3) rolls back, again TX(1) will roll back in practice. This
option is only available on some platforms, e.g. not Hibernate or
JTA, but it is the only one that works consistently.
So NESTED is best if the retry block contains any database access.
* Special Case: Transactions with Orthogonal Resources
Default propagation is always OK for simple cases where there are no
nested database transactions. Consider this (where the SESSION and
TX are not global XA resources, so their resources are orthogonal):
+---
0 | SESSION {
1 | input;
2 | RETRY {
3 | TX {
3.1 | database access;
| }
| }
| }
+---
Here there is a transactional message SESSION(0), but it doesn't
participate in other transactions with
<<<PlatformTransactionManager>>>, so doesn't propagate when TX(3)
starts. There is no database access outside the RETRY(2) block. If
TX(3) fails and then eventually succeeds on a retry, SESSION(0) can
commit (it can do this independent of a TX block). This is similar
to the vanilla "best-efforts-one-phase-commit" scenario - the worst
that can happen is a duplicate message when the RETRY(2) succeeds
and the SESSION(0) cannot commit, e.g. because the message system is
unavailable.
* Stateless Retry Cannot Recover
The distinction between an stateless and an stateful retry in the
{{{#Typical}typical}} example above is important. It is actually
ultimately a transactional constraint that forces the distiction,
and this constraint also makes it obvious why the distinction
exists.
We start with the observation that there is no way to skip an item
that failed and successfully commit the rest of the chunk unless we
wrap the item processing in a transaction. So we simplify the
{{{#Typical}typical}} batch execution plan to look like this:
+---
0 | REPEAT(until=exhausted) {
|
1 | TX {
2 | REPEAT(size=5) {
|
3 | RETRY(stateless) {
4 | TX {
4.1 | input;
4.2 | database access;
| }
5 | } RECOVER {
5.1 | skip;
| }
|
| }
| }
|
| }
+---
Here we have an stateless RETRY(3) with a RECOVER(5) path that kicks
in after the final attempt fails. The "stateless" label just means
that the block will be repeated without rethrowing any exception up
to some limit. This will only work if the transaction TX(4) has
propagation NESTED.
If the TX(3) has default propagation properties and it rolls back,
it will pollute the outer TX(1). The inner transaction is assumed by
the transaction manager to have corrupted the transactional
resource, and so it cannot be used again.
Support for NESTED propagation is sufficiently rare that we choose
not to support recovery with stateless retries in current versions of
Spring Batch. The same effect can always be achieved (at the
expense of repeating more processing) using the
{{{#Typical}typical}} pattern above.

196
src/site/fml/faq.fml Normal file
View File

@@ -0,0 +1,196 @@
<?xml version="1.0" encoding="UTF-8"?>
<faqs id="FAQ" title="Frequently Asked Questions">
<part id="General">
<faq id="layers">
<question>
There are 3 main layers of the architecture (application, container, and infrastructure), what is the
vision for how the Container layer might be used in future?
</question>
<answer>
<p>
The "layers" described are nicely segregated in terms of dependency. Each layer only depends (at
compile time) on layers below it.
</p>
<p>
Actually we have recognised that what we used to call the container layer actually is composed of
two distinct contexts, "Core" and "Execution". So the full catalogue of contexts is:
<ul>
<li>
<b>Application</b>
is the business logic. It is written by the application developer - the client of Spring
Batch - and only depends on the other Core interfaces for compilation and configuration.
</li>
<li>
<b>Core</b>
is the public API of Spring Batch, including the core batch domain of Job, Step,
configuration and Executor interfaces.
</li>
<li>
<b>Execution</b>
is the deployment, execution and management concerns. Different execution environments (e.g.
in a JEE container, out of container) are configured differently, but can execute the same
application business logic.
</li>
<li>
<b>Infrastructure</b>
is a set of low level tools, that are used to implement the execution and parts of the core
layers.
</li>
</ul>
<br />
</p>
<p>
The "execution" layer is fertile ground for collaboration and contributions from the community and
from projects in the field. The central interface is
<code>ExecutionService</code>
with methods for starting and stopping jobs. The vision for this is that there can be multiple
implementations of
<code>ExecutionService</code>
providing different architectural patterns, and delivering different levels of scalability and
robustness, without changing either the business logic or the job configuration. The initial 1.0
release of Spring Batch will have a single implementation
<code>SimpleExecutionService</code>
(formerly known as
<code>SimpleBatchContainer</code>
.)
</p>
</answer>
</faq>
<faq id="flexible">
<question>
What is the Spring Batch philosophy on the use of flexible strategies and default implementations?
</question>
<answer>
There are a great many extension points in Spring Batch for the framework developer (as opposed to the
implementor of business logic). We expect clients to create their own more specific strategies that can
be plugged in to control things like commit intervals (
<code>CompletionPolicy</code>
), rules about how to deal with exceptions (
<code>ExceptionHandler</code>
), and many others.
</answer>
</faq>
<faq id="quartz">
<question>How does Spring Batch differ from Quartz? Is there a place for them both in a solution?</question>
<answer>
<p>
Spring Batch and Quartz have different goals. Spring Batch provides functionality for processing
large volumes of data and Quartz provides functionality for scheduling tasks. So Quartz could
complement Spring Batch, but are not excluding technologies. A common combination would be to use
Quartz as a trigger for a Spring Batch job using a Cron expression and the Spring Core convenience
<code>SchedulerFactoryBean</code>
.
</p>
</answer>
</faq>
<faq id="schedulers">
<question>How do I schedule a job with Spring Batch?</question>
<answer>
<p>
Use a scheduling tool. There are plenty of them out there. Examples: Quartz, Control-M, Autosys.
Quartz doesn't have all the features of Control-M or Autosys - it is supposed to be lightweight. If
you want something even more lightweight you can just use the OS (cron, at, etc.).
</p>
<p>
Simple sequential dependencies can be implemented using the job-steps model of Spring Batch. We
think this is quite common. And in fact it makes it easier to correct a common mis-use of scehdulers
- having hundreds of jobs configured, many of which are not independent, but only depend on one
other.
</p>
</answer>
</faq>
<faq id="stable">
<question>How stable are the interfaces in Spring Batch?</question>
<answer>
<p>
We are still in the milestone release phase (1.0-m2 is in the pipeline). This means that we are
still adding functionality that we want to be part of a 1.0 release. We do not rule out changes to
package and interface names in this phase, but that said we think the basic domain concepts in
Spring Batch are sound enough to survive significant re-factoring. The bulk of the application
developer "touch points" have been stable for quite some time now, and we have several early adopter
projects already using snapshot releases.
</p>
<p>
The process from here is to collect feedback from the community and use that to decide on what extra
features need to be added to get us to 1.0. When we are feature complete we will move to the
"release candidate" phase, and the first release in that phase will be 1.0-rc1. When significant
issues are resolved (if there are any) we will promote the release through the "rc" numbers, until
we have a clean 1.0 release.
</p>
</answer>
</faq>
<faq id="parallel">
<question>
How will Spring Batch allow project to optimize for performance and scalability (through parallel
processing or other)?
</question>
<answer>
We see this as one of the roles of the Execution layer. A specific implementation (or implementations)
of the
<code>ExecutionService</code>
can deal with the concern of breaking apart the business logic and sharing it efficiently between
parallel processes or processors. There are a number of technologies that could play a role here. The
essence is just a set of concurrent remote calls to distributed agents that can handle some business
processing. Since the business processing is already typically modularised - e.g. input an item, process
it - Spring Batch can strategise the distribution in a number of ways. One implementation that we have
had some experience with (and have a prototype for) is a set of remote EJBs handling the business
processing. We switch off Home caching in the container and then send a specific range of primary keys
for the inputs to each of a number of remote calls. THe same basic strategy would work with any of the
Spring Remoting protocols (plain RMI, HttpInvoker, JMS, Hessian etc.) with little more than a couple of
lines change in the execution layer configuration.
</answer>
</faq>
<faq id="steps">
<question>What are the key concepts in the Spring Batch core domain?</question>
<answer>
<p>
In a nutshell: A JobConfiguration with a list of StepConfigurations is passed to an
ExecutionService. From this a Job is constructed consisting of a series of Steps, each of which is
executed by a StepExecutor. The StepExecutor contains all the strategies for deciding when to
complete, when to commit, when to abort and when to continue.
</p>
<p>
Many Jobs in practice consist of a single Step. Step is very useful and best practice for breaking a
Job down into logical units, rather than having to execute separate Jobs (potentially in separate OS
processes) which have no obvious logical connection.
</p>
<p>
Jobs can be executed once, or many times with different logical identifiers (JobRuntimeInformation).
It is also possible to restart a failed Job with the same or a modified input source, and identify
the resulting JobExecution as a separate entity. In this way the progress of a Job and itys history
of successful and failed executions can easily be tracked. The same argument applies to Steps, which
have their corresponding StepExecution entity.
</p>
</answer>
</faq>
<faq id="messaging-scaling">
<question>How can messaging be used to scale batch architectures?</question>
<answer>
There is a good deal of practical evidence from existing projects that a pipeline approach to batch
processing is highly beneficial, leading to resilience and high throughput. We are often faced with
mission-critical applications where audit trails are essential, and guaranteed processing is demanded,
but where there are extremely tight limits on performance under load, or where high throughput gives a
competitive advantage. Matt Welsh's work shows that a Staged Event Driven Architecture (SEDA) has
enormous benefits over more rigid processing architectures, and message-oriented middleware (JMS, AQ,
MQ, Tibco etc.) gives us a lot of resilience out of the box. There are particular benefits in a system
where there is feedback between downstream and upstream stages, so the number of consumers can be
adjusted to account for the amount of demand. So how does this fit into Spring Batch? Well it's a good
example of an
<code>ExecutionService</code>
or (more broadly) execution runtime if the deployment is grid- or cluster-based, or in any way involves
multiple OS processes.
</answer>
</faq>
<faq id="contributions">
<question>How can I contribute to Spring Batch?</question>
<answer>
Use JIRA and the forum to get involved in discussions about the product and its design. There is a
process for contributions and eventually becoming a committer. The process is pretty standard for all
Apache-licensed projects. You make contributions through JIRA (so sign up now); you assign the copyright
of any contributions using a standard Apache-like CLA (see the Apache one for example - ours might be
slightly different); when the contributions reach a certain level, or you somehow convince us otherwise
that you are going to be committed long term, even if part time, then you can become a committer.
</answer>
</faq>
</part>
</faqs>

BIN
src/site/resources/discussion/retry-with-mq.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 100 KiB

BIN
src/site/resources/images/ContainerLayer.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

BIN
src/site/resources/images/logos/i21-banner-1.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

BIN
src/site/resources/images/partitioned.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

36
src/site/site.xml Normal file
View File

@@ -0,0 +1,36 @@
<?xml version="1.0" encoding="ISO-8859-1"?>
<project name="${project.name}">
<bannerLeft>
<name>${project.name}</name>
<href>http://www.springframework.org/</href>
</bannerLeft>
<bannerRight>
<src>images/shim.gif</src>
</bannerRight>
<poweredBy>
<logo name="" href="" img="images/shim.gif"/>
</poweredBy>
<skin>
<groupId>org.springframework.maven.skins</groupId>
<artifactId>maven-spring-skin</artifactId>
<version>1.0.2</version>
</skin>
<body>
<links>
<item name="Home" href="index.html"/>
</links>
<menu name="Spring Batch">
<item name="Home" href="index.html"/>
<item name="Use Cases" href="cases/index.html"/>
<item name="Transactions" href="transactions.html"/>
<item name="Batch Processing Strategies" href="batch-processing-strategies.html"/>
<item name="General Batch Principles and Guidelines" href="batch-principles-guidelines.html"/>
<item name="Building" href="building.html"/>
<item name="FAQ" href="faq.html"/>
<item name="Changelog" href="changelog.html"/>
</menu>
<menu ref="modules"/>
<menu ref="reports"/>
</body>
</project>