Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

tidy up some docos

Browse Source
This commit is contained in:
dsyer
2009-03-20 10:09:31 +00:00
parent f22718d5f3
commit 2626e45995
6 changed files with 197 additions and 1672 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
src/site/apt/index.apt
View File

@@ -39,9 +39,9 @@ Introduction
The framework is oriented around application developers not needing to know any details of the framework - there are a few application developer interfaces that can be used for convenient construction of data processing pipelines, but apart from that we support as close to a POJO programming model as is practical. This is similar to the approach taken in Spring Core in the area of DAO implementation.
Spring Batch version 1.x was targeted at Java 1.4 and single-process, possibly multi-threaded execution. Spring Batch 2.0 will be a Java 5 only release, using all available language features with no compromises for backward compatibility with Java 2. We think this will provide a significantly improved programming model for batch application developers. Spring Batch 2.0 will also introduce new features enabling an application to scale to multiple processes.
Spring Batch version 1.x was targeted at Java 1.4 and single-process, possibly multi-threaded execution. Spring Batch 2.0 is a Java 5 only release, using all available language features with no compromises for backward compatibility with Java 2. We think this will provide a significantly improved programming model for batch application developers.
Framework tools for scaling to multiple processes are being developed in a parallel project hosted inside Spring Batch. These will provide more advanced technical services and features to enable extremely high-volume and high performance batch jobs though proven optimization and clustering techniques. Proven scaling techniques will be provided as strategies allowing users to spread the load across a pool of clustered servers. There are also discussions to leverage grid technologies as a scaling solution. These features are targeted for Spring Batch version 2.0.
Framework tools for scaling to multiple processes are available in Spring Batch 2.0. These provide advanced technical services and features to enable extremely high-volume and high performance batch jobs though proven optimization and clustering techniques. An SPI is provided with a simple implementation that works in a single process (multi-threaded). Various remoting and grid technologies can be used to implement the same SPI in a multi-process, clustered environment.
Matt Welsh's work shows that {{{http://www.eecs.harvard.edu/~mdw/proj/seda/}SEDA}} has enormous benefits over more rigid processing architectures, and messaging environments give us a lot of resilience out of the box. So we also want to enable a more SEDA flavoured execution environments, as well as supporting the more traditional ETL style approach. The key to unlocking the programming model is {{{http://www.springframework.org/projects/spring-integration}Spring Integration}}, where the choice of transport and distribution strategy can be made as late as possible. The key to the runtime requirements of deployment and manageability is going to be with the {{{http://www.springsource.com/products/suite/applicationplatform}SpringSource Application Platform}}. The same application code could be used in principle for a standalone tool processing a small amount of data, and a massive enterprise-scale bulk-processing engine.

656
src/site/docbook/reference/application.xml
View File

@@ -1,656 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter id="The Nature of Batch Applications">
<title>The Nature of Batch Applications</title>
<section>
<title id="s.0">Batch Application Overview</title>
<para>The Application layer is anything built on top of the spring batch framework.
Many enterprises solutions are composed of composite applications, meaning online web
applications, SOA Enabled Services, Enterprise Application Integration, and Batch. Despite
the ideal of zero latency applications and the interconnected enterprise many interfaces
still define the exchange of information through flat files. Increasingly these files
are provided in an XML format. The chapter will provide a more in depth coverage of
batch solution space and the kinds of problems that spring batch may provide solutions
for high volume processing.</para>
</section>
<section>
<title id="s.1">Batch Processing Strategies</title>
<para>To help design and implement batch systems, basic batch
application building blocks and patterns should be provided to the
designers and programmers in form of sample structure charts and
code shells. When starting to design a batch job, the business
logic should be decomposed into a series of steps which can be
implemented using the following standard building blocks:</para>
<itemizedlist>
<listitem>
<para>Conversion Applications: For each type of file supplied
by or generated to an external system, a conversion application
will need to be created to convert the transaction records
supplied into a standard format required for processing. This
type of batch application can partly or entirely consist of
translation utility modules (see Basic Batch Services).</para>
</listitem>
<listitem>
<para>Validation Applications: Validation applications ensure
that all input/output records are correct and consistent.
Validation is typically based on file headers and trailers,
checksums and validation algorithms as well as record level
cross-checks.</para>
</listitem>
<listitem>
<para>Extract Applications: An application that reads a set of
records from a database or input file, selects records based on
predefined rules, and writes the records to an output
file.</para>
</listitem>
<listitem>
<para>Extract/Update Applications: An application that reads
records from a database or an input file, and makes changes to
a database or an output file driven by the data found in each
input record.</para>
</listitem>
<listitem>
<para>Processing and Updating Applications: An application that
performs processing on input transactions from an extract or a
validation application. The processing will usually involve
reading a database to obtain data required for processing,
potentially updating the database and creating records for
output processing.</para>
</listitem>
<listitem>
<para>Output/Format Applications: Applications reading an input
file, restructures data from this record according to a
standard format, and produces an output file for printing or
transmission to another program or system.</para>
</listitem>
</itemizedlist>
<para><emphasis role="bold">Pre-processing
Capabilities</emphasis></para>
<para>Additionally a basic application shell should be provided for
business logic that cannot be built using the previously mentioned
building blocks.</para>
<para>In addition to the main building blocks, each application may
use one or more of standard utility steps, such as:</para>
<itemizedlist>
<listitem>
<para>Sort - A Program that reads an input file and produces an
output file where records have been re-sequenced according to a
sort key field in the records. Sorts are usually performed by
standard system utilities.</para>
</listitem>
<listitem>
<para>Split - A program that reads a single input file, and
writes each record to one of several output files based on a
field value. Splits can be tailored or performed by
parameter-driven standard system utilities.</para>
</listitem>
<listitem>
<para>Merge - A program that reads records from multiple input
files and produces one output file with combined data from the
input files. Merges can be tailored or performed by
parameter-driven standard system utilities.</para>
</listitem>
</itemizedlist>
<para>Batch applications can additionally be categorized by their
input source:</para>
<itemizedlist>
<listitem>
<para>Database-driven applications are driven by rows or values
retrieved from the database.</para>
</listitem>
<listitem>
<para>File-driven applications are driven by records or values
retrieved from a file</para>
</listitem>
</itemizedlist>
<para>The foundation of any batch system is the processing
strategy. Factors affecting the selection of the strategy include
estimated batch system volume, concurrency with on-line or with
another batch systems, available batch windows etc. Also with more
enterprises wanting to be up and running 24x7, it is more challenging to
establish batch windows.</para>
<para>Typical processing options for batch are:</para>
<itemizedlist>
<listitem>
<para>Normal processing in a batch window during off-line</para>
</listitem>
<listitem>
<para>Concurrent batch / on-line processing</para>
</listitem>
<listitem>
<para>Parallel processing of many different batch runs or jobs
at the same time</para>
</listitem>
<listitem>
<para>Streaming i.e. processing of many instances of the same
job at the same time</para>
</listitem>
<listitem>
<para>A combination of these</para>
</listitem>
</itemizedlist>
<para>The order in the list above reflects the implementation
complexity, processing in a batch window being the easiest and
streaming the most complex to implement.</para>
<para>Some or all of these options may be supported by a commercial
scheduler.</para>
<para>In the following section these processing options will be
discussed in more detail. It is important to notice that the commit
and locking strategy adopted by batch processes will be dependent
on the type of processing performed and as a rule of thumb, the
on-line locking should use the same principles. Therefore a batch
architecture cannot be simply an afterthought when designing an
overall architecture.</para>
<para>The locking strategy can use only normal database locks, or
an additional custom locking service can be implemented in the
architecture. The locking service would track database locking (for
example by storing the necessary information in a dedicated
db-table) and give or deny permissions to the application programs
requesting a db operation. Retry logic could also be implemented by
this architecture to avoid aborting a batch job in case of a lock
situation.</para>
<para><emphasis role="bold">1. Normal processing in a batch
window</emphasis> For simple batch processes running in a separate
batch window, where the data being updated is not required by
on-line users or other batch processes, concurrency is not an issue
and a single commit can be done at the end of the batch run.</para>
<para>In most cases a more robust approach is more appropriate. A
thing to keep in mind is that batch systems have a tendency to grow
as time goes by, both in terms of complexity and the data volumes
they will handle. If no locking strategy is in place and the system
still relies on a single commit point, modifying the batch programs
can be painful. Therefore, even with the simplest batch systems,
consider the need for commit logic depicted in the
[Restart/Recovery section|Restart &amp; Recovery] as well as the
information concerning the more complex cases below.</para>
<para><emphasis role="bold">2. Concurrent batch / on-line
processing</emphasis> Batch applications processing data that can
simultaneously be updated by on-line users, should not lock any
data (either in the database or in files) which could be required
by on-line users for more than a few seconds. Also updates should
be committed to the database at the end of every transaction or what's
referred to as a commit interval of size=1. This minimizes the portion
of data that is unavailable to other processes and the elapsed time the
data is unavailable.</para>
<para><emphasis role="bold">2. Concurrent batch / on-line
processing</emphasis> Batch applications processing data that can
simultaneously be updated by on-line users, should not lock any
data (either in the database or in files) which could be required
by on-line users for more than a few seconds. Also updates should
be committed to the database at the end of every transaction or what's
referred to as a commit interval of size=1. This minimizes the portion
of data that is unavailable to other processes and the elapsed time the
data is unavailable.</para>
<para>Another option to minimize physical locking is to have a
logical row-level locking implemented using either an Optimistic
Locking Pattern or a Pessimistic Locking Pattern.</para>
<itemizedlist>
<listitem>
<para>Optimistic locking assumes a low likelihood of record
contention. It typically means inserting a timestamp column in
each database table used concurrently by both batch and on-line
processing. When an application fetches a row for processing,
it also fetches the timestamp. As the application then tries to
update the processed row, the update uses the original
timestamp in the WHERE clause. If the timestamp matches, the
data and the timestamp will be updated successfully. If the
timestamp does not match, this indicates that another
application has updated the same row between the fetch and the
update attempt and therefore the update cannot be
performed.</para>
</listitem>
<listitem>
<para>Pessimistic locking is any locking strategy that assumes
there is a high likelihood of record contention and therefore
either a physical or logical lock needs to be obtained at
retrieval time. One type of pessimistic logical locking uses a
dedicated lock-column in the database table. When an
application retrieves the row for update, it sets a flag in the
lock column. With the flag in place, other applications
attempting to retrieve the same row will logically fail. When
the application that set the flag updates the row, it also
clears the flag, enabling the row to be retrieved by other
applications. Please note, that the integrity of data must be
maintained also between the initial fetch and the setting of
the flag, for example by using db locks (e.g.,SELECT FOR
UPDATE). Note also that this method suffers from the same
downside as physical locking except that it is somewhat easier
to manage building a time-out mechanism that will get the lock
released if the user goes to lunch while the record is
locked.</para>
</listitem>
</itemizedlist>
<para>These patterns are not necessarily suitable for batch
processing, but they might be used for concurrent batch and on-line
processing for example in cases where the database doesn&apos;t
support row-level locking. As a general rule, optimistic locking is
more suitable for on-line applications, while pessimistic locking
is more suitable for batch applications. Whenever logical locking
is used, the same scheme must be used for all applications
accessing data entities protected by logical locks.</para>
<para>Note that both of these solutions only address locking a
single record. Often we may need to lock a logically related group
of records. With physical locks, you have to manage these very
carefully in order to avoid potential deadlocks. With logical
locks, it is usually best to build a logical lock manager that
understands the logical record groups you want to protect and can
ensure that locks are coherent and non-deadlocking. This logical
lock manager usually uses its own tables for lock management,
contention reporting, time-out mechanism, etc.</para>
<para><emphasis role="bold">3. Parallel Processing</emphasis>
Parallel processing allows multiple batch runs / jobs to run in
parallel to minimize the total elapsed batch processing time. This
is not a problem as long as the jobs are not sharing the same
files, db-tables or index spaces. If they do, this service should
be implemented using partitioned data. Another option is to build
an architecture module for maintaining interdependencies using a
control table. A control table should contain a row for each shared
resource and whether it is in use by an application or not. The
batch architecture (Control Program Tasklet) or the application in a
parallel job would then retrieve information from that table to
determine if it can get access to the resource it needs or
not.</para>
<para>If the data access is not a problem, parallel processing can
be implemented in a mainframe environment using parallel job
classes, in order to ensure adequate CPU time for all the
processes. In an environment other than the mainframe, a similar
solution can be put in place with for example threads. The solution
has to be robust enough to ensure time slices for all the running
processes.</para>
<para>Other key issues in parallel processing include load
balancing and the availability of general system resources such as
files, database buffer pools etc. Also note that the control table
itself can easily become a critical resource.</para>
<para><emphasis role="bold">4. Partitioning</emphasis> Using
partitioning allows multiple versions of large batch applications
to run concurrently. The purpose of this is to reduce the elapsed
time required to process long batch jobs. Processes which can be
successfully partitioned are those where the input file can be
split and/or the main database tables partitioned to allow the
application to run against different sets of data.</para>
<para>In addition, processes which are partitioned must be designed
to only process their assigned data set. A partitioning
architecture has to be closely tied to the database design and the
database partitioning strategy. Please note, that the database
partitioning doesn&apos;t necessarily mean physical partitioning of
the database, although in most cases this is advisable. The
following picture illustrates the partitioning
approach:!app_style_batch_processing.png|align=center!</para>
<para>The architecture should be flexible enough to allow dynamic
configuration of the number of partitions. Both automatic and user
controlled configuration should be considered. Automatic
configuration may be based on parameters such as the input file
size and/or the number of input records.</para>
<para><emphasis role="bold">4.1 Streaming Approaches</emphasis> The
following lists some of the possible streaming approaches.
Selecting a streaming approach has to be done on a case-by-case
basis.</para>
<para><emphasis>1. Fixed and Even Break-Up of Record
Set</emphasis></para>
<para>This involves breaking the input record set into an even
number of portions (e.g. 10, where each portion will have exactly
1/10th of the entire record set). Each portion is then processed by
one instance of the batch/extract application.</para>
<para>In order to use this approach, preprocessing will be required
to split the recordset up. The result of this split will be a lower
and upper bound placement number which can be used as input to the
batch/extract application in order to restrict its processing to
its portion alone.</para>
<para>Preprocessing could be a large overhead as it has to
calculate and determine the bounds of each portion of the record
set.</para>
<para><emphasis>2. Breakup by a Key Column</emphasis></para>
<para>This involves breaking up the input record set by a key
column such as a location code, and assigning data from each key to
a batch instance. In order to achieve this, column values can
either be</para>
<para><emphasis>3. Assigned to a batch instance via a streaming
table (see below for details).</emphasis></para>
<para><emphasis>4. Assigned to a batch instance by a portion of the
value (e.g. values 0000-0999, 1000 - 1999, etc.)</emphasis></para>
<para>Under option 1, addition of new values will mean a manual
reconfiguration of the batch/extract to ensure that the new value
is added to a particular instance.</para>
<para>Under option 2, this will ensure that all values are covered
via an instance of the batch job. However, the number of values
processed by one instance is dependent on the distribution of
column values (i.e. there may be a large number of locations in the
0000-0999 range, and few in the 1000-1999 range). Under this
option, the data range should be designed with streaming in
mind.</para>
<para>Under both options, the optimal even distribution of records
to batch instances cannot be realized. There is no dynamic
configuration of the number of batch instances used.</para>
<para><emphasis>5. Breakup by Views</emphasis></para>
<para>This approach is basically breakup by a key column, but on
the database level. It involves breaking up the recordset into
views. These views will be used by each instance of the batch
application during its processing. The breakup will be done by
grouping the data.</para>
<para>With this option, each instance of a batch application will
have to be configured to hit a particular view (instead of the
master table). Also, with the addition of new data values, this new
group of data will have to be included into a view. There is no
dynamic configuration capability, as a change in the number of
instances will result in a change to the views.</para>
<para><emphasis>6. Addition of a Processing
Indicator</emphasis></para>
<para>This involves the addition of a new column to the input
table, which acts as an indicator. As a preprocessing step, all
indicators would be marked to non-processed. During the record
fetch stage of the batch application, records are read on the
condition that that record is marked non-processed, and once they
are read (with lock), they are marked processing. When that record
is completed, the indicator is updated to either complete or error.
Many instances of a batch application can be started without any
changes, as the additional column ensures that a record is only
processed once.</para>
<para>With this option, I/O on the table increased dynamically. In
the case of a updating batch application, this impact is reduced,
as a write will have to occur anyway.</para>
<para><emphasis>7. Extract Table to a Flat File</emphasis></para>
<para>This involves the extraction of the table into a file. This
file can then be split into multiple segments and used as input to
the batch instances.</para>
<para>With this option, the additional overhead of extracting the
table into a file, and splitting it, may cancel out the effect of
multi-streaming. Dynamic configuration can be achieved via changing
the file splitting script.</para>
<para><emphasis>8. Use of a Hashing Column</emphasis></para>
<para>This scheme involves the addition of a hash column
(key/index) to the database tables used to retrieve the driver
record. This hash column will have an indicator to determine which
instance of the batch application will process this particular row.
For example, if there are three batch instances to be started, then
an indicator of &apos;A&apos; will mark that row for processing by
instance 1, an indicator of &apos;B&apos; will mark that row for
processing by instance 2, etc.</para>
<para>The procedure used to retrieve the records would then have an
additional WHERE clause to select all rows marked by a particular
indicator. The inserts in this table would involve the addition of
the marker field, which would be defaulted to one of the instances
(e.g. &apos;A&apos;).</para>
<para>A simple batch application would be used to update the
indicators such as to redistribute the load between the different
instances. When a sufficiently large number of new rows have been
added, this batch can be run (anytime, except in the batch window)
to redistribute the new rows to other instances.</para>
<para>Additional instances of the batch application only require
the running of the batch application as above to redistribute the
indicators to cater for a new number of instances.</para>
<para>4.2 Database and Application design Principles</para>
<para>An architecture that supports multi-streamed applications
which run against partitioned database tables using the key column
approach, should include a central streaming repository for storing
streaming parameters. This provides flexibility and ensures
maintainability. The repository will generally consist of a single
table known as the streaming table.</para>
<para>Information stored in the streaming table will be static and
in general should be maintained by the DBA. The table should
consist of one row of information for each stream of a
multi-streamed application. The table should have a similar layout
to the following table:</para>
<para><anchor id="a.center"/>center || Streaming Table || | Program
ID Code Stream Number (Logical ID of the stream) Low Value of the
db key column for this stream High Value of the db key column for
this stream | center</para>
<para>On program start-up the program id and stream number should
be passed to the application from the architecture (Control
Processing Tasklet). These variables are used to read the streaming
table, to determine what range of data the application is to
process (if a key column approach is used). In addition the stream
number must be used throughout the processing to:</para>
<itemizedlist>
<listitem>
<para>Add to the output files/database updates in order for the
merge process to work properly</para>
</listitem>
<listitem>
<para>Report normal processing to the batch log and any errors
that occur during execution to the architecture error
handler</para>
</listitem>
</itemizedlist>
<para>4.3 Minimizing Deadlocks When applications run in parallel or
streamed, contention in database resources and deadlocks may occur.
It is critical that the database design team eliminates potential
contention situations as far as possible as part of the database
design.</para>
<para>Also ensure that the database index tables are designed with
deadlock prevention and performance in mind.</para>
<para>Deadlocks or hot spots often occur in administration or
architecture tables such as log tables, control tables, lock tables
etc.. The implications of these should be taken into account as
well. A realistic stress test is crucial for identifying the
possible bottlenecks in the architecture.</para>
<para>To minimize the impact of conflicts on data, the architecture
should provide services such as wait-and-retry intervals when
attaching to a database or when encountering a deadlock. This means
a built-in mechanism to react to certain database return codes and
instead of issuing an immediate error handling, waiting a
predetermined amount of time and retrying the database
operation.</para>
<para>4.4 Parameter Passing and Validation</para>
<para>The streaming architecture should be relatively transparent
to application developers. The architecture should perform all
tasks associated with running the application in a streamed mode
i.e.</para>
<itemizedlist>
<listitem>
<para>Retrieve streaming parameters before application
start-up</para>
</listitem>
<listitem>
<para>Validate streaming parameters before application
start-up</para>
</listitem>
<listitem>
<para>Pass parameters to application at start-up</para>
</listitem>
</itemizedlist>
<para>The validation should include checks to ensure that:</para>
<itemizedlist>
<listitem>
<para>the application has sufficient streams to cover the whole
data range</para>
</listitem>
<listitem>
<para>there are no gaps between streams</para>
</listitem>
</itemizedlist>
<para>If the database is partitioned, some additional validation
may be necessary to ensure that a single stream does not span
database partitions.</para>
<para>Also the architecture should take into consideration the
consolidation of streams. Key questions include:</para>
<itemizedlist>
<listitem>
<para>Must all the streams be finished before going into the
next job step?</para>
</listitem>
<listitem>
<para>What happens if one of the streams aborts?</para>
</listitem>
</itemizedlist>
</section>
<section>
<title id="bps.2">Batch Job Type Specific Concerns</title>
<para>Batch Jobs Types (e.g. conversion, pdf generation, report generation, high volume print, etc.) require different
technologies in the solution space. For example, conversion may require additional XML technologies, Adobe or iText for
PDF generation, different reporting options for report generation, etc. It is often helpful to isolate batch jobs at a
minimum by job type so that required dependencies for batch job types don't pollute other application styles. It is also
key to leaving the application decoupled by style and by execution environment allowing maximum flexibility at deployment
time.
</para>
</section>
</chapter>

546
src/site/docbook/reference/arch-overview.xml
View File

@@ -1,546 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter id="Container Architecture Overview" lang="en">
<title>Container Architecture Overview</title>
<section>
<title>Introduction</title>
<para></para>
</section>
<section>
<title id="s.1">Simple Container Architecture Overview</title>
<para></para>
<section>
<title id="s.1.1">Introduction</title>
<para>This chapter covers the overall spring batch architecture.
The Spring Container Archtiecture is made up of five logical
layers; 1) the Batch Application, 2) the Batch Application Layer,
3) the batch core layer, and 4) the batch infrastucture
layer.</para>
<informaltable colsep="1" frame="all" rowsep="1">
<tgroup cols="3">
<colspec align="center"/>
<colspec align="center"/>
<colspec align="left"/>
<tbody>
<row><entry><para>Provided
By</para></entry><entry><para>Layer</para></entry><entry><para>Description</para></entry></row>
<row><entry><para>Application
Developer</para></entry><entry><para>Batch
Application</para></entry><entry><para>This is where the
application developer writes their batch jobs and
tasklets.</para></entry></row>
<row><entry><para>Spring Batch Execution
Container</para></entry><entry><para>Container Application
Layer</para></entry><entry><para>Allows for extending and
overwriting of the batch support layer for custom
requirements. Facilities implemented in this layer could
migrate down to Batch Support Layer. This is also the layer
to add the project specific jars required by job types
(e.g. reporting jars like Crystal, Brio, etc, form
generation jars like Central Pro or Adobe,
etc).</para></entry></row>
<row><entry><para>Spring Batch Execution
Container</para></entry><entry><para>Container Support
Layer</para></entry><entry><para>Provides default
implementations of batch core services including I/O,
Restart, Partitioning, Statistics, and
configurations</para></entry></row>
<row><entry><para>Spring Batch Execution
Container</para></entry><entry><para>Container Core
Layer</para></entry><entry><para>Enables configuration,
Common Services &amp; Interfaces,
management</para></entry></row>
<row><entry><para>Spring Batch
Infrastructure</para></entry><entry><para>Batch-Infrastructure</para></entry><entry><para>Provides
IO support, Batch style transactions, advanced exception
handling, batch-template, batch-retry</para></entry></row>
</tbody>
</tgroup>
</informaltable>
<variablelist>
<varlistentry>
<term>Figure 2.0</term>
<listitem>
<para>- Batch Architecture Layers</para>
<para>The batch architecture is modeled after a container
architecture, meaning that there are managed resources
essential to high performance batch architectures that are
configured through a spring context. The following sect1s
will provide a quick review of each layer and their role in
the batch architecture.</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section>
<title id="s.1.2">Batch Applications</title>
<para></para>
</section>
<section>
<title id="s.1.3">Container Application Layer</title>
<para></para>
</section>
<section>
<title id="s.1.4">Container Support Layer</title>
<para>The batch support layer provides default implementations
for all interfaces, interceptors, advice and other core batch
services. Figure 2.3.1 illustrates the following logical
packages. !Batch Support.png! Although physically they break out
into many more than depicted, logically you can think of the
groupings in the following manner: * I/O Support packages *
Restart Support * Lifecycle Support packages * DAO support
layer</para>
<itemizedlist>
<listitem>
<para>I/O Support Packages</para>
<para>The I/O related packages are currently the richest
packages in the batch architecture. They are modeled after
Spring Patterns of Operations and Templates. For example,
you&apos;ll see FlatFileInputOperations accompanied with a
FlatFileInputTemplate. The FlatFileInputTemplate is wired up
with a File Descriptor, which contains a Record Descriptor
along with various other properties. With the File and Record
Descriptors the InputTemplate supports a callback method that
allows for the mapping of a record into an object. This
support applies to fixed length records, delimited records
and XML records. To further simplify this a
DefaultFlatFileDataProvider is supplied an input template,
which contains the field and record descriptions, along with
a line mapper that knows how to map the line to an object.
The next() operation on a record simply needs to
readAndMap(lineMapper) a record. This pattern is used over
again for XML and SQL input for simple mapping of input
records to objects.</para>
<para>In addition to declarative descriptions of the records
that can be re-used by multiple batch jobs, the I/O
facilities also support configurable validation strategies.
The two currently supported are Apache Commons Validator and
Spring&apos;s VALang.</para>
</listitem>
<listitem>
<para>Restart Support</para>
<para>The Restart Support provides implementations for a few
common restart strategies that will be discussed further in
the respective sect1. The following are provided
out-of-the-box: * IDList Restart Strategy - a strategy that
supports a batch application where the application does not
have a &quot;process&quot; flag and needs the batch
architecture to track which records have been processed. This
is not the ideal scenario. * Last Processed Restart Strategy
- when the record can be identified through a where and order
by only the last record(s) processed needs to be saved for
restart. * No Restart Strategy - some batch jobs simply
can&apos;t support restart. When they are re-run they are
considered to be a new instance of a batch job. * Sql Restart
Strategy - [need some additional javadoc for this
strategy].</para>
</listitem>
<listitem>
<para>Lifecycle Support</para>
</listitem>
</itemizedlist>
</section>
</section>
<section>
<title id="s.1.5">The Core Layer</title>
<para>The Batch Core interfaces and services are illustrated in
a simplified view of a package diagram. There are roughly seven
logical packages: * Core Spring Extensions * Core Batch Advice *
Core Batch Configuration * Core Batch Repository * Core Batch
Tasklet \ !Batch Core.png! [Figure 2.5] Batch Core Layer</para>
<para>In the actual physical packaging there are a few more
logical services that the batch execution environment provides. The following
sections will provide an introduction into each set of core batch
facilities.</para>
<itemizedlist>
<listitem>
<para>Core Spring Extensions</para>
<para>The Core Spring extensions provide the scaffolding for
a batch execution environment. This includes facilities for managing the
batch architecture in terms of launching, suspending and
stopping batch jobs. There is house keeping that goes on,
especially in concurrent batch jobs, related to ensuring that
batch jobs quiese properly. The lifecycle management provides
services for the proper initialization and subsequent
shutdown of batch resources and services. The batch
architecture is flexible in terms of how batch jobs may be
launched. For example, batch jobs can be started via JMX
facilities, scripts from the command line that launch a Java
VM. It can also support launching batch jobs through web
services or http. There are no restrictions. Finally, there
are standard batch error codes. These error codes can be
exposed to external utilities, like Schedulers, to ensure
that batch jobs expose the status of jobs to an operational
environment. This is especially important in the batch
context where the modus operandi is headless, meaning
unattended operation.</para>
</listitem>
<listitem>
<para>Core Batch Advice</para>
<para>Core Batch Advice is an inventory of the type of advice
that batch architectures will inject during the runtime of a
batch application. These are defined as a set of extensible
interfaces, with a number of default implementations in the
support layer that provide some of the most common types of
advice. Partition Advice is helpful with large datasets that
need to be &quot;chunked&quot; up and run concurrently for
better through put. Resource Advice is helpful for
registering interest in transactional information so that
file locations can be kept in sync with information processed
within a transaction. In addition, the resource is associated
with the correct step context and its associated
configuration properties. Skip advice is applied for records
that the tasklet is unable to process. Restart Advice is
helpful for Restartable jobs where for advising the job on
how to restart. There is considerable variability on how
restart can occur. For example, a job may be marking records
as &quot;processed&quot; and the restart advice will advise
the process with query that restarts the job at the last
successfully processed record. Finally, Statistics are vital
in operational environments to report on records processed,
records skipped and total number of records read. In
addition, certain batch jobs lend themselves to custom
reporting to expose additional business level information
like the number of trades processed or cases opened,
etc.</para>
</listitem>
<listitem>
<para>Core Batch Configuration</para>
<para>Batch configuration is considerably different from
online web applications or SOA based applications. The Core
Batch Configuration provides a place for configuring runtime
properties related to the batch application style. This
includes the ability to add Commit Policy. In a batch style
application it is often advantageous to keep the commit
interval as high as possible when processing Logical Units of
Work. Whereas in an online web application with declarative
transaction the transaction scope would be at the entrance to
a business service, a batch transaction scope may include
many logical units of work before a transaction commit is
executed. A Start Policy allows a configuration to tell the
batch job whether it is Restartable, and if so, what type of
restart to initiate. Some jobs are not restartable and care
should be taken to ensure that information is not applied
multiple times when the business rules do not allow for it.
Exception policies deal with what to do when exceptions
occur. This impacts logging policies and exception handling.
The architecture defines a common set of exceptions that
projects can apply handlers to like processing errors,
validation errors, parsing errors, missing configuration
parameters, etc.</para>
</listitem>
<listitem>
<para>2.5.4 Container Repository</para>
<para>This is an internal package for storing the state of a
batch job and any associated partition and step status.</para>
</listitem>
<listitem>
<para>Core Batch Tasklet</para>
<para>The core batch tasklet is where control is handed off to
the application. There are a number of patterns that have
been observed in processing batch data. Spring Core Batch
Tasklet implements the most common patterns and provides and
extension point for additional Tasklet processing
implementations. The basic idea of tasklet provides the
facilities for reading and processing data. The simplest
implementation of Tasklet, the ReadProcessTasklet, handles both
the input and output of data within one class. An alternative
implementation, the DataProviderProcessTasklet, provides
functionality for &apos;split processing&apos;. This type of
processing is characterized by separating the reading and
processing of batch data into two seperate classes:
DataProvider and TaskletProcessor. The DataProvider class
provides a solid means for reusablility and enforces good
architecture practices. Because an object *must* be returned
by the DataProider to continue processing, (Returning null
indicates processing should end) a developer is forced to
read in all relevant data, place it into domain or value
objects, and return the object. The TaskletProcessor will then
use this object within the business logic and final
output.</para>
</listitem>
</itemizedlist>
</section>
<section>
<title id="s.2">Container&apos;s Use of batch
infrastructure</title>
<itemizedlist>
<listitem>
<para>Infrastructure Provided I/O</para>
<para>The I/O core interfaces and implementations provide
facilities for simplifying the extraction of data from I/O
sources like files and database tables. The key concepts are
FieldDescriptors and FieldSets along with appropriate CallBack
Handlers. These are modeled after common spring operations and
templates like JdbcTemplate. Through the use of LineMappers a
developer needs only to describe a record format and write the
appropriate callback method that maps the parsed record into an
object of their choice. These can either be true POJO objects
or Value Objects (structures) that are subsequently available
for the tasklet to processs. The interface for Field Descriptors
also allows for a level of validation through the use of
Spring&apos;s VALang or Apache&apos;s Common Validator.</para>
</listitem>
<listitem>
<para>Core Batch Interceptors &amp; Interceptor
Services</para>
<itemizedlist>
<listitem>
<para>Batch Operations &amp; Batch Template</para>
<para>Interceptors and the associated services are the key
to how advise is applied in the batch architecture. The
interceptors are Point Cuts in the batch lifecycle that
allow the injection of advise. The shared lifecycle
behavior abstracted through the BatchLifeCycleInterceptor
defineds three methods; init, onError and finalize. All
subclasses of LifeCycleInterceptor define default behavior
for these three methods. The JobLifecycleInterceptor
further exposes the methods beforeJob(), beforeStep(),
afterJob(), and afterStep() allowing hooks into the
lifecycle for specific advise. The Batch Architecture
provides default implementations for all lifecycle point
cuts, or interception points. The Tasklet Interceptor, in
addition to the standard lifecycle methods, implements
logic around beforeLuw(), afterLuw(),
commitIntervalStarted() and commitIntervalCompleted().
Having well defined lifecycle interception points allows
for the easy insertion of custom advice into the batch
runtime environment.</para>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
</section>
<section>
<title id="s.3">Batch Execution Container Configurations</title>
<para>In addition to core facilities for configuring or wiring
together jobs and steps with required resources, policies, and
interceptors, spring batch allows considerable flexibility in how
scalability is achieved. More options for scalability will be
available in the future. The important key for scalability in Java
is the recognition that there is a limit to what one JVM may scale
up to in terms of number of threads, managed resources, memory
configuration, etc. The spring batch architecture allows for the
configuration of simple batch jobs where one VM and one process is
sufficient to do perform the work within a batch window all the way
through many threads distributed within a cluster of JEE servers.
The figure below illistrates the scalability spectrum.</para>
<para>This is not to be understood as the only way to scale batch
jobs as there are many factors. For example, other federated java
architectures hold potential like Teracotta or Gigaspaces although
there is no current implementation for these distributed models in
the current batch architecture. !scalability-model.png! [Figure
2.3.1] - Scalability Model</para>
<section>
<title id="s.3.1">Single VM Simple Batch Execution
Container - One Job, One Step, One Partition</title>
<para>The simplest configuration is one job with with step and
hence, one implied partition. Implied means that there is nothing
for the developer to consider because the default number of
partitions is one. There is typically one input source and one
output source in this simple configuration. See the Simple Tasklet
Job for an example of what this configuration looks like. A
simple configuration still typically configures a datasource
context, the batch configuration for describing the Job, Step,
along with the associated configured policies, field descriptors,
and line mappers. !SimpleTradeConfiguration.jpg! [Figure 2.3.1]
Simple Container Configuration</para>
<para>The details of this configuration will be covered
thoroughly in subsequent sect1s of the document but for now it
should be understood that Job, the Step, the input template, the
file descriptor with its associated line mapper, and the output
(e.g. the TradeWriter).</para>
</section>
<section>
<title id="s.3.2">Single VM Multi-threaded Batch Execution
Container Configuration - One Job, One Step, Multiple
Partitions</title>
<para>In a Single JVM using partitioning a multi-threaded
execution is supported. [This is still work in progress]</para>
</section>
<section>
<title id="s.3.3">Batch Execution Container Hosted in J2EE
Container - managed environment</title>
<para>The J2EE container model has fallen under fire over the
past few years for many valid reasons. There are some things that
the J2EE container do very well though that projects should
consider when planning for scalability with batch architectures.
Commercial and open source containers like WebSphere, BEA and
JBOSS typically:</para>
<itemizedlist>
<listitem>
<para>manage datasources effectively along with attendent
services like prepared statement caching.</para>
</listitem>
<listitem>
<para>manage transactions effectively including many
configurable properties for long lived transactions.</para>
</listitem>
<listitem>
<para>manage thread pools more effectively.</para>
</listitem>
<listitem>
<para>supply robust implementations of JTA, a requirement
when batch jobs output to multiple XA resources like JMS and
JDBC.</para>
</listitem>
<listitem>
<para>manage distribution effectively including domains,
clusters and cells</para>
</listitem>
<listitem>
<para>provide robust JMX management for configuring, managing
and administering distributed applications.</para>
</listitem>
<listitem>
<para>workload management facilities (clusters) provided by
J2EE vendors</para>
</listitem>
</itemizedlist>
<para>Projects are encouraged to deploy batch applications with
the simplest configuration possible, but when federated JVMs are
a requirement to process volumes of data within a batch window,
batch-in-container provides an effective way of distributing the
processing. Spring Batch supports this through a simple change in
configuration. [Work in progress on the exact implementation -
being released as part of M2].</para>
</section>
</section>
</chapter>

274
src/site/docbook/reference/container-overview.xml
View File

@@ -1,274 +0,0 @@
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<chapter id="container-overview">
<title>Overview of the Spring Batch Environment</title>
<section>
<title id="s.1">
Overview of the Spring Batch Simple Batch Execution
Environment
</title>
<para>
The diagram below provides an overview of the high level
components, technical services, and basic operations
required by a batch architecture. This architecture
framework is a blueprint that has been proven through
decades of implementations on the last several generations
of platforms (COBOL/Mainframe, C++/Unix, and now
Java/anywhere). The Simple Batch Execution Environment
provides a physical implementation of the layers, components
and technical services commonly found in robust,
maintainable systems used to address the creation of simple
to complex batch applications, with the infrastructure and
extensions to address very complex processing needs. The
materials below will walk through the details of the
diagram.
</para>
</section>
<section>
<title id="s.2">
Simple Batch Execution Environment high level flow and
interaction of the architecture.
</title>
<mediaobject>
<imageobject role="fo">
<imagedata
fileref="images/simple-batch-execution-env.jpg"
format="JPG" align="center"
width="66%" />
</imageobject>
<imageobject role="html">
<imagedata
fileref="images/simple-batch-execution-env.jpg" format="JPG"
align="center"
width="66%" />
</imageobject>
<caption>
<para>Figure 1: Batch Execution Environment</para>
</caption>
</mediaobject>
<para>
Tiers The application style is organized into four logical
tiers, which include Run, Job, Application, and Data tiers.
The primary goal for organizing an application according to
the tiers is to embed what is known as &quot;separation of
concerns&quot; within the system. Effective separation of
concerns results in reducing the impact of change to the
system.
</para>
<itemizedlist>
<listitem>
<para>
<emphasis role="bold">Run Tier:</emphasis>
The Run Tier is concerned with the scheduling and
launching of the application. A vendor product is
typically used in this tier to allow time-based and
interdependent scheduling of batch jobs as well as
providing parallel processing capabilities.
</para>
</listitem>
<listitem>
<para>
<emphasis role="bold">Job Tier:</emphasis>
The Job Tier is responsible for the overall
execution of a batch job. It sequentially executes
batch steps, ensuring that all steps are in the
correct state and all appropriate policies are
enforced.
</para>
</listitem>
<listitem>
<para>
<emphasis role="bold">Application Tier:</emphasis>
The Application Tier contains components required to
execute the program. It contains specific tasklets
that address the required batch functionality and
enforces policies around a tasklet execution (e.g.,
commit intervals, capture of statistics, etc.)
</para>
</listitem>
<listitem>
<para>
<emphasis role="bold">Data Tier:</emphasis>
The Data Tier provides the integration with the
physical data sources that might include databases,
files, or queues.
<emphasis role="bold">Note</emphasis>
: In some cases the Job tier can be completely
missing and in other cases one Job Script can start
several Batch Job instances.
</para>
</listitem>
</itemizedlist>
</section>
<section>
<title id="s.3">High Level Processing Flow</title>
<para>
The diagram above illustrates the flow and architecture
components in a typical batch run execution.
</para>
<para>Standard interaction is described as follows:</para>
<para>
<emphasis role="bold">1.</emphasis>
In the Run tier, a Scheduler starts a batch application by
invoking a Job Script. The Scheduler identifies what batch
process it wants to run by passing the name of the batch
process and any required additional parameters to the Job
Script.
</para>
<para>
<emphasis role="bold">2.</emphasis>
The Job Script initializes the program and executes any job
specific scripts prior to calling the Batch Launcher.
</para>
<para>
<emphasis role="bold">3.</emphasis>
The Batch Launcher starts the Batch Execution Environment
based upon any environment settings established in the
script.
</para>
<para>
<emphasis role="bold">3.1</emphasis>
The Batch Environment starts and controls the batch execution.
It initializes the Job execution environment with static
configuration items such as database settings, logging
levels and creates a Job based on the Job Configuration
created by a Batch Developer.
</para>
<para>
<emphasis role="bold">4</emphasis>
Based on configuration provided by a Batch Developer, the
Job sequentially executes steps after checking policies to
ensure that each step should be started. The status of the
job and step (start time, end time, status such as
&quot;started&quot; or &quot;completed&quot;) is stored at
various points during the process.
</para>
<para>
<emphasis role="bold">5.1</emphasis>
In order to maintain data integrity, at the application
tier, the Step acts as a controller to ensure that either an
entire group of actions completes successfully or that none
of the actions completes. This group of actions is referred
to as a logical unit of work (LUW). The Step controls the
overall execution of the Tasklet, ensuring that transaction
are committed at the appropriate time, and restart and
statistics information is stored appropriately. The first
thing the Step is responsible for is the initialization of
the data required to begin processing. The Step will
interact with other architecture components, such as the
Input Source, to setup the data required to be processed.
</para>
<para>
<emphasis role="bold">5.1.1</emphasis>
The Input Source provides services to access various data
sources. It provides location transparency to the Batch
Tasklet and hides the physical location details of the data.
</para>
<para>
<emphasis role="bold">5.2</emphasis>
Once the data is initialized by the Input Source, the Step
will call into the Tasklet to begin processing. The Tasklet
contains the business logic to define the LUW and the Step
repeatedly calls the Tasklets LUW to finish the business
function. The Step does this by first invoking the execute
method on the Tasklet in order to acquire a single
record/set of data for processing.
</para>
<para>
<emphasis role="bold">5.2.1</emphasis>
Before a record is returned to the Tasklet, it may be
validated by any number of validation Frameworks that can be
provided to an input source. A single record/set of data is
gathered by interacting with the Input Source.
</para>
<para>
<emphasis role="bold">5.3</emphasis>
Once a record/set has been obtained, the step calls the
tasklet to begin processing.
</para>
<para>
<emphasis role="bold">5.3.1</emphasis>
The Tasklet executes its internal business logic by calling
other Business Logic components as necessary. Based on the
business service, it requests or persists objects from the
data access components.
</para>
<para>
<emphasis role="bold">5.3.3</emphasis>
Data Access components can be leveraged retrieve or persist
domain objects.
</para>
<para>
<emphasis role="bold">5.3.4</emphasis>
Once the business logic has been executed, the resulting
output record is written out by utilizing the Output Source
interface. The Step will repeatedly call steps 5.2 -&gt; 5.3
for every record provided by the Input Source.
</para>
<para>
<emphasis role="bold">5.4</emphasis>
Once all of the records are processed, the Step calls the
Tasklet to perform any clean up activities such as closing
connections, exporting files, etc.
</para>
<para>
<emphasis role="bold">5.4.1</emphasis>
The Step is responsible for committing data associated with
the remaining logical units of work as well as performing
any finalization and administrative functions (e.g. closing
database connections).
</para>
<para>
Once the Step has completed finalization the control is
passed back to the Job, where any necessary logging or clean
up is executed for application termination and wrap-up --
provided there are no additional Steps to execute.
</para>
</section>
</chapter>

6
src/site/docbook/reference/glossary.xml
View File

@@ -63,8 +63,8 @@
<glossterm>Tasklet</glossterm>
<glossdef>
<para>An application program created by application developer to
process the business logic for an entire Step.</para>
<para>A component created by application developer to process the
business logic for a Step.</para>
</glossdef>
</glossentry>
@@ -213,4 +213,4 @@
</glossdef>
</glossentry>
</glossdiv>
</glossary>
</glossary>

383
src/site/docbook/reference/step.xml
View File

@@ -56,16 +56,16 @@
<para>Below is a code representation of the same concepts shown
above:</para>
<programlisting>
<programlisting><![CDATA[
List items = new Arraylist();
for(int i = 0; i &lt; commitInterval; i++){
for(int i = 0; i < commitInterval; i++){
Object item = itemReader.read()
Object processedItem = itemProcessor.process(item);
items.add(processedItem);
}
itemWriter.write(items);
</programlisting>
]]></programlisting>
<section>
<title>Configuring a Step</title>
@@ -75,14 +75,14 @@
potentially contain many collaborators. In order to ease configuration,
the Spring Batch namespace can be used:</para>
<programlisting>
&lt;job id="sampleJob"&gt;
&lt;step id="step1" job-repository="jobRepository" transaction-manager="transactionManager"&gt;
&lt;tasklet reader="itemReader" writer="itemWriter" commit-interval="10"/&gt;
&lt;/step&gt;
&lt;/job&gt;
<programlisting><![CDATA[
<job id="sampleJob">
<step id="step1" job-repository="jobRepository" transaction-manager="transactionManager">
<tasklet reader="itemReader" writer="itemWriter" commit-interval="10"/>
</step>
</job>
</programlisting>
]]></programlisting>
<para>The configuration above represents the only required dependencies
to create a item-oriented step:<itemizedlist>
@@ -255,23 +255,23 @@
<section>
<title>Step restart configuration example</title>
<programlisting>
&lt;job id="footballJob" restartable="true"&gt;
&lt;step id="playerload" next="gameLoad"&gt;
&lt;tasklet reader="playerFileItemReader" writer="playerWriter"
commit-interval="10" /&gt;
&lt;/step&gt;
&lt;step id="gameLoad" next="playerSummarization"&gt;
&lt;tasklet reader="gameFileItemReader" writer="gameWriter"
commit-interval="10" allow-start-if-complete="true"/&gt;
&lt;/step&gt;
&lt;step id="playerSummarization"&gt;
&lt;tasklet reader="playerSummarizationSource" writer="summaryWriter"
commit-interval="10" start-limit="3"/&gt;
&lt;/step&gt;
&lt;/job&gt;
<programlisting><![CDATA[
<job id="footballJob" restartable="true">
<step id="playerload" next="gameLoad">
<tasklet reader="playerFileItemReader" writer="playerWriter"
commit-interval="10" />
</step>
<step id="gameLoad" next="playerSummarization">
<tasklet reader="gameFileItemReader" writer="gameWriter"
commit-interval="10" allow-start-if-complete="true"/>
</step>
<step id="playerSummarization">
<tasklet reader="playerSummarizationSource" writer="summaryWriter"
commit-interval="10" start-limit="3"/>
</step>
</job>
</programlisting>
]]></programlisting>
<para>The above example configuration is for a job that loads in
information about football games and summarizes them. It contains
@@ -483,15 +483,15 @@
newlines. Prefixing a class name with the "+" symbol will indicate that
that exception should not cause rollback.</para>
<programlisting>
&lt;step id="step1"&gt;
&lt;tasklet reader="itemReader" writer="itemWriter" commit-interval="2" skip-limit="1"/&gt;
&lt;transaction-attributes&gt;
<programlisting><![CDATA[
<step id="step1">
<tasklet reader="itemReader" writer="itemWriter" commit-interval="2" skip-limit="1"/>
<transaction-attributes>
+org.springframework.batch.item.validator.ValidationException
&lt;/transaction-attributes&gt;
&lt;/step&gt;
</transaction-attributes>
</step>
</programlisting>
]]></programlisting>
<para>Transaction attributes can also be used to control other settings
such as isolation and propagation behavior. More information on setting
@@ -593,15 +593,15 @@
interface (or an extension thereof) can be applied to a step via the
listeners element:</para>
<programlisting>
&lt;step id="step1"&gt;
&lt;tasklet reader="reader" writer="writer" commit-interval="10"/&gt;
&lt;listeners&gt;
&lt;listener ref="stepListener"/&gt;
&lt;/listeners&gt;
&lt;/step&gt;
<programlisting><![CDATA[
<step id="step1">
<tasklet reader="reader" writer="writer" commit-interval="10"/>
<listeners>
<listener ref="stepListener"/>
</listeners>
</step>
</programlisting>
]]></programlisting>
<para>In addition to the <classname>StepListener</classname> interfaces,
annotations are provided to address the same concerns.</para>
@@ -614,7 +614,7 @@
for notification before a <classname>Step</classname> is started and
after it has ends, whether it ended normally or failed:</para>
<programlisting>
<programlisting><![CDATA[
public interface StepExecutionListener extends StepListener {
void beforeStep(StepExecution stepExecution);
@@ -622,7 +622,7 @@
ExitStatus afterStep(StepExecution stepExecution);
}
</programlisting>
]]></programlisting>
<para><classname>ExitStatus</classname> is the return type of
<methodname>afterStep</methodname> in order to allow listeners the
@@ -651,12 +651,12 @@
useful to perform logic before a chunk begins processing or after a
chunk has completed:</para>
<programlisting> public interface ChunkListener extends StepListener {
<programlisting><![CDATA[ public interface ChunkListener extends StepListener {
void beforeChunk();
void afterChunk();
}</programlisting>
}]]></programlisting>
<para>The <methodname>beforeChunk</methodname> method is called after
the transaction is started, but before <methodname>read</methodname>
@@ -685,14 +685,14 @@
<para>When discussing skip logic above, it was mentioned that it may
be beneficial to log out skipped records, so that they can be deal
with later. In the case of read errors, this can be done with an
<classname>ItemReaderListener:</classname><programlisting> public interface ItemReadListener&lt;T&gt; extends StepListener {
<classname>ItemReaderListener:</classname><programlisting><![CDATA[ public interface ItemReadListener<T> extends StepListener {
void beforeRead();
void afterRead(T item);
void onReadError(Exception ex);
}</programlisting></para>
}]]></programlisting></para>
<para>The <methodname>beforeRead</methodname> method will be called
before each call to <methodname>read</methodname> on the
@@ -727,14 +727,14 @@
<para>Just as with the <classname>ItemReadListener</classname>, the
processing of an item can be 'listened' to:</para>
<programlisting> public interface ItemProcessListener&lt;T, S&gt; extends StepListener {
<programlisting><![CDATA[ public interface ItemProcessListener<T, S> extends StepListener {
void beforeProcess(T item);
void afterProcess(T item, S result);
void onProcessError(T item, Exception e);
}</programlisting>
}]]></programlisting>
<para>The <methodname>beforeProcess</methodname> method will be called
before <methodname>process</methodname> on the
@@ -769,14 +769,14 @@
<para>The writing of an item can be 'listened' to with the
<classname>ItemWriteListener</classname>:</para>
<programlisting> public interface ItemWriteListener&lt;S&gt; extends StepListener {
<programlisting><![CDATA[ public interface ItemWriteListener<S> extends StepListener {
void beforeWrite(List&lt;? extends S&gt; items);
void beforeWrite(List<? extends S> items);
void afterWrite(List&lt;? extends S&gt; items);
void afterWrite(List<? extends S> items);
void onWriteError(Exception exception, List&lt;? extends S&gt; items);
}</programlisting>
void onWriteError(Exception exception, List<? extends S> items);
}]]></programlisting>
<para>The <methodname>beforeWrite</methodname> method will be called
before <methodname>write</methodname> on the
@@ -816,8 +816,8 @@
this reason, there is a separate interface for tracking skipped
items:</para>
<programlisting>
public interface SkipListener&lt;T,S&gt; extends StepListener {
<programlisting><![CDATA[
public interface SkipListener<T,S> extends StepListener {
void onSkipInRead(Throwable t);
@@ -826,7 +826,7 @@
void onSkipInWrite(S item, Throwable t);
}
</programlisting>
]]></programlisting>
<para><methodname>onSkipInRead</methodname> will be called whenever an
item is skipped while reading. It should be noted that rollbacks may
@@ -889,18 +889,19 @@
consist as a simple stored procedure call? You could implement the call as
an <classname>ItemReader</classname> and return null after the procedure
finishes, but it is a bit unnatural since there would need to be a no-op
<classname>ItemWriter</classname> and lots of overhead for transaction
handling, listeners, etc. Spring Batch provides an implementation of
<classname>Step</classname> for this scenario:
<classname>TaskletStep</classname>. As explained in Chapter 2, the
<classname>Tasklet</classname> is a simple interface that has one method,
<methodname>execute</methodname>, which will be a called once for the
whole <classname>Step</classname>. <classname> Tasklet</classname>
implementors might call a stored procedure, a script, or a simple SQL
update statement. To create a <classname>TaskletStep</classname>, the
'tasklet' attribute should reference a bean defining a
<classname>Tasklet</classname> object; no 'tasklet' element is needed
within the 'step':</para>
<classname>ItemWriter</classname>. Spring Batch provides the
<classname>TaskletStep</classname> for this scenario. </para>
<para>The <classname>Tasklet</classname> is a simple interface that has
one method, <methodname>execute</methodname>, which will be a called
repeatedly by the <classname>TaskletStep</classname> until it either
returns <literal>FINISHED</literal> or throws an exception to signal a
failure. Each call to the <classname>Tasklet</classname> is wrapped in a
transaction. <classname>Tasklet</classname> implementors might call a
stored procedure, a script, or a simple SQL update statement. To create a
<classname>TaskletStep</classname>, the 'tasklet' attribute should
reference a bean defining a <classname>Tasklet</classname> object; no
'tasklet' element is needed within the 'step':</para>
<programlisting>
&lt;step id="step1" <emphasis role="bold">tasklet="myTasklet"</emphasis> /&gt;
@@ -926,15 +927,15 @@
this class without having to write an adapter for the
<classname>Tasklet</classname> interface:</para>
<programlisting>
&lt;bean id="myTasklet" class="org.springframework.batch.core.step.tasklet.TaskletAdapter"&gt;
&lt;property name="targetObject"&gt;
&lt;bean class="org.mycompany.FooDao"&gt;
&lt;/property&gt;
&lt;property name="targetMethod" value-"updateFoo" /&gt;
&lt;/bean&gt;
<programlisting><![CDATA[
<bean id="myTasklet" class="org.springframework.batch.core.step.tasklet.TaskletAdapter">
<property name="targetObject">
<bean class="org.mycompany.FooDao">
</property>
<property name="targetMethod" value-"updateFoo" />
</bean>
</programlisting>
]]></programlisting>
</section>
<section>
@@ -949,7 +950,7 @@
project, is a <classname>Tasklet</classname> implementation with just
such a responsibility:</para>
<programlisting> public class FileDeletingTasklet implements Tasklet, InitializingBean {
<programlisting><![CDATA[ public class FileDeletingTasklet implements Tasklet, InitializingBean {
private Resource directory;
@@ -958,7 +959,7 @@
Assert.state(dir.isDirectory());
File[] files = dir.listFiles();
for (int i = 0; i &lt; files.length; i++) {
for (int i = 0; i < files.length; i++) {
boolean deleted = files[i].delete();
if (!deleted) {
throw new UnexpectedJobExecutionException("Could not delete file " + files[i].getPath());
@@ -974,7 +975,7 @@
public void afterPropertiesSet() throws Exception {
Assert.notNull(directory, "directory must be set");
}
}</programlisting>
}]]></programlisting>
<para>The above <classname>Tasklet</classname> implementation will
delete all files within a given directory. It should be noted that the
@@ -982,22 +983,22 @@
that is left is to reference the <classname>Tasklet</classname> from the
<classname>Step</classname>:</para>
<programlisting>
&lt;job id="taskletJob"&gt;
&lt;step id="deleteFilesInDir" tasklet="fileDeletingTasklet"/&gt;
&lt;/job&gt;
<programlisting><![CDATA[
<job id="taskletJob">
<step id="deleteFilesInDir" tasklet="fileDeletingTasklet"/>
</job>
&lt;bean id="fileDeletingTasklet"
class="org.springframework.batch.sample.tasklet.FileDeletingTasklet"&gt;
&lt;property name="directoryResource"&gt;
&lt;bean id="directory"
class="org.springframework.core.io.FileSystemResource"&gt;
&lt;constructor-arg value="target/test-outputs/test-dir" /&gt;
&lt;/bean&gt;
&lt;/property&gt;
&lt;/bean&gt;
<bean id="fileDeletingTasklet"
class="org.springframework.batch.sample.tasklet.FileDeletingTasklet">
<property name="directoryResource">
<bean id="directory"
class="org.springframework.core.io.FileSystemResource">
<constructor-arg value="target/test-outputs/test-dir" />
</bean>
</property>
</bean>
</programlisting>
]]></programlisting>
</section>
</section>
@@ -1034,18 +1035,18 @@
<para>This can be achieved using the 'next' attribute of the step
element:</para>
<para><programlisting>
&lt;job id="job"&gt;
&lt;step id="stepA" next="stepB" /&gt;
&lt;step id="stepB" next="stepC"/&gt;
&lt;step id="stepC" /&gt;
&lt;/job&gt;
<para><programlisting><![CDATA[
<job id="job">
<step id="stepA" next="stepB" />
<step id="stepB" next="stepC"/>
<step id="stepC" />
</job>
</programlisting>In the scenario above, 'step A' will execute first because it
is the first <classname>Step</classname> listed. If 'step A' completes
normally, then 'step B' will execute, and so on. However, if 'step A'
fails, then the entire <classname>Job</classname> will fail and 'step B'
will not execute.</para>
]]></programlisting>In the scenario above, 'step A' will execute first because
it is the first <classname>Step</classname> listed. If 'step A'
completes normally, then 'step B' will execute, and so on. However, if
'step A' fails, then the entire <classname>Job</classname> will fail and
'step B' will not execute.</para>
<note>
<para>With the Spring Batch namespace, the first step listed in the
@@ -1103,17 +1104,17 @@
<para>The next element specifies a pattern to match and the step to
execute next:</para>
<para><programlisting>
&lt;job id="job"&gt;
&lt;step id="stepA"&gt;
&lt;next on="FAILED" to="stepB" /&gt;
&lt;next on="*" to="stepC" /&gt;
&lt;/step&gt;
&lt;step id="stepB" next="stepC" /&gt;
&lt;step id="stepC" /&gt;
&lt;/job&gt;
<para><programlisting><![CDATA[
<job id="job">
<step id="stepA">
<next on="FAILED" to="stepB" />
<next on="*" to="stepC" />
</step>
<step id="stepB" next="stepC" />
<step id="stepC" />
</job>
</programlisting></para>
]]></programlisting></para>
<para>The "on" attribute of a transition element uses a simple
pattern-matching scheme to match the <classname>ExitStatus</classname>
@@ -1161,10 +1162,10 @@
it fails, and so on. The example above contains the following 'next'
element:</para>
<programlisting>
&lt;next on="FAILED" to="stepB" /&gt;
<programlisting><![CDATA[
<next on="FAILED" to="stepB" />
</programlisting>
]]></programlisting>
<para>At first glance, it would appear that the 'on' attribute
references the <classname>BatchStatus</classname> of the
@@ -1181,14 +1182,14 @@
code needs to be different? A good example comes from the skip sample
job within the samples project:</para>
<programlisting>
&lt;step id="step1"&gt;
&lt;end on="FAILED" /&gt;
&lt;next on="COMPLETED WITH SKIPS" to="errorPrint1" /&gt;
&lt;next on="*" to="step2" /&gt;
&lt;/step&gt;
<programlisting><![CDATA[
<step id="step1">
<end on="FAILED" />
<next on="COMPLETED WITH SKIPS" to="errorPrint1" />
<next on="*" to="step2" />
</step>
</programlisting>
]]></programlisting>
<para>The above step has three possibilities:</para>
@@ -1214,18 +1215,18 @@
change the exit code based on the condition of the execution having
skipped records:</para>
<programlisting> public class SkipCheckingListener extends StepExecutionListenerSupport {
<programlisting><![CDATA[ public class SkipCheckingListener extends StepExecutionListenerSupport {
public ExitStatus afterStep(StepExecution stepExecution) {
if (!stepExecution.getExitStatus().getExitCode().equals(ExitStatus.FAILED.getExitCode())
&amp;&amp; stepExecution.getSkipCount() &gt; 0) {
&& stepExecution.getSkipCount() > 0) {
return new ExitStatus("COMPLETED WITH SKIPS");
} else {
return null;
}
}
}</programlisting>
}]]></programlisting>
<para>The above code is a <classname>StepExecutionListener</classname>
that first checks to make sure the <classname>Step</classname> was
@@ -1253,7 +1254,7 @@
after the following step executes, the <classname>Job</classname> will
end:</para>
<para><programlisting> &lt;step id="stepC" /&gt;</programlisting></para>
<para><programlisting><![CDATA[ <step id="stepC" />]]></programlisting></para>
<para>If no transitions are defined for a <classname>Step</classname>,
then the <classname>Job</classname>'s statuses will be defined as
@@ -1312,12 +1313,12 @@
fails, the <classname>Job</classname> will not be restartable (because
the status is COMPLETED).</para>
<programlisting> &lt;step id="step1" next="step2"&gt;
&lt;step id="step2"&gt;
&lt;end on="FAILED"/&gt;
&lt;next on="*" to="step3"/&gt;
&lt;/step&gt;
&lt;step id="step3"&gt;</programlisting>
<programlisting><![CDATA[ <step id="step1" next="step2">
<step id="step2">
<end on="FAILED"/>
<next on="*" to="step3"/>
</step>
<step id="step3">]]></programlisting>
</section>
<section>
@@ -1341,12 +1342,12 @@
Additionally, if step2 fails, and the <classname>Job</classname> is
restarted, then execution will begin again on step2.</para>
<programlisting> &lt;step id="step1" next="step2"&gt;
&lt;step id="step2"&gt;
&lt;fail on="FAILED" exit-code="EARLY TERMINATION"/&gt;
&lt;next on="*" to="step3"/&gt;
&lt;/step&gt;
&lt;step id="step3"&gt;</programlisting>
<programlisting><![CDATA[ <step id="step1" next="step2">
<step id="step2">
<fail on="FAILED" exit-code="EARLY TERMINATION"/>
<next on="*" to="step3"/>
</step>
<step id="step3">]]></programlisting>
</section>
<section>
@@ -1364,10 +1365,10 @@
the job will then stop. Once it is restarted, execution will begin on
step2.</para>
<para><programlisting> &lt;step id="step1"&gt;
&lt;stop on="COMPLETED" restart="step2"/&gt;
&lt;/step&gt;
&lt;step id="step2"/&gt;</programlisting></para>
<para><programlisting><![CDATA[ <step id="step1">
<stop on="COMPLETED" restart="step2"/>
</step>
<step id="step2"/>]]></programlisting></para>
</section>
</section>
@@ -1380,7 +1381,7 @@
<classname>JobExecutionDecider</classname> can be used to assist in the
decision.</para>
<para><programlisting>
<para><programlisting><![CDATA[
public class MyDecider implements JobExecutionDecider {
public String decide(JobExecution jobExecution, StepExecution stepExecution) {
@@ -1394,27 +1395,27 @@
}
</programlisting></para>
]]></programlisting></para>
<para>In the job configuration, a "decision" tag will specify the
decider to use as well as all of the transitions.</para>
<para><programlisting>
&lt;job id="job"&gt;
&lt;step id="step1" next="decision" /&gt;
<para><programlisting><![CDATA[
<job id="job">
<step id="step1" next="decision" />
&lt;decision id="skipCheckingDecision" decider="decider"&gt;
&lt;next on="FAILED" to="step2" /&gt;
&lt;next on="COMPLETED" to="step3" /&gt;
&lt;/step&gt;
<decision id="skipCheckingDecision" decider="decider">
<next on="FAILED" to="step2" />
<next on="COMPLETED" to="step3" />
</step>
&lt;step id="step2" next="step3"/&gt;
&lt;step id="step3" /&gt;
&lt;/job&gt;
<step id="step2" next="step3"/>
<step id="step3" />
</job>
&lt;bean id="decider" class="com.MyDecider"/&gt;
<bean id="decider" class="com.MyDecider"/>
</programlisting></para>
]]></programlisting></para>
</section>
<section id="split-flows">
@@ -1431,16 +1432,16 @@
elements such as the 'next' attribute or the 'next', 'end', 'fail', or
'pause' elements.</para>
<programlisting> &lt;split id="split1" next="step4"&gt;
&lt;flow&gt;
&lt;step id="step1" next="step2"/&gt;
&lt;step id="step2"/&gt;
&lt;/flow&gt;
&lt;flow&gt;
&lt;step id="step3"/&gt;
&lt;/flow&gt;
&lt;/split&gt;
&lt;step id="step4"/&gt;</programlisting>
<programlisting><![CDATA[ <split id="split1" next="step4">
<flow>
<step id="step1" next="step2"/>
<step id="step2"/>
</flow>
<flow>
<step id="step3"/>
</flow>
</split>
<step id="step4"/>]]></programlisting>
</section>
</section>
@@ -1454,14 +1455,14 @@
Flat File resources can be configured using standard Spring
constructs:</para>
<programlisting>
&lt;bean id="flatFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"&gt;
&lt;property name="resource"
value="file://outputs/20070122.testStream.CustomerReportStep.TEMP.txt" /&gt;
&lt;/bean&gt;
<programlisting><![CDATA[
<bean id="flatFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource"
value="file://outputs/20070122.testStream.CustomerReportStep.TEMP.txt" />
</bean>
</programlisting>
]]></programlisting>
<para>The above <classname>Resource</classname> will load the file from
the file system location specified. Note that absolute locations have to
@@ -1471,13 +1472,13 @@
at runtime as a parameter to the job. This could be solved using '-D'
parameters, i.e. a system property:</para>
<programlisting>
&lt;bean id="flatFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"&gt;
&lt;property name="resource" value="${input.file.name}" /&gt;
&lt;/bean&gt;
<programlisting><![CDATA[
<bean id="flatFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="${input.file.name}" />
</bean>
</programlisting>
]]></programlisting>
<para>All that would be required for this solution to work would be a
system argument (-Dinput.file.name="file://file.txt"). (Note that although
@@ -1543,22 +1544,22 @@
must be added explicitly, either by using the <literal>batch</literal>
namespace:</para>
<programlisting>
&lt;beans xmlns="http://www.springframework.org/schema/beans"
<programlisting><![CDATA[
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="..."&gt;
xsi:schemaLocation="...">
...
&lt;/beans&gt;
</programlisting>
</beans>
]]></programlisting>
<para>or by including a bean definition explicitly for
the<classname>Step</classname> (but not both):</para>
<programlisting>
&lt;bean class="org.springframework.batch.core.scope.StepScope" /&gt;
<programlisting><![CDATA[
<bean class="org.springframework.batch.core.scope.StepScope" />
</programlisting>
]]></programlisting>
</section>
</section>
</chapter>
</chapter>