diff --git a/docs/models/Figures.ppt b/docs/models/Figures.ppt
index 081ba8c42..983683c5a 100644
Binary files a/docs/models/Figures.ppt and b/docs/models/Figures.ppt differ
diff --git a/docs/models/diagrams.ppt b/docs/models/diagrams.ppt
new file mode 100755
index 000000000..2b6a1392a
Binary files /dev/null and b/docs/models/diagrams.ppt differ
diff --git a/docs/src/site/docbook/reference/Discard.xml b/docs/src/site/docbook/reference/Discard.xml
deleted file mode 100644
index 4f66e8d3e..000000000
--- a/docs/src/site/docbook/reference/Discard.xml
+++ /dev/null
@@ -1,33 +0,0 @@
-
-
-
- Discard
-
-
- Support Stereotypes
-
- While item readers and writers serve as the main entry points for
- item-oriented processing, they might be supplemented by a number of
- support classes that perform specific tasks within the reader / writer
- lifecycle. These support stereotypes are useful for dividing the work of
- item readers and writers into reusable pieces, as well as abstracting away
- the details of processing, such as interaction with external systems.
- Additionally, they give us another opportunity to leverage the powerful
- configuration features of the Spring framework, as we can switch between
- several beans implementing these support interfaces without changing the
- driving item reader or writer.
-
-
- Item Transformers
-
- An item transformer is a class that is capable of taking an object
- and changing it somehow before processing occurs. For instance, an item
- transformer my alter an object by changing its properties or by
- replacing it with another object entirely, such as a wrapper or
- derivative object. It can also be defined as an adaptor, allowing an
- object of one type to be converted for use as an object of a second
- type.
-
-
-
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/batch-job-testing.xml b/docs/src/site/docbook/reference/batch-job-testing.xml
deleted file mode 100644
index ec54f8f81..000000000
--- a/docs/src/site/docbook/reference/batch-job-testing.xml
+++ /dev/null
@@ -1,17 +0,0 @@
-
-
- Batch Unit and Integration Tests
-
-
- Unit Testing
- Document Batch Job Unit Testing features. This includes the use of Mock Objects,
- embedded database (HSQLDB), etc.
-
-
-
- Integration Testing
- Document how to test against the targeted database, applications, etc.
-
-
-
diff --git a/docs/src/site/docbook/reference/batch-launch.xml b/docs/src/site/docbook/reference/batch-launch.xml
deleted file mode 100644
index 6795bcbc8..000000000
--- a/docs/src/site/docbook/reference/batch-launch.xml
+++ /dev/null
@@ -1,17 +0,0 @@
-
-
- Run Tier - Launching Batch Jobs
-
- Mapping Batch Error Codes to Launch Client Error Codes
- Mapping Batch Error Codes to Launch Client Error Codes
-
-
- Launch Batch from Command Line
- Document Command Line Launching
-
-
- Launch Batch On Demand
- Document Launching Batch Jobs on Demand
-
-
diff --git a/docs/src/site/docbook/reference/batch-performance-testing.xml b/docs/src/site/docbook/reference/batch-performance-testing.xml
deleted file mode 100644
index 8e2b30fdc..000000000
--- a/docs/src/site/docbook/reference/batch-performance-testing.xml
+++ /dev/null
@@ -1,52 +0,0 @@
-
-
- Batch Performance Testing
-
- Performance Testing Overview
- A batch performance test team needs to have the following at their disposal:
-
-
-
-
- Define performance Targets -
-
-
- Establishing the requirements for a performance testing environment -
-
-
- Performance Data - generating adequate volumes of realistic data for performance testing
-
-
- Performance Tools -
-
-
- Performance Team Roles - Tool SME's, performance DBA.
-
-
-
-
-
-
- Defining Performance Targets
-
-
-
-
- Establishing Performance requirements and installing the environment.
- Establishing the requirements for the performance environment.
-
-
-
- Performance Data
- Generating adequate volumes of realistic data for performance testing.
-
-
-
- Performance Team Roles
- Tool SME's, performance dba's, environment experts (OS, JVM, etc.)
-
-
diff --git a/docs/src/site/docbook/reference/common-patterns.xml b/docs/src/site/docbook/reference/common-patterns.xml
new file mode 100644
index 000000000..106183f4b
--- /dev/null
+++ b/docs/src/site/docbook/reference/common-patterns.xml
@@ -0,0 +1,232 @@
+
+
+
+ Common Batch Patterns
+
+
+ Introduction
+
+ Some batch jobs can be assembled purely from off-the-shelf
+ components in Spring Batch, mostly the ItemReader
+ and ItemWriter implementations. Where this is not
+ possible (the majority of cases) the main API entry points for application
+ developers are the Tasklet,
+ ItemReader, ItemWriter and
+ the various listener interfaces. Most simple batch jobs will be able to
+ use off-the-shelf input from a Spring Batch
+ ItemReader, but it is very often the case that
+ there are custom concerns in the processing and writing, which normally
+ leads developers to implement an ItemWriter, or
+ ItemTransformer.
+
+ Here we provide a few examples of common patterns in custom business
+ logic, mainly using the listener interfaces . It should be noted that an
+ ItemReader or ItemWriter can
+ implement the listener interfaces as well if appropriate.
+
+
+
+ Logging Item Processing and Failures
+
+ A common use case is the need for special handling of errors in a
+ step, item by item, perhaps logging to a special channel, or inserting a
+ record into a database. The StepHandlerStep
+ (created from the step factory beans) allows users to implement this use
+ case with a simple ItemReadListener, for errors on
+ read, and an ItemWriteListener, for errors on
+ write. The below code snippets illustrate a listener that logs both read
+ and write failures:
+
+ public class ItemFailureLoggerListener extends ItemListenerSupport {
+
+ private static Log logger = LogFactory.getLog("item.error");
+
+ public void onReadError(Exception ex) {
+ logger.error("Encountered error on read", e);
+ }
+
+ public void onWriteError(Exception ex, Object item) {
+ logger.error("Encountered error on write", e);
+ }
+
+}
+
+ Having implemented this listener it must be registered with the
+ step:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ ...
+ <property name="listeners">
+ <bean class="org.example...ItemFailureLoggerListener"/>
+ </property>
+ </bean>
+
+ Remember that if your listener does anything in an
+ onError() method, it will be inside a transaction that is
+ going to be rolled back. If you need to use a transactional resource such
+ as a database inside an onError() method, consider adding a
+ declarative transaction to that method (see Spring Core Reference Guide
+ for details), and giving its propagation attribute the value
+ REQUIRES_NEW.
+
+
+
+ Stopping a Job Manually for Business Reasons
+
+ Spring Batch provides a stop() method through the JobLauncher
+ interface, but this is really aimed at the operator, rather than the
+ application programmer. Sometimes it is more convenient or makes more
+ sense to stop a job execution from within the business logic.
+
+ The simplest thing to do is to throw a RuntimeException (one that
+ isn't retried indefinitely or skipped), For example, a custom exception
+ type could be used, as in the example below:
+
+ public class PoisonPillItemWriter extends AbstractItemWriter {
+
+ public void write(Object item) throws Exception {
+
+ if (isPoisonPill(item)) {
+ throw new PoisonPillException("Posion pill detected: "+item);
+ }
+
+ }
+
+}
+
+ Another simple way to stop a step from executing is to simply return
+ null from the ItemReader:
+
+ public class EarlyCompletionItemReader extends AbstractItemReader {
+
+ private ItemReader delegate;
+
+ public void setDelegate(ItemReader delegate) { ... }
+
+ public Object read() throws Exception {
+
+ Object item = delegate.read();
+
+ if (isEndItem(item)) {
+ return null; // end the step here
+ }
+
+ return item;
+
+ }
+
+}
+
+ The previous example actually relies on the fact that there is a
+ default implementation of the CompletionPolicy
+ strategy which signals a complete batch when the item to be processed is
+ null. A more sophisticated completion policy could be implemented and
+ injected into the Step through the
+ RepeatOperationsStepFactoryBean:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" >
+ ...
+ <property name="chunkOperations">
+ <bean class="org.springframework.batch.repeat.support.RepeatTemplate">
+ <property name="completionPolicy">
+ <bean class="org.example...SpecialCompletionPolicy"/>
+ </property>
+ </bean>
+ </property>
+ </bean>
+
+ An alternative is to set a flag in the
+ StepExecution, which is checked by the
+ Step implementations in the framework in between
+ item processing. To implement this alternative, we need access to the
+ current StepExecution, and this can be achieved by implementing a
+ StepListener and registering it with the Step. Here is an example of a
+ listener that sets the flag:
+
+ public class CustomItemWriter extends ItemListenerSupport implements StepListener {
+
+ private StepExecution stepExecution;
+
+ public void beforeStep(StepExecution stepExecution) {
+ this.stepExecution = stepExecution;
+ }
+
+ public void afterRead(Object item) {
+
+ if (isPoisonPill(item)) {
+ stepExecution.setTerminateOnly(true);
+ }
+
+ }
+
+}
+
+ The default behaviour here when the flag is set is for the step to
+ throw a JobInterruptedException. This can be
+ controlled through the StepInterruptionPolicy, but
+ the only choice is to throw or not throw an exception, so this is always
+ an abnormal ending to a job.
+
+
+
+ Adding a Footer Record
+
+ A very common requirement is to aggregate information during the
+ output process and to append a record at the end of a file summarizing the
+ data, or providing a checksum. This can also be achieved with a callbacks
+ in the step, normally as part of a custom
+ ItemWriter. In this case, since a job is
+ accumulating state that should not be lost if the job aborts, the
+ ItemStream interface should be implemented:
+
+ public class CustomItemWriter extends AbstractItemWriter implements
+ ItemStream, StepListener
+{
+
+ private static final String TOTAL_AMOUNT_KEY = "total.amount";
+
+ private ItemWriter delegate;
+
+ private double totalAmount = 0.0;
+
+ public void setDelegate(ItemWriter delegate) { ... }
+
+ public ExitStatus afterStep(StepExecution stepExecution) {
+ // Add the footer record here...
+ delegate.write("Total Amount Processed: " + totalAmount);
+ }
+
+ public void open(ExecutionContext executionContext) {
+ if (executionContext.containsKey(TOTAL_AMOUNT_KEY) {
+ totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY);
+ }
+ }
+
+ public void update(ExecutionContext executionContext) {
+ executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount);
+ }
+
+ public void write(Object item) {
+
+ delegate.write(item);
+ totalAmount += ((Trade) item).getAmount();
+
+ }
+
+}
+
+ The custom writer in the example is stateful (it maintains its total
+ in an instance variable totalAmount), but the state is
+ stored through the ItemStream interface in the
+ ExecutionContext. In this way we can be sure that
+ when the open() callback is received on a restart. The
+ framework garuntees we always get the last value that was committed. It
+ should be noted that it is not always necessary to implement ItemStream.
+ For example, if the ItemWriter is re-runnable, in the sense that it
+ maintains its own state in a transactional resource like a database, there
+ is no need to maintain state within the writer itself.
+
+
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/core.xml b/docs/src/site/docbook/reference/domain.xml
similarity index 51%
rename from docs/src/site/docbook/reference/core.xml
rename to docs/src/site/docbook/reference/domain.xml
index 26fd1a7d8..d04a1bab8 100644
--- a/docs/src/site/docbook/reference/core.xml
+++ b/docs/src/site/docbook/reference/domain.xml
@@ -4,130 +4,69 @@
The Domain Language of Batch
-
- Introduction
+ To any experienced batch architect, the overall concepts of batch
+ processing used in Spring Batch should be familiar and comfortable. There
+ are “Jobs” and “Steps” and developer supplied processing units called
+ ItemReaders and ItemWriters. However, because of the Spring patterns,
+ operations, templates, callbacks, and idioms, there are opportunities for
+ the following:
+
+ significant improvement in adherence to a clear separation of
+ concerns
+
- To any experienced batch architect, the overall concepts of batch
- processing used in Spring Batch should be familiar and comfortable. There
- are “Jobs” and “Steps” and developer supplied processing units called
- ItemReaders and ItemWriters. However, because of the Spring patterns,
- operations, templates, callbacks, and idioms, there are opportunities for
- the following:
-
- significant improvement in adherence to a clear separation of
- concerns
-
+
+ clearly delineated architectural layers and services provided as
+ interfaces
+
-
- clearly delineated architectural layers and services provided
- as interfaces
-
+
+ simple and default implementations that allowed for quick
+ adoption and ease of use out-of-the-box
+
-
- simple and default implementations that allowed for quick
- adoption and ease of use out-of-the-box
-
+
+ significantly enhanced extensibility
+
+
-
- significantly enhanced extensibility
-
-
+ The diagram below is simplified version of the batch reference
+ architecture that has been used for decades. It provides an overview of the
+ components that make up the domain language of batch processing. This
+ architecture framework is a blueprint that has been proven through decades
+ of implementations on the last several generations of platforms
+ (COBOL/Mainframe, C++/Unix, and now Java/anywhere). JCL and COBOL developers
+ are likely to be as comfortable with the concepts as C++, C# and Java
+ developers. Spring Batch provides a physical implementation of the layers,
+ components and technical services commonly found in robust, maintainable
+ systems used to address the creation of simple to complex batch
+ applications, with the infrastructure and extensions to address very complex
+ processing needs.
- The diagram below is only a slight variation of the batch reference
- architecture that has been used for decades. It provides an overview of
- the high level components, technical services, and basic operations
- required by a batch architecture. This architecture framework is a
- blueprint that has been proven through decades of implementations on the
- last several generations of platforms (COBOL/Mainframe, C++/Unix, and now
- Java/anywhere). JCL and COBOL developers are likely to be as comfortable
- with the concepts as C++, C# and Java developers. Spring Batch provides a
- physical implementation of the layers, components and technical services
- commonly found in robust, maintainable systems used to address the
- creation of simple to complex batch applications, with the infrastructure
- and extensions to address very complex processing needs.
-
+
+
+
+
+
+
+
+
+
+ Figure 2.1: Batch Stereotypes
+
+
+ The diagram above highlights the key concepts that make up the domain
+ language of batch. A Job has one to many steps, which has exactly one
+ ItemReader, ItemProcessor, and ItemWriter. A job needs to be launched
+ (JobLauncher), and meta data about the currently running process needs to be
+ stored (JobRepository)
- Batch Application Style Interactions and Services
-
-
-
-
-
-
-
-
-
-
- Figure 2.1: Batch Stereotypes
-
-
- The above diagram highlights the interactions and key services
- provided by the Spring Batch framework. The colors used are important to
- understanding the responsibilities of a developer in Spring Batch. Grey
- represents an external application such as an enterprise scheduler or a
- database. It's important to note that scheduling is grey, and should thus
- be considered separate from Spring Batch. Blue represents application
- architecture services. In most cases these are provided by Spring Batch
- with out of the box implementations, but an architecture team may make
- specific implementations that better address their specific needs. Yellow
- represents the pieces that must be configured by a developer. For example,
- a job schedule needs to be configured so that the job is kicked off at the
- appropriate time. A job configuration file also needs to be created, which
- defines how a job will be run. It is also worth noting that the
- ItemReader and ItemWriter
- used by an application may just as easily be a custom one made by a
- developer for their specific batch job, rather than one provided by Spring
- Batch or an architecture team.
-
- The Batch Application Style is organized into four logical tiers,
- which include Run, Job, Application, and Data. The primary goal for
- organizing an application according to the tiers is to embed what is known
- as "separation of concerns" within the system. These tiers can be
- conceptual but may prove effective in mapping the deployment of the
- artifacts onto physical components like Java runtimes and integration with
- data sources and targets. Effective separation of concerns results in
- reducing the impact of change to the system. The four conceptual tiers
- containing batch artifacts are:
-
-
-
- Run Tier: The Run Tier is
- concerned with the scheduling and launching of the application. A
- vendor product is typically used in this tier to allow time-based
- and interdependent scheduling of batch jobs as well as providing
- parallel processing capabilities.
-
-
-
- Job Tier: The Job Tier is
- responsible for the overall execution of a batch job. It
- sequentially executes batch steps, ensuring that all steps are in
- the correct state and all appropriate policies are enforced.
-
-
-
- Application Tier: The
- Application Tier contains components required to execute the
- program. It contains specific tasks that address required batch
- functionality and enforces policies around execution (e.g., commit
- intervals, capture of statistics, etc.)
-
-
-
- Data Tier: The Data Tier
- provides integration with the physical data sources that might
- include databases, files, or queues.
-
-
-
-
-
- Job Stereotypes
+ Job
This section describes stereotypes relating to the concept of a
batch job. A Job is an entity that encapsulates an
@@ -138,7 +77,8 @@
-
+
@@ -147,51 +87,40 @@
-
- Job
+ In Spring Batch, a Job is simply a continer for Steps. It combines
+ multiple steps that belong logically together in a flow and allows for
+ configuration of properties global to all steps, such as restartability.
+ The job configuration contains:
- A job is represented by a Spring bean that implements the
- Job interface and contains all of the information
- necessary to define the operations performed by a job. A job
- configuration is typically contained within a Spring XML configuration
- file and the job's name is determined by the "id" attribute associated
- with the job configuration bean. The job configuration contains
+
+
+ The simple name of the job
+
-
-
- The simple name of the job
-
+
+ Definition and ordering of Steps
+
-
- Definition and ordering of Steps
-
+
+ Whether or not the job is restartable
+
+
-
- Whether or not the job is restartable
-
-
+ A default simple implementation of the Job
+ interface is provided by Spring Batch in the form of the
+ SimpleJob class which creates some standard
+ functionality on top of Job, however the batch
+ namespace abstracts away the need to instaniate it directly. Instead, the
+ <job> tag can be used:
- A default simple implementation of the Job
- interface is provided by Spring Batch in the form of the
- SimpleJob class which creates some standard
- functionality on top of Job, namely a standard
- execution logic that all jobs should utilize. In general, all jobs
- should be defined using a bean of type
- SimpleJob:
+
+ <job id="footballJob">
+ <step name="playerload" next="gameLoad"/>
+ <step name="gameLoad" next="playerSummarization"/>
+ <step name="playerSummarization"/>
+ </job>
- <bean id="footballJob"
- class="org.springframework.batch.core.job.SimpleJob">
- <property name="steps">
- <list>
- <!-- Step Bean details ommitted for clarity -->
- <bean id="playerload" parent="simpleStep" />
- <bean id="gameLoad" parent="simpleStep" />
- <bean id="playerSummarization" parent="simpleStep" />
- </list>
- </property>
- <property name="restartable" value="true" />
- </bean>
-
+
JobInstance
@@ -238,13 +167,28 @@
another?" The answer is: JobParameters.
JobParameters are any set of parameters used to
start a batch job, which can be used for identification or even as
- reference data during the run. In the example above, where there are two
- instances, one for January 1st, and another for January 2nd, there is
- really only one Job, one that was started with a job parameter of
- 01-01-2008 and another that was started with a parameter of 01-02-2008.
- Thus, the contract can be defined as: JobInstance
- = Job + JobParameters.
- This allows a developer to effectively control how you a
+ reference data during the run:
+
+
+
+
+
+
+
+
+
+
+
+ In the example above, where there are two instances, one for
+ January 1st, and another for January 2nd, there is really only one Job,
+ one that was started with a job parameter of 01-01-2008 and another that
+ was started with a parameter of 01-02-2008. Thus, the contract can be
+ defined as: JobInstance =
+ Job + JobParameters. This
+ allows a developer to effectively control how a
JobInstance is defined, since they control what
parameters are passed in.
@@ -258,10 +202,10 @@
JobInstance corresponding to a given execution
will not be considered complete unless the execution completes
successfully. Using the EndOfDay Job described
- above as an example, consider a JobInstance for 01-01-2008 that failed
- the first time it was run. If it is ran again, with the same job
- parameters as the first run (01-01-2008), a new JobExecution will be
- created. However, there will still be only one
+ above as an example, consider a JobInstance for
+ 01-01-2008 that failed the first time it was run. If it is ran again,
+ with the same job parameters as the first run (01-01-2008), a new
+ JobExecution will be created. However, there will still be only one
JobInstance.
A Job defines what a job is and how it is
@@ -334,6 +278,15 @@
The 'property bag' containing any user data that needs to
be persisted between executions.
+
+
+ failureExceptions
+
+ The list of exceptions encountered during the execution
+ of a Job. These can be useful if more
+ than one exception is encountered during the failure of a
+ Job.
+
@@ -573,26 +526,29 @@
- Step Stereotypes
+ Step
A Step is a domain object that encapsulates
an independent, sequential phase of a batch job. Therefore, every
Job is composed entirely of one or more steps. A
- Step should be thought of as a unique processing
- stream that will be executed in sequence. For example, if you have one
- step that loads a file into a database, another that reads from the
- database, validates the data, preforms processing, and then writes to
- another table, and another that reads from that table and writes out to a
- file. Each of these steps will be performed completely before moving on to
- the next step. The file will be completely read into the database before
- step 2 can begin. As with Job, a
+ Step contains all of the information necessary to
+ define and control the actual batch processing. This is a necessarily
+ vague description because the contents of any given
+ Step are at the discretion of the developer writing
+ a Job. A Step can be as simple or complex as the
+ developer desires. A simple Step might load data
+ from a file into the database, requiring little or no code. (depending
+ upon the implementations used) A more complex Step
+ may have complicated business rules that are applied as part of the
+ processing. As with Job, a
Step has an individual
StepExecution that corresponds with a unique
JobExecution:
-
+
@@ -601,30 +557,6 @@
-
- Step
-
- A Step contains all of the information
- necessary to define and control the actual batch processing. This is a
- necessarily vague description because the contents of any given
- Step are at the discretion of the developer
- writing a Job. A Step can be as simple or complex
- as the developer desires. A simple Step might
- load data from a file into the database, requiring little or no code.
- (depending upon the implementations used) A more complex
- Step may have complicated business rules that are
- applied as part of the processing.
-
- Steps are defined by instantiating implementations of the
- Step interface. Two step implementation classes
- are available in the Spring Batch framework, and they are each discussed
- in detail in Chatper 4 of this guide. For most situations, the
- StepHandlerStep implementation is sufficient, but
- for situations where only one call is needed, such as a stored procedure
- call or a wrapper around existing script, a
- TaskletStep may be a better option.
-
-
StepExecution
@@ -753,181 +685,173 @@
+
-
- ExecutionContext
+
+ ExecutionContext
- An ExecutionContext represents a collection
- of key/value pairs that are persisted and controlled by the framework in
- order to allow developers a place to store persistent state that is
- scoped to a StepExecution or
- JobExecution. For those familiar with Quartz, it
- is very similar to JobDataMap. The best usage
- example is restart. Using flat file input as an example, while
- processing individual lines, the framework periodically persists the
- ExecutionContext at commit points. This allows
- the ItemReader to store its state in case a fatal
- error occurs during the run, or even if the power goes out. All that is
- needed is to put the current number of lines read into the context, and
- the framework will do the rest:
+ An ExecutionContext represents a collection
+ of key/value pairs that are persisted and controlled by the framework in
+ order to allow developers a place to store persistent state that is scoped
+ to a StepExecution or
+ JobExecution. For those familiar with Quartz, it is
+ very similar to JobDataMap. The best usage example
+ is to facilitate restart. Using flat file input as an example, while
+ processing individual lines, the framework periodically persists the
+ ExecutionContext at commit points. This allows the
+ ItemReader to store its state in case a fatal error
+ occurs during the run, or even if the power goes out. All that is needed
+ is to put the current number of lines read into the context, and the
+ framework will do the rest:
- executionContext.putLong(getKey(LINES_READ_COUNT), reader.getPosition());
+ executionContext.putLong(getKey(LINES_READ_COUNT), reader.getPosition());
- Using the EndOfDay example from the Job Stereotypes section as an
- example, assume there's one step: 'loadData', that loads a file into the
- database. After the first failed run, the meta data tables would look
- like the following:
+ Using the EndOfDay example from the Job Stereotypes section as an
+ example, assume there's one step: 'loadData', that loads a file into the
+ database. After the first failed run, the meta data tables would look like
+ the following:
-
- BATCH_JOB_INSTANCE
+
+ BATCH_JOB_INSTANCE
-
-
-
- JOB_INSTANCE_ID
+
+
+
+ JOB_INSTANCE_ID
- JOB_NAME
-
+ JOB_NAME
+
-
- 1
+
+ 1
- EndOfDayJob
-
-
-
-
- BATCH_JOB_PARAMS
+ EndOfDayJob
+
+
+
+
+ BATCH_JOB_PARAMS
-
-
-
- JOB_INSTANCE_ID
+
+
+
+ JOB_INSTANCE_ID
- TYPE_CD
+ TYPE_CD
- KEY_NAME
+ KEY_NAME
- DATE_VAL
-
+ DATE_VAL
+
-
- 1
+
+ 1
- DATE
+ DATE
- schedule.Date
+ schedule.Date
- 2008-01-01 00:00:00
-
-
-
-
- BATCH_JOB_EXECUTION
+ 2008-01-01 00:00:00
+
+
+
+
+ BATCH_JOB_EXECUTION
-
-
-
- JOB_EXECUTION_ID
+
+
+
+ JOB_EXECUTION_ID
- JOB_INSTANCE_ID
+ JOB_INSTANCE_ID
- START_TIME
+ START_TIME
- END_TIME
+ END_TIME
- STATUS
-
+ STATUS
+
-
- 1
+
+ 1
- 1
+ 1
- 2008-01-01 21:00:23.571
+ 2008-01-01 21:00:23.571
- 2008-01-01 21:30:17.132
+ 2008-01-01 21:30:17.132
- FAILED
-
-
-
-
- BATCH_STEP_EXECUTION
+ FAILED
+
+
+
+
+ BATCH_STEP_EXECUTION
-
-
-
- STEP_EXECUTION_ID
+
+
+
+ STEP_EXECUTION_ID
- JOB_EXECUTION_ID
+ JOB_EXECUTION_ID
- STEP_NAME
+ STEP_NAME
- START_TIME
+ START_TIME
- END_TIME
+ END_TIME
- STATUS
-
+ STATUS
+
-
- 1
+
+ 1
- 1
+ 1
- loadDate
+ loadDate
- 2008-01-01 21:00:23.571
+ 2008-01-01 21:00:23.571
- 2008-01-01 21:30:17.132
+ 2008-01-01 21:30:17.132
- FAILED
-
-
-
-
- BATCH_EXECUTION_CONTEXT
+ FAILED
+
+
+
+
+ BATCH_STEP_EXECUTION_CONTEXT
-
-
-
- EXECUTION_ID
+
+
+
+ STEP_EXECUTION_ID
- TYPE_CD
+ SHORT_CONTEXT
+
- KEY_NAME
+
+ 1
- LONG_VAL
-
+ {piece.count=40321}
+
+
+
+
In this case, the Step ran for 30 minutes
+ and processed 40,321 'pieces', which would represent lines in a file in
+ this scenario. This value will be updated just before each commit by the
+ framework, and can contain multiple rows corresponding to entries within
+ the ExecutionContext. Being notified before a
+ commit requires one of the various StepListeners, or an
+ ItemStream, which are discussed in more detail
+ later in this guide. As with the previous example, it is assumed that the
+ Job is restarted the next day. When it is restarted, the values from the
+ ExecutionContext of the last run are reconstituted
+ from the database, and when the ItemReader is
+ opened, it can check to see if it has any stored state in the context, and
+ initialize itself from there:
-
- 1
-
- LONG
-
- piece.count
-
- 40321
-
-
-
-
In this case, the Step ran for 30
- minutes and processed 40,321 'pieces', which would represent lines in a
- file in this scenario. This value will be updated just before each
- commit by the framework, and can contain multiple rows corresponding to
- entries within the ExecutionContext. Being
- notified before a commit requires one of the various StepListeners, or
- an ItemStream, which are discussed in more detail
- later in this guide. As with the previous example, it is assumed that
- the Job is restarted the next day. When it is restarted, the values from
- the ExecutionContext of the last run are
- reconstituted from the database, and when the
- ItemReader is opened, it can check to see if it
- has any stored state in the context, and initialize itself from
- there:
-
- if (executionContext.containsKey(getKey(LINES_READ_COUNT))) {
+ if (executionContext.containsKey(getKey(LINES_READ_COUNT))) {
log.debug("Initializing for restart. Restart data is: " + executionContext);
long lineCount = executionContext.getLong(getKey(LINES_READ_COUNT));
@@ -940,70 +864,75 @@
}
}
- In this case, after the above code is executed, the current line
- will be 40,322, allowing the Step to start again
- from where it left off. The ExecutionContext can
- also be used for statistics that need to be persisted about the run
- itself. For example, if a flat file contains orders for processing that
- exist across multiple lines, it may be necessary to store how many
- orders have been processed (which is much different from than the number
- of lines read) so that an email can be sent at the end of the
- Step with the total orders processed in the body.
- The framework handles storing this for the developer, in order to
- correctly scope it with an individual
- JobInstance. It can be very difficult to know
- whether an existing ExecutionContext should be
- used or not. For example, using the 'EndOfDay' example from above, when
- the 01-01 run starts again for the second time, the framework recognizes
- that it is the same JobInstance and on an
- individual Step basis, pulls the
- ExecutionContext out of the database and hands it
- as part of the StepExecution to the
- Step itself. Conversely, for the 01-02 run the
- framework recognizes that it is a different instance, so an empty
- context must be handed to the Step. There are
- many of these types of determinations that the framework makes for the
- developer to ensure the state is given to them at the correct time. It
- is also important to note that exactly one
- ExecutionContext exists per
- StepExecution at any given time. Clients of the
- ExecutionContext should be careful because this
- creates a shared keyspace, so care should be taken when putting values
- in to ensure no data is overwritten, however, the
- Step stores absolutely no data in the context, so
- there is no way to adversely affect the framework.
-
+ In this case, after the above code is executed, the current line
+ will be 40,322, allowing the Step to start again
+ from where it left off. The ExecutionContext can
+ also be used for statistics that need to be persisted about the run
+ itself. For example, if a flat file contains orders for processing that
+ exist across multiple lines, it may be necessary to store how many orders
+ have been processed (which is much different from than the number of lines
+ read) so that an email can be sent at the end of the
+ Step with the total orders processed in the body.
+ The framework handles storing this for the developer, in order to
+ correctly scope it with an individual JobInstance.
+ It can be very difficult to know whether an existing
+ ExecutionContext should be used or not. For
+ example, using the 'EndOfDay' example from above, when the 01-01 run
+ starts again for the second time, the framework recognizes that it is the
+ same JobInstance and on an individual
+ Step basis, pulls the
+ ExecutionContext out of the database and hands it
+ as part of the StepExecution to the
+ Step itself. Conversely, for the 01-02 run the
+ framework recognizes that it is a different instance, so an empty context
+ must be handed to the Step. There are many of these
+ types of determinations that the framework makes for the developer to
+ ensure the state is given to them at the correct time. It is also
+ important to note that exactly one ExecutionContext
+ exists per StepExecution at any given time. Clients
+ of the ExecutionContext should be careful because
+ this creates a shared keyspace, so care should be taken when putting
+ values in to ensure no data is overwritten, however, the
+ Step stores absolutely no data in the context, so
+ there is no way to adversely affect the framework.
+
+ It is also important to note that there is at least one
+ ExecutionContext per
+ JobExecution, and one for every
+ StepExecution. For example, consider the following
+ code snippet:
+
+
+ ExecutionContext ecStep = stepExecution.getExecutionContext();
+ ExecutionContext ecJob = jobExecution.getExecutionContext();
+ //ecStep does not equal ecJob
+
+
+
+ As noted in the comment, ecStep will not equal ecJob, they are two
+ different ExecutionContexts. The one scoped to the
+ Step will be saved at every commit point in the
+ Step, wheras the one scoped to the
+ Job will be saved in between every
+ Step execution.
JobRepository
JobRepository is the persistence mechanism
- for all of the Stereotypes mentioned above. When a job is first launched,
- a JobExecution is obtained by calling the
- repository's createJobExecution method, and
- during the course of execution, StepExecution and
- JobExecution are persisted by passing them to the
- repository:
+ for all of the Stereotypes mentioned above. It provides CRUD operations
+ for JobLauncher, Job, and
+ Step implementations. When a
+ Job is first launched, a
+ JobExecution is obtained from the repository, and
+ during the course of execution StepExecution and
+ JobExecution implementations are persisted by
+ passing them to the repository:
- public interface JobRepository {
+
+ <job-repository id="jobRepository"/>
- public JobExecution createJobExecution(Job job, JobParameters jobParameters)
- throws JobExecutionAlreadyRunningException, JobRestartException;
-
- void add(StepExecution stepExecution);
-
- void update(JobExecution jobExecution);
-
- void update(StepExecution stepExecution);
-
- void updateExecutionContext(StepExecution stepExecution);
-
- StepExecution getLastStepExecution(JobInstance jobInstance, Step step);
-
- int getStepExecutionCount(JobInstance jobInstance, Step step);
-
-}
@@ -1027,25 +956,6 @@
Job.
-
- JobLocator
-
- JobLocator represents an interface for
- locating a Job:
-
- public interface JobLocator {
-
- Job getJob(String name) throws NoSuchJobException;
- }
-
- This interface is very necessary due to the nature of Spring itself.
- Because it can't be guaranteed that one
- ApplicationContext equals one
- Job, an abstraction is needed to obtain a
- Job for a given name. It becomes especially useful
- when launching jobs from within a Java EE application server.
-
-
Item Reader
@@ -1080,15 +990,4 @@
that it's not valid, returning null indicates that it should not be
written out.
-
-
- Tasklet
-
- A Tasklet represents the execution of a
- logical unit of work, as defined by its implementation of the Spring Batch
- provided Tasklet interface. A
- Tasklet is useful for encapsulating processing
- logic that is not natural to split into read-(transform)-write phases,
- such as invoking a system command or a stored procedure.
-
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/execution.xml b/docs/src/site/docbook/reference/execution.xml
index 24484b117..b0469a99b 100644
--- a/docs/src/site/docbook/reference/execution.xml
+++ b/docs/src/site/docbook/reference/execution.xml
@@ -13,8 +13,7 @@
+ fileref="images/spring-batch-reference-model.png" />
@@ -457,8 +456,8 @@
</bean>
The databaseType property indicates the type of incrementer that
- must be used. Options include: "db2", "db2zos", "derby", "hsql", "mysql",
- "oracle", and "postgres".
+ must be used. Options include: "db2", "db2zos", "derby", "hsql",
+ "mysql", "oracle", and "postgres".
@@ -1152,9 +1151,9 @@
<property name="retryableExceptionClasses" value="org.springframework.dao.DeadlockLoserDataAccessException" />
</bean>
- The SkipLimitStepFactoryBean requires
- a limit for the number of times an individual item can be retried, and
- a list of Exceptions that are 'retryable'.
+ The SkipLimitStepFactoryBean requires a
+ limit for the number of times an individual item can be retried, and a
+ list of Exceptions that are 'retryable'.
@@ -1421,11 +1420,11 @@
<property name="jobRepository" ref="repository" />
</bean>
-
- TaskletStep will automatically register the tasklet as
- StepExecutionListener if it implements
- this interface
-
+
+ TaskletStep will automatically register the tasklet as
+ StepExecutionListener if it implements this
+ interface
+
TaskletAdapter
@@ -1565,16 +1564,14 @@
Logging Item Processing and Failures
- A common use case is the need for special handling of
- errors in a step, item by item, perhaps logging to a special
- channel, or inserting a record into a
- database. The StepHandlerStep (created
- from the step factory beans) allows users to implement this use
- case with a simple
- ItemReadListener, for errors on read, and an
- ItemWriteListener, for errors on write. The below
- code snippets illustrate a listener that logs both read and write
- failures:
+ A common use case is the need for special handling of errors in a
+ step, item by item, perhaps logging to a special channel, or inserting a
+ record into a database. The StepHandlerStep
+ (created from the step factory beans) allows users to implement this use
+ case with a simple ItemReadListener, for errors
+ on read, and an ItemWriteListener, for errors on
+ write. The below code snippets illustrate a listener that logs both read
+ and write failures:
public class ItemFailureLoggerListener extends ItemListenerSupport {
@@ -1771,4 +1768,4 @@
itself.
-
+
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/images/chunk-oriented-processing.png b/docs/src/site/docbook/reference/images/chunk-oriented-processing.png
new file mode 100644
index 000000000..5098aca16
Binary files /dev/null and b/docs/src/site/docbook/reference/images/chunk-oriented-processing.png differ
diff --git a/docs/src/site/docbook/reference/images/conditional-flow.png b/docs/src/site/docbook/reference/images/conditional-flow.png
new file mode 100644
index 000000000..68c7c573d
Binary files /dev/null and b/docs/src/site/docbook/reference/images/conditional-flow.png differ
diff --git a/docs/src/site/docbook/reference/images/job-heirarchy.png b/docs/src/site/docbook/reference/images/job-heirarchy.png
index a6c201c7f..afead4323 100644
Binary files a/docs/src/site/docbook/reference/images/job-heirarchy.png and b/docs/src/site/docbook/reference/images/job-heirarchy.png differ
diff --git a/docs/src/site/docbook/reference/images/job-launcher-sequence-async.png b/docs/src/site/docbook/reference/images/job-launcher-sequence-async.png
index f3886040f..8d7ab5d21 100644
Binary files a/docs/src/site/docbook/reference/images/job-launcher-sequence-async.png and b/docs/src/site/docbook/reference/images/job-launcher-sequence-async.png differ
diff --git a/docs/src/site/docbook/reference/images/job-launcher-sequence-sync.png b/docs/src/site/docbook/reference/images/job-launcher-sequence-sync.png
index 126e56fbd..7e7547d33 100644
Binary files a/docs/src/site/docbook/reference/images/job-launcher-sequence-sync.png and b/docs/src/site/docbook/reference/images/job-launcher-sequence-sync.png differ
diff --git a/docs/src/site/docbook/reference/images/job-stereotypes-parameters.png b/docs/src/site/docbook/reference/images/job-stereotypes-parameters.png
new file mode 100644
index 000000000..d9465b5e1
Binary files /dev/null and b/docs/src/site/docbook/reference/images/job-stereotypes-parameters.png differ
diff --git a/docs/src/site/docbook/reference/images/jobHeirarchyWithSteps.png b/docs/src/site/docbook/reference/images/jobHeirarchyWithSteps.png
index 912e32616..d8c55746e 100644
Binary files a/docs/src/site/docbook/reference/images/jobHeirarchyWithSteps.png and b/docs/src/site/docbook/reference/images/jobHeirarchyWithSteps.png differ
diff --git a/docs/src/site/docbook/reference/images/sequential-flow.png b/docs/src/site/docbook/reference/images/sequential-flow.png
new file mode 100644
index 000000000..0036b04db
Binary files /dev/null and b/docs/src/site/docbook/reference/images/sequential-flow.png differ
diff --git a/docs/src/site/docbook/reference/images/spring-batch-layers.png b/docs/src/site/docbook/reference/images/spring-batch-layers.png
index ff1764074..136617340 100644
Binary files a/docs/src/site/docbook/reference/images/spring-batch-layers.png and b/docs/src/site/docbook/reference/images/spring-batch-layers.png differ
diff --git a/docs/src/site/docbook/reference/images/spring-batch-reference-model.png b/docs/src/site/docbook/reference/images/spring-batch-reference-model.png
index 0da2b3bbc..7062d470b 100644
Binary files a/docs/src/site/docbook/reference/images/spring-batch-reference-model.png and b/docs/src/site/docbook/reference/images/spring-batch-reference-model.png differ
diff --git a/docs/src/site/docbook/reference/images/step.png b/docs/src/site/docbook/reference/images/step.png
new file mode 100644
index 000000000..1a321fafe
Binary files /dev/null and b/docs/src/site/docbook/reference/images/step.png differ
diff --git a/docs/src/site/docbook/reference/index.xml b/docs/src/site/docbook/reference/index.xml
index f16e82c2d..890d1edc0 100644
--- a/docs/src/site/docbook/reference/index.xml
+++ b/docs/src/site/docbook/reference/index.xml
@@ -48,17 +48,21 @@
-
+
+
+
+
+
-
-
+
+
diff --git a/docs/src/site/docbook/reference/job.xml b/docs/src/site/docbook/reference/job.xml
index adbbef628..92efd3f85 100644
--- a/docs/src/site/docbook/reference/job.xml
+++ b/docs/src/site/docbook/reference/job.xml
@@ -1,281 +1,253 @@
-
- Configuring and Executing A Job
+
+ Configuring and Running A Job
+
+ In Chapter 2, the overall architecture design was discussed, using the
+ following diagram as a guide:
+
+
+
+
+
+
+
+
+
+
+
+ While the Job object may seem like a simple container for steps, there
+ are many configuration options that developers should be aware of.
+ Furthermore, there are many considerations for how a
+ Job will be run and how its meta data will be stored
+ during that run. This chapter will explain the various configuration options
+ and runtime concerns of a Job.
- Introduction
+ Configuring a Job
- In Chapter 2, the overall architecture design was discussed, using
- the following diagram as a guide:
+ The only current implementation of the Job
+ interface is SimpleJob. Since a
+ Job is just a simple loop through a list of Steps,
+ this implementation should be sufficient for the majority of needs. It has
+ only three required dependencies: a name,
+ JobRepository, and a list of Steps.
-
-
-
-
+
+ <job id="footballJob">
+ <step name="playerload" next="gameLoad"/>
+ <step name="gameLoad" next="playerSummarization"/>
+ <step name="playerSummarization"/>
+ </job>
-
-
-
-
+
- When viewed from left to right, the diagram describes a basic flow
- for the execution of a batch job:
+ The namespace defaults to referencing a repository with an id of
+ 'jobRepository', which is a sensible default. However, this can be
+ overriden explicitely:
-
-
- A Scheduler kicks off a job script (usually some form of shell
- script)
-
+
+ <job id="footballJob" repository="specialRepository">
+ <step name="playerload" next="gameLoad"/>
+ <step name="gameLoad" next="playerSummarization"/>
+ <step name="playerSummarization"/>
+ </job>
-
- The script sets up the classpath appropriately, and starts the
- Java process. In most cases, using
- CommandLineJobRunner as the entry point
-
-
-
- The JobRunner finds the Job using the
- JobLocator, pulls together the
- JobParameters and launches the
- Job
-
-
-
- The JobLauncher retrieves a
- JobExecution from the
- JobRepository, and executes the
- Job
-
-
-
- The Job executes each
- Step.
-
-
-
- When execution is complete, the Step
- returns control back to the Job, and if no more
- steps exist, control is returned back to the original caller, in this
- case, the scheduler.
-
-
-
- This flow is perhaps a bit overly simplified, but describes the
- complete flow in the most basic terms. From here, each tier will be
- described in detail, using actual implementations and examples.
-
-
-
- JobLauncher
-
- The most basic implementation of the
- JobLauncher interface is the SimpleJobLauncher.
- It's only required dependency is a JobRepository,
- in order to obtain an execution:
-
- <bean id="jobLauncher"
- class="org.springframework.batch.execution.launch.SimpleJobLauncher">
- <property name="jobRepository" ref="jobRepository" />
- </bean>
-
- Once a JobExecution is obtained, it is passed
- to the execute method of Job, ultimately returning
- the JobExecution to the caller:
-
-
-
-
-
-
-
-
-
-
-
- The sequence is straightforward, and works well when launched from a
- scheduler, but causes issues when trying to launch from an HTTP request.
- In this scenario, the launching needs to be done asynchronously, so that
- the SimpleJobLauncher returns immediately to it's
- caller. This is because it is not good practice to keep an HTTP request
- open for the amount of time needed by long running processes such as
- batch. An example sequence is below:
-
-
-
-
-
-
-
-
-
-
-
- The SimpleJobLauncher can easily be
- configured to allow for this scenario by configuring a
- TaskExecutor:
-
- <bean id="jobLauncher"
- class="org.springframework.batch.execution.launch.SimpleJobLauncher">
- <property name="jobRepository" ref="jobRepository" />
- <property name="taskExecutor">
- <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
- </property>
- </bean>
-
- Any implementation of the spring TaskExecutor
- interface can be used to control how jobs are asynchronously
- executed.
-
-
-
- JobRepository
-
- The SimpleJobRepository is the only provided implementation of the
- JobRepository interface. It completely manages the
- various batch domain objects and ensures they are created and persisted
- correctly. The SimpleJobRepository uses three
- different DAO interfaces for the three major domain types it stores:
- JobInstanceDao,
- JobExecutionDao, and
- StepExecutionDao. The repository delegates to these
- DAOs to both persist the various domain objects and query for them during
- initialization. The following configuration shows a SimpleJobRepository
- configured with JDBC DAOs:
-
- <bean id="jobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository">
- <constructor-arg ref="jobInstanceDao" />
- <constructor-arg ref="jobExecutionDao" />
- <constructor-arg ref="stepExecutionDao" />
- </bean>
-
- <bean id="jobInstanceDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobInstanceDao" >
- <property name="jdbcTemplate" ref="jdbcTemplate" />
- <property name="jobIncrementer" ref="jobIncrementer" />
- </bean>
-
- <bean id="jobExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobExecutionDao" >
- <property name="jdbcTemplate" ref="jdbcTemplate" />
- <property name="jobExecutionIncrementer" ref="jobExecutionIncrementer" />
- </bean>
-
- <bean id="stepExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcStepExecutionDao" >
- <property name="jdbcTemplate" ref="jdbcTemplate" />
- <property name="stepExecutionIncrementer" ref="stepExecutionIncrementer" />
- </bean>
-
- <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate" >
- <property name="dataSource" ref="dataSource" />
- </bean>
-
- The configuration above isn't quite complete, each DAO
- implementation makes a reference to a Spring
- DataFieldMaxValueIncrementer.
- JobInstance, JobExecution,
- and StepExecution each have unique IDs, and the
- incrementers are used to create them.
+
- JobRepositoryFactoryBean
+ Restartability
- Including the incrementers, which must be database specific, the
- configuration above is verbose. In order to make this more manageable,
- the framework provides a FactoryBean for
- convenience: JobRepositoryFactoryBean.
+ One key concern when execution a batch job, is what happens when a
+ failed job is restarted? A Job is considered to have been 'restarted' if
+ the same JobInstance has more than one JobExecution. Ideally, all jobs
+ should be able to start up where they left off, but there are scenarios
+ where this is not possible. It is entirely up to
+ the developer to ensure that a new instance is always created in this
+ scenario. However, Spring Batch does provide some help. If a
+ Job should never be restarted, but should always be run as part of a new
+ JobInstance, then the restartable property may be
+ set to 'false':
- <bean id="jobRepository"
- class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean"
- <property name="databaseType" value="hsql" />
- <property name="dataSource" ref="dataSource" />
- <property name="transactionManager" ref="transactionManager" />
- </bean>
+
+ <job id="footballJob" restartable="false">
+ <step name="playerload" next="gameLoad"/>
+ <step name="gameLoad" next="playerSummarization"/>
+ <step name="playerSummarization"/>
+ </job>
- The databaseType property indicates the type of incrementer that
- must be used. Options include: "db2", "db2zos", "derby", "hsql",
- "mysql", "oracle", and "postgres".
+
+
+ To phrase it another way, setting restartable to false means "this
+ Job does not support being started again". Restarting a Job that is not
+ restartable will cause a JobRestartException to
+ be thrown:
+
+
+ Job job = new SimpleJob();
+ job.setRestartable(false);
+
+ JobParameters jobParameters = new JobParameters();
+
+ JobExecution firstExecution = jobRepository.createJobExecution(job, jobParameters);
+ jobRepository.saveOrUpdate(firstExecution);
+
+ try {
+ jobRepository.createJobExecution(job, jobParameters);
+ fail();
+ }
+ catch (JobRestartException e) {
+ // expected
+ }
+
+
+
+ This snippet of JUnit code shows how attempting to create a
+ JobExecution the first time for a non restartable
+ job will cause no issues. However, the second
+ attempt will throw a JobRestartException.
- In-Memory Repository
+ Intercepting Job execution
- There are scenarios in which you may not want to persist your
- domain objects to the database. One reason may be speed, storing domain
- objects at each commit point takes extra time. Another reason may be
- that you just don't need to persist status for a particular job. Spring
- batch provides a solution:
+ During the course of the execution of a
+ Job, it may be useful to be notified of various
+ events in its lifecycle so that custom code may be executed. The
+ SimpleJob allows for this by calling a
+ JobListener at the appropriate time:
- <bean id="simpleJobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository">
- <constructor-arg ref="mapJobInstanceDao" />
- <constructor-arg ref="mapJobExecutionDao" />
- <constructor-arg ref="mapStepExecutionDao" />
- </bean>
+
+ public interface JobExecutionListener {
- <bean id="mapJobInstanceDao"
- class="org.springframework.batch.core.repository.dao.MapJobInstanceDao" />
+ void beforeJob(JobExecution jobExecution);
- <bean id="mapJobExecutionDao"
- class="org.springframework.batch.core.repository.dao.MapJobExecutionDao" />
+ void afterJob(JobExecution jobExecution);
- <bean id="mapStepExecutionDao"
- class="org.springframework.batch.core.repository.dao.MapStepExecutionDao" />
+ }
- The Map* DAO implementations store the batch artifacts in a
- transactional map. So, the repository and DAOs may still be used
- normally, and are transactionally sound, but their contents will be lost
- when the class is destroyed.
+
- There is also a separate FactoryBean for the in-memory
- JobRepository, which reduces the amount of
- configuration required:
+ Listeners can be added to a SimpleJob via
+ the setJobListeners property:
- <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
+
+ <job id="footballJob">
+ <step name="playerload" next="gameLoad"/>
+ <step name="gameLoad" next="playerSummarization"/>
+ <step name="playerSummarization"/>
+ <listeners>
+ <listener class="org.springframework.batch.sample.SampleListener"/>
+ </listeners>
+ </job>
-
- Transaction Configuration For the JobRepository
+
- If either of the JobRepository factory beans are used,
- transactional advice will be automatically created around the
- repository. This is to ensure that the batch meta data, including
- state that is necessary for restarts after a failure, is persisted
- correctly. The behaviour of the framework is not well defined if the
- repository methods are not transactional. The isolation level in the
- create* method attributes is specified separately to
- ensure that when jobs are launched there if two processes are trying
- to launch the same job at the same time, only one will succeed. The
- default isolation level for that method is SERIALIZABLE, which is
- quite aggressive: READ_COMMITTED would work just as well;
- READ_UNCOMMITTED would be fine if two processes are not likely to
- collide in this way. However, since a call to the
- create* method is quite short, it is unlikely
- that the SERIALIZED will cause problems, as long as the database
- platform supports it. However, this can be overriden in the factory
- beans:
+ It should be noted that afterJob will be called regardless of the
+ success or failure of the Job. If success or
+ failure needs to be determined it can be obtained from the
+ JobExecution:
- <bean id="jobRepository"
- class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean"
- <property name="databaseType" value="hsql" />
- <property name="dataSource" ref="dataSource" />
- <property name="transactionManager" ref="transactionManager" />
- <property name="IsolationLevelForCreate" value="ISOLATION_REPEATABLE_READ" />
- </bean>
+
+ void afterJob(JobExecution jobExecution){
+ if( jobExecution.getStatus = BatchStatus.COMPLETED ){
+ //job success
+ }
+ else if(jobExecution.getStatus = BatchStatus.FAILED){
+ //job failure
+ }
- If the factory beans aren't used then it is also essential to
- configure the transactional behaviour of the repository using
- AOP:
+
+
-
+
+ JobFactory and Stateful Components in Steps
+
+ Unlike many traditional Spring applications, many of the
+ components of a batch application are stateful, the file readers and
+ writers are obvious examples. The recommended way to deal with this is
+ to create a fresh ApplicationContext for each job
+ execution. If the Job is launched from the
+ command line with CommandLineJobRunner this is
+ trivial. For more complex launching scenarios, where jobs are executed
+ in parallel or serially from the same process, some extra steps have to
+ be taken to ensure that the ApplicationContext is
+ refreshed. This is preferable to using prototype scope for the stateful
+ beans because then they would not receive lifecycle callbacks from the
+ container at the end of use. (e.g. through destroy-method in XML)
+
+ The strategy provided by Spring Batch to deal with this scenario
+ is the JobFactory, and the samples provide an
+ example of a specialized implementation that can load an
+ ApplicationContext and close it properly when the
+ job is finished. A relevant examples is
+ ClassPathXmlApplicationContextJobFactory and its
+ use in the adhoc-job-launcher-context.xml and the
+ quartz-job-launcher-context.xml, which can be found in the
+ Samples project.
+
+
+
+
+ Configuring a JobRepository
+
+ As described in Chatper 2, the JobRepository is used for basic CRUD
+ operations of the various persisted domain objects within Spring Batch,
+ such as JobExecution and StepExecution. It is required by many of the
+ major framework features, such as the JobLauncher,
+ Job, and Step. The batch
+ namespace abstract much of the implementation details of the JobRepository
+ implementations and their collaborators. However, there are still a few
+ configuration options available:
+
+
+ <job-repository id="jobRepository"
+ dataSource="dataSource"
+ transactionManager="transactionManager"
+ isolation-level-for-create="serializable"
+ table-prefix="BATCH_"
+ />
+
+
+
+ None of the configuration options listed above are required except
+ the id. If they are not set, the defaults shown above will be used. They
+ are shown above for awareness purposes.
+
+
+ Transaction Configuration For the JobRepository
+
+ If the namespace is used, transactional advice will be
+ automatically created around the repository. This is to ensure that the
+ batch meta data, including state that is necessary for restarts after a
+ failure, is persisted correctly. The behaviour of the framework is not
+ well defined if the repository methods are not transactional. The
+ isolation level in the create* method attributes is
+ specified separately to ensure that when jobs are launched, if two
+ processes are trying to launch the same job at the same time, only one
+ will succeed. The default isolation level for that method is
+ SERIALIZABLE, which is quite aggressive: READ_COMMITTED would work just
+ as well; READ_UNCOMMITTED would be fine if two processes are not likely
+ to collide in this way. However, since a call to the
+ create* method is quite short, it is unlikely
+ that the SERIALIZED will cause problems, as long as the database
+ platform supports it. However, this can be overriden:
+
+
+ <job-repository id="jobRepository"
+ isolation-level-for-create="ISOLATION_REPEATABLE_READ" />
+
+
+
+ If the namespace or factory beans aren't used then it is also
+ essential to configure the transactional behaviour of the repository
+ using AOP:
+
+
<aop:config>
<aop:advisor
pointcut="execution(* org.springframework.batch.core..*Repository+.*(..))"
@@ -290,11 +262,10 @@
- This fragment can be used as is, with almost no changes.
- Remember also to include the appropiate namespace declarations and to
- make sure spring-tx and spring-aop (or the whole of spring) is on the
- classpath.
-
+ This fragment can be used as is, with almost no changes. Remember
+ also to include the appropiate namespace declarations and to make sure
+ spring-tx and spring-aop (or the whole of spring) is on the
+ classpath.
Recommendations for Indexing Meta Data Tables
@@ -369,175 +340,131 @@
+
+
+ Changing the table prefix
+
+ Another modifiable property of the
+ JobRepository is the table prefix of the
+ meta-data tables. By default they are all prefaced with BATCH_.
+ BATCH_JOB_EXECUTION and BATCH_STEP_EXECUTION are two examples. However,
+ there are potential reasons to modify this prefix. If the schema names
+ needs to be prepended to the table names, or if more than one set of
+ meta data tables is needed within the same schema, then the table prefix
+ will need to be changed:
+
+
+ <job-repository id="jobRepository"
+ table-prefix="SYSTEM.TEST_"
+ />
+
+
+
+ Given the above changes, every query to the meta data tables will
+ be prefixed with "SYSTEM.TEST_". BATCH_JOB_EXECUTION will be referred to
+ as SYSTEM.TEST_JOB_EXECUTION.
+
+
+ Only the table prefix is configurable, the table and column
+ names are not.
+
+
+
+
+ In-Memory Repository
+
+ There are scenarios in which you may not want to persist your
+ domain objects to the database. One reason may be speed, storing domain
+ objects at each commit point takes extra time. Another reason may be
+ that you just don't need to persist status for a particular job. Spring
+ batch provides a solution:
+
+ <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
+
- Job
+ Configuring a JobLauncher
- The only current implementation of the Job
- interface is SimpleJob. Since a
- Job is just a simple loop through a list of Steps,
- this implementation should be sufficient for the majority of needs. It has
- only three required dependencies: a name,
- JobRepository, and a list of Steps.
+ The most basic implementation of the
+ JobLauncher interface is the SimpleJobLauncher.
+ It's only required dependency is a JobRepository,
+ in order to obtain an execution:
- <bean id="footballJob"
- class="org.springframework.batch.core.job.SimpleJob">
- <property name="steps">
- <list>
- <!-- Step Bean details ommitted for clarity -->
- <bean id="playerload" parent="simpleStep" />
- <bean id="gameLoad" parent="simpleStep" />
- <bean id="playerSummarization" parent="simpleStep" />
- </list>
- </property>
+ <bean id="jobLauncher"
+ class="org.springframework.batch.execution.launch.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
- Each Step will be executed in sequence until
- all have completed successfully. Any Step that fails will cause the entire
- job to fail.
+ Once a JobExecution is obtained, it is passed
+ to the execute method of Job, ultimately returning
+ the JobExecution to the caller:
-
- Restartability
+
+
+
+
- One key concern when execution a batch job, is what happens when a
- failed job is restarted? A Job is considered to have been 'restarted' if
- the same JobInstance has more than one JobExecution. Ideally, all jobs
- should be able to start up where they left off, but there are scenarios
- where this is not possible. It is entirely up to
- the developer to ensure that a new instance is always created in this
- scenario. However, Spring Batch does provide some help. If a
- Job should never be restarted, but should always be run as part of a new
- JobInstance, then the restartable property may be
- set to 'false':
+
+
+
+
- <bean id="footballJob"
- class="org.springframework.batch.core.job.SimpleJob">
- <property name="steps">
- <list>
- <!-- Step Bean details ommitted for clarity -->
- <bean id="playerload" parent="simpleStep" />
- <bean id="gameLoad" parent="simpleStep" />
- <bean id="playerSummarization" parent="simpleStep" />
- </list>
- </property>
+ The sequence is straightforward, and works well when launched from a
+ scheduler, but causes issues when trying to launch from an HTTP request.
+ In this scenario, the launching needs to be done asynchronously, so that
+ the SimpleJobLauncher returns immediately to it's
+ caller. This is because it is not good practice to keep an HTTP request
+ open for the amount of time needed by long running processes such as
+ batch. An example sequence is below:
+
+
+
+
+
+
+
+
+
+
+
+ The SimpleJobLauncher can easily be
+ configured to allow for this scenario by configuring a
+ TaskExecutor:
+
+ <bean id="jobLauncher"
+ class="org.springframework.batch.execution.launch.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
- <property name="restartable" value="false" />
+ <property name="taskExecutor">
+ <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
+ </property>
</bean>
- To phrase it another way, setting restartable to false means "this
- Job does not support being started again". Restarting a Job that is not
- restartable will cause a JobRestartException to
- be thrown:
-
- Job job = new SimpleJob();
- job.setRestartable(false);
-
- JobParameters jobParameters = new JobParameters();
-
- JobExecution firstExecution = jobRepository.createJobExecution(job, jobParameters);
- jobRepository.saveOrUpdate(firstExecution);
-
- try {
- jobRepository.createJobExecution(job, jobParameters);
- fail();
- }
- catch (JobRestartException e) {
- // expected
- }
-
- This snippet of JUnit code shows how attempting to create a
- JobExecution the first time for a non restartable
- job will cause no issues. However, the second
- attempt will throw a JobRestartException.
-
-
-
- Intercepting Job execution
-
- During the course of the execution of a
- Job, it may be useful to be notified of various
- events in its lifecycle so that custom code may be executed. The
- SimpleJob allows for this by calling a
- JobListener at the appropriate time:
-
- public interface JobListener {
-
- void beforeJob(JobExecution jobExecution);
-
- void afterJob(JobExecution jobExecution);
-
- void onError(JobExecution jobExecution, Throwable e);
-
- void onInterrupt(JobExecution jobExecution);
- }
-
- Listeners can be added to a SimpleJob via
- the setJobListeners property:
-
- <bean id="footballJob"
- class="org.springframework.batch.core.job.SimpleJob">
- <property name="steps">
- <list>
- <!-- Step Bean details ommitted for clarity -->
- <bean id="playerload" parent="simpleStep" />
- <bean id="gameLoad" parent="simpleStep" />
- <bean id="playerSummarization" parent="simpleStep" />
- </list>
- </property>
- <property name="jobRepository" ref="jobRepository" />
- <property name="jobListeners">
- <bean class="org.springframework.batch.core.listener.JobListenerSupport" />
- </property>
- </bean>
-
-
-
- JobFactory and Stateful Components in Steps
-
- Unlike many traditional Spring applications, many of the
- components of a batch application are stateful, the file readers and
- writers are obvious examples. The recommended way to deal with this is
- to create a fresh ApplicationContext for each job
- execution. If the Job is launched from the
- command line with CommandLineJobRunner this is
- trivial. For more complex launching scenarios, where jobs are executed
- in parallel or serially from the same process, some extra steps have to
- be taken to ensure that the ApplicationContext is
- refreshed. This is preferable to using prototype scope for the stateful
- beans because then they would not receive lifecycle callbacks from the
- container at the end of use. (e.g. through destroy-method in XML)
-
- The strategy provided by Spring Batch to deal with this scenario
- is the JobFactory, and the samples provide an
- example of a specialized implementation that can load an
- ApplicationContext and close it properly when the
- job is finished. A relevant examples is
- ClassPathXmlApplicationContextJobFactory and its
- use in the adhoc-job-launcher-context.xml and the
- quartz-job-launcher-context.xml, which can be found in the
- Samples project.
-
+ Any implementation of the spring TaskExecutor
+ interface can be used to control how jobs are asynchronously
+ executed.
Running a Job
- Regardless of whether the originator is a Scheduler or an HTTP
- request, a Job must be obtained, parameters must be parsed, and eventually
- a JobLauncher called:
-
-
-
-
-
-
-
-
-
-
+ At a minimum, launching a batch job requires two things: the Job to
+ be launched and a JobLauncher. Both can be contained within the same
+ context or different contexts. For example, if launching a job from the
+ command line, a new JVM will be instantiated for each Job, and thus every
+ job will have it's own JobLauncher. However, if
+ running from within a web container within the scope of an
+ HttpRequest, there will usually be one
+ JobLauncher, configured for asynchronous job
+ launching, that multiple requests will invoke to launch their jobs.
Running Jobs from the Command Line
@@ -765,265 +692,4 @@
-
-
- Job Tier
-
- The Job Tier is responsible for the overall execution of a batch
- job. It sequentially executes batch steps, ensuring that all steps are in
- the correct state and all appropriate policies are enforced:
-
-
-
-
-
-
-
-
-
-
-
- The job tier is entirely concerned with maintaining the three job
- stereotypes: Job,
- JobInstance, and
- JobExecution. The
- JobLauncher interacts with the
- JobRepository in order to create a
- JobExecution, and the Job
- stores the JobExecution using the
- repository.
-
-
-
- Examples of Customized Business Logic
-
-
- Some batch jobs can be assembled purely from off-the-shelf
- components in Spring Batch, mostly the ItemReader
- and ItemWriter implementations. Where this is not
- possible (the majority of cases) the main API entry points for
- application developers are the Tasklet,
- ItemReader, ItemWriter and
- the various listener interfaces. Most simple batch jobs will be able to
- use off-the-shelf input from a Spring Batch
- ItemReader, but it is very often the case that
- there are custom concerns in the processing and writing, which normally
- leads developers to implement an ItemWriter, or
- ItemTransformer.
-
- Here we provide a few examples of common patterns in custom
- business logic, mainly using the listener interfaces . It should be
- noted that an ItemReader or
- ItemWriter can implement the listener interfaces
- as well if appropriate.
-
-
-
- Logging Item Processing and Failures
-
- A common use case is the need for special handling of errors in a
- step, item by item, perhaps logging to a special channel, or inserting a
- record into a database. The StepHandlerStep
- (created from the step factory beans) allows users to implement this use
- case with a simple ItemReadListener, for errors
- on read, and an ItemWriteListener, for errors on
- write. The below code snippets illustrate a listener that logs both read
- and write failures:
-
- public class ItemFailureLoggerListener extends ItemListenerSupport {
-
- private static Log logger = LogFactory.getLog("item.error");
-
- public void onReadError(Exception ex) {
- logger.error("Encountered error on read", e);
- }
-
- public void onWriteError(Exception ex, Object item) {
- logger.error("Encountered error on write", e);
- }
-
-}
-
- Having implemented this listener it must be registered with the
- step:
-
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
- ...
- <property name="listeners">
- <bean class="org.example...ItemFailureLoggerListener"/>
- </property>
- </bean>
-
- Remember that if your listener does anything in an
- onError() method, it will be inside a transaction that is
- going to be rolled back. If you need to use a transactional resource
- such as a database inside an onError() method, consider
- adding a declarative transaction to that method (see Spring Core
- Reference Guide for details), and giving its propagation attribute the
- value REQUIRES_NEW.
-
-
-
- Stopping a Job Manually for Business Reasons
-
- Spring Batch provides a stop() method through the JobLauncher
- interface, but this is really aimed at the operator, rather than the
- application programmer. Sometimes it is more convenient or makes more
- sense to stop a job execution from within the business logic.
-
- The simplest thing to do is to throw a RuntimeException (one that
- isn't retried indefinitely or skipped), For example, a custom exception
- type could be used, as in the example below:
-
- public class PoisonPillItemWriter extends AbstractItemWriter {
-
- public void write(Object item) throws Exception {
-
- if (isPoisonPill(item)) {
- throw new PoisonPillException("Posion pill detected: "+item);
- }
-
- }
-
-}
-
- Another simple way to stop a step from executing is to simply
- return null from the
- ItemReader:
-
- public class EarlyCompletionItemReader extends AbstractItemReader {
-
- private ItemReader delegate;
-
- public void setDelegate(ItemReader delegate) { ... }
-
- public Object read() throws Exception {
-
- Object item = delegate.read();
-
- if (isEndItem(item)) {
- return null; // end the step here
- }
-
- return item;
-
- }
-
-}
-
- The previous example actually relies on the fact that there is a
- default implementation of the CompletionPolicy
- strategy which signals a complete batch when the item to be processed is
- null. A more sophisticated completion policy could be implemented and
- injected into the Step through the
- RepeatOperationsStepFactoryBean:
-
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" >
- ...
- <property name="chunkOperations">
- <bean class="org.springframework.batch.repeat.support.RepeatTemplate">
- <property name="completionPolicy">
- <bean class="org.example...SpecialCompletionPolicy"/>
- </property>
- </bean>
- </property>
- </bean>
-
- An alternative is to set a flag in the
- StepExecution, which is checked by the
- Step implementations in the framework in between
- item processing. To implement this alternative, we need access to the
- current StepExecution, and this can be achieved by implementing a
- StepListener and registering it with the Step. Here is an example of a
- listener that sets the flag:
-
- public class CustomItemWriter extends ItemListenerSupport implements StepListener {
-
- private StepExecution stepExecution;
-
- public void beforeStep(StepExecution stepExecution) {
- this.stepExecution = stepExecution;
- }
-
- public void afterRead(Object item) {
-
- if (isPoisonPill(item)) {
- stepExecution.setTerminateOnly(true);
- }
-
- }
-
-}
-
- The default behaviour here when the flag is set is for the step to
- throw a JobInterruptedException. This can be
- controlled through the StepInterruptionPolicy,
- but the only choice is to throw or not throw an exception, so this is
- always an abnormal ending to a job.
-
-
-
- Adding a Footer Record
-
- A very common requirement is to aggregate information during the
- output process and to append a record at the end of a file summarizing
- the data, or providing a checksum. This can also be achieved with a
- callbacks in the step, normally as part of a custom
- ItemWriter. In this case, since a job is
- accumulating state that should not be lost if the job aborts, the
- ItemStream interface should be
- implemented:
-
- public class CustomItemWriter extends AbstractItemWriter implements
- ItemStream, StepListener
-{
-
- private static final String TOTAL_AMOUNT_KEY = "total.amount";
-
- private ItemWriter delegate;
-
- private double totalAmount = 0.0;
-
- public void setDelegate(ItemWriter delegate) { ... }
-
- public ExitStatus afterStep(StepExecution stepExecution) {
- // Add the footer record here...
- delegate.write("Total Amount Processed: " + totalAmount);
- }
-
- public void open(ExecutionContext executionContext) {
- if (executionContext.containsKey(TOTAL_AMOUNT_KEY) {
- totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY);
- }
- }
-
- public void update(ExecutionContext executionContext) {
- executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount);
- }
-
- public void write(Object item) {
-
- delegate.write(item);
- totalAmount += ((Trade) item).getAmount();
-
- }
-
-}
-
- The custom writer in the example is stateful (it maintains its
- total in an instance variable totalAmount), but the
- state is stored through the ItemStream interface
- in the ExecutionContext. In this way we can be
- sure that when the open() callback is received on a
- restart. The framework garuntees we always get the last value that was
- committed. It should be noted that it is not always necessary to
- implement ItemStream. For example, if the ItemWriter is re-runnable, in
- the sense that it maintains its own state in a transactional resource
- like a database, there is no need to maintain state within the writer
- itself.
-
-
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/namespace.xml b/docs/src/site/docbook/reference/namespace.xml
deleted file mode 100644
index 02b35d94b..000000000
--- a/docs/src/site/docbook/reference/namespace.xml
+++ /dev/null
@@ -1,11 +0,0 @@
-
-
- Namespace Support
-
-
- Namespace Support
- Document Namespace support here.
-
-
-
diff --git a/docs/src/site/docbook/reference/outline.xml b/docs/src/site/docbook/reference/outline.xml
deleted file mode 100644
index 0a319f87b..000000000
--- a/docs/src/site/docbook/reference/outline.xml
+++ /dev/null
@@ -1,62 +0,0 @@
-
-
-
-
- The Spring Batch - Reference Documentation
- Wayne Lund, Waseem Malik, Lucas Ward, Scott Wintermute,
- Kerry O'Brien, Tomi Vanek
- May 2007
-
-
-
-
- Chapter 1: Spring Batch Introduction
- Overview of the Spring Batch Architecture - the Spring Batch Reference Model
-
-
-
- <ulink url="infrastructure.html">Chapter 2: The Spring Batch Infrastructure</ulink>
- Infrastructure covers Repeat Template, I/O facilities and the RetryTemplate
-
-
-
- <ulink url="core.html">Chapter 3: Spring Batch Core</ulink>
- Describe the domain language of batch and how the pieces fit together.
-
-
-
- <ulink url="execution.html">Chapter 4: Spring Batch Execution</ulink>
- Describe the simple batch execution environment.
-
-
-
- <ulink url="application.html">Chapter 5: Spring Batch Applications</ulink>
- Describe the solution space for spring batch.
-
-
-
- <ulink url="samples.html">Chapter 6: Spring Batch Samples</ulink>
- The documentation for samples goes here
-
-
-
- <ulink url="batch-job-testing.html">Chapter 7: Unit and Integration Testing Batch Jobs</ulink>
- The documentation for unit and integration testing of batch jobs goes here.
-
-
-
- <ulink url="batch-performance-testing.html">Chapter 8: Performance Testing Batch Jobs</ulink>
- How to performance test batch jobs.
-
-
-
- <ulink url="glossary.html">Chapter 9: Glossary</ulink>
- (Should this be Appendix A?) The Batch Glossary documents common terms used in the batch processing domain.
-
-
-
- More sections that may come later?
- More advanced info about building Spring Batch, how to contribute, JMS integration, management with JMX, the Spring Batch data model, integration with schedulers (eg quartz)
-
-
-
diff --git a/docs/src/site/docbook/reference/partitioned-containers.xml b/docs/src/site/docbook/reference/partitioned-containers.xml
deleted file mode 100644
index 2f245b529..000000000
--- a/docs/src/site/docbook/reference/partitioned-containers.xml
+++ /dev/null
@@ -1,10 +0,0 @@
-
-
- Batch IO Support
-
- Partitioned Containers
- Much stuff goes here.
-
-
-
diff --git a/docs/src/site/docbook/reference/spring-batch-intro.xml b/docs/src/site/docbook/reference/spring-batch-intro.xml
index 98152cf19..3628a8d6c 100644
--- a/docs/src/site/docbook/reference/spring-batch-intro.xml
+++ b/docs/src/site/docbook/reference/spring-batch-intro.xml
@@ -4,209 +4,201 @@
Spring Batch Introduction
+ Many applications within the enterprise domain require bulk processing
+ to perform business operations in mission critical environments. These
+ business operations include automated, complex processing of large volumes
+ of information that is most efficiently processed without user interaction.
+ These operations typically include time based events (e.g. month-end
+ calculations, notices or correspondence), periodic application of complex
+ business rules processed repetitively across very large data sets (e.g.
+ Insurance benefit determination or rate adjustments), or the integration of
+ information that is received from internal and external systems that
+ typically requires formatting, validation and processing in a transactional
+ manner into the system of record. Batch processing is used to process
+ billions of transactions every day for enterprises.
+
+ Spring Batch is a lightweight, comprehensive batch framework designed
+ to enable the development of robust batch applications vital for the daily
+ operations of enterprise systems. Spring Batch builds upon the productivity,
+ POJO-based development approach, and general ease of use capabilities people
+ have come to know from the Spring Framework, while making it easy for
+ developers to access and leverage more advance enterprise services when
+ necessary. Spring Batch is not a scheduling framework. There are many good
+ enterprise schedulers available in both the commercial and open source
+ spaces such as Quartz, Tivoli, Control-M, etc. It is intended to work in
+ conjunction with a scheduler, not replace a scheduler.
+
+ Spring Batch provides reusable functions that are essential in
+ processing large volumes of records, including logging/tracing, transaction
+ management, job processing statistics, job restart, skip, and resource
+ management. It also provides more advance technical services and features
+ that will enable extremely high-volume and high performance batch jobs
+ though optimization and partitioning techniques. Simple as well as complex,
+ high-volume batch jobs can leverage the framework in a highly scalable
+ manner to process significant volumes of information.
+
- Introduction
+ Background
- Many applications within the enterprise domain require bulk
- processing to perform business operations in mission critical
- environments. These business operations include automated, complex
- processing of large volumes of information that is most efficiently
- processed without user interaction. These operations typically include
- time based events (e.g. month-end calculations, notices or
- correspondence), periodic application of complex business rules processed
- repetitively across very large data sets (e.g. Insurance benefit
- determination or rate adjustments), or the integration of information that
- is received from internal and external systems that typically requires
- formatting, validation and processing in a transactional manner into the
- system of record. Batch processing is used to process billions of
- transactions every day for enterprises.
+ While open source software projects and associated communities have
+ focused greater attention on web-based and SOA messaging-based
+ architecture frameworks, there has been a notable lack of focus on
+ reusable architecture frameworks to accommodate Java-based batch
+ processing needs, despite continued needs to handle such processing within
+ enterprise IT environments. The lack of a standard, reusable batch
+ architecture has resulted in the proliferation of many one-off, in-house
+ solutions developed within client enterprise IT functions.
- Spring Batch is a lightweight, comprehensive batch framework
- designed to enable the development of robust batch applications vital for
- the daily operations of enterprise systems. Spring Batch builds upon the
- productivity, POJO-based development approach, and general ease of use
- capabilities people have come to know from the Spring Framework, while
- making it easy for developers to access and leverage more advance
- enterprise services when necessary. Spring Batch is not a scheduling
- framework. There are many good enterprise schedulers available in both the
- commercial and open source spaces such as Quartz, Tivoli, Control-M, etc.
- It is intended to work in conjunction with a scheduler, not replace a
- scheduler.
+ SpringSource and Accenture have collaborated to change this.
+ Accenture's hands-on industry and technical experience in implementing
+ batch architectures, SpringSource's depth of technical experience, and
+ Spring's proven programming model together mark a natural and powerful
+ partnership to create high-quality, market relevant software aimed at
+ filling an important gap in enterprise Java. Both companies are also
+ currently working with a number of clients solving similar problems
+ developing Spring-based batch architecture solutions. This has provided
+ some useful additional detail and real-life constraints helping to ensure
+ the solution can be applied to the real-world problems posed by clients.
+ For these reasons and many more, SpringSource and Accenture have teamed to
+ collaborate on the development of Spring Batch.
- Spring Batch provides reusable functions that are essential in
- processing large volumes of records, including logging/tracing,
- transaction management, job processing statistics, job restart, skip, and
- resource management. It also provides more advance technical services and
- features that will enable extremely high-volume and high performance batch
- jobs though optimization and partitioning techniques. Simple as well as
- complex, high-volume batch jobs can leverage the framework in a highly
- scalable manner to process significant volumes of information.
+ Accenture has contributed previously proprietary batch processing
+ architecture frameworks, based upon decades worth of experience in
+ building batch architectures with the last several generations of
+ platforms, (i.e., COBOL/Mainframe, C++/Unix, and now Java/anywhere) to the
+ Spring Batch project along with committer resources to drive support,
+ enhancements, and the future roadmap.
-
- Background
-
- While open source software projects and associated communities
- have focused greater attention on web-based and SOA messaging-based
- architecture frameworks, there has been a notable lack of focus on
- reusable architecture frameworks to accommodate Java-based batch
- processing needs, despite continued needs to handle such processing
- within enterprise IT environments. The lack of a standard, reusable
- batch architecture has resulted in the proliferation of many one-off,
- in-house solutions developed within client enterprise IT
- functions.
-
- SpringSource and Accenture have collaborated to change this.
- Accenture's hands-on industry and technical experience in implementing
- batch architectures, SpringSource's depth of technical experience, and
- Spring's proven programming model together mark a natural and powerful
- partnership to create high-quality, market relevant software aimed at
- filling an important gap in enterprise Java. Both companies are also
- currently working with a number of clients solving similar problems
- developing Spring-based batch architecture solutions. This has provided
- some useful additional detail and real-life constraints helping to
- ensure the solution can be applied to the real-world problems posed by
- clients. For these reasons and many more, SpringSource and Accenture
- have teamed to collaborate on the development of Spring Batch.
-
- Accenture has contributed previously proprietary batch processing
- architecture frameworks, based upon decades worth of experience in
- building batch architectures with the last several generations of
- platforms, (i.e., COBOL/Mainframe, C++/Unix, and now Java/anywhere) to
- the Spring Batch project along with committer resources to drive
- support, enhancements, and the future roadmap.
-
- The collaborative effort between Accenture and SpringSource aims
- to promote the standardization of software processing approaches,
- frameworks, and tools that can be consistently leveraged by enterprise
- users when creating batch applications. Companies and government
- agencies desiring to deliver standard, proven solutions to their
- enterprise IT environments will benefit from Spring Batch.
-
-
-
- Usage Scenarios
-
- A typical batch program generally reads a large number of records
- from a database, file, or queue, processes the data in some fashion, and
- then writes back data in a modified form. Spring Batch automates this
- basic batch iteration, providing the capability to process similar
- transactions as a set, typically in an offline environment without any
- user interaction. Batch jobs are part of most IT projects and Spring
- Batch is the only open source framework that provides a robust,
- enterprise-scale solution.
-
- Business Scenarios
-
- Commit batch process periodically
-
-
-
- Concurrent batch processing: parallel processing of a
- job
-
-
-
- Staged, enterprise message-driven processing
-
-
-
- Massively parallel batch processing
-
-
-
- Manual or scheduled restart after failure
-
-
-
- Sequential processing of dependent steps (with extensions to
- workflow-driven batches)
-
-
-
- Partial processing: skip records (e.g. on rollback)
-
-
-
- Whole-batch transaction: for cases with a small batch size
- or existing stored procedures/scripts
-
-
-
- Technical Objectives
-
- Batch developers use the Spring programming model:
- concentrate on business logic; let the framework take care of
- infrastructure.
-
-
-
- Clear separation of concerns between the infrastructure, the
- batch execution environment, and the batch application.
-
-
-
- Provide common, core execution services as interfaces that
- all projects can implement.
-
-
-
- Provide simple and default implementations of the core
- execution interfaces that can be used ‘out of the box’.
-
-
-
- Easy to configure, customize, and extend services, by
- leveraging the spring framework in all layers.
-
-
-
- All existing core services should be easy to replace or
- extend, without any impact to the infrastructure layer.
-
-
-
- Provide a simple deployment model, with the architecture
- JARs completely separate from the application, built using
- Maven.
-
-
-
-
-
- Spring Batch Architecture
-
- Spring Batch is designed with extensibility and a diverse group of
- end users in mind. The figure below shows a sketch of the layered
- architecture that supports the extensibility and ease of use for
- end-user developers.
-
-
-
-
-
-
-
-
- Figure 1.1: Spring Batch Layered
- Architecture
-
-
- This layered architecture highlights three major high level
- components: Application, Core, and Infrastructure. The application
- contains all batch jobs and custom code written by developers using
- Spring Batch. The Batch Core contains the core runtime classes necessary
- to launch and control a batch job. It includes things such as a
- JobLauncher, Job, and
- Step implementations. Both Application and Core
- are built on top of a common infrastructure. This infrastructure
- contains common readers and writers, and services such as the
- RetryTemplate, which are used both by application
- developers(ItemReader and
- ItemWriter) and the core framework itself.
- (retry)
-
+ The collaborative effort between Accenture and SpringSource aims to
+ promote the standardization of software processing approaches, frameworks,
+ and tools that can be consistently leveraged by enterprise users when
+ creating batch applications. Companies and government agencies desiring to
+ deliver standard, proven solutions to their enterprise IT environments
+ will benefit from Spring Batch.
-
+
+
+ Usage Scenarios
+
+ A typical batch program generally reads a large number of records
+ from a database, file, or queue, processes the data in some fashion, and
+ then writes back data in a modified form. Spring Batch automates this
+ basic batch iteration, providing the capability to process similar
+ transactions as a set, typically in an offline environment without any
+ user interaction. Batch jobs are part of most IT projects and Spring Batch
+ is the only open source framework that provides a robust, enterprise-scale
+ solution.
+
+ Business Scenarios
+
+ Commit batch process periodically
+
+
+
+ Concurrent batch processing: parallel processing of a
+ job
+
+
+
+ Staged, enterprise message-driven processing
+
+
+
+ Massively parallel batch processing
+
+
+
+ Manual or scheduled restart after failure
+
+
+
+ Sequential processing of dependent steps (with extensions to
+ workflow-driven batches)
+
+
+
+ Partial processing: skip records (e.g. on rollback)
+
+
+
+ Whole-batch transaction: for cases with a small batch size or
+ existing stored procedures/scripts
+
+
+
+ Technical Objectives
+
+ Batch developers use the Spring programming model: concentrate
+ on business logic; let the framework take care of
+ infrastructure.
+
+
+
+ Clear separation of concerns between the infrastructure, the
+ batch execution environment, and the batch application.
+
+
+
+ Provide common, core execution services as interfaces that all
+ projects can implement.
+
+
+
+ Provide simple and default implementations of the core
+ execution interfaces that can be used ‘out of the box’.
+
+
+
+ Easy to configure, customize, and extend services, by
+ leveraging the spring framework in all layers.
+
+
+
+ All existing core services should be easy to replace or
+ extend, without any impact to the infrastructure layer.
+
+
+
+ Provide a simple deployment model, with the architecture JARs
+ completely separate from the application, built using Maven.
+
+
+
+
+
+ Spring Batch Architecture
+
+ Spring Batch is designed with extensibility and a diverse group of
+ end users in mind. The figure below shows a sketch of the layered
+ architecture that supports the extensibility and ease of use for end-user
+ developers.
+
+
+
+
+
+
+
+
+ Figure 1.1: Spring Batch Layered
+ Architecture
+
+
+ This layered architecture highlights three major high level
+ components: Application, Core, and Infrastructure. The application
+ contains all batch jobs and custom code written by developers using Spring
+ Batch. The Batch Core contains the core runtime classes necessary to
+ launch and control a batch job. It includes things such as a
+ JobLauncher, Job, and
+ Step implementations. Both Application and Core are
+ built on top of a common infrastructure. This infrastructure contains
+ common readers and writers, and services such as the
+ RetryTemplate, which are used both by application
+ developers(ItemReader and
+ ItemWriter) and the core framework itself.
+ (retry)
+
+
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/step.xml b/docs/src/site/docbook/reference/step.xml
index d3a5520d4..1d89142d7 100644
--- a/docs/src/site/docbook/reference/step.xml
+++ b/docs/src/site/docbook/reference/step.xml
@@ -1,771 +1,858 @@
-
- Step
+
+ Configuring a Step
+
+ A Step is a domain object that encapsulates an
+ independent, sequential phase of a batch job and contains all of the
+ information necessary to define and control the actual batch processing.
+ This is a necessarily vague description because the contents of any given
+ Step are at the discretion of the developer writing a
+ Job. A Step can be as simple or complex as the
+ developer desires. A simple Step might load data from
+ a file into the database, requiring little or no code. (depending upon the
+ implementations used) A more complex Step may have
+ complicated business rules that are applied as part of the processing.
+
+
+
+
+
+
+
+
+
+
+
- Introduction
+ Chunk-Oriented Processing
- In Chapter 2, the overall architecture design was discussed, using
- the following diagram as a guide:
+ Spring Batch uses a 'Chunk Oriented' processing style within it's
+ most common implementation. Chunk oriented processing refers to reading
+ the data one at a time, and creating 'chunks' that will be written out,
+ within a transaction boundary. One item is read in from an
+ ItemReader, handed to an
+ ItemWriter, and aggregated. Once the number of
+ items read equals the commit interval, the entire chunk is written out via
+ the ItemWriter, and then the transaction is committed.
+ fileref="images/chunk-oriented-processing.png" scale="75"
+ width="" />
- When viewed from left to right, the diagram describes a basic flow
- for the execution of a batch job:
+ Below is a code representation of the same concepts shown above:
+
-
-
- A Scheduler kicks off a job script (usually some form of shell
- script)
-
+
+ List items = new Arraylist();
+ for(int i = 0; i < commitInterval; i++){
+ Object processedItem = itemProcessor.process(itemReader.read());
+ items.add(processedItem);
+ }
+ itemWriter.write(items);
-
- The script sets up the classpath appropriately, and starts the
- Java process. In most cases, using
- CommandLineJobRunner as the entry point
-
-
-
- The JobRunner finds the Job using the
- JobLocator, pulls together the
- JobParameters and launches the
- Job
-
-
-
- The JobLauncher retrieves a
- JobExecution from the
- JobRepository, and executes the
- Job
-
-
-
- The Job executes each
- Step in sequence.
-
-
-
- The Step calls read on the
- ItemReader, handing the resulting item to the
- ItemWriter until null is returned, periodically
- committing and storing status in the
- JobRepository.
-
-
-
- When execution is complete, the Step
- returns control back to the Job, and if no more
- steps exist, control is returned back to the original caller, in this
- case, the scheduler.
-
-
-
- This flow is perhaps a bit overly simplified, but describes the
- complete flow in the most basic terms. From here, each tier will be
- described in detail, using actual implementations and examples.
-
-
-
- Application Tier
-
- The Application tier is entirely concerned with the actual
- processing of input:
-
-
-
-
-
-
-
-
-
-
+
- StepHandlerStep
+ Configuring a Step
- The figure above shows a simple 'item-oriented' execution flow.
- One item is read in from an ItemReader, and then
- handed to an ItemWriter, until their are no more
- items left. When processing first begins, a transaction is started and
- periodically committed until the Step is
- complete. Given these basic requirements, the
- StepHandlerStep requires the following
- dependencies, at a minimum:
+ Despite the relatively short list of required dependencies for a
+ Step, it is an extremely complex class that can
+ potentially contain many collaborators. In order to ease configuration,
+ the Spring Batch namespace can be used:
-
-
- ItemReader - The
- ItemReader that provides items for
- processing.
-
+
+ <job id="sampleJob">
+ <step name="step1" job-repository="jobRepository" transaction-manager="transactionManager">
+ <tasklet reader="itemReader" writer="itemWriter" commit-interval="10"/>
+ </step>
+ </job>
-
- ItemWriter - The
- ItemWriter that processes the items provided
- by the ItemReader.
-
+
-
- PlatformTransactionManager - Spring
- transaction manager that will be used to begin and commit
- transactions during processing.
-
+ The configuration above represents the only required dependencies
+ to create a item-oriented step:
+
+ reader - The ItemReader that provides
+ items for processing.
+
-
- JobRepository - The
- JobRepository that will be used to
- periodically store the StepExecution and
- ExecutionContext during processing (just
- before committing).
-
-
+
+ writer - The ItemWriter that
+ processes the items provided by the
+ ItemReader.
+
+
+
+ transaction-manager - Spring's
+ PlatformTransactionManager that will be
+ used to begin and commit transactions during processing.
+
+
+
+ job-repository - The JobRepository
+ that will be used to periodically store the
+ StepExecution and
+ ExecutionContext during processing (just
+ before committing).
+
+
+
+ commit-interval - The number of items that will be processed
+ before the transaction is committed.
+
+
+
+ It should be noted that, job-repository defaults to
+ "jobRepository" and transaction-manager defaults to "transactionManger".
+ Furthermore, the ItemProcessor is not required, since the item could be
+ directly passed from the reader to the writer.
+
+
+
+ The Commit Interval
+
+ As mentioned above, a step reads in and writes out items,
+ periodically committing using the supplied
+ PlatformTransactionManager. With a
+ commit-interval of 1, it will commit after writing only one item. This
+ is less than ideal in many situations, since beginning and committing a
+ transaction is expensive. Ideally, it is preferable to process as many
+ items as possible in each transaction, which is completely dependent
+ upon the type of data being processed and the resources with which the
+ step is interacting. For this reason, the number of items that are
+ processed within a commit can be configured.
+
+
+ <job id="sampleJob">
+ <step name="step1" job-repository="jobRepository" transaction-manager="transactionManager">
+ <tasklet reader="itemReader" writer="itemWriter" commit-interval="10"/>
+ </step>
+ </job>
+
+
+
+ In the example above, 10 items will be processed within each
+ transaction. At the beginning of processing a transaction is begun, and
+ each time read is called on the
+ ItemReader, a counter is incremented. When it
+ reaches 10, the list of aggregated items is passed to the
+ ItemWriter, and the transaction will be
+ committed.
+
+
+
+ Configuring a Step for Restart
+
+ In the Chapter on configuring a Job, restarting a
+ Job was discussed. Restart has numerous impacts
+ on steps, and as such may require some specific configuration.
- SimpleStepFactoryBean
+ Setting a StartLimit
- Despite the relatively short list of required dependencies for
- an StepHandlerStep, it is an extremely complex
- class that can potentially contain many collaborators. In order to
- ease configuration, a SimpleStepFactoryBean can
- be used:
+ There are many scenarios where you may want to control the
+ number of times a Step may be started. An
+ example is a Step that may be run only once,
+ usually because it invalidates some resource that must be fixed
+ manually before it can be run again. This is configurable on the step
+ level, since different steps have different requirements. One Step
+ that may only be executed once can exist as part of the same
+ Job as Step that can be
+ run infinitely. Below is an example start limit configuration:
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
- <property name="transactionManager" ref="transactionManager" />
- <property name="jobRepository" ref="jobRepository" />
- <property name="itemReader" ref="itemReader" />
- <property name="itemWriter" ref="itemWriter" />
- </bean>
+
+ <step name="step1">
+ <tasklet reader="itemReader" writer="itemWriter" commit-interval="10" start-limit="1"/>
+ </step>
- The configuration above represents the only required
- dependencies of the factory bean. Attempting to instantiate a
- SimpleStepFactoryBean without at least those
- four dependencies will result in an exception being thrown during
- construction by the Spring container.
+
+
+ The simple step above can be run only once. Attempting to run it
+ again will cause an exception to be thrown. It should be noted that
+ the default value for the start-limit is
+ Integer.MAX_VALUE.
- Configuring a CommitInterval
+ Restarting a completed step
- As mentioned above, the StepHandlerStep
- reads in and writes out items, periodically commiting using the
- supplied PlatformTransactionManager. By
- default, it will commit after each item has been written. This is less
- than ideal in many situations, since beginning and commiting a
- transaction is expensive. Ideally, you would like to process as many
- items as possible in each transaction, which is completely dependant
- upon the type of data being processed and the resources that are being
- interacted with. For this reason, the number of items that are
- processed within a commit can be set as the commit interval:
+ In the case of a restartable job, there may be one or more steps
+ that should always be run, regardless of whether or not they were
+ successful the first time. An example might be a validation step, or a
+ Step that cleans up resources before
+ processing. During normal processing of a restarted job, any step with
+ a status of 'COMPLETED', meaning it has already been completed
+ successfully, will be skipped. Setting allow-start-if-complete to
+ "true" overrides this so that the step will always run:
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
- <property name="transactionManager" ref="transactionManager" />
- <property name="jobRepository" ref="jobRepository" />
- <property name="itemReader" ref="itemReader" />
- <property name="itemWriter" ref="itemWriter" />
- <property name="commitInterval" value="10" />
- </bean>
+
+ <step name="step1">
+ <tasklet reader="itemReader" writer="itemWriter" commit-interval="10" allow-start-if-complete="true"/>
+ </step>
- In this example, 10 items will be processed within each
- transaction. At the beginning of processing a transaction is begun,
- and each time read is called on the
- ItemReader, a counter is incremented. When it
- reaches 10, the transaction will be committed.
+
- Configuring a Step for Restart
+ Step restart configuration example
- Earlier in this chapter, restarting a Job
- was discussed. Restart has numerous impacts on steps, and as such may
- require some specific configuration.
+
+ <job id="footballJob" restartable="true">
+ <step name="playerload" next="gameLoad">
+ <tasklet reader="playerFileItemReader" writer="playerWriter"
+ commit-interval="10" />
+ </step>
+ <step name="gameLoad" next="playerSummarization">
+ <tasklet reader="gameFileItemReader" writer="gameWriter"
+ commit-interval="10" allow-start-if-complete="true"/>
+ </step>
+ <step name="playerSummarization">
+ <tasklet reader="playerSummarizationSource" writer="summaryWriter"
+ commit-interval="10" start-limit="3"/>
+ </step>
+ </job>
-
- There are many scenarios where you may want to control the
- number of times a Step may be started. An
- example is a Step that may be run only once,
- usually because it invalidates some resource that must be fixed
- manually before it can be run again. This is configurable on the
- step level, since different steps have different requirements. One
- Step that may only be executed once can exist as part of the same
- Job as Step that can
- be run infinitely. Below is an example start limit
- configuration:
+ The above example configuration is for a job that loads in
+ information about football games and summarizes them. It contains
+ three steps: playerLoad, gameLoad, and playerSummarization. The
+ playerLoad Step loads player information from a
+ flat file, while the gameLoad Step does the
+ same for games. The final Step,
+ playerSummarization, then summarizes the statistics for each player
+ based upon the provided games. It is assumed that the file loaded by
+ 'playerLoad' must be loaded only once, but that 'gameLoad' will load
+ any games found within a particular directory, deleting them after
+ they have been successfully loaded into the database. As a result, the
+ playerLoad Step contains no additional
+ configuration. It can be started almost limitlessly, and if complete
+ will be skipped. The 'gameLoad' Step, however,
+ needs to be run every time in case extra files have been dropped since
+ it last executed. It has 'allow-start-if-complete' set to 'true' in
+ order to always be started. (It is assumed that the database tables
+ games are loaded into has a process indicator on it, to ensure new
+ games can be properly found by the summarization step). The
+ summarization Step, which is the most important
+ in the Job, is configured to have a start limit
+ of 3. This is useful because if the step continually fails, a new exit
+ code will be returned to the operators that control job execution, and
+ it won't be allowed to start again until manual intervention has taken
+ place.
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
- <property name="transactionManager" ref="transactionManager" />
- <property name="jobRepository" ref="jobRepository" />
- <property name="itemReader" ref="itemReader" />
- <property name="itemWriter" ref="itemWriter" />
- <property name="commitInterval" value="10" />
- <property name="startLimit" value="1" />
- </bean>
+
+ This job is purely for example purposes and is not the same as
+ the footballJob found in the samples project.
+
- The simple step above can be run only once. Attempting to run
- it again will cause an exception to be thrown. It should be noted
- that the default value for startLimit is
- Integer.MAX_VALUE.
-
+ Run 1:
-
- Restarting a completed step
+
+
+ playerLoad is executed and completes successfully, adding
+ 400 players to the 'PLAYERS' table.
+
- In the case of a restartable job, there may be one or more
- steps that should always be run, regardless of whether or not they
- were successful the first time. An example might be a validation
- step, or a step that cleans up resources before processing. During
- normal processing of a restarted job, any step with a status of
- 'COMPLETED', meaning it has already been completed successfully,
- will be skipped. Setting allowStartIfComplete to true overrides this
- so that the step will always run:
+
+ gameLoad is executed and processes 11 files worth of game
+ data, loading their contents into the 'GAMES' table.
+
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
- <property name="transactionManager" ref="transactionManager" />
- <property name="jobRepository" ref="jobRepository" />
- <property name="itemReader" ref="itemReader" />
- <property name="itemWriter" ref="itemWriter" />
- <property name="commitInterval" value="10" />
- <property name="startLimit" value="1" />
- <property name="allowStartIfComplete" value="true" />
- </bean>
-
+
+ playerSummarization begins processing and fails after 5
+ minutes.
+
+
-
- Step restart configuration example
+ Run 2:
- <bean id="footballJob"
- class="org.springframework.batch.core.job.SimpleJob">
- <property name="steps">
- <list>
- <!-- Step Bean details ommitted for clarity -->
- <bean id="playerload" parent="simpleStep" />
- <bean id="gameLoad" parent="simpleStep" >
- <property name="allowStartIfComplete" value="true" />
- </bean>
- <bean id="playerSummarization" parent="simpleStep" >
- <property name="startLimit" value="2" />
- </bean>
- </list>
- </property>
- <property name="jobRepository" ref="jobRepository" />
- <property name="restartable" value="true" />
- </bean>
+
+
+ playerLoad is not run, since it has already completed
+ successfully, and allow-start-if-complete is 'false' (the
+ default).
+
- The above example configuration is for a job that loads in
- information about football games and summarizes them. It contains
- three steps: playerLoad, gameLoad, and playerSummarization. The
- playerLoad Step loads player information from
- a flat file, while the gameLoad
- Step does the same for games. The final
- Step, playerSummarization, then summarizes
- the statistics for each player based upon the provided games. It is
- assumed that the file loaded by 'playerLoad' must be loaded only
- once, but that 'gameLoad' will load any games found within a
- particular directory, deleting them after they have been
- successfully loaded into the database. As a result, the playerLoad
- Step contains no additional configuration. It
- can be started almost limitlessly, and if complete will be skipped.
- The 'gameLoad' Step, however, needs to be run
- everytime, in case extra files have been dropped since it last
- executed, so it has 'allowStartIfComplete' set to 'true' in order to
- always be started. (It is assumed that the database tables games are
- loaded into has a process indicator on it, to ensure new games can
- be properly found by the summarization step) The summarization
- step, which is the most important in the
- Job, is configured to have a start limit of
- 3. This is useful in case it continually fails, a new exit code will
- be returned to the operators that control job execution, and it
- won't be allowed to start again until manual intervention has taken
- place.
+
+ gameLoad is executed again and processes another 2 files,
+ loading their contents into the 'GAMES' table as well (with a
+ process indicator indicating they have yet to be processed)
+
-
- This job is purely for example purposes and is not the same
- as the footballJob found in the samples project.
-
+
+ playerSummarization begins processing of all remaining game
+ data (filtering using the process indicator) and fails again after
+ 30 minutes.
+
+
- Run 1:
+ Run 3:
-
-
- playerLoad is executed and completes successfully, adding
- 400 players to the 'PLAYERS' table.
-
+
+
+ playerLoad is not run, since it has already completed
+ successfully, and allow-start-if-complete is 'false' (the
+ default).
+
-
- gameLoad is executed and processes 11 files worth of game
- data, loading their contents into the 'GAMES' table.
-
+
+ gameLoad is executed again and processes another 2 files,
+ loading their contents into the 'GAMES' table as well (with a
+ process indicator indicating they have yet to be processed)
+
-
- playerSummarization begins processing and fails after 5
- minutes.
-
-
-
- Run 2:
-
-
-
- playerLoad is not run, since it has already completed
- succesfully, and allowStartIfComplete is false (the
- default).
-
-
-
- gameLoad is executed again and processes another 2 files,
- loading their contents into the 'GAMES' table as well (with a
- process indicator indicating they have yet to be
- processed)
-
-
-
- playerSummarization begins processing of all remaining
- game data (filtering using the process indicator) and fails
- again after 30 minutes.
-
-
-
- Run 3:
-
-
-
- playerLoad is not run, since it has already completed
- succesfully, and allowStartIfComplete is false (the
- default).
-
-
-
- gameLoad is executed again and processes another 2 files,
- loading their contents into the 'GAMES' table as well (with a
- process indicator indicating they have yet to be
- processed)
-
-
-
- playerSummarization is not start, and the job is
- immeadiately killed, since this is the third execution of
- playerSummarization, and it's limit is only 2. The limit must
- either be raised, or the Job must be
- executed as a new JobInstance.
-
-
-
+
+ playerSummarization is not start, and the job is immediately
+ killed, since this is the third execution of playerSummarization,
+ and it's limit is only 2. The limit must either be raised, or the
+ Job must be executed as a new
+ JobInstance.
+
+
+
+
+
+ Configuring Skip Logic
+
+ There are many scenarios where errors encountered while processing
+ should not result in Step failure, but should be
+ skipped instead. This is usually a decision that must be made by someone
+ who understands the data itself and what meaning it has. Financial data,
+ for example, may not be skippable because it results in money being
+ transferred, which needs to be completely accurate. Loading a list of
+ vendors, on the other hand, might allow for skips. If a vendor is not
+ loaded because it was formatted incorrectly or was missing necessary
+ information, then there probably won't be issues. Usually these bad
+ records are logged as well, which will be covered later when discussing
+ listeners.
+ <step name="step1">
+ <tasklet reader="flatFileItemReader" writer="itemWriter" commit-interval="10" skip-limit="10">
+ <skippable-exception-classes>
+ org.springframework.batch.item.file.FlatFileParseException
+ </skippable-exception-classes>
+ </tasklet>
+ </step>
+
+
+
+ In this example, a FlatFileItemReader is
+ used, and if at any point a
+ FlatFileParseException is thrown, it will be
+ skipped and counted against the total skip limit of 10. It should be
+ noted that any failures encountered while reading will not count against
+ the commit interval. In other words, the commit interval is only
+ incremented on writes (regardless of success or failure).
+
+
+
+ One problem with the example above is that any other exception
+ besides a FlatFileParseException will cause the
+ Job to fail. In certain scenarios this may be the
+ correct behaviour, however, in certain scenarios it may be easier to
+ identify which exceptions should cause failure and skip everything
+ else:
+ <step name="step1">
+ <tasklet reader="flatFileItemReader" writer="itemWriter" commit-interval="10" skip-limit="10">
+ <skippable-exception-classes>
+ java.lang.Exception
+ </skippable-exception-classes>
+ <fatal-exception-classes>
+ java.io.FileNotFoundException
+ </fatal-exception-classes>
+ </tasklet>
+ </step>
+
+
+
+ By setting the skippable exceptions to
+ java.lang.Exception, any exception that is thrown
+ will be skipped. However, the second list, 'fatal-exception-classes',
+ contains specific exceptions that should be fatal if encountered.
+
+
+
+ Configuring Retry Logic
+
+ In most cases you want an exception to cause either a skip or
+ Step failure. However, not all exceptions are
+ deterministic. If a FlatFileParseException is
+ encountered while reading, it will always be thrown for that record.
+ Resetting the ItemReader will not help. However,
+ for other exceptions, such as a
+ DeadlockLoserDataAccessException, which indicates
+ that the current process has attempted to update a record that another
+ process holds a lock on, waiting and trying again might result in
+ success. In this case, retry should be configured:
+
+
+ <step name="step1">
+ <tasklet reader="itemReader" writer="itemWriter" commit-interval="2" retry-limit="3">
+ <retryable-exception-classes>
+ org.springframework.dao.DeadlockLoserDataAccessException
+ </retryable-exception-classes>
+ </tasklet>
+ </step>
+
+
+
+ The Step allows a limit for the number of
+ times an individual item can be retried, and a list of exceptions that
+ are 'retryable'.
+
+
+
+ Controlling rollback
+
+ By default, regardless of retry or skip, any exceptions thrown
+ from the ItemWriter will cause the transaction
+ controlled by the Step to rollback. If skip is
+ configured as described above, exceptions thrown from the
+ ItemReader will not cause a rollback. However,
+ there are many scenarios in which exceptions thrown from the
+ ItemWriter should not cause a rollback because no
+ action has taken place to invalidate the transaction. For this reason,
+ the Step can be configured with a list of
+ exceptions that should not cause rollback. The transaction-attribute
+ attribute is a comma-separated list. Prefixing a class name with the "+"
+ symbol will indicate that exception should not cause rollback.
+
+
+ <step name="step1">
+ <tasklet reader="itemReader" writer="itemWriter" commit-interval="2" skip-limit="1"
+ transaction-attribute="+org.springframework.batch.item.validator.ValidationException">
+ </tasklet>
+ </step>
+
+
+
+ Transaction attributes can be used to control multiple other
+ settings such as isolation and propagation behaviour. More information
+ on setting transaction attributes can be found in the spring core
+ documentation.
+
+
+
+ Registering ItemStreams with the Step
+
+ The step has to take care of ItemStream
+ callbacks at the necessary points in its lifecycle. (for more
+ information on the ItemStream interface, please refer to the chapter on
+ Readers and Writers) This is vital if a step fails, and might need to be
+ restarted, because the ItemStream interface is
+ where the step gets the information it needs about persistent state
+ between executions.
+
+ If the ItemReader,
+ ItemProcessor, or
+ ItemWriter itself implements the
+ ItemStream interface, then these will be
+ registered automatically. Any other streams need to be registered
+ separately. This is often the case where there are indirect
+ dependencies, like delegates being injected into the reader and writer.
+ To a stream it can be injected into the Step
+ through the 'streams' element, as illustrated below:
+
+
+ <step name="step1">
+ <tasklet reader="itemReader" writer="compositeWriter" commit-interval="2">
+ <streams>
+ <stream ref="fileItemWriter1"/>
+ <stream ref="fileItemWriter2"/>
+ </streams>
+ </tasklet>
+ </step>
+
+ <beans:bean id="compositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter">
+ <beans:property name="delegates">
+ <beans:list>
+ <beans:ref bean="fileItemWriter1" />
+ <beans:ref bean="fileItemWriter2" />
+ </beans:list>
+ </beans:property>
+ </beans:bean>
+
+
+
+ In the example above, the
+ CompositeItemWriter is not an
+ ItemStream, but both of its delegates are.
+ Therefore, both delegate writers must be explicitly registered as
+ streams in order for the framework to handle them correctly. The
+ ItemReader does not need to explicitly registered
+ as a stream because it is a direct property of the
+ Step. The step will now be restartable and the
+ state of the reader and writer will be correctly persisted in case of a
+ failure.
+
+
+
+ Intercepting Step Execution
+
+ Just as with the Job, there are many events
+ during the execution of a Step where a user may
+ need to perform some functionality. For example, in order to write out
+ to a flat file that requires a footer, the
+ ItemWriter needs to be notified when the
+ Step has been completed, so that the footer can
+ written. This can be accomplished with one of many
+ Step scoped listeners.
+
+ Any class that implements the StepListener
+ interface (or an extension thereof) can be applied to a step via the
+ listeners element:
+
+
+ <step name="step1">
+ <tasklet reader="reader" writer="writer" commit-interval="10"/>
+ <listeners>
+ <listener ref="stepListener"/>
+ </listeners>
+ </step>
+
+
+
+ In addition to the StepListener interfaces,
+ annotations are provided address the same concerns.
- Configuring Skip Logic
+ StepExecutionListener
- There are many scenarios where errors encountered while
- processing should not result in Step failure,
- but should be skipped instead. This is usually a decision that must be
- made by someone who understands the data itself and what meaning it
- has. Financial data, for example, may not be skippable because it
- results in money being transferred, which needs to be completely
- accurate. Loading in a list of vendors, on the other hand, might allow
- for skips, since a vendor not being loaded because it was formatted
- incorrectly, or missing necessary information, won't cause issues.
- Usually these bad records are logged as well, which will be covered
- later when discussing listeners. Configuring skip handling requires
- using a new factory bean:
- SkipLimitStepFactoryBean <bean id="skipSample"
- class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
- <property name="skipLimit" value="10" />
- <property name="itemReader" ref="flatFileItemReader" />
- <property name="itemWriter" ref="itemWriter" />
- <property name="skippableExceptionClasses"
- value="org.springframework.batch.item.file.FlatFileParseException">
- </property>
- </bean>
+ StepExecutionListener represents the most
+ generic listener for Step execution. It allows
+ for notification before a Step is started and
+ after it has ends, whether it ended normally or failed:
- In this example, a FlatFileItemReader is
- used, and if at any point a FlatFileParseException is thrown, it will
- be skipped and counted against the total skip limit of 10. It should
- be noted that any failures encountered while reading will not count
- against the commit interval. In other words, the commit interval is
- only incremented on writes (regardless of success or failure).
-
-
-
- One problem with the example above is that any other exception
- besides a FlatFileParseException will cause the
- Job to fail. In certain scenarios this may be
- the correct behaviour, however, in certain scenarios it may be easier
- to identify which exceptions should cause failure and skip everything
- else: <bean id="skipSample"
- class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
- <property name="skipLimit" value="10" />
- <property name="itemReader" ref="flatFileItemReader" />
- <property name="itemWriter" ref="itemWriter" />
- <property name="skippableExceptionClasses"
- value="java.lang.Exception">
- <property name="fatalExceptionClasses"
- value="java.io.FileNotFoundException">
- </property>
- </befan>
-
- By setting the skippable exceptions to
- java.lang.Exception, any exception that is
- thrown will be skipped. However, the second list,
- 'fatalExceptionClasses', contains specific exceptions that should be
- fatal if encountered.
-
-
-
- Configuring Retry Logic
-
- In most cases you want an Exception to cause either a skip or
- Step failure. However, not all exceptions are
- deterministic. If a FlatFileParseException is encountered while
- reading, it will always be thrown for that record. Resseting the
- ItemReader will not help. However, for other
- exceptions, such as a
- DeadlockLoserDataAccessException, which
- indicates that the current process has attempted to update a record
- that another process holds a lock on, waiting and trying again might
- result in success. In this case, retry should be configured:
-
- <bean id="step1"
- class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
- <property name="itemReader" ref="itemGenerator" />
- <property name="itemWriter" ref="itemWriter" />
- <property name="retryLimit" value="3" />
- <property name="retryableExceptionClasses" value="org.springframework.dao.DeadlockLoserDataAccessException" />
- </bean>
-
- The SkipLimitStepFactoryBean requires a
- limit for the number of times an individual item can be retried, and a
- list of Exceptions that are 'retryable'.
-
-
-
- Controlling rollback
-
- By default, regardless of retry or skip, any exceptions thrown
- from the ItemWriter will cause the transaction
- controlled by the Step to rollback. If skip is
- configured as described above, exceptions thrown from the
- ItemReader will not cause a rollback. However,
- there are many scenarios in which exceptions thrown from the
- ItemWriter should not cause a rollback because
- no action has taken place to invalidate the transaction. For this
- reason, the SkipLimitStepFactoryBean can be
- configured with a list of exceptions that should not cause
- rollback:
-
- <bean id="step2"
- class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
- <property name="commitInterval" value="2" />
- <property name="skipLimit" value="1" />
- <!-- No rollback for exceptions that are marked with "+" in the tx attributes -->
- <property name="transactionAttribute"
- value="+org.springframework.batch.item.validator.ValidationException" />
- <property name="itemReader"
- ref="tradeSqlItemReader" />
- <property name="itemWriter"
- ref="itemTrackingWriter" />
- </bean>
-
- The TransactionAttribute property above
- can be used to control multiple other settings such as isolation and
- propagation behaviour. More information on setting transaction
- attributes can be found in the spring core documentation.
-
-
-
- Registering ItemStreams with the Step
-
- The step has to take care of ItemStream
- callbacks at the necessary points in its lifecycle. This is vital if a
- step fails, and might need to be restarted, because the
- ItemStream interface is where the step gets the
- information it needs about persistent state between executions. The
- factory beans that Spring Batch provides for convenient configuration
- of Step instances have features that allow
- streams to be registered with the step when it is configured.
-
- If the ItemReader or
- ItemWriter themselves implement the ItemStream
- interface, then these will be registered automatically. Any other
- streams need to be registered separately. This is often the case where
- there are indirect dependencies, like delegates being injected into
- the reader and writer. To register these they can be injected into the
- factory beans through the streams property, as illustrated
- below:
-
- <bean id="step1"
- class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
- <property name="streams" ref="fileItemReader" />
- <property name="itemReader">
- <bean
- class="org.springframework.batch.item.validator.ValidatingItemReader">
- <property name="itemReader" ref="itemReader" />
- <property name="validator" ref="fixedValidator" />
- </bean>
- </property>
- ...
-</bean>
-
- In the example above the main item reader is being set up to
- delegate to a bean called "fileItemReader", which itself is being
- registered as a stream directly. The step will now be restartable and
- the state of the reader will be correctly persisted in case of a
- failure.
-
-
-
- Intercepting Step Execution
-
- Just as with the Job, there are many
- events during the execution of a Step that a
- user may need notification of. For example, if writing out to a flat
- file that requires a footer, the ItemWriter
- needs to be notified when the Step has been
- completed, so that it can write the footer. This can be accomplished
- with one of many Step scoped listeners.
-
-
- StepExecutionListener
-
- StepExecutionListener represents the
- most generic listener for Step execution. It
- allows for notification before a Step is
- started, after it has completed, and if any errors are encountered
- during processing:
-
- public interface StepExecutionListener extends StepListener {
+
+ public interface StepExecutionListener extends StepListener {
void beforeStep(StepExecution stepExecution);
- ExitStatus onErrorInStep(StepExecution stepExecution, Throwable e);
-
ExitStatus afterStep(StepExecution stepExecution);
-}
+ }
- ExitStatus is the return type of
- onErrorInStep and
- afterStep in order to allow listeners the
- chance to modify the exit code that is returned upon completion of a
- Step. A
- StepExecutionListener can be applied to any
- step factory bean via the listeners property:
+
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
- <property name="transactionManager" ref="transactionManager" />
- <property name="jobRepository" ref="jobRepository" />
- <property name="itemReader" ref="itemReader" />
- <property name="itemWriter" ref="itemWriter" />
- <property name="commitInterval" value="10" />
- <property name="listeners" ref="stepListener" />
- </bean>
+ ExitStatus is the return type of
+ afterStep in order to allow listeners the
+ chance to modify the exit code that is returned upon completion of a
+ Step.
- Because all listeners extend the
- StepListener interface, they all may be
- applied to factory beans in the same way.
-
+ The annotations corresponding to this interface are:
-
- ChunkListener
+
+
+ @BeforeStep
+
- A chunk is defined as the items processed within the scope of
- a transaction. Committing a transaction commits a 'chunk'. It may be
- useful to be nofied before and after a chunk has completed, in which
- case the ChunkListener interface may be
- used:
+
+ @AfterStep
+
+
+
- public interface ChunkListener extends StepListener {
+
+ ChunkListener
+
+ A chunk is defined as the items processed within the scope of a
+ transaction. Committing a transaction, at each commit interval,
+ commits a 'chunk'. A ChunkListener may be
+ useful to perform logic before a chunk begins processing or after a
+ chunk has completed:
+
+ public interface ChunkListener extends StepListener {
void beforeChunk();
void afterChunk();
}
- The beforeChunk method is called
- after the transaction is started, but before
- read is called on the
- ItemReader. Conversely,
- afterChunk is called after the last call to
- write on the
- ItemWriter, but before the chunk has been
- committed.
-
+ The beforeChunk method is called after
+ the transaction is started, but before read
+ is called on the ItemReader. Conversely,
+ afterChunk is called after the last call to
+ write on the
+ ItemWriter, but before the chunk has been
+ committed.
-
- ItemReadListener
+ The annotations corresponding to this interface are:
- When discussing skip logic above, it was mentioned that it may
- be beneficial to log out skipped records, so that they can be deal
- with later. In the case of read errors, this can be done with an
- ItemReaderListener: public interface ItemReadListener extends StepListener {
+
+
+ @BeforeChunk
+
+
+
+ @AfterChunk
+
+
+
+
+
+ ItemReadListener
+
+ When discussing skip logic above, it was mentioned that it may
+ be beneficial to log out skipped records, so that they can be deal
+ with later. In the case of read errors, this can be done with an
+ ItemReaderListener: public interface ItemReadListener<T> extends StepListener {
void beforeRead();
- void afterRead(Object item);
+ void afterRead(T item);
void onReadError(Exception ex);
}
- The beforeRead method will be called
- before each call to read on the
- ItemReader. The
- afterRead method will be called after each
- successful call to read, and will be passed
- the item that was read. If there was an error while reading, the
- onReadError method will be called. The
- exception encounterd will be provided so that it can be
- logged.
-
+ The beforeRead method will be called
+ before each call to read on the
+ ItemReader. The
+ afterRead method will be called after each
+ successful call to read, and will be passed
+ the item that was read. If there was an error while reading, the
+ onReadError method will be called. The
+ exception encountered will be provided so that it can be
+ logged.
-
- ItemWriteListener
+ The annotations corresponding to this interface are:
- Just as with the ItemReaderListener, the writing of an item
- can be 'listened' to:
+
+
+ @BeforeRead
+
- public interface ItemWriteListener extends StepListener {
+
+ @AfterRead
+
- void beforeWrite(Object item);
+
+ @OnReadError
+
+
+
- void afterWrite(Object item);
+
+ ItemProcessListener
- void onWriteError(Exception ex, Object item);
+ Just as with the ItemReadListener, the
+ processing of an item can be 'listened' to:
+
+ public interface ItemProcessListener<T, S> extends StepListener {
+
+ void beforeProcess(T item);
+
+ void afterProcess(T item, S result);
+
+ void onProcessError(T item, Exception e);
}
- The beforeWrite method will be called
- before write on the
- ItemWriter, and is handed the item that will
- be written. The afterWrite method will be
- called after the item has been succesfully writen. If there was an
- error while writing, the onWriteError
- method will be called. The exception encountered and the item that
- was attempted to be written will be provided, so that they can be
- logged.
-
+ The beforeProcess method will be called
+ before process on the
+ ItemProcessor, and is handed the item that will
+ be processed. The afterProcess method will be
+ called after the item has been successfully processed. If there was an
+ error while processing, the onProcessError
+ method will be called. The exception encountered and the item that was
+ attempted to be processed will be provided, so that they can be
+ logged.
-
- SkipListener
+ The annotations corresponding to this interface are:
- Both ItemReadListener and
- ItemWriteListner provide a mechanism for
- being notified of errors, but neither one will inform you that a
- record has actually been skipped.
- onWriteError, for example, will be called
- even if an item is retried and successful. For this reason, there is
- a separate interface for tracking skipped items:
+
+
+ @BeforeProcess
+
-
- public interface SkipListener extends StepListener {
+
+ @AfterProcess
+
+
+
+ @OnProcessError
+
+
+
+
+
+ ItemWriteListener
+
+ The writing of an item can be 'listened' to with the
+ ItemWriteListener:
+
+ public interface ItemWriteListener<S> extends StepListener {
+
+ void beforeWrite(List<? extends S> items);
+
+ void afterWrite(List<? extends S> items);
+
+ void onWriteError(Exception exception, List<? extends S> items);
+}
+
+ The beforeWrite method will be called
+ before write on the
+ ItemWriter, and is handed the item that will be
+ written. The afterWrite method will be called
+ after the item has been successfully written. If there was an error
+ while writing, the onWriteError method will
+ be called. The exception encountered and the item that was attempted
+ to be written will be provided, so that they can be logged.
+
+ The annotations corresponding to this interface are:
+
+
+
+ @BeforeWrite
+
+
+
+ @AfterWrite
+
+
+
+ @OnWriteError
+
+
+
+
+
+ SkipListener
+
+ Both ItemReadListener and
+ ItemWriteListner provide a mechanism for being
+ notified of errors, but neither one will inform you that a record has
+ actually been skipped. onWriteError, for
+ example, will be called even if an item is retried and successful. For
+ this reason, there is a separate interface for tracking skipped
+ items:
+
+
+ public interface SkipListener<T,S> extends StepListener {
void onSkipInRead(Throwable t);
- void onSkipInWrite(Object item, Throwable t);
+ void onSkipInWrite(S item, Throwable t);
+
+ void onSkipInProcess(T item, Throwable t);
}
- onSkipInRead will be called whenever
- an item is skipped while reading. It should be noted that rollbacks
- may cause the same item to be registered as skipped more than once.
- onSkipInWrite will be called when an item
- is skipped while writing. Because the item has been read
- successfully (and not skipped), it is also provided the item itself
- as an argument.
-
+ onSkipInRead will be called whenever an
+ item is skipped while reading. It should be noted that rollbacks may
+ cause the same item to be registered as skipped more than once.
+ onSkipInWrite will be called when an item is
+ skipped while writing. Because the item has been read successfully
+ (and not skipped), it is also provided the item itself as an
+ argument.
+
+ The annotations corresponding to this interface are:
+
+
+
+ @OnSkipInRead
+
+
+
+ @OnSkipInWrite
+
+
+
+ @OnSkipInProcess
+
+
+
+
+
+ TaskletStep
+
+ Chunk oriented processing is not the only way to process in a
+ Step. What if a Step must
+ consist as a simple stored procedure call? You could implement the call as
+ an ItemReader and return null after the procedure
+ finishes, but it is a bit unnatural since there would need to be a no-op
+ ItemWriter and lots of overhead for transaction
+ handling, listeners, etc. Spring Batch provides an implementation of
+ Step for this scenario:
+ TaskletStep. As explained in Chapter 2, the
+ Tasklet is a simple interface that has one method,
+ execute, which will be a called once for the
+ whole Step. Tasklet
+ implementors might call a stored procedure, a script, or a simple SQL
+ update statement. To create a TaskletStep, the
+ 'tasklet' attribute should reference a bean defining a
+ Tasklet object; no 'tasklet' element is needed
+ within the 'step':
+
+
+ <step name="step1" tasklet="myTasklet" />
+
+
+
+
+ TaskletStep will automatically register the tasklet as
+ StepExecutionListener if it implements this
+ interface
+
+
+
+ TaskletAdapter
+
+ As with other adapters for the ItemWriter
+ and ItemReader interfaces, the
+ Tasklet interface contains an implementation that
+ allows for adapting itself to any pre-existing class:
+ TaskletAdapter. An example where this may be
+ useful is an existing DAO that is used to update a flag on a set of
+ records. The TaskletAdapter can be used to call
+ this class without having to write an adapter for the
+ Tasklet interface:
+
+
+ <bean id="myTasklet" class="org.springframework.batch.core.step.tasklet.TaskletAdapter">
+ <property name="targetObject">
+ <bean class="org.mycompany.FooDao">
+ </property>
+ <property name="targetMethod" value-"updateFoo" />
+ </bean>
+
+
+
- TaskletStep
+ Example Tasklet implementation
- Item oriented processing is not the only way to process in a
- Step. What if a Step must
- consist as a simple storec procedure call? You could implement the call
- as an ItemReader and return null after the
- procedure finishes, but it is a bit unnatural since there would need to
- be a no-op ItemWriter and lots of overhead for
- transaction handling, listeners, etc. Spring Batch provides an
- implementation of Step for this scenario:
- TaskletStep. As explained in Chapter 2, the
- Tasklet is a simple interface that has one
- method, execute, which will be a called once
- for the whole Step.
- Tasklet implementors might call a stored
- procedure, a script, or a simple SQL upate statement. Because there are
- less concerns, there are only two required dependencies for a
- TaskletStep: a Tasklet,
- and a JobRepository:
+ Many batch jobs contain steps that must be done before the main
+ processing begins in order to set up various resources, or after
+ processing has completed to cleanup those resources. In the case of a
+ job that works heavily with files, it is often necessary to delete
+ certain files locally after they have been uploaded successfully to
+ another location. The example below taken from the Spring Batch samples
+ project, is a Tasklet implementation with just
+ such a responsibility:
- <bean id="taskletStep"
- class="org.springframework.batch.core.step.tasklet.TaskletStep" />
- <property name="tasklet" ref="tasklet" />
- <property name="jobRepository" ref="repository" />
-</bean>
-
-
- TaskletStep will automatically register the tasklet as
- StepExecutionListener if it implements this
- interface
-
-
-
- TaskletAdapter
-
- As with other adapters for the ItemWriter
- and ItemReader interfaces, the
- Tasklet interface contains an implementation
- that allows for adapting itself to any pre-existing class:
- TaskletAdapter. An example where this may be
- useful is an existing DAO that is used to upate a flag on a set of
- records. The TaskletAdapter can be used to call
- this class without having to write an adapter for the
- Tasklet interface:
-
- <bean id="deleteFilesInDir" parent="taskletStep">
- <property name="tasklet">
- <bean class="org.springframework.batch.core.step.tasklet.TaskletAdapter">
- <property name="targetObject">
- <bean class="org.mycompany.FooDao">
- </property>
- <property name="targetMethod" value-"updateFoo" />
- </bean>
- </property>
- </bean>
-
-
-
- Example Tasklet implementation
-
- Many batch jobs contains steps that must be done before the main
- processing begins in order to set up various resources, or after
- processing has completed to cleanup those resources. In the case of a
- job that works heavily with files, it is often necessary to delete
- certain files locally after they have been uploaded successfully to
- another location. The example below taken from the Spring Batch
- samples project, is a Tasklet implementation
- with just such a responsibility:
-
- public class FileDeletingTasklet implements Tasklet, InitializingBean {
+ public class FileDeletingTasklet implements Tasklet, InitializingBean {
private Resource directory;
@@ -792,42 +879,42 @@
}
}
- The above Tasklet implementation will
- delete all files within a given directory. It should be noted that the
- execute method will only be called once. All
- that is left is to inject the Tasklet into a
- TaskletStep:
+ The above Tasklet implementation will
+ delete all files within a given directory. It should be noted that the
+ execute method will only be called once. All
+ that is left is to reference the Tasklet from the
+ Step:
- <bean id="taskletJob" parent="simpleJob">
- <property name="steps">
- <bean id="deleteFilesInDir" parent="taskletStep">
- <property name="tasklet">
- <bean class="org.springframework.batch.sample.tasklet.FileDeletingTasklet">
- <property name="directoryResource" ref="directory" />
- </bean>
- </property>
+
+ <job id="taskletJob">
+ <step name="deleteFilesInDir" tasklet="fileDeletingTasklet"/>
+ </job>
+
+ <bean id="fileDeletingTasklet"
+ class="org.springframework.batch.sample.tasklet.FileDeletingTasklet">
+ <property name="directoryResource">
+ <bean id="directory"
+ class="org.springframework.core.io.FileSystemResource">
+ <constructor-arg value="target/test-outputs/test-dir" />
</bean>
</property>
</bean>
- <bean id="directory"
- class="org.springframework.core.io.FileSystemResource">
- <constructor-arg value="target/test-outputs/test-dir" />
- </bean>
-
+
+
-
- Executing System Commands
+
+ Executing System Commands
- Many batch jobs may require that an external command be called
- from within the batch job. Such a process could be kicked off
- separately by the scheduler, but the advantage of common meta-data
- about the run would be lost. Furthermore, a multi-step job would also
- need to be split up into multiple jobs as well. Because the need is so
- common, Spring Batch provides a Tasklet
- implementation for calling system commands:
+ Many batch jobs may require that an external command be called
+ from within the batch job. Such a process could be kicked off separately
+ by the scheduler, but the advantage of common meta-data about the run
+ would be lost. Furthermore, a multi-step job would also need to be split
+ up into multiple jobs as well. Because the need is so common, Spring
+ Batch provides a Tasklet implementation for
+ calling system commands:
-
+
<bean class="org.springframework.batch.sample.tasklet.SystemCommandTasklet">
<property name="command" value="echo hello" />
<!-- 5 second timeout for the command to complete -->
@@ -835,263 +922,292 @@
</bean>
-
-
-
-
- Controlling Step Flow
-
-
-
-
-
-
- Configuring for Stop
-
-
-
-
-
- Programmatic flow decisions
-
-
-
- Examples of Customized Business Logic
+ Controlling Step Flow
+
+ With the ability to group steps together within an owning job, comes
+ the need to be able to control how the job 'flows' from one step to
+ another. The failure of a Step doesn't necessarily
+ mean that the Job should fail. Further, there may
+ be more than one type of 'success', which determines which
+ Step should be executed next. Depending upon how a
+ group of Steps is configured, certain steps may not even be processed at
+ all.
- Some batch jobs can be assembled purely from off-the-shelf
- components in Spring Batch, mostly the ItemReader
- and ItemWriter implementations. Where this is not
- possible (the majority of cases) the main API entry points for
- application developers are the Tasklet,
- ItemReader, ItemWriter and
- the various listener interfaces. Most simple batch jobs will be able to
- use off-the-shelf input from a Spring Batch
- ItemReader, but it is very often the case that
- there are custom concerns in the processing and writing, which normally
- leads developers to implement an ItemWriter, or
- ItemTransformer.
+ Sequential Flow
- Here we provide a few examples of common patterns in custom
- business logic, mainly using the listener interfaces . It should be
- noted that an ItemReader or
- ItemWriter can implement the listener interfaces
- as well if appropriate.
+ The simplest flow scenario is a job where all of the steps execute
+ sequentially:
+
+
+
+
+
+
+
+
+
+
+
+ This can be achieved using the 'next' attribute of
+ Step:
+
+
+ <job id="job">
+ <step name="stepA" next="stepB" />
+ <step name="stepB" next="stepC"/>
+ <step name="stepC" />
+ </job>
+
+In the scenario above, 'step A' will execute first. If 'step
+ A' completes normally, then 'step B' will execute and so on. However, if
+ 'step A' fails, then the entire Job will fail and
+ 'step B' will not execute.
- Logging Item Processing and Failures
+ Conditional Flow
- A common use case is the need for special handling of errors in a
- step, item by item, perhaps logging to a special channel, or inserting a
- record into a database. The StepHandlerStep
- (created from the step factory beans) allows users to implement this use
- case with a simple ItemReadListener, for errors
- on read, and an ItemWriteListener, for errors on
- write. The below code snippets illustrate a listener that logs both read
- and write failures:
+ In the example above, there's only two possibilities:
- public class ItemFailureLoggerListener extends ItemListenerSupport {
+
+
+ The Step is successful and the next Step should be
+ executed
+
- private static Log logger = LogFactory.getLog("item.error");
+
+ The Step failed and thus the Job should fail.
+
+
- public void onReadError(Exception ex) {
- logger.error("Encountered error on read", e);
- }
-
- public void onWriteError(Exception ex, Object item) {
- logger.error("Encountered error on write", e);
+ In many cases this may be sufficient. However, what about a
+ scenario in which the failure of a Step should trigger a different Step,
+ rather than causing failure?
+
+
+
+
+
+
+
+
+
+ In order to handle this scenario, the next step can be determined
+ based on the result of the step by adding a next element to the Step.
+ The "on" attribute uses a simple pattern-matching scheme to match the
+ exit code of the Step to the various next elements declared. Only two
+ special characters are allowed:
+
+
+
+ "*" will zero or more characters
+
+
+
+ "?" will match exactly one character
+
+
+
+ For example, "c*t" will match "cat" and "count", while "c?t" will
+ match "cat" but not "count".
+
+ Any number of "next" elements is allowed, but if the step has an
+ exit code that is not covered by a "next" element, then the framework
+ will throw an exception and the job will fail. It is important to note
+ that the framework will automatically order transitions from most
+ specific to least specific. So even if the "next" elements were swapped
+ for "stepA" below, an exit status of "FAILED" would still go to
+ "stepB".
+
+
+ <job id="job">
+ <step name="stepA">
+ <next on="FAILED" to="stepB" />
+ <next on="*" to="stepC" />
+ </step>
+ <step name="stepB" next="stepC" />
+ <step name="stepC" />
+ </job>
+
+
+
+
+ Batch Status vs. Exit Status
+
+ When configuring a Job for conditional
+ flow, it is important to understand the difference between
+ BatchStatus and
+ ExitStatus. BatchStatus
+ is an enumeration that is a property of both
+ JobExecution and
+ StepExecution, and is used by the framework to
+ record the status of a Job or
+ Step. It can be one of the following values:
+ COMPLETED, STARTING, STARTED, FAILED, STOPPING, STOPPED, or UNKNOWN.
+ Most of them are self explanatory, COMPLETED is the status set when a
+ step or job has completed successfully, FAILED is set when it fails,
+ and so on. The example above contains the following 'next'
+ element:
+
+
+ <next on="FAILED" to="stepB" />
+
+
+
+ At first glance, it would appear that the 'on' attribute
+ references the BatchStatus of the
+ Step it belongs to. However, it references the
+ ExitStatus of the Step.
+ As the name implies, ExitStatus represents the
+ status of a Step after it finishes execution.
+ More specifically, the 'next' element above references the
+ ExitCode of the
+ ExitStatus. To write it in English, it says:
+ "go to stepB if the exit code is FAILED". By default, the exit code is
+ always the same as the BatchStatus for the
+ Step, which is why the entry above works. However, what if the exit
+ code needs to be different? A good example comes from the skip sample
+ job, within the samples project:
+
+
+ <step name="step1">
+ <end on="FAILED" />
+ <next on="COMPLETED WITH SKIPS" to="errorPrint1" />
+ <next on="*" to="step2" />
+ </step>
+
+
+
+ The above step has three possibilities:
+
+
+
+ The step failed, in which case the job should fail.
+
+
+
+ The Step completed successfully.
+
+
+
+ The Step completed successfully, but with an exit code of
+ 'COMPLETED WITH SKIPS'. In this case, a different step should be
+ run to handle the errors.
+
+
+
+ The above configuration will work, however, something needs to
+ change the exit code based on the condition of the execution having
+ skipped records:
+
+ public class SkipCheckingListener implements StepExecutionListener {
+
+ public ExitStatus afterStep(StepExecution stepExecution) {
+ if (!stepExecution.getExitStatus().getExitCode().equals(ExitStatus.FAILED.getExitCode())
+ && stepExecution.getSkipCount() > 0) {
+ return new ExitStatus("COMPLETED WITH SKIPS");
+ } else {
+ return null;
}
+ }
+ ...
}
- Having implemented this listener it must be registered with the
- step:
-
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
- ...
- <property name="listeners">
- <bean class="org.example...ItemFailureLoggerListener"/>
- </property>
- </bean>
-
- Remember that if your listener does anything in an
- onError() method, it will be inside a transaction that is
- going to be rolled back. If you need to use a transactional resource
- such as a database inside an onError() method, consider
- adding a declarative transaction to that method (see Spring Core
- Reference Guide for details), and giving its propagation attribute the
- value REQUIRES_NEW.
+ The above code is a StepExecutionListener
+ that first checks to make sure the Step was
+ successful, and next if the skip count on the
+ StepExecution is higher than 0. If both
+ conditions are met, a new ExitStatus with an exit code of "COMPLETED
+ WITH SKIPS" is returned.
+
- Stopping a Job Manually for Business Reasons
+ Configuring for Stop
- Spring Batch provides a stop() method through the JobLauncher
- interface, but this is really aimed at the operator, rather than the
- application programmer. Sometimes it is more convenient or makes more
- sense to stop a job execution from within the business logic.
+ If it is desired that the batch job stop under certain conditions,
+ then either the "stop" tag or the "end" tag may be used.
- The simplest thing to do is to throw a RuntimeException (one that
- isn't retried indefinitely or skipped), For example, a custom exception
- type could be used, as in the example below:
+ The "stop" tag indicates the job should stop processing with an
+ exit status of "STOPPED". The "to" attribute tells the framework which
+ step should be first when the job is subsequently restarted. This
+ mechanism allows the job to pause temporarily.
- public class PoisonPillItemWriter extends AbstractItemWriter {
-
- public void write(Object item) throws Exception {
+ On the other hand, the "end" tag will stop the job but does not
+ allow for a "to" attribute. The "status" attribute is optional. It will
+ determine the exit status of the step if the flow ends at that location.
+ The only legal values for the "status" are "COMPLETED", "FAILED", and
+ "STOPPED". If no status is specified, then the default is
+ "COMPLETED".
- if (isPoisonPill(item)) {
- throw new PoisonPillException("Posion pill detected: "+item);
- }
+
+ <step name="step1">
+ <stop on="COMPLETED" to="step2"/>
+ </step>
+ <step name="step2">
+ <next on="FOO" to="step3"/>
+ <end on="*" status="FAILED"/>
+ </step>
+ <step name="step3" />
- }
-
-}
-
- Another simple way to stop a step from executing is to simply
- return null from the
- ItemReader:
-
- public class EarlyCompletionItemReader extends AbstractItemReader {
-
- private ItemReader delegate;
-
- public void setDelegate(ItemReader delegate) { ... }
-
- public Object read() throws Exception {
-
- Object item = delegate.read();
-
- if (isEndItem(item)) {
- return null; // end the step here
- }
-
- return item;
-
- }
-
-}
-
- The previous example actually relies on the fact that there is a
- default implementation of the CompletionPolicy
- strategy which signals a complete batch when the item to be processed is
- null. A more sophisticated completion policy could be implemented and
- injected into the Step through the
- RepeatOperationsStepFactoryBean:
-
- <bean id="simpleStep"
- class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" >
- ...
- <property name="chunkOperations">
- <bean class="org.springframework.batch.repeat.support.RepeatTemplate">
- <property name="completionPolicy">
- <bean class="org.example...SpecialCompletionPolicy"/>
- </property>
- </bean>
- </property>
- </bean>
-
- An alternative is to set a flag in the
- StepExecution, which is checked by the
- Step implementations in the framework in between
- item processing. To implement this alternative, we need access to the
- current StepExecution, and this can be achieved by implementing a
- StepListener and registering it with the Step. Here is an example of a
- listener that sets the flag:
-
- public class CustomItemWriter extends ItemListenerSupport implements StepListener {
-
- private StepExecution stepExecution;
-
- public void beforeStep(StepExecution stepExecution) {
- this.stepExecution = stepExecution;
- }
-
- public void afterRead(Object item) {
-
- if (isPoisonPill(item)) {
- stepExecution.setTerminateOnly(true);
- }
-
- }
-
-}
-
- The default behaviour here when the flag is set is for the step to
- throw a JobInterruptedException. This can be
- controlled through the StepInterruptionPolicy,
- but the only choice is to throw or not throw an exception, so this is
- always an abnormal ending to a job.
+
- Adding a Footer Record
+ Programmatic flow decisions
- A very common requirement is to aggregate information during the
- output process and to append a record at the end of a file summarizing
- the data, or providing a checksum. This can also be achieved with a
- callbacks in the step, normally as part of a custom
- ItemWriter. In this case, since a job is
- accumulating state that should not be lost if the job aborts, the
- ItemStream interface should be
- implemented:
+ In some situations, more information than the exit status may be
+ required to decide which step to execute next. In this case, a
+ JobExecutionDecider can be used to assist in the
+ decision.
- public class CustomItemWriter extends AbstractItemWriter implements
- ItemStream, StepListener
-{
+
+ public class MyDecider implements JobExecutionDecider {
- private static final String TOTAL_AMOUNT_KEY = "total.amount";
-
- private ItemWriter delegate;
-
- private double totalAmount = 0.0;
-
- public void setDelegate(ItemWriter delegate) { ... }
-
- public ExitStatus afterStep(StepExecution stepExecution) {
- // Add the footer record here...
- delegate.write("Total Amount Processed: " + totalAmount);
+ public String decide(JobExecution jobExecution, StepExecution stepExecution) {
+ if (someCondition) {
+ return "FAILED";
+ }
+ else {
+ return "COMPLETED";
+ }
}
- public void open(ExecutionContext executionContext) {
- if (executionContext.containsKey(TOTAL_AMOUNT_KEY) {
- totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY);
- }
- }
+ }
- public void update(ExecutionContext executionContext) {
- executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount);
- }
+
+
+ In the job configuration, a "decision" tag will specify the
+ decider to use as well as all of the transitions.
+
+
+ <job id="job">
+ <step name="step1" next="decision" />
- public void write(Object item) {
+ <decision id="skipCheckingDecision" decider="decider">
+ <next on="FAILED" to="step2" />
+ <next on="COMPLETED" to="step3" />
+ </step>
- delegate.write(item);
- totalAmount += ((Trade) item).getAmount();
+ <step name="step2" next="step3"/>
+ <step name="step3" />
+ </job>
- }
+ <bean id="decider" class="com.MyDecider"/>
-}
-
- The custom writer in the example is stateful (it maintains its
- total in an instance variable totalAmount), but the
- state is stored through the ItemStream interface
- in the ExecutionContext. In this way we can be
- sure that when the open() callback is received on a
- restart. The framework garuntees we always get the last value that was
- committed. It should be noted that it is not always necessary to
- implement ItemStream. For example, if the ItemWriter is re-runnable, in
- the sense that it maintains its own state in a transactional resource
- like a database, there is no need to maintain state within the writer
- itself.
+
\ No newline at end of file