diff --git a/docs/src/site/docbook/reference/job.xml b/docs/src/site/docbook/reference/job.xml
new file mode 100644
index 000000000..adbbef628
--- /dev/null
+++ b/docs/src/site/docbook/reference/job.xml
@@ -0,0 +1,1029 @@
+
+
+
+ Configuring and Executing A Job
+
+
+ Introduction
+
+ In Chapter 2, the overall architecture design was discussed, using
+ the following diagram as a guide:
+
+
+
+
+
+
+
+
+
+
+
+ When viewed from left to right, the diagram describes a basic flow
+ for the execution of a batch job:
+
+
+
+ A Scheduler kicks off a job script (usually some form of shell
+ script)
+
+
+
+ The script sets up the classpath appropriately, and starts the
+ Java process. In most cases, using
+ CommandLineJobRunner as the entry point
+
+
+
+ The JobRunner finds the Job using the
+ JobLocator, pulls together the
+ JobParameters and launches the
+ Job
+
+
+
+ The JobLauncher retrieves a
+ JobExecution from the
+ JobRepository, and executes the
+ Job
+
+
+
+ The Job executes each
+ Step.
+
+
+
+ When execution is complete, the Step
+ returns control back to the Job, and if no more
+ steps exist, control is returned back to the original caller, in this
+ case, the scheduler.
+
+
+
+ This flow is perhaps a bit overly simplified, but describes the
+ complete flow in the most basic terms. From here, each tier will be
+ described in detail, using actual implementations and examples.
+
+
+
+ JobLauncher
+
+ The most basic implementation of the
+ JobLauncher interface is the SimpleJobLauncher.
+ It's only required dependency is a JobRepository,
+ in order to obtain an execution:
+
+ <bean id="jobLauncher"
+ class="org.springframework.batch.execution.launch.SimpleJobLauncher">
+ <property name="jobRepository" ref="jobRepository" />
+ </bean>
+
+ Once a JobExecution is obtained, it is passed
+ to the execute method of Job, ultimately returning
+ the JobExecution to the caller:
+
+
+
+
+
+
+
+
+
+
+
+ The sequence is straightforward, and works well when launched from a
+ scheduler, but causes issues when trying to launch from an HTTP request.
+ In this scenario, the launching needs to be done asynchronously, so that
+ the SimpleJobLauncher returns immediately to it's
+ caller. This is because it is not good practice to keep an HTTP request
+ open for the amount of time needed by long running processes such as
+ batch. An example sequence is below:
+
+
+
+
+
+
+
+
+
+
+
+ The SimpleJobLauncher can easily be
+ configured to allow for this scenario by configuring a
+ TaskExecutor:
+
+ <bean id="jobLauncher"
+ class="org.springframework.batch.execution.launch.SimpleJobLauncher">
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="taskExecutor">
+ <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" />
+ </property>
+ </bean>
+
+ Any implementation of the spring TaskExecutor
+ interface can be used to control how jobs are asynchronously
+ executed.
+
+
+
+ JobRepository
+
+ The SimpleJobRepository is the only provided implementation of the
+ JobRepository interface. It completely manages the
+ various batch domain objects and ensures they are created and persisted
+ correctly. The SimpleJobRepository uses three
+ different DAO interfaces for the three major domain types it stores:
+ JobInstanceDao,
+ JobExecutionDao, and
+ StepExecutionDao. The repository delegates to these
+ DAOs to both persist the various domain objects and query for them during
+ initialization. The following configuration shows a SimpleJobRepository
+ configured with JDBC DAOs:
+
+ <bean id="jobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository">
+ <constructor-arg ref="jobInstanceDao" />
+ <constructor-arg ref="jobExecutionDao" />
+ <constructor-arg ref="stepExecutionDao" />
+ </bean>
+
+ <bean id="jobInstanceDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobInstanceDao" >
+ <property name="jdbcTemplate" ref="jdbcTemplate" />
+ <property name="jobIncrementer" ref="jobIncrementer" />
+ </bean>
+
+ <bean id="jobExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobExecutionDao" >
+ <property name="jdbcTemplate" ref="jdbcTemplate" />
+ <property name="jobExecutionIncrementer" ref="jobExecutionIncrementer" />
+ </bean>
+
+ <bean id="stepExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcStepExecutionDao" >
+ <property name="jdbcTemplate" ref="jdbcTemplate" />
+ <property name="stepExecutionIncrementer" ref="stepExecutionIncrementer" />
+ </bean>
+
+ <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate" >
+ <property name="dataSource" ref="dataSource" />
+ </bean>
+
+ The configuration above isn't quite complete, each DAO
+ implementation makes a reference to a Spring
+ DataFieldMaxValueIncrementer.
+ JobInstance, JobExecution,
+ and StepExecution each have unique IDs, and the
+ incrementers are used to create them.
+
+
+ JobRepositoryFactoryBean
+
+ Including the incrementers, which must be database specific, the
+ configuration above is verbose. In order to make this more manageable,
+ the framework provides a FactoryBean for
+ convenience: JobRepositoryFactoryBean.
+
+ <bean id="jobRepository"
+ class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean"
+ <property name="databaseType" value="hsql" />
+ <property name="dataSource" ref="dataSource" />
+ <property name="transactionManager" ref="transactionManager" />
+ </bean>
+
+ The databaseType property indicates the type of incrementer that
+ must be used. Options include: "db2", "db2zos", "derby", "hsql",
+ "mysql", "oracle", and "postgres".
+
+
+
+ In-Memory Repository
+
+ There are scenarios in which you may not want to persist your
+ domain objects to the database. One reason may be speed, storing domain
+ objects at each commit point takes extra time. Another reason may be
+ that you just don't need to persist status for a particular job. Spring
+ batch provides a solution:
+
+ <bean id="simpleJobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository">
+ <constructor-arg ref="mapJobInstanceDao" />
+ <constructor-arg ref="mapJobExecutionDao" />
+ <constructor-arg ref="mapStepExecutionDao" />
+ </bean>
+
+ <bean id="mapJobInstanceDao"
+ class="org.springframework.batch.core.repository.dao.MapJobInstanceDao" />
+
+ <bean id="mapJobExecutionDao"
+ class="org.springframework.batch.core.repository.dao.MapJobExecutionDao" />
+
+ <bean id="mapStepExecutionDao"
+ class="org.springframework.batch.core.repository.dao.MapStepExecutionDao" />
+
+ The Map* DAO implementations store the batch artifacts in a
+ transactional map. So, the repository and DAOs may still be used
+ normally, and are transactionally sound, but their contents will be lost
+ when the class is destroyed.
+
+ There is also a separate FactoryBean for the in-memory
+ JobRepository, which reduces the amount of
+ configuration required:
+
+ <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
+
+
+ Transaction Configuration For the JobRepository
+
+ If either of the JobRepository factory beans are used,
+ transactional advice will be automatically created around the
+ repository. This is to ensure that the batch meta data, including
+ state that is necessary for restarts after a failure, is persisted
+ correctly. The behaviour of the framework is not well defined if the
+ repository methods are not transactional. The isolation level in the
+ create* method attributes is specified separately to
+ ensure that when jobs are launched there if two processes are trying
+ to launch the same job at the same time, only one will succeed. The
+ default isolation level for that method is SERIALIZABLE, which is
+ quite aggressive: READ_COMMITTED would work just as well;
+ READ_UNCOMMITTED would be fine if two processes are not likely to
+ collide in this way. However, since a call to the
+ create* method is quite short, it is unlikely
+ that the SERIALIZED will cause problems, as long as the database
+ platform supports it. However, this can be overriden in the factory
+ beans:
+
+ <bean id="jobRepository"
+ class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean"
+ <property name="databaseType" value="hsql" />
+ <property name="dataSource" ref="dataSource" />
+ <property name="transactionManager" ref="transactionManager" />
+ <property name="IsolationLevelForCreate" value="ISOLATION_REPEATABLE_READ" />
+ </bean>
+
+ If the factory beans aren't used then it is also essential to
+ configure the transactional behaviour of the repository using
+ AOP:
+
+
+ <aop:config>
+ <aop:advisor
+ pointcut="execution(* org.springframework.batch.core..*Repository+.*(..))"
+ <advice-ref="txAdvice" />
+ </aop:config>
+
+ <tx:advice id="txAdvice" transaction-manager="transactionManager">
+ <tx:attributes>
+ <tx:method name="*" />
+ </tx:attributes>
+ </tx:advice>
+
+
+
+ This fragment can be used as is, with almost no changes.
+ Remember also to include the appropiate namespace declarations and to
+ make sure spring-tx and spring-aop (or the whole of spring) is on the
+ classpath.
+
+
+
+ Recommendations for Indexing Meta Data Tables
+
+ Spring Batch provides DDL samples for the meta-data tables in
+ the Core jar file for several common database platforms. Index
+ declarations are not included in that DDL because there are too many
+ variations in how users may want to index dependeing on their precise
+ platform, local conventions and also the business requirements of how
+ the jobs will be operated. The table below provides some indication as
+ to which columns are going to be used in a WHERE clause by the Dao
+ ipmlementations provided by Spring Batch, and how frequently they
+ might be used, so that individual projects can make up their own minds
+ about indexing.
+
+
+ Where clauses in SQL statements (exluding primary keys) and
+ their approximate frequency of use.
+
+
+
+
+ Default Table Name
+
+ Where Clause
+
+ Frequency
+
+
+
+ BATCH_JOB_INSTANCE
+
+ JOB_NAME = ? and JOB_KEY = ?
+
+ Every time a job is launched
+
+
+
+ BATCH_JOB_EXECUTION
+
+ JOB_INSTANCE_ID = ?
+
+ Every time a job is restarted
+
+
+
+ BATCH_EXECUTION_CONTEXT
+
+ EXECUTION_ID = ? and KEY_NAME = ?
+
+ On commit interval, a.k.a. chunk
+
+
+
+ BATCH_STEP_EXECUTION
+
+ VERSION = ?
+
+ On commit interval, a.k.a. chunk (and at start and end
+ of step)
+
+
+
+ BATCH_STEP_EXECUTION
+
+ STEP_NAME = ? and JOB_EXECUTION_ID = ?
+
+ Before each step execution
+
+
+
+
+
+
+
+
+
+ Job
+
+ The only current implementation of the Job
+ interface is SimpleJob. Since a
+ Job is just a simple loop through a list of Steps,
+ this implementation should be sufficient for the majority of needs. It has
+ only three required dependencies: a name,
+ JobRepository, and a list of Steps.
+
+ <bean id="footballJob"
+ class="org.springframework.batch.core.job.SimpleJob">
+ <property name="steps">
+ <list>
+ <!-- Step Bean details ommitted for clarity -->
+ <bean id="playerload" parent="simpleStep" />
+ <bean id="gameLoad" parent="simpleStep" />
+ <bean id="playerSummarization" parent="simpleStep" />
+ </list>
+ </property>
+ <property name="jobRepository" ref="jobRepository" />
+ </bean>
+
+ Each Step will be executed in sequence until
+ all have completed successfully. Any Step that fails will cause the entire
+ job to fail.
+
+
+ Restartability
+
+ One key concern when execution a batch job, is what happens when a
+ failed job is restarted? A Job is considered to have been 'restarted' if
+ the same JobInstance has more than one JobExecution. Ideally, all jobs
+ should be able to start up where they left off, but there are scenarios
+ where this is not possible. It is entirely up to
+ the developer to ensure that a new instance is always created in this
+ scenario. However, Spring Batch does provide some help. If a
+ Job should never be restarted, but should always be run as part of a new
+ JobInstance, then the restartable property may be
+ set to 'false':
+
+ <bean id="footballJob"
+ class="org.springframework.batch.core.job.SimpleJob">
+ <property name="steps">
+ <list>
+ <!-- Step Bean details ommitted for clarity -->
+ <bean id="playerload" parent="simpleStep" />
+ <bean id="gameLoad" parent="simpleStep" />
+ <bean id="playerSummarization" parent="simpleStep" />
+ </list>
+ </property>
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="restartable" value="false" />
+ </bean>
+
+ To phrase it another way, setting restartable to false means "this
+ Job does not support being started again". Restarting a Job that is not
+ restartable will cause a JobRestartException to
+ be thrown:
+
+ Job job = new SimpleJob();
+ job.setRestartable(false);
+
+ JobParameters jobParameters = new JobParameters();
+
+ JobExecution firstExecution = jobRepository.createJobExecution(job, jobParameters);
+ jobRepository.saveOrUpdate(firstExecution);
+
+ try {
+ jobRepository.createJobExecution(job, jobParameters);
+ fail();
+ }
+ catch (JobRestartException e) {
+ // expected
+ }
+
+ This snippet of JUnit code shows how attempting to create a
+ JobExecution the first time for a non restartable
+ job will cause no issues. However, the second
+ attempt will throw a JobRestartException.
+
+
+
+ Intercepting Job execution
+
+ During the course of the execution of a
+ Job, it may be useful to be notified of various
+ events in its lifecycle so that custom code may be executed. The
+ SimpleJob allows for this by calling a
+ JobListener at the appropriate time:
+
+ public interface JobListener {
+
+ void beforeJob(JobExecution jobExecution);
+
+ void afterJob(JobExecution jobExecution);
+
+ void onError(JobExecution jobExecution, Throwable e);
+
+ void onInterrupt(JobExecution jobExecution);
+ }
+
+ Listeners can be added to a SimpleJob via
+ the setJobListeners property:
+
+ <bean id="footballJob"
+ class="org.springframework.batch.core.job.SimpleJob">
+ <property name="steps">
+ <list>
+ <!-- Step Bean details ommitted for clarity -->
+ <bean id="playerload" parent="simpleStep" />
+ <bean id="gameLoad" parent="simpleStep" />
+ <bean id="playerSummarization" parent="simpleStep" />
+ </list>
+ </property>
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="jobListeners">
+ <bean class="org.springframework.batch.core.listener.JobListenerSupport" />
+ </property>
+ </bean>
+
+
+
+ JobFactory and Stateful Components in Steps
+
+ Unlike many traditional Spring applications, many of the
+ components of a batch application are stateful, the file readers and
+ writers are obvious examples. The recommended way to deal with this is
+ to create a fresh ApplicationContext for each job
+ execution. If the Job is launched from the
+ command line with CommandLineJobRunner this is
+ trivial. For more complex launching scenarios, where jobs are executed
+ in parallel or serially from the same process, some extra steps have to
+ be taken to ensure that the ApplicationContext is
+ refreshed. This is preferable to using prototype scope for the stateful
+ beans because then they would not receive lifecycle callbacks from the
+ container at the end of use. (e.g. through destroy-method in XML)
+
+ The strategy provided by Spring Batch to deal with this scenario
+ is the JobFactory, and the samples provide an
+ example of a specialized implementation that can load an
+ ApplicationContext and close it properly when the
+ job is finished. A relevant examples is
+ ClassPathXmlApplicationContextJobFactory and its
+ use in the adhoc-job-launcher-context.xml and the
+ quartz-job-launcher-context.xml, which can be found in the
+ Samples project.
+
+
+
+
+ Running a Job
+
+ Regardless of whether the originator is a Scheduler or an HTTP
+ request, a Job must be obtained, parameters must be parsed, and eventually
+ a JobLauncher called:
+
+
+
+
+
+
+
+
+
+
+
+
+ Running Jobs from the Command Line
+
+ For users that want to run their jobs from an enterprise
+ scheduler, the command line is the primary interface. This is because
+ most schedulers (with the exception of Quartz unless using the
+ NativeJob) work directly with operating system
+ processes, primarily kicked off with shell scripts. There are many ways
+ to launch a Java process besides a shell script, such as Perl, Ruby, or
+ even 'build tools' such as ant or maven. However, because most people
+ are familiar with shell scripts, this example will focus on them.
+
+
+ The CommandLineJobRunner
+
+ Because the script launching the job must kick off a Java
+ Virtual Machine, there needs to be a class with a main method to act
+ as the primary entry point. Spring Batch provides an implementation
+ that serves just this purpose:
+ CommandLineJobRunner. It's important to note
+ that this is just one way to bootstrap your application, but there are
+ many ways to launch a Java process, and this class should in no way be
+ viewed as definitive. It performs four tasks:
+
+
+
+ Loads the appropriate Application Context
+
+
+
+ Parses command line arguments into JobParameters
+
+
+
+ Locates the appropriate job based on arguments
+
+
+
+ Uses the JobLauncher provided in the application context to
+ launch the job.
+
+
+
+ All of these tasks are accomplished based completely upon the
+ arguments passed in. The following are required arguments:
+
+
+ CommandLineJobRunner arguments
+
+
+
+
+ jobPath
+
+ The location of the XML file that will be used to
+ create an ApplicationContext. This file
+ should contain everything needed to run the complete
+ Job
+
+
+
+ jobName
+
+ The name of the job to be run.
+
+
+
+
+
+ These arguments must be passed in with the path first and the
+ name second. All arguments after these are considered to be
+ JobParameters and must be in the format of 'name=value':
+
+ bash$ java CommandLineJobRunner endOfDayJob.xml endOfDay schedule.date(date)=2008/01/01
+
+ In most cases you would want to use a manifest to declare your
+ main class in a jar, but for simplicity, the class was used directly.
+ This example is using the same 'EndOfDay' example from Chapter 2. The
+ first argument is 'endOfDayJob.xml', which is the Spring
+ ApplicationContext containing the Job. The
+ second argument, 'endOfDay' represents the job name. The final
+ argument, 'schedule.date=01-01-2008' will be converted into
+ JobParameters. An example of the XML
+ configuration is below:
+
+ <bean id="endOfDay"
+ class="org.springframework.batch.core.job.SimpleJob">
+ <property name="steps">
+ <bean id="step1" parent="simpleStep" />
+ <!-- Step details removed for clarity -->
+ </property>
+ </bean>
+
+ <!-- Launcher details removed for clarity -->
+ <bean id="jobLauncher"
+ class="org.springframework.batch.core.launch.support.SimpleJobLauncher" />
+
+ This example is overly simplistic, since there are many more
+ requirements to a run a batch job in Spring Batch in general, but it
+ serves to show the two main requirements of the
+ CommandLineJobRunner:
+ Job and
+ JobLauncher
+
+
+
+ ExitCodes
+
+ When launching a batch job from the command-line, it is often
+ from an enterprise scheduler. Most schedulers are fairly dumb, and
+ work only at the process level. Meaning, they only know about some
+ operating system process such as a shell script that they're invoking.
+ In this scenario, the only way to communicate back to the scheduler
+ about the success or failure of a job is through return codes. A
+ number is returned to a scheduler that is told how to interpret the
+ result. In the simple case: 0 is success and 1 is failure. However,
+ there may be scenarios such as: If job A returns 4 kick off job B, if
+ it returns 5 kick off job C. This type of behavior is configured at
+ the scheduler level, but it is important that a processing framework
+ such as Spring Batch provide a way to return a numeric representation
+ of of the 'Exit Code' for a particular batch job. In Spring Batch this
+ is encapsulated within an ExitStatus, which is
+ covered in more detail in Chapter 5. For the purposes of discussing
+ exit codes, the only important thing to know is that an
+ ExitStatus has an exit code property that is
+ set by the framework (or the developer) and is returned as part of the
+ JobExecution returned from the
+ JobLauncher. The
+ CommandLineJobRunner converts this string value
+ to a number using the ExitCodeMapper
+ interface:
+
+ public interface ExitCodeMapper {
+
+ public int intValue(String exitCode);
+}
+
+ The essential contract of an
+ ExitCodeMapper is that, given a string exit
+ code, a number representation will be returned. The default
+ implementation used by the job runner is the SimpleJvmExitCodeMapper
+ that returns 0 for completion, 1 for generic errors, and 2 for any job
+ runner errors such as not being able to find a
+ Job in the provided context. If anything more
+ complex than the 3 values above is needed, then a custom
+ implementation of the ExitCodeMapper interface
+ must be supplied. Because the
+ CommandLineJobRunner is the class that creates
+ an ApplicationContext, and thus cannot be
+ 'wired together', any values that need to be overwritten must be
+ autowired. This means that if an implementation of
+ ExitCodeMapper is found within the BeanFactory,
+ it will be injected into the runner after the context is created. All
+ that needs to be done to provide your own
+ ExitCodeMapper is to declare the implementation
+ as a root level bean, and ensure it's part of the
+ ApplicationContext that is loaded by the
+ runner.
+
+
+
+
+ Running Jobs from within a container
+
+
+
+
+
+
+ Advanced Meta-Data Usage
+
+
+
+
+ Querying the repository
+
+
+
+
+
+
+
+ Stopping a Job
+
+ One of the most common reasons for wanting to launching a
+ job asynchronously is to be able to gracefully
+ stop it. This can be done through the
+ JobExecution returned by the
+ JobLauncher:
+
+ JobExecution jobExecution = launcher.run(getJob(), jobParameters);
+
+ //give job adequate time to start
+ Thread.sleep(1000);
+
+ assertEquals(BatchStatus.STARTED, jobExecution.getStatus());
+ assertTrue(jobExecution.isRunning());
+
+ jobExecution.stop();
+
+ //give job time to stop
+ Thread.sleep(1000);
+
+ assertEquals(BatchStatus.STOPPED, jobExecution.getStatus());
+ assertFalse(jobExecution.isRunning());
+
+ The shutdown is not immediate, since there is no way to force
+ immediate shutdown, especially if the execution is currently in
+ developer code that the framework has no control over, such as a
+ business service. What it does mean, is that as soon as control is
+ returned back to the framework, it will set the status of the current
+ StepExecution to
+ BatchStatus.STOPPED, save it, then do the same
+ for the JobExecution before finishing.
+
+
+
+
+
+
+ Job Tier
+
+ The Job Tier is responsible for the overall execution of a batch
+ job. It sequentially executes batch steps, ensuring that all steps are in
+ the correct state and all appropriate policies are enforced:
+
+
+
+
+
+
+
+
+
+
+
+ The job tier is entirely concerned with maintaining the three job
+ stereotypes: Job,
+ JobInstance, and
+ JobExecution. The
+ JobLauncher interacts with the
+ JobRepository in order to create a
+ JobExecution, and the Job
+ stores the JobExecution using the
+ repository.
+
+
+
+ Examples of Customized Business Logic
+
+
+ Some batch jobs can be assembled purely from off-the-shelf
+ components in Spring Batch, mostly the ItemReader
+ and ItemWriter implementations. Where this is not
+ possible (the majority of cases) the main API entry points for
+ application developers are the Tasklet,
+ ItemReader, ItemWriter and
+ the various listener interfaces. Most simple batch jobs will be able to
+ use off-the-shelf input from a Spring Batch
+ ItemReader, but it is very often the case that
+ there are custom concerns in the processing and writing, which normally
+ leads developers to implement an ItemWriter, or
+ ItemTransformer.
+
+ Here we provide a few examples of common patterns in custom
+ business logic, mainly using the listener interfaces . It should be
+ noted that an ItemReader or
+ ItemWriter can implement the listener interfaces
+ as well if appropriate.
+
+
+
+ Logging Item Processing and Failures
+
+ A common use case is the need for special handling of errors in a
+ step, item by item, perhaps logging to a special channel, or inserting a
+ record into a database. The StepHandlerStep
+ (created from the step factory beans) allows users to implement this use
+ case with a simple ItemReadListener, for errors
+ on read, and an ItemWriteListener, for errors on
+ write. The below code snippets illustrate a listener that logs both read
+ and write failures:
+
+ public class ItemFailureLoggerListener extends ItemListenerSupport {
+
+ private static Log logger = LogFactory.getLog("item.error");
+
+ public void onReadError(Exception ex) {
+ logger.error("Encountered error on read", e);
+ }
+
+ public void onWriteError(Exception ex, Object item) {
+ logger.error("Encountered error on write", e);
+ }
+
+}
+
+ Having implemented this listener it must be registered with the
+ step:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ ...
+ <property name="listeners">
+ <bean class="org.example...ItemFailureLoggerListener"/>
+ </property>
+ </bean>
+
+ Remember that if your listener does anything in an
+ onError() method, it will be inside a transaction that is
+ going to be rolled back. If you need to use a transactional resource
+ such as a database inside an onError() method, consider
+ adding a declarative transaction to that method (see Spring Core
+ Reference Guide for details), and giving its propagation attribute the
+ value REQUIRES_NEW.
+
+
+
+ Stopping a Job Manually for Business Reasons
+
+ Spring Batch provides a stop() method through the JobLauncher
+ interface, but this is really aimed at the operator, rather than the
+ application programmer. Sometimes it is more convenient or makes more
+ sense to stop a job execution from within the business logic.
+
+ The simplest thing to do is to throw a RuntimeException (one that
+ isn't retried indefinitely or skipped), For example, a custom exception
+ type could be used, as in the example below:
+
+ public class PoisonPillItemWriter extends AbstractItemWriter {
+
+ public void write(Object item) throws Exception {
+
+ if (isPoisonPill(item)) {
+ throw new PoisonPillException("Posion pill detected: "+item);
+ }
+
+ }
+
+}
+
+ Another simple way to stop a step from executing is to simply
+ return null from the
+ ItemReader:
+
+ public class EarlyCompletionItemReader extends AbstractItemReader {
+
+ private ItemReader delegate;
+
+ public void setDelegate(ItemReader delegate) { ... }
+
+ public Object read() throws Exception {
+
+ Object item = delegate.read();
+
+ if (isEndItem(item)) {
+ return null; // end the step here
+ }
+
+ return item;
+
+ }
+
+}
+
+ The previous example actually relies on the fact that there is a
+ default implementation of the CompletionPolicy
+ strategy which signals a complete batch when the item to be processed is
+ null. A more sophisticated completion policy could be implemented and
+ injected into the Step through the
+ RepeatOperationsStepFactoryBean:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" >
+ ...
+ <property name="chunkOperations">
+ <bean class="org.springframework.batch.repeat.support.RepeatTemplate">
+ <property name="completionPolicy">
+ <bean class="org.example...SpecialCompletionPolicy"/>
+ </property>
+ </bean>
+ </property>
+ </bean>
+
+ An alternative is to set a flag in the
+ StepExecution, which is checked by the
+ Step implementations in the framework in between
+ item processing. To implement this alternative, we need access to the
+ current StepExecution, and this can be achieved by implementing a
+ StepListener and registering it with the Step. Here is an example of a
+ listener that sets the flag:
+
+ public class CustomItemWriter extends ItemListenerSupport implements StepListener {
+
+ private StepExecution stepExecution;
+
+ public void beforeStep(StepExecution stepExecution) {
+ this.stepExecution = stepExecution;
+ }
+
+ public void afterRead(Object item) {
+
+ if (isPoisonPill(item)) {
+ stepExecution.setTerminateOnly(true);
+ }
+
+ }
+
+}
+
+ The default behaviour here when the flag is set is for the step to
+ throw a JobInterruptedException. This can be
+ controlled through the StepInterruptionPolicy,
+ but the only choice is to throw or not throw an exception, so this is
+ always an abnormal ending to a job.
+
+
+
+ Adding a Footer Record
+
+ A very common requirement is to aggregate information during the
+ output process and to append a record at the end of a file summarizing
+ the data, or providing a checksum. This can also be achieved with a
+ callbacks in the step, normally as part of a custom
+ ItemWriter. In this case, since a job is
+ accumulating state that should not be lost if the job aborts, the
+ ItemStream interface should be
+ implemented:
+
+ public class CustomItemWriter extends AbstractItemWriter implements
+ ItemStream, StepListener
+{
+
+ private static final String TOTAL_AMOUNT_KEY = "total.amount";
+
+ private ItemWriter delegate;
+
+ private double totalAmount = 0.0;
+
+ public void setDelegate(ItemWriter delegate) { ... }
+
+ public ExitStatus afterStep(StepExecution stepExecution) {
+ // Add the footer record here...
+ delegate.write("Total Amount Processed: " + totalAmount);
+ }
+
+ public void open(ExecutionContext executionContext) {
+ if (executionContext.containsKey(TOTAL_AMOUNT_KEY) {
+ totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY);
+ }
+ }
+
+ public void update(ExecutionContext executionContext) {
+ executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount);
+ }
+
+ public void write(Object item) {
+
+ delegate.write(item);
+ totalAmount += ((Trade) item).getAmount();
+
+ }
+
+}
+
+ The custom writer in the example is stateful (it maintains its
+ total in an instance variable totalAmount), but the
+ state is stored through the ItemStream interface
+ in the ExecutionContext. In this way we can be
+ sure that when the open() callback is received on a
+ restart. The framework garuntees we always get the last value that was
+ committed. It should be noted that it is not always necessary to
+ implement ItemStream. For example, if the ItemWriter is re-runnable, in
+ the sense that it maintains its own state in a transactional resource
+ like a database, there is no need to maintain state within the writer
+ itself.
+
+
+
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/step.xml b/docs/src/site/docbook/reference/step.xml
new file mode 100644
index 000000000..d3a5520d4
--- /dev/null
+++ b/docs/src/site/docbook/reference/step.xml
@@ -0,0 +1,1097 @@
+
+
+
+ Step
+
+
+ Introduction
+
+ In Chapter 2, the overall architecture design was discussed, using
+ the following diagram as a guide:
+
+
+
+
+
+
+
+
+
+
+
+ When viewed from left to right, the diagram describes a basic flow
+ for the execution of a batch job:
+
+
+
+ A Scheduler kicks off a job script (usually some form of shell
+ script)
+
+
+
+ The script sets up the classpath appropriately, and starts the
+ Java process. In most cases, using
+ CommandLineJobRunner as the entry point
+
+
+
+ The JobRunner finds the Job using the
+ JobLocator, pulls together the
+ JobParameters and launches the
+ Job
+
+
+
+ The JobLauncher retrieves a
+ JobExecution from the
+ JobRepository, and executes the
+ Job
+
+
+
+ The Job executes each
+ Step in sequence.
+
+
+
+ The Step calls read on the
+ ItemReader, handing the resulting item to the
+ ItemWriter until null is returned, periodically
+ committing and storing status in the
+ JobRepository.
+
+
+
+ When execution is complete, the Step
+ returns control back to the Job, and if no more
+ steps exist, control is returned back to the original caller, in this
+ case, the scheduler.
+
+
+
+ This flow is perhaps a bit overly simplified, but describes the
+ complete flow in the most basic terms. From here, each tier will be
+ described in detail, using actual implementations and examples.
+
+
+
+ Application Tier
+
+ The Application tier is entirely concerned with the actual
+ processing of input:
+
+
+
+
+
+
+
+
+
+
+
+
+ StepHandlerStep
+
+ The figure above shows a simple 'item-oriented' execution flow.
+ One item is read in from an ItemReader, and then
+ handed to an ItemWriter, until their are no more
+ items left. When processing first begins, a transaction is started and
+ periodically committed until the Step is
+ complete. Given these basic requirements, the
+ StepHandlerStep requires the following
+ dependencies, at a minimum:
+
+
+
+ ItemReader - The
+ ItemReader that provides items for
+ processing.
+
+
+
+ ItemWriter - The
+ ItemWriter that processes the items provided
+ by the ItemReader.
+
+
+
+ PlatformTransactionManager - Spring
+ transaction manager that will be used to begin and commit
+ transactions during processing.
+
+
+
+ JobRepository - The
+ JobRepository that will be used to
+ periodically store the StepExecution and
+ ExecutionContext during processing (just
+ before committing).
+
+
+
+
+ SimpleStepFactoryBean
+
+ Despite the relatively short list of required dependencies for
+ an StepHandlerStep, it is an extremely complex
+ class that can potentially contain many collaborators. In order to
+ ease configuration, a SimpleStepFactoryBean can
+ be used:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ <property name="transactionManager" ref="transactionManager" />
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="itemReader" ref="itemReader" />
+ <property name="itemWriter" ref="itemWriter" />
+ </bean>
+
+ The configuration above represents the only required
+ dependencies of the factory bean. Attempting to instantiate a
+ SimpleStepFactoryBean without at least those
+ four dependencies will result in an exception being thrown during
+ construction by the Spring container.
+
+
+
+ Configuring a CommitInterval
+
+ As mentioned above, the StepHandlerStep
+ reads in and writes out items, periodically commiting using the
+ supplied PlatformTransactionManager. By
+ default, it will commit after each item has been written. This is less
+ than ideal in many situations, since beginning and commiting a
+ transaction is expensive. Ideally, you would like to process as many
+ items as possible in each transaction, which is completely dependant
+ upon the type of data being processed and the resources that are being
+ interacted with. For this reason, the number of items that are
+ processed within a commit can be set as the commit interval:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ <property name="transactionManager" ref="transactionManager" />
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="itemReader" ref="itemReader" />
+ <property name="itemWriter" ref="itemWriter" />
+ <property name="commitInterval" value="10" />
+ </bean>
+
+ In this example, 10 items will be processed within each
+ transaction. At the beginning of processing a transaction is begun,
+ and each time read is called on the
+ ItemReader, a counter is incremented. When it
+ reaches 10, the transaction will be committed.
+
+
+
+ Configuring a Step for Restart
+
+ Earlier in this chapter, restarting a Job
+ was discussed. Restart has numerous impacts on steps, and as such may
+ require some specific configuration.
+
+
+ Setting a StartLimit
+
+ There are many scenarios where you may want to control the
+ number of times a Step may be started. An
+ example is a Step that may be run only once,
+ usually because it invalidates some resource that must be fixed
+ manually before it can be run again. This is configurable on the
+ step level, since different steps have different requirements. One
+ Step that may only be executed once can exist as part of the same
+ Job as Step that can
+ be run infinitely. Below is an example start limit
+ configuration:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ <property name="transactionManager" ref="transactionManager" />
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="itemReader" ref="itemReader" />
+ <property name="itemWriter" ref="itemWriter" />
+ <property name="commitInterval" value="10" />
+ <property name="startLimit" value="1" />
+ </bean>
+
+ The simple step above can be run only once. Attempting to run
+ it again will cause an exception to be thrown. It should be noted
+ that the default value for startLimit is
+ Integer.MAX_VALUE.
+
+
+
+ Restarting a completed step
+
+ In the case of a restartable job, there may be one or more
+ steps that should always be run, regardless of whether or not they
+ were successful the first time. An example might be a validation
+ step, or a step that cleans up resources before processing. During
+ normal processing of a restarted job, any step with a status of
+ 'COMPLETED', meaning it has already been completed successfully,
+ will be skipped. Setting allowStartIfComplete to true overrides this
+ so that the step will always run:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ <property name="transactionManager" ref="transactionManager" />
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="itemReader" ref="itemReader" />
+ <property name="itemWriter" ref="itemWriter" />
+ <property name="commitInterval" value="10" />
+ <property name="startLimit" value="1" />
+ <property name="allowStartIfComplete" value="true" />
+ </bean>
+
+
+
+ Step restart configuration example
+
+ <bean id="footballJob"
+ class="org.springframework.batch.core.job.SimpleJob">
+ <property name="steps">
+ <list>
+ <!-- Step Bean details ommitted for clarity -->
+ <bean id="playerload" parent="simpleStep" />
+ <bean id="gameLoad" parent="simpleStep" >
+ <property name="allowStartIfComplete" value="true" />
+ </bean>
+ <bean id="playerSummarization" parent="simpleStep" >
+ <property name="startLimit" value="2" />
+ </bean>
+ </list>
+ </property>
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="restartable" value="true" />
+ </bean>
+
+ The above example configuration is for a job that loads in
+ information about football games and summarizes them. It contains
+ three steps: playerLoad, gameLoad, and playerSummarization. The
+ playerLoad Step loads player information from
+ a flat file, while the gameLoad
+ Step does the same for games. The final
+ Step, playerSummarization, then summarizes
+ the statistics for each player based upon the provided games. It is
+ assumed that the file loaded by 'playerLoad' must be loaded only
+ once, but that 'gameLoad' will load any games found within a
+ particular directory, deleting them after they have been
+ successfully loaded into the database. As a result, the playerLoad
+ Step contains no additional configuration. It
+ can be started almost limitlessly, and if complete will be skipped.
+ The 'gameLoad' Step, however, needs to be run
+ everytime, in case extra files have been dropped since it last
+ executed, so it has 'allowStartIfComplete' set to 'true' in order to
+ always be started. (It is assumed that the database tables games are
+ loaded into has a process indicator on it, to ensure new games can
+ be properly found by the summarization step) The summarization
+ step, which is the most important in the
+ Job, is configured to have a start limit of
+ 3. This is useful in case it continually fails, a new exit code will
+ be returned to the operators that control job execution, and it
+ won't be allowed to start again until manual intervention has taken
+ place.
+
+
+ This job is purely for example purposes and is not the same
+ as the footballJob found in the samples project.
+
+
+ Run 1:
+
+
+
+ playerLoad is executed and completes successfully, adding
+ 400 players to the 'PLAYERS' table.
+
+
+
+ gameLoad is executed and processes 11 files worth of game
+ data, loading their contents into the 'GAMES' table.
+
+
+
+ playerSummarization begins processing and fails after 5
+ minutes.
+
+
+
+ Run 2:
+
+
+
+ playerLoad is not run, since it has already completed
+ succesfully, and allowStartIfComplete is false (the
+ default).
+
+
+
+ gameLoad is executed again and processes another 2 files,
+ loading their contents into the 'GAMES' table as well (with a
+ process indicator indicating they have yet to be
+ processed)
+
+
+
+ playerSummarization begins processing of all remaining
+ game data (filtering using the process indicator) and fails
+ again after 30 minutes.
+
+
+
+ Run 3:
+
+
+
+ playerLoad is not run, since it has already completed
+ succesfully, and allowStartIfComplete is false (the
+ default).
+
+
+
+ gameLoad is executed again and processes another 2 files,
+ loading their contents into the 'GAMES' table as well (with a
+ process indicator indicating they have yet to be
+ processed)
+
+
+
+ playerSummarization is not start, and the job is
+ immeadiately killed, since this is the third execution of
+ playerSummarization, and it's limit is only 2. The limit must
+ either be raised, or the Job must be
+ executed as a new JobInstance.
+
+
+
+
+
+
+ Configuring Skip Logic
+
+ There are many scenarios where errors encountered while
+ processing should not result in Step failure,
+ but should be skipped instead. This is usually a decision that must be
+ made by someone who understands the data itself and what meaning it
+ has. Financial data, for example, may not be skippable because it
+ results in money being transferred, which needs to be completely
+ accurate. Loading in a list of vendors, on the other hand, might allow
+ for skips, since a vendor not being loaded because it was formatted
+ incorrectly, or missing necessary information, won't cause issues.
+ Usually these bad records are logged as well, which will be covered
+ later when discussing listeners. Configuring skip handling requires
+ using a new factory bean:
+ SkipLimitStepFactoryBean <bean id="skipSample"
+ class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
+ <property name="skipLimit" value="10" />
+ <property name="itemReader" ref="flatFileItemReader" />
+ <property name="itemWriter" ref="itemWriter" />
+ <property name="skippableExceptionClasses"
+ value="org.springframework.batch.item.file.FlatFileParseException">
+ </property>
+ </bean>
+
+ In this example, a FlatFileItemReader is
+ used, and if at any point a FlatFileParseException is thrown, it will
+ be skipped and counted against the total skip limit of 10. It should
+ be noted that any failures encountered while reading will not count
+ against the commit interval. In other words, the commit interval is
+ only incremented on writes (regardless of success or failure).
+
+
+
+ One problem with the example above is that any other exception
+ besides a FlatFileParseException will cause the
+ Job to fail. In certain scenarios this may be
+ the correct behaviour, however, in certain scenarios it may be easier
+ to identify which exceptions should cause failure and skip everything
+ else: <bean id="skipSample"
+ class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
+ <property name="skipLimit" value="10" />
+ <property name="itemReader" ref="flatFileItemReader" />
+ <property name="itemWriter" ref="itemWriter" />
+ <property name="skippableExceptionClasses"
+ value="java.lang.Exception">
+ <property name="fatalExceptionClasses"
+ value="java.io.FileNotFoundException">
+ </property>
+ </befan>
+
+ By setting the skippable exceptions to
+ java.lang.Exception, any exception that is
+ thrown will be skipped. However, the second list,
+ 'fatalExceptionClasses', contains specific exceptions that should be
+ fatal if encountered.
+
+
+
+ Configuring Retry Logic
+
+ In most cases you want an Exception to cause either a skip or
+ Step failure. However, not all exceptions are
+ deterministic. If a FlatFileParseException is encountered while
+ reading, it will always be thrown for that record. Resseting the
+ ItemReader will not help. However, for other
+ exceptions, such as a
+ DeadlockLoserDataAccessException, which
+ indicates that the current process has attempted to update a record
+ that another process holds a lock on, waiting and trying again might
+ result in success. In this case, retry should be configured:
+
+ <bean id="step1"
+ class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
+ <property name="itemReader" ref="itemGenerator" />
+ <property name="itemWriter" ref="itemWriter" />
+ <property name="retryLimit" value="3" />
+ <property name="retryableExceptionClasses" value="org.springframework.dao.DeadlockLoserDataAccessException" />
+ </bean>
+
+ The SkipLimitStepFactoryBean requires a
+ limit for the number of times an individual item can be retried, and a
+ list of Exceptions that are 'retryable'.
+
+
+
+ Controlling rollback
+
+ By default, regardless of retry or skip, any exceptions thrown
+ from the ItemWriter will cause the transaction
+ controlled by the Step to rollback. If skip is
+ configured as described above, exceptions thrown from the
+ ItemReader will not cause a rollback. However,
+ there are many scenarios in which exceptions thrown from the
+ ItemWriter should not cause a rollback because
+ no action has taken place to invalidate the transaction. For this
+ reason, the SkipLimitStepFactoryBean can be
+ configured with a list of exceptions that should not cause
+ rollback:
+
+ <bean id="step2"
+ class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
+ <property name="commitInterval" value="2" />
+ <property name="skipLimit" value="1" />
+ <!-- No rollback for exceptions that are marked with "+" in the tx attributes -->
+ <property name="transactionAttribute"
+ value="+org.springframework.batch.item.validator.ValidationException" />
+ <property name="itemReader"
+ ref="tradeSqlItemReader" />
+ <property name="itemWriter"
+ ref="itemTrackingWriter" />
+ </bean>
+
+ The TransactionAttribute property above
+ can be used to control multiple other settings such as isolation and
+ propagation behaviour. More information on setting transaction
+ attributes can be found in the spring core documentation.
+
+
+
+ Registering ItemStreams with the Step
+
+ The step has to take care of ItemStream
+ callbacks at the necessary points in its lifecycle. This is vital if a
+ step fails, and might need to be restarted, because the
+ ItemStream interface is where the step gets the
+ information it needs about persistent state between executions. The
+ factory beans that Spring Batch provides for convenient configuration
+ of Step instances have features that allow
+ streams to be registered with the step when it is configured.
+
+ If the ItemReader or
+ ItemWriter themselves implement the ItemStream
+ interface, then these will be registered automatically. Any other
+ streams need to be registered separately. This is often the case where
+ there are indirect dependencies, like delegates being injected into
+ the reader and writer. To register these they can be injected into the
+ factory beans through the streams property, as illustrated
+ below:
+
+ <bean id="step1"
+ class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean">
+ <property name="streams" ref="fileItemReader" />
+ <property name="itemReader">
+ <bean
+ class="org.springframework.batch.item.validator.ValidatingItemReader">
+ <property name="itemReader" ref="itemReader" />
+ <property name="validator" ref="fixedValidator" />
+ </bean>
+ </property>
+ ...
+</bean>
+
+ In the example above the main item reader is being set up to
+ delegate to a bean called "fileItemReader", which itself is being
+ registered as a stream directly. The step will now be restartable and
+ the state of the reader will be correctly persisted in case of a
+ failure.
+
+
+
+ Intercepting Step Execution
+
+ Just as with the Job, there are many
+ events during the execution of a Step that a
+ user may need notification of. For example, if writing out to a flat
+ file that requires a footer, the ItemWriter
+ needs to be notified when the Step has been
+ completed, so that it can write the footer. This can be accomplished
+ with one of many Step scoped listeners.
+
+
+ StepExecutionListener
+
+ StepExecutionListener represents the
+ most generic listener for Step execution. It
+ allows for notification before a Step is
+ started, after it has completed, and if any errors are encountered
+ during processing:
+
+ public interface StepExecutionListener extends StepListener {
+
+ void beforeStep(StepExecution stepExecution);
+
+ ExitStatus onErrorInStep(StepExecution stepExecution, Throwable e);
+
+ ExitStatus afterStep(StepExecution stepExecution);
+}
+
+ ExitStatus is the return type of
+ onErrorInStep and
+ afterStep in order to allow listeners the
+ chance to modify the exit code that is returned upon completion of a
+ Step. A
+ StepExecutionListener can be applied to any
+ step factory bean via the listeners property:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ <property name="transactionManager" ref="transactionManager" />
+ <property name="jobRepository" ref="jobRepository" />
+ <property name="itemReader" ref="itemReader" />
+ <property name="itemWriter" ref="itemWriter" />
+ <property name="commitInterval" value="10" />
+ <property name="listeners" ref="stepListener" />
+ </bean>
+
+ Because all listeners extend the
+ StepListener interface, they all may be
+ applied to factory beans in the same way.
+
+
+
+ ChunkListener
+
+ A chunk is defined as the items processed within the scope of
+ a transaction. Committing a transaction commits a 'chunk'. It may be
+ useful to be nofied before and after a chunk has completed, in which
+ case the ChunkListener interface may be
+ used:
+
+ public interface ChunkListener extends StepListener {
+
+ void beforeChunk();
+
+ void afterChunk();
+ }
+
+ The beforeChunk method is called
+ after the transaction is started, but before
+ read is called on the
+ ItemReader. Conversely,
+ afterChunk is called after the last call to
+ write on the
+ ItemWriter, but before the chunk has been
+ committed.
+
+
+
+ ItemReadListener
+
+ When discussing skip logic above, it was mentioned that it may
+ be beneficial to log out skipped records, so that they can be deal
+ with later. In the case of read errors, this can be done with an
+ ItemReaderListener: public interface ItemReadListener extends StepListener {
+
+ void beforeRead();
+
+ void afterRead(Object item);
+
+ void onReadError(Exception ex);
+}
+
+ The beforeRead method will be called
+ before each call to read on the
+ ItemReader. The
+ afterRead method will be called after each
+ successful call to read, and will be passed
+ the item that was read. If there was an error while reading, the
+ onReadError method will be called. The
+ exception encounterd will be provided so that it can be
+ logged.
+
+
+
+ ItemWriteListener
+
+ Just as with the ItemReaderListener, the writing of an item
+ can be 'listened' to:
+
+ public interface ItemWriteListener extends StepListener {
+
+ void beforeWrite(Object item);
+
+ void afterWrite(Object item);
+
+ void onWriteError(Exception ex, Object item);
+}
+
+ The beforeWrite method will be called
+ before write on the
+ ItemWriter, and is handed the item that will
+ be written. The afterWrite method will be
+ called after the item has been succesfully writen. If there was an
+ error while writing, the onWriteError
+ method will be called. The exception encountered and the item that
+ was attempted to be written will be provided, so that they can be
+ logged.
+
+
+
+ SkipListener
+
+ Both ItemReadListener and
+ ItemWriteListner provide a mechanism for
+ being notified of errors, but neither one will inform you that a
+ record has actually been skipped.
+ onWriteError, for example, will be called
+ even if an item is retried and successful. For this reason, there is
+ a separate interface for tracking skipped items:
+
+
+ public interface SkipListener extends StepListener {
+
+ void onSkipInRead(Throwable t);
+
+ void onSkipInWrite(Object item, Throwable t);
+ }
+
+
+
+ onSkipInRead will be called whenever
+ an item is skipped while reading. It should be noted that rollbacks
+ may cause the same item to be registered as skipped more than once.
+ onSkipInWrite will be called when an item
+ is skipped while writing. Because the item has been read
+ successfully (and not skipped), it is also provided the item itself
+ as an argument.
+
+
+
+
+
+ TaskletStep
+
+ Item oriented processing is not the only way to process in a
+ Step. What if a Step must
+ consist as a simple storec procedure call? You could implement the call
+ as an ItemReader and return null after the
+ procedure finishes, but it is a bit unnatural since there would need to
+ be a no-op ItemWriter and lots of overhead for
+ transaction handling, listeners, etc. Spring Batch provides an
+ implementation of Step for this scenario:
+ TaskletStep. As explained in Chapter 2, the
+ Tasklet is a simple interface that has one
+ method, execute, which will be a called once
+ for the whole Step.
+ Tasklet implementors might call a stored
+ procedure, a script, or a simple SQL upate statement. Because there are
+ less concerns, there are only two required dependencies for a
+ TaskletStep: a Tasklet,
+ and a JobRepository:
+
+ <bean id="taskletStep"
+ class="org.springframework.batch.core.step.tasklet.TaskletStep" />
+ <property name="tasklet" ref="tasklet" />
+ <property name="jobRepository" ref="repository" />
+</bean>
+
+
+ TaskletStep will automatically register the tasklet as
+ StepExecutionListener if it implements this
+ interface
+
+
+
+ TaskletAdapter
+
+ As with other adapters for the ItemWriter
+ and ItemReader interfaces, the
+ Tasklet interface contains an implementation
+ that allows for adapting itself to any pre-existing class:
+ TaskletAdapter. An example where this may be
+ useful is an existing DAO that is used to upate a flag on a set of
+ records. The TaskletAdapter can be used to call
+ this class without having to write an adapter for the
+ Tasklet interface:
+
+ <bean id="deleteFilesInDir" parent="taskletStep">
+ <property name="tasklet">
+ <bean class="org.springframework.batch.core.step.tasklet.TaskletAdapter">
+ <property name="targetObject">
+ <bean class="org.mycompany.FooDao">
+ </property>
+ <property name="targetMethod" value-"updateFoo" />
+ </bean>
+ </property>
+ </bean>
+
+
+
+ Example Tasklet implementation
+
+ Many batch jobs contains steps that must be done before the main
+ processing begins in order to set up various resources, or after
+ processing has completed to cleanup those resources. In the case of a
+ job that works heavily with files, it is often necessary to delete
+ certain files locally after they have been uploaded successfully to
+ another location. The example below taken from the Spring Batch
+ samples project, is a Tasklet implementation
+ with just such a responsibility:
+
+ public class FileDeletingTasklet implements Tasklet, InitializingBean {
+
+ private Resource directory;
+
+ public ExitStatus execute() throws Exception {
+ File dir = directory.getFile();
+ Assert.state(dir.isDirectory());
+
+ File[] files = dir.listFiles();
+ for (int i = 0; i < files.length; i++) {
+ boolean deleted = files[i].delete();
+ if (!deleted) {
+ throw new UnexpectedJobExecutionException("Could not delete file " + files[i].getPath());
+ }
+ }
+ return ExitStatus.FINISHED;
+ }
+
+ public void setDirectoryResource(Resource directory) {
+ this.directory = directory;
+ }
+
+ public void afterPropertiesSet() throws Exception {
+ Assert.notNull(directory, "directory must be set");
+ }
+ }
+
+ The above Tasklet implementation will
+ delete all files within a given directory. It should be noted that the
+ execute method will only be called once. All
+ that is left is to inject the Tasklet into a
+ TaskletStep:
+
+ <bean id="taskletJob" parent="simpleJob">
+ <property name="steps">
+ <bean id="deleteFilesInDir" parent="taskletStep">
+ <property name="tasklet">
+ <bean class="org.springframework.batch.sample.tasklet.FileDeletingTasklet">
+ <property name="directoryResource" ref="directory" />
+ </bean>
+ </property>
+ </bean>
+ </property>
+ </bean>
+
+ <bean id="directory"
+ class="org.springframework.core.io.FileSystemResource">
+ <constructor-arg value="target/test-outputs/test-dir" />
+ </bean>
+
+
+
+ Executing System Commands
+
+ Many batch jobs may require that an external command be called
+ from within the batch job. Such a process could be kicked off
+ separately by the scheduler, but the advantage of common meta-data
+ about the run would be lost. Furthermore, a multi-step job would also
+ need to be split up into multiple jobs as well. Because the need is so
+ common, Spring Batch provides a Tasklet
+ implementation for calling system commands:
+
+
+ <bean class="org.springframework.batch.sample.tasklet.SystemCommandTasklet">
+ <property name="command" value="echo hello" />
+ <!-- 5 second timeout for the command to complete -->
+ <property name="timeout" value="5000" />
+ </bean>
+
+
+
+
+
+
+ Controlling Step Flow
+
+
+
+
+
+
+ Configuring for Stop
+
+
+
+
+
+ Programmatic flow decisions
+
+
+
+
+
+
+
+ Examples of Customized Business Logic
+
+
+ Some batch jobs can be assembled purely from off-the-shelf
+ components in Spring Batch, mostly the ItemReader
+ and ItemWriter implementations. Where this is not
+ possible (the majority of cases) the main API entry points for
+ application developers are the Tasklet,
+ ItemReader, ItemWriter and
+ the various listener interfaces. Most simple batch jobs will be able to
+ use off-the-shelf input from a Spring Batch
+ ItemReader, but it is very often the case that
+ there are custom concerns in the processing and writing, which normally
+ leads developers to implement an ItemWriter, or
+ ItemTransformer.
+
+ Here we provide a few examples of common patterns in custom
+ business logic, mainly using the listener interfaces . It should be
+ noted that an ItemReader or
+ ItemWriter can implement the listener interfaces
+ as well if appropriate.
+
+
+
+ Logging Item Processing and Failures
+
+ A common use case is the need for special handling of errors in a
+ step, item by item, perhaps logging to a special channel, or inserting a
+ record into a database. The StepHandlerStep
+ (created from the step factory beans) allows users to implement this use
+ case with a simple ItemReadListener, for errors
+ on read, and an ItemWriteListener, for errors on
+ write. The below code snippets illustrate a listener that logs both read
+ and write failures:
+
+ public class ItemFailureLoggerListener extends ItemListenerSupport {
+
+ private static Log logger = LogFactory.getLog("item.error");
+
+ public void onReadError(Exception ex) {
+ logger.error("Encountered error on read", e);
+ }
+
+ public void onWriteError(Exception ex, Object item) {
+ logger.error("Encountered error on write", e);
+ }
+
+}
+
+ Having implemented this listener it must be registered with the
+ step:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" >
+ ...
+ <property name="listeners">
+ <bean class="org.example...ItemFailureLoggerListener"/>
+ </property>
+ </bean>
+
+ Remember that if your listener does anything in an
+ onError() method, it will be inside a transaction that is
+ going to be rolled back. If you need to use a transactional resource
+ such as a database inside an onError() method, consider
+ adding a declarative transaction to that method (see Spring Core
+ Reference Guide for details), and giving its propagation attribute the
+ value REQUIRES_NEW.
+
+
+
+ Stopping a Job Manually for Business Reasons
+
+ Spring Batch provides a stop() method through the JobLauncher
+ interface, but this is really aimed at the operator, rather than the
+ application programmer. Sometimes it is more convenient or makes more
+ sense to stop a job execution from within the business logic.
+
+ The simplest thing to do is to throw a RuntimeException (one that
+ isn't retried indefinitely or skipped), For example, a custom exception
+ type could be used, as in the example below:
+
+ public class PoisonPillItemWriter extends AbstractItemWriter {
+
+ public void write(Object item) throws Exception {
+
+ if (isPoisonPill(item)) {
+ throw new PoisonPillException("Posion pill detected: "+item);
+ }
+
+ }
+
+}
+
+ Another simple way to stop a step from executing is to simply
+ return null from the
+ ItemReader:
+
+ public class EarlyCompletionItemReader extends AbstractItemReader {
+
+ private ItemReader delegate;
+
+ public void setDelegate(ItemReader delegate) { ... }
+
+ public Object read() throws Exception {
+
+ Object item = delegate.read();
+
+ if (isEndItem(item)) {
+ return null; // end the step here
+ }
+
+ return item;
+
+ }
+
+}
+
+ The previous example actually relies on the fact that there is a
+ default implementation of the CompletionPolicy
+ strategy which signals a complete batch when the item to be processed is
+ null. A more sophisticated completion policy could be implemented and
+ injected into the Step through the
+ RepeatOperationsStepFactoryBean:
+
+ <bean id="simpleStep"
+ class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" >
+ ...
+ <property name="chunkOperations">
+ <bean class="org.springframework.batch.repeat.support.RepeatTemplate">
+ <property name="completionPolicy">
+ <bean class="org.example...SpecialCompletionPolicy"/>
+ </property>
+ </bean>
+ </property>
+ </bean>
+
+ An alternative is to set a flag in the
+ StepExecution, which is checked by the
+ Step implementations in the framework in between
+ item processing. To implement this alternative, we need access to the
+ current StepExecution, and this can be achieved by implementing a
+ StepListener and registering it with the Step. Here is an example of a
+ listener that sets the flag:
+
+ public class CustomItemWriter extends ItemListenerSupport implements StepListener {
+
+ private StepExecution stepExecution;
+
+ public void beforeStep(StepExecution stepExecution) {
+ this.stepExecution = stepExecution;
+ }
+
+ public void afterRead(Object item) {
+
+ if (isPoisonPill(item)) {
+ stepExecution.setTerminateOnly(true);
+ }
+
+ }
+
+}
+
+ The default behaviour here when the flag is set is for the step to
+ throw a JobInterruptedException. This can be
+ controlled through the StepInterruptionPolicy,
+ but the only choice is to throw or not throw an exception, so this is
+ always an abnormal ending to a job.
+
+
+
+ Adding a Footer Record
+
+ A very common requirement is to aggregate information during the
+ output process and to append a record at the end of a file summarizing
+ the data, or providing a checksum. This can also be achieved with a
+ callbacks in the step, normally as part of a custom
+ ItemWriter. In this case, since a job is
+ accumulating state that should not be lost if the job aborts, the
+ ItemStream interface should be
+ implemented:
+
+ public class CustomItemWriter extends AbstractItemWriter implements
+ ItemStream, StepListener
+{
+
+ private static final String TOTAL_AMOUNT_KEY = "total.amount";
+
+ private ItemWriter delegate;
+
+ private double totalAmount = 0.0;
+
+ public void setDelegate(ItemWriter delegate) { ... }
+
+ public ExitStatus afterStep(StepExecution stepExecution) {
+ // Add the footer record here...
+ delegate.write("Total Amount Processed: " + totalAmount);
+ }
+
+ public void open(ExecutionContext executionContext) {
+ if (executionContext.containsKey(TOTAL_AMOUNT_KEY) {
+ totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY);
+ }
+ }
+
+ public void update(ExecutionContext executionContext) {
+ executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount);
+ }
+
+ public void write(Object item) {
+
+ delegate.write(item);
+ totalAmount += ((Trade) item).getAmount();
+
+ }
+
+}
+
+ The custom writer in the example is stateful (it maintains its
+ total in an instance variable totalAmount), but the
+ state is stored through the ItemStream interface
+ in the ExecutionContext. In this way we can be
+ sure that when the open() callback is received on a
+ restart. The framework garuntees we always get the last value that was
+ committed. It should be noted that it is not always necessary to
+ implement ItemStream. For example, if the ItemWriter is re-runnable, in
+ the sense that it maintains its own state in a transactional resource
+ like a database, there is no need to maintain state within the writer
+ itself.
+
+
+
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/whatsnew.xml b/docs/src/site/docbook/reference/whatsnew.xml
new file mode 100644
index 000000000..fc15f0f4d
--- /dev/null
+++ b/docs/src/site/docbook/reference/whatsnew.xml
@@ -0,0 +1,1103 @@
+
+
+
+ The Domain Language of Batch
+
+
+ Introduction
+
+ To any experienced batch architect, the overall concepts of batch
+ processing used in Spring Batch should be familiar and comfortable. There
+ are “Jobs” and “Steps” and developer supplied processing units called
+ ItemReaders and ItemWriters. However, because of the Spring patterns,
+ operations, templates, callbacks, and idioms, there are opportunities for
+ the following:
+
+ significant improvement in adherence to a clear separation of
+ concerns
+
+
+
+ clearly delineated architectural layers and services provided
+ as interfaces
+
+
+
+ simple and default implementations that allowed for quick
+ adoption and ease of use out-of-the-box
+
+
+
+ significantly enhanced extensibility
+
+
+
+ The diagram below is only a slight variation of the batch reference
+ architecture that has been used for decades. It provides an overview of
+ the high level components, technical services, and basic operations
+ required by a batch architecture. This architecture framework is a
+ blueprint that has been proven through decades of implementations on the
+ last several generations of platforms (COBOL/Mainframe, C++/Unix, and now
+ Java/anywhere). JCL and COBOL developers are likely to be as comfortable
+ with the concepts as C++, C# and Java developers. Spring Batch provides a
+ physical implementation of the layers, components and technical services
+ commonly found in robust, maintainable systems used to address the
+ creation of simple to complex batch applications, with the infrastructure
+ and extensions to address very complex processing needs.
+
+
+
+ Batch Application Style Interactions and Services
+
+
+
+
+
+
+
+
+
+
+ Figure 2.1: Batch Stereotypes
+
+
+ The above diagram highlights the interactions and key services
+ provided by the Spring Batch framework. The colors used are important to
+ understanding the responsibilities of a developer in Spring Batch. Grey
+ represents an external application such as an enterprise scheduler or a
+ database. It's important to note that scheduling is grey, and should thus
+ be considered separate from Spring Batch. Blue represents application
+ architecture services. In most cases these are provided by Spring Batch
+ with out of the box implementations, but an architecture team may make
+ specific implementations that better address their specific needs. Yellow
+ represents the pieces that must be configured by a developer. For example,
+ a job schedule needs to be configured so that the job is kicked off at the
+ appropriate time. A job configuration file also needs to be created, which
+ defines how a job will be run. It is also worth noting that the
+ ItemReader and ItemWriter
+ used by an application may just as easily be a custom one made by a
+ developer for their specific batch job, rather than one provided by Spring
+ Batch or an architecture team.
+
+ The Batch Application Style is organized into four logical tiers,
+ which include Run, Job, Application, and Data. The primary goal for
+ organizing an application according to the tiers is to embed what is known
+ as "separation of concerns" within the system. These tiers can be
+ conceptual but may prove effective in mapping the deployment of the
+ artifacts onto physical components like Java runtimes and integration with
+ data sources and targets. Effective separation of concerns results in
+ reducing the impact of change to the system. The four conceptual tiers
+ containing batch artifacts are:
+
+
+
+ Run Tier: The Run Tier is
+ concerned with the scheduling and launching of the application. A
+ vendor product is typically used in this tier to allow time-based
+ and interdependent scheduling of batch jobs as well as providing
+ parallel processing capabilities.
+
+
+
+ Job Tier: The Job Tier is
+ responsible for the overall execution of a batch job. It
+ sequentially executes batch steps, ensuring that all steps are in
+ the correct state and all appropriate policies are enforced.
+
+
+
+ Application Tier: The
+ Application Tier contains components required to execute the
+ program. It contains specific tasks that address required batch
+ functionality and enforces policies around execution (e.g., commit
+ intervals, capture of statistics, etc.)
+
+
+
+ Data Tier: The Data Tier
+ provides integration with the physical data sources that might
+ include databases, files, or queues.
+
+
+
+
+
+ Job Stereotypes
+
+ This section describes stereotypes relating to the concept of a
+ batch job. A Job is an entity that encapsulates an
+ entire batch process. As is common with other Spring projects, a
+ Job will be wired together via an XML configuration
+ file. This file may be referred to as the "job configuration". However,
+ Job is just the top of an overall hierarchy:
+
+
+
+
+
+
+
+
+
+
+
+
+ Job
+
+ A job is represented by a Spring bean that implements the
+ Job interface and contains all of the information
+ necessary to define the operations performed by a job. A job
+ configuration is typically contained within a Spring XML configuration
+ file and the job's name is determined by the "id" attribute associated
+ with the job configuration bean. The job configuration contains
+
+
+
+ The simple name of the job
+
+
+
+ Definition and ordering of Steps
+
+
+
+ Whether or not the job is restartable
+
+
+
+ A default simple implementation of the Job
+ interface is provided by Spring Batch in the form of the
+ SimpleJob class which creates some standard
+ functionality on top of Job, namely a standard
+ execution logic that all jobs should utilize. In general, all jobs
+ should be defined using a bean of type
+ SimpleJob:
+
+ <bean id="footballJob"
+ class="org.springframework.batch.core.job.SimpleJob">
+ <property name="steps">
+ <list>
+ <!-- Step Bean details ommitted for clarity -->
+ <bean id="playerload" parent="simpleStep" />
+ <bean id="gameLoad" parent="simpleStep" />
+ <bean id="playerSummarization" parent="simpleStep" />
+ </list>
+ </property>
+ <property name="restartable" value="true" />
+ </bean>
+
+
+
+ JobInstance
+
+ A JobInstance refers to the concept of a
+ logical job run. Let's consider a batch job that should be run once at
+ the end of the day, such as the 'EndOfDay' job from the diagram above.
+ There is one 'EndOfDay' Job, but each individual
+ run of the Job must be tracked separately. In the
+ case of this job, there will be one logical
+ JobInstance per day. For example, there will be a
+ January 1st run, and a January 2nd run. If the January 1st run fails the
+ first time and is run again the next day, it's still the January 1st
+ run. (Usually this corresponds with the data its processing as well,
+ meaning the January 1st run processes data for January 1st, etc) That is
+ to say, each JobInstance can have multiple
+ executions. (JobExecution is discussed in more
+ detail below) and only one JobInstance
+ corresponding to a particular Job can be running
+ at a given time. The definition of a JobInstance
+ has absolutely no bearing on the data the will be loaded. It is entirely
+ up to the ItemReader implementation used to
+ determine how data will be loaded. For example, in the EndOfDay
+ scenario, there may be a column on the data that indicates the
+ 'effective date' or 'schedule date' to which the data belongs. So, the
+ January 1st run would only load data from the 1st, and the January 2nd
+ run would only use data from the 2nd. Because this determination will
+ likely be a business decision, it is left up to the
+ ItemReader to decide. What using the same
+ JobInstance will determine, however, is whether
+ or not the 'state' (i.e. the ExecutionContext, which is discussed below)
+ from previous executions will be used. Using a new
+ JobInstance will mean 'start from the beginning'
+ and using an existing instance will generally mean 'start from where you
+ left off'.
+
+
+
+ JobParameters
+
+ Having discussed JobInstance and how it
+ differs from Job, the natural question to ask is:
+ "how is one JobInstance distinguished from
+ another?" The answer is: JobParameters.
+ JobParameters are any set of parameters used to
+ start a batch job, which can be used for identification or even as
+ reference data during the run. In the example above, where there are two
+ instances, one for January 1st, and another for January 2nd, there is
+ really only one Job, one that was started with a job parameter of
+ 01-01-2008 and another that was started with a parameter of 01-02-2008.
+ Thus, the contract can be defined as: JobInstance
+ = Job + JobParameters.
+ This allows a developer to effectively control how you a
+ JobInstance is defined, since they control what
+ parameters are passed in.
+
+
+
+ JobExecution
+
+ A JobExecution refers to the technical
+ concept of a single attempt to run a Job. An
+ execution may end in failure or success, but the
+ JobInstance corresponding to a given execution
+ will not be considered complete unless the execution completes
+ successfully. Using the EndOfDay Job described
+ above as an example, consider a JobInstance for 01-01-2008 that failed
+ the first time it was run. If it is ran again, with the same job
+ parameters as the first run (01-01-2008), a new JobExecution will be
+ created. However, there will still be only one
+ JobInstance.
+
+ A Job defines what a job is and how it is
+ to be executed, and JobInstance is a purely
+ organizational object to group executions together, primarily to enable
+ correct restart semantics. A JobExecution,
+ however, is the primary storage mechanism for what actually happened
+ during a run, and as such contains many more properties that must be
+ controlled and persisted:
+
+
+ JobExecution properties
+
+
+
+
+ status
+
+ A BatchStatus object that
+ indicates the status of the execution. While it's running, it's
+ BatchStatus.STARTED, if it fails it's BatchStatus.FAILED, and if
+ it finishes successfully it's BatchStatus.COMPLETED
+
+
+
+ startTime
+
+ A java.util.Date representing the
+ current system time when the execution was started.
+
+
+
+ endTime
+
+ A java.util.Date representing the
+ current system time when the execution finished, regardless of
+ whether or not it was successful.
+
+
+
+ exitStatus
+
+ The ExitStatus indicating the
+ result of the run. It is most important because it contains an
+ exit code that will be returned to the caller. See chapter 5 for
+ more details.
+
+
+
+ createTime
+
+ A java.util.Date representing the
+ current system time when the JobExecution was first persisted.
+ The job may not have been started yet (and thus has no start
+ time), but it will always have a createTime, which is required
+ by the framework for managing job level
+ ExecutionContexts.
+
+
+
+ lastUpdated
+
+ A java.util.Date representing the
+ last time a JobExecution was persisted.
+
+
+
+ executionContext
+
+ The 'property bag' containing any user data that needs to
+ be persisted between executions.
+
+
+
+ failureExceptions
+
+ The list of exceptions encountered during the execution
+ of a Job. These can be useful if more
+ than one exception is encountered during the failure of a
+ Job.
+
+
+
+
+
+ These properties are important because they will be persisted and
+ can be used to completely determine the status of an execution. For
+ example, if the EndOfDay job for 01-01 is executed at 9:00 PM, and fails
+ at 9:30, the following entries will be made in the batch meta data
+ tables:
+
+
+ BATCH_JOB_INSTANCE
+
+
+
+
+ JOB_INSTANCE_ID
+
+ JOB_NAME
+
+
+
+ 1
+
+ EndOfDayJob
+
+
+
+
+
+
+ BATCH_JOB_PARAMS
+
+
+
+
+ JOB_INSTANCE_ID
+
+ TYPE_CD
+
+ KEY_NAME
+
+ DATE_VAL
+
+
+
+ 1
+
+ DATE
+
+ schedule.Date
+
+ 2008-01-01 00:00:00
+
+
+
+
+
+
+ BATCH_JOB_EXECUTION
+
+
+
+
+ JOB_EXECUTION_ID
+
+ JOB_INSTANCE_ID
+
+ START_TIME
+
+ END_TIME
+
+ STATUS
+
+
+
+ 1
+
+ 1
+
+ 2008-01-01 21:00:23.571
+
+ 2008-01-01 21:30:17.132
+
+ FAILED
+
+
+
+
+
+
+ extra columns in the tables have been removed for added
+ clarity.
+
+
+ Now that the job has failed, let's assume that it took the entire
+ course of the night for the problem to be determined, so that the 'batch
+ window' is now closed. Assuming the window starts at 9:00 PM, the job
+ will be kicked off again for 01-01, starting where it left off and
+ completing successfully at 9:30. Because it's now the next day, the
+ 01-02 job must be run as well, which is kicked off just afterwards at
+ 9:31, and completes in it's normal one hour time at 10:30. There is no
+ requirement that one JobInstance be kicked off
+ after another, unless there is potential for the two jobs to attempt to
+ access the same data, causing issues with locking at the database level.
+ It is entirely up to the scheduler to determine when a
+ Job should be run. Since they're separate
+ JobInstances, Spring Batch will make no attempt to stop them from being
+ run concurrently. (Attempting to run the same
+ JobInstance while another is already running will
+ result in a JobExecutionAlreadyRunningException
+ being thrown) There should now be an extra entry in both the
+ JobInstance and
+ JobParameters tables, and two extra entries in
+ the JobExecution table:
+
+
+ BATCH_JOB_INSTANCE
+
+
+
+
+ JOB_INSTANCE_ID
+
+ JOB_NAME
+
+
+
+ 1
+
+ EndOfDayJob
+
+
+
+ 2
+
+ EndOfDayJob
+
+
+
+
+
+
+ BATCH_JOB_PARAMS
+
+
+
+
+ JOB_INSTANCE_ID
+
+ TYPE_CD
+
+ KEY_NAME
+
+ DATE_VAL
+
+
+
+ 1
+
+ DATE
+
+ schedule.Date
+
+ 2008-01-01 00:00:00
+
+
+
+ 2
+
+ DATE
+
+ schedule.Date
+
+ 2008-01-02 00:00:00
+
+
+
+
+
+
+ BATCH_JOB_EXECUTION
+
+
+
+
+ JOB_EXECUTION_ID
+
+ JOB_INSTANCE_ID
+
+ START_TIME
+
+ END_TIME
+
+ STATUS
+
+
+
+ 1
+
+ 1
+
+ 2008-01-01 21:00
+
+ 2008-01-01 21:30
+
+ FAILED
+
+
+
+ 2
+
+ 1
+
+ 2008-01-02 21:00
+
+ 2008-01-02 21:30
+
+ COMPLETED
+
+
+
+ 3
+
+ 2
+
+ 2008-01-02 21:31
+
+ 2008-01-02 22:29
+
+ COMPLETED
+
+
+
+
+
+
+
+
+ Step Stereotypes
+
+ A Step is a domain object that encapsulates
+ an independent, sequential phase of a batch job. Therefore, every
+ Job is composed entirely of one or more steps. A
+ Step should be thought of as a unique processing
+ stream that will be executed in sequence. For example, if you have one
+ step that loads a file into a database, another that reads from the
+ database, validates the data, preforms processing, and then writes to
+ another table, and another that reads from that table and writes out to a
+ file. Each of these steps will be performed completely before moving on to
+ the next step. The file will be completely read into the database before
+ step 2 can begin. As with Job, a
+ Step has an individual
+ StepExecution that corresponds with a unique
+ JobExecution:
+
+
+
+
+
+
+
+
+
+
+
+
+ Step
+
+ A Step contains all of the information
+ necessary to define and control the actual batch processing. This is a
+ necessarily vague description because the contents of any given
+ Step are at the discretion of the developer
+ writing a Job. A Step can be as simple or complex
+ as the developer desires. A simple Step might
+ load data from a file into the database, requiring little or no code.
+ (depending upon the implementations used) A more complex
+ Step may have complicated business rules that are
+ applied as part of the processing.
+
+ Steps are defined by instantiating implementations of the
+ Step interface. Two step implementation classes
+ are available in the Spring Batch framework, and they are each discussed
+ in detail in Chatper 4 of this guide. For most situations, the
+ StepHandlerStep implementation is sufficient, but
+ for situations where only one call is needed, such as a stored procedure
+ call or a wrapper around existing script, a
+ TaskletStep may be a better option.
+
+
+
+ StepExecution
+
+ A StepExecution represents a single attempt
+ to execute a Step. A new
+ StepExecution will be created each time a Step is
+ run, similar to JobExecution. However, if a step
+ fails to execute because the step before it fails, there will be no
+ execution persisted for it. A StepExecution will
+ only be created when it's Step is actually
+ started.
+
+ Step executions are represented by objects of the
+ StepExecution class. Each execution contains a
+ reference to its corresponding step and
+ JobExecution, and transaction related data such
+ as commit and rollback count and start and end times. Additionally, each
+ step execution will contain an ExecutionContext,
+ which contains any data a developer needs persisted across batch runs,
+ such as statistics or state information needed to restart. The following
+ is a listing of the properties for
+ StepExecution:
+
+
+ StepExecution properties
+
+
+
+
+ status
+
+ A BatchStatus object that
+ indicates the status of the execution. While it's running, the
+ status is BatchStatus.STARTED, if it fails the status is
+ BatchStatus.FAILED, and if it finishes successfully the status
+ is BatchStatus.COMPLETED
+
+
+
+ startTime
+
+ A java.util.Date representing the
+ current system time when the execution was started.
+
+
+
+ endTime
+
+ A java.util.Date representing the
+ current system time when the execution finished, regardless of
+ whether or not it was successful.
+
+
+
+ exitStatus
+
+ The ExitStatus indicating the
+ result of the execution. It is most important because it
+ contains an exit code that will be returned to the caller. See
+ chapter 5 for more details.
+
+
+
+ executionContext
+
+ The 'property bag' containing any user data that needs to
+ be persisted between executions.
+
+
+
+ readCount
+
+ The number of items that have been successfully
+ read
+
+
+
+ writeCount
+
+ The number of items that have been successfully
+ written
+
+
+
+ commitCount
+
+ The number transactions that have been committed for this
+ execution
+
+
+
+ rollbackCount
+
+ The number of times the business transaction controlled
+ by the Step has been rolled back.
+
+
+
+ readSkipCount
+
+ The number of times read has
+ failed, resulting in a skipped item.
+
+
+
+ processSkipCount
+
+ The number of times process has failed, resulting in a
+ skipped item.
+
+
+
+ filterCount
+
+ The number of items that have been 'filtered' by the
+ ItemProcessor
+
+
+
+ writeSkipCount
+
+ The number of times write has
+ failed, resulting in a skipped item.
+
+
+
+
+
+
+
+ ExecutionContext
+
+ An ExecutionContext represents a collection
+ of key/value pairs that are persisted and controlled by the framework in
+ order to allow developers a place to store persistent state that is
+ scoped to a StepExecution or
+ JobExecution. For those familiar with Quartz, it
+ is very similar to JobDataMap. The best usage
+ example is restart. Using flat file input as an example, while
+ processing individual lines, the framework periodically persists the
+ ExecutionContext at commit points. This allows
+ the ItemReader to store its state in case a fatal
+ error occurs during the run, or even if the power goes out. All that is
+ needed is to put the current number of lines read into the context, and
+ the framework will do the rest:
+
+ executionContext.putLong(getKey(LINES_READ_COUNT), reader.getPosition());
+
+ Using the EndOfDay example from the Job Stereotypes section as an
+ example, assume there's one step: 'loadData', that loads a file into the
+ database. After the first failed run, the meta data tables would look
+ like the following:
+
+
+ BATCH_JOB_INSTANCE
+
+
+
+
+ JOB_INSTANCE_ID
+
+ JOB_NAME
+
+
+
+ 1
+
+ EndOfDayJob
+
+
+
+
+ BATCH_JOB_PARAMS
+
+
+
+
+ JOB_INSTANCE_ID
+
+ TYPE_CD
+
+ KEY_NAME
+
+ DATE_VAL
+
+
+
+ 1
+
+ DATE
+
+ schedule.Date
+
+ 2008-01-01 00:00:00
+
+
+
+
+ BATCH_JOB_EXECUTION
+
+
+
+
+ JOB_EXECUTION_ID
+
+ JOB_INSTANCE_ID
+
+ START_TIME
+
+ END_TIME
+
+ STATUS
+
+
+
+ 1
+
+ 1
+
+ 2008-01-01 21:00:23.571
+
+ 2008-01-01 21:30:17.132
+
+ FAILED
+
+
+
+
+ BATCH_STEP_EXECUTION
+
+
+
+
+ STEP_EXECUTION_ID
+
+ JOB_EXECUTION_ID
+
+ STEP_NAME
+
+ START_TIME
+
+ END_TIME
+
+ STATUS
+
+
+
+ 1
+
+ 1
+
+ loadDate
+
+ 2008-01-01 21:00:23.571
+
+ 2008-01-01 21:30:17.132
+
+ FAILED
+
+
+
+
+ BATCH_EXECUTION_CONTEXT
+
+
+
+
+ EXECUTION_ID
+
+ TYPE_CD
+
+ KEY_NAME
+
+ LONG_VAL
+
+
+
+ 1
+
+ LONG
+
+ piece.count
+
+ 40321
+
+
+
+
In this case, the Step ran for 30
+ minutes and processed 40,321 'pieces', which would represent lines in a
+ file in this scenario. This value will be updated just before each
+ commit by the framework, and can contain multiple rows corresponding to
+ entries within the ExecutionContext. Being
+ notified before a commit requires one of the various StepListeners, or
+ an ItemStream, which are discussed in more detail
+ later in this guide. As with the previous example, it is assumed that
+ the Job is restarted the next day. When it is restarted, the values from
+ the ExecutionContext of the last run are
+ reconstituted from the database, and when the
+ ItemReader is opened, it can check to see if it
+ has any stored state in the context, and initialize itself from
+ there:
+
+ if (executionContext.containsKey(getKey(LINES_READ_COUNT))) {
+ log.debug("Initializing for restart. Restart data is: " + executionContext);
+
+ long lineCount = executionContext.getLong(getKey(LINES_READ_COUNT));
+
+ LineReader reader = getReader();
+
+ Object record = "";
+ while (reader.getPosition() < lineCount && record != null) {
+ record = readLine();
+ }
+ }
+
+ In this case, after the above code is executed, the current line
+ will be 40,322, allowing the Step to start again
+ from where it left off. The ExecutionContext can
+ also be used for statistics that need to be persisted about the run
+ itself. For example, if a flat file contains orders for processing that
+ exist across multiple lines, it may be necessary to store how many
+ orders have been processed (which is much different from than the number
+ of lines read) so that an email can be sent at the end of the
+ Step with the total orders processed in the body.
+ The framework handles storing this for the developer, in order to
+ correctly scope it with an individual
+ JobInstance. It can be very difficult to know
+ whether an existing ExecutionContext should be
+ used or not. For example, using the 'EndOfDay' example from above, when
+ the 01-01 run starts again for the second time, the framework recognizes
+ that it is the same JobInstance and on an
+ individual Step basis, pulls the
+ ExecutionContext out of the database and hands it
+ as part of the StepExecution to the
+ Step itself. Conversely, for the 01-02 run the
+ framework recognizes that it is a different instance, so an empty
+ context must be handed to the Step. There are
+ many of these types of determinations that the framework makes for the
+ developer to ensure the state is given to them at the correct time. It
+ is also important to note that exactly one
+ ExecutionContext exists per
+ StepExecution at any given time. Clients of the
+ ExecutionContext should be careful because this
+ creates a shared keyspace, so care should be taken when putting values
+ in to ensure no data is overwritten, however, the
+ Step stores absolutely no data in the context, so
+ there is no way to adversely affect the framework.
+
+
+
+
+ JobRepository
+
+ JobRepository is the persistence mechanism
+ for all of the Stereotypes mentioned above. When a job is first launched,
+ a JobExecution is obtained by calling the
+ repository's createJobExecution method, and
+ during the course of execution, StepExecution and
+ JobExecution are persisted by passing them to the
+ repository:
+
+ public interface JobRepository {
+
+ public JobExecution createJobExecution(Job job, JobParameters jobParameters)
+ throws JobExecutionAlreadyRunningException, JobRestartException;
+
+ void add(StepExecution stepExecution);
+
+ void update(JobExecution jobExecution);
+
+ void update(StepExecution stepExecution);
+
+ void updateExecutionContext(StepExecution stepExecution);
+
+ StepExecution getLastStepExecution(JobInstance jobInstance, Step step);
+
+ int getStepExecutionCount(JobInstance jobInstance, Step step);
+
+}
+
+
+
+
+ JobLauncher
+
+ JobLauncher represents a simple interface for
+ launching a Job with a given set of
+ JobParameters:
+
+ public interface JobLauncher {
+
+ public JobExecution run(Job job, JobParameters jobParameters) throws JobExecutionAlreadyRunningException,
+ JobRestartException;
+}
+
+
+ It is expected that implementations will obtain a valid
+ JobExecution from the
+ JobRepository and execute the
+ Job.
+
+
+
+ JobLocator
+
+ JobLocator represents an interface for
+ locating a Job:
+
+ public interface JobLocator {
+
+ Job getJob(String name) throws NoSuchJobException;
+ }
+
+ This interface is very necessary due to the nature of Spring itself.
+ Because it can't be guaranteed that one
+ ApplicationContext equals one
+ Job, an abstraction is needed to obtain a
+ Job for a given name. It becomes especially useful
+ when launching jobs from within a Java EE application server.
+
+
+
+ Item Reader
+
+ ItemReader is an abstraction that represents
+ the retrieval of input for a Step, one item at a
+ time. When the ItemReader has exhausted the items
+ it can provide, it will indicate this by returning null. More details
+ about the ItemReader interface and its various
+ implementations can be found in Chapter 3.
+
+
+
+ Item Writer
+
+ ItemWriter is an abstraction that represents
+ the output of a Step, one item at a time.
+ Generally, an item writer has no knowledge of the input it will receive
+ next, only the item that was passed in its current invocation. More
+ details about the ItemWriter interface and it's
+ various implementations can be found in Chapter 3.
+
+
+
+ Item Processor
+
+ ItemProcessor is an abstraction that
+ represents the business processing of an item. While the
+ ItemReader reads one item, and the
+ ItemWriter writes them, the
+ ItemProcessor provides access to transform or apply
+ other business processing. If while processing the item it's determined
+ that it's not valid, returning null indicates that it should not be
+ written out.
+
+
+
+ Tasklet
+
+ A Tasklet represents the execution of a
+ logical unit of work, as defined by its implementation of the Spring Batch
+ provided Tasklet interface. A
+ Tasklet is useful for encapsulating processing
+ logic that is not natural to split into read-(transform)-write phases,
+ such as invoking a system command or a stored procedure.
+
+
\ No newline at end of file