From d8c4eaf3a1163fa1d6d93308c3f7937cb9188c72 Mon Sep 17 00:00:00 2001 From: lucasward Date: Tue, 20 Jan 2009 19:48:55 +0000 Subject: [PATCH] BATCH-674:Upgrade reference documentation. Execution has been split into job and step each in their own chapters, along with a new chapter detailing what's new in 2.0. They are currently placeholders, and aren't referenced from the index yet. --- docs/src/site/docbook/reference/job.xml | 1029 ++++++++++++++++ docs/src/site/docbook/reference/step.xml | 1097 +++++++++++++++++ docs/src/site/docbook/reference/whatsnew.xml | 1103 ++++++++++++++++++ 3 files changed, 3229 insertions(+) create mode 100644 docs/src/site/docbook/reference/job.xml create mode 100644 docs/src/site/docbook/reference/step.xml create mode 100644 docs/src/site/docbook/reference/whatsnew.xml diff --git a/docs/src/site/docbook/reference/job.xml b/docs/src/site/docbook/reference/job.xml new file mode 100644 index 000000000..adbbef628 --- /dev/null +++ b/docs/src/site/docbook/reference/job.xml @@ -0,0 +1,1029 @@ + + + + Configuring and Executing A Job + +
+ Introduction + + In Chapter 2, the overall architecture design was discussed, using + the following diagram as a guide: + + + + + + + + + + + + When viewed from left to right, the diagram describes a basic flow + for the execution of a batch job: + + + + A Scheduler kicks off a job script (usually some form of shell + script) + + + + The script sets up the classpath appropriately, and starts the + Java process. In most cases, using + CommandLineJobRunner as the entry point + + + + The JobRunner finds the Job using the + JobLocator, pulls together the + JobParameters and launches the + Job + + + + The JobLauncher retrieves a + JobExecution from the + JobRepository, and executes the + Job + + + + The Job executes each + Step. + + + + When execution is complete, the Step + returns control back to the Job, and if no more + steps exist, control is returned back to the original caller, in this + case, the scheduler. + + + + This flow is perhaps a bit overly simplified, but describes the + complete flow in the most basic terms. From here, each tier will be + described in detail, using actual implementations and examples. +
+ +
+ JobLauncher + + The most basic implementation of the + JobLauncher interface is the SimpleJobLauncher. + It's only required dependency is a JobRepository, + in order to obtain an execution: + + <bean id="jobLauncher" + class="org.springframework.batch.execution.launch.SimpleJobLauncher"> + <property name="jobRepository" ref="jobRepository" /> + </bean> + + Once a JobExecution is obtained, it is passed + to the execute method of Job, ultimately returning + the JobExecution to the caller: + + + + + + + + + + + + The sequence is straightforward, and works well when launched from a + scheduler, but causes issues when trying to launch from an HTTP request. + In this scenario, the launching needs to be done asynchronously, so that + the SimpleJobLauncher returns immediately to it's + caller. This is because it is not good practice to keep an HTTP request + open for the amount of time needed by long running processes such as + batch. An example sequence is below: + + + + + + + + + + + + The SimpleJobLauncher can easily be + configured to allow for this scenario by configuring a + TaskExecutor: + + <bean id="jobLauncher" + class="org.springframework.batch.execution.launch.SimpleJobLauncher"> + <property name="jobRepository" ref="jobRepository" /> + <property name="taskExecutor"> + <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" /> + </property> + </bean> + + Any implementation of the spring TaskExecutor + interface can be used to control how jobs are asynchronously + executed. +
+ +
+ JobRepository + + The SimpleJobRepository is the only provided implementation of the + JobRepository interface. It completely manages the + various batch domain objects and ensures they are created and persisted + correctly. The SimpleJobRepository uses three + different DAO interfaces for the three major domain types it stores: + JobInstanceDao, + JobExecutionDao, and + StepExecutionDao. The repository delegates to these + DAOs to both persist the various domain objects and query for them during + initialization. The following configuration shows a SimpleJobRepository + configured with JDBC DAOs: + + <bean id="jobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository"> + <constructor-arg ref="jobInstanceDao" /> + <constructor-arg ref="jobExecutionDao" /> + <constructor-arg ref="stepExecutionDao" /> + </bean> + + <bean id="jobInstanceDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobInstanceDao" > + <property name="jdbcTemplate" ref="jdbcTemplate" /> + <property name="jobIncrementer" ref="jobIncrementer" /> + </bean> + + <bean id="jobExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobExecutionDao" > + <property name="jdbcTemplate" ref="jdbcTemplate" /> + <property name="jobExecutionIncrementer" ref="jobExecutionIncrementer" /> + </bean> + + <bean id="stepExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcStepExecutionDao" > + <property name="jdbcTemplate" ref="jdbcTemplate" /> + <property name="stepExecutionIncrementer" ref="stepExecutionIncrementer" /> + </bean> + + <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate" > + <property name="dataSource" ref="dataSource" /> + </bean> + + The configuration above isn't quite complete, each DAO + implementation makes a reference to a Spring + DataFieldMaxValueIncrementer. + JobInstance, JobExecution, + and StepExecution each have unique IDs, and the + incrementers are used to create them. + +
+ JobRepositoryFactoryBean + + Including the incrementers, which must be database specific, the + configuration above is verbose. In order to make this more manageable, + the framework provides a FactoryBean for + convenience: JobRepositoryFactoryBean. + + <bean id="jobRepository" + class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean" + <property name="databaseType" value="hsql" /> + <property name="dataSource" ref="dataSource" /> + <property name="transactionManager" ref="transactionManager" /> + </bean> + + The databaseType property indicates the type of incrementer that + must be used. Options include: "db2", "db2zos", "derby", "hsql", + "mysql", "oracle", and "postgres". +
+ +
+ In-Memory Repository + + There are scenarios in which you may not want to persist your + domain objects to the database. One reason may be speed, storing domain + objects at each commit point takes extra time. Another reason may be + that you just don't need to persist status for a particular job. Spring + batch provides a solution: + + <bean id="simpleJobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository"> + <constructor-arg ref="mapJobInstanceDao" /> + <constructor-arg ref="mapJobExecutionDao" /> + <constructor-arg ref="mapStepExecutionDao" /> + </bean> + + <bean id="mapJobInstanceDao" + class="org.springframework.batch.core.repository.dao.MapJobInstanceDao" /> + + <bean id="mapJobExecutionDao" + class="org.springframework.batch.core.repository.dao.MapJobExecutionDao" /> + + <bean id="mapStepExecutionDao" + class="org.springframework.batch.core.repository.dao.MapStepExecutionDao" /> + + The Map* DAO implementations store the batch artifacts in a + transactional map. So, the repository and DAOs may still be used + normally, and are transactionally sound, but their contents will be lost + when the class is destroyed. + + There is also a separate FactoryBean for the in-memory + JobRepository, which reduces the amount of + configuration required: + + <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" /> + +
+ Transaction Configuration For the JobRepository + + If either of the JobRepository factory beans are used, + transactional advice will be automatically created around the + repository. This is to ensure that the batch meta data, including + state that is necessary for restarts after a failure, is persisted + correctly. The behaviour of the framework is not well defined if the + repository methods are not transactional. The isolation level in the + create* method attributes is specified separately to + ensure that when jobs are launched there if two processes are trying + to launch the same job at the same time, only one will succeed. The + default isolation level for that method is SERIALIZABLE, which is + quite aggressive: READ_COMMITTED would work just as well; + READ_UNCOMMITTED would be fine if two processes are not likely to + collide in this way. However, since a call to the + create* method is quite short, it is unlikely + that the SERIALIZED will cause problems, as long as the database + platform supports it. However, this can be overriden in the factory + beans: + + <bean id="jobRepository" + class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean" + <property name="databaseType" value="hsql" /> + <property name="dataSource" ref="dataSource" /> + <property name="transactionManager" ref="transactionManager" /> + <property name="IsolationLevelForCreate" value="ISOLATION_REPEATABLE_READ" /> + </bean> + + If the factory beans aren't used then it is also essential to + configure the transactional behaviour of the repository using + AOP: + + + <aop:config> + <aop:advisor + pointcut="execution(* org.springframework.batch.core..*Repository+.*(..))" + <advice-ref="txAdvice" /> + </aop:config> + + <tx:advice id="txAdvice" transaction-manager="transactionManager"> + <tx:attributes> + <tx:method name="*" /> + </tx:attributes> + </tx:advice> + + + + This fragment can be used as is, with almost no changes. + Remember also to include the appropiate namespace declarations and to + make sure spring-tx and spring-aop (or the whole of spring) is on the + classpath. +
+ +
+ Recommendations for Indexing Meta Data Tables + + Spring Batch provides DDL samples for the meta-data tables in + the Core jar file for several common database platforms. Index + declarations are not included in that DDL because there are too many + variations in how users may want to index dependeing on their precise + platform, local conventions and also the business requirements of how + the jobs will be operated. The table below provides some indication as + to which columns are going to be used in a WHERE clause by the Dao + ipmlementations provided by Spring Batch, and how frequently they + might be used, so that individual projects can make up their own minds + about indexing. + + + Where clauses in SQL statements (exluding primary keys) and + their approximate frequency of use. + + + + + Default Table Name + + Where Clause + + Frequency + + + + BATCH_JOB_INSTANCE + + JOB_NAME = ? and JOB_KEY = ? + + Every time a job is launched + + + + BATCH_JOB_EXECUTION + + JOB_INSTANCE_ID = ? + + Every time a job is restarted + + + + BATCH_EXECUTION_CONTEXT + + EXECUTION_ID = ? and KEY_NAME = ? + + On commit interval, a.k.a. chunk + + + + BATCH_STEP_EXECUTION + + VERSION = ? + + On commit interval, a.k.a. chunk (and at start and end + of step) + + + + BATCH_STEP_EXECUTION + + STEP_NAME = ? and JOB_EXECUTION_ID = ? + + Before each step execution + + + +
+
+
+
+ +
+ Job + + The only current implementation of the Job + interface is SimpleJob. Since a + Job is just a simple loop through a list of Steps, + this implementation should be sufficient for the majority of needs. It has + only three required dependencies: a name, + JobRepository, and a list of Steps. + + <bean id="footballJob" + class="org.springframework.batch.core.job.SimpleJob"> + <property name="steps"> + <list> + <!-- Step Bean details ommitted for clarity --> + <bean id="playerload" parent="simpleStep" /> + <bean id="gameLoad" parent="simpleStep" /> + <bean id="playerSummarization" parent="simpleStep" /> + </list> + </property> + <property name="jobRepository" ref="jobRepository" /> + </bean> + + Each Step will be executed in sequence until + all have completed successfully. Any Step that fails will cause the entire + job to fail. + +
+ Restartability + + One key concern when execution a batch job, is what happens when a + failed job is restarted? A Job is considered to have been 'restarted' if + the same JobInstance has more than one JobExecution. Ideally, all jobs + should be able to start up where they left off, but there are scenarios + where this is not possible. It is entirely up to + the developer to ensure that a new instance is always created in this + scenario. However, Spring Batch does provide some help. If a + Job should never be restarted, but should always be run as part of a new + JobInstance, then the restartable property may be + set to 'false': + + <bean id="footballJob" + class="org.springframework.batch.core.job.SimpleJob"> + <property name="steps"> + <list> + <!-- Step Bean details ommitted for clarity --> + <bean id="playerload" parent="simpleStep" /> + <bean id="gameLoad" parent="simpleStep" /> + <bean id="playerSummarization" parent="simpleStep" /> + </list> + </property> + <property name="jobRepository" ref="jobRepository" /> + <property name="restartable" value="false" /> + </bean> + + To phrase it another way, setting restartable to false means "this + Job does not support being started again". Restarting a Job that is not + restartable will cause a JobRestartException to + be thrown: + + Job job = new SimpleJob(); + job.setRestartable(false); + + JobParameters jobParameters = new JobParameters(); + + JobExecution firstExecution = jobRepository.createJobExecution(job, jobParameters); + jobRepository.saveOrUpdate(firstExecution); + + try { + jobRepository.createJobExecution(job, jobParameters); + fail(); + } + catch (JobRestartException e) { + // expected + } + + This snippet of JUnit code shows how attempting to create a + JobExecution the first time for a non restartable + job will cause no issues. However, the second + attempt will throw a JobRestartException. +
+ +
+ Intercepting Job execution + + During the course of the execution of a + Job, it may be useful to be notified of various + events in its lifecycle so that custom code may be executed. The + SimpleJob allows for this by calling a + JobListener at the appropriate time: + + public interface JobListener { + + void beforeJob(JobExecution jobExecution); + + void afterJob(JobExecution jobExecution); + + void onError(JobExecution jobExecution, Throwable e); + + void onInterrupt(JobExecution jobExecution); + } + + Listeners can be added to a SimpleJob via + the setJobListeners property: + + <bean id="footballJob" + class="org.springframework.batch.core.job.SimpleJob"> + <property name="steps"> + <list> + <!-- Step Bean details ommitted for clarity --> + <bean id="playerload" parent="simpleStep" /> + <bean id="gameLoad" parent="simpleStep" /> + <bean id="playerSummarization" parent="simpleStep" /> + </list> + </property> + <property name="jobRepository" ref="jobRepository" /> + <property name="jobListeners"> + <bean class="org.springframework.batch.core.listener.JobListenerSupport" /> + </property> + </bean> +
+ +
+ JobFactory and Stateful Components in Steps + + Unlike many traditional Spring applications, many of the + components of a batch application are stateful, the file readers and + writers are obvious examples. The recommended way to deal with this is + to create a fresh ApplicationContext for each job + execution. If the Job is launched from the + command line with CommandLineJobRunner this is + trivial. For more complex launching scenarios, where jobs are executed + in parallel or serially from the same process, some extra steps have to + be taken to ensure that the ApplicationContext is + refreshed. This is preferable to using prototype scope for the stateful + beans because then they would not receive lifecycle callbacks from the + container at the end of use. (e.g. through destroy-method in XML) + + The strategy provided by Spring Batch to deal with this scenario + is the JobFactory, and the samples provide an + example of a specialized implementation that can load an + ApplicationContext and close it properly when the + job is finished. A relevant examples is + ClassPathXmlApplicationContextJobFactory and its + use in the adhoc-job-launcher-context.xml and the + quartz-job-launcher-context.xml, which can be found in the + Samples project. +
+
+ +
+ Running a Job + + Regardless of whether the originator is a Scheduler or an HTTP + request, a Job must be obtained, parameters must be parsed, and eventually + a JobLauncher called: + + + + + + + + + + + +
+ Running Jobs from the Command Line + + For users that want to run their jobs from an enterprise + scheduler, the command line is the primary interface. This is because + most schedulers (with the exception of Quartz unless using the + NativeJob) work directly with operating system + processes, primarily kicked off with shell scripts. There are many ways + to launch a Java process besides a shell script, such as Perl, Ruby, or + even 'build tools' such as ant or maven. However, because most people + are familiar with shell scripts, this example will focus on them. + +
+ The CommandLineJobRunner + + Because the script launching the job must kick off a Java + Virtual Machine, there needs to be a class with a main method to act + as the primary entry point. Spring Batch provides an implementation + that serves just this purpose: + CommandLineJobRunner. It's important to note + that this is just one way to bootstrap your application, but there are + many ways to launch a Java process, and this class should in no way be + viewed as definitive. It performs four tasks: + + + + Loads the appropriate Application Context + + + + Parses command line arguments into JobParameters + + + + Locates the appropriate job based on arguments + + + + Uses the JobLauncher provided in the application context to + launch the job. + + + + All of these tasks are accomplished based completely upon the + arguments passed in. The following are required arguments: + + + CommandLineJobRunner arguments + + + + + jobPath + + The location of the XML file that will be used to + create an ApplicationContext. This file + should contain everything needed to run the complete + Job + + + + jobName + + The name of the job to be run. + + + +
+ + These arguments must be passed in with the path first and the + name second. All arguments after these are considered to be + JobParameters and must be in the format of 'name=value': + + bash$ java CommandLineJobRunner endOfDayJob.xml endOfDay schedule.date(date)=2008/01/01 + + In most cases you would want to use a manifest to declare your + main class in a jar, but for simplicity, the class was used directly. + This example is using the same 'EndOfDay' example from Chapter 2. The + first argument is 'endOfDayJob.xml', which is the Spring + ApplicationContext containing the Job. The + second argument, 'endOfDay' represents the job name. The final + argument, 'schedule.date=01-01-2008' will be converted into + JobParameters. An example of the XML + configuration is below: + + <bean id="endOfDay" + class="org.springframework.batch.core.job.SimpleJob"> + <property name="steps"> + <bean id="step1" parent="simpleStep" /> + <!-- Step details removed for clarity --> + </property> + </bean> + + <!-- Launcher details removed for clarity --> + <bean id="jobLauncher" + class="org.springframework.batch.core.launch.support.SimpleJobLauncher" /> + + This example is overly simplistic, since there are many more + requirements to a run a batch job in Spring Batch in general, but it + serves to show the two main requirements of the + CommandLineJobRunner: + Job and + JobLauncher +
+ +
+ ExitCodes + + When launching a batch job from the command-line, it is often + from an enterprise scheduler. Most schedulers are fairly dumb, and + work only at the process level. Meaning, they only know about some + operating system process such as a shell script that they're invoking. + In this scenario, the only way to communicate back to the scheduler + about the success or failure of a job is through return codes. A + number is returned to a scheduler that is told how to interpret the + result. In the simple case: 0 is success and 1 is failure. However, + there may be scenarios such as: If job A returns 4 kick off job B, if + it returns 5 kick off job C. This type of behavior is configured at + the scheduler level, but it is important that a processing framework + such as Spring Batch provide a way to return a numeric representation + of of the 'Exit Code' for a particular batch job. In Spring Batch this + is encapsulated within an ExitStatus, which is + covered in more detail in Chapter 5. For the purposes of discussing + exit codes, the only important thing to know is that an + ExitStatus has an exit code property that is + set by the framework (or the developer) and is returned as part of the + JobExecution returned from the + JobLauncher. The + CommandLineJobRunner converts this string value + to a number using the ExitCodeMapper + interface: + + public interface ExitCodeMapper { + + public int intValue(String exitCode); +} + + The essential contract of an + ExitCodeMapper is that, given a string exit + code, a number representation will be returned. The default + implementation used by the job runner is the SimpleJvmExitCodeMapper + that returns 0 for completion, 1 for generic errors, and 2 for any job + runner errors such as not being able to find a + Job in the provided context. If anything more + complex than the 3 values above is needed, then a custom + implementation of the ExitCodeMapper interface + must be supplied. Because the + CommandLineJobRunner is the class that creates + an ApplicationContext, and thus cannot be + 'wired together', any values that need to be overwritten must be + autowired. This means that if an implementation of + ExitCodeMapper is found within the BeanFactory, + it will be injected into the runner after the context is created. All + that needs to be done to provide your own + ExitCodeMapper is to declare the implementation + as a root level bean, and ensure it's part of the + ApplicationContext that is loaded by the + runner. +
+
+ +
+ Running Jobs from within a container + + +
+
+ +
+ Advanced Meta-Data Usage + + + +
+ Querying the repository + + + +
+ JobExporer + + +
+
+ +
+ Stopping a Job + + One of the most common reasons for wanting to launching a + job asynchronously is to be able to gracefully + stop it. This can be done through the + JobExecution returned by the + JobLauncher: + + JobExecution jobExecution = launcher.run(getJob(), jobParameters); + + //give job adequate time to start + Thread.sleep(1000); + + assertEquals(BatchStatus.STARTED, jobExecution.getStatus()); + assertTrue(jobExecution.isRunning()); + + jobExecution.stop(); + + //give job time to stop + Thread.sleep(1000); + + assertEquals(BatchStatus.STOPPED, jobExecution.getStatus()); + assertFalse(jobExecution.isRunning()); + + The shutdown is not immediate, since there is no way to force + immediate shutdown, especially if the execution is currently in + developer code that the framework has no control over, such as a + business service. What it does mean, is that as soon as control is + returned back to the framework, it will set the status of the current + StepExecution to + BatchStatus.STOPPED, save it, then do the same + for the JobExecution before finishing. + +
+ JobOperator + + +
+
+
+ +
+ Job Tier + + The Job Tier is responsible for the overall execution of a batch + job. It sequentially executes batch steps, ensuring that all steps are in + the correct state and all appropriate policies are enforced: + + + + + + + + + + + + The job tier is entirely concerned with maintaining the three job + stereotypes: Job, + JobInstance, and + JobExecution. The + JobLauncher interacts with the + JobRepository in order to create a + JobExecution, and the Job + stores the JobExecution using the + repository. +
+ +
+ Examples of Customized Business Logic + +
+ Some batch jobs can be assembled purely from off-the-shelf + components in Spring Batch, mostly the ItemReader + and ItemWriter implementations. Where this is not + possible (the majority of cases) the main API entry points for + application developers are the Tasklet, + ItemReader, ItemWriter and + the various listener interfaces. Most simple batch jobs will be able to + use off-the-shelf input from a Spring Batch + ItemReader, but it is very often the case that + there are custom concerns in the processing and writing, which normally + leads developers to implement an ItemWriter, or + ItemTransformer. + + Here we provide a few examples of common patterns in custom + business logic, mainly using the listener interfaces . It should be + noted that an ItemReader or + ItemWriter can implement the listener interfaces + as well if appropriate. +
+ +
+ Logging Item Processing and Failures + + A common use case is the need for special handling of errors in a + step, item by item, perhaps logging to a special channel, or inserting a + record into a database. The StepHandlerStep + (created from the step factory beans) allows users to implement this use + case with a simple ItemReadListener, for errors + on read, and an ItemWriteListener, for errors on + write. The below code snippets illustrate a listener that logs both read + and write failures: + + public class ItemFailureLoggerListener extends ItemListenerSupport { + + private static Log logger = LogFactory.getLog("item.error"); + + public void onReadError(Exception ex) { + logger.error("Encountered error on read", e); + } + + public void onWriteError(Exception ex, Object item) { + logger.error("Encountered error on write", e); + } + +} + + Having implemented this listener it must be registered with the + step: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > + ... + <property name="listeners"> + <bean class="org.example...ItemFailureLoggerListener"/> + </property> + </bean> + + Remember that if your listener does anything in an + onError() method, it will be inside a transaction that is + going to be rolled back. If you need to use a transactional resource + such as a database inside an onError() method, consider + adding a declarative transaction to that method (see Spring Core + Reference Guide for details), and giving its propagation attribute the + value REQUIRES_NEW. +
+ +
+ Stopping a Job Manually for Business Reasons + + Spring Batch provides a stop() method through the JobLauncher + interface, but this is really aimed at the operator, rather than the + application programmer. Sometimes it is more convenient or makes more + sense to stop a job execution from within the business logic. + + The simplest thing to do is to throw a RuntimeException (one that + isn't retried indefinitely or skipped), For example, a custom exception + type could be used, as in the example below: + + public class PoisonPillItemWriter extends AbstractItemWriter { + + public void write(Object item) throws Exception { + + if (isPoisonPill(item)) { + throw new PoisonPillException("Posion pill detected: "+item); + } + + } + +} + + Another simple way to stop a step from executing is to simply + return null from the + ItemReader: + + public class EarlyCompletionItemReader extends AbstractItemReader { + + private ItemReader delegate; + + public void setDelegate(ItemReader delegate) { ... } + + public Object read() throws Exception { + + Object item = delegate.read(); + + if (isEndItem(item)) { + return null; // end the step here + } + + return item; + + } + +} + + The previous example actually relies on the fact that there is a + default implementation of the CompletionPolicy + strategy which signals a complete batch when the item to be processed is + null. A more sophisticated completion policy could be implemented and + injected into the Step through the + RepeatOperationsStepFactoryBean: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" > + ... + <property name="chunkOperations"> + <bean class="org.springframework.batch.repeat.support.RepeatTemplate"> + <property name="completionPolicy"> + <bean class="org.example...SpecialCompletionPolicy"/> + </property> + </bean> + </property> + </bean> + + An alternative is to set a flag in the + StepExecution, which is checked by the + Step implementations in the framework in between + item processing. To implement this alternative, we need access to the + current StepExecution, and this can be achieved by implementing a + StepListener and registering it with the Step. Here is an example of a + listener that sets the flag: + + public class CustomItemWriter extends ItemListenerSupport implements StepListener { + + private StepExecution stepExecution; + + public void beforeStep(StepExecution stepExecution) { + this.stepExecution = stepExecution; + } + + public void afterRead(Object item) { + + if (isPoisonPill(item)) { + stepExecution.setTerminateOnly(true); + } + + } + +} + + The default behaviour here when the flag is set is for the step to + throw a JobInterruptedException. This can be + controlled through the StepInterruptionPolicy, + but the only choice is to throw or not throw an exception, so this is + always an abnormal ending to a job. +
+ +
+ Adding a Footer Record + + A very common requirement is to aggregate information during the + output process and to append a record at the end of a file summarizing + the data, or providing a checksum. This can also be achieved with a + callbacks in the step, normally as part of a custom + ItemWriter. In this case, since a job is + accumulating state that should not be lost if the job aborts, the + ItemStream interface should be + implemented: + + public class CustomItemWriter extends AbstractItemWriter implements + ItemStream, StepListener +{ + + private static final String TOTAL_AMOUNT_KEY = "total.amount"; + + private ItemWriter delegate; + + private double totalAmount = 0.0; + + public void setDelegate(ItemWriter delegate) { ... } + + public ExitStatus afterStep(StepExecution stepExecution) { + // Add the footer record here... + delegate.write("Total Amount Processed: " + totalAmount); + } + + public void open(ExecutionContext executionContext) { + if (executionContext.containsKey(TOTAL_AMOUNT_KEY) { + totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY); + } + } + + public void update(ExecutionContext executionContext) { + executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount); + } + + public void write(Object item) { + + delegate.write(item); + totalAmount += ((Trade) item).getAmount(); + + } + +} + + The custom writer in the example is stateful (it maintains its + total in an instance variable totalAmount), but the + state is stored through the ItemStream interface + in the ExecutionContext. In this way we can be + sure that when the open() callback is received on a + restart. The framework garuntees we always get the last value that was + committed. It should be noted that it is not always necessary to + implement ItemStream. For example, if the ItemWriter is re-runnable, in + the sense that it maintains its own state in a transactional resource + like a database, there is no need to maintain state within the writer + itself. +
+
+ \ No newline at end of file diff --git a/docs/src/site/docbook/reference/step.xml b/docs/src/site/docbook/reference/step.xml new file mode 100644 index 000000000..d3a5520d4 --- /dev/null +++ b/docs/src/site/docbook/reference/step.xml @@ -0,0 +1,1097 @@ + + + + Step + +
+ Introduction + + In Chapter 2, the overall architecture design was discussed, using + the following diagram as a guide: + + + + + + + + + + + + When viewed from left to right, the diagram describes a basic flow + for the execution of a batch job: + + + + A Scheduler kicks off a job script (usually some form of shell + script) + + + + The script sets up the classpath appropriately, and starts the + Java process. In most cases, using + CommandLineJobRunner as the entry point + + + + The JobRunner finds the Job using the + JobLocator, pulls together the + JobParameters and launches the + Job + + + + The JobLauncher retrieves a + JobExecution from the + JobRepository, and executes the + Job + + + + The Job executes each + Step in sequence. + + + + The Step calls read on the + ItemReader, handing the resulting item to the + ItemWriter until null is returned, periodically + committing and storing status in the + JobRepository. + + + + When execution is complete, the Step + returns control back to the Job, and if no more + steps exist, control is returned back to the original caller, in this + case, the scheduler. + + + + This flow is perhaps a bit overly simplified, but describes the + complete flow in the most basic terms. From here, each tier will be + described in detail, using actual implementations and examples. +
+ +
+ Application Tier + + The Application tier is entirely concerned with the actual + processing of input: + + + + + + + + + + + +
+ StepHandlerStep + + The figure above shows a simple 'item-oriented' execution flow. + One item is read in from an ItemReader, and then + handed to an ItemWriter, until their are no more + items left. When processing first begins, a transaction is started and + periodically committed until the Step is + complete. Given these basic requirements, the + StepHandlerStep requires the following + dependencies, at a minimum: + + + + ItemReader - The + ItemReader that provides items for + processing. + + + + ItemWriter - The + ItemWriter that processes the items provided + by the ItemReader. + + + + PlatformTransactionManager - Spring + transaction manager that will be used to begin and commit + transactions during processing. + + + + JobRepository - The + JobRepository that will be used to + periodically store the StepExecution and + ExecutionContext during processing (just + before committing). + + + +
+ SimpleStepFactoryBean + + Despite the relatively short list of required dependencies for + an StepHandlerStep, it is an extremely complex + class that can potentially contain many collaborators. In order to + ease configuration, a SimpleStepFactoryBean can + be used: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > + <property name="transactionManager" ref="transactionManager" /> + <property name="jobRepository" ref="jobRepository" /> + <property name="itemReader" ref="itemReader" /> + <property name="itemWriter" ref="itemWriter" /> + </bean> + + The configuration above represents the only required + dependencies of the factory bean. Attempting to instantiate a + SimpleStepFactoryBean without at least those + four dependencies will result in an exception being thrown during + construction by the Spring container. +
+ +
+ Configuring a CommitInterval + + As mentioned above, the StepHandlerStep + reads in and writes out items, periodically commiting using the + supplied PlatformTransactionManager. By + default, it will commit after each item has been written. This is less + than ideal in many situations, since beginning and commiting a + transaction is expensive. Ideally, you would like to process as many + items as possible in each transaction, which is completely dependant + upon the type of data being processed and the resources that are being + interacted with. For this reason, the number of items that are + processed within a commit can be set as the commit interval: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > + <property name="transactionManager" ref="transactionManager" /> + <property name="jobRepository" ref="jobRepository" /> + <property name="itemReader" ref="itemReader" /> + <property name="itemWriter" ref="itemWriter" /> + <property name="commitInterval" value="10" /> + </bean> + + In this example, 10 items will be processed within each + transaction. At the beginning of processing a transaction is begun, + and each time read is called on the + ItemReader, a counter is incremented. When it + reaches 10, the transaction will be committed. +
+ +
+ Configuring a Step for Restart + + Earlier in this chapter, restarting a Job + was discussed. Restart has numerous impacts on steps, and as such may + require some specific configuration. + +
+ Setting a StartLimit + + There are many scenarios where you may want to control the + number of times a Step may be started. An + example is a Step that may be run only once, + usually because it invalidates some resource that must be fixed + manually before it can be run again. This is configurable on the + step level, since different steps have different requirements. One + Step that may only be executed once can exist as part of the same + Job as Step that can + be run infinitely. Below is an example start limit + configuration: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > + <property name="transactionManager" ref="transactionManager" /> + <property name="jobRepository" ref="jobRepository" /> + <property name="itemReader" ref="itemReader" /> + <property name="itemWriter" ref="itemWriter" /> + <property name="commitInterval" value="10" /> + <property name="startLimit" value="1" /> + </bean> + + The simple step above can be run only once. Attempting to run + it again will cause an exception to be thrown. It should be noted + that the default value for startLimit is + Integer.MAX_VALUE. +
+ +
+ Restarting a completed step + + In the case of a restartable job, there may be one or more + steps that should always be run, regardless of whether or not they + were successful the first time. An example might be a validation + step, or a step that cleans up resources before processing. During + normal processing of a restarted job, any step with a status of + 'COMPLETED', meaning it has already been completed successfully, + will be skipped. Setting allowStartIfComplete to true overrides this + so that the step will always run: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > + <property name="transactionManager" ref="transactionManager" /> + <property name="jobRepository" ref="jobRepository" /> + <property name="itemReader" ref="itemReader" /> + <property name="itemWriter" ref="itemWriter" /> + <property name="commitInterval" value="10" /> + <property name="startLimit" value="1" /> + <property name="allowStartIfComplete" value="true" /> + </bean> +
+ +
+ Step restart configuration example + + <bean id="footballJob" + class="org.springframework.batch.core.job.SimpleJob"> + <property name="steps"> + <list> + <!-- Step Bean details ommitted for clarity --> + <bean id="playerload" parent="simpleStep" /> + <bean id="gameLoad" parent="simpleStep" > + <property name="allowStartIfComplete" value="true" /> + </bean> + <bean id="playerSummarization" parent="simpleStep" > + <property name="startLimit" value="2" /> + </bean> + </list> + </property> + <property name="jobRepository" ref="jobRepository" /> + <property name="restartable" value="true" /> + </bean> + + The above example configuration is for a job that loads in + information about football games and summarizes them. It contains + three steps: playerLoad, gameLoad, and playerSummarization. The + playerLoad Step loads player information from + a flat file, while the gameLoad + Step does the same for games. The final + Step, playerSummarization, then summarizes + the statistics for each player based upon the provided games. It is + assumed that the file loaded by 'playerLoad' must be loaded only + once, but that 'gameLoad' will load any games found within a + particular directory, deleting them after they have been + successfully loaded into the database. As a result, the playerLoad + Step contains no additional configuration. It + can be started almost limitlessly, and if complete will be skipped. + The 'gameLoad' Step, however, needs to be run + everytime, in case extra files have been dropped since it last + executed, so it has 'allowStartIfComplete' set to 'true' in order to + always be started. (It is assumed that the database tables games are + loaded into has a process indicator on it, to ensure new games can + be properly found by the summarization step) The summarization + step, which is the most important in the + Job, is configured to have a start limit of + 3. This is useful in case it continually fails, a new exit code will + be returned to the operators that control job execution, and it + won't be allowed to start again until manual intervention has taken + place. + + + This job is purely for example purposes and is not the same + as the footballJob found in the samples project. + + + Run 1: + + + + playerLoad is executed and completes successfully, adding + 400 players to the 'PLAYERS' table. + + + + gameLoad is executed and processes 11 files worth of game + data, loading their contents into the 'GAMES' table. + + + + playerSummarization begins processing and fails after 5 + minutes. + + + + Run 2: + + + + playerLoad is not run, since it has already completed + succesfully, and allowStartIfComplete is false (the + default). + + + + gameLoad is executed again and processes another 2 files, + loading their contents into the 'GAMES' table as well (with a + process indicator indicating they have yet to be + processed) + + + + playerSummarization begins processing of all remaining + game data (filtering using the process indicator) and fails + again after 30 minutes. + + + + Run 3: + + + + playerLoad is not run, since it has already completed + succesfully, and allowStartIfComplete is false (the + default). + + + + gameLoad is executed again and processes another 2 files, + loading their contents into the 'GAMES' table as well (with a + process indicator indicating they have yet to be + processed) + + + + playerSummarization is not start, and the job is + immeadiately killed, since this is the third execution of + playerSummarization, and it's limit is only 2. The limit must + either be raised, or the Job must be + executed as a new JobInstance. + + +
+
+ +
+ Configuring Skip Logic + + There are many scenarios where errors encountered while + processing should not result in Step failure, + but should be skipped instead. This is usually a decision that must be + made by someone who understands the data itself and what meaning it + has. Financial data, for example, may not be skippable because it + results in money being transferred, which needs to be completely + accurate. Loading in a list of vendors, on the other hand, might allow + for skips, since a vendor not being loaded because it was formatted + incorrectly, or missing necessary information, won't cause issues. + Usually these bad records are logged as well, which will be covered + later when discussing listeners. Configuring skip handling requires + using a new factory bean: + SkipLimitStepFactoryBean <bean id="skipSample" + class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> + <property name="skipLimit" value="10" /> + <property name="itemReader" ref="flatFileItemReader" /> + <property name="itemWriter" ref="itemWriter" /> + <property name="skippableExceptionClasses" + value="org.springframework.batch.item.file.FlatFileParseException"> + </property> + </bean> + + In this example, a FlatFileItemReader is + used, and if at any point a FlatFileParseException is thrown, it will + be skipped and counted against the total skip limit of 10. It should + be noted that any failures encountered while reading will not count + against the commit interval. In other words, the commit interval is + only incremented on writes (regardless of success or failure). +
+ +
+ One problem with the example above is that any other exception + besides a FlatFileParseException will cause the + Job to fail. In certain scenarios this may be + the correct behaviour, however, in certain scenarios it may be easier + to identify which exceptions should cause failure and skip everything + else: <bean id="skipSample" + class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> + <property name="skipLimit" value="10" /> + <property name="itemReader" ref="flatFileItemReader" /> + <property name="itemWriter" ref="itemWriter" /> + <property name="skippableExceptionClasses" + value="java.lang.Exception"> + <property name="fatalExceptionClasses" + value="java.io.FileNotFoundException"> + </property> + </befan> + + By setting the skippable exceptions to + java.lang.Exception, any exception that is + thrown will be skipped. However, the second list, + 'fatalExceptionClasses', contains specific exceptions that should be + fatal if encountered. +
+ +
+ Configuring Retry Logic + + In most cases you want an Exception to cause either a skip or + Step failure. However, not all exceptions are + deterministic. If a FlatFileParseException is encountered while + reading, it will always be thrown for that record. Resseting the + ItemReader will not help. However, for other + exceptions, such as a + DeadlockLoserDataAccessException, which + indicates that the current process has attempted to update a record + that another process holds a lock on, waiting and trying again might + result in success. In this case, retry should be configured: + + <bean id="step1" + class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> + <property name="itemReader" ref="itemGenerator" /> + <property name="itemWriter" ref="itemWriter" /> + <property name="retryLimit" value="3" /> + <property name="retryableExceptionClasses" value="org.springframework.dao.DeadlockLoserDataAccessException" /> + </bean> + + The SkipLimitStepFactoryBean requires a + limit for the number of times an individual item can be retried, and a + list of Exceptions that are 'retryable'. +
+ +
+ Controlling rollback + + By default, regardless of retry or skip, any exceptions thrown + from the ItemWriter will cause the transaction + controlled by the Step to rollback. If skip is + configured as described above, exceptions thrown from the + ItemReader will not cause a rollback. However, + there are many scenarios in which exceptions thrown from the + ItemWriter should not cause a rollback because + no action has taken place to invalidate the transaction. For this + reason, the SkipLimitStepFactoryBean can be + configured with a list of exceptions that should not cause + rollback: + + <bean id="step2" + class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> + <property name="commitInterval" value="2" /> + <property name="skipLimit" value="1" /> + <!-- No rollback for exceptions that are marked with "+" in the tx attributes --> + <property name="transactionAttribute" + value="+org.springframework.batch.item.validator.ValidationException" /> + <property name="itemReader" + ref="tradeSqlItemReader" /> + <property name="itemWriter" + ref="itemTrackingWriter" /> + </bean> + + The TransactionAttribute property above + can be used to control multiple other settings such as isolation and + propagation behaviour. More information on setting transaction + attributes can be found in the spring core documentation. +
+ +
+ Registering ItemStreams with the Step + + The step has to take care of ItemStream + callbacks at the necessary points in its lifecycle. This is vital if a + step fails, and might need to be restarted, because the + ItemStream interface is where the step gets the + information it needs about persistent state between executions. The + factory beans that Spring Batch provides for convenient configuration + of Step instances have features that allow + streams to be registered with the step when it is configured. + + If the ItemReader or + ItemWriter themselves implement the ItemStream + interface, then these will be registered automatically. Any other + streams need to be registered separately. This is often the case where + there are indirect dependencies, like delegates being injected into + the reader and writer. To register these they can be injected into the + factory beans through the streams property, as illustrated + below: + + <bean id="step1" + class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> + <property name="streams" ref="fileItemReader" /> + <property name="itemReader"> + <bean + class="org.springframework.batch.item.validator.ValidatingItemReader"> + <property name="itemReader" ref="itemReader" /> + <property name="validator" ref="fixedValidator" /> + </bean> + </property> + ... +</bean> + + In the example above the main item reader is being set up to + delegate to a bean called "fileItemReader", which itself is being + registered as a stream directly. The step will now be restartable and + the state of the reader will be correctly persisted in case of a + failure. +
+ +
+ Intercepting Step Execution + + Just as with the Job, there are many + events during the execution of a Step that a + user may need notification of. For example, if writing out to a flat + file that requires a footer, the ItemWriter + needs to be notified when the Step has been + completed, so that it can write the footer. This can be accomplished + with one of many Step scoped listeners. + +
+ StepExecutionListener + + StepExecutionListener represents the + most generic listener for Step execution. It + allows for notification before a Step is + started, after it has completed, and if any errors are encountered + during processing: + + public interface StepExecutionListener extends StepListener { + + void beforeStep(StepExecution stepExecution); + + ExitStatus onErrorInStep(StepExecution stepExecution, Throwable e); + + ExitStatus afterStep(StepExecution stepExecution); +} + + ExitStatus is the return type of + onErrorInStep and + afterStep in order to allow listeners the + chance to modify the exit code that is returned upon completion of a + Step. A + StepExecutionListener can be applied to any + step factory bean via the listeners property: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > + <property name="transactionManager" ref="transactionManager" /> + <property name="jobRepository" ref="jobRepository" /> + <property name="itemReader" ref="itemReader" /> + <property name="itemWriter" ref="itemWriter" /> + <property name="commitInterval" value="10" /> + <property name="listeners" ref="stepListener" /> + </bean> + + Because all listeners extend the + StepListener interface, they all may be + applied to factory beans in the same way. +
+ +
+ ChunkListener + + A chunk is defined as the items processed within the scope of + a transaction. Committing a transaction commits a 'chunk'. It may be + useful to be nofied before and after a chunk has completed, in which + case the ChunkListener interface may be + used: + + public interface ChunkListener extends StepListener { + + void beforeChunk(); + + void afterChunk(); + } + + The beforeChunk method is called + after the transaction is started, but before + read is called on the + ItemReader. Conversely, + afterChunk is called after the last call to + write on the + ItemWriter, but before the chunk has been + committed. +
+ +
+ ItemReadListener + + When discussing skip logic above, it was mentioned that it may + be beneficial to log out skipped records, so that they can be deal + with later. In the case of read errors, this can be done with an + ItemReaderListener: public interface ItemReadListener extends StepListener { + + void beforeRead(); + + void afterRead(Object item); + + void onReadError(Exception ex); +} + + The beforeRead method will be called + before each call to read on the + ItemReader. The + afterRead method will be called after each + successful call to read, and will be passed + the item that was read. If there was an error while reading, the + onReadError method will be called. The + exception encounterd will be provided so that it can be + logged. +
+ +
+ ItemWriteListener + + Just as with the ItemReaderListener, the writing of an item + can be 'listened' to: + + public interface ItemWriteListener extends StepListener { + + void beforeWrite(Object item); + + void afterWrite(Object item); + + void onWriteError(Exception ex, Object item); +} + + The beforeWrite method will be called + before write on the + ItemWriter, and is handed the item that will + be written. The afterWrite method will be + called after the item has been succesfully writen. If there was an + error while writing, the onWriteError + method will be called. The exception encountered and the item that + was attempted to be written will be provided, so that they can be + logged. +
+ +
+ SkipListener + + Both ItemReadListener and + ItemWriteListner provide a mechanism for + being notified of errors, but neither one will inform you that a + record has actually been skipped. + onWriteError, for example, will be called + even if an item is retried and successful. For this reason, there is + a separate interface for tracking skipped items: + + + public interface SkipListener extends StepListener { + + void onSkipInRead(Throwable t); + + void onSkipInWrite(Object item, Throwable t); + } + + + + onSkipInRead will be called whenever + an item is skipped while reading. It should be noted that rollbacks + may cause the same item to be registered as skipped more than once. + onSkipInWrite will be called when an item + is skipped while writing. Because the item has been read + successfully (and not skipped), it is also provided the item itself + as an argument. +
+
+
+ +
+ TaskletStep + + Item oriented processing is not the only way to process in a + Step. What if a Step must + consist as a simple storec procedure call? You could implement the call + as an ItemReader and return null after the + procedure finishes, but it is a bit unnatural since there would need to + be a no-op ItemWriter and lots of overhead for + transaction handling, listeners, etc. Spring Batch provides an + implementation of Step for this scenario: + TaskletStep. As explained in Chapter 2, the + Tasklet is a simple interface that has one + method, execute, which will be a called once + for the whole Step. + Tasklet implementors might call a stored + procedure, a script, or a simple SQL upate statement. Because there are + less concerns, there are only two required dependencies for a + TaskletStep: a Tasklet, + and a JobRepository: + + <bean id="taskletStep" + class="org.springframework.batch.core.step.tasklet.TaskletStep" /> + <property name="tasklet" ref="tasklet" /> + <property name="jobRepository" ref="repository" /> +</bean> + + + TaskletStep will automatically register the tasklet as + StepExecutionListener if it implements this + interface + + +
+ TaskletAdapter + + As with other adapters for the ItemWriter + and ItemReader interfaces, the + Tasklet interface contains an implementation + that allows for adapting itself to any pre-existing class: + TaskletAdapter. An example where this may be + useful is an existing DAO that is used to upate a flag on a set of + records. The TaskletAdapter can be used to call + this class without having to write an adapter for the + Tasklet interface: + + <bean id="deleteFilesInDir" parent="taskletStep"> + <property name="tasklet"> + <bean class="org.springframework.batch.core.step.tasklet.TaskletAdapter"> + <property name="targetObject"> + <bean class="org.mycompany.FooDao"> + </property> + <property name="targetMethod" value-"updateFoo" /> + </bean> + </property> + </bean> +
+ +
+ Example Tasklet implementation + + Many batch jobs contains steps that must be done before the main + processing begins in order to set up various resources, or after + processing has completed to cleanup those resources. In the case of a + job that works heavily with files, it is often necessary to delete + certain files locally after they have been uploaded successfully to + another location. The example below taken from the Spring Batch + samples project, is a Tasklet implementation + with just such a responsibility: + + public class FileDeletingTasklet implements Tasklet, InitializingBean { + + private Resource directory; + + public ExitStatus execute() throws Exception { + File dir = directory.getFile(); + Assert.state(dir.isDirectory()); + + File[] files = dir.listFiles(); + for (int i = 0; i < files.length; i++) { + boolean deleted = files[i].delete(); + if (!deleted) { + throw new UnexpectedJobExecutionException("Could not delete file " + files[i].getPath()); + } + } + return ExitStatus.FINISHED; + } + + public void setDirectoryResource(Resource directory) { + this.directory = directory; + } + + public void afterPropertiesSet() throws Exception { + Assert.notNull(directory, "directory must be set"); + } + } + + The above Tasklet implementation will + delete all files within a given directory. It should be noted that the + execute method will only be called once. All + that is left is to inject the Tasklet into a + TaskletStep: + + <bean id="taskletJob" parent="simpleJob"> + <property name="steps"> + <bean id="deleteFilesInDir" parent="taskletStep"> + <property name="tasklet"> + <bean class="org.springframework.batch.sample.tasklet.FileDeletingTasklet"> + <property name="directoryResource" ref="directory" /> + </bean> + </property> + </bean> + </property> + </bean> + + <bean id="directory" + class="org.springframework.core.io.FileSystemResource"> + <constructor-arg value="target/test-outputs/test-dir" /> + </bean> +
+ +
+ Executing System Commands + + Many batch jobs may require that an external command be called + from within the batch job. Such a process could be kicked off + separately by the scheduler, but the advantage of common meta-data + about the run would be lost. Furthermore, a multi-step job would also + need to be split up into multiple jobs as well. Because the need is so + common, Spring Batch provides a Tasklet + implementation for calling system commands: + + + <bean class="org.springframework.batch.sample.tasklet.SystemCommandTasklet"> + <property name="command" value="echo hello" /> + <!-- 5 second timeout for the command to complete --> + <property name="timeout" value="5000" /> + </bean> + + +
+
+ +
+ Controlling Step Flow + + + +
+ Conditional Flow + + +
+ +
+ Configuring for Stop + + +
+ +
+ Programmatic flow decisions + + +
+
+
+ +
+ Examples of Customized Business Logic + +
+ Some batch jobs can be assembled purely from off-the-shelf + components in Spring Batch, mostly the ItemReader + and ItemWriter implementations. Where this is not + possible (the majority of cases) the main API entry points for + application developers are the Tasklet, + ItemReader, ItemWriter and + the various listener interfaces. Most simple batch jobs will be able to + use off-the-shelf input from a Spring Batch + ItemReader, but it is very often the case that + there are custom concerns in the processing and writing, which normally + leads developers to implement an ItemWriter, or + ItemTransformer. + + Here we provide a few examples of common patterns in custom + business logic, mainly using the listener interfaces . It should be + noted that an ItemReader or + ItemWriter can implement the listener interfaces + as well if appropriate. +
+ +
+ Logging Item Processing and Failures + + A common use case is the need for special handling of errors in a + step, item by item, perhaps logging to a special channel, or inserting a + record into a database. The StepHandlerStep + (created from the step factory beans) allows users to implement this use + case with a simple ItemReadListener, for errors + on read, and an ItemWriteListener, for errors on + write. The below code snippets illustrate a listener that logs both read + and write failures: + + public class ItemFailureLoggerListener extends ItemListenerSupport { + + private static Log logger = LogFactory.getLog("item.error"); + + public void onReadError(Exception ex) { + logger.error("Encountered error on read", e); + } + + public void onWriteError(Exception ex, Object item) { + logger.error("Encountered error on write", e); + } + +} + + Having implemented this listener it must be registered with the + step: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > + ... + <property name="listeners"> + <bean class="org.example...ItemFailureLoggerListener"/> + </property> + </bean> + + Remember that if your listener does anything in an + onError() method, it will be inside a transaction that is + going to be rolled back. If you need to use a transactional resource + such as a database inside an onError() method, consider + adding a declarative transaction to that method (see Spring Core + Reference Guide for details), and giving its propagation attribute the + value REQUIRES_NEW. +
+ +
+ Stopping a Job Manually for Business Reasons + + Spring Batch provides a stop() method through the JobLauncher + interface, but this is really aimed at the operator, rather than the + application programmer. Sometimes it is more convenient or makes more + sense to stop a job execution from within the business logic. + + The simplest thing to do is to throw a RuntimeException (one that + isn't retried indefinitely or skipped), For example, a custom exception + type could be used, as in the example below: + + public class PoisonPillItemWriter extends AbstractItemWriter { + + public void write(Object item) throws Exception { + + if (isPoisonPill(item)) { + throw new PoisonPillException("Posion pill detected: "+item); + } + + } + +} + + Another simple way to stop a step from executing is to simply + return null from the + ItemReader: + + public class EarlyCompletionItemReader extends AbstractItemReader { + + private ItemReader delegate; + + public void setDelegate(ItemReader delegate) { ... } + + public Object read() throws Exception { + + Object item = delegate.read(); + + if (isEndItem(item)) { + return null; // end the step here + } + + return item; + + } + +} + + The previous example actually relies on the fact that there is a + default implementation of the CompletionPolicy + strategy which signals a complete batch when the item to be processed is + null. A more sophisticated completion policy could be implemented and + injected into the Step through the + RepeatOperationsStepFactoryBean: + + <bean id="simpleStep" + class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" > + ... + <property name="chunkOperations"> + <bean class="org.springframework.batch.repeat.support.RepeatTemplate"> + <property name="completionPolicy"> + <bean class="org.example...SpecialCompletionPolicy"/> + </property> + </bean> + </property> + </bean> + + An alternative is to set a flag in the + StepExecution, which is checked by the + Step implementations in the framework in between + item processing. To implement this alternative, we need access to the + current StepExecution, and this can be achieved by implementing a + StepListener and registering it with the Step. Here is an example of a + listener that sets the flag: + + public class CustomItemWriter extends ItemListenerSupport implements StepListener { + + private StepExecution stepExecution; + + public void beforeStep(StepExecution stepExecution) { + this.stepExecution = stepExecution; + } + + public void afterRead(Object item) { + + if (isPoisonPill(item)) { + stepExecution.setTerminateOnly(true); + } + + } + +} + + The default behaviour here when the flag is set is for the step to + throw a JobInterruptedException. This can be + controlled through the StepInterruptionPolicy, + but the only choice is to throw or not throw an exception, so this is + always an abnormal ending to a job. +
+ +
+ Adding a Footer Record + + A very common requirement is to aggregate information during the + output process and to append a record at the end of a file summarizing + the data, or providing a checksum. This can also be achieved with a + callbacks in the step, normally as part of a custom + ItemWriter. In this case, since a job is + accumulating state that should not be lost if the job aborts, the + ItemStream interface should be + implemented: + + public class CustomItemWriter extends AbstractItemWriter implements + ItemStream, StepListener +{ + + private static final String TOTAL_AMOUNT_KEY = "total.amount"; + + private ItemWriter delegate; + + private double totalAmount = 0.0; + + public void setDelegate(ItemWriter delegate) { ... } + + public ExitStatus afterStep(StepExecution stepExecution) { + // Add the footer record here... + delegate.write("Total Amount Processed: " + totalAmount); + } + + public void open(ExecutionContext executionContext) { + if (executionContext.containsKey(TOTAL_AMOUNT_KEY) { + totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY); + } + } + + public void update(ExecutionContext executionContext) { + executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount); + } + + public void write(Object item) { + + delegate.write(item); + totalAmount += ((Trade) item).getAmount(); + + } + +} + + The custom writer in the example is stateful (it maintains its + total in an instance variable totalAmount), but the + state is stored through the ItemStream interface + in the ExecutionContext. In this way we can be + sure that when the open() callback is received on a + restart. The framework garuntees we always get the last value that was + committed. It should be noted that it is not always necessary to + implement ItemStream. For example, if the ItemWriter is re-runnable, in + the sense that it maintains its own state in a transactional resource + like a database, there is no need to maintain state within the writer + itself. +
+
+ \ No newline at end of file diff --git a/docs/src/site/docbook/reference/whatsnew.xml b/docs/src/site/docbook/reference/whatsnew.xml new file mode 100644 index 000000000..fc15f0f4d --- /dev/null +++ b/docs/src/site/docbook/reference/whatsnew.xml @@ -0,0 +1,1103 @@ + + + + The Domain Language of Batch + +
+ Introduction + + To any experienced batch architect, the overall concepts of batch + processing used in Spring Batch should be familiar and comfortable. There + are “Jobs” and “Steps” and developer supplied processing units called + ItemReaders and ItemWriters. However, because of the Spring patterns, + operations, templates, callbacks, and idioms, there are opportunities for + the following: + + significant improvement in adherence to a clear separation of + concerns + + + + clearly delineated architectural layers and services provided + as interfaces + + + + simple and default implementations that allowed for quick + adoption and ease of use out-of-the-box + + + + significantly enhanced extensibility + + + + The diagram below is only a slight variation of the batch reference + architecture that has been used for decades. It provides an overview of + the high level components, technical services, and basic operations + required by a batch architecture. This architecture framework is a + blueprint that has been proven through decades of implementations on the + last several generations of platforms (COBOL/Mainframe, C++/Unix, and now + Java/anywhere). JCL and COBOL developers are likely to be as comfortable + with the concepts as C++, C# and Java developers. Spring Batch provides a + physical implementation of the layers, components and technical services + commonly found in robust, maintainable systems used to address the + creation of simple to complex batch applications, with the infrastructure + and extensions to address very complex processing needs. +
+ +
+ Batch Application Style Interactions and Services + + + + + + + + + + + Figure 2.1: Batch Stereotypes + + + The above diagram highlights the interactions and key services + provided by the Spring Batch framework. The colors used are important to + understanding the responsibilities of a developer in Spring Batch. Grey + represents an external application such as an enterprise scheduler or a + database. It's important to note that scheduling is grey, and should thus + be considered separate from Spring Batch. Blue represents application + architecture services. In most cases these are provided by Spring Batch + with out of the box implementations, but an architecture team may make + specific implementations that better address their specific needs. Yellow + represents the pieces that must be configured by a developer. For example, + a job schedule needs to be configured so that the job is kicked off at the + appropriate time. A job configuration file also needs to be created, which + defines how a job will be run. It is also worth noting that the + ItemReader and ItemWriter + used by an application may just as easily be a custom one made by a + developer for their specific batch job, rather than one provided by Spring + Batch or an architecture team. + + The Batch Application Style is organized into four logical tiers, + which include Run, Job, Application, and Data. The primary goal for + organizing an application according to the tiers is to embed what is known + as "separation of concerns" within the system. These tiers can be + conceptual but may prove effective in mapping the deployment of the + artifacts onto physical components like Java runtimes and integration with + data sources and targets. Effective separation of concerns results in + reducing the impact of change to the system. The four conceptual tiers + containing batch artifacts are: + + + + Run Tier: The Run Tier is + concerned with the scheduling and launching of the application. A + vendor product is typically used in this tier to allow time-based + and interdependent scheduling of batch jobs as well as providing + parallel processing capabilities. + + + + Job Tier: The Job Tier is + responsible for the overall execution of a batch job. It + sequentially executes batch steps, ensuring that all steps are in + the correct state and all appropriate policies are enforced. + + + + Application Tier: The + Application Tier contains components required to execute the + program. It contains specific tasks that address required batch + functionality and enforces policies around execution (e.g., commit + intervals, capture of statistics, etc.) + + + + Data Tier: The Data Tier + provides integration with the physical data sources that might + include databases, files, or queues. + + +
+ +
+ Job Stereotypes + + This section describes stereotypes relating to the concept of a + batch job. A Job is an entity that encapsulates an + entire batch process. As is common with other Spring projects, a + Job will be wired together via an XML configuration + file. This file may be referred to as the "job configuration". However, + Job is just the top of an overall hierarchy: + + + + + + + + + + + +
+ Job + + A job is represented by a Spring bean that implements the + Job interface and contains all of the information + necessary to define the operations performed by a job. A job + configuration is typically contained within a Spring XML configuration + file and the job's name is determined by the "id" attribute associated + with the job configuration bean. The job configuration contains + + + + The simple name of the job + + + + Definition and ordering of Steps + + + + Whether or not the job is restartable + + + + A default simple implementation of the Job + interface is provided by Spring Batch in the form of the + SimpleJob class which creates some standard + functionality on top of Job, namely a standard + execution logic that all jobs should utilize. In general, all jobs + should be defined using a bean of type + SimpleJob: + + <bean id="footballJob" + class="org.springframework.batch.core.job.SimpleJob"> + <property name="steps"> + <list> + <!-- Step Bean details ommitted for clarity --> + <bean id="playerload" parent="simpleStep" /> + <bean id="gameLoad" parent="simpleStep" /> + <bean id="playerSummarization" parent="simpleStep" /> + </list> + </property> + <property name="restartable" value="true" /> + </bean> +
+ +
+ JobInstance + + A JobInstance refers to the concept of a + logical job run. Let's consider a batch job that should be run once at + the end of the day, such as the 'EndOfDay' job from the diagram above. + There is one 'EndOfDay' Job, but each individual + run of the Job must be tracked separately. In the + case of this job, there will be one logical + JobInstance per day. For example, there will be a + January 1st run, and a January 2nd run. If the January 1st run fails the + first time and is run again the next day, it's still the January 1st + run. (Usually this corresponds with the data its processing as well, + meaning the January 1st run processes data for January 1st, etc) That is + to say, each JobInstance can have multiple + executions. (JobExecution is discussed in more + detail below) and only one JobInstance + corresponding to a particular Job can be running + at a given time. The definition of a JobInstance + has absolutely no bearing on the data the will be loaded. It is entirely + up to the ItemReader implementation used to + determine how data will be loaded. For example, in the EndOfDay + scenario, there may be a column on the data that indicates the + 'effective date' or 'schedule date' to which the data belongs. So, the + January 1st run would only load data from the 1st, and the January 2nd + run would only use data from the 2nd. Because this determination will + likely be a business decision, it is left up to the + ItemReader to decide. What using the same + JobInstance will determine, however, is whether + or not the 'state' (i.e. the ExecutionContext, which is discussed below) + from previous executions will be used. Using a new + JobInstance will mean 'start from the beginning' + and using an existing instance will generally mean 'start from where you + left off'. +
+ +
+ JobParameters + + Having discussed JobInstance and how it + differs from Job, the natural question to ask is: + "how is one JobInstance distinguished from + another?" The answer is: JobParameters. + JobParameters are any set of parameters used to + start a batch job, which can be used for identification or even as + reference data during the run. In the example above, where there are two + instances, one for January 1st, and another for January 2nd, there is + really only one Job, one that was started with a job parameter of + 01-01-2008 and another that was started with a parameter of 01-02-2008. + Thus, the contract can be defined as: JobInstance + = Job + JobParameters. + This allows a developer to effectively control how you a + JobInstance is defined, since they control what + parameters are passed in. +
+ +
+ JobExecution + + A JobExecution refers to the technical + concept of a single attempt to run a Job. An + execution may end in failure or success, but the + JobInstance corresponding to a given execution + will not be considered complete unless the execution completes + successfully. Using the EndOfDay Job described + above as an example, consider a JobInstance for 01-01-2008 that failed + the first time it was run. If it is ran again, with the same job + parameters as the first run (01-01-2008), a new JobExecution will be + created. However, there will still be only one + JobInstance. + + A Job defines what a job is and how it is + to be executed, and JobInstance is a purely + organizational object to group executions together, primarily to enable + correct restart semantics. A JobExecution, + however, is the primary storage mechanism for what actually happened + during a run, and as such contains many more properties that must be + controlled and persisted: + + + JobExecution properties + + + + + status + + A BatchStatus object that + indicates the status of the execution. While it's running, it's + BatchStatus.STARTED, if it fails it's BatchStatus.FAILED, and if + it finishes successfully it's BatchStatus.COMPLETED + + + + startTime + + A java.util.Date representing the + current system time when the execution was started. + + + + endTime + + A java.util.Date representing the + current system time when the execution finished, regardless of + whether or not it was successful. + + + + exitStatus + + The ExitStatus indicating the + result of the run. It is most important because it contains an + exit code that will be returned to the caller. See chapter 5 for + more details. + + + + createTime + + A java.util.Date representing the + current system time when the JobExecution was first persisted. + The job may not have been started yet (and thus has no start + time), but it will always have a createTime, which is required + by the framework for managing job level + ExecutionContexts. + + + + lastUpdated + + A java.util.Date representing the + last time a JobExecution was persisted. + + + + executionContext + + The 'property bag' containing any user data that needs to + be persisted between executions. + + + + failureExceptions + + The list of exceptions encountered during the execution + of a Job. These can be useful if more + than one exception is encountered during the failure of a + Job. + + + +
+ + These properties are important because they will be persisted and + can be used to completely determine the status of an execution. For + example, if the EndOfDay job for 01-01 is executed at 9:00 PM, and fails + at 9:30, the following entries will be made in the batch meta data + tables: + + + BATCH_JOB_INSTANCE + + + + + JOB_INSTANCE_ID + + JOB_NAME + + + + 1 + + EndOfDayJob + + + +
+ + + BATCH_JOB_PARAMS + + + + + JOB_INSTANCE_ID + + TYPE_CD + + KEY_NAME + + DATE_VAL + + + + 1 + + DATE + + schedule.Date + + 2008-01-01 00:00:00 + + + +
+ + + BATCH_JOB_EXECUTION + + + + + JOB_EXECUTION_ID + + JOB_INSTANCE_ID + + START_TIME + + END_TIME + + STATUS + + + + 1 + + 1 + + 2008-01-01 21:00:23.571 + + 2008-01-01 21:30:17.132 + + FAILED + + + +
+ + + extra columns in the tables have been removed for added + clarity. + + + Now that the job has failed, let's assume that it took the entire + course of the night for the problem to be determined, so that the 'batch + window' is now closed. Assuming the window starts at 9:00 PM, the job + will be kicked off again for 01-01, starting where it left off and + completing successfully at 9:30. Because it's now the next day, the + 01-02 job must be run as well, which is kicked off just afterwards at + 9:31, and completes in it's normal one hour time at 10:30. There is no + requirement that one JobInstance be kicked off + after another, unless there is potential for the two jobs to attempt to + access the same data, causing issues with locking at the database level. + It is entirely up to the scheduler to determine when a + Job should be run. Since they're separate + JobInstances, Spring Batch will make no attempt to stop them from being + run concurrently. (Attempting to run the same + JobInstance while another is already running will + result in a JobExecutionAlreadyRunningException + being thrown) There should now be an extra entry in both the + JobInstance and + JobParameters tables, and two extra entries in + the JobExecution table: + + + BATCH_JOB_INSTANCE + + + + + JOB_INSTANCE_ID + + JOB_NAME + + + + 1 + + EndOfDayJob + + + + 2 + + EndOfDayJob + + + +
+ + + BATCH_JOB_PARAMS + + + + + JOB_INSTANCE_ID + + TYPE_CD + + KEY_NAME + + DATE_VAL + + + + 1 + + DATE + + schedule.Date + + 2008-01-01 00:00:00 + + + + 2 + + DATE + + schedule.Date + + 2008-01-02 00:00:00 + + + +
+ + + BATCH_JOB_EXECUTION + + + + + JOB_EXECUTION_ID + + JOB_INSTANCE_ID + + START_TIME + + END_TIME + + STATUS + + + + 1 + + 1 + + 2008-01-01 21:00 + + 2008-01-01 21:30 + + FAILED + + + + 2 + + 1 + + 2008-01-02 21:00 + + 2008-01-02 21:30 + + COMPLETED + + + + 3 + + 2 + + 2008-01-02 21:31 + + 2008-01-02 22:29 + + COMPLETED + + + +
+
+
+ +
+ Step Stereotypes + + A Step is a domain object that encapsulates + an independent, sequential phase of a batch job. Therefore, every + Job is composed entirely of one or more steps. A + Step should be thought of as a unique processing + stream that will be executed in sequence. For example, if you have one + step that loads a file into a database, another that reads from the + database, validates the data, preforms processing, and then writes to + another table, and another that reads from that table and writes out to a + file. Each of these steps will be performed completely before moving on to + the next step. The file will be completely read into the database before + step 2 can begin. As with Job, a + Step has an individual + StepExecution that corresponds with a unique + JobExecution: + + + + + + + + + + + +
+ Step + + A Step contains all of the information + necessary to define and control the actual batch processing. This is a + necessarily vague description because the contents of any given + Step are at the discretion of the developer + writing a Job. A Step can be as simple or complex + as the developer desires. A simple Step might + load data from a file into the database, requiring little or no code. + (depending upon the implementations used) A more complex + Step may have complicated business rules that are + applied as part of the processing. + + Steps are defined by instantiating implementations of the + Step interface. Two step implementation classes + are available in the Spring Batch framework, and they are each discussed + in detail in Chatper 4 of this guide. For most situations, the + StepHandlerStep implementation is sufficient, but + for situations where only one call is needed, such as a stored procedure + call or a wrapper around existing script, a + TaskletStep may be a better option. +
+ +
+ StepExecution + + A StepExecution represents a single attempt + to execute a Step. A new + StepExecution will be created each time a Step is + run, similar to JobExecution. However, if a step + fails to execute because the step before it fails, there will be no + execution persisted for it. A StepExecution will + only be created when it's Step is actually + started. + + Step executions are represented by objects of the + StepExecution class. Each execution contains a + reference to its corresponding step and + JobExecution, and transaction related data such + as commit and rollback count and start and end times. Additionally, each + step execution will contain an ExecutionContext, + which contains any data a developer needs persisted across batch runs, + such as statistics or state information needed to restart. The following + is a listing of the properties for + StepExecution: + + + StepExecution properties + + + + + status + + A BatchStatus object that + indicates the status of the execution. While it's running, the + status is BatchStatus.STARTED, if it fails the status is + BatchStatus.FAILED, and if it finishes successfully the status + is BatchStatus.COMPLETED + + + + startTime + + A java.util.Date representing the + current system time when the execution was started. + + + + endTime + + A java.util.Date representing the + current system time when the execution finished, regardless of + whether or not it was successful. + + + + exitStatus + + The ExitStatus indicating the + result of the execution. It is most important because it + contains an exit code that will be returned to the caller. See + chapter 5 for more details. + + + + executionContext + + The 'property bag' containing any user data that needs to + be persisted between executions. + + + + readCount + + The number of items that have been successfully + read + + + + writeCount + + The number of items that have been successfully + written + + + + commitCount + + The number transactions that have been committed for this + execution + + + + rollbackCount + + The number of times the business transaction controlled + by the Step has been rolled back. + + + + readSkipCount + + The number of times read has + failed, resulting in a skipped item. + + + + processSkipCount + + The number of times process has failed, resulting in a + skipped item. + + + + filterCount + + The number of items that have been 'filtered' by the + ItemProcessor + + + + writeSkipCount + + The number of times write has + failed, resulting in a skipped item. + + + +
+
+ +
+ ExecutionContext + + An ExecutionContext represents a collection + of key/value pairs that are persisted and controlled by the framework in + order to allow developers a place to store persistent state that is + scoped to a StepExecution or + JobExecution. For those familiar with Quartz, it + is very similar to JobDataMap. The best usage + example is restart. Using flat file input as an example, while + processing individual lines, the framework periodically persists the + ExecutionContext at commit points. This allows + the ItemReader to store its state in case a fatal + error occurs during the run, or even if the power goes out. All that is + needed is to put the current number of lines read into the context, and + the framework will do the rest: + + executionContext.putLong(getKey(LINES_READ_COUNT), reader.getPosition()); + + Using the EndOfDay example from the Job Stereotypes section as an + example, assume there's one step: 'loadData', that loads a file into the + database. After the first failed run, the meta data tables would look + like the following: + + + BATCH_JOB_INSTANCE + + + + + JOB_INSTANCE_ID + + JOB_NAME + + + + 1 + + EndOfDayJob + + + +
+ BATCH_JOB_PARAMS + + + + + JOB_INSTANCE_ID + + TYPE_CD + + KEY_NAME + + DATE_VAL + + + + 1 + + DATE + + schedule.Date + + 2008-01-01 00:00:00 + + + +
+ BATCH_JOB_EXECUTION + + + + + JOB_EXECUTION_ID + + JOB_INSTANCE_ID + + START_TIME + + END_TIME + + STATUS + + + + 1 + + 1 + + 2008-01-01 21:00:23.571 + + 2008-01-01 21:30:17.132 + + FAILED + + + +
+ BATCH_STEP_EXECUTION + + + + + STEP_EXECUTION_ID + + JOB_EXECUTION_ID + + STEP_NAME + + START_TIME + + END_TIME + + STATUS + + + + 1 + + 1 + + loadDate + + 2008-01-01 21:00:23.571 + + 2008-01-01 21:30:17.132 + + FAILED + + + +
+ BATCH_EXECUTION_CONTEXT + + + + + EXECUTION_ID + + TYPE_CD + + KEY_NAME + + LONG_VAL + + + + 1 + + LONG + + piece.count + + 40321 + + + +
In this case, the Step ran for 30 + minutes and processed 40,321 'pieces', which would represent lines in a + file in this scenario. This value will be updated just before each + commit by the framework, and can contain multiple rows corresponding to + entries within the ExecutionContext. Being + notified before a commit requires one of the various StepListeners, or + an ItemStream, which are discussed in more detail + later in this guide. As with the previous example, it is assumed that + the Job is restarted the next day. When it is restarted, the values from + the ExecutionContext of the last run are + reconstituted from the database, and when the + ItemReader is opened, it can check to see if it + has any stored state in the context, and initialize itself from + there: + + if (executionContext.containsKey(getKey(LINES_READ_COUNT))) { + log.debug("Initializing for restart. Restart data is: " + executionContext); + + long lineCount = executionContext.getLong(getKey(LINES_READ_COUNT)); + + LineReader reader = getReader(); + + Object record = ""; + while (reader.getPosition() < lineCount && record != null) { + record = readLine(); + } + } + + In this case, after the above code is executed, the current line + will be 40,322, allowing the Step to start again + from where it left off. The ExecutionContext can + also be used for statistics that need to be persisted about the run + itself. For example, if a flat file contains orders for processing that + exist across multiple lines, it may be necessary to store how many + orders have been processed (which is much different from than the number + of lines read) so that an email can be sent at the end of the + Step with the total orders processed in the body. + The framework handles storing this for the developer, in order to + correctly scope it with an individual + JobInstance. It can be very difficult to know + whether an existing ExecutionContext should be + used or not. For example, using the 'EndOfDay' example from above, when + the 01-01 run starts again for the second time, the framework recognizes + that it is the same JobInstance and on an + individual Step basis, pulls the + ExecutionContext out of the database and hands it + as part of the StepExecution to the + Step itself. Conversely, for the 01-02 run the + framework recognizes that it is a different instance, so an empty + context must be handed to the Step. There are + many of these types of determinations that the framework makes for the + developer to ensure the state is given to them at the correct time. It + is also important to note that exactly one + ExecutionContext exists per + StepExecution at any given time. Clients of the + ExecutionContext should be careful because this + creates a shared keyspace, so care should be taken when putting values + in to ensure no data is overwritten, however, the + Step stores absolutely no data in the context, so + there is no way to adversely affect the framework. +
+
+ +
+ JobRepository + + JobRepository is the persistence mechanism + for all of the Stereotypes mentioned above. When a job is first launched, + a JobExecution is obtained by calling the + repository's createJobExecution method, and + during the course of execution, StepExecution and + JobExecution are persisted by passing them to the + repository: + + public interface JobRepository { + + public JobExecution createJobExecution(Job job, JobParameters jobParameters) + throws JobExecutionAlreadyRunningException, JobRestartException; + + void add(StepExecution stepExecution); + + void update(JobExecution jobExecution); + + void update(StepExecution stepExecution); + + void updateExecutionContext(StepExecution stepExecution); + + StepExecution getLastStepExecution(JobInstance jobInstance, Step step); + + int getStepExecutionCount(JobInstance jobInstance, Step step); + +} + +
+ +
+ JobLauncher + + JobLauncher represents a simple interface for + launching a Job with a given set of + JobParameters: + + public interface JobLauncher { + + public JobExecution run(Job job, JobParameters jobParameters) throws JobExecutionAlreadyRunningException, + JobRestartException; +} + + + It is expected that implementations will obtain a valid + JobExecution from the + JobRepository and execute the + Job. +
+ +
+ JobLocator + + JobLocator represents an interface for + locating a Job: + + public interface JobLocator { + + Job getJob(String name) throws NoSuchJobException; + } + + This interface is very necessary due to the nature of Spring itself. + Because it can't be guaranteed that one + ApplicationContext equals one + Job, an abstraction is needed to obtain a + Job for a given name. It becomes especially useful + when launching jobs from within a Java EE application server. +
+ +
+ Item Reader + + ItemReader is an abstraction that represents + the retrieval of input for a Step, one item at a + time. When the ItemReader has exhausted the items + it can provide, it will indicate this by returning null. More details + about the ItemReader interface and its various + implementations can be found in Chapter 3. +
+ +
+ Item Writer + + ItemWriter is an abstraction that represents + the output of a Step, one item at a time. + Generally, an item writer has no knowledge of the input it will receive + next, only the item that was passed in its current invocation. More + details about the ItemWriter interface and it's + various implementations can be found in Chapter 3. +
+ +
+ Item Processor + + ItemProcessor is an abstraction that + represents the business processing of an item. While the + ItemReader reads one item, and the + ItemWriter writes them, the + ItemProcessor provides access to transform or apply + other business processing. If while processing the item it's determined + that it's not valid, returning null indicates that it should not be + written out. +
+ +
+ Tasklet + + A Tasklet represents the execution of a + logical unit of work, as defined by its implementation of the Spring Batch + provided Tasklet interface. A + Tasklet is useful for encapsulating processing + logic that is not natural to split into read-(transform)-write phases, + such as invoking a system command or a stored procedure. +
+ \ No newline at end of file