diff --git a/src/site/docbook/reference/job.xml b/src/site/docbook/reference/job.xml index 15c14d01e..4797ab6dd 100644 --- a/src/site/docbook/reference/job.xml +++ b/src/site/docbook/reference/job.xml @@ -4,8 +4,9 @@ Configuring and Running a Job - In , the overall architecture design was - discussed, using the following diagram as a guide: + In the domain section , the + overall architecture design was discussed, using the following + diagram as a guide: @@ -35,7 +36,7 @@ Job interface, however, the namespace abstracts away the differences in configuration. It has only three required dependencies: a name, JobRepository , and - a list of Step s. + a list of Steps. <job id="footballJob"> <step id="playerload" parent="s1" next="gameLoad"/> @@ -43,9 +44,11 @@ <step id="playerSummarization" parent="s3"/> </job> - The namespace defaults to referencing a repository with an id of - 'jobRepository', which is a sensible default. However, this can be - overridden explicitly: + The examples here use a parent bean definition to create the steps; + see the section on step configuration + for more options declaring specific step details inline. The XML namespace + defaults to referencing a repository with an id of 'jobRepository', which + is a sensible default. However, this can be overridden explicitly: <job id="footballJob" job-repository="specialRepository"> <step id="playerload" parent="s1" next="gameLoad"/> @@ -53,6 +56,12 @@ <step id="playerSummarization" parent="s3"/> </job> + In addition to steps a job configuration can contain other elements + that help with parallelisation (<split/>), + declarative flow control (<decision/>) and + externalization of flow definitions + (<flow/>). +
Restartability @@ -194,31 +203,26 @@ catch (JobRestartException e) { for more detailed information.
-
- JobFactory and Stateful Components in Steps +
+ JobParametersValidator - Unlike many traditional Spring applications, many of the - components of a batch application are stateful; the file readers and - writers are obvious examples. The recommended way to deal with this is - to create a fresh ApplicationContext for each job - execution. If the Job is launched from the - command line with CommandLineJobRunner, this is - trivial. For more complex launching scenarios where jobs are executed in - parallel or serially from the same process, some extra steps have to be - taken to ensure that the ApplicationContext is - refreshed. This is preferable to using prototype scope for the stateful - beans because then they would not receive lifecycle callbacks from the - container at the end of use. (e.g. through destroy-method in XML) + A job declared in the XML namespace or using any subclass of + AbstractJob can optionally declare a validator for the job parameters at + runtime. This is useful when for instance you need to assert that a job + is started with all its mandatory parameters. There is a + DefaultJobParametersValidator that can be used to constrain combinations + of simple mandatory and optional parameters, and for more complex + constraints you can implement the interface yourself. The configuration + of a validator is supported through the XML namespace through a child + element of the job, e.g>: - The strategy provided by Spring Batch to deal with this scenario - is the JobFactory, and the samples provide an - example of a specialized implementation that can load an - ApplicationContext and close it properly when the - job is finished. A relevant examples is - ClassPathXmlApplicationContextJobFactory and its - use in the adhoc-job-launcher-context.xml and the - quartz-job-launcher-context.xml, which can be found in the - Samples project. + <job id="job1" parent="baseJob3"> + <step id="step1" parent="standaloneStep"/> + <validator ref="paremetersValidator"/> +</job> + + The validator can be specified as a reference (as above) or as a + nested bean definition in the beans namespace.
@@ -537,11 +541,13 @@ catch (JobRestartException e) { bash$ java CommandLineJobRunner endOfDayJob.xml endOfDay schedule.date(date)=2008/01/01 - In most cases you would want to use a manifest to declare your - main class in a jar, but for simplicity, the class was used directly. - This example is using the same 'EndOfDay' example from . The first argument is 'endOfDayJob.xml', which is - the Spring ApplicationContext containing the + In most cases you would want to use a manifest to + declare your main class in a jar, but for simplicity, the + class was used directly. This example is using the same + 'EndOfDay' example from the domain + section. The first argument is 'endOfDayJob.xml', which + is the Spring ApplicationContext + containing the Job. The second argument, 'endOfDay' represents the job name. The final argument, 'schedule.date(date)=2008/01/01' will be converted into JobParameters. An diff --git a/src/site/docbook/reference/step.xml b/src/site/docbook/reference/step.xml index a635430e7..ca125584f 100644 --- a/src/site/docbook/reference/step.xml +++ b/src/site/docbook/reference/step.xml @@ -54,13 +54,13 @@ Below is a code representation of the same concepts shown above: - List items = new Arraylist(); +for(int i = 0; i < commitInterval; i++){ Object item = itemReader.read() Object processedItem = itemProcessor.process(item); items.add(processedItem); } -itemWriter.write(items);]]> +itemWriter.write(items);
Configuring a Step @@ -70,13 +70,13 @@ itemWriter.write(items);]]> potentially contain many collaborators. In order to ease configuration, the Spring Batch namespace can be used: - - - - - - -]]> + <job id="sampleJob" job-repository="jobRepository"> + <step id="step1"> + <tasklet transaction-manager="transactionManager"> + <chunk reader="itemReader" writer="itemWriter" commit-interval="10"/> + <tasklet> + </step> +</job> The configuration above represents the only required dependencies to create a item-oriented step: @@ -121,43 +121,6 @@ itemWriter.write(items);]]> writer.
-
- Referencing a Standalone Step - - While steps must exist within a Job to define the flow, it can - sometimes be useful to reference a 'standalone' Step. For example, if a - Step is used by multiple jobs it can be useful to declare it once and - reference it from multiple jobs. This can be achieved with the 'parent' - attribute: - - <job id="sampleJob" job-repository="jobRepository"> - <step id="step1" parent="standaloneStep" /> -</job> - -<step id="standaloneStep"> - <tasklet job-repository="jobRepository" transaction-manager="transactionManager"> - <chunk reader="itemReader" writer="itemWriter" commit-interval="10"/> - </tasklet> -</step> - - It should be noted that the id attribute is still required on the - step within the job element. This is for two reasons: - - - - The id will be used as the step name when persisting the - StepExecution. If the same standalone step is referenced in more - than one step in the job, an error will occur. - - - - When creating job flows, as described later in this chapter, - the next attribute should be referring to the step in the flow, not - the standalone step. - - -
-
Inheriting from a Parent Step @@ -175,17 +138,32 @@ itemWriter.write(items);]]> allowStartIfComplete=true. Additionally, the commitInterval will be '5' since it is overridden by the "concreteStep1": - - - - - + <step id="parentStep"> + <tasklet allow-start-if-complete="true"> + <chunk reader="itemReader" writer="itemWriter" commit-interval="10"/> + </tasklet> +</step> - - - - -]]> +<step id="concreteStep1" parent="parentStep"> + <tasklet start-limit="5"> + <chunk processor="itemProcessor" commit-interval="5"/> + </tasklet> +</step> + + The id attribute is still required on the step within the job + element. This is for two reasons: + + The id will be used as the step name when persisting the + StepExecution. If the same standalone step is referenced in more + than one step in the job, an error will occur. + + + + When creating job flows, as described later in this chapter, + the next attribute should be referring to the step in the flow, + not the standalone step. + +
Abstract Step @@ -205,17 +183,17 @@ itemWriter.write(items);]]> be abstract. The Step "concreteStep2" will have 'itemReader', 'itemWriter', and commitInterval=10. - - - - - + <step id="abstractParentStep" abstract="true"> + <tasklet> + <chunk commit-interval="10"/> + </tasklet> +</step> - - - - -]]> +<step id="concreteStep2" parent="abstractParentStep"> + <tasklet> + <chunk reader="itemReader" writer="itemWriter"/> + </tasklet> +</step>
@@ -236,20 +214,20 @@ itemWriter.write(items);]]> listenerOne and listenerTwo: - - - - - + <step id="listenersParentStep" abstract="true"> + <listeners> + <listener ref="listenerOne"/> + <listeners> +</step> - - - - - - - -]]> +<step id="concreteStep3" parent="listenersParentStep"> + <tasklet> + <chunk reader="itemReader" writer="itemWriter" commit-interval="5"/> + <listeners merge="true"> + <listener ref="listenerTwo"/> + <listeners> + </tasklet> +</step>
@@ -341,26 +319,26 @@ itemWriter.write(items);]]>
Step Restart Configuration Example - - - - - - - - - - - - - - - - -]]> + <job id="footballJob" restartable="true"> + <step id="playerload" next="gameLoad"> + <tasklet> + <chunk reader="playerFileItemReader" writer="playerWriter" + commit-interval="10" /> + </tasklet> + </step> + <step id="gameLoad" next="playerSummarization"> + <tasklet allow-start-if-complete="true"> + <chunk reader="gameFileItemReader" writer="gameWriter" + commit-interval="10"/> + </tasklet> + </step> + <step id="playerSummarization"> + <tasklet start-limit="3"> + <chunk reader="playerSummarizationSource" writer="summaryWriter" + commit-interval="10"/> + </tasklet> + </step> +</job> The above example configuration is for a job that loads in information about football games and summarizes them. It contains @@ -570,14 +548,14 @@ itemWriter.write(items);]]> the Step can be configured with a list of exceptions that should not cause rollback. - - - - - - - -]]> + <step id="step1"> + <tasklet> + <chunk reader="itemReader" writer="itemWriter" commit-interval="2"/> + <no-rollback-exception-classes> + <include class="org.springframework.batch.item.validator.ValidationException"/> + </no-rollback-exception-classes> + </tasklet> +</step>
Transactional Readers @@ -609,14 +587,14 @@ itemWriter.write(items);]]> transaction attributes can be found in the spring core documentation. - - - - <step id="step1"> + <tasklet> + <chunk reader="itemReader" writer="itemWriter" commit-interval="2"/> + <transaction-attributes isolation="DEFAULT" propagation="REQUIRED" - timeout="30"/> - -]]> + timeout="30"/> + </tasklet> +</step>
@@ -689,14 +667,14 @@ itemWriter.write(items);]]> interface (or an extension thereof) can be applied to a step via the listeners element: - - - - - - - -]]> + <step id="step1"> + <tasklet> + <chunk reader="reader" writer="writer" commit-interval="10"/> + <listeners> + <listener ref="stepListener"/> + </listeners> + </tasklet> +</step> An ItemReader, ItemWriter or @@ -728,13 +706,13 @@ itemWriter.write(items);]]> for notification before a Step is started and after it has ends, whether it ended normally or failed: - public interface StepExecutionListener extends StepListener { void beforeStep(StepExecution stepExecution); ExitStatus afterStep(StepExecution stepExecution); -}]]> +} ExitStatus is the return type of afterStep in order to allow listeners the @@ -763,13 +741,13 @@ itemWriter.write(items);]]> useful to perform logic before a chunk begins processing or after a chunk has completed: - public interface ChunkListener extends StepListener { void beforeChunk(); void afterChunk(); -}]]> +} The beforeChunk method is called after the transaction is started, but before read @@ -798,7 +776,7 @@ itemWriter.write(items);]]> When discussing skip logic above, it was mentioned that it may be beneficial to log out skipped records, so that they can be deal with later. In the case of read errors, this can be done with an - ItemReaderListener: extends StepListener { + ItemReaderListener:public interface ItemReadListener<T> extends StepListener { void beforeRead(); @@ -806,7 +784,7 @@ itemWriter.write(items);]]> void onReadError(Exception ex); -}]]> +} The beforeRead method will be called before each call to read on the @@ -841,7 +819,7 @@ itemWriter.write(items);]]> Just as with the ItemReadListener, the processing of an item can be 'listened' to: - extends StepListener { + public interface ItemProcessListener<T, S> extends StepListener { void beforeProcess(T item); @@ -849,7 +827,7 @@ itemWriter.write(items);]]> void onProcessError(T item, Exception e); -}]]> +} The beforeProcess method will be called before process on the @@ -884,15 +862,15 @@ itemWriter.write(items);]]> The writing of an item can be 'listened' to with the ItemWriteListener: - extends StepListener { + public interface ItemWriteListener<S> extends StepListener { - void beforeWrite(List items); + void beforeWrite(List<? extends S> items); - void afterWrite(List items); + void afterWrite(List<? extends S> items); - void onWriteError(Exception exception, List items); + void onWriteError(Exception exception, List<? extends S> items); -}]]> +} The beforeWrite method will be called before write on the @@ -932,8 +910,8 @@ itemWriter.write(items);]]> this reason, there is a separate interface for tracking skipped items: - extends StepListener { + + public interface SkipListener<T,S> extends StepListener { void onSkipInRead(Throwable t); @@ -941,7 +919,7 @@ itemWriter.write(items);]]> void onSkipInWrite(S item, Throwable t); -}]]> +} onSkipInRead will be called whenever an item is skipped while reading. It should be noted that rollbacks may @@ -1042,12 +1020,12 @@ itemWriter.write(items);]]> this class without having to write an adapter for the Tasklet interface: - - - - - -]]> + <bean id="myTasklet" class="org.springframework.batch.core.step.tasklet.TaskletAdapter"> + <property name="targetObject"> + <bean class="org.mycompany.FooDao"> + </property> + <property name="targetMethod" value="updateFoo" /> +</bean>
@@ -1062,7 +1040,7 @@ itemWriter.write(items);]]> project, is a Tasklet implementation with just such a responsibility: - public class FileDeletingTasklet implements Tasklet, InitializingBean { private Resource directory; @@ -1072,7 +1050,7 @@ itemWriter.write(items);]]> Assert.state(dir.isDirectory()); File[] files = dir.listFiles(); - for (int i = 0; i < files.length; i++) { + for (int i = 0; i < files.length; i++) { boolean deleted = files[i].delete(); if (!deleted) { throw new UnexpectedJobExecutionException("Could not delete file " + @@ -1089,7 +1067,7 @@ itemWriter.write(items);]]> public void afterPropertiesSet() throws Exception { Assert.notNull(directory, "directory must be set"); } -}]]> +} The above Tasklet implementation will delete all files within a given directory. It should be noted that the @@ -1097,21 +1075,21 @@ itemWriter.write(items);]]> that is left is to reference the Tasklet from the Step: - - - - - + <job id="taskletJob"> + <step id="deleteFilesInDir"> + <tasklet ref="fileDeletingTasklet"/> + </step> +</job> - - - - - - -]]> +<beans:bean id="fileDeletingTasklet" + class="org.springframework.batch.sample.tasklet.FileDeletingTasklet"> + <beans:property name="directoryResource"> + <beans:bean id="directory" + class="org.springframework.core.io.FileSystemResource"> + <beans:constructor-arg value="target/test-outputs/test-dir" /> + </beans:bean> + </beans:property> +</beans:bean>
@@ -1148,15 +1126,15 @@ itemWriter.write(items);]]> This can be achieved using the 'next' attribute of the step element: - - - - -]]>In the scenario above, 'step A' will execute first - because it is the first Step listed. If 'step A' - completes normally, then 'step B' will execute, and so on. However, if - 'step A' fails, then the entire Job will fail and - 'step B' will not execute. + <job id="job"> + <step id="stepA" parent="s1" next="stepB" /> + <step id="stepB" parent="s2" next="stepC"/> + <step id="stepC" parent="s3" /> +</job>In the scenario above, 'step A' will execute + first because it is the first Step listed. If + 'step A' completes normally, then 'step B' will execute, and so on. + However, if 'step A' fails, then the entire Job + will fail and 'step B' will not execute. With the Spring Batch namespace, the first step listed in the @@ -1214,14 +1192,14 @@ itemWriter.write(items);]]> The next element specifies a pattern to match and the step to execute next: - - - - - - - -]]> + <job id="job"> + <step id="stepA" parent="s1"> + <next on="*" to="stepB" /> + <next on="FAILED" to="stepC" /> + </step> + <step id="stepB" parent="s2" next="stepC" /> + <step id="stepC" parent="s3" /> +</job> The "on" attribute of a transition element uses a simple pattern-matching scheme to match the ExitStatus @@ -1269,7 +1247,7 @@ itemWriter.write(items);]]> it fails, and so on. The example above contains the following 'next' element: - ]]> + <next on="FAILED" to="stepB" /> At first glance, it would appear that the 'on' attribute references the BatchStatus of the @@ -1286,11 +1264,11 @@ itemWriter.write(items);]]> code needs to be different? A good example comes from the skip sample job within the samples project: - - - - -]]> + <step id="step1" parent="s1"> + <end on="FAILED" /> + <next on="COMPLETED WITH SKIPS" to="errorPrint1" /> + <next on="*" to="step2" /> +</step> The above step has three possibilities: @@ -1316,12 +1294,12 @@ itemWriter.write(items);]]> change the exit code based on the condition of the execution having skipped records: - public class SkipCheckingListener extends StepExecutionListenerSupport { public ExitStatus afterStep(StepExecution stepExecution) { String exitCode = stepExecution.getExitStatus().getExitCode(); - if (!exitCode.equals(ExitStatus.FAILED.getExitCode()) && - stepExecution.getSkipCount() > 0) { + if (!exitCode.equals(ExitStatus.FAILED.getExitCode()) && + stepExecution.getSkipCount() > 0) { return new ExitStatus("COMPLETED WITH SKIPS"); } else { @@ -1329,7 +1307,7 @@ itemWriter.write(items);]]> } } -}]]> +} The above code is a StepExecutionListener that first checks to make sure the Step was @@ -1357,7 +1335,7 @@ itemWriter.write(items);]]> after the following step executes, the Job will end: - ]]> + <step id="stepC" parent="s3"/> If no transitions are defined for a Step, then the Job's statuses will be defined as @@ -1416,14 +1394,14 @@ itemWriter.write(items);]]> fails, the Job will not be restartable (because the status is COMPLETED). - + <step id="step1" parent="s1" next="step2"> - - - - +<step id="step2" parent="s2"> + <end on="FAILED"/> + <next on="*" to="step3"/> +</step> -]]> +<step id="step3" parent="s3">
@@ -1447,14 +1425,14 @@ itemWriter.write(items);]]> Additionally, if step2 fails, and the Job is restarted, then execution will begin again on step2. - + <step id="step1" parent="s1" next="step2"> - - - - +<step id="step2" parent="s2"> + <fail on="FAILED" exit-code="EARLY TERMINATION"/> + <next on="*" to="step3"/> +</step> -]]> +<step id="step3" parent="s3">
@@ -1472,11 +1450,11 @@ itemWriter.write(items);]]> the job will then stop. Once it is restarted, execution will begin on step2. - - - + <step id="step1" parent="s1"> + <stop on="COMPLETED" restart="step2"/> +</step> -]]> +<step id="step2" parent="s2"/>
@@ -1489,7 +1467,7 @@ itemWriter.write(items);]]> JobExecutionDecider can be used to assist in the decision. - public class MyDecider implements JobExecutionDecider { public String decide(JobExecution jobExecution, StepExecution stepExecution) { if (someCondition) { return "FAILED"; @@ -1498,24 +1476,24 @@ itemWriter.write(items);]]> return "COMPLETED"; } } -}]]> +} In the job configuration, a "decision" tag will specify the decider to use as well as all of the transitions. - - + <job id="job"> + <step id="step1" parent="s1" next="decision" /> - - - - + <decision id="decision" decider="decider"> + <next on="FAILED" to="step2" /> + <next on="COMPLETED" to="step3" /> + </decision> - - - + <step id="step2" parent="s2" next="step3"/> + <step id="step3" parent="s3" /> +</job> -]]> +<beans:bean id="decider" class="com.MyDecider"/>
@@ -1532,16 +1510,95 @@ itemWriter.write(items);]]> elements such as the 'next' attribute or the 'next', 'end', 'fail', or 'pause' elements. - - - - - - - - - -]]> + <split id="split1" next="step4"> + <flow> + <step id="step1" parent="s1" next="step2"/> + <step id="step2" parent="s2"/> + </flow> + <flow> + <step id="step3" parent="s3"/> + </flow> +</split> +<step id="step4" parent="s4"/> +
+ +
+ Externalizing Flow Definitions and Dependencies Between + Jobs + + Part of the flow in a job can be externalized as a separate bean + definition, and then re-used. There are three ways to do this, and the + first is to simply declare the flow as a reference to one defined + elsewhere: + + <job id="job"> + <flow id="job1.flow1" parent="flow1" next="step3"/> + <step id="step3" parent="s3"/> +</job> + +<flow id="flow1"> + <step id="step1" parent="s1" next="step2"/> + <step id="step2" parent="s2"/> +</flow> + + The effect of defining an external flow like this is simply to + insert the steps from the external flow into the job as if they had been + declared inline. In this way many jobs can refer to the same template + flow and compose such templates into different logical flows. This is + also a good way to separate the integration testing of the individual + flows. + + The second form of an externalized flow is to use a + FlowStep. A FlowStep is an + implementation of the Step interface that + delegates processing to a flow defined as above with a + <flow/> element in XML. There is also support + for creating a FlowStep in XML directly: + + <job id="job"> + <step id="job1.flow1" flow="flow1" next="step3"/> + <step id="step3" parent="s3"/> +</job> + +<flow id="flow1"> + <step id="step1" parent="s1" next="step2"/> + <step id="step2" parent="s2"/> +</flow>The logic of execution of this job is the same + as the previous example, but the data stored in the job repository is + different: the Step "job1.flow1" gets its own + entry in the repository. This can be useful for monitoring and reporting + purposes, and moreover it can be used to give more structure to a partitioned step. + + The third form of an externalized flow is to use a + JobStep. A JobStep is + similar to a FlowStep, but actually creates and + launches a separate job execution for the steps in the flow specified. + Here is an example: + + <job id="jobStepJob" restartable="true"> + <step id="jobStepJob.step1" parent="jobStep"/> +</job> + +<job id="job" restartable="true">...</job> + +<bean id="jobStep" class="org.springframework.batch.core.step.job.JobStep"> + <property name="jobRepository" ref="jobRepository"/> + <property name="jobLauncher" ref="jobLauncher"/> + <property name="job" ref="job"/> + <property name="jobParametersExtractor"> + <bean class="org.springframework.batch.core.step.job.DefaultJobParametersExtractor"> + <property name="keys" value="input.file"/> + </bean> + </property> +</bean> + + Again this is useful when you want to have more granular options + for monitoring and reporting on jobs and steps. Using + JobStep is also often a good answer to the + question: "How do I create dependencies between jobs?". It is a good way + to break up a large system into smaller modules and control the flow of + jobs.
@@ -1555,11 +1612,11 @@ itemWriter.write(items);]]> Flat File resources can be configured using standard Spring constructs: - - -]]> + <bean id="flatFileItemReader" + class="org.springframework.batch.item.file.FlatFileItemReader"> + <property name="resource" + value="file://outputs/20070122.testStream.CustomerReportStep.TEMP.txt" /> +</bean> The above Resource will load the file from the file system location specified. Note that absolute locations have to @@ -1569,10 +1626,10 @@ itemWriter.write(items);]]> at runtime as a parameter to the job. This could be solved using '-D' parameters, i.e. a system property: - - -]]> + <bean id="flatFileItemReader" + class="org.springframework.batch.item.file.FlatFileItemReader"> + <property name="resource" value="${input.file.name}" /> +</bean> All that would be required for this solution to work would be a system argument (-Dinput.file.name="file://file.txt"). (Note that although @@ -1632,17 +1689,17 @@ itemWriter.write(items);]]> must be added explicitly, either by using the batch namespace: - <beans:beans xmlns="http://www.springframework.org/schema/beans" xmlns:batch="http://www.springframework.org/schema/batch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:schemaLocation="..."> + xsi:schemaLocation="..."> ... -]]> +</beans:beans> or by including a bean definition explicitly for theStep (but not both): - ]]> + <bean class="org.springframework.batch.core.scope.StepScope" /> - \ No newline at end of file +