diff --git a/src/models/Figures.ppt b/src/models/Figures.ppt index bcb2bd613..abe5b1139 100644 Binary files a/src/models/Figures.ppt and b/src/models/Figures.ppt differ diff --git a/src/models/diagrams.ppt b/src/models/diagrams.ppt deleted file mode 100755 index 2b6a1392a..000000000 Binary files a/src/models/diagrams.ppt and /dev/null differ diff --git a/src/site/docbook/reference/domain.xml b/src/site/docbook/reference/domain.xml index 6a9b99cbb..b0a3b9081 100644 --- a/src/site/docbook/reference/domain.xml +++ b/src/site/docbook/reference/domain.xml @@ -1,7 +1,7 @@ - + The Domain Language of Batch To any experienced batch architect, the overall concepts of batch diff --git a/src/site/docbook/reference/execution.xml b/src/site/docbook/reference/execution.xml deleted file mode 100644 index d2fa94978..000000000 --- a/src/site/docbook/reference/execution.xml +++ /dev/null @@ -1,1771 +0,0 @@ - - - - Configuring and Executing A Job - -
- Introduction - - In Chapter 2, the overall architecture design was discussed, using - the following diagram as a guide: - - - - - - - - - - - - When viewed from left to right, the diagram describes a basic flow - for the execution of a batch job: - - - - A Scheduler kicks off a job script (usually some form of shell - script) - - - - The script sets up the classpath appropriately, and starts the - Java process. In most cases, using - CommandLineJobRunner as the entry point - - - - The JobRunner finds the Job using the - JobLocator, pulls together the - JobParameters and launches the - Job - - - - The JobLauncher retrieves a - JobExecution from the - JobRepository, and executes the - Job - - - - The Job executes each - Step in sequence. - - - - The Step calls read on the - ItemReader, handing the resulting item to the - ItemWriter until null is returned, periodically - committing and storing status in the - JobRepository. - - - - When execution is complete, the Step - returns control back to the Job, and if no more - steps exist, control is returned back to the original caller, in this - case, the scheduler. - - - - This flow is perhaps a bit overly simplified, but describes the - complete flow in the most basic terms. From here, each tier will be - described in detail, using actual implementations and examples. -
- -
- Run Tier - - As its name suggests, this tier is entirely concerned with actually - running the job. Regardless of whether the originator is a Scheduler or an - HTTP request, a Job must be obtained, parameters must be parsed, and - eventually a JobLauncher called: - - - - - - - - - - - -
- Running Jobs from the Command Line - - For users that want to run their jobs from an enterprise - scheduler, the command line is the primary interface. This is because - most schedulers (with the exception of Quartz unless using the - NativeJob) work directly with operating system - processes, primarily kicked off with shell scripts. There are many ways - to launch a Java process besides a shell script, such as Perl, Ruby, or - even 'build tools' such as ant or maven. However, because most people - are familiar with shell scripts, this example will focus on them. - -
- The CommandLineJobRunner - - Because the script launching the job must kick off a Java - Virtual Machine, there needs to be a class with a main method to act - as the primary entry point. Spring Batch provides an implementation - that serves just this purpose: - CommandLineJobRunner. It's important to note - that this is just one way to bootstrap your application, but there are - many ways to launch a Java process, and this class should in no way be - viewed as definitive. It performs four tasks: - - - - Loads the appropriate Application Context - - - - Parses command line arguments into JobParameters - - - - Locates the appropriate job based on arguments - - - - Uses the JobLauncher provided in the application context to - launch the job. - - - - All of these tasks are accomplished based completely upon the - arguments passed in. The following are required arguments: - - - CommandLineJobRunner arguments - - - - - jobPath - - The location of the XML file that will be used to - create an ApplicationContext. This file - should contain everything needed to run the complete - Job - - - - jobName - - The name of the job to be run. - - - -
- - These arguments must be passed in with the path first and the - name second. All arguments after these are considered to be - JobParameters and must be in the format of 'name=value': - - bash$ java CommandLineJobRunner endOfDayJob.xml endOfDay schedule.date(date)=2008/01/01 - - In most cases you would want to use a manifest to declare your - main class in a jar, but for simplicity, the class was used directly. - This example is using the same 'EndOfDay' example from Chapter 2. The - first argument is 'endOfDayJob.xml', which is the Spring - ApplicationContext containing the Job. The - second argument, 'endOfDay' represents the job name. The final - argument, 'schedule.date=01-01-2008' will be converted into - JobParameters. An example of the XML - configuration is below: - - <bean id="endOfDay" - class="org.springframework.batch.core.job.SimpleJob"> - <property name="steps"> - <bean id="step1" parent="simpleStep" /> - <!-- Step details removed for clarity --> - </property> - </bean> - - <!-- Launcher details removed for clarity --> - <bean id="jobLauncher" - class="org.springframework.batch.core.launch.support.SimpleJobLauncher" /> - - This example is overly simplistic, since there are many more - requirements to a run a batch job in Spring Batch in general, but it - serves to show the two main requirements of the - CommandLineJobRunner: - Job and - JobLauncher -
- -
- ExitCodes - - When launching a batch job from the command-line, it is often - from an enterprise scheduler. Most schedulers are fairly dumb, and - work only at the process level. Meaning, they only know about some - operating system process such as a shell script that they're invoking. - In this scenario, the only way to communicate back to the scheduler - about the success or failure of a job is through return codes. A - number is returned to a scheduler that is told how to interpret the - result. In the simple case: 0 is success and 1 is failure. However, - there may be scenarios such as: If job A returns 4 kick off job B, if - it returns 5 kick off job C. This type of behavior is configured at - the scheduler level, but it is important that a processing framework - such as Spring Batch provide a way to return a numeric representation - of of the 'Exit Code' for a particular batch job. In Spring Batch this - is encapsulated within an ExitStatus, which is - covered in more detail in Chapter 5. For the purposes of discussing - exit codes, the only important thing to know is that an - ExitStatus has an exit code property that is - set by the framework (or the developer) and is returned as part of the - JobExecution returned from the - JobLauncher. The - CommandLineJobRunner converts this string value - to a number using the ExitCodeMapper - interface: - - public interface ExitCodeMapper { - - public int intValue(String exitCode); -} - - The essential contract of an - ExitCodeMapper is that, given a string exit - code, a number representation will be returned. The default - implementation used by the job runner is the SimpleJvmExitCodeMapper - that returns 0 for completion, 1 for generic errors, and 2 for any job - runner errors such as not being able to find a - Job in the provided context. If anything more - complex than the 3 values above is needed, then a custom - implementation of the ExitCodeMapper interface - must be supplied. Because the - CommandLineJobRunner is the class that creates - an ApplicationContext, and thus cannot be - 'wired together', any values that need to be overwritten must be - autowired. This means that if an implementation of - ExitCodeMapper is found within the BeanFactory, - it will be injected into the runner after the context is created. All - that needs to be done to provide your own - ExitCodeMapper is to declare the implementation - as a root level bean, and ensure it's part of the - ApplicationContext that is loaded by the - runner. -
-
-
- -
- Job Tier - - The Job Tier is responsible for the overall execution of a batch - job. It sequentially executes batch steps, ensuring that all steps are in - the correct state and all appropriate policies are enforced: - - - - - - - - - - - - The job tier is entirely concerned with maintaining the three job - stereotypes: Job, - JobInstance, and - JobExecution. The - JobLauncher interacts with the - JobRepository in order to create a - JobExecution, and the Job - stores the JobExecution using the - repository. - -
- SimpleJobLauncher - - The most basic implementation of the - JobLauncher interface is the SimpleJobLauncher. - It's only required dependency is a JobRepository, - in order to obtain an execution: - - <bean id="jobLauncher" - class="org.springframework.batch.execution.launch.SimpleJobLauncher"> - <property name="jobRepository" ref="jobRepository" /> - </bean> - - Once a JobExecution is obtained, it is - passed to the execute method of Job, ultimately - returning the JobExecution to the caller: - - - - - - - - - - - - The sequence is straightforward, and works well when launched from - a scheduler, but causes issues when trying to launch from an HTTP - request. In this scenario, the launching needs to be done - asynchronously, so that the SimpleJobLauncher - returns immediately to it's caller. This is because it is not good - practice to keep an HTTP request open for the amount of time needed by - long running processes such as batch. An example sequence is - below: - - - - - - - - - - - - The SimpleJobLauncher can easily be - configured to allow for this scenario by configuring a - TaskExecutor: - - <bean id="jobLauncher" - class="org.springframework.batch.execution.launch.SimpleJobLauncher"> - <property name="jobRepository" ref="jobRepository" /> - <property name="taskExecutor"> - <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" /> - </property> - </bean> - - Any implementation of the spring - TaskExecutor interface can be used to control how - jobs are asynchronously executed. - -
- Stopping a Job - - One of the most common reasons for wanting to launching a - job asynchronously is to be able to gracefully - stop it. This can be done through the - JobExecution returned by the - JobLauncher: - - JobExecution jobExecution = launcher.run(getJob(), jobParameters); - - //give job adequate time to start - Thread.sleep(1000); - - assertEquals(BatchStatus.STARTED, jobExecution.getStatus()); - assertTrue(jobExecution.isRunning()); - - jobExecution.stop(); - - //give job time to stop - Thread.sleep(1000); - - assertEquals(BatchStatus.STOPPED, jobExecution.getStatus()); - assertFalse(jobExecution.isRunning()); - - The shutdown is not immediate, since there is no way to force - immediate shutdown, especially if the execution is currently in - developer code that the framework has no control over, such as a - business service. What it does mean, is that as soon as control is - returned back to the framework, it will set the status of the current - StepExecution to - BatchStatus.STOPPED, save it, then do the same - for the JobExecution before finishing. -
-
- -
- SimpleJobRepository - - The SimpleJobRepository is the only provided implementation of the - JobRepository interface. It completely manages - the various batch domain objects and ensures they are created and - persisted correctly. The SimpleJobRepository uses - three different DAO interfaces for the three major domain types it - stores: JobInstanceDao, - JobExecutionDao, and - StepExecutionDao. The repository delegates to - these DAOs to both persist the various domain objects and query for them - during initialization. The following configuration shows a - SimpleJobRepository configured with JDBC DAOs: - - <bean id="jobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository"> - <constructor-arg ref="jobInstanceDao" /> - <constructor-arg ref="jobExecutionDao" /> - <constructor-arg ref="stepExecutionDao" /> - </bean> - - <bean id="jobInstanceDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobInstanceDao" > - <property name="jdbcTemplate" ref="jdbcTemplate" /> - <property name="jobIncrementer" ref="jobIncrementer" /> - </bean> - - <bean id="jobExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcJobExecutionDao" > - <property name="jdbcTemplate" ref="jdbcTemplate" /> - <property name="jobExecutionIncrementer" ref="jobExecutionIncrementer" /> - </bean> - - <bean id="stepExecutionDao" class="org.springframework.batch.core.repository.support.dao.JdbcStepExecutionDao" > - <property name="jdbcTemplate" ref="jdbcTemplate" /> - <property name="stepExecutionIncrementer" ref="stepExecutionIncrementer" /> - </bean> - - <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate" > - <property name="dataSource" ref="dataSource" /> - </bean> - - The configuration above isn't quite complete, each DAO - implementation makes a reference to a Spring - DataFieldMaxValueIncrementer. - JobInstance, JobExecution, - and StepExecution each have unique IDs, and the - incrementers are used to create them. - -
- JobRepositoryFactoryBean - - Including the incrementers, which must be database specific, the - configuration above is verbose. In order to make this more manageable, - the framework provides a FactoryBean for - convenience: JobRepositoryFactoryBean. - - <bean id="jobRepository" - class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean" - <property name="databaseType" value="hsql" /> - <property name="dataSource" ref="dataSource" /> - <property name="transactionManager" ref="transactionManager" /> - </bean> - - The databaseType property indicates the type of incrementer that - must be used. Options include: "db2", "db2zos", "derby", "hsql", - "mysql", "oracle", and "postgres". -
- -
- In-Memory Repository - - There are scenarios in which you may not want to persist your - domain objects to the database. One reason may be speed, storing - domain objects at each commit point takes extra time. Another reason - may be that you just don't need to persist status for a particular - job. Spring batch provides a solution: - - <bean id="simpleJobRepository" class="org.springframework.batch.core.repository.support.SimpleJobRepository"> - <constructor-arg ref="mapJobInstanceDao" /> - <constructor-arg ref="mapJobExecutionDao" /> - <constructor-arg ref="mapStepExecutionDao" /> - </bean> - - <bean id="mapJobInstanceDao" - class="org.springframework.batch.core.repository.dao.MapJobInstanceDao" /> - - <bean id="mapJobExecutionDao" - class="org.springframework.batch.core.repository.dao.MapJobExecutionDao" /> - - <bean id="mapStepExecutionDao" - class="org.springframework.batch.core.repository.dao.MapStepExecutionDao" /> - - The Map* DAO implementations store the batch artifacts in a - transactional map. So, the repository and DAOs may still be used - normally, and are transactionally sound, but their contents will be - lost when the class is destroyed. - - There is also a separate FactoryBean for the in-memory - JobRepository, which reduces the amount of - configuration required: - - <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" /> - -
- Transaction Configuration For the JobRepository - - If either of the JobRepository factory beans are used, - transactional advice will be automatically created around the - repository. This is to ensure that the batch meta data, including - state that is necessary for restarts after a failure, is persisted - correctly. The behaviour of the framework is not well defined if the - repository methods are not transactional. The isolation level in the - create* method attributes is specified separately to - ensure that when jobs are launched there if two processes are trying - to launch the same job at the same time, only one will succeed. The - default isolation level for that method is SERIALIZABLE, which is - quite aggressive: READ_COMMITTED would work just as well; - READ_UNCOMMITTED would be fine if two processes are not likely to - collide in this way. However, since a call to the - create* method is quite short, it is unlikely - that the SERIALIZED will cause problems, as long as the database - platform supports it. However, this can be overriden in the factory - beans: - - <bean id="jobRepository" - class="org.springframework.batch.execution.repository.JobRepositoryFactoryBean" - <property name="databaseType" value="hsql" /> - <property name="dataSource" ref="dataSource" /> - <property name="transactionManager" ref="transactionManager" /> - <property name="IsolationLevelForCreate" value="ISOLATION_REPEATABLE_READ" /> - </bean> - - If the factory beans aren't used then it is also essential to - configure the transactional behaviour of the repository using - AOP: - - - <aop:config> - <aop:advisor - pointcut="execution(* org.springframework.batch.core..*Repository+.*(..))" - <advice-ref="txAdvice" /> - </aop:config> - - <tx:advice id="txAdvice" transaction-manager="transactionManager"> - <tx:attributes> - <tx:method name="*" /> - </tx:attributes> - </tx:advice> - - - - This fragment can be used as is, with almost no changes. - Remember also to include the appropiate namespace declarations and - to make sure spring-tx and spring-aop (or the whole of spring) is on - the classpath. -
- -
- Recommendations for Indexing Meta Data Tables - - Spring Batch provides DDL samples for the meta-data tables in - the Core jar file for several common database platforms. Index - declarations are not included in that DDL because there are too many - variations in how users may want to index dependeing on their - precise platform, local conventions and also the business - requirements of how the jobs will be operated. The table below - provides some indication as to which columns are going to be used in - a WHERE clause by the Dao ipmlementations provided by Spring Batch, - and how frequently they might be used, so that individual projects - can make up their own minds about indexing. - - - Where clauses in SQL statements (exluding primary keys) and - their approximate frequency of use. - - - - - Default Table Name - - Where Clause - - Frequency - - - - BATCH_JOB_INSTANCE - - JOB_NAME = ? and JOB_KEY = ? - - Every time a job is launched - - - - BATCH_JOB_EXECUTION - - JOB_INSTANCE_ID = ? - - Every time a job is restarted - - - - BATCH_EXECUTION_CONTEXT - - EXECUTION_ID = ? and KEY_NAME = ? - - On commit interval, a.k.a. chunk - - - - BATCH_STEP_EXECUTION - - VERSION = ? - - On commit interval, a.k.a. chunk (and at start and - end of step) - - - - BATCH_STEP_EXECUTION - - STEP_NAME = ? and JOB_EXECUTION_ID = ? - - Before each step execution - - - -
-
-
-
- -
- SimpleJob - - The only current implementation of the Job - interface is SimpleJob. Since a - Job is just a simple loop through a list of - Steps, this implementation should be sufficient for the majority of - needs. It has only three required dependencies: a name, - JobRepository, and a list of Steps. - - <bean id="footballJob" - class="org.springframework.batch.core.job.SimpleJob"> - <property name="steps"> - <list> - <!-- Step Bean details ommitted for clarity --> - <bean id="playerload" parent="simpleStep" /> - <bean id="gameLoad" parent="simpleStep" /> - <bean id="playerSummarization" parent="simpleStep" /> - </list> - </property> - <property name="jobRepository" ref="jobRepository" /> - </bean> - - Each Step will be executed in sequence - until all have completed successfully. Any Step that fails will cause - the entire job to fail. - -
- Restartability - - One key concern when execution a batch job, is what happens when - a failed job is restarted? A Job is considered to have been - 'restarted' if the same JobInstance has more than one JobExecution. - Ideally, all jobs should be able to start up where they left off, but - there are scenarios where this is not possible. It is entirely up to the developer to ensure that a new - instance is always created in this scenario. However, - Spring Batch does provide some help. If a Job should never be - restarted, but should always be run as part of a new - JobInstance, then the restartable property may - be set to 'false': - - <bean id="footballJob" - class="org.springframework.batch.core.job.SimpleJob"> - <property name="steps"> - <list> - <!-- Step Bean details ommitted for clarity --> - <bean id="playerload" parent="simpleStep" /> - <bean id="gameLoad" parent="simpleStep" /> - <bean id="playerSummarization" parent="simpleStep" /> - </list> - </property> - <property name="jobRepository" ref="jobRepository" /> - <property name="restartable" value="false" /> - </bean> - - To phrase it another way, setting restartable to false means - "this Job does not support being started again". Restarting a Job that - is not restartable will cause a - JobRestartException to be thrown: - - Job job = new SimpleJob(); - job.setRestartable(false); - - JobParameters jobParameters = new JobParameters(); - - JobExecution firstExecution = jobRepository.createJobExecution(job, jobParameters); - jobRepository.saveOrUpdate(firstExecution); - - try { - jobRepository.createJobExecution(job, jobParameters); - fail(); - } - catch (JobRestartException e) { - // expected - } - - This snippet of JUnit code shows how attempting to create a - JobExecution the first time for a non - restartable job will cause no issues. However, - the second attempt will throw a - JobRestartException. -
- -
- Intercepting Job execution - - During the course of the execution of a - Job, it may be useful to be notified of various - events in its lifecycle so that custom code may be executed. The - SimpleJob allows for this by calling a - JobListener at the appropriate time: - - public interface JobListener { - - void beforeJob(JobExecution jobExecution); - - void afterJob(JobExecution jobExecution); - - void onError(JobExecution jobExecution, Throwable e); - - void onInterrupt(JobExecution jobExecution); - } - - Listeners can be added to a SimpleJob via - the setJobListeners property: - - <bean id="footballJob" - class="org.springframework.batch.core.job.SimpleJob"> - <property name="steps"> - <list> - <!-- Step Bean details ommitted for clarity --> - <bean id="playerload" parent="simpleStep" /> - <bean id="gameLoad" parent="simpleStep" /> - <bean id="playerSummarization" parent="simpleStep" /> - </list> - </property> - <property name="jobRepository" ref="jobRepository" /> - <property name="jobListeners"> - <bean class="org.springframework.batch.core.listener.JobListenerSupport" /> - </property> - </bean> -
-
- -
- JobFactory and Stateful Components in Steps - - Unlike many traditional Spring applications, many of the - components of a batch application are stateful, the file readers and - writers are obvious examples. The recommended way to deal with this is - to create a fresh ApplicationContext for each job - execution. If the Job is launched from the - command line with CommandLineJobRunner this is - trivial. For more complex launching scenarios, where jobs are executed - in parallel or serially from the same process, some extra steps have to - be taken to ensure that the ApplicationContext is - refreshed. This is preferable to using prototype scope for the stateful - beans because then they would not receive lifecycle callbacks from the - container at the end of use. (e.g. through destroy-method in XML) - - The strategy provided by Spring Batch to deal with this scenario - is the JobFactory, and the samples provide an - example of a specialized implementation that can load an - ApplicationContext and close it properly when the - job is finished. A relevant examples is - ClassPathXmlApplicationContextJobFactory and its - use in the adhoc-job-launcher-context.xml and the - quartz-job-launcher-context.xml, which can be found in the - Samples project. -
-
- -
- Application Tier - - The Application tier is entirely concerned with the actual - processing of input: - - - - - - - - - - - -
- StepHandlerStep - - The figure above shows a simple 'item-oriented' execution flow. - One item is read in from an ItemReader, and then - handed to an ItemWriter, until their are no more - items left. When processing first begins, a transaction is started and - periodically committed until the Step is - complete. Given these basic requirements, the - StepHandlerStep requires the following - dependencies, at a minimum: - - - - ItemReader - The - ItemReader that provides items for - processing. - - - - ItemWriter - The - ItemWriter that processes the items provided - by the ItemReader. - - - - PlatformTransactionManager - Spring - transaction manager that will be used to begin and commit - transactions during processing. - - - - JobRepository - The - JobRepository that will be used to - periodically store the StepExecution and - ExecutionContext during processing (just - before committing). - - - -
- SimpleStepFactoryBean - - Despite the relatively short list of required dependencies for - an StepHandlerStep, it is an extremely complex - class that can potentially contain many collaborators. In order to - ease configuration, a SimpleStepFactoryBean can - be used: - - <bean id="simpleStep" - class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > - <property name="transactionManager" ref="transactionManager" /> - <property name="jobRepository" ref="jobRepository" /> - <property name="itemReader" ref="itemReader" /> - <property name="itemWriter" ref="itemWriter" /> - </bean> - - The configuration above represents the only required - dependencies of the factory bean. Attempting to instantiate a - SimpleStepFactoryBean without at least those - four dependencies will result in an exception being thrown during - construction by the Spring container. -
- -
- Configuring a CommitInterval - - As mentioned above, the StepHandlerStep - reads in and writes out items, periodically commiting using the - supplied PlatformTransactionManager. By - default, it will commit after each item has been written. This is less - than ideal in many situations, since beginning and commiting a - transaction is expensive. Ideally, you would like to process as many - items as possible in each transaction, which is completely dependant - upon the type of data being processed and the resources that are being - interacted with. For this reason, the number of items that are - processed within a commit can be set as the commit interval: - - <bean id="simpleStep" - class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > - <property name="transactionManager" ref="transactionManager" /> - <property name="jobRepository" ref="jobRepository" /> - <property name="itemReader" ref="itemReader" /> - <property name="itemWriter" ref="itemWriter" /> - <property name="commitInterval" value="10" /> - </bean> - - In this example, 10 items will be processed within each - transaction. At the beginning of processing a transaction is begun, - and each time read is called on the - ItemReader, a counter is incremented. When it - reaches 10, the transaction will be committed. -
- -
- Configuring a Step for Restart - - Earlier in this chapter, restarting a Job - was discussed. Restart has numerous impacts on steps, and as such may - require some specific configuration. - -
- Setting a StartLimit - - There are many scenarios where you may want to control the - number of times a Step may be started. An - example is a Step that may be run only once, - usually because it invalidates some resource that must be fixed - manually before it can be run again. This is configurable on the - step level, since different steps have different requirements. One - Step that may only be executed once can exist as part of the same - Job as Step that can - be run infinitely. Below is an example start limit - configuration: - - <bean id="simpleStep" - class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > - <property name="transactionManager" ref="transactionManager" /> - <property name="jobRepository" ref="jobRepository" /> - <property name="itemReader" ref="itemReader" /> - <property name="itemWriter" ref="itemWriter" /> - <property name="commitInterval" value="10" /> - <property name="startLimit" value="1" /> - </bean> - - The simple step above can be run only once. Attempting to run - it again will cause an exception to be thrown. It should be noted - that the default value for startLimit is - Integer.MAX_VALUE. -
- -
- Restarting a completed step - - In the case of a restartable job, there may be one or more - steps that should always be run, regardless of whether or not they - were successful the first time. An example might be a validation - step, or a step that cleans up resources before processing. During - normal processing of a restarted job, any step with a status of - 'COMPLETED', meaning it has already been completed successfully, - will be skipped. Setting allowStartIfComplete to true overrides this - so that the step will always run: - - <bean id="simpleStep" - class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > - <property name="transactionManager" ref="transactionManager" /> - <property name="jobRepository" ref="jobRepository" /> - <property name="itemReader" ref="itemReader" /> - <property name="itemWriter" ref="itemWriter" /> - <property name="commitInterval" value="10" /> - <property name="startLimit" value="1" /> - <property name="allowStartIfComplete" value="true" /> - </bean> -
- -
- Step restart configuration example - - <bean id="footballJob" - class="org.springframework.batch.core.job.SimpleJob"> - <property name="steps"> - <list> - <!-- Step Bean details ommitted for clarity --> - <bean id="playerload" parent="simpleStep" /> - <bean id="gameLoad" parent="simpleStep" > - <property name="allowStartIfComplete" value="true" /> - </bean> - <bean id="playerSummarization" parent="simpleStep" > - <property name="startLimit" value="2" /> - </bean> - </list> - </property> - <property name="jobRepository" ref="jobRepository" /> - <property name="restartable" value="true" /> - </bean> - - The above example configuration is for a job that loads in - information about football games and summarizes them. It contains - three steps: playerLoad, gameLoad, and playerSummarization. The - playerLoad Step loads player information from - a flat file, while the gameLoad - Step does the same for games. The final - Step, playerSummarization, then summarizes - the statistics for each player based upon the provided games. It is - assumed that the file loaded by 'playerLoad' must be loaded only - once, but that 'gameLoad' will load any games found within a - particular directory, deleting them after they have been - successfully loaded into the database. As a result, the playerLoad - Step contains no additional configuration. It - can be started almost limitlessly, and if complete will be skipped. - The 'gameLoad' Step, however, needs to be run - everytime, in case extra files have been dropped since it last - executed, so it has 'allowStartIfComplete' set to 'true' in order to - always be started. (It is assumed that the database tables games are - loaded into has a process indicator on it, to ensure new games can - be properly found by the summarization step) The summarization - step, which is the most important in the - Job, is configured to have a start limit of - 3. This is useful in case it continually fails, a new exit code will - be returned to the operators that control job execution, and it - won't be allowed to start again until manual intervention has taken - place. - - - This job is purely for example purposes and is not the same - as the footballJob found in the samples project. - - - Run 1: - - - - playerLoad is executed and completes successfully, adding - 400 players to the 'PLAYERS' table. - - - - gameLoad is executed and processes 11 files worth of game - data, loading their contents into the 'GAMES' table. - - - - playerSummarization begins processing and fails after 5 - minutes. - - - - Run 2: - - - - playerLoad is not run, since it has already completed - succesfully, and allowStartIfComplete is false (the - default). - - - - gameLoad is executed again and processes another 2 files, - loading their contents into the 'GAMES' table as well (with a - process indicator indicating they have yet to be - processed) - - - - playerSummarization begins processing of all remaining - game data (filtering using the process indicator) and fails - again after 30 minutes. - - - - Run 3: - - - - playerLoad is not run, since it has already completed - succesfully, and allowStartIfComplete is false (the - default). - - - - gameLoad is executed again and processes another 2 files, - loading their contents into the 'GAMES' table as well (with a - process indicator indicating they have yet to be - processed) - - - - playerSummarization is not start, and the job is - immeadiately killed, since this is the third execution of - playerSummarization, and it's limit is only 2. The limit must - either be raised, or the Job must be - executed as a new JobInstance. - - -
-
- -
- Configuring Skip Logic - - There are many scenarios where errors encountered while - processing should not result in Step failure, - but should be skipped instead. This is usually a decision that must be - made by someone who understands the data itself and what meaning it - has. Financial data, for example, may not be skippable because it - results in money being transferred, which needs to be completely - accurate. Loading in a list of vendors, on the other hand, might allow - for skips, since a vendor not being loaded because it was formatted - incorrectly, or missing necessary information, won't cause issues. - Usually these bad records are logged as well, which will be covered - later when discussing listeners. Configuring skip handling requires - using a new factory bean: - SkipLimitStepFactoryBean <bean id="skipSample" - class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> - <property name="skipLimit" value="10" /> - <property name="itemReader" ref="flatFileItemReader" /> - <property name="itemWriter" ref="itemWriter" /> - <property name="skippableExceptionClasses" - value="org.springframework.batch.item.file.FlatFileParseException"> - </property> - </bean> - - In this example, a FlatFileItemReader is - used, and if at any point a FlatFileParseException is thrown, it will - be skipped and counted against the total skip limit of 10. It should - be noted that any failures encountered while reading will not count - against the commit interval. In other words, the commit interval is - only incremented on writes (regardless of success or failure). -
- -
- One problem with the example above is that any other exception - besides a FlatFileParseException will cause the - Job to fail. In certain scenarios this may be - the correct behaviour, however, in certain scenarios it may be easier - to identify which exceptions should cause failure and skip everything - else: <bean id="skipSample" - class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> - <property name="skipLimit" value="10" /> - <property name="itemReader" ref="flatFileItemReader" /> - <property name="itemWriter" ref="itemWriter" /> - <property name="skippableExceptionClasses" - value="java.lang.Exception"> - <property name="fatalExceptionClasses" - value="java.io.FileNotFoundException"> - </property> - </befan> - - By setting the skippable exceptions to - java.lang.Exception, any exception that is - thrown will be skipped. However, the second list, - 'fatalExceptionClasses', contains specific exceptions that should be - fatal if encountered. -
- -
- Configuring Retry Logic - - In most cases you want an Exception to cause either a skip or - Step failure. However, not all exceptions are - deterministic. If a FlatFileParseException is encountered while - reading, it will always be thrown for that record. Resseting the - ItemReader will not help. However, for other - exceptions, such as a - DeadlockLoserDataAccessException, which - indicates that the current process has attempted to update a record - that another process holds a lock on, waiting and trying again might - result in success. In this case, retry should be configured: - - <bean id="step1" - class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> - <property name="itemReader" ref="itemGenerator" /> - <property name="itemWriter" ref="itemWriter" /> - <property name="retryLimit" value="3" /> - <property name="retryableExceptionClasses" value="org.springframework.dao.DeadlockLoserDataAccessException" /> - </bean> - - The SkipLimitStepFactoryBean requires a - limit for the number of times an individual item can be retried, and a - list of Exceptions that are 'retryable'. -
- -
- Controlling rollback - - By default, regardless of retry or skip, any exceptions thrown - from the ItemWriter will cause the transaction - controlled by the Step to rollback. If skip is - configured as described above, exceptions thrown from the - ItemReader will not cause a rollback. However, - there are many scenarios in which exceptions thrown from the - ItemWriter should not cause a rollback because - no action has taken place to invalidate the transaction. For this - reason, the SkipLimitStepFactoryBean can be - configured with a list of exceptions that should not cause - rollback: - - <bean id="step2" - class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> - <property name="commitInterval" value="2" /> - <property name="skipLimit" value="1" /> - <!-- No rollback for exceptions that are marked with "+" in the tx attributes --> - <property name="transactionAttribute" - value="+org.springframework.batch.item.validator.ValidationException" /> - <property name="itemReader" - ref="tradeSqlItemReader" /> - <property name="itemWriter" - ref="itemTrackingWriter" /> - </bean> - - The TransactionAttribute property above - can be used to control multiple other settings such as isolation and - propagation behaviour. More information on setting transaction - attributes can be found in the spring core documentation. -
- -
- Registering ItemStreams with the Step - - The step has to take care of ItemStream - callbacks at the necessary points in its lifecycle. This is vital if a - step fails, and might need to be restarted, because the - ItemStream interface is where the step gets the - information it needs about persistent state between executions. The - factory beans that Spring Batch provides for convenient configuration - of Step instances have features that allow - streams to be registered with the step when it is configured. - - If the ItemReader or - ItemWriter themselves implement the ItemStream - interface, then these will be registered automatically. Any other - streams need to be registered separately. This is often the case where - there are indirect dependencies, like delegates being injected into - the reader and writer. To register these they can be injected into the - factory beans through the streams property, as illustrated - below: - - <bean id="step1" - class="org.springframework.batch.core.step.item.SkipLimitStepFactoryBean"> - <property name="streams" ref="fileItemReader" /> - <property name="itemReader"> - <bean - class="org.springframework.batch.item.validator.ValidatingItemReader"> - <property name="itemReader" ref="itemReader" /> - <property name="validator" ref="fixedValidator" /> - </bean> - </property> - ... -</bean> - - In the example above the main item reader is being set up to - delegate to a bean called "fileItemReader", which itself is being - registered as a stream directly. The step will now be restartable and - the state of the reader will be correctly persisted in case of a - failure. -
- -
- Intercepting Step Execution - - Just as with the Job, there are many - events during the execution of a Step that a - user may need notification of. For example, if writing out to a flat - file that requires a footer, the ItemWriter - needs to be notified when the Step has been - completed, so that it can write the footer. This can be accomplished - with one of many Step scoped listeners. - -
- StepExecutionListener - - StepExecutionListener represents the - most generic listener for Step execution. It - allows for notification before a Step is - started, after it has completed, and if any errors are encountered - during processing: - - public interface StepExecutionListener extends StepListener { - - void beforeStep(StepExecution stepExecution); - - ExitStatus onErrorInStep(StepExecution stepExecution, Throwable e); - - ExitStatus afterStep(StepExecution stepExecution); -} - - ExitStatus is the return type of - onErrorInStep and - afterStep in order to allow listeners the - chance to modify the exit code that is returned upon completion of a - Step. A - StepExecutionListener can be applied to any - step factory bean via the listeners property: - - <bean id="simpleStep" - class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > - <property name="transactionManager" ref="transactionManager" /> - <property name="jobRepository" ref="jobRepository" /> - <property name="itemReader" ref="itemReader" /> - <property name="itemWriter" ref="itemWriter" /> - <property name="commitInterval" value="10" /> - <property name="listeners" ref="stepListener" /> - </bean> - - Because all listeners extend the - StepListener interface, they all may be - applied to factory beans in the same way. -
- -
- ChunkListener - - A chunk is defined as the items processed within the scope of - a transaction. Committing a transaction commits a 'chunk'. It may be - useful to be nofied before and after a chunk has completed, in which - case the ChunkListener interface may be - used: - - public interface ChunkListener extends StepListener { - - void beforeChunk(); - - void afterChunk(); - } - - The beforeChunk method is called - after the transaction is started, but before - read is called on the - ItemReader. Conversely, - afterChunk is called after the last call to - write on the - ItemWriter, but before the chunk has been - committed. -
- -
- ItemReadListener - - When discussing skip logic above, it was mentioned that it may - be beneficial to log out skipped records, so that they can be deal - with later. In the case of read errors, this can be done with an - ItemReaderListener: public interface ItemReadListener extends StepListener { - - void beforeRead(); - - void afterRead(Object item); - - void onReadError(Exception ex); -} - - The beforeRead method will be called - before each call to read on the - ItemReader. The - afterRead method will be called after each - successful call to read, and will be passed - the item that was read. If there was an error while reading, the - onReadError method will be called. The - exception encounterd will be provided so that it can be - logged. -
- -
- ItemWriteListener - - Just as with the ItemReaderListener, the writing of an item - can be 'listened' to: - - public interface ItemWriteListener extends StepListener { - - void beforeWrite(Object item); - - void afterWrite(Object item); - - void onWriteError(Exception ex, Object item); -} - - The beforeWrite method will be called - before write on the - ItemWriter, and is handed the item that will - be written. The afterWrite method will be - called after the item has been succesfully writen. If there was an - error while writing, the onWriteError - method will be called. The exception encountered and the item that - was attempted to be written will be provided, so that they can be - logged. -
- -
- SkipListener - - Both ItemReadListener and - ItemWriteListner provide a mechanism for - being notified of errors, but neither one will inform you that a - record has actually been skipped. - onWriteError, for example, will be called - even if an item is retried and successful. For this reason, there is - a separate interface for tracking skipped items: - - - public interface SkipListener extends StepListener { - - void onSkipInRead(Throwable t); - - void onSkipInWrite(Object item, Throwable t); - } - - - - onSkipInRead will be called whenever - an item is skipped while reading. It should be noted that rollbacks - may cause the same item to be registered as skipped more than once. - onSkipInWrite will be called when an item - is skipped while writing. Because the item has been read - successfully (and not skipped), it is also provided the item itself - as an argument. -
-
-
- -
- TaskletStep - - Item oriented processing is not the only way to process in a - Step. What if a Step must - consist as a simple storec procedure call? You could implement the call - as an ItemReader and return null after the - procedure finishes, but it is a bit unnatural since there would need to - be a no-op ItemWriter and lots of overhead for - transaction handling, listeners, etc. Spring Batch provides an - implementation of Step for this scenario: - TaskletStep. As explained in Chapter 2, the - Tasklet is a simple interface that has one - method, execute, which will be a called once - for the whole Step. - Tasklet implementors might call a stored - procedure, a script, or a simple SQL upate statement. Because there are - less concerns, there are only two required dependencies for a - TaskletStep: a Tasklet, - and a JobRepository: - - <bean id="taskletStep" - class="org.springframework.batch.core.step.tasklet.TaskletStep" /> - <property name="tasklet" ref="tasklet" /> - <property name="jobRepository" ref="repository" /> -</bean> - - - TaskletStep will automatically register the tasklet as - StepExecutionListener if it implements this - interface - - -
- TaskletAdapter - - As with other adapters for the ItemWriter - and ItemReader interfaces, the - Tasklet interface contains an implementation - that allows for adapting itself to any pre-existing class: - TaskletAdapter. An example where this may be - useful is an existing DAO that is used to upate a flag on a set of - records. The TaskletAdapter can be used to call - this class without having to write an adapter for the - Tasklet interface: - - <bean id="deleteFilesInDir" parent="taskletStep"> - <property name="tasklet"> - <bean class="org.springframework.batch.core.step.tasklet.TaskletAdapter"> - <property name="targetObject"> - <bean class="org.mycompany.FooDao"> - </property> - <property name="targetMethod" value-"updateFoo" /> - </bean> - </property> - </bean> -
- -
- Example Tasklet implementation - - Many batch jobs contains steps that must be done before the main - processing begins in order to set up various resources, or after - processing has completed to cleanup those resources. In the case of a - job that works heavily with files, it is often necessary to delete - certain files locally after they have been uploaded successfully to - another location. The example below taken from the Spring Batch - samples project, is a Tasklet implementation - with just such a responsibility: - - public class FileDeletingTasklet implements Tasklet, InitializingBean { - - private Resource directory; - - public ExitStatus execute() throws Exception { - File dir = directory.getFile(); - Assert.state(dir.isDirectory()); - - File[] files = dir.listFiles(); - for (int i = 0; i < files.length; i++) { - boolean deleted = files[i].delete(); - if (!deleted) { - throw new UnexpectedJobExecutionException("Could not delete file " + files[i].getPath()); - } - } - return ExitStatus.FINISHED; - } - - public void setDirectoryResource(Resource directory) { - this.directory = directory; - } - - public void afterPropertiesSet() throws Exception { - Assert.notNull(directory, "directory must be set"); - } - } - - The above Tasklet implementation will - delete all files within a given directory. It should be noted that the - execute method will only be called once. All - that is left is to inject the Tasklet into a - TaskletStep: - - <bean id="taskletJob" parent="simpleJob"> - <property name="steps"> - <bean id="deleteFilesInDir" parent="taskletStep"> - <property name="tasklet"> - <bean class="org.springframework.batch.sample.tasklet.FileDeletingTasklet"> - <property name="directoryResource" ref="directory" /> - </bean> - </property> - </bean> - </property> - </bean> - - <bean id="directory" - class="org.springframework.core.io.FileSystemResource"> - <constructor-arg value="target/test-outputs/test-dir" /> - </bean> -
- -
- Executing System Commands - - Many batch jobs may require that an external command be called - from within the batch job. Such a process could be kicked off - separately by the scheduler, but the advantage of common meta-data - about the run would be lost. Furthermore, a multi-step job would also - need to be split up into multiple jobs as well. Because the need is so - common, Spring Batch provides a Tasklet - implementation for calling system commands: - - - <bean class="org.springframework.batch.sample.tasklet.SystemCommandTasklet"> - <property name="command" value="echo hello" /> - <!-- 5 second timeout for the command to complete --> - <property name="timeout" value="5000" /> - </bean> - - -
-
-
- -
- Examples of Customized Business Logic - -
- Some batch jobs can be assembled purely from off-the-shelf - components in Spring Batch, mostly the ItemReader - and ItemWriter implementations. Where this is not - possible (the majority of cases) the main API entry points for - application developers are the Tasklet, - ItemReader, ItemWriter and - the various listener interfaces. Most simple batch jobs will be able to - use off-the-shelf input from a Spring Batch - ItemReader, but it is very often the case that - there are custom concerns in the processing and writing, which normally - leads developers to implement an ItemWriter, or - ItemTransformer. - - Here we provide a few examples of common patterns in custom - business logic, mainly using the listener interfaces . It should be - noted that an ItemReader or - ItemWriter can implement the listener interfaces - as well if appropriate. -
- -
- Logging Item Processing and Failures - - A common use case is the need for special handling of errors in a - step, item by item, perhaps logging to a special channel, or inserting a - record into a database. The StepHandlerStep - (created from the step factory beans) allows users to implement this use - case with a simple ItemReadListener, for errors - on read, and an ItemWriteListener, for errors on - write. The below code snippets illustrate a listener that logs both read - and write failures: - - public class ItemFailureLoggerListener extends ItemListenerSupport { - - private static Log logger = LogFactory.getLog("item.error"); - - public void onReadError(Exception ex) { - logger.error("Encountered error on read", e); - } - - public void onWriteError(Exception ex, Object item) { - logger.error("Encountered error on write", e); - } - -} - - Having implemented this listener it must be registered with the - step: - - <bean id="simpleStep" - class="org.springframework.batch.core.step.item.SimpleStepFactoryBean" > - ... - <property name="listeners"> - <bean class="org.example...ItemFailureLoggerListener"/> - </property> - </bean> - - Remember that if your listener does anything in an - onError() method, it will be inside a transaction that is - going to be rolled back. If you need to use a transactional resource - such as a database inside an onError() method, consider - adding a declarative transaction to that method (see Spring Core - Reference Guide for details), and giving its propagation attribute the - value REQUIRES_NEW. -
- -
- Stopping a Job Manually for Business Reasons - - Spring Batch provides a stop() method through the JobLauncher - interface, but this is really aimed at the operator, rather than the - application programmer. Sometimes it is more convenient or makes more - sense to stop a job execution from within the business logic. - - The simplest thing to do is to throw a RuntimeException (one that - isn't retried indefinitely or skipped), For example, a custom exception - type could be used, as in the example below: - - public class PoisonPillItemWriter implements ItemWriter<T> { - - public void write(T item) throws Exception { - - if (isPoisonPill(item)) { - throw new PoisonPillException("Posion pill detected: "+item); - } - - } - -} - - Another simple way to stop a step from executing is to simply - return null from the - ItemReader: - - public class EarlyCompletionItemReader extends AbstractItemReader { - - private ItemReader delegate; - - public void setDelegate(ItemReader delegate) { ... } - - public Object read() throws Exception { - - Object item = delegate.read(); - - if (isEndItem(item)) { - return null; // end the step here - } - - return item; - - } - -} - - The previous example actually relies on the fact that there is a - default implementation of the CompletionPolicy - strategy which signals a complete batch when the item to be processed is - null. A more sophisticated completion policy could be implemented and - injected into the Step through the - RepeatOperationsStepFactoryBean: - - <bean id="simpleStep" - class="org.springframework.batch.core.step.item.RepeatOperationsStepFactoryBean" > - ... - <property name="chunkOperations"> - <bean class="org.springframework.batch.repeat.support.RepeatTemplate"> - <property name="completionPolicy"> - <bean class="org.example...SpecialCompletionPolicy"/> - </property> - </bean> - </property> - </bean> - - An alternative is to set a flag in the - StepExecution, which is checked by the - Step implementations in the framework in between - item processing. To implement this alternative, we need access to the - current StepExecution, and this can be achieved by implementing a - StepListener and registering it with the Step. Here is an example of a - listener that sets the flag: - - public class CustomItemWriter extends ItemListenerSupport implements StepListener { - - private StepExecution stepExecution; - - public void beforeStep(StepExecution stepExecution) { - this.stepExecution = stepExecution; - } - - public void afterRead(Object item) { - - if (isPoisonPill(item)) { - stepExecution.setTerminateOnly(true); - } - - } - -} - - The default behaviour here when the flag is set is for the step to - throw a JobInterruptedException. This can be - controlled through the StepInterruptionPolicy, - but the only choice is to throw or not throw an exception, so this is - always an abnormal ending to a job. -
- -
- Adding a Footer Record - - A very common requirement is to aggregate information during the - output process and to append a record at the end of a file summarizing - the data, or providing a checksum. This can also be achieved with a - callbacks in the step, normally as part of a custom - ItemWriter. In this case, since a job is - accumulating state that should not be lost if the job aborts, the - ItemStream interface should be - implemented: - - public class CustomItemWriter implements ItemWriter<Trade> - ItemStream, StepListener -{ - - private static final String TOTAL_AMOUNT_KEY = "total.amount"; - - private ItemWriter delegate; - - private double totalAmount = 0.0; - - public void setDelegate(ItemWriter delegate) { ... } - - public ExitStatus afterStep(StepExecution stepExecution) { - // Add the footer record here... - delegate.write("Total Amount Processed: " + totalAmount); - } - - public void open(ExecutionContext executionContext) { - if (executionContext.containsKey(TOTAL_AMOUNT_KEY) { - totalAmount = executionContext.getDouble(TOTAL_AMOUNT_KEY); - } - } - - public void update(ExecutionContext executionContext) { - executionContext.setDouble(TOTAL_AMOUNT_KEY, totalAmount); - } - - public void write(Trade item) { - - delegate.write(item); - totalAmount += item.getAmount(); - - } - -} - - The custom writer in the example is stateful (it maintains its - total in an instance variable totalAmount), but the - state is stored through the ItemStream interface - in the ExecutionContext. In this way we can be - sure that when the open() callback is received on a - restart. The framework garuntees we always get the last value that was - committed. It should be noted that it is not always necessary to - implement ItemStream. For example, if the ItemWriter is re-runnable, in - the sense that it maintains its own state in a transactional resource - like a database, there is no need to maintain state within the writer - itself. -
-
- diff --git a/src/site/docbook/reference/images/partitioning-overview.png b/src/site/docbook/reference/images/partitioning-overview.png new file mode 100644 index 000000000..a9cd82430 Binary files /dev/null and b/src/site/docbook/reference/images/partitioning-overview.png differ diff --git a/src/site/docbook/reference/images/partitioning-spi.png b/src/site/docbook/reference/images/partitioning-spi.png new file mode 100644 index 000000000..eeb7c76d1 Binary files /dev/null and b/src/site/docbook/reference/images/partitioning-spi.png differ diff --git a/src/site/docbook/reference/images/remote-chunking.png b/src/site/docbook/reference/images/remote-chunking.png new file mode 100644 index 000000000..004953a98 Binary files /dev/null and b/src/site/docbook/reference/images/remote-chunking.png differ diff --git a/src/site/docbook/reference/index.xml b/src/site/docbook/reference/index.xml index c82e5c161..119e08495 100644 --- a/src/site/docbook/reference/index.xml +++ b/src/site/docbook/reference/index.xml @@ -58,6 +58,8 @@ + + diff --git a/src/site/docbook/reference/readersAndWriters.xml b/src/site/docbook/reference/readersAndWriters.xml index 6eba54cdc..cf93a08b6 100644 --- a/src/site/docbook/reference/readersAndWriters.xml +++ b/src/site/docbook/reference/readersAndWriters.xml @@ -2511,7 +2511,7 @@ how a validator could be added. -
+
Preventing state persistence By default, all of the ItemReader and diff --git a/src/site/docbook/reference/repeat.xml b/src/site/docbook/reference/repeat.xml index ca0182375..e9cc6cd34 100644 --- a/src/site/docbook/reference/repeat.xml +++ b/src/site/docbook/reference/repeat.xml @@ -234,7 +234,7 @@ template.iterate(new RepeatCallback() { TaskExecutorRepeatTemplate, which uses the Spring TaskExecutor strategy to run the RepeatCallback. The default is to use a - SynchronousTaskExecutor, which has the effect of executing the whole + SynchronousTaskExecutor, which has the effect of executing the whole iteration in the same thread (the same as a normal RepeatTemplate).
diff --git a/src/site/docbook/reference/scalability.xml b/src/site/docbook/reference/scalability.xml new file mode 100644 index 000000000..01693b12a --- /dev/null +++ b/src/site/docbook/reference/scalability.xml @@ -0,0 +1,302 @@ + + + + Scaling and Parallel Processing + + Many batch processing problems can be solved with single threaded, + single process jobs, so it is always a good idea to properly check if that + meets your needs before thinking about more complex implementations. Measure + the performance of a realistic job and see if the simplest implementation + meets your needs first: you can read and write a file of several hundred + megabytes in well under a minute, even with bog standard hardware. + + When you are ready to start implementing a job with some parallel + processing, Spring Batch offers a range of options, which are described in + this chapter, although some features are covered elsewhere. At a high level + there are two modes of parallel processing: single process, multi-threaded; + and multi-process. These break down into categories as well, as + follows: + + + + Multi-threaded Step (single process) + + + + Parallel Steps (single process) + + + + Remote Chunking of Step (multi process) + + + + Partitioning a Step (single or multi process) + + + + Next we review the single-process options first, and then the + multi-process options. + +
+ Multi-threaded Step + + The simplest way to start parallel processing is to add a + TaskExecutor to your Step configuration, e.g. as an + attribute of the tasklet: + + + +]]> + + In this example the taskExecutor is a reference to another bean + definition, implementing the TaskExecutor + interface. TaskExecutor is a standard Spring + interface, so consult the Spring User Guide for details of available + implementations. The simplest multi-threaded + TaskExecutor is a + SimpleAsyncTaskExecutor. + + The result of the above configuration will be that the Step executes + by reading, processing and writing each chunk of items (each commit + interval) in a separate thread of execution. + + There are some practical limitations of using multi-threaded Steps + for some common Batch use cases. Many participants in a Step (e.g. readers + and writers) are stateful, and if the state is not segregated by thread, + then those components are not usable in a multi-threaded Step. In + particular most of the off-the-shelf readers and writers from Spring Batch + are not designed for multi-threaded use. It is, however, possible to work + with stateless or thread safe readers and writers, and there is a sample + (parallelJob) in the Spring Batch Samples that show the use of a process + indicator (see ) to keep + track of items that have been processed in a database input table. +
+ +
+ Parallel Steps + + As long as the application logic that needs to be parallelised can + be split into distinct responsibilities, and assigned to individual steps + then it can be parallelised in a single process. Parallel Step execution + is easy to configure and use, for example, to execute steps + (step1,step2) in parallel with + step3, you could configure a flow like this: + + + + + + + + + + + ]]> + + See the section on for more + detail. +
+ +
+ Remote Chunking + + In Remote Chunking the Step processing is split across multiple + processes, communicating with each other through some middleware. Here is + a picture of the pattern in action: + + + + + + + + The Master component is a single process, and the Slaves are + multiple remote processes. Clearly this pattern works best if the Master + is not a bottleneck, so the processing must be more expensive than the + reading of items (this is often the case in practice). + + The Master is just an implementation of a Spring Batch + Step, with the ItemWriter replaced with a generic + version that knows how to send chunks of items to the middleware as + messages. The Slaves are standard listeners for whatever middleware is + being used (e.g. with JMS they would be + MesssageListeners), and their role is to process + the chunks of items using a standard ItemWriter or + ItemProcessor plus + ItemWriter, through the + ChunkProcessor interface. One of the advantages of + using this pattern is that the reader, processor and writer components are + off-the-shelf (the same as would be used for a local execution of the + step). The items are divided up dynamically and work is shared through the + middleware, so if the listeners are all eager consumers, then load + balancing is automatic. + + The middleware has to be durable, with guaranteed delivery and + single consumer for each message. JMS is the obvious candidate, but other + options exist in the grid computing and shared memory product space (e.g. + Java Spaces). + + Spring Batch has a sub-project (Spring Batch Integration), providing + implementations of various patterns like this one using Spring + Integration. Spring Batch Integration is available in subversion for + people to use, but is not intended to be part of the official general + release of Spring Batch until it builds up more of a community of + users. +
+ +
+ Partitioning + + Spring Batch also provides an SPI for partitioning a Step execution + and executing it remotely. In this case the remote participants are simply + Step instances that could just as easily have been configured and used for + local processing. Here is a picture of the pattern in action: + + + + + + + + The Job is executing on the left hand side as a sequence of Steps, + and one of the Steps is labelled as a Master. The Slaves in this picture + are all identical instances of a Step, which could in fact take the place + of the Master resulting in the same outcome for the Job. The Slaves are + typically going to be remote services, but could also be local threads of + execution. The messages sent by the Master to the Slaves in this pattern + do not need to be durable, or have guaranteed delivery: Spring Batch + meta-data in the JobRepository will ensure that + each Slave is executed once and only once for each Job execution. + + The SPI in Spring Batch consists of a special implementation of Step + (the PartitionStep), and two strategy interfaces + that need to be implemented for the specific environment. The strategy + interfaces are PartitionHandler and + StepExecutionSplitter, and their role is show in + the sequence diagram below: + + + + + + + + The Step on the right in this case is the "remote" Slave, so + potentially there are many objects and or processes playing this role, and + the PartitionStep is shown driving the execution. The PartitionStep + configuration looks like this: + + + + + +]]> + + There is a simple example which can be copied and extended in the + unit test suite for Spring Batch Core (see + org.springframework.batch.core.partition + package). + +
+ PartitionHandler + + The PartitionHandler is the component that + knows about the fabric of the remoting or grid environment. It is able + to send StepExecution requests to the remote + Steps, wrapped in some fabric-specific format, like a DTO. It does not + have to know how to split up the input data, or how to aggregate the + result of multiple Step executions. Generally speaking it probably also + doesn't need to know about resilience or failover, since those are + features of the fabric in many cases, and anyway Spring Batch always + provides restartability independent of the fabric: a failed Job can + always be restarted and only the failed Steps will be + re-executed. + + The PartitionHandler interface can have + specialised implementations for a variety of fabric types: e.g. simple + RMI remoting, EJB remoting, custom web service, JMS, Java Spaces, shared + memory grids (like Terracotta or Coherence), grid execution fabrics + (like GridGain). Spring Batch does not contain implementations for any + proprietary grid or remoting fabrics. + + Spring Batch does however provide a useful implementation of + PartitionHandler that executes Steps locally in + separate threads of execution, using the + TaskExecutor strategy from Spring. The + implementation is called + TaskExecutorPartitionHandler, and it can be + configured like this: + + + + + +]]> + + The gridSize determines the number of separate + step executions to create, so it can be matched to the size of the + thread pool in the TaskExecutor, or else it can + be set to be larger than the number of threads available, in which case + the blocks of work are smaller. + + The TaskExecutorPartitionHandler is quite + useful for IO intensive Steps, like copying large numbers of files or + replicating filesystems into content management systems. +
+ +
+ StepExecutionSplitter + + The StepExecutionSplitter is responsible + for splitting up a StepExecution into blocks of + work, and providing input parameters for the remote Slaves in the form + of an ExecutionContext for each one. The + principal method for this in the interface is + + split(StepExecution stepExecution, int gridSize) + throws JobExecutionException; +}]]> + + So an execution instance for the Master step is passed in, along + with a hint about the grid size, and the splitter has to create a set of + partitioned StepExecution instances, each with a + different ExecutionContext. + + A convenient generic implementation of StepExecutionSplitter is + provided by Spring Batch, which handles concerns like interpreting the + grid size and handling restart. It is recommended that you use this + implementation (the SimpleStepExecutionSplitter) + and inject specific knowledge of the input data through its + Partitioner property. The Partitioner has a + simpler responsibility: to generate execution contexts as input + parameters for new step executions only (no need to worry about + restarts). It has a single method: + + partition(int gridSize); +}]]> + + The return value from this method associates a unique name for + each step execution (the String), with input + parameters in the form of an ExecutionContext. + The names show up later in the Batch meta data as the step name in the + partitioned StepExecutions. The + ExecutionContext is just a bag of name-value + pairs, so it might contain a range of primary keys, or line numbers, or + the location of an input file. The remote Step + then normally binds to the context input using #{...} + placeholders (late binding in step scope). +
+
+ \ No newline at end of file diff --git a/src/site/docbook/reference/step.xml b/src/site/docbook/reference/step.xml index 3412303cf..32b2a15dc 100644 --- a/src/site/docbook/reference/step.xml +++ b/src/site/docbook/reference/step.xml @@ -4,7 +4,7 @@ Configuring a Step - As discussed in , a + As discussed in , a Step is a domain object that encapsulates an independent, sequential phase of a batch job and contains all of the information necessary to define and control the actual batch processing. @@ -406,6 +406,7 @@
+ Configuring Fatal Exceptions One problem with the example above is that any other exception besides a FlatFileParseException will cause the Job to fail. In certain scenarios this may be the @@ -1414,7 +1415,7 @@ ]]>
-
+
Split Flows Every scenario described so far has involved a