diff --git a/docs/src/site/docbook/reference/domain.xml b/docs/src/site/docbook/reference/domain.xml index 11e3afb89..6a9b99cbb 100644 --- a/docs/src/site/docbook/reference/domain.xml +++ b/docs/src/site/docbook/reference/domain.xml @@ -21,8 +21,8 @@ - simple and default implementations that allowed for quick - adoption and ease of use out-of-the-box + simple and default implementations that allow for quick adoption + and ease of use out-of-the-box @@ -47,13 +47,13 @@ + format="PNG" /> + format="PNG" scale="80" /> Figure 2.1: Batch Stereotypes @@ -63,7 +63,7 @@ language of batch. A Job has one to many steps, which has exactly one ItemReader, ItemProcessor, and ItemWriter. A job needs to be launched (JobLauncher), and meta data about the currently running process needs to be - stored (JobRepository) + stored (JobRepository).
Job @@ -78,7 +78,7 @@ + scale="80" /> @@ -133,11 +133,11 @@ case of this job, there will be one logical JobInstance per day. For example, there will be a January 1st run, and a January 2nd run. If the January 1st run fails the - first time and is run again the next day, it's still the January 1st - run. (Usually this corresponds with the data its processing as well, - meaning the January 1st run processes data for January 1st, etc) That is - to say, each JobInstance can have multiple - executions. (JobExecution is discussed in more + first time and is run again the next day, it is still the January 1st + run. (Usually this corresponds with the data it is processing as well, + meaning the January 1st run processes data for January 1st, etc). + Therefore, each JobInstance can have multiple + executions (JobExecution is discussed in more detail below) and only one JobInstance corresponding to a particular Job can be running at a given time. The definition of a JobInstance @@ -151,11 +151,11 @@ likely be a business decision, it is left up to the ItemReader to decide. What using the same JobInstance will determine, however, is whether - or not the 'state' (i.e. The ExecutionContext, which is discussed below) - from previous executions will be used. Using a new - JobInstance will mean 'start from the beginning' - and using an existing instance will generally mean 'start from where you - left off'. + or not the 'state' (i.e. the ExecutionContext, + which is discussed below) from previous executions will be used. Using a + new JobInstance will mean 'start from the + beginning' and using an existing instance will generally mean 'start + from where you left off'.
@@ -165,15 +165,15 @@ differs from Job, the natural question to ask is: "how is one JobInstance distinguished from another?" The answer is: JobParameters. - JobParameters are any set of parameters used to - start a batch job, which can be used for identification or even as + JobParameters is a set of parameters used to + start a batch job. They can be used for identification or even as reference data during the run: + scale="80" /> @@ -204,10 +204,10 @@ will not be considered complete unless the execution completes successfully. Using the EndOfDay Job described above as an example, consider a JobInstance for - 01-01-2008 that failed the first time it was run. If it is ran again, + 01-01-2008 that failed the first time it was run. If it is run again with the same job parameters as the first run (01-01-2008), a new - JobExecution will be created. However, there will still be only one - JobInstance. + JobExecution will be created. However, there will + still be only one JobInstance. A Job defines what a job is and how it is to be executed, and JobInstance is a purely @@ -226,9 +226,9 @@ status A BatchStatus object that - indicates the status of the execution. While it's running, it's - BatchStatus.STARTED, if it fails it's BatchStatus.FAILED, and if - it finishes successfully it's BatchStatus.COMPLETED + indicates the status of the execution. While running, it's + BatchStatus.STARTED, if it fails, it's BatchStatus.FAILED, and + if it finishes successfully, it's BatchStatus.COMPLETED @@ -259,18 +259,19 @@ createTime A java.util.Date representing the - current system time when the JobExecution was first persisted. - The job may not have been started yet (and thus has no start - time), but it will always have a createTime, which is required - by the framework for managing job level - ExecutionContexts. + current system time when the JobExecution + was first persisted. The job may not have been started yet (and + thus has no start time), but it will always have a createTime, + which is required by the framework for managing job level + ExecutionContexts. lastUpdated A java.util.Date representing the - last time a JobExecution was persisted. + last time a JobExecution was + persisted. @@ -389,17 +390,17 @@ will be kicked off again for 01-01, starting where it left off and completing successfully at 9:30. Because it's now the next day, the 01-02 job must be run as well, which is kicked off just afterwards at - 9:31, and completes in it's normal one hour time at 10:30. There is no + 9:31, and completes in its normal one hour time at 10:30. There is no requirement that one JobInstance be kicked off after another, unless there is potential for the two jobs to attempt to access the same data, causing issues with locking at the database level. It is entirely up to the scheduler to determine when a Job should be run. Since they're separate - JobInstances, Spring Batch will make no attempt to stop them from being - run concurrently. (Attempting to run the same + JobInstances, Spring Batch will make no attempt + to stop them from being run concurrently. (Attempting to run the same JobInstance while another is already running will result in a JobExecutionAlreadyRunningException - being thrown) There should now be an extra entry in both the + being thrown). There should now be an extra entry in both the JobInstance and JobParameters tables, and two extra entries in the JobExecution table: @@ -554,7 +555,7 @@ + scale="80" /> @@ -568,12 +569,12 @@ A StepExecution represents a single attempt to execute a Step. A new - StepExecution will be created each time a Step is - run, similar to JobExecution. However, if a step - fails to execute because the step before it fails, there will be no - execution persisted for it. A StepExecution will - only be created when it's Step is actually - started. + StepExecution will be created each time a + Step is run, similar to + JobExecution. However, if a step fails to execute + because the step before it fails, there will be no execution persisted + for it. A StepExecution will only be created when + its Step is actually started. Step executions are represented by objects of the StepExecution class. Each execution contains a @@ -596,8 +597,8 @@ A BatchStatus object that indicates the status of the execution. While it's running, the - status is BatchStatus.STARTED, if it fails the status is - BatchStatus.FAILED, and if it finishes successfully the status + status is BatchStatus.STARTED, if it fails, the status is + BatchStatus.FAILED, and if it finishes successfully, the status is BatchStatus.COMPLETED @@ -670,15 +671,15 @@ processSkipCount - The number of times process has failed, resulting in a - skipped item. + The number of times process has + failed, resulting in a skipped item. filterCount The number of items that have been 'filtered' by the - ItemProcessor + ItemProcessor. @@ -848,14 +849,14 @@ this scenario. This value will be updated just before each commit by the framework, and can contain multiple rows corresponding to entries within the ExecutionContext. Being notified before a - commit requires one of the various StepListeners, or an - ItemStream, which are discussed in more detail - later in this guide. As with the previous example, it is assumed that the - Job is restarted the next day. When it is restarted, the values from the - ExecutionContext of the last run are reconstituted - from the database, and when the ItemReader is - opened, it can check to see if it has any stored state in the context, and - initialize itself from there: + commit requires one of the various StepListeners, + or an ItemStream, which are discussed in more + detail later in this guide. As with the previous example, it is assumed + that the Job is restarted the next day. When it is + restarted, the values from the ExecutionContext of + the last run are reconstituted from the database, and when the + ItemReader is opened, it can check to see if it has + any stored state in the context, and initialize itself from there: if (executionContext.containsKey(getKey(LINES_READ_COUNT))) { log.debug("Initializing for restart. Restart data is: " + executionContext); @@ -898,7 +899,7 @@ exists per StepExecution at any given time. Clients of the ExecutionContext should be careful because this creates a shared keyspace, so care should be taken when putting - values in to ensure no data is overwritten, however, the + values in to ensure no data is overwritten. However, the Step stores absolutely no data in the context, so there is no way to adversely affect the framework. @@ -915,7 +916,7 @@ - As noted in the comment, ecStep will not equal ecJob, they are two + As noted in the comment, ecStep will not equal ecJob; they are two different ExecutionContexts. The one scoped to the Step will be saved at every commit point in the Step, whereas the one scoped to the @@ -981,7 +982,7 @@ the output of a Step, one item at a time. Generally, an item writer has no knowledge of the input it will receive next, only the item that was passed in its current invocation. More - details about the ItemWriter interface and it's + details about the ItemWriter interface and its various implementations can be found in
@@ -994,10 +995,10 @@ ItemReader reads one item, and the ItemWriter writes them, the ItemProcessor provides access to transform or apply - other business processing. If while processing the item it's determined - that it's not valid, returning null indicates that it should not be - written out. More details about the ItemProcessor interface can be found - in . + other business processing. If, while processing the item, it is determined + that the item is not valid, returning null indicates that the item should + not be written out. More details about the ItemProcessor interface can be + found in .
diff --git a/docs/src/site/docbook/reference/job.xml b/docs/src/site/docbook/reference/job.xml index b87dfb1a6..e3c6db573 100644 --- a/docs/src/site/docbook/reference/job.xml +++ b/docs/src/site/docbook/reference/job.xml @@ -1,827 +1,853 @@ - - - - Configuring and Running A Job - - In , the overall architecture design was - discussed, using the following diagram as a guide: - - - - - - - - - - - - While the Job object may seem like a simple container for steps, there - are many configuration options that developers should be aware of. - Furthermore, there are many considerations for how a - Job will be run and how its meta data will be stored - during that run. This chapter will explain the various configuration options - and runtime concerns of a Job. - -
- Configuring a Job - - There are multiple implementations of the Job interface, however, the namespace abstracts away - the differences in configuration. It has only three required dependencies: - a name, JobRepository, and a list of Steps. - - - <job id="footballJob"> - <step id="playerload" next="gameLoad"/> - <step id="gameLoad" next="playerSummarization"/> - <step id="playerSummarization"/> - </job> - - - - The namespace defaults to referencing a repository with an id of - 'jobRepository', which is a sensible default. However, this can be - overridden explicitly: - - - <job id="footballJob" job-repository="specialRepository"> - <step id="playerload" next="gameLoad"/> - <step id="gameLoad" next="playerSummarization"/> - <step id="playerSummarization"/> - </job> - - - -
- Restartability - - One key concern when execution a batch job, is what happens when a - failed job is restarted? A Job is considered to have been 'restarted' if - the same JobInstance has more than one - JobExecution. Ideally, all jobs should be able to start up where they - left off, but there are scenarios where this is not possible. It is entirely up to the developer to ensure that a new - instance is always created in this scenario. However, Spring - Batch does provide some help. If a Job should never be restarted, but - should always be run as part of a new - JobInstance, then the restartable property may be - set to 'false': - - - <job id="footballJob" restartable="false"> - <step id="playerload" next="gameLoad"/> - <step id="gameLoad" next="playerSummarization"/> - <step id="playerSummarization"/> - </job> - - - - To phrase it another way, setting restartable to false means "this - Job does not support being started again". Restarting a Job that is not - restartable will cause a JobRestartException to - be thrown: - - - Job job = new SimpleJob(); - job.setRestartable(false); - - JobParameters jobParameters = new JobParameters(); - - JobExecution firstExecution = jobRepository.createJobExecution(job, jobParameters); - jobRepository.saveOrUpdate(firstExecution); - - try { - jobRepository.createJobExecution(job, jobParameters); - fail(); - } - catch (JobRestartException e) { - // expected - } - - - - This snippet of JUnit code shows how attempting to create a - JobExecution the first time for a non restartable - job will cause no issues. However, the second - attempt will throw a JobRestartException. -
- -
- Intercepting Job execution - - During the course of the execution of a - Job, it may be useful to be notified of various - events in its lifecycle so that custom code may be executed. The - SimpleJob allows for this by calling a - JobListener at the appropriate time: - - - public interface JobExecutionListener { - - void beforeJob(JobExecution jobExecution); - - void afterJob(JobExecution jobExecution); - - } - - - - Listeners can be added to a SimpleJob via - the setJobListeners property: - - - <job id="footballJob"> - <step id="playerload" next="gameLoad"/> - <step id="gameLoad" next="playerSummarization"/> - <step id="playerSummarization"/> - <listeners> - <listener class="org.springframework.batch.sample.SampleListener"/> - </listeners> - </job> - - - - It should be noted that afterJob will be called regardless of the - success or failure of the Job. If success or - failure needs to be determined it can be obtained from the - JobExecution: - - - void afterJob(JobExecution jobExecution){ - if( jobExecution.getStatus = BatchStatus.COMPLETED ){ - //job success - } - else if(jobExecution.getStatus = BatchStatus.FAILED){ - //job failure - } - } - - -
- -
- JobFactory and Stateful Components in Steps - - Unlike many traditional Spring applications, many of the - components of a batch application are stateful, the file readers and - writers are obvious examples. The recommended way to deal with this is - to create a fresh ApplicationContext for each job - execution. If the Job is launched from the - command line with CommandLineJobRunner this is - trivial. For more complex launching scenarios, where jobs are executed - in parallel or serially from the same process, some extra steps have to - be taken to ensure that the ApplicationContext is - refreshed. This is preferable to using prototype scope for the stateful - beans because then they would not receive lifecycle callbacks from the - container at the end of use. (e.g. through destroy-method in XML) - - The strategy provided by Spring Batch to deal with this scenario - is the JobFactory, and the samples provide an - example of a specialized implementation that can load an - ApplicationContext and close it properly when the - job is finished. A relevant examples is - ClassPathXmlApplicationContextJobFactory and its - use in the adhoc-job-launcher-context.xml and the - quartz-job-launcher-context.xml, which can be found in the - Samples project. -
-
- -
- Configuring a JobRepository - - As described in earlier, the JobRepository is used for basic CRUD - operations of the various persisted domain objects within Spring Batch, - such as JobExecution and StepExecution. It is required by many of the - major framework features, such as the JobLauncher, - Job, and Step. The batch - namespace abstract much of the implementation details of the - JobRepository implementations and their - collaborators. However, there are still a few configuration options - available: - - - <job-repository id="jobRepository" - dataSource="dataSource" - transactionManager="transactionManager" - isolation-level-for-create="serializable" - table-prefix="BATCH_" - /> - - - - None of the configuration options listed above are required except - the id. If they are not set, the defaults shown above will be used. They - are shown above for awareness purposes. - -
- Transaction Configuration For the JobRepository - - If the namespace is used, transactional advice will be - automatically created around the repository. This is to ensure that the - batch meta data, including state that is necessary for restarts after a - failure, is persisted correctly. The behaviour of the framework is not - well defined if the repository methods are not transactional. The - isolation level in the create* method attributes is - specified separately to ensure that when jobs are launched, if two - processes are trying to launch the same job at the same time, only one - will succeed. The default isolation level for that method is - SERIALIZABLE, which is quite aggressive: READ_COMMITTED would work just - as well; READ_UNCOMMITTED would be fine if two processes are not likely - to collide in this way. However, since a call to the - create* method is quite short, it is unlikely - that the SERIALIZED will cause problems, as long as the database - platform supports it. However, this can be overridden: - - - <job-repository id="jobRepository" - isolation-level-for-create="ISOLATION_REPEATABLE_READ" /> - - - - If the namespace or factory beans aren't used then it is also - essential to configure the transactional behaviour of the repository - using AOP: - - - <aop:config> - <aop:advisor - pointcut="execution(* org.springframework.batch.core..*Repository+.*(..))" - <advice-ref="txAdvice" /> - </aop:config> - - <tx:advice id="txAdvice" transaction-manager="transactionManager"> - <tx:attributes> - <tx:method name="*" /> - </tx:attributes> - </tx:advice> - - - - This fragment can be used as is, with almost no changes. Remember - also to include the appropriate namespace declarations and to make sure - spring-tx and spring-aop (or the whole of spring) is on the - classpath. -
- -
- Changing the table prefix - - Another modifiable property of the - JobRepository is the table prefix of the - meta-data tables. By default they are all prefaced with BATCH_. - BATCH_JOB_EXECUTION and BATCH_STEP_EXECUTION are two examples. However, - there are potential reasons to modify this prefix. If the schema names - needs to be prepended to the table names, or if more than one set of - meta data tables is needed within the same schema, then the table prefix - will need to be changed: - - - <job-repository id="jobRepository" - table-prefix="SYSTEM.TEST_" - /> - - - - Given the above changes, every query to the meta data tables will - be prefixed with "SYSTEM.TEST_". BATCH_JOB_EXECUTION will be referred to - as SYSTEM.TEST_JOB_EXECUTION. - - - Only the table prefix is configurable, the table and column - names are not. - -
- -
- In-Memory Repository - - There are scenarios in which you may not want to persist your - domain objects to the database. One reason may be speed, storing domain - objects at each commit point takes extra time. Another reason may be - that you just don't need to persist status for a particular job. Spring - batch provides a solution: - - <bean id="jobRepository" - class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" /> -
-
- -
- Configuring a JobLauncher - - The most basic implementation of the - JobLauncher interface is the - SimpleJobLauncher. It's only required dependency is - a JobRepository, in order to obtain an - execution: - - <bean id="jobLauncher" - class="org.springframework.batch.execution.launch.SimpleJobLauncher"> - <property name="jobRepository" ref="jobRepository" /> - </bean> - - Once a JobExecution is obtained, - it is passed to the execute method of Job, - ultimately returning the JobExecution to the - caller: - - - - - - - - - - - - The sequence is straightforward, and works well when launched from a - scheduler, but causes issues when trying to launch from an HTTP request. - In this scenario, the launching needs to be done asynchronously, so that - the SimpleJobLauncher returns immediately to it's - caller. This is because it is not good practice to keep an HTTP request - open for the amount of time needed by long running processes such as - batch. An example sequence is below: - - - - - - - - - - - - The SimpleJobLauncher can easily be - configured to allow for this scenario by configuring a - TaskExecutor: - - <bean id="jobLauncher" - class="org.springframework.batch.execution.launch.SimpleJobLauncher"> - <property name="jobRepository" ref="jobRepository" /> - <property name="taskExecutor"> - <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" /> - </property> - </bean> - - Any implementation of the spring TaskExecutor - interface can be used to control how jobs are asynchronously - executed. -
- -
- Running a Job - - At a minimum, launching a batch job requires two things: the Job to - be launched and a JobLauncher. Both can be - contained within the same context or different contexts. For example, if - launching a job from the command line, a new JVM will be instantiated for - each Job, and thus every job will have it's own - JobLauncher. However, if running from within a web - container within the scope of an HttpRequest, there - will usually be one JobLauncher, configured for - asynchronous job launching, that multiple requests will invoke to launch - their jobs. - -
- Running Jobs from the Command Line - - For users that want to run their jobs from an enterprise - scheduler, the command line is the primary interface. This is because - most schedulers (with the exception of Quartz unless using the - NativeJob) work directly with operating system - processes, primarily kicked off with shell scripts. There are many ways - to launch a Java process besides a shell script, such as Perl, Ruby, or - even 'build tools' such as ant or maven. However, because most people - are familiar with shell scripts, this example will focus on them. - -
- The CommandLineJobRunner - - Because the script launching the job must kick off a Java - Virtual Machine, there needs to be a class with a main method to act - as the primary entry point. Spring Batch provides an implementation - that serves just this purpose: - CommandLineJobRunner. It's important to note - that this is just one way to bootstrap your application, but there are - many ways to launch a Java process, and this class should in no way be - viewed as definitive. It performs four tasks: - - - - Loads the appropriate Application Context - - - - Parses command line arguments into JobParameters - - - - Locates the appropriate job based on arguments - - - - Uses the JobLauncher provided in the application context to - launch the job. - - - - All of these tasks are accomplished based completely upon the - arguments passed in. The following are required arguments: - - - CommandLineJobRunner arguments - - - - - jobPath - - The location of the XML file that will be used to - create an ApplicationContext. This file - should contain everything needed to run the complete - Job - - - - jobName - - The name of the job to be run. - - - -
- - These arguments must be passed in with the path first and the - name second. All arguments after these are considered to be - JobParameters and must be in the format of 'name=value': - - bash$ java CommandLineJobRunner endOfDayJob.xml endOfDay schedule.date(date)=2008/01/01 - - In most cases you would want to use a manifest to declare your - main class in a jar, but for simplicity, the class was used directly. - This example is using the same 'EndOfDay' example from . The first argument is 'endOfDayJob.xml', which is - the Spring ApplicationContext containing the - Job. The second argument, 'endOfDay' represents the job name. The - final argument, 'schedule.date=01-01-2008' will be converted into - JobParameters. An example of the XML - configuration is below: - - <bean id="endOfDay" - class="org.springframework.batch.core.job.SimpleJob"> - <property name="steps"> - <bean id="step1" parent="simpleStep" /> - <!-- Step details removed for clarity --> - </property> - </bean> - - <!-- Launcher details removed for clarity --> - <bean id="jobLauncher" - class="org.springframework.batch.core.launch.support.SimpleJobLauncher" /> - - This example is overly simplistic, since there are many more - requirements to a run a batch job in Spring Batch in general, but it - serves to show the two main requirements of the - CommandLineJobRunner: - Job and - JobLauncher -
- -
- ExitCodes - - When launching a batch job from the command-line, it is often - from an enterprise scheduler. Most schedulers are fairly dumb, and - work only at the process level. Meaning, they only know about some - operating system process such as a shell script that they're invoking. - In this scenario, the only way to communicate back to the scheduler - about the success or failure of a job is through return codes. A - number is returned to a scheduler that is told how to interpret the - result. In the simple case: 0 is success and 1 is failure. However, - there may be scenarios such as: If job A returns 4 kick off job B, if - it returns 5 kick off job C. This type of behavior is configured at - the scheduler level, but it is important that a processing framework - such as Spring Batch provide a way to return a numeric representation - of of the 'Exit Code' for a particular batch job. In Spring Batch this - is encapsulated within an ExitStatus, which is - covered in more detail in Chapter 5. For the purposes of discussing - exit codes, the only important thing to know is that an - ExitStatus has an exit code property that is - set by the framework (or the developer) and is returned as part of the - JobExecution returned from the - JobLauncher. The - CommandLineJobRunner converts this string value - to a number using the ExitCodeMapper - interface: - - public interface ExitCodeMapper { - - public int intValue(String exitCode); -} - - The essential contract of an - ExitCodeMapper is that, given a string exit - code, a number representation will be returned. The default - implementation used by the job runner is the SimpleJvmExitCodeMapper - that returns 0 for completion, 1 for generic errors, and 2 for any job - runner errors such as not being able to find a - Job in the provided context. If anything more - complex than the 3 values above is needed, then a custom - implementation of the ExitCodeMapper interface - must be supplied. Because the - CommandLineJobRunner is the class that creates - an ApplicationContext, and thus cannot be - 'wired together', any values that need to be overwritten must be - autowired. This means that if an implementation of - ExitCodeMapper is found within the BeanFactory, - it will be injected into the runner after the context is created. All - that needs to be done to provide your own - ExitCodeMapper is to declare the implementation - as a root level bean, and ensure it's part of the - ApplicationContext that is loaded by the - runner. -
-
- -
- Running Jobs from within a container - - -
-
- -
- Advanced Meta-Data Usage - - So far, both the JobLauncher and JobRepository interfaces have been - discussed. Together, they represent simple launching of a job, and basic - CRUD operations of batch domain objects: - - - - - - - - - - - - A JobLauncher uses the - JobRepository to create new - JobExecution objects, and run them. - Job and Step implementations - later use the same JobRepository for basic updates - of the same executions during the running of a Job. - The basic operations suffice for simple scenarios. However, in a large - batch environment with hundreds of batch jobs and complex scheduling - requirements, more advanced access of the meta data is required: - - - - - - - - - - - - The JobExplorer and - JobOperator interfaces, which will be discussed - below, add additional functionality for querying and controlling the meta - data. - -
- Querying the repository - - The most basic need before any advanced features is the ability to - query the repository for existing executions. This functionality is - provided by the JobExplorer interface: - - - public interface JobExplorer { - - List<JobInstance> getJobInstances(String jobName, int start, int count); - - JobExecution getJobExecution(Long executionId); - - StepExecution getStepExecution(Long jobExecutionId, Long stepExecutionId); - - JobInstance getJobInstance(Long instanceId); - - List<JobExecution> getJobExecutions(JobInstance jobInstance); - - Set<JobExecution> findRunningJobExecutions(String jobName); - } - - - - As is evident from the method signatures above, - JobExplorer is a read-only version of the - JobRepository, and like the - JobRepository, it can be easily configured via a - factory bean: - - - <bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean" - p:dataSource-ref="dataSource" /> - - - - Earlier in this - chapter, it was mentioned that the table prefix of the - JobRepository can be modified to allow for different versions or - schemas. Because the JobExplorer is working with the same tables, it too - needs the ability to set a prefix: - - - <bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean" - p:dataSource-ref="dataSource" p:tablePrefix="BATCH_" /> - - -
- -
- JobOperator - - As previously discussed, the JobRepository - provides CRUD operations on the meta-data, and the - JobExplorer provides read-only operations on the - meta-data. However, those operations are most useful when used together - to perform common monitoring tasks such as stopping, restarting, or - summarizing a Job, as is commonly done by batch operators. Spring Batch - provides for these types of operations via the - JobOperator interface: - - - public interface JobOperator { - - List<Long> getExecutions(long instanceId) throws NoSuchJobInstanceException; - - List<Long> getJobInstances(String jobName, int start, int count) throws NoSuchJobException; - - Set<Long> getRunningExecutions(String jobName) throws NoSuchJobException; - - String getParameters(long executionId) throws NoSuchJobExecutionException; - - Long start(String jobName, String parameters) - throws NoSuchJobException, JobInstanceAlreadyExistsException; - - Long restart(long executionId) - throws JobInstanceAlreadyCompleteException, NoSuchJobExecutionException, - NoSuchJobException, JobRestartException; - - Long startNextInstance(String jobName) - throws NoSuchJobException, JobParametersNotFoundException, JobRestartException, - JobExecutionAlreadyRunningException, JobInstanceAlreadyCompleteException; - - boolean stop(long executionId) throws NoSuchJobExecutionException, JobExecutionNotRunningException; - - String getSummary(long executionId) throws NoSuchJobExecutionException; - - Map<Long, String> getStepExecutionSummaries(long executionId) throws NoSuchJobExecutionException; - - Set<String> getJobNames(); - - } - - - - The above operations represent methods from many different - interfaces, such as JobLauncher, - JobRepository, - JobExplorer, and - JobRegistry. For this reason, the provided - implementation of JobOperator, - SimpleJobOperator, has many dependencies: - - - <bean id="jobOperator" class="org.springframework.batch.core.launch.support.SimpleJobOperator"> - <property name="jobExplorer"> - <bean class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean"> - <property name="dataSource" ref="dataSource" /> - </bean> - </property> - <property name="jobRepository" ref="jobRepository" /> - <property name="jobRegistry" ref="jobRegistry" /> - <property name="jobLauncher" ref="jobLauncher" /> - </bean> - - -
- -
- JobParametersIncrementer - - Most of the methods on JobOperator are - self-explanatory, and more detailed explanations can be found on the - javadoc - of the interface. However, the 'startNextInstance' method is - worth noting. This method will always start a new instance of a Job. - This can be extremely useful if there are serious issues in a - JobExecution, and the Job - needs to be started over again from the beginning. Unlike - JobLauncher though, which requires a new - JobParameters that will trigger a new JobInstance - if they are different than any previous one, the startNextInstance - method will use the JobParametersIncrementer tied to the Job to force - the Job to a new instance: - - - public interface JobParametersIncrementer { - - JobParameters getNext(JobParameters parameters); - } - - - - The contract of JobParametersIncrementer is - that, given a JobParameters, it - will return the 'next' parameter by incrementing any values it may - contain. This strategy is useful because the framework has no way of - knowing what changes to the JobParameters make it the 'next' instance. - For example, if the only value in JobParameters is a date, and the next - instance should be created, should that value be incremented by one day? - Or one week? (if the job is weekly for instance) The same can be said - for any numerical values that help to identify the Job, as shown - below: - - - public class SampleIncrementer implements JobParametersIncrementer { - - public JobParameters getNext(JobParameters parameters) { - if (parameters==null || parameters.isEmpty()) { - return new JobParametersBuilder().addLong("run.id", 1L).toJobParameters(); - } - long id = parameters.getLong("run.id",1L) + 1; - return new JobParametersBuilder().addLong("run.id", id).toJobParameters(); - } -} - - - - In this example, the value with a key of 'run.id' is used to - discriminate between JobInstances. If the JobParameters passed in is - null, it can be assumed that the Job has never been run before and thus - it's initial state can be returned. However, if not, the old value is - obtained, incremented by one, and returned. An incrementer can be - associated with Job via the 'incrementer' attribute in the - namespace: - - - <job id="footballJob" incrementer="sampleIncrementer"> - <step name="playerload" next="gameLoad"/> - <step name="gameLoad" next="playerSummarization"/> - <step name="playerSummarization"/> - </job> - - -
- -
- Stopping a Job - - One of the most common use cases of - JobOperator is gracefully stopping a - Job: - - - Set<Long> executions = jobOperator.getRunningExecutions("sampleJob"); - - jobOperator.stop(executions.iterator().next()); - - - The shutdown is not immediate, since there is no way to force - immediate shutdown, especially if the execution is currently in - developer code that the framework has no control over, such as a - business service. What it does mean, is that as soon as control is - returned back to the framework, it will set the status of the current - StepExecution to - BatchStatus.STOPPED, save it, then do the same - for the JobExecution before finishing. -
-
- + + + + Configuring and Running A Job + + In , the overall architecture design was + discussed, using the following diagram as a guide: + + + + + + + + + + + + While the Job object may seem like a simple + container for steps, there are many configuration options of which a + developers must be aware . Furthermore, there are many considerations for + how a Job will be run and how its meta-data will be + stored during that run. This chapter will explain the various configuration + options and runtime concerns of a Job. + +
+ Configuring a Job + + There are multiple implementations of the Job interface, however, the + namespace abstracts away the differences in configuration. It has only + three required dependencies: a name, JobRepository, + and a list of Steps. + + + <job id="footballJob"> + <step id="playerload" next="gameLoad"/> + <step id="gameLoad" next="playerSummarization"/> + <step id="playerSummarization"/> + </job> + + + + The namespace defaults to referencing a repository with an id of + 'jobRepository', which is a sensible default. However, this can be + overridden explicitly: + + + <job id="footballJob" job-repository="specialRepository"> + <step id="playerload" next="gameLoad"/> + <step id="gameLoad" next="playerSummarization"/> + <step id="playerSummarization"/> + </job> + + + + Note that if the job repository's default id is overridden, it must + be explicitly specified on every + Step as well as on + the Job. + +
+ Restartability + + One key issue when execution a batch job concerns the behavior of + a Job when it is restarted? The launching of a + Job is considered to be a 'restart' if a + JobExecution already exists for the particular + JobInstance. Ideally, all jobs should be able to + start up where they left off, but there are scenarios where this is not + possible. It is entirely up to the developer to + ensure that a new JobInstance is created in this scenario. + However, Spring Batch does provide some help. If a + Job should never be restarted, but should always + be run as part of a new JobInstance, then the + restartable property may be set to 'false': + + + <job id="footballJob" restartable="false"> + <step id="playerload" next="gameLoad"/> + <step id="gameLoad" next="playerSummarization"/> + <step id="playerSummarization"/> + </job> + + + + To phrase it another way, setting restartable to false means "this + Job does not support being started again". Restarting a Job that is not + restartable will cause a JobRestartException to + be thrown: + + + Job job = new SimpleJob(); + job.setRestartable(false); + + JobParameters jobParameters = new JobParameters(); + + JobExecution firstExecution = jobRepository.createJobExecution(job, jobParameters); + jobRepository.saveOrUpdate(firstExecution); + + try { + jobRepository.createJobExecution(job, jobParameters); + fail(); + } + catch (JobRestartException e) { + // expected + } + + + + This snippet of JUnit code shows how attempting to create a + JobExecution the first time for a non restartable + job will cause no issues. However, the second + attempt will throw a JobRestartException. +
+ +
+ Intercepting Job execution + + During the course of the execution of a + Job, it may be useful to be notified of various + events in its lifecycle so that custom code may be executed. The + SimpleJob allows for this by calling a + JobListener at the appropriate time: + + + public interface JobExecutionListener { + + void beforeJob(JobExecution jobExecution); + + void afterJob(JobExecution jobExecution); + + } + + + + JobListeners can be added to a + SimpleJob via the listeners element on the + job: + + + <job id="footballJob"> + <step id="playerload" next="gameLoad"/> + <step id="gameLoad" next="playerSummarization"/> + <step id="playerSummarization"/> + <listeners> + <listener class="org.springframework.batch.sample.SampleListener"/> + </listeners> + </job> + + + + It should be noted that afterJob will be + called regardless of the success or failure of the + Job. If success or failure needs to be determined + it can be obtained from the JobExecution: + + + void afterJob(JobExecution jobExecution){ + if( jobExecution.getStatus() == BatchStatus.COMPLETED ){ + //job success + } + else if(jobExecution.getStatus() == BatchStatus.FAILED){ + //job failure + } + } + + +
+ +
+ JobFactory and Stateful Components in Steps + + Unlike many traditional Spring applications, many of the + components of a batch application are stateful; the file readers and + writers are obvious examples. The recommended way to deal with this is + to create a fresh ApplicationContext for each job + execution. If the Job is launched from the + command line with CommandLineJobRunner, this is + trivial. For more complex launching scenarios where jobs are executed in + parallel or serially from the same process, some extra steps have to be + taken to ensure that the ApplicationContext is + refreshed. This is preferable to using prototype scope for the stateful + beans because then they would not receive lifecycle callbacks from the + container at the end of use. (e.g. through destroy-method in XML) + + The strategy provided by Spring Batch to deal with this scenario + is the JobFactory, and the samples provide an + example of a specialized implementation that can load an + ApplicationContext and close it properly when the + job is finished. A relevant examples is + ClassPathXmlApplicationContextJobFactory and its + use in the adhoc-job-launcher-context.xml and the + quartz-job-launcher-context.xml, which can be found in the + Samples project. +
+
+ +
+ Configuring a JobRepository + + As described in earlier, the JobRepository is + used for basic CRUD operations of the various persisted domain objects + within Spring Batch, such as JobExecution and + StepExecution. It is required by many of the major + framework features, such as the JobLauncher, + Job, and Step. The batch + namespace abstracts away many of the implementation details of the + JobRepository implementations and their + collaborators. However, there are still a few configuration options + available: + + + <job-repository id="jobRepository" + dataSource="dataSource" + transactionManager="transactionManager" + isolation-level-for-create="serializable" + table-prefix="BATCH_" + /> + + + + None of the configuration options listed above are required except + the id. If they are not set, the defaults shown above will be used. They + are shown above for awareness purposes. + +
+ Transaction Configuration For the JobRepository + + If the namespace is used, transactional advice will be + automatically created around the repository. This is to ensure that the + batch meta data, including state that is necessary for restarts after a + failure, is persisted correctly. The behavior of the framework is not + well defined if the repository methods are not transactional. The + isolation level in the create* method attributes is + specified separately to ensure that when jobs are launched, if two + processes are trying to launch the same job at the same time, only one + will succeed. The default isolation level for that method is + SERIALIZABLE, which is quite aggressive: READ_COMMITTED would work just + as well; READ_UNCOMMITTED would be fine if two processes are not likely + to collide in this way. However, since a call to the + create* method is quite short, it is unlikely + that the SERIALIZED will cause problems, as long as the database + platform supports it. However, this can be overridden: + + + <job-repository id="jobRepository" + isolation-level-for-create="ISOLATION_REPEATABLE_READ" /> + + + + If the namespace or factory beans aren't used then it is also + essential to configure the transactional behavior of the repository + using AOP: + + + <aop:config> + <aop:advisor + pointcut="execution(* org.springframework.batch.core..*Repository+.*(..))" + <advice-ref="txAdvice" /> + </aop:config> + + <tx:advice id="txAdvice" transaction-manager="transactionManager"> + <tx:attributes> + <tx:method name="*" /> + </tx:attributes> + </tx:advice> + + + + This fragment can be used as is, with almost no changes. Remember + also to include the appropriate namespace declarations and to make sure + spring-tx and spring-aop (or the whole of spring) are on the + classpath. +
+ +
+ Changing the table prefix + + Another modifiable property of the + JobRepository is the table prefix of the + meta-data tables. By default they are all prefaced with BATCH_. + BATCH_JOB_EXECUTION and BATCH_STEP_EXECUTION are two examples. However, + there are potential reasons to modify this prefix. If the schema names + needs to be prepended to the table names, or if more than one set of + meta data tables is needed within the same schema, then the table prefix + will need to be changed: + + + <job-repository id="jobRepository" + table-prefix="SYSTEM.TEST_" + /> + + + + Given the above changes, every query to the meta data tables will + be prefixed with "SYSTEM.TEST_". BATCH_JOB_EXECUTION will be referred to + as SYSTEM.TEST_JOB_EXECUTION. + + + Only the table prefix is configurable. The table and column + names are not. + +
+ +
+ In-Memory Repository + + There are scenarios in which you may not want to persist your + domain objects to the database. One reason may be speed; storing domain + objects at each commit point takes extra time. Another reason may be + that you just don't need to persist status for a particular job. For + this reason, Spring batch provides an in-memory Map version of the job + respository: + + <bean id="jobRepository" + class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" /> +
+
+ +
+ Configuring a JobLauncher + + The most basic implementation of the + JobLauncher interface is the + SimpleJobLauncher. Its only required dependency is + a JobRepository, in order to obtain an + execution: + + <bean id="jobLauncher" + class="org.springframework.batch.execution.launch.SimpleJobLauncher"> + <property name="jobRepository" ref="jobRepository" /> + </bean> + + Once a JobExecution is + obtained, it is passed to the execute method of + Job, ultimately returning the + JobExecution to the caller: + + + + + + + + + + + + The sequence is straightforward and works well when launched from a + scheduler. However, issues arise when trying to launch from an HTTP + request. In this scenario, the launching needs to be done asynchronously + so that the SimpleJobLauncher returns immediately + to its caller. This is because it is not good practice to keep an HTTP + request open for the amount of time needed by long running processes such + as batch. An example sequence is below: + + + + + + + + + + + + The SimpleJobLauncher can easily be + configured to allow for this scenario by configuring a + TaskExecutor: + + <bean id="jobLauncher" + class="org.springframework.batch.execution.launch.SimpleJobLauncher"> + <property name="jobRepository" ref="jobRepository" /> + <property name="taskExecutor"> + <bean class="org.springframework.core.task.SimpleAsyncTaskExecutor" /> + </property> + </bean> + + Any implementation of the spring TaskExecutor + interface can be used to control how jobs are asynchronously + executed. +
+ +
+ Running a Job + + At a minimum, launching a batch job requires two things: the + Job to be launched and a + JobLauncher. Both can be contained within the same + context or different contexts. For example, if launching a job from the + command line, a new JVM will be instantiated for each Job, and thus every + job will have its own JobLauncher. However, if + running from within a web container within the scope of an + HttpRequest, there will usually be one + JobLauncher, configured for asynchronous job + launching, that multiple requests will invoke to launch their jobs. + +
+ Running Jobs from the Command Line + + For users that want to run their jobs from an enterprise + scheduler, the command line is the primary interface. This is because + most schedulers (with the exception of Quartz unless using the + NativeJob) work directly with operating system + processes, primarily kicked off with shell scripts. There are many ways + to launch a Java process besides a shell script, such as Perl, Ruby, or + even 'build tools' such as ant or maven. However, because most people + are familiar with shell scripts, this example will focus on them. + +
+ The CommandLineJobRunner + + Because the script launching the job must kick off a Java + Virtual Machine, there needs to be a class with a main method to act + as the primary entry point. Spring Batch provides an implementation + that serves just this purpose: + CommandLineJobRunner. It's important to note + that this is just one way to bootstrap your application, but there are + many ways to launch a Java process, and this class should in no way be + viewed as definitive. The CommandLineJobRunner + performs four tasks: + + + + Load the appropriate + ApplicationContext + + + + Parse command line arguments into + JobParameters + + + + Locate the appropriate job based on arguments + + + + Use the JobLauncher provided in the + application context to launch the job. + + + + All of these tasks are accomplished using only the arguments + passed in. The following are required arguments: + + + CommandLineJobRunner arguments + + + + + jobPath + + The location of the XML file that will be used to + create an ApplicationContext. This file + should contain everything needed to run the complete + Job + + + + jobName + + The name of the job to be run. + + + +
+ + These arguments must be passed in with the path first and the + name second. All arguments after these are considered to be + JobParameters and must be in the format of 'name=value': + + bash$ java CommandLineJobRunner endOfDayJob.xml endOfDay schedule.date(date)=2008/01/01 + + In most cases you would want to use a manifest to declare your + main class in a jar, but for simplicity, the class was used directly. + This example is using the same 'EndOfDay' example from . The first argument is 'endOfDayJob.xml', which is + the Spring ApplicationContext containing the + Job. The second argument, 'endOfDay' represents + the job name. The final argument, 'schedule.date(date)=2008/01/01' + will be converted into JobParameters. An + example of the XML configuration is below: + + <job id="endOfDay"> + <steps> + <step id="step1" parent="simpleStep" /> + <!-- Step details removed for clarity --> + </steps> + </job> + + <!-- Launcher details removed for clarity --> + <beans: bean id="jobLauncher" + class="org.springframework.batch.core.launch.support.SimpleJobLauncher" /> + + This example is overly simplistic, since there are many more + requirements to a run a batch job in Spring Batch in general, but it + serves to show the two main requirements of the + CommandLineJobRunner: + Job and + JobLauncher +
+ +
+ ExitCodes + + When launching a batch job from the command-line, an enterprise + scheduler is often used. Most schedulers are fairly dumb and work only + at the process level. This means that they only know about some + operating system process such as a shell script that they're invoking. + In this scenario, the only way to communicate back to the scheduler + about the success or failure of a job is through return codes. A + return code is a number that is returned to a scheduler by the process + that indicates the result of the run. In the simplest case: 0 is + success and 1 is failure. However, there may be more complex + scenarios: If job A returns 4 kick off job B, and if it returns 5 kick + off job C. This type of behavior is configured at the scheduler level, + but it is important that a processing framework such as Spring Batch + provide a way to return a numeric representation of the 'Exit Code' + for a particular batch job. In Spring Batch this is encapsulated + within an ExitStatus, which is covered in more + detail in Chapter 5. For the purposes of discussing exit codes, the + only important thing to know is that an + ExitStatus has an exit code property that is + set by the framework (or the developer) and is returned as part of the + JobExecution returned from the + JobLauncher. The + CommandLineJobRunner converts this string value + to a number using the ExitCodeMapper + interface: + + public interface ExitCodeMapper { + + public int intValue(String exitCode); +} + + The essential contract of an + ExitCodeMapper is that, given a string exit + code, a number representation will be returned. The default + implementation used by the job runner is the SimpleJvmExitCodeMapper + that returns 0 for completion, 1 for generic errors, and 2 for any job + runner errors such as not being able to find a + Job in the provided context. If anything more + complex than the 3 values above is needed, then a custom + implementation of the ExitCodeMapper interface + must be supplied. Because the + CommandLineJobRunner is the class that creates + an ApplicationContext, and thus cannot be + 'wired together', any values that need to be overwritten must be + autowired. This means that if an implementation of + ExitCodeMapper is found within the BeanFactory, + it will be injected into the runner after the context is created. All + that needs to be done to provide your own + ExitCodeMapper is to declare the implementation + as a root level bean and ensure that it is part of the + ApplicationContext that is loaded by the + runner. +
+
+ +
+ Running Jobs from within a container + + +
+
+ +
+ Advanced Meta-Data Usage + + So far, both the JobLauncher and JobRepository interfaces have been + discussed. Together, they represent simple launching of a job, and basic + CRUD operations of batch domain objects: + + + + + + + + + + + + A JobLauncher uses the + JobRepository to create new + JobExecution objects and run them. + Job and Step implementations + later use the same JobRepository for basic updates + of the same executions during the running of a Job. + The basic operations suffice for simple scenarios, but in a large batch + environment with hundreds of batch jobs and complex scheduling + requirements, more advanced access of the meta data is required: + + + + + + + + + + + + The JobExplorer and + JobOperator interfaces, which will be discussed + below, add additional functionality for querying and controlling the meta + data. + +
+ Querying the repository + + The most basic need before any advanced features is the ability to + query the repository for existing executions. This functionality is + provided by the JobExplorer interface: + + + public interface JobExplorer { + + List<JobInstance> getJobInstances(String jobName, int start, int count); + + JobExecution getJobExecution(Long executionId); + + StepExecution getStepExecution(Long jobExecutionId, Long stepExecutionId); + + JobInstance getJobInstance(Long instanceId); + + List<JobExecution> getJobExecutions(JobInstance jobInstance); + + Set<JobExecution> findRunningJobExecutions(String jobName); + } + + + + As is evident from the method signatures above, + JobExplorer is a read-only version of the + JobRepository, and like the + JobRepository, it can be easily configured via a + factory bean: + + + <bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean" + p:dataSource-ref="dataSource" /> + + + + Earlier in this + chapter, it was mentioned that the table prefix of the + JobRepository can be modified to allow for + different versions or schemas. Because the + JobExplorer is working with the same tables, it + too needs the ability to set a prefix: + + + <bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean" + p:dataSource-ref="dataSource" p:tablePrefix="BATCH_" /> + + +
+ +
+ JobOperator + + As previously discussed, the JobRepository + provides CRUD operations on the meta-data, and the + JobExplorer provides read-only operations on the + meta-data. However, those operations are most useful when used together + to perform common monitoring tasks such as stopping, restarting, or + summarizing a Job, as is commonly done by batch operators. Spring Batch + provides for these types of operations via the + JobOperator interface: + + + public interface JobOperator { + + List<Long> getExecutions(long instanceId) throws NoSuchJobInstanceException; + + List<Long> getJobInstances(String jobName, int start, int count) throws NoSuchJobException; + + Set<Long> getRunningExecutions(String jobName) throws NoSuchJobException; + + String getParameters(long executionId) throws NoSuchJobExecutionException; + + Long start(String jobName, String parameters) + throws NoSuchJobException, JobInstanceAlreadyExistsException; + + Long restart(long executionId) + throws JobInstanceAlreadyCompleteException, NoSuchJobExecutionException, + NoSuchJobException, JobRestartException; + + Long startNextInstance(String jobName) + throws NoSuchJobException, JobParametersNotFoundException, JobRestartException, + JobExecutionAlreadyRunningException, JobInstanceAlreadyCompleteException; + + boolean stop(long executionId) throws NoSuchJobExecutionException, JobExecutionNotRunningException; + + String getSummary(long executionId) throws NoSuchJobExecutionException; + + Map<Long, String> getStepExecutionSummaries(long executionId) throws NoSuchJobExecutionException; + + Set<String> getJobNames(); + + } + + + + The above operations represent methods from many different + interfaces, such as JobLauncher, + JobRepository, + JobExplorer, and + JobRegistry. For this reason, the provided + implementation of JobOperator, + SimpleJobOperator, has many dependencies: + + + <bean id="jobOperator" class="org.springframework.batch.core.launch.support.SimpleJobOperator"> + <property name="jobExplorer"> + <bean class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean"> + <property name="dataSource" ref="dataSource" /> + </bean> + </property> + <property name="jobRepository" ref="jobRepository" /> + <property name="jobRegistry" ref="jobRegistry" /> + <property name="jobLauncher" ref="jobLauncher" /> + </bean> + + +
+ +
+ JobParametersIncrementer + + Most of the methods on JobOperator are + self-explanatory, and more detailed explanations can be found on the + javadoc + of the interface. However, the + startNextInstance method is worth noting. This + method will always start a new instance of a Job. + This can be extremely useful if there are serious issues in a + JobExecution and the Job + needs to be started over again from the beginning. Unlike + JobLauncher though, which requires a new + JobParameters object that will trigger a new + JobInstance if the parameters are different from + any previous set of parameters, the + startNextInstance method will use the + JobParametersIncrementer tied to the + Job to force the Job to a + new instance: + + + public interface JobParametersIncrementer { + + JobParameters getNext(JobParameters parameters); + + } + + + + The contract of JobParametersIncrementer is + that, given a JobParameters + object, it will return the 'next' JobParameters + object by incrementing any necessary values it may contain. This + strategy is useful because the framework has no way of knowing what + changes to the JobParameters make it the 'next' + instance. For example, if the only value in + JobParameters is a date, and the next instance + should be created, should that value be incremented by one day? Or one + week (if the job is weekly for instance)? The same can be said for any + numerical values that help to identify the Job, + as shown below: + + + public class SampleIncrementer implements JobParametersIncrementer { + + public JobParameters getNext(JobParameters parameters) { + if (parameters==null || parameters.isEmpty()) { + return new JobParametersBuilder().addLong("run.id", 1L).toJobParameters(); + } + long id = parameters.getLong("run.id",1L) + 1; + return new JobParametersBuilder().addLong("run.id", id).toJobParameters(); + } +} + + + + In this example, the value with a key of 'run.id' is used to + discriminate between JobInstances. If the + JobParameters passed in is null, it can be + assumed that the Job has never been run before + and thus its initial state can be returned. However, if not, the old + value is obtained, incremented by one, and returned. An incrementer can + be associated with Job via the 'incrementer' + attribute in the namespace: + + + <job id="footballJob" incrementer="sampleIncrementer"> + <step id="playerload" next="gameLoad"/> + <step id="gameLoad" next="playerSummarization"/> + <step id="playerSummarization"/> + </job> + + +
+ +
+ Stopping a Job + + One of the most common use cases of + JobOperator is gracefully stopping a + Job: + + + Set<Long> executions = jobOperator.getRunningExecutions("sampleJob"); + + jobOperator.stop(executions.iterator().next()); + + + The shutdown is not immediate, since there is no way to force + immediate shutdown, especially if the execution is currently in + developer code that the framework has no control over, such as a + business service. However, as soon as control is returned back to the + framework, it will set the status of the current + StepExecution to + BatchStatus.STOPPED, save it, then do the same + for the JobExecution before finishing. +
+
+ diff --git a/docs/src/site/docbook/reference/readersAndWriters.xml b/docs/src/site/docbook/reference/readersAndWriters.xml index b268f11b5..9e84d4710 100644 --- a/docs/src/site/docbook/reference/readersAndWriters.xml +++ b/docs/src/site/docbook/reference/readersAndWriters.xml @@ -1,2695 +1,2774 @@ - - - - ItemReaders and ItemWriters - - All batch processing can be described in its most simple form as - reading in large amounts of data, performing some type of calculation or - transformation, and writing the result out. Spring Batch provides three key - interfaces to help perform bulk reading and writing: - ItemReader, ItemProcessor and - ItemWriter. - -
- ItemReader - - Although a simple concept, an ItemReader is - the means for providing data from many different types of input. The most - general examples include: - - Flat File- Flat File Item Readers read lines of data from a - flat file that typically describe records with fields of data - defined by fixed positions in the file or delimited by some special - character (e.g. Comma). - - - - XML - XML ItemReaders process XML independently of - technologies used for parsing, mapping and validating objects. Input - data allows for the validation of and XML file against an XSD - schema. - - - - Database - A database resource is accessed that returns - resultsets which can be mapped to objects for processing. The - default SQL ItemReaders invoke a RowMapper to - return objects, keep track of the current row if restart is - required, basic statistics, and some transaction enhancements that - will be explained later. - - There are many more possibilities, but we'll focus on the - basic ones for this chapter. A complete list of all available ItemReaders - can be found in Appendix A. - - ItemReader is a basic interface for generic - input operations: - - public interface ItemReader<T> { - - T read() throws Exception, UnexpectedInputException, ParseException; - -} - - - The read method defines the most essential - contract of the ItemReader, calling it returns one - Item, returning null if no more items are left. An item might represent a - line in a file, a row in a database, or an element in an XML file. It is - generally expected that these will be mapped to a usable domain object - (i.e. Trade, Foo, etc) but there is no requirement in the contract to do - so. - - It is expected that implementations of the - ItemReader interface will be forward only. However, - if the underlying resource is transactional (such as a JMS queue) then - calling read may return the same logical item on subsequent calls in a - rollback scenario. It is also worth noting that a lack of items to process - by an ItemReader will not cause an exception to be - thrown. For example, a database ItemReader that is - configured with a query that returns 0 results will simply return null on - the first invocation of read. -
- -
- ItemWriter - - ItemWriter is similar in functionality to an - ItemReader, but with inverse operations. Resources - still need to be located, opened and closed but they differ in that an - ItemWriter writes out, rather than reading in. In - the case of databases or queues these may be inserts, updates or sends. - The format of the serialization of the output is specific for every batch - job. - - As with ItemReader, - ItemWriter is a fairly generic interface: - - public interface ItemWriter<T> { - - void write(List<? extends T> items) throws Exception; - -} - - - As with read on - ItemReader, write provides - the basic contract of ItemWriter, it will attempt - to write out the list of items passed in as long as it is open. Because it - is generally expected that items will be 'batched' together into a chunk - and then output, the interface accepts a list, rather than an item by - itself. After writing out the list, any flushing that may be necessary can - be performed before returning from the write method. For example, if - writing to a Hibernate DAO, multiple calls to write can be made, one for - each item. The writer can then call close on the hibernate Session before - returning. -
- -
- ItemProcessor - - The ItemReader and - ItemWriter interfaces are both very useful for - their specific tasks, but what if you want to insert business logic before - writing? One option for both reading and writing is to use the composite - pattern: create an ItemWriter that contains another - ItemWriter, or an ItemReader - that contains another ItemReader. For - example: - - public class CompositeItemWriter<T> implements ItemWriter<T> { - - ItemWriter<T> itemWriter; - - public CompositeItemWriter(ItemWriter<T> itemWriter) { - this.itemWriter = itemWriter; - } - - public void write(List<? extends T> items) throws Exception { - - //Add business logic here - - itemWriter.write(item); - } - - public void setDelegate(ItemWriter<T> itemWriter){ - this.itemWriter = itemWriter; - } -} - - The class above contains another ItemWriter - that it delgates to after having provided some business logic. This - pattern could easily be used for an ItemReader as - well, perhaps to obtain more reference data based upon the input that was - provided by the main ItemReader. It is also useful - if you need to control the call to write yourself. - However, if you only want to 'transform' the item passed in for writing - before it is actual written, there isn't much need to call - write yourself, you just want to modify the item. - For this scenario, Spring Batch provides the - ItemProcessor interface: - - public interface ItemProcessor<I, O> { - - O process(I item) throws Exception; -} - - An ItemProcessor is very simple, given one - object, transform it and return another. The object provided may or may - not be of the same type. The point is that business logic may be applied - within process, and is completely up to the developer to create. An - ItemProcessor can be wired directly into a step, - For example, assuming an ItemReader provides a - class of type Foo, and it needs to be converted to type Bar before being - written out. An ItemTransformer can be written that - performs the conversion: - - public class Foo {} - - public class Bar { - public Bar(Foo foo) {} - } - - public class FooProcessor implements ItemProcessor<Foo,Bar>{ - - //Perform simple transformation, convert a Foo to a Bar - public Bar transform(Foo foo) throws Exception { - return new Bar(foo); - } - } - - public class BarWriter implements ItemWriter<Bar>{ - - public void write(Bar bar) throws Exception { - //write bar - } - - //rest of class ommitted for clarity - } - - In the very simple example above, there is a class - Foo, a class Bar, and a - class FooProcessor that adheres to the - ItemProcessor interface. The transformation is - simple, but any type of transformation could be done here. The - BarWriter will be used to write out 'Bars', - throwing an exception if any other type is provided. Similarly, the - FooProcessor will throw an exception if anything but a - Foo is provided. The - FooProcessor can then be injected into a - Step: - - - <job id="ioSampleJob"> - <step name="step1"> - <tasklet reader="fooReader" processor="fooProcessor" writer="barWriter" commit-interval="2"/> - </step> - </job> - - - -
- Chaining ItemProcessors - - Performing a single transformation is useful in many scenarios, - but what if you want to 'chain' together multiple ItemProcessors? This - can be accomplished using the composite pattern mentioned previously. To - update the previous, single transformation, example, - Foo will be Transformed to - Bar, which will be transformed to - Foobar and written out: - - public class Foo {} - - public class Bar { - public Bar(Foo foo) {} - } - - public class Foobar{ - public Foobar(Bar bar){} - } - - public class FooProcessor implements ItemProcessor<Foo,Bar>{ - - //Perform simple transformation, convert a Foo to a Bar - public Bar transform(Foo foo) throws Exception { - return new Bar(foo); - } - } - - public class BarProcessor implements ItemProcessor<Bar,FooBar>{ - - public FooBar transform(Bar bar) throws Exception { - return new Foobar(bar); - } - } - - public class FoobarWriter implements ItemWriter<FooBar>{ - - public void write(Object item) throws Exception { - //write Foobar - } - - //rest of class ommitted for clarity - } - - A FooTransformer and - BarTransformer can be 'chained' together to give - the resultant Foobar: - - CompositeItemProcessor<Foo,Foobar> compositeProcessor = new CompositeItemProcessor<Foo,Foobar>(); - List itemProcessors = new ArrayList(); - itemProcessors.add(new FooTransformer()); - itemProcessors.add(new BarTransformer()); - compositeProcessor.setItemProcessors(itemProcessors); - - Just as with the previous example, the composite processor can be - configured into the Step: - - - <job id="ioSampleJob"> - <step name="step1"> - <tasklet reader="fooReader" processor="compositeProcessor" writer="foobarWriter" commit-interval="2"/> - </step> - </job> - - <bean id="compositeItemProcessor" - class="org.springframework.batch.item.support.CompositeItemProcessor"> - <property name="itemProcessors"> - <list> - <bean class="..FooProcessor" /> - <bean class="..BarProcessor" /> - </list> - </property> - </bean> - - -
-
- -
- ItemStream - - Both ItemReaders and ItemWriters serve their individual purposes - well, but there is a common concern among both of them that necessitates - another interface. In general, as part of the scope of a batch job, - readers and writers need to be opened, closed, and require a mechanism for - persisting state: - - public interface ItemStream { - - void open(ExecutionContext executionContext) throws StreamException; - - void update(ExecutionContext executionContext) throws ItemStreamException; - - void close(ExecutionContext executionContext) throws StreamException; -} - - - Before describing each method, its worth briefly mentioning the - ExecutionContext. Clients of an - ItemReader that also implements - ItemStream should call - open before any calls to - read, to open any resources such as files or - obtain connections. A similar restriction applies to an - ItemWriter that also implements - ItemStream. As mentioned in Chapter 2, if expected - data is found in the ExecutionContext, it may be - used to start the ItemReader or - ItemWriter at a location other than its initial - state. Conversely, close will be called to ensure - any resources allocated during open will be - released safely. update is called primarily to - ensure that any state currently being held is loaded into the provided - ExecutionContext. This method will be called before - committing, to ensure that the current state is persisted in the database - before commit. - - In the special case where the client of an - ItemStream is a Step (from - the Spring Batch Core), an ExecutionContext is - created for each StepExecution to allow users to - store the state of a particular execution, with the expectation that it - will be returned if the same JobInstance is started - again. For those familiar with Quartz, the semantics are very similar to a - Quartz JobDataMap. -
- -
- The Delegate Pattern and Registering with the Step - - Note that the CompositeItemWriter is an - example of the delegation pattern, which is common in Spring Batch. The - delegates themselves might implement callback interfaces like - ItemStream or StepListener. - If they do, and they are being used in conjunction with Spring Batch Core - as part of a Step in a Job, - then they almost certainly need to be registered manually with the - Step. Registration is automatic when the reader, - writer, or processor is directly wired into the Step. The delegates are - not known to the Step, so they need to be injected - as listeners or streams (or both if appropriate): - - - <job id="ioSampleJob"> - <step name="step1"> - <tasklet reader="fooReader" processor="fooProcessor" writer="compositeItemWriter" - commit-interval="2"/> - <streams> - <stream ref="barWriter" /> - </streams> - </step> - </job> - - <bean id="compositeItemWriter" - class="...CompositeItemWriter" > - <property name="delegate" ref="barWriter" /> - </bean> - - <bean id="barWriter" class="...BarWriter" /> - - -
- -
- Flat Files - - One of the most common mechanisms for interchanging bulk data has - always been the flat file. Unlike XML, which has an agreed upon standard - for defining how it is structured (XSD), anyone reading a flat file must - understand ahead of time exactly how the file is structured. In general, - all flat files fall into two types: Delimited and Fixed Length. - -
- The FieldSet - - When working with flat files in Spring Batch, regardless of - whether it is for input or output, one of the most important classes is - the FieldSet. Many architectures and libraries - contain abstractions for helping you read in from a file, but they - usually return a String or an array of Strings. This really only gets - you halfway there. A FieldSet is Spring Batch’s - abstraction for enabling the binding of fields from a file resource. It - allows developers to work with file input in much the same way as they - would work with database input. A FieldSet is - conceptually very similar to a Jdbc ResultSet. - FieldSets only require one argument, a String - array of tokens. Optionally, you can also configure in the names of the - fields so that the fields may be accessed either by index or name as - patterned after ResultSet: - - - String[] tokens = new String[]{"foo", "1", "true"}; - FieldSet fs = new DefaultFieldSet(tokens); - String name = fs.readString(0); - int value = fs.readInt(1); - boolean booleanValue = fs.readBoolean(2); - - - - There are many more options on the FieldSet - interface, such as Date, long, - BigDecimal, etc. The biggest advantage of the - FieldSet is that it provides consistent parsing - of flat file input. Rather than each batch job parsing differently in - potentially unexpected ways, it can be consistent, both when handling - errors caused by a format exception, or when doing simple data - conversions. -
- -
- FlatFileItemReader - - A flat file is any type of file that contains at most - two-dimensional (tabular) data. Reading flat files in the Spring Batch - framework is facilitated by the class - FlatFileItemReader, which provides basic - functionality for reading and parsing flat files. The two most important - required dependencies of FlatFileItemReader are - Resource and LineMapper. - The LineMapper interface will be - explored more in the next sections. The resource property represents a - Spring Core Resource. Documentation explaining - how to create beans of this type can be found in Spring - Framework, Chapter 4.Resources. Therefore, this - guide will not go into the details of creating - Resource objects. However, a simple example of a - file system resource can be found below: - Resource resource = new FileSystemResource("resources/trades.csv"); - - - In complex batch environments the directory structures are often - managed by the EAI infrastructure where drop zones for external - interfaces are established for moving files from ftp locations to batch - processing locations and vice versa. File moving utilities are beyond - the scope of the spring batch architecture but it is not unusual for - batch job streams to include file moving utilities as steps in the job - stream. Its sufficient that the batch architecture only needs to know - how to locate the files to be processed. Spring Batch begins the process - of feeding the data into the pipe from this starting point. However, - Spring - Integration provides many of these types of - services. - - The other properties in FlatFileItemReader - allow you to further specify how your data will be interpreted: - Flat File Item Reader Properties - - - - - - - Property - - Type - - Description - - - - - - encoding - - String - - Specifies what text encoding to use - - default is "ISO-8859-1" - - - - comments - - String[] - - Specifies line prefixes that indicate - comment rows - - - - linesToSkip - - int - - Number of lines to ignore at the top of - the file - - - - skippedLinesCallbackHandler - - LineCallbackHandler - - Interface which passes the raw line - content of the lines in the file to be skipped. If linesToSkip - is set to 2, then this interface will be called twice. - - - - firstLineIsHeader - - boolean - - Indicates that the first line of the file - is a header containing field names. If the column names have - not been set yet and the tokenizer extends - AbstractLineTokenizer, field names will be set automatically - from this line - - - - recordSeparatorPolicy - - RecordSeparatorPolicy - - Used to determine where the line endings - are and do things like continue over a line ending if inside a - quoted string. - - - -
- -
- LineMapper - - As with RowMapper, which takes a low - level construct such as ResultSet and returns an Object, - flat file procesing requires the same construct to convert a String - line into an Object: - public interface LineMapper<T> { - - T mapLine(String line, int lineNumber) throws Exception; - } - - - - The basic contract is that, given the current line, and the line - number its associated with, return a resulting domain object. This is - similar to RowMapper in that each line is - associated with it's line number, just as each row in a - ResultSet is tied to the row number it belongs - to. This allows for tying the line number to the resulting domain - object for identity comparison, or for more informative logging. - However, unlike RowMapper, the - LineMapper is given a raw line which, as - discussed above, only gets you halfway there. What is needed is - tokenization of the line into a FieldSet, which - can then be mapped to an object, as described below. -
- -
- LineTokenizer - - Because there can be many formats of flat file data, which all - need to be converted to a FieldSet so that a - useful domain object can be created from them, an abstraction for - turning a line of input into a FieldSet is - necessary. In Spring Batch, this is called a - LineTokenizer: - - - public interface LineTokenizer { - - FieldSet tokenize(String line); - - } - - - - The contract of a LineTokenizer is such - that, given a line of input (in theory the - String could encompass more than one line) a - FieldSet representing the line will be - returned. This can then be passed to a - FieldSetMapper. Spring Batch contains the - following LineTokenizer implementations: - - - - DelmitedLineTokenizer - Used for - files that separate records by a delimiter. The most common is a - comma, but pipes or semicolons are often used as well - - - - FixedLengthTokenizer - Used for - tokenizing files where each record is separated by a 'fixed width' - that must be defined per record. - - - - PrefixMatchingCompositeLineTokenizer - - Tokenizer that determines which among a list of Tokenizers - should be used on a particular line by checking against a - prefix. - - -
- -
- FieldSetMapper - - The FieldSetMapper interface defines a - single method, mapLine, which takes a - FieldSet object and maps its contents to an - object. This object may be a custom DTO or domain object, or it could - be as simple as an array, depending on your needs. The - FieldSetMapper is used in conjunction with the - LineTokenizer to translate a line of data from - a resource into an object of the desired type: - - - public interface FieldSetMapper<T> { - - T mapFieldSet(FieldSet fieldSet); - - } - - - - The pattern used is the same as RowMapper - used by JdbcTemplate. -
- -
- DefaultLineMapper - - Now that the basic interfaces for reading in flat files have - been defined, it becomes clear that three basic steps are - required: - - Read one line from the file. - - - - Pass the string line into the LineTokenizer#tokenize() - method, in order to retrieve a - FieldSet - - - - Pass the FieldSet returned from tokenizing to a - FieldSetMapper, returning the result from the ItemReader#read() - method - - - - The two interfaces described above represent two separate tasks: - converting a line into a FieldSet, and mapping - a FieldSet to a domain object. Becaue the input - of a LineTokenizer matches the input of the - LineMapper (a line), and the output of a - FieldSetMapper matches the output of the - LineMapper, and this is the deafult behavior - most users will need, a default implementation that uses both a - LineTokenizer and - FieldSetMapper is provided: - - - public class DefaultLineMapper<T> implements LineMapper<T>, InitializingBean { - - private LineTokenizer tokenizer; - - private FieldSetMapper<T> fieldSetMapper; - - public T mapLine(String line, int lineNumber) throws Exception { - return fieldSetMapper.mapFieldSet(tokenizer.tokenize(line)); - } - - public void setLineTokenizer(LineTokenizer tokenizer) { - this.tokenizer = tokenizer; - } - - public void setFieldSetMapper(FieldSetMapper<T> fieldSetMapper) { - this.fieldSetMapper = fieldSetMapper; - } - } - - - - The above functionality is provided in a default implementation, - rather than being built into the reader itself (as was done in - previous versions of the framework) in order to allow users greater - flexibility in controlling the parsing process, especially if access - to the raw line is needed. -
- -
- Simple Delimited File Reading Example - - The following example will be used to illustrate this using an - actual domain scenario. This particular batch job reads in football - players from the following file: ID,lastName,firstName,position,birthYear,debutYear - "AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996", - "AbduRa00,Abdullah,Rabih,rb,1975,1999", - "AberWa00,Abercrombie,Walter,rb,1959,1982", - "AbraDa00,Abramowicz,Danny,wr,1945,1967", - "AdamBo00,Adams,Bob,te,1946,1969", - "AdamCh00,Adams,Charlie,wr,1979,2003" - - The contents of this file will be mapped to the following Player - domain object: - public class Player implements Serializable { - - private String ID; - private String lastName; - private String firstName; - private String position; - private int birthYear; - private int debutYear; - - public String toString() { - - return "PLAYER:ID=" + ID + ",Last Name=" + lastName + - ",First Name=" + firstName + ",Position=" + position + - ",Birth Year=" + birthYear + ",DebutYear=" + - debutYear; - } - - // setters and getters... - } - - - In order to map a FieldSet into a Player - object, a FieldSetMapper that returns players - needs to be defined: - - - protected static class PlayerFieldSetMapper implements FieldSetMapper<Player> { - public Object mapLine(FieldSet fieldSet) { - Player player = new Player(); - - player.setID(fieldSet.readString(0)); - player.setLastName(fieldSet.readString(1)); - player.setFirstName(fieldSet.readString(2)); - player.setPosition(fieldSet.readString(3)); - player.setBirthYear(fieldSet.readInt(4)); - player.setDebutYear(fieldSet.readInt(5)); - - return player; - } - } - - - The file can then be read by correctly constructing a - FlatFileItemReader and calling - read: - - - FlatFileItemReader<Player> itemReader = new FlatFileItemReader<Player>(); - itemReader.setResource(new FileSystemResource("resources/players.csv")); - //DelimitedLineTokenizer defaults to comma as it's delimiter - LineMapper<Player> lineMapper = new DefaultLineMapper<Player>(); - lineMapper.setLineTokenizer(new DelimitedLineTokenizer()); - lineMapper.setFieldSetMapper(new PlayerFieldSetMapper()); - itemReader.setLineMapper(lineMapper); - itemReader.open(new ExecutionContext()); - Player player = itemReader.read(); - - - - Each call to read will return a new - Player object from each line in the file. When the end of the file is - reached, null will be returned. -
- -
- Mapping fields by name - - There is one additional functionality a - LineTokenizer that is similar in function to a - Jdbc ResultSet. The names of the fields can be - injected into the LineTokenizer to increase the - readability of the mapping function. First, the column names of all - fields in the flat file are injected into the - LineTokenizer: - - - tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); - - - a FieldSetMapper can this use this - information as follows: - - - public class PlayerMapper implements FieldSetMapper<Player> { - public Object mapLine(FieldSet fs) { - - if(fs == null){ - return null; - } - - Player player = new Player(); - player.setID(fs.readString("ID")); - player.setLastName(fs.readString("lastName")); - player.setFirstName(fs.readString("firstName")); - player.setPosition(fs.readString("position")); - player.setDebutYear(fs.readInt("debutYear")); - player.setBirthYear(fs.readInt("birthYear")); - - return player; - } - - } - -
- -
- Automapping FieldSets to Domain Objects - - For many, having to write a specific - FieldSetMapper is equally as cumbersome as - writing a specific RowMapper for a - JdbcTemplate. Spring Batch makes this easier by - providing a FieldSetMapper that automatically - maps fields by matching a field name with a setter on the object using - the JavaBean specification. Again using the football example, the - FieldSetMapper configuration looks like the - following: - - - <bean id="fieldSetMapper" - class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper"> - <property name="prototypeBeanName" value="player" /> - </bean> - - <bean id="player" - class="org.springframework.batch.sample.domain.Player" - scope="prototype" /> - - - - For each entry in the FieldSet, the - mapper will look for a corresponding setter on a new instance of the - Player object (for this reason, prototype scope - is required) in the same way the Spring container will look for - setters matching a property name. Each available field in the - FieldSet will be mapped, and the resultant - Player object will be returned, with no code - required. -
- -
- Fixed Length file formats - - So far only delimited files have been discussed in much detail, - however, they represent only half of the file reading picture. Many - organizations that use flat files use fixed length formats. An example - fixed length file is below: - - UK21341EAH4121131.11customer1 - UK21341EAH4221232.11customer2 - UK21341EAH4321333.11customer3 - UK21341EAH4421434.11customer4 - UK21341EAH4521535.11customer5 - - While this looks like one large field, it actually represent 4 - distinct fields: - - - - ISIN: Unique identifier for the item being order - 12 - characters long. - - - - Quantity: Number of this item being ordered - 3 characters - long. - - - - Price: Price of the item - 5 characters long. - - - - Customer: Id of the customer ordering the item - 9 - characters long. - - - - When configuring the - FixedLengthLineTokenizer, each of these lengths - must be provided in the form of ranges: - - - <bean id="fixedLengthLineTokenizer" - class="org.springframework.batch.io.file.transform.FixedLengthTokenizer"> - <property name="names" value="ISIN, Quantity, Price, Customer" /> - <property name="columns" value="1-12, 13-15, 16-20, 21-29" /> - </bean> - - - - This LineTokenizer will return the same - FieldSet as if a delimiter had been used, - allowing the same approach above to be used such as the - BeanWrapperFieldSetMapper, in a way that is - ignorant of how the actual line was parsed. - - It should be noted that supporting the above ranges requires a - specialized property editor be configured anywhere in the - ApplicationContext: - - - <bean id="customEditorConfigurer" - class="org.springframework.beans.factory.config.CustomEditorConfigurer"> - <property name="customEditors"> - <map> - <entry key="org.springframework.batch.item.file.transform.Range[]"> - <bean class="org.springframework.batch.item.file.transform.RangeArrayPropertyEditor" /> - </entry> - </map> - </property> - </bean> - - -
- -
- Multiple record types within a single file - - All of the file reading examples up to this point have all made - a key assumption for simplicity's sake: one record equals one line. - However, this may not always be the case. Its very common that a file - might have records spanning multiple lines with multiple formats. The - following excerpt from a file illustrates this: - - HEA;0013100345;2007-02-15 - NCU;Smith;Peter;;T;20014539;F - BAD;;Oak Street 31/A;;Small Town;00235;IL;US - SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States - BIN;VISA;VISA-12345678903 - LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47 - LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87 - SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS - FOT;2;2;267.34 - - Everything between the line starting with 'HEA' and the line - starting with 'FOT' is considered one record. The - PrefixMatchingCompositeLineTokenizer makes this easier by matching the - prefix in a line with a particular tokenizer: - - - <bean id="orderFileDescriptor" - class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer"> - <property name="tokenizers"> - <map> - <entry key="HEA" value-ref="headerRecordDescriptor" /> - <entry key="FOT" value-ref="footerRecordDescriptor" /> - <entry key="BCU" value-ref="businessCustomerLineDescriptor" /> - <entry key="NCU" value-ref="customerLineDescriptor" /> - <entry key="BAD" value-ref="billingAddressLineDescriptor" /> - <entry key="SAD" value-ref="shippingAddressLineDescriptor" /> - <entry key="BIN" value-ref="billingLineDescriptor" /> - <entry key="SIN" value-ref="shippingLineDescriptor" /> - <entry key="LIT" value-ref="itemLineDescriptor" /> - <entry key="" value-ref="defaultLineDescriptor" /> - </map> - </property> - </bean> - - - - This ensures that the line will be parsed correctly, which is - especially important for fixed length input. Any users of the - FlatFileItemReader in this scenario must - continue calling read until the footer for - the record is returned, allowing them to return a complete order as - one 'item'. -
- -
- Exception Handling in flat files - - There are many scenarios when tokenizing a line that cause - exceptions to be thrown. Many flat files are imperfect and contain - records that aren't formatted correctly. Many users choose to skip the - lines causing these errors, logging out the issue, original line, and - line number, for manual inspection later. (or by another batch job) - For this reason, Spring Batch provides a hierarchy of exceptions for - handling parse exceptions: - FlatFileParseException and - FlatFileFormatException. - FlatFileParseException is thrown by the - FlatFileItemReader when any errors are - encountered while trying to read a file. - FlatFileFormatException is thrown by - implementations of the LineTokenizer interface, - and indicates a more specific error encountered while - tokenizing. - -
- IncorrectTokenCountException - - Both DelimitedLineTokenizer and - FixedLengthLineTokenizer have the ability to - specify column names that can be used for creating a - FieldSet. However, if the number of column - names doesn't match the number of columns found while tokenizing a - line the FieldSet can't be created, and a - IncorrectTokenCountException is thrown, which contains the number of - tokens encountered, and the number expected: - - - tokenizer.setNames(new String[] {"A", "B", "C", "D"}); - - try{ - tokenizer.tokenize("a,b,c"); - } - catch(IncorrectTokenCountException e){ - assertEquals(4, e.getExpectedCount()); - assertEquals(3, e.getActualCount()); - } - - - - Because the tokenizer was configured with 4 columns, but only - 3 tokens were found in the file, an IncorrectTokenCountException was - thrown. -
- -
- IncorrectLineLengthException - - Files formatted in a fixed length format have additional - requirements when parsing because unlike a delimited format, each - column must strictly adhere to the width defined for it. If the - total line length doesn't add up to the widest value of this column, - an exception is thrown: - - - tokenizer.setColumns(new Range[] { new Range(1, 5), new Range(6, 10), new Range(11, 15) }); - try { - tokenizer.tokenize("12345"); - fail("Expected IncorrectLineLengthException"); - } - catch (IncorrectLineLengthException ex) { - assertEquals(15, ex.getExpectedLength()); - assertEquals(5, ex.getActualLength()); - } - - - - The configured ranges for the tokenizer above are: 1-5, 6-10, - and 11-15, thus the total length of the line expected is 15. - However, in this case a line of length 5 was passed in, causing an - IncorrectLineLengthException to be thrown. - Throwing an exception here rather than only mapping the first column - allows the processing of the line to fail earlier, and with more - information than it would if it failed while trying to read in - column 2 in a FieldSetMapper. However, there - are scenarios where the length of the line isn't always constant. - For this reason, validation of line length can be turned off via the - 'strict' property: - - - tokenizer.setColumns(new Range[] { new Range(1, 5), new Range(6, 10) }); - tokenizer.setStrict(false); - FieldSet tokens = tokenizer.tokenize("12345"); - assertEquals("12345", tokens.readString(0)); - assertEquals("", tokens.readString(1)); - - - - The above example is almost identical to the one before it, - except the tokenizer.setStrict(false) was called. This setting tells - the tokenizer to not enforce line lengths when tokenizing the line. - A FieldSet is now correctly created and - returned. However, it will only contain empty tokens for the - remaining values. -
-
-
- -
- FlatFileItemWriter - - Writing out to flat files has the same problems and issues that - reading in from a file must overcome. It must be able to write out in - either delimited or fixed length formats in a transactional - manner. - -
- LineAggregator - - Just as the LineTokenizer interface is - necessary to take an item and turn it into a string, file writing must - have a way to aggregate multiple fields into a single string for - writing to a file. In Spring Batch this is the - LineAggregator: - - - public interface LineAggregator<T> { - - public String aggregate(T item); - - } - - - - The LineAggregator is the opposite of a - LineTokenizer. - LineTokenizer takes a - String and returns a - FieldSet, whereas - LineAggregator takes an - item and returns a - String. - -
- PassThroughLineAggregator - - The most basic implementation of the LineAggregator interface - is the PassThroughLineAggregator, which simply assumes that the - object is already a string, or that it's string representation is - acceptable for writing: - - - public class PassThroughLineAggregator<T> implements LineAggregator<T> { - - public String aggregate(T item) { - return item.toString(); - } - } - - - - The above implementation is useful if direct control of - creating the string is required, but the advantages of a - FlatFileItemWriter, such as transaction and restart support, are - necessary. -
-
- -
- Simplified File Writing Example - - Now that the LineAggregator interface and - it's most basic implementation, PassThroughLineAggregator, has been - defined the basic flow of writing can be explained: - - - - The object to be written is passed to the - LineAggregator in order to obtain a - String. - - - - The returned String is written to the - configured file. - - - - The following excerpt from the - FlatFileItemWriter expresses this in - code: - - - public void write(T item) throws Exception { - write(lineAggregator.aggregate(item) + LINE_SEPARATOR); - } - - - - A simple configuration would look like the following: - - - <bean id="itemWriter" - class="org.springframework.batch.io.file.FlatFileItemWriter"> - <property name="resource" - value="file:target/test-outputs/20070122.testStream.multilineStep.txt" /> - <property name="lineAggregator"> - <bean class="org.springframework.batch.item.file.transform.PassThroughLineAggregator"/> - </property> - </bean> - - -
- -
- FieldExtractor - - The above example may be useful for the most basic uses of a - writing to a file. However, most users of the FlatFileItemWriter will - will have a domain object that needs to be written out, and thus must - be converted into a line. In file reading, the following was - required: - - Read one line from the file. - - - - Pass the string line into the LineTokenizer#tokenize() - method, in order to retrieve a - FieldSet - - - - Pass the FieldSet returned from tokenizing to a - FieldSetMapper, returning the result from the ItemReader#read() - method - - - - File writing has similar, but inverse steps: - - - - Pass the item to be written to the writer - - - - convert the fields on the item into an array - - - - aggregate the resulting array into a line - - - - Because there is no way for the framework to know which fields - from the object need to be written out, a FieldExtractor must be - written to accomplish the task: - - - public interface FieldExtractor<T> { - - Object[] extract(T item); - - } - - - - Implementations of the FieldExtractor - interface should create an array from the fields of the provided - object, which can then be written out with a delimited between the - elements, or as part of a field-width line. - -
- PassThroughFieldExtractor - - There are many cases where an array or something that can be - converted to an array, such as a Collection, - needs to be written out. For example, a List - could be passed through, in which case it only needs to be converted - to an Object array to be written out. For this type of scenario the - PassThroughFieldExtractor can be used. It should be noted, that if - the object passed in is not an array, and not a Collection, then an - Object array containing solely the item will be returned. -
- -
- BeanWrapperFieldExtractor - - As with the BeanWrapperFieldSetMapper - described in the file reading section, it is much preferrable to - configure how to convert an domain object to an object array, rather - than writing the conversion yourself. The - BeanWrapperFieldExtractor provides just this - type of functionality: - - - BeanWrapperFieldExtractor<Name> extractor = new BeanWrapperFieldExtractor<Name>(); - extractor.setNames(new String[] { "first", "last", "born" }); - - String first = "Alan"; - String last = "Turing"; - int born = 1912; - - Name n = new Name(first, last, born); - Object[] values = extractor.extract(n); - - assertEquals(first, values[0]); - assertEquals(last, values[1]); - assertEquals(born, values[2]); - - - - This extractor implementation has only one required property, - the names of the fields to map. Just as the - BeanWrapperFieldSetMapper needs field names - to map fields on the FieldSet to setters on the provided object, the - BeanWrapperFieldExtractor needs names to map - to getters for creating an object array. It's worth noting that the - order of the names determines the order of the fields within the - array. -
-
- -
- Delimited File Writing Example - - The most basic flat file format is one in which all fields are - separated by a delimiter. This can be accomplished using a - DelimitedLineAggregator. The example below writes out a simple domain - object that represents a credit to a customer account: - - - public class CustomerCredit { - - private int id; - - private String name; - - private BigDecimal credit; - - public CustomerCredit(int id, String name, BigDecimal credit) { - this.id = id; - this.name = name; - this.credit = credit; - } - - //getters and setters removed for clarity - } - - - - Because a domain object is being used, an implementation of the - FieldExtractor interface must be provided, along with the delimiter to - use: - - - <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"> - <property name="resource" ref="outputResource" /> - <property name="lineAggregator"> - <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator"> - <property name="delimiter" value=","/> - <property name="fieldExtractor"> - <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor"> - <property name="names" value="name,credit"/> - </bean> - </property> - </bean> - </property> - </bean> - - - - In this case, the - BeanWrapperFieldExtractor described earlier in - this chapter is used to turn the name and credit fields within - CustomerCredit into an object array, which is then written out with - commas between each field. -
- -
- Fixed Width File Writing Example - - Delimited is not the only type of flat file format, many prefer - to use a set width for each column to delineate between fields, which - is usually referred to as 'fixed width'. Spring Batch supports this in - file writing via the FormatterLineAggregator. Using the same - CustomerCredit domain object described above, it can be configured as - follows: - - - <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"> - <property name="resource" ref="outputResource" /> - <property name="lineAggregator"> - <bean class="org.springframework.batch.item.file.transform.FormatterLineAggregator"> - <property name="fieldExtractor"> - <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor"> - <property name="names" value="name,credit" /> - </bean> - </property> - <property name="format" value="%-9s%-2.0f" /> - </bean> - </property> - </bean> - - - - Most of the above example should look familiar. However, the - value of the format property is new: - - - <property name="format" value="%-9s%-2.0f" /> - - - - The underlying implementation is built using the same Formatter - added as part of Java 5. Most details on how to configure a formatter - can be found in the javadoc of Formatter. -
- -
- Handling file creation - - FlatFileItemReader has a very simple - relationship with file resources. When the reader is initialized, it - opens the file if it exists, and throws an exception if it does not. - File writing isn't quite so simple. At first glance it seems like a - similar straight forward contract should exist for - FlatFileItemWriter, if the file already exists, - throw an exception, if it does not, create it and start writing. - However, potentially restarting a Job can cause - issues. In normal restart scenarios, the contract is reversed, if the - file exists start writing to it from the last known good position, if - it does not, throw an exception. However, what happens if the file - name for this job is always the same? In this case, you would want to - delete the file if it exists, unless it's a restart. Because of this - possibility, the FlatFileItemWriter contains - the property, shouldDeleteIfExists. Setting - this property to true will cause an existing file with the same name - to be deleted when the writer is opened. -
-
-
- -
- XML Item Readers and Writers - - Spring Batch provides transactional infrastructure for both reading - XML records and mapping them to Java objects as well as writing Java - objects as XML records. - - - Constraints on streaming XML - - The StAX API is used for I/O as other standard XML parsing APIs do - not fit batch processing requirements (DOM loads the whole input into - memory at once and SAX controls the parsing process allowing the user - only to provide callbacks). - - - Lets take a closer look how XML input and output works in Spring - Batch. First, there are a few concepts that vary from file reading and - writing but are common across Spring Batch XML processing. With XML - processing, instead of lines of records (FieldSets) that need to be - tokenized, it is assumed an XML resource is a collection of 'fragments' - corresponding to individual records: - - - - - - - - - - - Figure 3.1: XML Input - - - The 'trade' tag is defined as the 'root element' in the scenario - above. Everything between '<trade>' and '</trade>' is - considered one 'fragment'. Spring Batch uses Object/XML Mapping (OXM) to - bind fragments to objects. However, Spring Batch is not tied to any - particular XML binding technology. Typical use is to delegate to Spring - OXM, which provides uniform abstraction for the most - popular OXM technologies. The dependency on Spring OXM is optional and you - can choose to implement Spring Batch specific interfaces if desired. The - relationship to the technologies that OXM supports can be shown as the - following: - - - - - - - - - - - Figure 3.2: OXM Binding - - - Now with an introduction to OXM and how one can use XML fragments to - represent records, let's take a closer look at readers and writers. - -
- StaxEventItemReader - - The StaxEventItemReader configuration - provides a typical setup for the processing of records from an XML input - stream. First, lets examine a set of XML records that the - StaxEventItemReader can process. - - -<?xml version="1.0" encoding="UTF-8"?> -<records> - <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain"> - <isin>XYZ0001</isin> - <quantity>5</quantity> - <price>11.39</price> - <customer>Customer1</customer> - </trade> - <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain"> - <isin>XYZ0002</isin> - <quantity>2</quantity> - <price>72.99</price> - <customer>Customer2c</customer> - </trade> - <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain"> - <isin>XYZ0003</isin> - <quantity>9</quantity> - <price>99.99</price> - <customer>Customer3</customer> - </trade> -</records> - - - - To be able to process the XML records the following is needed: - - - Root Element Name - Name of the root element of the fragment - that constitutes the object to be mapped. The example - configuration demonstrates this with the value of trade. - - - - Resource - Spring Resource that represents the file to be - read. - - - - FragmentDeserializer - UnMarshalling - facility provided by Spring OXM for mapping the XML fragment to an - object. - - - - - <bean id="itemReader" class="org.springframework.batch.item.xml.StaxEventItemReader"> - <property name="fragmentRootElementName" value="customer" /> - <property name="resource" value="data/iosample/input/input.xml" /> - <property name="unmarshaller" ref="customerCreditMarshaller" /> - </bean> - - <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller"> - <property name="aliases"> - <util:map id="aliases"> - <entry key="customer" - value="org.springframework.batch.sample.domain.trade.CustomerCredit" /> - <entry key="price" value="java.math.BigDecimal" /> - <entry key="name" value="java.lang.String" /> - </util:map> - </property> - </bean> - - - Notice that in this example we have chosen to use an - XStreamMarshaller that requires an alias passed - in as a map with the first key and value being the name of the fragment - (i.e. root element) and the object type to bind. Then, similar to a - FieldSet, the names of the other elements that - map to fields within the object type are described as key/value pairs in - the map. In the configuration file we can use a spring configuration - utility to describe the required alias as follows: - - - <bean id="itemReader" class="org.springframework.batch.item.xml.StaxEventItemReader"> - <property name="fragmentRootElementName" value="customer" /> - <property name="resource" value="data/iosample/input/input.xml" /> - <property name="unmarshaller" ref="customerCreditMarshaller" /> - </bean> - - <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller"> - <property name="aliases"> - <util:map id="aliases"> - <entry key="customer" - value="org.springframework.batch.sample.domain.trade.CustomerCredit" /> - <entry key="price" value="java.math.BigDecimal" /> - <entry key="name" value="java.lang.String" /> - </util:map> - </property> - </bean> - - - On input the reader reads the XML resource until it recognizes a - new fragment is about to start (by matching the tag name by default). - The reader creates a standalone XML document from the fragment (or at - least makes it appear so) and passes the document to a deserializer - (typically a wrapper around a Spring OXM - Unmarshaller) to map the XML to a Java - object. - - In summary, if you were to see this in scripted code like Java the - injection provided by the spring configuration would look something like - the following: - - - StaxEventItemReader xmlStaxEventItemReader = new StaxEventItemReader() - Resource resource = new ByteArrayResource(xmlResource.getBytes()) - - Map aliases = new HashMap(); - aliases.put("customer","org.springframework.batch.sample.domain.trade.CustomerCredit"); - aliases.put("price","java.math.BigDecimal"); - aliases.put("name","java.lang.String"); - Marshaller marshaller = new XStreamMarshaller(); - marshaller.setAliases(aliases); - xmlStaxEventItemReader.setUnmarshaller(marshaller); - xmlStaxEventItemReader.setResource(resource); - xmlStaxEventItemReader.setFragmentRootElementName("customer"); - xmlStaxEventItemReader.open(new ExecutionContext()); - - boolean hasNext = true - - CustomerCredit credit = null; - - while (hasNext) { - credit = xmlStaxEventItemReader.read(); - if (credit == null) { - hasNext = false; - } else { - println trade; - } - } - - -
- -
- StaxEventItemWriter - - Output works symmetrically to input. The - StaxEventItemWriter needs a - Resource, a serializer, and a rootTagName. A Java - object is passed to a serializer (typically a wrapper around Spring OXM - Marshaller) which writes to a - Resource using a custom event writer that filters - the StartDocument and - EndDocument events produced for each fragment by - the OXM tools. We'll show this in an example using the - MarshallingEventWriterSerializer. The Spring - configuration for this setup looks as follows: - - - <bean id="itemWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter"> - <property name="resource" ref="outputResource" /> - <property name="marshaller" ref="customerCreditMarshaller" /> - <property name="rootTagName" value="customers" /> - <property name="overwriteOutput" value="true" /> - </bean> - - - - The configuration sets up the three required properties and - optionally sets the overwriteOutput=true, mentioned earlier in the - chapter for specifying whether an existing file can be overwritten. It - should be noted the marshaller used for the writer is the exact same as - the one used in the reading example from earlier in the chapter: - - - <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller"> - <property name="aliases"> - <util:map id="aliases"> - <entry key="customer" - value="org.springframework.batch.sample.domain.trade.CustomerCredit" /> - <entry key="price" value="java.math.BigDecimal" /> - <entry key="name" value="java.lang.String" /> - </util:map> - </property> - </bean> - - - To summarize with a Java example, the following code illustrates - all of the points discussed, demonstrating the programmatic setup of the - required properties. - - StaxEventItemWriter staxItemWriter = new StaxEventItemWriter() - FileSystemResource resource = new FileSystemResource(File.createTempFile("StaxEventWriterOutputSourceTests", "xml")) - - Map aliases = new HashMap(); - aliases.put("customer","org.springframework.batch.sample.domain.trade.CustomerCredit"); - aliases.put("price","java.math.BigDecimal"); - aliases.put("name","java.lang.String"); - Marshaller marshaller = new XStreamMarshaller(); - marshaller.setAliases(aliases); - - staxItemWriter.setResource(resource); - staxItemWriter.setMarshaller(marshaller); - staxItemWriter.setRootTagName("trades"); - staxItemWriter.setOverwriteOutput(true); - - ExecutionContext executionContext = new ExecutionContext(); - staxItemWriter.open(executionContext); - CustomerCredit Credit = new CustomerCredit(); - trade.setPrice(11.39); - credit.setName("Customer1"); - staxItemWriter.write(trade); - -
-
- -
- Multi-File Input - - It is a common requirement to process multiple files within a single - Step. Assuming the files are all formatted the - same, the MultiResourceItemReader supports this - type of input for both XML and flat file processing. Consider the - following files in a directory: - - file-1.txt file-2.txt ignored.txt - - file-1.txt and file-2.txt are formatted the same and for business - reasons should be processed together. The - MuliResourceItemReader can be used to read in both - files by using wildcards: - - - <bean id="multiResourceReader" class="org.springframework.batch.item.SortedMultiResourceItemReader"> - <property name="resources" value="classpath:data/multiResourceJob/input/file-*.txt" /> - <property name="delegate" ref="flatFileItemReader" /> - </bean> - - - - The referenced delegate is a simple - FlatFileItemReader. The above configuration will - read input from both files, handling rollback and restart scenarios. It - should be noted that, as with any ItemReader, - adding extra input (in this case a file) could cause potential issues when - restarting. It is recommended that batch jobs work with their own - individual directories until completed successfully. -
- -
- Database - - Like most enterprise application styles, a database is the central - storage mechanism for batch. However, batch differs from other application - styles due to the sheer size of the datasets that must be worked with. The - Spring Core JdbcTemplate illustrates this problem - well. If you use JdbcTemplate with a - RowMapper, the RowMapper - will be called once for every result returned from the provided query. - This causes few issues in scenarios where the dataset is small, but the - large datasets often necessary for batch processing would cause any JVM to - crash quickly. If the SQL statement returns 1 million rows, the - RowMapper will be called 1 million times, holding - all returned results in memory until all rows have been read. Spring Batch - provides two types of solutions for this problem: Cursor and Paging - database ItemReaders. - -
- Cursor Based ItemReaders - - Using a database cursor is generally the default approach of most - batch developers, because it is the database's solution to the problem - of 'streaming' relational data. The Java - ResultSet class is essentially an object - orientated mechanism for manipulating a cursor. A - ResultSet maintains a cursor to the current row - of data. Calling next on a - ResultSet moves this cursor to the next row. - Spring Batch cursor based ItemReaders open the a cursor on - initialization, and move the cursor forward one row for every call to - read, returning a mapped object that can be - used for processing. The close method will then - be called to ensure all resources are freed up. The Spring core - JdbcTemplate gets around this problem by using - the callback pattern to completely map all rows in a - ResultSet and close before returning control back - to the method caller. However, in batch this must wait until the step is - complete. Below is a generic diagram of how a cursor based - ItemReader works, and while a SQL statement is - used as an example since it is so widely known, any technology could - implement the basic approach: - - - - - - - - - - - - The example illustrates the basic pattern. Given a 'FOO' table, - which has three columns: ID, NAME, and BAR, select all rows with an ID - greater than one but less than 7. This puts the beginning of the cursor - (row 1) on ID 2. The result of this row should be a completely mapped - Foo object, calling read() again, moves the cursor to the next row, - which is the Foo with an ID of 3. The results of these reads will be - written out after each read, thus allowing the - objects to be garbage collected. (Assuming no instance variables are - maintaining references to them) - -
- JdbcCursorItemReader - - JdbcCursorItemReader is the Jdbc - implementation of the cursor based technique. It works directly with a - ResultSet and requires a SQL statement to run - against a connection obtained from a - DataSource. The following database schema will - be used as an example: - - CREATE TABLE CUSTOMER ( - ID BIGINT IDENTITY PRIMARY KEY, - NAME VARCHAR(45), - CREDIT FLOAT -); - - Many people prefer to use a domain object for each row, so we'll - use an implementation of the RowMapper - interface to map a CustomerCredit - object: - - public class CustomerCreditRowMapper implements RowMapper { - - public static final String ID_COLUMN = "id"; - public static final String NAME_COLUMN = "name"; - public static final String CREDIT_COLUMN = "credit"; - - public Object mapRow(ResultSet rs, int rowNum) throws SQLException { - CustomerCredit customerCredit = new CustomerCredit(); - - customerCredit.setId(rs.getInt(ID_COLUMN)); - customerCredit.setName(rs.getString(NAME_COLUMN)); - customerCredit.setCredit(rs.getBigDecimal(CREDIT_COLUMN)); - - return customerCredit; - } - -} - - Because JdbcTemplate is so familiar to - users of Spring, and the JdbcCursorItemReader - shares key interfaces with it, it's useful to see an example of how to - read in this data with JdbcTemplate, in order - to contrast it with the ItemReader. For the - purposes of this example, let's assume there are 1,000 rows in the - CUSTOMER database. The first example will be using - JdbcTemplate: - - - //For simplicity sake, assume a dataSource has already been obtained - JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource); - List customerCredits = jdbcTemplate.query("SELECT ID, NAME, CREDIT from CUSTOMER", new CustomerCreditRowMapper()); - - - - After running this code snippet the customerCredits list will - contain 1,000 CustomerCredit objects. In the - query method, a connection will be obtained from the - DataSource, the provided SQL will be run - against it, and the mapRow method will be - called for each row in the ResultSet. Let's - contrast this with the approach of the - JdbcCursorItemReader: - - - JdbcCursorItemReader itemReader = new JdbcCursorItemReader(); - itemReader.setDataSource(dataSource); - itemReader.setSql("SELECT ID, NAME, CREDIT from CUSTOMER"); - itemReader.setMapper(new CustomerCreditRowMapper()); - int counter = 0; - ExecutionContext executionContext = new ExecutionContext(); - itemReader.open(executionContext); - Object customerCredit = new Object(); - while(customerCredit != null){ - customerCredit = itemReader.read(); - counter++; - } - itemReader.close(executionContext); - - - - After running this code snippet the counter will equal 1,000. If - the code above had put the returned customerCredit into a list, the - result would have been exactly the same as with the - JdbcTemplate example. However, the big - advantage of the ItemReader is that it allows - items to be 'streamed'. The read method can - be called once, and the item written out via an - ItemWriter, and then the next item obtained via - read. This allows item reading and writing to - be done in 'chunks' and committed periodically, which is the essence - of high performance batch processing. Furthermore, it is very easily - configured for injection into a Spring Batch - Step: - - - <bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader"> - <property name="dataSource" ref="dataSource"/> - <property name="sql" value="select ID, NAME, CREDIT from CUSTOMER"/> - <property name="mapper"> - <bean class="org.springframework.batch.sample.domain.trade.internal.CustomerCreditRowMapper"/> - </property> - </bean> - - - -
- Additional Properties - - Because there are so many varying options for opening a cursor - in Java, there are many properties on the - JdbcCustorItemReader that can be set: - - - JdbcCursorItemReader Properties - - - - - ignoreWarnings - - Determines whether or not SQLWarnings are logged or - cause an exception - default is true - - - - fetchSize - - Gives the Jdbc driver a hint as to the number of rows - that should be fetched from the database when more rows are - needed by the ResultSet object used - by the ItemReader. By default, no hint is given. - - - - maxRows - - Sets the limit for the maximum number of rows the - underlying ResultSet can hold at any - one time. - - - - queryTimeout - - Sets the number of seconds the driver will wait for a - Statement object to execute to the given number of seconds. - If the limit is exceeded, a - DataAccessEception is thrown. - (consult your driver vendor documentation for - details). - - - - verifyCursorPosition - - Because the same ResultSet - held by the ItemReader is passed to the - RowMapper, it's possible for users to - call ResultSet.next() themselves, which could cause issues - with the reader's internal count. Settings this value to - true will cause an exception to be thrown if the cursor - position is not the same after the - RowMapper call as it was - before. - - - - saveState - - Indicates whether or not the reader's state should be - saved in the ExecutionContext provided by - ItemStream#update(ExecutionContext) The default value is - false. - - - - driverSupportsAbsolute - - Defaults to false. Indicates whether the Jdbc driver - supports setting the absolute row on a - ResultSet. It is recommended that - this is set to true for Jdbc drivers that supports - ResultSet.absolute() as it may improve performance, - especially if a step fails while working with a large data - set. - - - - setUseSharedExtendedConnection - - Defaults to false. Indicates whether the connection - used for the cursor should be used by all other processing - thus sharing the same transaction. If this is set to false, - which is the default, then the cursor will be opened using - its own connection and will not participate in any - transactions started for the rest of the step processing. If - you set this flag to true then you must wrap the DataSource - in an - ExtendedConnectionDataSourceProxy to - prevent the connection from being closed and released after - each commit. When you set this option to true then the - statement used to open the cursor will be created with both - 'READ_ONLY' and 'HOLD_CUSORS_OVER_COMMIT' options. This - allows holding the cursor open over transaction start and - commits performed in the step processing. To use this - feature you need a database that supports this and a Jdbc - driver supporting Jdbc 3.0 or later. - - - -
-
-
- -
- HibernateCursorItemReader - - Just as normal Spring users make important decisions about - whether or not to use ORM solutions, which affects whether or not they - use a JdbcTemplate or a - HibernateTemplate, Spring Batch users have the - same options. HibernateCursorItemReader is the - Hibernate implementation of the cursor technique. Hibernate's usage in - batch has been fairly controversial. This has largely been because - hibernate was originally developed to support online application - styles. However, that doesn't mean it can't be used for batch - processing. The easiest approach for solving this problem is to use a - StatelessSession rather than a standard - session. This removes all of the caching and dirty checking hibernate - employs that can cause issues when using it in a batch scenario. For - more information on the differences between stateless and normal - hibernate sessions, refer to the documentation of your specific - hibernate release. The - HibernateCursorItemReader allows you to declare - an HQL statement and pass in a SessionFactory, - which will pass back one item per call to - read in the same basic fashion as the - JdbcCursorItemReader. Below is an example - configuration using the same 'customer credit' example as the JDBC - reader: - - - HibernateCursorItemReader itemReader = new HibernateCursorItemReader(); - itemReader.setQueryString("from CustomerCredit"); - //For simplicity sake, assume sessionFactory already obtained. - itemReader.setSessionFactory(sessionFactory); - itemReader.setUseStatelessSession(true); - int counter = 0; - ExecutionContext executionContext = new ExecutionContext(); - itemReader.open(executionContext); - Object customerCredit = new Object(); - while(customerCredit != null){ - customerCredit = itemReader.read(); - counter++; - } - itemReader.close(executionContext); - - - - This configured ItemReader will return - CustomerCredit objects in the exact same manner - as described by the JdbcCursorItemReader, - assuming hibernate mapping files have been created correctly for the - Customer table. The 'useStatelessSession' property defaults to true, - but has been added here to draw attention to the ability to switch it - on or off. It is also worth noting that the fetchSize of the - underlying cursor can be set via the setFetchSize property. As with - JdbcCursorItemReader, configuration is - straightforward: - - - <bean id="itemReader" - class="org.springframework.batch.item.database.HibernateCursorItemReader"> - <property name="sessionFactory" ref="sessionFactory" /> - <property name="queryString" value="from CustomerCredit" /> - </bean> - - -
-
- -
- Paging ItemReaders - - An alternative to using a database cursor is executing multiple - queries where each query is bringing back a portion of the results. We - refer to this portion as a page. Each query that is executed must - specify the starting row number and the number of rows that we want - returned for the page. - -
- JdbcPagingItemReader - - One implementation of a paging ItemReader - is the JdbcPagingItemReader. The - JdbcPagingItemReader needs a - PagingQueryProvider responsible for providing - the SQL queries used to retrieve the rows making up a page. Since each - database has its own strategy for providing paging support, we need to - use a different PagingQueryProvider for each - supported database type. There is also the - SimpleDelegatingPagingQueryProvider that will - auto-detect the database that is being used and determine the - appropriate PagingQueryProvider implementation. - This simplifies the configuration and is the recommended best - practice. - - The SimpleDelegatingPagingQueryProvider - requires that you specify a select clause and a from clause. You can - also provide an optional where clause. These clauses will be used to - build an SQL statement combined with the required sortKey. - - After the reader has been opened, it will pass back one item per - call to read in the same basic fashion as any - other ItemReader. The paging happens behind the - scenes when additional rows are needed. - - Below is an example configuration using a similar 'customer - credit' example as the cursor based ItemReaders above: - - <bean id="itemReader" - class="org.springframework.batch.item.database.JdbcPagingItemReader"> - <property name="dataSource" ref="dataSource"/> - <property name="queryProvider"> - <bean class="org.springframework.batch.item.database.support.SimpleDelegatingPagingQueryProvider"> - <property name="selectClause" value="select id, name, credit"/> - <property name="fromClause" value="from customer"/> - <property name="whereClause" value="where status=:status"/> - <property name="sortKey" value="id"/> - </bean> - </property> - <property name="parameterValues"> - <map> - <entry key="status" value="NEW"/> - </map> - </property> - <property name="pageSize" value="1000"/> - <property name="parameterizedRowMapper" ref="customerMapper"/> - </bean> - - - This configured ItemReader will return - CustomerCredit objects using the - ParameterizedRowMapper that must be specified. - The 'pageSize' property determines the number of entities read from - the database for each query execution. - - The 'parameterValues' property can be used to specify a Map of - parameter values for the query. If you use named parameters in the - where clause the key for each entry should match the name of the named - parameter. If you use a traditional '?' placeholder then the key for - each entry should be the number of the placeholder, starting with - 1. -
- -
- JpaPagingItemReader - - Another implementation of a paging - ItemReader is the - JpaPagingItemReader. JPA doesn't have a concept - similar to the Hibernate StatelessSession so we - have to use other features provided by the JPA specification. Since - JPA supports paging, this is a natural choice when it comes to using - JPA for batch processing. After each page is read the entities will - become detached and the persistence context will be cleared in order - to allow the entities to be garbage collected once the page is - processed. - - The JpaPagingItemReader allows you to - declare a JPQL statement and pass in a - EntityManagerFactory. It will then pass back - one item per call to read in the same basic - fashion as any other ItemReader. The paging - happens behind the scenes when additional entities are needed. Below - is an example configuration using the same 'customer credit' example - as the JDBC reader above: - - <bean id="itemReader" - class="org.springframework.batch.item.database.JpaPagingItemReader"> - <property name="entityManagerFactory" ref="entityManagerFactory"/> - <property name="queryString" value="select c from CustomerCredit c"/> - <property name="pageSize" value="1000"/> - </bean> - - - This configured ItemReader will return - CustomerCredit objects in the exact same manner - as described by the JdbcPagingItemReader above, - assuming the Customer object has the correct JPA annotations or ORM - mapping file. The 'pageSize' property determines the number of - entities read from the database for each query execution. -
- -
- IbatisPagingItemReader - - If you use IBATIS for your data access then you can use the - IbatisPagingItemReader which, as the name - indicates, is an implementation of a paging - ItemReader. IBATIS doesn't have direct support - for reading rows in pages but by providing a couple of standard - variables you can add paging support to your IBATIS queries. - - Here is an example of a configuration for a - IbatisPagingItemReader reading CustomerCredits - as in the examples above: - - <bean id="itemReader" - class="org.springframework.batch.item.database.IbatisPagingItemReader"> - <property name="sqlMapClient" ref="sqlMapClient"/> - <property name="queryId" value="getPagedCustomerCredits"/> - <property name="pageSize" value="1000"/> - </bean> - - - The IbatisPagingItemReader configuration - above references an IBATIS query called "getPagedCustomerCredits". - Here is an example of what that query should look like for - MySQL. - - <select id="getPagedCustomerCredits" resultMap="customerCreditResult"> - select id, name, credit from customer order by id asc LIMIT #_skiprows#, #_pagesize# - </select> - - - The _skiprows and - _pagesize variables are provided by the - IbatisPagingItemReader and there is also a - _page variable that can be used if necessary. - The syntax for the paging queries varies with the database used. Here - is an example for Oracle (unfortunately we need to use CDATA for some - operators since this belongs in an XML document): - - <select id="getPagedCustomerCredits" resultMap="customerCreditResult"> - select * from ( - select * from ( - select t.id, t.name, t.credit, ROWNUM ROWNUM_ from customer t order by id - ) where ROWNUM_ <![CDATA[ > ]]> ( #_page# * #_pagesize# ) - ) where ROWNUM <![CDATA[ <= ]]> #_pagesize# - </select> - -
-
- -
- Database ItemWriters - - While both Flat Files and XML have specific ItemWriters, there is - no exact equivalent in the database world. This is because transactions - provide all the functionality that is needed. ItemWriters are necessary - for files because they must act as if they're transactional, keeping - track of written items and flushing or clearing at the appropriate - times. Databases have no need for this functionality, since the write is - already contained in a transaction. Users can create their own DAOs that - implement the ItemWriter interface or use one - from a custom ItemWriter that's written for - generic processing concerns, either way, they should work without any - issues. One thing to look out for is the performance and error handling - capabilities that are provided by batching the outputs. This is most - common when using hibernate as an ItemWriter, but - could have the same issues when using Jdbc batch mode. Batching database - output doesn't have any inherent flaws, assuming we are careful to flush - and there are no errors in the data. However, any errors while writing - out can cause confusion because there is no way to know which individual - item caused an exception, or even if any individual item was - responsible, as illustrated below: - - - - - - - - - - If items are buffered before being written out, any - errors encountered will not be thrown until the buffer is flushed just - before a commit. For example, let's assume that 20 items will be written - per chunk, and the 15th item throws a DataIntegrityViolationException. - As far as the Step is concerned, all 20 item will be written out - successfully, since there's no way to know that an error will occur - until they are actually written out. Once - Session#flush() is - called, the buffer will be emptied and the exception will be hit. At - this point, there's nothing the Step can do, the - transaction must be rolled back. Normally, this exception might cause - the Item to be skipped (depending upon the skip/retry policies), and - then it won't be written out again. However, in the batched scenario, - there's no way for it to know which item caused the issue, the whole - buffer was being written out when the failure happened. The only way to - solve this issue is to flush after each item: - - - - - - - - - - - - This is a common use case, especially when using Hibernate, and - the simple guideline for implementations of - ItemWriter, is to flush on each call to - write(). Doing so allows for items to be - skipped reliably, with Spring Batch taking care internally of the - granularity of the calls to ItemWriter after an - error. -
-
- -
- Reusing Existing Services - - Batch systems are often used in conjunction with other application - styles. The most common is an online system, but it may also support - integration or even a thick client application by moving necessary bulk - data that each application style uses. For this reason, it is common that - many users want to reuse existing DAOs or other services within their - batch jobs. The Spring container itself makes this fairly easy by allowing - any necessary class to be injected. However, there may be cases where the - existing service needs to act as an ItemReader or - ItemWriter, either to satisfy the dependency of - another Spring Batch class, or because it truly is the main - ItemReader for a step. Its fairly trivial to write - an adaptor class for each service that needs wrapping, but because its - such a common concern, Spring Batch provides implementations: - ItemReaderAdapter and - ItemWriterAdapter. Both classes implement the - standard Spring method invoking the delegate pattern and are fairly simple - to set up. Below is an example of the reader: - - <bean id="itemReader" class="org.springframework.batch.item.adapter.ItemReaderAdapter"> - <property name="targetObject" ref="fooService" /> - <property name="targetMethod" value="generateFoo" /> - </bean> - - <bean id="fooService" class="org.springframework.batch.item.sample.FooService" /> - - One important point to note is that the contract of the targetMethod - must be the same as the contract for read: when - exhausted it will return null, otherwise an Object. - Anything else will prevent the framework from knowing when processing - should end, either causing an infinite loop or incorrect failure, - depending upon the implementation of the - ItemWriter. The ItemWriter - implementation is equally as simple: - - <bean id="itemWriter" class="org.springframework.batch.item.adapter.ItemWriterAdapter"> - <property name="targetObject" ref="fooService" /> - <property name="targetMethod" value="processFoo" /> - </bean> - - <bean id="fooService" class="org.springframework.batch.item.sample.FooService" /> - -
- -
- Validating Input - - During the course of this chapter, multiple approaches to parsing - input have been discussed. Each major implementation will throw an - exception if it is not 'well-formed'. The - FixedLengthTokenizer will throw an exception if a - range of data is missing. Similarly, attempting to access an index in a - RowMapper of FieldSetMapper - that doesn't exist or is in a different format than the one expected will - cause an exception to be thrown. All of these types of exceptions will be - thrown before read returns. However, they don't - address the issue of whether or not the returned item is valid. For - example, if one of the fields is an age, it obviously cannot be negative. - It will parse correctly, because it existed and is a number, but it won't - cause an exception. Since there are already a plethora of Validation - frameworks, Spring Batch does not attempt to provide yet another, but - rather provides a very simple interface that can be implemented by any - number of frameworks: - - - public interface Validator { - - void validate(Object value) throws ValidationException; - - } - - - - The contract is that the validate method - will throw an exception if the object is invalid, and return normally if - it is valid. Spring Batch provides an out of the box - ItemProcessor: - - <bean class="org.springframework.batch.item.validator.ValidatingItemProcessor"> - <property name="validator" ref="validator" /> - </bean> - - <bean id="validator" - class="org.springframework.batch.item.validator.SpringValidator"> - <property name="validator"> - <bean id="orderValidator" - class="org.springmodules.validation.valang.ValangValidator"> - <property name="valang"> - <value> - <![CDATA[ - { orderId : ? > 0 AND ? <= 9999999999 : 'Incorrect order ID' : 'error.order.id' } - { totalLines : ? = size(lineItems) : 'Bad count of order lines' - : 'error.order.lines.badcount'} - { customer.registered : customer.businessCustomer = FALSE OR ? = TRUE - : 'Business customer must be registered' - : 'error.customer.registration'} - { customer.companyName : customer.businessCustomer = FALSE OR ? HAS TEXT - : 'Company name for business customer is mandatory' - :'error.customer.companyname'} - ]]> - </value> - </property> - </bean> - </property> - </bean> - - - This simple example shows a simple - ValangValidator that is used to validate an order - object. The intent is not to show Valang functionality as much as to show - how a validator could be added. -
- -
- Preventing state persistence - - By default, all of the ItemReader and - ItemWriter implementations store their current - state in the ExecutionContext before it is - committed. However, this may not always be the desired behavior. For - example, many developers choose to make their database readers - 'rerunnable' by using a process indicator. An extra column is added to the - input data to indicate whether or not it has been processed. When a - particular record is being read (or written out) the processed flag is - flipped from false to true. The SQL statement can then contain an extra - statement in the where clause, such as: "where PROCESSED_IND = false", - thereby insuring that only unprocessed records will be returned in the - case of a restart. In this scenario, it is preferable to not store any - state, such as the current row number, since it will be irrelevant upon - restart. For this reason, all readers and writers include the 'saveState' - property: - - - <bean id="playerSummarizationSource" - class="org.springframework.batch.item.database.JdbcCursorItemReader"> - <property name="dataSource" ref="dataSource" /> - <property name="mapper"> - <bean class="org.springframework.batch.sample.mapping.PlayerSummaryMapper" /> - </property> - <property name="saveState" value="false" /> - <property name="sql"> - <value> - SELECT games.player_id, games.year_no, SUM(COMPLETES), - SUM(ATTEMPTS), SUM(PASSING_YARDS), SUM(PASSING_TD), - SUM(INTERCEPTIONS), SUM(RUSHES), SUM(RUSH_YARDS), - SUM(RECEPTIONS), SUM(RECEPTIONS_YARDS), SUM(TOTAL_TD) - from games, players where players.player_id = - games.player_id group by games.player_id, games.year_no - </value> - </property> - </bean> - - - - The ItemReader configured above will not make - any entries in the ExecutionContext for any - executions it participates in. -
- -
- Creating Custom ItemReaders and - ItemWriters - - So far in this chapter the basic contracts that exist for reading - and writing in Spring Batch and some common implementations have been - discussed. However, these are all fairly generic, and there are many - potential scenarios that may not be covered by out of the box - implementations. This section will show, using a simple example, how to - create a custom ItemReader and - ItemWriter implementation and implement their - contracts correctly. The ItemReader will also - implement ItemStream, in order to illustrate how to - make a reader or writer restartable. - -
- Custom ItemReader Example - - For the purpose of this example, a simple - ItemReader implementation that reads from a - provided list will be created. We'll start out by implementing the most - basic contract of ItemReader, - read: - - - public class CustomItemReader<T> implements ItemReader<T>{ - - List<T> items; - - public CustomItemReader(List<T> items) { - this.items = items; - } - - public T read() throws Exception, UnexpectedInputException, - NoWorkFoundException, ParseException { - - if (!items.isEmpty()) { - return items.remove(0); - } - return null; - } - } - - - - This very simple class takes a list of items, and returns one at a - time, removing it from the list. When the list empty, it returns null, - thus satisfying the most basic requirements of an - ItemReader, as illustrated below: - - List<String> items = new ArrayList<String>(); - items.add("1"); - items.add("2"); - items.add("3"); - - ItemReader itemReader = new CustomItemReader<String>(items); - assertEquals("1", itemReader.read()); - assertEquals("2", itemReader.read()); - assertEquals("3", itemReader.read()); - assertNull(itemReader.read()); - -
- Making the <classname>ItemReader</classname> - restartable - - The final challenge now is to make the - ItemReader restartable. Currently, if the power - goes out, and processing begins again, the - ItemReader must start at the beginning. This is - actually valid in many scenarios, but it is sometimes preferable that - a batch job starts off at where it left off. The key discriminant is - often whether the reader is stateful or stateless. A stateless reader - does not need to worry about restartability, but a stateful one has to - try and reconstitute its last known state on restart. For this reason, - we recommend that you keep custom readers stateless as far as - possible, so you don't have to worry about restartability. - - If you do need to store state, then in Spring Batch, this is - implemented with the ItemStream - interface: - - public class CustomItemReader<T> implements ItemReader<T>, ItemStream { - - List<T> items; - int currentIndex = 0; - private static final String CURRENT_INDEX = "current.index"; - - public CustomItemReader(List<T> items) { - this.items = items; - } - - public T read() throws Exception, UnexpectedInputException, - ParseException { - - if (currentIndex < items.size()) { - return items.get(currentIndex++); - } - - return null; - } - - public void open(ExecutionContext executionContext) throws ItemStreamException { - if(executionContext.containsKey(CURRENT_INDEX)){ - currentIndex = new Long(executionContext.getLong(CURRENT_INDEX)).intValue(); - } - else{ - currentIndex = 0; - } - } - - public void close(ExecutionContext executionContext) throws ItemStreamException {} - - public void update(ExecutionContext executionContext) throws ItemStreamException { - executionContext.putLong(CURRENT_INDEX, new Long(currentIndex).longValue()); - }; - } - - On each call to ItemStream - update method, the current index of the - ItemReader will be stored in the provided - ExecutionContext with a key of 'current.index'. - When the ItemStream open - method is called, the ExecutionContext is - checked to see if it contains an entry with that key, and if so the - current index is moved to that location. This is a fairly trivial - example, but it still meets the general contract: - - ExecutionContext executionContext = new ExecutionContext(); - ((ItemStream)itemReader).open(executionContext); - assertEquals("1", itemReader.read()); - ((ItemStream)itemReader).update(executionContext); - - List<String> items = new ArrayList<String>(); - items.add("1"); - items.add("2"); - items.add("3"); - itemReader = new CustomItemReader<String>(items); - - ((ItemStream)itemReader).open(executionContext); - assertEquals("2", itemReader.read()); - - Most ItemReaders have much more sophisticated restart logic. The - JdbcCursorItemReader, for example, stores the - row id of the last processed row in the Cursor. - - It is also worth noting that the key used within the - ExecutionContext should not be trivial. That is - because the same ExecutionContext is used for - all ItemStreams within a Step. In most cases, - simply prepending the key with the class name should be enough to - guarantee uniqueness. However, in the rare cases where two of the same - type of ItemStream are used in the same step - (which can happen if two files are need for output) then a more unique - name will be needed. For this reason, many of the Spring Batch - ItemReader and ItemWriters have a setName() property that allows this - key name to be overridden. -
-
- -
- Custom ItemWriter Example - - Implementing a Custom ItemWriter is similar - in many ways to the ItemReader example above, but - differs in enough ways as to warrant its own example. However, adding - restartability is essentially the same, so it won't be covered in this - example. As with the ItemReader example, a List - will be used in order to keep the example as simple as possible: - - public class CustomItemWriter<T> implements ItemWriter<T> { - - List<T> output = TransactionAwareProxyFactory.createTransactionalList(); - - public void write(List<? extends T> items) throws Exception { - output.addAll(items); - } - - public List<T> getOutput() { - return output; - } - } - -
- Making the <classname>ItemWriter</classname> - restartable - - To make the ItemWriter restartable we would follow the same - process as for the ItemReader, adding and - implementing the ItemStream interface to - synchronize the execution context. In the example we might have to - count the number of items processed and add that as a footer record. - If we needed to do that, we could implement - ItemStream in our - ItemWriter so that the counter was - reconstituted from the execution context if the stream was - re-opened. - - In many realistic cases, custom ItemWriters also delegate to - another writer that itself is restartable (e.g. when writing to a - file), or else it writes to a transactional resource so doesn't need - to be restartable because it is stateless. When you have a stateful - writer you should probably also be sure to implement - ItemStream as well as - ItemWriter. Remember also that the client of - the writer needs to be aware of the ItemStream, - so you may need to register it with a factory bean (e.g. one of the - StepFactoryBean implementations in Spring Batch - Core). -
-
-
- + + + + ItemReaders and ItemWriters + + All batch processing can be described in its most simple form as + reading in large amounts of data, performing some type of calculation or + transformation, and writing the result out. Spring Batch provides three key + interfaces to help perform bulk reading and writing: + ItemReader, ItemProcessor and + ItemWriter. + +
+ ItemReader + + Although a simple concept, an ItemReader is + the means for providing data from many different types of input. The most + general examples include: + + Flat File- Flat File Item Readers read lines of data from a + flat file that typically describe records with fields of data + defined by fixed positions in the file or delimited by some special + character (e.g. Comma). + + + + XML - XML ItemReaders process XML independently of + technologies used for parsing, mapping and validating objects. Input + data allows for the validation of an XML file against an XSD + schema. + + + + Database - A database resource is accessed to return + resultsets which can be mapped to objects for processing. The + default SQL ItemReaders invoke a RowMapper to + return objects, keep track of the current row if restart is + required, store basic statistics, and provide some transaction + enhancements that will be explained later. + + There are many more possibilities, but we'll focus on the + basic ones for this chapter. A complete list of all available ItemReaders + can be found in Appendix A. + + ItemReader is a basic interface for generic + input operations: + + public interface ItemReader<T> { + + T read() throws Exception, UnexpectedInputException, ParseException; + +} + + + The read method defines the most essential + contract of the ItemReader; calling it returns one + Item or null if no more items are left. An item might represent a line in + a file, a row in a database, or an element in an XML file. It is generally + expected that these will be mapped to a usable domain object (i.e. Trade, + Foo, etc) but there is no requirement in the contract to do so. + + It is expected that implementations of the + ItemReader interface will be forward only. However, + if the underlying resource is transactional (such as a JMS queue) then + calling read may return the same logical item on subsequent calls in a + rollback scenario. It is also worth noting that a lack of items to process + by an ItemReader will not cause an exception to be + thrown. For example, a database ItemReader that is + configured with a query that returns 0 results will simply return null on + the first invocation of read. +
+ +
+ ItemWriter + + ItemWriter is similar in functionality to an + ItemReader, but with inverse operations. Resources + still need to be located, opened and closed but they differ in that an + ItemWriter writes out, rather than reading in. In + the case of databases or queues these may be inserts, updates, or sends. + The format of the serialization of the output is specific to each batch + job. + + As with ItemReader, + ItemWriter is a fairly generic interface: + + public interface ItemWriter<T> { + + void write(List<? extends T> items) throws Exception; + +} + + + As with read on + ItemReader, write provides + the basic contract of ItemWriter; it will attempt + to write out the list of items passed in as long as it is open. Because it + is generally expected that items will be 'batched' together into a chunk + and then output, the interface accepts a list of items, rather than an + item by itself. After writing out the list, any flushing that may be + necessary can be performed before returning from the write method. For + example, if writing to a Hibernate DAO, multiple calls to write can be + made, one for each item. The writer can then call close on the hibernate + Session before returning. +
+ +
+ ItemProcessor + + The ItemReader and + ItemWriter interfaces are both very useful for + their specific tasks, but what if you want to insert business logic before + writing? One option for both reading and writing is to use the composite + pattern: create an ItemWriter that contains another + ItemWriter, or an ItemReader + that contains another ItemReader. For + example: + + public class CompositeItemWriter<T> implements ItemWriter<T> { + + ItemWriter<T> itemWriter; + + public CompositeItemWriter(ItemWriter<T> itemWriter) { + this.itemWriter = itemWriter; + } + + public void write(List<? extends T> items) throws Exception { + + //Add business logic here + + itemWriter.write(item); + } + + public void setDelegate(ItemWriter<T> itemWriter){ + this.itemWriter = itemWriter; + } +} + + The class above contains another ItemWriter + to which it delgates after having provided some business logic. This + pattern could easily be used for an ItemReader as + well, perhaps to obtain more reference data based upon the input that was + provided by the main ItemReader. It is also useful + if you need to control the call to write yourself. + However, if you only want to 'transform' the item passed in for writing + before it is actually written, there isn't much need to call + write yourself: you just want to modify the item. + For this scenario, Spring Batch provides the + ItemProcessor interface: + + public interface ItemProcessor<I, O> { + + O process(I item) throws Exception; +} + + An ItemProcessor is very simple; given one + object, transform it and return another. The provided object may or may + not be of the same type. The point is that business logic may be applied + within process, and is completely up to the developer to create. An + ItemProcessor can be wired directly into a step, + For example, assuming an ItemReader provides a + class of type Foo, and it needs to be converted to type Bar before being + written out. An ItemProcessor can be written that + performs the conversion: + + public class Foo {} + + public class Bar { + public Bar(Foo foo) {} + } + + public class FooProcessor implements ItemProcessor<Foo,Bar>{ + + //Perform simple transformation, convert a Foo to a Bar + public Bar process(Foo foo) throws Exception { + return new Bar(foo); + } + } + + public class BarWriter implements ItemWriter<Bar>{ + + public void write(List<? extends Bar> bars) throws Exception { + //write bars + } + + //rest of class ommitted for clarity + } + + In the very simple example above, there is a class + Foo, a class Bar, and a + class FooProcessor that adheres to the + ItemProcessor interface. The transformation is + simple, but any type of transformation could be done here. The + BarWriter will be used to write out + Bar objects, throwing an exception if any other + type is provided. Similarly, the FooProcessor will + throw an exception if anything but a Foo is + provided. The FooProcessor can then be injected + into a Step: + + + <job id="ioSampleJob"> + <step name="step1"> + <tasklet reader="fooReader" processor="fooProcessor" writer="barWriter" commit-interval="2"/> + </step> + </job> + + + +
+ Chaining ItemProcessors + + Performing a single transformation is useful in many scenarios, + but what if you want to 'chain' together multiple + ItemProcessors? This can be accomplished using + the composite pattern mentioned previously. To update the previous, + single transformation, example, Foo will be + transformed to Bar, which will be transformed to + Foobar and written out: + + public class Foo {} + + public class Bar { + public Bar(Foo foo) {} + } + + public class Foobar{ + public Foobar(Bar bar){} + } + + public class FooProcessor implements ItemProcessor<Foo,Bar>{ + + //Perform simple transformation, convert a Foo to a Bar + public Bar process(Foo foo) throws Exception { + return new Bar(foo); + } + } + + public class BarProcessor implements ItemProcessor<Bar,FooBar>{ + + public FooBar process(Bar bar) throws Exception { + return new Foobar(bar); + } + } + + public class FoobarWriter implements ItemWriter<FooBar>{ + + public void write(List<? extends FooBar> items) throws Exception { + //write items + } + + //rest of class ommitted for clarity + } + + A FooTransformer and + BarTransformer can be 'chained' together to give + the resultant Foobar: + + CompositeItemProcessor<Foo,Foobar> compositeProcessor = new CompositeItemProcessor<Foo,Foobar>(); + List itemProcessors = new ArrayList(); + itemProcessors.add(new FooTransformer()); + itemProcessors.add(new BarTransformer()); + compositeProcessor.setItemProcessors(itemProcessors); + + Just as with the previous example, the composite processor can be + configured into the Step: + + + <job id="ioSampleJob"> + <step name="step1"> + <tasklet reader="fooReader" processor="compositeProcessor" writer="foobarWriter" commit-interval="2"/> + </step> + </job> + + <bean id="compositeItemProcessor" + class="org.springframework.batch.item.support.CompositeItemProcessor"> + <property name="itemProcessors"> + <list> + <bean class="..FooProcessor" /> + <bean class="..BarProcessor" /> + </list> + </property> + </bean> + + +
+ +
+ Filtering Records + + One typical use for an item processor is to filter out records + before they are passed to the ItemWriter. Filtering is an action + distinct from skpping; skipping indicates that a record is invalid + whereas filtering simply indicates that a record should not be + written. + + For example, consider a batch job that reads a file containing + three different types of records: records to insert, records to update, + and records to delete. If record deletion is not supported by the + system, then we would not want to send any "delete" records to the + ItemWriter. But, since these records are not + actually bad records, we would want to filter them out, rather than + skip. As a result, the ItemWriter would receive only "insert" and + "update" records. + + To filter a record, one simply returns "null" from the + ItemProcessor. The framework will detect that the + result is "null" and avoid adding that item to the list of records + delivered to the ItemWriter. As usual, an + exception thrown from the ItemProcessor will + result in a skip. +
+
+ +
+ ItemStream + + Both ItemReaders and + ItemWriters serve their individual purposes well, + but there is a common concern among both of them that necessitates another + interface. In general, as part of the scope of a batch job, readers and + writers need to be opened, closed, and require a mechanism for persisting + state: + + public interface ItemStream { + + void open(ExecutionContext executionContext) throws ItemStreamException; + + void update(ExecutionContext executionContext) throws ItemStreamException; + + void close() throws ItemStreamException; +} + + + Before describing each method, we should mention the + ExecutionContext. Clients of an + ItemReader that also implement + ItemStream should call + open before any calls to + read in order to open any resources such as files + or to obtain connections. A similar restriction applies to an + ItemWriter that implements + ItemStream. As mentioned in Chapter 2, if expected + data is found in the ExecutionContext, it may be + used to start the ItemReader or + ItemWriter at a location other than its initial + state. Conversely, close will be called to ensure + that any resources allocated during open will be + released safely. update is called primarily to + ensure that any state currently being held is loaded into the provided + ExecutionContext. This method will be called before + committing, to ensure that the current state is persisted in the database + before commit. + + In the special case where the client of an + ItemStream is a Step (from + the Spring Batch Core), an ExecutionContext is + created for each StepExecution to allow users to + store the state of a particular execution, with the expectation that it + will be returned if the same JobInstance is started + again. For those familiar with Quartz, the semantics are very similar to a + Quartz JobDataMap. +
+ +
+ The Delegate Pattern and Registering with the Step + + Note that the CompositeItemWriter is an + example of the delegation pattern, which is common in Spring Batch. The + delegates themselves might implement callback interfaces like + ItemStream or StepListener. + If they do, and they are being used in conjunction with Spring Batch Core + as part of a Step in a Job, + then they almost certainly need to be registered manually with the + Step. A reader, writer, or processor that is + directly wired into the Step will be registered automatically if it + implements ItemStream or a + StepListener interface. But because the delegates + are not known to the Step, they need to be injected + as listeners or streams (or both if appropriate): + + + <job id="ioSampleJob"> + <step name="step1"> + <tasklet reader="fooReader" processor="fooProcessor" writer="compositeItemWriter" + commit-interval="2"/> + <streams> + <stream ref="barWriter" /> + </streams> + </step> + </job> + + <bean id="compositeItemWriter" + class="...CompositeItemWriter" > + <property name="delegate" ref="barWriter" /> + </bean> + + <bean id="barWriter" class="...BarWriter" /> + + +
+ +
+ Flat Files + + One of the most common mechanisms for interchanging bulk data has + always been the flat file. Unlike XML, which has an agreed upon standard + for defining how it is structured (XSD), anyone reading a flat file must + understand ahead of time exactly how the file is structured. In general, + all flat files fall into two types: Delimited and Fixed Length. Delimited + files are those in which fields are separated by a delimiter, such as a + comma. Fixed Length files have fields that are a set length. + +
+ The FieldSet + + When working with flat files in Spring Batch, regardless of + whether it is for input or output, one of the most important classes is + the FieldSet. Many architectures and libraries + contain abstractions for helping you read in from a file, but they + usually return a String or an array of Strings. This really only gets + you halfway there. A FieldSet is Spring Batch’s + abstraction for enabling the binding of fields from a file resource. It + allows developers to work with file input in much the same way as they + would work with database input. A FieldSet is + conceptually very similar to a Jdbc ResultSet. + FieldSets only require one argument, a String + array of tokens. Optionally, you can also configure in the names of the + fields so that the fields may be accessed either by index or name as + patterned after ResultSet: + + + String[] tokens = new String[]{"foo", "1", "true"}; + FieldSet fs = new DefaultFieldSet(tokens); + String name = fs.readString(0); + int value = fs.readInt(1); + boolean booleanValue = fs.readBoolean(2); + + + + There are many more options on the FieldSet + interface, such as Date, long, + BigDecimal, etc. The biggest advantage of the + FieldSet is that it provides consistent parsing + of flat file input. Rather than each batch job parsing differently in + potentially unexpected ways, it can be consistent, both when handling + errors caused by a format exception, or when doing simple data + conversions. +
+ +
+ FlatFileItemReader + + A flat file is any type of file that contains at most + two-dimensional (tabular) data. Reading flat files in the Spring Batch + framework is facilitated by the class + FlatFileItemReader, which provides basic + functionality for reading and parsing flat files. The two most important + required dependencies of FlatFileItemReader are + Resource and LineMapper. + The LineMapper interface will be + explored more in the next sections. The resource property represents a + Spring Core Resource. Documentation explaining + how to create beans of this type can be found in Spring + Framework, Chapter 4.Resources. Therefore, this + guide will not go into the details of creating + Resource objects. However, a simple example of a + file system resource can be found below: + Resource resource = new FileSystemResource("resources/trades.csv"); + + + In complex batch environments the directory structures are often + managed by the EAI infrastructure where drop zones for external + interfaces are established for moving files from ftp locations to batch + processing locations and vice versa. File moving utilities are beyond + the scope of the spring batch architecture but it is not unusual for + batch job streams to include file moving utilities as steps in the job + stream. It is sufficient that the batch architecture only needs to know + how to locate the files to be processed. Spring Batch begins the process + of feeding the data into the pipe from this starting point. However, + Spring + Integration provides many of these types of + services. + + The other properties in FlatFileItemReader + allow you to further specify how your data will be interpreted: + Flat File Item Reader Properties + + + + + + + Property + + Type + + Description + + + + + + encoding + + String + + Specifies what text encoding to use - + default is "ISO-8859-1" + + + + comments + + String[] + + Specifies line prefixes that indicate + comment rows + + + + linesToSkip + + int + + Number of lines to ignore at the top of + the file + + + + skippedLinesCallbackHandler + + LineCallbackHandler + + Interface which passes the raw line + content of the lines in the file to be skipped. If linesToSkip + is set to 2, then this interface will be called twice. + + + + firstLineIsHeader + + boolean + + Indicates that the first line of the file + is a header containing field names. If the column names have + not been set yet and the tokenizer extends + AbstractLineTokenizer, field names will be set automatically + from this line + + + + recordSeparatorPolicy + + RecordSeparatorPolicy + + Used to determine where the line endings + are and do things like continue over a line ending if inside a + quoted string. + + + +
+ +
+ LineMapper + + As with RowMapper, which takes a low + level construct such as ResultSet and returns + an Object, flat file processing requires the + same construct to convert a String line into an + Object: + public interface LineMapper<T> { + + T mapLine(String line, int lineNumber) throws Exception; + } + + + + The basic contract is that, given the current line and the line + number with which it is associated, the mapper should return a + resulting domain object. This is similar to + RowMapper in that each line is associated with + its line number, just as each row in a + ResultSet is tied to its row number. This + allows the line number to be tied to the resulting domain object for + identity comparison or for more informative logging. However, unlike + RowMapper, the + LineMapper is given a raw line which, as + discussed above, only gets you halfway there. The line must be + tokenized into a FieldSet, which can then be + mapped to an object, as described below. +
+ +
+ LineTokenizer + + An abstraction for turning a line of input into a line into a + FieldSet is necessary because there can be many + formats of flat file data that need to be converted to a + FieldSet. In Spring Batch, this interface is + the LineTokenizer: + + + public interface LineTokenizer { + + FieldSet tokenize(String line); + + } + + + + The contract of a LineTokenizer is such + that, given a line of input (in theory the + String could encompass more than one line), a + FieldSet representing the line will be + returned. This FieldSet can then be passed to a + FieldSetMapper. Spring Batch contains the + following LineTokenizer implementations: + + + + DelmitedLineTokenizer - Used for + files where fields in a record are separated by a delimiter. The + most common delimiter is a comma, but pipes or semicolons are + often used as well. + + + + FixedLengthTokenizer - Used for files + where fields in a record are each a 'fixed width'. The width of + each field must be defined for each record type. + + + + PrefixMatchingCompositeLineTokenizer + - Determines which among a list of + LineTokenizers should be used on a + particular line by checking against a prefix. + + +
+ +
+ FieldSetMapper + + The FieldSetMapper interface defines a + single method, mapLine, which takes a + FieldSet object and maps its contents to an + object. This object may be a custom DTO, a domain object, or a simple + array, depending on the needs of the job. The + FieldSetMapper is used in conjunction with the + LineTokenizer to translate a line of data from + a resource into an object of the desired type: + + + public interface FieldSetMapper<T> { + + T mapFieldSet(FieldSet fieldSet); + + } + + + + The pattern used is the same as the + RowMapper used by + JdbcTemplate. +
+ +
+ DefaultLineMapper + + Now that the basic interfaces for reading in flat files have + been defined, it becomes clear that three basic steps are + required: + + Read one line from the file. + + + + Pass the string line into the + LineTokenizer#tokenize() method, in + order to retrieve a FieldSet. + + + + Pass the FieldSet returned from + tokenizing to a FieldSetMapper, returning + the result from the ItemReader#read() + method. + + + + The two interfaces described above represent two separate tasks: + converting a line into a FieldSet, and mapping + a FieldSet to a domain object. Because the + input of a LineTokenizer matches the input of + the LineMapper (a line), and the output of a + FieldSetMapper matches the output of the + LineMapper, a default implementation that uses + both a LineTokenizer and + FieldSetMapper is provided. The + DefaultLineMapper represents the behavior most + users will need: + + + public class DefaultLineMapper<T> implements LineMapper<T>, InitializingBean { + + private LineTokenizer tokenizer; + + private FieldSetMapper<T> fieldSetMapper; + + public T mapLine(String line, int lineNumber) throws Exception { + return fieldSetMapper.mapFieldSet(tokenizer.tokenize(line)); + } + + public void setLineTokenizer(LineTokenizer tokenizer) { + this.tokenizer = tokenizer; + } + + public void setFieldSetMapper(FieldSetMapper<T> fieldSetMapper) { + this.fieldSetMapper = fieldSetMapper; + } + } + + + + The above functionality is provided in a default implementation, + rather than being built into the reader itself (as was done in + previous versions of the framework) in order to allow users greater + flexibility in controlling the parsing process, especially if access + to the raw line is needed. +
+ +
+ Simple Delimited File Reading Example + + The following example will be used to illustrate this using an + actual domain scenario. This particular batch job reads in football + players from the following file: ID,lastName,firstName,position,birthYear,debutYear + "AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996", + "AbduRa00,Abdullah,Rabih,rb,1975,1999", + "AberWa00,Abercrombie,Walter,rb,1959,1982", + "AbraDa00,Abramowicz,Danny,wr,1945,1967", + "AdamBo00,Adams,Bob,te,1946,1969", + "AdamCh00,Adams,Charlie,wr,1979,2003" + + The contents of this file will be mapped to the following + Player domain object: + public class Player implements Serializable { + + private String ID; + private String lastName; + private String firstName; + private String position; + private int birthYear; + private int debutYear; + + public String toString() { + + return "PLAYER:ID=" + ID + ",Last Name=" + lastName + + ",First Name=" + firstName + ",Position=" + position + + ",Birth Year=" + birthYear + ",DebutYear=" + + debutYear; + } + + // setters and getters... + } + + + In order to map a FieldSet into a + Player object, a + FieldSetMapper that returns players needs to be + defined: + + + protected static class PlayerFieldSetMapper implements FieldSetMapper<Player> { + public Player mapFieldSet(FieldSet fieldSet) { + Player player = new Player(); + + player.setID(fieldSet.readString(0)); + player.setLastName(fieldSet.readString(1)); + player.setFirstName(fieldSet.readString(2)); + player.setPosition(fieldSet.readString(3)); + player.setBirthYear(fieldSet.readInt(4)); + player.setDebutYear(fieldSet.readInt(5)); + + return player; + } + } + + + The file can then be read by correctly constructing a + FlatFileItemReader and calling + read: + + + FlatFileItemReader<Player> itemReader = new FlatFileItemReader<Player>(); + itemReader.setResource(new FileSystemResource("resources/players.csv")); + //DelimitedLineTokenizer defaults to comma as its delimiter + LineMapper<Player> lineMapper = new DefaultLineMapper<Player>(); + lineMapper.setLineTokenizer(new DelimitedLineTokenizer()); + lineMapper.setFieldSetMapper(new PlayerFieldSetMapper()); + itemReader.setLineMapper(lineMapper); + itemReader.open(new ExecutionContext()); + Player player = itemReader.read(); + + + + Each call to read will return a new + Player object from each line in the file. When the end of the file is + reached, null will be returned. +
+ +
+ Mapping fields by name + + There is one additional piece of functionality that is allowed + by both DelimitedLineTokenizer and + FixedLengthTokenizer that is similar in + function to a Jdbc ResultSet. The names of the + fields can be injected into either of these + LineTokenizer implementations to increase the + readability of the mapping function. First, the column names of all + fields in the flat file are injected into the tokenizer: + + + tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); + + + a FieldSetMapper can this use this + information as follows: + + + public class PlayerMapper implements FieldSetMapper<Player> { + public Player mapFieldSet(FieldSet fs) { + + if(fs == null){ + return null; + } + + Player player = new Player(); + player.setID(fs.readString("ID")); + player.setLastName(fs.readString("lastName")); + player.setFirstName(fs.readString("firstName")); + player.setPosition(fs.readString("position")); + player.setDebutYear(fs.readInt("debutYear")); + player.setBirthYear(fs.readInt("birthYear")); + + return player; + } + } + +
+ +
+ Automapping FieldSets to Domain Objects + + For many, having to write a specific + FieldSetMapper is equally as cumbersome as + writing a specific RowMapper for a + JdbcTemplate. Spring Batch makes this easier by + providing a FieldSetMapper that automatically + maps fields by matching a field name with a setter on the object using + the JavaBean specification. Again using the football example, the + BeanWrapperFieldSetMapper configuration looks + like the following: + + + <bean id="fieldSetMapper" + class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper"> + <property name="prototypeBeanName" value="player" /> + </bean> + + <bean id="player" + class="org.springframework.batch.sample.domain.Player" + scope="prototype" /> + + + + For each entry in the FieldSet, the + mapper will look for a corresponding setter on a new instance of the + Player object (for this reason, prototype scope + is required) in the same way the Spring container will look for + setters matching a property name. Each available field in the + FieldSet will be mapped, and the resultant + Player object will be returned, with no code + required. +
+ +
+ Fixed Length file formats + + So far only delimited files have been discussed in much detail, + however, they represent only half of the file reading picture. Many + organizations that use flat files use fixed length formats. An example + fixed length file is below: + + UK21341EAH4121131.11customer1 + UK21341EAH4221232.11customer2 + UK21341EAH4321333.11customer3 + UK21341EAH4421434.11customer4 + UK21341EAH4521535.11customer5 + + While this looks like one large field, it actually represent 4 + distinct fields: + + + + ISIN: Unique identifier for the item being order - 12 + characters long. + + + + Quantity: Number of this item being ordered - 3 characters + long. + + + + Price: Price of the item - 5 characters long. + + + + Customer: Id of the customer ordering the item - 9 + characters long. + + + + When configuring the + FixedLengthLineTokenizer, each of these lengths + must be provided in the form of ranges: + + + <bean id="fixedLengthLineTokenizer" + class="org.springframework.batch.io.file.transform.FixedLengthTokenizer"> + <property name="names" value="ISIN, Quantity, Price, Customer" /> + <property name="columns" value="1-12, 13-15, 16-20, 21-29" /> + </bean> + + + + Because the FixedLengthLineTokenizer uses + the same LineTokenizer interface as discussed + above, it will return the same FieldSet as if a + delimiter had been used. This allows the same approaches to be used in + handling its output, such as using the + BeanWrapperFieldSetMapper. + + It should be noted that supporting the above ranges requires a + specialized property editor be configured anywhere in the + ApplicationContext: + + + <bean id="customEditorConfigurer" + class="org.springframework.beans.factory.config.CustomEditorConfigurer"> + <property name="customEditors"> + <map> + <entry key="org.springframework.batch.item.file.transform.Range[]"> + <bean class="org.springframework.batch.item.file.transform.RangeArrayPropertyEditor" /> + </entry> + </map> + </property> + </bean> + + +
+ +
+ Multiple record types within a single file + + All of the file reading examples up to this point have all made + a key assumption for simplicity's sake: all of the records in a file + have the same format. However, this may not always be the case. It is + very common that a file might have records with different formats that + need to be tokenized differently and mapped to different objects. The + following excerpt from a file illustrates this: + + + USER;Smith;Peter;;T;20014539;F + LINEA;1044391041ABC037.49G201XX1383.12H + LINEB;2134776319DEF422.99M005LI + + + + In this file we have three types of records, "USER", "LINEA", + and "LINEB". A "USER" line corresponds to a User object. "LINEA" and + "LINEB" both correspond to Line objects, though a "LINEA" has more + information than a "LINEB". + + The ItemReader will read each line + individually, but we must specify different + LineTokenizer and + FieldSetMapper objects so that the + ItemWriter will recieve the correct items. The + PrefixMatchingCompositeLineMapper makes this + easy by allowing maps of prefixes to + LineTokenizers and prefixes to + FieldSetMappers to be configured: + + + <bean id="orderFileLineMapper" + class="org.springframework.batch.item.file.mapping.PrefixMatchingCompositeLineMapper"> + <property name="tokenizers"> + <map> + <entry key="USER" value-ref="userTokenizer" /> + <entry key="LINEA" value-ref="lineATokenizer" /> + <entry key="LINEB" value-ref="lineBTokenizer" /> + </map> + </property> + <property name="fieldSetMappers"> + <map> + <entry key="USER" value-ref="userFieldSetMapper" /> + <entry key="LINE" value-ref="lineFieldSetMapper" /> + </map> + </property> + </bean> + + + + In this example, "LINEA" and "LINEB" have separate + LineTokenizers but they both use the same + FieldSetMapper. + + The PrefixMatchingCompositeLineMapper + makes use of the PatternMatcher's + matchPattern method in order to select the + correct delegate for each line. The pattern will always match the most + specific pattern possible, regardless of the order in the + configuration. So if "LINE" and "LINEA" were both listed as prefixes, + "LINEA" would match prefix "LINEA", while "LINEB" would match prefix + "LINE". Additionally, the empty string ("") can serve as a default + prefix by matching any line not matched by any other prefix. + + + <entry key="" value-ref="defaultLineTokenizer" /> + + + + There is also a + PrefixMatchingCompositeLineTokenizer that can + be used for tokenization alone. + + It is also common for a flat file to contain records that each + span multiple lines. To handle this situation, a more complex strategy + is required. A demonstration of this common patter can be found in + . +
+ +
+ Exception Handling in flat files + + There are many scenarios when tokenizing a line may cause + exceptions to be thrown. Many flat files are imperfect and contain + records that aren't formatted correctly. Many users choose to skip + these erroneous lines, logging out the issue, original line, and line + number. These logs can later be inspected manually or or by another + batch job. For this reason, Spring Batch provides a hierarchy of + exceptions for handling parse exceptions: + FlatFileParseException and + FlatFileFormatException. + FlatFileParseException is thrown by the + FlatFileItemReader when any errors are + encountered while trying to read a file. + FlatFileFormatException is thrown by + implementations of the LineTokenizer interface, + and indicates a more specific error encountered while + tokenizing. + +
+ IncorrectTokenCountException + + Both DelimitedLineTokenizer and + FixedLengthLineTokenizer have the ability to + specify column names that can be used for creating a + FieldSet. However, if the number of column + names doesn't match the number of columns found while tokenizing a + line the FieldSet can't be created, and a + IncorrectTokenCountException is thrown, which + contains the number of tokens encountered, and the number + expected: + + + tokenizer.setNames(new String[] {"A", "B", "C", "D"}); + + try{ + tokenizer.tokenize("a,b,c"); + } + catch(IncorrectTokenCountException e){ + assertEquals(4, e.getExpectedCount()); + assertEquals(3, e.getActualCount()); + } + + + + Because the tokenizer was configured with 4 column names, but + only 3 tokens were found in the file, an + IncorrectTokenCountException was + thrown. +
+ +
+ IncorrectLineLengthException + + Files formatted in a fixed length format have additional + requirements when parsing because, unlike a delimited format, each + column must strictly adhere to its predefined width. If the total + line length doesn't add up to the widest value of this column, an + exception is thrown: + + + tokenizer.setColumns(new Range[] { new Range(1, 5), new Range(6, 10), new Range(11, 15) }); + try { + tokenizer.tokenize("12345"); + fail("Expected IncorrectLineLengthException"); + } + catch (IncorrectLineLengthException ex) { + assertEquals(15, ex.getExpectedLength()); + assertEquals(5, ex.getActualLength()); + } + + + + The configured ranges for the tokenizer above are: 1-5, 6-10, + and 11-15, thus the total length of the line expected is 15. + However, in this case a line of length 5 was passed in, causing an + IncorrectLineLengthException to be thrown. + Throwing an exception here rather than only mapping the first column + allows the processing of the line to fail earlier, and with more + information than it would if it failed while trying to read in + column 2 in a FieldSetMapper. However, there + are scenarios where the length of the line isn't always constant. + For this reason, validation of line length can be turned off via the + 'strict' property: + + + tokenizer.setColumns(new Range[] { new Range(1, 5), new Range(6, 10) }); + tokenizer.setStrict(false); + FieldSet tokens = tokenizer.tokenize("12345"); + assertEquals("12345", tokens.readString(0)); + assertEquals("", tokens.readString(1)); + + + + The above example is almost identical to the one before it, + except that tokenizer.setStrict(false) was called. This setting + tells the tokenizer to not enforce line lengths when tokenizing the + line. A FieldSet is now correctly created and + returned. However, it will only contain empty tokens for the + remaining values. +
+
+
+ +
+ FlatFileItemWriter + + Writing out to flat files has the same problems and issues that + reading in from a file must overcome. A step must be able to write out + in either delimited or fixed length formats in a transactional + manner. + +
+ LineAggregator + + Just as the LineTokenizer interface is + necessary to take an item and turn it into a + String, file writing must have a way to + aggregate multiple fields into a single string for writing to a file. + In Spring Batch this is the + LineAggregator: + + + public interface LineAggregator<T> { + + public String aggregate(T item); + + } + + + + The LineAggregator is the opposite of a + LineTokenizer. + LineTokenizer takes a + String and returns a + FieldSet, whereas + LineAggregator takes an + item and returns a + String. + +
+ PassThroughLineAggregator + + The most basic implementation of the LineAggregator interface + is the PassThroughLineAggregator, which + simply assumes that the object is already a string, or that its + string representation is acceptable for writing: + + + public class PassThroughLineAggregator<T> implements LineAggregator<T> { + + public String aggregate(T item) { + return item.toString(); + } + } + + + + The above implementation is useful if direct control of + creating the string is required, but the advantages of a + FlatFileItemWriter, such as transaction and + restart support, are necessary. +
+
+ +
+ Simplified File Writing Example + + Now that the LineAggregator interface and + its most basic implementation, + PassThroughLineAggregator, have been defined, + the basic flow of writing can be explained: + + + + The object to be written is passed to the + LineAggregator in order to obtain a + String. + + + + The returned String is written to the + configured file. + + + + The following excerpt from the + FlatFileItemWriter expresses this in + code: + + + public void write(T item) throws Exception { + write(lineAggregator.aggregate(item) + LINE_SEPARATOR); + } + + + + A simple configuration would look like the following: + + + <bean id="itemWriter" + class="org.springframework.batch.io.file.FlatFileItemWriter"> + <property name="resource" + value="file:target/test-outputs/20070122.testStream.multilineStep.txt" /> + <property name="lineAggregator"> + <bean class="org.springframework.batch.item.file.transform.PassThroughLineAggregator"/> + </property> + </bean> + + +
+ +
+ FieldExtractor + + The above example may be useful for the most basic uses of a + writing to a file. However, most users of the + FlatFileItemWriter will will have a domain + object that needs to be written out, and thus must be converted into a + line. In file reading, the following was required: + + Read one line from the file. + + + + Pass the string line into the + LineTokenizer#tokenize() method, in + order to retrieve a FieldSet + + + + Pass the FieldSet returned from + tokenizing to a FieldSetMapper, returning + the result from the ItemReader#read() + method + + + + File writing has similar, but inverse steps: + + + + Pass the item to be written to the writer + + + + convert the fields on the item into an array + + + + aggregate the resulting array into a line + + + + Because there is no way for the framework to know which fields + from the object need to be written out, a + FieldExtractor must be written to accomplish + the task of turning the item into an array: + + + public interface FieldExtractor<T> { + + Object[] extract(T item); + + } + + + + Implementations of the FieldExtractor + interface should create an array from the fields of the provided + object, which can then be written out with a delimiter between the + elements, or as part of a field-width line. + +
+ PassThroughFieldExtractor + + There are many cases where an array or something that can be + converted to an array, such as a Collection, + needs to be written out. For example, a List + could be passed through, in which case it only needs to be converted + to an Object array to be written out. For + this type of scenario the + PassThroughFieldExtractor can be used. It + should be noted, that if the object passed in is not an array, and + not a Collection, then an + Object array containing solely the item will + be returned. +
+ +
+ BeanWrapperFieldExtractor + + As with the BeanWrapperFieldSetMapper + described in the file reading section, it is often preferrable to + configure how to convert a domain object to an object array, rather + than writing the conversion yourself. The + BeanWrapperFieldExtractor provides just this + type of functionality: + + + BeanWrapperFieldExtractor<Name> extractor = new BeanWrapperFieldExtractor<Name>(); + extractor.setNames(new String[] { "first", "last", "born" }); + + String first = "Alan"; + String last = "Turing"; + int born = 1912; + + Name n = new Name(first, last, born); + Object[] values = extractor.extract(n); + + assertEquals(first, values[0]); + assertEquals(last, values[1]); + assertEquals(born, values[2]); + + + + This extractor implementation has only one required property, + the names of the fields to map. Just as the + BeanWrapperFieldSetMapper needs field names + to map fields on the FieldSet to setters on + the provided object, the + BeanWrapperFieldExtractor needs names to map + to getters for creating an object array. It is worth noting that the + order of the names determines the order of the fields within the + array. +
+
+ +
+ Delimited File Writing Example + + The most basic flat file format is one in which all fields are + separated by a delimiter. This can be accomplished using a + DelimitedLineAggregator. The example below + writes out a simple domain object that represents a credit to a + customer account: + + + public class CustomerCredit { + + private int id; + private String name; + private BigDecimal credit; + + //getters and setters removed for clarity + } + + + + Because a domain object is being used, an implementation of the + FieldExtractor interface must be provided, along with the delimiter to + use: + + + <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"> + <property name="resource" ref="outputResource" /> + <property name="lineAggregator"> + <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator"> + <property name="delimiter" value=","/> + <property name="fieldExtractor"> + <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor"> + <property name="names" value="name,credit"/> + </bean> + </property> + </bean> + </property> + </bean> + + + + In this case, the + BeanWrapperFieldExtractor described earlier in + this chapter is used to turn the name and credit fields within + CustomerCredit into an object array, which is + then written out with commas between each field. +
+ +
+ Fixed Width File Writing Example + + Delimited is not the only type of flat file format. Many prefer + to use a set width for each column to delineate between fields, which + is usually referred to as 'fixed width'. Spring Batch supports this in + file writing via the FormatterLineAggregator. + Using the same CustomerCredit domain object + described above, it can be configured as follows: + + + <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"> + <property name="resource" ref="outputResource" /> + <property name="lineAggregator"> + <bean class="org.springframework.batch.item.file.transform.FormatterLineAggregator"> + <property name="fieldExtractor"> + <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor"> + <property name="names" value="name,credit" /> + </bean> + </property> + <property name="format" value="%-9s%-2.0f" /> + </bean> + </property> + </bean> + + + + Most of the above example should look familiar. However, the + value of the format property is new: + + + <property name="format" value="%-9s%-2.0f" /> + + + + The underlying implementation is built using the same + Formatter added as part of Java 5. The Java + Formatter is based on the + printf functionality of the C programming + language. Most details on how to configure a formatter can be found in + the javadoc of Formatter. +
+ +
+ Handling file creation + + FlatFileItemReader has a very simple + relationship with file resources. When the reader is initialized, it + opens the file if it exists, and throws an exception if it does not. + File writing isn't quite so simple. At first glance it seems like a + similar straight forward contract should exist for + FlatFileItemWriter: if the file already exists, + throw an exception, and if it does not, create it and start writing. + However, potentially restarting a Job can cause + issues. In normal restart scenarios, the contract is reversed: if the + file exists, start writing to it from the last known good position, + and if it does not, throw an exception. However, what happens if the + file name for this job is always the same? In this case, you would + want to delete the file if it exists, unless it's a restart. Because + of this possibility, the FlatFileItemWriter + contains the property, shouldDeleteIfExists. + Setting this property to true will cause an existing file with the + same name to be deleted when the writer is opened. +
+
+
+ +
+ XML Item Readers and Writers + + Spring Batch provides transactional infrastructure for both reading + XML records and mapping them to Java objects as well as writing Java + objects as XML records. + + + Constraints on streaming XML + + The StAX API is used for I/O as other standard XML parsing APIs do + not fit batch processing requirements (DOM loads the whole input into + memory at once and SAX controls the parsing process allowing the user + only to provide callbacks). + + + Lets take a closer look how XML input and output works in Spring + Batch. First, there are a few concepts that vary from file reading and + writing but are common across Spring Batch XML processing. With XML + processing, instead of lines of records (FieldSets) that need to be + tokenized, it is assumed an XML resource is a collection of 'fragments' + corresponding to individual records: + + + + + + + + + + + Figure 3.1: XML Input + + + The 'trade' tag is defined as the 'root element' in the scenario + above. Everything between '<trade>' and '</trade>' is + considered one 'fragment'. Spring Batch uses Object/XML Mapping (OXM) to + bind fragments to objects. However, Spring Batch is not tied to any + particular XML binding technology. Typical use is to delegate to Spring + OXM, which provides uniform abstraction for the most + popular OXM technologies. The dependency on Spring OXM is optional and you + can choose to implement Spring Batch specific interfaces if desired. The + relationship to the technologies that OXM supports can be shown as the + following: + + + + + + + + + + + Figure 3.2: OXM Binding + + + Now with an introduction to OXM and how one can use XML fragments to + represent records, let's take a closer look at readers and writers. + +
+ StaxEventItemReader + + The StaxEventItemReader configuration + provides a typical setup for the processing of records from an XML input + stream. First, lets examine a set of XML records that the + StaxEventItemReader can process. + + +<?xml version="1.0" encoding="UTF-8"?> +<records> + <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain"> + <isin>XYZ0001</isin> + <quantity>5</quantity> + <price>11.39</price> + <customer>Customer1</customer> + </trade> + <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain"> + <isin>XYZ0002</isin> + <quantity>2</quantity> + <price>72.99</price> + <customer>Customer2c</customer> + </trade> + <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain"> + <isin>XYZ0003</isin> + <quantity>9</quantity> + <price>99.99</price> + <customer>Customer3</customer> + </trade> +</records> + + + + To be able to process the XML records the following is needed: + + + Root Element Name - Name of the root element of the fragment + that constitutes the object to be mapped. The example + configuration demonstrates this with the value of trade. + + + + Resource - Spring Resource that represents the file to be + read. + + + + FragmentDeserializer - Unmarshalling + facility provided by Spring OXM for mapping the XML fragment to an + object. + + + + + <bean id="itemReader" class="org.springframework.batch.item.xml.StaxEventItemReader"> + <property name="fragmentRootElementName" value="customer" /> + <property name="resource" value="data/iosample/input/input.xml" /> + <property name="unmarshaller" ref="customerCreditMarshaller" /> + </bean> + + <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller"> + <property name="aliases"> + <util:map id="aliases"> + <entry key="customer" + value="org.springframework.batch.sample.domain.trade.CustomerCredit" /> + <entry key="price" value="java.math.BigDecimal" /> + <entry key="name" value="java.lang.String" /> + </util:map> + </property> + </bean> + + + Notice that in this example we have chosen to use an + XStreamMarshaller that requires an alias passed + in as a map with the first key and value being the name of the fragment + (i.e. root element) and the object type to bind. Then, similar to a + FieldSet, the names of the other elements that + map to fields within the object type are described as key/value pairs in + the map. In the configuration file we can use a Spring configuration + utility to describe the required alias as follows: + + + <bean id="itemReader" class="org.springframework.batch.item.xml.StaxEventItemReader"> + <property name="fragmentRootElementName" value="customer" /> + <property name="resource" value="data/iosample/input/input.xml" /> + <property name="unmarshaller" ref="customerCreditMarshaller" /> + </bean> + + <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller"> + <property name="aliases"> + <util:map id="aliases"> + <entry key="customer" + value="org.springframework.batch.sample.domain.trade.CustomerCredit" /> + <entry key="price" value="java.math.BigDecimal" /> + <entry key="name" value="java.lang.String" /> + </util:map> + </property> + </bean> + + + On input the reader reads the XML resource until it recognizes + that a new fragment is about to start (by matching the tag name by + default). The reader creates a standalone XML document from the fragment + (or at least makes it appear so) and passes the document to a + deserializer (typically a wrapper around a Spring OXM + Unmarshaller) to map the XML to a Java + object. + + In summary, this procedure is analogous to the following scripted + Java code which uses the injection provided by the Spring + configuration: + + + StaxEventItemReader xmlStaxEventItemReader = new StaxEventItemReader() + Resource resource = new ByteArrayResource(xmlResource.getBytes()) + + Map aliases = new HashMap(); + aliases.put("customer","org.springframework.batch.sample.domain.trade.CustomerCredit"); + aliases.put("price","java.math.BigDecimal"); + aliases.put("name","java.lang.String"); + Marshaller marshaller = new XStreamMarshaller(); + marshaller.setAliases(aliases); + xmlStaxEventItemReader.setUnmarshaller(marshaller); + xmlStaxEventItemReader.setResource(resource); + xmlStaxEventItemReader.setFragmentRootElementName("customer"); + xmlStaxEventItemReader.open(new ExecutionContext()); + + boolean hasNext = true + + CustomerCredit credit = null; + + while (hasNext) { + credit = xmlStaxEventItemReader.read(); + if (credit == null) { + hasNext = false; + } else { + println trade; + } + } + + +
+ +
+ StaxEventItemWriter + + Output works symmetrically to input. The + StaxEventItemWriter needs a + Resource, a serializer, and a rootTagName. A Java + object is passed to a serializer (typically a wrapper around Spring OXM + Marshaller) which writes to a + Resource using a custom event writer that filters + the StartDocument and + EndDocument events produced for each fragment by + the OXM tools. We'll show this in an example using the + MarshallingEventWriterSerializer. The Spring + configuration for this setup looks as follows: + + + <bean id="itemWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter"> + <property name="resource" ref="outputResource" /> + <property name="marshaller" ref="customerCreditMarshaller" /> + <property name="rootTagName" value="customers" /> + <property name="overwriteOutput" value="true" /> + </bean> + + + + The configuration sets up the three required properties and + optionally sets the overwriteOutput=true, mentioned earlier in the + chapter for specifying whether an existing file can be overwritten. It + should be noted the marshaller used for the writer is the exact same as + the one used in the reading example from earlier in the chapter: + + + <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller"> + <property name="aliases"> + <util:map id="aliases"> + <entry key="customer" + value="org.springframework.batch.sample.domain.trade.CustomerCredit" /> + <entry key="price" value="java.math.BigDecimal" /> + <entry key="name" value="java.lang.String" /> + </util:map> + </property> + </bean> + + + To summarize with a Java example, the following code illustrates + all of the points discussed, demonstrating the programmatic setup of the + required properties: + + StaxEventItemWriter staxItemWriter = new StaxEventItemWriter() + FileSystemResource resource = new FileSystemResource(File.createTempFile("StaxEventWriterOutputSourceTests", "xml")) + + Map aliases = new HashMap(); + aliases.put("customer","org.springframework.batch.sample.domain.trade.CustomerCredit"); + aliases.put("price","java.math.BigDecimal"); + aliases.put("name","java.lang.String"); + Marshaller marshaller = new XStreamMarshaller(); + marshaller.setAliases(aliases); + + staxItemWriter.setResource(resource); + staxItemWriter.setMarshaller(marshaller); + staxItemWriter.setRootTagName("trades"); + staxItemWriter.setOverwriteOutput(true); + + ExecutionContext executionContext = new ExecutionContext(); + staxItemWriter.open(executionContext); + CustomerCredit Credit = new CustomerCredit(); + trade.setPrice(11.39); + credit.setName("Customer1"); + staxItemWriter.write(trade); + +
+
+ +
+ Multi-File Input + + It is a common requirement to process multiple files within a single + Step. Assuming the files all have the same + formatting, the MultiResourceItemReader supports + this type of input for both XML and flat file processing. Consider the + following files in a directory: + + file-1.txt file-2.txt ignored.txt + + file-1.txt and file-2.txt are formatted the same and for business + reasons should be processed together. The + MuliResourceItemReader can be used to read in both + files by using wildcards: + + + <bean id="multiResourceReader" class="org.springframework.batch.item.file.MultiResourceItemReader"> + <property name="resources" value="classpath:data/multiResourceJob/input/file-*.txt" /> + <property name="delegate" ref="flatFileItemReader" /> + </bean> + + + + The referenced delegate is a simple + FlatFileItemReader. The above configuration will + read input from both files, handling rollback and restart scenarios. It + should be noted that, as with any ItemReader, + adding extra input (in this case a file) could cause potential issues when + restarting. It is recommended that batch jobs work with their own + individual directories until completed successfully. +
+ +
+ Database + + Like most enterprise application styles, a database is the central + storage mechanism for batch. However, batch differs from other application + styles due to the sheer size of the datasets with which the system must + work. The Spring Core JdbcTemplate illustrates this + problem well. If you use JdbcTemplate with a + RowMapper, the RowMapper + will be called once for every result returned from the provided query. + This causes few issues in scenarios where the dataset is small, but the + large datasets often necessary for batch processing would cause any JVM to + crash quickly. If the SQL statement returns 1 million rows, the + RowMapper will be called 1 million times, holding + all returned results in memory until all rows have been read. Spring Batch + provides two types of solutions for this problem: Cursor and Paging + database ItemReaders. + +
+ Cursor Based ItemReaders + + Using a database cursor is generally the default approach of most + batch developers, because it is the database's solution to the problem + of 'streaming' relational data. The Java + ResultSet class is essentially an object + orientated mechanism for manipulating a cursor. A + ResultSet maintains a cursor to the current row + of data. Calling next on a + ResultSet moves this cursor to the next row. + Spring Batch cursor based ItemReaders open the a cursor on + initialization, and move the cursor forward one row for every call to + read, returning a mapped object that can be + used for processing. The close method will then + be called to ensure all resources are freed up. The Spring core + JdbcTemplate gets around this problem by using + the callback pattern to completely map all rows in a + ResultSet and close before returning control back + to the method caller. However, in batch this must wait until the step is + complete. Below is a generic diagram of how a cursor based + ItemReader works, and while a SQL statement is + used as an example since it is so widely known, any technology could + implement the basic approach: + + + + + + + + + + + + This example illustrates the basic pattern. Given a 'FOO' table, + which has three columns: ID, NAME, and BAR, select all rows with an ID + greater than 1 but less than 7. This puts the beginning of the cursor + (row 1) on ID 2. The result of this row should be a completely mapped + Foo object. Calling read() again moves the + cursor to the next row, which is the Foo with an ID of 3. The results of + these reads will be written out after each + read, thus allowing the objects to be garbage + collected (assuming no instance variables are maintaining references to + them). + +
+ JdbcCursorItemReader + + JdbcCursorItemReader is the Jdbc + implementation of the cursor based technique. It works directly with a + ResultSet and requires a SQL statement to run + against a connection obtained from a + DataSource. The following database schema will + be used as an example: + + CREATE TABLE CUSTOMER ( + ID BIGINT IDENTITY PRIMARY KEY, + NAME VARCHAR(45), + CREDIT FLOAT + ); + + Many people prefer to use a domain object for each row, so we'll + use an implementation of the RowMapper + interface to map a CustomerCredit + object: + + public class CustomerCreditRowMapper implements RowMapper { + + public static final String ID_COLUMN = "id"; + public static final String NAME_COLUMN = "name"; + public static final String CREDIT_COLUMN = "credit"; + + public Object mapRow(ResultSet rs, int rowNum) throws SQLException { + CustomerCredit customerCredit = new CustomerCredit(); + + customerCredit.setId(rs.getInt(ID_COLUMN)); + customerCredit.setName(rs.getString(NAME_COLUMN)); + customerCredit.setCredit(rs.getBigDecimal(CREDIT_COLUMN)); + + return customerCredit; + } +} + + Because JdbcTemplate is so familiar to + users of Spring, and the JdbcCursorItemReader + shares key interfaces with it, it is useful to see an example of how + to read in this data with JdbcTemplate, in + order to contrast it with the ItemReader. For + the purposes of this example, let's assume there are 1,000 rows in the + CUSTOMER database. The first example will be using + JdbcTemplate: + + + //For simplicity sake, assume a dataSource has already been obtained + JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource); + List customerCredits = jdbcTemplate.query("SELECT ID, NAME, CREDIT from CUSTOMER", new CustomerCreditRowMapper()); + + + + After running this code snippet the customerCredits list will + contain 1,000 CustomerCredit objects. In the + query method, a connection will be obtained from the + DataSource, the provided SQL will be run + against it, and the mapRow method will be + called for each row in the ResultSet. Let's + contrast this with the approach of the + JdbcCursorItemReader: + + + JdbcCursorItemReader itemReader = new JdbcCursorItemReader(); + itemReader.setDataSource(dataSource); + itemReader.setSql("SELECT ID, NAME, CREDIT from CUSTOMER"); + itemReader.setMapper(new CustomerCreditRowMapper()); + int counter = 0; + ExecutionContext executionContext = new ExecutionContext(); + itemReader.open(executionContext); + Object customerCredit = new Object(); + while(customerCredit != null){ + customerCredit = itemReader.read(); + counter++; + } + itemReader.close(executionContext); + + + + After running this code snippet the counter will equal 1,000. If + the code above had put the returned customerCredit into a list, the + result would have been exactly the same as with the + JdbcTemplate example. However, the big + advantage of the ItemReader is that it allows + items to be 'streamed'. The read method can + be called once, and the item written out via an + ItemWriter, and then the next item obtained via + read. This allows item reading and writing to + be done in 'chunks' and committed periodically, which is the essence + of high performance batch processing. Furthermore, it is very easily + configured for injection into a Spring Batch + Step: + + + <bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader"> + <property name="dataSource" ref="dataSource"/> + <property name="sql" value="select ID, NAME, CREDIT from CUSTOMER"/> + <property name="mapper"> + <bean class="org.springframework.batch.sample.domain.trade.internal.CustomerCreditRowMapper"/> + </property> + </bean> + + + +
+ Additional Properties + + Because there are so many varying options for opening a cursor + in Java, there are many properties on the + JdbcCustorItemReader that can be set: + + + JdbcCursorItemReader Properties + + + + + ignoreWarnings + + Determines whether or not SQLWarnings are logged or + cause an exception - default is true + + + + fetchSize + + Gives the Jdbc driver a hint as to the number of rows + that should be fetched from the database when more rows are + needed by the ResultSet object used + by the ItemReader. By default, no + hint is given. + + + + maxRows + + Sets the limit for the maximum number of rows the + underlying ResultSet can hold at any + one time. + + + + queryTimeout + + Sets the number of seconds the driver will wait for a + Statement object to execute to the + given number of seconds. If the limit is exceeded, a + DataAccessEception is thrown. + (Consult your driver vendor documentation for + details). + + + + verifyCursorPosition + + Because the same ResultSet + held by the ItemReader is passed to + the RowMapper, it is possible for + users to call ResultSet.next() + themselves, which could cause issues with the reader's + internal count. Setting this value to true will cause an + exception to be thrown if the cursor position is not the + same after the RowMapper call as it + was before. + + + + saveState + + Indicates whether or not the reader's state should be + saved in the ExecutionContext + provided by + ItemStream#update(ExecutionContext) + The default value is false. + + + + driverSupportsAbsolute + + Defaults to false. Indicates whether the Jdbc driver + supports setting the absolute row on a + ResultSet. It is recommended that + this is set to true for Jdbc drivers that supports + ResultSet.absolute() as it may + improve performance, especially if a step fails while + working with a large data set. + + + + setUseSharedExtendedConnection + + Defaults to false. Indicates whether the connection + used for the cursor should be used by all other processing + thus sharing the same transaction. If this is set to false, + which is the default, then the cursor will be opened using + its own connection and will not participate in any + transactions started for the rest of the step processing. If + you set this flag to true then you must wrap the + DataSource in an + ExtendedConnectionDataSourceProxy to + prevent the connection from being closed and released after + each commit. When you set this option to true then the + statement used to open the cursor will be created with both + 'READ_ONLY' and 'HOLD_CUSORS_OVER_COMMIT' options. This + allows holding the cursor open over transaction start and + commits performed in the step processing. To use this + feature you need a database that supports this and a Jdbc + driver supporting Jdbc 3.0 or later. + + + +
+
+
+ +
+ HibernateCursorItemReader + + Just as normal Spring users make important decisions about + whether or not to use ORM solutions, which affect whether or not they + use a JdbcTemplate or a + HibernateTemplate, Spring Batch users have the + same options. HibernateCursorItemReader is the + Hibernate implementation of the cursor technique. Hibernate's usage in + batch has been fairly controversial. This has largely been because + Hibernate was originally developed to support online application + styles. However, that doesn't mean it can't be used for batch + processing. The easiest approach for solving this problem is to use a + StatelessSession rather than a standard + session. This removes all of the caching and dirty checking hibernate + employs that can cause issues in a batch scenario. For more + information on the differences between stateless and normal hibernate + sessions, refer to the documentation of your specific hibernate + release. The HibernateCursorItemReader allows + you to declare an HQL statement and pass in a + SessionFactory, which will pass back one item + per call to read in the same basic fashion as + the JdbcCursorItemReader. Below is an example + configuration using the same 'customer credit' example as the JDBC + reader: + + + HibernateCursorItemReader itemReader = new HibernateCursorItemReader(); + itemReader.setQueryString("from CustomerCredit"); + //For simplicity sake, assume sessionFactory already obtained. + itemReader.setSessionFactory(sessionFactory); + itemReader.setUseStatelessSession(true); + int counter = 0; + ExecutionContext executionContext = new ExecutionContext(); + itemReader.open(executionContext); + Object customerCredit = new Object(); + while(customerCredit != null){ + customerCredit = itemReader.read(); + counter++; + } + itemReader.close(executionContext); + + + + This configured ItemReader will return + CustomerCredit objects in the exact same manner + as described by the JdbcCursorItemReader, + assuming hibernate mapping files have been created correctly for the + Customer table. The 'useStatelessSession' property defaults to true, + but has been added here to draw attention to the ability to switch it + on or off. It is also worth noting that the fetchSize of the + underlying cursor can be set via the setFetchSize property. As with + JdbcCursorItemReader, configuration is + straightforward: + + + <bean id="itemReader" + class="org.springframework.batch.item.database.HibernateCursorItemReader"> + <property name="sessionFactory" ref="sessionFactory" /> + <property name="queryString" value="from CustomerCredit" /> + </bean> + + +
+
+ +
+ Paging ItemReaders + + An alternative to using a database cursor is executing multiple + queries where each query is bringing back a portion of the results. We + refer to this portion as a page. Each query that is executed must + specify the starting row number and the number of rows that we want + returned for the page. + +
+ JdbcPagingItemReader + + One implementation of a paging ItemReader + is the JdbcPagingItemReader. The + JdbcPagingItemReader needs a + PagingQueryProvider responsible for providing + the SQL queries used to retrieve the rows making up a page. Since each + database has its own strategy for providing paging support, we need to + use a different PagingQueryProvider for each + supported database type. There is also the + SimpleDelegatingPagingQueryProvider that will + auto-detect the database that is being used and determine the + appropriate PagingQueryProvider implementation. + This simplifies the configuration and is the recommended best + practice. + + The SimpleDelegatingPagingQueryProvider + requires that you specify a select clause and a from clause. You can + also provide an optional where clause. These clauses will be used to + build an SQL statement combined with the required sortKey. + + After the reader has been opened, it will pass back one item per + call to read in the same basic fashion as any + other ItemReader. The paging happens behind the + scenes when additional rows are needed. + + Below is an example configuration using a similar 'customer + credit' example as the cursor based ItemReaders above: + + <bean id="itemReader" + class="org.springframework.batch.item.database.JdbcPagingItemReader"> + <property name="dataSource" ref="dataSource"/> + <property name="queryProvider"> + <bean class="org.springframework.batch.item.database.support.SimpleDelegatingPagingQueryProvider"> + <property name="selectClause" value="select id, name, credit"/> + <property name="fromClause" value="from customer"/> + <property name="whereClause" value="where status=:status"/> + <property name="sortKey" value="id"/> + </bean> + </property> + <property name="parameterValues"> + <map> + <entry key="status" value="NEW"/> + </map> + </property> + <property name="pageSize" value="1000"/> + <property name="parameterizedRowMapper" ref="customerMapper"/> + </bean> + + + This configured ItemReader will return + CustomerCredit objects using the + ParameterizedRowMapper that must be specified. + The 'pageSize' property determines the number of entities read from + the database for each query execution. + + The 'parameterValues' property can be used to specify a Map of + parameter values for the query. If you use named parameters in the + where clause the key for each entry should match the name of the named + parameter. If you use a traditional '?' placeholder then the key for + each entry should be the number of the placeholder, starting with + 1. +
+ +
+ JpaPagingItemReader + + Another implementation of a paging + ItemReader is the + JpaPagingItemReader. JPA doesn't have a concept + similar to the Hibernate StatelessSession so we + have to use other features provided by the JPA specification. Since + JPA supports paging, this is a natural choice when it comes to using + JPA for batch processing. After each page is read, the entities will + become detached and the persistence context will be cleared in order + to allow the entities to be garbage collected once the page is + processed. + + The JpaPagingItemReader allows you to + declare a JPQL statement and pass in a + EntityManagerFactory. It will then pass back + one item per call to read in the same basic + fashion as any other ItemReader. The paging + happens behind the scenes when additional entities are needed. Below + is an example configuration using the same 'customer credit' example + as the JDBC reader above: + + <bean id="itemReader" + class="org.springframework.batch.item.database.JpaPagingItemReader"> + <property name="entityManagerFactory" ref="entityManagerFactory"/> + <property name="queryString" value="select c from CustomerCredit c"/> + <property name="pageSize" value="1000"/> + </bean> + + + This configured ItemReader will return + CustomerCredit objects in the exact same manner + as described by the JdbcPagingItemReader above, + assuming the Customer object has the correct JPA annotations or ORM + mapping file. The 'pageSize' property determines the number of + entities read from the database for each query execution. +
+ +
+ IbatisPagingItemReader + + If you use IBATIS for your data access then you can use the + IbatisPagingItemReader which, as the name + indicates, is an implementation of a paging + ItemReader. IBATIS doesn't have direct support + for reading rows in pages but by providing a couple of standard + variables you can add paging support to your IBATIS queries. + + Here is an example of a configuration for a + IbatisPagingItemReader reading CustomerCredits + as in the examples above: + + <bean id="itemReader" + class="org.springframework.batch.item.database.IbatisPagingItemReader"> + <property name="sqlMapClient" ref="sqlMapClient"/> + <property name="queryId" value="getPagedCustomerCredits"/> + <property name="pageSize" value="1000"/> + </bean> + + + The IbatisPagingItemReader configuration + above references an IBATIS query called "getPagedCustomerCredits". + Here is an example of what that query should look like for + MySQL. + + <select id="getPagedCustomerCredits" resultMap="customerCreditResult"> + select id, name, credit from customer order by id asc LIMIT #_skiprows#, #_pagesize# + </select> + + + The _skiprows and + _pagesize variables are provided by the + IbatisPagingItemReader and there is also a + _page variable that can be used if necessary. + The syntax for the paging queries varies with the database used. Here + is an example for Oracle (unfortunately we need to use CDATA for some + operators since this belongs in an XML document): + + <select id="getPagedCustomerCredits" resultMap="customerCreditResult"> + select * from ( + select * from ( + select t.id, t.name, t.credit, ROWNUM ROWNUM_ from customer t order by id + ) where ROWNUM_ <![CDATA[ > ]]> ( #_page# * #_pagesize# ) + ) where ROWNUM <![CDATA[ <= ]]> #_pagesize# + </select> + +
+
+ +
+ Database ItemWriters + + While both Flat Files and XML have specific ItemWriters, there is + no exact equivalent in the database world. This is because transactions + provide all the functionality that is needed. ItemWriters are necessary + for files because they must act as if they're transactional, keeping + track of written items and flushing or clearing at the appropriate + times. Databases have no need for this functionality, since the write is + already contained in a transaction. Users can create their own DAOs that + implement the ItemWriter interface or use one + from a custom ItemWriter that's written for + generic processing concerns, either way, they should work without any + issues. One thing to look out for is the performance and error handling + capabilities that are provided by batching the outputs. This is most + common when using hibernate as an ItemWriter, but + could have the same issues when using Jdbc batch mode. Batching database + output doesn't have any inherent flaws, assuming we are careful to flush + and there are no errors in the data. However, any errors while writing + out can cause confusion because there is no way to know which individual + item caused an exception, or even if any individual item was + responsible, as illustrated below: + + + + + + + + + + If items are buffered before being written out, any + errors encountered will not be thrown until the buffer is flushed just + before a commit. For example, let's assume that 20 items will be written + per chunk, and the 15th item throws a DataIntegrityViolationException. + As far as the Step is concerned, all 20 item will be written out + successfully, since there's no way to know that an error will occur + until they are actually written out. Once + Session#flush() is + called, the buffer will be emptied and the exception will be hit. At + this point, there's nothing the Step can do, the + transaction must be rolled back. Normally, this exception might cause + the Item to be skipped (depending upon the skip/retry policies), and + then it won't be written out again. However, in the batched scenario, + there's no way for it to know which item caused the issue, the whole + buffer was being written out when the failure happened. The only way to + solve this issue is to flush after each item: + + + + + + + + + + + + This is a common use case, especially when using Hibernate, and + the simple guideline for implementations of + ItemWriter, is to flush on each call to + write(). Doing so allows for items to be + skipped reliably, with Spring Batch taking care internally of the + granularity of the calls to ItemWriter after an + error. +
+
+ +
+ Reusing Existing Services + + Batch systems are often used in conjunction with other application + styles. The most common is an online system, but it may also support + integration or even a thick client application by moving necessary bulk + data that each application style uses. For this reason, it is common that + many users want to reuse existing DAOs or other services within their + batch jobs. The Spring container itself makes this fairly easy by allowing + any necessary class to be injected. However, there may be cases where the + existing service needs to act as an ItemReader or + ItemWriter, either to satisfy the dependency of + another Spring Batch class, or because it truly is the main + ItemReader for a step. It is fairly trivial to + write an adaptor class for each service that needs wrapping, but because + it is such a common concern, Spring Batch provides implementations: + ItemReaderAdapter and + ItemWriterAdapter. Both classes implement the + standard Spring method invoking the delegate pattern and are fairly simple + to set up. Below is an example of the reader: + + <bean id="itemReader" class="org.springframework.batch.item.adapter.ItemReaderAdapter"> + <property name="targetObject" ref="fooService" /> + <property name="targetMethod" value="generateFoo" /> + </bean> + + <bean id="fooService" class="org.springframework.batch.item.sample.FooService" /> + + One important point to note is that the contract of the targetMethod + must be the same as the contract for read: when + exhausted it will return null, otherwise an Object. + Anything else will prevent the framework from knowing when processing + should end, either causing an infinite loop or incorrect failure, + depending upon the implementation of the + ItemWriter. The ItemWriter + implementation is equally as simple: + + <bean id="itemWriter" class="org.springframework.batch.item.adapter.ItemWriterAdapter"> + <property name="targetObject" ref="fooService" /> + <property name="targetMethod" value="processFoo" /> + </bean> + + <bean id="fooService" class="org.springframework.batch.item.sample.FooService" /> + +
+ +
+ Validating Input + + During the course of this chapter, multiple approaches to parsing + input have been discussed. Each major implementation will throw an + exception if it is not 'well-formed'. The + FixedLengthTokenizer will throw an exception if a + range of data is missing. Similarly, attempting to access an index in a + RowMapper of FieldSetMapper + that doesn't exist or is in a different format than the one expected will + cause an exception to be thrown. All of these types of exceptions will be + thrown before read returns. However, they don't + address the issue of whether or not the returned item is valid. For + example, if one of the fields is an age, it obviously cannot be negative. + It will parse correctly, because it existed and is a number, but it won't + cause an exception. Since there are already a plethora of Validation + frameworks, Spring Batch does not attempt to provide yet another, but + rather provides a very simple interface that can be implemented by any + number of frameworks: + + + public interface Validator { + + void validate(Object value) throws ValidationException; + + } + + + + The contract is that the validate method + will throw an exception if the object is invalid, and return normally if + it is valid. Spring Batch provides an out of the box + ItemProcessor: + + <bean class="org.springframework.batch.item.validator.ValidatingItemProcessor"> + <property name="validator" ref="validator" /> + </bean> + + <bean id="validator" + class="org.springframework.batch.item.validator.SpringValidator"> + <property name="validator"> + <bean id="orderValidator" + class="org.springmodules.validation.valang.ValangValidator"> + <property name="valang"> + <value> + <![CDATA[ + { orderId : ? > 0 AND ? <= 9999999999 : 'Incorrect order ID' : 'error.order.id' } + { totalLines : ? = size(lineItems) : 'Bad count of order lines' + : 'error.order.lines.badcount'} + { customer.registered : customer.businessCustomer = FALSE OR ? = TRUE + : 'Business customer must be registered' + : 'error.customer.registration'} + { customer.companyName : customer.businessCustomer = FALSE OR ? HAS TEXT + : 'Company name for business customer is mandatory' + :'error.customer.companyname'} + ]]> + </value> + </property> + </bean> + </property> + </bean> + + + This simple example shows a simple + ValangValidator that is used to validate an order + object. The intent is not to show Valang functionality as much as to show + how a validator could be added. +
+ +
+ Preventing state persistence + + By default, all of the ItemReader and + ItemWriter implementations store their current + state in the ExecutionContext before it is + committed. However, this may not always be the desired behavior. For + example, many developers choose to make their database readers + 'rerunnable' by using a process indicator. An extra column is added to the + input data to indicate whether or not it has been processed. When a + particular record is being read (or written out) the processed flag is + flipped from false to true. The SQL statement can then contain an extra + statement in the where clause, such as "where PROCESSED_IND = false", + thereby ensuring that only unprocessed records will be returned in the + case of a restart. In this scenario, it is preferable to not store any + state, such as the current row number, since it will be irrelevant upon + restart. For this reason, all readers and writers include the 'saveState' + property: + + + <bean id="playerSummarizationSource" + class="org.springframework.batch.item.database.JdbcCursorItemReader"> + <property name="dataSource" ref="dataSource" /> + <property name="mapper"> + <bean class="org.springframework.batch.sample.mapping.PlayerSummaryMapper" /> + </property> + <property name="saveState" value="false" /> + <property name="sql"> + <value> + SELECT games.player_id, games.year_no, SUM(COMPLETES), + SUM(ATTEMPTS), SUM(PASSING_YARDS), SUM(PASSING_TD), + SUM(INTERCEPTIONS), SUM(RUSHES), SUM(RUSH_YARDS), + SUM(RECEPTIONS), SUM(RECEPTIONS_YARDS), SUM(TOTAL_TD) + from games, players where players.player_id = + games.player_id group by games.player_id, games.year_no + </value> + </property> + </bean> + + + + The ItemReader configured above will not make + any entries in the ExecutionContext for any + executions in which it participates. +
+ +
+ Creating Custom ItemReaders and + ItemWriters + + So far in this chapter the basic contracts that exist for reading + and writing in Spring Batch and some common implementations have been + discussed. However, these are all fairly generic, and there are many + potential scenarios that may not be covered by out of the box + implementations. This section will show, using a simple example, how to + create a custom ItemReader and + ItemWriter implementation and implement their + contracts correctly. The ItemReader will also + implement ItemStream, in order to illustrate how to + make a reader or writer restartable. + +
+ Custom ItemReader Example + + For the purpose of this example, a simple + ItemReader implementation that reads from a + provided list will be created. We'll start out by implementing the most + basic contract of ItemReader, + read: + + + public class CustomItemReader<T> implements ItemReader<T>{ + + List<T> items; + + public CustomItemReader(List<T> items) { + this.items = items; + } + + public T read() throws Exception, UnexpectedInputException, + NoWorkFoundException, ParseException { + + if (!items.isEmpty()) { + return items.remove(0); + } + return null; + } + } + + + + This very simple class takes a list of items, and returns them one + at a time, removing each from the list. When the list is empty, it + returns null, thus satisfying the most basic requirements of an + ItemReader, as illustrated below: + + List<String> items = new ArrayList<String>(); + items.add("1"); + items.add("2"); + items.add("3"); + + ItemReader itemReader = new CustomItemReader<String>(items); + assertEquals("1", itemReader.read()); + assertEquals("2", itemReader.read()); + assertEquals("3", itemReader.read()); + assertNull(itemReader.read()); + +
+ Making the <classname>ItemReader</classname> + restartable + + The final challenge now is to make the + ItemReader restartable. Currently, if the power + goes out, and processing begins again, the + ItemReader must start at the beginning. This is + actually valid in many scenarios, but it is sometimes preferable that + a batch job starts where it left off. The key discriminant is often + whether the reader is stateful or stateless. A stateless reader does + not need to worry about restartability, but a stateful one has to try + and reconstitute its last known state on restart. For this reason, we + recommend that you keep custom readers stateless if possible, so you + don't have to worry about restartability. + + If you do need to store state, then the + ItemStream interface should be used: + + public class CustomItemReader<T> implements ItemReader<T>, ItemStream { + + List<T> items; + int currentIndex = 0; + private static final String CURRENT_INDEX = "current.index"; + + public CustomItemReader(List<T> items) { + this.items = items; + } + + public T read() throws Exception, UnexpectedInputException, + ParseException { + + if (currentIndex < items.size()) { + return items.get(currentIndex++); + } + + return null; + } + + public void open(ExecutionContext executionContext) throws ItemStreamException { + if(executionContext.containsKey(CURRENT_INDEX)){ + currentIndex = new Long(executionContext.getLong(CURRENT_INDEX)).intValue(); + } + else{ + currentIndex = 0; + } + } + + public void update(ExecutionContext executionContext) throws ItemStreamException { + executionContext.putLong(CURRENT_INDEX, new Long(currentIndex).longValue()); + }; + + public void close() throws ItemStreamException {} + } + + On each call to the ItemStream + update method, the current index of the + ItemReader will be stored in the provided + ExecutionContext with a key of 'current.index'. + When the ItemStream open + method is called, the ExecutionContext is + checked to see if it contains an entry with that key. If the key is + found, then the current index is moved to that location. This is a + fairly trivial example, but it still meets the general + contract: + + ExecutionContext executionContext = new ExecutionContext(); + ((ItemStream)itemReader).open(executionContext); + assertEquals("1", itemReader.read()); + ((ItemStream)itemReader).update(executionContext); + + List<String> items = new ArrayList<String>(); + items.add("1"); + items.add("2"); + items.add("3"); + itemReader = new CustomItemReader<String>(items); + + ((ItemStream)itemReader).open(executionContext); + assertEquals("2", itemReader.read()); + + Most ItemReaders have much more sophisticated restart logic. The + JdbcCursorItemReader, for example, stores the + row id of the last processed row in the Cursor. + + It is also worth noting that the key used within the + ExecutionContext should not be trivial. That is + because the same ExecutionContext is used for + all ItemStreams within a + Step. In most cases, simply prepending the key + with the class name should be enough to guarantee uniqueness. However, + in the rare cases where two of the same type of + ItemStream are used in the same step (which can + happen if two files are need for output) then a more unique name will + be needed. For this reason, many of the Spring Batch + ItemReader and + ItemWriter implementations have a + setName() property that allows this key name + to be overridden. +
+
+ +
+ Custom ItemWriter Example + + Implementing a Custom ItemWriter is similar + in many ways to the ItemReader example above, but + differs in enough ways as to warrant its own example. However, adding + restartability is essentially the same, so it won't be covered in this + example. As with the ItemReader example, a + List will be used in order to keep the example as + simple as possible: + + public class CustomItemWriter<T> implements ItemWriter<T> { + + List<T> output = TransactionAwareProxyFactory.createTransactionalList(); + + public void write(List<? extends T> items) throws Exception { + output.addAll(items); + } + + public List<T> getOutput() { + return output; + } + } + +
+ Making the <classname>ItemWriter</classname> + restartable + + To make the ItemWriter restartable we would follow the same + process as for the ItemReader, adding and + implementing the ItemStream interface to + synchronize the execution context. In the example we might have to + count the number of items processed and add that as a footer record. + If we needed to do that, we could implement + ItemStream in our + ItemWriter so that the counter was + reconstituted from the execution context if the stream was + re-opened. + + In many realistic cases, custom ItemWriters also delegate to + another writer that itself is restartable (e.g. when writing to a + file), or else it writes to a transactional resource so doesn't need + to be restartable because it is stateless. When you have a stateful + writer you should probably also be sure to implement + ItemStream as well as + ItemWriter. Remember also that the client of + the writer needs to be aware of the ItemStream, + so you may need to register it as a stream in the configuration + xml. +
+
+
+ diff --git a/docs/src/site/docbook/reference/schema-appendix.xml b/docs/src/site/docbook/reference/schema-appendix.xml index 49e06c15e..aa7957e22 100644 --- a/docs/src/site/docbook/reference/schema-appendix.xml +++ b/docs/src/site/docbook/reference/schema-appendix.xml @@ -1,612 +1,613 @@ - - - - Meta-Data Schema - -
- Overview - - The Spring Batch Meta-Data tables very closely match the Domain - objects that represent them in Java. For example, - JobInstance, JobExecution, - JobParameters, and - StepExecution map to BATCH_JOB_INSTANCE, - BATCH_JOB_EXECUTION, BATCH_JOB_PARAMS, and BATCH_STEP_EXECUTION, - respectively. ExecutionContext maps to both - BATCH_JOB_EXECUTION_CONTEXT and BATCH_STEP_EXECUTION_CONTEXT. The - JobRepository is responsible for saving and storing - each Java object into it's correct table. The following appendix describes - the meta-data tables in detail, along with many of the design decisions - that were made when creating them. When viewing the various table creation - statements below, it is important to realize that the data types used are - as generic as possible. Spring Batch provides many schemas as examples, - which all have varying data types due to variations in individual database - vendors' handling of data types. Below is an ERD model of all 6 tables and - their relationships to one another: - - - - - - - - - - - -
- Version - - Many of the database tables discussed in this appendix contain a - version column. This column is important because Spring Batch employs an - optimistic locking strategy when dealing with updates to the database. - This means that each time a record is 'touched' (updated) the value in - the version column is incremented by one. When the repository goes back - to try and save the value, if the version number has change it will - throw OptimisticLockingFailureException, - indicating there has been an error with concurrent access. This check is - necessary, since even though different batch jobs may be running in - different machines, they are all using the same database tables. -
- -
- Identity - - BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, and BATCH_STEP_EXECUTION - each contain columns ending in _ID, which act as primary keys for their - respective tables. However, they are not database generated keys, but - rather are generated by separate sequences. This is necessary because - after inserting one of the domain objects into the database, the key it - is given needs to be set on the actual object, so that they can be - uniquely identified in Java. Newer database drivers (Jdbc 3.0 and up) - support this feature with database generated keys, but rather than - requiring it, sequences were used. Each variation of the schema will - contain some form of the following: - - CREATE SEQUENCE BATCH_STEP_EXECUTION_SEQ; -CREATE SEQUENCE BATCH_JOB_EXECUTION_SEQ; -CREATE SEQUENCE BATCH_JOB_SEQ; - - Many database vendors don't support sequences. In these cases, - work arounds are used, such as the following for mySQL: - - CREATE TABLE BATCH_STEP_EXECUTION_SEQ (ID BIGINT NOT NULL) type=MYISAM; -INSERT INTO BATCH_STEP_EXECUTION_SEQ values(0); -CREATE TABLE BATCH_JOB_EXECUTION_SEQ (ID BIGINT NOT NULL) type=MYISAM; -INSERT INTO BATCH_JOB_EXECUTION_SEQ values(0); -CREATE TABLE BATCH_JOB_SEQ (ID BIGINT NOT NULL) type=MYISAM; -INSERT INTO BATCH_JOB_SEQ values(0); - - In the above case, a table is used in place of each sequence. The - Spring core class MySQLMaxValueIncrementer will - then increment the one column in this sequence in order to give similar - functionality. -
-
- -
- BATCH_JOB_INSTANCE - - The BATCH_JOB_INSTANCE table holds all information relevant to a - JobInstance, and serves as the top of the overall - hierarchy. The following generic DDL statement is used to create - it: - - CREATE TABLE BATCH_JOB_INSTANCE ( - JOB_INSTANCE_ID BIGINT PRIMARY KEY , - VERSION BIGINT, - JOB_NAME VARCHAR(100) NOT NULL , - JOB_KEY VARCHAR(2500) -); - - Below are descriptions of each column in the table: - - - - JOB_INSTANCE_ID: The unique id that will identify the instance, - which is also the primary key. The value of this column should be - obtainable by calling the getId method on - JobInstance. - - - - VERSION: See above section. - - - - JOB_NAME: Name of the job obtained from the - Job object. Because it is required to identify - the instance, it must not be null. - - - - JOB_KEY: A serialization of the - JobParameters that uniquely identifies separate - instances of the same job from one another. - (JobInstances with the same job name must have - different JobParameters, and thus, different - JOB_KEY values). - - -
- -
- BATCH_JOB_PARAMS - - The BATCH_JOB_PARAMS table holds all information relevant to the - JobParameters object. It contains 0 or more - key/value pairs that together uniquely identify a - JobInstance and serve as a record of the parameters - a job was run with. It should be noted that the table has been - denormalized. Rather than creating a separate table for each type, there - is one table with a column indicating the type: - - CREATE TABLE BATCH_JOB_PARAMS ( - JOB_INSTANCE_ID BIGINT NOT NULL , - TYPE_CD VARCHAR(6) NOT NULL , - KEY_NAME VARCHAR(100) NOT NULL , - STRING_VAL VARCHAR(250) , - DATE_VAL TIMESTAMP DEFAULT NULL, - LONG_VAL BIGINT , - DOUBLE_VAL DOUBLE PRECISION, - constraint JOB_INSTANCE_PARAMS_FK foreign key (JOB_INSTANCE_ID) - references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID) -); - - Below are descriptions for each column: - - - - JOB_INSTANCE_ID: Foreign Key from the BATCH_JOB_INSTANCE table - that indicates the job instance the parameter entry belongs to. It - should be noted that multiple rows (i.e key/value pairs) may exist for - each instance. - - - - TYPE_CD: String representation of the type of value stored, - which can be either a string, date, long, or double. Because the type - must be known, it cannot be null. - - - - KEY_NAME: The parameter key. - - - - STRING_VAL: Parameter value, if the type is string. - - - - DATE_VAL: Parameter value, if the type is date. - - - - LONG_VAL: Parameter value, if the type is a long. - - - - DOUBLE_VAL: Parameter value, if the type is double. - - - - It is worth noting that there is no primary key for this table. This - is simply because the framework has no use for one, and thus doesn't - require it. If a user so chooses, one may be added with a database - generated key, without causing any issues to the framework itself. -
- -
- BATCH_JOB_EXECUTION - - The BATCH_JOB_EXECUTION table holds all information relevant to the - JobExecution object. Every time a - Job is run there will always be a new - JobExecution, and a new row in this table: - - CREATE TABLE BATCH_JOB_EXECUTION ( - JOB_EXECUTION_ID BIGINT PRIMARY KEY , - VERSION BIGINT, - JOB_INSTANCE_ID BIGINT NOT NULL, - CREATE_TIME TIMESTAMP NOT NULL, - START_TIME TIMESTAMP DEFAULT NULL, - END_TIME TIMESTAMP DEFAULT NULL, - STATUS VARCHAR(10), - EXIT_CODE VARCHAR(20), - EXIT_MESSAGE VARCHAR(2500), - LAST_UPDATED TIMESTAMP, - constraint JOB_INSTANCE_EXECUTION_FK foreign key (JOB_INSTANCE_ID) - references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID) -) ; - - Below are descriptions for each column: - - - - JOB_EXECUTION_ID: Primary key that uniquely identifies this - execution. The value of this column is obtainable by calling the - getId method of the - JobExecution object. - - - - VERSION: See above section. - - - - JOB_INSTANCE_ID: Foreign key from the BATCH_JOB_INSTANCE table - indicating the instance to which this execution belongs. There may be - more than one execution per instance. - - - - CREATE_TIME: Timestamp representing the time that the execution - was created. - - - - START_TIME: Timestamp representing the time the execution was - started. - - - - END_TIME: Timestamp representing the time the execution was - finished, regardless of success or failure. An empty value in this - column even though the job is not currently running indicates that - there has been some type of error and the framework was unable to - perform a last save before failing. - - - - STATUS: Character string representing the status of the - execution. This may be COMPLETED, STARTED, etc. The object - representation of this column is the - BatchStatus enumeration. - - - - EXIT_CODE: Character string representing the exit code of the - execution. In the case of a command line job, this may be converted - into a number. - - - - EXIT_MESSAGE: Character string representing a more detailed - description of how the job exited. In the case of failure, this might - include as much of the stack trace as is possible. - - - - LAST_UPDATED: Timestamp representing the last time this - execution was persisted. - - -
- -
- BATCH_STEP_EXECUTION - - The BATCH_STEP_EXECUTION table holds all information relevant to the - StepExecution object. This table is very similar in - many ways to the BATCH_JOB_EXECUTION table and there will always be at - least one entry per Step for each - JobExecution created: - - CREATE TABLE BATCH_STEP_EXECUTION ( - STEP_EXECUTION_ID BIGINT PRIMARY KEY , - VERSION BIGINT NOT NULL, - STEP_NAME VARCHAR(100) NOT NULL, - JOB_EXECUTION_ID BIGINT NOT NULL, - START_TIME TIMESTAMP NOT NULL , - END_TIME TIMESTAMP DEFAULT NULL, - STATUS VARCHAR(10), - COMMIT_COUNT BIGINT , - READ_COUNT BIGINT , - FILTER_COUNT BIGINT , - WRITE_COUNT BIGINT , - READ_SKIP_COUNT BIGINT , - WRITE_SKIP_COUNT BIGINT , - PROCESS_SKIP_COUNT BIGINT , - ROLLBACK_COUNT BIGINT , - EXIT_CODE VARCHAR(20) , - EXIT_MESSAGE VARCHAR(2500) , - LAST_UPDATED TIMESTAMP, - constraint JOB_EXECUTION_STEP_FK foreign key (JOB_EXECUTION_ID) - references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID) -) ; - - Below are descriptions for each column: - - - - STEP_EXECUTION_ID: Primary key that uniquely identifies this - execution. The value of this column should be obtainable by calling - the getId method of the - StepExecution object. - - - - VERSION: See above section. - - - - STEP_NAME: The name of the step to which this execution - belongs. - - - - JOB_EXECUTION_ID: Foreign key from the BATCH_JOB_EXECUTION table - indicating the JobExecution to which this StepExecution belongs. There - may be only one StepExecution for a given - JobExecution for a given - Step name. - - - - START_TIME: Timestamp representing the time the execution was - started. - - - - END_TIME: Timestamp representing the time the execution was - finished, regardless of success or failure. An empty value in this - column even though the job is not currently running indicates that - there has been some type of error and the framework was unable to - perform a last save before failing. - - - - STATUS: Character string representing the status of the - execution. This may be COMPLETED, STARTED, etc. The object - representation of this column is the - BatchStatus enumeration. - - - - COMMIT_COUNT: The number of times in which the step has - committed a transaction during this execution. - - - - READ_COUNT: The number of items read during this - execution. - - - - FILTER_COUNT: The number of items filtered out of this - execution. - - - - WRITE_COUNT: The number of items written during this - execution. - - - - READ_SKIP_COUNT: The number of items skipped on read during this - execution. - - - - WRITE_SKIP_COUNT: The number of items skipped on write during - this execution. - - - - PROCESS_SKIP_COUNT: The number of items skipped during - processing during this execution. - - - - ROLLBACK_COUNT: The number of rollbacks during this - execution. - - - - EXIT_CODE: Character string representing the exit code of the - execution. In the case of a command line job, this may be converted - into a number. - - - - EXIT_MESSAGE: Character string representing a more detailed - description of how the job exited. In the case of failure, this might - include as much of the stack trace as is possible. - - - - LAST_UPDATED: Timestamp representing the last time this - execution was persisted. - - -
- -
- BATCH_JOB_EXECUTION_CONTEXT - - The BATCH_JOB_EXECUTION_CONTEXT table holds all information relevant - to an Job's - ExecutionContext. There is exactly one - ExecutionContext per - StepExecution, and it contains all of the job-level - data that is needed for a particular job execution. This data typically - represents the state that must be retrieved after a failure so that a - JobInstance can 'start from where it left - off'. - - CREATE TABLE BATCH_JOB_EXECUTION_CONTEXT ( - JOB_EXECUTION_ID BIGINT PRIMARY KEY, - SHORT_CONTEXT VARCHAR(2500) NOT NULL, - SERIALIZED_CONTEXT CLOB, - constraint JOB_EXEC_CTX_FK foreign key (JOB_EXECUTION_ID) - references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID) -) ; - - Below are descriptions for each column: - - - - JOB_EXECUTION_ID: Foreign key representing the - JobExecution to which the context belongs. - There may be more than one row associated to a given execution. - - - - SHORT_CONTEXT: A string version of the - SERIALIZED_CONTEXT. - - - - SERIALIZED_CONTEXT: The entire context, serialized. - - -
- -
- BATCH_STEP_EXECUTION_CONTEXT - - The BATCH_STEP_EXECUTION_CONTEXT table holds all information - relevant to an Step's - ExecutionContext. There is exactly one - ExecutionContext per - StepExecution, and it contains all of the data that - needs to persisted for a particular step execution. This data typically - represents the state that must be retrieved after a failure so that a - JobInstance can 'start from where it left - off'. - - CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT ( - STEP_EXECUTION_ID BIGINT PRIMARY KEY, - SHORT_CONTEXT VARCHAR(2500) NOT NULL, - SERIALIZED_CONTEXT CLOB, - constraint STEP_EXEC_CTX_FK foreign key (STEP_EXECUTION_ID) - references BATCH_STEP_EXECUTION(STEP_EXECUTION_ID) -) ; - - Below are descriptions for each column: - - - - STEP_EXECUTION_ID: Foreign key representing the - StepExecution to which the context belongs. - There may be more than one row associated to a given execution. - - - - SHORT_CONTEXT: A string version of the - SERIALIZED_CONTEXT. - - - - SERIALIZED_CONTEXT: The entire context, serialized. - - -
- -
- Archiving - - Because there are entries in multiple tables every time a batch job - is run, it is common to create an archive strategy for the meta-data - tables. The tables themselves are designed to show a record of what - happened in the past, and generally won't affect the run of any job, with - a couple of notable exceptions pertaining to restart: - - - - The framework will use the meta-data tables to determine if a - particular JobInstance has been run before. If it has been run, and - the job is not restartable, then an exception will be thrown. - - - - If an entry for a JobInstance is removed without having - completed successfully, the framework will think that the job is new, - rather than a restart. - - - - If a job is restarted, the framework will use any data that has - been persisted to the ExecutionContext to restore the Job's state. - Therefore, removing any entries from this table for jobs that haven't - completed successfully will prevent them from starting at the correct - point if run again. - - -
- -
- Recommendations for Indexing Meta Data Tables - - Spring Batch provides DDL samples for the meta-data tables in the - Core jar file for several common database platforms. Index declarations - are not included in that DDL because there are too many variations in how - users may want to index depending on their precise platform, local - conventions and also the business requirements of how the jobs will be - operated. The table below provides some indication as to which columns are - going to be used in a WHERE clause by the Dao implementations provided by - Spring Batch, and how frequently they might be used, so that individual - projects can make up their own minds about indexing. - - - Where clauses in SQL statements (exluding primary keys) and their - approximate frequency of use. - - - - - Default Table Name - - Where Clause - - Frequency - - - - BATCH_JOB_INSTANCE - - JOB_NAME = ? and JOB_KEY = ? - - Every time a job is launched - - - - BATCH_JOB_EXECUTION - - JOB_INSTANCE_ID = ? - - Every time a job is restarted - - - - BATCH_EXECUTION_CONTEXT - - EXECUTION_ID = ? and KEY_NAME = ? - - On commit interval, a.k.a. chunk - - - - BATCH_STEP_EXECUTION - - VERSION = ? - - On commit interval, a.k.a. chunk (and at start and end of - step) - - - - BATCH_STEP_EXECUTION - - STEP_NAME = ? and JOB_EXECUTION_ID = ? - - Before each step execution - - - -
-
- + + + + Meta-Data Schema + +
+ Overview + + The Spring Batch Meta-Data tables very closely match the Domain + objects that represent them in Java. For example, + JobInstance, JobExecution, + JobParameters, and + StepExecution map to BATCH_JOB_INSTANCE, + BATCH_JOB_EXECUTION, BATCH_JOB_PARAMS, and BATCH_STEP_EXECUTION, + respectively. ExecutionContext maps to both + BATCH_JOB_EXECUTION_CONTEXT and BATCH_STEP_EXECUTION_CONTEXT. The + JobRepository is responsible for saving and storing + each Java object into its correct table. The following appendix describes + the meta-data tables in detail, along with many of the design decisions + that were made when creating them. When viewing the various table creation + statements below, it is important to realize that the data types used are + as generic as possible. Spring Batch provides many schemas as examples, + which all have varying data types due to variations in individual database + vendors' handling of data types. Below is an ERD model of all 6 tables and + their relationships to one another: + + + + + + + + + + + +
+ Version + + Many of the database tables discussed in this appendix contain a + version column. This column is important because Spring Batch employs an + optimistic locking strategy when dealing with updates to the database. + This means that each time a record is 'touched' (updated) the value in + the version column is incremented by one. When the repository goes back + to try and save the value, if the version number has change it will + throw OptimisticLockingFailureException, + indicating there has been an error with concurrent access. This check is + necessary since, even though different batch jobs may be running in + different machines, they are all using the same database tables. +
+ +
+ Identity + + BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, and BATCH_STEP_EXECUTION + each contain columns ending in _ID. These fields act as primary keys for + their respective tables. However, they are not database generated keys, + but rather they are generated by separate sequences. This is necessary + because after inserting one of the domain objects into the database, the + key it is given needs to be set on the actual object so that they can be + uniquely identified in Java. Newer database drivers (Jdbc 3.0 and up) + support this feature with database generated keys, but rather than + requiring it, sequences were used. Each variation of the schema will + contain some form of the following: + + CREATE SEQUENCE BATCH_STEP_EXECUTION_SEQ; +CREATE SEQUENCE BATCH_JOB_EXECUTION_SEQ; +CREATE SEQUENCE BATCH_JOB_SEQ; + + Many database vendors don't support sequences. In these cases, + work arounds are used, such as the following for mySQL: + + CREATE TABLE BATCH_STEP_EXECUTION_SEQ (ID BIGINT NOT NULL) type=MYISAM; +INSERT INTO BATCH_STEP_EXECUTION_SEQ values(0); +CREATE TABLE BATCH_JOB_EXECUTION_SEQ (ID BIGINT NOT NULL) type=MYISAM; +INSERT INTO BATCH_JOB_EXECUTION_SEQ values(0); +CREATE TABLE BATCH_JOB_SEQ (ID BIGINT NOT NULL) type=MYISAM; +INSERT INTO BATCH_JOB_SEQ values(0); + + In the above case, a table is used in place of each sequence. The + Spring core class MySQLMaxValueIncrementer will + then increment the one column in this sequence in order to give similar + functionality. +
+
+ +
+ BATCH_JOB_INSTANCE + + The BATCH_JOB_INSTANCE table holds all information relevant to a + JobInstance, and serves as the top of the overall + hierarchy. The following generic DDL statement is used to create + it: + + CREATE TABLE BATCH_JOB_INSTANCE ( + JOB_INSTANCE_ID BIGINT PRIMARY KEY , + VERSION BIGINT, + JOB_NAME VARCHAR(100) NOT NULL , + JOB_KEY VARCHAR(2500) +); + + Below are descriptions of each column in the table: + + + + JOB_INSTANCE_ID: The unique id that will identify the instance, + which is also the primary key. The value of this column should be + obtainable by calling the getId method on + JobInstance. + + + + VERSION: See above section. + + + + JOB_NAME: Name of the job obtained from the + Job object. Because it is required to identify + the instance, it must not be null. + + + + JOB_KEY: A serialization of the + JobParameters that uniquely identifies separate + instances of the same job from one another. + (JobInstances with the same job name must have + different JobParameters, and thus, different + JOB_KEY values). + + +
+ +
+ BATCH_JOB_PARAMS + + The BATCH_JOB_PARAMS table holds all information relevant to the + JobParameters object. It contains 0 or more + key/value pairs that together uniquely identify a + JobInstance and serve as a record of the parameters + a job was run with. It should be noted that the table has been + denormalized. Rather than creating a separate table for each type, there + is one table with a column indicating the type: + + CREATE TABLE BATCH_JOB_PARAMS ( + JOB_INSTANCE_ID BIGINT NOT NULL , + TYPE_CD VARCHAR(6) NOT NULL , + KEY_NAME VARCHAR(100) NOT NULL , + STRING_VAL VARCHAR(250) , + DATE_VAL TIMESTAMP DEFAULT NULL, + LONG_VAL BIGINT , + DOUBLE_VAL DOUBLE PRECISION, + constraint JOB_INSTANCE_PARAMS_FK foreign key (JOB_INSTANCE_ID) + references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID) +); + + Below are descriptions for each column: + + + + JOB_INSTANCE_ID: Foreign Key from the BATCH_JOB_INSTANCE table + that indicates the job instance the parameter entry belongs to. It + should be noted that multiple rows (i.e key/value pairs) may exist for + each instance. + + + + TYPE_CD: String representation of the type of value stored, + which can be either a string, date, long, or double. Because the type + must be known, it cannot be null. + + + + KEY_NAME: The parameter key. + + + + STRING_VAL: Parameter value, if the type is string. + + + + DATE_VAL: Parameter value, if the type is date. + + + + LONG_VAL: Parameter value, if the type is a long. + + + + DOUBLE_VAL: Parameter value, if the type is double. + + + + It is worth noting that there is no primary key for this table. This + is simply because the framework has no use for one, and thus doesn't + require it. If a user so chooses, one may be added with a database + generated key, without causing any issues to the framework itself. +
+ +
+ BATCH_JOB_EXECUTION + + The BATCH_JOB_EXECUTION table holds all information relevant to the + JobExecution object. Every time a + Job is run there will always be a new + JobExecution, and a new row in this table: + + CREATE TABLE BATCH_JOB_EXECUTION ( + JOB_EXECUTION_ID BIGINT PRIMARY KEY , + VERSION BIGINT, + JOB_INSTANCE_ID BIGINT NOT NULL, + CREATE_TIME TIMESTAMP NOT NULL, + START_TIME TIMESTAMP DEFAULT NULL, + END_TIME TIMESTAMP DEFAULT NULL, + STATUS VARCHAR(10), + EXIT_CODE VARCHAR(20), + EXIT_MESSAGE VARCHAR(2500), + LAST_UPDATED TIMESTAMP, + constraint JOB_INSTANCE_EXECUTION_FK foreign key (JOB_INSTANCE_ID) + references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID) +) ; + + Below are descriptions for each column: + + + + JOB_EXECUTION_ID: Primary key that uniquely identifies this + execution. The value of this column is obtainable by calling the + getId method of the + JobExecution object. + + + + VERSION: See above section. + + + + JOB_INSTANCE_ID: Foreign key from the BATCH_JOB_INSTANCE table + indicating the instance to which this execution belongs. There may be + more than one execution per instance. + + + + CREATE_TIME: Timestamp representing the time that the execution + was created. + + + + START_TIME: Timestamp representing the time the execution was + started. + + + + END_TIME: Timestamp representing the time the execution was + finished, regardless of success or failure. An empty value in this + column even though the job is not currently running indicates that + there has been some type of error and the framework was unable to + perform a last save before failing. + + + + STATUS: Character string representing the status of the + execution. This may be COMPLETED, STARTED, etc. The object + representation of this column is the + BatchStatus enumeration. + + + + EXIT_CODE: Character string representing the exit code of the + execution. In the case of a command line job, this may be converted + into a number. + + + + EXIT_MESSAGE: Character string representing a more detailed + description of how the job exited. In the case of failure, this might + include as much of the stack trace as is possible. + + + + LAST_UPDATED: Timestamp representing the last time this + execution was persisted. + + +
+ +
+ BATCH_STEP_EXECUTION + + The BATCH_STEP_EXECUTION table holds all information relevant to the + StepExecution object. This table is very similar in + many ways to the BATCH_JOB_EXECUTION table and there will always be at + least one entry per Step for each + JobExecution created: + + CREATE TABLE BATCH_STEP_EXECUTION ( + STEP_EXECUTION_ID BIGINT PRIMARY KEY , + VERSION BIGINT NOT NULL, + STEP_NAME VARCHAR(100) NOT NULL, + JOB_EXECUTION_ID BIGINT NOT NULL, + START_TIME TIMESTAMP NOT NULL , + END_TIME TIMESTAMP DEFAULT NULL, + STATUS VARCHAR(10), + COMMIT_COUNT BIGINT , + READ_COUNT BIGINT , + FILTER_COUNT BIGINT , + WRITE_COUNT BIGINT , + READ_SKIP_COUNT BIGINT , + WRITE_SKIP_COUNT BIGINT , + PROCESS_SKIP_COUNT BIGINT , + ROLLBACK_COUNT BIGINT , + EXIT_CODE VARCHAR(20) , + EXIT_MESSAGE VARCHAR(2500) , + LAST_UPDATED TIMESTAMP, + constraint JOB_EXECUTION_STEP_FK foreign key (JOB_EXECUTION_ID) + references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID) +) ; + + Below are descriptions for each column: + + + + STEP_EXECUTION_ID: Primary key that uniquely identifies this + execution. The value of this column should be obtainable by calling + the getId method of the + StepExecution object. + + + + VERSION: See above section. + + + + STEP_NAME: The name of the step to which this execution + belongs. + + + + JOB_EXECUTION_ID: Foreign key from the BATCH_JOB_EXECUTION table + indicating the JobExecution to which this StepExecution belongs. There + may be only one StepExecution for a given + JobExecution for a given + Step name. + + + + START_TIME: Timestamp representing the time the execution was + started. + + + + END_TIME: Timestamp representing the time the execution was + finished, regardless of success or failure. An empty value in this + column even though the job is not currently running indicates that + there has been some type of error and the framework was unable to + perform a last save before failing. + + + + STATUS: Character string representing the status of the + execution. This may be COMPLETED, STARTED, etc. The object + representation of this column is the + BatchStatus enumeration. + + + + COMMIT_COUNT: The number of times in which the step has + committed a transaction during this execution. + + + + READ_COUNT: The number of items read during this + execution. + + + + FILTER_COUNT: The number of items filtered out of this + execution. + + + + WRITE_COUNT: The number of items written during this + execution. + + + + READ_SKIP_COUNT: The number of items skipped on read during this + execution. + + + + WRITE_SKIP_COUNT: The number of items skipped on write during + this execution. + + + + PROCESS_SKIP_COUNT: The number of items skipped during + processing during this execution. + + + + ROLLBACK_COUNT: The number of rollbacks during this + execution. + + + + EXIT_CODE: Character string representing the exit code of the + execution. In the case of a command line job, this may be converted + into a number. + + + + EXIT_MESSAGE: Character string representing a more detailed + description of how the job exited. In the case of failure, this might + include as much of the stack trace as is possible. + + + + LAST_UPDATED: Timestamp representing the last time this + execution was persisted. + + +
+ +
+ BATCH_JOB_EXECUTION_CONTEXT + + The BATCH_JOB_EXECUTION_CONTEXT table holds all information relevant + to an Job's + ExecutionContext. There is exactly one + Job ExecutionContext per + JobExecution, and it contains all of the job-level + data that is needed for a particular job execution. This data typically + represents the state that must be retrieved after a failure so that a + JobInstance can 'start from where it left + off'. + + CREATE TABLE BATCH_JOB_EXECUTION_CONTEXT ( + JOB_EXECUTION_ID BIGINT PRIMARY KEY, + SHORT_CONTEXT VARCHAR(2500) NOT NULL, + SERIALIZED_CONTEXT CLOB, + constraint JOB_EXEC_CTX_FK foreign key (JOB_EXECUTION_ID) + references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID) +) ; + + Below are descriptions for each column: + + + + JOB_EXECUTION_ID: Foreign key representing the + JobExecution to which the context belongs. + There may be more than one row associated to a given execution. + + + + SHORT_CONTEXT: A string version of the + SERIALIZED_CONTEXT. + + + + SERIALIZED_CONTEXT: The entire context, serialized. + + +
+ +
+ BATCH_STEP_EXECUTION_CONTEXT + + The BATCH_STEP_EXECUTION_CONTEXT table holds all information + relevant to an Step's + ExecutionContext. There is exactly one + ExecutionContext per + StepExecution, and it contains all of the data that + needs to persisted for a particular step execution. This data typically + represents the state that must be retrieved after a failure so that a + JobInstance can 'start from where it left + off'. + + CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT ( + STEP_EXECUTION_ID BIGINT PRIMARY KEY, + SHORT_CONTEXT VARCHAR(2500) NOT NULL, + SERIALIZED_CONTEXT CLOB, + constraint STEP_EXEC_CTX_FK foreign key (STEP_EXECUTION_ID) + references BATCH_STEP_EXECUTION(STEP_EXECUTION_ID) +) ; + + Below are descriptions for each column: + + + + STEP_EXECUTION_ID: Foreign key representing the + StepExecution to which the context belongs. + There may be more than one row associated to a given execution. + + + + SHORT_CONTEXT: A string version of the + SERIALIZED_CONTEXT. + + + + SERIALIZED_CONTEXT: The entire context, serialized. + + +
+ +
+ Archiving + + Because there are entries in multiple tables every time a batch job + is run, it is common to create an archive strategy for the meta-data + tables. The tables themselves are designed to show a record of what + happened in the past, and generally won't affect the run of any job, with + a couple of notable exceptions pertaining to restart: + + + + The framework will use the meta-data tables to determine if a + particular JobInstance has been run before. If it has been run, and + the job is not restartable, then an exception will be thrown. + + + + If an entry for a JobInstance is removed without having + completed successfully, the framework will think that the job is new, + rather than a restart. + + + + If a job is restarted, the framework will use any data that has + been persisted to the ExecutionContext to restore the Job's state. + Therefore, removing any entries from this table for jobs that haven't + completed successfully will prevent them from starting at the correct + point if run again. + + +
+ +
+ Recommendations for Indexing Meta Data Tables + + Spring Batch provides DDL samples for the meta-data tables in the + Core jar file for several common database platforms. Index declarations + are not included in that DDL because there are too many variations in how + users may want to index depending on their precise platform, local + conventions and also the business requirements of how the jobs will be + operated. The table below provides some indication as to which columns are + going to be used in a WHERE clause by the Dao implementations provided by + Spring Batch, and how frequently they might be used, so that individual + projects can make up their own minds about indexing. + + + Where clauses in SQL statements (exluding primary keys) and their + approximate frequency of use. + + + + + Default Table Name + + Where Clause + + Frequency + + + + BATCH_JOB_INSTANCE + + JOB_NAME = ? and JOB_KEY = ? + + Every time a job is launched + + + + BATCH_JOB_EXECUTION + + JOB_INSTANCE_ID = ? + + Every time a job is restarted + + + + BATCH_EXECUTION_CONTEXT + + EXECUTION_ID = ? and KEY_NAME = ? + + On commit interval, a.k.a. chunk + + + + BATCH_STEP_EXECUTION + + VERSION = ? + + On commit interval, a.k.a. chunk (and at start and end of + step) + + + + BATCH_STEP_EXECUTION + + STEP_NAME = ? and JOB_EXECUTION_ID = ? + + Before each step execution + + + +
+
+ diff --git a/docs/src/site/docbook/reference/whatsnew.xml b/docs/src/site/docbook/reference/whatsnew.xml index f40fdcee1..d192fe432 100644 --- a/docs/src/site/docbook/reference/whatsnew.xml +++ b/docs/src/site/docbook/reference/whatsnew.xml @@ -1,389 +1,408 @@ - - - - What's new in Spring Batch 2.0 - - The Spring Batch 2.0 release has six major themes: - - - - Java 5 - - - - Non Sequential Step Execution - - - - Chunk oriented processing - - - - Meta Data enhancements - - - - Scalability - - - - Configuration - - - -
- Java 5 - - The 1.x release of Spring Batch were all based on Java 1.4. This - prevented the framework from using many enhancements provided in Java 5 - such as generics, parameterized types, etc. The entire framework has been - updated to utilize these features. Java 1.4 is no - longer supported. Most of the interfaces developers work with - have been updated to support generic types. As an example, the ItemReader - interface from 1.1 is below: - - - public interface ItemReader { - - Object read() throws Exception; - - void mark() throws MarkFailedException; - - void reset() throws ResetFailedException; - } - - - - As you can see, the read method returns an - Object. The 2.0 version is below: - - - public interface ItemReader<T> { - - T read() throws Exception, UnexpectedInputException, ParseException; - - } - - - - As you can see, ItemReader now supports the - generic type, T, which is returned from read. You may also notice that - mark and reset have been removed. This is due to step processing strategy - changes, which are discussed below. Many other interfaces have been - similarly updated. -
- -
- Chunk Oriented Processing - - Previously, the default processing strategy provided by Spring Batch - was item-oriented processing: - - - - - - - - - - - - In item-oriented processing, the ItemReader returns one Object (the - 'item) which is then handed to the ItemWriter, periodically committing - when the number of items hits the commit interval. For example, if the - commit interval is 5, ItemReader and ItemWriter will each be called 5 - times. This is illustrated in a simplified code example below: - - - for(int i = 0; i < commitInterval; i++){ - Object item = itemReader.read(); - itemWriter.write(item); - } - - - - Both the ItemReader and ItemWriter interfaces were completely geared - toward this approach: - - - public interface ItemReader { - - Object read() throws Exception; - - void mark() throws MarkFailedException; - - void reset() throws ResetFailedException; - } - - - - public interface ItemWriter { - - void write(Object item) throws Exception; - - void flush() throws FlushFailedException; - - void clear() throws ClearFailedException; - } - - - - Because the 'scope' of the processing was one item, supporting - rollback scenarios required additional methods, which is what mark, reset, - flush, and clear provided. If, after successfully reading and writing 2 - items, the third had an error while writing, the transaction would need to - be rolled back. In this case, the clear method on the writer would be - called, indicating that it should clear its buffer, and reset would be - called on the ItemReader, indicating that it should return back to the - last position it was at when mark was called. (Both mark and flush are - called on commit) - - In 2.0, this strategy has been changed to a chunk-oriented - approach: - - - - - - - - - - - - Using the same example from above, if the commit interval is five, - read will be called 5 times, and write once. The items read will be - aggregated into a list, that will ultimately be written out, as the - simplified example below illustrates: - - - List items = new Arraylist(); - for(int i = 0; i < commitInterval; i++){ - items.add(itemReader.read()); - } - itemWriter.write(items); - - - - This approach not only allows for much simpler processing and - scalability approaches, it also makes the ItemReader and ItemWriter - interfaces much cleaner: - - - public interface ItemReader<T> { - - T read() throws Exception, UnexpectedInputException, ParseException; - - } - - - - - public interface ItemWriter<T> { - - void write(List<? extends T> items) throws Exception; - - } - - - - As you can see, the interfaces no longer contain the mark, reset, - flush, and clear methods. This makes the creation of readers and writers - much more straightforward for developers. In the case of - ItemReader, the interface is now forward-only. The - framework will buffer read items for developers in the case of rollback. - (There are exceptions if the underlying resource is transactional see: - ) ItemWriter is also simplified, - since it gets the entire 'chunk' of items at once, rather than one at a - time, it can decide to flush any resources (such as a file or hibernate - session) before returning control to the Step. More - detailed information on chunk-oriented processing can be found in . Reader and writer implementation - information can be found in 80 - -
- ItemProcessor - - Previously, Steps had only two dependencies, - ItemReader and - ItemWriter: - - - - - - - - - - - - The basic configuration above is fairly robust. However, there are - many cases where the item needs to be transformed before writing. In 1.x - this can be achieved using the composite pattern: - - - - - - - - - - - - This approach works, however, it requires an extra layer between - either the reader or the writer and the Step. - Furthermore, the ItemWriter would need to be - registered separately as an ItemStream with the - Step. For this reason, the ItemTransfomer was - renamed to ItemProcessor and moved up to the same level as ItemReader - and ItemWriter: - - - - - - - - - - -
-
- -
- Configuration enhancements - - Until 2.0, the only option for configuring batch jobs has been - normal spring bean configuration. However, in 2.0 there is a new namespace - for configuration. For example, in 1.1, configuring a job looked like the - following: - - - <bean id="footballJob" - class="org.springframework.batch.core.job.SimpleJob"> - <property name="steps"> - <list> - <!-- Step Bean details ommitted for clarity --> - <bean id="playerload" parent="simpleStep" /> - <bean id="gameLoad" parent="simpleStep" /> - <bean id="playerSummarization" parent="simpleStep" /> - </list> - </property> - <property name="jobRepository" ref="jobRepository" /> - </bean> - - - - In 2.0, the equivalent would be: - - - <job id="footballJob"> - <step name="playerload" next="gameLoad"/> - <step name="gameLoad" next="playerSummarization"/> - <step name="playerSummarization"/> - </job> - - - - More information on how to configure Jobs and Steps with the new - namespace can be found in , and . -
- -
- Meta Data access improvements - - The JobRepository interface represents basic - CRUD operations with Job meta-data. However, it can - be useful to query the meta-data. For that reason, the - JobExplorer and JobOperator - interfaces have been created: - - - - - - - - - - - - More information on the new meta data features can be found in . It is also worth noting that Jobs can now - be stopped via the database, removing the requirement to maintain a handle - to the JobExecution on the JVM the job was launched - in. -
- -
- Non Sequential Step Execution - - 2.0 has also seen improvements in how steps can be configured. - Rather than requiring that they solely be sequential: - - - - - - - - - - - - They may now be conditional: - - - - - - - - - - - - This new 'conditional flow' support is made easy to configure via - the new namespace: - - - <job id="job"> - <step name="stepA"> - <next on="FAILED" to="stepB" /> - <next on="*" to="stepC" /> - </step> - <step name="stepB" next="stepC" /> - <step name="stepC" /> - </job> - - - - More details on how to configure non sequential steps can be found - in -
- + + + + What's new in Spring Batch 2.0 + + The Spring Batch 2.0 release has six major themes: + + + + Java 5 + + + + Non Sequential Step Execution + + + + Chunk oriented processing + + + + Meta Data enhancements + + + + Scalability + + + + Configuration + + + +
+ Java 5 + + The 1.x releases of Spring Batch were all based on Java 1.4. This + prevented the framework from using many enhancements provided in Java 5 + such as generics, parameterized types, etc. The entire framework has been + updated to utilize these features. As a result, Java + 1.4 is no longer supported. Most of the interfaces developers + work with have been updated to support generic types. As an example, the + ItemReader interface from 1.1 is below: + + + public interface ItemReader { + + Object read() throws Exception; + + void mark() throws MarkFailedException; + + void reset() throws ResetFailedException; + } + + + + As you can see, the read method returns an + Object. The 2.0 version is below: + + + public interface ItemReader<T> { + + T read() throws Exception, UnexpectedInputException, ParseException; + + } + + + + As you can see, ItemReader now supports the + generic type, T, which is returned from read. You may also notice that + mark and reset have been + removed. This is due to step processing strategy changes, which are + discussed below. Many other interfaces have been similarly updated. +
+ +
+ Chunk Oriented Processing + + Previously, the default processing strategy provided by Spring Batch + was item-oriented processing: + + + + + + + + + + + + In item-oriented processing, the ItemReader + returns one Object (the 'item') which is then + handed to the ItemWriter, periodically committing + when the number of items hits the commit interval. For example, if the + commit interval is 5, ItemReader and + ItemWriter will each be called 5 times. This is + illustrated in a simplified code example below: + + + for(int i = 0; i < commitInterval; i++){ + Object item = itemReader.read(); + itemWriter.write(item); + } + + + + Both the ItemReader and + ItemWriter interfaces were completely geared toward + this approach: + + + public interface ItemReader { + + Object read() throws Exception; + + void mark() throws MarkFailedException; + + void reset() throws ResetFailedException; + } + + + + public interface ItemWriter { + + void write(Object item) throws Exception; + + void flush() throws FlushFailedException; + + void clear() throws ClearFailedException; + } + + + + Because the 'scope' of the processing was one item, supporting + rollback scenarios required additional methods, which is what + mark, reset, + flush, and clear + provided. If, after successfully reading and writing 2 items, the third + has an error while writing, the transaction would need to be rolled back. + In this case, the clear method on the writer + would be called, indicating that it should clear + its buffer, and reset would be called on the + ItemReader, indicating that it should return back + to the last position it was at when mark was + called. (Both mark and + flush are called on commit) + + In 2.0, this strategy has been changed to a chunk-oriented + approach: + + + + + + + + + + + + Using the same example from above, if the commit interval is five, + read will be called 5 times, and write once. The items read will be + aggregated into a list, that will ultimately be written out, as the + simplified example below illustrates: + + + List items = new Arraylist(); + for(int i = 0; i < commitInterval; i++){ + items.add(itemReader.read()); + } + itemWriter.write(items); + + + + This approach not only allows for much simpler processing and + scalability approaches, it also makes the + ItemReader and ItemWriter + interfaces much cleaner: + + + public interface ItemReader<T> { + + T read() throws Exception, UnexpectedInputException, ParseException; + + } + + + + + public interface ItemWriter<T> { + + void write(List<? extends T> items) throws Exception; + + } + + + + As you can see, the interfaces no longer contain the + mark, reset, + flush, and clear + methods. This makes the creation of readers and writers much more + straightforward for developers. In the case of + ItemReader, the interface is now forward-only. The + framework will buffer read items for developers in the case of rollback + (though there are exceptions if the underlying resource is transactional + see: ). + ItemWriter is also simplified, since it gets the + entire 'chunk' of items at once, rather than one at a time, it can decide + to flush any resources (such as a file or hibernate session) before + returning control to the Step. More detailed + information on chunk-oriented processing can be found in . Reader and writer implementation + information can be found in . + +
+ ItemProcessor + + Previously, Steps had only two + dependencies, ItemReader and + ItemWriter: + + + + + + + + + + + + The basic configuration above is fairly robust. However, there are + many cases where the item needs to be transformed before writing. In 1.x + this can be achieved using the composite pattern: + + + + + + + + + + + + This approach works. However, it requires an extra layer between + either the reader or the writer and the Step. + Furthermore, the ItemWriter would need to be + registered separately as an ItemStream with the + Step. For this reason, the + ItemTransfomer was renamed to + ItemProcessor and moved up to the same level as + ItemReader and + ItemWriter: + + + + + + + + + + +
+
+ +
+ Configuration enhancements + + Until 2.0, the only option for configuring batch jobs has been + normal spring bean configuration. However, in 2.0 there is a new namespace + for configuration. For example, in 1.1, configuring a job looked like the + following: + + + <bean id="footballJob" + class="org.springframework.batch.core.job.SimpleJob"> + <property name="steps"> + <list> + <!-- Step Bean details ommitted for clarity --> + <bean id="playerload" parent="simpleStep" /> + <bean id="gameLoad" parent="simpleStep" /> + <bean id="playerSummarization" parent="simpleStep" /> + </list> + </property> + <property name="jobRepository" ref="jobRepository" /> + </bean> + + + + In 2.0, the equivalent would be: + + + <job id="footballJob"> + <step name="playerload" next="gameLoad"/> + <step name="gameLoad" next="playerSummarization"/> + <step name="playerSummarization"/> + </job> + + + + More information on how to configure Jobs and Steps with the new + namespace can be found in , and . +
+ +
+ Meta Data access improvements + + The JobRepository interface represents basic + CRUD operations with Job meta-data. However, it may + also be useful to query the meta-data. For that reason, the + JobExplorer and JobOperator + interfaces have been created: + + + + + + + + + + + + More information on the new meta data features can be found in . It is also worth noting that Jobs can now + be stopped via the database, removing the requirement to maintain a handle + to the JobExecution on the JVM the job was launched + in. +
+ +
+ Non Sequential Step Execution + + 2.0 has also seen improvements in how steps can be configured. + Rather than requiring that they solely be sequential: + + + + + + + + + + + + They may now be conditional: + + + + + + + + + + + + This new 'conditional flow' support is made easy to configure via + the new namespace: + + + <job id="job"> + <step name="stepA"> + <next on="FAILED" to="stepB" /> + <next on="*" to="stepC" /> + </step> + <step name="stepB" next="stepC" /> + <step name="stepC" /> + </job> + + + + More details on how to configure non sequential steps can be found + in . +
+