diff --git a/docs/models/Batch Presentation Diagrams.vsd b/docs/models/Batch Presentation Diagrams.vsd index d25c7da09..2cd6bc886 100644 Binary files a/docs/models/Batch Presentation Diagrams.vsd and b/docs/models/Batch Presentation Diagrams.vsd differ diff --git a/docs/src/site/docbook/reference/core.xml b/docs/src/site/docbook/reference/core.xml index 52c809f89..6af6ee987 100644 --- a/docs/src/site/docbook/reference/core.xml +++ b/docs/src/site/docbook/reference/core.xml @@ -2,20 +2,17 @@ - Spring Batch Core - the Domain language of Batch + The Domain Language of Batch
Introduction To any experienced batch architect, the overall concepts of batch - processing described above should be familiar and comfortable. There are - “Jobs” and “Steps” and a developer supplied processing units called - ItemReaders and ItemWriters. The following diagram is only a slight - variation of the batch reference architecture that has been used for - decades. JCL and COBOL developers are likely to be as comfortable with the - concepts as C++, C# and Java developers. However, because of the Spring - patterns, operations, templates, callbacks, and idioms, there are - opportunities for + processing used in Spring Batch should be familiar and comfortable. There + are “Jobs” and “Steps” and developer supplied processing units called + ItemReaders and ItemWriters. However, because of the Spring patterns, + operations, templates, callbacks, and idioms, there are opportunities for + significant improvement in adherence to a clear separation of concerns, @@ -36,17 +33,19 @@ - The diagram below provides an overview of the high level components, - technical services, and basic operations required by a batch architecture. - This architecture framework is a blueprint that has been proven through - decades of implementations on the last several generations of platforms - (COBOL/Mainframe, C++/Unix, and now Java/anywhere). The Simple Batch - Execution Environment provides a physical implementation of the layers, - components and technical services commonly found in robust, maintainable - systems used to address the creation of simple to complex batch - applications, with the infrastructure and extensions to address very - complex processing needs. The materials below will walk through the - details of the diagram. + The diagram below is only a slight variation of the batch reference + architecture that has been used for decades. It provides an overview of + the high level components, technical services, and basic operations + required by a batch architecture. This architecture framework is a + blueprint that has been proven through decades of implementations on the + last several generations of platforms (COBOL/Mainframe, C++/Unix, and now + Java/anywhere). JCL and COBOL developers are likely to be as comfortable + with the concepts as C++, C# and Java developers. Spring Batch provides a + physical implementation of the layers, components and technical services + commonly found in robust, maintainable systems used to address the + creation of simple to complex batch applications, with the infrastructure + and extensions to address very complex processing needs. The materials + below will walk through the details of the diagram.
@@ -56,21 +55,37 @@ - Figure 1: Batch Stereotypes + Figure 2.1: Batch Stereotypes - The application style is organized into four logical tiers, which - include Run, Job, Application, and Data tiers. The primary goal for + The colors used on the above diagram are extremely important. Grey + represents an external application such as an enterprise scheduler or a + database. It's important to note that scheduling is grey, and should thus + be considered separate from Spring Batch. Blue represents application + architecture services. In most cases these are provided by Spring Batch + with out of the box implementations, but an architecture time may make + specific implementations that better address their specific needs. Yellow + represents the pieces that must be configured by a developer. For example, + they need to configure their job schedule so that the job is kicked off at + the appropriate time. They also need to create a job configuration that + defines how their job will be run. It is also worth noting that the + ItemReader and ItemWriter + used by an application may just as easily be a custom one made by the + developer for the specific batch job, rather than one provided by Spring + Batch or even an architecture team. + + The Batch Application Style is organized into four logical tiers, + which include Run, Job, Application, and Data. The primary goal for organizing an application according to the tiers is to embed what is known as "separation of concerns" within the system. These tiers can be conceptual but may they prove effective in mapping the deployment of the @@ -111,91 +126,35 @@ completely missing and in other cases one Job Script can start several Batch Job instances. - In addition the components describe the batch interaction - and services stereotypes that are the domain language and interfaces - implemented by developers in constructing a batch solution. As the diagram - illustrates, custom applicaton archifacts, generally created by the - developer, are the following: - - - - Job Scripts - - - - JobConfigurations - - - - Tasklet - - - - Business Logic - - - - The application architect needs to consider the batch execution - environment with the following issues: - - - - Define how batch jobs will be launched - - - - Job construction and Configuration - - - - Step construction and Configuration - - - - ItemReaders - - - - ItemWriters - - - - Data Access Strategies - - - - The grey icons indicate the technologies selected as part of the - batch solution that are not part of the final solution and entail items - like: - - - - Schedulers (e.g. Quartz, Tivoli, etc.) - - - - Physical Resources in the Data Tier that are the source and - target of ItemReaders and Writers like Message Queues, Databases, - Files and Print Queues. - - +
- Job Stereotypes + Job Stereotypes This section describes stereotypes relating to the concept of a batch job. A job is an entity that encapsulates an entire batch process. The file containing the job may sometimes be referred to as the "job - configuration. + configuration". However, Job is just the top of an + overall hierarchy: + + + + + + + + + +
Job The job could be described as the heart of the Spring Batch framework. It is represented by a Spring bean that implements the - Job interface and contains all of the - information necessary to define the operations performed by a job. A job + Job interface and contains all of the information + necessary to define the operations performed by a job. A job configuration is typically contained within a Spring XML configuration file and the job's name is determined by the "id" attribute associated with the job configuration bean. The job configuration contains @@ -206,12 +165,7 @@ - Definition and ordering of [Step Configurations|#Step - Configuration] - - - - The limit of how many times this job may be started + Definition and ordering of Steps @@ -219,117 +173,388 @@ - A default simple implementation of the Job interface is provided by Spring Batch in the - form of the SimpleJob class which creates some standard functionality on - top of Job, namely a standard execution - logic that all jobs should utilize. In general, all job configurations - should be defined using a bean of type SimpleJob. + A default simple implementation of the Job + interface is provided by Spring Batch in the form of the + SimpleJob class which creates some standard + functionality on top of Job, namely a standard + execution logic that all jobs should utilize. In general, all jobs + should be defined using a bean of type + SimpleJob: + + <bean id="footballJob" + class="org.springframework.batch.execution.job.SimpleJob"> + <property name="steps"> + <list> + <!-- Step Bean details ommitted for clarity --> + <bean id="playerload" parent="simpleStep" /> + <bean id="gameLoad" parent="simpleStep" /> + <bean id="playerSummarization" parent="simpleStep" /> + </list> + </property> + <property name="restartable" value="true" /> + </bean>
- Job Instance + JobInstance - A job instance refers to the business concept of a single job - invocation. In other words, suppose you have a job called "foo" that is - run three times a day. There will be one "foo" configuration, and each - time "foo" is supposed to run would be an instance of the "foo" job. - Each instance would be uniquely identified as each one represents a - distinct batch need. Further, each instance might have attempted several - times to complete its work. Each attempt is represented by a JobExecution, described below. A job instance is - not considered to be complete until an associated job execution - completes successfully. As such, a single job instance may have many - executions. To keep track of this, every job instance provides a - reference to the last execution attempt. - - For example, a unique instance might be identified by just a job - name, or by the combination of a job name and a scheduled date. Using - this second type of identification, we might have two distinct - instances, "foo-01-01-2008" and "foo-01-02-2008." Although these two - instances would share the same configuration, they would each have their - own set of executions and the successful completion of one instance - would not affect the status of the other. - - Job instances are represented by objects of the JobInstance class, which are created when the job - is executed. Each job instance contains references to related [Step - Instances|#Step Instance] and a set of job parameters, represented by - the JobParameters that uniquely - identifies this job instance. + A JobInstance refers to the concept of a + logical job run. Let's consider a batch job that should be run once at + the end of the day, such as the 'EndOfDay' job from the diagram above. + There is a one 'EndOfDay' Job, but each + individual run of the Job must be tracked + separately. In the case of this job, there will be one logical + JobInstance per day. For example, there will be a + January 1st run, and a January 2nd run. If the January 1st run fails the + first time and is run again the next day, it's still the January 1st + run. (Usually this corresponds with the data its processing as well, + meaning the January 1st run processes data for January 1st, etc) That is + to say, each JobInstance can have multiple + executions. (JobExecution is discussed in more + detail below) and only one instance can be running at a given time. +
- Job Parameters + JobParameters - Job Parameters represent parameters to a job that are used to - either uniquely identify an instance, or to help drive processing. One - example of identifying a job would be 'Schedule Date'. This is a common - batch use case where an 'effective date' for the data is used. For - example, you may be processing on thursday for a scheduled date of - Monday. Once the JobInstance with a schedule date of monday has - completed successfully, a schedule date of Tuesday will be used for the - next run. You will then be given a new JobInstance by the framework, - since your parameters are different. This allows you to effectively - control how you define a 'JobInstance', since you control what - parameters are passed in. + Having discussed JobInstance and how it + differs from Job, the natural question to ask is, + "how is one JobInstance distinguished from another?" The answer is: + JobParameters. + JobParameters are any set of parameters used to + start a batch job, which can be used for identification or even as + reference data during the run. In the example above, where there are two + instances, one for January 1st, and another for January 2nd, there is + really only one Job, one that was started with a job parameter of + 01-01-2008 and another that was started with a parameter of 01-02-2008. + Thus, the contract can be defined as: JobInstance + = Job + JobParameters. + This allows you to effectively control how you define a + JobInstance, since you control what parameters + are passed in.
- Job Execution + JobExecution - A job execution refers to the technical concept of a single - attempt to run a job. It is a single attempt to execute the logic - represented by a job instance. A job execution may end in failure or - success, but the job instance corresponding to a given execution will - not be marked as complete unless the execution completes - successfully. + A JobExecution refers to the technical + concept of a single attempt to run a Job. An + execution may end in failure or success, but the + JobInstance corresponding to a given execution + will not be marked as complete unless the execution completes + successfully. For instance, if we have a + JobInstance of the EndOfDay job for 01-01-2008, + as described above, that fails to successfully complete its work the + first time it is run, when we attempt to run it again (with the same job + parameters of 01-01-2008), a new job execution will be created. - For instance, if we have a job instance "foo-01-01-2008" that - fails to successfully complete its work the first time it is run, when - we attempt to run it again, a new job execution will be created. If our - "foo" configuration is restartable, we may begin our second job - execution from a restart point. Otherwise, our job execution will start - from the beginning. In either case, we will see that our single job - instance has had two job executions. + A Job defines what a job is and defines how it is to be executed, + and JobInstance is a purely organization object + to group executions together, primarily to enable correct restart. A + JobExecution, however, is the primary storage + mechanism for what actually happened during a run, and as such contains + many more properties that must be controlled and persisted: - Job executions are represented by objects of the JobExecution class. These job executions are - created by an implementation of the JobRepository interface from a given JobInstance corresponding to a unique JobParameters object. Each job execution contains - a reference to its corresponding job instance and related Step - Executions. + + JobExecution properties + + + + + status + + A BatchStatus object that + indicates the status of the execution. While it's running, it's + BatchStatus.STARTED, if it fails it's BatchStatus.FAILED, and if + it finishes successfully it's BatchStatus.COMPLETED + + + + startTime + + A java.util.Date representing the + current system time when the execution was started. + + + + endTime + + A java.util.Date representing the + current system time when the execution finished, regardless of + whether or not it was successful. + + + + exitStatus + + The ExitStatus indicating the + result of the run. It is most important because it contains an + exit code that will be returned to the caller. See chapter 5 for + more details. + + + +
+ + These properties are important because they will be persisted and + can be used to completely determine the status of an execution. For + example, if the EndOfDay job for 01-01 is executed at 9:00 PM, and fails + at 9:30, the following entries will be in the batch meta data + tables: + + + BATCH_JOB_INSTANCE + + + + + JOB_INSTANCE_ID + + JOB_NAME + + + + 1 + + EndOfDayJob + + + +
+ + + BATCH_JOB_PARAMS + + + + + JOB_INSTANCE_ID + + TYPE_CD + + KEY_NAME + + DATE_VAL + + + + 1 + + DATE + + schedule.Date + + 2008-01-01 00:00:00 + + + +
+ + + BATCH_JOB_EXECUTION + + + + + JOB_EXECUTION_ID + + JOB_INSTANCE_ID + + START_TIME + + END_TIME + + STATUS + + + + 1 + + 1 + + 2008-01-01 21:00:23.571 + + 2008-01-01 21:30:17.132 + + FAILED + + + +
+ + + extra columns in the table have been removed for added + clarity. + + + Now that the job has failed, let's assume that it took the entire + course of the night for the problem to be determined, so that the 'batch + window' is now closed. Assuming the window starts at 9:00 PM, the job + will be kicked off again for 01-01, starting where it left off and + completing successfully at 9:30. Because it's now the next day, the + 01-02 job must be run as well, which is kicked off just afterwards at + 9:31, and completes in it's normal one hour time at 10:30. There should + now be an extra entry in both the job instance and job parameters table, + and two extra entries in the job execution table: + + + BATCH_JOB_INSTANCE + + + + + JOB_INSTANCE_ID + + JOB_NAME + + + + 1 + + EndOfDayJob + + + + 2 + + EndOfDayJob + + + +
+ + + BATCH_JOB_PARAMS + + + + + JOB_INSTANCE_ID + + TYPE_CD + + KEY_NAME + + DATE_VAL + + + + 1 + + DATE + + schedule.Date + + 2008-01-01 00:00:00 + + + + 2 + + DATE + + schedule.Date + + 2008-01-02 00:00:00 + + + +
+ + + BATCH_JOB_EXECUTION + + + + + JOB_EXECUTION_ID + + JOB_INSTANCE_ID + + START_TIME + + END_TIME + + STATUS + + + + 1 + + 1 + + 2008-01-01 21:00 + + 2008-01-01 21:30 + + FAILED + + + + 2 + + 1 + + 2008-01-02 21:00 + + 2008-01-02 21:30 + + COMPLETED + + + + 3 + + 2 + + 2008-01-02 21:31 + + 2008-01-02 22:29 + + COMPLETED + + + +
Step Stereotypes - This section describes stereotypes relating to the concept of a - batch step. A step is an entity that encapsulates a single, independent - phase of a batch job. Therefore, every batch job is composed entirely of - one or more batch steps. Steps should be thought of as unique processing - streams that will be executed in sequence. For example, if you have one - step that loads a file into a database, another that reads from the - database, validates the data, preforms processing, and then writes to - another table, and another that reads from that table and writes out to a - file. Each of these steps will be performed completely before moving on to - the next step. The file will be completely read into the database before - step 2 can begin. + A Step is an entity that encapsulates a + single, independent phase of a batch job. Therefore, every batch job is + composed entirely of one or more batch steps. Steps should be thought of + as unique processing streams that will be executed in sequence. For + example, if you have one step that loads a file into a database, another + that reads from the database, validates the data, preforms processing, and + then writes to another table, and another that reads from that table and + writes out to a file. Each of these steps will be performed completely + before moving on to the next step. The file will be completely read into + the database before step 2 can begin. As with Job, a Step has individual + executions, that correspond with unique JobExecutions: + + + + + + + + + +
- Step + Step - A batch step contains all of the information necessary to define a - discrete set of business logic within a job. This is a necessarily vague - description because the contents of any given step are at the discretion - of the developer writing jobs. A step can be as narrowly defined as a - single line of code or as broadly defined as necessary to complete the - entire work of your job. There are several factors that will affect the - breadth of your step configurations. + A Step contains all of the information + necessary to define a discrete set of business logic within a job. This + is a necessarily vague description because the contents of any given + step are at the discretion of the developer writing a job. A step can be + as narrowly defined as a single line of code or as broadly defined as + necessary to complete the entire work of a job. There are several + factors that will affect the breadth of step configurations. @@ -351,53 +576,108 @@ Steps are defined by instantiating implementations of the - Step interface. Two step implementation - classes are available in the Spring Batch framework, and they are each - discussed in detail in other sections of this guide. For most - situations, the ItemOrientedStep - implementation is sufficient, but custom control flow behavior and - transaction management behavior can also be configured by using a - RepeatOperationsStep. + Step interface. Two step implementation classes + are available in the Spring Batch framework, and they are each discussed + in detail in other sections of this guide. For most situations, the + ItemOrientedStep implementation is sufficient, + but for situations where only one call is needed, such as a stored + procedure call or a wrapper around existing script, a + TaskletStep may be the better option.
- Step Execution + StepExecution - A step execution represents the technical concept of a single - attempt to execute a step. It is a single attempt to execute the logic - represented by a step instance. + A StepExecution represents the technical + concept of a single attempt to execute a Step. + For instance, using the example from + JobExecution, if we have a job instance + "EndOfJob-01-01-2008" that fails to successfully complete its work the + first time it is run, when we attempt to run it again, a new + StepExecution will be created. Each of these step + executions may represent a different invocation of the batch framework, + but they will all correspond to the same + JobInstance. - For instance, if we have a step instance "foo-01-01-2008#step1" - that fails to successfully complete its work the first time it is run, - when we attempt to run it again, a new step execution will be created. - Each of these step executions may represent a different invocation of - the batch framework, but they will all correspond to the same step - instance. + Step executions are represented by objects of the + StepExecution class. Each execution contains a + reference to its corresponding step and job execution, and transaction + related data such as commit and rollback count and start and end times. + Additionally, each step execution will contain an + ExecutionContext, which contains any data a + developer needs persisted across batch runs, such as statistics or state + information needed to restart. The following is a listing of the + properties for StepExecution: - Step executions are represented by objects of the StepExecution class. These step executions are - created by an implementation of the Job - interface from a given JobExecution. - Each step execution contains a reference to its corresponding step and - job execution, and transaction related data such as commit and rollback - count and start and end times. Additionally, each step execution will - contain a set of execution attributes, which will contain statistics and - restart data. + + StepExecution properties + + + + + status + + A BatchStatus object that + indicates the status of the execution. While it's running, it's + BatchStatus.STARTED, if it fails it's BatchStatus.FAILED, and if + it finishes successfully it's BatchStatus.COMPLETED + + + + startTime + + A java.util.Date representing the + current system time when the execution was started. + + + + endTime + + A java.util.Date representing the + current system time when the execution finished, regardless of + whether or not it was successful. + + + + exitStatus + + The ExitStatus indicating the + result of the run. It is most important because it contains an + exit code that will be returned to the caller. See chapter 5 for + more details. + + + + executionContext + + The 'property bag' containing any user data that needs to + be persisted between batch runs. + + + + + + + + + +
Tasklets - A tasklet represents the execution of a logical unit of work, as - defined by its implementation of the Spring Batch provided Tasklet - interface. Tasklets are used when defining step configurations to - specify the work done by the step. Subsequently, the logic in a tasklet - is atomic in terms of transactions. A transaction will never commit - until an entire tasklet execution is complete (unless an exception - occurs - a transaction might either commit or rollback if that behavior - is specified in the step's exception management strategy). There is a - specific implementation of the Step interface, TaskletStep, that works - directly with a Tasklet. + A Tasklet represents the execution of a + logical unit of work, as defined by its implementation of the Spring + Batch provided Tasklet interface. Tasklets are + used when defining step configurations to specify the work done by the + step. Subsequently, the logic in a tasklet is atomic in terms of + transactions. A transaction will never commit until an entire tasklet + execution is complete (unless an exception occurs - a transaction might + either commit or rollback if that behavior is specified in the step's + exception management strategy). There is a specific implementation of + the Step interface, TaskletStep, that works directly with a + Tasklet.
diff --git a/docs/src/site/docbook/reference/images/cursorExample.png b/docs/src/site/docbook/reference/images/cursorExample.png new file mode 100644 index 000000000..757d5b639 Binary files /dev/null and b/docs/src/site/docbook/reference/images/cursorExample.png differ diff --git a/docs/src/site/docbook/reference/images/drivingQueryExample.png b/docs/src/site/docbook/reference/images/drivingQueryExample.png new file mode 100644 index 000000000..adabc0a54 Binary files /dev/null and b/docs/src/site/docbook/reference/images/drivingQueryExample.png differ diff --git a/docs/src/site/docbook/reference/images/drivingQueryJob.png b/docs/src/site/docbook/reference/images/drivingQueryJob.png new file mode 100644 index 000000000..230a7157f Binary files /dev/null and b/docs/src/site/docbook/reference/images/drivingQueryJob.png differ diff --git a/docs/src/site/docbook/reference/images/job-heirarchy.png b/docs/src/site/docbook/reference/images/job-heirarchy.png new file mode 100644 index 000000000..a6c201c7f Binary files /dev/null and b/docs/src/site/docbook/reference/images/job-heirarchy.png differ diff --git a/docs/src/site/docbook/reference/images/jobHeirarchyWithSteps.png b/docs/src/site/docbook/reference/images/jobHeirarchyWithSteps.png new file mode 100644 index 000000000..912e32616 Binary files /dev/null and b/docs/src/site/docbook/reference/images/jobHeirarchyWithSteps.png differ diff --git a/docs/src/site/docbook/reference/images/oxm-fragments.png b/docs/src/site/docbook/reference/images/oxm-fragments.png new file mode 100644 index 000000000..f77af97e7 Binary files /dev/null and b/docs/src/site/docbook/reference/images/oxm-fragments.png differ diff --git a/docs/src/site/docbook/reference/images/spring-batch-layers.png b/docs/src/site/docbook/reference/images/spring-batch-layers.png new file mode 100644 index 000000000..ff1764074 Binary files /dev/null and b/docs/src/site/docbook/reference/images/spring-batch-layers.png differ diff --git a/docs/src/site/docbook/reference/images/spring-batch-reference-model.png b/docs/src/site/docbook/reference/images/spring-batch-reference-model.png new file mode 100644 index 000000000..ae1ff22ef Binary files /dev/null and b/docs/src/site/docbook/reference/images/spring-batch-reference-model.png differ diff --git a/docs/src/site/docbook/reference/images/xmlinput.png b/docs/src/site/docbook/reference/images/xmlinput.png new file mode 100644 index 000000000..f270346c2 Binary files /dev/null and b/docs/src/site/docbook/reference/images/xmlinput.png differ diff --git a/docs/src/site/docbook/reference/index.xml b/docs/src/site/docbook/reference/index.xml index 48e864a7e..8ebfc7cea 100644 --- a/docs/src/site/docbook/reference/index.xml +++ b/docs/src/site/docbook/reference/index.xml @@ -1,41 +1,51 @@ +"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"> - - Spring Batch - Reference Documentation - Spring Batch 1.0 - - - Dave - Syer - - - Wayne - Lund - - - Scott - Wintermute - - - - - Copies of this document may be made for your own use and - for distribution to others, provided that you do not - charge any fee for such copies and further provided that - each copy contains this Copyright Notice, whether - distributed in print or electronically. - - - - - - - - - - - - + + Spring Batch - Reference Documentation + Spring Batch 1.0 + + + + Dave + + Syer + + + + Wayne + + Lund + + + + Lucas Ward + + + + + Copies of this document may be made for your own use and for + distribution to others, provided that you do not charge any fee for such + copies and further provided that each copy contains this Copyright + Notice, whether distributed in print or electronically. + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/src/site/docbook/reference/readersAndWriters.xml b/docs/src/site/docbook/reference/readersAndWriters.xml index 5898dbbfd..6f08441ca 100644 --- a/docs/src/site/docbook/reference/readersAndWriters.xml +++ b/docs/src/site/docbook/reference/readersAndWriters.xml @@ -8,7 +8,7 @@ Introduction All batch processing can be described in its most simple form as - reading in large ammounts of data, performing some type of calculation or + reading in large amounts of data, performing some type of calculation or transformation, and writing the result out. Spring Batch provides two key interfaces to help perform bulk reading and writing: ItemReader and ItemWriter @@ -25,7 +25,7 @@ Flat File- Flat File Item Readers read lines of data from a flat file that typically describe records with fields of data defined by fixed positions in the file or delimited by some special - character (e.g. comma). + character (e.g. Comma). @@ -42,7 +42,7 @@ of the current row if restart is required, basic statistics, and some transaction enhancements that will be explained later. - There are many more possbilities, but we'll focus on the + There are many more possibilities, but we'll focus on the basic ones for this chapter. A complete list of all available ItemReaders can be found in Appendix A. @@ -59,31 +59,36 @@ } - The read() method defines the most essential contract of the - ItemReader, calling it returns one Item, returning null if no more items - are left. An item might represent a line in a file, a row in a database, - or an element in an xml file. It is generally expected that these will be - mapped to a useable domain object (i.e. Trade or Foo, etc) but there is no - requirement in the contract to do so. + The read method defines the most essential + contract of the ItemReader, calling it returns one + Item, returning null if no more items are left. An item might represent a + line in a file, a row in a database, or an element in an XML file. It is + generally expected that these will be mapped to a usable domain object + (i.e. Trade or Foo, etc) but there is no requirement in the contract to do + so. - The mark() and reset() methods are important due to the - transactional nature of batch processing. Mark() will be called before - reading begins. Calling reset() at anytime will position the ItemReader to - its position when mark() was last called. The semantics are very similar - to java.io.Reader. + The mark and reset + methods are important due to the transactional nature of batch processing. + Mark() will be called before reading begins. Calling + reset at anytime will position the + ItemReader to its position when + mark was last called. The semantics are very + similar to java.io.Reader.
ItemWriter - Item Writers are similar in functionality to an ItemReader with the - exception that the operations are reversed. They still need to be located, - opened and closed but they differ in the case that we write out, rather - than reading in. In the case of databases or queues these may be inserts, - updates or sends. The format of the serialization of the output source is - specific for every batch job. + ItemWriter is similar in functionality to an + ItemReader with the exception that the operations + are reversed. They still need to be located, opened and closed but they + differ in the case that we write out, rather than reading in. In the case + of databases or queues these may be inserts, updates or sends. The format + of the serialization of the output source is specific for every batch + job. - As with ItemReader, ItemWriter is a fairly generic interface: + As with ItemReader, + ItemWriter is a fairly generic interface: public interface ItemWriter { @@ -95,16 +100,20 @@ } - As with read() on ItemReader, write provides the basic contract of - ItemWriter, it will attempt to write out the item passed in as long as it - is open. As with mark() and reset(), flush() and clear() are necessary due - to the nature of batch processing. Because it is generally expected that - items will be 'batched' together into a chunk, and then output, it is - expected that an ItemWriter will perform some type of buffering. flush() - will empty the buffer by actually writing the items out, whereas clear - will simply throw the contents of the buffer away. In most cases, a Step - implementation will call flush() before a commit and clear() in case of - rollback. + As with read on + ItemReader, write provides the basic contract of + ItemWriter, it will attempt to write out the item + passed in as long as it is open. As with mark and + reset, flush and + clear are necessary due to the nature of batch + processing. Because it is generally expected that items will be 'batched' + together into a chunk, and then output, it is expected that an + ItemWriter will perform some type of buffering. + flush will empty the buffer by actually writing + the items out, whereas clear will simply throw + the contents of the buffer away. In most cases, a Step implementation will + call flush before a commit and + clear in case of rollback.
@@ -126,32 +135,41 @@ } - Before describing each method, it's worth breifly mentioning the - ExecutionContext. Clients of an ItemReader that is also an ItemStream - should call open() before any calls to read, to open any resources such as - files or obtain connections. A similar restriction applies to an - ItemWriter that is also an ItemStream. As mentioned before, if expected - data is found in the ExecutionContext, it may be used to start the - ItemReader or ItemWriter at a location other than its initial state. - Converely, close will be called to ensure any resources allocated during - open will be released safely. Update() is called primarily to ensure that - any state currently being held is loaded into the provided - ExecutionContext. This method will be called before committing, to ensure - that the current state is persisted in the database before commit. + Before describing each method, it's worth briefly mentioning the + ExecutionContext. Clients of an + ItemReader that is also an + ItemStream should call + open before any calls to + read, to open any resources such as files or + obtain connections. A similar restriction applies to an + ItemWriter that is also an + ItemStream. As mentioned before, if expected data + is found in the ExecutionContext, it may be used to + start the ItemReader or + ItemWriter at a location other than its initial + state. Conversely, close will be called to ensure any resources allocated + during open will be released safely. + update is called primarily to ensure that any + state currently being held is loaded into the provided + ExecutionContext. This method will be called before + committing, to ensure that the current state is persisted in the database + before commit. - In the special case where the client of an ItemStream is a Step - (from the Spring Batch Core), an ExecutionContext is created for each - StepExecution to allow users to store the state of a particular execution, - with the expectation that it will be returned if the same JobInstance is - started again. For those familiar with Quartz, the semantics are very - similar to a Quartz JobDataMap. + In the special case where the client of an + ItemStream is a Step (from + the Spring Batch Core), an ExecutionContext is + created for each StepExecution to allow users to + store the state of a particular execution, with the expectation that it + will be returned if the same JobInstance is started + again. For those familiar with Quartz, the semantics are very similar to a + Quartz JobDataMap.
Flat Files One of the most common mechanisms for interchanging bulk data has - always been the flat file. Unlike XML, which has an aggreed upon standard + always been the flat file. Unlike XML, which has an agreed upon standard for defining how it is structured (XSD), anyone reading a flat file must understand ahead of time exactly how the file is structured. In general, all flat files fall into two general types: Delimited and Fixed @@ -162,30 +180,34 @@ When working with flat files in Spring Batch, regardless of whether it is for input or output, one of the most important classes is - the FieldSet. Many architectures and libraries contain abstractions for - helping you read in from a file, but they usually return a String or an - array of Strings. This really only gets you halfway there. A FieldSet is - Spring Batch’s abstraction for enabling the binding of fields from a - file resource. It allows developers to work with file input in much the - same way as they would work with database input. A FieldSet is - conceptually very similar to a Jdbc ResultSet. FieldSets only require - one argument, a String array of tokens. Optionally you can also - configure in the names of the fields so that the fields may be accessed - either by index or name as patterned after ResultSet. In code it means - it's as simple as: + the FieldSet. Many architectures and libraries + contain abstractions for helping you read in from a file, but they + usually return a String or an array of Strings. This really only gets + you halfway there. A FieldSet is Spring Batch’s + abstraction for enabling the binding of fields from a file resource. It + allows developers to work with file input in much the same way as they + would work with database input. A FieldSet is + conceptually very similar to a Jdbc ResultSet. + FieldSets only require one argument, a String + array of tokens. Optionally, you can also configure in the names of the + fields so that the fields may be accessed either by index or name as + patterned after ResultSet. In code it means it's + as simple as: - String[] tokens = new String[]{"foo", "1", "true"}; -FieldSet fs = new DefaultFieldSet(tokens); -String name = fs.readString(0); -int value = fs.readInt(1); -boolean booleanValue = fs.readBoolean(2); + String[] tokens = new String[]{"foo", "1", "true"}; + FieldSet fs = new DefaultFieldSet(tokens); + String name = fs.readString(0); + int value = fs.readInt(1); + boolean booleanValue = fs.readBoolean(2); - There are many more options on the FieldSet interface, such as - Date, long, BigDecimal, etc. The biggest advantage of the FieldSet is - that it provides consistent parsing of flat file input. Rather than each - batch job parsing differenty in potentially unexpected ways, it can be - consistent, both when erroring out due to a format exception, or when - doing simple data conversions. + There are many more options on the FieldSet + interface, such as Date, long, + BigDecimal, etc. The biggest advantage of the + FieldSet is that it provides consistent parsing + of flat file input. Rather than each batch job parsing differently in + potentially unexpected ways, it can be consistent, both when erroring + out due to a format exception, or when doing simple data + conversions.
@@ -195,31 +217,27 @@ boolean booleanValue = fs.readBoolean(2); reading from some type of file. A flat file is basically any type of file that contains at most two-dimensional (tabular) data. Reading flat files in the Spring Batch framework is facilitated by the class - FlatFileItemReader, which provides - basic functionality for reading and parsing flat files. In addition, - there are default implementations of the Skippable and ItemStream interfaces that solve the majority of + FlatFileItemReader, which provides basic + functionality for reading and parsing flat files. In addition, there are + default implementations of the ItemReader and + ItemStream interfaces that solve the majority of file processing needs. - The FlatFileItemReader class has - several properties. The three most important of these properties are - resource, fieldSetMapper and tokenizer, which define the resource from which - data will be read and the method by which the read data will be - converted to distinct fields. The fieldSetMapper and tokenizer interfaces will be explored more in the - next sections. In addition, we'll explore integration with the file - system via the resource property. The resource property represents a Spring Core - Resource. Documentation explaining how - to create beans of this type can be found in The FlatFileItemReader class has several + properties. The three most important of these properties are + Resource, FieldSetMapper + and LineTokenizer, which define the resource from + which data will be read and the method by which the read data will be + converted int distinct fields. The FieldSetMapper + and LineTokenizer interfaces will be explored + more in the next sections. In addition, we'll explore integration with + the file system via the resource property. The resource property + represents a Spring Core Resource. Documentation + explaining how to create beans of this type can be found in Spring Framework, Chapter 4.Resources. Therefore, this - guide will not go into the details of creating Resource objects except to make a couple of + guide will not go into the details of creating + Resource objects except to make a couple of points on the locating files to process within a batch environment. Tokenizers and field set mappers will be discussed a bit later. @@ -241,9 +259,10 @@ boolean booleanValue = fs.readBoolean(2); process of feeding the data into the pipe from this starting point. - The flat file reader uses a ResourceLineReader object to read from - the file. Optionally, you can specify a RecordSeparatorPolicy through the + The flat file reader uses a + ResourceLineReader object to read from the file. + Optionally, you can specify a + RecordSeparatorPolicy through the recordSeparatorPolicy property. This can be used to configure more low-level features, such as what constitutes the end of a line and whether to continue quoted strings over newlines, among other @@ -312,64 +331,71 @@ boolean booleanValue = fs.readBoolean(2);
FieldSetMapper - Field set mappers used by the FlatFileItemReader implement the - FieldSetMapper interface. This interface defines a single method, - mapLine, which takes a FieldSet object and maps its contents to some - Object. This object may be a custom DTO or domain object, or it could - be as simple as an array, depending on your needs. The FieldSetMapper is used in conjunction with the - tokenizer to translate a line of data from a resource into an object - of the desired type: + Field set mappers used by the + FlatFileItemReader implement the + FieldSetMapper interface. This interface + defines a single method, mapLine, which takes a FieldSet object and + maps its contents to some Object. This object may be a custom DTO or + domain object, or it could be as simple as an array, depending on your + needs. The FieldSetMapper is used in + conjunction with the LineTokenizer to translate + a line of data from a resource into an object of the desired + type: - public interface FieldSetMapper { + public interface FieldSetMapper { - public Object mapLine(FieldSet fs); + public Object mapLine(FieldSet fs); -} + } - As you can see, the pattern used is exatly the same as RowMapper - used by JdbcTemplate. + As you can see, the pattern used is exatly the same as + RowMapper used by + JdbcTemplate.
LineTokenizer Because there can be many formats of flat file data, which all - need to be converted to a FieldSet so that a FieldSetMapper can create - a useful domain object from them, an abstraction for turning a line of - input into a FieldSet is necessary. In Spring Batch, this is called - the LineTokenizer: + need to be converted to a FieldSet so that a + FieldSetMapper can create a useful domain + object from them, an abstraction for turning a line of input into a + FieldSet is necessary. In Spring Batch, this is + called a LineTokenizer: - public interface LineTokenizer { + public interface LineTokenizer { - FieldSet tokenize(String line); + FieldSet tokenize(String line); -} + } - The contract of a LineTokenizer is such that, given a line of - input (in theory the String could encompass more than one line) a - FieldSet representing the line will be returned. This will then be - based to a FieldSetMapper. Spring Batch contains the following - LineTokenizers: + The contract of a LineTokenizer is such + that, given a line of input (in theory the + String could encompass more than one line) a + FieldSet representing the line will be + returned. This will then be based to a + FieldSetMapper. Spring Batch contains the + following LineTokenizers: - DelmitedLineTokenizer - Used for files that separate records - by a delimiter. The most common is a comma, but pipes or - semicolons are often used as well + DelmitedLineTokenizer - Used for + files that separate records by a delimiter. The most common is a + comma, but pipes or semicolons are often used as well - FixedLengthTokenizer - Used for tokenizing files where each - record is separated by a 'fixed width' that must be defined per - record. + FixedLengthTokenizer - Used for + tokenizing files where each record is separated by a 'fixed width' + that must be defined per record. - PrefixMatchingCompositeLineTokenizer - Tokenizer that - determines which among a list of Tokenizers should be used on a - particular line by checking against a prefix. + PrefixMatchingCompositeLineTokenizer + - Tokenizer that determines which among a list of Tokenizers + should be used on a particular line by checking against a + prefix.
@@ -389,7 +415,8 @@ boolean booleanValue = fs.readBoolean(2); Pass the string line into the LineTokenizer#tokenize() - method, in order to retrieve a FieldSet + method, in order to retrieve a + FieldSet @@ -401,89 +428,90 @@ boolean booleanValue = fs.readBoolean(2); The following example will be used to illustrate this using an actual domain scenario. This particular batch job reads in football - players from the following file:ID,lastName,firstName,position,birthYear,debutYear -AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996 -AbduRa00,Abdullah,Rabih,rb,1975,1999 -AberWa00,Abercrombie,Walter,rb,1959,1982 -AbraDa00,Abramowicz,Danny,wr,1945,1967 -AdamBo00,Adams,Bob,te,1946,1969 -AdamCh00,Adams,Charlie,wr,1979,2003 + players from the following file: ID,lastName,firstName,position,birthYear,debutYear + "AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996", + "AbduRa00,Abdullah,Rabih,rb,1975,1999", + "AberWa00,Abercrombie,Walter,rb,1959,1982", + "AbraDa00,Abramowicz,Danny,wr,1945,1967", + "AdamBo00,Adams,Bob,te,1946,1969", + "AdamCh00,Adams,Charlie,wr,1979,2003" We want to map this data to the following Player domain object: - public class Player implements Serializable { + public class Player implements Serializable { - private String ID; - private String lastName; - private String firstName; - private String position; - private int birthYear; - private int debutYear; + private String ID; + private String lastName; + private String firstName; + private String position; + private int birthYear; + private int debutYear; - public String toString() { + public String toString() { - return "PLAYER:ID=" + ID + ",Last Name=" + lastName + - ",First Name=" + firstName + ",Position=" + position + - ",Birth Year=" + birthYear + ",DebutYear=" + - debutYear; - } - - // setters and getters... - } + return "PLAYER:ID=" + ID + ",Last Name=" + lastName + + ",First Name=" + firstName + ",Position=" + position + + ",Birth Year=" + birthYear + ",DebutYear=" + + debutYear; + } + + // setters and getters... + } - In order to map a FieldSet into our Player object, we need to - create a FieldSetMapper that returns players: + In order to map a FieldSet into our + Player object, we need to create a + FieldSetMapper that returns players: - protected static class PlayerFieldSetMapper implements FieldSetMapper { - public Object mapLine(FieldSet fieldSet) { - Player player = new Player(); + protected static class PlayerFieldSetMapper implements FieldSetMapper { + public Object mapLine(FieldSet fieldSet) { + Player player = new Player(); - player.setID(fieldSet.readString(0)); - player.setLastName(fieldSet.readString(1)); - player.setFirstName(fieldSet.readString(2)); - player.setPosition(fieldSet.readString(3)); - player.setBirthYear(fieldSet.readInt(4)); - player.setDebutYear(fieldSet.readInt(5)); + player.setID(fieldSet.readString(0)); + player.setLastName(fieldSet.readString(1)); + player.setFirstName(fieldSet.readString(2)); + player.setPosition(fieldSet.readString(3)); + player.setBirthYear(fieldSet.readInt(4)); + player.setDebutYear(fieldSet.readInt(5)); - return player; - } - } - + return player; + } + } We can then read in from the filed by correctly constructing our FlatFileItemReader and calling read(): - FlatFileItemReader itemReader = new FlatFileItemReader(); -itemReader.setResource = new FileSystemResource("resources/players.csv"); -//DelimitedLineTokenizer defaults to comma as it's delimiter -itemReader.setLineTokenizer(new DelimitedLineTokenizer()); -itemReader.setFieldSetMapper(new PlayerFieldSetMapper()); -itemReader.read(); + FlatFileItemReader itemReader = new FlatFileItemReader(); + itemReader.setResource = new FileSystemResource("resources/players.csv"); + //DelimitedLineTokenizer defaults to comma as it's delimiter + itemReader.setLineTokenizer(new DelimitedLineTokenizer()); + itemReader.setFieldSetMapper(new PlayerFieldSetMapper()); + itemReader.read(); - Each call to read will return a new Player object from each line - in the file. When the end of the file is reached, null will be - returned. + Each call to read will return a new + Player object from each line in the file. When the end of the file is + reached, null will be returned.
Mapping fields by name There is one additional functionality that is similar in - function to a JDBC ResultSet. The names of the fields can be injected - into the Tokenizer to increase the readability of the mapping - function. We can expose this behavior by adding the following. First, - we tell the tokenizer what the names of the fields in the fieldset - are: + function to a JDBC ResultSet. The names of the + fields can be injected into the LineTokenizer + to increase the readability of the mapping function. We can expose + this behavior by adding the following. First, we tell the + LineTokenizer what the names of the fields in + the fieldset are: - tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); + tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); - and provide a mapper that uses this information as - follows: + and provide a FieldSetMapper that uses + this information as follows: public class PlayerMapper implements FieldSetMapper { @@ -510,29 +538,32 @@ itemReader.read();
Automapping FieldSets to Domain Objects - For many, having to write a specific FieldSetMapper is equally - as cumbersome as writing a specific RowMapper for a JdbcTemplate. - Spring Batch makes this easier by providing a FieldSetMapper that - automatically maps fields by matching a field name with a setter using - the JavaBean spec. Again using the footbal example, the FieldSetMapper + For many, having to write a specific + FieldSetMapper is equally as cumbersome as + writing a specific RowMapper for a + JdbcTemplate. Spring Batch makes this easier by providing a + FieldSetMapper that automatically maps fields + by matching a field name with a setter using the JavaBean spec. Again + using the football example, the FieldSetMapper configuration looks like the following: - <bean id="fieldSetMapper" - class="org.springframework.batch.io.file.mapping.BeanWrapperFieldSetMapper"> - <property name="prototypeBeanName" value="player" /> -</bean> + <bean id="fieldSetMapper" + class="org.springframework.batch.io.file.mapping.BeanWrapperFieldSetMapper"> + <property name="prototypeBeanName" value="player" /> + </bean> -<bean id="person" - class="org.springframework.batch.sample.domain.Player" - scope="prototype" /> + <bean id="person" + class="org.springframework.batch.sample.domain.Player" + scope="prototype" /> - For each entry in the FieldSet, the mapper will look for a - corresponding setter on a new instance of the Player object (for this - reason, prototype scope is required) in the same way the Spring - container will look for setters matching a property name. Each - available field in the FieldSet will be mapped, and the resultant - Player object will be returned, only there was no code - required. + For each entry in the FieldSet, the + mapper will look for a corresponding setter on a new instance of the + Player object (for this reason, prototype scope + is required) in the same way the Spring container will look for + setters matching a property name. Each available field in the + FieldSet will be mapped, and the resultant + Player object will be returned, only there was + no code required.
@@ -541,13 +572,13 @@ itemReader.read(); So far only delimited files have been discussed in much detail, however, they respresent only half of the file reading picture. Many organizations that use flat files use fixed length formats. An example - field length file is below: + fixed length file is below: - UK21341EAH4121131.11customer1 -UK21341EAH4221232.11customer2 -UK21341EAH4321333.11customer3 -UK21341EAH4421434.11customer4 -UK21341EAH4521535.11customer5 + UK21341EAH4121131.11customer1 + UK21341EAH4221232.11customer2 + UK21341EAH4321333.11customer3 + UK21341EAH4421434.11customer4 + UK21341EAH4521535.11customer5 While this looks like one large field, it actually represent 4 distinct fields: @@ -573,19 +604,21 @@ UK21341EAH4521535.11customer5 - When configuring the FixedLengthLineTokenizer, each of these - lengths must be provided in the form of ranges: + When configuring the + FixedLengthLineTokenizer, each of these lengths + must be provided in the form of ranges: - <bean id="fixedLengthLineTokenizer" - class="org.springframework.batch.io.file.transform.FixedLengthTokenizer"> - <property name="names" value="ISIN, Quantity, Price, Customer" /> - <property name="columns" value="1-12, 13-15, 16-20, 21-29" /> -</bean> + <bean id="fixedLengthLineTokenizer" + class="org.springframework.batch.io.file.transform.FixedLengthTokenizer"> + <property name="names" value="ISIN, Quantity, Price, Customer" /> + <property name="columns" value="1-12, 13-15, 16-20, 21-29" /> + </bean> - This LineTokenizer will return the same FieldSet as if a - dlimiter had been used, allowing the same approachs above to be used - such as the BeanWrapperFieldSetMapper, in a way that is ignorant of - how the actual line was parsed. + This LineTokenizer will return the same + FieldSet as if a dlimiter had been used, + allowing the same approachs above to be used such as the + BeanWrapperFieldSetMapper, in a way that is + ignorant of how the actual line was parsed.
@@ -593,48 +626,49 @@ UK21341EAH4521535.11customer5 All of the file reading examples up to this point have all made a key assumption for simplicity's sake: one record equals one line. - However, this may not always be the case. It's very common that a file + However, this may not always be the case. Its very common that a file might have records spanning multiple lines with multiple formats. The following excerpt from a file illustrates this: - HEA;0013100345;2007-02-15 -NCU;Smith;Peter;;T;20014539;F -BAD;;Oak Street 31/A;;Small Town;00235;IL;US -SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States -BIN;VISA;VISA-12345678903 -LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47 -LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87 -SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS -FOT;2;2;267.34 + HEA;0013100345;2007-02-15 + NCU;Smith;Peter;;T;20014539;F + BAD;;Oak Street 31/A;;Small Town;00235;IL;US + SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States + BIN;VISA;VISA-12345678903 + LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47 + LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87 + SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS + FOT;2;2;267.34 Everything between the line starting with 'HEA' and the line starting with 'FOT' is considered one record. The PrefixMatchingCompositeLineTokenizer makes this easier by matching the prefix in a line with a particular tokenizer: - <bean id="orderFileDescriptor" - class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer"> - <property name="tokenizers"> - <map> - <entry key="HEA" value-ref="headerRecordDescriptor" /> - <entry key="FOT" value-ref="footerRecordDescriptor" /> - <entry key="BCU" value-ref="businessCustomerLineDescriptor" /> - <entry key="NCU" value-ref="customerLineDescriptor" /> - <entry key="BAD" value-ref="billingAddressLineDescriptor" /> - <entry key="SAD" value-ref="shippingAddressLineDescriptor" /> - <entry key="BIN" value-ref="billingLineDescriptor" /> - <entry key="SIN" value-ref="shippingLineDescriptor" /> - <entry key="LIT" value-ref="itemLineDescriptor" /> - <entry key="" value-ref="defaultLineDescriptor" /> - </map> - </property> -</bean> + <bean id="orderFileDescriptor" + class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer"> + <property name="tokenizers"> + <map> + <entry key="HEA" value-ref="headerRecordDescriptor" /> + <entry key="FOT" value-ref="footerRecordDescriptor" /> + <entry key="BCU" value-ref="businessCustomerLineDescriptor" /> + <entry key="NCU" value-ref="customerLineDescriptor" /> + <entry key="BAD" value-ref="billingAddressLineDescriptor" /> + <entry key="SAD" value-ref="shippingAddressLineDescriptor" /> + <entry key="BIN" value-ref="billingLineDescriptor" /> + <entry key="SIN" value-ref="shippingLineDescriptor" /> + <entry key="LIT" value-ref="itemLineDescriptor" /> + <entry key="" value-ref="defaultLineDescriptor" /> + </map> + </property> + </bean> This ensures that the line will be parsed correctly, which is especially important for fixed length input, with the correct field - names. Any users of the FlatFileItemReader in this scenario must - continue calling read() until the footer for the record is returned, - allowing them to return a complete order as one 'item'. + names. Any users of the FlatFileItemReader in + this scenario must continue calling read + until the footer for the record is returned, allowing them to return a + complete order as one 'item'.
@@ -643,73 +677,89 @@ FOT;2;2;267.34 Writing out to flat files has the same problems and issues that reading in from a file must overcome. It must be able to write out in - either dlimited or fixed length formats in a transactional - mannger. + either delimited or fixed length formats in a transactional + manger.
LineAggregator - Just like file reading's LineTokenizer interface is necessary to - take a string and split it into tokens, file writing must have a way - to aggregate multiple fields into a single string for writing to a - file. In Spring Batch this is the LineAggregator: + Just like file reading's LineTokenizer + interface is necessary to take a string and split it into tokens, file + writing must have a way to aggregate multiple fields into a single + string for writing to a file. In Spring Batch this is the + LineAggregator: - public interface LineAggregator { + public interface LineAggregator { - public String aggregate(FieldSet fieldSet); -} + public String aggregate(FieldSet fieldSet); + + } - The LineAggregator is exactly the opposite of a LineTokenizer. - LineTokenizer takes a string and returns a FieldSet, wheras - LineAggreator takes a FieldSet and returns a string. As with reading - there are two types: DelimitedLineAggregator and - FixedLengthLineAggregator. + The LineAggregator is exactly the + opposite of a LineTokenizer. + LineTokenizer takes a + String and returns a + FieldSet, whereas + LineAggregator takes a + FieldSet and returns a + String. As with reading there are two types: + DelimitedLineAggregator and + FixedLengthLineAggregator.
FieldSetCreator - Because the LineAggregator interface uses a FieldSet as it's - mechanism for converting to a string, there needs to be an interface - that describes how to convert from an object into a FieldSet: + Because the LineAggregator interface uses a + FieldSet as it's mechanism for converting to a + string, there needs to be an interface that describes how to convert + from an object into a FieldSet: - public interface FieldSetCreator { + public interface FieldSetCreator { - FieldSet mapItem(Object data); + FieldSet mapItem(Object data); -} + } - As with LineTokenizer and LineAggregator, FieldSetCreator is the - polar opposite of FieldSetMapper. FieldSetMapper takes a FieldSet and - returns a mapped object, whereas a FieldSetCreator takes an Object and - returns a FieldSet. + As with LineTokenizer and + LineAggregator, + FieldSetCreator is the polar opposite of + FieldSetMapper. + FieldSetMapper takes a + FieldSet and returns a mapped object, whereas a + FieldSetCreator takes an Object and returns a + FieldSet.
Simple Delimited File Writing Example - Now that both the LineAggregator and FieldSetCreator interfaces - have been defined, the basic flow of writing can be explained: + Now that both the LineAggregator and + FieldSetCreator interfaces have been defined, + the basic flow of writing can be explained: - The object to be written is passed to the FieldSetCreator in - order to obtain a FieldSet. + The object to be written is passed to the + FieldSetCreator in order to obtain a + FieldSet. - The returned FieldSet is passed to the LineAggregator + The returned FieldSet is passed to + the LineAggregator - The returned string is written to the configured - file. + The returned String is written to the + configured file. - The following excerpt from the FlatFileItemWriter expresses this - in code: + The following excerpt from the + FlatFileItemWriter expresses this in + code: public void write(Object data) throws Exception { FieldSet fieldSet = fieldSetCreator.mapItem(data); @@ -732,21 +782,23 @@ FOT;2;2;267.34
Handling file creation - FlatFileItemReader has a very simple relationship with file - resources. When the reader is initialized, it opens the file if it - exists, and throws an exception if it does not. File writing isn't - quite so simple. At first glance it seems like a similiar straight - forward contract should exist for FlatFileItemWriter, if the file - already exists, throw an exception, if it does not, create it and - start writing. Job restart throws a bit of a kink into this. In the - normal restart scenario, the contract is reversed, if the file exists - start writing to it from the last known good position, if it does not, - throw an exception. However, what happens if the file name for this - job is always the same? In this case, you would want to delete the - file if it exists, unless it's a restart. Because of this possibility, - the FlatFileItemWriter contains the property, shouldDeleteIfExists. - Setting this property to true will cause an existing file with the - same name to be deleted when the writer is opened. + FlatFileItemReader has a very simple + relationship with file resources. When the reader is initialized, it + opens the file if it exists, and throws an exception if it does not. + File writing isn't quite so simple. At first glance it seems like a + similar straight forward contract should exist for + FlatFileItemWriter, if the file already exists, + throw an exception, if it does not, create it and start writing. Job + restart throws a bit of a kink into this. In the normal restart + scenario, the contract is reversed, if the file exists start writing + to it from the last known good position, if it does not, throw an + exception. However, what happens if the file name for this job is + always the same? In this case, you would want to delete the file if it + exists, unless it's a restart. Because of this possibility, the + FlatFileItemWriter contains the property, + shouldDeleteIfExists. Setting this property + to true will cause an existing file with the same name to be deleted + when the writer is opened.
@@ -779,15 +831,11 @@ FOT;2;2;267.34 - + - + Figure X: XML Inputs @@ -805,14 +853,12 @@ FOT;2;2;267.34 - - @@ -826,9 +872,10 @@ FOT;2;2;267.34
StaxEventItemReader - The StaxEventItemReader configuration provides a typical setup for - the processing of records from an XML input stream. First, lets examine - a set of xml records that the StaxEventItemReader can process. + The StaxEventItemReader configuration + provides a typical setup for the processing of records from an XML input + stream. First, lets examine a set of XML records that the + StaxEventItemReader can process. <?xml version="1.0" encoding="UTF-8"?> @@ -869,9 +916,9 @@ FOT;2;2;267.34 - Fragment Deserializer - this is the UnMarshalling facility - provided by Spring OXM for mapping the XML fragment to an - object. + FragmentDeserializer - this is the + UnMarshalling facility provided by Spring OXM for mapping the XML + fragment to an object. @@ -893,13 +940,13 @@ FOT;2;2;267.34 Notice that in this example we have chosen to use an - XStreamMarshaller that requires an alias passed in as a map with the - first key and value being the name of the fragment (i.e. root element) - and the object type to bind. Then, similar to a FieldSet, the names of - the other elements that map to fields within the object type are - described as key/value pairs in the map. In the configuration file we - can use a spring configuration utility to describe the required alias as - follows: + XStreamMarshaller that requires an alias passed + in as a map with the first key and value being the name of the fragment + (i.e. root element) and the object type to bind. Then, similar to a + FieldSet, the names of the other elements that + map to fields within the object type are described as key/value pairs in + the map. In the configuration file we can use a spring configuration + utility to describe the required alias as follows: <util:map id="aliases"> @@ -916,8 +963,9 @@ FOT;2;2;267.34 new fragment is about to start (by matching the tag name by default). The reader creates a standalone XML document from the fragment (or at least makes it appear so) and passes the document to a deserializer - (typically a wrapper around a Spring WS Unmarshaller) to map the XML to - a Java object. + (typically a wrapper around a Spring OXM + Unmarshaller) to map the XML to a Java + object. In summary, if you were to see this in scripted code like Java the injection provided by the spring configuration would look something like @@ -957,14 +1005,16 @@ FOT;2;2;267.34
StaxEventItemWriter - Output works symetrically to input. The XMLItemWriter needs a - resource, a serializer, and a rootTagName. A java object is passed to a - serializer (typically a wrapper around Spring WS Marshaller) which - writes to output using a custom event writer that filters the - StartDocument and EndDocument events produced for each fragment by the - OXM tools. We'll show this in an example using the - MarshallingEventWriterSerializer. The Spring configuration for this - setup looks as follows: + Output works symmetrically to input. The + StaxEventItemWriter needs a + Resource, a serializer, and a rootTagName. A java + object is passed to a serializer (typically a wrapper around Spring OXM + Marshaller) which writes to output using a custom + event writer that filters the StartDocument and EndDocument events + produced for each fragment by the OXM tools. We'll show this in an + example using the + MarshallingEventWriterSerializer. The Spring + configuration for this setup looks as follows: <bean class="org.springframework.batch.item.xml.StaxEventItemWriter" id="tradeStaxWriter"> <property name="resource"value="file:target/test-outputs/20070918.testStream.xmlFileStep.output.xml" /> @@ -977,7 +1027,8 @@ FOT;2;2;267.34 The configuration sets up the three required properties and optionally sets the overwriteOutput=true, mentioned earlier in the chapter for specifying whether an existing file can be overwritten. The - TradeMarshallingSerializer is configured as follows: + TradeMarshallingSerializer is configured as + follows: <bean class="org.springframework.batch.item.xml.oxm.MarshallingEventWriterSerializer" id="tradeMarshallingSerializer"> <constructor-arg> @@ -1032,39 +1083,49 @@ FOT;2;2;267.34 Like most enterprise application styles, a database is the central storage mechanism for batch. However, batch differs from other application styles due to the sheer size of the datasets that must be worked with. The - Spring Core JdbcTemplate illustrates this problem well. If you use - JdbcTemplate with a RowMapper, the RowMapper will be called once for every - result returned from the provided query. This causes few issues in - scenarios where the dataset is small, but the large datasets often - necessary for batch processing would cause any JVM to crash quickly. If - the sql statement returns 1 million rows, the RowMapper will be called 1 - million times. Spring Batch provides two types of solutions for this - problem: Cursor and DrivingQuery ItemReaders. + Spring Core JdbcTemplate illustrates this problem + well. If you use JdbcTemplate with a + RowMapper, the RowMapper + will be called once for every result returned from the provided query. + This causes few issues in scenarios where the dataset is small, but the + large datasets often necessary for batch processing would cause any JVM to + crash quickly. If the sql statement returns 1 million rows, the + RowMapper will be called 1 million times. Spring + Batch provides two types of solutions for this problem: Cursor and + DrivingQuery ItemReaders.
Cursor Based ItemReaders Using a database cursor is generally the default approach of most batch developers. This is because it is the database's solution to the - problem of 'streaming' relational data. The Java ResultSet class is - essentially an object orientated mechanism for manipulating a cursor. A - ResultSet maintains a cursor to the current row of data. Calling next() - on a ResultSet moves this cursor to the next row. Spring Batch cursor - based ItemReaders open the a cursor on initialization, and move the - cursor forward one row for every call to read(), returning a mapped - object that can be used for processing. The close() method will then be - called to ensure all resources are freed up. The Spring core - JdbcTemplate gets around this problem by using the callback pattern to - completely map all rows in a ResultSet and close before returning - control back to the method caller. However, in batch this must wait - until the step is complete. Below is a generic diagram of how a cursor - based ItemReader works, and while a SQL statement is used as an example - since it is so widely known, any technolog could implement the basic - approach: + problem of 'streaming' relational data. The Java + ResultSet class is essentially an object + orientated mechanism for manipulating a cursor. A + ResultSet maintains a cursor to the current row + of data. Calling next on a + ResultSet moves this cursor to the next row. + Spring Batch cursor based ItemReaders open the a cursor on + initialization, and move the cursor forward one row for every call to + read, returning a mapped object that can be + used for processing. The close method will then + be called to ensure all resources are freed up. The Spring core + JdbcTemplate gets around this problem by using + the callback pattern to completely map all rows in a + ResultSet and close before returning control back + to the method caller. However, in batch this must wait until the step is + complete. Below is a generic diagram of how a cursor based + ItemReader works, and while a SQL statement is + used as an example since it is so widely known, any technology could + implement the basic approach: - - + + + + + + @@ -1078,10 +1139,12 @@ FOT;2;2;267.34
JdbcCursorItemReader - JdbcCursorItemReader is the jdbc implementation of the cursor - based technique. It works directly with a ResultSet and requires a SQL - statement to run against a connection obtained from a DataSource. The - following database schema will be used as an example: + JdbcCursorItemReader is the JDBC + implementation of the cursor based technique. It works directly with a + ResultSet and requires a SQL statement to run + against a connection obtained from a + DataSource. The following database schema will + be used as an example: CREATE TABLE CUSTOMER ( ID BIGINT IDENTITY PRIMARY KEY, @@ -1090,8 +1153,9 @@ FOT;2;2;267.34 ); Many people prefer to use a domain object for each row, so we'll - use an implementation of the RowMapper interface to map a - CustomerCredit object: + use an implementation of the RowMapper + interface to map a CustomerCredit + object: public class CustomerCreditRowMapper implements RowMapper { @@ -1111,23 +1175,26 @@ FOT;2;2;267.34 } - Because JdbcTemplate is so familiar to users of Spring, and the - JdbcCursorItemReader shares key interfaces with it, it's useful to see - an example of how to read in this data with JdbcTemplate, in order to - contrast it with the item reader. For the purposes of this example, + Because JdbcTemplate is so familiar to + users of Spring, and the JdbcCursorItemReader + shares key interfaces with it, it's useful to see an example of how to + read in this data with JdbcTemplate, in order + to contrast it with the item reader. For the purposes of this example, let's assume there are 1,000 rows in the CUSTOMER database. The first - example will be using JdbcTemplate: + example will be using JdbcTemplate: //For simplicity sake, assume a dataSource has already been obtained JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource); List customerCredits = jdbcTemplate.query("SELECT ID, NAME, CREDIT from CUSTOMER", new CustomerCreditRowMapper()); After running this code snippet the customerCredits list will - contain 1,000 CustomerCredit objects. In the query method, a - connection will be obtained from the DataSource, the provided SQL will - be run against it, and RowMapper.mapRow() will be called for each row - in the ResultSet. Let's constrast this with the approach of the - JdbcCursorItemReader: + contain 1,000 CustomerCredit objects. In the + query method, a connection will be obtained from the + DataSource, the provided SQL will be run + against it, and the mapRow method will be + called for each row in the ResultSet. Let's + constrast this with the approach of the + JdbcCursorItemReader: JdbcCursorItemReader itemReader = new JdbcCursorItemReader(); itemReader.setDataSource(dataSource); @@ -1145,20 +1212,22 @@ itemReader.close(executionContext); After running this code snippet the counter will equal 1,000. If the code above had put the returned customerCredit into a list, the - result would have been exactly the same as with the JdbcTemplate - example. However, the big advantage of the item reader is that it - allows items to be 'streamed'. The read() method can be called once, - and the item written out via an ItemWriter, and then the next item - obtained via read(). This allows item reading and writing to be done - in 'chunks' and committed periodically, which is the essence of high - performance batch processing. + result would have been exactly the same as with the + JdbcTemplate example. However, the big + advantage of the ItemReader is that it allows + items to be 'streamed'. The read method can + be called once, and the item written out via an + ItemWriter, and then the next item obtained via + read. This allows item reading and writing to + be done in 'chunks' and committed periodically, which is the essence + of high performance batch processing.
Additional Properties Because there are so many varying options for opening a cursor - in java, there are many properties on the JdbcCustorItemReader that - can be set: + in Java, there are many properties on the + JdbcCustorItemReader that can be set: JdbcCursorItemReader Properties @@ -1177,15 +1246,16 @@ itemReader.close(executionContext);Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are - needed by the ResultSet object used by the ItemReader. By - default, no hint is given. + needed by the ResultSet object used + by the ItemReader. By default, no hint is given. maxRows Sets the limits for the maximum number of rows the - underlying ResultSet can hold at any one time. + underlying ResultSet can hold at any + one time. @@ -1193,7 +1263,8 @@ itemReader.close(executionContext); Sets the number of seconds the driver will wait for a Statement object to execute to the given number of seconds. - If the limit is exceeded, a DataAccessEception is thrown. + If the limit is exceeded, a + DataAccessEception is thrown. (consult your driver vendor documentation for details). @@ -1201,12 +1272,14 @@ itemReader.close(executionContext); verifyCursorPosition - Because the same ResultSet held by the ItemReader is - passed to the RowMapper, it's possible for users to call - ResultSet.next() themselves, which could cause issues with - the reader's internal count. Settings this value to true - will cause an exception to be thrown if the cursor position - is not the same after the RowMapper call as it was + Because the same ResultSet + held by the ItemReader is passed to the + RowMapper, it's possible for users to + call ResultSet.next() themselves, which could cause issues + with the reader's internal count. Settings this value to + true will cause an exception to be thrown if the cursor + position is not the same after the + RowMapper call as it was before. @@ -1229,22 +1302,27 @@ itemReader.close(executionContext);Just as normal Spring users make important decisions about whether or not to use ORM solutions, which affects whether or not they - use a JdbcTemplate or a HibernateTemplate, Spring Batch users have the - same options. HibernateCursorItemReader is the Hibernate - implementation of the cursor technique. Hibernate's usage in batch has - been fairly controversial. This has largely been because hibernate was - originally developed to support online application styles. However, - that doesn't mean it can't be used for batch processing. The easiest - approach for solving this problem is to use a StatelessSession rather - than a standard session. This removes all of the caching and dirty - checking hibernate employs that can cause issues when using it in a - batch scenario. For more information on the differences between - Stateless and normal hibernate sessions, refer to the documentation of - your specific hibernate release. The HibernateCursorItemReader allows - you to declare an HQL statement and pass in a sessionFactory, which - will pass back one item per call to read() in the same basic fashion - as the JdbcCursorItemReader. Below is an example configuration using - the same 'customer credit' example as the jdbc reader: + use a JdbcTemplate or a + HibernateTemplate, Spring Batch users have the + same options. HibernateCursorItemReader is the + Hibernate implementation of the cursor technique. Hibernate's usage in + batch has been fairly controversial. This has largely been because + hibernate was originally developed to support online application + styles. However, that doesn't mean it can't be used for batch + processing. The easiest approach for solving this problem is to use a + StatelessSession rather than a standard + session. This removes all of the caching and dirty checking hibernate + employs that can cause issues when using it in a batch scenario. For + more information on the differences between stateless and normal + hibernate sessions, refer to the documentation of your specific + hibernate release. The + HibernateCursorItemReader allows you to declare + an HQL statement and pass in a SessionFactory, + which will pass back one item per call to + read in the same basic fashion as the + JdbcCursorItemReader. Below is an example + configuration using the same 'customer credit' example as the JDBC + reader: HibernateCursorItemReader itemReader = new HibernateCursorItemReader(); itemReader.setQueryString("from CustomerCredit"); @@ -1262,8 +1340,9 @@ itemReader.close(executionContext); itemReader.close(executionContext); - This configured ItemReader will return CustomerCredit objects in - the exact same manner as described by the JdbcCursorItemReader, + This configured ItemReader will return + CustomerCredit objects in the exact same manner + as described by the JdbcCursorItemReader, assuming hibernate mapping files have been created correctly for the Customer table. The 'useStatelessSession' property default to true, but has been added here to draw attention to the ability to switch it @@ -1286,8 +1365,12 @@ itemReader.close(executionContext); illustrates: - - + + + + + + @@ -1299,23 +1382,28 @@ itemReader.close(executionContext); Foo object: - - + + + + + + As you can see, an existing DAO can be used to obtain a full 'Foo' object using the key obtained from the driving query. In Spring Batch, - driving query style input is implemented with a DrivingQueryItemReader, - which has only one dependency: a KeyCollector + driving query style input is implemented with a + DrivingQueryItemReader, which has only one + dependency: a KeyCollector
KeyCollector As the previous example illustrates, the DrivingQueryItemReader is fairly simple. It simply iteratoes over a list of keys. However, - the real complication is how those keys are obtained. The KeyCollector - interface abstracts this: + the real complication is how those keys are obtained. The + KeyCollector interface abstracts this: public interface KeyCollector { @@ -1324,16 +1412,18 @@ itemReader.close(executionContext); void updateContext(Object key, ExecutionContext executionContext); } - The primary method in this interface is the retrieveKeys() - method. It is expected that this method will return the keys to be - processed regardless of whether or not it is a restart scenario. For - example, if a job starts processing keys 1 through 1,000, and fails - after processing key 500, upon restarting keys 500 through 1,000 - should be returned. This functionality is made possible by the - saveState method, which saves the provided key (which should be the - current key being processed) in the provided ExecutionContext. The - retrieveKeys method can then use this value to retrieve a subset of - the original keys: + The primary method in this interface is the + retrieveKeys method. It is expected that this + method will return the keys to be processed regardless of whether or + not it is a restart scenario. For example, if a job starts processing + keys 1 through 1,000, and fails after processing key 500, upon + restarting keys 500 through 1,000 should be returned. This + functionality is made possible by the + saveState method, which saves the provided + key (which should be the current key being processed) in the provided + ExecutionContext. The + retrieveKeys method can then use this value + to retrieve a subset of the original keys: ExecutionContext executionContext = new ExecutionContext(); List keys = keyStrategy.retrieveKeys(executionContext); @@ -1342,11 +1432,14 @@ itemReader.close(executionContext); keys = keyStrategy.retrieveKeys(executionContext); //keys should now contains 500 through 1,000 - This generalization illustrates the KeyCollector contract. If we - assume that initially calling retrieveKeys returned 1,000 keys (1 - through 1,000), calling updateContext() with key 500 should mean that - calling retrieveKeys again with the same execution context will return - 500 keys (501 through 1,000). + This generalization illustrates the + KeyCollector contract. If we assume that + initially calling retrieveKeys returned 1,000 + keys (1 through 1,000), calling updateContext + with key 500 should mean that calling + retrieveKeys again with the same + ExecutionContext will return 500 keys (501 + through 1,000).
@@ -1354,8 +1447,8 @@ itemReader.close(executionContext); The most common driving query scenario is that of a input that has only one column that represents it's key. This is implemented as - the SingleColumnJdbcKeyCollector class, which has the following - options: + the SingleColumnJdbcKeyCollector class, which + has the following options:
SinglecolumnJdbcKeyCollector properties @@ -1398,7 +1491,7 @@ itemReader.close(executionContext);
The following code helps illustrate how to setup and use a - SingleColumnJdbcKeyCollector: + SingleColumnJdbcKeyCollector: SingleColumnJdbcKeyCollector keyCollector = new SingleColumnJdbcKeyCollector(getJdbcTemplate(), "SELECT ID from T_FOOS order by ID"); @@ -1452,11 +1545,13 @@ itemReader.close(executionContext); keyStrategy.updateContext(new Long(3), executionContext); This tells the key collector to update the provided - ExecutionContext with the key of three. This will normally be called - by the DrivingQueryItemReader, but is called directly for simplicities - sake. By calling retrieveKeys with the ExecutionContext that was - updated to contain 3, the argument of 3 will be passed to the - restartSql: + ExecutionContext with the key of three. This + will normally be called by the + DrivingQueryItemReader, but is called directly + for simplicities sake. By calling + retrieveKeys with the + ExecutionContext that was updated to contain 3, + the argument of 3 will be passed to the restartSql: keyCollector.setRestartSql("SELECT ID from T_FOOS where ID > ? order by ID"); @@ -1467,20 +1562,23 @@ itemReader.close(executionContext);
Mapping multiple column keys - The SingleColumnJdbcKeyCollector is extremely useful for - generating keys, but only if one column uniquely identifies your - record. What if more than one column is required to be able to - uniquely identify your record? This should be a minority scenario, but - it is still possible. In this case, the MultipleColumnJdbcKeyCollector - should be used. It allows for mapping multiple columns by sacrificing - simplicity. The properties needed to use the multiple column collector - are the same as the single column version except one difference: - instead of a regular RowMaper, an ExecutionContextRowMapper must be - provided. Just like the single column version, it requires a normal - sql statement and a restart sql statement. However, because the - restart sql statement will require more than one argument, there needs - to be more complex handling of how keys are mapped to an execution - context. An ExecutionContextRowMapper provides this: + The SingleColumnJdbcKeyCollector is + extremely useful for generating keys, but only if one column uniquely + identifies your record. What if more than one column is required to be + able to uniquely identify your record? This should be a minority + scenario, but it is still possible. In this case, the + MultipleColumnJdbcKeyCollector should be used. + It allows for mapping multiple columns by sacrificing simplicity. The + properties needed to use the multiple column collector are the same as + the single column version except one difference: instead of a regular + RowMaper, an + ExecutionContextRowMapper must be provided. + Just like the single column version, it requires a normal SQL + statement and a restart SQL statement. However, because the restart + SQL statement will require more than one argument, there needs to be + more complex handling of how keys are mapped to an execution context. + An ExecutionContextRowMapper provides + this: public interface ExecutionContextRowMapper extends RowMapper { @@ -1490,15 +1588,18 @@ itemReader.close(executionContext); } - The ExecutionContextRowMapper interface extends the standard - RowMapper interface to allow for multiple keys to be stored in an - ExecutionContext, and a PreparedStatementSetter be created so that - arguments to a the restart sql statement can be set for the key + The ExecutionContextRowMapper interface + extends the standard RowMapper interface to + allow for multiple keys to be stored in an + ExecutionContext, and a + PreparedStatementSetter be created so that + arguments to a the restart SQL statement can be set for the key returned. - By default a implementation of the ExecutionContextRowMapper - that uses a Map will be used. It is recommended that this - implementation not be overriden. However, if a specific type of key + By default a implementation of the + ExecutionContextRowMapper that uses a + Map will be used. It is recommended that this + implementation not be overridden. However, if a specific type of key needs to be returned, then a new implementation can be provided.
@@ -1508,9 +1609,10 @@ itemReader.close(executionContext); Jdbc is not the only option available for key collectors, iBatis can be used as well. The usage of iBatis doesn't change the basic - requirements of a KeyCollector: query, restart query, and dataSource. - However, because iBatis is used, both queries are simply iBatis query - ids, and the data source is a SqlMapClient. + requirements of a KeyCollector: query, restart + query, and DataSource. However, because iBatis + is used, both queries are simply iBatis query ids, and the data source + is a SqlMapClient.
@@ -1519,43 +1621,46 @@ itemReader.close(executionContext); While both Flat Files and XML have specific ItemWriters, there is no exact equivalent in the database world. This is because transactions - give all the functionality that is needed. ItemWriters are necessary for - files because they must act like as if they're transactional, keeping + provide all the functionality that is needed. ItemWriters are necessary + for files because they must act as if they're transactional, keeping track of written items and flushing or clearing at the appropriate times. Databases have no need for this functionality, since the write is already contained in a transaction. Users can create their own DAO's - that implement the ItemWriter interface or use one from a custom - ItemWriter that's written for generic processing concerns, either way, - they should work without any issues. The one exception to this is - buffered output. This is most common when using hibernate as an - ItemWriter, but could have the same issues when using Jdbc batch mode. - Buffering database output doesn't have any inherent flaws, assuming - there are no errors in the data. However, any errors while writing out - can cause issues because there is no way to know which individual item - caused an exception. An example would be a record that causes a + that implement the ItemWriter interface or use + one from a custom ItemWriter that's written for + generic processing concerns, either way, they should work without any + issues. The one exception to this is buffered output. This is most + common when using hibernate as an ItemWriter, but + could have the same issues when using Jdbc batch mode. Buffering + database output doesn't have any inherent flaws, assuming there are no + errors in the data. However, any errors while writing out can cause + issues because there is no way to know which individual item caused an + exception. An example would be a record that causes a DataIntegrityViolationException, perhaps because of a primary key violation. If items are buffered before being written out, this error will not be thrown until the buffer is flushed just before a commit. For example, let's assume that 20 items will be written per chunk, and the - 15th item will have the DataIntegrityViolationException. As far as the - Step is concerned, all 20 item will be written out successfully, since + 15th item throws a DataIntegrityViolationException. As far as the Step + is concerned, all 20 item will be written out successfully, since there's no way to know that and error will occur until they are actually - written out. Once ItemWriter#flush() is called, the buffer will be - emptied and the exception will be hit. At this point, there's nothing - the Step can do, the transaction must be rolled back. Normally, this - exception will cause the Item to be skipped (depending upon the - skip/retry policies), and then it won't be written out again. However, - in this scenario, there's no way for it to know which item caused the - issue, the whole buffer was being written out when the failure happened. - Because this is a common enough use case, especially when using - Hibernate, Spring Batch provides a common implementation to help: - HibernateAwareItemWriter. The HibernateAwareItemWriter solves the - problem in a straightforward way: if a chunk fails the first time, on - subsequent runs it will be flushed and the transaction committed after - each itme. This effectively lowers the commit interval to one for the - length of the chunk. Doing so allows for items to be skipped reliably. - The following example illustrates how to configure the - HibernateAwareItemWriter: + written out. Once + ItemWriter#flush() is + called, the buffer will be emptied and the exception will be hit. At + this point, there's nothing the Step can do, the transaction must be + rolled back. Normally, this exception will cause the Item to be skipped + (depending upon the skip/retry policies), and then it won't be written + out again. However, in this scenario, there's no way for it to know + which item caused the issue, the whole buffer was being written out when + the failure happened. Because this is a common enough use case, + especially when using Hibernate, Spring Batch provides an implementation + to help: HibernateAwareItemWriter. The + HibernateAwareItemWriter solves the problem in a + straightforward way: if a chunk fails the first time, on subsequent runs + it will be flushed and the transaction committed after each time. This + effectively lowers the commit interval to one for the length of the + chunk. Doing so allows for items to be skipped reliably. The following + example illustrates how to configure the + HibernateAwareItemWriter: <bean id="hibernateItemWriter" class="org.springframework.batch.item.database.HibernateAwareItemWriter"> @@ -1582,14 +1687,16 @@ itemReader.close(executionContext); many users want to reuse existing DAOs or other services within their batch jobs. The Spring container itself makes this fairly easy by allowing any necessary class to be injected. However, there may be cases where the - existing service needs to act as an ItemReader or ItemWriter, either to - satisfy the dependency of another Spring Batch class, or because it truly - is the main ItemReader for a step. It's fairly trivial to write an adaptor - class for each service that needs wrapping, but because it's such a common - concern, Spring Batch provides implementations: ItemReaderAdapter and - ItemWriterAdapter. Both classes implement the standard Spring method - invoking delegator pattern and are fairly simple to set up. Below is an - example of the reader: + existing service needs to act as an ItemReader or + ItemWriter, either to satisfy the dependency of + another Spring Batch class, or because it truly is the main + ItemReader for a step. It's fairly trivial to write + an adaptor class for each service that needs wrapping, but because it's + such a common concern, Spring Batch provides implementations: + ItemReaderAdapter and + ItemWriterAdapter. Both classes implement the + standard Spring method invoking delegator pattern and are fairly simple to + set up. Below is an example of the reader: <bean id="itemReader" class="org.springframework.batch.item.adapter.ItemReaderAdapter"> <property name="targetObject" ref="fooService" /> @@ -1599,12 +1706,14 @@ itemReader.close(executionContext); <bean id="fooService" class="org.springframework.batch.item.sample.FooService" /> One important point to note is that the contract of the targetMethod - must be the same as the contract for read(). That is, when exhausted it - will return null, otherwise an Object. Anything else will prevent the - framework from correctly knowing when processing should end, either - causing an infinite loop or incorrect failure, depending upon the - implementation of the ItemWriter. The ItemWriter implementation is equally - as simple: + must be the same as the contract for read. That + is, when exhausted it will return null, otherwise an + Object. Anything else will prevent the framework + from correctly knowing when processing should end, either causing an + infinite loop or incorrect failure, depending upon the implementation of + the ItemWriter. The + ItemWriter implementation is equally as + simple: <bean id="itemWriter" class="org.springframework.batch.item.adapter.ItemWriterAdapter"> <property name="targetObject" ref="fooService" /> @@ -1620,19 +1729,20 @@ itemReader.close(executionContext); During the course of this chapter, multiple approaches to parsing input have been discussed. Each major implementation will throw exception - if it is not 'well-formed'. The FixedLengthTokenizer will throw an - exception if a range of data is missing. Similarly, attempting to access - an index in a RowMapper of FieldSetMapper that doesn't exist or is in a - different format than the one expected will cause an exception to be - thrown. All of these types of exceptions will be thrown before - ItemReader#read() returns. However, they don't address the issue of - whether or not the returned item is valid. For example, if one of the - fields is an age, it obviously cannot be negative. It will parse - correctly, because it existed and is a number, but it won't cause an - exception. Since there are already a plethora of Validation frameworks, - Spring Batch does not attempt to provide yet another, but rather provides - a very simple interface that can be implemented by any number of - frameworks: + if it is not 'well-formed'. The + FixedLengthTokenizer will throw an exception if a + range of data is missing. Similarly, attempting to access an index in a + RowMapper of FieldSetMapper + that doesn't exist or is in a different format than the one expected will + cause an exception to be thrown. All of these types of exceptions will be + thrown before read returns. However, they don't + address the issue of whether or not the returned item is valid. For + example, if one of the fields is an age, it obviously cannot be negative. + It will parse correctly, because it existed and is a number, but it won't + cause an exception. Since there are already a plethora of Validation + frameworks, Spring Batch does not attempt to provide yet another, but + rather provides a very simple interface that can be implemented by any + number of frameworks: public interface Validator { @@ -1640,10 +1750,11 @@ itemReader.close(executionContext); } - The contract is that the validate() method will throw an exception - if the object is invalid, and return normally if it is valid. Spring Batch - provides an out-of-the box ItemReader that delegates to another ItemReader - and validates the returned item: + The contract is that the validate method + will throw an exception if the object is invalid, and return normally if + it is valid. Spring Batch provides an out of the box + ItemReader that delegates to another + ItemReader and validates the returned item: <bean class="org.springframework.batch.item.validator.ValidatingItemReader"> <property name="itemReader"> @@ -1672,9 +1783,10 @@ itemReader.close(executionContext); </bean> - This simple example shows a simple ValangValidator that is used to - validate an order object. The intent is not to show Valang funtionality as - much as to show how a validator could be added. + This simple example shows a simple + ValangValidator that is used to validate an order + object. The intent is not to show Valang functionality as much as to show + how a validator could be added.
@@ -1686,17 +1798,20 @@ itemReader.close(executionContext); discussed. However, these are all fairly generic, and there are many potential scenarios that may not be covered by out of the box implementations. This section will show, using a simple example, how to - create a custom ItemReader and ItemWriter implementation and implement - their contracts correctly. Each one will also implement ItemStream, in - order to illustrate how to make a reader or writer restartable. + create a custom ItemReader and + ItemWriter implementation and implement their + contracts correctly. The ItemReader will also + implement ItemStream, in order to illustrate how to + make a reader or writer restartable.
Custom Restartable ItemReader Example - For the purpose of this example, a simple ItemReader - implementation that reads from a provided list will be created. We'll - start out by implementing the most basic contract of ItemReader, - read(): + For the purpose of this example, a simple + ItemReader implementation that reads from a + provided list will be created. We'll start out by implementing the most + basic contract of ItemReader, + read: public class CustomItemReader implements ItemReader{ @@ -1722,8 +1837,8 @@ itemReader.close(executionContext); This very simple class takes a list of items, and returns one at a time, removing it from the list. When the list empty, it returns null, - thus satisfying the most basic requirements of an ItemReader, as - illustrated below: + thus satisfying the most basic requirements of an + ItemReader, as illustrated below: List items = new ArrayList(); items.add("1"); @@ -1736,14 +1851,15 @@ itemReader.close(executionContext); assertEquals("3", itemReader.read()); assertNull(itemReader.read()); - This most basic ItemReader will work, but what happens if the - transaction needs to be rolled back? This will usually caused by an - error in the ItemWriter, since the ItmReader generally won't do anything - that invalidates the transaction, but without supporting it, there would - be erroneous results. ItemReaders are notified about rollbacks via the - mark() and reset() methods. In the example above they're empty, but - we'll need to add code to them in order to support the rollback - scenario: + This most basic ItemReader will work, but + what happens if the transaction needs to be rolled back? This will + usually caused by an error in the ItemWriter, since the ItmReader + generally won't do anything that invalidates the transaction, but + without supporting it, there would be erroneous results. ItemReaders are + notified about rollbacks via the mark and + reset methods. In the example above they're + empty, but we'll need to add code to them in order to support the + rollback scenario: public class CustomItemReader implements ItemReader{ @@ -1773,11 +1889,12 @@ itemReader.close(executionContext); }; } - The CustomItemReader has now been modified to keep track of where - it is currently, and where it was when mark() was last called. This - allows the new ItemReader to fulfill the basic contract that calling - reset() returns the ItemReader to the state it was in when mark() was - last called: + The CustomItemReader has now been modified + to keep track of where it is currently, and where it was when mark() was + last called. This allows the new ItemReader to + fulfill the basic contract that calling reset + returns the ItemReader to the state it was in + when mark was last called: //Assume same setup as last example, a list with "1", "2", and "3" itemReader.mark(); @@ -1788,22 +1905,28 @@ itemReader.close(executionContext); In most real world scenarios, there will likely be some kind of underlying resource that will require tracking. In the case of a file, - mark() will hold the current location within the file, and reset will - move it back. The JdbcCursorItemReader, for example, holds on to the - current row number, and on reset moves the cursor back by calling - ResultSet#absolute(int), which moves the current cursor to the row - number supplied. The CustomItemReader now completely adheres to the - entire ItemReader contract. Read will return the appropriates items, - returning null when empty, and reset() returns the ItemReader back to - it's state as of the last call to mark(), allowing for correct support - of a rollback. (It's assumed a Step implementation will call mark() and - reset()) The final challenge now is to make the ItemReader restartable. - Currently, if the power goes out, and processing begins again, the - ItemReader must start at the beginning. This is actually valid in many - scenarios, but due to the large datasets often used in batch, it's - generally preferable that a batch job starts off at where it left off. - In Spring Batch, this is implemented with the ItemStream - interface: + mark will hold the current location within the + file, and reset will move it back. The + JdbcCursorItemReader, for example, holds on to + the current row number, and on reset moves the cursor back by calling + the ResultSet absolute + method, which moves the current cursor to the row number supplied. The + CustomItemReader now completely adheres to the + entire ItemReader contract. + read will return the appropriates items, + returning null when empty, and reset returns + the ItemReader back to it's state as of the last + call to mark, allowing for correct support of a + rollback. (It's assumed a Step implementation + will call mark and + reset) The final challenge now is to make the + ItemReader restartable. Currently, if the power + goes out, and processing begins again, the + ItemReader must start at the beginning. This is + actually valid in many scenarios, but due to the large datasets often + used in batch, it's generally preferable that a batch job starts off at + where it left off. In Spring Batch, this is implemented with the + ItemStream interface: public class CustomItemReader implements ItemReader, ItemStream{ @@ -1850,12 +1973,15 @@ itemReader.close(executionContext); public void close(ExecutionContext executionContext) throws ItemStreamException {} } - On each call to ItemStream#update(), the current index of the - ItemReader will be stored in the provided ExecutionContext with a key of - 'current.index'. When ItemStream#open() is called, the ExecutionContext - is checked to see if it contains an entry with that key, and if so the - current index is moved to that location. This is a fairly trivial - example, but it still meets the general contract: + On each call to ItemStream + update method, the current index of the + ItemReader will be stored in the provided + ExecutionContext with a key of 'current.index'. + When the ItemStream open + method is called, the ExecutionContext is checked + to see if it contains an entry with that key, and if so the current + index is moved to that location. This is a fairly trivial example, but + it still meets the general contract: ExecutionContext executionContext = new ExecutionContext(); ((ItemStream)itemReader).open(executionContext); @@ -1872,28 +1998,31 @@ itemReader.close(executionContext); assertEquals("2", itemReader.read()); Most ItemReaders have much more sophisticated restart logic. The - DrivingQueryItemReader, for example, only loads up the remaining keys to - be processed, rather than loading all of them and then moving to the - correct index. It's also worth noting that the key used within the - ExecutionContext should not be trivial. That is because the same - ExecutionContext is used for all ItemStreams within a Step. In most - cases, simply prepending the key with the class name should be enough to - garuntee uniqueness. However, in the rare cases where two of the same - type of ItemStream are used in the same step (which can happen if two - files are need for output) then a more unique name will be needed. For - this reason, many of the Spring Batch ItemReader and ItemWriters have a - setName() property that allows this key name to be overriden. + DrivingQueryItemReader, for example, only loads + up the remaining keys to be processed, rather than loading all of them + and then moving to the correct index. It's also worth noting that the + key used within the ExecutionContext should not + be trivial. That is because the same + ExecutionContext is used for all ItemStreams + within a Step. In most cases, simply prepending + the key with the class name should be enough to guarantee uniqueness. + However, in the rare cases where two of the same type of + ItemStream are used in the same step (which can + happen if two files are need for output) then a more unique name will be + needed. For this reason, many of the Spring Batch ItemReader and + ItemWriters have a setName() property that allows this key name to be + overridden.
Custom ItemWriter Example - Implementing a Custom ItemWriter is similar in many ways to the - ItemReader example above, but differs in enough ways as to warrant its - own example. However, adding restartability is essentially the same, so - it won't be covered in this example. As with the ItemReader example, a - List will be used in order to keep the example as simple as - possible: + Implementing a Custom ItemWriter is similar + in many ways to the ItemReader example above, but + differs in enough ways as to warrant its own example. However, adding + restartability is essentially the same, so it won't be covered in this + example. As with the ItemReader example, a List + will be used in order to keep the example as simple as possible: public class CustomItemWriter implements ItemWriter{ @@ -1909,15 +2038,18 @@ itemReader.close(executionContext); } The example is extremely simple, but it's worth showing to - illustrate an ItemWriter that doesn't respond to rollbacks and commits - (i.e. clear() and flush()). If your potential writer is such that it - doesn't need to care about rollback or commit, likely because it's - writing to a database, then there is little value to the ItemWriter - interface in that scenario other than using it to meet another class's - requirement for an implementation of the ItemWriter interface. In that - case, the ItemWriterAdapter would be a better solution. However, if it - does need to be transactional, then flush() and clear() should be - implemented to allow for a buffering solution: + illustrate an ItemWriter that doesn't respond to + rollbacks and commits (i.e. clear and + flush). If your potential writer is such that + it doesn't need to care about rollback or commit, likely because it's + writing to a database, then there is little value to the + ItemWriter interface in that scenario other than + using it to meet another class's requirement for an implementation of + the ItemWriter interface. In that case, the + ItemWriterAdapter would be a better solution. + However, if it does need to be transactional, then + flush and clear should + be implemented to allow for a buffering solution: public class CustomItemWriter implements ItemWriter{ @@ -1939,10 +2071,12 @@ itemReader.close(executionContext); } } - The ItemWriter buffers all output, only writing to the actual - output (in this case by added to a list) when ItemWriter#flush() is - called. The contents of the buffer are thrown away when - ItemStream#clear() is called. + The ItemWriter buffers all output, only + writing to the actual output (in this case by added to a list) when the + ItemWriter flush method + is called. The contents of the buffer are thrown away when + ItemStream clear is + called.
\ No newline at end of file diff --git a/docs/src/site/docbook/reference/spring-batch-intro.xml b/docs/src/site/docbook/reference/spring-batch-intro.xml index 971175f35..45d063ccf 100644 --- a/docs/src/site/docbook/reference/spring-batch-intro.xml +++ b/docs/src/site/docbook/reference/spring-batch-intro.xml @@ -14,7 +14,7 @@ processed without user interaction. These operations typically include time based events (e.g. month-end calculations, notices or correspondence), periodic application of complex business rules processed - repetitively across very large data sets (e.g. insurance benefit + repetitively across very large data sets (e.g. Insurance benefit determination or rate adjustments), or the integration of information that is received from internal and external systems that typically requires formatting, validation and processing in a transactional manner into the @@ -29,7 +29,7 @@ making it easy for developers to access and leverage more advance enterprise services when necessary. Spring Batch is not a scheduling framework. There are many good enterprise schedulers available in both the - commerical and open source spaces such as Quartz, Tivoli, Control-M, etc. + commercial and open source spaces such as Quartz, Tivoli, Control-M, etc. It is intended to work in conjunction with a scheduler, not replace a scheduler. @@ -42,80 +42,6 @@ complex, high-volume batch jobs can leverage the framework in a highly scalable manner to process significant volumes of information. -
- Spring Batch Architecture - - Spring Batch is designed with extensibility and a diverse group of - end users in mind. The figure below shows a sketch of the layered - architecture that supports the extensibility and ease of use for - end-user developers. - - - - - - - - - Figure 1.1: Batch Execution - Environments - -
- -
- Supporting Batch Execution Environments - - Spring Batch Architecture showing potential execution environment - implementations support different platforms and end-user goals from the - same blocks of business logic in the Application Layer. The initial - release provides an Infrastructure layer in the form of low level tools. - There is also a simple batch execution environment with sample jobs, - using the infrastructure in its implementation. The batch execution - environment provides robust features for traceability and management of - the batch lifecycle. A key goal is that the management of the batch - process (locating a job and its input, starting, scheduling, restarting, - and finally processing to created results) should be as easy as possible - for developers. - - The Infrastructure provides the ability to batch operations - together, and to retry an piece of work if there is an exception. Both - requirements have a transactional flavour, and similar concepts are - relevant (propagation, synchronisation). They also both lend themselves - to the template programming model common in Spring, c.f. - TransactionTemplate, JdbcTemplate, - JmsTemplate. - - The Simple Batch Execution environment is the first execution - environment available. It provides a robust set of integrated features - including logging/tracing, transaction management, job processing - statistics, job restart, skip, and resource management to enable the - management of the full lifecycle of traditional batch processing. A - number of sample jobs are packaged with this execution environment and - are described in detail to more clearly articulate usage and - capabilities of the execution environment. - - The runtime dependencies of infrastructure, core and execution are - shown in the figure below. - - - - - - - - - Figure 1.2: Runtime Dependencies - -
-
Background @@ -156,110 +82,129 @@ agencies desiring to deliver standard, proven solutions to their enterprise IT environments will benefit from Spring Batch.
-
-
- Usage Scenarios +
+ Usage Scenarios - Spring Batch provides a technical framework and programming model to - support long-running processes that perform a given set of tasks - repetitively. A typical batch program generally reads a large number of - records from a database, file, or queue, processes the data in some - fashion, and then writes back data in a modified form. Spring Batch - automates this basic batch iteration, providing the capability to process - similar transactions as a set, typically in an offline environment without - any user interaction. Batch jobs are part of most IT projects and Spring - Batch is the only open source framework that provides a robust, - enterprise-scale solution. Batch processing is an application style for - many enterprise data processing pipelines (e.g. payment and settlement - systems), and the lack of a standard architecture has led many projects to - create their own custom architecture at significant development and - maintenance costs. + A typical batch program generally reads a large number of records + from a database, file, or queue, processes the data in some fashion, and + then writes back data in a modified form. Spring Batch automates this + basic batch iteration, providing the capability to process similar + transactions as a set, typically in an offline environment without any + user interaction. Batch jobs are part of most IT projects and Spring + Batch is the only open source framework that provides a robust, + enterprise-scale solution. - Business Scenarios - - Commit batch process periodically - + Business Scenarios + + Commit batch process periodically + - - Concurrent batch processing: parallel processing of a - job - + + Concurrent batch processing: parallel processing of a + job + - - Staged, enterprise message-driven processing - + + Staged, enterprise message-driven processing + - - Massively parallel batch processing - + + Massively parallel batch processing + - - Manual or scheduled restart after failure - + + Manual or scheduled restart after failure + - - Sequential processing of dependent steps (with extensions to - workflow-driven batches) - + + Sequential processing of dependent steps (with extensions to + workflow-driven batches) + - - Partial processing: skip records (e.g. on rollback) - + + Partial processing: skip records (e.g. on rollback) + - - Whole-batch transaction: for cases with a simple enough data - model or a small batch size - - + + Whole-batch transaction: for cases with a small batch size + or existing stored procedures/scripts + + - Technical Objectives - - Batch developers use the Spring programming model: concentrate - on business logic; let the framework take care of - infrastructure. - + Technical Objectives + + Batch developers use the Spring programming model: + concentrate on business logic; let the framework take care of + infrastructure. + - - Clear separation of concerns between the infrastructure, the - batch execution environment, and the batch application. - + + Clear separation of concerns between the infrastructure, the + batch execution environment, and the batch application. + - - Provide common, core execution services as interfaces that all - projects can implement. - + + Provide common, core execution services as interfaces that + all projects can implement. + - - Provide simple and default implementations of the core - execution interfaces that can be used ‘out of the box’. - + + Provide simple and default implementations of the core + execution interfaces that can be used ‘out of the box’. + - - Easy to configure, customize, and extend services, by - leveraging the spring framework in all layers. - + + Easy to configure, customize, and extend services, by + leveraging the spring framework in all layers. + - - All existing execution environment services should be easy to - replace or extend, without any impact to the infrastructure - layer. - + + All existing core services should be easy to replace or + extend, without any impact to the infrastructure layer. + - - Provide a simple deployment model, with the architecture JARs - completely separate from the application, built using Maven. - - -
+ + Provide a simple deployment model, with the architecture + JARs completely separate from the application, built using + Maven. + + +
-
- How To Get Started +
+ Spring Batch Architecture - There are a number of sample applications that can be used to get - started with Spring Batch. They can be found in the samples project. They - are executed either from the command line or as unit tests. See Chapter 6: Practical Examples for Spring Batch - as a starting point. + Spring Batch is designed with extensibility and a diverse group of + end users in mind. The figure below shows a sketch of the layered + architecture that supports the extensibility and ease of use for + end-user developers. + + + + + + + + + Figure 1.1: Spring Batch Layered + Architecture + + + This layered architecture highlights three major high level + components: Application, Core, and Infrastructure. The application + contains all batch jobs and custom code written by developers using + Spring Batch. The Batch Core contains the core runtime classes necessary + to launch and control a batch job. It includes things such as a + JobLauncher, Job, and + Step implementations. Both Application and Core + are built on top of a common infrastructure. This infrastructure + contains common readers and writers, and services such as the + RetryTemplate, which are used both by application + developers(readers and writers) and the core framework itself. + (retry) +
\ No newline at end of file diff --git a/docs/src/site/resources/reference/images/cursorExample.png b/docs/src/site/resources/reference/images/cursorExample.png new file mode 100755 index 000000000..757d5b639 Binary files /dev/null and b/docs/src/site/resources/reference/images/cursorExample.png differ diff --git a/docs/src/site/resources/reference/images/drivingQueryExample.png b/docs/src/site/resources/reference/images/drivingQueryExample.png new file mode 100755 index 000000000..adabc0a54 Binary files /dev/null and b/docs/src/site/resources/reference/images/drivingQueryExample.png differ diff --git a/docs/src/site/resources/reference/images/drivingQueryJob.png b/docs/src/site/resources/reference/images/drivingQueryJob.png new file mode 100755 index 000000000..230a7157f Binary files /dev/null and b/docs/src/site/resources/reference/images/drivingQueryJob.png differ