From eb744db9f4cb3d1d9fe856cda8c595446ef23d6b Mon Sep 17 00:00:00 2001 From: lucasward Date: Thu, 6 Mar 2008 05:45:00 +0000 Subject: [PATCH] Continued cleanup of Reference Documentation. Repeat and Retry have been separated into their own chapters, Infrastructure is now just ItemReaders and ItemWriters. Added a lot of extra initial documentation about the expected contract of ItemReader, ItemWriter, and ItemStream. --- docs/src/site/docbook/reference/core.xml | 119 ++- docs/src/site/docbook/reference/glossary.xml | 11 + docs/src/site/docbook/reference/index.xml | 8 +- .../site/docbook/reference/infrastructure.xml | 802 ------------------ .../docbook/reference/readersAndWriters.xml | 615 ++++++++++++++ docs/src/site/docbook/reference/repeat.xml | 76 ++ docs/src/site/docbook/reference/retry.xml | 12 + 7 files changed, 794 insertions(+), 849 deletions(-) delete mode 100644 docs/src/site/docbook/reference/infrastructure.xml create mode 100644 docs/src/site/docbook/reference/readersAndWriters.xml create mode 100644 docs/src/site/docbook/reference/repeat.xml create mode 100644 docs/src/site/docbook/reference/retry.xml diff --git a/docs/src/site/docbook/reference/core.xml b/docs/src/site/docbook/reference/core.xml index 9ba6f0999..eea52b948 100644 --- a/docs/src/site/docbook/reference/core.xml +++ b/docs/src/site/docbook/reference/core.xml @@ -263,8 +263,17 @@
Job Parameters - Job Parameters represent parameters to a job that - + Job Parameters represent parameters to a job that are used to + either uniquely identify an instance, or to help drive processing. One + example of identifying a job would be 'Schedule Date'. This is a common + batch use case where an 'effective date' for the data is used. For + example, you may be processing on thursday for a scheduled date of + Monday. Once the JobInstance with a schedule date of monday has + completed successfully, a schedule date of Tuesday will be used for the + next run. You will then be given a new JobInstance by the framework, + since your parameters are different. This allows you to effectively + control how you define a 'JobInstance', since you control what + parameters are passed in.
@@ -288,11 +297,11 @@ Job executions are represented by objects of the JobExecution class. These job executions are created by an implementation of the JobRepository interface from a given - JobInstance corresponding to a unique - JobParameters object. Each job - execution contains a reference to its corresponding job instance and - related Step Executions. + role="bold">JobRepository interface from a given JobInstance corresponding to a unique JobParameters object. Each job execution contains + a reference to its corresponding job instance and related Step + Executions.
@@ -302,24 +311,25 @@ This section describes stereotypes relating to the concept of a batch step. A step is an entity that encapsulates a single, independent phase of a batch job. Therefore, every batch job is composed entirely of - one or more batch steps. Steps should be thought of as unique processing streams that - will be executed in sequence. For example, if you have one step that loads a file into a database, - another that reads from the database, validates the data, preforms processing, and then - writes to another table, and another that reads from that table and writes out to a file. Each - of these steps will be performed completely before moving on to the next step. The file will - be completely read into the database before step 2 can begin. + one or more batch steps. Steps should be thought of as unique processing + streams that will be executed in sequence. For example, if you have one + step that loads a file into a database, another that reads from the + database, validates the data, preforms processing, and then writes to + another table, and another that reads from that table and writes out to a + file. Each of these steps will be performed completely before moving on to + the next step. The file will be completely read into the database before + step 2 can begin.
Step - A batch step contains all of the information necessary to define - a discrete set of business logic within a job. This is a - necessarily vague description because the contents of any given step - are at the discretion of the developer writing jobs. A - step can be as narrowly defined as a single line of code or as broadly - defined as necessary to complete the entire work of your job. There are - several factors that will affect the breadth of your step - configurations. + A batch step contains all of the information necessary to define a + discrete set of business logic within a job. This is a necessarily vague + description because the contents of any given step are at the discretion + of the developer writing jobs. A step can be as narrowly defined as a + single line of code or as broadly defined as necessary to complete the + entire work of your job. There are several factors that will affect the + breadth of your step configurations. @@ -340,15 +350,14 @@ - Steps are defined by instantiating implementations - of the Step interface. Two step implementation - classes are available in the Spring Batch - framework, and they are each discussed in detail in other sections of - this guide. For most situations, the ItemOrientedStep implementation is sufficient, but - custom control flow behavior and transaction management behavior can - also be configured by using a RepeatOperationsStep. + Steps are defined by instantiating implementations of the + Step interface. Two step implementation + classes are available in the Spring Batch framework, and they are each + discussed in detail in other sections of this guide. For most + situations, the ItemOrientedStep + implementation is sufficient, but custom control flow behavior and + transaction management behavior can also be configured by using a + RepeatOperationsStep.
@@ -367,14 +376,13 @@ Step executions are represented by objects of the StepExecution class. These step executions are - created by an implementation of the Job interface from a given JobExecution. Each step execution contains a - reference to its corresponding step and job execution, and - transaction related data such as commit and rollback count and start and - end times. Additionally, each step execution will contain a set of - execution attributes, which will contain statistics and restart - data. + created by an implementation of the Job + interface from a given JobExecution. + Each step execution contains a reference to its corresponding step and + job execution, and transaction related data such as commit and rollback + count and start and end times. Additionally, each step execution will + contain a set of execution attributes, which will contain statistics and + restart data.
@@ -387,14 +395,14 @@ is atomic in terms of transactions. A transaction will never commit until an entire tasklet execution is complete (unless an exception occurs - a transaction might either commit or rollback if that behavior - is specified in the step's exception management strategy). There is a + is specified in the step's exception management strategy). There is a specific implementation of the Step interface, TaskletStep, that works directly with a Tasklet.
- Processing Stereotypes + Item-Oriented Processing Stereotypes A powerful batch processing paradigm implemented by the Spring Batch framework is the concept of item-oriented processing. That is, doing work @@ -413,7 +421,7 @@ framework's dependency injection capabilities.
- Item Reader + Item Readers An item reader is an object that is used to retrieve the inputs for a step, one at a time. When the item reader has exhausted the @@ -424,7 +432,7 @@
- Item Writer + Item Writers/Processors An item writer is an object that is used to perform processing for a step, one item at a time. Generally, an item writer has no @@ -439,6 +447,33 @@
+
+ Support Stereotypes + + While item readers and writers serve as the main entry points for + item-oriented processing, they might be supplemented by a number of + support classes that perform specific tasks within the reader / writer + lifecycle. These support stereotypes are useful for dividing the work of + item readers and writers into reusable pieces, as well as abstracting away + the details of processing, such as interaction with external systems. + Additionally, they give us another opportunity to leverage the powerful + configuration features of the Spring framework, as we can switch between + several beans implementing these support interfaces without changing the + driving item reader or writer. + +
+ Item Transformers + + An item transformer is a class that is capable of taking an object + and changing it somehow before processing occurs. For instance, an item + transformer my alter an object by changing its properties or by + replacing it with another object entirely, such as a wrapper or + derivative object. It can also be defined as an adaptor, allowing an + object of one type to be converted for use as an object of a second + type. +
+
+
High Level Processing Flow diff --git a/docs/src/site/docbook/reference/glossary.xml b/docs/src/site/docbook/reference/glossary.xml index 2331816df..a528a40fa 100644 --- a/docs/src/site/docbook/reference/glossary.xml +++ b/docs/src/site/docbook/reference/glossary.xml @@ -103,6 +103,17 @@ + + Item + + + An item represents the smallest ammount of complete data for processing. In the most + simple terms this might mean a line in a file, a row in a database table, or a particular + element in an XML file. + + + + Logicial Unit of Work diff --git a/docs/src/site/docbook/reference/index.xml b/docs/src/site/docbook/reference/index.xml index 23d489573..48e864a7e 100644 --- a/docs/src/site/docbook/reference/index.xml +++ b/docs/src/site/docbook/reference/index.xml @@ -31,12 +31,10 @@ - + - - - - + + diff --git a/docs/src/site/docbook/reference/infrastructure.xml b/docs/src/site/docbook/reference/infrastructure.xml deleted file mode 100644 index ffb551c86..000000000 --- a/docs/src/site/docbook/reference/infrastructure.xml +++ /dev/null @@ -1,802 +0,0 @@ - - - - The Spring Batch Infrastructure - -
- Introduction to the - Spring Batch Infrastructure - - Spring Batch is a Pipe and Filters architecture. The Spring Batch - Infrastructure implements key services that enable a high volume of - throughput . These include: - - I/O infrastructure components - I/O Compoments are grouped - into a few simple concepts but provide a high degree of robustness - in record at a time processing. The key interfaces in this regard - are Item Readers and Item Writers. An ItemReader is responsible for providing data - to the pipe. An ItemWriter is a - basic interface for generic output operatrions. - - - - Validation- interface to support pluggable validation - strategies to ensure the integrity of the input source items or, in - other words, object level validation. - - - - RepeatTemplates- the Repeat Template is responsible for - repeatedly invoking an operation on the Input Provider pulling input - items from an input source until there are no more items to be - processed. - - - - RetryTemplates - a mechanism for attempting to reprocess an - input item that has thrown an exception. - - - - Support for Statistics - an interface that dependent projects - can use to implement application specific statistics. - - - - Transaction semantics for batch - support facilities for - giving transaction extensions used by the batch architecture. - - - - - - The Batch Lifecycle is simple. Data comes in one side of the pipe. - It is then parsed, validated and transformed and handed off for business - logic processing. That processing can be as simple as loading records into - a database or as complicated as supporting batch job styles of generating - reports, conversion, pdf generation, generation of high volume print - formats, etc. Spring Batch provides a framework for simplifying the - handling of input and output resources so that developers can concentrate - on what needs to happen during the processing steps. -
- -
- Item Readers - - Although a simple concept, ItemReaders are the means for providing data from - many different types of input sources. The table provide below list an - inventory of ItemReaders currently available. In addition it is easy to - create a custom ItemReader. A list of the available ItemReaders can be - found in Appendix A. - - The Item Reader is a basic interface for generic input operations. - Subclasses implementing this interface will be responsible for reading - records from an item stream and also possibly for mapping these records to - objects. Generally it is the responsibility of implementing class to - decide which technology to use for mapping and how it should be - configured. A picture of the I/O hierarhcy is helpful in understanding - their place within the spring batch infrastructure. - - - - A description of ItemReaders and ItemWriters types that spring batch - supports are roughly divided into four areas of which one is for - simplifying the use of processing Items in memory: - - - - List Item Reader - allows for processing in memory collections - of Items. The ListItemReader is useful for testing but would - probably not be used in typical batch scenarios - - - - File Item Readers- File Item Readers read lines of data from a - flat file that typically describe records with fields of data - defined by fixed positions in the file or delimited by some special - character (e.g. a comma). There is a line tokenizer associated with - input sources and a line aggregator associated with the output - source. - - - - XML - an XML input and output sources process XML - independently of technologies used for parsing, mapping and - validating objects. Input data allows for the validation of and XML - file against and XSD schema. The input template provides for - restart, skip, statistics and transaction features by implementing - the corresponding interfaces. - - - - SQL - a database resource accessed that returns resultsets - that can be mapped to objects for processing. The default SQL Input - Sources invoke a RowMapper to return objects, keep track of the - current row if restart is required, basic statistics, and some - transaction enhancements that will be explained later. - - - - JMS - An ItemReader for JMS uses a JmsTemplate. The template - should have a default destination, which will be used to provide - items in read(). If a recovery step is needed, set the error - destination and the item will be sent there if processing fails in - an external retry. - - - - An ItemReader interface is a very - simple abstraction for defining the set of inputs that a step in a batch - job will use. The next sections will describe how to create custom Item - Readers and what the basic properties and behaviors of Item - Readers. - -
- List Item Readers and Common Custom Item - Reader Behavior - - The ListItemReader, as mentioned - above, is useful for testing and probably not too useful as something - used in typical batch processing. One instructive use is to see how - narrow the responsiblity of ItemReaders are. They simply provide a - method that allows us to continue reading items until the items are - exhausted much like an iterator. In addition,, it is expected that - projects will create custom Item Readers. As a means of illustrating the - standard properties and behaviors of other framework-provided - ItemReaders like mapping unstructured items into objects through the use - of tokenizing we will extend the ListItemReader to supporting mapping. - The ItemReader interface defines a single method called read(). The read() method returns the next object to be - provided, much like an iterator. The definition of this method will - contain the logic that decides what object to return, performs any - object construction or other work that needs to occur, and finally - returns the object. We inherit this behavior from ListItemReader. We - will add two methods, - setFieldSetMapper(), to enable the mapping behavior and - setTokenizer(), to enabling parsing of - List Items. It this example the items in the list are a simple array of - delimited strings.. - - Here is our custom list item Reader that supplies mapping or - binding behavior as follows: - protected static class ListPlayerReader extends ListItemReader { - private FieldSetMapper fieldSetMapper; - private LineTokenizer tokenizer = null; - - public ListPlayerReader(List list) { - super(list); - } - - public void setFieldSetMapper(FieldSetMapper fieldSetMapper) { - this.fieldSetMapper = fieldSetMapper; - } - - public void setTokenizer(LineTokenizer tokenizer) { - this.tokenizer = tokenizer; - } - - - } - - - We will tag it as an Player Reader for reasons you'll see next as - we map Player objects from input strings. In this example we have - inherited the read() behavior that allows us to read from a List in - memory and provided a way to map arbitrary streams into objects and - added the ability to map FieldSets to objects. We will see how to take - advantage of this next. - -
- Understanding Field Set Mappers - - A FieldSet is Spring Batch’s abstraction for enabling the - binding of fields from a file data source. It allows developers to - work with file input in much the same way as they would work with - database input. A FieldSet is conceptually very similar to a Jdbc - Result Set. FieldSets only require one argument, a list of tokens. - Optionally you can also configure in the names of the fields so that - the fields may be accessed either by index or name as patterned after - the JdbcResultSet. In code it means it's as simple as: - - Field set mappers used by the flat file reader classes implement - the FieldSetMapper interface. This interface defines a single method, - mapLine, which takes a FieldSet object and maps its contents to some - Object. This object may be a custom DTO or domain object, or it could - be as simple as an array, depending on your needs. The field set - mapper is used in conjunction with the tokenizer to translate a line - of data from a resource into an object of the desired type. - - For example, suppose our file or list consists of players has - the following fields and the start of the data looks like the - following: - ID,lastName,firstName,position,birthYear,debutYear - "AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996", - "AbduRa00,Abdullah,Rabih,rb,1975,1999", - "AberWa00,Abercrombie,Walter,rb,1959,1982", - "AbraDa00,Abramowicz,Danny,wr,1945,1967", - "AdamBo00,Adams,Bob,te,1946,1969", - "AdamCh00,Adams,Charlie,wr,1979,2003" - - - We want to map this data to the following Player object: - - public class Player implements Serializable { - - private String ID; - private String lastName; - private String firstName; - private String position; - private int birthYear; - private int debutYear; - - public String toString() { - - return "PLAYER:ID=" + ID + ",Last Name=" + lastName + - ",First Name=" + firstName + ",Position=" + position + - ",Birth Year=" + birthYear + ",DebutYear=" + - debutYear; - } - - // setters and getters... - } - - - We can now inject a fieldset mapper into the ListPlayerReader, - for example, that can take advantage of a PlayerFieldSetMapper for - transforming a line that consists of one item separated by delimiters - into a domain object - Player in this - case. We inject programmatically by invoking the following: - - itemReader.setFieldSetMapper(fieldSetMapper); - - and define the fieldSetMapper class in the following - declaration: - - - protected static class PlayerFieldSetMapper implements FieldSetMapper { - public Object mapLine(FieldSet fieldSet) { - Player player = new Player(); - - player.setID(fieldSet.readString(0)); - player.setLastName(fieldSet.readString(1)); - player.setFirstName(fieldSet.readString(2)); - player.setPosition(fieldSet.readString(3)); - player.setBirthYear(fieldSet.readInt(4)); - player.setDebutYear(fieldSet.readInt(5)); - - return player; - } - } - - - There is one additional preference that can be used that is - similar in function to the jdbc fieldset. The names of the fields can - be injected into the Tokenizer to increase the readability of the - mapping function. We can expose this behavior by adding the following. - First, we tell the tokenizer what the names of the fields in the - fieldset are: - - - tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); - - - and provide a mapper that uses this information as - follows: - - - public class PlayerMapper implements FieldSetMapper { - public Object mapLine(FieldSet fs) { - - if(fs == null){ - return null; - } - - Player player = new Player(); - player.setID(fs.readString("ID")); - player.setLastName(fs.readString("lastName")); - player.setFirstName(fs.readString("firstName")); - player.setPosition(fs.readString("position")); - player.setDebutYear(fs.readInt("debutYear")); - player.setBirthYear(fs.readInt("birthYear")); - - return player; - } - - } - -
- -
- Configuring and Using - LineTokenizers - - We have already mentioned tokenizers in the preceding section. - In order to separate the structure of the raw records, LineTokenizer, - or one of it subclasses, is used to parse data obtained from the an - ItemReader, most typically a file. Flat File Item Readers, as - mentioned above, typically process records in two forms, fixed and - delimited. A fixed length input record is where the fields are - assigned fixed locations within a line of a file. In the preceding - section we gave an example of a comma delimited record layout as - follows: - - - AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996 - AbduRa00,Abdullah,Rabih,rb,1975,1999 - AberWa00,Abercrombie,Walter,rb,1959,1982 - AbraDa00,Abramowicz,Danny,wr,1945,1967 - AdamBo00,Adams,Bob,te,1946,1969 - AdamCh00,Adams,Charlie,wr,1979,2003 - - - Here, rather than fix fields to positions within a line or - record, fields are simply separated by some predefined symbol. We - illustrate with commas in this example as it is a very familiar format - to developers who have experience with csv file formats exported from - spreadsheets. - - On the other hand a fixed record format might look like the - following: - - - 12345678901234567890123456789012345678901234567890 - AbduKa00Abdul-Jabbar Karim rb19741996 - AbduRa00Abdullah Rabih rb19751999 - AberWa00Abercrombie Walter rb19591982 - AbraDa00Abramowicz Danny wr19451967 - AdamBo00Adams Bob te19461969 - AdamCh00Adams Charlie wr19792003 - - - One can see that each field in the record starts at the same - position in the record. It's fixed from a starting position to an end - of position for each field, which may include a user defined end - position such as EOL. - - Neither of these formats are particularly self describing but - are still very much in use in flat file exchanges between system - interfaces. Both formats share in common the requirement to read in a - line of data (a String) and parse it into tokens that can be mapped to - an object (or objects) to be passed to the ItemProcessor. As you can - see, there are two required dependencies of the input source; the - first is a resource to read in, which is the file to process. The - second dependency is a LineTokenizer, which will be discused - below. - - The interface for a LineTokenizer is very simple, given a - string; it will return a FieldSet that wraps the results from - tokenizing the provided string. The tokens are created through a - LineTokenizer and a - FieldSetMapper is used to map the - FieldSet to an object. The - FieldSet is configured as a property for an Item - Provider, which wraps an Input Source. Programatically you can inject - the tokenizer as demonstrated previously on the ItemReader via the - method call toYou can see this in the following example: - - And, as you can see, the field names will get passed in the the - mapper. The actual mapping provided by the developer would then look - as simple as: - - itemReader.setTokenizer(tokenizer); -
-
- -
- Flat File Item Readers - - One of the most common tasks performed in batch jobs involve - reading from some type of file. A flat file is basically any type of - file that contains at most two-dimensional (tabular) data. Flat files - include several formats of file. Some common types of flat files one - might encounter are: - - - - Fixed Width Files - - - - Delimited Files (e.g.Fixed Width Files Comma-Separated - Values). - - - - (XML data files will be discussed separately). Reading flat files - in the Spring Batch framework is facilitated by the class FlatFileItemReader, which provides basic - functionality for reading and parsing flat files. In addition, there are - default implementations of the Skippable and - ItemStream interfaces that solve the majority of file - processing needs. - -
- Flat File Item Reader - Properties - - The FlatFileItemReader reader - class has several properties. The three most important of these - properties are resource, fieldSetMapper and tokenizer, which define the resource from which - data will be read and the method by which the read data will be - converted to distinct fields. We explored fieldSetMapper and tokenizer while reviewing how to create a - custom ItemReader. We'll revisit - these properties in light of how we use them with the FlatFileItemReader. In addition, we'll explore - integration with the file system via the resource property. The - resource property represents a Spring - Core Resource. Documentation - explaining how to create beans of this type can be found in Spring - Framework, Chapter 4.Resources. Therefore, this - guide will not go into the details of creating Resource objects except to make a couple of - points on the locating files to process within a batch environment. - Tokenizers and field set mappers will be discussed a bit later. - - As mentioned, the location of the file is defined by the - resource property. There are only a few methods exposed through a - resource service. A resource is used to help locate, open, and close - resources. It can be as simple as: - Resource resource = new FileSystemResource("resources/trades.csv"); - - - In complex batch environments the directory structures are often - managed by the EAI infrastructure where drop zones for external - interfaces are established for moving files from ftp locations to - batch processing locations and vice versa. File moving utilities are - beyond the scope of the spring batch architecture but it is not - unusual for batch job streams to include file moving utilities as - steps in the job stream. It's sufficient to know that the batch - architecture only needs to know how to locate the files to be - processed. Spring Batch begins the process of feeding the data into - the pipe from this starting point. - - The flat file reader uses a ResourceLineReader object to read - from the file. Optionally, you can specify a RecordSeparatorPolicy through property - recordSeparatorPolicy. This can be used to configure more low-level - features, such as what constitutes the end of a line and whether to - continue quoted strings over newlines, among other things. - - The other properties in the flat file readers allow you to - further specify how your data will be interpreted: - Flat File Item Reader Properties - - - - - - - Property - - Type - - Description - - - - - - encoding - - String - - Specifies what text encoding to use - - default is "ISO-8859-1" - - - - comments - - String[] - - Specifies line prefixes that indicate - comment rows - - - - linesToSkip - - int - - Number of lines to ignore at the top of - the file - - - - firstLineIsHeader - - boolean - - Indicates that the first line of the - file is a header containing field names. If the column names - have not been set yet and the tokenizer extends - AbstractLineTokenizer, field names will be set automatically - from this line - - - -
-
- - -
-
- -
- XML Item Readers and Writers - - Spring Batch provides transactional infrastructure for both reading - XML records and mapping them to Java objects as well as writing Java - objects as XML records. - - - Constraints on streaming XML - - StAX API is used for I/O as other standard XML APIs do not fit - batch processing requirements (DOM loads the whole input into memory at - once and SAX controls the parsing process allowing the user only to - provide callbacks). - - - Spring Batch is not tied to any particular OXM technology. Typical - use is to delegate OXM - to Spring WS, which provides uniform abstraction for - the most popular OXM technologies. However dependency on Spring WS is - optional and you can choose to implement Spring Batch specific interfaces - if desired. The relationship to the technologies that OXM supports can be - shown as the following: - - - - - - - - - - - Figure X: OXM Binding - - - Lets take a closer look how XML input and output work in batch. It - is assumed the XML resource is a collection of 'fragments' corresponding - to individual records. Note that OXM tools are designed to work with - standalone XML documents rather than XML fragments cut out of an XML - document, therefore the Spring Batch infrastructure needs to work around - this fact (as described below). - - On input the reader reads the XML resource until it recognizes a new - fragment is about to start (by matching the tag name by default). The - reader creates a standalone XML document from the fragment (or at least - makes it appear so) and passes the document to a deserializer (typically a - wrapper around Spring WS Unmarshaller) to map the XML to a Java - object. - - - - - - - - - - - Figure X: XML Inputs - - - Output works symetrically to input. Java object is passed to a - serializer (typically a wrapper around Spring WS Marshaller) which writes - to output using a custom event writer that filters the StartDocument and - EndDocument events produced for each fragment by the OXM tools. - - For example configuration of XML input and output see the sample - xmlStaxJob. //TODO inline the example once it is not subject to change + - show sample input file -
- -
- Item Writers - - The Item Writers are similar in functionality to the input source - with the exception that the operations are reversed. They still need to be - located, opened and closed but they differ in the case that we write to - output sources. In the case of databases or queues these may be inserts, - updates or sends. The format of the serialization of the output source is - specific for every batch job. -
- -
- SQL Sources - - SQL input sources can be configured for various reasons, for - example: - - - - a staging table for large volumes of sorted data that was loaded - from flat files - - - - the beginning of an outbound collection of data targeted for an - external flat file interface - - - - the target of a triggered event like "collect all cases that can - be automatically closed" - - Spring Batch supports two approaches for accessing a SQL Input - Source; 1) a cursor driven input source and 2) an indexed based Input - Query. The cursor driven input source is named because it utilizes a - jdbc cursor to stream over the SQL input source whereas an indexed - based input query is designed for easy division of the input into - ranges. - - - - -
- -
- Item Providers and Processors - - We finally arrive at the Item Provider, We've already alluded to - Item Providers in some of the code samples above. -
- -
- Validating Input - - -
- -
- Repeat Templates - - One of the most fundamental concepts in the batch architecture is - the Repeat Template. The Repeat Template is responsible for repeatedly - invoking an operation on the Input Provider pulling input items from an - input source until there are no more items to be processed. One - interesting analogy used by Dierk Koenig in the book "Groovy in Action" is - a boiler vs. a continuous-flow heater. In this analogy he illustrates how - XML parsers can typically be divided into those that read the entire input - before process begins like DOM Parsers vs. those that stream over the - input like SAX parsers. Spring Batch is a continuous-flow heater and uses - the RepeatTemplate as the mechanism to keep the hot water or input stream - in constant flow. - - Many times batch processes are not only working on non-transaction - input sources like files but the output is a transactional resource such - as a queue or database. A common scenario when a batch job is a datastream - coming from a flat file interface is to have a file or files as input - sources and a database resource as the output source. In this case the - repeat templates can be used like the following: - - In this batch scenario an outer RepeatTemplate initialies the - continuous flow, a TransactionTemplate wraps the input and output - resources and an inner RepeatTemplate manages the commit interval or - chunks of data to be processed. The Business Logic occurs in the input and - output of single items. Of course this is a simplistic view of how batch - really works. Input can be quite complex with multiple files and - complicated validation scenarios. Conversely, the output source can also - be quite complex in determining how the records will be stored in the - database. Spring Batch makes no assumptions about how simple or complex - the business processing is within the RepeatTemplates. It's only job is to - keep the flow moving from the Item Provider to the Item Processor as - quickly as possible. The repeat template can process records irrespective - of the batch architecture. A simple example would be: - - RepeatTemplate template = new RepeatTemplate(); - Resource resource = new FileSystemResource("resources/trades.csv"); - TradeProcessor executor = new TradeProcessor(); - TradeItemProvider provider = null; - try { - provider = new TradeItemProvider(resource); - } catch (Exception e) { - // TODO Auto-generated catch block - e.printStackTrace(); - } - template.iterate(new ItemProviderRepeatCallback(provider, executor)); - - - A RepeatTemplate has an exception policy that can be - leveraged -
- -
- Retry Template - - The retry template is used as a way to overcome failures in the - stream -
- diff --git a/docs/src/site/docbook/reference/readersAndWriters.xml b/docs/src/site/docbook/reference/readersAndWriters.xml new file mode 100644 index 000000000..b1a3907a5 --- /dev/null +++ b/docs/src/site/docbook/reference/readersAndWriters.xml @@ -0,0 +1,615 @@ + + + + ItemReaders and ItemWriters + +
+ Introduction + + All batch processing can be described in its most simple form as + reading in large ammounts of data, performing some type of calculation or + transformation, and writing the result back out. Spring Batch provides two + key interfaces to help perform bulk reading and writing: ItemReader and + ItemWriter +
+ +
+ ItemReader + + Although a simple concept, ItemReaders are the means for providing data from + many different types of input. The most general examples include: + + + Flat File- Flat File Item Readers read lines of data from a + flat file that typically describe records with fields of data + defined by fixed positions in the file or delimited by some special + character (e.g. comma). + + + + XML - XML ItemReaders process XML independently of + technologies used for parsing, mapping and validating objects. Input + data allows for the validation of and XML file against and XSD + schema. + + + + SQL - A database resource accessed that returns resultsets + that can be mapped to objects for processing. The default SQL Input + Sources invoke a RowMapper to return objects, keep track of the + current row if restart is required, basic statistics, and some + transaction enhancements that will be explained later. + + + + JMS - An ItemReader for JMS using JmsTemplate. The template + should have a default destination, which will be used to provide + items in read(). If a recovery step is needed, set the error + destination and the item will be sent there if processing fails in + an external retry. + + There are many more possbilities, but we'll focus on the + basic ones for this chapter. A complete list of all available ItemReaders + can be found in Appendix A. + + The Item Reader is a basic interface for generic input + operations: + + public interface ItemReader { + + Object read() throws Exception; + + void mark() throws MarkFailedException; + + void reset() throws ResetFailedException; +} + + + The read() method defines the most essential contract of the + ItemReader, calling it returns one Item, returning null if no more items + are left. An item might represent a line in a file, a row in a database, + or an element in an xml file. It is generally expected that these will be + mapped to a useable domain object (i.e. Trade or Foo, etc) but there is no + requirement in the contract to do so. + + mark() and reset() are important methods due to the transactional + nature of batch processing. Mark() will be called before reading begins. + Calling reset() at anytime will position the ItemReader to its position + when Mark() was last called. The semantics are very similar to + java.io.Reader. +
+ +
+ ItemWriter + + Item Writers are similar in functionality to an ItemReader with the + exception that the operations are reversed. They still need to be located, + opened and closed but they differ in the case that we write out, rather + than reading in. In the case of databases or queues these may be inserts, + updates or sends. The format of the serialization of the output source is + specific for every batch job. + + As with ItemReader, ItemWriter is a fairly generic interface: + + public interface ItemWriter { + + void write(Object item) throws Exception; + + void flush() throws FlushFailedException; + + void clear() throws ClearFailedException; +} + + + As with read() on ItemReader, write provides the basic contract of + ItemWriter, it will attempt to write out the item passed in as long as it + is open. As with mark() and reset(), flush() and clear() are necessary due + to the nature of batch processing. Because it is generally expected that + items will be 'batched' together into a chunk, and then output, it is + expected that an ItemWriter will perform some type of buffering. flush() + will empty the buffer by actually writing the items out, whereas clear + will simply throw the contents of the buffer away. In most cases, a Step + implementation will call flush() before a commit and clear() in case of + rollback. +
+ +
+ ItemStream + + Both ItemReaders and ItemWriters serve their individual purposes + well, but there is a common concern among both of them that necessitates + another interface. In general, as part of the scope of a batch job, + readers and writers need to be opened, closed, and require a mechanism for + persisting state: + + public interface ItemStream { + + void open(ExecutionContext executionContext) throws StreamException; + + void update(ExecutionContext executionContext); + + void close(ExecutionContext executionContext) throws StreamException; +} + + + Before describing each method, it's worth breifly mentioning the + ExecutionContext. An ExecutionContext is created for Each StepExecution to + allow users to store the state of a particular execution, with the + expectation that it will be returned if the same JobInstance is started + again. For those familiar with Quartz, the semantics are very similar to a + Quartz JobDataMap. Open() should be called before any calls to read or + write and is expected to open any resources such as files or obtain + connections. As mentioned before, if expected data is found in the + ExecutionContext, it may be used to start the ItemReader or ItemWriter at + a location other than its initial state. Converely, close will be called + to ensure any resources allocated during open will be released safely. + Update() is called primarily to ensure that any state currently being held + is loaded into the provided ExecutionContext. In most cases, this method + will be called before committing, to ensure that the current state is + persisted in the database before commit. +
+ +
+ List Item Readers and Common Custom Item + Reader Behavior + + The ListItemReader, as mentioned + above, is useful for testing and probably not too useful as something used + in typical batch processing. One instructive use is to see how narrow the + responsiblity of ItemReaders are. They simply provide a method that allows + us to continue reading items until the items are exhausted much like an + iterator. In addition,, it is expected that projects will create custom + Item Readers. As a means of illustrating the standard properties and + behaviors of other framework-provided ItemReaders like mapping + unstructured items into objects through the use of tokenizing we will + extend the ListItemReader to supporting mapping. The ItemReader interface + defines a single method called read(). + The read() method returns the next object + to be provided, much like an iterator. The definition of this method will + contain the logic that decides what object to return, performs any object + construction or other work that needs to occur, and finally returns the + object. We inherit this behavior from ListItemReader. We will add two + methods, setFieldSetMapper(), to enable + the mapping behavior and setTokenizer(), + to enabling parsing of List Items. It this example the items in the list + are a simple array of delimited strings.. + + Here is our custom list item Reader that supplies mapping or binding + behavior as follows: + protected static class ListPlayerReader extends ListItemReader { + private FieldSetMapper fieldSetMapper; + private LineTokenizer tokenizer = null; + + public ListPlayerReader(List list) { + super(list); + } + + public void setFieldSetMapper(FieldSetMapper fieldSetMapper) { + this.fieldSetMapper = fieldSetMapper; + } + + public void setTokenizer(LineTokenizer tokenizer) { + this.tokenizer = tokenizer; + } + + + } + + + We will tag it as an Player Reader for reasons you'll see next as we + map Player objects from input strings. In this example we have inherited + the read() behavior that allows us to read from a List in memory and + provided a way to map arbitrary streams into objects and added the ability + to map FieldSets to objects. We will see how to take advantage of this + next. +
+ +
+ Flat Files + + Since the beginning of batch processing, one of the most common + mechanisms for interchanging bulk data has been the flat file. Unlike XML, + which has an aggreed upon standard for defining how it is structured + (XSD), the person reading a flat file must understand ahead of time + exactly how the file is structured. In general, all flat files fall into + two general types: Delimited and Fixed Length. + +
+ The FieldSet + + When working with flat files in Spring Batch, regardless of + whether it is for input or output, one of the most important classes is + the FieldSet. Many architectures and libraries contain abstractions for + helping you read in from a file, but they usually return a String or + array of Strings. This really only gets you halfway there. A FieldSet is + Spring Batch’s abstraction for enabling the binding of fields from a + file resource. It allows developers to work with file input in much the + same way as they would work with database input. A FieldSet is + conceptually very similar to a Jdbc ResultSet. FieldSets only require + one argument, a String array of tokens. Optionally you can also + configure in the names of the fields so that the fields may be accessed + either by index or name as patterned after ResultSet. In code it means + it's as simple as: + + String[] tokens = new String[]{"foo", "1", "true"}; +FieldSet fs = new DefaultFieldSet(tokens); +String name = fs.readString(0); +int value = fs.readInt(1); +boolean booleanValue = fs.readBoolean(2); + + There are many more options on the FieldSet interface, such as + Date, long, BigDecimal, etc. The biggest advantage of the FieldSet is + that it provides consistent parsing of flat file input. Rather than each + batch job parsing differenty in potentially unexpected ways, it can be + consistent, both when erroring out due to a format exception, or when + doing simple data conversions. +
+ +
+ FlatFileItemReader + + One of the most common tasks performed in batch jobs involve + reading from some type of file. A flat file is basically any type of + file that contains at most two-dimensional (tabular) data. Reading flat + files in the Spring Batch framework is facilitated by the class + FlatFileItemReader, which provides + basic functionality for reading and parsing flat files. In addition, + there are default implementations of the Skippable and ItemStream interfaces that solve the majority of + file processing needs. + + The FlatFileItemReader class has + several properties. The three most important of these properties are + resource, fieldSetMapper and tokenizer, which define the resource from which + data will be read and the method by which the read data will be + converted to distinct fields. We explored fieldSetMapper and tokenizer while reviewing how to create a custom + ItemReader. We'll revisit these + properties in light of how we use them with the FlatFileItemReader. In addition, we'll explore + integration with the file system via the resource property. The + resource property represents a Spring + Core Resource. Documentation explaining + how to create beans of this type can be found in Spring + Framework, Chapter 4.Resources. Therefore, this + guide will not go into the details of creating Resource objects except to make a couple of + points on the locating files to process within a batch environment. + Tokenizers and field set mappers will be discussed a bit later. + + As mentioned, the location of the file is defined by the resource + property. There are only a few methods exposed through a resource + service. A resource is used to help locate, open, and close resources. + It can be as simple as: + Resource resource = new FileSystemResource("resources/trades.csv"); + + + In complex batch environments the directory structures are often + managed by the EAI infrastructure where drop zones for external + interfaces are established for moving files from ftp locations to batch + processing locations and vice versa. File moving utilities are beyond + the scope of the spring batch architecture but it is not unusual for + batch job streams to include file moving utilities as steps in the job + stream. It's sufficient to know that the batch architecture only needs + to know how to locate the files to be processed. Spring Batch begins the + process of feeding the data into the pipe from this starting + point. + + The flat file reader uses a ResourceLineReader object to read from + the file. Optionally, you can specify a RecordSeparatorPolicy through property + recordSeparatorPolicy. This can be used to configure more low-level + features, such as what constitutes the end of a line and whether to + continue quoted strings over newlines, among other things. + + The other properties in the flat file readers allow you to further + specify how your data will be interpreted: + Flat File Item Reader Properties + + + + + + + Property + + Type + + Description + + + + + + encoding + + String + + Specifies what text encoding to use - + default is "ISO-8859-1" + + + + comments + + String[] + + Specifies line prefixes that indicate + comment rows + + + + linesToSkip + + int + + Number of lines to ignore at the top of + the file + + + + firstLineIsHeader + + boolean + + Indicates that the first line of the file + is a header containing field names. If the column names have + not been set yet and the tokenizer extends + AbstractLineTokenizer, field names will be set automatically + from this line + + + +
+ +
+ The FieldSet + + A FieldSet is Spring Batch’s abstraction for enabling the + binding of fields from a file data source. It allows developers to + work with file input in much the same way as they would work with + database input. A FieldSet is conceptually very similar to a Jdbc + Result Set. FieldSets only require one argument, a list of tokens. + Optionally you can also configure in the names of the fields so that + the fields may be accessed either by index or name as patterned after + the JdbcResultSet. In code it means it's as simple as: + + Field set mappers used by the flat file reader classes implement + the FieldSetMapper interface. This interface defines a single method, + mapLine, which takes a FieldSet object and maps its contents to some + Object. This object may be a custom DTO or domain object, or it could + be as simple as an array, depending on your needs. The field set + mapper is used in conjunction with the tokenizer to translate a line + of data from a resource into an object of the desired type. + + For example, suppose our file or list consists of players has + the following fields and the start of the data looks like the + following: + ID,lastName,firstName,position,birthYear,debutYear + "AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996", + "AbduRa00,Abdullah,Rabih,rb,1975,1999", + "AberWa00,Abercrombie,Walter,rb,1959,1982", + "AbraDa00,Abramowicz,Danny,wr,1945,1967", + "AdamBo00,Adams,Bob,te,1946,1969", + "AdamCh00,Adams,Charlie,wr,1979,2003" + + + We want to map this data to the following Player object: + + public class Player implements Serializable { + + private String ID; + private String lastName; + private String firstName; + private String position; + private int birthYear; + private int debutYear; + + public String toString() { + + return "PLAYER:ID=" + ID + ",Last Name=" + lastName + + ",First Name=" + firstName + ",Position=" + position + + ",Birth Year=" + birthYear + ",DebutYear=" + + debutYear; + } + + // setters and getters... + } + + + We can now inject a fieldset mapper into the ListPlayerReader, + for example, that can take advantage of a PlayerFieldSetMapper for + transforming a line that consists of one item separated by delimiters + into a domain object - Player in this + case. We inject programmatically by invoking the following: + + itemReader.setFieldSetMapper(fieldSetMapper); + + and define the fieldSetMapper class in the following + declaration: + + + protected static class PlayerFieldSetMapper implements FieldSetMapper { + public Object mapLine(FieldSet fieldSet) { + Player player = new Player(); + + player.setID(fieldSet.readString(0)); + player.setLastName(fieldSet.readString(1)); + player.setFirstName(fieldSet.readString(2)); + player.setPosition(fieldSet.readString(3)); + player.setBirthYear(fieldSet.readInt(4)); + player.setDebutYear(fieldSet.readInt(5)); + + return player; + } + } + + + There is one additional preference that can be used that is + similar in function to the jdbc fieldset. The names of the fields can + be injected into the Tokenizer to increase the readability of the + mapping function. We can expose this behavior by adding the following. + First, we tell the tokenizer what the names of the fields in the + fieldset are: + + + tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); + + + and provide a mapper that uses this information as + follows: + + + public class PlayerMapper implements FieldSetMapper { + public Object mapLine(FieldSet fs) { + + if(fs == null){ + return null; + } + + Player player = new Player(); + player.setID(fs.readString("ID")); + player.setLastName(fs.readString("lastName")); + player.setFirstName(fs.readString("firstName")); + player.setPosition(fs.readString("position")); + player.setDebutYear(fs.readInt("debutYear")); + player.setBirthYear(fs.readInt("birthYear")); + + return player; + } + + } + +
+
+
+ +
+ XML Item Readers and Writers + + Spring Batch provides transactional infrastructure for both reading + XML records and mapping them to Java objects as well as writing Java + objects as XML records. + + + Constraints on streaming XML + + StAX API is used for I/O as other standard XML APIs do not fit + batch processing requirements (DOM loads the whole input into memory at + once and SAX controls the parsing process allowing the user only to + provide callbacks). + + + Spring Batch is not tied to any particular OXM technology. Typical + use is to delegate OXM + to Spring WS, which provides uniform abstraction for + the most popular OXM technologies. However dependency on Spring WS is + optional and you can choose to implement Spring Batch specific interfaces + if desired. The relationship to the technologies that OXM supports can be + shown as the following: + + + + + + + + + + + Figure X: OXM Binding + + + Lets take a closer look how XML input and output work in batch. It + is assumed the XML resource is a collection of 'fragments' corresponding + to individual records. Note that OXM tools are designed to work with + standalone XML documents rather than XML fragments cut out of an XML + document, therefore the Spring Batch infrastructure needs to work around + this fact (as described below). + + On input the reader reads the XML resource until it recognizes a new + fragment is about to start (by matching the tag name by default). The + reader creates a standalone XML document from the fragment (or at least + makes it appear so) and passes the document to a deserializer (typically a + wrapper around Spring WS Unmarshaller) to map the XML to a Java + object. + + + + + + + + + + + Figure X: XML Inputs + + + Output works symetrically to input. Java object is passed to a + serializer (typically a wrapper around Spring WS Marshaller) which writes + to output using a custom event writer that filters the StartDocument and + EndDocument events produced for each fragment by the OXM tools. + + For example configuration of XML input and output see the sample + xmlStaxJob. //TODO inline the example once it is not subject to change + + show sample input file +
+ +
+ SQL Sources + + SQL input sources can be configured for various reasons, for + example: + + + + a staging table for large volumes of sorted data that was loaded + from flat files + + + + the beginning of an outbound collection of data targeted for an + external flat file interface + + + + the target of a triggered event like "collect all cases that can + be automatically closed" + + Spring Batch supports two approaches for accessing a SQL Input + Source; 1) a cursor driven input source and 2) an indexed based Input + Query. The cursor driven input source is named because it utilizes a + jdbc cursor to stream over the SQL input source whereas an indexed + based input query is designed for easy division of the input into + ranges + + +
+ +
+ +
+ +
+ Validating Input + + +
+ \ No newline at end of file diff --git a/docs/src/site/docbook/reference/repeat.xml b/docs/src/site/docbook/reference/repeat.xml new file mode 100644 index 000000000..21203cdbf --- /dev/null +++ b/docs/src/site/docbook/reference/repeat.xml @@ -0,0 +1,76 @@ + + + + Repeat + +
+ Repeat Templates + + One of the most fundamental concepts in the batch architecture is + the Repeat Template. The Repeat Template is responsible for repeatedly + invoking an operation on the Input Provider pulling input items from an + input source until there are no more items to be processed. One + interesting analogy used by Dierk Koenig in the book "Groovy in Action" is + a boiler vs. a continuous-flow heater. In this analogy he illustrates how + XML parsers can typically be divided into those that read the entire input + before process begins like DOM Parsers vs. those that stream over the + input like SAX parsers. Spring Batch is a continuous-flow heater and uses + the RepeatTemplate as the mechanism to keep the hot water or input stream + in constant flow. + + Many times batch processes are not only working on non-transaction + input sources like files but the output is a transactional resource such + as a queue or database. A common scenario when a batch job is a datastream + coming from a flat file interface is to have a file or files as input + sources and a database resource as the output source. In this case the + repeat templates can be used like the following: + + In this batch scenario an outer RepeatTemplate initialies the + continuous flow, a TransactionTemplate wraps the input and output + resources and an inner RepeatTemplate manages the commit interval or + chunks of data to be processed. The Business Logic occurs in the input and + output of single items. Of course this is a simplistic view of how batch + really works. Input can be quite complex with multiple files and + complicated validation scenarios. Conversely, the output source can also + be quite complex in determining how the records will be stored in the + database. Spring Batch makes no assumptions about how simple or complex + the business processing is within the RepeatTemplates. It's only job is to + keep the flow moving from the Item Provider to the Item Processor as + quickly as possible. The repeat template can process records irrespective + of the batch architecture. A simple example would be: + + RepeatTemplate template = new RepeatTemplate(); + Resource resource = new FileSystemResource("resources/trades.csv"); + TradeProcessor executor = new TradeProcessor(); + TradeItemProvider provider = null; + try { + provider = new TradeItemProvider(resource); + } catch (Exception e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + template.iterate(new ItemProviderRepeatCallback(provider, executor)); + + + A RepeatTemplate has an exception policy that can be + leveraged +
+ \ No newline at end of file diff --git a/docs/src/site/docbook/reference/retry.xml b/docs/src/site/docbook/reference/retry.xml new file mode 100644 index 000000000..ed92363f1 --- /dev/null +++ b/docs/src/site/docbook/reference/retry.xml @@ -0,0 +1,12 @@ + + + + Retry + +
+ + + +
+ \ No newline at end of file