diff --git a/docs/src/site/docbook/reference/common-patterns.xml b/docs/src/site/docbook/reference/common-patterns.xml
index 106183f4b..2fc1ac0d1 100644
--- a/docs/src/site/docbook/reference/common-patterns.xml
+++ b/docs/src/site/docbook/reference/common-patterns.xml
@@ -4,27 +4,23 @@
Common Batch Patterns
-
- Introduction
+ Some batch jobs can be assembled purely from off-the-shelf components
+ in Spring Batch, mostly the ItemReader and
+ ItemWriter implementations. Where this is not
+ possible (the majority of cases) the main API entry points for application
+ developers are the Tasklet,
+ ItemReader, ItemWriter and the
+ various listener interfaces. Most simple batch jobs will be able to use
+ off-the-shelf input from a Spring Batch ItemReader,
+ but it is very often the case that there are custom concerns in the
+ processing and writing, which normally leads developers to implement an
+ ItemWriter, or
+ ItemTransformer.
- Some batch jobs can be assembled purely from off-the-shelf
- components in Spring Batch, mostly the ItemReader
- and ItemWriter implementations. Where this is not
- possible (the majority of cases) the main API entry points for application
- developers are the Tasklet,
- ItemReader, ItemWriter and
- the various listener interfaces. Most simple batch jobs will be able to
- use off-the-shelf input from a Spring Batch
- ItemReader, but it is very often the case that
- there are custom concerns in the processing and writing, which normally
- leads developers to implement an ItemWriter, or
- ItemTransformer.
-
- Here we provide a few examples of common patterns in custom business
- logic, mainly using the listener interfaces . It should be noted that an
- ItemReader or ItemWriter can
- implement the listener interfaces as well if appropriate.
-
+ Here we provide a few examples of common patterns in custom business
+ logic, mainly using the listener interfaces . It should be noted that an
+ ItemReader or ItemWriter can
+ implement the listener interfaces as well if appropriate.
Logging Item Processing and Failures
@@ -229,4 +225,57 @@
maintains its own state in a transactional resource like a database, there
is no need to maintain state within the writer itself.
+
+
+ Driving Query Based ItemReaders
+
+ In the chapter on readers and writers, database input using paging
+ was discussed. Many database vendors, such as DB2, have extremely
+ pessimistic locking strategies that can cause issues if the table being
+ read also needs to be used by other portions of the online application.
+ Furthermore, opening cursors over extremely large datasets can cause
+ issues on certain vendors. Therefore, many projects prefer to use a
+ 'Driving Query' approach to reading in data. This approach works by
+ iterating over keys, rather than the entire object that needs to be
+ returned, as the following example illustrates:
+
+
+
+
+
+
+
+
+
+
+
+ As you can see, this example uses the same 'FOO' table as was used
+ in the cursor based example. However, rather than selecting the entire
+ row, only the ID's were selected in the SQL statement. So, rather than a
+ FOO object being returned from read, an Integer
+ will be returned. This number can then be used to query for the 'details',
+ which is a complete Foo object:
+
+
+
+
+
+
+
+
+
+
+
+ As you can see, an existing DAO can be used to obtain a full 'Foo'
+ object using the key obtained from the driving query. In Spring Batch,
+ driving query style input is implemented with a
+ DrivingQueryItemReader, which has only one
+ dependency: a KeyCollector
+
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/images/oxm-fragments.png b/docs/src/site/docbook/reference/images/oxm-fragments.png
index f77af97e7..0e6df14d1 100644
Binary files a/docs/src/site/docbook/reference/images/oxm-fragments.png and b/docs/src/site/docbook/reference/images/oxm-fragments.png differ
diff --git a/docs/src/site/docbook/reference/images/xmlinput.png b/docs/src/site/docbook/reference/images/xmlinput.png
index f270346c2..d123488c8 100644
Binary files a/docs/src/site/docbook/reference/images/xmlinput.png and b/docs/src/site/docbook/reference/images/xmlinput.png differ
diff --git a/docs/src/site/docbook/reference/readersAndWriters.xml b/docs/src/site/docbook/reference/readersAndWriters.xml
index d4464101c..be82d6c85 100644
--- a/docs/src/site/docbook/reference/readersAndWriters.xml
+++ b/docs/src/site/docbook/reference/readersAndWriters.xml
@@ -4,16 +4,12 @@
ItemReaders and ItemWriters
-
- Introduction
-
- All batch processing can be described in its most simple form as
- reading in large amounts of data, performing some type of calculation or
- transformation, and writing the result out. Spring Batch provides two key
- interfaces to help perform bulk reading and writing:
- ItemReader and
- ItemWriter.
-
+ All batch processing can be described in its most simple form as
+ reading in large amounts of data, performing some type of calculation or
+ transformation, and writing the result out. Spring Batch provides two key
+ interfaces to help perform bulk reading and writing:
+ ItemReader and
+ ItemWriter.
ItemReader
@@ -50,12 +46,12 @@
ItemReader is a basic interface for generic
input operations:
- {
+ public interface ItemReader<T> {
T read() throws Exception, UnexpectedInputException, ParseException;
}
-]]>
+
The read method defines the most essential
contract of the ItemReader, calling it returns one
@@ -67,12 +63,10 @@
It is expected that implementations of the
ItemReader interface will be forward only. However,
- if the underlying resource is transactional (such as a JMS queue) thehn
+ if the underlying resource is transactional (such as a JMS queue) then
calling read may return the same logical item on subsequent calls in a
- rollback scenario.
-
- It is also worth noting that a lack of items to process by an
- ItemReader will not cause an exception to be
+ rollback scenario. It is also worth noting that a lack of items to process
+ by an ItemReader will not cause an exception to be
thrown. For example, a database ItemReader that is
configured with a query that returns 0 results will simply return null on
the first invocation of read.
@@ -82,7 +76,7 @@
ItemWriter
ItemWriter is similar in functionality to an
- ItemReader, but with reversed operations. Resources
+ ItemReader, but with inverse operations. Resources
still need to be located, opened and closed but they differ in that an
ItemWriter writes out, rather than reading in. In
the case of databases or queues these may be inserts, updates or sends.
@@ -92,12 +86,12 @@
As with ItemReader,
ItemWriter is a fairly generic interface:
- {
+ public interface ItemWriter<T> {
- void write(List items) throws Exception;
+ void write(List<? extends T> items) throws Exception;
}
-]]>
+
As with read on
ItemReader, write provides
@@ -121,15 +115,15 @@
readers and writers need to be opened, closed, and require a mechanism for
persisting state:
- public interface ItemStream {
void open(ExecutionContext executionContext) throws StreamException;
- void update(ExecutionContext executionContext);
+ void update(ExecutionContext executionContext) throws ItemStreamException;
void close(ExecutionContext executionContext) throws StreamException;
}
-]]>
+
Before describing each method, its worth briefly mentioning the
ExecutionContext. Clients of an
@@ -168,8 +162,7 @@
always been the flat file. Unlike XML, which has an agreed upon standard
for defining how it is structured (XSD), anyone reading a flat file must
understand ahead of time exactly how the file is structured. In general,
- all flat files fall into two general types: Delimited and Fixed
- Length.
+ all flat files fall into two types: Delimited and Fixed Length.
The FieldSet
@@ -189,11 +182,11 @@
fields so that the fields may be accessed either by index or name as
patterned after ResultSet:
- String[] tokens = new String[]{"foo", "1", "true"};
FieldSet fs = new DefaultFieldSet(tokens);
String name = fs.readString(0);
int value = fs.readInt(1);
- boolean booleanValue = fs.readBoolean(2);]]>
+ boolean booleanValue = fs.readBoolean(2);
There are many more options on the FieldSet
interface, such as Date, long,
@@ -212,23 +205,20 @@
two-dimensional (tabular) data. Reading flat files in the Spring Batch
framework is facilitated by the class
FlatFileItemReader, which provides basic
- functionality for reading and parsing flat files. The three most
- important required dependencies of
- FlatFileItemReader are
- Resource, FieldSetMapper
- and LineTokenizer. The
- FieldSetMapper and
- LineTokenizer interfaces will be explored more in
- the next sections. The resource property represents a Spring Core
- Resource. Documentation explaining how to create
- beans of this type can be found in FlatFileItemReader are
+ Resource and LineMapper.
+ The LineMapper interface will be
+ explored more in the next sections. The resource property represents a
+ Spring Core Resource. Documentation explaining
+ how to create beans of this type can be found in Spring
Framework, Chapter 4.Resources. Therefore, this
guide will not go into the details of creating
Resource objects. However, a simple example of a
- file system resource can be found below:
Resource resource = new FileSystemResource("resources/trades.csv");
- ]]>
+
In complex batch environments the directory structures are often
managed by the EAI infrastructure where drop zones for external
@@ -238,7 +228,11 @@
batch job streams to include file moving utilities as steps in the job
stream. Its sufficient that the batch architecture only needs to know
how to locate the files to be processed. Spring Batch begins the process
- of feeding the data into the pipe from this starting point.
+ of feeding the data into the pipe from this starting point. However,
+ Spring
+ Integration provides many of these types of
+ services.
The other properties in FlatFileItemReader
allow you to further specify how your data will be interpreted:
@@ -285,6 +279,16 @@
the file
+
+ skippedLinesCallbackHandler
+
+ LineCallbackHandler
+
+ Interface which passes the raw line
+ content of the lines in the file to be skipped. If linesToSkip
+ is set to 2, then this interface will be called twice.
+
+
firstLineIsHeader
@@ -316,32 +320,23 @@
As with RowMapper, which takes a low
level construct such as ResultSet and returns an Object,
flat file procesing requires the same construct to convert a String
- line into an Object: {
+ line into an Object:public interface LineMapper<T> {
T mapLine(String line, int lineNumber) throws Exception;
-}]]>
-
+}
-
- FieldSetMapper
-
- The FieldSetMapper interface defines a
- single method, mapLine, which takes a
- FieldSet object and maps its contents to an
- object. This object may be a custom DTO or domain object, or it could
- be as simple as an array, depending on your needs. The
- FieldSetMapper is used in conjunction with the
- LineTokenizer to translate a line of data from
- a resource into an object of the desired type:
-
- {
-
- T mapFieldSet(FieldSet fieldSet);
-
- }]]>
-
- The pattern used is the same as RowMapper
- used by JdbcTemplate.
+ The basic constract is that, given the current line, and the
+ line number its associated with, return a resulting domain object.
+ This is similiar to RowMapper in that each line
+ is associated with it's line number, just as each row in a resultset
+ is tied to the row number it belongs to. This allows for tying the
+ line number to the resulting domain object for identitiy comparison,
+ or for more informative logging. However, unlike
+ RowMapper, the
+ LineMapper is given a raw line which, as
+ discussed above, only gets you halfway there. What is needed is
+ tokenization of the line into a FieldSet, which
+ can then be mapped to an object, as described below.
@@ -349,24 +344,24 @@
Because there can be many formats of flat file data, which all
need to be converted to a FieldSet so that a
- FieldSetMapper can create a useful domain
- object from them, an abstraction for turning a line of input into a
- FieldSet is necessary. In Spring Batch, this is
- called a LineTokenizer:
+ useful domain object can be created from them, an abstraction for
+ turning a line of input into a FieldSet is
+ necessary. In Spring Batch, this is called a
+ LineTokenizer:
- public interface LineTokenizer {
FieldSet tokenize(String line);
- }]]>
+ }
The contract of a LineTokenizer is such
that, given a line of input (in theory the
String could encompass more than one line) a
FieldSet representing the line will be
- returned. This will then be passed to a
+ returned. This can then be passed to a
FieldSetMapper. Spring Batch contains the
- following LineTokenizers:
+ following LineTokenizer implementations:
@@ -391,58 +386,103 @@
- Simple Delimited File Reading Example
+ FieldSetMapper
+
+ The FieldSetMapper interface defines a
+ single method, mapLine, which takes a
+ FieldSet object and maps its contents to an
+ object. This object may be a custom DTO or domain object, or it could
+ be as simple as an array, depending on your needs. The
+ FieldSetMapper is used in conjunction with the
+ LineTokenizer to translate a line of data from
+ a resource into an object of the desired type:
+
+ public interface FieldSetMapper<T> {
+
+ T mapFieldSet(FieldSet fieldSet);
+
+ }
+
+ The pattern used is the same as RowMapper
+ used by JdbcTemplate.
+
+
+
+ DefaultLineMapper
Now that the basic interfaces for reading in flat files have
- been defined, a simple example explaining how they work together is
- helpful. In it's most simple form, the flow when reading a line from a
- file is the following:
+ been defined, it becomes clear that three basic steps are
+ required:
+
+ Read one line from the file.
+
-
-
- Read one line from the file.
-
+
+ Pass the string line into the LineTokenizer#tokenize()
+ method, in order to retrieve a
+ FieldSet
+
-
- Pass the string line into the LineTokenizer#tokenize()
- method, in order to retrieve a
- FieldSet
-
+
+ Pass the FieldSet returned from tokenizing to a
+ FieldSetMapper, returning the result from the ItemReader#read()
+ method
+
+
-
- Pass the FieldSet returned from tokenizing to a
- FieldSetMapper, returning the result from the ItemReader#read()
- method
-
-
+ The two interfaces described above represent two separate tasks:
+ converting a line into a FieldSet, and mapping
+ a FieldSet to a domain object. Becaue the input
+ of a LineTokenizer matches the input of the
+ LineMapper (a line), and the output of a
+ FieldSetMapper matches the output of the
+ LineMapper, and this is the deafult behavior
+ most users will need, a default implementation that uses both a
+ LineTokenizer and
+ FieldSetMapper is provided:
- In code, the above flow looks like the following:
+
+ public class DefaultLineMapper<T> implements LineMapper<T>, InitializingBean {
-
+ private FieldSetMapper<T> fieldSetMapper;
-
- Exception handling has been removed for clarity.
-
+ public T mapLine(String line, int lineNumber) throws Exception {
+ return fieldSetMapper.mapFieldSet(tokenizer.tokenize(line));
+ }
+
+ public void setLineTokenizer(LineTokenizer tokenizer) {
+ this.tokenizer = tokenizer;
+ }
+
+ public void setFieldSetMapper(FieldSetMapper<T> fieldSetMapper) {
+ this.fieldSetMapper = fieldSetMapper;
+ }
+}
+
+ The above functionality is provided in a default implementation,
+ rather than being built into the reader itself (as was done in
+ previous versions of the framework) in order to allow users greater
+ flexibility in controlling the parsing process, especially if access
+ to the raw line is needed.
+
+
+
+ Simple Delimited File Reading Example
The following example will be used to illustrate this using an
actual domain scenario. This particular batch job reads in football
- players from the following file: ID,lastName,firstName,position,birthYear,debutYear
"AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996",
"AbduRa00,Abdullah,Rabih,rb,1975,1999",
"AberWa00,Abercrombie,Walter,rb,1959,1982",
"AbraDa00,Abramowicz,Danny,wr,1945,1967",
"AdamBo00,Adams,Bob,te,1946,1969",
- "AdamCh00,Adams,Charlie,wr,1979,2003" ]]>
+ "AdamCh00,Adams,Charlie,wr,1979,2003"
The contents of this file will be mapped to the following Player
- domain object:
public class Player implements Serializable {
private String ID;
@@ -462,14 +502,14 @@
// setters and getters...
}
- ]]>
+
In order to map a FieldSet into a Player
object, a FieldSetMapper that returns players
needs to be defined:
- {
+
+ protected static class PlayerFieldSetMapper implements FieldSetMapper<Player> {
public Object mapLine(FieldSet fieldSet) {
Player player = new Player();
@@ -482,22 +522,24 @@
return player;
}
- } ]]>
+ }
The file can then be read by correctly constructing a
FlatFileItemReader and calling
read:
- itemReader = new FlatFileItemReader();
+
+ FlatFileItemReader<Player> itemReader = new FlatFileItemReader<Player>();
itemReader.setResource(new FileSystemResource("resources/players.csv"));
//DelimitedLineTokenizer defaults to comma as it's delimiter
- itemReader.setLineTokenizer(new DelimitedLineTokenizer());
- itemReader.setFieldSetMapper(new PlayerFieldSetMapper());
+ LineMapper<Player> lineMapper = new DefaultLineMapper<Player>();
+ lineMapper.setLineTokenizer(new DelimitedLineTokenizer());
+ lineMapper.setFieldSetMapper(new PlayerFieldSetMapper());
+ itemReader.setLineMapper(lineMapper);
itemReader.open(new ExecutionContext());
Player player = itemReader.read();
-]]>
+
Each call to read will return a new
Player object from each line in the file. When the end of the file is
@@ -515,9 +557,9 @@
fields in the flat file are injected into the
LineTokenizer:
-
tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"});
- ]]>
+
a FieldSetMapper can this use this
information as follows:
@@ -553,18 +595,18 @@
JdbcTemplate. Spring Batch makes this easier by providing a
FieldSetMapper that automatically maps fields
by matching a field name with a setter on the object using the
- JavaBean spec. Again using the football example, the
+ JavaBean specification. Again using the football example, the
FieldSetMapper configuration looks like the
following:
-
-
-
+ <bean id="fieldSetMapper"
+ class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
+ <property name="prototypeBeanName" value="player" />
+ </bean>
- ]]>
+ scope="prototype" />
For each entry in the FieldSet, the
mapper will look for a corresponding setter on a new instance of the
@@ -584,11 +626,11 @@
organizations that use flat files use fixed length formats. An example
fixed length file is below:
- UK21341EAH4121131.11customer1
UK21341EAH4221232.11customer2
UK21341EAH4321333.11customer3
UK21341EAH4421434.11customer4
- UK21341EAH4521535.11customer5]]>
+ UK21341EAH4521535.11customer5
While this looks like one large field, it actually represent 4
distinct fields:
@@ -618,14 +660,14 @@
FixedLengthLineTokenizer, each of these lengths
must be provided in the form of ranges:
-
-
-
-
+
+ <bean id="fixedLengthLineTokenizer"
+ class="org.springframework.batch.io.file.transform.FixedLengthTokenizer">
+ <property name="names" value="ISIN, Quantity, Price, Customer" />
+ <property name="columns" value="1-12, 13-15, 16-20, 21-29" />
+ </bean>
-]]>
+
This LineTokenizer will return the same
FieldSet as if a dlimiter had been used,
@@ -637,18 +679,18 @@
specialized property editor be configured anywhere in the
ApplicationContext:
-
-
-
-
-
+
+ <bean id="customEditorConfigurer" class="org.springframework.beans.factory.config.CustomEditorConfigurer">
+ <property name="customEditors">
+ <map>
+ <entry key="org.springframework.batch.item.file.transform.Range[]">
+ <bean class="org.springframework.batch.item.file.transform.RangeArrayPropertyEditor" />
+ </entry>
+ </map>
+ </property>
+ </bean>
-]]>
+
@@ -660,7 +702,7 @@
might have records spanning multiple lines with multiple formats. The
following excerpt from a file illustrates this:
- HEA;0013100345;2007-02-15
NCU;Smith;Peter;;T;20014539;F
BAD;;Oak Street 31/A;;Small Town;00235;IL;US
SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States
@@ -668,30 +710,30 @@
LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47
LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87
SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS
- FOT;2;2;267.34]]>
+ FOT;2;2;267.34
Everything between the line starting with 'HEA' and the line
starting with 'FOT' is considered one record. The
PrefixMatchingCompositeLineTokenizer makes this easier by matching the
prefix in a line with a particular tokenizer:
-
-
-
-
- ]]>
+ <bean id="orderFileDescriptor"
+ class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer">
+ <property name="tokenizers">
+ <map>
+ <entry key="HEA" value-ref="headerRecordDescriptor" />
+ <entry key="FOT" value-ref="footerRecordDescriptor" />
+ <entry key="BCU" value-ref="businessCustomerLineDescriptor" />
+ <entry key="NCU" value-ref="customerLineDescriptor" />
+ <entry key="BAD" value-ref="billingAddressLineDescriptor" />
+ <entry key="SAD" value-ref="shippingAddressLineDescriptor" />
+ <entry key="BIN" value-ref="billingLineDescriptor" />
+ <entry key="SIN" value-ref="shippingLineDescriptor" />
+ <entry key="LIT" value-ref="itemLineDescriptor" />
+ <entry key="" value-ref="defaultLineDescriptor" />
+ </map>
+ </property>
+ </bean>
This ensures that the line will be parsed correctly, which is
especially important for fixed length input. Any users of the
@@ -733,7 +775,7 @@
IncorrectTokenCountException is thrown, which contains the number of
tokens encountered, and the number expected:
-
tokenizer.setNames(new String[] {"A", "B", "C", "D"});
try{
@@ -744,7 +786,7 @@
assertEquals(3, e.getActualCount());
}
-]]>
+
Because the tokenizer was configured with 4 columns, but only
3 tokens were found in the file, an IncorrectTokenCountException was
@@ -760,7 +802,7 @@
total line length doesn't add up to the widest value of this column,
an exception is thrown:
-
tokenizer.setColumns(new Range[] { new Range(1, 5), new Range(6, 10), new Range(11, 15) });
try {
tokenizer.tokenize("12345");
@@ -771,7 +813,7 @@
assertEquals(5, ex.getActualLength());
}
-]]>
+
The configured ranges for the tokenizer above are: 1-5, 6-10,
and 11-15, thus the total length of the line expected is 15.
@@ -785,14 +827,14 @@
For this reason, validation of line length can be turned off via the
'strict' property:
-
tokenizer.setColumns(new Range[] { new Range(1, 5), new Range(6, 10) });
- tokenizer.setStrict(false);
+ tokenizer.setStrict(false);
FieldSet tokens = tokenizer.tokenize("12345");
assertEquals("12345", tokens.readString(0));
assertEquals("", tokens.readString(1));
-]]>
+
The above example is almost identical to the one before it,
except the tokenizer.setStrict(false) was called. This setting tells
@@ -821,11 +863,11 @@
writing to a file. In Spring Batch this is the
LineAggregator:
- {
+ public interface LineAggregator<T> {
public String aggregate(T item);
- }]]>
+ }
The LineAggregator is the opposite of a
LineTokenizer.
@@ -834,16 +876,39 @@
FieldSet, whereas
LineAggregator takes an
item and returns a
- String. As with reading there are two types:
- DelimitedLineAggregator and
- FixedLengthLineAggregator.
+ String.
+
+
+ PassThroughLineAggregator
+
+ The most basic implementation of the LineAggreagot interface
+ is the PassThroughLineAggregator, which simply assumes that the
+ object is already a string, or that it's string representation is
+ acceptable for writing:
+
+
+ public class PassThroughLineAggregator<T> implements LineAggregator<T> {
+
+ public String aggregate(T item) {
+ return item.toString();
+ }
+ }
+
+
+
+ The above implementation is useful if direct control of
+ creating the string is required, but the advantages of a
+ FlatFileItemWriter, such as transaction and restart support, are
+ necessary.
+
- Simple Delimited File Writing Example
+ Simplified File Writing Example
- Now that the LineAggregator interface has
- been defined, the basic flow of writing can be explained:
+ Now that the LineAggregator interface and
+ it's most basic implementation, PassThroughLineAggregator, has been
+ defined the basic flow of writing can be explained:
@@ -862,21 +927,227 @@
FlatFileItemWriter expresses this in
code:
-
+ public void write(T item) throws Exception {
+ write(lineAggregator.aggregate(item) + LINE_SEPARATOR);
+ }
- A simple configuration with the smallest ammount of setters
- would look like the following:
+ A simple configuration would look like the following:
-
-
-
-
-
- ]]>
+ <bean id="itemWriter"
+ class="org.springframework.batch.io.file.FlatFileItemWriter">
+ <property name="resource"
+ value="file:target/test-outputs/20070122.testStream.multilineStep.txt" />
+ <property name="lineAggregator">
+ <bean class="org.springframework.batch.item.file.transform.PassThroughLineAggregator"/>
+ </property>
+ </bean>
+
+
+
+ FieldExtractor
+
+ The above example may be useful for the most basic uses of a
+ writing to a file. However, most users of the FlatFileItemWriter will
+ will have a domain object that needs to be written out, and thus must
+ be converted into a line. In file reading, the following was
+ required:
+
+ Read one line from the file.
+
+
+
+ Pass the string line into the LineTokenizer#tokenize()
+ method, in order to retrieve a
+ FieldSet
+
+
+
+ Pass the FieldSet returned from tokenizing to a
+ FieldSetMapper, returning the result from the ItemReader#read()
+ method
+
+
+
+ File writing has similiar, but inverse steps:
+
+
+
+ Pass the item to be written to the writer
+
+
+
+ convert the fields on the item into an array
+
+
+
+ aggregate the resulting array into a line
+
+
+
+ Because there is no way for the framework to know which fields
+ from the object need to be written out, a FieldExtractor must be
+ written to accomplish the task:
+
+
+ public interface FieldExtractor<T> {
+
+ Object[] extract(T item);
+
+ }
+
+
+
+ Implementations of the FieldExtractor
+ interface should create an array from the fields of the provided
+ object, which can then be written out with a delimited between the
+ elements, or as part of a field-width line.
+
+
+ PassThroughFieldExtractor
+
+ There are many cases where an array or something that can be
+ converted to an array, such as a Collection,
+ needs to be written out. For example, a List
+ could be passed through, in which case it only needs to be converted
+ to an Object array to be written out. For this type of scenario the
+ PassThroughFieldExtractor can be used. It should be noted, that if
+ the object passed in is not an array, and not a Collection, then an
+ Object array containing solely the item will be returned.
+
+
+
+ BeanWrapperFieldExtractor
+
+ As with the BeanWrapperFieldSetMapper
+ described in the file reading section, it is much preferrable to
+ configure how to convert an domain object to an object array, rather
+ than writing the conversion yourself. The
+ BeanWrapperFieldExtractor provides just this
+ type of functionality:
+
+
+ BeanWrapperFieldExtractor<Name> extractor = new BeanWrapperFieldExtractor<Name>();
+ extractor.setNames(new String[] { "first", "last", "born" });
+
+ String first = "Alan";
+ String last = "Turing";
+ int born = 1912;
+
+ Name n = new Name(first, last, born);
+ Object[] values = extractor.extract(n);
+
+ assertEquals(first, values[0]);
+ assertEquals(last, values[1]);
+ assertEquals(born, values[2]);
+
+
+
+ This extractor implementation has only one required property,
+ the names of the fields to map. Just as the
+ BeanWrapperFieldSetMapper needs field names
+ to map fields on the FieldSet to setters on the provided object, the
+ BeanWrapperFieldExtractor needs names to map
+ to getters for creating an object array. It's worth noting that the
+ order of the names determines the order of the fields within the
+ array.
+
+
+
+
+ Delimited File Writing Example
+
+ The most basic flat file format is one in which all fields are
+ separated by a delimiter. This can be accomplished using a
+ DelimitedLineAggregator. The example below writes out a simple domain
+ object that represents a credit to a customer account:
+
+
+ public class CustomerCredit {
+
+ private int id;
+
+ private String name;
+
+ private BigDecimal credit;
+
+ public CustomerCredit(int id, String name, BigDecimal credit) {
+ this.id = id;
+ this.name = name;
+ this.credit = credit;
+ }
+
+ //getters and setters removed for clarity
+ }
+
+
+
+ Because a domain object is being used, an implementation of the
+ FieldExtractor interface must be provided, along with the delimiter to
+ use:
+
+
+ <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
+ <property name="resource" ref="outputResource" />
+ <property name="lineAggregator">
+ <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
+ <property name="delimiter" value=","/>
+ <property name="fieldExtractor">
+ <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
+ <property name="names" value="name,credit"/>
+ </bean>
+ </property>
+ </bean>
+ </property>
+ </bean>
+
+
+
+ In this case, the
+ BeanWrapperFieldExtractor described earlier in
+ this chapter is used to turn the name and credit fields within
+ CustomerCredit into an object array, which is then written out with
+ commas between each field.
+
+
+
+ Fixed Width File Writing Example
+
+ Delimited is not the only type of flat file format, many prefer
+ to use a set width for each column to deliniate between fields, which
+ is usually referred to as 'fixed width'. Spring Batch supports this in
+ file writting via the FormatterLineAggregator. Using the same
+ CustomerCredit domain object described above, it can be configured as
+ follows:
+
+
+ <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
+ <property name="resource" ref="outputResource" />
+ <property name="lineAggregator">
+ <bean class="org.springframework.batch.item.file.transform.FormatterLineAggregator">
+ <property name="fieldExtractor">
+ <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
+ <property name="names" value="name,credit" />
+ </bean>
+ </property>
+ <property name="format" value="%-9s%-2.0f" />
+ </bean>
+ </property>
+ </bean>
+
+
+
+ Most of the above example should look familiar. However, the
+ value of the format property is new:
+
+
+ <property name="format" value="%-9s%-2.0f" />
+
+
+
+ The underlying implementation is built using the same Formatter
+ added as part of Java 5. Most details on how to configure a formatter
+ can be found in the javadoc of Formatter.
@@ -930,12 +1201,12 @@
+ format="PNG" width="50%" />
+ width="50%" />
Figure 3.1: XML Input
@@ -957,20 +1228,19 @@
+ format="PNG" width="50%" />
+ format="PNG" width="50%" />
Figure 3.2: OXM Binding
Now with an introduction to OXM and how one can use XML fragments to
- represent records, let's take a closer look at Item Readers and Item
- Writers.
+ represent records, let's take a closer look at readers and writers.
StaxEventItemReader
@@ -980,28 +1250,30 @@
stream. First, lets examine a set of XML records that the
StaxEventItemReader can process.
-
-
-
- XYZ0001
- 5
- 11.39
- Customer1
-
-
- XYZ0002
- 2
- 72.99
- Customer2c
-
-
- XYZ0003
- 9
- 99.99
- Customer3
-
-]]>
+
+<?xml version="1.0" encoding="UTF-8"?>
+<records>
+ <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain">
+ <isin>XYZ0001</isin>
+ <quantity>5</quantity>
+ <price>11.39</price>
+ <customer>Customer1</customer>
+ </trade>
+ <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain">
+ <isin>XYZ0002</isin>
+ <quantity>2</quantity>
+ <price>72.99</price>
+ <customer>Customer2c</customer>
+ </trade>
+ <trade xmlns="http://springframework.org/batch/sample/io/oxm/domain">
+ <isin>XYZ0003</isin>
+ <quantity>9</quantity>
+ <price>99.99</price>
+ <customer>Customer3</customer>
+ </trade>
+</records>
+
+
To be able to process the XML records the following is needed:
@@ -1023,22 +1295,24 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- ]]>
+
+ <bean id="itemReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
+ <property name="fragmentRootElementName" value="customer" />
+ <property name="resource" value="data/iosample/input/input.xml" />
+ <property name="unmarshaller" ref="customerCreditMarshaller" />
+ </bean>
+
+ <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller">
+ <property name="aliases">
+ <util:map id="aliases">
+ <entry key="customer"
+ value="org.springframework.batch.sample.domain.trade.CustomerCredit" />
+ <entry key="price" value="java.math.BigDecimal" />
+ <entry key="name" value="java.lang.String" />
+ </util:map>
+ </property>
+ </bean>
+
Notice that in this example we have chosen to use an
XStreamMarshaller that requires an alias passed
@@ -1049,16 +1323,24 @@
the map. In the configuration file we can use a spring configuration
utility to describe the required alias as follows:
-
-
-
-
-
-
-
- ]]>
+
+ <bean id="itemReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
+ <property name="fragmentRootElementName" value="customer" />
+ <property name="resource" value="data/iosample/input/input.xml" />
+ <property name="unmarshaller" ref="customerCreditMarshaller" />
+ </bean>
+
+ <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller">
+ <property name="aliases">
+ <util:map id="aliases">
+ <entry key="customer"
+ value="org.springframework.batch.sample.domain.trade.CustomerCredit" />
+ <entry key="price" value="java.math.BigDecimal" />
+ <entry key="name" value="java.lang.String" />
+ </util:map>
+ </property>
+ </bean>
+
On input the reader reads the XML resource until it recognizes a
new fragment is about to start (by matching the tag name by default).
@@ -1072,35 +1354,35 @@
injection provided by the spring configuration would look something like
the following:
-
StaxEventItemReader xmlStaxEventItemReader = new StaxEventItemReader()
Resource resource = new ByteArrayResource(xmlResource.getBytes())
Map aliases = new HashMap();
- aliases.put("trade","org.springframework.batch.sample.domain.Trade");
- aliases.put("isin","java.lang.String");
- aliases.put("quantity","long");
+ aliases.put("customer","org.springframework.batch.sample.domain.trade.CustomerCredit");
aliases.put("price","java.math.BigDecimal");
- aliases.put("customer","java.lang.String");
+ aliases.put("name","java.lang.String");
Marshaller marshaller = new XStreamMarshaller();
marshaller.setAliases(aliases);
- xmlStaxEventItemReader.setFragmentDeserializer(new UnmarshallingEventReaderDeserializer(marshaller));
+ xmlStaxEventItemReader.setUnmarshaller(marshaller);
xmlStaxEventItemReader.setResource(resource);
- xmlStaxEventItemReader.setFragmentRootElementName("trade");
+ xmlStaxEventItemReader.setFragmentRootElementName("customer");
xmlStaxEventItemReader.open(new ExecutionContext());
boolean hasNext = true
-
+
+ CustomerCredit credit = null;
+
while (hasNext) {
- trade = xmlStaxEventItemReader.read();
- if (trade == null) {
+ credit = xmlStaxEventItemReader.read();
+ if (credit == null) {
hasNext = false;
} else {
println trade;
}
}
-]]>
+
@@ -1118,167 +1400,95 @@
MarshallingEventWriterSerializer. The Spring
configuration for this setup looks as follows:
-
-
-
-
-
-
-]]>
+
+ <bean id="itemWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter">
+ <property name="resource" ref="outputResource" />
+ <property name="marshaller" ref="customerCreditMarshaller" />
+ <property name="rootTagName" value="customers" />
+ <property name="overwriteOutput" value="true" />
+ </bean>
+
+
The configuration sets up the three required properties and
optionally sets the overwriteOutput=true, mentioned earlier in the
- chapter for specifying whether an existing file can be overwritten. The
- TradeMarshallingSerializer is configured as
- follows:
+ chapter for specifying whether an existing file can be overwritten. It
+ should be noted the marshaller used for the writer is the exact same as
+ the one used in the reading example from ealier in the chapter:
- <bean class="org.springframework.batch.item.xml.oxm.MarshallingEventWriterSerializer" id="tradeMarshallingSerializer">
- <constructor-arg>
- <bean class="org.springframework.oxm.xstream.XStreamMarshaller">
- <property name="aliases" ref="aliases" />
- </bean>
- </constructor-arg>
-</bean>
+
+ <bean id="customerCreditMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller">
+ <property name="aliases">
+ <util:map id="aliases">
+ <entry key="customer"
+ value="org.springframework.batch.sample.domain.trade.CustomerCredit" />
+ <entry key="price" value="java.math.BigDecimal" />
+ <entry key="name" value="java.lang.String" />
+ </util:map>
+ </property>
+ </bean>
+
To summarize with a Java example, the following code illustrates
all of the points discussed, demonstrating the programmatic setup of the
required properties.
- StaxEventItemWriter staxItemWriter = new StaxEventItemWriter()
FileSystemResource resource = new FileSystemResource(File.createTempFile("StaxEventWriterOutputSourceTests", "xml"))
Map aliases = new HashMap();
- aliases.put("trade","org.springframework.batch.sample.domain.Trade");
- aliases.put("isin","java.lang.String");
- aliases.put("quantity","long");
+ aliases.put("customer","org.springframework.batch.sample.domain.trade.CustomerCredit");
aliases.put("price","java.math.BigDecimal");
- aliases.put("customer","java.lang.String");
- XStreamMarshaller marshaller = new XStreamMarshaller()
- marshaller.setAliases(aliases)
+ aliases.put("name","java.lang.String");
+ Marshaller marshaller = new XStreamMarshaller();
+ marshaller.setAliases(aliases);
- MarshallingEventWriterSerializer tradeMarshallingSerializer = new MarshallingEventWriterSerializer(marshaller)
+ staxItemWriter.setResource(resource);
+ staxItemWriter.setMarshaller(marshaller);
+ staxItemWriter.setRootTagName("trades");
+ staxItemWriter.setOverwriteOutput(true);
- staxItemWriter.setResource(resource)
- staxItemWriter.setSerializer(tradeMarshallingSerializer)
- staxItemWriter.setRootTagName("trades")
- staxItemWriter.setOverwriteOutput(true)
-
- ExecutionContext executionContext = new ExecutionContext()
- staxItemWriter.open(executionContext)
- Trade trade = new Trade()
- trade.isin = "XYZ0001"
- trade.quantity =5
- trade.price = 11.39
- trade.customer = "Customer1"
- println trade
- staxItemWriter.write(trade)
- staxItemWriter.flush()]]>
-
- For a complete example configuration of XML input and output and a
- corresponding Job see the sample xmlStaxJob.
+ ExecutionContext executionContext = new ExecutionContext();
+ staxItemWriter.open(executionContext);
+ CustomerCredit Credit = new CustomerCredit();
+ trade.setPrice(11.39);
+ credit.setName("Customer1");
+ staxItemWriter.write(trade);
+
-
- Creating File Names at Runtime
-
- Both the XML and Flat File examples above use the Spring
- Resource abstraction to obtain the file to read or
- write from. This works because Resource has a
- getFile method, that returns a
- java.io.File. Both XML and Flat File resources can
- be configured using standard Spring constructs:
-
-
-
- ]]>
-
- The above Resource will load the file from
- the file system, at the location specificied. Note that absolute locations
- have to start with a double slash ("//"). In most spring applications,
- this solution is good enough because the names of these are known at
- compile time. However, in batch scenarios, the file name may need to be
- determined at runtime as a parameter to the job. This could be solved
- using '-D' parameters, i.e. a system property:
-
-
-
-]]>
-
- All that would be required for this solution to work would be a
- system argument (-Dinput.file.name="file://file.txt"). (Note that although
- a PropertyPlaceholderConfigurer can be used here,
- it is not necessary if the system property is always set because the
- ResourceEditor in Spring already filters and does
- placeholder replacement on system properties.)
-
- Often in a batch setting it is preferable to parameterize the file
- name in the JobParameters of the job, instead of
- through system properties, and access them that way. To allow for this,
- Spring Batch provides the
- StepExecutionResourceProxy. The proxy can use
- either job name, step name, or any values from the
- JobParameters, by surrounding them with %:
-
-
-
- ]]>
-
- Assuming a job name of 'fooJob', and a step name of 'fooStep', and
- the key-value pair of 'file.name="fileName.txt"' is in the
- JobParameters the job is started with, the
- following filename will be passed as the Resource:
- "//fooJob/fooStep/fileName.txt". It should be noted
- that in order for the proxy to have access to the
- StepExecution, it must be registered as a
- StepListener:
-
-
-
- ]]>
-
- The StepListener interface will be discussed
- in more detail in Chapter 4. For now, it is sufficient to know that the
- proxy must be registered.
-
-
Multi-File Input
It is a common requirement to process multiple files within a single
- Step. Assuming the files are all formatted the same, the
- MultiResourceItemReader supports this type of input
- for both XML and FlatFile processing. Consider the following files in a
- directory:
+ Step. Assuming the files are all formatted the
+ same, the MultiResourceItemReader supports this
+ type of input for both XML and flat file processing. Consider the
+ following files in a directory:
-
+ file-1.txt file-2.txt ignored.txt
file-1.txt and file-2.txt are formatted the same and for business
reasons should be processed together. The
MuliResourceItemReader can be used to read in both
files by using wildcards:
-
-
-
-
+
+ <bean id="multiResourceReader" class="org.springframework.batch.item.SortedMultiResourceItemReader">
+ <property name="resources" value="classpath:data/multiResourceJob/input/file-*.txt" />
+ <property name="delegate" ref="flatFileItemReader" />
+ </bean>
-]]>
+
The referenced delegate is a simple
FlatFileItemReader. The above configuration will
read input from both files, handling rollback and restart scenarios. It
- should be noted that, as with any ItemReader, adding extra input (in this
- case a file) could cause potential issues when restarting. It is
- recommended that batch jobs work with their own individual directories
- until completed successfully.
+ should be noted that, as with any ItemReader,
+ adding extra input (in this case a file) could cause potential issues when
+ restarting. It is recommended that batch jobs work with their own
+ individual directories until completed successfully.
@@ -1296,15 +1506,15 @@
crash quickly. If the SQL statement returns 1 million rows, the
RowMapper will be called 1 million times, holding
all returned results in memory until all rows have been read. Spring Batch
- provides two types of solutions for this problem: Cursor and DrivingQuery
- ItemReaders.
+ provides two types of solutions for this problem: Cursor and Paging
+ database ItemReaders.
Cursor Based ItemReaders
Using a database cursor is generally the default approach of most
- batch developers. This is because it is the database's solution to the
- problem of 'streaming' relational data. The Java
+ batch developers, because it is the database's solution to the problem
+ of 'streaming' relational data. The Java
ResultSet class is essentially an object
orientated mechanism for manipulating a cursor. A
ResultSet maintains a cursor to the current row
@@ -1357,18 +1567,18 @@
DataSource. The following database schema will
be used as an example:
- CREATE TABLE CUSTOMER (
ID BIGINT IDENTITY PRIMARY KEY,
NAME VARCHAR(45),
CREDIT FLOAT
-);]]>
+);
Many people prefer to use a domain object for each row, so we'll
use an implementation of the RowMapper
interface to map a CustomerCredit
object:
- public class CustomerCreditRowMapper implements RowMapper {
public static final String ID_COLUMN = "id";
public static final String NAME_COLUMN = "name";
@@ -1384,7 +1594,7 @@
return customerCredit;
}
-}]]>
+}
Because JdbcTemplate is so familiar to
users of Spring, and the JdbcCursorItemReader
@@ -1395,12 +1605,12 @@
CUSTOMER database. The first example will be using
JdbcTemplate:
-
//For simplicity sake, assume a dataSource has already been obtained
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
List customerCredits = jdbcTemplate.query("SELECT ID, NAME, CREDIT from CUSTOMER", new CustomerCreditRowMapper());
-]]>
+
After running this code snippet the customerCredits list will
contain 1,000 CustomerCredit objects. In the
@@ -1411,7 +1621,7 @@
constrast this with the approach of the
JdbcCursorItemReader:
-
JdbcCursorItemReader itemReader = new JdbcCursorItemReader();
itemReader.setDataSource(dataSource);
itemReader.setSql("SELECT ID, NAME, CREDIT from CUSTOMER");
@@ -1426,7 +1636,7 @@
}
itemReader.close(executionContext);
-]]>
+
After running this code snippet the counter will equal 1,000. If
the code above had put the returned customerCredit into a list, the
@@ -1438,7 +1648,20 @@
ItemWriter, and then the next item obtained via
read. This allows item reading and writing to
be done in 'chunks' and committed periodically, which is the essence
- of high performance batch processing.
+ of high performance batch processing. Furthermore, it is very easily
+ configured for injection into a Spring Batch
+ Step:
+
+
+ <bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
+ <property name="dataSource" ref="dataSource"/>
+ <property name="sql" value="select ID, NAME, CREDIT from CUSTOMER"/>
+ <property name="mapper">
+ <bean class="org.springframework.batch.sample.domain.trade.internal.CustomerCreditRowMapper"/>
+ </property>
+ </bean>
+
+
Additional Properties
@@ -1509,6 +1732,40 @@
ItemStream#update(ExecutionContext) The default value is
false.
+
+
+ driverSupportsAbsolute
+
+ Defaults to false. Indicates whether the JDBC driver
+ supports setting the absolute row on a
+ ResultSet. It is recommended that
+ this is set to true for JDBC drivers that supports
+ ResultSet.absolute() as it may improve performance,
+ especially if a step fails while working with a large data
+ set.
+
+
+
+ setUseSharedExtendedConnection
+
+ Defaults to false. Indicates whether the connection
+ used for the cursor should be used by all other processing
+ thus sharing the same transaction. If this is set to false,
+ which is the default, then the cursor will be opened using
+ its own connection and will not participate in any
+ transactions started for the rest of the step processing. If
+ you set this flag to true then you must wrap the DataSource
+ in an
+ ExtendedConnectionDataSourceProxy to
+ prevent the connection from being closed and released after
+ each commit. When you set this option to true then the
+ statement used to open the cursor will be created with both
+ 'READ_ONLY' and 'HOLD_CUSORS_OVER_COMMIT' options. This
+ allows holding the cursor open over transaction start and
+ commits performed in the step processing. To use this
+ feature you need a database that supports this and a JDBC
+ driver supporting JDBC 3.0 or later.
+
@@ -1542,7 +1799,7 @@
configuration using the same 'customer credit' example as the JDBC
reader:
-
HibernateCursorItemReader itemReader = new HibernateCursorItemReader();
itemReader.setQueryString("from CustomerCredit");
//For simplicity sake, assume sessionFactory already obtained.
@@ -1558,7 +1815,7 @@
}
itemReader.close(executionContext);
-]]>
+
This configured ItemReader will return
CustomerCredit objects in the exact same manner
@@ -1567,7 +1824,18 @@
Customer table. The 'useStatelessSession' property defaults to true,
but has been added here to draw attention to the ability to switch it
on or off. It is also worth noting that the fetchSize of the
- underlying cursor can be set via the setFetchSize property.
+ underlying cursor can be set via the setFetchSize property. As with
+ JdbcCursorItemReader, configuration is
+ straightforward:
+
+
+ <bean id="itemReader"
+ class="org.springframework.batch.item.database.HibernateCursorItemReader">
+ <property name="sessionFactory" ref="sessionFactory" />
+ <property name="queryString" value="from CustomerCredit" />
+ </bean>
+
+
@@ -1592,10 +1860,10 @@
use a different PagingQueryProvider for each
supported database type. There is also the
SimpleDelegatingPagingQueryProvider that will
- auto-detect the database that is being used and use that to adjust to
- use the appropriate PagingQueryProvider
- implementation. This simplifies the configuration and is the
- recommended best practice.
+ auto-detect the database that is being used and determine the
+ appropriate PagingQueryProvider implementation.
+ This simplifies the configuration and is the recommended best
+ practice.
The SimpleDelegatingPagingQueryProvider
requires that you specify a select clause and a from clause. You can
@@ -1610,26 +1878,26 @@
Below is an example configuration using a similar 'customer
credit' example as the cursor based ItemReaders above:
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>
+ <bean id="itemReader"
+ class="org.springframework.batch.item.database.JdbcPagingItemReader">
+ <property name="dataSource" ref="dataSource"/>
+ <property name="queryProvider">
+ <bean class="org.springframework.batch.item.database.support.SimpleDelegatingPagingQueryProvider">
+ <property name="selectClause" value="select id, name, credit"/>
+ <property name="fromClause" value="from customer"/>
+ <property name="whereClause" value="where status=:status"/>
+ <property name="sortKey" value="id"/>
+ </bean>
+ </property>
+ <property name="parameterValues">
+ <map>
+ <entry key="status" value="NEW"/>
+ </map>
+ </property>
+ <property name="pageSize" value="1000"/>
+ <property name="parameterizedRowMapper" ref="customerMapper"/>
+ </bean>
+
This configured ItemReader will return
CustomerCredit objects using the
@@ -1668,13 +1936,13 @@
is an example configuration using the same 'customer credit' example
as the JDBC reader above:
-
-
-
-
-
-]]>
+ <bean id="itemReader"
+ class="org.springframework.batch.item.database.JpaPagingItemReader">
+ <property name="entityManagerFactory" ref="entityManagerFactory"/>
+ <property name="queryString" value="select c from CustomerCredit c"/>
+ <property name="pageSize" value="1000"/>
+ </bean>
+
This configured ItemReader will return
CustomerCredit objects in the exact same manner
@@ -1698,23 +1966,23 @@
IbatisPagingItemReader reading CustomerCredits
as in the examples above:
-
-
-
-
-
-]]>
+ <bean id="itemReader"
+ class="org.springframework.batch.item.database.IbatisPagingItemReader">
+ <property name="sqlMapClient" ref="sqlMapClient"/>
+ <property name="queryId" value="getPagedCustomerCredits"/>
+ <property name="pageSize" value="1000"/>
+ </bean>
+
The IbatisPagingItemReader configuration
above references an IBATIS query called "getPagedCustomerCredits".
Here is an example of what that query should look like for
MySQL.
-
+ <select id="getPagedCustomerCredits" resultMap="customerCreditResult">
select id, name, credit from customer order by id asc LIMIT #_skiprows#, #_pagesize#
-
-]]>
+ </select>
+
The _skiprows and
_pagesize variables are provided by the
@@ -1735,278 +2003,6 @@
-
- Driving Query Based ItemReaders
-
- In the previous section, Cursor based database input was
- discussed. However, it isn't the only option. Many database vendors,
- such as DB2, have extremely pessimistic locking strategies that can
- cause issues if the table being read also needs to be used by other
- portions of the online application. Furthermore, opening cursors over
- extremely large datasets can cause issues on certain vendors. Therefore,
- many projects prefer to use a 'Driving Query' approach to reading in
- data. This approach works by iterating over keys, rather than the entire
- object that needs to be returned, as the following example
- illustrates:
-
-
-
-
-
-
-
-
-
-
-
- As you can see, this example uses the same 'FOO' table as was used
- in the cursor based example. However, rather than selecting the entire
- row, only the ID's were selected in the SQL statement. So, rather than a
- FOO object being returned from read, an Integer
- will be returned. This number can then be used to query for the
- 'details', which is a complete Foo object:
-
-
-
-
-
-
-
-
-
-
-
- As you can see, an existing DAO can be used to obtain a full 'Foo'
- object using the key obtained from the driving query. In Spring Batch,
- driving query style input is implemented with a
- DrivingQueryItemReader, which has only one
- dependency: a KeyCollector
-
-
- KeyCollector
-
- As the previous example illustrates, the DrivingQueryItemReader
- is fairly simple. It simply iterates over a list of keys. However, the
- real complication is how those keys are obtained. The
- KeyCollector interface abstracts this:
-
-
-
- The primary method in this interface is the
- retrieveKeys method. It is expected that this
- method will return the keys to be processed regardless of whether or
- not it is a restart scenario. For example, if a job starts processing
- keys 1 through 1,000, and fails after processing key 500, upon
- restarting keys 500 through 1,000 should be returned. This
- functionality is made possible by the
- updateContext method, which saves the
- provided key (which should be the current key being processed) in the
- provided ExecutionContext. The
- retrieveKeys method can then use this value
- to retrieve a subset of the original keys:
-
-
-
- This generalization illustrates the
- KeyCollector contract. If we assume that
- initially calling retrieveKeys returned 1,000
- keys (1 through 1,000), calling updateContext
- with key 500 should mean that calling
- retrieveKeys again with the same
- ExecutionContext will return 500 keys (501
- through 1,000).
-
-
-
- SingleColumnJdbcKeyCollector
-
- The most common driving query scenario is that of input that has
- only one column that represents its key. This is implemented as the
- SingleColumnJdbcKeyCollector class, which has
- the following options:
-
-
- SinglecolumnJdbcKeyCollector properties
-
-
-
-
- jdbcTemplate
-
- The JdbcTemplate to be used to query the
- database
-
-
-
- sql
-
- The sql statement to query the database with. It should
- return only one value.
-
-
-
- restartSql
-
- The sql statement to use in the case of restart.
- Because only one key will be used, this query should require
- only one argument.
-
-
-
- keyMapper
-
- The RowMapper implementation to be used to map the keys
- to objects. By default, this is a Spring Core
- SingleColumnRowMapper, which maps them to well known types
- such as Integer, String, etc. For more information, check the
- documentation of your specific Spring release.
-
-
-
-
-
- The following code helps illustrate how to setup and use a
- SingleColumnJdbcKeyCollector:
-
- ? order by ID");
-
- ExecutionContext executionContext = new ExecutionContext();
-
- List keys = keyStrategy.retrieveKeys(new ExecutionContext());
-
- for (int i = 0; i < keys.size(); i++) {
- System.out.println(keys.get(i));
- }]]>
-
- If this code were run in the proper environment with the correct
- database tables setup, then it would output the following:
-
-
-
- Now, let's modify the code slightly to show what would happen if
- the code were started again after a restart, having failed after
- processing key 3 successfully:
-
- ? order by ID");
-
- ExecutionContext executionContext = new ExecutionContext();
-
- keyStrategy.updateContext(new Long(3), executionContext);
-
- List keys = keyStrategy.retrieveKeys(executionContext);
-
- for (int i = 0; i < keys.size(); i++) {
- System.out.println(keys.get(i));
- }]]>
-
- Running this code snippet would result in the following:
-
-
-
- The key difference between the two examples is the following
- line:
-
-
-
- This tells the key collector to update the provided
- ExecutionContext with the key of three. This
- will normally be called by the
- DrivingQueryItemReader, but is called directly
- for simplicities sake. By calling
- retrieveKeys with the
- ExecutionContext that was updated to contain 3,
- the argument of 3 will be passed to the restartSql:
-
- ? order by ID");]]>
-
- This will cause only keys 4 and 5 to be returned, since they are
- the only ones with an ID greater than 3.
-
-
-
- Mapping multiple column keys
-
- The SingleColumnJdbcKeyCollector is
- extremely useful for generating keys, but only if one column uniquely
- identifies your record. What if more than one column is required to be
- able to uniquely identify your record? This should be a minority
- scenario, but it is still possible. In this case, the
- MultipleColumnJdbcKeyCollector should be used.
- It allows for mapping multiple columns by sacrificing simplicity. The
- properties needed to use the multiple column collector are the same as
- the single column version except one difference: instead of a regular
- RowMaper, an
- ExecutionContextRowMapper must be provided.
- Just like the single column version, it requires a normal SQL
- statement and a restart SQL statement. However, because the restart
- SQL statement will require more than one argument, there needs to be
- more complex handling of how keys are mapped to an execution context.
- An ExecutionContextRowMapper provides
- this:
-
-
-
- The ExecutionContextRowMapper interface
- extends the standard RowMapper interface to
- allow for multiple keys to be stored in an
- ExecutionContext, and a
- PreparedStatementSetter be created so that
- arguments to a the restart SQL statement can be set for the key
- returned.
-
- By default a implementation of the
- ExecutionContextRowMapper that uses a
- Map will be used. It is recommended that this
- implementation not be overridden. However, if a specific type of key
- needs to be returned, then a new implementation can be
- provided.
-
-
-
- iBatisKeyCollector
-
- Jdbc is not the only option available for key collectors, iBatis
- can be used as well. The usage of iBatis doesn't change the basic
- requirements of a KeyCollector: query, restart
- query, and DataSource. However, because iBatis
- is used, both queries are simply iBatis query ids, and the data source
- is a SqlMapClient.
-
-
-
Database ItemWriters
@@ -2027,7 +2023,7 @@
output doesn't have any inherent flaws, assuming we are careful to flush
and there are no errors in the data. However, any errors while writing
out can cause confusion because there is no way to know which individual
- item caused an exception, or even ideed if any individual item was
+ item caused an exception, or even if any individual item was
responsible, as illustrated below:
@@ -2098,33 +2094,33 @@
standard Spring method invoking delegator pattern and are fairly simple to
set up. Below is an example of the reader:
-
-
-
-
+ <bean id="itemReader" class="org.springframework.batch.item.adapter.ItemReaderAdapter">
+ <property name="targetObject" ref="fooService" />
+ <property name="targetMethod" value="generateFoo" />
+ </bean>
- ]]>
+ <bean id="fooService" class="org.springframework.batch.item.sample.FooService" />
One important point to note is that the contract of the targetMethod
must be the same as the contract for read: when
exhausted it will return null, otherwise an Object.
- Anything else will prevent the framework from correctly knowing when
- processing should end, either causing an infinite loop or incorrect
- failure, depending upon the implementation of the
+ Anything else will prevent the framework from knowing when processing
+ should end, either causing an infinite loop or incorrect failure,
+ depending upon the implementation of the
ItemWriter. The ItemWriter
implementation is equally as simple:
-
-
-
-
+ <bean id="itemWriter" class="org.springframework.batch.item.adapter.ItemWriterAdapter">
+ <property name="targetObject" ref="fooService" />
+ <property name="targetMethod" value="processFoo" />
+ </bean>
-
-]]>
+ <bean id="fooService" class="org.springframework.batch.item.sample.FooService" />
+
- Item Transforming
+ ItemProcessor
The ItemReader and
ItemWriter interfaces have been discussed in detail
@@ -2135,141 +2131,145 @@
that contains another ItemReader. For
example:
- implements ItemWriter {
+ public class CompositeItemWriter<T> implements ItemWriter<T> {
- ItemWriter itemWriter;
+ ItemWriter<T> itemWriter;
- public CompositeItemWriter(ItemWriter itemWriter) {
+ public CompositeItemWriter(ItemWriter<T> itemWriter) {
this.itemWriter = itemWriter;
}
- public void write(T item) throws Exception {
+ public void write(List<? extends T> items) throws Exception {
//Add business logic here
itemWriter.write(item);
}
- public void clear() throws ClearFailedException {
- itemWriter.clear();
+ public void setDelegate(ItemWriter<T> itemWriter){
+ this.itemWriter = itemWriter;
}
-
- public void flush() throws FlushFailedException {
- itemWriter.flush();
- }
-}]]>
+}
The class above contains another ItemWriter
- that it delgates to after having provided some business logic. It should
- be noted that the clear and
- flush methods must be propogated as well so that
- the delegate ItemWriter is notified. This pattern
- could easily be used for an ItemReader as well,
- perhaps to obtain more reference data based upon the input that was
+ that it delgates to after having provided some business logic. This
+ pattern could easily be used for an ItemReader as
+ well, perhaps to obtain more reference data based upon the input that was
provided by the main ItemReader. This pattern is
very useful if you need to control the call to
write yourself. However, if you only want to
'transform' the item passed in for writing before it is actual written,
there isn't much need to call write yourself, you
just want to modify the item. For this scenario, Spring Batch provides the
- ItemTransformer interface:
+ ItemProcessor interface:
- {
+ public interface ItemProcessor<I, O> {
O process(I item) throws Exception;
-}]]>
+}
- An ItemTransformer is very simple, given one
+ An ItemProcessor is very simple, given one
object, transorm it and return another. The object provided may or may not
be of the same type. The point is that business logic may be applied
- within transform, and is completely up to the developer to create. An
- ItemTransformer is used as part of the
- ItemTransformerItemWriter, which accepts an
- ItemWriter and an
- ItemTransformer, passing the item first to the
- transformer, before writing it. For example, assuming an
- ItemReader provides a class of type Foo, and it
- needs to be converted to type Bar before being written out. An
- ItemTransformer can be written that performs the
- conversion:
+ within process, and is completely up to the developer to create. An
+ ItemProcessor can be wired direclty into a step,
+ For example, assuming an ItemReader provides a
+ class of type Foo, and it needs to be converted to type Bar before being
+ written out. An ItemTransformer can be written that
+ performs the conversion:
- public class Foo {}
public class Bar {
public Bar(Foo foo) {}
}
- public class FooTransformer implements ItemProcessor{
+ public class FooProcessor implements ItemProcessor<Foo,Bar>{
//Perform simple transformation, convert a Foo to a Bar
- public Object transform(Object item) throws Exception {
- assertTrue(item instanceof Foo);
- Foo foo = (Foo)item;
+ public Bar transform(Foo foo) throws Exception {
return new Bar(foo);
}
}
- public class BarWriter implements ItemWriter{
+ public class BarWriter implements ItemWriter<Bar>{
- public void write(Object item) throws Exception {
- assertTrue(item instanceof Bar);
+ public void write(Bar bar) throws Exception {
+ //write bar
}
//rest of class ommitted for clarity
- }]]>
+ }
In the very simple example above, there is a class
Foo, a class Bar, and a
- class FooTransformer that adheres to the
- ItemTransformer interface. The transformation is
+ class FooProcessor that adheres to the
+ ItemProcessor interface. The transformation is
simple, but any type of transformation could be done here. The
BarWriter will be used to write out 'Bars',
throwing an exception if any other type is provided. Similarly, the
- FooTransformer will throw an exception if anything but a
- Foo is provided. An
- ItemTransformerItemWriter can then be used like a
- normal ItemWriter. It will be passed a Foo for
- writing, which will be passed to the transformer, and a
- Bar returned. The resulting
- Bar will then be written:
+ FooProcessor will throw an exception if anything but a
+ Foo is provided. The
+ FooProcessor can then be injected into a
+ Step:
-
+
+ <job id="ioSampleJob">
+ <step name="step1">
+ <tasklet reader="fooReader" processor="fooProcessor" writer="barWriter" commit-interval="2"/>
+ </step>
+ </job>
+
+
The Delegate Pattern and Registering with the Step
- Note that the ItemTransformerItemWriter and
- the CompositeItemWriter are examples of a
- delegation pattern, which is common in Spring Batch. The delegates
- themselves might implement callback interfaces like
+ Note that the CompositeItemWriter is an
+ example of the delegation pattern, which is common in Spring Batch. The
+ delegates themselves might implement callback interfaces like
ItemStream or
StepListener. If they do, and they are being used
in conjunction with Spring Batch Core as part of a
Step in a Job, then they
almost certainly need to be registered manually with the
- Step. Registration is automatic when using the
- factory beans (*StepFactoryBean) , but only for
- the ItemReader and
- ItemWriter injected directly. The delegates are
+ Step. Registration is automatic when the reader,
+ writer, or processor is directly wired into the Step. The delegates are
not known to the Step, so they need to be
- injected as listeners or streams (or both if appropriate).
+ injected as listeners or streams (or both if appropriate):
+
+
+ <job id="ioSampleJob">
+ <step name="step1">
+ <tasklet reader="fooReader" processor="fooProcessor" writer="compositeItemWriter" commit-interval="2"/>
+ <streams>
+ <stream ref="barWriter" />
+ </streams>
+ </step>
+ </job>
+
+ <bean id="compositeItemWriter"
+ class="...CompositeItemWriter" >
+ <property name="delegate" ref="barWriter" />
+ </bean>
+
+ <bean id="barWriter" class="...BarWriter" />
+
+
- Chaining ItemTransformers
+ Chaining ItemProcessors
Performing a single transformation is useful in many scenarios,
- but what if you want to 'chain' together multiple ItemTransformers? This
- can be accomplished using a
- CompositeItemTransformer. To update the previous,
- single transformation, example, Foo will be
- Transformed to Bar, which will be transformed to
+ but what if you want to 'chain' together multiple ItemProcessors? This
+ can be accomplished using the composite pattern mentioned previously. To
+ update the previous, single transformation, example,
+ Foo will be Transformed to
+ Bar, which will be transformed to
Foobar and written out:
- public class Foo {}
public class Bar {
public Bar(Foo foo) {}
@@ -2279,54 +2279,61 @@
public Foobar(Bar bar){}
}
- public class FooTransformer implements ItemTransformer{
+ public class FooProcessor implements ItemProcessor<Foo,Bar>{
//Perform simple transformation, convert a Foo to a Bar
- public Object transform(Object item) throws Exception {
- assertTrue(item instanceof Foo);
- Foo foo = (Foo)item;
+ public Bar transform(Foo foo) throws Exception {
return new Bar(foo);
}
}
- public class BarTransformer implements ItemTransformer{
+ public class BarProcessor implements ItemProcessor<Bar,FooBar>{
- public Object transform(Object item) throws Exception {
- assertTrue(item instanceof Bar);
- return new Foobar((Bar)item);
+ public FooBar transform(Bar bar) throws Exception {
+ return new Foobar(bar);
}
}
- public class FoobarWriter implements ItemWriter{
+ public class FoobarWriter implements ItemWriter<FooBar>{
public void write(Object item) throws Exception {
- assertTrue(item instanceof Foobar);
+ //write Foobar
}
//rest of class ommitted for clarity
- }]]>
+ }
A FooTransformer and
BarTransformer can be 'chained' together to give
the resultant Foobar:
-
+ CompositeItemProcessor<Foo,Foobar> compositeProcessor = new CompositeItemProcessor<Foo,Foobar>();
+ List itemProcessors = new ArrayList();
+ itemProcessors.add(new FooTransformer());
+ itemProcessors.add(new BarTransformer());
+ compositeProcessor.setItemProcessors(itemProcessors);
- The compositeTransformer could be said to accept a
- Foo and return a Foobar.
- Clients of the composite transformer don't need to know that there are
- actually two separate transformations taking place. By updating the
- example from above to use the composite transformer, the correct class
- can be passed to FoobarWriter:
+ Just as with the previous example, the compsite processor can be
+ configured into the Step:
- ItemTransformerItemWriter itemTransformerItemWriter = new ItemTransformerItemWriter();
- itemTransformerItemWriter.setItemTransformer(compositeTransformer);
- itemTransformerItemWriter.setDelegate(new FoobarWriter());
- itemTransformerItemWriter.write(new Foo());
+
+ <job id="ioSampleJob">
+ <step name="step1">
+ <tasklet reader="fooReader" processor="compositeProcessor" writer="foobarWriter" commit-interval="2"/>
+ </step>
+ </job>
+
+ <bean id="compositeItemProcessor"
+ class="org.springframework.batch.item.support.CompositeItemProcessor">
+ <property name="itemProcessors">
+ <list>
+ <bean class="..FooProcessor" />
+ <bean class="..BarProcessor" />
+ </list>
+ </property>
+ </bean>
+
+
@@ -2334,8 +2341,8 @@
Validating Input
During the course of this chapter, multiple approaches to parsing
- input have been discussed. Each major implementation will throw exception
- if it is not 'well-formed'. The
+ input have been discussed. Each major implementation will throw an
+ exception if it is not 'well-formed'. The
FixedLengthTokenizer will throw an exception if a
range of data is missing. Similarly, attempting to access an index in a
RowMapper of FieldSetMapper
@@ -2350,22 +2357,18 @@
rather provides a very simple interface that can be implemented by any
number of frameworks:
- public interface Validator {
void validate(Object value) throws ValidationException;
- }]]>
+ }
The contract is that the validate method
will throw an exception if the object is invalid, and return normally if
it is valid. Spring Batch provides an out of the box
- ItemReader that delegates to another
- ItemReader and validates the returned item:
+ ItemProcessor:
- <bean class="org.springframework.batch.item.validator.ValidatingItemReader">
- <property name="itemReader">
- <bean class="org.springframework.batch.sample.item.reader.OrderItemReader" />
- </property>
+ <bean class="org.springframework.batch.item.validator.ValidatingItemProcessor">
<property name="validator" ref="validator" />
</bean>
@@ -2393,24 +2396,6 @@
ValangValidator that is used to validate an order
object. The intent is not to show Valang functionality as much as to show
how a validator could be added.
-
-
- The Delegate Pattern and Registering with the Step
-
- Note that the ValidatingItemReader is
- another example of a delegation pattern, and the delegates themselves
- might implement callback interfaces like
- ItemStream or
- StepListener. If they do, and they are being used
- in conjunction with Spring Batch Core as part of a step in a job, then
- they almost certainly need to be registered manually with the
- Step. Registration is automatic when using the
- factory beans (*StepFactoryBean) , but only for
- the ItemReader and
- ItemWriter injected directly - the delegates are
- not known to the step, so they need to be injected as listeners or
- streams (or both if appropriate).
-
@@ -2483,11 +2468,11 @@
basic contract of ItemReader,
read:
- implements ItemReader{
+ public class CustomItemReader<T> implements ItemReader<T>{
- List items;
+ List<T> items;
- public CustomItemReader(List items) {
+ public CustomItemReader(List<T> items) {
this.items = items;
}
@@ -2499,23 +2484,23 @@
}
return null;
}
- }]]>
+ }
This very simple class takes a list of items, and returns one at a
time, removing it from the list. When the list empty, it returns null,
thus satisfying the most basic requirements of an
ItemReader, as illustrated below:
- items = new ArrayList();
+ List<String> items = new ArrayList<String>();
items.add("1");
items.add("2");
items.add("3");
- ItemReader itemReader = new CustomItemReader(items);
+ ItemReader itemReader = new CustomItemReader<String>(items);
assertEquals("1", itemReader.read());
assertEquals("2", itemReader.read());
assertEquals("3", itemReader.read());
- assertNull(itemReader.read());]]>
+ assertNull(itemReader.read());
Making the <classname>ItemReader</classname>
@@ -2537,20 +2522,20 @@
implemented with the <classname>ItemStream</classname>
interface:</para>
- <programlisting><![CDATA[ public class CustomItemReader<T> implements ItemReader<T>, ItemStream {
+ <programlisting> public class CustomItemReader<T> implements ItemReader<T>, ItemStream {
- List<T> items;
+ List<T> items;
int currentIndex = 0;
private static final String CURRENT_INDEX = "current.index";
- public CustomItemReader(List<T> items) {
+ public CustomItemReader(List<T> items) {
this.items = items;
}
public T read() throws Exception, UnexpectedInputException,
ParseException {
- if (currentIndex < items.size()) {
+ if (currentIndex < items.size()) {
return items.get(currentIndex++);
}
@@ -2571,7 +2556,7 @@
public void update(ExecutionContext executionContext) throws ItemStreamException {
executionContext.putLong(CURRENT_INDEX, new Long(currentIndex).longValue());
};
- }]]></programlisting>
+ }</programlisting>
<para>On each call to <classname>ItemStream</classname>
<methodname>update</methodname> method, the current index of the
@@ -2583,24 +2568,23 @@
current index is moved to that location. This is a fairly trivial
example, but it still meets the general contract:</para>
- <programlisting><![CDATA[ ExecutionContext executionContext = new ExecutionContext();
+ <programlisting> ExecutionContext executionContext = new ExecutionContext();
((ItemStream)itemReader).open(executionContext);
assertEquals("1", itemReader.read());
((ItemStream)itemReader).update(executionContext);
- List<String> items = new ArrayList<String>();
+ List<String> items = new ArrayList<String>();
items.add("1");
items.add("2");
items.add("3");
- itemReader = new CustomItemReader<String>(items);
+ itemReader = new CustomItemReader<String>(items);
((ItemStream)itemReader).open(executionContext);
- assertEquals("2", itemReader.read());]]></programlisting>
+ assertEquals("2", itemReader.read());</programlisting>
<para>Most ItemReaders have much more sophisticated restart logic. The
- <classname>DrivingQueryItemReader</classname>, for example, only loads
- up the remaining keys to be processed, rather than loading all of them
- and then moving to the correct index.</para>
+ <classname>JdbcCursorItemReader</classname>, for example, stores the
+ row id of the last processed row in the Cursor.</para>
<para>It is also worth noting that the key used within the
<classname>ExecutionContext</classname> should not be trivial. That is
@@ -2626,18 +2610,18 @@
example. As with the <classname>ItemReader</classname> example, a List
will be used in order to keep the example as simple as possible:</para>
- <programlisting><![CDATA[ public class CustomItemWriter<T> implements ItemWriter<T> {
+ <programlisting> public class CustomItemWriter<T> implements ItemWriter<T> {
- List<T> output = TransactionAwareProxyFactory.createTransactionalList();
+ List<T> output = TransactionAwareProxyFactory.createTransactionalList();
- public void write(List<? extends T> items) throws Exception {
+ public void write(List<? extends T> items) throws Exception {
output.addAll(items);
}
- public List<T> getOutput() {
+ public List<T> getOutput() {
return output;
}
- }]]></programlisting>
+ }</programlisting>
<section>
<title>Making the <classname>ItemWriter</classname>
diff --git a/docs/src/site/docbook/reference/schema-appendix.xml b/docs/src/site/docbook/reference/schema-appendix.xml
index 36f1263f2..ace579cfd 100644
--- a/docs/src/site/docbook/reference/schema-appendix.xml
+++ b/docs/src/site/docbook/reference/schema-appendix.xml
@@ -8,10 +8,13 @@
<title>Overview
The Spring Batch Meta-Data tables very closely match the Domain
- objects that represent them in Java. For example, JobInstance,
- JobExecution, JobParameters, StepExecution, and ExecutionContext map to
- BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, BATCH_JOB_PARAMS,
- BATCH_STEP_EXECUTION, BATCH_STEP_EXECUTION_CONTEXT, respectively. The
+ objects that represent them in Java. For example,
+ JobInstance, JobExecution,
+ JobParameters, and
+ StepExecution map to BATCH_JOB_INSTANCE,
+ BATCH_JOB_EXECUTION, BATCH_JOB_PARAMS, and BATCH_STEP_EXECUTION,
+ respectively. ExecutionContext maps to both
+ BATCH_JOB_EXECUTION_CONTEXT and BATCH_STEP_EXECUTION_CONTEXT. The
JobRepository is responsible for saving and storing
each Java object into it's correct table. The following appendix describes
the meta-data tables in detail, along with many of the design decisions
@@ -19,7 +22,7 @@
statements below, it is important to realize that the data types used are
as generic as possible. Spring Batch provides many schemas as examples,
which all have varying data types due to variations in individual database
- vendors' handling of data types. Below is an ERD model of all 5 tables and
+ vendors' handling of data types. Below is an ERD model of all 6 tables and
their relationships to one another:
@@ -122,7 +125,9 @@ INSERT INTO BATCH_JOB_SEQ values(0);
JOB_KEY: A serialization of the
JobParameters that uniquely identifies separate
instances of the same job from one another.
- (JobInstances with the same job name
+ (JobInstances with the same job name must have
+ different JobParameters, and thus, different
+ JOB_KEY values).
@@ -131,11 +136,12 @@ INSERT INTO BATCH_JOB_SEQ values(0);
BATCH_JOB_PARAMS
The BATCH_JOB_PARAMS table holds all information relevant to the
- JobParameters object. It contains 0 or more key/value pairs that together
- uniquely identify a JobInstance and serve as a
- record of the parameters a job was run with. It should be noted that the
- table has been denormalized. Rather than creating a separate table for
- each type, there is one table with a column indicating the type:
+ JobParameters object. It contains 0 or more
+ key/value pairs that together uniquely identify a
+ JobInstance and serve as a record of the parameters
+ a job was run with. It should be noted that the table has been
+ denormalized. Rather than creating a separate table for each type, there
+ is one table with a column indicating the type:
CREATE TABLE BATCH_JOB_PARAMS (
JOB_INSTANCE_ID BIGINT NOT NULL ,
@@ -204,12 +210,13 @@ INSERT INTO BATCH_JOB_SEQ values(0);
JOB_EXECUTION_ID BIGINT PRIMARY KEY ,
VERSION BIGINT,
JOB_INSTANCE_ID BIGINT NOT NULL,
+ CREATE_TIME TIMESTAMP NOT NULL,
START_TIME TIMESTAMP DEFAULT NULL,
END_TIME TIMESTAMP DEFAULT NULL,
STATUS VARCHAR(10),
- CONTINUABLE CHAR(1),
EXIT_CODE VARCHAR(20),
EXIT_MESSAGE VARCHAR(2500),
+ LAST_UPDATED TIMESTAMP,
constraint JOB_INSTANCE_EXECUTION_FK foreign key (JOB_INSTANCE_ID)
references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID)
) ;
@@ -234,6 +241,11 @@ INSERT INTO BATCH_JOB_SEQ values(0);
more than one execution per instance.
+
+ CREATE_TIME: Timestamp representing the time that the execution
+ was created.
+
+
START_TIME: Timestamp representing the time the execution was
started.
@@ -254,11 +266,6 @@ INSERT INTO BATCH_JOB_SEQ values(0);
BatchStatus enumeration.
-
- CONTINUABLE: Character indicating whether or not the execution
- is currently able to continue. 'Y' for yes and 'N' for no.
-
-
EXIT_CODE: Character string representing the exit code of the
execution. In the case of a command line job, this may be converted
@@ -270,6 +277,11 @@ INSERT INTO BATCH_JOB_SEQ values(0);
description of how the job exited. In the case of failure, this might
include as much of the stack trace as is possible.
+
+
+ LAST_UPDATED: Timestamp representing the last time this
+ execution was persisted.
+
@@ -291,10 +303,16 @@ INSERT INTO BATCH_JOB_SEQ values(0);
END_TIME TIMESTAMP DEFAULT NULL,
STATUS VARCHAR(10),
COMMIT_COUNT BIGINT ,
- ITEM_COUNT BIGINT ,
- CONTINUABLE CHAR(1),
- EXIT_CODE VARCHAR(20),
- EXIT_MESSAGE VARCHAR(2500),
+ READ_COUNT BIGINT ,
+ FILTER_COUNT BIGINT ,
+ WRITE_COUNT BIGINT ,
+ READ_SKIP_COUNT BIGINT ,
+ WRITE_SKIP_COUNT BIGINT ,
+ PROCESS_SKIP_COUNT BIGINT ,
+ ROLLBACK_COUNT BIGINT ,
+ EXIT_CODE VARCHAR(20) ,
+ EXIT_MESSAGE VARCHAR(2500) ,
+ LAST_UPDATED TIMESTAMP,
constraint JOB_EXECUTION_STEP_FK foreign key (JOB_EXECUTION_ID)
references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID)
) ;
@@ -352,13 +370,38 @@ INSERT INTO BATCH_JOB_SEQ values(0);
- ITEM_COUNT: The number of items that have been written out
- during this execution.
+ READ_COUNT: The number of items read during this
+ execution.
- CONTINUABLE: Character indicating whether or not the execution
- is currently able to continue. 'Y' for yes and 'N' for no.
+ FILTER_COUNT: The number of items filtered out of this
+ execution.
+
+
+
+ WRITE_COUNT: The number of items written during this
+ execution.
+
+
+
+ READ_SKIP_COUNT: The number of items skipped on read during this
+ execution.
+
+
+
+ WRITE_SKIP_COUNT: The number of items skipped on write during
+ this execution.
+
+
+
+ PROCESS_SKIP_COUNT: The number of items skipped during
+ processing during this execution.
+
+
+
+ ROLLBACK_COUNT: The number of rollbacks during this
+ execution.
@@ -372,88 +415,94 @@ INSERT INTO BATCH_JOB_SEQ values(0);
description of how the job exited. In the case of failure, this might
include as much of the stack trace as is possible.
+
+
+ LAST_UPDATED: Timestamp representing the last time this
+ execution was persisted.
+
- BATCH_EXECUTION_CONTEXT
+ BATCH_JOB_EXECUTION_CONTEXT
- The BATCH_STEP_EXECUTION_CONTEXT table holds all information
- relevant to an ExecutionContext. There is exactly
- one ExecutionContext per
- StepExecution, and it contains all user defined
- key/value pairs that need to persisted for a particular job run. This data
- is typically state that must be retrieved back after a failure so that a
- JobInstance can 'start from where it left off'. As
- with the BATCH_JOB_PARAMS table, this table has been denormalized and uses
- a column to determine the type:
+ The BATCH_JOB_EXECUTION_CONTEXT table holds all information relevant
+ to an Job's
+ ExecutionContext. There is exactly one
+ ExecutionContext per
+ StepExecution, and it contains all of the job-level
+ data that is needed for a particular job execution. This data typically
+ represents the state that must be retrieved after a failure so that a
+ JobInstance can 'start from where it left
+ off'.
- CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT (
- EXECUTION_ID BIGINT NOT NULL ,
- DISCRIMINATOR VARCHAR2(1) NOT NULL,
- TYPE_CD VARCHAR(6) NOT NULL ,
- KEY_NAME VARCHAR(1000) NOT NULL ,
- STRING_VAL VARCHAR(1000) ,
- DATE_VAL TIMESTAMP DEFAULT NULL ,
- LONG_VAL VARCHAR(10) ,
- DOUBLE_VAL DOUBLE PRECISION ,
- OBJECT_VAL BLOB,
+ CREATE TABLE BATCH_JOB_EXECUTION_CONTEXT (
+ JOB_EXECUTION_ID BIGINT PRIMARY KEY,
+ SHORT_CONTEXT VARCHAR(2500) NOT NULL,
+ SERIALIZED_CONTEXT CLOB,
+ constraint JOB_EXEC_CTX_FK foreign key (JOB_EXECUTION_ID)
+ references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID)
) ;
Below are descriptions for each column:
- EXECUTION_ID: Foreign key representing the
- StepExecution or
+ JOB_EXECUTION_ID: Foreign key representing the
JobExecution to which the context belongs.
There may be more than one row associated to a given execution.
- DISCRIMINATOR: Character indicating whether or not the entry is
- job or step scoped. (i.e. does it belong to the JobExecution or
- StepExecution)
+ SHORT_CONTEXT: A string version of the
+ SERIALIZED_CONTEXT.
- TYPE_CD: String representation of the type of value stored,
- which can be either a character string, date, long, or double. Because
- the type must be known, it cannot be null.
-
-
-
- KEY_NAME: The Parameter key.
-
-
-
- STRING_VAL: Parameter value, if the type is string.
-
-
-
- DATE_VAL: Parameter value, if the type is date.
-
-
-
- LONG_VAL: Parameter value, if the type is a long.
-
-
-
- DOUBLE_VAL: Parameter value, if the type is double.
-
-
-
- OBJECT_VAL: Parameter value, if the type is a blob.
+ SERIALIZED_CONTEXT: The entire context, serialized.
+
- When an ExecutionContext is stored, values that are one of the well
- known types above will be stored as their respective type. Any unknown
- type will be serialized to a blob and stored in the OBJECT_VAL column. As
- with BATCH_JOB_PARAMS, there is no primary key for this table. This is
- simply because the framework has no use for one, and thus doesn't require
- it. If a user so chooses, one may be added with a database generated key,
- without causing any issues to the framework itself.
+
+ BATCH_STEP_EXECUTION_CONTEXT
+
+ The BATCH_STEP_EXECUTION_CONTEXT table holds all information
+ relevant to an Step's
+ ExecutionContext. There is exactly one
+ ExecutionContext per
+ StepExecution, and it contains all of the data that
+ needs to persisted for a particular step execution. This data typically
+ represents the state that must be retrieved after a failure so that a
+ JobInstance can 'start from where it left
+ off'.
+
+ CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT (
+ STEP_EXECUTION_ID BIGINT PRIMARY KEY,
+ SHORT_CONTEXT VARCHAR(2500) NOT NULL,
+ SERIALIZED_CONTEXT CLOB,
+ constraint STEP_EXEC_CTX_FK foreign key (STEP_EXECUTION_ID)
+ references BATCH_STEP_EXECUTION(STEP_EXECUTION_ID)
+) ;
+
+ Below are descriptions for each column:
+
+
+
+ STEP_EXECUTION_ID: Foreign key representing the
+ StepExecution to which the context belongs.
+ There may be more than one row associated to a given execution.
+
+
+
+ SHORT_CONTEXT: A string version of the
+ SERIALIZED_CONTEXT.
+
+
+
+ SERIALIZED_CONTEXT: The entire context, serialized.
+
+
@@ -463,23 +512,28 @@ INSERT INTO BATCH_JOB_SEQ values(0);
is run, it is common to create an archive strategy for the meta-data
tables. The tables themselves are designed to show a record of what
happened in the past, and generally won't affect the run of any job, with
- a couple of notable exceptions:
+ a couple of notable exceptions pertaining to restart:
- Restart: Because the ExecutionContext is persisted, removing any
- entries from this table of jobs that haven't completed successfully,
- will prevent them from starting at the correct point if run again.
- Furthermore, if an entry for a JobInstance is removed without having
- completed successfully, the framework will think that the job is new,
- rather than a restart.
+ The framework will use the meta-data tables to determine if a
+ particular JobInstance has been run before. If it has been run, and
+ the job is not restartable, then an exception will be thrown.
- Determining if an instance has been run: The framework will use
- the meta-data tables to determine if a particular JobInstance has been
- run before, and if it has an exception will be thrown.
+ If an entry for a JobInstance is removed without having
+ completed successfully, the framework will think that the job is new,
+ rather than a restart.
+
+
+
+ If a job is restarted, the framework will use any data that has
+ been persisted to the ExecutionContext to restore the Job's state.
+ Therefore, removing any entries from this table for jobs that haven't
+ completed successfully will prevent them from starting at the correct
+ point if run again.
-
\ No newline at end of file
+
diff --git a/docs/src/site/docbook/reference/step.xml b/docs/src/site/docbook/reference/step.xml
index 1d89142d7..60b787113 100644
--- a/docs/src/site/docbook/reference/step.xml
+++ b/docs/src/site/docbook/reference/step.xml
@@ -1209,5 +1209,75 @@
+
+
+ Creating File Names at Runtime
+
+ Both the XML and Flat File examples above use the Spring
+ Resource abstraction to obtain the file to read
+ or write from. This works because Resource has a
+ getFile method, that returns a
+ java.io.File. Both XML and Flat File resources
+ can be configured using standard Spring constructs:
+
+ <bean id="flatFileItemReader"
+ class="org.springframework.batch.item.file.FlatFileItemReader">
+ <property name="resource"
+ value="file://outputs/20070122.testStream.CustomerReportStep.TEMP.txt" />
+ </bean>
+
+ The above Resource will load the file from
+ the file system, at the location specificied. Note that absolute
+ locations have to start with a double slash ("//"). In most spring
+ applications, this solution is good enough because the names of these
+ are known at compile time. However, in batch scenarios, the file name
+ may need to be determined at runtime as a parameter to the job. This
+ could be solved using '-D' parameters, i.e. a system property:
+
+ <bean id="flatFileItemReader"
+ class="org.springframework.batch.item.file.FlatFileItemReader">
+ <property name="resource" value="${input.file.name}" />
+</bean>
+
+ All that would be required for this solution to work would be a
+ system argument (-Dinput.file.name="file://file.txt"). (Note that
+ although a PropertyPlaceholderConfigurer can be
+ used here, it is not necessary if the system property is always set
+ because the ResourceEditor in Spring already
+ filters and does placeholder replacement on system properties.)
+
+ Often in a batch setting it is preferable to parameterize the file
+ name in the JobParameters of the job, instead of
+ through system properties, and access them that way. To allow for this,
+ Spring Batch provides the
+ StepExecutionResourceProxy. The proxy can use
+ either job name, step name, or any values from the
+ JobParameters, by surrounding them with %:
+
+ <bean id="inputFile"
+ class="org.springframework.batch.core.resource.StepExecutionResourceProxy" />
+ <property name="filePattern" value="//%JOB_NAME%/%STEP_NAME%/%file.name%" />
+ </bean>
+
+ Assuming a job name of 'fooJob', and a step name of 'fooStep', and
+ the key-value pair of 'file.name="fileName.txt"' is in the
+ JobParameters the job is started with, the
+ following filename will be passed as the
+ Resource:
+ "//fooJob/fooStep/fileName.txt". It should be noted
+ that in order for the proxy to have access to the
+ StepExecution, it must be registered as a
+ StepListener:
+
+ <bean id="fooStep" parent="abstractStep"
+ p:itemReader-ref="itemReader"
+ p:itemWriter-ref="itemWriter">
+ <property name="listeners" ref="inputFile" />
+ </bean>
+
+ The StepListener interface will be
+ discussed in more detail in Chapter 4. For now, it is sufficient to know
+ that the proxy must be registered.
+
\ No newline at end of file
diff --git a/docs/src/site/docbook/reference/testing.xml b/docs/src/site/docbook/reference/testing.xml
index fa586c5d8..4ec5fa4aa 100644
--- a/docs/src/site/docbook/reference/testing.xml
+++ b/docs/src/site/docbook/reference/testing.xml
@@ -9,180 +9,108 @@
documentation covers how to unit and integration test with Spring in great
detail, so it won't be repeated here. It is important, however, to think
about how to 'end to end' test a batch job, which is what this chapter will
- focus on.
+ focus on. The spring-batch-test project includes classes that will help
+ factillitate this end-to-end test approach.
+
+
+ Creating a Unit Test Class
+
+ In order for the unit test to run a batch job, the framework must
+ load the job's ApplicationContext. Two annotations are used to trigger
+ this:
+
+
+
+ @RunWith(SpringJUnit4ClassRunner.class):
+ Indicates that the class should use Spring's JUnit facilities
+
+
+
+ @ContextConfiguration(locations = {...}):
+ Indicates which xml files contain the ApplicationContext.
+
+
+
+
+ @RunWith(SpringJUnit4ClassRunner.class)
+ @ContextConfiguration(locations = { "/simple-job-launcher-context.xml", "/jobs/skipSampleJob.xml" })
+ public class SkipSampleFunctionalTests extends AbstractJobTests { ... }
+
+
+
End To End Testing Batch Jobs
'End To End' testing can be defined as testing the complete run of a
- batch job from beginning to end. If the job reads from a file, then writes
- into the database, this type of testing ensures that any preconditions are
- met (reference data, correct file, etc) and then runs the job, verifying
- afterwards that all records that should be in the database are present and
- correct. Below is an example from one of the Spring Batch sample jobs, the
- 'fixedLengthImportJob'. It reads from a flat file (in fixed length format)
- and loads the records into the database. The following unit test code
- assures it processes correctly:
+ batch job from beginning to end. This allows for a test that sets up a
+ test condition, executes the job, and verifies the end result.
- //fixed-length file is expected on input
- protected void validatePreConditions() throws Exception{
- BufferedReader reader = null;
- reader = new BufferedReader(new FileReader(fileLocator.getFile()));
- String line;
- while ((line = reader.readLine()) != null) {
- assertEquals(LINE_LENGTH, line.length());
- }
+ In the example below, the batch job reads from the database and
+ writes to a flat file. The test method begins by setting up the database
+ with test data. It clears the CUSTOMER table and then inserts 10 new
+ records. The test then launches the Job using the
+ launchJob() method. The
+ launchJob() method is provided by the
+ AbstractJobTests parent class. Also provided by the
+ super class is launchJob(JobParameters), which
+ allows the test to give particular parameters. The
+ launchJob() method returns the
+ JobExecution object which is useful for asserting
+ particular information about the Job run. In the
+ case below, the test verifies that the Job ended
+ with status "COMPLETED".
+
+
+ @RunWith(SpringJUnit4ClassRunner.class)
+ @ContextConfiguration(locations = { "/simple-job-launcher-context.xml", "/jobs/skipSampleJob.xml" })
+ public class SkipSampleFunctionalTests extends AbstractJobTests {
+
+ private SimpleJdbcTemplate simpleJdbcTemplate;
+
+ @Autowired
+ public void setDataSource(DataSource dataSource) {
+ this.simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
+ }
+
+ @Transactional
+ @Test
+ public void testJob() throws Exception {
+ simpleJdbcTemplate.update("delete from CUSTOMER");
+ for (int i = 1; i <= 10; i++) {
+ simpleJdbcTemplate.update("insert into CUSTOMER values (?, 0, ?, 100000)", i, "customer" + i);
+ }
+
+ JobExecution jobExecution = this.launchJob();
+
+ Assert.assertEquals("COMPLETED", jobExecution.getExitStatus());
+ }
}
- //Check that records have been correctly written to database
- protected void validatePostConditions() throws Exception {
-
- inputSource.open(new ExecutionContext());
-
- jdbcTemplate.query("SELECT ID, ISIN, QUANTITY, PRICE, CUSTOMER FROM trade ORDER BY id",
- new RowCallbackHandler() {
-
- public void processRow(ResultSet rs) throws SQLException {
- Trade trade;
- try {
- trade = (Trade)inputSource.read();
- }
- catch (Exception e) {
- throw new IllegalStateException(e.getMessage());
- }
- assertEquals(trade.getIsin(), rs.getString(2));
- assertEquals(trade.getQuantity(),rs.getLong(3));
- assertEquals(trade.getPrice(), rs.getBigDecimal(4));
- assertEquals(trade.getCustomer(), rs.getString(5));
- }});
-
- assertNull(inputSource.read());
- }
-
- In the first method, validatePreConditions,
- the input file is checked to ensure it is correctly formatted. Because it
- is common to add extra lines to the file to test additional use cases,
- this test ensures that the fixed length lines are the length they should
- be. If they are not, it is much preferred to fail in this phase, rather
- than the job (correctly) failing during the run and causing needless
- debugging.
-
- In the second method, validatePostconditions, the database is
- checked to ensure all data has been written correctly. This is arguably
- the most important part of the test. In this case, it reads one line from
- the file, and one row from the database, and checks each column one by one
- for accuracy. It's important to not hard-code the data that should be
- present in the database into the test class. Instead, use the input file
- (bypassing the job) to check the output. This allows you to quickly add
- additional test cases to your file without having to add them to code. The
- same would be true for database to database jobs, or database to file
- jobs. It is preferable to be able to add additional rows to the database
- input without having to add them to the hard coded list in the test
- class.
+
- Extending Unit Test frameworks
+ Validating Output Files
- Because most unit testing of complete batch jobs will take place in
- the development environment (i.e. eclipse) it's important to be able to
- launch these tests in the same way you would launch any unit test. In the
- following examples JUnit 4 will be used, but any testing framework could
- be substituted. The Spring Batch samples contain many 'sample jobs' that
- are unit tested using this technique. The most important step is being
- able to launch the job within a unit test. This requires the use of the
- JobLauncher interface that is discussed in chapters
- 2 and 4. A Job and
- JobLauncher must be obtained from an
- ApplicationContext, and then launched. The
- following abstract class from Spring Batch Samples illustrates
- this:
+ When a batch job writes to the database, it is easy to query the
+ database to verify that the output is as expected. However, if the batch
+ job writes to a file, it is equally important that the output be verified.
+ Spring Batch provides a class AssertFile to
+ facilitate the verification of output files. The method
+ assertFileEquals takes two
+ File objects (or two
+ Resource objects) and asserts, line by line, that
+ the two files have the same content. Therefore, it is possible to create a
+ file with the expected output and to compare it to the actual
+ result:
- public abstract class AbstractBatchLauncherTests implements ApplicationContextAware {
+
+ private static final String EXPECTED_FILE = "src/main/resources/data/iosample/input/multiLine.txt";
+ private static final String OUTPUT_FILE = "target/test-outputs/multiLineOutput.txt";
- JobLauncher launcher;
- private Job job;
- private JobParameters jobParameters = new JobParameters();
+ AssertFile.assertFileEquals(new FileSystemResource(EXPECTED_FILE), new FileSystemResource(OUTPUT_FILE));
- @Test
- public void testLaunchJob() throws Exception {
- launcher.run(job, jobParameters);
- }
-
- @Autowired
- public void setLauncher(JobLauncher bootstrap) {
- this.launcher = bootstrap;
- }
-
- @Autowired
- public void setJob(Job job) {
- this.job = job;
- }
-}
-
-
- Few additional convenience properties are left out from the real class definition for clarity.
-
-
- Only two classes
- are needed: The Job to be run, and the JobLauncher to run it. These properties
- are declared to be autowired from the job's application context . Empty
- JobParameters are used in the example above.
- However, if the job requires specific parameters they could be coded in
- subclasses with an abstract method, or using a factory bean in the
- ApplicationContext for testing purposes. Because
- none of the sample jobs require this, an empty
- JobParameters is used. One simple JUnit test case
- is present in the file, which actually launches the job. If any exceptions
- are thrown or assertions fail, it will act the same way as any other unit
- test and display as a failed test due to errors or assertion failure.
- Because of the best practice for validation mentioned earlier in the
- chapter, this class is extended further to allow for separate validation
- before and after the job is run:
-
-
- public abstract class AbstractValidatingBatchLauncherTests extends AbstractBatchLauncherTests {
-
- @Test
- public void testLaunchJob() throws Exception {
- validatePreConditions();
- super.testLaunchJob();
- validatePostConditions();
- }
-
- /**
- * Make sure input data meets expectations
- */
- protected void validatePreConditions() throws Exception {}
-
- /**
- * Make sure job did what it was expected to do.
- */
- protected abstract void validatePostConditions() throws Exception;
-
-}
-
-
- In the class above, the testLaunchJob
- method is overridden to call the two abstract methods for validation.
- Before actually running the job,
- validatePreConditions is called (it should be
- noted that it's not required), and then after the job completes
- successfully, validatePostConidtions is
- called.
-
- Finally to create an executable test the abstract superclass needs to be subclassed.
- Spring-specific annotations ensure the appropriate application context is loaded and
- required properties are injected before executing the test. In this case the XML file name
- is derived from the class name, so FixedLengthImportJobFunctionalTests-context.xml
- (see the "Testing" chapter of Spring reference documentation for more details)
-
-
- @RunWith(SpringJUnit4ClassRunner.class)
- @ContextConfiguration()
- public class FixedLengthImportJobFunctionalTests extends AbstractValidatingBatchLauncherTests {...}
-
-