Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Cleaned up content around ItemReaders, Mappers and Tokenizers.

Browse Source
This commit is contained in:
wxlund
2008-03-04 16:27:15 +00:00
parent f22e8d04b7
commit 895df88eeb
2 changed files with 170 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
docs/.project
View File

@@ -1,17 +1,29 @@
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>docs</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
<buildCommand>
<name>org.eclipse.jdt.core.javabuilder</name>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>org.eclipse.jdt.core.javanature</nature>
</natures>
</projectDescription>
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>docs</name>
<comment>@key 32303037303533312D313020646F63732F77786C756E64 </comment>
<projects>
</projects>
<buildSpec>
<buildCommand>
<name>org.eclipse.jdt.core.javabuilder</name>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>com.soyatec.additional.Builder</name>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>org.codehaus.groovy.eclipse.groovyBuilder</name>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>org.eclipse.jdt.core.javanature</nature>
<nature>com.soyatec.additional.Nature</nature>
<nature>org.codehaus.groovy.eclipse.groovyNature</nature>
</natures>
</projectDescription>

275
docs/src/site/docbook/reference/infrastructure.xml
View File

@@ -409,9 +409,16 @@
}
</programlisting></para>
<para>Now by adding an PlayerFieldSetMapper that creates a Player for
each FieldSet Item and accessing the data within a record, we have
mapping logic that looks like the following:</para>
<para>We can now inject a fieldset mapper into the ListPlayerReader,
for example, that can take advantage of a PlayerFieldSetMapper for
transforming a line that consists of one item separated by delimiters
into a domain object - <emphasis role="bold">Player</emphasis> in this
case. We inject programmatically by invoking the following:</para>
<para><programlisting> itemReader.setFieldSetMapper(fieldSetMapper);</programlisting></para>
<para>and define the fieldSetMapper class in the following
declaration:</para>
<para><programlisting>
protected static class PlayerFieldSetMapper implements FieldSetMapper {
@@ -429,23 +436,77 @@
}
}
</programlisting></para>
<para>There is one additional preference that can be used that is
similar in function to the jdbc fieldset. The names of the fields can
be injected into the Tokenizer to increase the readability of the
mapping function. We can expose this behavior by adding the following.
First, we tell the tokenizer what the names of the fields in the
fieldset are:</para>
<para><programlisting>
tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"});
</programlisting></para>
<para>and provide a mapper that uses this information as
follows:</para>
<para><programlisting>
public class PlayerMapper implements FieldSetMapper {
public Object mapLine(FieldSet fs) {
if(fs == null){
return null;
}
Player player = new Player();
player.setID(fs.readString("ID"));
player.setLastName(fs.readString("lastName"));
player.setFirstName(fs.readString("firstName"));
player.setPosition(fs.readString("position"));
player.setDebutYear(fs.readInt("debutYear"));
player.setBirthYear(fs.readInt("birthYear"));
return player;
}
}
</programlisting></para>
</section>
<section>
<title id="infrastructure.1.3">Configuring and Using
LineTokenizers</title>
<para>To separate the structure of the raw records, LineTokenizer, or
one of it subclasses, is used to parse data obtained from the an
<para>We have already mentioned tokenizers in the preceding section.
In order to separate the structure of the raw records, LineTokenizer,
or one of it subclasses, is used to parse data obtained from the an
ItemReader, most typically a file. Flat File Item Readers, as
mentioned above, typically process records in two forms, fixed and
delimited. A fixed length input record is where the fields are
assigned fixed locations within a line of a file. An example would be:
<programlisting>
assigned fixed locations within a line of a file. In the preceding
section we gave an example of a comma delimited record layout as
follows:</para>
<para><programlisting>
AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996
AbduRa00,Abdullah,Rabih,rb,1975,1999
AberWa00,Abercrombie,Walter,rb,1959,1982
AbraDa00,Abramowicz,Danny,wr,1945,1967
AdamBo00,Adams,Bob,te,1946,1969
AdamCh00,Adams,Charlie,wr,1979,2003
</programlisting></para>
<para>Here, rather than fix fields to positions within a line or
record, fields are simply separated by some predefined symbol. We
illustrate with commas in this example as it is a very familiar format
to developers who have experience with csv file formats exported from
spreadsheets.</para>
<para>On the other hand a fixed record format might look like the
following:</para>
<para><programlisting>
12345678901234567890123456789012345678901234567890
AbduKa00Abdul-Jabbar Karim rb19741996
AbduRa00Abdullah Rabih rb19751999
@@ -455,36 +516,11 @@
AdamCh00Adams Charlie wr19792003
</programlisting></para>
<para>One can see that each field in the record starts at the same
position in the record. It's fixed from a starting position to an end
of position for each field, which may include a user defined end
position such as EOL.</para>
On the other hand a delimited record format might look like the following:
<para>
<programlisting>
AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996
AbduRa00,Abdullah,Rabih,rb,1975,1999
AberWa00,Abercrombie,Walter,rb,1959,1982
AbraDa00,Abramowicz,Danny,wr,1945,1967
AdamBo00,Adams,Bob,te,1946,1969
AdamCh00,Adams,Charlie,wr,1979,2003
</programlisting>
</para>
<para>Here, rather than fix fields to positions within a line or
record, fields are simply separated by some predefined symbol. We
illustrate with commas in this example as it is a very familiar format
to developers who have experience with to exported csv files from
spreadsheets.</para>
<para>Neither of these formats are particularly self describing but
are still very much in use in flat file exchanges between system
interfaces. Both formats share in common the requirement to read in a
@@ -495,51 +531,22 @@
second dependency is a LineTokenizer, which will be discused
below.</para>
<para>The interface for a LineTokenizer is very simple, given a
string; it will return a FieldSet that wraps the results from
tokenizing the provided string. The tokens are created through a
<emphasis>LineTokenizer</emphasis> and a
<emphasis>FieldSetMapper</emphasis> is used to map the
<emphasis>FieldSet</emphasis> to an object. The framework provides a
few convenience classes, the <emphasis>FieldSetInputSource</emphasis>
and the <emphasis>SimpleFlatFileInputSource</emphasis>. They provide a
convenient way to read the FieldSet. The <emphasis>FieldSet</emphasis>
is configured as a property for an Item Provider, which wraps an Input
Source. You can see this in the following example:</para>
<emphasis>FieldSet</emphasis> to an object. The
<emphasis>FieldSet</emphasis> is configured as a property for an Item
Provider, which wraps an Input Source. Programatically you can inject
the tokenizer as demonstrated previously on the ItemReader via the
method call toYou can see this in the following example:</para>
<para>And, as you can see, the field names will get passed in the the
mapper. The actual mapping provided by the developer would then look
as simple as:</para>
<para>
<programlisting>
public class PlayerMapper implements FieldSetMapper {
public Object mapLine(FieldSet fs) {
if(fs == null){
return null;
}
Player player = new player();
player.setID(fs.readString("ID"));
player.setLastName(fs.readString("lastName"));
player.setFirstName(fs.readString("firstName"));
player.setPosition(fs.readString("position"));
player.setDebutYear(fs.readInt("debutYear"));
player.setBirthYear(fs.readInt("birthYear"));
return player;
}
}
</programlisting>
</para>
<programlisting> itemReader.setTokenizer(tokenizer);</programlisting>
</section>
</section>
@@ -563,10 +570,11 @@ public class PlayerMapper implements FieldSetMapper {
</listitem>
</itemizedlist>
<para>Reading flat files in the Spring Batch framework is facilitated by
the class <emphasis role="bold">FlatFileItemReader</emphasis>, which
provides basic functionality for reading and parsing flat files. In
addition, there are default implementations of the <emphasis
<para>(XML data files will be discussed separately). Reading flat files
in the Spring Batch framework is facilitated by the class <emphasis
role="bold">FlatFileItemReader</emphasis>, which provides basic
functionality for reading and parsing flat files. In addition, there are
default implementations of the <emphasis
role="bold">Skippable</emphasis> and <emphasis role="bold">
ItemStream</emphasis> interfaces that solve the majority of file
processing needs.</para>
@@ -584,20 +592,19 @@ public class PlayerMapper implements FieldSetMapper {
converted to distinct fields. We explored <emphasis
role="bold">fieldSetMapper</emphasis> and <emphasis
role="bold">tokenizer</emphasis> while reviewing how to create a
custom ItemReader. We'll revisit these properties in light of how we
use them with the <emphasis role="bold">FlatFileItemReader</emphasis>.
In addition, we'll explore integration with the file system via the
resource property. The <emphasis role="bold">resource</emphasis>
property represents a Spring Core <emphasis
role="bold">Resource</emphasis>. Documentation explaining how to
create beans of this type can be found in <ulink
custom <emphasis role="bold">ItemReader</emphasis>. We'll revisit
these properties in light of how we use them with the <emphasis
role="bold">FlatFileItemReader</emphasis>. In addition, we'll explore
integration with the file system via the resource property. The
<emphasis role="bold">resource</emphasis> property represents a Spring
Core <emphasis role="bold">Resource</emphasis>. Documentation
explaining how to create beans of this type can be found in <ulink
url="http://static.springframework.org/spring/docs/2.5.x/reference/resources.html"><citetitle>Spring
Framework, Chapter 4.Resources</citetitle></ulink>. Therefore, this
guide will not go into the details of creating <emphasis
role="bold">Resource</emphasis> objects except to make a couple of
points on the locating files to process within a batch environment.
Tokenizers and field set mappers will be discussed a bit
later.&gt;</para>
Tokenizers and field set mappers will be discussed a bit later.</para>
<para>As mentioned, the location of the file is defined by the
resource property. There are only a few methods exposed through a
@@ -707,62 +714,62 @@ public class PlayerMapper implements FieldSetMapper {
</para>
-->
</section>
<section>
<title id="infrastructure.2.3">XML Item Readers and Writers</title>
<para>Spring Batch provides transactional infrastructure for both reading
XML records and mapping them to Java objects as well as writing Java
objects as XML records.</para>
<para>StAX API is used for I/O as other standard XML APIs do not fit batch
processing requirements (DOM loads the whole input into memory at once and
SAX controls the parsing process allowing the user only to provide
callbacks).</para>
<para>Spring Batch is not tied to any particular OXM technology. Typical
use is to delegate OXM to Spring WS which provides uniform abstraction for
the most popular OXM technologies. However dependency on Spring WS is
optional and you can choose to implement Spring Batch specific interfaces
if desired.</para>
<para>Lets take a closer look how XML input and output work in batch. It
is assumed the XML resource is a collection of 'fragments' corresponding
to individual records. Note that OXM tools are designed to work with
standalone XML documents rather than XML fragments cut out of an XML
document, therefore the Spring Batch infrastructure needs to work around
this fact (as described below).</para>
<para>On input the reader reads the XML resource until it recognizes a new
fragment is about to start (by matching the tag name by default). The
reader creates a standalone XML document from the fragment (or at least
makes it appear so) and passes the document to a deserializer (typically a
wrapper around Spring WS Unmarshaller) to map the XML to a Java
object.</para>
<para>Output works symetrically to input. Java object is passed to a
serializer (typically a wrapper around Spring WS Marshaller) which writes
to output using a custom event writer that filters the StartDocument and
EndDocument events produced for each fragment by the OXM tools.</para>
<para>For example configuration of XML input and output see the sample
xmlStaxJob. //TODO inline the example once it is not subject to change +
show sample input file</para>
</section>
</section>
<section>
<title id="infrastructure.1.4">Item Writers</title>
<title id="infrastructure.2.3">XML Item Readers and Writers</title>
<para>The Item Writers are similar in functionality to the input source
with the exception that the operations are reversed. They still need to
be located, opened and closed but they differ in the case that we write
to output sources. In the case of databases or queues these may be
inserts, updates or sends. The format of the serialization of the output
source is specific for every batch job.</para>
<para>Spring Batch provides transactional infrastructure for both
reading XML records and mapping them to Java objects as well as writing
Java objects as XML records.</para>
<para>StAX API is used for I/O as other standard XML APIs do not fit
batch processing requirements (DOM loads the whole input into memory at
once and SAX controls the parsing process allowing the user only to
provide callbacks).</para>
<para>Spring Batch is not tied to any particular OXM technology. Typical
use is to delegate OXM to Spring WS which provides uniform abstraction
for the most popular OXM technologies. However dependency on Spring WS
is optional and you can choose to implement Spring Batch specific
interfaces if desired.</para>
<para>Lets take a closer look how XML input and output work in batch. It
is assumed the XML resource is a collection of 'fragments' corresponding
to individual records. Note that OXM tools are designed to work with
standalone XML documents rather than XML fragments cut out of an XML
document, therefore the Spring Batch infrastructure needs to work around
this fact (as described below).</para>
<para>On input the reader reads the XML resource until it recognizes a
new fragment is about to start (by matching the tag name by default).
The reader creates a standalone XML document from the fragment (or at
least makes it appear so) and passes the document to a deserializer
(typically a wrapper around Spring WS Unmarshaller) to map the XML to a
Java object.</para>
<para>Output works symetrically to input. Java object is passed to a
serializer (typically a wrapper around Spring WS Marshaller) which
writes to output using a custom event writer that filters the
StartDocument and EndDocument events produced for each fragment by the
OXM tools.</para>
<para>For example configuration of XML input and output see the sample
xmlStaxJob. //TODO inline the example once it is not subject to change +
show sample input file</para>
</section>
<section>
<title id="infrastructure.1.4">Item Writers</title>
<para>The Item Writers are similar in functionality to the input source
with the exception that the operations are reversed. They still need to be
located, opened and closed but they differ in the case that we write to
output sources. In the case of databases or queues these may be inserts,
updates or sends. The format of the serialization of the output source is
specific for every batch job.</para>
</section>
<section>
<title id="infrastructure.2.2">SQL Sources</title>