diff --git a/docs/src/site/docbook/reference/readersAndWriters.xml b/docs/src/site/docbook/reference/readersAndWriters.xml index b1a3907a5..40e933ab9 100644 --- a/docs/src/site/docbook/reference/readersAndWriters.xml +++ b/docs/src/site/docbook/reference/readersAndWriters.xml @@ -209,12 +209,12 @@
Flat Files - Since the beginning of batch processing, one of the most common - mechanisms for interchanging bulk data has been the flat file. Unlike XML, - which has an aggreed upon standard for defining how it is structured - (XSD), the person reading a flat file must understand ahead of time - exactly how the file is structured. In general, all flat files fall into - two general types: Delimited and Fixed Length. + One of the most common mechanisms for interchanging bulk data has + always been the flat file. Unlike XML, which has an aggreed upon standard + for defining how it is structured (XSD), anyone reading a flat file must + understand ahead of time exactly how the file is structured. In general, + all flat files fall into two general types: Delimited and Fixed + Length.
The FieldSet @@ -222,7 +222,7 @@ When working with flat files in Spring Batch, regardless of whether it is for input or output, one of the most important classes is the FieldSet. Many architectures and libraries contain abstractions for - helping you read in from a file, but they usually return a String or + helping you read in from a file, but they usually return a String or an array of Strings. This really only gets you halfway there. A FieldSet is Spring Batch’s abstraction for enabling the binding of fields from a file resource. It allows developers to work with file input in much the @@ -267,16 +267,14 @@ boolean booleanValue = fs.readBoolean(2); role="bold">fieldSetMapper and tokenizer, which define the resource from which data will be read and the method by which the read data will be - converted to distinct fields. We explored fieldSetMapper and tokenizer while reviewing how to create a custom - ItemReader. We'll revisit these - properties in light of how we use them with the FlatFileItemReader. In addition, we'll explore - integration with the file system via the resource property. The - resource property represents a Spring - Core Resource. Documentation explaining - how to create beans of this type can be found in tokenizer interfaces will be explored more in the + next sections. In addition, we'll explore integration with the file + system via the resource property. The resource property represents a Spring Core + Resource. Documentation explaining how + to create beans of this type can be found in Spring Framework, Chapter 4.Resources. Therefore, this guide will not go into the details of creating The flat file reader uses a ResourceLineReader object to read from the file. Optionally, you can specify a RecordSeparatorPolicy through property - recordSeparatorPolicy. This can be used to configure more low-level - features, such as what constitutes the end of a line and whether to - continue quoted strings over newlines, among other things. + role="bold">RecordSeparatorPolicy through the + recordSeparatorPolicy property. This can be used to configure more + low-level features, such as what constitutes the end of a line and + whether to continue quoted strings over newlines, among other + things. The other properties in the flat file readers allow you to further specify how your data will be interpreted: @@ -370,38 +369,106 @@ boolean booleanValue = fs.readBoolean(2);
- The FieldSet + FieldSetMapper - A FieldSet is Spring Batch’s abstraction for enabling the - binding of fields from a file data source. It allows developers to - work with file input in much the same way as they would work with - database input. A FieldSet is conceptually very similar to a Jdbc - Result Set. FieldSets only require one argument, a list of tokens. - Optionally you can also configure in the names of the fields so that - the fields may be accessed either by index or name as patterned after - the JdbcResultSet. In code it means it's as simple as: - - Field set mappers used by the flat file reader classes implement - the FieldSetMapper interface. This interface defines a single method, + Field set mappers used by the FlatFileItemReader implement the + FieldSetMapper interface. This interface defines a single method, mapLine, which takes a FieldSet object and maps its contents to some Object. This object may be a custom DTO or domain object, or it could - be as simple as an array, depending on your needs. The field set - mapper is used in conjunction with the tokenizer to translate a line - of data from a resource into an object of the desired type. + be as simple as an array, depending on your needs. The FieldSetMapper is used in conjunction with the + tokenizer to translate a line of data from a resource into an object + of the desired type: - For example, suppose our file or list consists of players has - the following fields and the start of the data looks like the - following: - ID,lastName,firstName,position,birthYear,debutYear - "AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996", - "AbduRa00,Abdullah,Rabih,rb,1975,1999", - "AberWa00,Abercrombie,Walter,rb,1959,1982", - "AbraDa00,Abramowicz,Danny,wr,1945,1967", - "AdamBo00,Adams,Bob,te,1946,1969", - "AdamCh00,Adams,Charlie,wr,1979,2003" - + public interface FieldSetMapper { + + public Object mapLine(FieldSet fs); - We want to map this data to the following Player object: +} + + As you can see, the pattern used is exatly the same as RowMapper + used by JdbcTemplate. +
+ +
+ LineTokenizer + + Because there can be many formats of flat file data, which all + need to be converted to a FieldSet so that a FieldSetMapper can create + a useful domain object from them, an abstraction for turning a line of + input into a FieldSet is necessary. In Spring Batch, this is called + the LineTokenizer: + + public interface LineTokenizer { + + FieldSet tokenize(String line); + +} + + + The contract of a LineTokenizer is such that, given a line of + input (in theory the String could encompass more than one line) a + FieldSet representing the line will be returned. This will then be + based to a FieldSetMapper. Spring Batch contains the following + LineTokenizers: + + + + DelmitedLineTokenizer - Used for files that separate records + by a delimiter. The most common is a comma, but pipes or + semicolons are often used as well + + + + FixedLengthTokenizer - Used for tokenizing files where each + record is separated by a 'fixed width' that must be defined per + record. + + + + PrefixMatchingCompositeLineTokenizer - Tokenizer that + determines which among a list of Tokenizers should be used on a + particular line by checking against a prefix. + + +
+ +
+ Simple Delimited File Reading Example + + Now that the basic interfaces for reading in flat files have + been defined, a simple example explaining how they work together is + helpful. In it's most simple form, the flow when reading a line form a + file is this: + + + + Read one line from the file. + + + + Pass the string line into the LineTokenizer#tokenize() + method, in order to retrieve a FieldSet + + + + Pass the FieldSet returned from tokenizing to a + FieldSetMapper, returning the result from the ItemReader#read() + method + + + + The following example will be used to illustrate this using an + actual domain scenario. This particular batch job reads in football + players from the following file:ID,lastName,firstName,position,birthYear,debutYear +"AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996", +"AbduRa00,Abdullah,Rabih,rb,1975,1999", +"AberWa00,Abercrombie,Walter,rb,1959,1982", +"AbraDa00,Abramowicz,Danny,wr,1945,1967", +"AdamBo00,Adams,Bob,te,1946,1969", +"AdamCh00,Adams,Charlie,wr,1979,2003" + + We want to map this data to the following Player domain object: public class Player implements Serializable { @@ -424,16 +491,8 @@ boolean booleanValue = fs.readBoolean(2); } - We can now inject a fieldset mapper into the ListPlayerReader, - for example, that can take advantage of a PlayerFieldSetMapper for - transforming a line that consists of one item separated by delimiters - into a domain object - Player in this - case. We inject programmatically by invoking the following: - - itemReader.setFieldSetMapper(fieldSetMapper); - - and define the fieldSetMapper class in the following - declaration: + In order to map a FieldSet into our Player object, we need to + create a FieldSetMapper that returns players: protected static class PlayerFieldSetMapper implements FieldSetMapper { @@ -452,41 +511,301 @@ boolean booleanValue = fs.readBoolean(2); } - There is one additional preference that can be used that is - similar in function to the jdbc fieldset. The names of the fields can - be injected into the Tokenizer to increase the readability of the - mapping function. We can expose this behavior by adding the following. - First, we tell the tokenizer what the names of the fields in the - fieldset are: + We can then read in from the filed by correctly constructing our + FlatFileItemReader and calling read(): + + FlatFileItemReader itemReader = new FlatFileItemReader(); +itemReader.setResource = new FileSystemResource("resources/players.csv"); +//DelimitedLineTokenizer defaults to comma as it's delimiter +itemReader.setLineTokenizer = new DelimitedLineTokenizer(); +itemReader.setFieldSetMapper = new PlayerFieldSetMapper(); +itemReader.read(); + + + Each call to read will return a new Player object from each line + in the file. When the end of the file is reached, null will be + returned. +
+ +
+ Mapping fields by name + + There is one additional functionality that is similar in + function to a JDBC ResultSet. The names of the fields can be injected + into the Tokenizer to increase the readability of the mapping + function. We can expose this behavior by adding the following. First, + we tell the tokenizer what the names of the fields in the fieldset + are: - tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); + tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); and provide a mapper that uses this information as follows: - public class PlayerMapper implements FieldSetMapper { - public Object mapLine(FieldSet fs) { + public class PlayerMapper implements FieldSetMapper { + public Object mapLine(FieldSet fs) { - if(fs == null){ - return null; - } + if(fs == null){ + return null; + } - Player player = new Player(); - player.setID(fs.readString("ID")); - player.setLastName(fs.readString("lastName")); - player.setFirstName(fs.readString("firstName")); - player.setPosition(fs.readString("position")); - player.setDebutYear(fs.readInt("debutYear")); - player.setBirthYear(fs.readInt("birthYear")); + Player player = new Player(); + player.setID(fs.readString("ID")); + player.setLastName(fs.readString("lastName")); + player.setFirstName(fs.readString("firstName")); + player.setPosition(fs.readString("position")); + player.setDebutYear(fs.readInt("debutYear")); + player.setBirthYear(fs.readInt("birthYear")); - return player; - } - + return player; } - + + } +
+ +
+ BeanWrapperFieldSetMapper + + For many, having to write a specific FieldSetMapper is equally + as cumbersome as writing a specific RowMapper for a JdbcTemplate. + Spring Batch makes this easier by providing a FieldSetMapper that + automatically maps fields by matching a field name with a setter using + the JavaBean spec. Again using the footbal example, the FieldSetMapper + configuration looks like the following: + + <bean id="fieldSetMapper" + class="org.springframework.batch.io.file.mapping.BeanWrapperFieldSetMapper"> + <property name="prototypeBeanName" value="player" /> +</bean> + +<bean id="person" + class="org.springframework.batch.sample.domain.Player" + scope="prototype" /> + + For each entry in the FieldSet, the mapper will look for a + corresponding setter on a new instance of the Player object (for this + reason, prototype scope is required) in the same way the Spring + container will look for setters matching a property name. Each + available field in the FieldSet will be mapped, and the resultant + Player object will be returned, only there was no code + required. +
+ +
+ FixedLengthLineTokenizer + + So far only delimited files have been discussed in much detail, + however, they respresent only half of the file reading picture. Many + organizations that use flat files use fixed length formats. An example + field length file is below: + + UK21341EAH4121131.11customer1 +UK21341EAH4221232.11customer2 +UK21341EAH4321333.11customer3 +UK21341EAH4421434.11customer4 +UK21341EAH4521535.11customer5 + + While this looks like one large field, it actually represent 4 + distinct fields: + + + + ISIN: Unique identifier for the item being order - 12 + characters long. + + + + Quantity: Number of this item being ordered - 3 characters + long. + + + + Price: Price of the item - 4 characters long. + + + + Customer: Id of the customer ordering the item - 8 + characters long. + + + + When configuring the FixedLengthLineTokenizer, each of these + lengths must be provided in the form of ranges: + + <bean id="fixedLengthLineTokenizer" + class="org.springframework.batch.io.file.transform.FixedLengthTokenizer"> + <property name="names" value="ISIN, Quantity, Price, Customer" /> + <property name="columns" value="1-12, 13-15, 16-20, 21-29" /> +</bean> + + This LineTokenizer will return the same FieldSet as if a + dlimiter had been used, allowing the same approachs above to be used + such as the BeanWrapperFieldSetMapper, in a way that is ignorant of + how the actual line was parsed. +
+ +
+ PrefixMatchingCompositeLineTokenizer + + All of the file reading examples up to this point have all made + a key assumption for simplicity's sake: one record equals one line. + However, this may not always be the case. It's very common that a file + might have records spanning multiple lines with multiple formats. The + following excerpt from a file illustrates this: + + HEA;0013100345;2007-02-15 +NCU;Smith;Peter;;T;20014539;F +BAD;;Oak Street 31/A;;Small Town;00235;IL;US +SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States +BIN;VISA;VISA-12345678903 +LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47 +LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87 +SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS +FOT;2;2;267.34 + + Everything between the line starting with 'HEA' and the line + starting with 'FOT' is considered one record. The + PrefixMatchingCompositeLineTokenizer makes this easier by matching the + prefix in a line with a particular tokenizer: + + <bean id="orderFileDescriptor" + class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer"> + <property name="tokenizers"> + <map> + <entry key="HEA" value-ref="headerRecordDescriptor" /> + <entry key="FOT" value-ref="footerRecordDescriptor" /> + <entry key="BCU" value-ref="businessCustomerLineDescriptor" /> + <entry key="NCU" value-ref="customerLineDescriptor" /> + <entry key="BAD" value-ref="billingAddressLineDescriptor" /> + <entry key="SAD" value-ref="shippingAddressLineDescriptor" /> + <entry key="BIN" value-ref="billingLineDescriptor" /> + <entry key="SIN" value-ref="shippingLineDescriptor" /> + <entry key="LIT" value-ref="itemLineDescriptor" /> + <entry key="" value-ref="defaultLineDescriptor" /> + </map> + </property> +</bean> + + This ensures that the line will be parsed correctly, which is + especially important for fixed length input, with the correct field + names. Any users of the FlatFileItemReader in this scenario must + continue calling read() until the footer for the record is returned, + allowing them to return a complete order as one 'item'. +
+
+ +
+ FlatFileItemWriter + + Writing out to flat files has the same problems and issues that + reading in from a file must overcome. It must be able to write out in + either dlimited or fixed length formats in a transactional + mannger. + +
+ LineAggregator + + Just like file reading's LineTokenizer interface is necessary to + take a string and split it into tokens, file writing must have a way + to aggregate multiple fields into a single string for writing to a + file. In Spring Batch this is the LineAggregator: + + public interface LineAggregator { + + public String aggregate(FieldSet fieldSet); +} + + + The LineAggregator is exactly the opposite of a LineTokenizer. + LineTokenizer takes a string and returns a FieldSet, wheras + LineAggreator takes a FieldSet and returns a string. As with reading + there are two types: DelimitedLineAggregator and + FixedLengthLineAggregator. +
+ +
+ FieldSetCreator + + Because the LineAggregator interface uses a FieldSet as it's + mechanism for converting to a string, there needs to be an interface + that describes how to convert from an object into a FieldSet: + + public interface FieldSetCreator { + + FieldSet mapItem(Object data); + +} + + As with LineTokenizer and LineAggregator, FieldSetCreator is the + polar opposite of FieldSetMapper. FieldSetMapper takes a FieldSet and + returns a mapped object, whereas a FieldSetCreator takes an Object and + returns a FieldSet. +
+ +
+ Simple Delimited File Writing Example + + Now that both the LineAggregator and FieldSetCreator interfaces + have been defined, the basic flow of writing can be explained: + + + + The object to be written is passed to the FieldSetCreator in + order to obtain a FieldSet. + + + + The returned FieldSet is passed to the LineAggregator + + + + The returned string is written to the configured + file. + + + + The following excerpt from the FlatFileItemWriter expresses this + in code: + + public void write(Object data) throws Exception { + FieldSet fieldSet = fieldSetCreator.mapItem(data); + getOutputState().write(lineAggregator.aggregate(fieldSet) + LINE_SEPARATOR); +} + + A simple configuration with the smallest ammount of setters + would look like the following: + + <bean id="itemWriter" + class="org.springframework.batch.io.file.FlatFileItemWriter"> + <property name="resource" + value="file:target/test-outputs/20070122.testStream.multilineStep.txt" /> + <property name="fieldSetCreator"> + <bean class="org.springframework.batch.io.file.mapping.PassThroughFieldSetMapper"/> + </property> +</bean> +
+ +
+ Handling file creation + + FlatFileItemReader has a very simple relationship with file + resources. When the reader is initialized, it opens the file if it + exists, and throws an exception if it does not. File writing isn't + quite so simple. At first glance it seems like a similiar straight + forward contract should exist for FlatFileItemWriter, if the file + already exists, throw an exception, if it does not, create it and + start writing. Job restart throws a bit of a kink into this. In the + normal restart scenario, the contract is reversed, if the file exists + start writing to it from the last known good position, if it does not, + throw an exception. However, what happens if the file name for this + job is always the same? In this case, you would want to delete the + file if it exists, unless it's a restart. Because of this possibility, + the FlatFileItemWriter contains the property, shouldDeleteIfExists. + Setting this property to true will cause an existing file with the + same name to be deleted when the writer is opened.
@@ -603,10 +922,6 @@ boolean booleanValue = fs.readBoolean(2); -
- -
-
Validating Input