diff --git a/docs/src/site/docbook/reference/common-patterns.xml b/docs/src/site/docbook/reference/common-patterns.xml index 2fc1ac0d1..7ea604469 100644 --- a/docs/src/site/docbook/reference/common-patterns.xml +++ b/docs/src/site/docbook/reference/common-patterns.xml @@ -278,4 +278,118 @@ DrivingQueryItemReader, which has only one dependency: a KeyCollector - \ No newline at end of file + +
+ Multi-Line Records + + While it is usually the case with flat files that one each record is + confined to a single line, it is common that a file might have records + spanning multiple lines with multiple formats. The following excerpt from + a file illustrates this: + + HEA;0013100345;2007-02-15 + NCU;Smith;Peter;;T;20014539;F + BAD;;Oak Street 31/A;;Small Town;00235;IL;US + FOT;2;2;267.34 + + Everything between the line starting with 'HEA' and the line + starting with 'FOT' is considered one record. There are a few + considerations that must be made in order to handle this situation + correctly: + + + + Instead of reading one record at a time, the + ItemReader must read every line of the + multi-line record as a group, so that it can be passed to the + ItemWriter intact. + + + + Each line type may need to be tokenized differently. + + + + Because a single record spans multiple lines, and we may not know + how many lines there are, the ItemReader must be + careful to always read an entire record. In order to do this, a custom + ItemReader should be implemented as a wrapper for + the FlatFileItemReader. + + <bean id="itemReader" + class="org.springframework.batch.sample.iosample.internal.MultiLineTradeItemReader"> + <property name="delegate"> + <bean class="org.springframework.batch.item.file.FlatFileItemReader"> + <property name="resource" value="data/iosample/input/multiLine.txt" /> + <property name="lineMapper"> + <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper"> + <property name="lineTokenizer" ref="orderFileTokenizer"/> + <property name="fieldSetMapper"> + <bean class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" /> + </property> + </bean> + </property> + </bean> + </property> + </bean> + + To ensure that each line is tokenized properly, which is especially + important for fixed length input, the + PrefixMatchingCompositeLineTokenizer can be used on + the delegate FlatFileItemReader. See for more details. The delegate + reader will then use a PassThroughFieldSetMapper to + deliver a FieldSet for each line back to the + wrapping ItemReader. + + <bean id="orderFileTokenizer" + class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer"> + <property name="tokenizers"> + <map> + <entry key="HEA" value-ref="headerRecordTokenizer" /> + <entry key="FOT" value-ref="footerRecordTokenizer" /> + <entry key="NCU" value-ref="customerLineTokenizer" /> + <entry key="BAD" value-ref="billingAddressLineTokenizer" /> + </map> + </property> + </bean> + + This wrapper will have to be able recognize the end of a record so + that it can continually call read() on its + delegate until the end is reached. For each line that is read, the wrapper + should build up the item to be returned. Once the footer is reached, the + item can be returned for delivery to the + ItemProcessor and + ItemWriter. + + private FlatFileItemReader<FieldSet> delegate; + + public Trade read() throws Exception { + Trade t = null; + + for (FieldSet line = null; (line = this.delegate.read()) != null;) { + String prefix = line.readString(0); + if (prefix.equals("HEA")) { + t = new Trade(); // Record must start with header + } + else if (prefix.equals("NCU")) { + Assert.notNull(t, "No header was found."); + t.setLast(line.readString(1)); + t.setFirst(line.readString(2)); + ... + } + else if (prefix.equals("BAD")) { + Assert.notNull(t, "No header was found."); + t.setCity(line.readString(4)); + t.setState(line.readString(6)); + ... + } + else if (prefix.equals("FOT")) { + return t; // Record must end with footer + } + } + Assert.isNull(t, "No 'END' was found."); + return null; + } +
+ diff --git a/docs/src/site/docbook/reference/readersAndWriters.xml b/docs/src/site/docbook/reference/readersAndWriters.xml index 523d983c5..b6e0184a2 100644 --- a/docs/src/site/docbook/reference/readersAndWriters.xml +++ b/docs/src/site/docbook/reference/readersAndWriters.xml @@ -931,57 +931,85 @@ -
+
Multiple record types within a single file All of the file reading examples up to this point have all made - a key assumption for simplicity's sake: one record equals one line. - However, this may not always be the case. Its very common that a file - might have records spanning multiple lines with multiple formats. The + a key assumption for simplicity's sake: all of the records in a file + have the same format. However, this may not always be the case. Its + very common that a file might have records with different formats that + need to be tokenized differently and mapped to different objects. The following excerpt from a file illustrates this: - HEA;0013100345;2007-02-15 - NCU;Smith;Peter;;T;20014539;F - BAD;;Oak Street 31/A;;Small Town;00235;IL;US - SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States - BIN;VISA;VISA-12345678903 - LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47 - LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87 - SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS - FOT;2;2;267.34 + + USER;Smith;Peter;;T;20014539;F + LINEA;1044391041ABC037.49G201XX1383.12H + LINEB;2134776319DEF422.99M005LI - Everything between the line starting with 'HEA' and the line - starting with 'FOT' is considered one record. The - PrefixMatchingCompositeLineTokenizer makes this easier by matching the - prefix in a line with a particular tokenizer: + + + In this file we have three types of records, "USER", "LINEA", + and "LINEB". A "USER" line corresponds to a User object. "LINEA" and + "LINEB" both correspond to Line objects, though a "LINEA" has more + information than a "LINEB". + + The ItemReader will read each line + individually, but we must specify different + LineTokenizer and + FieldSetMapper objects so that the + ItemWriter will recieve the correct items. The + PrefixMatchingCompositeLineMapper makes this + easy by allowing maps of prefixes to + LineTokenizers and prefixes to + FieldSetMappers to be configured: - <bean id="orderFileDescriptor" - class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer"> + <bean id="orderFileLineMapper" + class="org.springframework.batch.item.file.mapping.PrefixMatchingCompositeLineMapper"> <property name="tokenizers"> - <map> - <entry key="HEA" value-ref="headerRecordDescriptor" /> - <entry key="FOT" value-ref="footerRecordDescriptor" /> - <entry key="BCU" value-ref="businessCustomerLineDescriptor" /> - <entry key="NCU" value-ref="customerLineDescriptor" /> - <entry key="BAD" value-ref="billingAddressLineDescriptor" /> - <entry key="SAD" value-ref="shippingAddressLineDescriptor" /> - <entry key="BIN" value-ref="billingLineDescriptor" /> - <entry key="SIN" value-ref="shippingLineDescriptor" /> - <entry key="LIT" value-ref="itemLineDescriptor" /> - <entry key="" value-ref="defaultLineDescriptor" /> - </map> + <map> + <entry key="USER" value-ref="userTokenizer" /> + <entry key="LINEA" value-ref="lineATokenizer" /> + <entry key="LINEB" value-ref="lineBTokenizer" /> + </map> + </property> + <property name="fieldSetMappers"> + <map> + <entry key="USER" value-ref="userFieldSetMapper" /> + <entry key="LINE" value-ref="lineFieldSetMapper" /> + </map> </property> </bean> - This ensures that the line will be parsed correctly, which is - especially important for fixed length input. Any users of the - FlatFileItemReader in this scenario must - continue calling read until the footer for - the record is returned, allowing them to return a complete order as - one 'item'. + In this example, "LINEA" and "LINEB" have separate + LineTokenizers but they both use the same + FieldSetMapper. + + The PrefixMatchingCompositeLineMapper + makes use of the PatternMatcher's + matchPattern method in order to select the + correct delegate for each line. The pattern will always match the most + specific pattern possible, regardless of the order in the + configuration. So if "LINE" and "LINEA" were both listed as prefixes, + "LINEA" would match prefix "LINEA", while "LINEB" would match prefix + "LINE". Additionally, the empty string ("") can serve as a default + prefix by matching any line not matched by any other prefix. + + + <entry key="" value-ref="defaultLineTokenizer" /> + + + + There is also a + PrefixMatchingCompositeLineTokenizer that can + be used for tokenization alone. + + It is also common for a flat file to contain records that each + span multiple lines. To handle this situation, a more complex strategy + is required. A demonstration of this common patter can be found in + .