BATCH-1051: Created section in Common Patterns to explain how to handle multi-line records
This commit is contained in:
dhgarrette
@@ -278,4 +278,118 @@
|
||||
<classname>DrivingQueryItemReader</classname>, which has only one
|
||||
dependency: a <classname>KeyCollector</classname></para>
|
||||
</section>
|
||||
</chapter>
|
||||
|
||||
<section id="multiLineRecords">
|
||||
<title>Multi-Line Records</title>
|
||||
|
||||
<para>While it is usually the case with flat files that one each record is
|
||||
confined to a single line, it is common that a file might have records
|
||||
spanning multiple lines with multiple formats. The following excerpt from
|
||||
a file illustrates this:</para>
|
||||
|
||||
<programlisting> HEA;0013100345;2007-02-15
|
||||
NCU;Smith;Peter;;T;20014539;F
|
||||
BAD;;Oak Street 31/A;;Small Town;00235;IL;US
|
||||
FOT;2;2;267.34</programlisting>
|
||||
|
||||
<para>Everything between the line starting with 'HEA' and the line
|
||||
starting with 'FOT' is considered one record. There are a few
|
||||
considerations that must be made in order to handle this situation
|
||||
correctly:</para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>Instead of reading one record at a time, the
|
||||
<classname>ItemReader</classname> must read every line of the
|
||||
multi-line record as a group, so that it can be passed to the
|
||||
<classname>ItemWriter</classname> intact.</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Each line type may need to be tokenized differently.</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
|
||||
<para>Because a single record spans multiple lines, and we may not know
|
||||
how many lines there are, the <classname>ItemReader</classname> must be
|
||||
careful to always read an entire record. In order to do this, a custom
|
||||
<classname>ItemReader</classname> should be implemented as a wrapper for
|
||||
the <classname>FlatFileItemReader</classname>. </para>
|
||||
|
||||
<programlisting> <bean id="itemReader"
|
||||
class="org.springframework.batch.sample.iosample.internal.MultiLineTradeItemReader">
|
||||
<property name="delegate">
|
||||
<bean class="org.springframework.batch.item.file.FlatFileItemReader">
|
||||
<property name="resource" value="data/iosample/input/multiLine.txt" />
|
||||
<property name="lineMapper">
|
||||
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
|
||||
<property name="lineTokenizer" ref="orderFileTokenizer"/>
|
||||
<property name="fieldSetMapper">
|
||||
<bean class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
|
||||
</property>
|
||||
</bean>
|
||||
</property>
|
||||
</bean>
|
||||
</property>
|
||||
</bean></programlisting>
|
||||
|
||||
<para>To ensure that each line is tokenized properly, which is especially
|
||||
important for fixed length input, the
|
||||
<classname>PrefixMatchingCompositeLineTokenizer</classname> can be used on
|
||||
the delegate <classname>FlatFileItemReader</classname>. See <xref
|
||||
linkend="prefixMatchingLineMapper" /> for more details. The delegate
|
||||
reader will then use a <classname>PassThroughFieldSetMapper</classname> to
|
||||
deliver a <classname>FieldSet</classname> for each line back to the
|
||||
wrapping <classname>ItemReader</classname>.</para>
|
||||
|
||||
<programlisting> <bean id="orderFileTokenizer"
|
||||
class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer">
|
||||
<property name="tokenizers">
|
||||
<map>
|
||||
<entry key="HEA" value-ref="headerRecordTokenizer" />
|
||||
<entry key="FOT" value-ref="footerRecordTokenizer" />
|
||||
<entry key="NCU" value-ref="customerLineTokenizer" />
|
||||
<entry key="BAD" value-ref="billingAddressLineTokenizer" />
|
||||
</map>
|
||||
</property>
|
||||
</bean></programlisting>
|
||||
|
||||
<para>This wrapper will have to be able recognize the end of a record so
|
||||
that it can continually call <methodname>read()</methodname> on its
|
||||
delegate until the end is reached. For each line that is read, the wrapper
|
||||
should build up the item to be returned. Once the footer is reached, the
|
||||
item can be returned for delivery to the
|
||||
<classname>ItemProcessor</classname> and
|
||||
<classname>ItemWriter</classname>.</para>
|
||||
|
||||
<programlisting> private FlatFileItemReader<FieldSet> delegate;
|
||||
|
||||
public Trade read() throws Exception {
|
||||
Trade t = null;
|
||||
|
||||
for (FieldSet line = null; (line = this.delegate.read()) != null;) {
|
||||
String prefix = line.readString(0);
|
||||
if (prefix.equals("HEA")) {
|
||||
t = new Trade(); // Record must start with header
|
||||
}
|
||||
else if (prefix.equals("NCU")) {
|
||||
Assert.notNull(t, "No header was found.");
|
||||
t.setLast(line.readString(1));
|
||||
t.setFirst(line.readString(2));
|
||||
...
|
||||
}
|
||||
else if (prefix.equals("BAD")) {
|
||||
Assert.notNull(t, "No header was found.");
|
||||
t.setCity(line.readString(4));
|
||||
t.setState(line.readString(6));
|
||||
...
|
||||
}
|
||||
else if (prefix.equals("FOT")) {
|
||||
return t; // Record must end with footer
|
||||
}
|
||||
}
|
||||
Assert.isNull(t, "No 'END' was found.");
|
||||
return null;
|
||||
}</programlisting>
|
||||
</section>
|
||||
</chapter>
|
||||
|
||||
@@ -931,57 +931,85 @@
|
||||
</programlisting>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<section id="prefixMatchingLineMapper">
|
||||
<title>Multiple record types within a single file</title>
|
||||
|
||||
<para>All of the file reading examples up to this point have all made
|
||||
a key assumption for simplicity's sake: one record equals one line.
|
||||
However, this may not always be the case. Its very common that a file
|
||||
might have records spanning multiple lines with multiple formats. The
|
||||
a key assumption for simplicity's sake: all of the records in a file
|
||||
have the same format. However, this may not always be the case. Its
|
||||
very common that a file might have records with different formats that
|
||||
need to be tokenized differently and mapped to different objects. The
|
||||
following excerpt from a file illustrates this:</para>
|
||||
|
||||
<programlisting> HEA;0013100345;2007-02-15
|
||||
NCU;Smith;Peter;;T;20014539;F
|
||||
BAD;;Oak Street 31/A;;Small Town;00235;IL;US
|
||||
SAD;Smith, Elizabeth;Elm Street 17;;Some City;30011;FL;United States
|
||||
BIN;VISA;VISA-12345678903
|
||||
LIT;1044391041;37.49;0;0;4.99;2.99;1;45.47
|
||||
LIT;2134776319;221.99;5;0;7.99;2.99;1;221.87
|
||||
SIN;UPS;EXP;DELIVER ONLY ON WEEKDAYS
|
||||
FOT;2;2;267.34</programlisting>
|
||||
<programlisting>
|
||||
USER;Smith;Peter;;T;20014539;F
|
||||
LINEA;1044391041ABC037.49G201XX1383.12H
|
||||
LINEB;2134776319DEF422.99M005LI
|
||||
|
||||
<para>Everything between the line starting with 'HEA' and the line
|
||||
starting with 'FOT' is considered one record. The
|
||||
PrefixMatchingCompositeLineTokenizer makes this easier by matching the
|
||||
prefix in a line with a particular tokenizer:</para>
|
||||
</programlisting>
|
||||
|
||||
<para>In this file we have three types of records, "USER", "LINEA",
|
||||
and "LINEB". A "USER" line corresponds to a User object. "LINEA" and
|
||||
"LINEB" both correspond to Line objects, though a "LINEA" has more
|
||||
information than a "LINEB".</para>
|
||||
|
||||
<para>The <classname>ItemReader </classname>will read each line
|
||||
individually, but we must specify different
|
||||
<classname>LineTokenizer</classname> and
|
||||
<classname>FieldSetMapper</classname> objects so that the
|
||||
<classname>ItemWriter</classname> will recieve the correct items. The
|
||||
<classname>PrefixMatchingCompositeLineMapper</classname> makes this
|
||||
easy by allowing maps of prefixes to
|
||||
<classname>LineTokenizer</classname>s and prefixes to
|
||||
<classname>FieldSetMapper</classname>s to be configured:</para>
|
||||
|
||||
<programlisting>
|
||||
<bean id="orderFileDescriptor"
|
||||
class="org.springframework.batch.io.file.transform.PrefixMatchingCompositeLineTokenizer">
|
||||
<bean id="orderFileLineMapper"
|
||||
class="org.springframework.batch.item.file.mapping.PrefixMatchingCompositeLineMapper">
|
||||
<property name="tokenizers">
|
||||
<map>
|
||||
<entry key="HEA" value-ref="headerRecordDescriptor" />
|
||||
<entry key="FOT" value-ref="footerRecordDescriptor" />
|
||||
<entry key="BCU" value-ref="businessCustomerLineDescriptor" />
|
||||
<entry key="NCU" value-ref="customerLineDescriptor" />
|
||||
<entry key="BAD" value-ref="billingAddressLineDescriptor" />
|
||||
<entry key="SAD" value-ref="shippingAddressLineDescriptor" />
|
||||
<entry key="BIN" value-ref="billingLineDescriptor" />
|
||||
<entry key="SIN" value-ref="shippingLineDescriptor" />
|
||||
<entry key="LIT" value-ref="itemLineDescriptor" />
|
||||
<entry key="" value-ref="defaultLineDescriptor" />
|
||||
</map>
|
||||
<map>
|
||||
<entry key="USER" value-ref="userTokenizer" />
|
||||
<entry key="LINEA" value-ref="lineATokenizer" />
|
||||
<entry key="LINEB" value-ref="lineBTokenizer" />
|
||||
</map>
|
||||
</property>
|
||||
<property name="fieldSetMappers">
|
||||
<map>
|
||||
<entry key="USER" value-ref="userFieldSetMapper" />
|
||||
<entry key="LINE" value-ref="lineFieldSetMapper" />
|
||||
</map>
|
||||
</property>
|
||||
</bean>
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>This ensures that the line will be parsed correctly, which is
|
||||
especially important for fixed length input. Any users of the
|
||||
<classname>FlatFileItemReader</classname> in this scenario must
|
||||
continue calling <methodname>read</methodname> until the footer for
|
||||
the record is returned, allowing them to return a complete order as
|
||||
one 'item'.</para>
|
||||
<para>In this example, "LINEA" and "LINEB" have separate
|
||||
<classname>LineTokenizer</classname>s but they both use the same
|
||||
<classname>FieldSetMapper</classname>.</para>
|
||||
|
||||
<para>The <classname>PrefixMatchingCompositeLineMapper</classname>
|
||||
makes use of the <classname>PatternMatcher</classname>'s
|
||||
<classname>matchPattern</classname> method in order to select the
|
||||
correct delegate for each line. The pattern will always match the most
|
||||
specific pattern possible, regardless of the order in the
|
||||
configuration. So if "LINE" and "LINEA" were both listed as prefixes,
|
||||
"LINEA" would match prefix "LINEA", while "LINEB" would match prefix
|
||||
"LINE". Additionally, the empty string ("") can serve as a default
|
||||
prefix by matching any line not matched by any other prefix.</para>
|
||||
|
||||
<programlisting>
|
||||
<entry key="" value-ref="defaultLineTokenizer" />
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>There is also a
|
||||
<classname>PrefixMatchingCompositeLineTokenizer</classname> that can
|
||||
be used for tokenization alone.</para>
|
||||
|
||||
<para>It is also common for a flat file to contain records that each
|
||||
span multiple lines. To handle this situation, a more complex strategy
|
||||
is required. A demonstration of this common patter can be found in
|
||||
<xref linkend="multiLineRecords" />.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
|
||||
Reference in New Issue
Block a user