BATCH-1042: Added section on scalability to what's new section.
This commit is contained in:
lucasward
@@ -114,7 +114,7 @@
|
||||
detail.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<section id="remoteChunking">
|
||||
<title>Remote Chunking</title>
|
||||
|
||||
<para>In Remote Chunking the Step processing is split across multiple
|
||||
@@ -162,7 +162,7 @@
|
||||
users.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<section id="partitioning">
|
||||
<title>Partitioning</title>
|
||||
|
||||
<para>Spring Batch also provides an SPI for partitioning a Step execution
|
||||
|
||||
@@ -1,408 +1,463 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
|
||||
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
|
||||
<chapter id="whatsNew">
|
||||
<title>What's new in Spring Batch 2.0</title>
|
||||
|
||||
<para>The Spring Batch 2.0 release has six major themes:</para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>Java 5</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Non Sequential Step Execution</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Chunk oriented processing</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Meta Data enhancements</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Scalability</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Configuration</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
|
||||
<section>
|
||||
<title id="s.2.1.6">Java 5</title>
|
||||
|
||||
<para>The 1.x releases of Spring Batch were all based on Java 1.4. This
|
||||
prevented the framework from using many enhancements provided in Java 5
|
||||
such as generics, parameterized types, etc. The entire framework has been
|
||||
updated to utilize these features. As a result, <emphasis role="bold">Java
|
||||
1.4 is no longer supported.</emphasis> Most of the interfaces developers
|
||||
work with have been updated to support generic types. As an example, the
|
||||
<classname>ItemReader</classname> interface from 1.1 is below:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader {
|
||||
|
||||
Object read() throws Exception;
|
||||
|
||||
void mark() throws MarkFailedException;
|
||||
|
||||
void reset() throws ResetFailedException;
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>As you can see, the <methodname>read</methodname> method returns an
|
||||
<classname>Object</classname>. The 2.0 version is below:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader<T> {
|
||||
|
||||
T read() throws Exception, UnexpectedInputException, ParseException;
|
||||
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>As you can see, <classname>ItemReader</classname> now supports the
|
||||
generic type, T, which is returned from read. You may also notice that
|
||||
<methodname>mark</methodname> and <methodname>reset</methodname> have been
|
||||
removed. This is due to step processing strategy changes, which are
|
||||
discussed below. Many other interfaces have been similarly updated.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Chunk Oriented Processing</title>
|
||||
|
||||
<para>Previously, the default processing strategy provided by Spring Batch
|
||||
was item-oriented processing:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center"
|
||||
fileref="images/item-oriented-processing.png" scale="90"
|
||||
width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/item-oriented-processing.png" scale="90"
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>In item-oriented processing, the <classname>ItemReader</classname>
|
||||
returns one <classname>Object</classname> (the 'item') which is then
|
||||
handed to the <classname>ItemWriter</classname>, periodically committing
|
||||
when the number of items hits the commit interval. For example, if the
|
||||
commit interval is 5, <classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname> will each be called 5 times. This is
|
||||
illustrated in a simplified code example below:</para>
|
||||
|
||||
<programlisting>
|
||||
for(int i = 0; i < commitInterval; i++){
|
||||
Object item = itemReader.read();
|
||||
itemWriter.write(item);
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>Both the <classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname> interfaces were completely geared toward
|
||||
this approach:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader {
|
||||
|
||||
Object read() throws Exception;
|
||||
|
||||
void mark() throws MarkFailedException;
|
||||
|
||||
void reset() throws ResetFailedException;
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<programlisting> public interface ItemWriter {
|
||||
|
||||
void write(Object item) throws Exception;
|
||||
|
||||
void flush() throws FlushFailedException;
|
||||
|
||||
void clear() throws ClearFailedException;
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>Because the 'scope' of the processing was one item, supporting
|
||||
rollback scenarios required additional methods, which is what
|
||||
<methodname>mark</methodname>, <methodname>reset</methodname>,
|
||||
<methodname>flush</methodname>, and <methodname>clear</methodname>
|
||||
provided. If, after successfully reading and writing 2 items, the third
|
||||
has an error while writing, the transaction would need to be rolled back.
|
||||
In this case, the <methodname>clear</methodname> method on the writer
|
||||
would be called, indicating that it should <methodname>clear</methodname>
|
||||
its buffer, and <methodname>reset</methodname> would be called on the
|
||||
<classname>ItemReader</classname>, indicating that it should return back
|
||||
to the last position it was at when <methodname>mark</methodname> was
|
||||
called. (Both <methodname>mark</methodname> and
|
||||
<methodname>flush</methodname> are called on commit)</para>
|
||||
|
||||
<para>In 2.0, this strategy has been changed to a chunk-oriented
|
||||
approach:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center"
|
||||
fileref="images/simplified-chunk-oriented-processing.png"
|
||||
scale="90" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/simplified-chunk-oriented-processing.png"
|
||||
scale="90" width="60%" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>Using the same example from above, if the commit interval is five,
|
||||
read will be called 5 times, and write once. The items read will be
|
||||
aggregated into a list, that will ultimately be written out, as the
|
||||
simplified example below illustrates:</para>
|
||||
|
||||
<programlisting>
|
||||
List items = new Arraylist();
|
||||
for(int i = 0; i < commitInterval; i++){
|
||||
items.add(itemReader.read());
|
||||
}
|
||||
itemWriter.write(items);
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>This approach not only allows for much simpler processing and
|
||||
scalability approaches, it also makes the
|
||||
<classname>ItemReader</classname> and <classname>ItemWriter</classname>
|
||||
interfaces much cleaner:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader<T> {
|
||||
|
||||
T read() throws Exception, UnexpectedInputException, ParseException;
|
||||
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemWriter<T> {
|
||||
|
||||
void write(List<? extends T> items) throws Exception;
|
||||
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>As you can see, the interfaces no longer contain the
|
||||
<methodname>mark</methodname>, <methodname>reset</methodname>,
|
||||
<methodname>flush</methodname>, and <methodname>clear</methodname>
|
||||
methods. This makes the creation of readers and writers much more
|
||||
straightforward for developers. In the case of
|
||||
<classname>ItemReader</classname>, the interface is now forward-only. The
|
||||
framework will buffer read items for developers in the case of rollback
|
||||
(though there are exceptions if the underlying resource is transactional
|
||||
see: <xref linkend="transactionalReaders" />).
|
||||
<classname>ItemWriter</classname> is also simplified, since it gets the
|
||||
entire 'chunk' of items at once, rather than one at a time, it can decide
|
||||
to flush any resources (such as a file or hibernate session) before
|
||||
returning control to the <classname>Step</classname>. More detailed
|
||||
information on chunk-oriented processing can be found in <xref
|
||||
linkend="chunkOrientedProcessing" />. Reader and writer implementation
|
||||
information can be found in <xref linkend="readersAndWriters" />.</para>
|
||||
|
||||
<section>
|
||||
<title>ItemProcessor</title>
|
||||
|
||||
<para>Previously, <classname>Step</classname>s had only two
|
||||
dependencies, <classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname>:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/1-1-step.png" scale="80"
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/1-1-step.png" scale="80"
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>The basic configuration above is fairly robust. However, there are
|
||||
many cases where the item needs to be transformed before writing. In 1.x
|
||||
this can be achieved using the composite pattern:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/composite-transformer.png"
|
||||
scale="" width="70%" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/composite-transformer.png"
|
||||
scale="65" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>This approach works. However, it requires an extra layer between
|
||||
either the reader or the writer and the <classname>Step</classname>.
|
||||
Furthermore, the <classname>ItemWriter</classname> would need to be
|
||||
registered separately as an <classname>ItemStream</classname> with the
|
||||
<classname>Step</classname>. For this reason, the
|
||||
<classname>ItemTransfomer</classname> was renamed to
|
||||
<classname>ItemProcessor</classname> and moved up to the same level as
|
||||
<classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname>:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/step.png" scale=""
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center" contentwidth="480"
|
||||
fileref="images/step.png"
|
||||
scale="60" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Configuration enhancements</title>
|
||||
|
||||
<para>Until 2.0, the only option for configuring batch jobs has been
|
||||
normal spring bean configuration. However, in 2.0 there is a new namespace
|
||||
for configuration. For example, in 1.1, configuring a job looked like the
|
||||
following:</para>
|
||||
|
||||
<programlisting>
|
||||
<bean id="footballJob"
|
||||
class="org.springframework.batch.core.job.SimpleJob">
|
||||
<property name="steps">
|
||||
<list>
|
||||
<!-- Step Bean details ommitted for clarity -->
|
||||
<bean id="playerload" parent="simpleStep" />
|
||||
<bean id="gameLoad" parent="simpleStep" />
|
||||
<bean id="playerSummarization" parent="simpleStep" />
|
||||
</list>
|
||||
</property>
|
||||
<property name="jobRepository" ref="jobRepository" />
|
||||
</bean>
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>In 2.0, the equivalent would be:</para>
|
||||
|
||||
<programlisting>
|
||||
<job id="footballJob">
|
||||
<step id="playerload" next="gameLoad"/>
|
||||
<step id="gameLoad" next="playerSummarization"/>
|
||||
<step id="playerSummarization"/>
|
||||
</job>
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>More information on how to configure Jobs and Steps with the new
|
||||
namespace can be found in <xref linkend="configureJob" />, and <xref
|
||||
linkend="configureStep" />.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Meta Data access improvements</title>
|
||||
|
||||
<para>The <classname>JobRepository</classname> interface represents basic
|
||||
CRUD operations with <classname>Job</classname> meta-data. However, it may
|
||||
also be useful to query the meta-data. For that reason, the
|
||||
<classname>JobExplorer</classname> and <classname>JobOperator</classname>
|
||||
interfaces have been created:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/job-repository-advanced.png"
|
||||
scale="90" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/job-repository-advanced.png"
|
||||
scale="70" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>More information on the new meta data features can be found in <xref
|
||||
linkend="advancedMetaData" />. It is also worth noting that Jobs can now
|
||||
be stopped via the database, removing the requirement to maintain a handle
|
||||
to the <classname>JobExecution</classname> on the JVM the job was launched
|
||||
in.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Non Sequential Step Execution</title>
|
||||
|
||||
<para>2.0 has also seen improvements in how steps can be configured.
|
||||
Rather than requiring that they solely be sequential:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/sequential-flow.png"
|
||||
scale="" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/sequential-flow.png"
|
||||
scale="80" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>They may now be conditional:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/conditional-flow.png"
|
||||
scale="80" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/conditional-flow.png"
|
||||
scale="80" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>This new 'conditional flow' support is made easy to configure via
|
||||
the new namespace:</para>
|
||||
|
||||
<programlisting>
|
||||
<job id="job">
|
||||
<step id="stepA">
|
||||
<next on="FAILED" to="stepB" />
|
||||
<next on="*" to="stepC" />
|
||||
</step>
|
||||
<step id="stepB" next="stepC" />
|
||||
<step id="stepC" />
|
||||
</job>
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>More details on how to configure non sequential steps can be found
|
||||
in <xref linkend="controllingStepFlow" />.</para>
|
||||
</section>
|
||||
</chapter>
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
|
||||
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
|
||||
<chapter id="whatsNew">
|
||||
<title>What's new in Spring Batch 2.0</title>
|
||||
|
||||
<para>The Spring Batch 2.0 release has six major themes:</para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>Java 5</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Non Sequential Step Execution</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Chunk oriented processing</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Meta Data enhancements</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Scalability</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>Configuration</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
|
||||
<section>
|
||||
<title id="s.2.1.6">Java 5</title>
|
||||
|
||||
<para>The 1.x releases of Spring Batch were all based on Java 1.4. This
|
||||
prevented the framework from using many enhancements provided in Java 5
|
||||
such as generics, parameterized types, etc. The entire framework has been
|
||||
updated to utilize these features. As a result, <emphasis role="bold">Java
|
||||
1.4 is no longer supported.</emphasis> Most of the interfaces developers
|
||||
work with have been updated to support generic types. As an example, the
|
||||
<classname>ItemReader</classname> interface from 1.1 is below:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader {
|
||||
|
||||
Object read() throws Exception;
|
||||
|
||||
void mark() throws MarkFailedException;
|
||||
|
||||
void reset() throws ResetFailedException;
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>As you can see, the <methodname>read</methodname> method returns an
|
||||
<classname>Object</classname>. The 2.0 version is below:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader<T> {
|
||||
|
||||
T read() throws Exception, UnexpectedInputException, ParseException;
|
||||
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>As you can see, <classname>ItemReader</classname> now supports the
|
||||
generic type, T, which is returned from read. You may also notice that
|
||||
<methodname>mark</methodname> and <methodname>reset</methodname> have been
|
||||
removed. This is due to step processing strategy changes, which are
|
||||
discussed below. Many other interfaces have been similarly updated.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Chunk Oriented Processing</title>
|
||||
|
||||
<para>Previously, the default processing strategy provided by Spring Batch
|
||||
was item-oriented processing:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center"
|
||||
fileref="images/item-oriented-processing.png" scale="90"
|
||||
width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/item-oriented-processing.png" scale="90"
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>In item-oriented processing, the <classname>ItemReader</classname>
|
||||
returns one <classname>Object</classname> (the 'item') which is then
|
||||
handed to the <classname>ItemWriter</classname>, periodically committing
|
||||
when the number of items hits the commit interval. For example, if the
|
||||
commit interval is 5, <classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname> will each be called 5 times. This is
|
||||
illustrated in a simplified code example below:</para>
|
||||
|
||||
<programlisting>
|
||||
for(int i = 0; i < commitInterval; i++){
|
||||
Object item = itemReader.read();
|
||||
itemWriter.write(item);
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>Both the <classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname> interfaces were completely geared toward
|
||||
this approach:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader {
|
||||
|
||||
Object read() throws Exception;
|
||||
|
||||
void mark() throws MarkFailedException;
|
||||
|
||||
void reset() throws ResetFailedException;
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<programlisting> public interface ItemWriter {
|
||||
|
||||
void write(Object item) throws Exception;
|
||||
|
||||
void flush() throws FlushFailedException;
|
||||
|
||||
void clear() throws ClearFailedException;
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>Because the 'scope' of the processing was one item, supporting
|
||||
rollback scenarios required additional methods, which is what
|
||||
<methodname>mark</methodname>, <methodname>reset</methodname>,
|
||||
<methodname>flush</methodname>, and <methodname>clear</methodname>
|
||||
provided. If, after successfully reading and writing 2 items, the third
|
||||
has an error while writing, the transaction would need to be rolled back.
|
||||
In this case, the <methodname>clear</methodname> method on the writer
|
||||
would be called, indicating that it should <methodname>clear</methodname>
|
||||
its buffer, and <methodname>reset</methodname> would be called on the
|
||||
<classname>ItemReader</classname>, indicating that it should return back
|
||||
to the last position it was at when <methodname>mark</methodname> was
|
||||
called. (Both <methodname>mark</methodname> and
|
||||
<methodname>flush</methodname> are called on commit)</para>
|
||||
|
||||
<para>In 2.0, this strategy has been changed to a chunk-oriented
|
||||
approach:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center"
|
||||
fileref="images/simplified-chunk-oriented-processing.png"
|
||||
scale="90" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center"
|
||||
fileref="images/simplified-chunk-oriented-processing.png"
|
||||
scale="90" width="60%" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>Using the same example from above, if the commit interval is five,
|
||||
read will be called 5 times, and write once. The items read will be
|
||||
aggregated into a list, that will ultimately be written out, as the
|
||||
simplified example below illustrates:</para>
|
||||
|
||||
<programlisting>
|
||||
List items = new Arraylist();
|
||||
for(int i = 0; i < commitInterval; i++){
|
||||
items.add(itemReader.read());
|
||||
}
|
||||
itemWriter.write(items);
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>This approach not only allows for much simpler processing and
|
||||
scalability approaches, it also makes the
|
||||
<classname>ItemReader</classname> and <classname>ItemWriter</classname>
|
||||
interfaces much cleaner:</para>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemReader<T> {
|
||||
|
||||
T read() throws Exception, UnexpectedInputException, ParseException;
|
||||
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<programlisting>
|
||||
public interface ItemWriter<T> {
|
||||
|
||||
void write(List<? extends T> items) throws Exception;
|
||||
|
||||
}
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>As you can see, the interfaces no longer contain the
|
||||
<methodname>mark</methodname>, <methodname>reset</methodname>,
|
||||
<methodname>flush</methodname>, and <methodname>clear</methodname>
|
||||
methods. This makes the creation of readers and writers much more
|
||||
straightforward for developers. In the case of
|
||||
<classname>ItemReader</classname>, the interface is now forward-only. The
|
||||
framework will buffer read items for developers in the case of rollback
|
||||
(though there are exceptions if the underlying resource is transactional
|
||||
see: <xref linkend="transactionalReaders" />).
|
||||
<classname>ItemWriter</classname> is also simplified, since it gets the
|
||||
entire 'chunk' of items at once, rather than one at a time, it can decide
|
||||
to flush any resources (such as a file or hibernate session) before
|
||||
returning control to the <classname>Step</classname>. More detailed
|
||||
information on chunk-oriented processing can be found in <xref
|
||||
linkend="chunkOrientedProcessing" />. Reader and writer implementation
|
||||
information can be found in <xref linkend="readersAndWriters" />.</para>
|
||||
|
||||
<section>
|
||||
<title>ItemProcessor</title>
|
||||
|
||||
<para>Previously, <classname>Step</classname>s had only two
|
||||
dependencies, <classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname>:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/1-1-step.png" scale="80"
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center" fileref="images/1-1-step.png" scale="80"
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>The basic configuration above is fairly robust. However, there are
|
||||
many cases where the item needs to be transformed before writing. In 1.x
|
||||
this can be achieved using the composite pattern:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/composite-transformer.png"
|
||||
scale="" width="70%" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center" fileref="images/composite-transformer.png"
|
||||
scale="65" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>This approach works. However, it requires an extra layer between
|
||||
either the reader or the writer and the <classname>Step</classname>.
|
||||
Furthermore, the <classname>ItemWriter</classname> would need to be
|
||||
registered separately as an <classname>ItemStream</classname> with the
|
||||
<classname>Step</classname>. For this reason, the
|
||||
<classname>ItemTransfomer</classname> was renamed to
|
||||
<classname>ItemProcessor</classname> and moved up to the same level as
|
||||
<classname>ItemReader</classname> and
|
||||
<classname>ItemWriter</classname>:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/step.png" scale=""
|
||||
width="50%" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center" contentwidth="480"
|
||||
fileref="images/step.png" scale="60" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Configuration enhancements</title>
|
||||
|
||||
<para>Until 2.0, the only option for configuring batch jobs has been
|
||||
normal spring bean configuration. However, in 2.0 there is a new namespace
|
||||
for configuration. For example, in 1.1, configuring a job looked like the
|
||||
following:</para>
|
||||
|
||||
<programlisting>
|
||||
<bean id="footballJob"
|
||||
class="org.springframework.batch.core.job.SimpleJob">
|
||||
<property name="steps">
|
||||
<list>
|
||||
<!-- Step Bean details ommitted for clarity -->
|
||||
<bean id="playerload" parent="simpleStep" />
|
||||
<bean id="gameLoad" parent="simpleStep" />
|
||||
<bean id="playerSummarization" parent="simpleStep" />
|
||||
</list>
|
||||
</property>
|
||||
<property name="jobRepository" ref="jobRepository" />
|
||||
</bean>
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>In 2.0, the equivalent would be:</para>
|
||||
|
||||
<programlisting>
|
||||
<job id="footballJob">
|
||||
<step id="playerload" next="gameLoad"/>
|
||||
<step id="gameLoad" next="playerSummarization"/>
|
||||
<step id="playerSummarization"/>
|
||||
</job>
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>More information on how to configure Jobs and Steps with the new
|
||||
namespace can be found in <xref linkend="configureJob" />, and <xref
|
||||
linkend="configureStep" />.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Meta Data access improvements</title>
|
||||
|
||||
<para>The <classname>JobRepository</classname> interface represents basic
|
||||
CRUD operations with <classname>Job</classname> meta-data. However, it may
|
||||
also be useful to query the meta-data. For that reason, the
|
||||
<classname>JobExplorer</classname> and <classname>JobOperator</classname>
|
||||
interfaces have been created:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/job-repository-advanced.png"
|
||||
scale="90" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center" fileref="images/job-repository-advanced.png"
|
||||
scale="70" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>More information on the new meta data features can be found in <xref
|
||||
linkend="advancedMetaData" />. It is also worth noting that Jobs can now
|
||||
be stopped via the database, removing the requirement to maintain a handle
|
||||
to the <classname>JobExecution</classname> on the JVM the job was launched
|
||||
in.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Non Sequential Step Execution</title>
|
||||
|
||||
<para>2.0 has also seen improvements in how steps can be configured.
|
||||
Rather than requiring that they solely be sequential:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/sequential-flow.png"
|
||||
scale="" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center" fileref="images/sequential-flow.png"
|
||||
scale="80" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>They may now be conditional:</para>
|
||||
|
||||
<mediaobject>
|
||||
<imageobject role="html">
|
||||
<imagedata align="center" fileref="images/conditional-flow.png"
|
||||
scale="80" width="" />
|
||||
</imageobject>
|
||||
|
||||
<imageobject role="fo">
|
||||
<imagedata align="center" fileref="images/conditional-flow.png"
|
||||
scale="80" width="" />
|
||||
</imageobject>
|
||||
</mediaobject>
|
||||
|
||||
<para>This new 'conditional flow' support is made easy to configure via
|
||||
the new namespace:</para>
|
||||
|
||||
<programlisting>
|
||||
<job id="job">
|
||||
<step id="stepA">
|
||||
<next on="FAILED" to="stepB" />
|
||||
<next on="*" to="stepC" />
|
||||
</step>
|
||||
<step id="stepB" next="stepC" />
|
||||
<step id="stepC" />
|
||||
</job>
|
||||
|
||||
</programlisting>
|
||||
|
||||
<para>More details on how to configure non sequential steps can be found
|
||||
in <xref linkend="controllingStepFlow" />.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Scalability</title>
|
||||
|
||||
<para>Spring Batch 1.x was always intended as a single VM, possibly
|
||||
multi-threaded model, but many features were built into it that support
|
||||
parallel execution in multiple processes. Many projects have successfully
|
||||
implemented a scalable solution relying on the quality of service features
|
||||
of Spring Batch to ensure that processing only happens in the correct
|
||||
sequence. In 2.0 those features have been exposed more explicitly. There
|
||||
are two approaches to scalability: remote chunking, and
|
||||
partitioning.</para>
|
||||
|
||||
<section>
|
||||
<title>Remote Chunking</title>
|
||||
|
||||
<para>Remote chunking is a technique for dividing up the work of a step
|
||||
without any explicit knowledge of the structure of the data. Any input
|
||||
source can be split up dynamically by reading it in a single process (as
|
||||
per normal in 1.x) and sending the items as a chunk to a remote worker
|
||||
process. The remote process implements a listener pattern, responding to
|
||||
the request, processing the data and sending an asynchronous reply. The
|
||||
transport for the request and reply has to be durable with guaranteed
|
||||
delivery and a single consumer, and those features are readily available
|
||||
with any JMS implementation. But Spring Batch is building the remote
|
||||
chunking feature on top of Spring Integration, therefore it is agnostic
|
||||
to the actual implementation of the message middleware. More details can
|
||||
be found in <xref linkend="remoteChunking" /></para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Partitioning</title>
|
||||
|
||||
<para>Partitioning is an alternative approach which in contrast depends
|
||||
on having some knowledge of the structure of the input data, like a
|
||||
range of primary keys, or the name of a file to process. The advantage
|
||||
of this model is that the processors of each element in a partition can
|
||||
act as if they are a single step in a normal Spring Batch job. They
|
||||
don't have to implement any special or new patterns, which makes them
|
||||
easy to configure and test. Partitioning in principle is more scalable
|
||||
than remote chunking because there is no serialization bottleneck
|
||||
arising from reading all the input data in one place.</para>
|
||||
|
||||
<para>In Spring Batch 2.0 partitioning is supported by two interfaces:
|
||||
<classname>PartitionHandler</classname> and
|
||||
<classname>StepExecutionSplitter</classname>. The
|
||||
<classname>PartitionHandler</classname> is the one that knows about the
|
||||
execution fabric - it has to transmit requests to remote steps and
|
||||
collect the results using whatever grid or remoting technology is
|
||||
available. <classname>PartitionHandler</classname> is an SPI, and Spring
|
||||
Batch provides one implementation out of the box for local execution
|
||||
through a <classname>TaskExecutor</classname>. This will be useful
|
||||
immediately when parallel processing of heavily IO bound tasks is
|
||||
required, since in those cases remote execution only complicates the
|
||||
deployment and doesn't necessarily help much with the performance. Other
|
||||
implementations will be specific to the execution fabric. (e.g. one of
|
||||
the grid providers such as IBM, Oracle, Terracotta, Appistry etc.),
|
||||
Spring Batch makes no preference for any of grid provider over another.
|
||||
More details can be found in <xref linkend="partitioning" /></para>
|
||||
</section>
|
||||
</section>
|
||||
</chapter>
|
||||
|
||||
Reference in New Issue
Block a user