117 lines
12 KiB
HTML
117 lines
12 KiB
HTML
<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE html><html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xmlns:m="http://www.w3.org/1998/Math/MathML" xmlns:pls="http://www.w3.org/2005/01/pronunciation-lexicon" xmlns:ssml="http://www.w3.org/2001/10/synthesis" xmlns:svg="http://www.w3.org/2000/svg"><head><title>Partitioning</title><link rel="stylesheet" type="text/css" href="docbook-epub.css"/><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"/><link rel="prev" href="ch07s03.xhtml" title="Remote Chunking"/><link rel="next" href="ch08.xhtml" title="Chapter 8. Repeat"/></head><body><header/><section class="section" title="Partitioning" epub:type="subchapter" id="partitioning"><div class="titlepage"><div><div><h2 class="title" style="clear: both">Partitioning</h2></div></div></div><p>Spring Batch also provides an SPI for partitioning a Step execution
|
||
and executing it remotely. In this case the remote participants are simply
|
||
Step instances that could just as easily have been configured and used for
|
||
local processing. Here is a picture of the pattern in action:</p><div style="text-align: center; " class="mediaobject"><img style="text-align: middle; " src="images/partitioning-overview.png" width="432"/></div><p>The Job is executing on the left hand side as a sequence of Steps,
|
||
and one of the Steps is labelled as a Master. The Slaves in this picture
|
||
are all identical instances of a Step, which could in fact take the place
|
||
of the Master resulting in the same outcome for the Job. The Slaves are
|
||
typically going to be remote services, but could also be local threads of
|
||
execution. The messages sent by the Master to the Slaves in this pattern
|
||
do not need to be durable, or have guaranteed delivery: Spring Batch
|
||
meta-data in the <code class="classname">JobRepository</code> will ensure that
|
||
each Slave is executed once and only once for each Job execution.</p><p>The SPI in Spring Batch consists of a special implementation of Step
|
||
(the <code class="classname">PartitionStep</code>), and two strategy interfaces
|
||
that need to be implemented for the specific environment. The strategy
|
||
interfaces are <code class="classname">PartitionHandler</code> and
|
||
<code class="classname">StepExecutionSplitter</code>, and their role is show in
|
||
the sequence diagram below:</p><div style="text-align: center; " class="mediaobject"><img style="text-align: middle; " src="images/partitioning-spi.png" width="486"/></div><p>The Step on the right in this case is the "remote" Slave, so
|
||
potentially there are many objects and or processes playing this role, and
|
||
the PartitionStep is shown driving the execution. The PartitionStep
|
||
configuration looks like this:</p><pre class="programlisting"><step id="step1.master">
|
||
<partition step="step1" partitioner="partitioner">
|
||
<handler grid-size="10" task-executor="taskExecutor"/>
|
||
</partition>
|
||
</step></pre><p>Similar to the multi-threaded step's throttle-limit
|
||
attribute, the grid-size attribute prevents the task executor from
|
||
being saturated with requests from a single step.</p><p>There is a simple example which can be copied and extended in the
|
||
unit test suite for Spring Batch Samples (see
|
||
<code class="classname">*PartitionJob.xml</code> configuration).</p><p>Spring Batch creates step executions for the partitions called
|
||
"step1:partition0", etc., so many people prefer to call the master step
|
||
"step1:master" for consistency. With Spring 3.0 you can do this using an
|
||
alias for the step (specifying the <code class="literal">name</code> attribute
|
||
instead of the <code class="literal">id</code>). </p><section class="section" title="PartitionHandler" epub:type="division" id="partitionHandler"><div class="titlepage"><div><div><h3 class="title">PartitionHandler</h3></div></div></div><p>The <code class="classname">PartitionHandler</code> is the component that
|
||
knows about the fabric of the remoting or grid environment. It is able
|
||
to send <code class="classname">StepExecution</code> requests to the remote
|
||
Steps, wrapped in some fabric-specific format, like a DTO. It does not
|
||
have to know how to split up the input data, or how to aggregate the
|
||
result of multiple Step executions. Generally speaking it probably also
|
||
doesn't need to know about resilience or failover, since those are
|
||
features of the fabric in many cases, and anyway Spring Batch always
|
||
provides restartability independent of the fabric: a failed Job can
|
||
always be restarted and only the failed Steps will be
|
||
re-executed.</p><p><code class="classname">The PartitionHandler</code> interface can have
|
||
specialized implementations for a variety of fabric types: e.g. simple
|
||
RMI remoting, EJB remoting, custom web service, JMS, Java Spaces, shared
|
||
memory grids (like Terracotta or Coherence), grid execution fabrics
|
||
(like GridGain). Spring Batch does not contain implementations for any
|
||
proprietary grid or remoting fabrics.</p><p>Spring Batch does however provide a useful implementation of
|
||
<code class="classname">PartitionHandler</code> that executes Steps locally in
|
||
separate threads of execution, using the
|
||
<code class="classname">TaskExecutor</code> strategy from Spring. The
|
||
implementation is called
|
||
<code class="classname">TaskExecutorPartitionHandler</code>, and it is the
|
||
default for a step configured with the XML namespace as above. It can
|
||
also be configured explicitly like this:</p><pre class="programlisting"><step id="step1.master">
|
||
<partition step="step1" handler="handler"/>
|
||
</step>
|
||
|
||
<bean class="org.spr...TaskExecutorPartitionHandler">
|
||
<property name="taskExecutor" ref="taskExecutor"/>
|
||
<property name="step" ref="step1" />
|
||
<property name="gridSize" value="10" />
|
||
</bean></pre><p>The <code class="literal">gridSize</code> determines the number of separate
|
||
step executions to create, so it can be matched to the size of the
|
||
thread pool in the <code class="classname">TaskExecutor</code>, or else it can
|
||
be set to be larger than the number of threads available, in which case
|
||
the blocks of work are smaller.</p><p>The <code class="classname">TaskExecutorPartitionHandler</code> is quite
|
||
useful for IO intensive Steps, like copying large numbers of files or
|
||
replicating filesystems into content management systems. It can also be
|
||
used for remote execution by providing a Step implementation that is a
|
||
proxy for a remote invocation (e.g. using Spring Remoting).</p></section><section class="section" title="Partitioner" epub:type="division" id="stepExecutionSplitter"><div class="titlepage"><div><div><h3 class="title">Partitioner</h3></div></div></div><p>The Partitioner has a simpler responsibility: to generate
|
||
execution contexts as input parameters for new step executions only (no
|
||
need to worry about restarts). It has a single method:</p><pre class="programlisting">public interface Partitioner {
|
||
Map<String, ExecutionContext> partition(int gridSize);
|
||
}</pre><p>The return value from this method associates a unique name for
|
||
each step execution (the <code class="classname">String</code>), with input
|
||
parameters in the form of an <code class="classname">ExecutionContext</code>.
|
||
The names show up later in the Batch meta data as the step name in the
|
||
partitioned <code class="classname">StepExecutions</code>. The
|
||
<code class="classname">ExecutionContext</code> is just a bag of name-value
|
||
pairs, so it might contain a range of primary keys, or line numbers, or
|
||
the location of an input file. The remote <code class="classname">Step</code>
|
||
then normally binds to the context input using <code class="literal">#{...}</code>
|
||
placeholders (late binding in step scope), as illustrated in the next
|
||
section.</p><p>The names of the step executions (the keys in the
|
||
<code class="classname">Map</code> returned by
|
||
<code class="classname">Partitioner</code>) need to be unique amongst the step
|
||
executions of a Job, but do not have any other specific requirements.
|
||
The easiest way to do this, and to make the names meaningful for users,
|
||
is to use a prefix+suffix naming convention, where the prefix is the
|
||
name of the step that is being executed (which itself is unique in the
|
||
<code class="classname">Job</code>), and the suffix is just a counter. There is
|
||
a <code class="classname">SimplePartitioner</code> in the framework that uses
|
||
this convention.</p><p>An optional interface
|
||
<code class="classname">PartitioneNameProvider</code> can be used to
|
||
provide the partition names separately from the partitions
|
||
themselves. If a <code class="classname">Partitioner</code> implements
|
||
this interface then on a restart only the names will be queried.
|
||
If partitioning is expensive this can be a useful optimisation.
|
||
Obviously the names provided by the
|
||
<code class="classname">PartitioneNameProvider</code> must match those
|
||
provided by the <code class="classname">Partitioner</code>.</p></section><section class="section" title="Binding Input Data to Steps" epub:type="division" id="bindingInputDataToSteps"><div class="titlepage"><div><div><h3 class="title">Binding Input Data to Steps</h3></div></div></div><p>It is very efficient for the steps that are executed by the
|
||
PartitionHandler to have identical configuration, and for their input
|
||
parameters to be bound at runtime from the ExecutionContext. This is
|
||
easy to do with the StepScope feature of Spring Batch (covered in more
|
||
detail in the section on <a class="xref" href="ch05s04.xhtml" title="Late Binding of Job and Step Attributes">Late Binding</a>). For example
|
||
if the <code class="classname">Partitioner</code> creates
|
||
<code class="classname">ExecutionContext</code> instances with an attribute key
|
||
<code class="literal">fileName</code>, pointing to a different file (or
|
||
directory) for each step invocation, the
|
||
<code class="classname">Partitioner</code> output might look like this:</p><div class="table" id="d5e3165"><div class="table-title">Table 7.1. Example step execution name to execution context provided by
|
||
Partitioner targeting directory processing</div><div class="table-contents"><table style="border-collapse: collapse; border-top: 0.5pt solid ; border-bottom: 0.5pt solid ; border-left: 0.5pt solid ; border-right: 0.5pt solid ; "><colgroup><col/><col/></colgroup><tbody><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; "><span class="bold"><strong>Step Execution Name
|
||
(key)</strong></span></td><td style="border-bottom: 0.5pt solid ; "><span class="bold"><strong>ExecutionContext
|
||
(value)</strong></span></td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">filecopy:partition0</td><td style="border-bottom: 0.5pt solid ; ">fileName=/home/data/one</td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">filecopy:partition1</td><td style="border-bottom: 0.5pt solid ; ">fileName=/home/data/two</td></tr><tr><td style="border-right: 0.5pt solid ; ">filecopy:partition2</td><td>fileName=/home/data/three</td></tr></tbody></table></div></div><p>Then the file name can be bound to a step using late binding to
|
||
the execution context:</p><pre class="programlisting"><bean id="itemReader" scope="step"
|
||
class="org.spr...MultiResourceItemReader">
|
||
<property name="resource" value="<span class="bold"><strong>#{stepExecutionContext[fileName]}/*</strong></span>"/>
|
||
</bean></pre></section></section><footer/></body></html> |