Tidy up old terminology in use cases.
This commit is contained in:
@@ -10,8 +10,8 @@ Use Case: Asynchronous Chunk Processing
|
||||
* Goal
|
||||
|
||||
Increased the efficiency of chunk processing by having it execute
|
||||
asynchronously: each record in a separate thread. Maintain
|
||||
transactional intergrity of the chunk.
|
||||
asynchronously: in multiple threads. Maintain transactional
|
||||
intergrity of the chunk.
|
||||
|
||||
* Scope
|
||||
|
||||
@@ -119,11 +119,11 @@ Use Case: Asynchronous Chunk Processing
|
||||
have its own threads - how would each one be able to guide its child
|
||||
processes to participate in the same transaction?
|
||||
|
||||
* Beware a framework that extracts data from an <<<ItemProvider>>>
|
||||
* Beware a framework that extracts data from an <<<ItemReader>>>
|
||||
before executing the business logic (e.g. in a
|
||||
<<<ItemProcessor>>>). It is not enough to allow concurrent
|
||||
<<<ItemWriter>>>). It is not enough to allow concurrent
|
||||
processing but simply insist that the individual records are
|
||||
processed transactionally because the <<<ItemProvider>>> will then
|
||||
processed transactionally because the <<<ItemReader>>> will then
|
||||
not be able to participate in the transaction - its next record has
|
||||
already been passed to the consumer when the transaction starts, so
|
||||
if there is a rollback then the record is lost.
|
||||
@@ -131,14 +131,14 @@ Use Case: Asynchronous Chunk Processing
|
||||
This is the origin of the signature:
|
||||
|
||||
+---
|
||||
public interface ItemProvider {
|
||||
public interface ItemReader {
|
||||
Object next();
|
||||
}
|
||||
+---
|
||||
|
||||
There is no peeking and no iteratror-style <<<hasNext>>>. If there
|
||||
There is no peeking and no iterator-style <<<hasNext>>>. If there
|
||||
is a processing problem, transactional clients of the
|
||||
<<<ItemProvider>>> throw an exception <after> the provider's
|
||||
<<<ItemReader>>> throw an exception <after> the provider's
|
||||
<<<next()>>> has been called, but in the same thread (so that
|
||||
transactional semantics are preserved and the data provider reverts
|
||||
to its previous state).
|
||||
@@ -155,5 +155,5 @@ public interface RepeatCallback {
|
||||
so we can return an object, which is null when the processing has
|
||||
finished.
|
||||
|
||||
In the end we decided against the <<<Object>>> retrun type and went
|
||||
with a boolean flag to signal (false) for no more processing.
|
||||
In the end we decided against the <<<Object>>> return type and went
|
||||
with an exit status to signal for no more processing.
|
||||
|
||||
@@ -120,14 +120,14 @@ Use Case: Commit Batch Process Periodically
|
||||
{{{simple.html}simple}} use case). The iterator could be more than
|
||||
just a loop that might terminate early: here it could also manage
|
||||
the file cursor on the input source. In this design there is a
|
||||
<<<ItemProvider>>> interface that can take care of termination and
|
||||
<<<ItemReader>>> interface that can take care of termination and
|
||||
iteration (e.g. iterator-like method signatures).
|
||||
|
||||
* Another design idea (more encapsulated and more in keeping with
|
||||
existing Spring practice) is to make the data source transaction
|
||||
aware, and for the client use it like a database resource, through a
|
||||
template. In this case there is a <<<FileInputTemplate>>>. The
|
||||
<<<ItemProvider>>> needs to be aware of the data source template, so
|
||||
<<<ItemReader>>> needs to be aware of the data source template, so
|
||||
that it can terminate when the data is exhausted.
|
||||
|
||||
In this version of events there are two kinds of resource in play.
|
||||
@@ -193,7 +193,7 @@ batchTemplate.iterate(chunkCallback);
|
||||
+---
|
||||
|
||||
The transaction boundary is demarcated at the chunk level
|
||||
(<<<chunkCallback.doWithRepeat()>>>). Thw termination policy depends
|
||||
(<<<chunkCallback.doWithRepeat()>>>). The termination policy depends
|
||||
only on a data source eventually returning null.
|
||||
|
||||
* N.B. the chunkSize can be dynamic. E.g., if the chunk is long
|
||||
@@ -201,7 +201,7 @@ batchTemplate.iterate(chunkCallback);
|
||||
in case the batch has to be terminated.
|
||||
|
||||
* Chunking can also be implemented simply in an
|
||||
<<<ExecutionHandler>>>. The handler just buffers records up to a
|
||||
<<<ItemHandler>>>. The handler just buffers records up to a
|
||||
chunk size, and then executes them all in one step (which might be
|
||||
transactional). This is easier to implement, and easier to
|
||||
configure for the clients, but cannot easily be made both concurrent
|
||||
|
||||
@@ -76,7 +76,7 @@ Use Case: Copy File to File
|
||||
* With some external limitations the write-only file source can be
|
||||
implemented so that within a single JVM it will behave like a
|
||||
transactional database datasource. We can provide a
|
||||
<<<FileOutputTemplate>>> that hides the resource acquisition and
|
||||
<<<FlatFileItemWriter>>> that hides the resource acquisition and
|
||||
release, and interacts with an existing transaction to provide the
|
||||
transactional behaviour that is required.
|
||||
|
||||
|
||||
@@ -116,7 +116,26 @@ Use Case: Massively Parallel Batch Processing
|
||||
|
||||
** Chunking
|
||||
|
||||
|
||||
The messages from a dispatcher to worker processes consist of a
|
||||
chunk of items - a set of items to be processed together in a single
|
||||
transaction (or as the worker sees fit). The dispatcher is usually
|
||||
single threaded, but this is only a restriction based on the input
|
||||
data type (if it is a file it is difficult to read in parallel and
|
||||
maintain restartability). Using a process indicator the dispatcher
|
||||
could be reading from a database table in a multi-threaded model.
|
||||
|
||||
The main restriction is that for restartability the messages between
|
||||
the dispatcher and workers has to be durable (i.e. JMS or
|
||||
equivalent). If there is a durable middleware there are no in
|
||||
principle difficulties with this approach.
|
||||
|
||||
The practicalities deserve some discussion. In particular the
|
||||
dispatcher has to co-ordinate asynchronous replies from its workers,
|
||||
and also has to avoid overwhelming the workers (so there should be
|
||||
some throttling). As long as the middleware is durable the
|
||||
dispatcher can simply wait for replies whenever it thinks there are
|
||||
workers working. It needs to record this expectation in a durable
|
||||
form as well, as part of an <<<ExecutionContext>>> for the step.
|
||||
|
||||
** Partitioning
|
||||
|
||||
|
||||
@@ -69,12 +69,12 @@ Use Case: Manual Restart After Failure
|
||||
(persist / rehydrate).
|
||||
|
||||
* The initial condition is naturally under control of the
|
||||
<<<DataProvider>>>. The client need not know about the persistence
|
||||
<<<ItemReader>>>. The client need not know about the persistence
|
||||
and rehydration. In fact explicit persistence and rehydration might
|
||||
be overkill - just relying on the transaction semantics might be
|
||||
adequate in a lot of cases. The <<<DataProvider>>> would have to be
|
||||
adequate in a lot of cases. The <<<ItemReader>>> would have to be
|
||||
aware of the transactions, which we assume are normally demarcated
|
||||
in the <<<ExecutionHandler>>>. Since the point at which persistence
|
||||
in the <<<Step>>>. Since the point at which persistence
|
||||
is needed is tied to transaction commits, there may have to be some
|
||||
transaction synchronization.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user