                                    ------
                                    Spring Integration Batch
                                    ------
                                    Dave Syer
                                    ------
                                    March 2008

Overview of the Spring Integration Batch Module

  Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring Integration.  Here is a list.  These are features that can extend Spring Batch, or use Spring batch features in the context of Spring Integration.  Work in progress waiting for community feedback.  Many issues to do with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result of these use cases being prototyped.

*---+---+---+---+---+
|<<ID>>|<<Description>>|<<Status>>|<<Sub-package>>|<<Comments>>|
*----
|1|{{{Triggers}Message triggers job}}|Complete|launch|Complete.  Also lots of opportunities with monitoring progress.|
*----
|2|{{{Chunking}Chunking and multi-VM job execution}}|Complete|chunk|Failures might need some analysis.  Use of stateful StepExecutionListener requires use of step scope.|
*----
|3|{{{Aggregator}Asynchronous Aggregator}}|Unstarted| | |
*----
|4|{{{jobs}Stateful and non-linear jobs}} -> job = flow|Complete|job|Simple use cases work well with Spring Batch 2.0 and no Integration features.|
*----
|5|{{{Flexible}Flexible item processing model}} (as message flow) -> step = flow|Complete|item|Complete (v. simple using MessagingGateway).  Unit tests only.|
*----
|6|{{{repeat}Automatic repeat / retry}}|Complete|retry (unit test)|Unit tests only, since it just uses existing features.|
*----
|7|{{{files}Restartable file processing}}|Complete|file|Seems to hang together.  Not tested thoroughly, but apparently someone is using it.|
*----
|8|{{{async}Asynchronous item processing}}|Complete|async|A general purpose ItemProcesor that returns a Future.|
*----

  Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes.  If we implement 1, then we also don't need to do any more scheduling and triggering in Spring Batch.

  Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one.  We implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration later (with probably a split of repeat/retry out of Batch at that time).

* Message {Triggers} Job

  Description:

    [[1]] User sends message to channel (maybe through a scheduler)

    [[1]] System interprets message payload as parameters for JobLauncher

    [[1]] System launches job execution
 
    [[1]] If message had a replyTo, System acknowledges with JobExecution

    [[1]] User accepts response and uses it to monitor progress

  Variation:

    [[1]] System waits for job to finish and replies when it is over

    [[1]] User polls for replies and gets notification about end of execution

  Variation:

    [[1]] User wants to block on send and only receive response when job is done

* {Chunking} and Multi-VM

  Description:

    [[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit)

    [[1]] Worker thread picks up chunk and processes it

    [[1]] Worker thread replies to response channel
 
    [[1]] Step picks up reply and aggregates the counts

    [[1]] Step blocks until all the requests are satisfied

  TODO: failure modes

* Asynchronous {Aggregator}

  Job is executed over long period.  Many jobs can be executing concurrently.  

  Description:

    [[1]] Input stage for each job: System reads all items and marks with the job instance id in a durable repository (staging table)

    [[1]] System sends each item (or chunks of items that can be processed together as appropriate) to a channel

    [[1]] Items flow through message pipeline, occasionally pausing until certain conditions are met, possibly for days at a time

    [[1]] Aggregator sits and waits for all items in a job to be finished and then wraps up

* Stateful and non-linear {jobs}

  Dependencies beyween steps and conditional flow between steps.  Each handler node in a message flow is a step execution, with all the robustness guarantees from the Spring Batch meta data.

  Description:

    [[1]] User launches job
 
    [[1]] System sends message to channel containing job execution

    [[1]] Handler accepts message and executes a step

    [[1]] Handler translates result of step execution into the same form that it accepted the original request

    [[1]] System routes message to next handler, possibly dynamically based on data in the message

    [[1]] Next handler does the same... until one of the routing decisions leads to a reply channel

    [[1]] System receives reply and transfers information to job execution (e.g. status) as necessary

  Variation: failure in one of the handlers

  Variation: restart after failure

* {Flexible} item processing model

  Description:

    [[1]] Step hands item to ItemWriter

    [[1]] Item is converted to message and sent to synchronous flow

    [[1]] Handler accepts message and does something with item

    [[1]] System routes result to next handler, possibly dynamically

  Variation: failure

    [[1]] Handler throws exception

    [[1]] System propagates exception up to ItemWriter (forces rollback under normal circs - hence synchronous flow)

* Automatic repeat / retry

  Description ({repeat}):

    [[1]] User sends message to channel

    [[1]] System start a transaction and reseives message, then processes it

    [[1]] User sends another message

    [[1]] System receives and processes it in the same transaction

    [[1]] ... repeat ...

    [[1]] System determines that batch is complete and commits transaction

* Restartable file processing

  Large {files} need to be processed, so message payload of file contents is not practical.  One line or XML event per message with failover and restartability from Spring Batch.

  Description:

    [[1]] User triggers file processing (sends message, copies file to directory, etc.)

    [[1]] System starts new job

    [[1]] System processes file line by line (or even by event), wrapping each one as a message and sending it to a synchronous flow

    [[1]] System commits periodically (as determined by Spring Batch step configuration)

  Variation: failure and restart

    [[1]] Item processing fails

    [[1]] System aborts job and sends message to failure channel (or failure message to normal reply channel)

    [[1]] Operator fixes problem and triggers restart (another message channel?)

    [[1]] System restarts job for same file at point where it left off

    [[1]] System completes processing

    [[1]] System sends sucess message to reply channel

  Variation: send to asynchronous flow.  Same as main use case but item message is sent to asynchronous flow.  Not as robust because if the lights go out then meesages will be lost, but at least a large file can be split into smaller chunks.

* Asynchronous item processing

  This is actually a variation on {{{Flexible}flexible item processing model}}.

  Description ({async}):

    [[1]] ItemProcessor executes in background (non-transactionally)

    [[1]] ItemWriter collects outputs from futures before phyically writing data

