Add initial 3.0 documentation
* What's new
* JSR-352
* Convert Spring Batch Integration apt files into docbook and use as chapter
* Remove site dir from Spring Batch Integration where converted apt files used
to live
This commit is contained in:
Chris Schaefer
committed by
Michael Minella
Michael Minella
parent
de7df9c1f1
commit
1ad8bd803a
@@ -1,68 +0,0 @@
|
||||
------
|
||||
Remote Chunking
|
||||
------
|
||||
Dave Syer
|
||||
------
|
||||
March 2008
|
||||
|
||||
Remote Chunking Implementation
|
||||
|
||||
* Basic Use Case
|
||||
|
||||
Description:
|
||||
|
||||
[[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit)
|
||||
|
||||
[[1]] Worker thread picks up chunk and processes it
|
||||
|
||||
[[1]] Worker thread replies to response channel
|
||||
|
||||
[[1]] Step picks up reply if there is one and aggregates the counts
|
||||
|
||||
[[1]] Step reapeats until no more input data
|
||||
|
||||
[[1]] Step blocks until all the outstanding requests are satisfied
|
||||
|
||||
* Implementation
|
||||
|
||||
A ChunkProcessor acts as a kind of Throttling Asynchronous Messaging
|
||||
Gateway, which isn't a pattern that is supported out of the box with
|
||||
Spring Integration. There is a SimpleMessagingGateway that provides
|
||||
programmatic access to send and receive payloads (instead of
|
||||
messages), so the pattern can be manually implemented in the
|
||||
ChunkProcessor.
|
||||
|
||||
The current implementation is in the form of an ItemWriter
|
||||
(ChunkMessageChannelItemWriter) which is a StepExecutionListener
|
||||
(blocks and waits for the outstanding responses in the afterStep).
|
||||
The ChunkProcessor can then simply be a vanilla implementation from
|
||||
Spring Batch.
|
||||
|
||||
The ChunkMessageChannelItemWriter implements the Throttling part of
|
||||
the pattern by keeping track of the number of outstanding requests
|
||||
(which it has to do anyway) and blocking until a response arrives if
|
||||
the number is above a configurable limit. It wouldn't be necessary
|
||||
to do this manually in the writer if the messages were only going
|
||||
over local MessageChannels: the requests would either be processed
|
||||
serially in a single thread, or else there would be a thread pool
|
||||
with limited size controlling the workers. But since the messages
|
||||
are going to JMS we need to either explicitly throttle in the writer
|
||||
(or else rely on vendor features for producer flow control),
|
||||
otherwise the JMS Queue could easily be overwhelmed and start
|
||||
barfing (which happened in one of the early prototypes on an
|
||||
Accenture project).
|
||||
|
||||
Throttling MessageChannel.send() might be something Spring
|
||||
Integration could do, but it only makes sense really in the context
|
||||
of this gateway pattern (because you need something to react against
|
||||
to decide when to release another send).
|
||||
|
||||
The gateway is used to send requests to the workers, and then to
|
||||
receive responses in the same thread, but only waiting for a
|
||||
response when the step is complete. To implement this with JMS
|
||||
backed channels we need a Spring Integration inbound adapter that
|
||||
translates PollableChannel.receive() into
|
||||
JmsTemplate.receiveAndConvert(). In the unlikely event of a problem
|
||||
in the receiver the JMS message should roll back.
|
||||
JmsDestinationPollingAdapter actually almost does what we need but
|
||||
there is no support for configuring it without a scheduled poller.
|
||||
@@ -1,187 +0,0 @@
|
||||
------
|
||||
Spring Integration Batch
|
||||
------
|
||||
Dave Syer
|
||||
------
|
||||
March 2008
|
||||
|
||||
Overview of the Spring Integration Batch Module
|
||||
|
||||
Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring Integration. Here is a list. These are features that can extend Spring Batch, or use Spring batch features in the context of Spring Integration. Work in progress waiting for community feedback. Many issues to do with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result of these use cases being prototyped.
|
||||
|
||||
*---+---+---+---+---+
|
||||
|<<ID>>|<<Description>>|<<Status>>|<<Sub-package>>|<<Comments>>|
|
||||
*----
|
||||
|1|{{{Triggers}Message triggers job}}|Complete|launch|Complete. Also lots of opportunities with monitoring progress.|
|
||||
*----
|
||||
|2|{{{Chunking}Chunking and multi-VM job execution}}|Complete|chunk|Failures might need some analysis. Use of stateful StepExecutionListener requires use of step scope.|
|
||||
*----
|
||||
|3|{{{Aggregator}Asynchronous Aggregator}}|Unstarted| | |
|
||||
*----
|
||||
|4|{{{jobs}Stateful and non-linear jobs}} -> job = flow|Complete|job|Simple use cases work well with Spring Batch 2.0 and no Integration features.|
|
||||
*----
|
||||
|5|{{{Flexible}Flexible item processing model}} (as message flow) -> step = flow|Complete|item|Complete (v. simple using MessagingGateway). Unit tests only.|
|
||||
*----
|
||||
|6|{{{repeat}Automatic repeat / retry}}|Complete|retry (unit test)|Unit tests only, since it just uses existing features.|
|
||||
*----
|
||||
|7|{{{files}Restartable file processing}}|Complete|file|Seems to hang together. Not tested thoroughly, but apparently someone is using it.|
|
||||
*----
|
||||
|8|{{{async}Asynchronous item processing}}|Complete|async|A general purpose ItemProcesor that returns a Future.|
|
||||
*----
|
||||
|
||||
Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes. If we implement 1, then we also don't need to do any more scheduling and triggering in Spring Batch.
|
||||
|
||||
Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one. We implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration later (with probably a split of repeat/retry out of Batch at that time).
|
||||
|
||||
* Message {Triggers} Job
|
||||
|
||||
Description:
|
||||
|
||||
[[1]] User sends message to channel (maybe through a scheduler)
|
||||
|
||||
[[1]] System interprets message payload as parameters for JobLauncher
|
||||
|
||||
[[1]] System launches job execution
|
||||
|
||||
[[1]] If message had a replyTo, System acknowledges with JobExecution
|
||||
|
||||
[[1]] User accepts response and uses it to monitor progress
|
||||
|
||||
Variation:
|
||||
|
||||
[[1]] System waits for job to finish and replies when it is over
|
||||
|
||||
[[1]] User polls for replies and gets notification about end of execution
|
||||
|
||||
Variation:
|
||||
|
||||
[[1]] User wants to block on send and only receive response when job is done
|
||||
|
||||
* {Chunking} and Multi-VM
|
||||
|
||||
Description:
|
||||
|
||||
[[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit)
|
||||
|
||||
[[1]] Worker thread picks up chunk and processes it
|
||||
|
||||
[[1]] Worker thread replies to response channel
|
||||
|
||||
[[1]] Step picks up reply and aggregates the counts
|
||||
|
||||
[[1]] Step blocks until all the requests are satisfied
|
||||
|
||||
TODO: failure modes
|
||||
|
||||
* Asynchronous {Aggregator}
|
||||
|
||||
Job is executed over long period. Many jobs can be executing concurrently.
|
||||
|
||||
Description:
|
||||
|
||||
[[1]] Input stage for each job: System reads all items and marks with the job instance id in a durable repository (staging table)
|
||||
|
||||
[[1]] System sends each item (or chunks of items that can be processed together as appropriate) to a channel
|
||||
|
||||
[[1]] Items flow through message pipeline, occasionally pausing until certain conditions are met, possibly for days at a time
|
||||
|
||||
[[1]] Aggregator sits and waits for all items in a job to be finished and then wraps up
|
||||
|
||||
* Stateful and non-linear {jobs}
|
||||
|
||||
Dependencies beyween steps and conditional flow between steps. Each handler node in a message flow is a step execution, with all the robustness guarantees from the Spring Batch meta data.
|
||||
|
||||
Description:
|
||||
|
||||
[[1]] User launches job
|
||||
|
||||
[[1]] System sends message to channel containing job execution
|
||||
|
||||
[[1]] Handler accepts message and executes a step
|
||||
|
||||
[[1]] Handler translates result of step execution into the same form that it accepted the original request
|
||||
|
||||
[[1]] System routes message to next handler, possibly dynamically based on data in the message
|
||||
|
||||
[[1]] Next handler does the same... until one of the routing decisions leads to a reply channel
|
||||
|
||||
[[1]] System receives reply and transfers information to job execution (e.g. status) as necessary
|
||||
|
||||
Variation: failure in one of the handlers
|
||||
|
||||
Variation: restart after failure
|
||||
|
||||
* {Flexible} item processing model
|
||||
|
||||
Description:
|
||||
|
||||
[[1]] Step hands item to ItemWriter
|
||||
|
||||
[[1]] Item is converted to message and sent to synchronous flow
|
||||
|
||||
[[1]] Handler accepts message and does something with item
|
||||
|
||||
[[1]] System routes result to next handler, possibly dynamically
|
||||
|
||||
Variation: failure
|
||||
|
||||
[[1]] Handler throws exception
|
||||
|
||||
[[1]] System propagates exception up to ItemWriter (forces rollback under normal circs - hence synchronous flow)
|
||||
|
||||
* Automatic repeat / retry
|
||||
|
||||
Description ({repeat}):
|
||||
|
||||
[[1]] User sends message to channel
|
||||
|
||||
[[1]] System start a transaction and reseives message, then processes it
|
||||
|
||||
[[1]] User sends another message
|
||||
|
||||
[[1]] System receives and processes it in the same transaction
|
||||
|
||||
[[1]] ... repeat ...
|
||||
|
||||
[[1]] System determines that batch is complete and commits transaction
|
||||
|
||||
* Restartable file processing
|
||||
|
||||
Large {files} need to be processed, so message payload of file contents is not practical. One line or XML event per message with failover and restartability from Spring Batch.
|
||||
|
||||
Description:
|
||||
|
||||
[[1]] User triggers file processing (sends message, copies file to directory, etc.)
|
||||
|
||||
[[1]] System starts new job
|
||||
|
||||
[[1]] System processes file line by line (or even by event), wrapping each one as a message and sending it to a synchronous flow
|
||||
|
||||
[[1]] System commits periodically (as determined by Spring Batch step configuration)
|
||||
|
||||
Variation: failure and restart
|
||||
|
||||
[[1]] Item processing fails
|
||||
|
||||
[[1]] System aborts job and sends message to failure channel (or failure message to normal reply channel)
|
||||
|
||||
[[1]] Operator fixes problem and triggers restart (another message channel?)
|
||||
|
||||
[[1]] System restarts job for same file at point where it left off
|
||||
|
||||
[[1]] System completes processing
|
||||
|
||||
[[1]] System sends sucess message to reply channel
|
||||
|
||||
Variation: send to asynchronous flow. Same as main use case but item message is sent to asynchronous flow. Not as robust because if the lights go out then meesages will be lost, but at least a large file can be split into smaller chunks.
|
||||
|
||||
* Asynchronous item processing
|
||||
|
||||
This is actually a variation on {{{Flexible}flexible item processing model}}.
|
||||
|
||||
Description ({async}):
|
||||
|
||||
[[1]] ItemProcessor executes in background (non-transactionally)
|
||||
|
||||
[[1]] ItemWriter collects outputs from futures before phyically writing data
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
<?xml version="1.0" encoding="ISO-8859-1"?>
|
||||
<project name="Spring Batch: ${project.name}">
|
||||
|
||||
<body>
|
||||
|
||||
<links>
|
||||
<item name="${project.name}" href="index.html"/>
|
||||
</links>
|
||||
|
||||
<menu name="Documentation">
|
||||
<item name="${project.name}" href="index.html"/>
|
||||
</menu>
|
||||
|
||||
<menu ref="reports"/>
|
||||
|
||||
</body>
|
||||
</project>
|
||||
Reference in New Issue
Block a user