                                    ------
                                    Remote Chunking
                                    ------
                                    Dave Syer
                                    ------
                                    March 2008

Remote Chunking Implementation

* Basic Use Case

  Description:

    [[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit)

    [[1]] Worker thread picks up chunk and processes it

    [[1]] Worker thread replies to response channel
 
    [[1]] Step picks up reply if there is one and aggregates the counts

    [[1]] Step reapeats until no more input data

    [[1]] Step blocks until all the outstanding requests are satisfied

* Implementation

  A ChunkProcessor acts as a kind of Throttling Asynchronous Messaging
  Gateway, which isn't a pattern that is supported out of the box with
  Spring Integration.  There is a SimpleMessagingGateway that provides
  programmatic access to send and receive payloads (instead of
  messages), so the pattern can be manually implemented in the
  ChunkProcessor.

  The current implementation is in the form of an ItemWriter
  (ChunkMessageChannelItemWriter) which is a StepExecutionListener
  (blocks and waits for the outstanding responses in the afterStep).
  The ChunkProcessor can then simply be a vanilla implementation from
  Spring Batch.

  The ChunkMessageChannelItemWriter implements the Throttling part of
  the pattern by keeping track of the number of outstanding requests
  (which it has to do anyway) and blocking until a response arrives if
  the number is above a configurable limit.  It wouldn't be necessary
  to do this manually in the writer if the messages were only going
  over local MessageChannels: the requests would either be processed
  serially in a single thread, or else there would be a thread pool
  with limited size controlling the workers.  But since the messages
  are going to JMS we need to either explicitly throttle in the writer
  (or else rely on vendor features for producer flow control),
  otherwise the JMS Queue could easily be overwhelmed and start
  barfing (which happened in one of the early prototypes on an
  Accenture project).

  Throttling MessageChannel.send() might be something Spring
  Integration could do, but it only makes sense really in the context
  of this gateway pattern (because you need something to react against
  to decide when to release another send).

  The gateway is used to send requests to the workers, and then to
  receive responses in the same thread, but only waiting for a
  response when the step is complete.  To implement this with JMS
  backed channels we need a Spring Integration inbound adapter that
  translates PollableChannel.receive() into
  JmsTemplate.receiveAndConvert().  In the unlikely event of a problem
  in the receiver the JMS message should roll back.
  JmsDestinationPollingAdapter actually almost does what we need but
  there is no support for configuring it without a scheduled poller.
