() {
+ public Integer call() throws Exception {
+
+ int count = 0;
+
+ while (reader.read() != null) {
+ count++;
+ }
+ return count;
+ }
+});
+----
+
+[[validatingOutputFiles]]
+
+
+=== Validating Output Files
+
+When a batch job writes to the database, it is easy to query the
+ database to verify that the output is as expected. However, if the batch
+ job writes to a file, it is equally important that the output be verified.
+ Spring Batch provides a class `AssertFile` to
+ facilitate the verification of output files. The method
+ `assertFileEquals` takes two
+ `File` objects (or two
+ `Resource` objects) and asserts, line by line, that
+ the two files have the same content. Therefore, it is possible to create a
+ file with the expected output and to compare it to the actual
+ result:
+
+
+[source, java]
+----
+private static final String EXPECTED_FILE = "src/main/resources/data/input.txt";
+private static final String OUTPUT_FILE = "target/test-outputs/output.txt";
+
+AssertFile.assertFileEquals(new FileSystemResource(EXPECTED_FILE),
+ new FileSystemResource(OUTPUT_FILE));
+----
+
+[[mockingDomainObjects]]
+
+
+=== Mocking Domain Objects
+
+Another common issue encountered while writing unit and integration
+ tests for Spring Batch components is how to mock domain objects. A good
+ example is a `StepExecutionListener`, as illustrated
+ below:
+
+
+[source, java]
+----
+public class NoWorkFoundStepExecutionListener extends StepExecutionListenerSupport {
+
+ public ExitStatus afterStep(StepExecution stepExecution) {
+ if (stepExecution.getReadCount() == 0) {
+ throw new NoWorkFoundException("Step has not processed any items");
+ }
+ return stepExecution.getExitStatus();
+ }
+}
+----
+
+The above listener is provided by the framework and checks a
+ `StepExecution` for an empty read count, thus
+ signifying that no work was done. While this example is fairly simple, it
+ serves to illustrate the types of problems that may be encountered when
+ attempting to unit test classes that implement interfaces requiring Spring
+ Batch domain objects. Consider the above listener's unit test:
+
+
+[source, java]
+----
+private NoWorkFoundStepExecutionListener tested = new NoWorkFoundStepExecutionListener();
+
+@Test
+public void testAfterStep() {
+ StepExecution stepExecution = new StepExecution("NoProcessingStep",
+ new JobExecution(new JobInstance(1L, new JobParameters(),
+ "NoProcessingJob")));
+
+ stepExecution.setReadCount(0);
+
+ try {
+ tested.afterStep(stepExecution);
+ fail();
+ } catch (NoWorkFoundException e) {
+ assertEquals("Step has not processed any items", e.getMessage());
+ }
+}
+----
+
+Because the Spring Batch domain model follows good object orientated
+ principles, the `StepExecution` requires a
+ `JobExecution`, which requires a
+ `JobInstance` and
+ `JobParameters` in order to create a valid
+ `StepExecution`. While this is good in a solid domain
+ model, it does make creating stub objects for unit testing verbose. To
+ address this issue, the Spring Batch test module includes a factory for
+ creating domain objects: `MetaDataInstanceFactory`.
+ Given this factory, the unit test can be updated to be more
+ concise:
+
+
+[source, java]
+----
+private NoWorkFoundStepExecutionListener tested = new NoWorkFoundStepExecutionListener();
+
+@Test
+public void testAfterStep() {
+ StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution();
+
+ stepExecution.setReadCount(0);
+
+ try {
+ tested.afterStep(stepExecution);
+ fail();
+ } catch (NoWorkFoundException e) {
+ assertEquals("Step has not processed any items", e.getMessage());
+ }
+}
+----
+
+The above method for creating a simple
+ `StepExecution` is just one convenience method
+ available within the factory. A full method listing can be found in its
+ link:$$http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/test/MetaDataInstanceFactory.html$$[Javadoc].
+
diff --git a/src/site/docbook/reference/transaction-appendix.xml b/spring-batch-docs/asciidoc/transaction-appendix.adoc
similarity index 59%
rename from src/site/docbook/reference/transaction-appendix.xml
rename to spring-batch-docs/asciidoc/transaction-appendix.adoc
index b00c1bb79..0dd3b5ee4 100644
--- a/src/site/docbook/reference/transaction-appendix.xml
+++ b/spring-batch-docs/asciidoc/transaction-appendix.adoc
@@ -1,18 +1,25 @@
-
-
-
- Batch Processing and Transactions
+:batch-asciidoc: http://docs.spring.io/spring-batch/reference/html/
+:toc: left
+:toclevels: 4
-
- Simple Batching with No Retry
+[[transactions]]
- Consider the following simple example of a nested batch with no
+[appendix]
+== Batch Processing and Transactions
+
+[[transactionsNoRetry]]
+
+
+=== Simple Batching with No Retry
+
+Consider the following simple example of a nested batch with no
retries. This is a very common scenario for batch processing, where
an input source is processed until exhausted, but we commit
- periodically at the end of a "chunk" of processing.
+ periodically at the end of a "chunk" of processing.
+
+
+----
-
1 | REPEAT(until=exhausted) {
|
2 | TX {
@@ -23,26 +30,30 @@
| }
|
| }
-
+
+----
- The input operation (3.1) could be a message-based receive
+The input operation (3.1) could be a message-based receive
(e.g. JMS), or a file-based read, but to recover and continue
processing with a chance of completing the whole job, it must be
transactional. The same applies to the operation at (3.2) - it must
- be either transactional or idempotent.
+ be either transactional or idempotent.
- If the chunk at REPEAT(3) fails because of a database exception at
- (3.2), then TX(2) will roll back the whole chunk.
-
+If the chunk at REPEAT(3) fails because of a database exception at
+ (3.2), then TX(2) will roll back the whole chunk.
-
- Simple Stateless Retry
+[[transactionStatelessRetry]]
- It is also useful to use a retry for an operation which is not
+
+=== Simple Stateless Retry
+
+It is also useful to use a retry for an operation which is not
transactional, like a call to a web-service or other remote
- resource. For example:
+ resource. For example:
+
+
+----
-
0 | TX {
1 | input;
1.1 | output;
@@ -50,24 +61,28 @@
2.1 | remote access;
| }
| }
-
+
+----
- This is actually one of the most useful applications of a retry,
+This is actually one of the most useful applications of a retry,
since a remote call is much more likely to fail and be retryable
than a database update. As long as the remote access (2.1)
eventually succeeds, the transaction TX(0) will commit. If the
remote access (2.1) eventually fails, then the transaction TX(0) is
- guaranteed to roll back.
-
+ guaranteed to roll back.
-
- Typical Repeat-Retry Pattern
+[[repeatRetry]]
- The most typical batch processing pattern is to add a retry to the
+
+=== Typical Repeat-Retry Pattern
+
+The most typical batch processing pattern is to add a retry to the
inner block of the chunk in the Simple Batching example.
- Consider this:
+ Consider this:
+
+
+----
-
1 | REPEAT(until=exhausted, exception=not critical) {
|
2 | TX {
@@ -85,41 +100,39 @@
| }
|
| }
-
+
+----
- The inner RETRY(4) block is marked as "stateful" - see the
+The inner RETRY(4) block is marked as "stateful" - see the
typical use case for a description of a stateful
retry. This means that if the the retry PROCESS(5) block fails, the
- behaviour of the RETRY(4) is as follows.
-
-
-
- Throw an exception, rolling back the transaction TX(2) at the
+ behaviour of the RETRY(4) is as follows.
+
+
+* Throw an exception, rolling back the transaction TX(2) at the
chunk level, and allowing the item to be re-presented to the input
- queue.
-
-
- When the item re-appears, it might be retried depending on the
+ queue.
+
+
+* When the item re-appears, it might be retried depending on the
retry policy in place, executing PROCESS(5) again. The second and
- subsequent attempts might fail again and rethrow the exception.
-
-
- Eventually the item re-appears for the final time: the retry
+ subsequent attempts might fail again and rethrow the exception.
+
+
+* Eventually the item re-appears for the final time: the retry
policy disallows another attempt, so PROCESS(5) is never
executed. In this case we follow a RECOVER(6) path, effectively
- "skipping" the item that was received and is being processed.
-
-
+ "skipping" the item that was received and is being processed.
- Notice that the notation used for the RETRY(4) in the plan above
+Notice that the notation used for the RETRY(4) in the plan above
shows explictly that the the input step (4.1) is part of the retry.
It also makes clear that there are two alternate paths for
processing: the normal case is denoted by PROCESS(5), and the
recovery path is a separate block, RECOVER(6). The two alternate
paths are completely distinct: only one is ever taken in normal
- circumstances.
+ circumstances.
- In special cases (e.g. a special TranscationValidException
+In special cases (e.g. a special TranscationValidException
type), the retry policy might be able to determine that the
RECOVER(6) path can be taken on the last attempt after PROCESS(5)
has just failed, instead of waiting for the item to be re-presented.
@@ -127,17 +140,17 @@
knowledge of what has happened inside the PROCESS(5) block, which is
not usually available - e.g. if the output included write
access before the failure, then the exception should be rethrown to
- ensure transactional integrity.
+ ensure transactional integrity.
- The completion policy in the outer, REPEAT(1) is crucial to the
+The completion policy in the outer, REPEAT(1) is crucial to the
success of the above plan. If the output(5.1) fails it may throw an
exception (it usually does, as described), in which case the
transaction TX(2) fails and the exception could propagate up through
the outer batch REPEAT(1). We do not want the whole batch to stop
because the RETRY(4) might still be successful if we try again, so
- we add the exception=not critical to the outer REPEAT(1).
+ we add the exception=not critical to the outer REPEAT(1).
- Note, however, that if the TX(2) fails and we do try again, by
+Note, however, that if the TX(2) fails and we __do__ try again, by
virtue of the outer completion policy, the item that is next
processed in the inner REPEAT(3) is not guaranteed to be the one
that just failed. It might well be, but it depends on the
@@ -149,18 +162,21 @@
after 10 consecutive attempts, but not necessarily at the same item.
This is consistent with the overall retry strategy: it is the inner
RETRY(4) that is aware of the history of each item, and can decide
- whether or not to have another attempt at it.
-
+ whether or not to have another attempt at it.
-
- Asynchronous Chunk Processing
+[[asyncChunkProcessing]]
- The inner batches or chunks in the typical example
+
+=== Asynchronous Chunk Processing
+
+The inner batches or chunks in the typical example
above can be executed concurrently by configuring the outer batch to
- use an AsyncTaskExecutor . The outer batch waits for all the
- chunks to complete before completing.
+ use an AsyncTaskExecutor. The outer batch waits for all the
+ chunks to complete before completing.
+
+
+----
-
1 | REPEAT(until=exhausted, concurrent, exception=not critical) {
|
2 | TX {
@@ -178,19 +194,23 @@
| }
|
| }
-
-
+
+----
-
- Asynchronous Item Processing
+[[asyncItemProcessing]]
- The individual items in chunks in the typical
+
+=== Asynchronous Item Processing
+
+The individual items in chunks in the typical
can also in principle be processed concurrently. In this case the
transaction boundary has to move to the level of the individual
item, so that each transaction is on a single thread:
-
+
+
+
+----
-
1 | REPEAT(until=exhausted, exception=not critical) {
|
2 | REPEAT(size=5, concurrent) {
@@ -208,26 +228,30 @@
| }
|
| }
-
+
+----
- This plan sacrifices the optimisation benefit, that the simple plan
+This plan sacrifices the optimisation benefit, that the simple plan
had, of having all the transactional resources chunked together. It
is only useful if the cost of the processing (5) is much higher than
- the cost of transaction management (3).
-
+ the cost of transaction management (3).
-
- Interactions Between Batching and Transaction Propagation
+[[transactionPropagation]]
- There is a tighter coupling between batch-retry and TX management
+
+=== Interactions Between Batching and Transaction Propagation
+
+There is a tighter coupling between batch-retry and TX management
than we would ideally like. In particular a stateless retry cannot
be used to retry database operations with a transaction manager that
doesn't support NESTED propagation.
-
+
- For a simple example using retry without repeat, consider this:
+For a simple example using retry without repeat, consider this:
+
+
+----
-
1 | TX {
|
1.1 | input;
@@ -239,16 +263,19 @@
| }
|
| }
-
+
+----
- Again, and for the same reason, the inner transaction TX(3) can
+Again, and for the same reason, the inner transaction TX(3) can
cause the outer transaction TX(1) to fail, even if the RETRY(2) is
- eventually successful.
+ eventually successful.
- Unfortunately the same effect percolates from the retry block up to
- the surrounding repeat batch if there is one:
+Unfortunately the same effect percolates from the retry block up to
+ the surrounding repeat batch if there is one:
+
+
+----
-
1 | TX {
|
2 | REPEAT(size=5) {
@@ -262,45 +289,46 @@
| }
|
| }
-
-
- Now if TX(3) rolls back it can pollute the whole batch at TX(1) and
- force it to roll back at the end.
-
- What about non-default propagation?
-
-
- In the last example PROPAGATION_REQUIRES_NEW at TX(3) will
+----
+
+Now if TX(3) rolls back it can pollute the whole batch at TX(1) and
+ force it to roll back at the end.
+
+What about non-default propagation?
+
+
+* In the last example PROPAGATION_REQUIRES_NEW at TX(3) will
prevent the outer TX(1) from being polluted if both transactions
are eventually successful. But if TX(3) commits and TX(1) rolls
back, then TX(3) stays committed, so we violate the transaction
contract for TX(1). If TX(3) rolls back, TX(1) does not necessarily (but it probably
will in practice because the retry will throw a roll back
- exception).
-
-
- PROPAGATION_NESTED at TX(3) works as we require in the retry
+ exception).
+
+
+* PROPAGATION_NESTED at TX(3) works as we require in the retry
case (and for a batch with skips): TX(3) can commit, but
subsequently be rolled back by the outer transaction TX(1). If
TX(3) rolls back, again TX(1) will roll back in practice. This
option is only available on some platforms, e.g. not Hibernate or
- JTA, but it is the only one that works consistently.
-
-
+ JTA, but it is the only one that works consistently.
- So NESTED is best if the retry block contains any database access.
-
+So NESTED is best if the retry block contains any database access.
-
- Special Case: Transactions with Orthogonal Resources
+[[specialTransactionOrthonogonal]]
- Default propagation is always OK for simple cases where there are no
+
+=== Special Case: Transactions with Orthogonal Resources
+
+Default propagation is always OK for simple cases where there are no
nested database transactions. Consider this (where the SESSION and
TX are not global XA resources, so their resources are orthogonal):
-
+
+
+
+----
-
0 | SESSION {
1 | input;
2 | RETRY {
@@ -309,36 +337,40 @@
| }
| }
| }
-
+
+----
- Here there is a transactional message SESSION(0), but it doesn't
+Here there is a transactional message SESSION(0), but it doesn't
participate in other transactions with
- PlatformTransactionManager , so doesn't propagate when TX(3)
+ PlatformTransactionManager, so doesn't propagate when TX(3)
starts. There is no database access outside the RETRY(2) block. If
TX(3) fails and then eventually succeeds on a retry, SESSION(0) can
commit (it can do this independent of a TX block). This is similar
to the vanilla "best-efforts-one-phase-commit" scenario - the worst
that can happen is a duplicate message when the RETRY(2) succeeds
and the SESSION(0) cannot commit, e.g. because the message system is
- unavailable.
-
+ unavailable.
-
- Stateless Retry Cannot Recover
+[[statelessRetryCannotRecover]]
- The distinction between a stateless and a stateful retry in the
+
+=== Stateless Retry Cannot Recover
+
+The distinction between a stateless and a stateful retry in the
typical example above is important. It is actually
ultimately a transactional constraint that forces the distinction,
and this constraint also makes it obvious why the distinction
exists.
-
+
- We start with the observation that there is no way to skip an item
+We start with the observation that there is no way to skip an item
that failed and successfully commit the rest of the chunk unless we
wrap the item processing in a transaction. So we simplify the
- typical batch execution plan to look like this:
+ typical batch execution plan to look like this:
+
+
+----
-
0 | REPEAT(until=exhausted) {
|
1 | TX {
@@ -357,23 +389,23 @@
| }
|
| }
-
+
+----
- Here we have a stateless RETRY(3) with a RECOVER(5) path that kicks
+Here we have a stateless RETRY(3) with a RECOVER(5) path that kicks
in after the final attempt fails. The "stateless" label just means
that the block will be repeated without rethrowing any exception up
to some limit. This will only work if the transaction TX(4) has
- propagation NESTED.
+ propagation NESTED.
- If the TX(3) has default propagation properties and it rolls back,
+If the TX(3) has default propagation properties and it rolls back,
it will pollute the outer TX(1). The inner transaction is assumed by
the transaction manager to have corrupted the transactional
- resource, and so it cannot be used again.
+ resource, and so it cannot be used again.
- Support for NESTED propagation is sufficiently rare that we choose
+Support for NESTED propagation is sufficiently rare that we choose
not to support recovery with stateless retries in current versions of
Spring Batch. The same effect can always be achieved (at the
expense of repeating more processing) using the
- typical pattern above.
-
-
\ No newline at end of file
+ typical pattern above.
+
diff --git a/spring-batch-docs/asciidoc/whatsnew.adoc b/spring-batch-docs/asciidoc/whatsnew.adoc
new file mode 100644
index 000000000..324f79a73
--- /dev/null
+++ b/spring-batch-docs/asciidoc/whatsnew.adoc
@@ -0,0 +1,51 @@
+:batch-asciidoc: http://docs.spring.io/spring-batch/reference/html/
+:toc: left
+:toclevels: 4
+
+[[whatsNew]]
+
+== What's New in Spring Batch 4.0
+
+The Spring Batch 4.0 release has three major themes:
+
+
+* Java 8 Requirement
+
+
+* Dependencies re-baseline
+
+
+* Builders for ItemReaders and ItemWriters
+
+[[whatsNewJava]]
+
+
+=== Java 8 Requirement
+
+Spring Batch has historically followed Spring Framework's baselines for both
+ java version as well as third party dependencies. With Spring Batch 4, the Spring
+ Framework version is being upgraded to Spring Framework 5. As such, the java
+ version requirement for Spring Batch is also increasing to Java 8.
+
+
+[[whatsNewDependencies]]
+
+
+=== Dependencies re-baseline
+
+In order to continue to integrate with supported versions of the third party
+ libraries Spring Batch utilizes, Spring Batch 4 is updating the dependencies across
+ the board. The new dependency versions are in alignment with Spring Framework 5.
+
+
+[[whatsNewBuilders]]
+
+
+=== Provide builders for the ItemReaders and ItemWriters
+
+Spring Batch 4 is providing a collection of builders for all of the ItemReaders
+ and ItemWriters that come with the framework. As of this release, builders for the
+ `FlatFileItemReader`, `FlatFileItemWriter`, `JdbcCursorItemReader`, and
+ `JdbcBatchItemWriter` are available. More information can be found in the javadoc
+ for Spring Batch.
+
diff --git a/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/ldif/MappingLdifReader.java b/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/ldif/MappingLdifReader.java
index cde3301cb..e9213274b 100644
--- a/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/ldif/MappingLdifReader.java
+++ b/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/ldif/MappingLdifReader.java
@@ -36,11 +36,6 @@ import org.springframework.util.ClassUtils;
* object which can be consumed and manipulated as necessary by {@link org.springframework.batch.item.ItemProcessor ItemProcessor} or any
* output service.
*
- * {@link LdifReader LdifReader} usage is mimics that of the FlatFileItemReader for all intensive purposes. Adjustments have been made to
- * process records instead of lines, however. As such, the {@link #recordsToSkip recordsToSkip} attribute indicates the number of records
- * from the top of the file that should not be processed. Implementations of the {@link RecordCallbackHandler RecordCallbackHandler}
- * interface can be used to execute operations on those skipped records.
- *
* As with the {@link org.springframework.batch.item.file.FlatFileItemReader FlatFileItemReader}, the {@link #strict strict} option
* differentiates between whether or not to require the resource to exist before processing. In the case of a value set to false, a warning
* is logged instead of an exception being thrown.
diff --git a/src/site/apt/articles.apt b/src/site/apt/articles.apt
deleted file mode 100644
index aaf2584b5..000000000
--- a/src/site/apt/articles.apt
+++ /dev/null
@@ -1,22 +0,0 @@
- ------
- Spring Batch in the Media
- ------
- Dave Syer
- ------
- November 2007
-
-Spring Batch In the Media
-
- * http://www.theserverside.com/tt/articles/article.tss?l=SpringBatchOverview
-
- * http://www.theserverside.com/news/thread.tss?thread_id=47506#242493
-
- * http://blog.decaresystems.ie/index.php/2007/04/12/spring-batch/
-
- * http://www.itweek.co.uk/itweek/news/2189502/accenture-launches-batch
-
- * http://www.theserverside.com/tt/articles/article.tss?l=SpringBatchOverview
-
- * http://www.infoq.com/interviews/johnson-spring-portfolio
-
- * http://www.infoq.com/news/2008/06/spring-batch
diff --git a/src/site/apt/building.apt b/src/site/apt/building.apt
deleted file mode 100644
index 8d14e5cc9..000000000
--- a/src/site/apt/building.apt
+++ /dev/null
@@ -1,377 +0,0 @@
- ------
- Building Spring Batch
- ------
- Dave Syer
- ------
- April 2007
-
-Building Spring Batch
-
- Spring Batch is organised as a reactor build in Maven (m2). To
- build from the command line use
-
-+---
-$ mvn install
-+---
-
- or the goal of your choice (compile, test, etc.). This builds the
- artifact (e.g. jar file) from the project in the current directory,
- and deploys it to you local m2 repo at
- <<<${user.home}/.m2/repository>>>. If there are any dependency resolution
- problems try
-
-+---
-$ mvn install -P bootstrap
-+---
-
- which enables some additional, non-standard repositories (which
- should not be present in ther deployed artifacts). You should only
- need to do this once, because then all the dependencies will be
- installed in your local repository. To get the source code (where
- available) for all dependencies, you can use
-
-+---
-$ mvn dependency:sources -P bootstrap
-+---
-
- See below for instructions on how
- to build the documentation and web site.
-
- By default the whole project (including subprojects) will be built
- using Maven's "reactor" plugin. This can be expensive. To build
- only one module, cd to that directory first. Or at the top level
- use -N (for non-recursive) to exclude subprojects.
-
-+---
-$ mvn -N install
-+---
-
-* Skipping Tests
-
- The profile <> skips all the tests, so
-
-+---
-mvn -o install -P fast
-+---
-
- is the quickest way to update your local repo (assuming the tests
- are OK). It is equivalent of setting <<<-Dmaven.test.skip=true>>>.
-
-* Running Individual Tests
-
- The standard way to do this with Maven is -Dtest= with the class name (not fully qualified), e.g.
-
-+---
-$ mvn test -Dtest=FootballJobFunctionalTests
-+---
-
- In the samples you can also add additional system properties, which will be used to override bean properties. This can be done with an argLine property, e.g.
-
-+---
-$ mvn test -Dtest=FootballJobFunctionalTests -DargLine='-Dplayer.file.name=player.csv -Dgames.file.name=games.csv'
-+---
-
- or by specifying forkMode=never (in which case the test is run in the same process as Maven):
-
-+---
-$ mvn test -DforkMode=never -Dtest=FootballJobFunctionalTests -Dplayer.file.name=player.csv -Dgames.file.name=games.csv -Djob.commit.interval=50
-+---
-
-* Eclipse IDE
-
- Our policy is to commit Eclipse (and only Eclipse) meta data to
- source control. This will work out of the box for you if you use
- the (excellent) Q4E Eclipse-plugin
- (http://q4e.googlecode.com/svn/trunk/updatesite). With this plugin,
- each of the reactor modules at the top level builds independently
- and feeds changes into other projects in your workspace. It is not
- recommended to use the Maven Eclipse plugin (<<>>) because it cannot track dependencies across the
- Eclipse workspace. It will also create Eclipse meta-data every time
- you run it, conflicting with the version under source control.
-
-* Dependencies
-
- If you get multiple versions of the same jar across projects, or a
- jar is appearing in the classpath that you don't think is necessary,
- look into the dependency structure and try and exclude it from
- wherever it is being transitively included. To see the dependencies
- for a project look in the site for the dependency report.
- Alternatively (very useful for quickly locating a rogue jar) use
-
-+---
-$ mvn -P snapshots dependency:tree
-+---
-
- We use the "snapshots" profile here so that we get a snapshot of the
- dependency plugin (older versions did not have the tree goal, but
- newer versions are not stable enough to use in production).
-
-* Subversion and Line Endings
-
- Please use
-
-+---
-*.xml = svn:eol-style=LF
-*.sql = svn:eol-style=LF
-*.txt = svn:eol-style=LF
-*.java = svn:eol-style=LF
-*.apt = svn:eol-style=LF
-*.properties = svn:eol-style=LF
-+---
-
- in your <<<~/.subversion>>> (or <<\Application Data\Subversion/config>>>). If anyone forgets to do that then the property can be recursively set using Tortoise (type in the property key and value and use the recursive checkbox).
-
-* Documentation
-
- With the exception of reference docs, please put content in the
- project that it is most closely associated with. Here is a
- {{{./sitemap.html}site map}} to help you decide.
-
-** Quotidian Web Content
-
- Maven allows you to choose from a range of source format for
- building web content. For Spring Batch we prefer the "almost plain
- text" version. See files under <<>> in all the projects
- for examples, and also refer to the
- {{{http://maven.apache.org/guides/mini/guide-apt-format.html}Apt
- Format Guide}} on the Maven website.
-
- N.B. you put .apt source files in a subdirectory called <<>>,
- but they are moved to the top level when the site is built. Thus
- <<>> becomes <<>>.
-
-*** Using emacs to edit .apt files
-
- Because the .apt format relies on indentation in plain text files,
- the emacs auto-fill feature in text mode makes editing very
- convenient. Put this in your .emacs
-
-+---
-(setq auto-mode-alist (cons '("\\.apt\\'" . text-mode) auto-mode-alist))
-+---
-
- Then use <<>> to auto-fill the current paragraph. Emacs
- adjusts the indentation of all the lines to match the first one (or
- the first two if the second is different.
-
- If anyone knows how to do this with Eclipse or other editors, let us
- know and we'll put a note here.
-
-** Reference Guide
-
- The <<>> project is reserved for reference guides in the
- normal Spring docbook format. Each chapter of the reference guide
- is in a separate xml file under <<>>.
- The easiest way to work with the reference guide is to cd to the
- <<>> module, and run Maven from there.
-
- Use the DTD with a validating XML editor (e.g. Eclipse) to explore
- the docbook format. Also look at existing examples in Spring Batch
- and in the Core Spring Framework source code.
-
- [Section numbers] There is no need to explicitly create section numbers in the
- XML - this is done for you by the build when everything is stitched
- together into a book.
-
- [Source location] You put docbook .xml source files in a
- subdirectory called <<>>, but they are moved to the top
- level when the site is built. Thus
- <<>> becomes
- <<>>.
-
- [XMLMind] If you use {{{http://www.xmlmind.com}XMLMind}} to edit the
- reference guide add the following line to
- <<</addon/config/docbook/common.incl>>>:
-
-+---
-
- Chapter Title
-
-
-+---
-
- Create a file with the template above, and put it in
- <<>>. Use lower case, dash separated file names
- (XML style), e.g. <<>>.
-
- Add the chapter to the master book in <<>> using
-
-+---
->>, and
- then refer to the file using the << element:
-
-+---
-
-
-
-
-
-
-
- Figure 1: the figure caption...
-
-
-
-+---
-
-* Program Listings in Docbook (Including XML)
-
- Use CDATA to save you from having to use the HTML escapes for all
- the special characters. E.g.
-
-+---
-
-]]>
-
-+---
-
-* Dynamic Editing
-
- To see your changes to web site content as soon as you have typed
- it, use
-
-+---
-mvn site:run
-+---
-
- and go to http://localhost:8080.
-
- In a project with unit tests, you can skip the tests and go straight
- to the documentation using
-
-+---
-mvn -o site:run -P fast
-+---
-
- If you are offline, or want to speed things up a bit, the "-o" stops
- Maven from trying to resolve dependencies on the internet.
-
- Use -N to build only the current project, not subprojects, So this
- is pretty useful at the top level:
-
-+---
-mvn -N -o site:run -P fast
-+---
-
- In the <<>> project the docbook reference guide shows up at
- http://localhost:8080/reference/*.html, where * is the name of an
- xml file with a chapter in it. There is no link to these pages on
- the site because the real docbook generated output is much nicer,
- but this is still pretty useful for debugging and dynamic
- editing.
-
- Note that the formatting is a bit limited compared to the whole
- docbook stylesheet - Maven uses Doxia to squish all of docbook into
- some simple wiki-like formatting rules. In particular it can't
- generate the index page in the format we need it, so you may see
- errors from <<>> if you visit that page. One of the
- features is that the <<<\>>> syntax we use to build the
- index and table of contents in the docbook-generated pages does not
- work. Images are another problem. Use the generated content from
- <<>> to view these artifacts.
-
-* Building and deploying the web site
-
- There is a bug in the m2 reactor (MNG-740) which means that we have
- to install the parent pom to the local repo first.
-
- So do it this way:
-
-+---
-$ mvn install -P fast
-$ mvn -P staging clean site site:deploy
-+---
-
- Remove "-P staging" to deploy to the real website (requires ssh
- access to static.springframework.org).
-
- The "-P staging" is to deploy to <<>>, so we
- don't get accidental updates to the site. To test the site contents
- navigate with your browser to that directory. The site:stage goal
- deos not work properly for this build: all the subprojects are not
- integrated into the staging site, so use site:deploy instead.
-
- The static website content is not deleted during the deployment
- process - merely replaced. If you need to clean everything up from
- scratch you need to delete the contents on the server as well
- (using ssh).
-
-Problems?
-
- Make sure your source code is up to date. Delete everything from
- your local Spring Batch repo
- <<<${user.home}/.m2/repository/org/springframework/batch>>>. If
- necessary, delete a project or directory and update from SVN again.
-
- Try
-
-+---
-$ mvn install
-+---
-
- or
-
-+---
-$ mvn clean install
-+---
-
- or
-
-+---
-$ mvn clean install -P fast
-+---
-
- from the top level, and
-
-+---
-$ mvn -U ...
-+---
-
- from wherever you are (top level or sub-project). The latter will
- update any older plugins you have in your local Maven repository.
- Some people have had trouble building the web site without this.
-
- If you get <<>> e.g. building the site, use
- MAVEN_OPTS to boost the heap size (on the command line if you have a
- sensible shell):
-
-+---
-$ MAVEN_OPTS=-Xmx256m mvn site
-+---
diff --git a/src/site/apt/cases/async.apt b/src/site/apt/cases/async.apt
deleted file mode 100644
index 2255d5ba3..000000000
--- a/src/site/apt/cases/async.apt
+++ /dev/null
@@ -1,159 +0,0 @@
- ------
- Asynchronous Chunk Processing Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Asynchronous Chunk Processing
-
-* Goal
-
- Increased the efficiency of chunk processing by having it execute
- asynchronously: in multiple threads. Maintain transactional
- intergrity of the chunk.
-
-* Scope
-
- * All chunks might conceivably benefit from parallel processing, so
- we don't want any unnecessary restrictions on the batch operation,
- or its implementation. A should be possible for Client to write a
- batch operation without reference to the fact that it might run in
- an asynchronous chunk.
-
-* Preconditions
-
- * Input data exists with non-trivial size: chunks contain more than
- one record.
-
- * Batch processing of a record is slow, or can be delayed, so that
- the asynchronous processing can take longer than launching the
- threads.
-
- * A chunk can be made to fail after at least one record is
- processed.
-
-* Success
-
- * A chunk is processed and the results inspected to verify that all
- records were processed.
-
- * Transactional behaviour is verified by rolling back a chunk and
- verifying that no records were processed.
-
-* Description
-
- The vanilla case proceeds as for normal {{{./chunks.html}chunk
- processing}}, but:
-
- [[1]] Within a chunk, Container processes records in parallel.
-
- [[1]] At the end of a chunk, Container waits for the last record
- to be processed (with a timeout if the wait is long).
-
-* Variations
-
-** Rollback on Failure
-
- If there is an exception in one of the record processing threads,
- the whole chunk should roll back:
-
- [[1]] Client throws exception in record processing.
-
- [[1]] Container catahes exception and attempts to abort other
- running processes.
-
- [[1]] Container waits for running processes to abort (or finish
- normally, but preferably to abort).
-
- [[1]] Container propagates the exception and signals transaction to
- rollback.
-
-** Timeout
-
- If there is a timeout during a chunk, it might happen before the
- chunk has finished, or while waiting for the processes to complete
- before exiting.
-
- [[1]] At end of chunk, Container is waiting for all processes to
- finish. It times out, according to a parameter set by the
- Operator.
-
- [[1]] Container does not start any new processes, and attempts to
- abort running processes.
-
- [[1]] Container waits for running processes to abort (or finish
- normally, but preferably to abort).
-
- [[1]] Container throws a time out exception and signals chunk
- transaction to rollback.
-
-* Implementation
-
- * The implementation of this use case could be tricky in the general
- case. In particular, the transactional nature is going to be hard
- or impossible to maintain across multiple threads without the
- individual processes being aware of the transaction, and (perhaps)
- without global (XA) transaction support.
-
- A "normal" local transaction is thread bound - i.e. it only executes
- in one thread. If the code inside the transaction creates new
- threads, then they might not finish processing before the parent
- exits and the transaction wants to finish. The transaction needs to
- wait for the sub-processes before committing, or (more difficult)
- rolling back. The rollback case basically forces us to a model of
- one transaction per thread, and therefore to one transaction per
- data item in a concurrent environment.
-
- Otherwise some transactional semantics might be respected in a
- parallel process, but others certainly will not be because
- synchronizations and resources are managed at the level of the
- thread where the transaction started. If the transaction manager is
- a local one (not XA) there is little hope even that the datasource
- resource would be the same for all the parallel threads and the
- parent method.
-
- If we use a global transaction manager to make the parallel
- processes transactional, how will they know which transaction to
- participate in? There could be many active chunks, and each would
- have its own threads - how would each one be able to guide its child
- processes to participate in the same transaction?
-
- * Beware a framework that extracts data from an <<>>
- before executing the business logic (e.g. in a
- <<>>). It is not enough to allow concurrent
- processing but simply insist that the individual records are
- processed transactionally because the <<>> will then
- not be able to participate in the transaction - its next record has
- already been passed to the consumer when the transaction starts, so
- if there is a rollback then the record is lost.
-
- This is the origin of the signature:
-
-+---
-public interface ItemReader {
- Object next();
-}
-+---
-
- There is no peeking and no iterator-style <<>>. If there
- is a processing problem, transactional clients of the
- <<>> throw an exception the provider's
- <<>> has been called, but in the same thread (so that
- transactional semantics are preserved and the data provider reverts
- to its previous state).
-
- This means that in the callback interface also picks up an
- <<>> return type
-
-+---
-public interface RepeatCallback {
- Object doInIteration(BatchContext context);
-}
-+---
-
- so we can return an object, which is null when the processing has
- finished.
-
- In the end we decided against the <<>> return type and went
- with an exit status to signal for no more processing.
diff --git a/src/site/apt/cases/chunks.apt b/src/site/apt/cases/chunks.apt
deleted file mode 100644
index 0f3e129d7..000000000
--- a/src/site/apt/cases/chunks.apt
+++ /dev/null
@@ -1,208 +0,0 @@
- ------
- Commit Periodically Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Commit Batch Process Periodically
-
-* Goal
-
- Read a file line-by-line and process into database inserts, for
- example using the Jdbc API. Commit periodically, and if there is a
- fault where the database transaction rolls back, then the file
- reader is reset to the place it was after the last successful
- commit.
-
- To develop a batch process to achieve the goal above should be as
- simple a process as possible. The more that can be done with simple
- POJOs and Spring configuration the better.
-
-* Scope
-
- To keep things simple for now, assume that:
-
- * All lines in the input file are in the same format and each line
- generates a single database insert (or a fixed number).
-
- * The file is read synchronously by a single consumer.
-
-* Preconditions
-
- * A file exists in the right format, with a sufficiently large
- number of lines to be realistic.
-
- * A mechanism exists to force a rollback at a non-trivial position
- (not during the first commit), but produce a successful operation
- on the second try.
-
- * A framework for retry exists, so that the case above can be
- tested.
-
-* Success
-
- Integration test confirms that
-
- * All data are processed and records inserted successfully.
-
- * When a rollback occurs and the retry is successful, the complete
- dataset is processed (same result as successful run).
-
- * Batch operations can be implemented without framework code (or
- with minimal dependencies, e.g. through interfaces). Launching
- the batch might require access to framework code.
-
-* Description
-
- The vanilla successful batch use case proceeds as follows:
-
- [[1]] Container starts a transaction.
-
- [[1]] Container makes resources available, e.g. opens file and
- creates <<>> for it.
-
- [[1]] Client reads a line from the file, and converts it to a
- database statement, then runs it.
-
- [[1]] Container increments counter.
-
- [[1]] Repeat previous two steps until a counter is equal to chunk
- size.
-
- [[1]] Container commits database transaction.
-
- [[1]] Repeat chunk processing until input source is exhausted.
-
-* Variations
-
-** Non-fatal Chunk Failure
-
- If there is an unrecoverable database exception during execution of
- client code:
-
- [[1]] Container rolls back current transaction.
-
- [[1]] Container resets input source to the point it was at before
- failure.
-
- [[1]] Container retries chunk.
-
-** Fatal Chunk Failure
-
- If there is an error in the input data in the middle of a chunk
- (could be manifested as database exception, e.g. uniqueness
- exception, or nullable exception):
-
- [[1]] Container rolls back current transaction.
-
- [[1]] Container terminates batch and notifies client of precise
- details, including the line number of error, and the last line
- that was committed (last of the previous chunk).
-
- There is no need to reset the input source because the error is
- fatal.
-
- To restart:
-
- [[1]] Operator truncates the input file so the completed chunks
- are not repeated.
-
- [[1]] Operator fixes bad line (if there was one), and starts the
- batch process wit hthe same parameters.
-
- Variations on this theme are also necessary, e.g. a tolerance for a
- small number of bad records in the input data.
-
-* Implementation
-
- * The concept of a batch iterator seems relevant here (see also the
- {{{./simple.html}simple}} use case). The iterator could be more than
- just a loop that might terminate early: here it could also manage
- the file cursor on the input source. In this design there is a
- <<>> interface that can take care of termination and
- iteration (e.g. iterator-like method signatures).
-
- * Another design idea (more encapsulated and more in keeping with
- existing Spring practice) is to make the data source transaction
- aware, and for the client use it like a database resource, through a
- template. In this case there is a <<>>. The
- <<>> needs to be aware of the data source template, so
- that it can terminate when the data is exhausted.
-
- In this version of events there are two kinds of resource in play.
- The transaction itself, and the data sources that are aware of the
- transaction. The comparison with <<>>
- and <<>> is obvious. The client is often completely
- unaware of the transaction manager, which is applied through an
- interceptor, whereas the data source is used explicitly with its own
- API through a template. The Client can concentrate on his domain,
- and not be concerned with infrastructure or resource handling.
-
- * The analogy with <<>> is even stronger. If the input
- data came from JMS instead of a file, we would hardly have to do
- anything to implement very robust chunking. JMS is the obvious best
- practice and already provides all the transactional semantics we
- need for chunking - simply roll back a transaction and the records
- processed return to the message system for delivery to the next
- consumer. Bad records can be sent to a bad message queue for
- independent processing. JMS might ssem like overkill for a lot of
- batch processes, but it is tempting to say that if the robustness is
- needed then the we should take that as a sign that installing and
- configuring JMS is worth the extra effort.
-
- * Naturally we do not want to insist that the client code is aware
- of the transaction that is surrounding it - this would be the normal
- practice familiar from the Spring programming model. Should a
- client need access to transaction-scoped resources, the usual way to
- do that is to wrap the transactional resource (data source etc.) in
- a proxy that uses a synchronization, or a more generic thread-bound
- resource (using <<>>). The aim
- is to retain this separation in a batch operation. The batch
- framework itself might provide some of these synchronizations.
-
- * The {{{./simple.html}Simple Batch Repeat}} is actually a pretty good
- model for the chunk processing in this use case. This observation
- leads to another: that a batch of chunks is a nested (or composed)
- batch - the outer termination policy is dependent only on the data
- source having further records to process, the inner one is a simple
- iterator (with a check for empty data). A simplified programming
- model for this is
-
-+---
-RepeatCallback chunkCallback = new RepeatCallback() {
-
- public boolean doInIteration(RepeatContext context) {
-
- int count = 0;
-
- do {
-
- Object result = callback.doWithRepeat(context);
-
- } while (result!=null && count++>>). The termination policy depends
- only on a data source eventually returning null.
-
- * N.B. the chunkSize can be dynamic. E.g., if the chunk is long
- during a nightime batch window, and short when the window is over,
- in case the batch has to be terminated.
-
- * Chunking can also be implemented simply in an
- <<>>. The handler just buffers records up to a
- chunk size, and then executes them all in one step (which might be
- transactional). This is easier to implement, and easier to
- configure for the clients, but cannot easily be made both concurrent
- and transactional.
diff --git a/src/site/apt/cases/file-to-file.apt b/src/site/apt/cases/file-to-file.apt
deleted file mode 100644
index 53a6bc45d..000000000
--- a/src/site/apt/cases/file-to-file.apt
+++ /dev/null
@@ -1,89 +0,0 @@
- ------
- Copy File to File
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Copy File to File
-
-* Goal
-
- Read a file line-by-line and process into a file in a different
- format (possibly different number of lines). Commit periodically
- and in the event of an error both data sources (input and output)
- rollback to the last known good point.
-
-* Scope
-
- To keep things simple for now, assume that:
-
- * All lines in the file are in the same format and the final
- output is an aggregate.
-
- * The files are read and written synchronously by a single
- consumer.
-
- * This use case requires two kinds of transactional file source.
- One is read-only and the other is write-only. Only one consumer
- can use the write-only source at a time.
-
-* Preconditions
-
- * An input file exists in the right format, with a sufficiently
- large number of lines to be realistic.
-
-* Success
-
- Integration test confirms that
-
- * All data are processed and output produced successfully.
-
-* Description
-
- Very similar to the use case {{{./chunks.html}Copy File to
- Database}}, but involving transactional access to an output source
- which is a file. Also we are introducing the idea of an aggregate
- function for the output.
-
- The vanilla successful case proceeds as in the file to database
- version, except that:
-
- [[1]] A successful chunk results in a line in an intermediate file
- output source.
-
- [[1]] After all chunks are successfully processed the intermediate
- file is itself processed in a single transaction to complete the
- aggregate. The output is itself sent to an output channel
- (e.g. database or file).
-
-* Variations
-
- * Chunk failure variations proceed as in the use case
- {{{./chunks.html}Copy File to Database}}. In the case of a
- restart after fatal failure, the intermediate output file need does
- not need to be reset or re-created.
-
-* Implementation
-
- * The write-only file source is new in this use case. It has a
- similar flavour to the read-only version, but also has more serious
- implications for implementation and usage. Since a file system is
- not inherently transactional, when we create the write-only data
- source we are assuming that consumers will play by the rules,
- principally that there is only one consumer at a time.
-
- * With some external limitations the write-only file source can be
- implemented so that within a single JVM it will behave like a
- transactional database datasource. We can provide a
- <<>> that hides the resource acquisition and
- release, and interacts with an existing transaction to provide the
- transactional behaviour that is required.
-
- * File-based transactional resources are a lot like messaging
- clients. We can send a message (write a line) through a sender
- client, and receive a message (read a line) through a consumer
- client. In the case of a transaction rollback, all sent messages
- are guaranteed not to reach consumers, and all received messages are
- returned to the queue. Maybe ActiveMQ has a file transport already?
- Mule definitely does, but it isn't transactional.
diff --git a/src/site/apt/cases/index.apt b/src/site/apt/cases/index.apt
deleted file mode 100644
index c544aa4cb..000000000
--- a/src/site/apt/cases/index.apt
+++ /dev/null
@@ -1,113 +0,0 @@
- ------
- Use Cases
- ------
- Dave Syer
- ------
- January 2007
-
-Use Cases for Spring Batch
-
- These are more like scenarios or flows than real use cases in formal
- UML terms, but they serve a useful purpose as both. We don't want
- to be over formal, and probably code is being written and tested at
- the same time as these use cases. But there are many stakeholders
- in this project, and use cases are a useful resource to make sure
- they are all agreed on scope and certain implementation details.
-
- * {{{./simple.html}Simple Batch Repeat}}
-
- * {{{./retry.html}Automatic Retry After Failure}}
-
- * {{{./chunks.html}Commit Batch Process Periodically}}: chunk
- processing.
-
- * {{{./async.html}Asynchronous Chunk Processing}}: parallel
- processing within a chunk.
-
- * {{{./file-to-file.html}Copy File to File in a Batch}}
-
- * {{{./parallel.html}Massively Parallel Batch Processing}}. Spring
- Batch 1.0 does not contain any implementations of this use case,
- but it is quite feasible to implement them using the framework as
- a starting point. 1.1 has some prototype code under the Integration
- module.
-
- * {{{./restart.html}Manual Restart After Failure}}
-
- * {{{./steps.html}Sequential Processing of Dependent Steps}}
-
- * {{{./partial.html}Partial Processing}}: skip records (e.g. on rollback).
-
- * Whole-Batch Transaction - transactional support for the whole
- batch, not just chunks. Quite a common requirement, but not
- always practical using normal transaction support. May require a
- staging area, and a decision after it is full about whether to
- copy it in one big batch (e.g. using native database tools) or
- chunk it (e.g. if it is now in a form for which chunk failure is
- easier to deal with).
-
- * {{{./scheduled.html}Scheduled Processing}}: Batch Jobs controlled
- by scheduler (e.g. start, stop, suspend, kill). Spring Batch does
- not intend to implement the scheduler concerns, but needs
- to provide enough information that a scheduler can act
- appropriately.
-
- * Non-Sequential Processing of Steps (Conditional Branching)
-
- * {{{./pause.html}Pause and Resume Job Execution}}
-
-
-* Actors
-
- The following actors are involved in the use cases (Container and
- Client being the most common / important).
-
-** Client or Business Domain
-
- Code written by the batch developer.
-
- One aim us that the client is a POJO - the batch behaviour, boundary
- conditions, transactions etc. can be dealt with by the Container in
- such as way that the client does not need to know about them. The
- client may have access to framework abstractions, like templated
- data sources (<<>> etc.), but these should work the
- same whether they are in a batch or not.
-
-** Container
-
- An application that converts user requests for batch jobs into
- running processes. Container concerns are robustness, traceability,
- manageability.
-
-** Framework
-
- The Framework is the infrastructure code that the Container depends
- on, and possibly spi implementations where knowledge of the
- non-business logic resides.
-
- The Framework provides two kinds of infrastruture (as per usual
- Spring cornerstones and ):
-
- * For cross-cutting concerns there are interceptors that can be
- wrapped around client code without it needing any knowledge of the
- Framework at all. An existing parallel is with transaction
- support - the client code can use <<>>
- directly, but does not always need to.
-
- * Concrete abstractions that allow access to resources in a
- uniform way without needing to know the details of how they are
- provided (e.g. partitioned). Client code can use these
- abstractions like it would a use a <<>>.
-
-** Operator
-
- The batch operator is not a developer. Tools are provided for the
- Operator to be able to stop and start a batch, and to monitor the
- progress and status of on ongoing or finished batch.
-
-** Business User
-
- The Operator has technical skills, e.g. a member of an application
- support team, but may need help with business-related decisions.
- For instance if input data are bad, he would not expect to be able
- to fix them alone because they might be bad for a business reason.
diff --git a/src/site/apt/cases/parallel.apt b/src/site/apt/cases/parallel.apt
deleted file mode 100644
index c61cacf2a..000000000
--- a/src/site/apt/cases/parallel.apt
+++ /dev/null
@@ -1,278 +0,0 @@
- ------
- Parallel Processing Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Massively Parallel Batch Processing
-
-* Goal
-
- Support efficient processing of really large batch jobs (100K -
- 1000K records) through parallel processing, across multiple
- processes or physical or virtual machines. The goals of other use
- cases should not be compromised, e.g. we need to be able to start
- and stop a batch job easily (for non developer), and trace the
- progress and failure points of a batch. The client code should not
- be aware of whether the processing is parallel or serial.
-
-* Scope
-
- * Any batch operation that reads data item-by-item from an input
- source is capable of being scaled up by parallelizing.
-
- * The initial implementation might concentrate on multiple threads
- in a single process. Ultimately we need to be able to support
- multiple processes each one running in an application server
- (e.g. so that jobs that require EJBs can be used).
-
-* Preconditions
-
- * A data source with multiple chunks (commitable units) - more chunks
- than parallel processes.
-
- * A way for the framework to launch parallel processes.
-
-* Success
-
- * A batch completes successfully, and the results are verified.
-
- * A batch fails in one of the nodes, and when restarted processes
- the remaining records.
-
-* Description
-
- [[1]] Framework splits input data into partitions.
-
- [[1]] Framework sends input data (or references to them) to
- processing nodes.
-
- [[1]] Processing nodes act independently, converting the input data
- and sending it transactionally to output source (as per normal
- single process batch).
-
- [[1]] Framework collects status data from individual nodes for
- reporting and auditing.
-
- [[1]] When all nodes are complete Framework decides that batch is
- complete finishes processing.
-
-* Variations
-
- Two failure cases can be distinguished, bad input data on a node and
- an internal node failure have different implications for how to
- proceed. In both cases, however
-
- [[1]] Framework catches exception and classifies it. Rolls back
- current transaction to preserve state of data (input and output).
-
- [[1]] Framework saves state for restart from last known good
- point, including a pointer to the next input record.
-
- Then if a processing node detects bad data in the input source, it
- cannot be restarted or re-distributed because the data need to be
- modified for a successful outcome.
-
- [[1]] Framework alerts Operator of the location and nature of the
- failure.
-
- [[1]] Operator waits for batch to finish - the overall status will
- be a failure, but most of the data might be consumed.
-
- [[1]] Operator fixes problem and restarts batch.
-
- [[1]] Framework does not re-process data that has already been
- processed successfully. The parallel processing nodes are used as
- before.
-
- [[1]] Batch completes normally.
-
- If a processing node fails unrecoverably (e.g. after retry timeout),
- but with no indication that the input data were bad, then the data
- can be re-used: Framework returns unprocessed input data, and
- redistributes it to other nodes.
-
-* Implementation
-
- * There are actually two approaches to this problem, which are
- largely complementary.
-
- [[1]] The model dynamically assigned chunks of items to
- be processed and sends them to durable middleware. Worker
- processes pick them up and process them, sending back a message
- about the status. This approach works best if the dispatching is
- efficient compared to the processing.
-
- [[1]] The approach is more like running multiple
- jobs in parallel, with input data partitioned into larger pieces,
- and not split any further by the dispatcher. The item reading
- happens in the worker processes. This approach is necessary if
- the dispatcher in the model becomes a bottle neck.
-
- Generally, chunking is easier to implement than partitioning, but
- there are tools available for implementing both patterns
- efficiently.
-
-** Chunking
-
- The messages from a dispatcher to worker processes consist of a
- chunk of items - a set of items to be processed together in a single
- transaction (or as the worker sees fit). The dispatcher is usually
- single threaded, but this is only a restriction based on the input
- data type (if it is a file it is difficult to read in parallel and
- maintain restartability). Using a process indicator the dispatcher
- could be reading from a database table in a multi-threaded model.
-
- The main restriction is that for restartability the messages between
- the dispatcher and workers has to be durable (i.e. JMS or
- equivalent). If there is a durable middleware there are no in
- principle difficulties with this approach.
-
- The practicalities deserve some discussion. In particular the
- dispatcher has to co-ordinate asynchronous replies from its workers,
- and also has to avoid overwhelming the workers (so there should be
- some throttling). As long as the middleware is durable the
- dispatcher can simply wait for replies whenever it thinks there are
- workers working. It needs to record this expectation in a durable
- form as well, as part of an <<>> for the step.
-
-** Partitioning
-
- The hard thing about this use case is the partitioning of input (and
- output) sources. Ideally, this has to be done in such a way that
- the individual operations are unaware that they are participating in
- a batch farm. Partitioning has to be at least partially
- deterministic because restarts have to be able to ignore data that
- have already been processed successfully.
-
- Consider two examples: a file input source and a JDBC (SQL query)
- based input source. Each provides its own challenges.
-
-*** File Data Source
-
- * If each node reads the whole file there could be a performance
- issue. They would all need to have instructions about which lines
- to process.
-
- * If each record of input data is a line, this isn't so bad. Each
- node can have a range of line numbers to process. The only problem
- is knowing how many lines there are, and how many nodes, so that the
- job can be partitionaed efficiently.
-
- * But if each input record can span a variable number of lines (not
- that unlikely in practice), then we can't use line numbers
-
- * Maybe the best solution is to use middleware anyway. A single
- process parses the file and sends it to a message queue, item by
- item (or chunk by chunk). The integration pattern could then be a
- simple Eager Consumer, assuming that all records are processed
- independently. The messaging semantics would simply have to ensure
- that a consumer can roll back and return the input records to a
- queue for another consumer to retry.
-
- * For large batches a real messaging infrastructure (JMS etc.) with
- guaranteed delivery would be a benefit, but might be seen as
- overkill for a system that didn't otherwise require it. In this
- case we could imagine the partitioning process being one of simply
- dividing the input file up into smaller files, which are then
- processed by individual nodes independently. The integration
- pattern is then different - more like a Router.
-
- * What would parallel processing look like to the client? We can
- make it completely transparent if we assume that the client only
- ever implements <<>> and <<>>. The
- client code is unaware of the partitioning of its data source.
-
- * Parallelisation could also take place at the level of the
- <<>> - we could proxy the data provider and wrap it in
- a partitioning proxy:
-
-+---
-
-
-
-
- ...
-
-
-
-
-
- ...
-
-
-+---
-
-*** SQL Data Source Partitioning
-
- * If each node is allowed to do its own query or queries to
- determine the input data:
-
- * Each node has to be given a way to narrow the query so that they
- don't all use the same data. There is no easy universal way to
- achieve this, and in the general case we have to know in advance
- when we are going to execute in a parallel or as a single process.
- Maybe a range of primary keys would work as a special case that we
- could support as a strategy.
-
- * Maybe we could assume that all nodes execute precisely the same
- query, and then provide a way to add a cursor to the result set,
- so it can be treated a bit more like a file.
-
- * We might be forced to use a distributed transaction to ensure
- that all the nodes see the same data. This would be unfortunate,
- but possibly necessary. It would be up to the client to configure
- distributed transactions if that was required, otherwise the
- result might be unpredictable if data can be added to an input
- source while it is being read.
-
- * If only one query is done by the Framework and the results shared
- out amongst the nodes we face the issue of how to send the data
- between nodes. Performance problems might ensue. Plus (more
- seriously) the individual nodes would now need a different
- implementation if they were acting in a parallel cluster to the
- vanilla serial processing case - a single node would do the query
- and work directly with the results, whereas in a parallel
- environment it would be one step removed from the actual query.
- This breaks our encapsulation design goal.
-
- * When considering the approach to partitioning the data source
- we should follow closely the discussion above on partitioning a file
- input source. If the client is to remain unaware of the batch
- parameters, then an interceptor looks like the best approach.
-
- If each node prefers to do its own query then an interceptor would
- have to catch the call to a JDBC template and modify the query
- dynamically. This is quite a scary thing to be doing - it might end
- up with us needing to parse the SQL and add where clauses. Maybe a
- client should be forced to specify (in the case of a parallel batch)
- how his query should be partitioned. For example:
-
-+---
-
-
-
- SELECT * from T_INPUT
-
-
-
- SELECT * from T_INPUT where ID>=? and ID
-
-
-
-+---
-
- It would be an error to run a batch in parallel if the partition
- query had not been provided.
-
- * What happens if the data source changes between failed execution
- and restart? We can't legislate for that because it is outside the
- realm of what can be controlled through a transaction. A restart
- might produce different results than the original failed batch would
- have done were it successful.
diff --git a/src/site/apt/cases/partial.apt b/src/site/apt/cases/partial.apt
deleted file mode 100644
index 015ff5a19..000000000
--- a/src/site/apt/cases/partial.apt
+++ /dev/null
@@ -1,154 +0,0 @@
- ------
- Partial Processing Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Partial Processing
-
-* Goal
-
- Support partial processing of a batch, without having to interrupt
- or manually restart, but enabling corrective action to be taken
- after the process has finished to complete the processing of failed
- records. A batch that is going to fail completely can be be
- identified as soon as possible, but one which is substantially
- alright can run as far as possible to prevent costly duplication.
- Records that are skipped are reported in such a way that they can be
- easily identified by the Operator and / or Business User and a new
- batch created to finish the original goal. By the same token, in
- the case of an aborted batch where a minority of records are
- processed successfully first time, it should be possible to identify
- the successful records and exclude them from data presented on
- restart.
-
-* Scope
-
- Any batch should be configurable to support partial processing.
-
-* Preconditions
-
- * A data source with a small number of bad records exists.
-
-* Success
-
- * A test data set with a small number of bad records is run through
- the batch processer and completes normally. Operator confirms
- that the good recirds are all processed and then fixes and
- resubmits the bad records, and confirms that they are also
- correctly processed with no duplicates.
-
-* Description
-
- The vanilla flow proceeds as follows:
-
- [[1]] Batch processing begins as per normal (see for example
- {{{./chunks.apt}chunk processing use case}}).
-
- [[1]] A record is processed. This step repeats until...
-
- [[1]] Container detects a bad record, e.g. by catching a
- classified execption.
-
- [[1]] Container logs the exception in a way that identifies the
- bad record easily and immediately to the Operator.
-
- [[1]] Container stores an identifier for the bad record (or the
- whole record) in a location designated to the Operator for that
- purpose.
-
- [[1]] Container determines that the batch can still succeed
- despite the cumulative number or nature of bad records - the bad
- record is skipped. Container goes back to normal processing, and
- eventually completes the whole batch.
-
-* Variations
-
-** Abort Batch Early
-
- The batch cannot skip all records. After each failure the decision
- about whether to coninue has to be made:
-
- [[1]] When a record is processed successfully, Container logs the
- event in a form that can be used later to identify successful
- records in case the batch is aborted.
-
- [[1]] Container determines that a sufficiently large fraction of
- the records processed so far have failed. The faction relevant is
- to be specified through configuration meta data (not specified by
- business logic).
-
- [[1]] Container aborts the batch with a clear signal to the
- Operator that it has aborted owing to an unacceptable number of
- errors.
-
-* Implementation
-
- * When the decision to abort is taken, Container may have
- successfully processed a small number of records and the
- corresponding transactions might have committed. Those records that
- were successfully processed on the first attempt are easy to
- exclude from the restart, if transactional semantics are respected
- by the item processing.
-
- * The decision to abort is based on exception classification. Each
- time an item is processed, the framework needs to catch exceptions
- and classify them as
-
- * fatal: signals an abort - rethrow.
-
- * transient: nominally fatal, but the operation is retryable.
-
- * non-fatal: signals a skip.
-
- The transient failure is really just a sub-type of fatal case. It
- is treated differently by the {{{./retry.html}retry framework}} but
- not necessarily by the vanilla batch.
-
- * Actually we can't decide what action to take simply on the
- evidence of the current exception. What we need to do is decide,
- potentially based on the whole history of exceptions in a given
- batch, whether the latest one should trigger an abort. E.g. a
- simple and sensible policy would be to abort if the total number of
- exceptions reaches a threshold, either absolute or relative to the
- number of items processed.
-
- * So how does it look? In the template...
-
-+---
-public void iterate(RepeatCallback callback) {
-
- ...
-
- try {
- result = callback.doInIteration(context);
- } catch (Exception e) {
- handleException(e); // Maybe re-throw, maybe not...
- }
-
- ...
-
-}
-+---
-
- If the callback was transactional it has already rolled back. If
- the whole <<>> was transactional we need to rethrow
-
- * If the processing is asynchronous, the template has to execute in
- a separate thread (see {{{./async.html}asynchronous example}}). In
- this case the whole thread (i.e. the <<>>) has to be
- transactional. Whoever is counting failed items needs to be
- poooling information from multiple threads.
-
- * It may also be the role of the framework to translate exceptions
- into a batch-specific hierarchy. This is not the same concern as
- exception classification (as done for instance by the Spring Jdbc
- and Jms templates). Exception classification might also be of
- value, but the argument is not as clear cut as the existing core
- templates, where there is an underlying Jave EE API checked
- exception to convert. In the absence of a batch-specific exception
- hierarchy definition, we could choose to leave exception translation
- out of the batch framework.
-
-
diff --git a/src/site/apt/cases/pause.apt b/src/site/apt/cases/pause.apt
deleted file mode 100644
index 15b568a3b..000000000
--- a/src/site/apt/cases/pause.apt
+++ /dev/null
@@ -1,121 +0,0 @@
- ------
- Pause Resume Use Case
- ------
- Dave Syer
- ------
- October 2008
-
-Use Case: Pause and Resume Job Execution
-
-* Goal
-
- Allow a job to pause itself and await further instructions. A
- paused status indicates to a user that the job is waiting, either
- for a manual signal to proceed, or for a remote worker to finish
- doing something asynchronously. For instance, a job may require
- manual verification of business condition before continuing - a
- sanity check on critical data. Assume that a job execution could
- receive hundreds of resume signals, and this is a "normal"
- situation, so it does not create a horrible mess in the history of
- the execution - e.g. looking like hundreds of restarts.
-
-* Scope
-
- * The instruction to pause comes from processing logic, not from an
- external signal (like an interrupt). A variation where the signal
- comes from outside might be a useful extension, but isn't explicitly
- included here.
-
-* Preconditions
-
- * A job is configured and one of its components can send the signal to pause
-
- * The launching interface has the ability to resume a paused job
-
- * The execution meta data can be inspected to verify that a pause has occurred
-
-* Success
-
- * User launches job and verifies that it has paused at a certain point
-
- * User resumes job and verifies that it completes successfully.
-
- * The end state is indistinguishable from a successful completion of
- the job in one attempt
-
-* Description
-
- The vanilla successful case proceeds as follows:
-
- [[1]] User launches a new job execution.
-
- [[1]] Framework begins processing, and successfully executes one
- or more steps.
-
- [[1]] At the end of a step Framework encounters condition that
- signals it should pause (e.g. a status flag).
-
- [[1]] Framework gracefully exits the job execution, marking it as
- paused so that it can be identifed as such when asked to resume.
- Often the framework will also be configured to notify a user that
- the pause has occurred, so that some business condition can be
- verified manually.
-
- [[1]] User requests the job execution be resumed.
-
- [[1]] Framework picks up where it left off, ignoring steps that
- have already successfully executed and starting with the one after
- the pause.
-
- [[1]] Job finishes processing and Framework marks it as
- sucessfully completed, just as if it hadn't paused in the first
- place.
-
-* Variations
-
- * The agent that causes the job to resume is not a User but a remote
- worker process.
-
- * Two agents request a resume at the same time. One of them has to
- lose (an exception is acceptable).
-
- * A step pauses in the middle of execution. The job picks it
- up and start where it left off, just like in a restart.
-
- * More than one step was executing when the pause signal was
- detected. Framework allows steps that are executing in process to
- complete (or pause) before exiting the job execution.
-
- * More than one step is in a paused state when the job resumes.
- Requires no special treatment from Framework: if those steps were
- active when the pause reached the job level on the last run, then
- they will be processed in the same way on a resume (presumably in
- multiple threads).
-
-* Implementation
-
- * A new <<>>.
-
- * The <<>> interface may not need any more than it already has:
-
-+---
-public interface JobLauncher {
-
- public JobExecution run(Job job, JobParameters jobParameters) throws ....;
-
-}
-+---
-
- In the case that the last execution failed, we already pick up from
- where we left off with a new <<>>. The only
- difference now is that we don't need a new <<>>, so we
- have to be careful about concurrency - what happens if two agents
- try to resume the job at once. To be safe we can treat this the
- same way as a restart - lock the <<>> table in the
- database by setting a TX isolation attribute on the
- <<>>.
-
- * When we resume we need to wind forward through the job execution
- and look at all step executions to see if they are active. Once the
- <<>> has been identified the process should be no
- different to a restart.
diff --git a/src/site/apt/cases/restart.apt b/src/site/apt/cases/restart.apt
deleted file mode 100644
index 8f703d5ef..000000000
--- a/src/site/apt/cases/restart.apt
+++ /dev/null
@@ -1,86 +0,0 @@
- ------
- Restart Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Manual Restart After Failure
-
-* Goal
-
- Restart a failed or interrupted batch and have it pick up where it
- left off (within limits of transaction boundaries) to save time and
- resources. A key goal is that the management of the batch process
- (locating a job and its input and results, starting, scheduling,
- restarting) should be as easy as possible for a non-developer, like
- an application support team with some business back up.
-
-* Scope
-
- Any batch should be able to restart gracefully, even if (depending
- on chosen execution or client implementation) it might have to go
- right back to the beginning.
-
-* Preconditions
-
- * It is possible to identify exception conditions under which a
- restart will be able to carry on processing a batch from where it
- left off.
-
- * There exists a persistent storage mechanism for the initial
- conditions.
-
-* Success
-
- * Force a batch to fail, and then fix the problem and restart. See
- successful completion with no duplicate results.
-
-* Description
-
- [[1]] A batch operation encounters an exception which forces the
- process to stop processing.
-
- [[1]] Framework catches exception and classifies it.
-
- [[1]] Framework logs event with enough information to identify the
- location of the job and the nature of the problem.
-
- [[1]] Framework saves initial condition from last commit point, to
- enable restart to start from the last known good operation.
-
- [[1]] Operator fixes problem (e.g. makes missing resource available,
- edits input file).
-
- [[1]] Operator restarts batch.
-
- [[1]] Framework loads initial conditions and continues processing.
-
-* Variations
-
- * Some restarts might lend themsleves to being handled automatically
- - see the use case {{{./retry.html}Automatic Retry}}.
-
-* Implementation
-
- * The saving of initial conditions needs to be strategised. In some
- cases saving a native serialization to a file will suffice. In
- others a database might be used, or some custom serialization
- (persist / rehydrate).
-
- * The initial condition is naturally under control of the
- <<>>. The client need not know about the persistence
- and rehydration. In fact explicit persistence and rehydration might
- be overkill - just relying on the transaction semantics might be
- adequate in a lot of cases. The <<>> would have to be
- aware of the transactions, which we assume are normally demarcated
- in the <<>>. Since the point at which persistence
- is needed is tied to transaction commits, there may have to be some
- transaction synchronization.
-
- * The persistence of initial conditions is a cross cutting concern.
- It may lend itself (along with the application of an execution
- handler generally) to being implemented as an aspect. Compare the
- <<>>, where the most common usage is via an
- interceptor, but occasionally the template is used directly by
- client code.
diff --git a/src/site/apt/cases/retry.apt b/src/site/apt/cases/retry.apt
deleted file mode 100644
index 5ea21cac7..000000000
--- a/src/site/apt/cases/retry.apt
+++ /dev/null
@@ -1,279 +0,0 @@
- ------
- Automatic Retry Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Automatic Retry
-
-* Goal
-
- Support automatic retry of an operation if it fails in certain
- pre-determined ways. Client code is not aware of the details of
- when and how many times to retry the operation, and various
- strategies for those details are available. The decision about
- whether to retry or abandon lies with the Framework, but is
- parameterisable through some retry meta data.
-
- Retryable operations are usually transactional, but this can be
- provided by a normal transaction template or interceptor
- (transaction meta data are independent of the retry meta data).
-
-* Scope
-
- Any operation can be retried, but there are restrictions on nesting
- transactions (normally an inner transaction needs to be
- propagation=NESTED).
-
-* Preconditions
-
- An operation exists that can be forced to fail and is able to
- succeed on a retry.
-
-* Success
-
- * Verify that an operation fails and then succeeds on a retry.
-
- * Verify that back off policy (time between retries) can be
- strategised without changing client code.
-
- * Verify that the retry policy can be strategised, and can be used
- to change the number of retry attempts depending on the type of
- exception thrown in the retry block.
-
-* Description
-
- Successful retry proceeds as follows:
-
- [[1]] Framework executes an operation provided by Client.
-
- [[1]] The operation fails and Framework catches an exception,
- classified as retryable.
-
- [[1]] Framework waits for a pre-defined back off period. The
- period is not be fixed, but is strategised so that different
- policies can be applied. The most common and useful policy is an
- exponentially increasing back off delay, with a ceiling.
-
- [[1]] Framework repeats the operation.
-
- [[1]] Processing is successful.
-
- [[1]] Framework stores and / or logs statistics about the retry
- for management purposes. Details?
-
-* Variations
-
- The following variations are supported.
-
-** Retry Failure
-
- A retry can fail for a number of reasons. E.g. if the number of
- retries is too high, or there is a timeout, or an exception of
- another sort that cannot be classified as retryable.
-
- [[1]] Last retry attempt fails and Framework determines that
- another retry is not permitted by the current policy.
-
- [[1]] Framework records status for management purposes.
-
- [[1]] Framework throws a recognisable exception?
-
- [[1]] Control may return to client (if the exception was caught),
- or the processing may end.
-
-** Transient and Non-transient Failures
-
- We may wish to classify exceptions into (at least) three types, and
- vary the retry policy based on the classification:
-
- * Transient failures come from resources that are external and may
- have independent lifecycles to the client process. Examples are
- database deadlock, network connectivity. It is always worth
- retrying on a transient failure, and normally we can keep retrying
- (if not forever then for a very long time), in the belief that
- eventually the resource will become available again.
-
- * Non-transient failures can be retried a few times. This is the
- default.
-
- * Non-retryable failures like a configuration or input data error
- should not be retried (they will always fail the same way).
-
-** Early Termination
-
- Normally client code is unaware of the Framework, but occasionally
- emergency measures might be taken inside client code where all
- further retry attempts are vetoed for the current block.
-
-** Stateful Retry
-
- A stateful (or external) retry is used to force a roll back of an
- external message (or other data) resource, so that the message will
- be re-delivered. The implementation has to be stateful so it can
- remember the context for the failed message next time it is
- delivered. The additional features of a stateful retry, as opposed
- to a normal rollback, are that:
-
- * A message can be retried indefinitely or up to a set number of
- times, after which an error processing route is taken.
-
- * A back-off delay is used at the of the retry
- before any other transactional resources are enlisted.
-
-* {Implementation}
-
- * The vanilla case and most of the variations can be achieved with a
- simple template approach:
-
-+---
-RetryTemplate retryTemplate = new RetryTemplate();
-retryTemplate.setRetryPolicy(new SimpleRetryPolicy(5));
-Object result = retryTemplate.execute(new RetryCallback() {
- public Object doWithRetry(RetryContext context) throws Throwable {
- // do some processing
- return result;
- }
-});
-+---
-
- * Schematically we can represent the implementation of the [retry}
- template as follows:
-
-+---
-1 | TRY {
-1.1 | do something;
-2 | } FAIL {
-2.1 | if (retry limit reached) {
-2.2 | rethrow exception;
- | } else {
-2.3 | TRY(1) again;
- | }
- | }
-+---
-
- * The template has policies for back off and retry (whether or not
- to retry the last exception). The example above shows the retry
- policy being set to simply retry all exceptions up to a limit of 5
- times.
-
- * The <<>> has an API that allows clients to override
- the retry policy. The context can also be accessed as a thread
- local from a static convenience class, in the case that the callback
- is implemented as a wrapper around a POJO.
-
- * External retry is the most difficult variation to implement, and
- doesn't fit naturally into the template model above. Two things
- depend on the retry count - back-off delay and the decision to
- follow the recovery path - so it needs to be available at the
- beginning of every processing block.
-
- We will discuss the implementation from a JMS-flavoured viewpoint,
- where the current item being processed is a message. This can be
- generalised to more generic data types, as long as the item can be
- rejected transactionally to signal that we require it to be
- re-delivered to this or another consumer.
-
- Consider this pattern, which is very typical:
-
-+---
-1 | SESSION {
-2 | receive;
-3 | RETRY {
- | remote access;
- | }
- | }
-+---
-
- A <<>> is responsible for the RETRY(3) block. But
- we can't put the same wrapper around the whole process:
-
-+---
-0 | RETRY { // Do not do this!
-1 | SESSION {
-2 | receive;
-3 | RETRY {
- | remote access;
- | }
- | }
- | }
-+---
-
- because the receive(2) might not get the same message back on the
- second and subsequent attempts (another consumer might get it, or it
- might come out of order). So external retry has a different flow -
- it might be a different implementation of the same interface, or a
- different parameterisation of the normal retry template.
-
- We can break down the implementation of an external retry into steps
- as follows:
-
-+---
-1 | SESSION {
-2 | receive;
-3 | TRY {
-3.1 | if (already processed) {
-3.2 | backoff;
- | }
-4 | RETRY {
- | remote access;
- | }
-5 | } FAIL {
-5.1 | if (retry limit reached) {
-5.2 | recover;
- | } else {
-5.3 | rethrow exception;
- | }
- | }
- | }
-+---
-
- Decisions (3.1) and (5.1) require knowledge of the history of
- processing the current message. Note that the action on failure is
- the opposite to the vanilla case {{{Implementation}above}} - if the retry
- limit is not reached then we rethrow the exception.
-
- If the retry limit is not reached then the rethrow(5.3) causes the
- SESSION(1) to roll back, and the message will be re-delivered.
- RETRY(4) is a normal retry with a template.
-
- The retry logic is easy to implement - the hard bit is that the
- policies depend on the history of the message. This requires some
- special retry and back off policies that are aware of the history:
-
- * When a message arrives, at the beginning of the TRY(3) above, we
- need to update our knowledge of its history.
-
- * The backoff policy can decide whether to back off immediately
- when it is initialized at step (3.1).
-
- * The retry decision at (5.1) has to be aware of the history as
- well as some simple exception classification rules.
-
- * If the retry cannot proceed the retry policy can take steps to
- recover (5.2), e.g. send the current message to an error queue.
- The exception should not propagate in this case.
-
- * If we fail and rethrow (5.3), then we need to store the
- knowledge of the message history somewhere where another consumer
- can access it.
-
- There is a small conundrum about what value to return from the
- TRY(3) block if it ultimately fails (5.2) - a normal retry never
- completes unless it is successful, but an external retry can
- complete if it is unsuccessful. The obvious choice is to return
- null. It probably won't matter in a messaging application anyway
- because the client of the retry block probably isn't expecting
- anything. It may matter if the TRY(3) block is part of a batch
- because the batch template uses null as a signal that the current
- batch is complete. But on the other hand it might be a good
- strategy to close the batch if processing a message fails.
-
- With JMS there is no indication in the <<>> how many times
- it has been rejected - only a flag <<>> to show
- that it has failed at least once. To count the number of retries,
- we have to store a global map of messages (ids) to retry counts
- (within a single VM - for more than one OS process each one has to
- be independent).
-
diff --git a/src/site/apt/cases/scheduled.apt b/src/site/apt/cases/scheduled.apt
deleted file mode 100644
index 7b65ce82d..000000000
--- a/src/site/apt/cases/scheduled.apt
+++ /dev/null
@@ -1,42 +0,0 @@
- ------
- Scheduler Managed Use Case
- ------
- Wayne Lund, Dave Syer
- ------
- May 2007
-
-Use Case: Scheduler Managed Processing
-
-* Goal
-
- Ensure that an Enterprise Scheduler can interact with the Batch Launcher to start, stop,
- suspend and/or kill a batch job.
-
-* Scope
-
- * Batch jobs tends to run within carefully planned job stream
- schedules. At a minimum this requires an integration between the
- Batch Launcher (in the abstract) and the scheduler's control
- mechanism to start and stop batch jobs and then to understand the
- results of the batch job execution (e.g. COMPLETED, ABENDED, etc.)
- so that subsequent actions may be taken.
-
- * Spring Batch does not aim to implement the scheduling concerns as
- such (other tools are available for that). The framework, does need
- to provide the information that such tools need to decide when to
- act and what to do (e.g. exit code mapping).
-
-* Preconditions
-
- * A mechanism has been established for the scheduler to launch a batch job. This is often times
- a simple unix or dos shell script.
-
- * A mapping of exit codes to the error code numbers that the scheduler is expecting on the exiting
- of a batch job.
-
-* Success
-
- * Batch Jobs are launched and managed by scheduler
-
-* Description
-
diff --git a/src/site/apt/cases/simple.apt b/src/site/apt/cases/simple.apt
deleted file mode 100644
index b24c77233..000000000
--- a/src/site/apt/cases/simple.apt
+++ /dev/null
@@ -1,290 +0,0 @@
- ------
- Simple Batch Repeat Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Simple Batch Repeat
-
-* Goal
-
- Repeat a simple operation such as processing a data item, or a
- message, up to a fixed number of times, normally with a transaction
- scoped to the whole batch. Transaction resources are shared between
- the operations in the batch, leading to performance benefits.
-
-* Scope
-
- The operation to be repeated:
-
- * Can expect to use and manage its own I/O or datastore resources,
- but not necessarily transactions;
-
- * May need to introspect the batch status (as a variation);
-
- * Executes synchronously or asynchronously (as a variation).
-
- * Is stateless - this is not a framework restriction in principle,
- but simplifies the implementation for now. See in the
- {{{store}Implementation}} section below for some notes on
- stateful synchronisation;
-
- * Should be implementable as a POJO if desired.
-
-* Preconditions
-
- Client code can locate and acquire all the resources it needs for
- the batched operation, and can force transactions to rollback for
- testing purposes.
-
-* Success
-
- * Verify that a successful batch executed a fixed number of times.
-
- * Verify that a batch completes early but successfully if an
- underlying transaction times out.
-
- * Terminate a batch by failing one of the operations, and verify
- that the preceding operations rolled back (subject to batch meta
- data).
-
- * Execute a batch asynchronously and verify that the correct number
- of operations is performed.
-
-* Description
-
- We are often interested in a specific scenario of this use case
- where the batched operation is:
-
- * Read a message or data item from an endpoint like a JMS
- Destination.
-
- * Do some business processing involving database reads and writes.
-
- The vanilla successful batch use case proceeds as follows:
-
- [[1]] Framework starts a batch, acquiring resources as needed and
- creating a context for the execution.
-
- [[1]] Client provides a batch operation in the form of a source of
- data items and a processor acting on the data item.
-
- [[1]] Framework executes batch operation.
-
- [[1]] Repeat the last step until the batch size is reached.
-
- [[1]] Framework commits the batch. All database changes are
- committed and received messages removed from the endpoints.
-
-* Variations
-
-** Rollback
-
- If one of the operations rolls back it will throw an exception.
- Normal transaction semantics determine what happens next. Usually
- (in the scenario described above) there is an outer transaction for
- the whole batch, which rolls back as well: all the messages remain
- unsent, and all the data remain uncommitted. A retry will receive
- exactly the same initial conditions.
-
-** Timeout
-
- The batch size is not fixed. The use case proceeds as above, but in
- the middle of a batch operation execution:
-
-
- [[1]] Framework determines that the batch has timed out operation
- (e.g. while it was waiting for an incoming message).
-
- [[1]] Framework commits the batch with all operations so far
- complete - possibly a smaller than normal size.
-
-** Asynchronous Processing
-
- Instead of the Framework waiting for each operation to complete it
- could spin them off independently into separate threads or a work
- queue. The batch still has to have a definite endpoint, so the
- Framework waits for all the operations to finish or fail
- before cmpleting the batch.
-
-** Introspection of Batch Context
-
- Client may wish to inspect the state of the ongoing batch operation,
- and potentially force an early completion.
-
-* {Implementation}
-
- * The completion of the batch loop is handled by a policy delegate
- that we can use to strategise the concept of a loop that might
- complete early. This can cover both the timeout variation and the
- vanilla use case flow.
-
- * What form should the batch template (<<>>)
- interface take? We might start with something like this:
-
-+---
-batchTemplate.iterate(new RepeatCallback() {
-
- public boolean doInIteration() {
- // do stuff
- }
-
-});
-+---
-
- * A nice tool for a batch operation in a callback is an iterator
- through a data set or message endpoint (<<>>), coupled
- with a handler for processing the item. This adds a potential
- implementation of <<>> that knows about the
- <<>> and adds a processor object. E.g. as an
- anonymous inner class:
-
-+---
-final ItemProvider provider = new JmsItemProvider();
-final ItemProcessor processor = new ItemProcessor() {
- public void process(Object data) {
- // do something with the data (a record)
- }
-};
-
-batchTemplate.execute(new RepeatCallback() {
-
- public boolean doInIteration() {
- Object data = provider.next();
- if (data!=null) {
- processor.process(data);
- }
- return data!=null;
- }
-
-});
-+---
-
- * Is a batch template with callback the best implementation? Could
- we perhaps use or re-use <<>> somehow? Which is
- better for the client:
-
-+---
-batchTemplate.iterate(new RepeatCallback() {
-
- public boolean doInIteration() {
- // do stuff
- }
-
-});
-+---
-
- where the batch template might itself use a <<>>
- internally, or
-
-+---
-batchTemplate.iterate(new Runnable() {
-
- public void run() {
- // do stuff with data
- };
-
-});
-+---
-
- where the batch template is a <<>>. Probably the
- former because it is more encapsulated: it gives the framework more
- freedom to implement the template in any way it needs to, e.g. to
- accommodate more complicated use cases.
-
- * To {store} up SQL operations until the end of a batch, and take
- advantage of JDBC driver efficiencies, the client needs to store
- some state during the batch, and also register a transaction
- synchronisation. For this kind of scenario we introduce an
- interceptor framework in the template execution. The template calls
- back to interceptors, which themselves can strategise clean up and
- close-type behaviour:
-
-+---
-public class RepeatTemplate implements RepeatOperations {
-
- public void iterate(RepeatCallback callback) {
-
- // set up the batch
- interceptors.open();
-
- while (running) {
-
- // allow interceptor to pre-process and veto continuation
- interceptor.before();
-
- // continue only if batch is ongoing
- if (running = callback.doInIteration()!=null) {
- interceptor.after();
- }
-
- }
-
- // clean up or commit the whole batch
- interceptor.close();
-
- }
-}
-+---
-
- The <<>> can be stateful, and can store up inserts
- until the end of the batch. If the <<>> is
- transactional then they will only happen if the transaction is
- successful.
-
- This way the client can even decide to use a batch interceptor
- that runs in its own transaction at the end of the batch.
-
- * There is no need for an overall batch timeout because the inner
- operations are synchronous and have their own timeout metadata
- though transaction definitions. The whole batch (outer transaction)
- may still have a timeout attribute, and then there is a corner case
- where the batch operations are all successful, but because they all
- took a long time the whole batch rolls back because of the timeout.
-
- * The context of the ongoing batch is closely linked with the
- completion policy. The completion policy is pluggable into the
- batch template, and acts as a factory for context objects which can
- then be inspected by Client in the callback. For example:
-
-+---
-public class RepeatTemplate implements RepeatOperations {
-
- public void iterate(RepeatCallback callback) {
-
- // set up the batch session
- RepeatContext context = completionPolicy.start();
-
- while (!completionPolicy.isComplete(context)) {
-
- // callback gets the context as an argument
- callback.doInIteration(context);
-
- completionPolicy.update(context);
- }
-
- }
-}
-+---
-
- * The example above provides Client the opportunity to inspect the
- context through the callback interface. If Client is a POJO,
- Framework has to create a callback and wrap it, in which case there
- needs to be a global accessor for the current context or session.
- The template is then responsible for registering the current context
- with a <<>>. E.g.client code can look
- at the session and mark it as complete if desired
- (c.f. <<>>):
-
-+---
-public Object doMyBatch() {
-
- // do some processing
-
- // something bad happened...
- RepeatContext context = RepeatSynchronizationManager.getContext();
- context.setCompleteOnly();
-
-}
-+---
diff --git a/src/site/apt/cases/steps.apt b/src/site/apt/cases/steps.apt
deleted file mode 100644
index c0340464b..000000000
--- a/src/site/apt/cases/steps.apt
+++ /dev/null
@@ -1,176 +0,0 @@
- ------
- Batch: Sequential Steps Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Sequential Processing of Dependent Steps
-
-* Goal
-
- Compose a batch operation from a sequence of dependent steps.
- Define and implement the operation only once, and allow restart
- after failure without having to change configuration, and without
- having to repeat steps that were successful.
-
- A sub-goal is to allow the progress of a batch through the steps to
- be traced accurately for reporting and auditing purposes. This
- requires the steps to be uniquely identified.
-
-* Scope
-
- * Simple linear sequence of steps. Slightly more complicated
- requirements can be handled by putting independent steps in a
- sequence (no need for splits and joins).
-
-* Preconditions
-
- * A non-trivial sequence is defined:
-
- * more than one step:
-
- * the effects of each step can be measured.
-
- * The sequence can be interrupted or artificially terminated in the
- second or subsequent step.
-
-* Success
-
- * A non-trivial sequence executes successfully. The progress and
- success of each step can be verified by the tester.
-
- * The same sequence is forced to fail on second step in such a way
- that the first step result is not suspected of being in error,
- e.g. by interrupting it. When it is restarted the first step is not
- repeated, and the sequence is successful.
-
- * The same sequence is forced to fail on second step in such a way
- that the first step result is obviously in error, even though it
- completed normally. When the batch is restarted the first step
- repeated, and the sequence is successful.
-
-* Description
-
- The vanilla successful case proceeds as follows:
-
- [[1]] Framework logs the start of a step, uniquely indentifying
- the initial conditions.
-
- [[1]] Framework stores internal state so that initial conditions
- can be re-created in the event of a restart.
-
- [[1]] Step execution proceeds as per one of the other use cases
- (e.g. {{{file-to-database.html}Copy File to Database}}), including
- transactional behaviour.
-
- [[1]] Client instructs Framework to store internal state needed by
- further steps (e.g. cached reference data).
-
- [[1]] Framework logs successful completion of step, and stores
-
- [[1]] Repeat for next and subsequent steps. Internal state is
- passed from one state to the next.
-
-* Variations
-
-** Internal Failure of Step
-
- If a step fails internally, e.g. because of resource becoming
- temporarily unavailable, the sequence can be restarted without
- repeating the previous steps.
-
- [[1]] Operator fixes resource problem (e.g. starts web service).
-
- [[1]] Operator restarts batch with no configuration or input data
- changes.
-
- [[1]] Framework resumes batch from the last commit point of the
- failed step.
-
- [[1]] Sequence completes normally.
-
- The process above could be carried out by the framework entirely (no
- need for operator intervention) if a retry policy is in effect.
-
-** Failure of Step Owing to Bad Initial State
-
- If a step fails because it receives bad data from an earlier step,
- the Framework cannot recover without intervention.
-
- [[1]] Operator attempts to restart without doing anything to fix
- the problem.
-
- [[1]] Framework detects bad initial state immediately and fails
- fast.
-
- If the original problem can be located and fixed (e.g. input data
- for earlier step is revised):
-
- [[1]] Operator restarts batch signalling to framework which step
- to begin with.
-
- [[1]] Framework locates initial state for the first step to be
- executed.
-
- [[1]] Framework starts execution from the beginning of the desired
- state. This time the input data are different, so the sequence
- can complete normally.
-
-* Implementation
-
- * The need to save state for subsequent steps leads to the
- introduction of a batch context concept. And the need for
- initialising restarts leads to the context being serializable,
- either natively or by some pluggable strategy (this is covered in
- the {{{./restart.html}Restart after Failure}} use case).
-
- Unfortunately, the need for {{{./parallel.html}parallel processing}}
- and automatic {{{./restart.html}restart}} also makes it practically
- impossible for steps to handle the context at the level of a single
- thread of execution, where the client needs to implement business
- logic. If a step is executing in parallel, then each node needs to
- be able to restart independently, but the context needs to be a
- single object that can be passed on to the next step (unless all the
- steps are parallelised with the same multiplicity, which might not
- be efficient in general).
-
- Thus batch context must be defined and managed by the template or
- execution handler.
-
- * The requirement for steps might have implications for the
- implementer of the batch operation (the client). Obviously a client
- defines the sequence of steps according to the business requirement,
- but ideally we would like him to be unaware of the reporting and
- restart infrastructure. Maybe an array of callbacks works (the
- callback interface is irrelevant, except that it accepts a context
- object as an argument):
-
-+---
-batchTemplate.iterate(new RepeatCallback[] {
-
- new RepeatCallback() {
- public boolean doInIteration(RepeatContext context) {
- // do stuff for step one
- };
- },
-
- new RepeatCallback() {
- public boolean doInIteration(RepeatContext context) {
- // do stuff for step two - the context
- // is the same...
- };
- }
-
-});
-+---
-
- Notice that there is no need for the context to be set explicitly
- before executing the callback. The context is handled internally to
- the batch template using an analogue of the
- <<>>.
-
- * If we prefer that clients never need to know about batch
- templates, then the code above needs to be automated. This would be
- where an additional domain layer might come into play
- (c.f. <<>>).
diff --git a/src/site/apt/cases/template.apt b/src/site/apt/cases/template.apt
deleted file mode 100644
index ed359bed6..000000000
--- a/src/site/apt/cases/template.apt
+++ /dev/null
@@ -1,22 +0,0 @@
- ------
- Template Use Case
- ------
- Dave Syer
- ------
- January 2007
-
-Use Case: Template
-
-* Goal
-
-* Scope
-
-* Preconditions
-
-* Success
-
-* Description
-
-* Variations
-
-* Implementation
\ No newline at end of file
diff --git a/src/site/apt/downloads.apt b/src/site/apt/downloads.apt
deleted file mode 100644
index 60a3629d4..000000000
--- a/src/site/apt/downloads.apt
+++ /dev/null
@@ -1,133 +0,0 @@
- ---------
- Downloads
- ---------
- Dave Syer, Ben Hale, Michael Minella
- ------
- December 2007, February 2009, July 2009
-
-Spring Batch Downloads
-
- The current GA release is <<2.2.6.RELEASE>>, the latest snapshots are <<3.0.0.BUILD-SNAPSHOT>>. The version 2.0.x and 1.x branches are now in maintenance (the last release was <<2.0.4.RELEASE>>).
-
-For runtime concerns and a container for running a Job as a service see the {{{http://docs.spring.io/spring-batch-admin/}Spring Batch Admin}} project and the {{{http://docs.spring.io/spring-batch-admin/getting-started.html}getting started}} link there.
-
-* Zip Downloads
-
- There is a ZIP artifact containing the release JARs called <<>>. This file contains the JAR files for the release, including source code and the samples.
-
- * Full releases: {{{http://docs.spring.io/downloads/nightly/release-download.php?project=BATCH}here}}.
-
- * Milestones: {{{http://docs.spring.io/downloads/nightly/milestone-download.php?project=BATCH}here}}.
-
-Source code can also be browsed and downloaded at {{{https://github.com/spring-projects/spring-batch}Github}}.
-
-* Maven Artifacts
-
- Traditional "spring-*" artifacts are deployed on the Maven central repo. Full releases in this format also go in our s3 repository, browseable {{{http://shrub.appspot.com/maven.springframework.org/release/org/springframework/batch/}here}}:
-
-+---------------
-
- spring-releases
- Spring Maven RELEASE Repository
- http://repo.spring.io/release
-
-+---------------
-
- You can see the internal and external project dependencies in the <<>> files and also in the dependency reports for each module on this website. You will probably need <<>> and <<>>. Source code is packaged in a separate jar file in the same directory, and the samples are also bundled in the same way.
-
- Individual dependencies can then by added like so (inside a \ element at the top level):
-
-+---------------
-
-