Re-organize documentation resources

This commit also adds scaling diagrams [1]
to Figures.ppt. Those diagrams were not under
source control.

[1]: https://docs.spring.io/spring-batch/docs/current/reference/html/spring-batch-integration.html#externalizing-batch-process-execution

spring-batch-docs/src/main/asciidoc/appendix.adoc

@@ -0,0 +1,135 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[listOfReadersAndWriters]]
+
+[appendix]
+== List of ItemReaders and ItemWriters
+
+[[itemReadersAppendix]]
+
+=== Item Readers
+
+.Available Item Readers
+[options="header"]
+|===============
+|Item Reader|Description
+|AbstractItemCountingItemStreamItemReader|Abstract base class that provides basic
+            restart capabilities by counting the number of items returned from
+            an `ItemReader`.
+|AggregateItemReader|An `ItemReader` that delivers a list as its
+            item, storing up objects from the injected `ItemReader` until they
+            are ready to be packed out as a collection. This class must be used 
+            as a wrapper for a custom `ItemReader` that can identify the record 
+            boundaries. The custom reader should mark the beginning and end of 
+            records by returning an `AggregateItem` which responds `true` to its 
+            query methods `isHeader()` and `isFooter()`. Note that this reader
+            is not part of the library of readers provided by Spring Batch
+            but given as a sample in `spring-batch-samples`.
+|AmqpItemReader|Given a Spring `AmqpTemplate`, it provides
+            synchronous receive methods. The `receiveAndConvert()` method
+            lets you receive POJO objects.
+|KafkaItemReader|An `ItemReader` that reads messages from an Apache Kafka topic.
+It can be configured to read messages from multiple partitions of the same topic.
+This reader stores message offsets in the execution context to support restart capabilities.
+|FlatFileItemReader|Reads from a flat file. Includes `ItemStream`
+            and `Skippable` functionality. See link:readersAndWriters.html#flatFileItemReader[`FlatFileItemReader`].
+|HibernateCursorItemReader|Reads from a cursor based on an HQL query. See
+            link:readersAndWriters.html#cursorBasedItemReaders[`Cursor-based ItemReaders`].
+|HibernatePagingItemReader|Reads from a paginated HQL query
+|ItemReaderAdapter|Adapts any class to the
+            `ItemReader` interface.
+|JdbcCursorItemReader|Reads from a database cursor via JDBC. See
+            link:readersAndWriters.html#cursorBasedItemReaders[`Cursor-based ItemReaders`].
+|JdbcPagingItemReader|Given an SQL statement, pages through the rows,
+            such that large datasets can be read without running out of
+            memory.
+|JmsItemReader|Given a Spring `JmsOperations` object and a JMS
+            Destination or destination name to which to send errors, provides items
+            received through the injected `JmsOperations#receive()`
+            method.
+|JpaPagingItemReader|Given a JPQL statement, pages through the
+            rows, such that large datasets can be read without running out of
+            memory.
+|ListItemReader|Provides the items from a list, one at a
+            time.
+|MongoItemReader|Given a `MongoOperations` object and a JSON-based MongoDB
+            query, provides items received from the `MongoOperations#find()` method.
+|Neo4jItemReader|Given a `Neo4jOperations` object and the components of a
+            Cyhper query, items are returned as the result of the Neo4jOperations.query
+            method.
+|RepositoryItemReader|Given a Spring Data `PagingAndSortingRepository` object,
+            a `Sort`, and the name of method to execute, returns items provided by the
+            Spring Data repository implementation.
+|StoredProcedureItemReader|Reads from a database cursor resulting from the
+            execution of a database stored procedure. See link:readersAndWriters.html#StoredProcedureItemReader[`StoredProcedureItemReader`]
+|StaxEventItemReader|Reads via StAX. see link:readersAndWriters.html#StaxEventItemReader[`StaxEventItemReader`].
+|JsonItemReader|Reads items from a Json document. see link:readersAndWriters.html#JsonItemReader[`JsonItemReader`].
+
+|===============
+
+
+[[itemWritersAppendix]]
+
+
+=== Item Writers
+
+.Available Item Writers
+[options="header"]
+|===============
+|Item Writer|Description
+|AbstractItemStreamItemWriter|Abstract base class that combines the
+            `ItemStream` and
+            `ItemWriter` interfaces.
+|AmqpItemWriter|Given a Spring `AmqpTemplate`, it provides
+            for a synchronous `send` method. The `convertAndSend(Object)`
+             method lets you send POJO objects.
+|CompositeItemWriter|Passes an item to the `write` method of each
+            in an injected `List` of `ItemWriter` objects.
+|FlatFileItemWriter|Writes to a flat file. Includes `ItemStream` and
+            Skippable functionality. See link:readersAndWriters.html#flatFileItemWriter[`FlatFileItemWriter`].
+|GemfireItemWriter|Using a `GemfireOperations` object, items are either written
+            or removed from the Gemfire instance based on the configuration of the delete
+            flag.
+|HibernateItemWriter|This item writer is Hibernate-session aware
+            and handles some transaction-related work that a non-"hibernate-aware"
+            item writer would not need to know about and then delegates
+            to another item writer to do the actual writing.
+|ItemWriterAdapter|Adapts any class to the
+            `ItemWriter` interface.
+|JdbcBatchItemWriter|Uses batching features from a
+            `PreparedStatement`, if available, and can
+            take rudimentary steps to locate a failure during a
+            `flush`.
+|JmsItemWriter|Using a `JmsOperations` object, items are written
+            to the default queue through the `JmsOperations#convertAndSend()` method.
+|JpaItemWriter|This item writer is JPA EntityManager-aware
+            and handles some transaction-related work that a non-"JPA-aware"
+            `ItemWriter` would not need to know about and
+            then delegates to another writer to do the actual writing.
+|KafkaItemWriter|Using a `KafkaTemplate` object, items are written to the default topic through the
+            `KafkaTemplate#sendDefault(Object, Object)` method using a `Converter` to map the key from the item.
+            A delete flag can also be configured to send delete events to the topic.
+|MimeMessageItemWriter|Using Spring's `JavaMailSender`, items of type `MimeMessage`
+            are sent as mail messages.
+|MongoItemWriter|Given a `MongoOperations` object, items are written
+            through the `MongoOperations.save(Object)` method.  The actual write is delayed
+            until the last possible moment before the transaction commits.
+|Neo4jItemWriter|Given a `Neo4jOperations` object, items are persisted through the
+            `save(Object)` method or deleted through the `delete(Object)` per the
+            `ItemWriter's` configuration
+|PropertyExtractingDelegatingItemWriter|Extends `AbstractMethodInvokingDelegator`
+            creating arguments on the fly. Arguments are created by retrieving
+            the values from the fields in the item to be processed (through a
+            `SpringBeanWrapper`), based on an injected array of field
+            names.
+|RepositoryItemWriter|Given a Spring Data `CrudRepository` implementation,
+            items are saved through the method specified in the configuration.
+|StaxEventItemWriter|Uses a `Marshaller` implementation to
+            convert each item to XML and then writes it to an XML file using
+            StAX.
+|JsonFileItemWriter|Uses a `JsonObjectMarshaller` implementation to
+            convert each item to Json and then writes it to an Json file.
+
+|===============

spring-batch-docs/src/main/asciidoc/common-patterns.adoc

@@ -0,0 +1,774 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[commonPatterns]]
+
+== Common Batch Patterns
+
+ifndef::onlyonetoggle[]
+include::toggle.adoc[]
+endif::onlyonetoggle[]
+
+
+Some batch jobs can be assembled purely from off-the-shelf components in Spring Batch.
+For instance, the `ItemReader` and `ItemWriter` implementations can be configured to
+cover a wide range of scenarios. However, for the majority of cases, custom code must be
+written. The main API entry points for application developers are the `Tasklet`, the
+`ItemReader`, the `ItemWriter`, and the various listener interfaces. Most simple batch
+jobs can use off-the-shelf input from a Spring Batch `ItemReader`, but it is often the
+case that there are custom concerns in the processing and writing that require developers
+to implement an `ItemWriter` or `ItemProcessor`.
+
+In this chapter, we provide a few examples of common patterns in custom business logic.
+These examples primarily feature the listener interfaces. It should be noted that an
+`ItemReader` or `ItemWriter` can implement a listener interface as well, if appropriate.
+
+[[loggingItemProcessingAndFailures]]
+=== Logging Item Processing and Failures
+
+A common use case is the need for special handling of errors in a step, item by item,
+perhaps logging to a special channel or inserting a record into a database. A
+chunk-oriented `Step` (created from the step factory beans) lets users implement this use
+case with a simple `ItemReadListener` for errors on `read` and an `ItemWriteListener` for
+errors on `write`. The following code snippet illustrates a listener that logs both read
+and write failures:
+
+[source, java]
+----
+public class ItemFailureLoggerListener extends ItemListenerSupport {
+
+    private static Log logger = LogFactory.getLog("item.error");
+
+    public void onReadError(Exception ex) {
+        logger.error("Encountered error on read", e);
+    }
+
+    public void onWriteError(Exception ex, List<? extends Object> items) {
+        logger.error("Encountered error on write", ex);
+    }
+}
+----
+
+Having implemented this listener, it must be registered with a step.
+
+[role="xmlContent"]
+The following example shows how to register a listener with a step in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<step id="simpleStep">
+...
+<listeners>
+    <listener>
+        <bean class="org.example...ItemFailureLoggerListener"/>
+    </listener>
+</listeners>
+</step>
+----
+
+[role="javaContent"]
+The following example shows how to register a listener with a step Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Step simpleStep() {
+	return this.stepBuilderFactory.get("simpleStep")
+				...
+				.listener(new ItemFailureLoggerListener())
+				.build();
+}
+----
+
+IMPORTANT: if your listener does anything in an `onError()` method, it must be inside
+a transaction that is going to be rolled back. If you need to use a transactional
+resource, such as a database, inside an `onError()` method, consider adding a declarative
+transaction to that method (see Spring Core Reference Guide for details), and giving its
+propagation attribute a value of `REQUIRES_NEW`.
+
+[[stoppingAJobManuallyForBusinessReasons]]
+=== Stopping a Job Manually for Business Reasons
+
+Spring Batch provides a `stop()` method through the `JobOperator` interface, but this is
+really for use by the operator rather than the application programmer. Sometimes, it is
+more convenient or makes more sense to stop a job execution from within the business
+logic.
+
+The simplest thing to do is to throw a `RuntimeException` (one that is neither retried
+indefinitely nor skipped). For example, a custom exception type could be used, as shown
+in the following example:
+
+[source, java]
+----
+public class PoisonPillItemProcessor<T> implements ItemProcessor<T, T> {
+
+    @Override
+    public T process(T item) throws Exception {
+        if (isPoisonPill(item)) {
+            throw new PoisonPillException("Poison pill detected: " + item);
+        }
+        return item;
+    }
+}
+----
+
+Another simple way to stop a step from executing is to return `null` from the
+`ItemReader`, as shown in the following example:
+
+[source, java]
+----
+public class EarlyCompletionItemReader implements ItemReader<T> {
+
+    private ItemReader<T> delegate;
+
+    public void setDelegate(ItemReader<T> delegate) { ... }
+
+    public T read() throws Exception {
+        T item = delegate.read();
+        if (isEndItem(item)) {
+            return null; // end the step here
+        }
+        return item;
+    }
+
+}
+----
+The previous example actually relies on the fact that there is a default implementation
+of the `CompletionPolicy` strategy that signals a complete batch when the item to be
+processed is `null`. A more sophisticated completion policy could be implemented and
+injected into the `Step` through the `SimpleStepFactoryBean`.
+
+[role="xmlContent"]
+The following example shows how to inject a completion policy into a step in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<step id="simpleStep">
+    <tasklet>
+        <chunk reader="reader" writer="writer" commit-interval="10"
+               chunk-completion-policy="completionPolicy"/>
+    </tasklet>
+</step>
+
+<bean id="completionPolicy" class="org.example...SpecialCompletionPolicy"/>
+----
+
+[role="javaContent"]
+The following example shows how to inject a completion policy into a step in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Step simpleStep() {
+	return this.stepBuilderFactory.get("simpleStep")
+				.<String, String>chunk(new SpecialCompletionPolicy())
+				.reader(reader())
+				.writer(writer())
+				.build();
+}
+----
+
+An alternative is to set a flag in the `StepExecution`, which is checked by the `Step`
+implementations in the framework in between item processing. To implement this
+alternative, we need access to the current `StepExecution`, and this can be achieved by
+implementing a `StepListener` and registering it with the `Step`. The following example
+shows a listener that sets the flag:
+
+[source, java]
+----
+public class CustomItemWriter extends ItemListenerSupport implements StepListener {
+
+    private StepExecution stepExecution;
+
+    public void beforeStep(StepExecution stepExecution) {
+        this.stepExecution = stepExecution;
+    }
+
+    public void afterRead(Object item) {
+        if (isPoisonPill(item)) {
+            stepExecution.setTerminateOnly();
+       }
+    }
+
+}
+----
+
+When the flag is set, the default behavior is for the step to throw a
+`JobInterruptedException`. This behavior can be controlled through the
+`StepInterruptionPolicy`. However, the only choice is to throw or not throw an exception,
+so this is always an abnormal ending to a job.
+
+[[addingAFooterRecord]]
+===  Adding a Footer Record
+
+Often, when writing to flat files, a "`footer`" record must be appended to the end of the
+file, after all processing has be completed. This can be achieved using the
+`FlatFileFooterCallback` interface provided by Spring Batch. The `FlatFileFooterCallback`
+(and its counterpart, the `FlatFileHeaderCallback`) are optional properties of the
+`FlatFileItemWriter` and can be added to an item writer.
+
+[role="xmlContent"]
+The following example shows how to use the `FlatFileHeaderCallback` and the
+`FlatFileFooterCallback` in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<bean id="itemWriter" class="org.spr...FlatFileItemWriter">
+    <property name="resource" ref="outputResource" />
+    <property name="lineAggregator" ref="lineAggregator"/>
+    <property name="headerCallback" ref="headerCallback" />
+    <property name="footerCallback" ref="footerCallback" />
+</bean>
+----
+
+[role="javaContent"]
+The following example shows how to use the `FlatFileHeaderCallback` and the
+`FlatFileFooterCallback` in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public FlatFileItemWriter<String> itemWriter(Resource outputResource) {
+	return new FlatFileItemWriterBuilder<String>()
+			.name("itemWriter")
+			.resource(outputResource)
+			.lineAggregator(lineAggregator())
+			.headerCallback(headerCallback())
+			.footerCallback(footerCallback())
+			.build();
+}
+----
+
+The footer callback interface has just one method that is called when the footer must be
+written, as shown in the following interface definition:
+
+[source, java]
+----
+public interface FlatFileFooterCallback {
+
+    void writeFooter(Writer writer) throws IOException;
+
+}
+----
+
+[[writingASummaryFooter]]
+==== Writing a Summary Footer
+
+A common requirement involving footer records is to aggregate information during the
+output process and to append this information to the end of the file. This footer often
+serves as a summarization of the file or provides a checksum.
+
+For example, if a batch job is writing `Trade` records to a flat file, and there is a
+requirement that the total amount from all the `Trades` is placed in a footer, then the
+following `ItemWriter` implementation can be used:
+
+[source, java]
+----
+public class TradeItemWriter implements ItemWriter<Trade>,
+                                        FlatFileFooterCallback {
+
+    private ItemWriter<Trade> delegate;
+
+    private BigDecimal totalAmount = BigDecimal.ZERO;
+
+    public void write(List<? extends Trade> items) throws Exception {
+        BigDecimal chunkTotal = BigDecimal.ZERO;
+        for (Trade trade : items) {
+            chunkTotal = chunkTotal.add(trade.getAmount());
+        }
+
+        delegate.write(items);
+
+        // After successfully writing all items
+        totalAmount = totalAmount.add(chunkTotal);
+    }
+
+    public void writeFooter(Writer writer) throws IOException {
+        writer.write("Total Amount Processed: " + totalAmount);
+    }
+
+    public void setDelegate(ItemWriter delegate) {...}
+}
+----
+
+This `TradeItemWriter` stores a `totalAmount` value that is increased with the `amount`
+from each `Trade` item written. After the last `Trade` is processed, the framework calls
+`writeFooter`, which puts the `totalAmount` into the file. Note that the `write` method
+makes use of a temporary variable, `chunkTotal`, that stores the total of the
+`Trade` amounts in the chunk. This is done to ensure that, if a skip occurs in the
+`write` method, the `totalAmount` is left unchanged. It is only at the end of the `write`
+method, once we are guaranteed that no exceptions are thrown, that we update the
+`totalAmount`.
+
+In order for the `writeFooter` method to be called, the `TradeItemWriter` (which
+implements `FlatFileFooterCallback`) must be wired into the `FlatFileItemWriter` as the
+`footerCallback`.
+
+[role="xmlContent"]
+The following example shows how to wire the `TradeItemWriter` in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<bean id="tradeItemWriter" class="..TradeItemWriter">
+    <property name="delegate" ref="flatFileItemWriter" />
+</bean>
+
+<bean id="flatFileItemWriter" class="org.spr...FlatFileItemWriter">
+   <property name="resource" ref="outputResource" />
+   <property name="lineAggregator" ref="lineAggregator"/>
+   <property name="footerCallback" ref="tradeItemWriter" />
+</bean>
+----
+
+[role="javaContent"]
+The following example shows how to wire the `TradeItemWriter` in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public TradeItemWriter tradeItemWriter() {
+	TradeItemWriter itemWriter = new TradeItemWriter();
+
+	itemWriter.setDelegate(flatFileItemWriter(null));
+
+	return itemWriter;
+}
+
+@Bean
+public FlatFileItemWriter<String> flatFileItemWriter(Resource outputResource) {
+	return new FlatFileItemWriterBuilder<String>()
+			.name("itemWriter")
+			.resource(outputResource)
+			.lineAggregator(lineAggregator())
+			.footerCallback(tradeItemWriter())
+			.build();
+}
+----
+
+The way that the `TradeItemWriter` has been written so far functions correctly only if
+the `Step` is not restartable. This is because the class is stateful (since it stores the
+`totalAmount`), but the `totalAmount` is not persisted to the database. Therefore, it
+cannot be retrieved in the event of a restart. In order to make this class restartable,
+the `ItemStream` interface should be implemented along with the methods `open` and
+`update`, as shown in the following example:
+
+[source, java]
+----
+public void open(ExecutionContext executionContext) {
+    if (executionContext.containsKey("total.amount") {
+        totalAmount = (BigDecimal) executionContext.get("total.amount");
+    }
+}
+
+public void update(ExecutionContext executionContext) {
+    executionContext.put("total.amount", totalAmount);
+}
+----
+
+The update method stores the most current version of `totalAmount` to the
+`ExecutionContext` just before that object is persisted to the database. The open method
+retrieves any existing `totalAmount` from the `ExecutionContext` and uses it as the
+starting point for processing, allowing the `TradeItemWriter` to pick up on restart where
+it left off the previous time the `Step` was run.
+
+[[drivingQueryBasedItemReaders]]
+=== Driving Query Based ItemReaders
+
+In the link:readersAndWriters.html[chapter on readers and writers], database input using
+paging was discussed. Many database vendors, such as DB2, have extremely pessimistic
+locking strategies that can cause issues if the table being read also needs to be used by
+other portions of the online application. Furthermore, opening cursors over extremely
+large datasets can cause issues on databases from certain vendors. Therefore, many
+projects prefer to use a 'Driving Query' approach to reading in data. This approach works
+by iterating over keys, rather than the entire object that needs to be returned, as the
+following image illustrates:
+
+.Driving Query Job
+image::{batch-asciidoc}images/drivingQueryExample.png[Driving Query Job, scaledwidth="60%"]
+
+As you can see, the example shown in the preceding image uses the same 'FOO' table as was
+used in the cursor-based example. However, rather than selecting the entire row, only the
+IDs were selected in the SQL statement. So, rather than a `FOO` object being returned
+from `read`, an `Integer` is returned. This number can then be used to query for the
+'details', which is a complete `Foo` object, as shown in the following image:
+
+.Driving Query Example
+image::{batch-asciidoc}images/drivingQueryJob.png[Driving Query Example, scaledwidth="60%"]
+
+An `ItemProcessor` should be used to transform the key obtained from the driving query
+into a full `Foo` object. An existing DAO can be used to query for the full object based
+on the key.
+
+[[multiLineRecords]]
+=== Multi-Line Records
+
+While it is usually the case with flat files that each record is confined to a single
+line, it is common that a file might have records spanning multiple lines with multiple
+formats. The following excerpt from a file shows an example of such an arrangement:
+
+----
+HEA;0013100345;2007-02-15
+NCU;Smith;Peter;;T;20014539;F
+BAD;;Oak Street 31/A;;Small Town;00235;IL;US
+FOT;2;2;267.34
+----
+Everything between the line starting with 'HEA' and the line starting with 'FOT' is
+considered one record. There are a few considerations that must be made in order to
+handle this situation correctly:
+
+* Instead of reading one record at a time, the `ItemReader` must read every line of the
+multi-line record as a group, so that it can be passed to the `ItemWriter` intact.
+* Each line type may need to be tokenized differently.
+
+Because a single record spans multiple lines and because we may not know how many lines
+there are, the `ItemReader` must be careful to always read an entire record. In order to
+do this, a custom `ItemReader` should be implemented as a wrapper for the
+`FlatFileItemReader`.
+
+[role="xmlContent"]
+The following example shows how to implement a custom `ItemReader` in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<bean id="itemReader" class="org.spr...MultiLineTradeItemReader">
+    <property name="delegate">
+        <bean class="org.springframework.batch.item.file.FlatFileItemReader">
+            <property name="resource" value="data/iosample/input/multiLine.txt" />
+            <property name="lineMapper">
+                <bean class="org.spr...DefaultLineMapper">
+                    <property name="lineTokenizer" ref="orderFileTokenizer"/>
+                    <property name="fieldSetMapper" ref="orderFieldSetMapper"/>
+                </bean>
+            </property>
+        </bean>
+    </property>
+</bean>
+----
+
+[role="javaContent"]
+The following example shows how to implement a custom `ItemReader` in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public MultiLineTradeItemReader itemReader() {
+	MultiLineTradeItemReader itemReader = new MultiLineTradeItemReader();
+
+	itemReader.setDelegate(flatFileItemReader());
+
+	return itemReader;
+}
+
+@Bean
+public FlatFileItemReader flatFileItemReader() {
+	FlatFileItemReader<Trade> reader = new FlatFileItemReaderBuilder<>()
+			.name("flatFileItemReader")
+			.resource(new ClassPathResource("data/iosample/input/multiLine.txt"))
+			.lineTokenizer(orderFileTokenizer())
+			.fieldSetMapper(orderFieldSetMapper())
+			.build();
+	return reader;
+}
+----
+
+To ensure that each line is tokenized properly, which is especially important for
+fixed-length input, the `PatternMatchingCompositeLineTokenizer` can be used on the
+delegate `FlatFileItemReader`. See
+link:readersAndWriters.html#flatFileItemReader[`FlatFileItemReader` in the Readers and
+Writers chapter] for more details. The delegate reader then uses a
+`PassThroughFieldSetMapper` to deliver a `FieldSet` for each line back to the wrapping
+`ItemReader`.
+
+[role="xmlContent"]
+The following example shows how to ensure that each line is properly tokenized in XML:
+
+.XML Content
+[source, xml, role="xmlContent"]
+----
+<bean id="orderFileTokenizer" class="org.spr...PatternMatchingCompositeLineTokenizer">
+    <property name="tokenizers">
+        <map>
+            <entry key="HEA*" value-ref="headerRecordTokenizer" />
+            <entry key="FOT*" value-ref="footerRecordTokenizer" />
+            <entry key="NCU*" value-ref="customerLineTokenizer" />
+            <entry key="BAD*" value-ref="billingAddressLineTokenizer" />
+        </map>
+    </property>
+</bean>
+----
+
+[role="javaContent"]
+The following example shows how to ensure that each line is properly tokenized in Java:
+
+.Java Content
+[source, java, role="javaContent"]
+----
+@Bean
+public PatternMatchingCompositeLineTokenizer orderFileTokenizer() {
+	PatternMatchingCompositeLineTokenizer tokenizer =
+			new PatternMatchingCompositeLineTokenizer();
+
+	Map<String, LineTokenizer> tokenizers = new HashMap<>(4);
+
+	tokenizers.put("HEA*", headerRecordTokenizer());
+	tokenizers.put("FOT*", footerRecordTokenizer());
+	tokenizers.put("NCU*", customerLineTokenizer());
+	tokenizers.put("BAD*", billingAddressLineTokenizer());
+
+	tokenizer.setTokenizers(tokenizers);
+
+	return tokenizer;
+}
+----
+
+This wrapper has to be able to recognize the end of a record so that it can continually
+call `read()` on its delegate until the end is reached. For each line that is read, the
+wrapper should build up the item to be returned. Once the footer is reached, the item can
+be returned for delivery to the `ItemProcessor` and `ItemWriter`, as  shown in the
+following example:
+
+[source, java]
+----
+private FlatFileItemReader<FieldSet> delegate;
+
+public Trade read() throws Exception {
+    Trade t = null;
+
+    for (FieldSet line = null; (line = this.delegate.read()) != null;) {
+        String prefix = line.readString(0);
+        if (prefix.equals("HEA")) {
+            t = new Trade(); // Record must start with header
+        }
+        else if (prefix.equals("NCU")) {
+            Assert.notNull(t, "No header was found.");
+            t.setLast(line.readString(1));
+            t.setFirst(line.readString(2));
+            ...
+        }
+        else if (prefix.equals("BAD")) {
+            Assert.notNull(t, "No header was found.");
+            t.setCity(line.readString(4));
+            t.setState(line.readString(6));
+          ...
+        }
+        else if (prefix.equals("FOT")) {
+            return t; // Record must end with footer
+        }
+    }
+    Assert.isNull(t, "No 'END' was found.");
+    return null;
+}
+----
+
+[[executingSystemCommands]]
+=== Executing System Commands
+
+Many batch jobs require that an external command be called from within the batch job.
+Such a process could be kicked off separately by the scheduler, but the advantage of
+common metadata about the run would be lost. Furthermore, a multi-step job would also
+need to be split up into multiple jobs as well.
+
+Because the need is so common, Spring Batch provides a `Tasklet` implementation for
+calling system commands.
+
+[role="xmlContent"]
+The following example shows how to call an external command in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<bean class="org.springframework.batch.core.step.tasklet.SystemCommandTasklet">
+    <property name="command" value="echo hello" />
+    <!-- 5 second timeout for the command to complete -->
+    <property name="timeout" value="5000" />
+</bean>
+----
+
+[role="javaContent"]
+The following example shows how to call an external command in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public SystemCommandTasklet tasklet() {
+	SystemCommandTasklet tasklet = new SystemCommandTasklet();
+
+	tasklet.setCommand("echo hello");
+	tasklet.setTimeout(5000);
+
+	return tasklet;
+}
+----
+
+[[handlingStepCompletionWhenNoInputIsFound]]
+=== Handling Step Completion When No Input is Found
+
+In many batch scenarios, finding no rows in a database or file to process is not
+exceptional. The `Step` is simply considered to have found no work and completes with 0
+items read. All of the `ItemReader` implementations provided out of the box in Spring
+Batch default to this approach. This can lead to some confusion if nothing is written out
+even when input is present (which usually happens if a file was misnamed or some similar
+issue arises). For this reason, the metadata itself should be inspected to determine how
+much work the framework found to be processed. However, what if finding no input is
+considered exceptional? In this case, programmatically checking the metadata for no items
+processed and causing failure is the best solution. Because this is a common use case,
+Spring Batch provides a listener with exactly this functionality, as shown in
+the class definition for `NoWorkFoundStepExecutionListener`:
+
+[source, java]
+----
+public class NoWorkFoundStepExecutionListener extends StepExecutionListenerSupport {
+
+    public ExitStatus afterStep(StepExecution stepExecution) {
+        if (stepExecution.getReadCount() == 0) {
+            return ExitStatus.FAILED;
+        }
+        return null;
+    }
+
+}
+----
+
+The preceding `StepExecutionListener` inspects the `readCount` property of the
+`StepExecution` during the 'afterStep' phase to determine if no items were read. If that
+is the case, an exit code `FAILED` is returned, indicating that the `Step` should fail.
+Otherwise, `null` is returned, which does not affect the status of the `Step`.
+
+[[passingDataToFutureSteps]]
+=== Passing Data to Future Steps
+
+It is often useful to pass information from one step to another. This can be done through
+the `ExecutionContext`. The catch is that there are two `ExecutionContexts`: one at the
+`Step` level and one at the `Job` level. The `Step` `ExecutionContext` remains only as
+long as the step, while the `Job` `ExecutionContext` remains through the whole `Job`. On
+the other hand, the `Step` `ExecutionContext` is updated every time the `Step` commits a
+chunk, while the `Job` `ExecutionContext` is updated only at the end of each `Step`.
+
+The consequence of this separation is that all data must be placed in the `Step`
+`ExecutionContext` while the `Step` is executing. Doing so ensures that the data is
+stored properly while the `Step` runs. If data is stored to the `Job` `ExecutionContext`,
+then it is not persisted during `Step` execution. If the `Step` fails, that data is lost.
+
+[source, java]
+----
+public class SavingItemWriter implements ItemWriter<Object> {
+    private StepExecution stepExecution;
+
+    public void write(List<? extends Object> items) throws Exception {
+        // ...
+
+        ExecutionContext stepContext = this.stepExecution.getExecutionContext();
+        stepContext.put("someKey", someObject);
+    }
+
+    @BeforeStep
+    public void saveStepExecution(StepExecution stepExecution) {
+        this.stepExecution = stepExecution;
+    }
+}
+----
+
+To make the data available to future `Steps`, it must be "`promoted`" to the `Job`
+`ExecutionContext` after the step has finished. Spring Batch provides the
+`ExecutionContextPromotionListener` for this purpose. The listener must be configured
+with the keys related to the data in the `ExecutionContext` that must be promoted. It can
+also, optionally, be configured with a list of exit code patterns for which the promotion
+should occur (`COMPLETED` is the default). As with all listeners, it must be registered
+on the `Step`.
+
+[role="xmlContent"]
+The following example shows how to promote a step to the `Job` `ExecutionContext` in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<job id="job1">
+    <step id="step1">
+        <tasklet>
+            <chunk reader="reader" writer="savingWriter" commit-interval="10"/>
+        </tasklet>
+        <listeners>
+            <listener ref="promotionListener"/>
+        </listeners>
+    </step>
+
+    <step id="step2">
+       ...
+    </step>
+</job>
+
+<beans:bean id="promotionListener" class="org.spr....ExecutionContextPromotionListener">
+    <beans:property name="keys">
+        <list>
+            <value>someKey</value>
+        </list>
+    </beans:property>
+</beans:bean>
+----
+
+[role="xmlContent"]
+The following example shows how to promote a step to the `Job` `ExecutionContext` in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Job job1() {
+	return this.jobBuilderFactory.get("job1")
+				.start(step1())
+				.next(step1())
+				.build();
+}
+
+@Bean
+public Step step1() {
+	return this.stepBuilderFactory.get("step1")
+				.<String, String>chunk(10)
+				.reader(reader())
+				.writer(savingWriter())
+				.listener(promotionListener())
+				.build();
+}
+
+@Bean
+public ExecutionContextPromotionListener promotionListener() {
+	ExecutionContextPromotionListener listener = new ExecutionContextPromotionListener();
+
+	listener.setKeys(new String[] {"someKey"});
+
+	return listener;
+}
+----
+
+Finally, the saved values must be retrieved from the `Job` `ExecutionContext`, as shown
+in the following example:
+
+[source, java]
+----
+public class RetrievingItemWriter implements ItemWriter<Object> {
+    private Object someObject;
+
+    public void write(List<? extends Object> items) throws Exception {
+        // ...
+    }
+
+    @BeforeStep
+    public void retrieveInterstepData(StepExecution stepExecution) {
+        JobExecution jobExecution = stepExecution.getJobExecution();
+        ExecutionContext jobContext = jobExecution.getExecutionContext();
+        this.someObject = jobContext.get("someKey");
+    }
+}
+----

spring-batch-docs/src/main/asciidoc/domain.adoc

@@ -0,0 +1,659 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[domainLanguageOfBatch]]
+
+== The Domain Language of Batch
+
+ifndef::onlyonetoggle[]
+include::toggle.adoc[]
+endif::onlyonetoggle[]
+
+To any experienced batch architect, the overall concepts of batch processing used in
+Spring Batch should be familiar and comfortable. There are "Jobs" and "Steps" and
+developer-supplied processing units called `ItemReader` and `ItemWriter`. However,
+because of the Spring patterns, operations, templates, callbacks, and idioms, there are
+opportunities for the following:
+
+* Significant improvement in adherence to a clear separation of concerns.
+* Clearly delineated architectural layers and services provided as interfaces.
+* Simple and default implementations that allow for quick adoption and ease of use
+out-of-the-box.
+* Significantly enhanced extensibility.
+
+The following diagram is a simplified version of the batch reference architecture that
+has been used for decades. It provides an overview of the components that make up the
+domain language of batch processing. This architecture framework is a blueprint that has
+been proven through decades of implementations on the last several generations of
+platforms (COBOL/Mainframe, C++/Unix, and now Java/anywhere). JCL and COBOL developers
+are likely to be as comfortable with the concepts as C++, C#, and Java developers. Spring
+Batch provides a physical implementation of the layers, components, and technical
+services commonly found in the robust, maintainable systems that are used to address the
+creation of simple to complex batch applications, with the infrastructure and extensions
+to address very complex processing needs.
+
+.Batch Stereotypes
+image::{batch-asciidoc}images/spring-batch-reference-model.png[Figure 2.1: Batch Stereotypes, scaledwidth="60%"]
+
+The preceding diagram highlights the key concepts that make up the domain language of
+Spring Batch. A Job has one to many steps, each of which has exactly one `ItemReader`,
+one `ItemProcessor`, and one `ItemWriter`. A job needs to be launched (with
+`JobLauncher`), and metadata about the currently running process needs to be stored (in
+`JobRepository`).
+
+=== Job
+
+This section describes stereotypes relating to the concept of a batch job. A `Job` is an
+entity that encapsulates an entire batch process. As is common with other Spring
+projects, a `Job` is wired together with either an XML configuration file or Java-based
+configuration. This configuration may be referred to as the "job configuration". However,
+`Job` is just the top of an overall hierarchy, as shown in the following diagram:
+
+.Job Hierarchy
+image::{batch-asciidoc}images/job-heirarchy.png[Job Hierarchy, scaledwidth="60%"]
+
+In Spring Batch, a `Job` is simply a container for `Step` instances. It combines multiple
+steps that belong logically together in a flow and allows for configuration of properties
+global to all steps, such as restartability. The job configuration contains:
+
+* The simple name of the job.
+* Definition and ordering of `Step` instances.
+* Whether or not the job is restartable.
+
+ifdef::backend-html5[]
+[role="javaContent"]
+For those who use Java configuration, Spring Batch provices a default implementation of
+the Job interface in the form of the `SimpleJob` class, which creates some standard
+functionality on top of `Job`. When using java based configuration, a collection of
+builders is made available for the instantiation of a `Job`, as shown in the following
+example:
+
+[source, java, role="javaContent"]
+----
+@Bean
+public Job footballJob() {
+    return this.jobBuilderFactory.get("footballJob")
+                     .start(playerLoad())
+                     .next(gameLoad())
+                     .next(playerSummarization())
+                     .end()
+                     .build();
+}
+----
+
+[role="xmlContent"]
+For those who use XML configuration, Spring Batch provides a default implementation of the
+`Job` interface in the form of the `SimpleJob` class, which creates some standard
+functionality on top of `Job`. However, the batch namespace abstracts away the need to
+instantiate it directly. Instead, the `<job>` element can be used, as shown in the
+following example:
+
+[source, xml, role="xmlContent"]
+----
+<job id="footballJob">
+    <step id="playerload" next="gameLoad"/>
+    <step id="gameLoad" next="playerSummarization"/>
+    <step id="playerSummarization"/>
+</job>
+----
+endif::backend-html5[]
+
+ifdef::backend-pdf[]
+Spring Batch provides a default implementation of the Job interface in the form of the
+`SimpleJob` class, which creates some standard functionality on top of `Job`. When using
+Java-based configuration, a collection of builders are made available for the
+instantiation of a `Job`, as shown in the following example:
+
+[source, java]
+----
+@Bean
+public Job footballJob() {
+    return this.jobBuilderFactory.get("footballJob")
+                     .start(playerLoad())
+                     .next(gameLoad())
+                     .next(playerSummarization())
+                     .end()
+                     .build();
+}
+----
+
+However, when using XML configuration, the batch namespace abstracts away the need to
+instantiate it directly. Instead, the `<job>` tag can be used as shown in the following
+example:
+
+[source, xml]
+----
+<job id="footballJob">
+    <step id="playerload" next="gameLoad"/>
+    <step id="gameLoad" next="playerSummarization"/>
+    <step id="playerSummarization"/>
+</job>
+----
+endif::backend-pdf[]
+
+==== JobInstance
+
+A `JobInstance` refers to the concept of a logical job run. Consider a batch job that
+should be run once at the end of the day, such as the 'EndOfDay' `Job` from the preceding
+diagram. There is one 'EndOfDay' job, but each individual run of the `Job` must be
+tracked separately. In the case of this job, there is one logical `JobInstance` per day.
+For example, there is a January 1st run, a January 2nd run, and so on. If the January 1st
+run fails the first time and is run again the next day, it is still the January 1st run.
+(Usually, this corresponds with the data it is processing as well, meaning the January
+1st run processes data for January 1st). Therefore, each `JobInstance` can have multiple
+executions (`JobExecution` is discussed in more detail later in this chapter), and only
+one `JobInstance` corresponding to a particular `Job` and identifying `JobParameters` can
+run at a given time.
+
+The definition of a `JobInstance` has absolutely no bearing on the data to be loaded.
+It is entirely up to the `ItemReader` implementation to determine how data is loaded. For
+example, in the EndOfDay scenario, there may be a column on the data that indicates the
+'effective date' or 'schedule date' to which the data belongs. So, the January 1st run
+would load only data from the 1st, and the January 2nd run would use only data from the
+2nd. Because this determination is likely to be a business decision, it is left up to the
+`ItemReader` to decide. However, using the same `JobInstance` determines whether or not
+the 'state' (that is, the `ExecutionContext`, which is discussed later in this chapter)
+from previous executions is used. Using a new `JobInstance` means 'start from the
+beginning', and using an existing instance generally means 'start from where you left
+off'.
+
+==== JobParameters
+
+Having discussed `JobInstance` and how it differs from Job, the natural question to ask
+is: "How is one `JobInstance` distinguished from another?" The answer is:
+`JobParameters`. A `JobParameters` object holds a set of parameters used to start a batch
+job. They can be used for identification or even as reference data during the run, as
+shown in the following image:
+
+.Job Parameters
+image::{batch-asciidoc}images/job-stereotypes-parameters.png[Job Parameters, scaledwidth="60%"]
+
+In the preceding example, where there are two instances, one for January 1st, and another
+for January 2nd, there is really only one `Job`, but it has two `JobParameter` objects:
+one that was started with a job parameter of 01-01-2017 and another that was started with
+a parameter of 01-02-2017. Thus, the contract can be defined as: `JobInstance` = `Job`
+ + identifying `JobParameters`. This allows a developer to effectively control how a
+`JobInstance` is defined, since they control what parameters are passed in.
+
+NOTE: Not all job parameters are required to contribute to the identification of a
+`JobInstance`.  By default, they do so. However, the framework also allows the submission
+of a `Job` with parameters that do not contribute to the identity of a `JobInstance`.
+
+==== JobExecution
+
+A `JobExecution` refers to the technical concept of a single attempt to run a Job. An
+execution may end in failure or success, but the `JobInstance` corresponding to a given
+execution is not considered to be complete unless the execution completes successfully.
+Using the EndOfDay `Job` described previously as an example, consider a `JobInstance` for
+01-01-2017 that failed the first time it was run. If it is run again with the same
+identifying job parameters as the first run (01-01-2017), a new `JobExecution` is
+created. However, there is still only one `JobInstance`.
+
+A `Job` defines what a job is and how it is to be executed, and a `JobInstance` is a
+purely organizational object to group executions together, primarily to enable correct
+restart semantics. A `JobExecution`, however, is the primary storage mechanism for what
+actually happened during a run and contains many more properties that must be controlled
+and persisted, as shown in the following table:
+
+.JobExecution Properties
+
+|===
+|Property |Definition
+|Status
+|A `BatchStatus` object that indicates the status of the execution. While running, it is
+`BatchStatus#STARTED`. If it fails, it is `BatchStatus#FAILED`. If it finishes
+successfully, it is `BatchStatus#COMPLETED`
+
+|startTime
+|A `java.util.Date` representing the current system time when the execution was started.
+This field is empty if the job has yet to start.
+
+|endTime
+|A `java.util.Date` representing the current system time when the execution finished,
+regardless of whether or not it was successful. The field is empty if the job has yet to
+finish.
+
+|exitStatus
+|The `ExitStatus`, indicating the result of the run. It is most important, because it
+contains an exit code that is returned to the caller. See chapter 5 for more details. The
+field is empty if the job has yet to finish.
+
+|createTime
+|A `java.util.Date` representing the current system time when the `JobExecution` was
+first persisted. The job may not have been started yet (and thus has no start time), but
+it always has a createTime, which is required by the framework for managing job level
+`ExecutionContexts`.
+
+|lastUpdated
+|A `java.util.Date` representing the last time a `JobExecution` was persisted. This field
+is empty if the job has yet to start.
+
+|executionContext
+|The "property bag" containing any user data that needs to be persisted between
+executions.
+
+|failureExceptions
+|The list of exceptions encountered during the execution of a `Job`. These can be useful
+if more than one exception is encountered during the failure of a `Job`.
+|===
+
+These properties are important because they are persisted and can be used to completely
+determine the status of an execution. For example, if the EndOfDay job for 01-01 is
+executed at 9:00 PM and fails at 9:30, the following entries are made in the batch
+metadata tables:
+
+.BATCH_JOB_INSTANCE
+
+|===
+|JOB_INST_ID |JOB_NAME
+|1
+|EndOfDayJob
+|===
+
+.BATCH_JOB_EXECUTION_PARAMS
+|===
+|JOB_EXECUTION_ID|TYPE_CD|KEY_NAME|DATE_VAL|IDENTIFYING
+|1
+|DATE
+|schedule.Date
+|2017-01-01
+|TRUE
+|===
+
+.BATCH_JOB_EXECUTION
+|===
+|JOB_EXEC_ID|JOB_INST_ID|START_TIME|END_TIME|STATUS
+|1
+|1
+|2017-01-01 21:00
+|2017-01-01 21:30
+|FAILED
+|===
+
+NOTE: Column names may have been abbreviated or removed for the sake of clarity and
+formatting.
+
+Now that the job has failed, assume that it took the entire night for the problem to be
+determined, so that the 'batch window' is now closed. Further assuming that the window
+starts at 9:00 PM, the job is kicked off again for 01-01, starting where it left off and
+completing successfully at 9:30. Because it is now the next day, the 01-02 job must be
+run as well, and it is kicked off just afterwards at 9:31 and completes in its normal one
+hour time at 10:30. There is no requirement that one `JobInstance` be kicked off after
+another, unless there is potential for the two jobs to attempt to access the same data,
+causing issues with locking at the database level. It is entirely up to the scheduler to
+determine when a `Job` should be run. Since they are separate `JobInstances`, Spring
+Batch makes no attempt to stop them from being run concurrently. (Attempting to run the
+same `JobInstance` while another is already running results in a
+`JobExecutionAlreadyRunningException` being thrown). There should now be an extra entry
+in both the `JobInstance` and `JobParameters` tables and two extra entries in the
+`JobExecution` table, as shown in the following tables:
+
+.BATCH_JOB_INSTANCE
+|===
+|JOB_INST_ID |JOB_NAME
+|1
+|EndOfDayJob
+
+|2
+|EndOfDayJob
+|===
+
+.BATCH_JOB_EXECUTION_PARAMS
+|===
+|JOB_EXECUTION_ID|TYPE_CD|KEY_NAME|DATE_VAL|IDENTIFYING
+|1
+|DATE
+|schedule.Date
+|2017-01-01 00:00:00
+|TRUE
+
+|2
+|DATE
+|schedule.Date
+|2017-01-01 00:00:00
+|TRUE
+
+|3
+|DATE
+|schedule.Date
+|2017-01-02 00:00:00
+|TRUE
+|===
+
+.BATCH_JOB_EXECUTION
+|===
+|JOB_EXEC_ID|JOB_INST_ID|START_TIME|END_TIME|STATUS
+|1
+|1
+|2017-01-01 21:00
+|2017-01-01 21:30
+|FAILED
+
+|2
+|1
+|2017-01-02 21:00
+|2017-01-02 21:30
+|COMPLETED
+
+|3
+|2
+|2017-01-02 21:31
+|2017-01-02 22:29
+|COMPLETED
+|===
+
+NOTE: Column names may have been abbreviated or removed for the sake of clarity and
+formatting.
+
+=== Step
+
+A `Step` is a domain object that encapsulates an independent, sequential phase of a batch
+job. Therefore, every Job is composed entirely of one or more steps. A `Step` contains
+all of the information necessary to define and control the actual batch processing. This
+is a necessarily vague description because the contents of any given `Step` are at the
+discretion of the developer writing a `Job`. A `Step` can be as simple or complex as the
+developer desires. A simple `Step` might load data from a file into the database,
+requiring little or no code (depending upon the implementations used). A more complex
+`Step` may have complicated business rules that are applied as part of the processing. As
+with a `Job`, a `Step` has an individual `StepExecution` that correlates with a unique
+`JobExecution`, as shown in the following image:
+
+.Job Hierarchy With Steps
+image::{batch-asciidoc}images/jobHeirarchyWithSteps.png[Figure 2.1: Job Hierarchy With Steps, scaledwidth="60%"]
+
+==== StepExecution
+
+A `StepExecution` represents a single attempt to execute a `Step`. A new `StepExecution`
+is created each time a `Step` is run, similar to `JobExecution`. However, if a step fails
+to execute because the step before it fails, no execution is persisted for it. A
+`StepExecution` is created only when its `Step` is actually started.
+
+`Step` executions are represented by objects of the `StepExecution` class. Each execution
+contains a reference to its corresponding step and `JobExecution` and transaction related
+data, such as commit and rollback counts and start and end times. Additionally, each step
+execution contains an `ExecutionContext`, which contains any data a developer needs to
+have persisted across batch runs, such as statistics or state information needed to
+restart. The following table lists the properties for `StepExecution`:
+
+.StepExecution Properties
+|===
+|Property|Definition
+|Status
+|A `BatchStatus` object that indicates the status of the execution. While running, the
+status is `BatchStatus.STARTED`. If it fails, the status is `BatchStatus.FAILED`. If it
+finishes successfully, the status is `BatchStatus.COMPLETED`.
+
+|startTime
+|A `java.util.Date` representing the current system time when the execution was started.
+This field is empty if the step has yet to start.
+
+|endTime
+
+|A `java.util.Date` representing the current system time when the execution finished,
+regardless of whether or not it was successful. This field is empty if the step has yet to
+exit.
+
+|exitStatus
+|The `ExitStatus` indicating the result of the execution. It is most important, because
+it contains an exit code that is returned to the caller. See chapter 5 for more details.
+This field is empty if the job has yet to exit.
+
+|executionContext
+|The "property bag" containing any user data that needs to be persisted between
+executions.
+
+|readCount
+|The number of items that have been successfully read.
+
+|writeCount
+|The number of items that have been successfully written.
+
+|commitCount
+|The number of transactions that have been committed for this execution.
+
+|rollbackCount
+|The number of times the business transaction controlled by the `Step` has been rolled
+back.
+
+|readSkipCount
+|The number of times `read` has failed, resulting in a skipped item.
+
+|processSkipCount
+|The number of times `process` has failed, resulting in a skipped item.
+
+|filterCount
+|The number of items that have been 'filtered' by the `ItemProcessor`.
+
+|writeSkipCount
+|The number of times `write` has failed, resulting in a skipped item.
+|===
+
+=== ExecutionContext
+
+An `ExecutionContext` represents a collection of key/value pairs that are persisted and
+controlled by the framework in order to allow developers a place to store persistent
+state that is scoped to a `StepExecution` object or a `JobExecution` object. For those
+familiar with Quartz, it is very similar to JobDataMap. The best usage example is to
+facilitate restart. Using flat file input as an example, while processing individual
+lines, the framework periodically persists the `ExecutionContext` at commit points. Doing
+so allows the `ItemReader` to store its state in case a fatal error occurs during the run
+or even if the power goes out. All that is needed is to put the current number of lines
+read into the context, as shown in the following example, and the framework will do the
+rest:
+
+[source, java]
+----
+executionContext.putLong(getKey(LINES_READ_COUNT), reader.getPosition());
+----
+
+Using the EndOfDay example from the `Job` Stereotypes section as an example, assume there
+is one step, 'loadData', that loads a file into the database. After the first failed run,
+the metadata tables would look like the following example:
+
+.BATCH_JOB_INSTANCE
+|===
+|JOB_INST_ID|JOB_NAME
+|1
+|EndOfDayJob
+|===
+
+.BATCH_JOB_EXECUTION_PARAMS
+|===
+|JOB_INST_ID|TYPE_CD|KEY_NAME|DATE_VAL
+|1
+|DATE
+|schedule.Date
+|2017-01-01
+|===
+
+.BATCH_JOB_EXECUTION
+|===
+|JOB_EXEC_ID|JOB_INST_ID|START_TIME|END_TIME|STATUS
+|1
+|1
+|2017-01-01 21:00
+|2017-01-01 21:30
+|FAILED
+|===
+
+.BATCH_STEP_EXECUTION
+|===
+|STEP_EXEC_ID|JOB_EXEC_ID|STEP_NAME|START_TIME|END_TIME|STATUS
+|1
+|1
+|loadData
+|2017-01-01 21:00
+|2017-01-01 21:30
+|FAILED
+|===
+
+.BATCH_STEP_EXECUTION_CONTEXT
+|===
+|STEP_EXEC_ID|SHORT_CONTEXT
+|1
+|{piece.count=40321}
+|===
+
+In the preceding case, the `Step` ran for 30 minutes and processed 40,321 'pieces', which
+would represent lines in a file in this scenario. This value is updated just before each
+commit by the framework and can contain multiple rows corresponding to entries within the
+`ExecutionContext`. Being notified before a commit requires one of the various
+`StepListener` implementations (or an `ItemStream`), which are discussed in more detail
+later in this guide. As with the previous example, it is assumed that the `Job` is
+restarted the next day. When it is restarted, the values from the `ExecutionContext` of
+the last run are reconstituted from the database. When the `ItemReader` is opened, it can
+check to see if it has any stored state in the context and initialize itself from there,
+as shown in the following example:
+
+[source, java]
+----
+if (executionContext.containsKey(getKey(LINES_READ_COUNT))) {
+    log.debug("Initializing for restart. Restart data is: " + executionContext);
+
+    long lineCount = executionContext.getLong(getKey(LINES_READ_COUNT));
+
+    LineReader reader = getReader();
+
+    Object record = "";
+    while (reader.getPosition() < lineCount && record != null) {
+        record = readLine();
+    }
+}
+----
+In this case, after the above code runs, the current line is 40,322, allowing the `Step`
+to start again from where it left off. The `ExecutionContext` can also be used for
+statistics that need to be persisted about the run itself. For example, if a flat file
+contains orders for processing that exist across multiple lines, it may be necessary to
+store how many orders have been processed (which is much different from the number of
+lines read), so that an email can be sent at the end of the `Step` with the total number
+of orders processed in the body. The framework handles storing this for the developer, in
+order to correctly scope it with an individual `JobInstance`. It can be very difficult to
+know whether an existing `ExecutionContext` should be used or not. For example, using the
+'EndOfDay' example from above, when the 01-01 run starts again for the second time, the
+framework recognizes that it is the same `JobInstance` and on an individual `Step` basis,
+pulls the `ExecutionContext` out of the database, and hands it (as part of the
+`StepExecution`) to the `Step` itself. Conversely, for the 01-02 run, the framework
+recognizes that it is a different instance, so an empty context must be handed to the
+`Step`. There are many of these types of determinations that the framework makes for the
+developer, to ensure the state is given to them at the correct time. It is also important
+to note that exactly one `ExecutionContext` exists per `StepExecution` at any given time.
+Clients of the `ExecutionContext` should be careful, because this creates a shared
+keyspace. As a result, care should be taken when putting values in to ensure no data is
+overwritten. However, the `Step` stores absolutely no data in the context, so there is no
+way to adversely affect the framework.
+
+It is also important to note that there is at least one `ExecutionContext` per
+`JobExecution` and one for every `StepExecution`. For example, consider the following
+code snippet:
+
+[source, java]
+----
+ExecutionContext ecStep = stepExecution.getExecutionContext();
+ExecutionContext ecJob = jobExecution.getExecutionContext();
+//ecStep does not equal ecJob
+----
+
+As noted in the comment, `ecStep` does not equal `ecJob`. They are two different
+`ExecutionContexts`. The one scoped to the `Step` is saved at every commit point in the
+`Step`, whereas the one scoped to the Job is saved in between every `Step` execution.
+
+=== JobRepository
+
+`JobRepository` is the persistence mechanism for all of the Stereotypes mentioned above.
+It provides CRUD operations for `JobLauncher`, `Job`, and `Step` implementations. When a
+`Job` is first launched, a `JobExecution` is obtained from the repository, and, during
+the course of execution, `StepExecution` and `JobExecution` implementations are persisted
+by passing them to the repository.
+
+[role="xmlContent"]
+The Spring Batch XML namespace provides support for configuring a `JobRepository` instance
+with the `<job-repository>` tag, as shown in the following example:
+
+[source, xml, role="xmlContent"]
+----
+<job-repository id="jobRepository"/>
+----
+
+[role="javaContent"]
+When using Java configuration, the `@EnableBatchProcessing` annotation provides a
+`JobRepository` as one of the components automatically configured out of the box.
+
+=== JobLauncher
+
+`JobLauncher` represents a simple interface for launching a `Job` with a given set of
+`JobParameters`, as shown in the following example:
+
+[source, java]
+----
+public interface JobLauncher {
+
+public JobExecution run(Job job, JobParameters jobParameters)
+            throws JobExecutionAlreadyRunningException, JobRestartException,
+                   JobInstanceAlreadyCompleteException, JobParametersInvalidException;
+}
+----
+It is expected that implementations obtain a valid `JobExecution` from the
+`JobRepository` and execute the `Job`.
+
+=== Item Reader
+
+`ItemReader` is an abstraction that represents the retrieval of input for a `Step`, one
+item at a time. When the `ItemReader` has exhausted the items it can provide, it
+indicates this by returning `null`. More details about the `ItemReader` interface and its
+various implementations can be found in
+<<readersAndWriters.adoc#readersAndWriters,Readers And Writers>>.
+
+=== Item Writer
+
+`ItemWriter` is an abstraction that represents the output of a `Step`, one batch or chunk
+of items at a time.  Generally, an `ItemWriter` has no knowledge of the input it should
+receive next and knows only the item that was passed in its current invocation. More
+details about the `ItemWriter` interface and its various implementations can be found in
+<<readersAndWriters.adoc#readersAndWriters,Readers And Writers>>.
+
+=== Item Processor
+
+`ItemProcessor` is an abstraction that represents the business processing of an item.
+While the `ItemReader` reads one item, and the `ItemWriter` writes them, the
+`ItemProcessor` provides an access point to transform or apply other business processing.
+If, while processing the item, it is determined that the item is not valid, returning
+`null` indicates that the item should not be written out. More details about the
+`ItemProcessor` interface can be found in
+<<readersAndWriters.adoc#readersAndWriters,Readers And Writers>>.
+
+[role="xmlContent"]
+=== Batch Namespace
+
+Many of the domain concepts listed previously need to be configured in a Spring
+`ApplicationContext`. While there are implementations of the interfaces above that can be
+used in a standard bean definition, a namespace has been provided for ease of
+configuration, as shown in the following example:
+
+[source, xml, role="xmlContent"]
+----
+<beans:beans xmlns="http://www.springframework.org/schema/batch"
+xmlns:beans="http://www.springframework.org/schema/beans"
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+xsi:schemaLocation="
+   http://www.springframework.org/schema/beans
+   https://www.springframework.org/schema/beans/spring-beans.xsd
+   http://www.springframework.org/schema/batch
+   https://www.springframework.org/schema/batch/spring-batch.xsd">
+
+<job id="ioSampleJob">
+    <step id="step1">
+        <tasklet>
+            <chunk reader="itemReader" writer="itemWriter" commit-interval="2"/>
+        </tasklet>
+    </step>
+</job>
+
+</beans:beans>
+----
+
+[role="xmlContent"]
+As long as the batch namespace has been declared, any of its elements can be used. More
+information on configuring a Job can be found in <<job.adoc#configureJob,Configuring and
+Running a Job>>. More information on configuring a `Step` can be found in
+<<step.adoc#configureStep,Configuring a Step>>.

spring-batch-docs/src/main/asciidoc/footer/index-footer.adoc

@@ -0,0 +1,11 @@
+'''
+Lucas Ward, Dave Syer, Thomas Risberg, Robert Kasanicky, Dan Garrette, Wayne Lund,
+Michael Minella, Chris Schaefer, Gunnar Hillert, Glenn Renfro, Jay Bryant, Mahmoud Ben Hassine
+
+Copyright © 2009 - 2020 Pivotal, Inc. All Rights
+Reserved.
+
+Copies of this document may be made for your own use and for
+distribution to others, provided that you do not charge any fee for such
+copies and further provided that each copy contains this Copyright
+Notice, whether distributed in print or electronically.

spring-batch-docs/src/main/asciidoc/glossary.adoc

@@ -0,0 +1,103 @@
+[[glossary]]
+[appendix]
+== Glossary
+[glossary]
+=== Spring Batch Glossary
+
+Batch::
+  An accumulation of business transactions over time.
+
+Batch Application Style::
+  Term used to designate batch as an application style in its own right, similar to
+  online, Web, or SOA. It has standard elements of input, validation, transformation of
+  information to business model, business processing, and output. In addition, it
+  requires monitoring at a macro level.
+
+Batch Processing::
+  The handling of a batch of many business transactions that have accumulated over a
+  period of time (such as an hour, a day, a week, a month, or a year). It is the
+  application of a process or set of processes to many data entities or objects in a
+  repetitive and predictable fashion with either no manual element or a separate manual
+  element for error processing.
+
+Batch Window::
+  The time frame within which a batch job must complete. This can be constrained by other
+  systems coming online, other dependent jobs needing to execute, or other factors
+  specific to the batch environment.
+
+Step::
+  The main batch task or unit of work. It initializes the business logic and controls the
+  transaction environment, based on commit interval setting and other factors.
+
+Tasklet::
+  A component created by an application developer to process the business logic for a
+  Step.
+
+Batch Job Type::
+  Job types describe application of jobs for particular types of processing. Common areas
+  are interface processing (typically flat files), forms processing (either for online
+  PDF generation or print formats), and report processing.
+
+Driving Query::
+  A driving query identifies the set of work for a job to do. The job then breaks that
+  work into individual units of work. For instance, a driving query might be to identify
+  all financial transactions that have a status of "pending transmission" and send them
+  to a partner system. The driving query returns a set of record IDs to process. Each
+  record ID then becomes a unit of work. A driving query may involve a join (if the
+  criteria for selection falls across two or more tables) or it may work with a single
+  table.
+
+Item::
+  An item represents the smallest amount of complete data for processing. In the simplest
+  terms, this might be a line in a file, a row in a database table, or a particular
+  element in an XML file.
+
+Logicial Unit of Work (LUW)::
+  A batch job iterates through a driving query (or other input source, such as a file) to
+  perform the set of work that the job must accomplish. Each iteration of work performed
+  is a unit of work.
+
+Commit Interval::
+  A set of LUWs processed within a single transaction.
+
+Partitioning::
+  Splitting a job into multiple threads where each thread is responsible for a subset of
+  the overall data to be processed. The threads of execution may be within the same JVM
+  or they may span JVMs in a clustered environment that supports workload balancing.
+
+Staging Table::
+  A table that holds temporary data while it is being processed.
+
+Restartable::
+  A job that can be executed again and assumes the same identity as when run initially.
+  In other words, it is has the same job instance ID.
+
+Rerunnable::
+  A job that is restartable and manages its own state in terms of the previous run's
+  record processing. An example of a rerunnable step is one based on a driving query. If
+  the driving query can be formed so that it limits the processed rows when the job is
+  restarted, then it is re-runnable. This is managed by the application logic. Often, a
+  condition is added to the `where` statement to limit the rows returned by the driving
+  query with logic resembling "and processedFlag!= true".
+
+Repeat::
+  One of the most basic units of batch processing, it defines by repeatability calling a
+  portion of code until it is finished and while there is no error. Typically, a batch
+  process would be repeatable as long as there is input.
+
+Retry::
+  Simplifies the execution of operations with retry semantics most frequently associated
+  with handling transactional output exceptions. Retry is slightly different from repeat,
+  rather than continually calling a block of code, retry is stateful and continually
+  calls the same block of code with the same input, until it either succeeds or some type
+  of retry limit has been exceeded. It is only generally useful when a subsequent
+  invocation of the operation might succeed because something in the environment has
+  improved.
+
+Recover::
+  Recover operations handle an exception in such a way that a repeat process is able to
+  continue.
+
+Skip::
+  Skip is a recovery strategy often used on file input sources as the strategy for
+  ignoring bad input records that failed validation.

spring-batch-docs/src/main/asciidoc/header/index-header.adoc

@@ -0,0 +1,3 @@
+= Spring Batch - Reference Documentation
+
+:batch-asciidoc: https://docs.spring.io/spring-batch/docs/current/reference/html/

spring-batch-docs/src/main/asciidoc/index-single.adoc

@@ -0,0 +1,47 @@
+:doctype: book
+:toc: left
+:toclevels: 4
+:sectnums:
+:onlyonetoggle: true
+
+include::header/index-header.adoc[]
+
+include::toggle.adoc[]
+
+include::spring-batch-intro.adoc[]
+
+include::whatsnew.adoc[]
+
+include::domain.adoc[]
+
+include::job.adoc[]
+
+include::step.adoc[]
+
+include::readersAndWriters.adoc[]
+
+include::processor.adoc[]
+
+include::scalability.adoc[]
+
+include::repeat.adoc[]
+
+include::retry.adoc[]
+
+include::testing.adoc[]
+
+include::common-patterns.adoc[]
+
+include::jsr-352.adoc[]
+
+include::spring-batch-integration.adoc[]
+
+include::monitoring-and-metrics.adoc[]
+
+include::appendix.adoc[]
+
+include::schema-appendix.adoc[]
+
+include::transaction-appendix.adoc[]
+
+include::glossary.adoc[]

spring-batch-docs/src/main/asciidoc/index.adoc

@@ -0,0 +1,49 @@
+include::header/index-header.adoc[]
+
+// ======================================================================================
+
+Welcome to the Spring Batch reference documentation! This documentation is also available
+as single link:index-single.html[html] and link:../pdf/spring-batch-reference.pdf[pdf] documents.
+
+The reference documentation is divided into several sections:
+
+[horizontal]
+<<spring-batch-intro.adoc#spring-batch-intro,Spring Batch Introduction>> :: Background, usage
+ scenarios and general guidelines.
+<<whatsnew.adoc#whatsNew,What's new in Spring Batch 5.0>> :: New features introduced in version 5.0.
+<<domain.adoc#domainLanguageOfBatch,The Domain Language of Batch>> :: Core concepts and abstractions
+of the Batch domain language.
+<<job.adoc#configureJob,Configuring and Running a Job>> :: Job configuration, execution and
+administration.
+<<step.adoc#configureStep,Configuring a Step>> :: Step configuration, different types of steps,
+controlling step flow.
+<<readersAndWriters.adoc#readersAndWriters,Item reading and writing>> :: `ItemReader`
+and `ItemWriter` interfaces and how to use them.
+<<processor.adoc#itemProcessor,Item processing>> :: `ItemProcessor` interface and how to use it.
+<<scalability.adoc#scalability,Scaling and Parallel Processing>> :: Multi-threaded steps,
+parallel steps, remote chunking and partitioning.
+<<repeat.adoc#repeat,Repeat>> :: Completion policies and exception handling of repetitive actions.
+<<retry.adoc#retry,Retry>> :: Retry and backoff policies of retryable operations.
+<<testing.adoc#testing,Unit Testing>> :: Job and Step testing facilities and APIs.
+<<common-patterns.adoc#commonPatterns, Common Patterns>> :: Common batch processing patterns
+and guidelines.
+<<jsr-352.adoc#jsr-352,JSR-352 Support>> :: JSR-352 support, similarities and differences
+with Spring Batch.
+<<spring-batch-integration.adoc#springBatchIntegration,Spring Batch Integration>> :: Integration
+between Spring Batch and Spring Integration projects.
+<<monitoring-and-metrics.adoc#monitoring-and-metrics,Monitoring and metrics>> :: Batch jobs
+monitoring and metrics
+
+The following appendices are available:
+
+[horizontal]
+<<appendix.adoc#listOfReadersAndWriters,List of ItemReaders and ItemWriters>> :: List of
+all item readers and writers provided out-of-the box.
+<<schema-appendix.adoc#metaDataSchema,Meta-Data Schema>> :: Core tables used by the Batch
+domain model.
+<<transaction-appendix.adoc#transactions,Batch Processing and Transactions>> :: Transaction
+boundaries, propagation and isolation levels used in Spring Batch.
+<<glossary.adoc#glossary,Glossary>> :: Glossary of common terms, concepts and vocabulary of
+the Batch domain.
+
+include::footer/index-footer.adoc[]

spring-batch-docs/src/main/asciidoc/js/DocumentToggle.js

@@ -0,0 +1,79 @@
+$(document).ready(function(){
+
+    var BATCH_LANGUAGES = ["java", "xml", "both"];
+    var $xmlButton = $("#xmlButton");
+    var $javaButton = $("#javaButton");
+    var $bothButton = $("#bothButton");
+
+    var $xmlContent = $("*.xmlContent");
+    var $xmlContentAll = $("*.xmlContent > *");
+
+    var $javaContent = $("*.javaContent");
+    var $javaContentAll = $("*.javaContent > *");
+
+    // Initial cookie handler. This part remembers the
+    // reader's choice and sets the toggle accordingly.
+    var lang = window.localStorage.getItem("docToggle");
+    if (BATCH_LANGUAGES.indexOf(lang) === -1) {
+        lang = "java";
+        $javaButton.prop("checked", true);
+        setJava();
+    } else {
+        if (lang === "xml") {
+            $xmlButton.prop("checked", true);
+            setXml();
+        }
+        if (lang === "java") {
+            $javaButton.prop("checked", true);
+            setJava();
+        }
+        if (lang === "both") {
+            $javaButton.prop("checked", true);
+            setBoth();
+        }
+    }
+
+    // Click handlers
+    $xmlButton.on("click", function() {
+        setXml();
+    });
+    $javaButton.on("click", function() {
+        setJava();
+    });
+    $bothButton.on("click", function() {
+        setBoth();
+    });
+
+    // Functions to do the work of handling the reader's choice, whether through a click
+    // or through a cookie. 3652 days is 10 years, give or take a leap day.
+    function setXml() {
+        $xmlContent.show();
+        $javaContent.hide();
+        $javaContentAll.addClass("js-toc-ignore");
+        $xmlContentAll.removeClass("js-toc-ignore");
+        window.dispatchEvent(new Event("tocRefresh"));
+        tocbot.refresh();
+        window.localStorage.setItem('docToggle', 'xml');
+    }
+
+    function setJava() {
+        $javaContent.show();
+        $xmlContent.hide();
+        $xmlContentAll.addClass("js-toc-ignore");
+        $javaContentAll.removeClass("js-toc-ignore");
+        window.dispatchEvent(new Event("tocRefresh"));
+        tocbot.refresh();
+        window.localStorage.setItem('docToggle', 'java');
+    }
+
+    function setBoth() {
+        $javaContent.show();
+        $xmlContent.show();
+        $javaContentAll.removeClass("js-toc-ignore");
+        $xmlContentAll.removeClass("js-toc-ignore");
+        window.dispatchEvent(new Event("tocRefresh"));
+        tocbot.refresh();
+        window.localStorage.setItem('docToggle', 'both');
+    }
+
+});

spring-batch-docs/src/main/asciidoc/js/Redirect.js

@@ -0,0 +1,62 @@
+$(document).ready(function(){
+
+    redirect();
+
+    function redirect() {
+        var anchorMap = {
+            "#domain": "#domainLanguageOfBatch",
+            "#domainJob": "#job",
+            "#domainJobInstance": "#jobinstance",
+            "#domainJobParameters": "#jobparameters",
+            "#domainJobExecution": "#jobexecution",
+            "#d5e455": "#jobexecution",
+            "#d5e497": "#jobexecution",
+            "#d5e507": "#jobexecution",
+            "#d5e523": "#jobexecution",
+            "#d5e550": "#jobexecution",
+            "#d5e563": "#jobexecution",
+            "#d5e591": "#jobexecution",
+            "#domainStep": "#step",
+            "#domainStepExecution": "#stepexecution",
+            "#d5e655": "#stepexecution",
+            "#domainExecutionContext": "#executioncontext",
+            "#d5e721": "#executioncontext",
+            "#d5e731": "#executioncontext",
+            "#d5e745": "#executioncontext",
+            "#d5e761": "#executioncontext",
+            "#d5e779": "#executioncontext",
+            "#domainJobRepository": "#jobrepository",
+            "#domainJobLauncher": "#joblauncher",
+            "#domainItemReader": "#item-reader",
+            "#domainItemWriter": "#item-writer",
+            "#domainItemProcessor": "#item-processor",
+            "#domainBatchNamespace": "#batch-namespace",
+            "#d5e970": "#jobparametersvalidator",
+            "#d5e1130": "#commandLineJobRunner",
+            "#d5e1232": "#jobregistry",
+            "#d5e1237": "#jobregistrybeanpostprocessor",
+            "#d5e1242": "#automaticjobregistrar",
+            "#d5e1320": "#aborting-a-job",
+            "#filiteringRecords": "#filteringRecords",
+            "#d5e2247": "#flatFileItemReader",
+            "#d5e2769": "#JdbcCursorItemReaderProperties",
+            "#stepExecutionSplitter": "#partitioner",
+            "#d5e3182": "#bindingInputDataToSteps",
+            "#d5e3241": "#repeatStatus",
+            "#d5e3531": "#testing-step-scoped-components",
+            "#patterns": "#commonPatterns",
+            "#d5e3959": "#item-based-processing",
+            "#d5e3969": "#custom-checkpointing",
+            "#available-attributes-of-the-job-launching-gateway": "#availableAttributesOfTheJobLaunchingGateway",
+            "#d5e4425": "#itemReadersAppendix",
+            "#d5e4494": "#itemWritersAppendix",
+            "#d5e4788": "#recommendationsForIndexingMetaDataTables"
+        };
+        var baseUrl = window.location.origin + window.location.pathname;
+        var anchor = window.location.hash;
+        if (anchor && anchorMap[anchor] != null) {
+            window.location.replace(baseUrl + anchorMap[anchor]);
+        }
+    }
+
+});

spring-batch-docs/src/main/asciidoc/js/js.cookie.js

@@ -0,0 +1,165 @@
+/*!
+ * JavaScript Cookie v2.1.4
+ * https://github.com/js-cookie/js-cookie
+ *
+ * Copyright 2006, 2015 Klaus Hartl & Fagner Brack
+ * Released under the MIT license
+ */
+;(function (factory) {
+    var registeredInModuleLoader = false;
+    if (typeof define === 'function' && define.amd) {
+        define(factory);
+        registeredInModuleLoader = true;
+    }
+    if (typeof exports === 'object') {
+        module.exports = factory();
+        registeredInModuleLoader = true;
+    }
+    if (!registeredInModuleLoader) {
+        var OldCookies = window.Cookies;
+        var api = window.Cookies = factory();
+        api.noConflict = function () {
+            window.Cookies = OldCookies;
+            return api;
+        };
+    }
+}(function () {
+    function extend () {
+        var i = 0;
+        var result = {};
+        for (; i < arguments.length; i++) {
+            var attributes = arguments[ i ];
+            for (var key in attributes) {
+                result[key] = attributes[key];
+            }
+        }
+        return result;
+    }
+
+    function init (converter) {
+        function api (key, value, attributes) {
+            var result;
+            if (typeof document === 'undefined') {
+                return;
+            }
+
+            // Write
+
+            if (arguments.length > 1) {
+                attributes = extend({
+                    path: '/'
+                }, api.defaults, attributes);
+
+                if (typeof attributes.expires === 'number') {
+                    var expires = new Date();
+                    expires.setMilliseconds(expires.getMilliseconds() + attributes.expires * 864e+5);
+                    attributes.expires = expires;
+                }
+
+                // We're using "expires" because "max-age" is not supported by IE
+                attributes.expires = attributes.expires ? attributes.expires.toUTCString() : '';
+
+                try {
+                    result = JSON.stringify(value);
+                    if (/^[\{\[]/.test(result)) {
+                        value = result;
+                    }
+                } catch (e) {}
+
+                if (!converter.write) {
+                    value = encodeURIComponent(String(value))
+                        .replace(/%(23|24|26|2B|3A|3C|3E|3D|2F|3F|40|5B|5D|5E|60|7B|7D|7C)/g, decodeURIComponent);
+                } else {
+                    value = converter.write(value, key);
+                }
+
+                key = encodeURIComponent(String(key));
+                key = key.replace(/%(23|24|26|2B|5E|60|7C)/g, decodeURIComponent);
+                key = key.replace(/[\(\)]/g, escape);
+
+                var stringifiedAttributes = '';
+
+                for (var attributeName in attributes) {
+                    if (!attributes[attributeName]) {
+                        continue;
+                    }
+                    stringifiedAttributes += '; ' + attributeName;
+                    if (attributes[attributeName] === true) {
+                        continue;
+                    }
+                    stringifiedAttributes += '=' + attributes[attributeName];
+                }
+                return (document.cookie = key + '=' + value + stringifiedAttributes);
+            }
+
+            // Read
+
+            if (!key) {
+                result = {};
+            }
+
+            // To prevent the for loop in the first place assign an empty array
+            // in case there are no cookies at all. Also prevents odd result when
+            // calling "get()"
+            var cookies = document.cookie ? document.cookie.split('; ') : [];
+            var rdecode = /(%[0-9A-Z]{2})+/g;
+            var i = 0;
+
+            for (; i < cookies.length; i++) {
+                var parts = cookies[i].split('=');
+                var cookie = parts.slice(1).join('=');
+
+                if (cookie.charAt(0) === '"') {
+                    cookie = cookie.slice(1, -1);
+                }
+
+                try {
+                    var name = parts[0].replace(rdecode, decodeURIComponent);
+                    cookie = converter.read ?
+                        converter.read(cookie, name) : converter(cookie, name) ||
+                        cookie.replace(rdecode, decodeURIComponent);
+
+                    if (this.json) {
+                        try {
+                            cookie = JSON.parse(cookie);
+                        } catch (e) {}
+                    }
+
+                    if (key === name) {
+                        result = cookie;
+                        break;
+                    }
+
+                    if (!key) {
+                        result[name] = cookie;
+                    }
+                } catch (e) {}
+            }
+
+            return result;
+        }
+
+        api.set = api;
+        api.get = function (key) {
+            return api.call(api, key);
+        };
+        api.getJSON = function () {
+            return api.apply({
+                json: true
+            }, [].slice.call(arguments));
+        };
+        api.defaults = {};
+
+        api.remove = function (key, attributes) {
+            api(key, '', extend(attributes, {
+                expires: -1
+            }));
+        };
+
+        api.withConverter = init;
+
+        return api;
+    }
+
+    return init(function () {});
+}));

spring-batch-docs/src/main/asciidoc/jsfiles/DocumentToggle.js

@@ -0,0 +1,62 @@
+$(document).ready(function(){
+
+    // Make Java the default
+    setJava();
+
+    // Initial cookie handler. This part remembers the reader's choice and sets the toggle
+    // accordingly.
+    var docToggleCookieString = Cookies.get("docToggle");
+    if (docToggleCookieString != null) {
+        if (docToggleCookieString === "xml") {
+            $("#xmlButton").prop("checked", true);
+            setXml();
+        } else if (docToggleCookieString === "java") {
+            $("#javaButton").prop("checked", true);
+            setJava();
+        } else if (docToggleCookieString === "both") {
+            $("#bothButton").prop("checked", true);
+            setBoth();
+        }
+    }
+
+    // Click handlers
+    $("#xmlButton").on("click", function() {
+        setXml();
+    });
+    $("#javaButton").on("click", function() {
+        setJava();
+    });
+    $("#bothButton").on("click", function() {
+        setBoth();
+    });
+
+    // Functions to do the work of handling the reader's choice, whether through a click
+    // or through a cookie. 3652 days is 10 years, give or take a leap day.
+    function setXml() {
+        $("*.xmlContent").show();
+        $("*.javaContent").hide();
+        $("*.javaContent > *").addClass("js-toc-ignore");
+        $("*.xmlContent > *").removeClass("js-toc-ignore");
+        window.dispatchEvent(new Event("tocRefresh"));
+        Cookies.set('docToggle', 'xml', { expires: 3652 });
+    };
+
+    function setJava() {
+        $("*.javaContent").show();
+        $("*.xmlContent").hide();
+        $("*.xmlContent > *").addClass("js-toc-ignore");
+        $("*.javaContent > *").removeClass("js-toc-ignore");
+        window.dispatchEvent(new Event("tocRefresh"));
+        Cookies.set('docToggle', 'java', { expires: 3652 });
+    };
+
+    function setBoth() {
+        $("*.javaContent").show();
+        $("*.xmlContent").show();
+        $("*.javaContent > *").removeClass("js-toc-ignore");
+        $("*.xmlContent > *").removeClass("js-toc-ignore");
+        window.dispatchEvent(new Event("tocRefresh"));
+        Cookies.set('docToggle', 'both', { expires: 3652 });
+    };
+
+});

spring-batch-docs/src/main/asciidoc/jsr-352.adoc

@@ -0,0 +1,620 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[jsr-352]]
+
+== JSR-352 Support
+
+ifndef::onlyonetoggle[]
+include::toggle.adoc[]
+endif::onlyonetoggle[]
+
+As of Spring Batch 3.0 support for JSR-352 has been fully implemented. This section is not a replacement for
+the spec itself and instead, intends to explain how the JSR-352 specific concepts apply to Spring Batch.
+Additional information on JSR-352 can be found via the
+JCP here: link:$$https://jcp.org/en/jsr/detail?id=352$$[https://jcp.org/en/jsr/detail?id=352]
+
+[[jsrGeneralNotes]]
+
+
+=== General Notes about Spring Batch and JSR-352
+
+Spring Batch and JSR-352 are structurally the same.  They both have jobs that are made up of steps.  They
+both have readers, processors, writers, and listeners.  However, their interactions are subtly different.
+For example, the `org.springframework.batch.core.SkipListener#onSkipInWrite(S item, Throwable t)`
+within Spring Batch receives two parameters: the item that was skipped and the Exception that caused the
+skip.  The JSR-352 version of the same method
+(`javax.batch.api.chunk.listener.SkipWriteListener#onSkipWriteItem(List<Object> items, Exception ex)`)
+also receives two parameters.  However the first one is a `List` of all the items
+within the current chunk with the second being the `Exception` that caused the skip.
+Because of these differences, it is important to note that there are two paths to execute a job within
+Spring Batch: either a traditional Spring Batch job or a JSR-352 based job.  While the use of Spring Batch
+artifacts (readers, writers, etc) will work within a job configured with JSR-352's JSL and executed with the
+`JsrJobOperator`, they will behave according to the rules of JSR-352.  It is also
+important to note that batch artifacts that have been developed against the JSR-352 interfaces will not work
+within a traditional Spring Batch job.
+
+[[jsrSetup]]
+
+
+=== Setup
+
+[[jsrSetupContexts]]
+
+
+==== Application Contexts
+
+All JSR-352 based jobs within Spring Batch consist of two application contexts.  A parent context, that
+contains beans related to the infrastructure of Spring Batch such as the `JobRepository`,
+`PlatformTransactionManager`, etc and a child context that consists of the configuration
+of the job to be run.  The parent context is defined via the `jsrBaseContext.xml` provided
+by the framework.  This context may be overridden by setting the `JSR-352-BASE-CONTEXT` system
+property.
+
+
+[NOTE]
+====
+The base context is not processed by the JSR-352 processors for things like property injection so
+no components requiring that additional processing should be configured there.
+
+====
+
+
+[[jsrSetupLaunching]]
+
+
+==== Launching a JSR-352 based job
+
+JSR-352 requires a very simple path to executing a batch job.  The following code is all that is needed to
+execute your first batch job:
+
+
+
+[source, java]
+----
+JobOperator operator = BatchRuntime.getJobOperator();
+jobOperator.start("myJob", new Properties());
+----
+
+While that is convenient for developers, the devil is in the details.  Spring Batch bootstraps a bit of
+infrastructure behind the scenes that a developer may want to override.  The following is bootstrapped the
+first time `BatchRuntime.getJobOperator()` is called:
+
+|===============
+|__Bean Name__|__Default Configuration__|__Notes__
+|
+									dataSource
+								|
+									Apache DBCP BasicDataSource with configured values.
+								|
+									By default, HSQLDB is bootstrapped.
+
+|`transactionManager`|`org.springframework.jdbc.datasource.DataSourceTransactionManager`|
+									References the dataSource bean defined above.
+
+|
+									A Datasource initializer
+								||
+									This is configured to execute the scripts configured via the
+									`batch.drop.script` and `batch.schema.script` properties.  By
+									default, the schema scripts for HSQLDB are executed.  This behavior can be disabled by setting the
+									`batch.data.source.init` property.
+
+|
+									jobRepository
+								|
+									A JDBC based `SimpleJobRepository`.
+								|
+									This `JobRepository` uses the previously mentioned data source and transaction
+									manager.  The schema's table prefix is configurable (defaults to BATCH_) via the
+									`batch.table.prefix` property.
+
+|
+									jobLauncher
+								|`org.springframework.batch.core.launch.support.SimpleJobLauncher`|
+									Used to launch jobs.
+
+|
+									batchJobOperator
+								|`org.springframework.batch.core.launch.support.SimpleJobOperator`|
+									The `JsrJobOperator` wraps this to provide most of it's functionality.
+
+|
+									jobExplorer
+								|`org.springframework.batch.core.explore.support.JobExplorerFactoryBean`|
+									Used to address lookup functionality provided by the `JsrJobOperator`.
+
+|
+									jobParametersConverter
+								|`org.springframework.batch.core.jsr.JsrJobParametersConverter`|
+									JSR-352 specific implementation of the `JobParametersConverter`.
+
+|
+									jobRegistry
+								|`org.springframework.batch.core.configuration.support.MapJobRegistry`|
+									Used by the `SimpleJobOperator`.
+
+|
+									placeholderProperties
+								|`org.springframework.beans.factory.config.PropertyPlaceholderConfigure`|
+									Loads the properties file `batch-${ENVIRONMENT:hsql}.properties` to configure
+									the properties mentioned above.  ENVIRONMENT is a System property (defaults to `hsql`)
+									that can be used to specify any of the supported databases Spring Batch currently
+									supports.
+
+
+|===============
+
+
+
+
+
+
+[NOTE]
+====
+None of the above beans are optional for executing JSR-352 based jobs.  All may be overridden to
+provide customized functionality as needed.
+====
+
+
+[[dependencyInjection]]
+
+
+=== Dependency Injection
+
+JSR-352 is based heavily on the Spring Batch programming model.  As such, while not explicitly requiring a
+formal dependency injection implementation, DI of some kind implied.  Spring Batch supports all three
+methods for loading batch artifacts defined by JSR-352:
+
+
+* Implementation Specific Loader: Spring Batch is built upon Spring and so supports
+Spring dependency injection within JSR-352 batch jobs.
+* Archive Loader: JSR-352 defines the existing of a `batch.xml` file that provides mappings
+between a logical name and a class name.  This file must be found within the `/META-INF/`
+directory if it is used.
+* Thread Context Class Loader: JSR-352 allows configurations to specify batch artifact
+implementations in their JSL by providing the fully qualified class name inline. Spring
+Batch supports this as well in JSR-352 configured jobs.
+
+To use Spring dependency injection within a JSR-352 based batch job consists of
+configuring batch artifacts using a Spring application context as beans.  Once the beans
+have been defined, a job can refer to them as it would any bean defined within the
+`batch.xml` file.
+
+[role="xmlContent"]
+The following example shows how to use Spring dependency injection within a JSR-352 based
+batch job in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<?xml version="1.0" encoding="UTF-8"?>
+<beans xmlns="http://www.springframework.org/schema/beans"
+          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+          xsi:schemaLocation="http://www.springframework.org/schema/beans
+                              https://www.springframework.org/schema/beans/spring-beans.xsd
+                              http://xmlns.jcp.org/xml/ns/javaee
+                              https://xmlns.jcp.org/xml/ns/javaee/jobXML_1_0.xsd">
+
+    <!-- javax.batch.api.Batchlet implementation -->
+    <bean id="fooBatchlet" class="io.spring.FooBatchlet">
+            <property name="prop" value="bar"/>
+    </bean>
+
+    <!-- Job is defined using the JSL schema provided in JSR-352 -->
+    <job id="fooJob" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0">
+        <step id="step1">
+            <batchlet ref="fooBatchlet"/>
+        </step>
+    </job>
+</beans>
+----
+
+[role="javaContent"]
+The following example shows how to use Spring dependency injection within a JSR-352 based
+batch job in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Configuration
+public class BatchConfiguration {
+
+    @Bean
+    public Batchlet fooBatchlet() {
+        FooBatchlet batchlet = new FooBatchlet();
+        batchlet.setProp("bar");
+        return batchlet;
+       }
+}
+
+
+<?xml version="1.0" encoding="UTF-8"?>
+<job id="fooJob" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0">
+    <step id="step1" >
+        <batchlet ref="fooBatchlet" />
+    </step>
+</job>
+----
+
+The assembly of Spring contexts (imports, etc) works with JSR-352 jobs just as it would with any other
+Spring based application.  The only difference with a JSR-352 based job is that the entry point for the
+context definition will be the job definition found in /META-INF/batch-jobs/.
+
+To use the thread context class loader approach, all you need to do is provide the fully qualified class
+name as the ref.  It is important to note that when using this approach or the `batch.xml` approach, the class
+referenced requires a no argument constructor which will be used to create the bean.
+
+
+[source, xml]
+----
+<?xml version="1.0" encoding="UTF-8"?>
+<job id="fooJob" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0">
+    <step id="step1" >
+        <batchlet ref="io.spring.FooBatchlet" />
+    </step>
+</job>
+
+----
+
+[[jsrJobProperties]]
+
+
+=== Batch Properties
+
+[[jsrPropertySupport]]
+
+
+==== Property Support
+
+JSR-352 allows for properties to be defined at the Job, Step and batch artifact level by way of
+configuration in the JSL. Batch properties are configured at each level in the following way:
+
+
+[source, xml]
+----
+<properties>
+    <property name="propertyName1" value="propertyValue1"/>
+    <property name="propertyName2" value="propertyValue2"/>
+</properties>
+----
+
+
+`Properties` may be configured on any batch artifact.
+
+[[jsrBatchPropertyAnnotation]]
+
+
+==== @BatchProperty annotation
+
+`Properties` are referenced in batch artifacts by annotating class fields with the
+`@BatchProperty` and `@Inject` annotations (both annotations
+are required by the spec). As defined by JSR-352, fields for properties must be String typed. Any type
+conversion is up to the implementing developer to perform.
+
+An `javax.batch.api.chunk.ItemReader` artifact could be configured with a
+properties block such as the one described above and accessed as such:
+
+
+[source, java]
+----
+public class MyItemReader extends AbstractItemReader {
+    @Inject
+    @BatchProperty
+    private String propertyName1;
+
+    ...
+}
+----
+
+
+The value of the field "propertyName1" will be "propertyValue1"
+
+[[jsrPropertySubstitution]]
+
+
+==== Property Substitution
+
+Property substitution is provided by way of operators and simple conditional expressions. The general
+usage is `#{operator['key']}`.
+
+Supported operators:
+
+* `jobParameters`: access job parameter values that the job was started/restarted with.
+* `jobProperties`: access properties configured at the job level of the JSL.
+* `systemProperties`: access named system properties.
+* `partitionPlan`: access named property from the partition plan of a partitioned step.
+
+----
+#{jobParameters['unresolving.prop']}?:#{systemProperties['file.separator']}
+----
+
+The left hand side of the assignment is the expected value, the right hand side is the
+default value. In the preceding
+ example, the result will resolve to a value of the system property file.separator as
+ #{jobParameters['unresolving.prop']} is assumed to not be resolvable. If neither
+expressions can be resolved, an empty String will be returned. Multiple conditions can be
+used, which are separated by a ';'.
+
+
+[[jsrProcessingModels]]
+
+=== Processing Models
+
+JSR-352 provides the same two basic processing models that Spring Batch does:
+
+* Item based processing - Using an `javax.batch.api.chunk.ItemReader`, an optional
+`javax.batch.api.chunk.ItemProcessor`, and an `javax.batch.api.chunk.ItemWriter`.
+* Task based processing - Using a `javax.batch.api.Batchlet`
+implementation.  This processing model is the same as the
+`org.springframework.batch.core.step.tasklet.Tasklet` based processing
+currently available.
+
+
+==== Item based processing
+
+Item based processing in this context is a chunk size being set by the number of items read by an
+`ItemReader`.  To configure a step this way, specify the
+`item-count` (which defaults to 10) and optionally configure the
+`checkpoint-policy` as item (this is the default).
+
+
+[source, xml]
+----
+...
+<step id="step1">
+    <chunk checkpoint-policy="item" item-count="3">
+        <reader ref="fooReader"/>
+        <processor ref="fooProcessor"/>
+        <writer ref="fooWriter"/>
+    </chunk>
+</step>
+...
+----
+
+If item-based checkpointing is chosen, an additional attribute `time-limit` is supported.
+This sets a time limit for how long the number of items specified has to be processed.  If
+the timeout is reached, the chunk will complete with however many items have been read by
+then regardless of what the `item-count` is configured to be.
+
+
+==== Custom checkpointing
+
+JSR-352 calls the process around the commit interval within a step "checkpointing".
+Item-based checkpointing is one approach as mentioned above. However, this is not robust
+enough in many cases. Because of this, the spec allows for the implementation of a custom
+checkpointing algorithm by implementing the `javax.batch.api.chunk.CheckpointAlgorithm`
+interface. This functionality is functionally the same as Spring Batch's custom completion
+policy.  To use an implementation of `CheckpointAlgorithm`, configure your step with the
+custom `checkpoint-policy` as shown below where `fooCheckpointer` refers to an
+implementation of `CheckpointAlgorithm`.
+
+
+[source, xml]
+----
+...
+<step id="step1">
+    <chunk checkpoint-policy="custom">
+        <checkpoint-algorithm ref="fooCheckpointer"/>
+        <reader ref="fooReader"/>
+        <processor ref="fooProcessor"/>
+        <writer ref="fooWriter"/>
+    </chunk>
+</step>
+...
+----
+
+[[jsrRunningAJob]]
+
+=== Running a job
+
+The entrance to executing a JSR-352 based job is through the
+`javax.batch.operations.JobOperator`. Spring Batch provides its own implementation of
+this interface (`org.springframework.batch.core.jsr.launch.JsrJobOperator`).  This
+implementation is loaded via the `javax.batch.runtime.BatchRuntime`.  Launching a
+JSR-352 based batch job is implemented as follows:
+
+
+[source, java]
+----
+
+JobOperator jobOperator = BatchRuntime.getJobOperator();
+long jobExecutionId = jobOperator.start("fooJob", new Properties());
+
+----
+
+The above code does the following:
+
+* Bootstraps a base `ApplicationContext`: In order to provide batch functionality, the
+framework needs some infrastructure bootstrapped. This occurs once per JVM. The
+components that are bootstrapped are similar to those provided by
+`@EnableBatchProcessing`. Specific details can be found in the javadoc for the
+`JsrJobOperator`.
+* Loads an `ApplicationContext` for the job requested: In the example
+above, the framework looks in /META-INF/batch-jobs for a file named fooJob.xml and load a
+context that is a child of the shared context mentioned previously.
+* Launch the job: The job defined within the context will be executed asynchronously.
+The `JobExecution's` ID will be returned.
+
+[NOTE]
+====
+All JSR-352 based batch jobs are executed asynchronously.
+====
+
+When `JobOperator#start` is called using `SimpleJobOperator`, Spring Batch determines if
+the call is an initial run or a retry of a previously executed run. Using the JSR-352
+based `JobOperator#start(String jobXMLName, Properties jobParameters)`, the framework
+will always create a new JobInstance (JSR-352 job parameters are non-identifying). In order to
+restart a job, a call to
+`JobOperator#restart(long executionId, Properties restartParameters)` is required.
+
+
+[[jsrContexts]]
+
+=== Contexts
+
+JSR-352 defines two context objects that are used to interact with the meta-data of a job or step from
+within a batch artifact: `javax.batch.runtime.context.JobContext` and
+`javax.batch.runtime.context.StepContext`.  Both of these are available in any step
+level artifact (`Batchlet`, `ItemReader`, etc) with the
+`JobContext` being available to job level artifacts as well
+(`JobListener` for example).
+
+To obtain a reference to the `JobContext` or `StepContext`
+within the current scope, simply use the `@Inject` annotation:
+
+
+[source, java]
+----
+@Inject
+JobContext jobContext;
+
+----
+
+
+[NOTE]
+.@Autowire for JSR-352 contexts
+====
+Using Spring's @Autowire is not supported for the injection of these contexts.
+====
+
+
+In Spring Batch, the `JobContext` and `StepContext` wrap their
+corresponding execution objects (`JobExecution` and
+`StepExecution` respectively).  Data stored through
+`StepContext#setPersistentUserData(Serializable data)` is stored in the
+Spring Batch `StepExecution#executionContext`.
+
+[[jsrStepFlow]]
+
+
+=== Step Flow
+
+Within a JSR-352 based job, the flow of steps works similarly as it does within Spring Batch.
+However, there are a few subtle differences:
+
+
+* Decision's are steps - In a regular Spring Batch job, a decision is a state that does not
+have an independent `StepExecution` or any of the rights and
+responsibilities that go along with being a full step..  However, with JSR-352, a decision
+is a step just like any other and will behave just as any other steps (transactionality,
+it gets a `StepExecution`, etc).  This means that they are treated the
+same as any other step on restarts as well.
+
+* `next` attribute and step transitions - In a regular job, these are
+allowed to appear together in the same step.  JSR-352 allows them to both be used in the
+same step with the next attribute taking precedence in evaluation.
+
+* Transition element ordering - In a standard Spring Batch job, transition elements are
+sorted from most specific to least specific and evaluated in that order.  JSR-352 jobs
+evaluate transition elements in the order they are specified in the XML.
+
+
+
+
+[[jsrScaling]]
+
+
+=== Scaling a JSR-352 batch job
+
+Traditional Spring Batch jobs have four ways of scaling (the last two capable of being executed across
+multiple JVMs):
+
+* Split - Running multiple steps in parallel.
+
+
+* Multiple threads - Executing a single step via multiple threads.
+
+
+* Partitioning - Dividing the data up for parallel processing (manager/worker).
+
+
+* Remote Chunking - Executing the processor piece of logic remotely.
+
+
+
+
+JSR-352 provides two options for scaling batch jobs.  Both options support only a single JVM:
+
+* Split - Same as Spring Batch
+
+
+* Partitioning - Conceptually the same as Spring Batch however implemented slightly different.
+
+
+
+
+
+[[jsrPartitioning]]
+
+
+==== Partitioning
+
+Conceptually, partitioning in JSR-352 is the same as it is in Spring Batch.  Meta-data is provided
+to each worker to identify the input to be processed, with the workers reporting back to the manager the
+results upon completion. However, there are some important differences:
+
+* Partitioned `Batchlet` - This will run multiple instances of the
+configured `Batchlet` on multiple threads.  Each instance will have
+it's own set of properties as provided by the JSL or the
+`PartitionPlan`
+
+
+* `PartitionPlan` - With Spring Batch's partitioning, an
+`ExecutionContext` is provided for each partition.  With JSR-352, a
+single `javax.batch.api.partition.PartitionPlan` is provided with an
+array of `Properties` providing the meta-data for each partition.
+
+
+
+* `PartitionMapper` - JSR-352 provides two ways to generate partition
+meta-data.  One is via the JSL (partition properties).  The second is via an implementation
+of the `javax.batch.api.partition.PartitionMapper` interface.
+Functionally, this interface is similar to the
+`org.springframework.batch.core.partition.support.Partitioner`
+interface provided by Spring Batch in that it provides a way to programmatically generate
+meta-data for partitioning.
+
+
+* `StepExecutions` - In Spring Batch, partitioned steps are run as
+manager/worker. Within JSR-352, the same configuration occurs.  However, the worker steps do
+not get official `StepExecutions`.  Because of that, calls to
+`JsrJobOperator#getStepExecutions(long jobExecutionId)` will only
+return the `StepExecution` for the manager.
+
+[NOTE]
+====
+The child `StepExecutions` still exist in the job repository and are available
+through the `JobExplorer`.
+====
+
+
+* Compensating logic - Since Spring Batch implements the manager/worker logic of
+partitioning using steps, `StepExecutionListeners` can be used to
+handle compensating logic if something goes wrong.  However, since the workers JSR-352
+provides a collection of other components for the ability to provide compensating logic when
+errors occur and to dynamically set the exit status.  These components include the following:
+
+|===============
+|__Artifact Interface__|__Description__
+|`javax.batch.api.partition.PartitionCollector`|Provides a way for worker steps to send information back to the
+manager.  There is one instance per worker thread.
+|`javax.batch.api.partition.PartitionAnalyzer`|End point that receives the information collected by the
+`PartitionCollector` as well as the resulting
+statuses from a completed partition.
+|`javax.batch.api.partition.PartitionReducer`|Provides the ability to provide compensating logic for a partitioned
+step.
+
+|===============
+
+
+[[jsrTesting]]
+
+=== Testing
+
+Since all JSR-352 based jobs are executed asynchronously, it can be difficult to determine when a job has
+completed.  To help with testing, Spring Batch provides the
+`org.springframework.batch.test.JsrTestUtils`.  This utility class provides the
+ability to start a job and restart a job and wait for it to complete.  Once the job completes, the
+associated `JobExecution` is returned.

spring-batch-docs/src/main/asciidoc/monitoring-and-metrics.adoc

@@ -0,0 +1,90 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[monitoring-and-metrics]]
+
+== Monitoring and metrics
+
+Since version 4.2, Spring Batch provides support for batch monitoring and metrics
+based on link:$$https://micrometer.io/$$[Micrometer]. This section describes
+which metrics are provided out-of-the-box and how to contribute custom metrics.
+
+[[built-in-metrics]]
+
+=== Built-in metrics
+
+Metrics collection does not require any specific configuration. All metrics provided
+by the framework are registered in
+link:$$https://micrometer.io/docs/concepts#_global_registry$$[Micrometer's global registry]
+under the `spring.batch` prefix. The following table explains all the metrics in details:
+
+|===============
+|__Metric Name__|__Type__|__Description__|__Tags__
+|`spring.batch.job`|`TIMER`|Duration of job execution|`name`, `status`
+|`spring.batch.job.active`|`LONG_TASK_TIMER`|Currently active jobs|`name`
+|`spring.batch.step`|`TIMER`|Duration of step execution|`name`, `job.name`, `status`
+|`spring.batch.item.read`|`TIMER`|Duration of item reading|`job.name`, `step.name`, `status`
+|`spring.batch.item.process`|`TIMER`|Duration of item processing|`job.name`, `step.name`, `status`
+|`spring.batch.chunk.write`|`TIMER`|Duration of chunk writing|`job.name`, `step.name`, `status`
+|===============
+
+NOTE: The `status` tag can be either `SUCCESS` or `FAILURE`.
+
+[[custom-metrics]]
+
+=== Custom metrics
+
+If you want to use your own metrics in your custom components, we recommend using
+Micrometer APIs directly. The following is an example of how to time a `Tasklet`:
+
+[source, java]
+----
+import io.micrometer.core.instrument.Metrics;
+import io.micrometer.core.instrument.Timer;
+
+import org.springframework.batch.core.StepContribution;
+import org.springframework.batch.core.scope.context.ChunkContext;
+import org.springframework.batch.core.step.tasklet.Tasklet;
+import org.springframework.batch.repeat.RepeatStatus;
+
+public class MyTimedTasklet implements Tasklet {
+
+	@Override
+	public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) {
+		Timer.Sample sample = Timer.start(Metrics.globalRegistry);
+		String status = "success";
+		try {
+			// do some work
+		} catch (Exception e) {
+			// handle exception
+			status = "failure";
+		} finally {
+			sample.stop(Timer.builder("my.tasklet.timer")
+					.description("Duration of MyTimedTasklet")
+					.tag("status", status)
+					.register(Metrics.globalRegistry));
+		}
+		return RepeatStatus.FINISHED;
+	}
+}
+----
+
+[[disabling-metrics]]
+
+=== Disabling metrics
+
+Metrics collection is a concern similar to logging. Disabling logs is typically
+done by configuring the logging library and this is no different for metrics.
+There is no feature in Spring Batch to disable micrometer's metrics, this should
+be done on micrometer's side. Since Spring Batch stores metrics in the global
+registry of micrometer with the `spring.batch` prefix, it is possible to configure
+micrometer to ignore/deny batch metrics with the following snippet:
+
+[source, java]
+----
+Metrics.globalRegistry.config().meterFilter(MeterFilter.denyNameStartsWith("spring.batch"))
+----
+
+Please refer to micrometer's link:$$http://micrometer.io/docs/concepts#_meter_filters$$[reference documentation]
+for more details.

spring-batch-docs/src/main/asciidoc/processor.adoc

@@ -0,0 +1,379 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[itemProcessor]]
+== Item processing
+
+ifndef::onlyonetoggle[]
+include::toggle.adoc[]
+endif::onlyonetoggle[]
+
+The <<readersAndWriters.adoc#readersAndWriters,ItemReader and ItemWriter interfaces>> are both very useful for their specific
+tasks, but what if you want to insert business logic before writing? One option for both
+reading and writing is to use the composite pattern: Create an `ItemWriter` that contains
+another `ItemWriter` or an `ItemReader` that contains another `ItemReader`. The following
+code shows an example:
+
+[source, java]
+----
+public class CompositeItemWriter<T> implements ItemWriter<T> {
+
+    ItemWriter<T> itemWriter;
+
+    public CompositeItemWriter(ItemWriter<T> itemWriter) {
+        this.itemWriter = itemWriter;
+    }
+
+    public void write(List<? extends T> items) throws Exception {
+        //Add business logic here
+       itemWriter.write(items);
+    }
+
+    public void setDelegate(ItemWriter<T> itemWriter){
+        this.itemWriter = itemWriter;
+    }
+}
+----
+
+The preceding class contains another `ItemWriter` to which it delegates after having
+provided some business logic. This pattern could easily be used for an `ItemReader` as
+well, perhaps to obtain more reference data based upon the input that was provided by the
+main `ItemReader`. It is also useful if you need to control the call to `write` yourself.
+However, if you only want to 'transform' the item passed in for writing before it is
+actually written, you need not `write` yourself. You can just modify the item. For this
+scenario, Spring Batch provides the `ItemProcessor` interface, as shown in the following
+interface definition:
+
+[source, java]
+----
+public interface ItemProcessor<I, O> {
+
+    O process(I item) throws Exception;
+}
+----
+
+An `ItemProcessor` is simple. Given one object, transform it and return another. The
+provided object may or may not be of the same type. The point is that business logic may
+be applied within the process, and it is completely up to the developer to create that
+logic. An `ItemProcessor` can be wired directly into a step. For example, assume an
+`ItemReader` provides a class of type `Foo` and that it needs to be converted to type `Bar`
+before being written out. The following example shows an `ItemProcessor` that performs
+the conversion:
+
+[source, java]
+----
+public class Foo {}
+
+public class Bar {
+    public Bar(Foo foo) {}
+}
+
+public class FooProcessor implements ItemProcessor<Foo, Bar> {
+    public Bar process(Foo foo) throws Exception {
+        //Perform simple transformation, convert a Foo to a Bar
+        return new Bar(foo);
+    }
+}
+
+public class BarWriter implements ItemWriter<Bar> {
+    public void write(List<? extends Bar> bars) throws Exception {
+        //write bars
+    }
+}
+----
+
+In the preceding example, there is a class `Foo`, a class `Bar`, and a class
+`FooProcessor` that adheres to the `ItemProcessor` interface. The transformation is
+simple, but any type of transformation could be done here. The `BarWriter` writes `Bar`
+objects, throwing an exception if any other type is provided. Similarly, the
+`FooProcessor` throws an exception if anything but a `Foo` is provided. The
+`FooProcessor` can then be injected into a `Step`, as shown in the following example:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<job id="ioSampleJob">
+    <step name="step1">
+        <tasklet>
+            <chunk reader="fooReader" processor="fooProcessor" writer="barWriter"
+                   commit-interval="2"/>
+        </tasklet>
+    </step>
+</job>
+----
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Job ioSampleJob() {
+	return this.jobBuilderFactory.get("ioSampleJob")
+				.start(step1())
+				.end()
+				.build();
+}
+
+@Bean
+public Step step1() {
+	return this.stepBuilderFactory.get("step1")
+				.<Foo, Bar>chunk(2)
+				.reader(fooReader())
+				.processor(fooProcessor())
+				.writer(barWriter())
+				.build();
+}
+----
+
+A difference between `ItemProcessor` and `ItemReader` or `ItemWriter` is that an `ItemProcessor`
+is optional for a `Step`.
+
+[[chainingItemProcessors]]
+=== Chaining ItemProcessors
+
+Performing a single transformation is useful in many scenarios, but what if you want to
+'chain' together multiple `ItemProcessor` implementations? This can be accomplished using
+the composite pattern mentioned previously. To update the previous, single
+transformation, example, `Foo` is transformed to `Bar`, which is transformed to `Foobar`
+and written out, as shown in the following example:
+
+[source, java]
+----
+public class Foo {}
+
+public class Bar {
+    public Bar(Foo foo) {}
+}
+
+public class Foobar {
+    public Foobar(Bar bar) {}
+}
+
+public class FooProcessor implements ItemProcessor<Foo, Bar> {
+    public Bar process(Foo foo) throws Exception {
+        //Perform simple transformation, convert a Foo to a Bar
+        return new Bar(foo);
+    }
+}
+
+public class BarProcessor implements ItemProcessor<Bar, Foobar> {
+    public Foobar process(Bar bar) throws Exception {
+        return new Foobar(bar);
+    }
+}
+
+public class FoobarWriter implements ItemWriter<Foobar>{
+    public void write(List<? extends Foobar> items) throws Exception {
+        //write items
+    }
+}
+----
+
+A `FooProcessor` and a `BarProcessor` can be 'chained' together to give the resultant
+`Foobar`, as shown in the following example:
+
+
+[source, java]
+----
+CompositeItemProcessor<Foo,Foobar> compositeProcessor =
+                                      new CompositeItemProcessor<Foo,Foobar>();
+List itemProcessors = new ArrayList();
+itemProcessors.add(new FooProcessor());
+itemProcessors.add(new BarProcessor());
+compositeProcessor.setDelegates(itemProcessors);
+----
+
+Just as with the previous example, the composite processor can be configured into the
+`Step`:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<job id="ioSampleJob">
+    <step name="step1">
+        <tasklet>
+            <chunk reader="fooReader" processor="compositeItemProcessor" writer="foobarWriter"
+                   commit-interval="2"/>
+        </tasklet>
+    </step>
+</job>
+
+<bean id="compositeItemProcessor"
+      class="org.springframework.batch.item.support.CompositeItemProcessor">
+    <property name="delegates">
+        <list>
+            <bean class="..FooProcessor" />
+            <bean class="..BarProcessor" />
+        </list>
+    </property>
+</bean>
+----
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Job ioSampleJob() {
+	return this.jobBuilderFactory.get("ioSampleJob")
+				.start(step1())
+				.end()
+				.build();
+}
+
+@Bean
+public Step step1() {
+	return this.stepBuilderFactory.get("step1")
+				.<Foo, Foobar>chunk(2)
+				.reader(fooReader())
+				.processor(compositeProcessor())
+				.writer(foobarWriter())
+				.build();
+}
+
+@Bean
+public CompositeItemProcessor compositeProcessor() {
+	List<ItemProcessor> delegates = new ArrayList<>(2);
+	delegates.add(new FooProcessor());
+	delegates.add(new BarProcessor());
+
+	CompositeItemProcessor processor = new CompositeItemProcessor();
+
+	processor.setDelegates(delegates);
+
+	return processor;
+}
+----
+
+[[filteringRecords]]
+=== Filtering Records
+
+One typical use for an item processor is to filter out records before they are passed to
+the `ItemWriter`. Filtering is an action distinct from skipping. Skipping indicates that
+a record is invalid, while filtering simply indicates that a record should not be
+written.
+
+For example, consider a batch job that reads a file containing three different types of
+records: records to insert, records to update, and records to delete. If record deletion
+is not supported by the system, then we would not want to send any "delete" records to
+the `ItemWriter`. But, since these records are not actually bad records, we would want to
+filter them out rather than skip them. As a result, the `ItemWriter` would receive only
+"insert" and "update" records.
+
+To filter a record, you can return `null` from the `ItemProcessor`. The framework detects
+that the result is `null` and avoids adding that item to the list of records delivered to
+the `ItemWriter`. As usual, an exception thrown from the `ItemProcessor` results in a
+skip.
+
+[[validatingInput]]
+=== Validating Input
+
+In the <<readersAndWriters.adoc#readersAndWriters,ItemReaders and ItemWriters>> chapter, multiple approaches to parsing input have been
+discussed. Each major implementation throws an exception if it is not 'well-formed'. The
+`FixedLengthTokenizer` throws an exception if a range of data is missing. Similarly,
+attempting to access an index in a `RowMapper` or `FieldSetMapper` that does not exist or
+is in a different format than the one expected causes an exception to be thrown. All of
+these types of exceptions are thrown before `read` returns. However, they do not address
+the issue of whether or not the returned item is valid. For example, if one of the fields
+is an age, it obviously cannot be negative. It may parse correctly, because it exists and
+is a number, but it does not cause an exception. Since there are already a plethora of
+validation frameworks, Spring Batch does not attempt to provide yet another. Rather, it
+provides a simple interface, called `Validator`, that can be implemented by any number of
+frameworks, as shown in the following interface definition:
+
+[source, java]
+----
+public interface Validator<T> {
+
+    void validate(T value) throws ValidationException;
+
+}
+----
+
+The contract is that the `validate` method throws an exception if the object is invalid
+and returns normally if it is valid. Spring Batch provides an out of the box
+`ValidatingItemProcessor`, as shown in the following bean definition:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<bean class="org.springframework.batch.item.validator.ValidatingItemProcessor">
+    <property name="validator" ref="validator" />
+</bean>
+
+<bean id="validator" class="org.springframework.batch.item.validator.SpringValidator">
+	<property name="validator">
+		<bean class="org.springframework.batch.sample.domain.trade.internal.validator.TradeValidator"/>
+	</property>
+</bean>
+----
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public ValidatingItemProcessor itemProcessor() {
+	ValidatingItemProcessor processor = new ValidatingItemProcessor();
+
+	processor.setValidator(validator());
+
+	return processor;
+}
+
+@Bean
+public SpringValidator validator() {
+	SpringValidator validator = new SpringValidator();
+
+	validator.setValidator(new TradeValidator());
+
+	return validator;
+}
+----
+
+You can also use the `BeanValidatingItemProcessor` to validate items annotated with
+the Bean Validation API (JSR-303) annotations. For example, given the following type `Person`:
+
+[source, java]
+----
+class Person {
+
+    @NotEmpty
+    private String name;
+
+    public Person(String name) {
+     this.name = name;
+    }
+
+    public String getName() {
+     return name;
+    }
+
+    public void setName(String name) {
+     this.name = name;
+    }
+
+}
+----
+
+you can validate items by declaring a `BeanValidatingItemProcessor` bean in your
+application context and register it as a processor in your chunk-oriented step:
+
+[source, java]
+----
+@Bean
+public BeanValidatingItemProcessor<Person> beanValidatingItemProcessor() throws Exception {
+    BeanValidatingItemProcessor<Person> beanValidatingItemProcessor = new BeanValidatingItemProcessor<>();
+    beanValidatingItemProcessor.setFilter(true);
+
+    return beanValidatingItemProcessor;
+}
+----
+
+[[faultTolerant]]
+=== Fault Tolerance
+
+When a chunk is rolled back, items that have been cached during reading may be
+reprocessed.  If a step is configured to be fault tolerant (typically by using skip or
+retry processing), any `ItemProcessor` used should be implemented in a way that is
+idempotent.  Typically that would consist of performing no changes on the input item for
+the `ItemProcessor` and only updating the
+instance that is the result.

spring-batch-docs/src/main/asciidoc/repeat.adoc

@@ -0,0 +1,274 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[repeat]]
+
+== Repeat
+
+ifndef::onlyonetoggle[]
+include::toggle.adoc[]
+endif::onlyonetoggle[]
+
+[[repeatTemplate]]
+
+=== RepeatTemplate
+
+Batch processing is about repetitive actions, either as a simple optimization or as part
+of a job. To strategize and generalize the repetition and to provide what amounts to an
+iterator framework, Spring Batch has the `RepeatOperations` interface. The
+`RepeatOperations` interface has the following definition:
+
+[source, java]
+----
+public interface RepeatOperations {
+
+    RepeatStatus iterate(RepeatCallback callback) throws RepeatException;
+
+}
+----
+
+The callback is an interface, shown in the following definition, that lets you insert
+some business logic to be repeated:
+
+[source, java]
+----
+public interface RepeatCallback {
+
+    RepeatStatus doInIteration(RepeatContext context) throws Exception;
+
+}
+----
+
+The callback is executed repeatedly until the implementation determines that the
+iteration should end. The return value in these interfaces is an enumeration that can
+either be `RepeatStatus.CONTINUABLE` or `RepeatStatus.FINISHED`. A `RepeatStatus`
+enumeration conveys information to the caller of the repeat operations about whether
+there is any more work to do. Generally speaking, implementations of `RepeatOperations`
+should inspect the `RepeatStatus` and use it as part of the decision to end the
+iteration. Any callback that wishes to signal to the caller that there is no more work to
+do can return `RepeatStatus.FINISHED`.
+
+The simplest general purpose implementation of `RepeatOperations` is `RepeatTemplate`, as
+shown in the following example:
+
+
+[source, java]
+----
+RepeatTemplate template = new RepeatTemplate();
+
+template.setCompletionPolicy(new SimpleCompletionPolicy(2));
+
+template.iterate(new RepeatCallback() {
+
+    public RepeatStatus doInIteration(RepeatContext context) {
+        // Do stuff in batch...
+        return RepeatStatus.CONTINUABLE;
+    }
+
+});
+----
+
+In the preceding example, we return `RepeatStatus.CONTINUABLE`, to show that there is
+more work to do. The callback can also return `RepeatStatus.FINISHED`, to signal to the
+caller that there is no more work to do. Some iterations can be terminated by
+considerations intrinsic to the work being done in the callback. Others are effectively
+infinite loops as far as the callback is concerned and the completion decision is
+delegated to an external policy, as in the case shown in the preceding example.
+
+[[repeatContext]]
+
+==== RepeatContext
+
+The method parameter for the `RepeatCallback` is a `RepeatContext`. Many callbacks ignore
+the context. However, if necessary, it can be used as an attribute bag to store transient
+data for the duration of the iteration. After the `iterate` method returns, the context
+no longer exists.
+
+If there is a nested iteration in progress, a `RepeatContext` has a parent context. The
+parent context is occasionally useful for storing data that need to be shared between
+calls to `iterate`. This is the case, for instance, if you want to count the number of
+occurrences of an event in the iteration and remember it across subsequent calls.
+
+[[repeatStatus]]
+
+==== RepeatStatus
+
+`RepeatStatus` is an enumeration used by Spring Batch to indicate whether processing has
+finished. It has two possible `RepeatStatus` values, described in the following table:
+
+.RepeatStatus Properties
+
+|===============
+|__Value__|__Description__
+|CONTINUABLE|There is more work to do.
+|FINISHED|No more repetitions should take place.
+
+|===============
+
+`RepeatStatus` values can also be combined with a logical AND operation by using the
+`and()` method in `RepeatStatus`. The effect of this is to do a logical AND on the
+continuable flag. In other words, if either status is `FINISHED`, then the result is
+`FINISHED`.
+
+[[completionPolicies]]
+
+=== Completion Policies
+
+Inside a `RepeatTemplate`, the termination of the loop in the `iterate` method is
+determined by a `CompletionPolicy`, which is also a factory for the `RepeatContext`. The
+`RepeatTemplate` has the responsibility to use the current policy to create a
+`RepeatContext` and pass that in to the `RepeatCallback` at every stage in the iteration.
+After a callback completes its `doInIteration`, the `RepeatTemplate` has to make a call
+to the `CompletionPolicy` to ask it to update its state (which will be stored in the
+`RepeatContext`). Then it asks the policy if the iteration is complete.
+
+Spring Batch provides some simple general purpose implementations of `CompletionPolicy`.
+`SimpleCompletionPolicy` allows execution up to a fixed number of times (with
+`RepeatStatus.FINISHED` forcing early completion at any time).
+
+Users might need to implement their own completion policies for more complicated
+decisions. For example, a batch processing window that prevents batch jobs from executing
+once the online systems are in use would require a custom policy.
+
+[[repeatExceptionHandling]]
+
+=== Exception Handling
+
+If there is an exception thrown inside a `RepeatCallback`, the `RepeatTemplate` consults
+an `ExceptionHandler`, which can decide whether or not to re-throw the exception.
+
+The following listing shows the `ExceptionHandler` interface definition:
+
+[source, java]
+----
+public interface ExceptionHandler {
+
+    void handleException(RepeatContext context, Throwable throwable)
+        throws Throwable;
+
+}
+----
+
+A common use case is to count the number of exceptions of a given type and fail when a
+limit is reached. For this purpose, Spring Batch provides the
+`SimpleLimitExceptionHandler` and a slightly more flexible
+`RethrowOnThresholdExceptionHandler`. The `SimpleLimitExceptionHandler` has a limit
+property and an exception type that should be compared with the current exception. All
+subclasses of the provided type are also counted. Exceptions of the given type are
+ignored until the limit is reached, and then they are rethrown. Exceptions of other types
+are always rethrown.
+
+An important optional property of the `SimpleLimitExceptionHandler` is the boolean flag
+called `useParent`. It is `false` by default, so the limit is only accounted for in the
+current `RepeatContext`. When set to `true`, the limit is kept across sibling contexts in
+a nested iteration (such as a set of chunks inside a step).
+
+[[repeatListeners]]
+
+=== Listeners
+
+Often, it is useful to be able to receive additional callbacks for cross-cutting concerns
+across a number of different iterations. For this purpose, Spring Batch provides the
+`RepeatListener` interface. The `RepeatTemplate` lets users register `RepeatListener`
+implementations, and they are given callbacks with the `RepeatContext` and `RepeatStatus`
+where available during the iteration.
+
+The `RepeatListener` interface has the following definition:
+
+[source, java]
+----
+public interface RepeatListener {
+    void before(RepeatContext context);
+    void after(RepeatContext context, RepeatStatus result);
+    void open(RepeatContext context);
+    void onError(RepeatContext context, Throwable e);
+    void close(RepeatContext context);
+}
+----
+
+The `open` and `close` callbacks come before and after the entire iteration. `before`,
+`after`, and `onError` apply to the individual `RepeatCallback` calls.
+
+Note that, when there is more than one listener, they are in a list, so there is an
+order. In this case, `open` and `before` are called in the same order while `after`,
+`onError`, and `close` are called in reverse order.
+
+[[repeatParallelProcessing]]
+
+=== Parallel Processing
+
+Implementations of `RepeatOperations` are not restricted to executing the callback
+sequentially. It is quite important that some implementations are able to execute their
+callbacks in parallel. To this end, Spring Batch provides the
+`TaskExecutorRepeatTemplate`, which uses the Spring `TaskExecutor` strategy to run the
+`RepeatCallback`. The default is to use a `SynchronousTaskExecutor`, which has the effect
+of executing the whole iteration in the same thread (the same as a normal
+`RepeatTemplate`).
+
+[[declarativeIteration]]
+
+=== Declarative Iteration
+
+Sometimes there is some business processing that you know you want to repeat every time
+it happens. The classic example of this is the optimization of a message pipeline. It is
+more efficient to process a batch of messages, if they are arriving frequently, than to
+bear the cost of a separate transaction for every message. Spring Batch provides an AOP
+interceptor that wraps a method call in a `RepeatOperations` object for just this
+purpose. The `RepeatOperationsInterceptor` executes the intercepted method and repeats
+according to the `CompletionPolicy` in the provided `RepeatTemplate`.
+
+[role="xmlContent"]
+The following example shows declarative iteration using the Spring AOP namespace to
+repeat a service call to a method called `processMessage` (for more detail on how to
+configure AOP interceptors, see the Spring User Guide):
+
+[source, xml, role="xmlContent"]
+----
+<aop:config>
+    <aop:pointcut id="transactional"
+        expression="execution(* com..*Service.processMessage(..))" />
+    <aop:advisor pointcut-ref="transactional"
+        advice-ref="retryAdvice" order="-1"/>
+</aop:config>
+
+<bean id="retryAdvice" class="org.spr...RepeatOperationsInterceptor"/>
+----
+
+[role="javaContent"]
+The following example demonstrates using Java configuration to
+repeat a service call to a method called `processMessage` (for more detail on how to
+configure AOP interceptors, see the Spring User Guide):
+
+[source, java, role="javaContent"]
+----
+@Bean
+public MyService myService() {
+	ProxyFactory factory = new ProxyFactory(RepeatOperations.class.getClassLoader());
+	factory.setInterfaces(MyService.class);
+	factory.setTarget(new MyService());
+
+	MyService service = (MyService) factory.getProxy();
+	JdkRegexpMethodPointcut pointcut = new JdkRegexpMethodPointcut();
+	pointcut.setPatterns(".*processMessage.*");
+
+	RepeatOperationsInterceptor interceptor = new RepeatOperationsInterceptor();
+
+	((Advised) service).addAdvisor(new DefaultPointcutAdvisor(pointcut, interceptor));
+
+	return service;
+}
+----
+
+The preceding example uses a default `RepeatTemplate` inside the interceptor. To change
+the policies, listeners, and other details, you can inject an instance of
+`RepeatTemplate` into the interceptor.
+
+If the intercepted method returns `void`, then the interceptor always returns
+`RepeatStatus.CONTINUABLE` (so there is a danger of an infinite loop if the
+`CompletionPolicy` does not have a finite end point). Otherwise, it returns
+`RepeatStatus.CONTINUABLE` until the return value from the intercepted method is `null`,
+at which point it returns `RepeatStatus.FINISHED`. Consequently, the business logic
+inside the target method can signal that there is no more work to do by returning `null`
+or by throwing an exception that is re-thrown by the `ExceptionHandler` in the provided
+`RepeatTemplate`.

spring-batch-docs/src/main/asciidoc/retry.adoc

@@ -0,0 +1,22 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[retry]]
+
+== Retry
+
+To make processing more robust and less prone to failure, it sometimes helps to
+automatically retry a failed operation in case it might succeed on a subsequent attempt.
+Errors that are susceptible to intermittent failure are often transient in nature.
+Examples include remote calls to a web service that fails because of a network glitch or a
+`DeadlockLoserDataAccessException` in a database update.
+
+[NOTE]
+====
+The retry functionality was pulled out of Spring Batch as of 2.2.0.
+It is now part of a new library, https://github.com/spring-projects/spring-retry[Spring Retry].
+Spring Batch still relies on Spring Retry to automate retry operations within the framework.
+Please refer to the reference documentation of Spring Retry for details about
+key APIs and how to use them.
+====

spring-batch-docs/src/main/asciidoc/scalability.adoc

@@ -0,0 +1,503 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[scalability]]
+
+== Scaling and Parallel Processing
+
+ifndef::onlyonetoggle[]
+include::toggle.adoc[]
+endif::onlyonetoggle[]
+
+Many batch processing problems can be solved with single threaded, single process jobs,
+so it is always a good idea to properly check if that meets your needs before thinking
+about more complex implementations. Measure the performance of a realistic job and see if
+the simplest implementation meets your needs first. You can read and write a file of
+several hundred megabytes in well under a minute, even with standard hardware.
+
+When you are ready to start implementing a job with some parallel processing, Spring
+Batch offers a range of options, which are described in this chapter, although some
+features are covered elsewhere. At a high level, there are two modes of parallel
+processing:
+
+* Single process, multi-threaded
+* Multi-process
+
+These break down into categories as well, as follows:
+
+* Multi-threaded Step (single process)
+* Parallel Steps (single process)
+* Remote Chunking of Step (multi process)
+* Partitioning a Step (single or multi process)
+
+First, we review the single-process options. Then we review the multi-process options.
+
+[[multithreadedStep]]
+
+=== Multi-threaded Step
+
+The simplest way to start parallel processing is to add a `TaskExecutor` to your Step
+configuration.
+
+[role="xmlContent"]
+For example, you might add an attribute of the `tasklet`, as follows:
+
+[source, xml, role="xmlContent"]
+----
+<step id="loading">
+    <tasklet task-executor="taskExecutor">...</tasklet>
+</step>
+----
+
+[role="javaContent"]
+When using java configuration, a `TaskExecutor` can be added to the step,
+as shown in the following example:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public TaskExecutor taskExecutor() {
+    return new SimpleAsyncTaskExecutor("spring_batch");
+}
+
+@Bean
+public Step sampleStep(TaskExecutor taskExecutor) {
+	return this.stepBuilderFactory.get("sampleStep")
+				.<String, String>chunk(10)
+				.reader(itemReader())
+				.writer(itemWriter())
+				.taskExecutor(taskExecutor)
+				.build();
+}
+----
+
+In this example, the `taskExecutor` is a reference to another bean definition that
+implements the `TaskExecutor` interface.
+https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/core/task/TaskExecutor.html[`TaskExecutor`]
+is a standard Spring interface, so consult the Spring User Guide for details of available
+implementations. The simplest multi-threaded `TaskExecutor` is a
+`SimpleAsyncTaskExecutor`.
+
+The result of the above configuration is that the `Step` executes by reading, processing,
+and writing each chunk of items (each commit interval) in a separate thread of execution.
+Note that this means there is no fixed order for the items to be processed, and a chunk
+might contain items that are non-consecutive compared to the single-threaded case. In
+addition to any limits placed by the task executor (such as whether it is backed by a
+thread pool), there is a throttle limit in the tasklet configuration which defaults to 4.
+You may need to increase this to ensure that a thread pool is fully utilized.
+
+[role="xmlContent"]
+For example you might increase the throttle-limit, as shown in the following example:
+
+[source, xml, role="xmlContent"]
+----
+<step id="loading"> <tasklet
+    task-executor="taskExecutor"
+    throttle-limit="20">...</tasklet>
+</step>
+----
+
+[role="javaContent"]
+When using Java configuration, the builders provide access to the throttle limit, as shown
+in the following example:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Step sampleStep(TaskExecutor taskExecutor) {
+	return this.stepBuilderFactory.get("sampleStep")
+				.<String, String>chunk(10)
+				.reader(itemReader())
+				.writer(itemWriter())
+				.taskExecutor(taskExecutor)
+				.throttleLimit(20)
+				.build();
+}
+----
+
+Note also that there may be limits placed on concurrency by any pooled resources used in
+your step, such as a `DataSource`.  Be sure to make the pool in those resources at least
+as large as the desired number of concurrent threads in the step.
+
+There are some practical limitations of using multi-threaded `Step` implementations for
+some common batch use cases. Many participants in a `Step` (such as readers and writers)
+are stateful. If the state is not segregated by thread, then those components are not
+usable in a multi-threaded `Step`. In particular, most of the off-the-shelf readers and
+writers from Spring Batch are not designed for multi-threaded use. It is, however,
+possible to work with stateless or thread safe readers and writers, and there is a sample
+(called `parallelJob`) in the
+https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples[Spring
+Batch Samples] that shows the use of a process indicator (see
+<<readersAndWriters.adoc#process-indicator,Preventing State Persistence>>) to keep track
+of items that have been processed in a database input table.
+
+Spring Batch provides some implementations of `ItemWriter` and `ItemReader`.  Usually,
+they say in the Javadoc if they are thread safe or not or what you have to do to avoid
+problems in a concurrent environment.  If there is no information in the Javadoc, you can
+check the implementation to see if there is any state.  If a reader is not thread safe,
+you can decorate it with the provided `SynchronizedItemStreamReader` or use it in your own
+synchronizing delegator. You can synchronize the call to `read()` and as long as the
+processing and writing is the most expensive part of the chunk, your step may still
+complete much faster than it would in a single threaded configuration.
+
+[[scalabilityParallelSteps]]
+
+
+=== Parallel Steps
+
+As long as the application logic that needs to be parallelized can be split into distinct
+responsibilities and assigned to individual steps, then it can be parallelized in a
+single process. Parallel Step execution is easy to configure and use.
+
+[role="xmlContent"]
+For example, executing steps `(step1,step2)` in parallel with `step3` is straightforward,
+as shown in the following example:
+
+[source, xml, role="xmlContent"]
+----
+<job id="job1">
+    <split id="split1" task-executor="taskExecutor" next="step4">
+        <flow>
+            <step id="step1" parent="s1" next="step2"/>
+            <step id="step2" parent="s2"/>
+        </flow>
+        <flow>
+            <step id="step3" parent="s3"/>
+        </flow>
+    </split>
+    <step id="step4" parent="s4"/>
+</job>
+
+<beans:bean id="taskExecutor" class="org.spr...SimpleAsyncTaskExecutor"/>
+----
+
+[role="javaContent"]
+When using Java configuration, executing steps `(step1,step2)` in parallel with `step3`
+is straightforward, as shown in the following example:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Job job() {
+    return jobBuilderFactory.get("job")
+        .start(splitFlow())
+        .next(step4())
+        .build()        //builds FlowJobBuilder instance
+        .build();       //builds Job instance
+}
+
+@Bean
+public Flow splitFlow() {
+    return new FlowBuilder<SimpleFlow>("splitFlow")
+        .split(taskExecutor())
+        .add(flow1(), flow2())
+        .build();
+}
+
+@Bean
+public Flow flow1() {
+    return new FlowBuilder<SimpleFlow>("flow1")
+        .start(step1())
+        .next(step2())
+        .build();
+}
+
+@Bean
+public Flow flow2() {
+    return new FlowBuilder<SimpleFlow>("flow2")
+        .start(step3())
+        .build();
+}
+
+@Bean
+public TaskExecutor taskExecutor() {
+    return new SimpleAsyncTaskExecutor("spring_batch");
+}
+----
+
+The configurable task executor is used to specify which `TaskExecutor`
+implementation should be used to execute the individual flows. The default is
+`SyncTaskExecutor`, but an asynchronous `TaskExecutor` is required to run the steps in
+parallel. Note that the job ensures that every flow in the split completes before
+aggregating the exit statuses and transitioning.
+
+See the section on <<step.adoc#split-flows,Split Flows>> for more detail.
+
+[[remoteChunking]]
+
+=== Remote Chunking
+
+In remote chunking, the `Step` processing is split across multiple processes,
+communicating with each other through some middleware. The following image shows the
+pattern:
+
+.Remote Chunking
+image::{batch-asciidoc}images/remote-chunking.png[Remote Chunking, scaledwidth="60%"]
+
+The manager component is a single process, and the workers are multiple remote processes.
+This pattern works best if the manager is not a bottleneck, so the processing must be more
+expensive than the reading of items (as is often the case in practice).
+
+The manager is an implementation of a Spring Batch `Step` with the `ItemWriter` replaced
+by a generic version that knows how to send chunks of items to the middleware as
+messages. The workers are standard listeners for whatever middleware is being used (for
+example, with JMS, they would be `MesssageListener` implementations), and their role is
+to process the chunks of items using a standard `ItemWriter` or `ItemProcessor` plus
+`ItemWriter`, through the `ChunkProcessor` interface. One of the advantages of using this
+pattern is that the reader, processor, and writer components are off-the-shelf (the same
+as would be used for a local execution of the step). The items are divided up dynamically
+and work is shared through the middleware, so that, if the listeners are all eager
+consumers, then load balancing is automatic.
+
+The middleware has to be durable, with guaranteed delivery and a single consumer for each
+message. JMS is the obvious candidate, but other options (such as JavaSpaces) exist in
+the grid computing and shared memory product space.
+
+See the section on
+<<spring-batch-integration.adoc#remote-chunking,Spring Batch Integration - Remote Chunking>>
+for more detail.
+
+[[partitioning]]
+
+=== Partitioning
+
+Spring Batch also provides an SPI for partitioning a `Step` execution and executing it
+remotely. In this case, the remote participants are `Step` instances that could just as
+easily have been configured and used for local processing. The following image shows the
+pattern:
+
+.Partitioning
+image::{batch-asciidoc}images/partitioning-overview.png[Partitioning Overview, scaledwidth="60%"]
+
+The `Job` runs on the left-hand side as a sequence of `Step` instances, and one of the
+`Step` instances is labeled as a manager. The workers in this picture are all identical
+instances of a `Step`, which could in fact take the place of the manager, resulting in the
+same outcome for the `Job`. The workers are typically going to be remote services but
+could also be local threads of execution. The messages sent by the manager to the workers
+in this pattern do not need to be durable or have guaranteed delivery. Spring Batch
+metadata in the `JobRepository` ensures that each worker is executed once and only once for
+each `Job` execution.
+
+The SPI in Spring Batch consists of a special implementation of `Step` (called the
+`PartitionStep`) and two strategy interfaces that need to be implemented for the specific
+environment. The strategy interfaces are `PartitionHandler` and `StepExecutionSplitter`,
+and their role is shown in the following sequence diagram:
+
+.Partitioning SPI
+image::{batch-asciidoc}images/partitioning-spi.png[Partitioning SPI, scaledwidth="60%"]
+
+The `Step` on the right in this case is the "`remote`" worker, so, potentially, there are
+many objects and or processes playing this role, and the `PartitionStep` is shown driving
+the execution.
+
+[role="xmlContent"]
+The following example shows the `PartitionStep` configuration when using XML
+configuration:
+
+[source, xml, role="xmlContent"]
+----
+<step id="step1.manager">
+    <partition step="step1" partitioner="partitioner">
+        <handler grid-size="10" task-executor="taskExecutor"/>
+    </partition>
+</step>
+----
+
+[role="javaContent"]
+The following example shows the `PartitionStep` configuration when using Java
+configuration:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Step step1Manager() {
+    return stepBuilderFactory.get("step1.manager")
+        .<String, String>partitioner("step1", partitioner())
+        .step(step1())
+        .gridSize(10)
+        .taskExecutor(taskExecutor())
+        .build();
+}
+----
+
+Similar to the multi-threaded step's `throttle-limit` attribute, the `grid-size`
+attribute prevents the task executor from being saturated with requests from a single
+step.
+
+There is a simple example that can be copied and extended in the unit test suite for
+https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples/src/main/resources/jobs[Spring
+Batch Samples] (see `partition*Job.xml` configuration).
+
+Spring Batch creates step executions for the partitions called "step1:partition0", and so
+on. Many people prefer to call the manager step "step1:manager" for consistency. You can
+use an alias for the step (by specifying the `name` attribute instead of the `id`
+attribute).
+
+[[partitionHandler]]
+
+==== PartitionHandler
+
+The `PartitionHandler` is the component that knows about the fabric of the remoting or
+grid environment. It is able to send `StepExecution` requests to the remote `Step`
+instances, wrapped in some fabric-specific format, like a DTO. It does not have to know
+how to split the input data or how to aggregate the result of multiple `Step` executions.
+Generally speaking, it probably also does not need to know about resilience or failover,
+since those are features of the fabric in many cases. In any case, Spring Batch always
+provides restartability independent of the fabric. A failed `Job` can always be restarted
+and only the failed `Steps` are re-executed.
+
+The `PartitionHandler` interface can have specialized implementations for a variety of
+fabric types, including simple RMI remoting, EJB remoting, custom web service, JMS, Java
+Spaces, shared memory grids (like Terracotta or Coherence), and grid execution fabrics
+(like GridGain). Spring Batch does not contain implementations for any proprietary grid
+or remoting fabrics.
+
+Spring Batch does, however, provide a useful implementation of `PartitionHandler` that
+executes `Step` instances locally in separate threads of execution, using the
+`TaskExecutor` strategy from Spring. The implementation is called
+`TaskExecutorPartitionHandler`.
+
+[role="xmlContent"]
+The `TaskExecutorPartitionHandler` is the default for a step configured with the XML
+namespace shown previously. It can also be configured explicitly, as shown in the
+following example:
+
+[source, xml, role="xmlContent"]
+----
+<step id="step1.manager">
+    <partition step="step1" handler="handler"/>
+</step>
+
+<bean class="org.spr...TaskExecutorPartitionHandler">
+    <property name="taskExecutor" ref="taskExecutor"/>
+    <property name="step" ref="step1" />
+    <property name="gridSize" value="10" />
+</bean>
+----
+
+[role="javaContent"]
+The `TaskExecutorPartitionHandler` can be configured explicitly within java configuration,
+as shown in the following example:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public Step step1Manager() {
+    return stepBuilderFactory.get("step1.manager")
+        .partitioner("step1", partitioner())
+        .partitionHandler(partitionHandler())
+        .build();
+}
+
+@Bean
+public PartitionHandler partitionHandler() {
+    TaskExecutorPartitionHandler retVal = new TaskExecutorPartitionHandler();
+    retVal.setTaskExecutor(taskExecutor());
+    retVal.setStep(step1());
+    retVal.setGridSize(10);
+    return retVal;
+}
+----
+
+The `gridSize` attribute determines the number of separate step executions to create, so
+it can be matched to the size of the thread pool in the `TaskExecutor`. Alternatively, it
+can be set to be larger than the number of threads available, which makes the blocks of
+work smaller.
+
+The `TaskExecutorPartitionHandler` is useful for IO-intensive `Step` instances, such as
+copying large numbers of files or replicating filesystems into content management
+systems. It can also be used for remote execution by providing a `Step` implementation
+that is a proxy for a remote invocation (such as using Spring Remoting).
+
+[[partitioner]]
+
+==== Partitioner
+
+The `Partitioner` has a simpler responsibility: to generate execution contexts as input
+parameters for new step executions only (no need to worry about restarts). It has a
+single method, as shown in the following interface definition:
+
+[source, java]
+----
+public interface Partitioner {
+    Map<String, ExecutionContext> partition(int gridSize);
+}
+----
+
+The return value from this method associates a unique name for each step execution (the
+`String`) with input parameters in the form of an `ExecutionContext`. The names show up
+later in the Batch metadata as the step name in the partitioned `StepExecutions`. The
+`ExecutionContext` is just a bag of name-value pairs, so it might contain a range of
+primary keys, line numbers, or the location of an input file. The remote `Step` then
+normally binds to the context input using `#{...}` placeholders (late binding in step
+scope), as illustrated in the next section.
+
+The names of the step executions (the keys in the `Map` returned by `Partitioner`) need
+to be unique amongst the step executions of a `Job` but do not have any other specific
+requirements. The easiest way to do this (and to make the names meaningful for users) is
+to use a prefix+suffix naming convention, where the prefix is the name of the step that
+is being executed (which itself is unique in the `Job`), and the suffix is just a
+counter. There is a `SimplePartitioner` in the framework that uses this convention.
+
+An optional interface called `PartitionNameProvider` can be used to provide the partition
+names separately from the partitions themselves. If a `Partitioner` implements this
+interface, then, on a restart, only the names are queried. If partitioning is expensive,
+this can be a useful optimization. The names provided by the `PartitionNameProvider` must
+match those provided by the `Partitioner`.
+
+[[bindingInputDataToSteps]]
+
+==== Binding Input Data to Steps
+
+It is very efficient for the steps that are executed by the `PartitionHandler` to have
+identical configuration and for their input parameters to be bound at runtime from the
+`ExecutionContext`. This is easy to do with the StepScope feature of Spring Batch
+(covered in more detail in the section on <<step.adoc#late-binding,Late Binding>>). For
+example, if the `Partitioner` creates `ExecutionContext` instances with an attribute key
+called `fileName`, pointing to a different file (or directory) for each step invocation,
+the `Partitioner` output might resemble the content of the following table:
+
+.Example step execution name to execution context provided by `Partitioner` targeting directory processing
+|===============
+|__Step Execution Name (key)__|__ExecutionContext (value)__
+|filecopy:partition0|fileName=/home/data/one
+|filecopy:partition1|fileName=/home/data/two
+|filecopy:partition2|fileName=/home/data/three
+|===============
+
+Then the file name can be bound to a step using late binding to the execution context.
+
+[role="xmlContent"]
+The following example shows how to define late binding in XML:
+
+.XML Configuration
+[source, xml, role="xmlContent"]
+----
+<bean id="itemReader" scope="step"
+      class="org.spr...MultiResourceItemReader">
+    <property name="resources" value="#{stepExecutionContext[fileName]}/*"/>
+</bean>
+----
+
+[role="xmlContent"]
+The following example shows how to define late binding in Java:
+
+.Java Configuration
+[source, java, role="javaContent"]
+----
+@Bean
+public MultiResourceItemReader itemReader(
+	@Value("#{stepExecutionContext['fileName']}/*") Resource [] resources) {
+	return new MultiResourceItemReaderBuilder<String>()
+			.delegate(fileReader())
+			.name("itemReader")
+			.resources(resources)
+			.build();
+}
+----

spring-batch-docs/src/main/asciidoc/schema-appendix.adoc

@@ -0,0 +1,403 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[metaDataSchema]]
+[appendix]
+== Meta-Data Schema
+
+[[metaDataSchemaOverview]]
+=== Overview
+
+The Spring Batch Metadata tables closely match the Domain objects that represent them in
+Java. For example, `JobInstance`, `JobExecution`, `JobParameters`, and `StepExecution`
+map to `BATCH_JOB_INSTANCE`, `BATCH_JOB_EXECUTION`, `BATCH_JOB_EXECUTION_PARAMS`, and
+`BATCH_STEP_EXECUTION`, respectively. `ExecutionContext` maps to both
+`BATCH_JOB_EXECUTION_CONTEXT` and `BATCH_STEP_EXECUTION_CONTEXT`. The `JobRepository` is
+responsible for saving and storing each Java object into its correct table. This appendix
+describes the metadata tables in detail, along with many of the design decisions that
+were made when creating them. When viewing the various table creation statements below,
+it is important to realize that the data types used are as generic as possible. Spring
+Batch provides many schemas as examples, all of which have varying data types, due to
+variations in how individual database vendors handle data types. The following image
+shows an ERD model of all 6 tables and their relationships to one another:
+
+.Spring Batch Meta-Data ERD
+image::{batch-asciidoc}images/meta-data-erd.png[Spring Batch Meta-Data ERD, scaledwidth="60%"]
+
+[[exampleDDLScripts]]
+==== Example DDL Scripts
+
+The Spring Batch Core JAR file contains example scripts to create the relational tables
+for a number of database platforms (which are, in turn, auto-detected by the job
+repository factory bean or namespace equivalent). These scripts can be used as is or
+modified with additional indexes and constraints as desired. The file names are in the
+form `schema-\*.sql`, where "*" is the short name of the target database platform.
+The scripts are in the package `org.springframework.batch.core`.
+
+[[migrationDDLScripts]]
+==== Migration DDL Scripts
+
+Spring Batch provides migration DDL scripts that you need to execute when you upgrade versions.
+These scripts can be found in the Core Jar file under `org/springframework/batch/core/migration`.
+Migration scripts are organized into folders corresponding to version numbers in which they were introduced:
+
+* `2.2`: contains scripts needed if you are migrating from a version before `2.2` to version `2.2`
+* `4.1`: contains scripts needed if you are migrating from a version before `4.1` to version `4.1`
+
+[[metaDataVersion]]
+==== Version
+
+Many of the database tables discussed in this appendix contain a version column. This
+column is important because Spring Batch employs an optimistic locking strategy when
+dealing with updates to the database. This means that each time a record is 'touched'
+(updated) the value in the version column is incremented by one. When the repository goes
+back to save the value, if the version number has changed it throws an
+`OptimisticLockingFailureException`, indicating there has been an error with concurrent
+access. This check is necessary, since, even though different batch jobs may be running
+in different machines, they all use the same database tables.
+
+[[metaDataIdentity]]
+==== Identity
+
+`BATCH_JOB_INSTANCE`, `BATCH_JOB_EXECUTION`, and `BATCH_STEP_EXECUTION` each contain
+columns ending in `_ID`. These fields act as primary keys for their respective tables.
+However, they are not database generated keys. Rather, they are generated by separate
+sequences. This is necessary because, after inserting one of the domain objects into the
+database, the key it is given needs to be set on the actual object so that they can be
+uniquely identified in Java. Newer database drivers (JDBC 3.0 and up) support this
+feature with database-generated keys. However, rather than require that feature,
+sequences are used. Each variation of the schema contains some form of the following
+statements:
+
+[source, sql]
+----
+CREATE SEQUENCE BATCH_STEP_EXECUTION_SEQ;
+CREATE SEQUENCE BATCH_JOB_EXECUTION_SEQ;
+CREATE SEQUENCE BATCH_JOB_SEQ;
+----
+
+Many database vendors do not support sequences. In these cases, work-arounds are used,
+such as the following statements for MySQL:
+
+[source, sql]
+----
+CREATE TABLE BATCH_STEP_EXECUTION_SEQ (ID BIGINT NOT NULL) type=InnoDB;
+INSERT INTO BATCH_STEP_EXECUTION_SEQ values(0);
+CREATE TABLE BATCH_JOB_EXECUTION_SEQ (ID BIGINT NOT NULL) type=InnoDB;
+INSERT INTO BATCH_JOB_EXECUTION_SEQ values(0);
+CREATE TABLE BATCH_JOB_SEQ (ID BIGINT NOT NULL) type=InnoDB;
+INSERT INTO BATCH_JOB_SEQ values(0);
+----
+
+In the preceding case, a table is used in place of each sequence. The Spring core class,
+`MySQLMaxValueIncrementer`, then increments the one column in this sequence in order to
+give similar functionality.
+
+[[metaDataBatchJobInstance]]
+=== `BATCH_JOB_INSTANCE`
+
+The `BATCH_JOB_INSTANCE` table holds all information relevant to a `JobInstance`, and
+serves as the top of the overall hierarchy. The following generic DDL statement is used
+to create it:
+
+[source, sql]
+----
+CREATE TABLE BATCH_JOB_INSTANCE  (
+  JOB_INSTANCE_ID BIGINT  PRIMARY KEY ,
+  VERSION BIGINT,
+  JOB_NAME VARCHAR(100) NOT NULL ,
+  JOB_KEY VARCHAR(2500)
+);
+----
+
+The following list describes each column in the table:
+
+* `JOB_INSTANCE_ID`: The unique ID that identifies the instance. It is also the primary
+key. The value of this column should be obtainable by calling the `getId` method on
+`JobInstance`.
+* `VERSION`: See <<metaDataVersion>>.
+* `JOB_NAME`: Name of the job obtained from the `Job` object. Because it is required to
+identify the instance, it must not be null.
+* `JOB_KEY`: A serialization of the `JobParameters` that uniquely identifies separate
+instances of the same job from one another. (`JobInstances` with the same job name must
+have different `JobParameters` and, thus, different `JOB_KEY` values).
+
+[[metaDataBatchJobParams]]
+=== `BATCH_JOB_EXECUTION_PARAMS`
+
+The `BATCH_JOB_EXECUTION_PARAMS` table holds all information relevant to the
+`JobParameters` object. It contains 0 or more key/value pairs passed to a `Job` and
+serves as a record of the parameters with which a job was run. For each parameter that
+contributes to the generation of a job's identity, the `IDENTIFYING` flag is set to true.
+Note that the table has been denormalized. Rather than creating a separate table for each
+type, there is one table with a column indicating the type, as shown in the following
+listing:
+
+[source, sql]
+----
+CREATE TABLE BATCH_JOB_EXECUTION_PARAMS  (
+	JOB_EXECUTION_ID BIGINT NOT NULL ,
+	TYPE_CD VARCHAR(6) NOT NULL ,
+	KEY_NAME VARCHAR(100) NOT NULL ,
+	STRING_VAL VARCHAR(250) ,
+	DATE_VAL DATETIME DEFAULT NULL ,
+	LONG_VAL BIGINT ,
+	DOUBLE_VAL DOUBLE PRECISION ,
+	IDENTIFYING CHAR(1) NOT NULL ,
+	constraint JOB_EXEC_PARAMS_FK foreign key (JOB_EXECUTION_ID)
+	references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID)
+);
+----
+
+The following list describes each column:
+
+* `JOB_EXECUTION_ID`: Foreign key from the `BATCH_JOB_EXECUTION` table that indicates the
+job execution to which the parameter entry belongs. Note that multiple rows (that is,
+key/value pairs) may exist for each execution.
+* TYPE_CD: String representation of the type of value stored, which can be a string, a
+date, a long, or a double. Because the type must be known, it cannot be null.
+* KEY_NAME: The parameter key.
+* STRING_VAL: Parameter value, if the type is string.
+* DATE_VAL: Parameter value, if the type is date.
+* LONG_VAL: Parameter value, if the type is long.
+* DOUBLE_VAL: Parameter value, if the type is double.
+* IDENTIFYING: Flag indicating whether the parameter contributed to the identity of the
+related `JobInstance`.
+
+Note that there is no primary key for this table. This is because the framework has no
+use for one and, thus, does not require it. If need be, you can add a primary key may be
+added with a database generated key without causing any issues to the framework itself.
+
+[[metaDataBatchJobExecution]]
+=== `BATCH_JOB_EXECUTION`
+
+The `BATCH_JOB_EXECUTION` table holds all information relevant to the `JobExecution`
+object. Every time a `Job` is run, there is always a new `JobExecution`, and a new row in
+this table. The following listing shows the definition of the `BATCH_JOB_EXECUTION`
+table:
+
+[source, sql]
+----
+CREATE TABLE BATCH_JOB_EXECUTION  (
+  JOB_EXECUTION_ID BIGINT  PRIMARY KEY ,
+  VERSION BIGINT,
+  JOB_INSTANCE_ID BIGINT NOT NULL,
+  CREATE_TIME TIMESTAMP NOT NULL,
+  START_TIME TIMESTAMP DEFAULT NULL,
+  END_TIME TIMESTAMP DEFAULT NULL,
+  STATUS VARCHAR(10),
+  EXIT_CODE VARCHAR(20),
+  EXIT_MESSAGE VARCHAR(2500),
+  LAST_UPDATED TIMESTAMP,
+  JOB_CONFIGURATION_LOCATION VARCHAR(2500) NULL,
+  constraint JOB_INSTANCE_EXECUTION_FK foreign key (JOB_INSTANCE_ID)
+  references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID)
+) ;
+----
+
+The following list describes each column:
+
+* `JOB_EXECUTION_ID`: Primary key that uniquely identifies this execution. The value of
+this column is obtainable by calling the `getId` method of the `JobExecution` object.
+* `VERSION`: See <<metaDataVersion>>.
+* `JOB_INSTANCE_ID`: Foreign key from the `BATCH_JOB_INSTANCE` table. It indicates the
+instance to which this execution belongs. There may be more than one execution per
+instance.
+* `CREATE_TIME`: Timestamp representing the time when the execution was created.
+* `START_TIME`: Timestamp representing the time when the execution was started.
+* `END_TIME`: Timestamp representing the time when the execution finished, regardless of
+success or failure. An empty value in this column when the job is not currently running
+indicates that there has been some type of error and the framework was unable to perform
+a last save before failing.
+* `STATUS`: Character string representing the status of the execution. This may be
+`COMPLETED`, `STARTED`, and others. The object representation of this column is the
+`BatchStatus` enumeration.
+* `EXIT_CODE`: Character string representing the exit code of the execution. In the case
+of a command-line job, this may be converted into a number.
+* `EXIT_MESSAGE`: Character string representing a more detailed description of how the
+job exited. In the case of failure, this might include as much of the stack trace as is
+possible.
+* `LAST_UPDATED`: Timestamp representing the last time this execution was persisted.
+
+[[metaDataBatchStepExecution]]
+=== `BATCH_STEP_EXECUTION`
+
+The BATCH_STEP_EXECUTION table holds all information relevant to the `StepExecution`
+object. This table is similar in many ways to the `BATCH_JOB_EXECUTION` table, and there
+is always at least one entry per `Step` for each `JobExecution` created. The following
+listing shows the definition of the `BATCH_STEP_EXECUTION` table:
+
+[source, sql]
+----
+CREATE TABLE BATCH_STEP_EXECUTION  (
+  STEP_EXECUTION_ID BIGINT  PRIMARY KEY ,
+  VERSION BIGINT NOT NULL,
+  STEP_NAME VARCHAR(100) NOT NULL,
+  JOB_EXECUTION_ID BIGINT NOT NULL,
+  START_TIME TIMESTAMP NOT NULL ,
+  END_TIME TIMESTAMP DEFAULT NULL,
+  STATUS VARCHAR(10),
+  COMMIT_COUNT BIGINT ,
+  READ_COUNT BIGINT ,
+  FILTER_COUNT BIGINT ,
+  WRITE_COUNT BIGINT ,
+  READ_SKIP_COUNT BIGINT ,
+  WRITE_SKIP_COUNT BIGINT ,
+  PROCESS_SKIP_COUNT BIGINT ,
+  ROLLBACK_COUNT BIGINT ,
+  EXIT_CODE VARCHAR(20) ,
+  EXIT_MESSAGE VARCHAR(2500) ,
+  LAST_UPDATED TIMESTAMP,
+  constraint JOB_EXECUTION_STEP_FK foreign key (JOB_EXECUTION_ID)
+  references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID)
+) ;
+----
+
+The following list describes for each column:
+
+* `STEP_EXECUTION_ID`: Primary key that uniquely identifies this execution. The value of
+this column should be obtainable by calling the `getId` method of the `StepExecution`
+object.
+* `VERSION`: See <<metaDataVersion>>.
+* `STEP_NAME`: The name of the step to which this execution belongs.
+* `JOB_EXECUTION_ID`: Foreign key from the `BATCH_JOB_EXECUTION` table. It indicates the
+`JobExecution` to which this `StepExecution` belongs. There may be only one
+`StepExecution` for a given `JobExecution` for a given `Step` name.
+* `START_TIME`: Timestamp representing the time when the execution was started.
+* `END_TIME`: Timestamp representing the time the when execution was finished, regardless
+of success or failure. An empty value in this column, even though the job is not
+currently running, indicates that there has been some type of error and the framework was
+unable to perform a last save before failing.
+* `STATUS`: Character string representing the status of the execution. This may be
+`COMPLETED`, `STARTED`, and others. The object representation of this column is the
+`BatchStatus` enumeration.
+* `COMMIT_COUNT`: The number of times in which the step has committed a transaction
+during this execution.
+* `READ_COUNT`: The number of items read during this execution.
+* `FILTER_COUNT`: The number of items filtered out of this execution.
+* `WRITE_COUNT`: The number of items written and committed during this execution.
+* `READ_SKIP_COUNT`: The number of items skipped on read during this execution.
+* `WRITE_SKIP_COUNT`: The number of items skipped on write during this execution.
+* `PROCESS_SKIP_COUNT`: The number of items skipped during processing during this
+execution.
+* `ROLLBACK_COUNT`: The number of rollbacks during this execution. Note that this count
+includes each time rollback occurs, including rollbacks for retry and those in the skip
+recovery procedure.
+* `EXIT_CODE`: Character string representing the exit code of the execution. In the case
+of a command-line job, this may be converted into a number.
+* `EXIT_MESSAGE`: Character string representing a more detailed description of how the
+job exited. In the case of failure, this might include as much of the stack trace as is
+possible.
+* `LAST_UPDATED`: Timestamp representing the last time this execution was persisted.
+
+[[metaDataBatchJobExecutionContext]]
+=== `BATCH_JOB_EXECUTION_CONTEXT`
+
+The `BATCH_JOB_EXECUTION_CONTEXT` table holds all information relevant to the
+`ExecutionContext` of a `Job`. There is exactly one `Job` `ExecutionContext` per
+`JobExecution`, and it contains all of the job-level data that is needed for a particular
+job execution. This data typically represents the state that must be retrieved after a
+failure, so that a `JobInstance` can "start from where it left off". The following
+listing shows the definition of the `BATCH_JOB_EXECUTION_CONTEXT` table:
+
+[source, sql]
+----
+CREATE TABLE BATCH_JOB_EXECUTION_CONTEXT  (
+  JOB_EXECUTION_ID BIGINT PRIMARY KEY,
+  SHORT_CONTEXT VARCHAR(2500) NOT NULL,
+  SERIALIZED_CONTEXT CLOB,
+  constraint JOB_EXEC_CTX_FK foreign key (JOB_EXECUTION_ID)
+  references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID)
+) ;
+----
+
+The following list describes each column:
+
+* `JOB_EXECUTION_ID`: Foreign key representing the `JobExecution` to which the context
+belongs. There may be more than one row associated with a given execution.
+* `SHORT_CONTEXT`: A string version of the `SERIALIZED_CONTEXT`.
+* `SERIALIZED_CONTEXT`: The entire context, serialized.
+
+[[metaDataBatchStepExecutionContext]]
+=== `BATCH_STEP_EXECUTION_CONTEXT`
+
+The `BATCH_STEP_EXECUTION_CONTEXT` table holds all information relevant to the
+`ExecutionContext` of a `Step`. There is exactly one `ExecutionContext` per
+`StepExecution`, and it contains all of the data that
+needs to be persisted for a particular step execution. This data typically represents the
+state that must be retrieved after a failure, so that a `JobInstance` can 'start from
+where it left off'. The following listing shows the definition of the
+`BATCH_STEP_EXECUTION_CONTEXT` table:
+
+[source, sql]
+----
+CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT  (
+  STEP_EXECUTION_ID BIGINT PRIMARY KEY,
+  SHORT_CONTEXT VARCHAR(2500) NOT NULL,
+  SERIALIZED_CONTEXT CLOB,
+  constraint STEP_EXEC_CTX_FK foreign key (STEP_EXECUTION_ID)
+  references BATCH_STEP_EXECUTION(STEP_EXECUTION_ID)
+) ;
+----
+
+The following list describes each column:
+
+* `STEP_EXECUTION_ID`: Foreign key representing the `StepExecution` to which the context
+belongs. There may be more than one row associated to a given execution.
+* `SHORT_CONTEXT`: A string version of the `SERIALIZED_CONTEXT`.
+* `SERIALIZED_CONTEXT`: The entire context, serialized.
+
+[[metaDataArchiving]]
+=== Archiving
+
+Because there are entries in multiple tables every time a batch job is run, it is common
+to create an archive strategy for the metadata tables. The tables themselves are designed
+to show a record of what happened in the past and generally do not affect the run of any
+job, with a few notable exceptions pertaining to restart:
+
+* The framework uses the metadata tables to determine whether a particular `JobInstance`
+has been run before. If it has been run and if the job is not restartable, then an
+exception is thrown.
+* If an entry for a `JobInstance` is removed without having completed successfully, the
+framework thinks that the job is new rather than a restart.
+* If a job is restarted, the framework uses any data that has been persisted to the
+`ExecutionContext` to restore the `Job's` state. Therefore, removing any entries from
+this table for jobs that have not completed successfully prevents them from starting at
+the correct point if run again.
+
+[[multiByteCharacters]]
+=== International and Multi-byte Characters
+
+If you are using multi-byte character sets (such as Chinese or Cyrillic) in your business
+processing, then those characters might need to be persisted in the Spring Batch schema.
+Many users find that simply changing the schema to double the length of the `VARCHAR`
+columns is enough.  Others prefer to configure the
+<<job.adoc#configuringJobRepository,JobRepository>> with `max-varchar-length` half the
+value of the `VARCHAR` column length.  Some users have also reported that they use
+`NVARCHAR` in place of `VARCHAR` in their schema definitions.  The best result depends on
+the database platform and the way the database server has been configured locally.
+
+[[recommendationsForIndexingMetaDataTables]]
+=== Recommendations for Indexing Meta Data Tables
+
+Spring Batch provides DDL samples for the metadata tables in the core jar file for
+several common database platforms. Index declarations are not included in that DDL,
+because there are too many variations in how users may want to index, depending on their
+precise platform, local conventions, and the business requirements of how the jobs are
+operated. The following below provides some indication as to which columns are going to
+be used in a `WHERE` clause by the DAO implementations provided by Spring Batch and how
+frequently they might be used, so that individual projects can make up their own minds
+about indexing:
+
+.Where clauses in SQL statements (excluding primary keys) and their approximate frequency of use.
+
+|===============
+|Default Table Name|Where Clause|Frequency
+|BATCH_JOB_INSTANCE|JOB_NAME = ? and JOB_KEY = ?|Every time a job is launched
+|BATCH_JOB_EXECUTION|JOB_INSTANCE_ID = ?|Every time a job is restarted
+|BATCH_STEP_EXECUTION|VERSION = ?|On commit interval, a.k.a. chunk (and at start and end of
+            step)
+|BATCH_STEP_EXECUTION|STEP_NAME = ? and JOB_EXECUTION_ID = ?|Before each step execution
+
+|===============

spring-batch-docs/src/main/asciidoc/spring-batch-intro.adoc

@@ -0,0 +1,550 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[spring-batch-intro]]
+
+== Spring Batch Introduction
+
+Many applications within the enterprise domain require bulk processing to perform
+business operations in mission critical environments. These business operations include:
+
+* Automated, complex processing of large volumes of information that is most efficiently
+processed without user interaction. These operations typically include time-based events
+(such as month-end calculations, notices, or correspondence).
+* Periodic application of complex business rules processed repetitively across very large
+data sets (for example, insurance benefit determination or rate adjustments).
+* Integration of information that is received from internal and external systems that
+typically requires formatting, validation, and processing in a transactional manner into
+the system of record. Batch processing is used to process billions of transactions every
+day for enterprises.
+
+Spring Batch is a lightweight, comprehensive batch framework designed to enable the
+development of robust batch applications vital for the daily operations of enterprise
+systems. Spring Batch builds upon the characteristics of the Spring Framework that people
+have come to expect (productivity, POJO-based development approach, and general ease of
+use), while making it easy for developers to access and leverage more advance enterprise
+services when necessary. Spring Batch is not a scheduling framework. There are many good
+enterprise schedulers (such as Quartz, Tivoli, Control-M, etc.) available in both the
+commercial and open source spaces.  It is intended to work in conjunction with a
+scheduler, not replace a scheduler.
+
+Spring Batch provides reusable functions that are essential in processing large volumes
+of records, including logging/tracing, transaction management, job processing statistics,
+job restart, skip, and resource management. It also provides more advanced technical
+services and features that enable extremely high-volume and high performance batch jobs
+through optimization and partitioning techniques. Spring Batch can be used in both simple
+use cases (such as reading a file into a database or running a stored procedure) as well
+as complex, high volume use cases (such as moving high volumes of data between databases,
+transforming it, and so on). High-volume batch jobs can leverage the framework in a
+highly scalable manner to process significant volumes of information.
+
+[[springBatchBackground]]
+
+=== Background
+
+While open source software projects and associated communities have focused greater
+attention on web-based and microservices-based architecture frameworks, there has been a
+notable lack of focus on reusable architecture frameworks to accommodate Java-based batch
+processing needs, despite continued needs to handle such processing within enterprise IT
+environments. The lack of a standard, reusable batch architecture has resulted in the
+proliferation of many one-off, in-house solutions developed within client enterprise IT
+functions.
+
+SpringSource (now Pivotal) and Accenture collaborated to change this. Accenture's
+hands-on industry and technical experience in implementing batch architectures,
+SpringSource's depth of technical experience, and Spring's proven programming model
+together made a natural and powerful partnership to create high-quality, market-relevant
+software aimed at filling an important gap in enterprise Java. Both companies worked with
+a number of clients who were solving similar problems by developing Spring-based batch
+architecture solutions. This provided some useful additional detail and real-life
+constraints that helped to ensure the solution can be applied to the real-world problems
+posed by clients.
+
+Accenture contributed previously proprietary batch processing architecture frameworks to
+the Spring Batch project, along with committer resources to drive support, enhancements,
+and the existing feature set. Accenture's contribution was based upon decades of
+experience in building batch architectures with the last several generations of
+platforms: COBOL/Mainframe, C++/Unix, and now Java/anywhere.
+
+The collaborative effort between Accenture and SpringSource aimed to promote the
+standardization of software processing approaches, frameworks, and tools that can be
+consistently leveraged by enterprise users when creating batch applications. Companies
+and government agencies desiring to deliver standard, proven solutions to their
+enterprise IT environments can benefit from Spring Batch.
+
+[[springBatchUsageScenarios]]
+
+=== Usage Scenarios
+
+A typical batch program generally:
+
+* Reads a large number of records from a database, file, or queue.
+* Processes the data in some fashion.
+* Writes back data in a modified form.
+
+Spring Batch automates this basic batch iteration, providing the capability to process
+similar transactions as a set, typically in an offline environment without any user
+interaction. Batch jobs are part of most IT projects, and Spring Batch is the only open
+source framework that provides a robust, enterprise-scale solution.
+
+Business Scenarios
+
+* Commit batch process periodically
+* Concurrent batch processing: parallel processing of a job
+* Staged, enterprise message-driven processing
+* Massively parallel batch processing
+* Manual or scheduled restart after failure
+* Sequential processing of dependent steps (with extensions to workflow-driven batches)
+* Partial processing: skip records (for example, on rollback)
+* Whole-batch transaction, for cases with a small batch size or existing stored
+procedures/scripts
+
+Technical Objectives
+
+* Batch developers use the Spring programming model: Concentrate on business logic and
+let the framework take care of infrastructure.
+* Clear separation of concerns between the infrastructure, the batch execution
+environment, and the batch application.
+* Provide common, core execution services as interfaces that all projects can implement.
+* Provide simple and default implementations of the core execution interfaces that can be
+used 'out of the box'.
+* Easy to configure, customize, and extend services, by leveraging the spring framework
+in all layers.
+* All existing core services should be easy to replace or extend, without any impact to
+the infrastructure layer.
+* Provide a simple deployment model, with the architecture JARs completely separate from
+the application, built using Maven.
+
+[[springBatchArchitecture]]
+=== Spring Batch Architecture
+// TODO Make a separate document
+Spring Batch is designed with extensibility and a diverse group of end users in mind. The
+figure below shows the layered architecture that supports the extensibility and ease of
+use for end-user developers.
+
+.Spring Batch Layered Architecture
+image::{batch-asciidoc}images/spring-batch-layers.png[Figure 1.1: Spring Batch Layered Architecture, scaledwidth="60%"]
+
+This layered architecture highlights three major high-level components: Application,
+Core, and Infrastructure. The application contains all batch jobs and custom code written
+by developers using Spring Batch. The Batch Core contains the core runtime classes
+necessary to launch and control a batch job. It includes implementations for
+`JobLauncher`, `Job`, and `Step`. Both Application and Core are built on top of a common
+infrastructure. This infrastructure contains common readers and writers and services
+(such as the `RetryTemplate`), which are used both by application developers(readers and
+writers, such as `ItemReader` and `ItemWriter`) and the core framework itself (retry,
+which is its own library).
+
+[[batchArchitectureConsiderations]]
+=== General Batch Principles and Guidelines
+
+The following key principles, guidelines, and general considerations should be considered
+when building a batch solution.
+
+* Remember that a batch architecture typically affects on-line architecture and vice
+versa. Design with both architectures and environments in mind using common building
+blocks when possible.
+
+* Simplify as much as possible and avoid building complex logical structures in single
+batch applications.
+
+* Keep the processing and storage of data physically close together (in other words, keep
+your data where your processing occurs).
+
+* Minimize system resource use, especially I/O. Perform as many operations as possible in
+internal memory.
+
+* Review application I/O (analyze SQL statements) to ensure that unnecessary physical I/O
+is avoided. In particular, the following four common flaws need to be looked for:
+** Reading data for every transaction when the data could be read once and cached or kept
+in the working storage.
+** Rereading data for a transaction where the data was read earlier in the same
+transaction.
+** Causing unnecessary table or index scans.
+** Not specifying key values in the WHERE clause of an SQL statement.
+
+* Do not do things twice in a batch run. For instance, if you need data summarization for
+reporting purposes, you should (if possible) increment stored totals when data is being
+initially processed, so your reporting application does not have to reprocess the same
+data.
+
+* Allocate enough memory at the beginning of a batch application to avoid time-consuming
+reallocation during the process.
+
+* Always assume the worst with regard to data integrity. Insert adequate checks and
+record validation to maintain data integrity.
+
+* Implement checksums for internal validation where possible. For example, flat files
+should have a trailer record telling the total of records in the file and an aggregate of
+the key fields.
+
+* Plan and execute stress tests as early as possible in a production-like environment
+with realistic data volumes.
+
+* In large batch systems, backups can be challenging, especially if the system is running
+concurrent with on-line on a 24-7 basis. Database backups are typically well taken care
+of in the on-line design, but file backups should be considered to be just as important.
+If the system depends on flat files, file backup procedures should not only be in place
+and documented but be regularly tested as well.
+
+[[batchProcessingStrategy]]
+=== Batch Processing Strategies
+
+To help design and implement batch systems, basic batch application building blocks and
+patterns should be provided to the designers and programmers in the form of sample
+structure charts and code shells. When starting to design a batch job, the business logic
+should be decomposed into a series of steps that can be implemented using the following
+standard building blocks:
+
+* __Conversion Applications:__ For each type of file supplied by or generated to an
+external system, a conversion application must be created to convert the transaction
+records supplied into a standard format required for processing. This type of batch
+application can partly or entirely consist of translation utility modules (see Basic
+Batch Services).
+// TODO Add a link to "Basic Batch Services", once you discover where that content is.
+* __Validation Applications:__ Validation applications ensure that all input/output
+records are correct and consistent. Validation is typically based on file headers and
+trailers, checksums and validation algorithms, and record level cross-checks.
+* __Extract Applications:__ An application that reads a set of records from a database or
+input file, selects records based on predefined rules, and writes the records to an
+output file.
+* __Extract/Update Applications:__ An application that reads records from a database or
+an input file and makes changes to a database or an output file driven by the data found
+in each input record.
+* __Processing and Updating Applications:__ An application that performs processing on
+input transactions from an extract or a validation application. The processing usually
+involves reading a database to obtain data required for processing, potentially updating
+the database and creating records for output processing.
+* __Output/Format Applications:__ Applications that read an input file, restructure data
+from this record according to a standard format, and produce an output file for printing
+or transmission to another program or system.
+
+Additionally, a basic application shell should be provided for business logic that cannot
+be built using the previously mentioned building blocks.
+// TODO What is an example of such a system?
+
+In addition to the main building blocks, each application may use one or more of standard
+utility steps, such as:
+
+
+* Sort: A program that reads an input file and produces an output file where records
+have been re-sequenced according to a sort key field in the records. Sorts are usually
+performed by standard system utilities.
+* Split: A program that reads a single input file and writes each record to one of
+several output files based on a field value. Splits can be tailored or performed by
+parameter-driven standard system utilities.
+* Merge: A program that reads records from multiple input files and produces one output
+file with combined data from the input files. Merges can be tailored or performed by
+parameter-driven standard system utilities.
+
+Batch applications can additionally be categorized by their input source:
+
+* Database-driven applications are driven by rows or values retrieved from the database.
+* File-driven applications are driven by records or values retrieved from a file.
+* Message-driven applications are driven by messages retrieved from a message queue.
+
+The foundation of any batch system is the processing strategy. Factors affecting the
+selection of the strategy include: estimated batch system volume, concurrency with
+on-line systems or with other batch systems, available batch windows. (Note that, with
+more enterprises wanting to be up and running 24x7, clear batch windows are
+disappearing).
+
+Typical processing options for batch are (in increasing order of implementation
+complexity):
+
+* Normal processing during a batch window in off-line mode.
+* Concurrent batch or on-line processing.
+* Parallel processing of many different batch runs or jobs at the same time.
+* Partitioning (processing of many instances of the same job at the same time).
+* A combination of the preceding options.
+
+Some or all of these options may be supported by a commercial scheduler.
+
+The following section discusses these processing options in more detail. It is important
+to notice that, as a rule of thumb, the commit and locking strategy adopted by batch
+processes depends on the type of processing performed and that the on-line locking
+strategy should also use the same principles. Therefore, the batch architecture cannot be
+simply an afterthought when designing an overall architecture.
+
+The locking strategy can be to use only normal database locks or to implement an
+additional custom locking service in the architecture. The locking service would track
+database locking (for example, by storing the necessary information in a dedicated
+db-table) and give or deny permissions to the application programs requesting a db
+operation. Retry logic could also be implemented by this architecture to avoid aborting a
+batch job in case of a lock situation.
+
+*1. Normal processing in a batch window* For simple batch processes running in a separate
+batch window where the data being updated is not required by on-line users or other batch
+processes, concurrency is not an issue and a single commit can be done at the end of the
+batch run.
+
+In most cases, a more robust approach is more appropriate. Keep in mind that batch
+systems have a tendency to grow as time goes by, both in terms of complexity and the data
+volumes they handle. If no locking strategy is in place and the system still relies on a
+single commit point, modifying the batch programs can be painful. Therefore, even with
+the simplest batch systems, consider the need for commit logic for restart-recovery
+options as well as the information concerning the more complex cases described later in
+this section.
+
+*2. Concurrent batch or on-line processing* Batch applications processing data that can
+be simultaneously updated by on-line users should not lock any data (either in the
+database or in files) which could be required by on-line users for more than a few
+seconds. Also, updates should be committed to the database at the end of every few
+transactions. This minimizes the portion of data that is unavailable to other processes
+and the elapsed time the data is unavailable.
+
+Another option to minimize physical locking is to have logical row-level locking
+implemented with either an Optimistic Locking Pattern or a Pessimistic Locking Pattern.
+
+
+* Optimistic locking assumes a low likelihood of record contention. It typically means
+inserting a timestamp column in each database table used concurrently by both batch and
+on-line processing. When an application fetches a row for processing, it also fetches the
+timestamp. As the application then tries to update the processed row, the update uses the
+original timestamp in the WHERE clause. If the timestamp matches, the data and the
+timestamp are updated. If the timestamp does not match, this indicates that another
+application has updated the same row between the fetch and the update attempt. Therefore,
+the update cannot be performed.
+
+
+* Pessimistic locking is any locking strategy that assumes there is a high likelihood of
+record contention and therefore either a physical or logical lock needs to be obtained at
+retrieval time. One type of pessimistic logical locking uses a dedicated lock-column in
+the database table. When an application retrieves the row for update, it sets a flag in
+the lock column. With the flag in place, other applications attempting to retrieve the
+same row logically fail. When the application that sets the flag updates the row, it also
+clears the flag, enabling the row to be retrieved by other applications. Please note that
+the integrity of data must be maintained also between the initial fetch and the setting
+of the flag, for example by using db locks (such as `SELECT FOR UPDATE`). Note also that
+this method suffers from the same downside as physical locking except that it is somewhat
+easier to manage building a time-out mechanism that gets the lock released if the user
+goes to lunch while the record is locked.
+
+These patterns are not necessarily suitable for batch processing, but they might be used
+for concurrent batch and on-line processing (such as in cases where the database does not
+support row-level locking). As a general rule, optimistic locking is more suitable for
+on-line applications, while pessimistic locking is more suitable for batch applications.
+Whenever logical locking is used, the same scheme must be used for all applications
+accessing data entities protected by logical locks.
+
+Note that both of these solutions only address locking a single record. Often, we may
+need to lock a logically related group of records. With physical locks, you have to
+manage these very carefully in order to avoid potential deadlocks. With logical locks, it
+is usually best to build a logical lock manager that understands the logical record
+groups you want to protect and that can ensure that locks are coherent and
+non-deadlocking. This logical lock manager usually uses its own tables for lock
+management, contention reporting, time-out mechanism, and other concerns.
+
+*3. Parallel Processing* Parallel processing allows multiple batch runs or jobs to run in
+parallel to minimize the total elapsed batch processing time. This is not a problem as
+long as the jobs are not sharing the same files, db-tables, or index spaces. If they do,
+this service should be implemented using partitioned data. Another option is to build an
+architecture module for maintaining interdependencies by using a control table. A control
+table should contain a row for each shared resource and whether it is in use by an
+application or not. The batch architecture or the application in a parallel job would
+then retrieve information from that table to determine if it can get access to the
+resource it needs or not.
+
+If the data access is not a problem, parallel processing can be implemented through the
+use of additional threads to process in parallel.  In the mainframe environment, parallel
+job classes have traditionally been used, in order to ensure adequate CPU time for all
+the processes. Regardless, the solution has to be robust enough to ensure time slices for
+all the running processes.
+
+Other key issues in parallel processing include load balancing and the availability of
+general system resources such as files, database buffer pools, and so on. Also note that
+the control table itself can easily become a critical resource.
+
+*4. Partitioning* Using partitioning allows multiple versions of large batch applications
+to run concurrently. The purpose of this is to reduce the elapsed time required to
+process long batch jobs. Processes that can be successfully partitioned are those where
+the input file can be split and/or the main database tables partitioned to allow the
+application to run against different sets of data.
+
+In addition, processes which are partitioned must be designed to only process their
+assigned data set. A partitioning architecture has to be closely tied to the database
+design and the database partitioning strategy. Note that database partitioning does not
+necessarily mean physical partitioning of the database, although in most cases this is
+advisable. The following picture illustrates the partitioning approach:
+
+.Partitioned Process
+image::{batch-asciidoc}images/partitioned.png[Figure 1.2: Partitioned Process, scaledwidth="60%"]
+
+
+The architecture should be flexible enough to allow dynamic configuration of the number
+of partitions. Both automatic and user controlled configuration should be considered.
+Automatic configuration may be based on parameters such as the input file size and the
+number of input records.
+
+*4.1 Partitioning Approaches* Selecting a partitioning approach has to be done on a
+case-by-case basis. The  following list describes some of the possible partitioning
+approaches:
+
+_1. Fixed and Even Break-Up of Record Set_
+
+This involves breaking the input record set into an even number of portions (for example,
+10, where each portion has exactly 1/10th of the entire record set). Each portion is then
+processed by one instance of the batch/extract application.
+
+In order to use this approach, preprocessing is required to split the record set up. The
+result of this split will be a lower and upper bound placement number which can be used
+as input to the batch/extract application in order to restrict its processing to only its
+portion.
+
+Preprocessing could be a large overhead, as it has to calculate and determine the bounds
+of each portion of the record set.
+
+_2. Break up by a Key Column_
+
+This involves breaking up the input record set by a key column, such as a location code,
+and assigning data from each key to a batch instance. In order to achieve this, column
+values can be either:
+
+* Assigned to a batch instance by a partitioning table (described later in this
+section).
+
+* Assigned to a batch instance by a portion of the value (such as 0000-0999, 1000 - 1999,
+and so on).
+
+Under option 1, adding new values means a manual reconfiguration of the batch/extract to
+ensure that the new value is added to a particular instance.
+
+Under option 2, this ensures that all values are covered via an instance of the batch
+job. However, the number of values processed by one instance is dependent on the
+distribution of column values (there may be a large number of locations in the 0000-0999
+range, and few in the 1000-1999 range). Under this option, the data range should be
+designed with partitioning in mind.
+
+Under both options, the optimal even distribution of records to batch instances cannot be
+realized. There is no dynamic configuration of the number of batch instances used.
+
+_3. Breakup by Views_
+
+This approach is basically breakup by a key column but on the database level. It involves
+breaking up the record set into views. These views are used by each instance of the batch
+application during its processing. The breakup is done by grouping the data.
+
+With this option, each instance of a batch application has to be configured to hit a
+particular view (instead of the master table). Also, with the addition of new data
+values, this new group of data has to be included into a view. There is no dynamic
+configuration capability, as a change in the number of instances results in a change to
+the views.
+
+_4. Addition of a Processing Indicator_
+
+This involves the addition of a new column to the input table, which acts as an
+indicator. As a preprocessing step, all indicators are marked as being non-processed.
+During the record fetch stage of the batch application, records are read on the condition
+that that record is marked as being non-processed, and once they are read (with lock),
+they are marked as being in processing. When that record is completed, the indicator is
+updated to either complete or error. Many instances of a batch application can be started
+without a change, as the additional column ensures that a record is only processed once.
+// TODO On completion, what is the record marked as? Same for on error. (I expected a
+// sentence or two on the order of "On completion, indicators are marked as being
+// complete.")
+
+With this option, I/O on the table increases dynamically. In the case of an updating
+batch application, this impact is reduced, as a write must occur anyway.
+
+_5. Extract Table to a Flat File_
+
+This involves the extraction of the table into a file. This file can then be split into
+multiple segments and used as input to the batch instances.
+
+With this option, the additional overhead of extracting the table into a file and
+splitting it may cancel out the effect of multi-partitioning. Dynamic configuration can
+be achieved by changing the file splitting script.
+
+_6. Use of a Hashing Column_
+
+This scheme involves the addition of a hash column (key/index) to the database tables
+used to retrieve the driver record. This hash column has an indicator to determine which
+instance of the batch application processes this particular row. For example, if there
+are three batch instances to be started, then an indicator of 'A' marks a row for
+processing by instance 1, an indicator of 'B' marks a row for processing by instance 2,
+and an indicator of 'C' marks a row for processing by instance 3.
+
+The procedure used to retrieve the records would then have an additional `WHERE` clause
+to select all rows marked by a particular indicator. The inserts in this table would
+involve the addition of the marker field, which would be defaulted to one of the
+instances (such as 'A').
+
+A simple batch application would be used to update the indicators, such as to
+redistribute the load between the different instances. When a sufficiently large number
+of new rows have been added, this batch can be run (anytime, except in the batch window)
+to redistribute the new rows to other instances.
+// TODO Why not in the batch window?
+
+Additional instances of the batch application only require the running of the batch
+application as described in the preceding paragraphs to redistribute the indicators to
+work with a new number of instances.
+
+*4.2 Database and Application Design Principles*
+
+An architecture that supports multi-partitioned applications which run against
+partitioned database tables using the key column approach should include a central
+partition repository for storing partition parameters. This provides flexibility and
+ensures maintainability. The repository generally consists of a single table, known as
+the partition table.
+
+Information stored in the partition table is static and, in general, should be maintained
+by the DBA. The table should consist of one row of information for each partition of a
+multi-partitioned application. The table should have columns for  Program ID Code,
+Partition Number (logical ID of the partition), Low Value of the db key column for this
+partition, and High Value of the db key column for this partition.
+
+On program start-up, the program `id` and partition number should be passed to the
+application from the architecture (specifically, from the Control Processing Tasklet). If
+a key column approach is used, these variables are used to read the partition table in
+order to determine what range of data the application is to process. In addition the
+partition number must be used throughout the processing to:
+
+* Add to the output files/database updates in order for the merge process to work
+properly.
+* Report normal processing to the batch log and any errors to the architecture error
+handler.
+
+*4.3 Minimizing Deadlocks*
+
+When applications run in parallel or are partitioned, contention in database resources
+and deadlocks may occur. It is critical that the database design team eliminates
+potential contention situations as much as possible as part of the database design.
+
+Also, the developers must ensure that the database index tables are designed with
+deadlock prevention and performance in mind.
+
+Deadlocks or hot spots often occur in administration or architecture tables, such as log
+tables, control tables, and lock tables. The implications of these should be taken into
+account as well. A realistic stress test is crucial for identifying the possible
+bottlenecks in the architecture.
+
+To minimize the impact of conflicts on data, the architecture should provide services
+such as wait-and-retry intervals when attaching to a database or when encountering a
+deadlock. This means a built-in mechanism to react to certain database return codes and,
+instead of issuing an immediate error, waiting a predetermined amount of time and
+retrying the database operation.
+
+*4.4 Parameter Passing and Validation*
+
+The partition architecture should be relatively transparent to application developers.
+The architecture should perform all tasks associated with running the application in a
+partitioned mode, including:
+
+* Retrieving partition parameters before application start-up.
+* Validating partition parameters before application start-up.
+* Passing parameters to the application at start-up.
+
+The validation should include checks to ensure that:
+
+* The application has sufficient partitions to cover the whole data range.
+* There are no gaps between partitions.
+
+If the database is partitioned, some additional validation may be necessary to ensure
+that a single partition does not span database partitions.
+
+Also, the architecture should take into consideration the consolidation of partitions.
+Key questions include:
+
+* Must all the partitions be finished before going into the next job step?
+* What happens if one of the partitions aborts?

spring-batch-docs/src/main/asciidoc/testing.adoc

@@ -0,0 +1,379 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[testing]]
+
+== Unit Testing
+
+ifndef::onlyonetoggle[]
+include::toggle.adoc[]
+endif::onlyonetoggle[]
+
+As with other application styles, it is extremely important to unit test any code written
+as part of a batch job. The Spring core documentation covers how to unit and integration
+test with Spring in great detail, so it is not be repeated here. It is important, however,
+to think about how to 'end to end' test a batch job, which is what this chapter covers.
+The spring-batch-test project includes classes that facilitate this end-to-end test
+approach.
+
+[[creatingUnitTestClass]]
+
+
+=== Creating a Unit Test Class
+
+In order for the unit test to run a batch job, the framework must load the job's
+ApplicationContext. Two annotations are used to trigger this behavior:
+
+* `@RunWith(SpringJUnit4ClassRunner.class)`: Indicates that the class should use Spring's
+JUnit facilities
+* `@ContextConfiguration(...)`: Indicates which resources to configure the
+`ApplicationContext` with.
+
+Starting from v4.1, it is also possible to inject Spring Batch test utilities
+like the `JobLauncherTestUtils` and `JobRepositoryTestUtils` in the test context
+using the `@SpringBatchTest` annotation.
+
+[NOTE]
+====
+It should be noted that `JobLauncherTestUtils` requires a `Job` bean and that
+`JobRepositoryTestUtils` requires a `DataSource` bean. Since `@SpringBatchTest`
+registers a `JobLauncherTestUtils` and a `JobRepositoryTestUtils` in the test
+context, it is expected that the test context contains a single autowire candidate
+for a `Job` and a `DataSource` (either a single bean definition or one that is
+annotated with `org.springframework.context.annotation.Primary`).
+====
+
+[role="javaContent"]
+The following Java example shows the annotations in use:
+
+.Using Java Configuration
+[source, java, role="javaContent"]
+----
+@SpringBatchTest
+@RunWith(SpringRunner.class)
+@ContextConfiguration(classes=SkipSampleConfiguration.class)
+public class SkipSampleFunctionalTests { ... }
+----
+
+[role="xmlContent"]
+The following XML example shows the annotations in use:
+
+.Using XML Configuration
+[source, java, role="xmlContent"]
+----
+@SpringBatchTest
+@RunWith(SpringRunner.class)
+@ContextConfiguration(locations = { "/simple-job-launcher-context.xml",
+                                    "/jobs/skipSampleJob.xml" })
+public class SkipSampleFunctionalTests { ... }
+----
+
+[[endToEndTesting]]
+
+
+=== End-To-End Testing of Batch Jobs
+
+'End To End' testing can be defined as testing the complete run of a batch job from
+beginning to end. This allows for a test that sets up a test condition, executes the job,
+and verifies the end result.
+
+Consider an example of a batch job that reads from the database and writes to a flat file.
+The test method begins by setting up the database with test data. It clears the CUSTOMER
+table and then inserts 10 new records. The test then launches the `Job` by using the
+`launchJob()` method. The `launchJob()` method is provided by the `JobLauncherTestUtils`
+class. The `JobLauncherTestUtils` class also provides the `launchJob(JobParameters)`
+method, which allows the test to give particular parameters. The `launchJob()` method
+returns the `JobExecution` object, which is useful for asserting particular information
+about the `Job` run. In the following case, the test verifies that the `Job` ended with
+status "COMPLETED".
+
+[role="xmlContent"]
+The following listing shows the example in XML:
+
+.XML Based Configuration
+[source, java, role="xmlContent"]
+----
+@SpringBatchTest
+@RunWith(SpringRunner.class)
+@ContextConfiguration(locations = { "/simple-job-launcher-context.xml",
+                                    "/jobs/skipSampleJob.xml" })
+public class SkipSampleFunctionalTests {
+
+    @Autowired
+    private JobLauncherTestUtils jobLauncherTestUtils;
+
+    private SimpleJdbcTemplate simpleJdbcTemplate;
+
+    @Autowired
+    public void setDataSource(DataSource dataSource) {
+        this.simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
+    }
+
+    @Test
+    public void testJob() throws Exception {
+        simpleJdbcTemplate.update("delete from CUSTOMER");
+        for (int i = 1; i <= 10; i++) {
+            simpleJdbcTemplate.update("insert into CUSTOMER values (?, 0, ?, 100000)",
+                                      i, "customer" + i);
+        }
+
+        JobExecution jobExecution = jobLauncherTestUtils.launchJob();
+
+
+        Assert.assertEquals("COMPLETED", jobExecution.getExitStatus().getExitCode());
+    }
+}
+----
+
+[role="javaContent"]
+The following listing shows the example in Java:
+
+.Java Based Configuration
+[source, java, role="javaContent"]
+----
+@SpringBatchTest
+@RunWith(SpringRunner.class)
+@ContextConfiguration(classes=SkipSampleConfiguration.class)
+public class SkipSampleFunctionalTests {
+
+    @Autowired
+    private JobLauncherTestUtils jobLauncherTestUtils;
+
+    private SimpleJdbcTemplate simpleJdbcTemplate;
+
+    @Autowired
+    public void setDataSource(DataSource dataSource) {
+        this.simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
+    }
+
+    @Test
+    public void testJob() throws Exception {
+        simpleJdbcTemplate.update("delete from CUSTOMER");
+        for (int i = 1; i <= 10; i++) {
+            simpleJdbcTemplate.update("insert into CUSTOMER values (?, 0, ?, 100000)",
+                                      i, "customer" + i);
+        }
+
+        JobExecution jobExecution = jobLauncherTestUtils.launchJob();
+
+
+        Assert.assertEquals("COMPLETED", jobExecution.getExitStatus().getExitCode());
+    }
+}
+----
+
+[[testingIndividualSteps]]
+
+
+=== Testing Individual Steps
+
+For complex batch jobs, test cases in the end-to-end testing approach may become
+unmanageable. It these cases, it may be more useful to have test cases to test individual
+steps on their own. The `AbstractJobTests` class contains a method called `launchStep`,
+which takes a step name and runs just that particular `Step`. This approach allows for
+more targeted tests letting the test set up data for only that step and to validate its
+results directly. The following example shows how to use the `launchStep` method to load a
+`Step` by name:
+
+[source, java]
+----
+JobExecution jobExecution = jobLauncherTestUtils.launchStep("loadFileStep");
+----
+
+
+
+=== Testing Step-Scoped Components
+
+Often, the components that are configured for your steps at runtime use step scope and
+late binding to inject context from the step or job execution. These are tricky to test as
+standalone components, unless you have a way to set the context as if they were in a step
+execution. That is the goal of two components in Spring Batch:
+`StepScopeTestExecutionListener` and `StepScopeTestUtils`.
+
+The listener is declared at the class level, and its job is to create a step execution
+context for each test method, as shown in the following example:
+
+[source, java]
+----
+@ContextConfiguration
+@TestExecutionListeners( { DependencyInjectionTestExecutionListener.class,
+    StepScopeTestExecutionListener.class })
+@RunWith(SpringRunner.class)
+public class StepScopeTestExecutionListenerIntegrationTests {
+
+    // This component is defined step-scoped, so it cannot be injected unless
+    // a step is active...
+    @Autowired
+    private ItemReader<String> reader;
+
+    public StepExecution getStepExecution() {
+        StepExecution execution = MetaDataInstanceFactory.createStepExecution();
+        execution.getExecutionContext().putString("input.data", "foo,bar,spam");
+        return execution;
+    }
+
+    @Test
+    public void testReader() {
+        // The reader is initialized and bound to the input data
+        assertNotNull(reader.read());
+    }
+
+}
+----
+
+There are two `TestExecutionListeners`. One is the regular Spring Test framework, which
+handles dependency injection from the configured application context to inject the reader.
+The other is the Spring Batch `StepScopeTestExecutionListener`. It works by looking for a
+factory method in the test case for a `StepExecution`, using that as the context for the
+test method, as if that execution were active in a `Step` at runtime. The factory method
+is detected by its signature (it must return a `StepExecution`). If a factory method is
+not provided, then a default `StepExecution` is created.
+
+Starting from v4.1, the `StepScopeTestExecutionListener` and
+`JobScopeTestExecutionListener` are imported as test execution listeners
+if the test class is annotated with `@SpringBatchTest`. The preceding test
+example can be configured as follows:
+
+[source, java]
+----
+@SpringBatchTest
+@RunWith(SpringRunner.class)
+@ContextConfiguration
+public class StepScopeTestExecutionListenerIntegrationTests {
+
+    // This component is defined step-scoped, so it cannot be injected unless
+    // a step is active...
+    @Autowired
+    private ItemReader<String> reader;
+
+    public StepExecution getStepExecution() {
+        StepExecution execution = MetaDataInstanceFactory.createStepExecution();
+        execution.getExecutionContext().putString("input.data", "foo,bar,spam");
+        return execution;
+    }
+
+    @Test
+    public void testReader() {
+        // The reader is initialized and bound to the input data
+        assertNotNull(reader.read());
+    }
+
+}
+----
+
+The listener approach is convenient if you want the duration of the step scope to be the
+execution of the test method. For a more flexible but more invasive approach, you can use
+the `StepScopeTestUtils`. The following example counts the number of items available in
+the reader shown in the previous example:
+
+[source, java]
+----
+int count = StepScopeTestUtils.doInStepScope(stepExecution,
+    new Callable<Integer>() {
+      public Integer call() throws Exception {
+
+        int count = 0;
+
+        while (reader.read() != null) {
+           count++;
+        }
+        return count;
+    }
+});
+----
+
+[[validatingOutputFiles]]
+
+=== Validating Output Files
+
+When a batch job writes to the database, it is easy to query the database to verify that
+the output is as expected. However, if the batch job writes to a file, it is equally
+important that the output be verified. Spring Batch provides a class called `AssertFile`
+to facilitate the verification of output files. The method called `assertFileEquals` takes
+two `File` objects (or two `Resource` objects) and asserts, line by line, that the two
+files have the same content. Therefore, it is possible to create a file with the expected
+output and to compare it to the actual result, as shown in the following example:
+
+[source, java]
+----
+private static final String EXPECTED_FILE = "src/main/resources/data/input.txt";
+private static final String OUTPUT_FILE = "target/test-outputs/output.txt";
+
+AssertFile.assertFileEquals(new FileSystemResource(EXPECTED_FILE),
+                            new FileSystemResource(OUTPUT_FILE));
+----
+
+[[mockingDomainObjects]]
+
+
+=== Mocking Domain Objects
+
+Another common issue encountered while writing unit and integration tests for Spring Batch
+components is how to mock domain objects. A good example is a `StepExecutionListener`, as
+illustrated in the following code snippet:
+
+[source, java]
+----
+public class NoWorkFoundStepExecutionListener extends StepExecutionListenerSupport {
+
+    public ExitStatus afterStep(StepExecution stepExecution) {
+        if (stepExecution.getReadCount() == 0) {
+            return ExitStatus.FAILED;
+        }
+        return null;
+    }
+}
+----
+
+The preceding listener example is provided by the framework and checks a `StepExecution`
+for an empty read count, thus signifying that no work was done. While this example is
+fairly simple, it serves to illustrate the types of problems that may be encountered when
+attempting to unit test classes that implement interfaces requiring Spring Batch domain
+objects. Consider the following unit test for the listener's in the preceding example:
+
+[source, java]
+----
+private NoWorkFoundStepExecutionListener tested = new NoWorkFoundStepExecutionListener();
+
+@Test
+public void noWork() {
+    StepExecution stepExecution = new StepExecution("NoProcessingStep",
+                new JobExecution(new JobInstance(1L, new JobParameters(),
+                                 "NoProcessingJob")));
+
+    stepExecution.setExitStatus(ExitStatus.COMPLETED);
+    stepExecution.setReadCount(0);
+
+    ExitStatus exitStatus = tested.afterStep(stepExecution);
+    assertEquals(ExitStatus.FAILED.getExitCode(), exitStatus.getExitCode());
+}
+----
+
+Because the Spring Batch domain model follows good object-oriented principles, the
+`StepExecution` requires a `JobExecution`, which requires a `JobInstance` and
+`JobParameters`, to create a valid `StepExecution`. While this is good in a solid domain
+model, it does make creating stub objects for unit testing verbose. To address this issue,
+the Spring Batch test module includes a factory for creating domain objects:
+`MetaDataInstanceFactory`. Given this factory, the unit test can be updated to be more
+concise, as shown in the following example:
+
+[source, java]
+----
+private NoWorkFoundStepExecutionListener tested = new NoWorkFoundStepExecutionListener();
+
+@Test
+public void testAfterStep() {
+    StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution();
+
+    stepExecution.setExitStatus(ExitStatus.COMPLETED);
+    stepExecution.setReadCount(0);
+
+    ExitStatus exitStatus = tested.afterStep(stepExecution);
+    assertEquals(ExitStatus.FAILED.getExitCode(), exitStatus.getExitCode());
+}
+----
+
+The preceding method for creating a simple `StepExecution` is just one convenience method
+available within the factory. A full method listing can be found in its
+link:$$http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/test/MetaDataInstanceFactory.html$$[Javadoc].

spring-batch-docs/src/main/asciidoc/toggle.adoc

@@ -0,0 +1,15 @@
+ifdef::backend-html5[]
++++
+<div>
+  <script type="text/javascript" src="js/jquery-3.2.1.min.js"></script>
+  <script type="text/javascript" src="js/js.cookie.js"></script>
+  <script type="text/javascript" src="js/DocumentToggle.js"></script>
+  <script type="text/javascript" src="js/Redirect.js"></script>
+  <div class="docToggle-button">
+    <input id="xmlButton" type="radio" name="docToggle" value="XML"><label for="xmlButton">XML</label>
+    <input id="javaButton" type="radio" name="docToggle" value="Java" checked><label for="javaButton">Java</label>
+    <input id="bothButton" type="radio" name="docToggle" value="Both" checked><label for="bothButton">Both</label>
+  </div>
+</div>
++++
+endif::backend-html5[]

spring-batch-docs/src/main/asciidoc/transaction-appendix.adoc

@@ -0,0 +1,345 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[transactions]]
+
+[appendix]
+== Batch Processing and Transactions
+
+[[transactionsNoRetry]]
+=== Simple Batching with No Retry
+
+Consider the following simple example of a nested batch with no retries. It shows a
+common scenario for batch processing: An input source is processed until exhausted, and
+we commit periodically at the end of a "chunk" of processing.
+
+----
+
+1   |  REPEAT(until=exhausted) {
+|
+2   |    TX {
+3   |      REPEAT(size=5) {
+3.1 |        input;
+3.2 |        output;
+|      }
+|    }
+|
+|  }
+
+----
+
+The input operation (3.1) could be a message-based receive (such as from JMS), or a
+file-based read, but to recover and continue processing with a chance of completing the
+whole job, it must be transactional. The same applies to the operation at 3.2. It must
+be either transactional or idempotent.
+
+If the chunk at `REPEAT` (3) fails because of a database exception at 3.2, then `TX` (2)
+must roll back the whole chunk.
+
+[[transactionStatelessRetry]]
+=== Simple Stateless Retry
+
+It is also useful to use a retry for an operation which is not transactional, such as a
+call to a web-service or other remote resource, as shown in the following example:
+
+----
+
+0   |  TX {
+1   |    input;
+1.1 |    output;
+2   |    RETRY {
+2.1 |      remote access;
+|    }
+|  }
+
+----
+
+This is actually one of the most useful applications of a retry, since a remote call is
+much more likely to fail and be retryable than a database update.  As long as the remote
+access (2.1) eventually succeeds, the transaction, `TX` (0), commits.  If the remote
+access (2.1) eventually fails, then the transaction, `TX` (0), is guaranteed to roll
+back.
+
+[[repeatRetry]]
+=== Typical Repeat-Retry Pattern
+
+The most typical batch processing pattern is to add a retry to the inner block of the
+chunk, as shown in the following example:
+
+----
+
+1   |  REPEAT(until=exhausted, exception=not critical) {
+|
+2   |    TX {
+3   |      REPEAT(size=5) {
+|
+4   |        RETRY(stateful, exception=deadlock loser) {
+4.1 |          input;
+5   |        } PROCESS {
+5.1 |          output;
+6   |        } SKIP and RECOVER {
+|          notify;
+|        }
+|
+|      }
+|    }
+|
+|  }
+
+----
+
+The inner `RETRY` (4) block is marked as "stateful". See <<transactionsNoRetry,the
+typical use case>> for a description of a stateful retry.  This means that if the
+retry `PROCESS` (5) block fails, the behavior of the `RETRY` (4) is as follows:
+
+. Throw an exception, rolling back the transaction, `TX` (2), at the chunk level, and
+allowing the item to be re-presented to the input queue.
+. When the item re-appears, it might be retried depending on the retry policy in place,
+executing `PROCESS` (5) again.  The second and subsequent attempts might fail again and
+re-throw the exception.
+. Eventually, the item reappears for the final time. The retry policy disallows another
+attempt, so `PROCESS` (5) is never executed. In this case, we follow the `RECOVER` (6)
+path, effectively "skipping" the item that was received and is being processed.
+
+Note that the notation used for the `RETRY` (4) in the plan above explicitly shows that
+the input step (4.1) is part of the retry. It also makes clear that there are two
+alternate paths for processing: the normal case, as denoted by `PROCESS` (5), and the
+recovery path, as denoted in a separate block by `RECOVER` (6).  The two alternate paths
+are completely distinct. Only one is ever taken in normal circumstances.
+
+In special cases (such as a special `TranscationValidException` type), the retry policy
+might be able to determine that the `RECOVER` (6) path can be taken on the last attempt
+after `PROCESS` (5) has just failed, instead of waiting for the item to be re-presented.
+This is not the default behavior, because it requires detailed knowledge of what has
+happened inside the `PROCESS` (5) block, which is not usually available. For example, if
+the output included write access before the failure, then the exception should be
+re-thrown to ensure transactional integrity.
+
+The completion policy in the outer `REPEAT` (1) is crucial to the success of the above
+plan.  If the output (5.1) fails, it may throw an exception (it usually does, as
+described), in which case the transaction, `TX` (2), fails, and the exception could
+propagate up through the outer batch `REPEAT` (1).  We do not want the whole batch to
+stop, because the `RETRY` (4) might still be successful if we try again, so we add
+`exception=not critical` to the outer `REPEAT` (1).
+
+Note, however, that if the `TX` (2) fails and we __do__ try again, by virtue of the outer
+completion policy, the item that is next processed in the inner `REPEAT` (3) is not
+guaranteed to be the one that just failed.  It might be, but it depends on the
+implementation of the input (4.1).  Thus, the output (5.1) might fail again on either a
+new item or the old one.  The client of the batch should not assume that each `RETRY` (4)
+attempt is going to process the same items as the last one that failed.  For example, if
+the termination policy for `REPEAT` (1) is to fail after 10 attempts, it fails after 10
+consecutive attempts but not necessarily at the same item. This is consistent with the
+overall retry strategy. The inner `RETRY` (4) is aware of the history of each item and
+can decide whether or not to have another attempt at it.
+
+[[asyncChunkProcessing]]
+=== Asynchronous Chunk Processing
+
+The inner batches or chunks in the <<repeatRetry,typical example>> can be executed
+concurrently by configuring the outer batch to use an `AsyncTaskExecutor`.  The outer
+batch waits for all the chunks to complete before completing. The following example shows
+asynchronous chunk processing:
+
+----
+
+1   |  REPEAT(until=exhausted, concurrent, exception=not critical) {
+|
+2   |    TX {
+3   |      REPEAT(size=5) {
+|
+4   |        RETRY(stateful, exception=deadlock loser) {
+4.1 |          input;
+5   |        } PROCESS {
+|          output;
+6   |        } RECOVER {
+|          recover;
+|        }
+|
+|      }
+|    }
+|
+|  }
+
+----
+
+[[asyncItemProcessing]]
+=== Asynchronous Item Processing
+
+The individual items in chunks in the <<repeatRetry,typical example>> can also, in
+principle, be processed concurrently.  In this case, the transaction boundary has to move
+to the level of the individual item, so that each transaction is on a single thread, as
+shown in the following example:
+
+----
+
+1   |  REPEAT(until=exhausted, exception=not critical) {
+|
+2   |    REPEAT(size=5, concurrent) {
+|
+3   |      TX {
+4   |        RETRY(stateful, exception=deadlock loser) {
+4.1 |          input;
+5   |        } PROCESS {
+|          output;
+6   |        } RECOVER {
+|          recover;
+|        }
+|      }
+|
+|    }
+|
+|  }
+
+----
+
+This plan sacrifices the optimization benefit, which the simple plan had, of having all
+the transactional resources chunked together.  It is only useful if the cost of the
+processing (5) is much higher than the cost of transaction management (3).
+
+[[transactionPropagation]]
+=== Interactions Between Batching and Transaction Propagation
+
+There is a tighter coupling between batch-retry and transaction management than we would
+ideally like.  In particular, a stateless retry cannot be used to retry database
+operations with a transaction manager that does not support NESTED propagation.
+
+The following example uses retry without repeat:
+
+----
+
+1   |  TX {
+|
+1.1 |    input;
+2.2 |    database access;
+2   |    RETRY {
+3   |      TX {
+3.1 |        database access;
+|      }
+|    }
+|
+|  }
+
+----
+
+Again, and for the same reason, the inner transaction, `TX` (3), can cause the outer
+transaction, `TX` (1), to fail, even if the `RETRY` (2) is eventually successful.
+
+Unfortunately, the same effect percolates from the retry block up to the surrounding
+repeat batch if there is one, as shown in the following example:
+
+----
+
+1   |  TX {
+|
+2   |    REPEAT(size=5) {
+2.1 |      input;
+2.2 |      database access;
+3   |      RETRY {
+4   |        TX {
+4.1 |          database access;
+|        }
+|      }
+|    }
+|
+|  }
+
+----
+
+Now, if TX (3) rolls back, it can pollute the whole batch at TX (1) and force it to roll
+back at the end.
+
+What about non-default propagation?
+
+* In the preceding example, `PROPAGATION_REQUIRES_NEW` at `TX` (3) prevents the outer
+`TX` (1) from being polluted if both transactions are eventually successful.  But if `TX`
+(3) commits and `TX` (1) rolls back, then `TX` (3) stays committed, so we violate the
+transaction contract for `TX` (1).  If `TX` (3) rolls back, `TX` (1) does not necessarily
+(but it probably does in practice, because the retry throws a roll back exception).
+
+* `PROPAGATION_NESTED` at `TX` (3) works as we require in the retry case (and for a
+batch with skips): `TX` (3) can commit but subsequently be rolled back by the outer
+transaction, `TX` (1).  If `TX` (3) rolls back, `TX` (1) rolls back in practice.  This
+option is only available on some platforms, not including Hibernate or
+JTA, but it is the only one that consistently works.
+
+Consequently, the `NESTED` pattern is best if the retry block contains any database
+access.
+
+[[specialTransactionOrthonogonal]]
+=== Special Case: Transactions with Orthogonal Resources
+
+Default propagation is always OK for simple cases where there are no nested database
+transactions.  Consider the following example, where the `SESSION` and `TX` are not
+global `XA` resources, so their resources are orthogonal:
+
+----
+
+0   |  SESSION {
+1   |    input;
+2   |    RETRY {
+3   |      TX {
+3.1 |        database access;
+|      }
+|    }
+|  }
+
+----
+
+Here there is a transactional message `SESSION` (0), but it does not participate in other
+transactions with `PlatformTransactionManager`, so it does not propagate when `TX` (3)
+starts.  There is no database access outside the `RETRY` (2) block. If `TX` (3) fails and
+then eventually succeeds on a retry, `SESSION` (0) can commit (independently of a `TX`
+block).  This is similar to the vanilla "best-efforts-one-phase-commit" scenario. The
+worst that can happen is a duplicate message when the `RETRY` (2) succeeds and the
+`SESSION` (0) cannot commit (for example, because the message system is unavailable).
+
+[[statelessRetryCannotRecover]]
+=== Stateless Retry Cannot Recover
+
+The distinction between a stateless and a stateful retry in the typical example above is
+important.  It is actually ultimately a transactional constraint that forces the
+distinction, and this constraint also makes it obvious why the distinction exists.
+
+We start with the observation that there is no way to skip an item that failed and
+successfully commit the rest of the chunk unless we wrap the item processing in a
+transaction.  Consequently, we simplify the typical batch execution plan to be as
+follows:
+
+----
+
+0   |  REPEAT(until=exhausted) {
+|
+1   |    TX {
+2   |      REPEAT(size=5) {
+|
+3   |        RETRY(stateless) {
+4   |          TX {
+4.1 |            input;
+4.2 |            database access;
+|          }
+5   |        } RECOVER {
+5.1 |          skip;
+|        }
+|
+|      }
+|    }
+|
+|  }
+
+----
+
+The preceding example shows a stateless `RETRY` (3) with a `RECOVER` (5) path that kicks
+in after the final attempt fails.  The `stateless` label means that the block is repeated
+without re-throwing any exception up to some limit.  This only works if the transaction,
+`TX` (4), has propagation NESTED.
+
+If the inner `TX` (4) has default propagation properties and rolls back, it pollutes the
+outer `TX` (1). The inner transaction is assumed by the transaction manager to have
+corrupted the transactional resource, so it cannot be used again.
+
+Support for NESTED propagation is sufficiently rare that we choose not to support
+recovery with stateless retries in the current versions of Spring Batch.  The same effect
+can always be achieved (at the expense of repeating more processing) by using the
+typical pattern above.

spring-batch-docs/src/main/asciidoc/whatsnew.adoc

@@ -0,0 +1,9 @@
+:batch-asciidoc: ./
+:toc: left
+:toclevels: 4
+
+[[whatsNew]]
+
+== What's New in Spring Batch 5.0
+
+TDB

spring-batch-docs/src/main/javadoc/overview.html

@@ -0,0 +1,17 @@
+<html>
+<body>
+<p>
+	This document is the API specification for <a href="https://github.com/spring-projects/spring-batch" target="_top">Spring Batch</a>
+</p>
+<div id="overviewBody">
+	<p>
+		For further API reference and developer documentation, see the
+		<a href="https://docs.spring.io/spring-batch/docs/current/reference/html/index.html" target="_top">
+		Spring Batch reference documentation</a>.
+		That documentation contains more detailed, developer-targeted
+		descriptions, with conceptual overviews, definitions of terms,
+		workarounds, and working code examples.
+	</p>
+</div>
+</body>
+</html>