From 1ad8bd803a3d456ebe3ab4c5826c853477010e8e Mon Sep 17 00:00:00 2001 From: Chris Schaefer Date: Fri, 7 Mar 2014 13:21:06 -0500 Subject: [PATCH] Add initial 3.0 documentation * What's new * JSR-352 * Convert Spring Batch Integration apt files into docbook and use as chapter * Remove site dir from Spring Batch Integration where converted apt files used to live --- .../src/site/apt/chunking.apt | 68 --- .../src/site/apt/index.apt | 187 -------- spring-batch-integration/src/site/site.xml | 17 - src/site/docbook/reference/appendix.xml | 7 +- src/site/docbook/reference/index.xml | 13 +- src/site/docbook/reference/jsr-352.xml | 368 +++++++++++++++ .../docbook/reference/readersAndWriters.xml | 2 + .../reference/spring-batch-integration.xml | 433 ++++++++++++++++++ src/site/docbook/reference/step.xml | 12 + src/site/docbook/reference/whatsnew.xml | 126 ++--- 10 files changed, 855 insertions(+), 378 deletions(-) delete mode 100644 spring-batch-integration/src/site/apt/chunking.apt delete mode 100644 spring-batch-integration/src/site/apt/index.apt delete mode 100644 spring-batch-integration/src/site/site.xml create mode 100644 src/site/docbook/reference/jsr-352.xml create mode 100644 src/site/docbook/reference/spring-batch-integration.xml diff --git a/spring-batch-integration/src/site/apt/chunking.apt b/spring-batch-integration/src/site/apt/chunking.apt deleted file mode 100644 index 4173f2814..000000000 --- a/spring-batch-integration/src/site/apt/chunking.apt +++ /dev/null @@ -1,68 +0,0 @@ - ------ - Remote Chunking - ------ - Dave Syer - ------ - March 2008 - -Remote Chunking Implementation - -* Basic Use Case - - Description: - - [[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit) - - [[1]] Worker thread picks up chunk and processes it - - [[1]] Worker thread replies to response channel - - [[1]] Step picks up reply if there is one and aggregates the counts - - [[1]] Step reapeats until no more input data - - [[1]] Step blocks until all the outstanding requests are satisfied - -* Implementation - - A ChunkProcessor acts as a kind of Throttling Asynchronous Messaging - Gateway, which isn't a pattern that is supported out of the box with - Spring Integration. There is a SimpleMessagingGateway that provides - programmatic access to send and receive payloads (instead of - messages), so the pattern can be manually implemented in the - ChunkProcessor. - - The current implementation is in the form of an ItemWriter - (ChunkMessageChannelItemWriter) which is a StepExecutionListener - (blocks and waits for the outstanding responses in the afterStep). - The ChunkProcessor can then simply be a vanilla implementation from - Spring Batch. - - The ChunkMessageChannelItemWriter implements the Throttling part of - the pattern by keeping track of the number of outstanding requests - (which it has to do anyway) and blocking until a response arrives if - the number is above a configurable limit. It wouldn't be necessary - to do this manually in the writer if the messages were only going - over local MessageChannels: the requests would either be processed - serially in a single thread, or else there would be a thread pool - with limited size controlling the workers. But since the messages - are going to JMS we need to either explicitly throttle in the writer - (or else rely on vendor features for producer flow control), - otherwise the JMS Queue could easily be overwhelmed and start - barfing (which happened in one of the early prototypes on an - Accenture project). - - Throttling MessageChannel.send() might be something Spring - Integration could do, but it only makes sense really in the context - of this gateway pattern (because you need something to react against - to decide when to release another send). - - The gateway is used to send requests to the workers, and then to - receive responses in the same thread, but only waiting for a - response when the step is complete. To implement this with JMS - backed channels we need a Spring Integration inbound adapter that - translates PollableChannel.receive() into - JmsTemplate.receiveAndConvert(). In the unlikely event of a problem - in the receiver the JMS message should roll back. - JmsDestinationPollingAdapter actually almost does what we need but - there is no support for configuring it without a scheduled poller. diff --git a/spring-batch-integration/src/site/apt/index.apt b/spring-batch-integration/src/site/apt/index.apt deleted file mode 100644 index 5b6a23462..000000000 --- a/spring-batch-integration/src/site/apt/index.apt +++ /dev/null @@ -1,187 +0,0 @@ - ------ - Spring Integration Batch - ------ - Dave Syer - ------ - March 2008 - -Overview of the Spring Integration Batch Module - - Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring Integration. Here is a list. These are features that can extend Spring Batch, or use Spring batch features in the context of Spring Integration. Work in progress waiting for community feedback. Many issues to do with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result of these use cases being prototyped. - -*---+---+---+---+---+ -|<>|<>|<>|<>|<>| -*---- -|1|{{{Triggers}Message triggers job}}|Complete|launch|Complete. Also lots of opportunities with monitoring progress.| -*---- -|2|{{{Chunking}Chunking and multi-VM job execution}}|Complete|chunk|Failures might need some analysis. Use of stateful StepExecutionListener requires use of step scope.| -*---- -|3|{{{Aggregator}Asynchronous Aggregator}}|Unstarted| | | -*---- -|4|{{{jobs}Stateful and non-linear jobs}} -> job = flow|Complete|job|Simple use cases work well with Spring Batch 2.0 and no Integration features.| -*---- -|5|{{{Flexible}Flexible item processing model}} (as message flow) -> step = flow|Complete|item|Complete (v. simple using MessagingGateway). Unit tests only.| -*---- -|6|{{{repeat}Automatic repeat / retry}}|Complete|retry (unit test)|Unit tests only, since it just uses existing features.| -*---- -|7|{{{files}Restartable file processing}}|Complete|file|Seems to hang together. Not tested thoroughly, but apparently someone is using it.| -*---- -|8|{{{async}Asynchronous item processing}}|Complete|async|A general purpose ItemProcesor that returns a Future.| -*---- - - Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes. If we implement 1, then we also don't need to do any more scheduling and triggering in Spring Batch. - - Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one. We implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration later (with probably a split of repeat/retry out of Batch at that time). - -* Message {Triggers} Job - - Description: - - [[1]] User sends message to channel (maybe through a scheduler) - - [[1]] System interprets message payload as parameters for JobLauncher - - [[1]] System launches job execution - - [[1]] If message had a replyTo, System acknowledges with JobExecution - - [[1]] User accepts response and uses it to monitor progress - - Variation: - - [[1]] System waits for job to finish and replies when it is over - - [[1]] User polls for replies and gets notification about end of execution - - Variation: - - [[1]] User wants to block on send and only receive response when job is done - -* {Chunking} and Multi-VM - - Description: - - [[1]] Step flushes chunk as message to outgoing channel (repeat up to throttle limit) - - [[1]] Worker thread picks up chunk and processes it - - [[1]] Worker thread replies to response channel - - [[1]] Step picks up reply and aggregates the counts - - [[1]] Step blocks until all the requests are satisfied - - TODO: failure modes - -* Asynchronous {Aggregator} - - Job is executed over long period. Many jobs can be executing concurrently. - - Description: - - [[1]] Input stage for each job: System reads all items and marks with the job instance id in a durable repository (staging table) - - [[1]] System sends each item (or chunks of items that can be processed together as appropriate) to a channel - - [[1]] Items flow through message pipeline, occasionally pausing until certain conditions are met, possibly for days at a time - - [[1]] Aggregator sits and waits for all items in a job to be finished and then wraps up - -* Stateful and non-linear {jobs} - - Dependencies beyween steps and conditional flow between steps. Each handler node in a message flow is a step execution, with all the robustness guarantees from the Spring Batch meta data. - - Description: - - [[1]] User launches job - - [[1]] System sends message to channel containing job execution - - [[1]] Handler accepts message and executes a step - - [[1]] Handler translates result of step execution into the same form that it accepted the original request - - [[1]] System routes message to next handler, possibly dynamically based on data in the message - - [[1]] Next handler does the same... until one of the routing decisions leads to a reply channel - - [[1]] System receives reply and transfers information to job execution (e.g. status) as necessary - - Variation: failure in one of the handlers - - Variation: restart after failure - -* {Flexible} item processing model - - Description: - - [[1]] Step hands item to ItemWriter - - [[1]] Item is converted to message and sent to synchronous flow - - [[1]] Handler accepts message and does something with item - - [[1]] System routes result to next handler, possibly dynamically - - Variation: failure - - [[1]] Handler throws exception - - [[1]] System propagates exception up to ItemWriter (forces rollback under normal circs - hence synchronous flow) - -* Automatic repeat / retry - - Description ({repeat}): - - [[1]] User sends message to channel - - [[1]] System start a transaction and reseives message, then processes it - - [[1]] User sends another message - - [[1]] System receives and processes it in the same transaction - - [[1]] ... repeat ... - - [[1]] System determines that batch is complete and commits transaction - -* Restartable file processing - - Large {files} need to be processed, so message payload of file contents is not practical. One line or XML event per message with failover and restartability from Spring Batch. - - Description: - - [[1]] User triggers file processing (sends message, copies file to directory, etc.) - - [[1]] System starts new job - - [[1]] System processes file line by line (or even by event), wrapping each one as a message and sending it to a synchronous flow - - [[1]] System commits periodically (as determined by Spring Batch step configuration) - - Variation: failure and restart - - [[1]] Item processing fails - - [[1]] System aborts job and sends message to failure channel (or failure message to normal reply channel) - - [[1]] Operator fixes problem and triggers restart (another message channel?) - - [[1]] System restarts job for same file at point where it left off - - [[1]] System completes processing - - [[1]] System sends sucess message to reply channel - - Variation: send to asynchronous flow. Same as main use case but item message is sent to asynchronous flow. Not as robust because if the lights go out then meesages will be lost, but at least a large file can be split into smaller chunks. - -* Asynchronous item processing - - This is actually a variation on {{{Flexible}flexible item processing model}}. - - Description ({async}): - - [[1]] ItemProcessor executes in background (non-transactionally) - - [[1]] ItemWriter collects outputs from futures before phyically writing data - diff --git a/spring-batch-integration/src/site/site.xml b/spring-batch-integration/src/site/site.xml deleted file mode 100644 index 4aac57281..000000000 --- a/spring-batch-integration/src/site/site.xml +++ /dev/null @@ -1,17 +0,0 @@ - - - - - - - - - - - - - - - - - diff --git a/src/site/docbook/reference/appendix.xml b/src/site/docbook/reference/appendix.xml index 41bdf3cbe..a947278f1 100644 --- a/src/site/docbook/reference/appendix.xml +++ b/src/site/docbook/reference/appendix.xml @@ -78,7 +78,8 @@ Reads via iBATIS based on a query. Pages through the rows so that large datasets can be read without - running out of memory. See HOWTO - Read from a Database + running out of memory. See HOWTO - Read from a Database. This + ItemReader is now deprecated as of Spring Batch 3.0. @@ -238,8 +239,8 @@ IbatisBatchItemWriter - Writes items in a batch using the SqlMapClientTemplate - execute() method + Writes items in a batch using the iBatis API's + directly. This ItemWriter is deprecated as of Spring Batch 3.0. diff --git a/src/site/docbook/reference/index.xml b/src/site/docbook/reference/index.xml index 94a31c0b8..751f283d6 100644 --- a/src/site/docbook/reference/index.xml +++ b/src/site/docbook/reference/index.xml @@ -42,11 +42,10 @@ Minella - - Chris - Schaefer - - + + Chris + Schaefer + @@ -79,6 +78,10 @@ + + + + diff --git a/src/site/docbook/reference/jsr-352.xml b/src/site/docbook/reference/jsr-352.xml new file mode 100644 index 000000000..1ae89eca9 --- /dev/null +++ b/src/site/docbook/reference/jsr-352.xml @@ -0,0 +1,368 @@ + + + + JSR-352 Support + + As of Spring Batch 3.0 support for JSR-352 has been fully implemented. This section will briefly describe high level JSR-352 specific concepts. It is encouraged to read the full JSR-352 spec, located at the JSR-352 home page: https://jcp.org/en/jsr/detail?id=352 + +
+ Batch Contexts + + JSR-352 Batch Contexts provide information about the batch job and interaction with the batch runtime. Context objects can be injected into fields of batch artifact classes using the @Inject annotation. + +
+ JobContext + + The JobContext represents the entire batch job. There is one JobContext per job execution and exists for the life of the Job. +
+ +
+ StepContext + + The StepContext represents the current executing step. There is one StepContext per step execution and exists for the life of the Step. +
+
+
+ Configuring and Running a Job + +
+ Configuring a Job + +
+ JSR-352 namespace + + JSR-352 API classes reside in the package namespace of javax.batch. The XML namespace also known as the Job Specification Language (JSL) in the language in which Jobs are defined in. The JSL is represented by an XML schema. Refer to the JSR-352 spec document or the XSD itself for more details. +
+
+ batch.xml configuration + + The batch.xml file contains a mapping of batch artifacts to be loaded and used by the runtime. Batch artifact definitions consist of a reference identifier and a fully qualified class name. The batch.xml file resides in the META-INF directory. +
+
+ Spring configuration + + In addition to configuring batch artifacts in the batch.xml file, Spring Batch allows you to configure batch artifacts in the typical Spring bean fashion. To utilize this configuration method simply define your JSR-352 job inside a Spring configuration file and reference your Spring beans via the ref tag as shown in the snippet below. + + <?xml version="1.0" encoding="UTF-8"?> +<beans xmlns="http://www.springframework.org/schema/beans" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://www.springframework.org/schema/beans + http://www.springframework.org/schema/beans/spring-beans.xsd + http://xmlns.jcp.org/xml/ns/javaee + http://xmlns.jcp.org/xml/ns/javaee/jobXML_1_0.xsd"> + + <job id="job1" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0"> + <step id="step1"> + <chunk> + <reader ref="testReader"/> + </chunk> + </step> + ... + </job> + + <bean id="testReader" class="TestItemReader" scope="step"/> +</beans> +
+
+
+ Running a Job + +
+ JsrJobOperator + + The JsrJobOperator is a javax.batch.operations.JobOperator implementation which supports various operations for dealing with batch jobs. Job XML files are expected to reside in the META-INF/batch-jobs directory. + + Typically used JobOperator methods for operating a job include: + + + start(String jobXMLName, Properties jobParameters) +restart(long executionId, Properties restartParameters) +stop(long executionId) +abandon(long executionId) + + When providing the jobXMLname to the start method of JobOperator, the META-INF/batch-jobs prefix nor the .xml file extension should be added. +
+
+
+ Intercepting Job Execution + +
+ JobListener + + As with Spring Batch, JSR-352 allows for Job level listeners. Job listeners are configured as a child element of the Job in the JSL and implement the javax.batch.api.listener.JobListener interface. +
+
+
+
+ Job Properties + +
+ Property Support + + JSR-352 allows for properties to be defined at the Job, Step and batch artifact level by way of configuration in the JSL. Batch properties are configured at each level in the following way: + + + <properties> + <property name="propertyName1" value="propertyValue1"/> + <property name="propertyName2" value="propertyValue2"/> +</properties> + + Properties may be configured on any batch artifact. +
+
+ @BatchProperty annotation + + Properties are referenced in batch artifacts by annotating class fields with the @BatchProperty and @Inject annotations. As defined by JSR-352, fields for properties must be String typed. Any type conversion is up to the implementing developer to perform. + + An ItemReader artifact could be configured with a properties block such as the one described above and accessed as such: + public class MyItemReader implements ItemReader { + @Inject + @BatchProperty + private String propertyName1; + + ... +} + + The value of the field "propertyName1" will be "propertyValue1" +
+
+ Property Substitution + + Property substitution is provided by way of operators and simple conditional expressions. The general usage is #{operator['key']}. + Supported operators: + + + + jobParameters - access job parameter values that the job was started/restarted with. + + + jobProperties - access properties configured at the job level of the JSL. + + + systemProperties - access named system properties. + + + partitionPlan - access named property from the parition plan of a partitioned step. + + + + + #{jobParameters['unresolving.prop']}?:#{systemProperties['file.separator']} + The left hand side of the assignment is the expected value, the right hand side is the default value. In this example, the result will resolve to a value of the system property file.separator as #{jobParameters['unresolving.prop']} is assumed to not be resolvable. If neither expressions can be resolved, an empty String will be returned. Multiple conditions can be used, which are separated by a ';'. + +
+
+
+ Retry + + JSR-352 provides support for retry by way of listeners for chunk level artifacts. Listeners are configured at the step level contained in a listeners block. + +
+ Retry Listeners + + JSR-352 provides listener interfaces for various cases of retry handling: + + + + + + + + + Artifact Interface + + + Listener Interface + + + + javax.batch.api.chunk.ItemReader + javax.batch.api.chunk.listener.RetryReadListener + + + javax.batch.api.chunk.ItemProcessor + javax.batch.api.chunk.listener.RetryProcessListener + + + javax.batch.api.chunk.ItemWriter + javax.batch.api.chunk.listener.RetryWriteListener + + + + +
+
+
+ Configuring a Step + +
+ Batchlet + + Task oriented step's in the JSR-352 implementation are backed by implementations of javax.batch.api.Batchlet. A Batchlet is analogous to a Spring Batch Tasklet. +
+
+ Controlling Step Flow + +
+ Decider + + JSR-352 decision support for steps, splits and flows are implemented via implementations of the javax.batch.api.Decider interface. +
+
+ Conditional Flow + + JSR-352 allows for both the next attribute and a next element to be present as transition elements. If both are present, the next value specified in the Step attribute will be preferred. +
+
+ Configuring For Stop + + JSR-352 transition elements are matched against in the order they are defined in the JSL. +
+
+
+ Intercepting Step Execution + + JSR-352 provides support for intercepting Step execution by way of listeners. Listeners are configured at the step level contained in a listeners block. + + + + + + + + + Interception Point + + + Listener Interface + + + + Step + javax.batch.api.listener.StepListener + + + Chunk + javax.batch.api.chunk.listener.ChunkListener + + + Item Reading + javax.batch.api.chunk.listener.ItemReadListener + + + Item Processing + javax.batch.api.chunk.listener.ItemProcessListener + + + Item Writing + javax.batch.api.chunk.listener.ItemWriteListener + + + Skip Read + javax.batch.api.chunk.listener.SkipReadListener + + + Skip Process + javax.batch.api.chunk.listener.SkipProcessListener + + + Skip Write + javax.batch.api.chunk.listener.SkipWriteListener + + + + +
+
+ Completion Policies + +
+ Checkpoint Algorithm + + By default, completion is based on reaching the end of the chunk dictated by its specified (or default) size. A custom algorithm can be created by implementing the javax.batch.api.chunk.CheckpointAlgorithm interface and referencing it via the chunk attribute "checkpoint-policy". +
+
+
+
+ ItemReaders and ItemWriters + + JSR-352 provides interfaces to read, process and write data. + + + + + + + + + Role + + + Interface + + + + Reading + javax.batch.api.chunk.ItemReader + + + Processing + javax.batch.api.chunk.ItemProcessor + + + Writing + javax.batch.api.chunk.ItemProcessor + + + + +
+
+ Scaling + +
+ Partitioning + +
+ Partition Plan + + A partition plan defines the attributes of a partitioned step. Both chunk oriented and task oriented Step's may be partitioned. Some of these items include the number of partitions, threads and the partitions properties. For example consider the following partitioned Step sample: + ... +<step> + ... + <partition> + <plan partitions="10" threads="2"> + <properties> + <property name="partitionProperty1" value="value1"/> + </properties> + </plan> + </partition> + ... +</step> +... + +
+
+ Partition Mapper + + The Partition Mapper provides a way to programmatically calculate the number of partitions and threads for a partitioned Step. The Partition Mapper is a child element of partition and implementations should implement the javax.batch.api.partition.PartitionMapper interface. +
+
+ Partition Reducer + + The Partition Reducer provides a way to programmatically intercept the partitioned Step's lifecycle. The Partition Reducer is a child element of partition and implementations should implement the javax.batch.api.partition.PartitionReducer interface. +
+
+ Partition Collector + + The Partition Collector sends results from each partition to the Partition Analyzer at the end of each checkpoint and again at the end of the partition. The Partition Collector is a child element of partition and implementations should implement the javax.batch.api.partition.PartitionCollector interface. +
+
+ Partition Analyzer + + The Partition Analyzer is a collection point for data returned by the Partition Collector's and can be used to implement custom exit status handling for the individual partitions. The Partition Analyzer is a child element of a partition and implements should implement the javax.batch.api.partition.PartitionAnalyzer interface. +
+
+
+ diff --git a/src/site/docbook/reference/readersAndWriters.xml b/src/site/docbook/reference/readersAndWriters.xml index ddee910c3..04a43ea1f 100644 --- a/src/site/docbook/reference/readersAndWriters.xml +++ b/src/site/docbook/reference/readersAndWriters.xml @@ -2255,6 +2255,8 @@ itemReader.close(executionContext);
IbatisPagingItemReader + This reader is deprecated as of Spring Batch 3.0. + If you use IBATIS for your data access then you can use the IbatisPagingItemReader which, as the name indicates, is an implementation of a paging diff --git a/src/site/docbook/reference/spring-batch-integration.xml b/src/site/docbook/reference/spring-batch-integration.xml new file mode 100644 index 000000000..0d9dacbcb --- /dev/null +++ b/src/site/docbook/reference/spring-batch-integration.xml @@ -0,0 +1,433 @@ + + + + Spring Batch Integration + +
+ Overview of the Spring Integration Batch Module + Many use cases in Spring Batch look like they might be efficiently and concisely implemented in Spring + Integration. Here is a list. These are features that can extend Spring Batch, or use Spring batch features + in the context of Spring Integration. Work in progress waiting for community feedback. Many issues to do + with transactionality and synchronous execution have been raised and fixed in Spring Integration as a result + of these use cases being prototyped. + + + + + + + + + + + + ID + + + Description + + + Status + + + Sub-package + + + Comments + + + + 1 + + Message triggers job + + Complete + launch + Complete. Also lots of opportunities with monitoring progress. + + + 2 + + Chunking and multi-VM job execution + + Complete + chunk + Failures might need some analysis. Use of stateful StepExecutionListener requires use of + step scope. + + + + 3 + + Asynchronous Aggregator + + Unstarted + + + + + 4 + + Stateful and non-linear jobs + -> job = flow + + Complete + job + Simple use cases work well with Spring Batch 2.0 and no Integration features. + + + 5 + + Flexible item processing model + (as message flow) -> step = flow + + Complete + item + Complete (v. simple using MessagingGateway). Unit tests only. + + + 6 + + Automatic repeat / retry + + Complete + retry (unit test) + Unit tests only, since it just uses existing features. + + + 7 + + Restartable file processing + + Complete + file + Seems to hang together. Not tested thoroughly, but apparently someone is using it. + + + + 8 + + Asynchronous item processing + + Complete + async + A general purpose ItemProcesor that returns a Future. + + + + + Numbers 2, 4, 5 have also been identified as high level Spring Batch 2.0 Features or themes. If we + implement 1, then we also don't need to do any more scheduling and triggering in Spring Batch. + + Number 6 from the list (repeat/retry) is more of a Spring Integration pattern than a Spring Batch one. We + implemented it in Spring Batch first, with an eye to seeing about pushing it out into Spring Integration + later (with probably a split of repeat/retry out of Batch at that time). + +
+ Message<anchor id="Triggers"/>Triggers<!-- anchor_end --> Job + + Description: + + + User sends message to channel (maybe through a scheduler) + + + System interprets message payload as parameters for JobLauncher + + + System launches job execution + + + If message had a replyTo, System acknowledges with JobExecution + + + User accepts response and uses it to monitor progress + + + Variation: + + + System waits for job to finish and replies when it is over + + + User polls for replies and gets notification about end of execution + + + Variation: + + + User wants to block on send and only receive response when job is done + + +
+
+ <anchor id="Chunking"/>Chunking<!-- anchor_end --> and Multi-VM + + Description: + + + Step flushes chunk as message to outgoing channel (repeat up to throttle limit) + + + Worker thread picks up chunk and processes it + + + Worker thread replies to response channel + + + Step picks up reply and aggregates the counts + + + Step blocks until all the requests are satisfied + + + TODO: failure modes +
+
+ Asynchronous<anchor id="Aggregator"/>Aggregator<!-- anchor_end --> + + Job is executed over long period. Many jobs can be executing concurrently. + Description: + + + Input stage for each job: System reads all items and marks with the job instance id in a + durable repository (staging table) + + + + System sends each item (or chunks of items that can be processed together as appropriate) to a + channel + + + + Items flow through message pipeline, occasionally pausing until certain conditions are met, + possibly for days at a time + + + + Aggregator sits and waits for all items in a job to be finished and then wraps up + + +
+
+ Stateful and non-linear<anchor id="jobs"/>jobs<!-- anchor_end --> + + Dependencies beyween steps and conditional flow between steps. Each handler node in a message flow is + a step execution, with all the robustness guarantees from the Spring Batch meta data. + + Description: + + + User launches job + + + System sends message to channel containing job execution + + + Handler accepts message and executes a step + + + Handler translates result of step execution into the same form that it accepted the original + request + + + + System routes message to next handler, possibly dynamically based on data in the message + + + + Next handler does the same... until one of the routing decisions leads to a reply channel + + + + System receives reply and transfers information to job execution (e.g. status) as necessary + + + + Variation: failure in one of the handlers + Variation: restart after failure +
+
+ <anchor id="Flexible"/>Flexible<!-- anchor_end --> item processing model + + Description: + + + Step hands item to ItemWriter + + + Item is converted to message and sent to synchronous flow + + + Handler accepts message and does something with item + + + System routes result to next handler, possibly dynamically + + + Variation: failure + + + Handler throws exception + + + System propagates exception up to ItemWriter (forces rollback under normal circs - hence + synchronous flow) + + + +
+
+ Automatic repeat / retry + Description (repeat): + + + + User sends message to channel + + + System start a transaction and reseives message, then processes it + + + User sends another message + + + System receives and processes it in the same transaction + + + ... repeat ... + + + System determines that batch is complete and commits transaction + + +
+
+ Restartable file processing + Largefiles need to be processed, so message payload of file + contents is not practical. One line or XML event per message with failover and restartability from + Spring Batch. + + Description: + + + User triggers file processing (sends message, copies file to directory, etc.) + + + System starts new job + + + System processes file line by line (or even by event), wrapping each one as a message and + sending it to a synchronous flow + + + + System commits periodically (as determined by Spring Batch step configuration) + + + Variation: failure and restart + + + Item processing fails + + + System aborts job and sends message to failure channel (or failure message to normal reply + channel) + + + + Operator fixes problem and triggers restart (another message channel?) + + + System restarts job for same file at point where it left off + + + System completes processing + + + System sends sucess message to reply channel + + + Variation: send to asynchronous flow. Same as main use case but item message is sent to asynchronous + flow. Not as robust because if the lights go out then meesages will be lost, but at least a large file + can be split into smaller chunks. + +
+
+ Asynchronous item processing + This is actually a variation onflexible item processing model. + + Description (async): + + + + ItemProcessor executes in background (non-transactionally) + + + ItemWriter collects outputs from futures before phyically writing data + + +
+
+
+ Remote Chunking Implementation +
+ Basic Use Case + Description: + + + Step flushes chunk as message to outgoing channel (repeat up to throttle limit) + + + Worker thread picks up chunk and processes it + + + Worker thread replies to response channel + + + Step picks up reply if there is one and aggregates the counts + + + Step reapeats until no more input data + + + Step blocks until all the outstanding requests are satisfied + + +
+
+ Implementation + A ChunkProcessor acts as a kind of Throttling Asynchronous Messaging Gateway, which isn't a + pattern that is supported out of the box with Spring Integration. There is a SimpleMessagingGateway that + provides programmatic access to send and receive payloads (instead of messages), so the pattern can be + manually implemented in the ChunkProcessor. + + The current implementation is in the form of an ItemWriter (ChunkMessageChannelItemWriter) which is a + StepExecutionListener (blocks and waits for the outstanding responses in the afterStep). The + ChunkProcessor can then simply be a vanilla implementation from Spring Batch. + + The ChunkMessageChannelItemWriter implements the Throttling part of the pattern by keeping track of + the number of outstanding requests (which it has to do anyway) and blocking until a response arrives if + the number is above a configurable limit. It wouldn't be necessary to do this manually in the + writer if the messages were only going over local MessageChannels: the requests would either be + processed serially in a single thread, or else there would be a thread pool with limited size + controlling the workers. But since the messages are going to JMS we need to either explicitly throttle + in the writer (or else rely on vendor features for producer flow control), otherwise the JMS Queue could + easily be overwhelmed and start barfing (which happened in one of the early prototypes on an Accenture + project). + + Throttling MessageChannel.send() might be something Spring Integration could do, but it only makes + sense really in the context of this gateway pattern (because you need something to react against to + decide when to release another send). + + The gateway is used to send requests to the workers, and then to receive responses in the same thread, + but only waiting for a response when the step is complete. To implement this with JMS backed channels we + need a Spring Integration inbound adapter that translates PollableChannel.receive() into + JmsTemplate.receiveAndConvert(). In the unlikely event of a problem in the receiver the JMS message + should roll back. JmsDestinationPollingAdapter actually almost does what we need but there is no support + for configuring it without a scheduled poller. + +
+
+ \ No newline at end of file diff --git a/src/site/docbook/reference/step.xml b/src/site/docbook/reference/step.xml index b73710892..9630fc4b8 100644 --- a/src/site/docbook/reference/step.xml +++ b/src/site/docbook/reference/step.xml @@ -1727,5 +1727,17 @@ itemWriter.write(items); <bean class="org.springframework.batch.core.scope.StepScope" />
+ +
+ Job Scope + + Job scope, introduced in Spring Batch 3.0 is similar to Step scope in configuration but is a scope for the job context and ensures there is only one instance of a particular bean for the entire Job. To Job scope your beans set the scope to "job": + + <bean id="jobScopedBean" scope="step" class="com.test.MyJobScopedBean"/> + + + + +
diff --git a/src/site/docbook/reference/whatsnew.xml b/src/site/docbook/reference/whatsnew.xml index 0c66f1d51..0d9515df2 100644 --- a/src/site/docbook/reference/whatsnew.xml +++ b/src/site/docbook/reference/whatsnew.xml @@ -1,130 +1,60 @@ + "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"> - What's New in Spring Batch 2.2 + What's New in Spring Batch 3.0 - The Spring Batch 2.2 release has six major themes: + The Spring Batch 3.0 release has five major themes: - Spring Data Integration + JSR-352 Support - Java Configuration + Promote Spring Batch Integration to Spring Batch - Spring Retry + Upgrade to Support Spring 4 and Java 8 - Job Parameters + JobScope Support + + + + SQLite Support -
- Spring Data Integration +
+ JSR-352 Support - Since the 2.0 release of Spring Batch, the Spring Data project has brought - support for the NoSQL movement to Spring. The 2.2 release of Spring Batch has added - support for MongoDB, Neo4j and Gemfire natively through the Spring Data abstractions. - - This release has also added support for writing to any custom Spring Data Repository a - user may write. The RepositoryItemReader and - RepositoryItemWriter each wrap a repository implementation ( - PagingAndSortingRepository and CrudRepository - respectively) to retrieve data from and persist data to. + The JSR-352 (Batch Applications for the Java Platform) specification has been fully implemented. See for more details.
-
- Java Configuration +
+ Promote Spring Batch Integration to Spring Batch - Until 2.2.0 the only option for configuring a job was via XML (either through the batch DSL or - by hand). However, in 2.2.0, Java based configuration has been added as a way to define Spring Batch - Jobs. To support this new configuration option, an annotation and builder classes have been added. What - was previously defined as this: - - <batch> - <job-repository/> - - <job id="myJob"> - <step id="step1".../> - <step id="step2".../> - </job> - - <beans:bean id="transactionManager".../> - - <beans:bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"> - <beans:property name="jobRepository" ref="jobRepository"/> - </beans:bean> -</batch> - - - Can now be configured using the @EnableBatchProcessing annotation and the - provided JobBuilderFactory and StepBuilderFactory as show below: - - @Configuration - @EnableBatchProcessing - @Import(DataSourceCnfiguration.class) - public class AppConfig { - - @Autowired - private JobBuilderFactory jobs; - - @Bean - public Job job() { - return jobs.get("myJob").start(step1()).next(step2()).build(); - } - - @Bean - protected Step step1() { - ... - } - - @Bean - protected Step step2() { - ... - } -} - - The @EnableBatchProcessing annotation makes a number - of common dependencies available for autowiring by default. This list includes a - JobRepsitory, JobLauncher, - JobRegistry, PlatformTransactionManager, - JobBuilderFactory, and a StepBuilderFactory. - More information on how to configure Jobs and Steps with the new - Java config can be found in + The Spring Batch Integration project has been merged into the main Spring Batch code base. Spring Batch Integration provides support for Remote Chunking, Asynchronous handing and so on. See for more details.
-
- Spring Retry +
+ Upgrade to Support Spring 4 and Java 8 - The ability to retry an operation via the RetryTemplate - has always been a feature of Spring Batch. That ability has been identified as a - useful feature for other frameworks (Spring Integration for example). With the 2.2.0 - release, the retry logic has been extracted from Spring Batch into it's own library - called Spring Retry. With this change, there are two main impacts. The first is - that the majority of the org.springframework.batch.retry package - has been moved into this new library. With that move, the package name has also - dropped the batch to become org.springframework.retry. + Spring Batch now depends on Spring 4.0 and has been tested for compatibility against Java 8. Java 6 still remains the default source/target version for the Spring Batch distribution.
-
- Job Parameters +
+ JobScope Support - Prior to the 2.2.0 release of Spring Batch, all parameters pass to a job execution - were used as part of the identity of the job. This limited the ability to change job - parameters during a rerun of a job. To accommodate this use case, 2.2.0 introduced the - idea of non-identifying job parameters. - - By default, job parameters in 2.2.0 are still identifying. However, Spring Batch - now allows a user to specify a parameter not be used in the identity of a job instance. - In order to support this change, the domain model for batch changed. Before 2.2.0, job - parameters were associated with a JobInstance. 2.2.0 and beyond, - they are associated with a JobExecution. This also required the - underlying database schema for the job repository to change. + A new "job" scope has been added, see section for more details.
+
+ SQLite Support + + SQLite support has been added along with database schema scripts. +