diff --git a/src/site/apt/blotter.apt b/src/site/apt/blotter.apt deleted file mode 100644 index 3e58e8133..000000000 --- a/src/site/apt/blotter.apt +++ /dev/null @@ -1,104 +0,0 @@ - ------ - Spring Batch-Retry Comments - ------ - Dave Syer - ------ - February 2007 - -Open Comments and Questions - -* Batches and Asynchronous JMS - - There are a large number of common concerns between - <<>> and <<>>. In - fact one could imagine <<>> being a - simple example of a batch, something like: - -+--- -RepeatTemplate template = new RepeatTemplate(); - -template.setTaskExecutor(new SimpleAsyncTaskExecutor()); - -template.setTerminationPolicy(new TerminateNeverPolicy()); - -template.execute(new JmsItemProviderCallback(jmsTemplate)); -+--- - - * Instead of setting the transactionManager in - <<>>, wrap the callback in a - transaction proxy. - - * Instead of setting sessionTransacted=true in the - <<>>, set it on the - <<>>. - - I think that would give me 80% of the functionality in a - <<>> without any changes to - <<>>. Maybe the other 20% would be useful additions - to <<>> anyway (like being able to stop and start). - Maybe there is a case for sharing some code, e.g. a base class. - Maybe the batch project should be an offshoot from the core.task - package. It certainly looks like the - <<>> could be a lot simpler (and - easier to test), if it delegates transactional properties to - something not a lot different from a <<>>. - -Resolved - -* Using <<>> in Batches - - * What can you do with a <<>> that you couldn't do with - a <<>>? Maybe <<>> should be a - <<>>, or use one to execute the batch? Probably the - latter would work best, on the basis of preferring composition to - inheritance generally. - -* Asynchronous Batching - - * Can you run a batch asynchronously? Would need a thread-safe - <<>> that can be shared amongst participating - threads, and used to determine termination conditions. - - * If <<>> used a <<>> to execute its - tasks, asynchronous batch might be as simple as using an - asynchronous <<>> internally - the same - <<>> would be able to operate in both modes, just - by changing the <<>>. - -* Using RetryContext to Stash State for the Policies - - E.g. in <<>>: - -+--- - protected void setupContext(RetryCallback callback, - RetryContext context) { - - if (callback instanceof AttributeAccessor) { - AttributeAccessor accessor = (AttributeAccessor) callback; - String[] names = accessor.attributeNames(); - for (int i = 0; i < names.length; i++) { - String name = names[i]; - context.setAttribute(name, accessor.getAttribute(name)); - } - } - } - -+--- - - But this is pants: it makes the callback stateful. We can't store - state in the callback to do with the current item becaue the - callback might (will?) be shared between attempts in a concurrent - system. - - So what to do? Nothing - the state for retry is nothing to do with - the callback, and it is natural to store it in the context. - -* Asynchronous Batching - Thread Safe Context - - * What would happen if several threads were sharing a context object - via a synchronization manager in a thread local (like - <<>>)? The context itself had - better be thread safe, otherwise the concurrent peers might assume - that they have the only copy of the context and try and modify it. - - diff --git a/src/site/apt/cases/parallel.apt b/src/site/apt/cases/parallel.apt index fd00e1e64..d9944f2d8 100644 --- a/src/site/apt/cases/parallel.apt +++ b/src/site/apt/cases/parallel.apt @@ -32,7 +32,7 @@ Use Case: Massively Parallel Batch Processing * A data source with multiple chunks (commitable units) - more chunks than parallel processes. - * A way for the container to launch parallel processes. + * A way for the framework to launch parallel processes. * Success @@ -43,19 +43,19 @@ Use Case: Massively Parallel Batch Processing * Description - [[1]] Container splits input data into partitions. + [[1]] Framework splits input data into partitions. - [[1]] Container sends input data (or references to them) to + [[1]] Framework sends input data (or references to them) to processing nodes. [[1]] Processing nodes act independently, converting the input data and sending it transactionally to output source (as per normal single process batch). - [[1]] Container collects status data from individual nodes for + [[1]] Framework collects status data from individual nodes for reporting and auditing. - [[1]] When all nodes are complete Container decides that batch is + [[1]] When all nodes are complete Framework decides that batch is complete finishes processing. * Variations @@ -64,17 +64,17 @@ Use Case: Massively Parallel Batch Processing an internal node failure have different implications for how to proceed. In both cases, however - [[1]] Container catches exception and classifies it. Rolls back + [[1]] Framework catches exception and classifies it. Rolls back current transaction to preserve state of data (input and output). - [[1]] Container saves state for restart from last known good + [[1]] Framework saves state for restart from last known good point, including a pointer to the next input record. Then if a processing node detects bad data in the input source, it cannot be restarted or re-distributed because the data need to be modified for a successful outcome. - [[1]] Container alerts Operator of the location and nature of the + [[1]] Framework alerts Operator of the location and nature of the failure. [[1]] Operator waits for batch to finish - the overall status will @@ -82,7 +82,7 @@ Use Case: Massively Parallel Batch Processing [[1]] Operator fixes problem and restarts batch. - [[1]] Container does not re-process data that has already been + [[1]] Framework does not re-process data that has already been processed successfully. The parallel processing nodes are used as before. @@ -90,7 +90,7 @@ Use Case: Massively Parallel Batch Processing If a processing node fails unrecoverably (e.g. after retry timeout), but with no indication that the input data were bad, then the data - can be re-used: Container returns unprocessed input data, and + can be re-used: Framework returns unprocessed input data, and redistributes it to other nodes. * Implementation @@ -192,7 +192,7 @@ batchTemplate.iterate(new ItemProviderRepeatCallback(provider, processor)); result might be unpredictable if data can be added to an input source while it is being read. - * If only one query is done by the Container and the results shared + * If only one query is done by the Framework and the results shared out amongst the nodes we face the issue of how to send the data between nodes. Performance problems might ensue. Plus (more seriously) the individual nodes would now need a different diff --git a/src/site/apt/cases/restart.apt b/src/site/apt/cases/restart.apt index bd4296ec9..8a71279d8 100644 --- a/src/site/apt/cases/restart.apt +++ b/src/site/apt/cases/restart.apt @@ -19,7 +19,7 @@ Use Case: Manual Restart After Failure * Scope Any batch should be able to restart gracefully, even if (depending - on chosen container or client implementation) it might have to go + on chosen execution or client implementation) it might have to go right back to the beginning. * Preconditions @@ -28,8 +28,8 @@ Use Case: Manual Restart After Failure restart will be able to carry on processing a batch from where it left off. - * There exists a presistent storage mechanism for the initial - conditions. + * There exists a persistent storage mechanism for the initial + conditions. * Success @@ -41,12 +41,12 @@ Use Case: Manual Restart After Failure [[1]] A batch operation encounters an exception which forces the process to stop processing. - [[1]] Container catches exception and classifies it. + [[1]] Framework catches exception and classifies it. - [[1]] Container logs event with enough information to identify the + [[1]] Framework logs event with enough information to identify the location of the job and the nature of the problem. - [[1]] Container saves initial condition from last commit point, to + [[1]] Framework saves initial condition from last commit point, to enable restart to start from the last known good operation. [[1]] Operator fixes problem (e.g. makes missing resource available, @@ -54,7 +54,7 @@ Use Case: Manual Restart After Failure [[1]] Operator restarts batch. - [[1]] Container loads initial conditions and continues processing. + [[1]] Framework loads initial conditions and continues processing. * Variations diff --git a/src/site/apt/cases/steps.apt b/src/site/apt/cases/steps.apt index 4d21c1667..cce7226e1 100644 --- a/src/site/apt/cases/steps.apt +++ b/src/site/apt/cases/steps.apt @@ -54,20 +54,20 @@ Use Case: Sequential Processing of Dependent Steps The vanilla successful case proceeds as follows: - [[1]] Container logs the start of a step, uniquely indentifying + [[1]] Framework logs the start of a step, uniquely indentifying the initial conditions. - [[1]] Container stores internal state so that initial conditions + [[1]] Framework stores internal state so that initial conditions can be re-created in the event of a restart. [[1]] Step execution proceeds as per one of the other use cases (e.g. {{{file-to-database.html}Copy File to Database}}), including transactional behaviour. - [[1]] Client instructs Container to store internal state needed by + [[1]] Client instructs Framework to store internal state needed by further steps (e.g. cached reference data). - [[1]] Container logs successful completion of step, and stores + [[1]] Framework logs successful completion of step, and stores [[1]] Repeat for next and subsequent steps. Internal state is passed from one state to the next. @@ -85,35 +85,35 @@ Use Case: Sequential Processing of Dependent Steps [[1]] Operator restarts batch with no configuration or input data changes. - [[1]] Container resumes batch from the last commit point of the + [[1]] Framework resumes batch from the last commit point of the failed step. [[1]] Sequence completes normally. - The process above could be carried out by the container entirely (no + The process above could be carried out by the framework entirely (no need for operator intervention) if a retry policy is in effect. ** Failure of Step Owing to Bad Initial State If a step fails because it receives bad data from an earlier step, - the Container cannot recover without intervention. + the Framework cannot recover without intervention. [[1]] Operator attempts to restart without doing anything to fix the problem. - [[1]] Container detects bad initial state immediately and fails + [[1]] Framework detects bad initial state immediately and fails fast. If the original problem can be located and fixed (e.g. input data for earlier step is revised): - [[1]] Operator restarts batch signalling to container which step + [[1]] Operator restarts batch signalling to framework which step to begin with. - [[1]] Container locates initial state for the first step to be + [[1]] Framework locates initial state for the first step to be executed. - [[1]] Container starts execution from the beginning of the desired + [[1]] Framework starts execution from the beginning of the desired state. This time the input data are different, so the sequence can complete normally. diff --git a/src/site/apt/changelog.apt b/src/site/apt/changelog.apt index 1d789e3a6..8f2bd585c 100644 --- a/src/site/apt/changelog.apt +++ b/src/site/apt/changelog.apt @@ -1,7 +1,3 @@ Changelog: Spring Batch - See the individual subprojects for their changelogs: - - * {{{spring-batch-infrastructure/changelog.html}Infrastructure}} - - * {{{spring-batch-container/changelog.html}Container}} + See the changes report generated from {{{issue-tracking.html}JIRA}}. diff --git a/src/site/apt/index.apt b/src/site/apt/index.apt index e22508960..7b81f5889 100644 --- a/src/site/apt/index.apt +++ b/src/site/apt/index.apt @@ -11,75 +11,29 @@ Introduction Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. Spring Batch builds upon the productivity, POJO-based development approach, and general ease of use capabilities people have come to know from the Spring Framework, while making it easy for developers to access and leverage more advance enterprise services when necessary. - Spring Batch provides reusable functions that are essential in - processing large volumes of records, including logging/tracing, - transaction management, job processing statistics, job restart, - skip, and resource management. It also provides more advance - technical services and features that will enable extremely - high-volume and high performance batch jobs though optimization and - partitioning techniques. Simple as well as complex, high-volume - batch jobs can leverage the framework in a highly scalable manner to - process significant volumes of information. + Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advance technical services and features that will enable extremely high-volume and high performance batch jobs though optimization and partitioning techniques. Simple as well as complex, high-volume batch jobs can leverage the framework in a highly scalable manner to process significant volumes of information. - Spring Batch is part of the - {{{http://www.springframework.org/sub-projects}Spring Portfolio}}. + Spring Batch is part of the {{{http://www.springframework.org/sub-projects}Spring Portfolio}}. * Spring Batch Architecture - Spring Batch is designed with extensibility and a diverse group of - end users in mind. The figure below shows a sketch of the layered - architecture that supports the extensibility and ease of use for - end-user developers. + Spring Batch is designed with extensibility and a diverse group of end users in mind. The figure below shows a sketch of the layered architecture that supports the extensibility and ease of use for end-user developers. -[images/ContainerLayer.png] Spring Batch Architecture showing -Infrastructure and Container Layers. Potential container -implementations support different platforms and end-user goals from -the same blocks of business logic in the Application Layer. +[images/ExecutionEnvironment.png] Spring Batch Architecture showing Infrastructure and Execution Layers. Potential execution strategy implementations support different platforms and end-user goals from the same blocks of business logic in the Application Layer. - The initial release provides an Infrastructure layer in the form of - low level tools. There is also a simple Container application, - using the infrastructure in its implementation. The container - provides robust features for traceability and management of the - batch lifecycle. A key goal is that the management of the batch - process (locating a job and its input, starting, scheduling, - restarting, and finally processing to created results) should be as - easy as possible for developers. + The initial release provides an Infrastructure layer in the form of low level tools. There is also a simple execution environment, using the infrastructure in its implementation. The execution environment provides robust features for traceability and management of the batch lifecycle. A key goal is that the management of the batch process (locating a job and its input, starting, scheduling, restarting, and finally processing to created results) should be as easy as possible for developers. - The Infrastructure provides the ability to batch operations - together, and to retry an piece of work if there is an exception. - Both requirements have a transactional flavour, and similar concepts - are relevant (propagation, synchronisation). They also both lend - themselves to the template programming model common in Spring, - c.f. <<>>, <<>>, - <<>>. + The Infrastructure provides the ability to batch operations together, and to retry an piece of work if there is an exception. Both requirements have a transactional flavour, and similar concepts are relevant (propagation, synchronisation). They also both lend themselves to the template programming model common in Spring, c.f. <<>>, <<>>, <<>>. - The Simple Batch Execution Container is the first container available. It provides a robust set of integrated features including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management to enable the management of the full lifecycle of traditional batch processing. A number of sample jobs are packaged with this container and are described in detail to more clearly articulate usage and capabilities of the container. + The simple batch execution environment is the reference implementation, expressed through the execution module of the Spring Batch product. It provides a robust set of integrated features including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management to enable the management of the full lifecycle of traditional batch processing. A number of sample jobs are packaged with this environment and are described in detail to more clearly articulate its usage and capabilities. * Roadmap - Once the framework is released it can be used immediately to - simplify batch optimisations and automatic retries. The framework - is oriented around application developers not needing to know any - details of the framework - there are a few application developer - interfaces that can be used for convenient construction of data - processing pipelines, but apart from that we support as close to a - POJO programming model as is practical. This is similar to the - approach taken in Spring Core in the area of DAO implementation. + Once the framework is released it can be used immediately to simplify batch optimisations and automatic retries. The framework is oriented around application developers not needing to know any details of the framework - there are a few application developer interfaces that can be used for convenient construction of data processing pipelines, but apart from that we support as close to a POJO programming model as is practical. This is similar to the approach taken in Spring Core in the area of DAO implementation. - A Partitioned Batch Execution Container is also being developed that will provide alternate scaling solutions. This container will provide more advance technical services and features to enable extremely high-volume and high performance batch jobs though proven optimization and partitioning techniques. Proven scaling techniques will be provided as partitioned strategies allowing users to spread the load across a pool of clustered J2EE application servers. There are also discussions to leverage grid technologies as an alternate scaling solution. + A partitioned step execution strategy is also being developed that will provide alternate scaling solutions. This will provide more advanced technical services and features to enable extremely high-volume and high performance batch jobs though proven optimization and partitioning techniques. Proven scaling techniques will be provided as partitioned strategies allowing users to spread the load across a pool of clustered J2EE application servers. There are also discussions to leverage grid technologies as an alternate scaling solution. - Matt Welsh's work shows that - {{{http://www.eecs.harvard.edu/~mdw/proj/seda/}SEDA}} has enormous - benefits over more rigid processing architectures, and messaging - containers give us a lot of resilience out of the box. So we also - want to provide a more SEDA flavoured container, or container - support, as well as supporting the more traditional ETL style - approach. There might be a tie in with Mule and/or other ESB tools - here, giving the benefit of a very scalable architecture, where the - choice of transport and distribution strategy can be made as late as - possible. The same application code could be used in principle for - a standalone tool processing a small amount of data, and a massive - enterprise-scale bulk-processing engine. + Matt Welsh's work shows that {{{http://www.eecs.harvard.edu/~mdw/proj/seda/}SEDA}} has enormous benefits over more rigid processing architectures, and messaging environments give us a lot of resilience out of the box. So we also want to provide a more SEDA flavoured execution environments, as well as supporting the more traditional ETL style approach. There might be a tie in with Mule and/or other ESB tools here, giving the benefit of a very scalable architecture, where the choice of transport and distribution strategy can be made as late as possible. The same application code could be used in principle for a standalone tool processing a small amount of data, and a massive enterprise-scale bulk-processing engine. * Background @@ -90,9 +44,4 @@ the same blocks of business logic in the Application Layer. Accenture is contributing previously proprietary batch processing architecture frameworks -- based upon decades worth of experience in building batch architectures with the last several generations of platforms (i.e., COBOL/Mainframe, C++/Unix, and now Java/anywhere) -- to the Spring Batch project along with committer resources to drive support, enhancements, and the future roadmap. The collaborative effort between Accenture and Interface21 aims to promote the standardization of software processing approaches, frameworks, and tools that can be consistently leveraged by enterprise users when creating batch applications. Companies and government agencies desiring to deliver standard, proven solutions to their enterprise IT environments will benefit from Spring Batch. - - -* Links: - - * A discussion {{{blotter.html}blotter}}. diff --git a/src/site/apt/sitemap.apt b/src/site/apt/sitemap.apt index 6f46215b9..a7bf864a5 100644 --- a/src/site/apt/sitemap.apt +++ b/src/site/apt/sitemap.apt @@ -18,7 +18,9 @@ Spring Batch Site Map * Integration Tests - reports on tests of infrastructure - * Container - CI and technical information about container layer + * Core - CI and technical information about core domain + + * Execution - CI and technical information about execution (implementation of core) * Main Site @@ -53,11 +55,11 @@ Spring Batch Site Map * Infrastructure - How to use the core API - * Simple Container + * Simple Execution Environment - * Partitioning Container + * Partitioning Execution Environment - * Other Containers? + * Other Execution Environments? * Changelog @@ -89,9 +91,16 @@ Spring Batch Site Map * CI Reports (same as for infrastructure). -* Container +* Core - Should the use case go here (showing which ones are implemented)? + * Changelog + + * Project information (duplicated from Main Site - Maven "feature" - + TODO: find a way to switch them off in sub-projects) + + * CI Reports (same as for infrastructure). + +* Execution * Changelog @@ -99,3 +108,13 @@ Spring Batch Site Map * CI Reports (same as for infrastructure). +* Integration Tests + + * Changelog + + * Project information (duplicated from Main Site - Maven "feature" - + TODO: find a way to switch them off in sub-projects) + + * CI Reports (same as for infrastructure). + + diff --git a/src/site/apt/snapshots.apt b/src/site/apt/snapshots.apt new file mode 100644 index 000000000..86f84688a --- /dev/null +++ b/src/site/apt/snapshots.apt @@ -0,0 +1,33 @@ + --------- + Snapshots + --------- + Dave Syer + ------ + August 2007 + +Snapshot Builds + + These builds are provided for testing and development purposes only. They are built by a Bamboo process + automatically using the latest snapshot from Subversion. + + The snapshots are deloyed to a Maven2 snapshot repository. To use it, just add the following repository to your POM: + ++--------------- + + spring-s3 + Springframework Maven SNAPSHOT Repository + http://s3.amazonaws.com/maven.springframework.org/snapshot + ++--------------- + + Individual dependencies can then by added like so: + ++--------------- + + org.springframework.batch + spring-batch-core + 1.0-m2-SNAPSHOT + ++--------------- + + Additionally, you can browse the repository with a web browser {{{http://s3browse.com/explore/maven.springframework.org/snapshot/org/springframework/batch}here}}. diff --git a/src/site/ppt/ExecutionEnvironment.ppt b/src/site/ppt/ExecutionEnvironment.ppt new file mode 100644 index 000000000..70875e745 Binary files /dev/null and b/src/site/ppt/ExecutionEnvironment.ppt differ diff --git a/src/site/ppt/Figures.ppt b/src/site/ppt/Figures.ppt new file mode 100644 index 000000000..5126f2e09 Binary files /dev/null and b/src/site/ppt/Figures.ppt differ diff --git a/src/site/resources/images/ContainerLayer.png b/src/site/resources/images/ContainerLayer.png deleted file mode 100644 index 0ae74bfe3..000000000 Binary files a/src/site/resources/images/ContainerLayer.png and /dev/null differ diff --git a/src/site/resources/images/ExecutionEnvironment.png b/src/site/resources/images/ExecutionEnvironment.png new file mode 100644 index 000000000..e574236b3 Binary files /dev/null and b/src/site/resources/images/ExecutionEnvironment.png differ diff --git a/src/site/resources/images/RuntimeDependencies.png b/src/site/resources/images/RuntimeDependencies.png new file mode 100644 index 000000000..a8834c84b Binary files /dev/null and b/src/site/resources/images/RuntimeDependencies.png differ diff --git a/src/site/site.xml b/src/site/site.xml index e58b0ed8e..6b15c906d 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -30,6 +30,7 @@ +