Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Refactor parent website content to reflect container -> core / execution changes

Browse Source
This commit is contained in:
dsyer
2007-08-20 21:09:24 +00:00
parent eb8ba63d4e
commit 821084c580
14 changed files with 99 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
src/site/apt/blotter.apt
View File

@@ -1,104 +0,0 @@
------
Spring Batch-Retry Comments
------
Dave Syer
------
February 2007
Open Comments and Questions
* Batches and Asynchronous JMS
There are a large number of common concerns between
<<<DefaultMessageListenerContainer>>> and <<<RepeatTemplate>>>. In
fact one could imagine <<<DefaultMessageListenerContainer>>> being a
simple example of a batch, something like:
+---
RepeatTemplate template = new RepeatTemplate();
template.setTaskExecutor(new SimpleAsyncTaskExecutor());
template.setTerminationPolicy(new TerminateNeverPolicy());
template.execute(new JmsItemProviderCallback(jmsTemplate));
+---
* Instead of setting the transactionManager in
<<<DefaultMessageListenerContainer>>>, wrap the callback in a
transaction proxy.
* Instead of setting sessionTransacted=true in the
<<<DefaultMessageListenerContainer>>>, set it on the
<<<JmsTemplate>>>.
I think that would give me 80% of the functionality in a
<<<DefaultMessageListenerContainer>>> without any changes to
<<<RepeatTemplate>>>. Maybe the other 20% would be useful additions
to <<<RepeatTemplate>>> anyway (like being able to stop and start).
Maybe there is a case for sharing some code, e.g. a base class.
Maybe the batch project should be an offshoot from the core.task
package. It certainly looks like the
<<<DefaultMessageListenerContainer>>> could be a lot simpler (and
easier to test), if it delegates transactional properties to
something not a lot different from a <<<RepeatTemplate>>>.
Resolved
* Using <<<TaskExecutor>>> in Batches
* What can you do with a <<<RepeatTemplate>>> that you couldn't do with
a <<<TaskExecutor>>>? Maybe <<<RepeatTemplate>>> should be a
<<<TaskExecutor>>>, or use one to execute the batch? Probably the
latter would work best, on the basis of preferring composition to
inheritance generally.
* Asynchronous Batching
* Can you run a batch asynchronously? Would need a thread-safe
<<<RepeatContext>>> that can be shared amongst participating
threads, and used to determine termination conditions.
* If <<<RepeatTemplate>>> used a <<<TaskExecutor>>> to execute its
tasks, asynchronous batch might be as simple as using an
asynchronous <<<TaskExecutor>>> internally - the same
<<<RepeatTemplate>>> would be able to operate in both modes, just
by changing the <<<TaskExecutor>>>.
* Using RetryContext to Stash State for the Policies
E.g. in <<<RetryTemplate>>>:
+---
protected void setupContext(RetryCallback callback,
RetryContext context) {
if (callback instanceof AttributeAccessor) {
AttributeAccessor accessor = (AttributeAccessor) callback;
String[] names = accessor.attributeNames();
for (int i = 0; i < names.length; i++) {
String name = names[i];
context.setAttribute(name, accessor.getAttribute(name));
}
}
}
+---
But this is pants: it makes the callback stateful. We can't store
state in the callback to do with the current item becaue the
callback might (will?) be shared between attempts in a concurrent
system.
So what to do? Nothing - the state for retry is nothing to do with
the callback, and it is natural to store it in the context.
* Asynchronous Batching - Thread Safe Context
* What would happen if several threads were sharing a context object
via a synchronization manager in a thread local (like
<<<TransactionSynchronizationManager>>>)? The context itself had
better be thread safe, otherwise the concurrent peers might assume
that they have the only copy of the context and try and modify it.

22
src/site/apt/cases/parallel.apt
View File

@@ -32,7 +32,7 @@ Use Case: Massively Parallel Batch Processing
* A data source with multiple chunks (commitable units) - more chunks
than parallel processes.
* A way for the container to launch parallel processes.
* A way for the framework to launch parallel processes.
* Success
@@ -43,19 +43,19 @@ Use Case: Massively Parallel Batch Processing
* Description
[[1]] Container splits input data into partitions.
[[1]] Framework splits input data into partitions.
[[1]] Container sends input data (or references to them) to
[[1]] Framework sends input data (or references to them) to
processing nodes.
[[1]] Processing nodes act independently, converting the input data
and sending it transactionally to output source (as per normal
single process batch).
[[1]] Container collects status data from individual nodes for
[[1]] Framework collects status data from individual nodes for
reporting and auditing.
[[1]] When all nodes are complete Container decides that batch is
[[1]] When all nodes are complete Framework decides that batch is
complete finishes processing.
* Variations
@@ -64,17 +64,17 @@ Use Case: Massively Parallel Batch Processing
an internal node failure have different implications for how to
proceed. In both cases, however
[[1]] Container catches exception and classifies it. Rolls back
[[1]] Framework catches exception and classifies it. Rolls back
current transaction to preserve state of data (input and output).
[[1]] Container saves state for restart from last known good
[[1]] Framework saves state for restart from last known good
point, including a pointer to the next input record.
Then if a processing node detects bad data in the input source, it
cannot be restarted or re-distributed because the data need to be
modified for a successful outcome.
[[1]] Container alerts Operator of the location and nature of the
[[1]] Framework alerts Operator of the location and nature of the
failure.
[[1]] Operator waits for batch to finish - the overall status will
@@ -82,7 +82,7 @@ Use Case: Massively Parallel Batch Processing
[[1]] Operator fixes problem and restarts batch.
[[1]] Container does not re-process data that has already been
[[1]] Framework does not re-process data that has already been
processed successfully. The parallel processing nodes are used as
before.
@@ -90,7 +90,7 @@ Use Case: Massively Parallel Batch Processing
If a processing node fails unrecoverably (e.g. after retry timeout),
but with no indication that the input data were bad, then the data
can be re-used: Container returns unprocessed input data, and
can be re-used: Framework returns unprocessed input data, and
redistributes it to other nodes.
* Implementation
@@ -192,7 +192,7 @@ batchTemplate.iterate(new ItemProviderRepeatCallback(provider, processor));
result might be unpredictable if data can be added to an input
source while it is being read.
* If only one query is done by the Container and the results shared
* If only one query is done by the Framework and the results shared
out amongst the nodes we face the issue of how to send the data
between nodes. Performance problems might ensue. Plus (more
seriously) the individual nodes would now need a different

14
src/site/apt/cases/restart.apt
View File

@@ -19,7 +19,7 @@ Use Case: Manual Restart After Failure
* Scope
Any batch should be able to restart gracefully, even if (depending
on chosen container or client implementation) it might have to go
on chosen execution or client implementation) it might have to go
right back to the beginning.
* Preconditions
@@ -28,8 +28,8 @@ Use Case: Manual Restart After Failure
restart will be able to carry on processing a batch from where it
left off.
* There exists a presistent storage mechanism for the initial
conditions.
* There exists a persistent storage mechanism for the initial
conditions.
* Success
@@ -41,12 +41,12 @@ Use Case: Manual Restart After Failure
[[1]] A batch operation encounters an exception which forces the
process to stop processing.
[[1]] Container catches exception and classifies it.
[[1]] Framework catches exception and classifies it.
[[1]] Container logs event with enough information to identify the
[[1]] Framework logs event with enough information to identify the
location of the job and the nature of the problem.
[[1]] Container saves initial condition from last commit point, to
[[1]] Framework saves initial condition from last commit point, to
enable restart to start from the last known good operation.
[[1]] Operator fixes problem (e.g. makes missing resource available,
@@ -54,7 +54,7 @@ Use Case: Manual Restart After Failure
[[1]] Operator restarts batch.
[[1]] Container loads initial conditions and continues processing.
[[1]] Framework loads initial conditions and continues processing.
* Variations

22
src/site/apt/cases/steps.apt
View File

@@ -54,20 +54,20 @@ Use Case: Sequential Processing of Dependent Steps
The vanilla successful case proceeds as follows:
[[1]] Container logs the start of a step, uniquely indentifying
[[1]] Framework logs the start of a step, uniquely indentifying
the initial conditions.
[[1]] Container stores internal state so that initial conditions
[[1]] Framework stores internal state so that initial conditions
can be re-created in the event of a restart.
[[1]] Step execution proceeds as per one of the other use cases
(e.g. {{{file-to-database.html}Copy File to Database}}), including
transactional behaviour.
[[1]] Client instructs Container to store internal state needed by
[[1]] Client instructs Framework to store internal state needed by
further steps (e.g. cached reference data).
[[1]] Container logs successful completion of step, and stores
[[1]] Framework logs successful completion of step, and stores
[[1]] Repeat for next and subsequent steps. Internal state is
passed from one state to the next.
@@ -85,35 +85,35 @@ Use Case: Sequential Processing of Dependent Steps
[[1]] Operator restarts batch with no configuration or input data
changes.
[[1]] Container resumes batch from the last commit point of the
[[1]] Framework resumes batch from the last commit point of the
failed step.
[[1]] Sequence completes normally.
The process above could be carried out by the container entirely (no
The process above could be carried out by the framework entirely (no
need for operator intervention) if a retry policy is in effect.
** Failure of Step Owing to Bad Initial State
If a step fails because it receives bad data from an earlier step,
the Container cannot recover without intervention.
the Framework cannot recover without intervention.
[[1]] Operator attempts to restart without doing anything to fix
the problem.
[[1]] Container detects bad initial state immediately and fails
[[1]] Framework detects bad initial state immediately and fails
fast.
If the original problem can be located and fixed (e.g. input data
for earlier step is revised):
[[1]] Operator restarts batch signalling to container which step
[[1]] Operator restarts batch signalling to framework which step
to begin with.
[[1]] Container locates initial state for the first step to be
[[1]] Framework locates initial state for the first step to be
executed.
[[1]] Container starts execution from the beginning of the desired
[[1]] Framework starts execution from the beginning of the desired
state. This time the input data are different, so the sequence
can complete normally.

6
src/site/apt/changelog.apt
View File

@@ -1,7 +1,3 @@
Changelog: Spring Batch
See the individual subprojects for their changelogs:
* {{{spring-batch-infrastructure/changelog.html}Infrastructure}}
* {{{spring-batch-container/changelog.html}Container}}
See the changes report generated from {{{issue-tracking.html}JIRA}}.

71
src/site/apt/index.apt
View File

@@ -11,75 +11,29 @@ Introduction
Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. Spring Batch builds upon the productivity, POJO-based development approach, and general ease of use capabilities people have come to know from the Spring Framework, while making it easy for developers to access and leverage more advance enterprise services when necessary.
Spring Batch provides reusable functions that are essential in
processing large volumes of records, including logging/tracing,
transaction management, job processing statistics, job restart,
skip, and resource management. It also provides more advance
technical services and features that will enable extremely
high-volume and high performance batch jobs though optimization and
partitioning techniques. Simple as well as complex, high-volume
batch jobs can leverage the framework in a highly scalable manner to
process significant volumes of information.
Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advance technical services and features that will enable extremely high-volume and high performance batch jobs though optimization and partitioning techniques. Simple as well as complex, high-volume batch jobs can leverage the framework in a highly scalable manner to process significant volumes of information.
Spring Batch is part of the
{{{http://www.springframework.org/sub-projects}Spring Portfolio}}.
Spring Batch is part of the {{{http://www.springframework.org/sub-projects}Spring Portfolio}}.
* Spring Batch Architecture
Spring Batch is designed with extensibility and a diverse group of
end users in mind. The figure below shows a sketch of the layered
architecture that supports the extensibility and ease of use for
end-user developers.
Spring Batch is designed with extensibility and a diverse group of end users in mind. The figure below shows a sketch of the layered architecture that supports the extensibility and ease of use for end-user developers.
[images/ContainerLayer.png] Spring Batch Architecture showing
Infrastructure and Container Layers. Potential container
implementations support different platforms and end-user goals from
the same blocks of business logic in the Application Layer.
[images/ExecutionEnvironment.png] Spring Batch Architecture showing Infrastructure and Execution Layers. Potential execution strategy implementations support different platforms and end-user goals from the same blocks of business logic in the Application Layer.
The initial release provides an Infrastructure layer in the form of
low level tools. There is also a simple Container application,
using the infrastructure in its implementation. The container
provides robust features for traceability and management of the
batch lifecycle. A key goal is that the management of the batch
process (locating a job and its input, starting, scheduling,
restarting, and finally processing to created results) should be as
easy as possible for developers.
The initial release provides an Infrastructure layer in the form of low level tools. There is also a simple execution environment, using the infrastructure in its implementation. The execution environment provides robust features for traceability and management of the batch lifecycle. A key goal is that the management of the batch process (locating a job and its input, starting, scheduling, restarting, and finally processing to created results) should be as easy as possible for developers.
The Infrastructure provides the ability to batch operations
together, and to retry an piece of work if there is an exception.
Both requirements have a transactional flavour, and similar concepts
are relevant (propagation, synchronisation). They also both lend
themselves to the template programming model common in Spring,
c.f. <<<TransactionTemplate>>>, <<<JdbcTemplate>>>,
<<<JmsTemplate>>>.
The Infrastructure provides the ability to batch operations together, and to retry an piece of work if there is an exception. Both requirements have a transactional flavour, and similar concepts are relevant (propagation, synchronisation). They also both lend themselves to the template programming model common in Spring, c.f. <<<TransactionTemplate>>>, <<<JdbcTemplate>>>, <<<JmsTemplate>>>.
The Simple Batch Execution Container is the first container available. It provides a robust set of integrated features including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management to enable the management of the full lifecycle of traditional batch processing. A number of sample jobs are packaged with this container and are described in detail to more clearly articulate usage and capabilities of the container.
The simple batch execution environment is the reference implementation, expressed through the execution module of the Spring Batch product. It provides a robust set of integrated features including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management to enable the management of the full lifecycle of traditional batch processing. A number of sample jobs are packaged with this environment and are described in detail to more clearly articulate its usage and capabilities.
* Roadmap
Once the framework is released it can be used immediately to
simplify batch optimisations and automatic retries. The framework
is oriented around application developers not needing to know any
details of the framework - there are a few application developer
interfaces that can be used for convenient construction of data
processing pipelines, but apart from that we support as close to a
POJO programming model as is practical. This is similar to the
approach taken in Spring Core in the area of DAO implementation.
Once the framework is released it can be used immediately to simplify batch optimisations and automatic retries. The framework is oriented around application developers not needing to know any details of the framework - there are a few application developer interfaces that can be used for convenient construction of data processing pipelines, but apart from that we support as close to a POJO programming model as is practical. This is similar to the approach taken in Spring Core in the area of DAO implementation.
A Partitioned Batch Execution Container is also being developed that will provide alternate scaling solutions. This container will provide more advance technical services and features to enable extremely high-volume and high performance batch jobs though proven optimization and partitioning techniques. Proven scaling techniques will be provided as partitioned strategies allowing users to spread the load across a pool of clustered J2EE application servers. There are also discussions to leverage grid technologies as an alternate scaling solution.
A partitioned step execution strategy is also being developed that will provide alternate scaling solutions. This will provide more advanced technical services and features to enable extremely high-volume and high performance batch jobs though proven optimization and partitioning techniques. Proven scaling techniques will be provided as partitioned strategies allowing users to spread the load across a pool of clustered J2EE application servers. There are also discussions to leverage grid technologies as an alternate scaling solution.
Matt Welsh's work shows that
{{{http://www.eecs.harvard.edu/~mdw/proj/seda/}SEDA}} has enormous
benefits over more rigid processing architectures, and messaging
containers give us a lot of resilience out of the box. So we also
want to provide a more SEDA flavoured container, or container
support, as well as supporting the more traditional ETL style
approach. There might be a tie in with Mule and/or other ESB tools
here, giving the benefit of a very scalable architecture, where the
choice of transport and distribution strategy can be made as late as
possible. The same application code could be used in principle for
a standalone tool processing a small amount of data, and a massive
enterprise-scale bulk-processing engine.
Matt Welsh's work shows that {{{http://www.eecs.harvard.edu/~mdw/proj/seda/}SEDA}} has enormous benefits over more rigid processing architectures, and messaging environments give us a lot of resilience out of the box. So we also want to provide a more SEDA flavoured execution environments, as well as supporting the more traditional ETL style approach. There might be a tie in with Mule and/or other ESB tools here, giving the benefit of a very scalable architecture, where the choice of transport and distribution strategy can be made as late as possible. The same application code could be used in principle for a standalone tool processing a small amount of data, and a massive enterprise-scale bulk-processing engine.
* Background
@@ -90,9 +44,4 @@ the same blocks of business logic in the Application Layer.
Accenture is contributing previously proprietary batch processing architecture frameworks -- based upon decades worth of experience in building batch architectures with the last several generations of platforms (i.e., COBOL/Mainframe, C++/Unix, and now Java/anywhere) -- to the Spring Batch project along with committer resources to drive support, enhancements, and the future roadmap.
The collaborative effort between Accenture and Interface21 aims to promote the standardization of software processing approaches, frameworks, and tools that can be consistently leveraged by enterprise users when creating batch applications. Companies and government agencies desiring to deliver standard, proven solutions to their enterprise IT environments will benefit from Spring Batch.
* Links:
* A discussion {{{blotter.html}blotter}}.

31
src/site/apt/sitemap.apt
View File

@@ -18,7 +18,9 @@ Spring Batch Site Map
* Integration Tests - reports on tests of infrastructure
* Container - CI and technical information about container layer
* Core - CI and technical information about core domain
* Execution - CI and technical information about execution (implementation of core)
* Main Site
@@ -53,11 +55,11 @@ Spring Batch Site Map
* Infrastructure - How to use the core API
* Simple Container
* Simple Execution Environment
* Partitioning Container
* Partitioning Execution Environment
* Other Containers?
* Other Execution Environments?
* Changelog
@@ -89,9 +91,16 @@ Spring Batch Site Map
* CI Reports (same as for infrastructure).
* Container
* Core
Should the use case go here (showing which ones are implemented)?
* Changelog
* Project information (duplicated from Main Site - Maven "feature" -
TODO: find a way to switch them off in sub-projects)
* CI Reports (same as for infrastructure).
* Execution
* Changelog
@@ -99,3 +108,13 @@ Spring Batch Site Map
* CI Reports (same as for infrastructure).
* Integration Tests
* Changelog
* Project information (duplicated from Main Site - Maven "feature" -
TODO: find a way to switch them off in sub-projects)
* CI Reports (same as for infrastructure).

33
src/site/apt/snapshots.apt Normal file
View File

@@ -0,0 +1,33 @@
---------
Snapshots
---------
Dave Syer
------
August 2007
Snapshot Builds
These builds are provided for testing and development purposes only. They are built by a Bamboo process
automatically using the latest snapshot from Subversion.
The snapshots are deloyed to a Maven2 snapshot repository. To use it, just add the following repository to your POM:
+---------------
<repository>
<id>spring-s3</id>
<name>Springframework Maven SNAPSHOT Repository</name>
<url>http://s3.amazonaws.com/maven.springframework.org/snapshot</url>
</repository>
+---------------
Individual dependencies can then by added like so:
+---------------
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-core</artifactId>
<version>1.0-m2-SNAPSHOT</version>
</dependency>
+---------------
Additionally, you can browse the repository with a web browser {{{http://s3browse.com/explore/maven.springframework.org/snapshot/org/springframework/batch}here}}.

BIN
src/site/ppt/ExecutionEnvironment.ppt Normal file
View File

Binary file not shown.

BIN
src/site/ppt/Figures.ppt Normal file
View File

Binary file not shown.

BIN
src/site/resources/images/ContainerLayer.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 30 KiB

BIN
src/site/resources/images/ExecutionEnvironment.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.7 KiB

BIN
src/site/resources/images/RuntimeDependencies.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB

1
src/site/site.xml
View File

@@ -30,6 +30,7 @@
<item name="Building" href="building.html"/>
<item name="FAQ" href="faq.html"/>
<item name="Changelog" href="changelog.html"/>
<item name="Downloads" href="snapshots.html"/>
</menu>
<menu ref="modules"/>
<menu ref="reports"/>