Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated site (tidied and added migration guide) for m3 release.

Browse Source
This commit is contained in:
dsyer
2007-12-04 18:54:55 +00:00
parent 34b0ca7a23
commit c2ba752725
7 changed files with 403 additions and 317 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
src/.project
View File

@@ -19,5 +19,10 @@
<type>1</type>
<locationURI>PARENT_DIR/pom.xml</locationURI>
</link>
<link>
<name>parent</name>
<type>2</type>
<locationURI>PARENT_DIR/parent</locationURI>
</link>
</linkedResources>
</projectDescription>

4
src/site/apt/changelog.apt
View File

@@ -1,3 +1,7 @@
Changelog: Spring Batch
See the changes report generated from {{{issue-tracking.html}JIRA}}.
There is a pre-built {{{http://opensource.atlassian.com/projects/spring/secure/IssueNavigator.jspa?reset=true&mode=hide&pid=10090&status=5&status=6&updated:previous=-1w&sorter/field=updated&sorter/order=DESC}Resolved Recently}} query that might be useful.
You can also browse the upcoming releases and look at the {{{http://opensource.atlassian.com/projects/spring/browse/BATCH?report=com.atlassian.jira.plugin.system.project:roadmap-panel}Road Map}} query.

10
src/site/apt/getting-started.apt Normal file
View File

@@ -0,0 +1,10 @@
------
Spring Batch Getting Started
------
Dave Syer
------
December 2007
Spring Batch Getting Started
* Work in Progress

67
src/site/apt/migration/1.0-m2-m3.apt Normal file
View File

@@ -0,0 +1,67 @@
------
Spring Batch Upgrade
------
Robert Kasanicky
------
December 2007
Updating Spring Batch from 1.0-m2 to 1.0-m3
This is a description of what needed to be done to migrate the samples from m2 to the new m3 release.
[Tip:] if you use SpringIDE, make sure it tracks all your configuration files,
so that errors are immediately visible.
* VARIOUS
* OutputSource interface has been renamed to ItemWriter, implementations have been renamed correspondingly
(e.g. FlatFileOutputSource is now FlatFileItemWriter). ItemWriter no longer extends ResourceLifecycle
(which declares methods open() and close). Often you don't need to call these methods at all, in case
you do you should implement InitializingBean and DisposableBean interfaces and declare the bean
in step scope.
* Sql renamed to Jdbc (e.g. SqlInputSource -> JdbcInputSource).
* Database input sources split into driving and cursor packages (cursor makes a single query and iterates
over the result set, driving query for keys and then make a new query for each key).
* ExceptionHandler interface has been changed to handle single throwable instead of a collection of throwables
and it has become responsible for deciding about step completion. See javadoc for more details.
* FlatFileInputSources now include FieldSet mapping logic - you can inject a FieldSetMapper
into the input source, so you no longer need to handle this in ItemProvider. Therefore
FieldSetInputSource interface and FlatFileItemProvider have been removed. Where FlatFileItemProvider
was used you can use the InputSourceItemProvider and inject the mapper into the input source.
* Validation has been removed from input sources and moved upwards to ValidatingItemProvider.
* XML related classes have been moved from io.stax package under io.file
* Fixed-length tokenizer accept ranges property instead of array of lengths, e.g. "2,3"
(see javadoc for more details and fixedLengthImportJob.xml for example usage).
* CONTAINER CONFIGURATION
* Use the value of JobConfiguration name property as the bean id.
* Simplest approach to update is to use the simple-container-definition.xml,
data-source-context.xml (plus hibernate-context.xml) and batch.properties
from M3 samples and replace the post-processor bean in your job xml file with
"\<bean parent="jobConfigurationRegistryBeanPostProcessor"/\>"
* JobLauncher has been moved from execution.bootstrap to execution.launch
* JobExecutor facade has been removed, it's properties are now injected directly
into SimpleJobLauncher intead of the facade itself.
* DefaultStepExecutorFactory is now SimpleStepExecutorFactory located in simple subpackage
and needs a reference to jobRepository instead of value for StepExecutor name.
* ScheduledJobIdentifier has a new jobKey property replaces jobRun and jobStream.
* SimpleStepConfiguration has been moved one package up (from execution.step.simple to execution.step).
* Transaction manager class is specified in batch.properties

629
src/site/fml/faq.fml
View File

@@ -1,315 +1,314 @@
<?xml version="1.0" encoding="UTF-8"?>
<faqs id="FAQ" title="Frequently Asked Questions">
<part id="General">
<faq id="release">
<question>
When will Batch 1.0 be out? My understanding is that the
current product can be used on projects, is that
correct?
</question>
<answer>
We expect m3, m4, rc1 before 1.0 final, so realistically
not soon. You can track the progress and planning in
JIRA
(http://opensource.atlassian.com/projects/spring/browse/BATCH).
But it is already usable at m2 as long as there is an
understanding that upgrading might take a small amount
of effort - there are 5 or 6 projects using it, with at
least one in production. That project upgraded from m1
to m2 (which was distinctly less trivial than m2 to m3
is likely to be) in less than a day. We intend to keep
changes from here to 1.0 as close to internal as
possible, so the client touch points are pretty stable.
</answer>
</faq>
<faq id="stable">
<question>
How stable are the interfaces in Spring Batch?
</question>
<answer>
<p>
We are still in the milestone release phase (1.0-m3
is in the pipeline). This means that we are still
adding functionality that we want to be part of a
1.0 release. We do not rule out changes to package
and interface names in this phase, but that said we
think the basic domain concepts in Spring Batch are
sound enough to survive significant re-factoring.
The bulk of the application developer "touch points"
have been stable for quite some time now, and we
have several early adopter projects already using
snapshot releases.
</p>
<p>
The process from here is to collect feedback from
the community and use that to decide on what extra
features need to be added to get us to 1.0. When we
are feature complete we will move to the "release
candidate" phase, and the first release in that
phase will be 1.0-rc1. When significant issues are
resolved (if there are any) we will promote the
release through the "rc" numbers, until we have a
clean 1.0 release.
</p>
</answer>
</faq>
<faq id="threading">
<question>
When will support for the more complex job execution
classes appear? Our Client has a number of
multi-threaded batch jobs.
</question>
<answer>
Multi-threaded execution in a single VM is perfectly
possible with 1.0 - but we recommend exercising caution
in the analysis of such requirements (is it really
necessary?). Several people have tried it and we haven't
had any issues yet (except documentation requests).
</answer>
</faq>
<faq id="layers">
<question>
There are 4 main layers of the architecture
(application, core, execution, and infrastructure), what
is the vision for how the execution layer might be used
in future? What happened to the "container" layer?
</question>
<answer>
<p>
The "layers" described are nicely segregated in
terms of dependency. Each layer only depends (at
compile time) on layers below it.
</p>
<p>
We recognised that what we used to call the
container layer actually is composed of two distinct
contexts, "Core" and "Execution". So the full
catalogue of contexts is:
<ul>
<li>
<b>Application</b>
is the business logic. It is written by the
application developer - the client of Spring
Batch - and only depends on the other Core
interfaces for compilation and
configuration.
</li>
<li>
<b>Core</b>
is the public API of Spring Batch, including
the core batch domain of Job, Step,
configuration and Executor interfaces.
</li>
<li>
<b>Execution</b>
is the deployment, execution and management
concerns. Different execution environments
(e.g. in a JEE container, out of container)
are configured differently, but can execute
the same application business logic.
</li>
<li>
<b>Infrastructure</b>
is a set of low level tools, that are used
to implement the execution and parts of the
core layers.
</li>
</ul>
<br />
</p>
<p>
The "execution" layer is fertile ground for
collaboration and contributions from the community
and from projects in the field. The central
interface is
<code>JobExecutorFacade</code>
with methods for starting and stopping jobs. The
vision for this is that there can be multiple
implementations of
<code>JobExecutorFacade</code>
providing different architectural patterns, and
delivering different levels of scalability and
robustness, without changing either the business
logic or the job configuration. The initial 1.0
release of Spring Batch will have a single
implementation
<code>SimpleJobExecutorFacade</code>
(formerly known as
<code>SimpleBatchContainer</code>
.)
</p>
</answer>
</faq>
<faq id="flexible">
<question>
What is the Spring Batch philosophy on the use of
flexible strategies and default implementations?
</question>
<answer>
There are a great many extension points in Spring Batch
for the framework developer (as opposed to the
implementor of business logic). We expect clients to
create their own more specific strategies that can be
plugged in to control things like commit intervals (
<code>CompletionPolicy</code>
), rules about how to deal with exceptions (
<code>ExceptionHandler</code>
), and many others.
</answer>
</faq>
<faq id="quartz">
<question>
How does Spring Batch differ from Quartz? Is there a
place for them both in a solution?
</question>
<answer>
<p>
Spring Batch and Quartz have different goals. Spring
Batch provides functionality for processing large
volumes of data and Quartz provides functionality
for scheduling tasks. So Quartz could complement
Spring Batch, but are not excluding technologies. A
common combination would be to use Quartz as a
trigger for a Spring Batch job using a Cron
expression and the Spring Core convenience
<code>SchedulerFactoryBean</code>
.
</p>
</answer>
</faq>
<faq id="schedulers">
<question>
How do I schedule a job with Spring Batch?
</question>
<answer>
<p>
Use a scheduling tool. There are plenty of them out
there. Examples: Quartz, Control-M, Autosys. Quartz
doesn't have all the features of Control-M or
Autosys - it is supposed to be lightweight. If you
want something even more lightweight you can just
use the OS (cron, at, etc.).
</p>
<p>
Simple sequential dependencies can be implemented
using the job-steps model of Spring Batch. We think
this is quite common. And in fact it makes it easier
to correct a common mis-use of scehdulers - having
hundreds of jobs configured, many of which are not
independent, but only depend on one other.
</p>
</answer>
</faq>
<faq id="parallel">
<question>
How will Spring Batch allow project to optimize for
performance and scalability (through parallel processing
or other)?
</question>
<answer>
We see this as one of the roles of the Execution layer.
A specific implementation (or implementations) of the
<code>StepExecutor</code>
can deal with the concern of breaking apart the business
logic and sharing it efficiently between parallel
processes or processors. There are a number of
technologies that could play a role here. The essence is
just a set of concurrent remote calls to distributed
agents that can handle some business processing. Since
the business processing is already typically modularised
- e.g. input an item, process it - Spring Batch can
strategise the distribution in a number of ways. One
implementation that we have had some experience with
(and have a prototype for) is a set of remote EJBs
handling the business processing. We switch off Home
caching in the container and then send a specific range
of primary keys for the inputs to each of a number of
remote calls. THe same basic strategy would work with
any of the Spring Remoting protocols (plain RMI,
HttpInvoker, JMS, Hessian etc.) with little more than a
couple of lines change in the execution layer
configuration.
</answer>
</faq>
<faq id="steps">
<question>
What are the key concepts in the Spring Batch core
domain?
</question>
<answer>
<p>
In a nutshell: A JobConfiguration with a list of
StepConfigurations is passed to an ExecutionService.
From this a Job is constructed consisting of a
series of Steps, each of which is executed by a
StepExecutor. The StepExecutor contains all the
strategies for deciding when to complete, when to
commit, when to abort and when to continue.
</p>
<p>
Many Jobs in practice consist of a single Step. Step
is very useful and best practice for breaking a Job
down into logical units, rather than having to
execute separate Jobs (potentially in separate OS
processes) which have no obvious logical connection.
</p>
<p>
Jobs can be executed once, or many times with
different logical identifiers (JobIdentifier). It is
also possible to restart a failed Job with the same
or a modified input source, and identify the
resulting JobExecution as a separate entity. In this
way the progress of a Job and its history of
successful and failed executions can easily be
tracked. The same argument applies to Steps, which
have their corresponding StepExecution entity.
</p>
</answer>
</faq>
<faq id="messaging-scaling">
<question>
How can messaging be used to scale batch architectures?
</question>
<answer>
There is a good deal of practical evidence from existing
projects that a pipeline approach to batch processing is
highly beneficial, leading to resilience and high
throughput. We are often faced with mission-critical
applications where audit trails are essential, and
guaranteed processing is demanded, but where there are
extremely tight limits on performance under load, or
where high throughput gives a competitive advantage.
Matt Welsh's work shows that a Staged Event Driven
Architecture (SEDA) has enormous benefits over more
rigid processing architectures, and message-oriented
middleware (JMS, AQ, MQ, Tibco etc.) gives us a lot of
resilience out of the box. There are particular benefits
in a system where there is feedback between downstream
and upstream stages, so the number of consumers can be
adjusted to account for the amount of demand. So how
does this fit into Spring Batch? Well it's a good
example of an
<code>StepExecutor</code>
or (more broadly) execution runtime if the deployment is
grid- or cluster-based, or in any way involves multiple
OS processes.
</answer>
</faq>
<faq id="contributions">
<question>How can I contribute to Spring Batch?</question>
<answer>
Use JIRA and the forum to get involved in discussions
about the product and its design. There is a process for
contributions and eventually becoming a committer. The
process is pretty standard for all Apache-licensed
projects. You make contributions through JIRA (so sign
up now); you assign the copyright of any contributions
using a standard Apache-like CLA (see the Apache one for
example - ours might be slightly different); when the
contributions reach a certain level, or you convince us
otherwise that you are going to be committed long term,
even if part time, then you can become a committer.
</answer>
</faq>
</part>
</faqs>
<?xml version="1.0" encoding="UTF-8"?>
<faqs id="FAQ" title="Frequently Asked Questions">
<part id="General">
<faq id="release">
<question>
When will Batch 1.0 be out? My understanding is that the
current product can be used on projects, is that
correct?
</question>
<answer>
We expect m3, m4, rc1 before 1.0 final, so realistically
not soon. You can track the progress and planning in
JIRA
(http://opensource.atlassian.com/projects/spring/browse/BATCH).
But it is already usable at m3 as long as there is an
understanding that upgrading might take a small amount
of effort - there are 5 or 6 projects using it, with at
least one in production. That project upgraded from m1
to m2 in less than a day. We intend to keep changes from
here to 1.0 as close to internal as possible, so the
client touch points are pretty stable.
</answer>
</faq>
<faq id="stable">
<question>
How stable are the interfaces in Spring Batch?
</question>
<answer>
<p>
We are still in the milestone release phase (1.0-m4
is in the pipeline). This means that we are still
adding functionality that we want to be part of a
1.0 release. We do not rule out changes to package
and interface names in this phase, but that said we
think the basic domain concepts in Spring Batch are
sound enough to survive significant re-factoring.
The bulk of the application developer "touch points"
have been stable for quite some time now, and we
have several early adopter projects already using
snapshot releases.
</p>
<p>
The process from here is to collect feedback from
the community and use that to decide on what extra
features need to be added to get us to 1.0. When we
are feature complete we will move to the "release
candidate" phase, and the first release in that
phase will be 1.0-rc1. We only expect one release
candidate, but if there is enough demand for new
features, or significant problems occur in rc1, then
we might need an rc2. We aim for 2 weeks elapsed
time between release candidates (and between the
last milestone and rc1).
</p>
</answer>
</faq>
<faq id="threading">
<question>
When will support for the more complex job execution
classes appear? Our Client has a number of
multi-threaded batch jobs.
</question>
<answer>
Multi-threaded execution in a single VM is perfectly
possible with 1.0 - but we recommend exercising caution
in the analysis of such requirements (is it really
necessary?). Several people have tried it and there are
some issues with restartability (plus the usual
documentation requests), but otherwise it should work.
With a little more effort starting in m4 it should work
even better, but we aren't planning to solve the
restartability problem until after 1.0.
</answer>
</faq>
<faq id="layers">
<question>
There are 4 main layers of the architecture
(application, core, execution, and infrastructure), what
is the vision for how the execution layer might be used
in future? What happened to the "container" layer?
</question>
<answer>
<p>
The "layers" described are nicely segregated in
terms of dependency. Each layer only depends (at
compile time) on layers below it.
</p>
<p>
We recognised that what we used to call the
container layer actually is composed of two distinct
contexts, "Core" and "Execution". So the full
catalogue of contexts is:
<ul>
<li>
<b>Application</b>
is the business logic. It is written by the
application developer - the client of Spring
Batch - and only depends on the other Core
interfaces for compilation and
configuration.
</li>
<li>
<b>Core</b>
is the public API of Spring Batch, including
the core batch domain of Job, Step,
configuration and Executor interfaces.
</li>
<li>
<b>Execution</b>
is the deployment, execution and management
concerns. Different execution environments
(e.g. in a JEE container, out of container)
are configured differently, but can execute
the same application business logic.
</li>
<li>
<b>Infrastructure</b>
is a set of low level tools, that are used
to implement the execution and parts of the
core layers.
</li>
</ul>
<br />
</p>
<p>
The "execution" layer is fertile ground for
collaboration and contributions from the community
and from projects in the field. The central
interface is
<code>JobLauncher</code>
with methods for starting and stopping jobs. The
vision for this is that there can be multiple
implementations of
<code>JobLauncher</code>
providing different architectural patterns, and
delivering different levels of scalability and
robustness, without changing either the business
logic or the job configuration.
</p>
</answer>
</faq>
<faq id="flexible">
<question>
What is the Spring Batch philosophy on the use of
flexible strategies and default implementations?
</question>
<answer>
There are a great many extension points in Spring Batch
for the framework developer (as opposed to the
implementor of business logic). We expect clients to
create their own more specific strategies that can be
plugged in to control things like commit intervals (
<code>CompletionPolicy</code>
), rules about how to deal with exceptions (
<code>ExceptionHandler</code>
), and many others.
</answer>
</faq>
<faq id="quartz">
<question>
How does Spring Batch differ from Quartz? Is there a
place for them both in a solution?
</question>
<answer>
<p>
Spring Batch and Quartz have different goals. Spring
Batch provides functionality for processing large
volumes of data and Quartz provides functionality
for scheduling tasks. So Quartz could complement
Spring Batch, but are not excluding technologies. A
common combination would be to use Quartz as a
trigger for a Spring Batch job using a Cron
expression and the Spring Core convenience
<code>SchedulerFactoryBean</code>
.
</p>
</answer>
</faq>
<faq id="schedulers">
<question>
How do I schedule a job with Spring Batch?
</question>
<answer>
<p>
Use a scheduling tool. There are plenty of them out
there. Examples: Quartz, Control-M, Autosys. Quartz
doesn't have all the features of Control-M or
Autosys - it is supposed to be lightweight. If you
want something even more lightweight you can just
use the OS (cron, at, etc.).
</p>
<p>
Simple sequential dependencies can be implemented
using the job-steps model of Spring Batch. We think
this is quite common. And in fact it makes it easier
to correct a common mis-use of scehdulers - having
hundreds of jobs configured, many of which are not
independent, but only depend on one other.
</p>
</answer>
</faq>
<faq id="parallel">
<question>
How will Spring Batch allow project to optimize for
performance and scalability (through parallel processing
or other)?
</question>
<answer>
We see this as one of the roles of the Execution layer.
A specific implementation (or implementations) of the
<code>StepExecutor</code>
can deal with the concern of breaking apart the business
logic and sharing it efficiently between parallel
processes or processors. There are a number of
technologies that could play a role here. The essence is
just a set of concurrent remote calls to distributed
agents that can handle some business processing. Since
the business processing is already typically modularised
- e.g. input an item, process it - Spring Batch can
strategise the distribution in a number of ways. One
implementation that we have had some experience with
(and have a prototype for) is a set of remote EJBs
handling the business processing. We switch off Home
caching in the container and then send a specific range
of primary keys for the inputs to each of a number of
remote calls. The same basic strategy would work with
any of the Spring Remoting protocols (plain RMI,
HttpInvoker, JMS, Hessian etc.) with little more than a
couple of lines change in the execution layer
configuration.
</answer>
</faq>
<faq id="steps">
<question>
What are the key concepts in the Spring Batch core
domain?
</question>
<answer>
<p>
In a nutshell: A JobConfiguration with a list of
StepConfigurations is passed to a JobExecutor.
From this a Job is constructed consisting of a
series of Steps, each of which is executed by a
StepExecutor. The StepExecutor contains all the
strategies for deciding when to complete, when to
commit, when to abort and when to continue.
</p>
<p>
Many Jobs in practice consist of a single Step. Step
is very useful and best practice for breaking a Job
down into logical units, rather than having to
execute separate Jobs (potentially in separate OS
processes) which have no obvious logical connection.
</p>
<p>
Jobs can be executed once, or many times with
different logical identifiers (JobIdentifier). It is
also possible to restart a failed Job with the same
or a modified input source, and identify the
resulting JobExecution as a separate entity. In this
way the progress of a Job and its history of
successful and failed executions can easily be
tracked. The same argument applies to Steps, which
have their corresponding StepExecution entity.
</p>
</answer>
</faq>
<faq id="messaging-scaling">
<question>
How can messaging be used to scale batch architectures?
</question>
<answer>
There is a good deal of practical evidence from existing
projects that a pipeline approach to batch processing is
highly beneficial, leading to resilience and high
throughput. We are often faced with mission-critical
applications where audit trails are essential, and
guaranteed processing is demanded, but where there are
extremely tight limits on performance under load, or
where high throughput gives a competitive advantage.
Matt Welsh's work shows that a Staged Event Driven
Architecture (SEDA) has enormous benefits over more
rigid processing architectures, and message-oriented
middleware (JMS, AQ, MQ, Tibco etc.) gives us a lot of
resilience out of the box. There are particular benefits
in a system where there is feedback between downstream
and upstream stages, so the number of consumers can be
adjusted to account for the amount of demand. So how
does this fit into Spring Batch? Well it's a good
example of an
<code>StepExecutor</code>
or (more broadly) execution runtime if the deployment is
grid- or cluster-based, or in any way involves multiple
OS processes.
</answer>
</faq>
<faq id="contributions">
<question>How can I contribute to Spring Batch?</question>
<answer>
Use JIRA and the forum to get involved in discussions
about the product and its design. There is a process for
contributions and eventually becoming a committer. The
process is pretty standard for all Apache-licensed
projects. You make contributions through JIRA (so sign
up now); you assign the copyright of any contributions
using a standard Apache-like CLA (see the Apache one for
example - ours might be slightly different); when the
contributions reach a certain level, or you convince us
otherwise that you are going to be committed long term,
even if part time, then you can become a committer.
</answer>
</faq>
</part>
</faqs>

BIN
src/site/ppt/Figures.ppt
View File

Binary file not shown.

5
src/site/site.xml
View File

@@ -28,9 +28,10 @@
<item name="Batch Processing Strategies" href="batch-processing-strategies.html"/>
<item name="General Batch Principles and Guidelines" href="batch-principles-guidelines.html"/>
<item name="Building" href="building.html"/>
<item name="Getting Started" href="getting-started.html"/>
<!-- item name="Getting Started" href="getting-started.html"/-->
<item name="FAQ" href="faq.html"/>
<item name="Recent Changes" href="http://opensource.atlassian.com/projects/spring/browse/BATCH"/>
<item name="Changelog" href="changelog.html"/>
<item name="Migrating" href="migration/1.0-m2-m3.html"/>
<item name="Downloads" href="downloads.html"/>
</menu>
<menu ref="modules"/>