diff --git a/src/.project b/src/.project index 44c6d4e5f..8aaaa6d5e 100644 --- a/src/.project +++ b/src/.project @@ -19,5 +19,10 @@ 1 PARENT_DIR/pom.xml + + parent + 2 + PARENT_DIR/parent + diff --git a/src/site/apt/changelog.apt b/src/site/apt/changelog.apt index 8f2bd585c..6ee9c36cb 100644 --- a/src/site/apt/changelog.apt +++ b/src/site/apt/changelog.apt @@ -1,3 +1,7 @@ Changelog: Spring Batch See the changes report generated from {{{issue-tracking.html}JIRA}}. + + There is a pre-built {{{http://opensource.atlassian.com/projects/spring/secure/IssueNavigator.jspa?reset=true&mode=hide&pid=10090&status=5&status=6&updated:previous=-1w&sorter/field=updated&sorter/order=DESC}Resolved Recently}} query that might be useful. + + You can also browse the upcoming releases and look at the {{{http://opensource.atlassian.com/projects/spring/browse/BATCH?report=com.atlassian.jira.plugin.system.project:roadmap-panel}Road Map}} query. diff --git a/src/site/apt/getting-started.apt b/src/site/apt/getting-started.apt new file mode 100644 index 000000000..55f3fd6ff --- /dev/null +++ b/src/site/apt/getting-started.apt @@ -0,0 +1,10 @@ + ------ + Spring Batch Getting Started + ------ + Dave Syer + ------ + December 2007 + +Spring Batch Getting Started + +* Work in Progress diff --git a/src/site/apt/migration/1.0-m2-m3.apt b/src/site/apt/migration/1.0-m2-m3.apt new file mode 100644 index 000000000..0629cebb9 --- /dev/null +++ b/src/site/apt/migration/1.0-m2-m3.apt @@ -0,0 +1,67 @@ + ------ + Spring Batch Upgrade + ------ + Robert Kasanicky + ------ + December 2007 + +Updating Spring Batch from 1.0-m2 to 1.0-m3 + + This is a description of what needed to be done to migrate the samples from m2 to the new m3 release. + + [Tip:] if you use SpringIDE, make sure it tracks all your configuration files, +so that errors are immediately visible. + +* VARIOUS + + * OutputSource interface has been renamed to ItemWriter, implementations have been renamed correspondingly +(e.g. FlatFileOutputSource is now FlatFileItemWriter). ItemWriter no longer extends ResourceLifecycle +(which declares methods open() and close). Often you don't need to call these methods at all, in case +you do you should implement InitializingBean and DisposableBean interfaces and declare the bean +in step scope. + + * Sql renamed to Jdbc (e.g. SqlInputSource -> JdbcInputSource). + + * Database input sources split into driving and cursor packages (cursor makes a single query and iterates +over the result set, driving query for keys and then make a new query for each key). + + * ExceptionHandler interface has been changed to handle single throwable instead of a collection of throwables +and it has become responsible for deciding about step completion. See javadoc for more details. + + * FlatFileInputSources now include FieldSet mapping logic - you can inject a FieldSetMapper +into the input source, so you no longer need to handle this in ItemProvider. Therefore +FieldSetInputSource interface and FlatFileItemProvider have been removed. Where FlatFileItemProvider +was used you can use the InputSourceItemProvider and inject the mapper into the input source. + + * Validation has been removed from input sources and moved upwards to ValidatingItemProvider. + + * XML related classes have been moved from io.stax package under io.file + + * Fixed-length tokenizer accept ranges property instead of array of lengths, e.g. "2,3" +(see javadoc for more details and fixedLengthImportJob.xml for example usage). + + +* CONTAINER CONFIGURATION + + * Use the value of JobConfiguration name property as the bean id. + + * Simplest approach to update is to use the simple-container-definition.xml, +data-source-context.xml (plus hibernate-context.xml) and batch.properties +from M3 samples and replace the post-processor bean in your job xml file with +"\" + + * JobLauncher has been moved from execution.bootstrap to execution.launch + + * JobExecutor facade has been removed, it's properties are now injected directly +into SimpleJobLauncher intead of the facade itself. + + * DefaultStepExecutorFactory is now SimpleStepExecutorFactory located in simple subpackage +and needs a reference to jobRepository instead of value for StepExecutor name. + + * ScheduledJobIdentifier has a new jobKey property replaces jobRun and jobStream. + + * SimpleStepConfiguration has been moved one package up (from execution.step.simple to execution.step). + + * Transaction manager class is specified in batch.properties + + diff --git a/src/site/fml/faq.fml b/src/site/fml/faq.fml index db6c3cde9..a9c6df090 100644 --- a/src/site/fml/faq.fml +++ b/src/site/fml/faq.fml @@ -1,315 +1,314 @@ - - - - - - When will Batch 1.0 be out? My understanding is that the - current product can be used on projects, is that - correct? - - - We expect m3, m4, rc1 before 1.0 final, so realistically - not soon. You can track the progress and planning in - JIRA - (http://opensource.atlassian.com/projects/spring/browse/BATCH). - But it is already usable at m2 as long as there is an - understanding that upgrading might take a small amount - of effort - there are 5 or 6 projects using it, with at - least one in production. That project upgraded from m1 - to m2 (which was distinctly less trivial than m2 to m3 - is likely to be) in less than a day. We intend to keep - changes from here to 1.0 as close to internal as - possible, so the client touch points are pretty stable. - - - - - How stable are the interfaces in Spring Batch? - - -

- We are still in the milestone release phase (1.0-m3 - is in the pipeline). This means that we are still - adding functionality that we want to be part of a - 1.0 release. We do not rule out changes to package - and interface names in this phase, but that said we - think the basic domain concepts in Spring Batch are - sound enough to survive significant re-factoring. - The bulk of the application developer "touch points" - have been stable for quite some time now, and we - have several early adopter projects already using - snapshot releases. -

-

- The process from here is to collect feedback from - the community and use that to decide on what extra - features need to be added to get us to 1.0. When we - are feature complete we will move to the "release - candidate" phase, and the first release in that - phase will be 1.0-rc1. When significant issues are - resolved (if there are any) we will promote the - release through the "rc" numbers, until we have a - clean 1.0 release. -

- - - - - When will support for the more complex job execution - classes appear? Our Client has a number of - multi-threaded batch jobs. - - - Multi-threaded execution in a single VM is perfectly - possible with 1.0 - but we recommend exercising caution - in the analysis of such requirements (is it really - necessary?). Several people have tried it and we haven't - had any issues yet (except documentation requests). - - - - - There are 4 main layers of the architecture - (application, core, execution, and infrastructure), what - is the vision for how the execution layer might be used - in future? What happened to the "container" layer? - - -

- The "layers" described are nicely segregated in - terms of dependency. Each layer only depends (at - compile time) on layers below it. -

-

- We recognised that what we used to call the - container layer actually is composed of two distinct - contexts, "Core" and "Execution". So the full - catalogue of contexts is: -

    -
  • - Application - is the business logic. It is written by the - application developer - the client of Spring - Batch - and only depends on the other Core - interfaces for compilation and - configuration. -
    • -
  • - Core - is the public API of Spring Batch, including - the core batch domain of Job, Step, - configuration and Executor interfaces. -
    • -
  • - Execution - is the deployment, execution and management - concerns. Different execution environments - (e.g. in a JEE container, out of container) - are configured differently, but can execute - the same application business logic. -
    • -
  • - Infrastructure - is a set of low level tools, that are used - to implement the execution and parts of the - core layers. -
    • -
-
-

-

- The "execution" layer is fertile ground for - collaboration and contributions from the community - and from projects in the field. The central - interface is - JobExecutorFacade - with methods for starting and stopping jobs. The - vision for this is that there can be multiple - implementations of - JobExecutorFacade - providing different architectural patterns, and - delivering different levels of scalability and - robustness, without changing either the business - logic or the job configuration. The initial 1.0 - release of Spring Batch will have a single - implementation - SimpleJobExecutorFacade - (formerly known as - SimpleBatchContainer - .) -

- - - - - What is the Spring Batch philosophy on the use of - flexible strategies and default implementations? - - - There are a great many extension points in Spring Batch - for the framework developer (as opposed to the - implementor of business logic). We expect clients to - create their own more specific strategies that can be - plugged in to control things like commit intervals ( - CompletionPolicy - ), rules about how to deal with exceptions ( - ExceptionHandler - ), and many others. - - - - - How does Spring Batch differ from Quartz? Is there a - place for them both in a solution? - - -

- Spring Batch and Quartz have different goals. Spring - Batch provides functionality for processing large - volumes of data and Quartz provides functionality - for scheduling tasks. So Quartz could complement - Spring Batch, but are not excluding technologies. A - common combination would be to use Quartz as a - trigger for a Spring Batch job using a Cron - expression and the Spring Core convenience - SchedulerFactoryBean - . -

- - - - - How do I schedule a job with Spring Batch? - - -

- Use a scheduling tool. There are plenty of them out - there. Examples: Quartz, Control-M, Autosys. Quartz - doesn't have all the features of Control-M or - Autosys - it is supposed to be lightweight. If you - want something even more lightweight you can just - use the OS (cron, at, etc.). -

-

- Simple sequential dependencies can be implemented - using the job-steps model of Spring Batch. We think - this is quite common. And in fact it makes it easier - to correct a common mis-use of scehdulers - having - hundreds of jobs configured, many of which are not - independent, but only depend on one other. -

- - - - - How will Spring Batch allow project to optimize for - performance and scalability (through parallel processing - or other)? - - - We see this as one of the roles of the Execution layer. - A specific implementation (or implementations) of the - StepExecutor - can deal with the concern of breaking apart the business - logic and sharing it efficiently between parallel - processes or processors. There are a number of - technologies that could play a role here. The essence is - just a set of concurrent remote calls to distributed - agents that can handle some business processing. Since - the business processing is already typically modularised - - e.g. input an item, process it - Spring Batch can - strategise the distribution in a number of ways. One - implementation that we have had some experience with - (and have a prototype for) is a set of remote EJBs - handling the business processing. We switch off Home - caching in the container and then send a specific range - of primary keys for the inputs to each of a number of - remote calls. THe same basic strategy would work with - any of the Spring Remoting protocols (plain RMI, - HttpInvoker, JMS, Hessian etc.) with little more than a - couple of lines change in the execution layer - configuration. - - - - - What are the key concepts in the Spring Batch core - domain? - - -

- In a nutshell: A JobConfiguration with a list of - StepConfigurations is passed to an ExecutionService. - From this a Job is constructed consisting of a - series of Steps, each of which is executed by a - StepExecutor. The StepExecutor contains all the - strategies for deciding when to complete, when to - commit, when to abort and when to continue. -

-

- Many Jobs in practice consist of a single Step. Step - is very useful and best practice for breaking a Job - down into logical units, rather than having to - execute separate Jobs (potentially in separate OS - processes) which have no obvious logical connection. -

-

- Jobs can be executed once, or many times with - different logical identifiers (JobIdentifier). It is - also possible to restart a failed Job with the same - or a modified input source, and identify the - resulting JobExecution as a separate entity. In this - way the progress of a Job and its history of - successful and failed executions can easily be - tracked. The same argument applies to Steps, which - have their corresponding StepExecution entity. -

- - - - - How can messaging be used to scale batch architectures? - - - There is a good deal of practical evidence from existing - projects that a pipeline approach to batch processing is - highly beneficial, leading to resilience and high - throughput. We are often faced with mission-critical - applications where audit trails are essential, and - guaranteed processing is demanded, but where there are - extremely tight limits on performance under load, or - where high throughput gives a competitive advantage. - Matt Welsh's work shows that a Staged Event Driven - Architecture (SEDA) has enormous benefits over more - rigid processing architectures, and message-oriented - middleware (JMS, AQ, MQ, Tibco etc.) gives us a lot of - resilience out of the box. There are particular benefits - in a system where there is feedback between downstream - and upstream stages, so the number of consumers can be - adjusted to account for the amount of demand. So how - does this fit into Spring Batch? Well it's a good - example of an - StepExecutor - or (more broadly) execution runtime if the deployment is - grid- or cluster-based, or in any way involves multiple - OS processes. - - - - How can I contribute to Spring Batch? - - Use JIRA and the forum to get involved in discussions - about the product and its design. There is a process for - contributions and eventually becoming a committer. The - process is pretty standard for all Apache-licensed - projects. You make contributions through JIRA (so sign - up now); you assign the copyright of any contributions - using a standard Apache-like CLA (see the Apache one for - example - ours might be slightly different); when the - contributions reach a certain level, or you convince us - otherwise that you are going to be committed long term, - even if part time, then you can become a committer. - - - - + + + + + + When will Batch 1.0 be out? My understanding is that the + current product can be used on projects, is that + correct? + + + We expect m3, m4, rc1 before 1.0 final, so realistically + not soon. You can track the progress and planning in + JIRA + (http://opensource.atlassian.com/projects/spring/browse/BATCH). + But it is already usable at m3 as long as there is an + understanding that upgrading might take a small amount + of effort - there are 5 or 6 projects using it, with at + least one in production. That project upgraded from m1 + to m2 in less than a day. We intend to keep changes from + here to 1.0 as close to internal as possible, so the + client touch points are pretty stable. + + + + + How stable are the interfaces in Spring Batch? + + +

+ We are still in the milestone release phase (1.0-m4 + is in the pipeline). This means that we are still + adding functionality that we want to be part of a + 1.0 release. We do not rule out changes to package + and interface names in this phase, but that said we + think the basic domain concepts in Spring Batch are + sound enough to survive significant re-factoring. + The bulk of the application developer "touch points" + have been stable for quite some time now, and we + have several early adopter projects already using + snapshot releases. +

+

+ The process from here is to collect feedback from + the community and use that to decide on what extra + features need to be added to get us to 1.0. When we + are feature complete we will move to the "release + candidate" phase, and the first release in that + phase will be 1.0-rc1. We only expect one release + candidate, but if there is enough demand for new + features, or significant problems occur in rc1, then + we might need an rc2. We aim for 2 weeks elapsed + time between release candidates (and between the + last milestone and rc1). +

+ + + + + When will support for the more complex job execution + classes appear? Our Client has a number of + multi-threaded batch jobs. + + + Multi-threaded execution in a single VM is perfectly + possible with 1.0 - but we recommend exercising caution + in the analysis of such requirements (is it really + necessary?). Several people have tried it and there are + some issues with restartability (plus the usual + documentation requests), but otherwise it should work. + With a little more effort starting in m4 it should work + even better, but we aren't planning to solve the + restartability problem until after 1.0. + + + + + There are 4 main layers of the architecture + (application, core, execution, and infrastructure), what + is the vision for how the execution layer might be used + in future? What happened to the "container" layer? + + +

+ The "layers" described are nicely segregated in + terms of dependency. Each layer only depends (at + compile time) on layers below it. +

+

+ We recognised that what we used to call the + container layer actually is composed of two distinct + contexts, "Core" and "Execution". So the full + catalogue of contexts is: +

    +
  • + Application + is the business logic. It is written by the + application developer - the client of Spring + Batch - and only depends on the other Core + interfaces for compilation and + configuration. +
    • +
  • + Core + is the public API of Spring Batch, including + the core batch domain of Job, Step, + configuration and Executor interfaces. +
    • +
  • + Execution + is the deployment, execution and management + concerns. Different execution environments + (e.g. in a JEE container, out of container) + are configured differently, but can execute + the same application business logic. +
    • +
  • + Infrastructure + is a set of low level tools, that are used + to implement the execution and parts of the + core layers. +
    • +
+
+

+

+ The "execution" layer is fertile ground for + collaboration and contributions from the community + and from projects in the field. The central + interface is + JobLauncher + with methods for starting and stopping jobs. The + vision for this is that there can be multiple + implementations of + JobLauncher + providing different architectural patterns, and + delivering different levels of scalability and + robustness, without changing either the business + logic or the job configuration. +

+ + + + + What is the Spring Batch philosophy on the use of + flexible strategies and default implementations? + + + There are a great many extension points in Spring Batch + for the framework developer (as opposed to the + implementor of business logic). We expect clients to + create their own more specific strategies that can be + plugged in to control things like commit intervals ( + CompletionPolicy + ), rules about how to deal with exceptions ( + ExceptionHandler + ), and many others. + + + + + How does Spring Batch differ from Quartz? Is there a + place for them both in a solution? + + +

+ Spring Batch and Quartz have different goals. Spring + Batch provides functionality for processing large + volumes of data and Quartz provides functionality + for scheduling tasks. So Quartz could complement + Spring Batch, but are not excluding technologies. A + common combination would be to use Quartz as a + trigger for a Spring Batch job using a Cron + expression and the Spring Core convenience + SchedulerFactoryBean + . +

+ + + + + How do I schedule a job with Spring Batch? + + +

+ Use a scheduling tool. There are plenty of them out + there. Examples: Quartz, Control-M, Autosys. Quartz + doesn't have all the features of Control-M or + Autosys - it is supposed to be lightweight. If you + want something even more lightweight you can just + use the OS (cron, at, etc.). +

+

+ Simple sequential dependencies can be implemented + using the job-steps model of Spring Batch. We think + this is quite common. And in fact it makes it easier + to correct a common mis-use of scehdulers - having + hundreds of jobs configured, many of which are not + independent, but only depend on one other. +

+ + + + + How will Spring Batch allow project to optimize for + performance and scalability (through parallel processing + or other)? + + + We see this as one of the roles of the Execution layer. + A specific implementation (or implementations) of the + StepExecutor + can deal with the concern of breaking apart the business + logic and sharing it efficiently between parallel + processes or processors. There are a number of + technologies that could play a role here. The essence is + just a set of concurrent remote calls to distributed + agents that can handle some business processing. Since + the business processing is already typically modularised + - e.g. input an item, process it - Spring Batch can + strategise the distribution in a number of ways. One + implementation that we have had some experience with + (and have a prototype for) is a set of remote EJBs + handling the business processing. We switch off Home + caching in the container and then send a specific range + of primary keys for the inputs to each of a number of + remote calls. The same basic strategy would work with + any of the Spring Remoting protocols (plain RMI, + HttpInvoker, JMS, Hessian etc.) with little more than a + couple of lines change in the execution layer + configuration. + + + + + What are the key concepts in the Spring Batch core + domain? + + +

+ In a nutshell: A JobConfiguration with a list of + StepConfigurations is passed to a JobExecutor. + From this a Job is constructed consisting of a + series of Steps, each of which is executed by a + StepExecutor. The StepExecutor contains all the + strategies for deciding when to complete, when to + commit, when to abort and when to continue. +

+

+ Many Jobs in practice consist of a single Step. Step + is very useful and best practice for breaking a Job + down into logical units, rather than having to + execute separate Jobs (potentially in separate OS + processes) which have no obvious logical connection. +

+

+ Jobs can be executed once, or many times with + different logical identifiers (JobIdentifier). It is + also possible to restart a failed Job with the same + or a modified input source, and identify the + resulting JobExecution as a separate entity. In this + way the progress of a Job and its history of + successful and failed executions can easily be + tracked. The same argument applies to Steps, which + have their corresponding StepExecution entity. +

+ + + + + How can messaging be used to scale batch architectures? + + + There is a good deal of practical evidence from existing + projects that a pipeline approach to batch processing is + highly beneficial, leading to resilience and high + throughput. We are often faced with mission-critical + applications where audit trails are essential, and + guaranteed processing is demanded, but where there are + extremely tight limits on performance under load, or + where high throughput gives a competitive advantage. + Matt Welsh's work shows that a Staged Event Driven + Architecture (SEDA) has enormous benefits over more + rigid processing architectures, and message-oriented + middleware (JMS, AQ, MQ, Tibco etc.) gives us a lot of + resilience out of the box. There are particular benefits + in a system where there is feedback between downstream + and upstream stages, so the number of consumers can be + adjusted to account for the amount of demand. So how + does this fit into Spring Batch? Well it's a good + example of an + StepExecutor + or (more broadly) execution runtime if the deployment is + grid- or cluster-based, or in any way involves multiple + OS processes. + + + + How can I contribute to Spring Batch? + + Use JIRA and the forum to get involved in discussions + about the product and its design. There is a process for + contributions and eventually becoming a committer. The + process is pretty standard for all Apache-licensed + projects. You make contributions through JIRA (so sign + up now); you assign the copyright of any contributions + using a standard Apache-like CLA (see the Apache one for + example - ours might be slightly different); when the + contributions reach a certain level, or you convince us + otherwise that you are going to be committed long term, + even if part time, then you can become a committer. + + + + diff --git a/src/site/ppt/Figures.ppt b/src/site/ppt/Figures.ppt index e3c0c354e..519f59798 100644 Binary files a/src/site/ppt/Figures.ppt and b/src/site/ppt/Figures.ppt differ diff --git a/src/site/site.xml b/src/site/site.xml index 9668a2333..2d530dcdf 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -28,9 +28,10 @@ - + - + +