diff --git a/src/site/apt/features.apt b/src/site/apt/features.apt index 6d04d1686..f11cd38a5 100644 --- a/src/site/apt/features.apt +++ b/src/site/apt/features.apt @@ -3,13 +3,15 @@ ------ Dave Syer ------ - July 2007, February 2008 + July 2007, February 2008, January 2009 Spring Batch Features and Roadmap -* 1.0 Features + See also {{{migration/2.0-highlights.html}details of main themes of 2.0}}. - The following features are supported by Spring Batch 1.0: +* 2.0 Features + + The following features are supported by Spring Batch 2.0: ** Optimisation and Infrastructure @@ -35,15 +37,13 @@ Spring Batch Features and Roadmap * ItemWriter abstraction and implementations for flat files and xml (the Sql case is just a regular Jdbc Dao). - * ItemReader and ItenWriter implementations are generally - ItemStreams. An ItemStream encapsulates stream-like behaviour that - is needed for transaction synchronization (mark/reset). It also - provides the facility to be restored from a persistent - ExecutionContext so that jobs can fail and be restarted in - another process. + * ItemReader and ItemWriter implementations are generally + ItemStreams. An ItemStream provides the facility to be restored + from a persistent ExecutionContext so that jobs can fail and be + restarted in another process. * For modifying an item before it is written, there is the - ItemTransformer abstraction. ItemTransformer and ItemWriter are the + ItemProcessor abstraction. ItemProcessor and ItemWriter are the two most common application developer touch points. ** Core Domain @@ -58,14 +58,25 @@ Spring Batch Features and Roadmap * Step is the corresponding point for a StepExecution. Step is the main strategy for different scaling, distribution and processing - approaches. The 1.0 release contains implementations for in-process - execution (single VM). See below (under Execution). + approaches. The 2.0 release contains implementations for in-process + execution (single VM), and a PartitionStep as part of an SPI for + remote execution of steps. See below (under Execution). * The most commonly used implementation of Step is a wrapper for an ItemReader and an ItemWriter. There is also a special implementation that wraps a Tasklet, which can be used to execute a single action like a stored procedure call. + * FactoryBeans are provided for creating Step instances with the + most common features. See in particular + FaultTolerantStepFactoryBean for a factory that provides convenient + configuration points for skips and retries. + + * Late binding of environment properties, job parameters and + execution context values into a Step when it starts. A custom + Spring Scope takes care of deferring the initialization of + components until a step is executing. + ** Job Execution and Management * A simple JobLauncher to launch jobs. Start a new one or restart @@ -80,10 +91,6 @@ Spring Batch Features and Roadmap represenation of the state of the job - can be augmented by developers). - * StepHandlerStep - uses an ItemReader to obtain the next record to - process, and hands it to an ItemWriter if it is not null. It can - run a StepExecution in the same process (VM). - * Adjustiable exception handling strategies allowing fault tolerance through skipping bad records. @@ -100,22 +107,35 @@ Spring Batch Features and Roadmap * A set of listener callbacks that users can implement and register with a Step to add custom behaviour like footer records. + * Remote chunking of steps. The step proceeds as in the single JVM + case, but each chunk is passed on to the remote processes. The + remote execution is an asynchronous listener of some sort + (e.g. message-driven component or web service). Implemented using + {{{http://www.springsource.org/spring-integration}Spring + Integration}} in a Batch sub-project (spring-batch-integration). + + * Partitioning - steps execute concurrently and optionally in + separate processes. Feedback loop between consumers and producers + to prevent overflows. Spring Batch provides an SPI for Partitioning + and an implementation for local (multi-threaded, single JVM) + execution. + + * OSGi support. Deploy the Spring Batch framework as a set of OSGi + services. Deploy individual jobs or groups of jobs as additional + bundles that depend on the core. Spring Batch JAR files are also + OSGi bundles, and can be deployed easily in + {{{http://www.springsource.com/dmserver}SpringSource dm Server}}. + + * Non-sequential models for Job configuration (branching and + descision support). + ** Samples * A range of samples is available as a separate module. They all use a common simple configuration and extend in various ways to show the different features of the Execution module. -* Roadmap (Beyond 1.0). - - * Remote or distributed execution of steps. The step proceeds as in - the single JVM case, but each chunk is passed on to the remote - processes. The remote execution is an asynchronous listener of some - sort (e.g. message-driven component or web service). - - * Asynchronous pipeline processing - steps execute concurrently and - optionally in separate processes. Feedback loop between consumers - and producers to prevent overflows. +* Roadmap (Beyond 2.0). * Issue tracking - a job is not finished until all issues with its executions are resolved. Spring Batch can provide hooks to @@ -127,13 +147,11 @@ Spring Batch Features and Roadmap richer options for detailed outcome reports), but also who has executed the job, what changes they made to runtime parameters. - * OSGi support. Deploy the Spring Batch framework as a set of OSGi - services. Deploy individual jobs or groups of jobs as additional - bundles that depend on the core. +* SpringSource Enterprise Batch - * Non-sequential models for Job configuration (branching and - descision support). + * The plan is for {{{http://www.springsource.com}SpringSource}} to + provide an enterprise product that deals with runtime concerns, as + opposed to programming and configuration. -* No Plans Yet to Support - - * Triggering. + * Triggering. Other runtime concerns, like monitoring and managemtn + of jobs and historical executions. diff --git a/src/site/apt/migration/2.0-highlights.apt b/src/site/apt/migration/2.0-highlights.apt new file mode 100644 index 000000000..364473af4 --- /dev/null +++ b/src/site/apt/migration/2.0-highlights.apt @@ -0,0 +1,163 @@ + ------ + Spring Batch 2.0 Highlights + ------ + Dave Syer + ------ + January 2009 + +Spring Batch 2.0 Highlights + + Here we outline the main themes of Spring Batch 2.0, and highlight the changes from 1.x. + +* Spring Batch 2.0 Themes + + The four main themes of the new release are + + * Java 5 and Spring 3.0 + + * Non-sequential execution + + * Scalability + + * Configuration: annotations and XML namespace + + so we'll cover each of those areas separately and describe what they mean and the impact of the changes on Spring Batch existing users. There is more detail below for features that are already implemented, which is mostly in the first category with some enabling features in other areas. + + There are no changes to the physical layout of the project in Spring Batch 2.0.0.M2 (same old downloads, same basic layout of Java packages). We have not removed any features, but we have taken the opportunity to revise a couple of APIs, and there are some minor changes for people updating projects from 1.x. Spring Batch is immature enough and we were adding some pretty big features, so we decided a major version change was a good opportunity to have a bit of a clean out. We don't expect anyone to have any difficulty upgrading, and if you are an existing user this page will help you to get the measure of the changes. + +* Java 5 + + As you may know, Spring 3.0 is going to be the first major release of Spring to target Java 5 exclusively (I'll leave it to Juergen and Arjen to clarify that in more detail). Now that Sun has put an "{{{http://java.dzone.com/articles/rip%E2%80%A6jdk-14}End of Service Life}}" stamp on the JDK 1.4 it seems appropriate, and there are some great new features on Spring 3.0 that we want to take advantage of. + +** Type Safety + + Most of the work in the 2.0.0.M1 release of Spring Batch went into converting the existing code to Java 5, taking advantage of generics and parameterised types wherever we could. This gives users of the framework a much nicer programming experience, allowing compile time checks for type safety and ultimately reducing maintenance costs for projects using Spring Batch. For instance the in the <<>>, one of the central interfaces and user extension points in Spring Batch, we now have a typesafe <<>> method: + ++--- +public interface ItemReader { + S read(); +} ++--- + + A further point to note here is that the old (1.x) framework callbacks <<>> and <<>> have gone from this interface, making it more friendly to end users, and preventing misunderstanding about what the framework requires and when. The same concerns (or mark and reset) are now handled internally in the <<>> implementations that the framework provides. + +** Chunk-oriented Processing + + Similar changes, and one slightly more radical, have also occurred in the partner <<>> interface, used by the framework for writing data: + ++--- +public interface ItemWriter { + void write(List items); +} ++--- + + The old framework callbacks for <<>> and <<>> have gone from this interface too, and to compensate for that, the <<>> method has a new signature. The bottom line here is that we have moved to a chunk-oriented processing paradigm internally to the framework. This is actually much more natural in a batch framework than the old item-oriented approach because for performance reasons we often need to buffer and flush, and the old interface made that awkward for users. Now you can do all the batching you need inside the <<>> method. + +** Step Factory Bean Changes + + A side effect of the chunk-oriented approach to processing is a change in the step factory bean implementations. The old <<>> has been renamed to <<>>, and it can still be used for most common use cases as a replacement for the old factory bean. By default it creates a step implementation that buffers input items across rollbacks, so that the new <<>> and <<>> interfaces work properly with non-transactional input sources (like files). For input sources that re-present the items after a rollback, clients have to set a flag in the factory bean (<<>> is the name of the flag as of M2) - <<>> is the only relevant reader in the framework and not many projects use it so this isn't a big change. + +** Business Processing + + A related new feature is that there is a new kid on the block in the form of the <<>>: + ++---- +public interface ItemProcessor { + T process(S item); +} ++---- + + In 1.x the transformation between input items of type <<>> and output items of type <<>> had to be hidden inside one of the other participants (usually the <<>>). Now we have genericised this concern and placed it at the same level of importance in the framework as its siblings, <<>> and <<>>. Users of 1.x might recognise the traces of the old <<>> interface here, which has now been removed. + +** A More Useful Tasklet Interface + + Many people, looking at Spring Batch 1.x, have asked "what if my business logic is not reading and writing?" To answer this question more satisfactorily we have modified the <<>> interface. In Spring Batch 1.x it is fairly bland - little more than a <<>> in fact, but in 2.0 we have given it some more flexibility and slotted it more into the mainstream of the framework (e.g. the central chunk-oriented step implementation is now implemented as a <<>>). Here is the new interface: + ++---- +public interface Tasklet { + ExitStatus execute(StepContribution contribution, + AttributeAccessor attributes); +} ++---- + + The idea is that the tasklet can now contribute more back to the enclosing step, and this makes it a much more flexible platform for implementing business logic. The <<>> was already part of the 1.x API, but it wasn't very publicly exposed. Its role is to collect updates to the current <<>> without the programmer having to worry about concurrent modifications in another thread. This also tells us that the <<>> will be called repeatedly (instead of just once per step in the 1.x framework), and so it can be used to carry out a greater range of business processing tasks. The <<>> is a chunk-scoped bag of key-value pairs. The tasklet can use this to store intermediate results that will be preserved across a rollback. + +** Late Binding of Job and Step Attributes + + Late binding of <<>> and <<>> attributes to step components is possible in a very generic way. This is a much requested feature, and we have some workarounds for special cases in 1.x, like the <<>> for binding to a filename as an input parameter to a job. A more generic solution available in Spring Batch 2.0 is to allow those step attributes to be bound to arbitrary components, by defining them in an appropriate Spring scope, e.g. + ++---- + + + + + ... + + + ... + ++---- + + The item reader needs to be bound to a file name that is only available at runtime. To do this we have declared it as scope="step" and used the Spring EL binding pattern <<<#{...}>>> to bind in a job parameter. The same pattern works with step and job level execution context attributes (binding at step execution time). + + Step scoped beans are also a good solution to the old Spring Batch problem of how to keep your steps thread safe. If a step relies on a stateful component like a <<>>, then it only has to define that component as scope="step" and the framework creates a lazy initializing proxy for it, and it will be created as needed once per step execution. There's still noting wrong with creating a new <<>> for each job execution, which is the current best practice in 1.x for keeping threads from colliding across jobs. But now step scope gives you another option, and one that is probably easier for most Spring users to get to grips with. + +* Spring 3.0 + + Spring Batch 2.0 depends on Spring 2.5.6 (the lastest stable version at the time the development of Spring Batch was in progress). The plan is to add a Spring 3.0 dependence for Spring Batch 2.1 (while keeping the option of 2.5.6). This provides some useful new features in particular in the configuration of jobs and steps using late binding with Spring Expression Language (EL), which has the same syntax as we are using for late binding in Spring Batch 2.0, but has more features and is rather more flexible. + +* Non-sequential Execution + + In 1.x the model of a job was always as a linear sequence of steps, and if one step failed, then the job failed. Although many jobs still fit that pattern so it hasn't gone away, in 2.0 we are lifting that restriction by introducing some new features. These are planned for the M3 release so the implementation details might change, but the idea is to support three features: + + * Conditional execution: branching to a different step based on the <<>> of the last one. This includes the ability to branch on a FAILED status, which implies that a step failure is no longer fatal for a job. + + * Pause execution and wait for explicit instruction to proceed. This is useful for instance where there is a business rule that forces manual intervention to check the validity of business critical data. + + * Parallel execution of multiple steps. Where steps are independent the user can specify which branches can be executed in parallel. + + These features are available through the custom XML namespace described briefly below. + +* Scalability + + Spring Batch 1.x was always intended as a single VM, possibly multi-threaded model, but we built a lot of features into it that support parallel execution in multiple processes. Many projects have successfully implemented a scalable solution relying on the quality of service features of Spring Batch to ensure that processing only happens in the correct sequence. In 2.0 we expose those features more explicitly. There are two approaches to scalability, and we support both: remote chunking, and partitioning. + +** Remote Chunking + + Remote chunking is a technique for dividing up the work of a step without any explicit knowledge of the structure of the data. Any input source can be split up dynamically by reading it in a single process (as per normal in 1.x) and sending the items as a chunk to a remote worker process. The remote process implements a listener pattern, responding to the request, processing the data and sending an asynchronous reply. The transport for the request and reply has to be durable with guaranteed delivery and a single consumer, and those features are readily available with any JMS implementation. But Spring Batch is building the remote chunking feature on top of {{{http://www.springframework.org/spring-integration}Spring Integration}}, so actually it is agnostic to the actual implementation of the message middleware. + +** Partitioning + + Partitioning is an alternative approach which in contrast depends on having some knowledge of the structure of the input data, like a range of primary keys, or the name of a file to process. The advantage of this model is that the processors of each element in a partition can act as if they are a single step in a normal Spring Batch job. They don't have to implement any special or new patterns, which makes them easy to configure and test. Partitioning in principle is more scalable than remote chunking because there is no serialization bottleneck arising from reading all the input data in one place. + + In Spring Batch 2.0 partitioning is supported by two interfaces: <<>> and <<>>. The <<>> is the one that knows about the execution fabric - it has to transmit requests to remote steps and collect the results using whatever grid or remoting technology is available. <<>> is an SPI, and we provide one implementation out of the box for local execution through a <<>>. This will be useful immediately to a number of projects we have seen where parallel processing of heavily IO bound tasks is required, since in those cases remote execution only complicates the deployment and doesn't necessarily help much with the performance. Other implementations will be specific to the execution fabric, e.g. one of the grid providers (IBM, Oracle, Terracotta, Appistry etc.), and we don't want to imply a preference for any of those over the others in Spring Batch. + + SpringSource is planning an Enterprise Batch product that will provide a full runtime solution for partitioning and remote chunking, as well as admin and scheduling concerns. + +* Configuration: annotations and XML namespace + + The idea behind using annotations to implement batch logic is by analogy with Spring @MVC. The net effect is that instead of having to implement and possibly register a bunch of interfaces (reader, writer, processor, listeners, etc.) you would just annotate a POJO and plug it into a step. There are method level annotations corresponding to the various interfaces, and parameter level annotations and factory methods corresponding to job, step and chunk level attributes (a bit like <<<@ModelParameter>>> and <<<@RequestParameter>>> in Spring @MVC. + + A XML namespace for Spring Batch makes configuration of common things even easier. For example, the <<>> we mentioned above has an XML configuration option, so a simple conditional execution might look like this: + ++---- + + + + + + + + + ++---- + + The first step ("gamesLoad") is followed immediately by "playerLoad", but if that fails, then we can go to an alternative step ("compensate") instead of finishing the job normally with the "summarize" step. The name= attribute of the <<<<step>>> element in the XML is a bean reference, so the implementation of <<>> is defined elsewhere (possibly through annotations). + +* Database Schema Changes + + There are a couple of tidying up tasks and extensions to the data model in the meta data schema. We provide update scripts for anyone moving from 1.x to 2.0, so they shouldn't cause any problems, and will definitely make it easier to navigate and interact with the meta data. For those of you who are new to Spring Batch, some of the key benefits of the framework are the quality of service features like restartability and idempotence (process business data once and only once). We implement these features through shared state in a relational database (in most use cases), and the definition of the data model in this database has changed slightly in 2.0. + + The main changes are to do with the storage of <<>>, which used to be centralised in one table, even though the context can be associated either with a <<>> or a <<>>. The new model will be more popular with DBAs because it makes the relationships more transparent in the DDL. We also started storing the context values in JSON, to make them easier to read and track for human users (the context for a single entity is all stored in one row in a table, instead of many). + + We have also added some more statistics for the counting and accounting of items executed and skipped, splitting out counts for total items read, processed and written at each stage. For steps (or tasklets) that do not split their execution into read, process, write, this is more comprehensive than is needed, but for the majority use case it is more appropriate than just storing an overall item count. diff --git a/src/site/apt/migration/index.apt b/src/site/apt/migration/index.apt index 778ea0a43..42989ea82 100644 --- a/src/site/apt/migration/index.apt +++ b/src/site/apt/migration/index.apt @@ -12,6 +12,8 @@ You can also browse the upcoming releases and look at the {{{http://opensource.atlassian.com/projects/spring/browse/BATCH?report=com.atlassian.jira.plugin.system.project:roadmap-panel}Road Map}} query. Links: + + * {{{2.0-highlights.html}Highlights of changes between 1.x and 2.0}} * {{{2.0-m2-m3.html}2.0.0.M2 to 2.0.0.M3}}