Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[BATCH-429] Merge docos over from execution

Browse Source
This commit is contained in:
dsyer
2008-03-07 15:52:55 +00:00
parent 2b0e75b3bf
commit 58a178a5b4
7 changed files with 883 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
spring-batch-core/src/site/apt/executable.apt Normal file
View File

@@ -0,0 +1,38 @@
Tried to create an executable jar with this:
+---
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>org.springframework.batch.container.bootstrap.BatchCommandLineLauncher</mainClass>
<addClasspath>true</addClasspath>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
+---
But the resulting MANIFEST.MF is rubbish. Look at the classpath
(where did that come from)?
+---
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: dsyer
Build-Jdk: 1.5.0_09
Main-Class: org.springframework.batch.container.bootstrap.BatchCommand
LineLauncher
Class-Path: spring-2.1-m2.jar commons-logging-1.1.jar log4j-1.2.12.jar
dom4j-1.6.1.jar commons-lang-2.1.jar spring-batch-infrastructure-1.0
-m2-SNAPSHOT.jar antlr-2.7.6.jar commons-collections-2.1.1.jar hibern
ate-3.2.3.ga.jar spring-mock-2.1-m2.jar ehcache-1.2.3.jar
+---

46
spring-batch-core/src/site/apt/glossary.apt Normal file
View File

@@ -0,0 +1,46 @@
------
Glossary
------
Wayne Lund
------
May 2007
[[1]]<<Batch>>: An accumulation of business transactions over time.
[[2]]<<Batch Application Style>>: Term used to designate batch as an application style in its own right similar to online, Web or SOA. It has standard elements of input, validation, transformation of information to business model, business processing and output. In addition, it requires monitoring at a macro level.
[[3]]<<Batch Processing>>: The handling of a batch of many business transactions that have accumulated over a period of time (e.g. an hour, day, week, month, or year). It is the application of a process, or set of processes, to many data entities or objects in a repetitive and predictable fashion with either no manual element, or a separate manual element for error processing.
[[4]]<<Batch Window>>: The time frame within which a batch job must complete. This can be constrained by other systems coming online, other dependent jobs needing to execute or other factors specific to the batch environment.
[[5]]<<Step Controller>>: It is the main batch task or Unit of Work controller. It initializes the module, and controls the transaction environment based on commit interval setting, etc.
[[6]]<<Tasklet>>: The main application program created by application developer to process the business logic for each LUW.
[[7]]<<Batch Job Type>>: Job Types describe application of jobs for particular type of processing. Common areas are interface processing (typically flat files), forms processing (either for online pdf generation or print formats), report processing. s
[[8]]<<Driving Query>>: A driving query identifies the set of work for a job to do; the job then breaks that work into individual units of work. For instance, identify all financial transactions that have a status of "pending transmission" and send them to our partner system. The driving query returns a set of record IDs to process; each record ID then becomes a unit of work. A driving query may involve a join (if the criteria for selection falls across two or more tables) or it may work with a single table.
[[9]]<<Logicial Unit of Work (LUW)>>: A batch job iterates through a driving query (or another input source such as a file) to perform the set of work that the job must accomplish. Each iteration of work performed is a unit of work.
[[10]]<<Commit Interval>>: A set of LUWs constitute a commit interval.
[[11]]<<Partitioning>>: Splitting a job into multiple threads where each thread is responsible for a subset of the overall data to be processed. The threads of execution may be within the same JVM or they may span JVMs in a clustered environment that supports workload balancing.
[[12]]<<Staging Table>>: A table that holds temporary data while it is being processed.
[[13]]<<Restartable>>: - a job that can be executed again and will assume the same identity as when run initially. In othewords, it is has the same job instance id.
[[14]]Rerunnable - a job that is restartable and manages it's own state in terms of previous run's record processing. Note>>: Rerunnable is tied to the driving query. If the query can be formed so that it will limit the processed rows when the job is restarted than re-runnable = true. Often times a condition is added to the where statement to limit the rows returned by the driving query with something like "and processedFlag != true".
---------------------------------------------------------------------
Note: If its false the architecture assumes responsibility for tracking which rows have been processed. There is a default strategy for tracking the last record processed by partition. Most batch jobs only have one partition. The option is only valid for a restartable job. The reason being is that we have to persist the restart data which is only available on a restartable job.
In DSL it is the following:
StartOver ::= restartable = false. Restartable ::= true | false
If (Restartable)
re-runnable ::= true | false
We don't persist restart information for a non-restartable job. As you can see, it doesn't make sense. Rerunnable has always confused the best of us.
----------------------------------------------------------------------------

72
spring-batch-core/src/site/apt/index.apt
View File

@@ -5,9 +5,9 @@
------
August 2007
Overview of the Spring Batch Core Domain
Overview of the Spring Batch Core
The Spring Batch Core Domain consists of a public API for launching,
The Spring Batch Core Domain consists of an API for launching,
monitoring and managing batch jobs.
[images/core-domain-overview.png] The Spring Batch Core Domain with
@@ -34,20 +34,19 @@ dependencies to infrastructure indicated schematically.
[images/core-domain-extended.png] The Spring Batch Core Domain
extended to include the datababase entities and identifier strategy.
A <<<JobConfiguration>>> can be re-used to create multiple job
A <<<Job>>> can be re-used to create multiple job
instances and this is reflected in the figure above showing an
extended picture of the core domain. When a <<<JobConfiguration>>>
is launched the <<<JobExecutor>>> first checks to see if a job with
the same <<<JobIdentifier>>> was already executed. We expect one of
the following outcomes, depending on the <<<JobExecutor>>>
implementation and <<<JobConfiguration>>>:
extended picture of the core domain. When a <<<Job>>>
is launched it first checks to see if a job with
the same <<<JobParameters>>> was already executed. We expect one of
the following outcomes, depending on the <<<Job>>>:
* If the job was not previously launched then it can be created
and executed. A new <<<JobInstance>>> is created and stored in a
repository (usually a database). A new <<<JobExecution>>> is also
created to track the progress of this particular execution.
* If the job was previously launched the <<<JobConfiguration>>>
* If the job was previously launched the <<<Job>>>
has a flag indicating whether or not to continue and launch a new
execution (was this expected?). The decision is parameterised to
depend on whether or not the job failed last time it was executed.
@@ -58,3 +57,58 @@ extended to include the datababase entities and identifier strategy.
doesn't need to be distinguished from previous runs. In either
case a new <<<JobExecution>>> is created and stored to monitor
this execution of the <<<JobInstance>>>.
Overview of the Spring Batch Simple Batch Execution Container
The diagram below provides an overview of the high level components, technical services, and basic operations required by a batch architecture. This architecture framework is a blueprint that has been proven through decades of implementations on the last several generations of platforms (COBOL/Mainframe, C++/Unix, and now Java/anywhere). The Simple Batch Execution Container provides a physical implementation of the layers, components and technical services commonly found in robust, maintainable systems used to address the creation of simple to complex batch applications, with the infrastructure and extensions to address very complex processing needs. The materials below will walk through the details of the diagram.
[images/simple-batch-execution-container.jpg] Simple Batch Execution Container high level flow and interaction of the architecture.
Tiers
The application style is organized into four logical tiers, which include Run, Job, Application, and Data tiers. The primary goal for organizing an application according to the tiers is to embed what is known as "separation of concerns" within the system. Effective separation of concerns results in reducing the impact of change to the system.
* <<Run Tier:>> The Run Tier is concerned with the scheduling and launching of the application. A vendor product is typically used in this tier to allow time-based and interdependent scheduling of batch jobs as well as providing parallel processing capabilities.
* <<Job Tier:>> The Job Tier is responsible for the overall execution of a batch job. It sequentially executes batch steps, ensuring that all steps are in the correct state and all appropriate policies are enforced.
* <<Application Tier:>> The Application Tier contains components required to execute the program. It contains specific modules that address the required batch functionality and enforces policies around a module execution (e.g., commit intervals, capture of statistics, etc.)
* <<Data Tier:>> The Data Tier provides the integration with the physical data sources that might include databases, files, or queues. <<Note>>: In some cases the Job tier can be completely missing and in other cases one Job Script can start several Batch Job instances.
High Level Processing Flow
The diagram above illustrates the flow and architecture components in a typical batch run execution.
Standard interaction is described as follows:
<<1.>> In the Run tier, a Scheduler starts a batch application by invoking a Job Script. The Scheduler identifies what batch process it wants to run by passing the name of the batch process and any required additional parameters to the Job Script.
<<2.>> The Job Script initializes the program and executes any job specific scripts prior to calling the Batch Launcher.
<<3.>> The Batch Launcher starts the Batch Execution Container based upon any environment settings established in the script. (NOTE: A Batch Execution Container is not a Java EE container)
<<3.1>> The Batch Container starts and controls the batch execution. It initializes the Job execution environment with static configuration items such as database settings, logging levels and creates a Job based on the Job Configuration created by a Batch Developer.
<<4>> Based on configuration provided by a Batch Developer, the Job sequentially executes steps after checking policies to ensure that each step should be started. The status of the job and step (start time, end time, status such as "started" or "completed") is stored at various points during the process.
<<5.1>> In order to maintain data integrity, at the application tier, the Step acts as a controller to ensure that either an entire group of actions completes successfully or that none of the actions completes. This group of actions is referred to as a logical unit of work (LUW). The Step controls the overall execution of the Tasklet, ensuring that transaction are committed at the appropriate time, and restart and statistics information is stored appropriately. The first thing the Step is responsible for is the initialization of the data required to begin processing. The Step will interact with other architecture components, such as the Input Source, to setup the data required to be processed.
<<5.1.1>> The Input Source provides services to access various data sources. It provides location transparency to the Batch Tasklet and hides the physical location details of the data.
<<5.2>> Once the data is initialized by the Input Source, the Step will call into the Tasklet to begin processing. The Tasklet contains the business logic to define the LUW and the Step repeatedly calls the Tasklets LUW to finish the business function. The Step does this by first invoking the execute method on the Tasklet in order to acquire a single record/set of data for processing.
<<5.2.1>> Before a record is returned to the Tasklet, it may be validated by any number of validation Frameworks that can be provided to an input source. A single record/set of data is gathered by interacting with the Input Source.
<<5.3>> Once a record/set has been obtained, the step calls the module to begin processing.
<<5.3.1>> The Tasklet executes its internal business logic by calling other Business Logic components as necessary. Based on the business service, it can requests or persists objects from the data access components.
<<5.3.3>> Data Access components can be leveraged retrieve or persist domain objects.
<<5.3.4>> Once the business logic has been executed, the resulting output record is written out by utilizing the Output Source interface. The Step will repeatedly call steps 4.2 \-> 4.4 for every record provided by the Input Source.
<<5.4>> Once all of the records are processed, the Step calls the Tasklet to perform any clean up activities such as closing connections, exporting files, etc.
<<5.4.1>> The Step is responsible for committing data associated with the remaining logical units of work as well as performing any finalization and administrative functions (e.g. closing database connections).
Once the Step has completed finalization the control is passed back to the Job, where any necessary logging or clean up is executed for application termination and wrap-up -- provided there are no additional Steps to execute.

201
spring-batch-core/src/site/apt/introduction.apt Normal file
View File

@@ -0,0 +1,201 @@
------
Batch Processing Strategy
------
Scott Wintermute
------
May 2007
Batch Processing Strategy
To help design and implement batch systems, basic batch application building blocks and patterns should be provided to the designers and programmers in form of sample structure charts and code shells. When starting to design a batch job, the business logic should be decomposed into a series of steps which can be implemented using the following standard building blocks:
* Conversion Applications: For each type of file supplied by or generated to an external system, a conversion application will need to be created to convert the transaction records supplied into a standard format required for processing. This type of batch application can partly or entirely consist of translation utility modules (see Basic Batch Services).
* Validation Applications: Validation applications ensure that all input/output records are correct and consistent. Validation is typically based on file headers and trailers, checksums and validation algorithms as well as record level cross-checks.
* Extract Applications: An application that reads a set of records from a database or input file, selects records based on predefined rules, and writes the records to an output file.
* Extract/Update Applications: An application that reads records from a database or an input file, and makes changes to a database or an output file driven by the data found in each input record.
* Processing and Updating Applications: An application that performs processing on input transactions from an extract or a validation application. The processing will usually involve reading a database to obtain data required for processing, potentially updating the database and creating records for output processing.
* Output/Format Applications: Applications reading an input file, restructures data from this record according to a standard format, and produces an output file for printing or transmission to another program or system.
<<Pre-processing Capabilities>>
Additionally a basic application shell should be provided for business logic that cannot be built using the previously mentioned building blocks.
In addition to the main building blocks, each application may use one or more of standard utility steps, such as:
* Sort - A Program that reads an input file and produces an output file where records have been re-sequenced according to a sort key field in the records. Sorts are usually performed by standard system utilities.
* Split - A program that reads a single input file, and writes each record to one of several output files based on a field value. Splits can be tailored or performed by parameter-driven standard system utilities.
* Merge - A program that reads records from multiple input files and produces one output file with combined data from the input files. Merges can be tailored or performed by parameter-driven standard system utilities.
Batch applications can additionally be categorized by their input source:
* Database-driven applications are driven by rows or values retrieved from the database.
* File-driven applications are driven by records or values retrieved from a file
The foundation of any batch system is the processing strategy. Factors affecting the selection of the strategy include estimated batch system volume, concurrency with on-line or with another batch systems, available batch windows etc. Also with more enterprises wanting to be up and running 24x7, leaving no obvious batch windows.
Typical processing options for batch are:
* Normal processing in a batch window during off-line
* Concurrent batch / on-line processing
* Parallel processing of many different batch runs or jobs at the same time
* Streaming i.e. processing of many instances of the same job at the same time
* A combination of these
The order in the list above reflects the implementation complexity, processing in a batch window being the easiest and streaming the most complex to implement.
Some or all of these options may be supported by a commercial scheduler.
In the following section these processing options will be discussed in more detail. It is important to notice that the commit and locking strategy adopted by batch processes will be dependent on the type of processing performed and as a rule of thumb, the on-line locking should use the same principles. Therefore a batch architecture cannot be simply an afterthought when designing an overall architecture.
The locking strategy can use only normal database locks, or an additional custom locking service can be implemented in the architecture. The locking service would track database locking (for example by storing the necessary information in a dedicated db-table) and give or deny permissions to the application programs requesting a db operation. Retry logic could also be implemented by this architecture to avoid aborting a batch job in case of a lock situation.
<<1. Normal processing in a batch window>>
For simple batch processes running in a separate batch window, where the data being updated is not required by on-line users or other batch processes, concurrency is not an issue and a single commit can be done at the end of the batch run.
In most cases a more robust approach is more appropriate. A thing to keep in mind is that batch systems have a tendency to grow as time goes by, both in terms of complexity and the data volumes they will handle. If no locking strategy is in place and the system still relies on a single commit point, modifying the batch programs can be painful. Therefore, even with the simplest batch systems, consider the need for commit logic depicted in the [Restart/Recovery section|Restart & Recovery] as well as the information concerning the more complex cases below.
<<2. Concurrent batch / on-line processing>>
Batch applications processing data that can simultaneously be updated by on-line users, should not lock any data (either in the database or in files) which could be required by on-line users for more than a few seconds. Also updates should be committed to the database at the end of every few transaction. This minimizes the portion of data that is unavailable to other processes and the elapsed time the data is unavailable.
Another option to minimize physical locking is to have a logical row-level locking implemented using either an Optimistic Locking Pattern or a Pessimistic Locking Pattern.
* Optimistic locking assumes a low likelihood of record contention. It typically means inserting a timestamp column in each database table used concurrently by both batch and on-line processing. When an application fetches a row for processing, it also fetches the timestamp. As the application then tries to update the processed row, the update uses the original timestamp in the WHERE clause. If the timestamp matches, the data and the timestamp will be updated successfully. If the timestamp does not match, this indicates that another application has updated the same row between the fetch and the update attempt and therefore the update cannot be performed.
* Pessimistic locking is any locking strategy that assumes there is a high likelihood of record contention and therefore either a physical or logical lock needs to be obtained at retrieval time. One type of pessimistic logical locking uses a dedicated lock-column in the database table. When an application retrieves the row for update, it sets a flag in the lock column. With the flag in place, other applications attempting to retrieve the same row will logically fail. When the application that set the flag updates the row, it also clears the flag, enabling the row to be retrieved by other applications. Please note, that the integrity of data must be maintained also between the initial fetch and the setting of the flag, for example by using db locks (e.g.,SELECT FOR UPDATE). Note also that this method suffers from the same downside as physical locking except that it is somewhat easier to manage building a time-out mechanism that will get the lock released if the user goes to lunch while the record is locked.
These patterns are not necessarily suitable for batch processing, but they might be used for concurrent batch and on-line processing for example in cases where the database doesn't support row-level locking. As a general rule, optimistic locking is more suitable for on-line applications, while pessimistic locking is more suitable for batch applications. Whenever logical locking is used, the same scheme must be used for all applications accessing data entities protected by logical locks.
Note that both of these solutions only address locking a single record. Often we may need to lock a logically related group of records. With physical locks, you have to manage these very carefully in order to avoid potential deadlocks. With logical locks, it is usually best to build a logical lock manager that understands the logical record groups you want to protect and can ensure that locks are coherent and non-deadlocking. This logical lock manager usually uses its own tables for lock management, contention reporting, time-out mechanism, etc.
<<3. Parallel Processing>>
Parallel processing allows multiple batch runs / jobs to run in parallel to minimize the total elapsed batch processing time. This is not a problem as long as the jobs are not sharing the same files, db-tables or index spaces. If they do, this service should be implemented using partitioned data. Another option is to build an architecture module for maintaining interdependencies using a control table. A control table should contain a row for each shared resource and whether it is in use by an application or not. The batch architecture (Control Program Tasklet) or the application in a parallel job would then retrieve information from that table to determine if it can get access to the resource it needs or not.
If the data access is not a problem, parallel processing can be implemented in a mainframe environment using parallel job classes, in order to ensure adequate CPU time for all the processes. In an environment other than the mainframe, a similar solution can be put in place with for example threads. The solution has to be robust enough to ensure time slices for all the running processes.
Other key issues in parallel processing include load balancing and the availability of general system resources such as files, database buffer pools etc. Also note that the control table itself can easily become a critical resource.
<<4. Partitioning>>
Using partitioning allows multiple versions of large batch applications to run in concurrent. The purpose of this is to reduce the elapsed time required to process long batch jobs. Processes which can be successfully partitioned are those where the input file can be split and/or the main database tables partitioned to allow the application to run against different sets of data.
In addition, processes which are partitioned must be designed to only process their assigned data set. A partitioning architecture has to be closely tied to the database design and the database partitioning strategy. Please note, that the database partitioning doesn't necessarily mean physical partitioning of the database, although in most cases this is advisable. The following picture illustrates the partitioning approach:!app_style_batch_processing.png|align=center!
The architecture should be flexible enough to allow dynamic configuration of the number of partitions. Both automatic and user controlled configuration should be considered. Automatic configuration may be based on parameters such as the input file size and/or the number of input records.
<<4.1 Streaming Approaches>>
The following lists some of the possible streaming approaches. Selecting a streaming approach has to be done on a case-by-case basis.
<1. Fixed and Even Break-Up of Record Set>
This involves breaking the input record set into an even number of portions (e.g. 10, where each portion will have exactly 1/10th of the entire record set). Each portion is then processed by one instance of the batch/extract application.
In order to use this approach, preprocessing will be required to split the recordset up. The result of this split will be a lower and upper bound placement number which can be used as input to the batch/extract application in order to restrict its processing to its portion alone.
Preprocessing could be a large overhead as it has to calculate and determine the bounds of each portion of the record set.
<2. Breakup by a Key Column>
This involves breaking up the input record set by a key column such as a location code, and assigning data from each key to a batch instance. In order to achieve this, column values can either be
<3. Assigned to a batch instance via a streaming table (see below for details).>
<4. Assigned to a batch instance by a portion of the value (e.g. values 0000-0999, 1000 - 1999, etc.)>
Under option 1, addition of new values will mean a manual reconfiguration of the batch/extract to ensure that the new value is added to a particular instance.
Under option 2, this will ensure that all values are covered via an instance of the batch job. However, the number of values processed by one instance is dependent on the distribution of column values (i.e. there may be a large number of locations in the 0000-0999 range, and few in the 1000-1999 range). Under this option, the data range should be designed with streaming in mind.
Under both options, the optimal even distribution of records to batch instances cannot be realized. There is no dynamic configuration of the number of batch instances used.
<5. Breakup by Views>
This approach is basically breakup by a key column, but on the database level. It involves breaking up the recordset into views. These views will be used by each instance of the batch application during its processing. The breakup will be done by grouping the data.
With this option, each instance of a batch application will have to be configured to hit a particular view (instead of the master table). Also, with the addition of new data values, this new group of data will have to be included into a view. There is no dynamic configuration capability, as a change in the number of instances will result in a change to the views.
<6. Addition of a Processing Indicator>
This involves the addition of a new column to the input table, which acts as an indicator. As a preprocessing step, all indicators would be marked to non-processed. During the record fetch stage of the batch application, records are read on the condition that that record is marked non-processed, and once they are read (with lock), they are marked processing. When that record is completed, the indicator is updated to either complete or error. Many instances of a batch application can be started without an change, as the additional column ensures that a record is only processed once.
With this option, I/O on the table increased dynamically. In the case of a updating batch application, this impact is reduced, as a write will have to occur anyway.
<7. Extract Table to a Flat File>
This involves the extraction of the table into a file. This file can then be split into multiple segments and used as input to the batch instances.
With this option, the additional overhead of extracting the table into a file, and splitting it, may cancel out the effect of multi-streaming. Dynamic configuration can be achieved via changing the file splitting script.
<8. Use of a Hashing Column>
This scheme involves the addition of a hash column (key/index) to the database tables used to retrieve the driver record. This hash column will have an indicator to determine which instance of the batch application will process this particular row. For example, if there are three batch instances to be started, then an indicator of 'A' will mark that row for processing by instance 1, an indicator of 'B' will mark that row for processing by instance 2, etc.
The procedure used to retrieve the records would then have an additional WHERE clause to select all rows marked by a particular indicator. The inserts in this table would involve the addition of the marker field, which would be defaulted to one of the instances (e.g. 'A').
A simple batch application would be used to update the indicators such as to redistribute the load between the different instances. When a sufficiently large number of new rows have been added, this batch can be run (anytime, except in the batch window) to redistribute the new rows to other instances.
Additional instances of the batch application only require the running of the batch application as above to redistribute the indicators to cater for a new number of instances.
4.2 Database and Application design Principles
An architecture that supports multi-streamed applications which run against partitioned database tables using the key column approach, should include a central streaming repository for storing streaming parameters. This provides flexibility and ensures maintainability. The repository will generally consist of a single table known as the streaming table.
Information stored in the streaming table will be static and in general should be maintained by the DBA. The table should consist of one row of information for each stream of a multi-streamed application. The table should have a similar layout to the following table:
{center}
|| Streaming Table ||
| Program ID Code
Stream Number (Logical ID of the stream)
Low Value of the db key column for this stream
High Value of the db key column for this stream |
{center}
On program start-up the program id and stream number should be passed to the application from the architecture (Control Processing Tasklet). These variables are used to read the streaming table, to determine what range of data the application is to process (if a key column approach is used). In addition the stream number must be used throughout the processing to:
* Add to the output files/database updates in order for the merge process to work properly
* Report normal processing to the batch log and any errors that occur during execution to the architecture error handler
4.3 Minimizing Deadlocks
When applications run in parallel or streamed, contention in database resources and deadlocks may occur. It is critical that the database design team eliminates potential contention situations as far as possible as part of the database design.
Also ensure that the database index tables are designed with deadlock prevention and performance in mind.
Deadlocks or hot spots often occur in administration or architecture tables such as log tables, control tables, lock tables etc.. The implications of these should be taken into account as well. A realistic stress test is crucial for identifying the possible bottlenecks in the architecture.
To minimize the impact of conflicts on data, the architecture should provide services such as wait-and-retry intervals when attaching to a database or when encountering a deadlock. This means a built-in mechanism to react to certain database return codes and instead of issuing an immediate error handling, waiting a predetermined amount of time and retrying the database operation.
4.4 Parameter Passing and Validation
The streaming architecture should be relatively transparent to application developers. The architecture should perform all tasks associated with running the application in a streamed mode i.e.
* Retrieve streaming parameters before application start-up
* Validate streaming parameters before application start-up
* Pass parameters to application at start-up
The validation should include checks to ensure that:
* the application has sufficient streams to cover the whole data range
* there are no gaps between streams
If the database is partitioned, some additional validation may be necessary to ensure that a single stream does not span database partitions.
Also the architecture should take into consideration the consolidation of streams. Key questions include:
* Must all the streams be finished before going into the next job step?
* What happens if one of the streams aborts?

124
spring-batch-core/src/site/apt/outline.apt Normal file
View File

@@ -0,0 +1,124 @@
------------------------------------------
The Spring Batch - Reference Documentation
----------------------------------------
Wayne Lund, Waseem Malik, Lucas Ward, Scott Wintermute,
Kerry O'Brien, Tomi Vanek
-------------------------------------------
May 2007
Preface
*1. {{{introduction.html}Spring Container Batch Processing}}
*1.1. Overview
*1.2 Usage Scenarios
*2. {{{overview.html}Architecture Overview}}
**2.1. Introduction to Architecture Layers
**2.2. Batch Applications
**2.3 Container Application layer
**2.4 Container Support Layer
**2.5 The Container Core Layer
**2.6 Using the Spring-batch infrastructure
*2.6.1 Infrastructure Provided I/O Support
*2.6.2 Infrastucture Provided Base Services
*2.7. Batch Execution Container Configurations
*2.7.1. Single VM Simple Batch Execution Container - One Job, One Step, One Partition
*2.7.2. Single VM Multi-threaded Batch Execution Container Configuration - One Job, One Step, Multiple Partitions
*2.7.3 Batch Execution Container Hosted in J2EE Container - managed environment
*3. Core Batch Services
*3.1. Launching Batch Jobs
*3.2. Mapping Batch Error Codes to Launch Client Error Codes
*3.2.1. Returning error codes to Enterprise Schedulers with command line interfaces
*3.2.2. Web Request - returning error codes to "On-Demand Batch Requests"
*3.3. Job Services
*3.3.1. Step Configuration
*3.3.2. Job Status Service
*3.3.3. Job Status
*3.3.4. Job Statistics
*3.4. Step Services
*3.4.1. Step Execution
*3.4.2. Restart Services
*3.4.3. Skip Services
*3.4.5. Step Statistics
*3.5. Data Providers - Operations, Templates and Convenience Callbacks
*3.5.1. Wrapping input sources
*3.5.2. Validation of input
*3.5.3. Delimited File Data Providers
*3.5.4. Fixed Position File Data Providers
*3.5.5. XML File Data Providers
*3.5.6. SQL Input Source Data Provider
*3.6. Transaction Management
*3.6.1. Transaction Synchronization with non-transactional resources
*3.7. Partitioning Batch Jobs
*3.7.1. Partitioning Strategies
*3.7.2. Partitioning Job Steps
*3.7.3. Partition Status
*3.7.4. Partition Statistics
*3.7.5. Handling Exceptions within partitions
*4. [Batch in a J2EE Container]
*5. [Testing Batch Jobs]
*5.1. Unit Testing & Mock Objects Provided by the Framework
*5.2. Integration Testing
*5.3. Performance Testing Batch Jobs
*6. {{{samples.html}Practical Examples for Spring Batch}}
*6.1. Sample Applications|Spring Reference-Application Job-Map
*6.2. Running the Sample Batch Applications
*7. [INCUB:Batch XML Schema]
*8. {{{glossary.html}Glossary}}

145
spring-batch-core/src/site/apt/overview.apt Normal file
View File

@@ -0,0 +1,145 @@
------
Architecture Overview
------
Wayne Lund
------
May 2007
2. Architecture Overview
*2.1 Introduction
This chapter covers the overall spring batch architecture. The Spring Container Archtiecture is made up of five logical layers; 1) the Batch Application, 2) the Batch Application Layer, 3) the batch core layer, and 4) the batch infrastucture layer.
*----------*----------------*------------+
|Provided By | Layer | Description
*----------*----------------*------------*
| Application Developer | Batch Application | This is where the application writes their batch jobs and modules. |
*----------*----------------*------------*
| Spring Batch Execution Container | Container Application Layer | Allows for extending and overwriting of the batch support layer for custom requirements. Facilities implemented in this layer could migrate down to Batch Support Layer. This is also the layer to add the project specific jars required by job types (e.g. reporting jars like Crystal, Brio, etc, form generation jars like Central Pro or Adobe, etc). |
*----------*----------------*------------*
| Spring Batch Execution Container | Container Support Layer | Provides default implementations of batch core services including I/O, Restart, Partitioning, Statistics, and configurations |
*----------*----------------*------------*
| Spring Batch Execution Container | Container Core Layer | Enables configuration, Common Services & Interfaces, management |
*----------*----------------*------------*
| Spring Batch Infrastructure | Batch-Infrastructure | Provides IO support, Batch style transactions, advanced exception handling, batch-template, batch-retry |
*----------*----------------*------------*
[Figure 2.0] - Batch Architecture Layers
The batch architecture is modeled after a container architecture, meaning that there are managed resources essential to high performance batch architectures that are configured through a spring context. The following sections will provide a quick review of each layer and their role in the batch architecture.
*2.2 Batch Applications
*2.3 Container Application Layer
*2.4 Container Support Layer
The batch support layer provides default implementations for all interfaces, interceptors, advice and other core batch services. Figure 2.3.1 illustrates the following logical packages. !Batch Support.png!
Although physically they break out into many more than depicted, logically you can think of the groupings in the following manner:
* I/O Support packages
* Restart Support
* Lifecycle Support packages
* DAO support layer
**2.4.1 I/O Support Packages
The I/O related packages are currently the richest packages in the batch architecture. They are modeled after Spring Patterns of Operations and Templates. For example, you'll see FlatFileInputOperations accompanied with a FlatFileInputTemplate. The FlatFileInputTemplate is wired up with a File Descriptor, which contains a Record Descriptor along with various other properties. With the File and Record Descriptors the InputTemplate supports a callback method that allows for the mapping of a record into an object. This support applies to fixed length records, delimited records and XML records. To further simplify this a DefaultFlatFileDataProvider is supplied an input template, which contains the field and record descriptions, along with a line mapper that knows how to map the line to an object. The next() operation on a record simply needs to readAndMap(lineMapper) a record. This pattern is used over again for XML and SQL input for simple mapping of input records to objects.
In addition to declarative descriptions of the records that can be re-used by multiple batch jobs, the I/O facilities also support configurable validation strategies. The two currently supported are Apache Commons Validator and Spring's VALang.
**2.4.2 Restart Support
The Restart Support provides implementations for a few common restart strategies that will be discussed further in the respective section. The following are provided out-of-the-box:
* IDList Restart Strategy - a strategy that supports a batch application where the application does not have a "process" flag and needs the batch architecture to track which records have been processed. This is not the ideal scenario.
* Last Processed Restart Strategy - when the record can be identified through a where and order by only the last record(s) processed needs to be saved for restart.
* No Restart Strategy - some batch jobs simply can't support restart. When they are re-run they are considered to be a new instance of a batch job.
* Sql Restart Strategy - \[need some additional javadoc for this strategy\].
**2.4.3 Lifecycle Support
*2.5. The Container Core Layer
The Batch Core interfaces and services are illiustrated in a simplied view of a package diagram. There are roughly seven logical packages:
* Core Spring Extensions
* Core Batch Advice
* Core Batch Configuration
* Core Batch Repository
* Core Batch Tasklet
\\ !Batch Core.png!
[Figure 2.5] Batch Core Layer
In the actual physical packaging there are a few more packages but the above illistration serves as an overview of the logical services that the batch container provides. The following sections will provide an introduction into each set of core batch facilities.
**2.5.1 Core Spring Extensions
The Core Spring extensions provide the scaffolding for a batch container. This includes facilities for managing the batch architecture in terms of launching, suspending and stopping batch jobs. There is house keeping that goes on, especially in concurrent batch jobs, related to ensuring that batch jobs quiese properly. The lifecycle management provides services for the proper initialization and subsequent shutdown of batch resources and services. The batch architecture is flexible in terms of how batch jobs may be launched. For example, batch jobs can be started via JMX facilities, scripts from the command line that launch a Java VM. It can also support launching batch jobs through web services or http. There are no restrictions. Finally, there are standard batch error codes. These error codes can be exposed to external utilities, like Schedulers, to ensure that batch jobs expose the status of jobs to an operational environment. This is especially important in the batch context where the modus operandi is headless, meaning unattended operation.
**2.5.2 Core Batch Advice
Core Batch Advice is an inventory of the type of advice that batch architectures will inject during the runtime of a batch application. These are defined as a set of extensible interfaces, with a number of default implementations in the support layer that provide some of the most common types of advice. Partition Advice is helpful with large datasets that need to be "chunked" up and run concurrently for better through put. Resource Advice is helpful for registering interest in transactional information so that file locations can be kept in sync with information processed within a transaction. In addition, the resource is associated with the correct step context and its associated configuration properties. Skip advice is applied for records that the module is unable to process. Restart Advice is helpful for Restartable jobs where for advising the job on how to restart. There is considerable variability on how restart can occur. For example, a job may be marking records as "processed" and the restart advice will advise the process with query that restarts the job at the last successfully processed record. Finally, Statistics are vital in operational environments to report on records processed, records skipped and total number of records read. In addition, certain batch jobs lend themselves to custom reporting to expose additional business level information like the number of trades processed or cases opened, etc.
**2.5.3 Core Batch Configuration
Batch configuration is considerably different from online web applications or SOA based applications. The Core Batch Configuration provides a place for configuring runtime properties related to the batch application style. This includes the ability to add Commit Policy. In a batch style application it is often advantageous to keep the commit interval as high as possible when processing Logical Units of Work. Whereas in an online web application with declarative transaction the transaction scope would be at the entrance to a business service, a batch transaction scope may include many logical units of work before a transaction commit is executed. A Start Policy allows a configuration to tell the batch job whether it is Restartable, and if so, what type of restart to initiate. Some jobs are not restartable and care should be taken to ensure that information is not applied multiple times when the business rules do not allow for it. Exception policies deal with what to do when exceptions occur. This impacts logging policies and exception handling. The architecture defines a common set of exceptions that projects can apply handlers to like processing errors, validation errors, parsing errors, missing configuration parameters, etc.
**2.5.4 Container Repository
This is an internal package for storing the state of a batch job and any associated partition and step status.
**2.5.5 Core Batch Tasklet
The core batch module is where control is handed off to the application. There are a number of patterns that have been observed in processing batch data. Spring Core Batch Tasklet implements the most common patterns and provides and extension point for additional Tasklet processing implementations. The basic idea of module provides the facilities for reading and processing data. The simplest implementation of Tasklet, the ReadProcessTasklet, handles both the input and output of data within one class. An alternative implementation, the DataProviderProcessTasklet, provides functionality for 'split processing'. This type of processing is characterized by separating the reading and processing of batch data into two seperate classes: DataProvider and TaskletProcessor. The DataProvider class provides a solid means for reusablility and enforces good architecture practices. Because an object \*must\* be returned by the DataProider to continue processing, (Returning null indicates processing should end) a developer is forced to read in all relevant data, place it into domain or value objects, and return the object. The TaskletProcessor will then use this object within the business logic and final output.
2.6 Container's Use of batch infrastructure
**2.6.1 Infrastructure Provided I/O
The I/O core interfaces and implementations provide facilities for simplifying the extraction of data from I/O sources like files and database tables. The key concepts are FieldDescriptors and FieldSets along with appropriate CallBack Handlers. These are modeled after common spring operations and templates like JdbcTemplate. Through the use of LineMappers a developer needs only to describe a record format and write the appropriate callback method that maps the parsed record into an object of their choice. These can either be true POJO objects or Value Objects (structures) that are subsequently available for the module to processs. The interface for Field Descriptors also allows for a level of validation through the use of Spring's VALang or Apache's Common Validator.
**2.6.2 Core Batch Interceptors & Interceptor Services
* Batch Operations & Batch Template
Interceptors and the associated services are the key to how advise is applied in the batch architecture. The interceptors are Point Cuts in the batch lifecycle that allow the injection of advise. The shared lifecycle behavior abstracted through the BatchLifeCycleInterceptor defineds three methods; init, onError and finalize. All subclasses of LifeCycleInterceptor define default behavior for these three methods. The JobLifecycleInterceptor further exposes the methods beforeJob(), beforeStep(), afterJob(), and afterStep() allowing hooks into the lifecycle for specific advise. The Batch Architecture provides default implementations for all lifecycle point cuts, or interception points. The Tasklet Interceptor, in addition to the standard lifecycle methods, implements logic around beforeLuw(), afterLuw(), commitIntervalStarted() and commitIntervalCompleted(). Having well defined lifecycle interception points allows for the easy insertion of custom advice into the batch runtime environment.
2.7 Batch Esecution Container Configurations
In addition to core facilities for configuring or wiring together jobs and steps with required resources, policies, and interceptors, spring batch allows considerable flexibility in how scalability is achieved. More options for scalability will be available in the future. The important key for scalability in Java is the recognition that there is a limit to what one JVM may scale up to in terms of number of threads, managed resources, memory configuration, etc. The spring batch architecture allows for the configuration of simple batch jobs where one VM and one process is sufficient to do perform the work within a batch window all the way through many threads distributed within a cluster of JEE servers. The figure below illistrates the scalability spectrum.
This is not to be understood as the only way to scale batch jobs as there are many factors. For example, other federated java architectures hold potential like Teracotta or Gigaspaces although there is no current implementation for these distributed models in the current batch architecture.
!scalability-model.png!
[Figure 2.3.1] - Scalability Model
*2.7.1. Single VM Simple Batch Execution Container - One Job, One Step, One Partition
The simplest configuration is one job with with step and hence, one implied partition. Implied means that there is nothing for the developer to consider because the default number of partitions is one. There is typically one input source and one output source in this simple configuration. See the Simple Tasklet Job for an example of what this configuration looks like. A simple configuration still typically configures a datasource context, the batch configuration for describing the Job, Step, along with the associated configured policies, field descriptors, and line mappers. !SimpleTradeConfiguration.jpg!
[Figure 2.3.1] Simple Container Configuration
The details of this configuration will be covered thoroughly in subsequent sections of the document but for now it should be understood that Job, the Step, the input template, the file descriptor with its associated line mapper, and the output (e.g. the TradeWriter).
*2.3.2. Single VM Multi-threaded Batch Execution Container Configuration - One Job, One Step, Multiple Partitions
In a Single JVM using partitioning a multi-threaded execution is supported. \[This is still work in progress\]
*2.3.3. Batch Execution Container Hosted in J2EE Container - managed environment
The J2EE container model has fallen under fire over the past few years for many valid reasons. There are some things that the J2EE container do very well though that projects should consider when planning for scalability with batch architectures. Commercial and open source containers like WebSphere, BEA and JBOSS typically:
* manage datasources effectively along with attendent services like prepared statement caching.
* manage transactions effectively including many configurable properties for long lived transactions.
* manage thread pools more effectively.
* supply robust implementations of JTA, a requirement when batch jobs output to multiple XA resources like JMS and JDBC.
* manage distribution effectively including domains, clusters and cells
* provide robust JMX management for configuring, managing and administering distributed applications.
* workload management facilities (clusters) provided by J2EE vendors
Projects are encouraged to deploy batch applications with the simplest configuration possible, but when federated JVMs are a requirement to process volumes of data within a batch window, batch-in-container provides an effective way of distributing the processing. Spring Batch supports this through a simple change in configuration. \[Work in progress on the exact implementation - being released as part of M2\].

266
spring-batch-core/src/site/apt/samples.apt Normal file
View File

@@ -0,0 +1,266 @@
------
Sample Container Applications
------
Wayne Lund
------
May 2007
Overview of Batch Reference Applications
There is considerable variability in the types of input and output formats in batch jobs. There is also a number of options to consider in terms of how the types of strategies that will be used to handle skips, recovery, and statistics. However, when approaching a new batch job there are a few standard questions to answer to help determine how the job will be written and how to utilize the services offered by the spring batch framework. Consider the following:
* How do I configure this batch job? In the reference applications the pattern is to follow the convention of <nameOf>Job.xml. Each section with identify the XML definition.
* What is the input source? Each sample batch job will identify its input source.
* What is my output source? Each sample batch job will identify its output source.
* How are records read and validated from the input source? This refers to the input type and its format (e.g. flat file with fixed position, comma separated or XML, etc.)
* What is the policy of the job if a input record fails the validation step? The most important aspect is whether the record can be skipped so that processing can be continued.
* How will I process the data and write to the output source? How and what business logic is being applied to the processing of a record.
* How do I recover from an exception while operating on the output source? There are numerous recovery strategies that can be applied to handling errors on transactional targets. The reference applications will provide a feeling for some of the choices.
* Can I restart the job and if so which strategy will I use to restart the job? The reference applications will show some of the options available to jobs and what the decision criteria is for the respective choices.
[Samples] Reference Applications Table of Features
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| Job / Feature | delimited input | fixed-length input | xml input | db driving query input | db cursor input | delimited output | fixed-length output | db output | skip | restart | quartz scheduling |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| simpleTaskletJob | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| fixedLengthImport | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| multi-line order | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| quartzBatch | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| simple skip sample | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| Skip And Restart Sample | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| SQL Cursor Trade Job | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| Trade Job | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
| XML Job | | | | | | | | | | | |
*--------------*-----------------*--------------------*-----------*------------------------*-----------------*------------------*---------------------*-----------*------*---------*-------------------*
{Simple Tasklet Job}
The goal is to show the simplest use of the batch framework with a single job with a single step&nbsp;where the module processes&nbsp;one input source to one output source.
<<Description:>> This job is defined by simpleTaskletJob.xml file. Job itself is defined by element simpleTaskletJob. Each job consists of several steps, these steps are defined in steps property. In this example we have only one step. Each step defines module that is responsible for . In this case processing will be handled by SimpleTradeTasklet class. Each module must implement execute() method. All processing of business data should be handled by this method. In this example execute() method tries to read the data from defined input source using read() method and if the data exists, it is processed using process() method. If there is no data to read, method returns false to signal, that there is nothing for further processing.
<<Method read()>> gets the data from the input template defined and maps it to an object using mapper defined in XML definition. This sample uses FlatFileInputTemplate class as input template. This template reads the whole line from the file and pass it to tokenizer which knows the structure of the line. Location of the file is defined by fileLocatorStrategy property, structurte of the line is defined by fixedFileDescriptor. Result of parsing the line is stored in FieldSet, which is used by mapper to create value object. In our example we use DefaultLineMapper which creates an instance of Trade class.
<<Method process()>> is quite simple - just writes trade object using DbTradeWriter class. This class writes values obtained from an object to the database.
<<Specific information:>> This job has whole logic implemented in Tasklet. It is not using Data provider as well as Tasklet processor, which is typical way how to handle data.
<<XML definition:>> simpleTaskletJob.xml
\[Note: we need to document Spring IDE in setup and installation so we can use to describe the project. Also, if we could also publish we can provide links to the graphics from docs. This is a sample only\].
Visualization of the spring configuration through Spring-IDE exposes the structure of a job configuration. The following is the visualization of the Simple Tasklet Job configuration. See {{{http://springide.org/blog/}Spring IDE}}.
[images/simple-module-job-configuration.jpg]Spring IDE Graph of Simple Tasklet Job Configuration.
[Figure:]\ Simple Tasklet Job Configuration
For simplicity we are only displaying the job configuration itself and leaving out the details of the supporting container configuration. The source view of the configuration is as follows:
[]
--------------------------------------------------------------------------------------
<import resource="BatchArchConfig.xml" />
<bean id="simpleTaskletJob" parent="Job">
<property name="name" value="fixedLengthImportJob" />
<property name="steps">
<list>
<bean id="tradeStep" parent="Step">
<property name="name" value="ImportTradeDataStep" />
<property name="module">
<bean class="com.accenture.adsj.refapp.batch.module.SimpleTradeTasklet">
<property name="inputTemplate" ref="fileInputTemplate" />
<property name="tradeDbWriter" ref="tradeWriter" />
</bean>
</property>
<property name="commitFrequency" value="5" />
<property name="startPolicy">
<bean class="org.springframework.batch.container.conf.StartPolicy">
<property name="ignoreComplete" value="true" />
<property name="restartEnabled" value="true" />
<property name="startlimit" value="12" />
</bean>
</property>
<property name="exceptionPolicy">
<bean class="org.springframework.batch.container.conf.ExceptionPolicy">
<property name="totalExceptionLimit" value="20" />
<property name="transactionInvalidExceptionLimit" value="20" />
<property name="transactionValidExceptionLimit" value="5" />
</bean>
</property>
</bean>
</list>
</property>
</bean>
<bean id="tradeWriter" class="com.accenture.adsj.refapp.batch.dao.DbTradeWriter">
<property name="jdbcTemplate" ref="jdbcTemplate" />
<property name="incrementer">
<bean parent="incrementerParent">
<property name="incrementerName" value="TRADE_SEQ" />
</bean>
</property>
</bean>
<bean id="fileInputTemplate" class="org.springframework.batch.container.io.file.support.FlatFileInputTemplate">
<property name="name" value="FileInputSource" />
<property name="fileLocatorStrategy" ref="fileLocator" />
<property name="tokenizer">
<bean class="org.springframework.batch.container.io.file.support.FixedLineTokenizer">
<property name="fileDescriptor" ref="fixedFileDescriptor" />
</bean>
</property>
</bean>
<bean id="fixedFileDescriptor" class="org.springframework.batch.container.io.support.DefaultFileDescriptor">
<property name="recordDescriptors">
<bean class="org.springframework.batch.container.io.support.DefaultRecordDescriptor">
<property name="fieldDescriptors">
<list>
<bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
<property name="name" value="ISIN" />
<property name="length" value="12" />
</bean>
<bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
<property name="name" value="Quantity" />
<property name="length" value="3" />
</bean>
<bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
<property name="name" value="Price" />
<property name="length" value="5" />
</bean>
<bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
<property name="name" value="Customer" />
<property name="length" value="9" />
</bean>
</list>
</property>
</bean>
</property>
</bean>
<bean id="tradeLineMapper" class="com.accenture.adsj.refapp.batch.mapping.TradeRowMapper" />
<bean class="com.accenture.adsj.refapp.batch.advice.LogAdvice" id="logAdvice" />
<aop:config>
<aop:aspect id="logging" ref="logAdvice">
<aop:around pointcut-ref="pointcut" method="doBasicLogging" />
<aop:pointcut id="pointcut" expression="execution(* org.springframework.batch.container.dao.*.*(..))" />
</aop:aspect>
</aop:config>
</beans>
-----------------------------------------------------------------------------------------------------------------------
You should take the time to make sure you understand the relationship of the xml configuration with the visualization as provided by Spring IDE. \[Note: this will be updated when we use the namespace handler\].
<<Input source:>> file with fixed row structure
In this example we are using a simple fixed length record structure that can be found in the project at <REFAPP_INSTALL_HOME>/testBatchRoot/job_data/simpleTaskletJob/input/20070122.teststream.ImportTradeDataStep.txt. There's generally a considerable amount of thought that goes into architecting the folder structures for batch file management. See [provide a link to DefaultFileStrategy]. The only point to note here is the ImportTradeDataStep matches the name of the step in the configuration and the fixed length records look like:
[]
------------------------------------------------------------------------------------
20070122.teststream.ImportTradeDataStep.txt
UK21341EAH4597898.34customer1
UK21341EAH4611218.12customer2
UK21341EAH4724512.78customer2
UK21341EAH48108109.25customer3
UK21341EAH49854123.39customer4
------------------------------------------------------------------------------------
Looking back to the configuration file you will see where this is documented in the propery of the DefaultRecordDescriptor. You can see the following:
*--------------*-----------------*
|| FieldName | Length ||
*--------------*-----------------*
| ISIN | 12 |
*--------------*-----------------*
| Quantity | 3 |
*--------------*-----------------*
| Price | 5 |
*--------------*-----------------*
| Customer | 9 |
*--------------*-----------------*
<<Output target:>> database
<<Data Provider:>> data provider is not used, all functionality is implemented directly in Tasklet.
<<Tasklet processor:>> module processor is not used, all functionality is implemented directly in Tasklet.
Fixed Length Import Job
The goal is to demonstrate a typical scenarion of importing data from a fixed-length file to database
<<Description:>> This job shows a more typical scenario, when reading input data and processing the data is cleanly separated. The data provider is responsible for reading input and mapping each record to a domain object, which is then passed to the module processor. The module processor handles the processing of the domain objects, in this case it only writes them to database.
<<XML definition:>> fixedLengthImportJob.xml
<<Input source:>> file with fixed row structure
<<Output target:>> database
<<Data Provider:>> DefaultFlatFileDataProvider which uses the injected FlatFileInputTemplate to read input and the DefaultLineMapper to map each line to an object according to the file descriptor.
<<Tasklet processor:>> module processor does not do any special processing, it just writes the data to database using a DAO object (called OutputSource in this case, because it is specialized for writing to database, it has no methods for reading data).
Multiline Order Job
The goal is to demostrate how to handle a more complex file input format, where a record meant for processing inludes nested records and spans multiple lines
<<XML definition:>> multilineOrderJob.xml
<<Input source:>> file with multiline records
<<Output target:>> file with multiline records
<<Data Provider:>> OrderDataProvider is an example of a non-default programmatic data provider. It reads input until it detects that the multiline record has finished and encapsulates the record in a single domain object.
<<Tasklet processor:>> module processor passes the object to a an injected 'report service' which in this case writes the output to a file do demonstrate how to use the FlatFileOutputTemplate for writing multiline output according to a file descriptor.
Quartz Batch
The goal is to demonstrate how to schedule job execution using Quartz scheduler
<<XML definition:>> quartzBatch.xml
<<Description:>> First, declares launcher beans. Each launcher bean is able to launch a job using injected arguments. Second, triggers are declared saying when the launchers should be run. Last, there is the scheduler bean, where the triggers are registered.
Simple Skip Sample
Skip And Restart Sample
SQL Cursor Trade Job
Trade Job
The goal is to show a reasonably complex scenario, that would resemble the real-life usage of the framework.
<<Description:>> This job has 3 steps. First, data about trades is imported from a file to database. Second, the data about trades is read from the database and credit on customer accounts is decreased appropriately. Last, a report about customers is exported to a file.
<<XML definition:>> tradeJob.xml - the job definition, tradeJobIo.xml - input and output configuration, tradeJobAop.xml - optional AOP logging
<<Description:>> This job has 3 steps. First, data about trades is imported from a file to database. Second, the data about trades is read from the database and credit on customer accounts is decreased appropriately. Last, a report about customers is exported to a file.
XML Job