# Sample Jobs

## Overview of Batch Samples

There is considerable variability in the types of input and output formats in batch jobs. There is also a number of options to consider in terms of how the types of strategies that will be used to handle skips, recovery, and statistics. However, when approaching a new batch job there are a few standard questions to answer to help determine how the job will be written and how to utilize the services offered by the spring batch framework. Consider the following:

- How do I configure this batch job? In the reference applications the pattern is to follow the convention of nameOf Job.xml. Each sample will identify the XML definition used to configure the job. Job configurations that leverage a common execution environment have many common items in their respective configurations.

- What is the input source? Each sample batch job will identify its input source.

- What is my output source? Each sample batch job will identify its output source.

- How are records read and validated from the input source? This refers to the input type and its format (e.g. flat file with fixed position, comma separated or XML, etc.)

- What is the policy of the job if a input record fails the validation step? The most important aspect is whether the record can be skipped so that processing can be continued.

- How will I process the data and write to the output source? How and what business logic is being applied to the processing of a record.

- How do I recover from an exception while operating on the output source? There are numerous recovery strategies that can be applied to handling errors on transactional targets. The reference applications will provide a feeling for some of the choices.

- Can I restart the job and if so which strategy will I use to restart the job? The reference applications will show some of the options available to jobs and what the decision criteria is for the respective choices.

  Samples

    Reference Applications Table of Features

    Job / Feature | delimited input | fixed-length input | xml input | db driving query input | db cursor input | delimited output | fixed-length output | db output | skip | restart | quartz scheduling
    simpleTaskletJob |  |  |  |  |  |  |  |  |  |  |
    fixedLengthImport |  |  |  |  |  |  |  |  |  |  |
    multi-line order |  |  |  |  |  |  |  |  |  |  |
    quartzBatch |  |  |  |  |  |  |  |  |  |  |
    simple skip sample |  |  |  |  |  |  |  |  |  |  |
    Skip And Restart Sample |  |  |  |  |  |  |  |  |  |  |
    SQL Cursor Trade Job |  |  |  |  |  |  |  |  |  |  |
    Trade Job |  |  |  |  |  |  |  |  |  |  |
    XML Job |  |  |  |  |  |  |  |  |  |  |

## Common Sample Test Structures

The sample applications, although executing within unit test frameworks are actually integration tests that configure the simple batch execution environment, the job with its respective steps and tasklets, and wire in the infrastructure services used by the job. On a quick inspection the sample jobs, especially if new to Spring, appear to use a lot of auto magic. However, this is simply the power of Spring to help configure applications and allow developers to focus on the application and not infrastructure. The common test structures are the following;

- All test cases subclass AbstractLifecycleSpringContextTests: This class extends a convenience class provided by Spring, AbstractDependencyInjectionSpringContextTests, that is populated by Dependency Injection. This effectively serves as a substitute Batch Launcher that conveniently describes the location of the application context and loads the beans enlisted by the job into the application context. In addition, once the batch execution environment has completed the wiring of the application context, the Batch Bootstrap process launches the BatchLauncher via its start() method.

- All test cases have a common configuration structure: If using Spring IDE you will see the following configuration hierarchy:

  - The Job Specific Configuration

  - The Simple Batch Execution Environment Definition

  - The Data Source Context

## Simple Tasklet Job

The goal is to show the simplest use of the batch framework with a single job with a single step&nbsp;where the tasklet processes&nbsp;one input source to one output source.

Description: This job is defined by simpleTaskletJob.xml file. Job itself is defined by element simpleTaskletJob. Each job consists of several steps, these steps are defined in steps property. In this example we have only one step. The step defines a tasklet that is responsible for processing trades. In this case processing will be handled by SimpleTradeTasklet class. Each tasklet must implement execute() method. All processing of business data should be handled by this method. In this example the tradeFieldSetMapper obtains the data from the input source and maps the line to the Trade object.

                    trade = (Trade) tradeFieldSetMapper.mapLine(inputSource.readFieldSet());

If data exists and an object is returned it is simply passed to the output source. If there is no data to read an ExitStatus with the status of FINISHED is returned from the Tasklet.

Method read() gets the data from the input template defined and maps it to an object using mapper defined in XML definition. This sample uses FlatFileInputTemplate class as input template. This template reads the whole line from the file and pass it to tokenizer which knows the structure of the line. Location of the file is defined by fileLocatorStrategy property, structurte of the line is defined by fixedFileDescriptor. Result of parsing the line is stored in FieldSet, which is used by mapper to create value object. In our example we use DefaultLineMapper which creates an instance of Trade class.

Method process() is quite simple - just writes trade object using DbTradeWriter class. This class writes values obtained from an object to the database.

Specific information: This job has whole logic implemented in Tasklet. It is not using Data provider as well as Tasklet processor, which is typical way how to handle data.

XML definition: simpleTaskletJob.xml

[Note: we need to document Spring IDE in setup and installation so we can use to describe the project. Also, if we could also publish we can provide links to the graphics from docs. This is a sample only].

Visualization of the spring configuration through Spring-IDE exposes the structure of a job configuration. The following is the visualization of the Simple Tasklet Job configuration. See Spring IDE .

## Spring IDE Graph of Simple Tasklet Job Configuration.

Figure:

  Simple Tasklet Job Configuration

  For simplicity we are only displaying the job configuration itself and leaving out the details of the supporting batch execution environment configuration. The source view of the configuration is as follows:

      <import resource="BatchArchConfig.xml" />
            <bean id="simpleTaskletJob" parent="Job">
                    <property name="name" value="fixedLengthImportJob" />
                    <property name="steps">
                            <list>
                                    <bean id="tradeStep" parent="Step">
                                            <property name="name" value="ImportTradeDataStep" />
                                            <property name="module">
                                                    <bean class="com.accenture.adsj.refapp.batch.module.SimpleTradeTasklet">
                                                            <property name="inputTemplate" ref="fileInputTemplate" />
                                                            <property name="tradeDbWriter" ref="tradeWriter" />
                                                    </bean>
                                            </property>
                                            <property name="commitFrequency" value="5" />
                                            <property name="startPolicy">
                                                    <bean class="org.springframework.batch.container.conf.StartPolicy">
                                                            <property name="ignoreComplete" value="true" />
                                                            <property name="restartEnabled" value="true" />
                                                            <property name="startlimit" value="12" />
                                                    </bean>
                                            </property>
                                            <property name="exceptionPolicy">
                                                    <bean class="org.springframework.batch.container.conf.ExceptionPolicy">
                                                            <property name="totalExceptionLimit" value="20" />
                                                            <property name="transactionInvalidExceptionLimit" value="20" />
                                                            <property name="transactionValidExceptionLimit" value="5" />
                                                    </bean>
                                            </property>
                                    </bean>
                            </list>
                    </property>
            </bean>
            <bean id="tradeWriter" class="com.accenture.adsj.refapp.batch.dao.DbTradeWriter">
                    <property name="jdbcTemplate" ref="jdbcTemplate" />
                    <property name="incrementer">
                            <bean parent="incrementerParent">
                                    <property name="incrementerName" value="TRADE_SEQ" />
                            </bean>
                    </property>
            </bean>
            <bean id="fileInputTemplate" class="org.springframework.batch.container.io.file.support.FlatFileInputTemplate">
                    <property name="name" value="FileInputSource" />
                    <property name="fileLocatorStrategy" ref="fileLocator" />
                    <property name="tokenizer">
                            <bean class="org.springframework.batch.container.io.file.support.FixedLineTokenizer">
                                    <property name="fileDescriptor" ref="fixedFileDescriptor" />
                            </bean>
                    </property>
            </bean>

    <bean id="fixedFileDescriptor" class="org.springframework.batch.container.io.support.DefaultFileDescriptor">
            <property name="recordDescriptors">
                    <bean class="org.springframework.batch.container.io.support.DefaultRecordDescriptor">
                            <property name="fieldDescriptors">
                                    <list>
                                            <bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
                                                    <property name="name" value="ISIN" />
                                                    <property name="length" value="12" />
                                            </bean>
                                            <bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
                                                    <property name="name" value="Quantity" />
                                                    <property name="length" value="3" />
                                            </bean>
                                            <bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
                                                    <property name="name" value="Price" />
                                                    <property name="length" value="5" />
                                            </bean>
                                            <bean class="org.springframework.batch.container.io.support.DefaultFieldDescriptor">
                                                    <property name="name" value="Customer" />
                                                    <property name="length" value="9" />
                                            </bean>
                                    </list>
                            </property>
                    </bean>
            </property>
    </bean>
    <bean id="tradeLineMapper" class="com.accenture.adsj.refapp.batch.mapping.TradeRowMapper" />
    <bean class="com.accenture.adsj.refapp.batch.advice.LogAdvice" id="logAdvice" />
    <aop:config>
            <aop:aspect id="logging" ref="logAdvice">
                    <aop:around pointcut-ref="pointcut" method="doBasicLogging" />
                    <aop:pointcut id="pointcut" expression="execution(* org.springframework.batch.container.dao.*.*(..))" />
            </aop:aspect>
    </aop:config>
     </beans>

You should take the time to make sure you understand the relationship of the xml configuration with the visualization as provided by Spring IDE. [Note: this will be updated when we use the namespace handler].

Input source: file with fixed row structure

In this example we are using a simple fixed length record structure that can be found in the project at REFAPP_INSTALL_HOME/testBatchRoot/job_data/simpleTaskletJob/input/20070122.teststream.ImportTradeDataStep.txt. There's generally a considerable amount of thought that goes into architecting the folder structures for batch file management. See [provide a link to DefaultFileStrategy]. The only point to note here is the ImportTradeDataStep matches the name of the step in the configuration and the fixed length records look like:

      20070122.teststream.ImportTradeDataStep.txt

      UK21341EAH4597898.34customer1
      UK21341EAH4611218.12customer2
      UK21341EAH4724512.78customer2
      UK21341EAH48108109.25customer3
      UK21341EAH49854123.39customer4

Looking back to the configuration file you will see where this is documented in the propery of the DefaultRecordDescriptor. You can see the following:

FieldName | Length
ISIN | 12
Quantity | 3
Price | 5
Customer | 9

Output target: database

Data Provider: data provider is not used, all functionality is implemented directly in Tasklet.

Tasklet processor: module processor is not used, all functionality is implemented directly in Tasklet.

## Fixed Length Import Job

The goal is to demonstrate a typical scenarion of importing data from a fixed-length file to database

Description: This job shows a more typical scenario, when reading input data and processing the data is cleanly separated. The data provider is responsible for reading input and mapping each record to a domain object, which is then passed to the module processor. The module processor handles the processing of the domain objects, in this case it only writes them to database.

XML definition: fixedLengthImportJob.xml

Input source: file with fixed row structure

Output target: database

Data Provider: DefaultFlatFileDataProvider which uses the injected FlatFileInputTemplate to read input and the DefaultLineMapper to map each line to an object according to the file descriptor.

Tasklet processor: module processor does not do any special processing, it just writes the data to database using a DAO object (called OutputSource in this case, because it is specialized for writing to database, it has no methods for reading data).

## Multiline Order Job

The goal is to demostrate how to handle a more complex file input format, where a record meant for processing inludes nested records and spans multiple lines

XML definition: multilineOrderJob.xml

Input source: file with multiline records

Output target: file with multiline records

Data Provider: OrderDataProvider is an example of a non-default programmatic data provider. It reads input until it detects that the multiline record has finished and encapsulates the record in a single domain object.

Tasklet processor: module processor passes the object to a an injected 'report service' which in this case writes the output to a file do demonstrate how to use the FlatFileOutputTemplate for writing multiline output according to a file descriptor.

## Quartz Batch

The goal is to demonstrate how to schedule job execution using Quartz scheduler

XML definition: quartzBatch.xml

Description: First, declares launcher beans. Each launcher bean is able to launch a job using injected arguments. Second, triggers are declared saying when the launchers should be run. Last, there is the scheduler bean, where the triggers are registered.

## Simple Skip Sample

Document how skip works.

## Skip And Restart Sample

Document how Skip and Restart Sample Works

## SQL Cursor Trade Job

Document how SQL Cursor Trade Job works

## Trade Job

The goal is to show a reasonably complex scenario, that would resemble the real-life usage of the framework.

Description: This job has 3 steps. First, data about trades is imported from a file to database. Second, the data about trades is read from the database and credit on customer accounts is decreased appropriately. Last, a report about customers is exported to a file.

XML definition: tradeJob.xml - the job definition, tradeJobIo.xml - input and output configuration, tradeJobAop.xml - optional AOP logging

Description: This job has 3 steps. First, data about trades is imported from a file to database. Second, the data about trades is read from the database and credit on customer accounts is decreased appropriately. Last, a report about customers is exported to a file.

## XML Job

Document how the sample XML job works

## Football Job

The final Job is an Football statistics loading job. We’ll give it the id of “footballjob” in our configuration file. Before diving into the batch job, we’ll examine the two input files that need to be loaded. First is ‘player.csv’, which can be found in the samples project under src/main/resources/data/footballjob/input/. Each line within this file represents a player, with a unique id, the player’s name, position, etc:

            AbduKa00,Abdul-Jabbar,Karim,rb,1974,1996
            AbduRa00,Abdullah,Rabih,rb,1975,1999
            AberWa00,Abercrombie,Walter,rb,1959,1982
            AbraDa00,Abramowicz,Danny,wr,1945,1967
            AdamBo00,Adams,Bob,te,1946,1969
            AdamCh00,Adams,Charlie,wr,1979,2003

One of the first noticeable characteristics of the file is that each data element is separated by a comma, a format most are familiar with known as ‘CSV’. Other separators such as pipes or semicolons could just as easily be used to delineate between unique elements. In general, it falls into one of two types of flat file formats: delimited or fixed length. Because both input files in this example are comma delimited, we’ll skip over fixed length for now, other than to say that the only difference between the two types is that fixed length formatting determines the separation between elements by assigning each element a ‘fixed length’ in which to reside, rather than using a character that hopefully doesn’t exist in the data itself to separate individual elements.

The second file, ‘games.csv’ is formatted the same as the previous example, and resides in the same directory:

            AbduKa00,1996,mia,10,nwe,0,0,0,0,0,29,104,,16,2
            AbduKa00,1996,mia,11,clt,0,0,0,0,0,18,70,,11,2
            AbduKa00,1996,mia,12,oti,0,0,0,0,0,18,59,,0,0
            AbduKa00,1996,mia,13,pit,0,0,0,0,0,16,57,,0,0
            AbduKa00,1996,mia,14,rai,0,0,0,0,0,18,39,,7,0
            AbduKa00,1996,mia,15,nyg,0,0,0,0,0,17,96,,14,0

Each line in the file represents an individual player’s performance in a particular game, containing such statistics as passing yards, receptions, rushes, and total touchdowns.

Our example batch job is going to load both files into a database, and then combine each to summarize how each player performed for a particular year. Although this example is fairly trivial, it shows multiple types of input, and the general style is a common batch scenario. That is, summarizing a very large dataset so that it can be more easily manipulated or viewed by an online web-based application. In an enterprise solution the third step, the reporting step, could be implemented through the use of Eclipse BIRT or one of the many Java Reporting Engines. Given this description, we can then easily divide our batch job up into 3 ‘steps’: one to load the player data, one to load the game data, and one to produce a summary report:

NOTE:One of the nice features of Spring is a project called Spring IDE. When you download the project you can install Spring IDE and add the Spring configurations to the IDE project. This is not a tutorial on Spring IDE but the visual view into Spring beans is helpful in understanding the structure of a Job Configuration. Spring IDE produces the following diagram:

Figure 3 - Spring Bean Job Configuration

This corresponds exactly with the footballJob.xml job configuration file which can be found in the jobs folder under src/main/resources. When you drill down into the footballjob you will see that the configuration has a list of steps:

            <property name="steps">
                <list>
                    <bean id="playerload"> ... </bean>
                    <bean id="gameLoad"> ... </bean
                    <bean id="playerSummarization"> ... </bean>
                </list>
            </property>

The step is run until there is no more input to process, which in this case would mean that each file has been completely processed. To describe it in a more narrative form: The first step, playerLoad, begins executing by grabbing one line of input from the file, and parsing it into a domain object. That domain object is then passed to a dao, which writes it out to the PLAYERS table. This action is repeated until there are no more lines in the file, causing the playerLoad step to finish. Next, the gameLoad step does the same for the games input file, inserting into the GAMES table. Once finished, the playerSummarization step can begin. Unlike the first two steps, playerSummarization’s input comes from the database, using a Sql statement to combine the GAMES and PLAYERS table. Each returned row is packaged into a domain object and written out to the PLAYER_SUMMARY table.

Now that we’ve discussed the entire flow of the batch job, we can dive deeper into the first step: playerLoad:

            <bean id="playerload"        class="org.springframework.batch...SimpleStepConfiguration">
                    <property name="commitInterval" value="100" />
                    <property name="tasklet">
                            <bean class="org.springframework...RestartableItemProviderTasklet">
                                    <property name="itemProvider">...</property>
                                    <property name="itemProcessor">...</property>
                            </bean>
                    </property>
            </bean>

The root bean in this case is a StepConfiguration, which can be considered a ‘blueprint’ of sorts that tells the execution environment basic details about how the batch job should be executed. It contains two properties: (others have been removed for greater clarity) commitInterval and tasklet. The Tasklet is the main abstraction representing the developer’s business logic within the batch job. After performing all necessary startup, the framework will periodically delegate to the Tasklet. In this way, the developer can remain solely concerned with their business logic. In this case, the Tasklet has been split into two classes:

- Item Provider – the item provider is the source of the information pipe. At the most basic level input is read in from an input source, parsed into a domain object and returned. In this way, the good batch architecture practice of ensuring all data has been read before beginning processing can be enforced, along with providing a possible avenue for reuse.

- Item Processorr – this is the business logic. At a high level, the ItemProcessor takes the item returned from the ItemProvider and ‘processes’ it. In our case it’s a data access object that is simply responsible for inserting a record into the PLAYERS table. As you can see the developer does very little.

Clearly, the developer does very litt. Simply provide a job configuration with a configured number of steps, an Item Provider associated to some type of input source, and Item Processor associated to some type of output source and a little mapping of data from flat records to objects and the pipe is ready wired for processing.

The other property to the StepConfiguration, commitInterval, gives the framework vital information about how to control transactions during the batch run. Due to the large amount of data involved in batch processing, it is often advantageous to ‘batch’ together multiple Logical Units of Work into one transaction, since starting and committing a transaction is extremely expensive. For example, in the playerLoad step, the framework calls the execute() method on the Tasklet, which then calls next() on the ItemProvider. The ItemProvider reads one record from the file, then returns a domain object representation which is passed to the processor. The processor then writes the one record to the database. It can then be said that one iteration = one call to Tasklet.execute() = one line of the file. Therefore, setting your commitInterval to 5 would result in the framework committing a transaction after 5 lines have been read from the file, with 5 resultant entries in the PLAYERS table.

Following the general flow of the batch job, the next step is to describe how each line of the file will be parsed from its string representation into a domain object. The first thing the provider will need is an InputSource, which is provided as part of the Spring Batch infrastructure. Because the input is flat-file based, a FlatFileInputSource is used:

    <bean id="playerFileInputSource"
    class="org.springframework.batch.io.file.support.DefaultFlatFileInputSource">
      <property name="resource">
        <bean class="org.springframework.core.io.ClassPathResource">
            <constructor-arg value="data/footballjob/input/player.csv" />
        </bean>
      </property>
      <property name="tokenizer">
        <bean class = "org.springframework.batch.io.file.support.transform.DelimitedLineTokenizer">
          <property name="names"
              value="ID,lastName,firstName,position,birthYear,debutYear" />
        </bean>
      </property>
    </bean>

There are two required dependencies of the input source; the first is a resource to read in, which is the file to process. The second dependency is a LineTokenizer. The interface for a LineTokenizer is very simple, given a string; it will return a FieldSet that wraps the results from splitting the provided string. A FieldSet is Spring Batch’s abstraction for flat file data. It allows developers to work with file input in much the same way as they would work with database input. All the developers need to provide is a FieldSetMapper (similar to a Spring RowMapper) that will map the provided FieldSet into an Object. Simply by providing the names of each token to the LineTokenizer, the ItemProvider can pass the FieldSet into our PlayerMapper, which implements the FieldSetMapper interface. There is a single method, mapLine(), which maps FieldSets the same way that developers are comfortable mapping ResultSets into Java Objects, either by index or fieldname. This behavior is by intention and design similar to the RowMapper passed into a JdbcTemplate. You can see this below:

    public class PlayerMapper implements FieldSetMapper {

            public Object mapLine(FieldSet fs) {

                    if(fs == null){
                            return null;
                    }

                    Player player = new Player();
                    player.setID(fs.readString("ID"));
                    player.setLastName(fs.readString("lastName"));
                    player.setFirstName(fs.readString("firstName"));
                    player.setPosition(fs.readString("position"));
                    player.setDebutYear(fs.readInt("debutYear"));
                    player.setBirthYear(fs.readInt("birthYear"));

                    return player;
            }
    }

The flow of the ItemProvider, in this case, starts with a call to readFieldSet on the InputSource. The next line in the file is read in as a String and passed into the provided LineTokenizer. The LineTokenizer splits the line at every comma, and creates a FieldSet using the created String array and the array of names passed in. (Note: it is only necessary to provide the names if you wish to access the field by name, rather than by index).

Once the domain representation of the data has been returned by the provider, (i.e. an Player object) it is passed to the ItemProcessor, which is essentially a Dao that uses a Spring JdbcTemplate to insert a new row in the PLAYERS table.

The next step, gameLoad, works almost exactly the same as the playerLoad step, except the games file is used.

The final step, playerSummarization, is much like the previous two steps, it is split into a provider that reads from an InputSource and returns a domain object to the processor. However, in this case, the input source is the database, not a file:

    <bean id="playerSummarizationSource"
            class="org.springframework.batch.io.sql.SqlCursorInputSource">
            <property name="dataSource" ref="dataSource" />
            <property name="mapper">
                    <bean class="sample.mapping.PlayerSummaryMapper" />
            </property>
            <property name="sql">
                    <value>
                            SELECT games.player_id, games.year, SUM(COMPLETES),
                            SUM(ATTEMPTS), SUM(PASSING_YARDS), SUM(PASSING_TD),
                            SUM(INTERCEPTIONS), SUM(RUSHES), SUM(RUSH_YARDS),
                            SUM(RECEPTIONS), SUM(RECEPTIONS_YARDS), SUM(TOTAL_TD)
                            from games, players where players.player_id =
                            games.player_id group by games.player_id, games.year
                    </value>
            </property>
    </bean>

The SqlCursorInputSource has three dependences:

- A DataSource

- The SqlRowMapper to use for each row.

- The Sql statement used to create the Cursor.

When the step is first started, a query will be run against the database to open a cursor, and each call to inputSource.read() will move the ‘cursor’ to the next row, using the provided RowMapper to return the correct object. As with the previous two steps, each record returned by the provider will be written out to the database in the PLAYER_SUMMARY table. Finally to run this sample application you can execute the JUnit test “FootballJobFunctionalTests”, and you’ll see an output showing each of the records as they are processed. Please keep in mind that AoP is used to wrap the ItemProcessors and output each record as it is processed to the logger, which will greatly impact performance.