Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

BATCH-674: Updated readers and writers, testing, and schema appendix.

Browse Source
This commit is contained in:
lucasward
2009-01-27 05:17:49 +00:00
parent 2898861782
commit 6858782f02
7 changed files with 1233 additions and 1148 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
docs/src/site/docbook/reference/common-patterns.xml
View File

@@ -4,27 +4,23 @@
<chapter id="patterns">
<title>Common Batch Patterns</title>
<section>
<title>Introduction</title>
<para>Some batch jobs can be assembled purely from off-the-shelf components
in Spring Batch, mostly the <classname>ItemReader</classname> and
<classname>ItemWriter</classname> implementations. Where this is not
possible (the majority of cases) the main API entry points for application
developers are the <classname>Tasklet</classname>,
<classname>ItemReader</classname>, <classname>ItemWriter</classname> and the
various listener interfaces. Most simple batch jobs will be able to use
off-the-shelf input from a Spring Batch <classname>ItemReader</classname>,
but it is very often the case that there are custom concerns in the
processing and writing, which normally leads developers to implement an
<classname>ItemWriter</classname>, or
<classname>ItemTransformer</classname>.</para>
<para>Some batch jobs can be assembled purely from off-the-shelf
components in Spring Batch, mostly the <classname>ItemReader</classname>
and <classname>ItemWriter</classname> implementations. Where this is not
possible (the majority of cases) the main API entry points for application
developers are the <classname>Tasklet</classname>,
<classname>ItemReader</classname>, <classname>ItemWriter</classname> and
the various listener interfaces. Most simple batch jobs will be able to
use off-the-shelf input from a Spring Batch
<classname>ItemReader</classname>, but it is very often the case that
there are custom concerns in the processing and writing, which normally
leads developers to implement an <classname>ItemWriter</classname>, or
<classname>ItemTransformer</classname>.</para>
<para>Here we provide a few examples of common patterns in custom business
logic, mainly using the listener interfaces . It should be noted that an
<classname>ItemReader</classname> or <classname>ItemWriter</classname> can
implement the listener interfaces as well if appropriate.</para>
</section>
<para>Here we provide a few examples of common patterns in custom business
logic, mainly using the listener interfaces . It should be noted that an
<classname>ItemReader</classname> or <classname>ItemWriter</classname> can
implement the listener interfaces as well if appropriate.</para>
<section>
<title>Logging Item Processing and Failures</title>
@@ -229,4 +225,57 @@
maintains its own state in a transactional resource like a database, there
is no need to maintain state within the writer itself.</para>
</section>
<section>
<title>Driving Query Based ItemReaders</title>
<para>In the chapter on readers and writers, database input using paging
was discussed. Many database vendors, such as DB2, have extremely
pessimistic locking strategies that can cause issues if the table being
read also needs to be used by other portions of the online application.
Furthermore, opening cursors over extremely large datasets can cause
issues on certain vendors. Therefore, many projects prefer to use a
'Driving Query' approach to reading in data. This approach works by
iterating over keys, rather than the entire object that needs to be
returned, as the following example illustrates:</para>
<mediaobject>
<imageobject role="html">
<imagedata align="center" fileref="images/drivingQueryExample.png"
width="50%" />
</imageobject>
<imageobject role="fo">
<imagedata align="center"
fileref="src/site/docbook/reference/images/drivingQueryExample.png"
width="66%" />
</imageobject>
</mediaobject>
<para>As you can see, this example uses the same 'FOO' table as was used
in the cursor based example. However, rather than selecting the entire
row, only the ID's were selected in the SQL statement. So, rather than a
FOO object being returned from <classname>read</classname>, an Integer
will be returned. This number can then be used to query for the 'details',
which is a complete Foo object:</para>
<mediaobject>
<imageobject role="html">
<imagedata align="center" fileref="images/drivingQueryJob.png"
width="66%" />
</imageobject>
<imageobject role="fo">
<imagedata align="center"
fileref="src/site/docbook/reference/images/drivingQueryJob.png"
width="66%" />
</imageobject>
</mediaobject>
<para>As you can see, an existing DAO can be used to obtain a full 'Foo'
object using the key obtained from the driving query. In Spring Batch,
driving query style input is implemented with a
<classname>DrivingQueryItemReader</classname>, which has only one
dependency: a <classname>KeyCollector</classname></para>
</section>
</chapter>

BIN
docs/src/site/docbook/reference/images/oxm-fragments.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 28 KiB

BIN
docs/src/site/docbook/reference/images/xmlinput.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 52 KiB

1732
docs/src/site/docbook/reference/readersAndWriters.xml
View File

File diff suppressed because it is too large Load Diff

244
docs/src/site/docbook/reference/schema-appendix.xml
View File

@@ -8,10 +8,13 @@
<title>Overview</title>
<para>The Spring Batch Meta-Data tables very closely match the Domain
objects that represent them in Java. For example, JobInstance,
JobExecution, JobParameters, StepExecution, and ExecutionContext map to
BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, BATCH_JOB_PARAMS,
BATCH_STEP_EXECUTION, BATCH_STEP_EXECUTION_CONTEXT, respectively. The
objects that represent them in Java. For example,
<classname>JobInstance</classname>, <classname>JobExecution</classname>,
<classname>JobParameters</classname>, and
<classname>StepExecution</classname> map to BATCH_JOB_INSTANCE,
BATCH_JOB_EXECUTION, BATCH_JOB_PARAMS, and BATCH_STEP_EXECUTION,
respectively. <classname>ExecutionContext</classname> maps to both
BATCH_JOB_EXECUTION_CONTEXT and BATCH_STEP_EXECUTION_CONTEXT. The
<classname>JobRepository</classname> is responsible for saving and storing
each Java object into it's correct table. The following appendix describes
the meta-data tables in detail, along with many of the design decisions
@@ -19,7 +22,7 @@
statements below, it is important to realize that the data types used are
as generic as possible. Spring Batch provides many schemas as examples,
which all have varying data types due to variations in individual database
vendors' handling of data types. Below is an ERD model of all 5 tables and
vendors' handling of data types. Below is an ERD model of all 6 tables and
their relationships to one another:</para>
<mediaobject>
@@ -122,7 +125,9 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
<para>JOB_KEY: A serialization of the
<classname>JobParameters</classname> that uniquely identifies separate
instances of the same job from one another.
(<classname>JobInstances</classname> with the same job name</para>
(<classname>JobInstances</classname> with the same job name must have
different <classname>JobParameters</classname>, and thus, different
JOB_KEY values).</para>
</listitem>
</itemizedlist>
</section>
@@ -131,11 +136,12 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
<title>BATCH_JOB_PARAMS</title>
<para>The BATCH_JOB_PARAMS table holds all information relevant to the
JobParameters object. It contains 0 or more key/value pairs that together
uniquely identify a <classname>JobInstance</classname> and serve as a
record of the parameters a job was run with. It should be noted that the
table has been denormalized. Rather than creating a separate table for
each type, there is one table with a column indicating the type:</para>
<classname>JobParameters</classname> object. It contains 0 or more
key/value pairs that together uniquely identify a
<classname>JobInstance</classname> and serve as a record of the parameters
a job was run with. It should be noted that the table has been
denormalized. Rather than creating a separate table for each type, there
is one table with a column indicating the type:</para>
<programlisting>CREATE TABLE BATCH_JOB_PARAMS (
JOB_INSTANCE_ID BIGINT NOT NULL ,
@@ -204,12 +210,13 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
JOB_EXECUTION_ID BIGINT PRIMARY KEY ,
VERSION BIGINT,
JOB_INSTANCE_ID BIGINT NOT NULL,
CREATE_TIME TIMESTAMP NOT NULL,
START_TIME TIMESTAMP DEFAULT NULL,
END_TIME TIMESTAMP DEFAULT NULL,
STATUS VARCHAR(10),
CONTINUABLE CHAR(1),
EXIT_CODE VARCHAR(20),
EXIT_MESSAGE VARCHAR(2500),
LAST_UPDATED TIMESTAMP,
constraint JOB_INSTANCE_EXECUTION_FK foreign key (JOB_INSTANCE_ID)
references BATCH_JOB_INSTANCE(JOB_INSTANCE_ID)
) ;</programlisting>
@@ -234,6 +241,11 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
more than one execution per instance.</para>
</listitem>
<listitem>
<para>CREATE_TIME: Timestamp representing the time that the execution
was created.</para>
</listitem>
<listitem>
<para>START_TIME: Timestamp representing the time the execution was
started.</para>
@@ -254,11 +266,6 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
<classname>BatchStatus</classname> enumeration.</para>
</listitem>
<listitem>
<para>CONTINUABLE: Character indicating whether or not the execution
is currently able to continue. 'Y' for yes and 'N' for no.</para>
</listitem>
<listitem>
<para>EXIT_CODE: Character string representing the exit code of the
execution. In the case of a command line job, this may be converted
@@ -270,6 +277,11 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
description of how the job exited. In the case of failure, this might
include as much of the stack trace as is possible.</para>
</listitem>
<listitem>
<para>LAST_UPDATED: Timestamp representing the last time this
execution was persisted.</para>
</listitem>
</itemizedlist>
</section>
@@ -291,10 +303,16 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
END_TIME TIMESTAMP DEFAULT NULL,
STATUS VARCHAR(10),
COMMIT_COUNT BIGINT ,
ITEM_COUNT BIGINT ,
CONTINUABLE CHAR(1),
EXIT_CODE VARCHAR(20),
EXIT_MESSAGE VARCHAR(2500),
READ_COUNT BIGINT ,
FILTER_COUNT BIGINT ,
WRITE_COUNT BIGINT ,
READ_SKIP_COUNT BIGINT ,
WRITE_SKIP_COUNT BIGINT ,
PROCESS_SKIP_COUNT BIGINT ,
ROLLBACK_COUNT BIGINT ,
EXIT_CODE VARCHAR(20) ,
EXIT_MESSAGE VARCHAR(2500) ,
LAST_UPDATED TIMESTAMP,
constraint JOB_EXECUTION_STEP_FK foreign key (JOB_EXECUTION_ID)
references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID)
) ;</programlisting>
@@ -352,13 +370,38 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
</listitem>
<listitem>
<para>ITEM_COUNT: The number of items that have been written out
during this execution.</para>
<para>READ_COUNT: The number of items read during this
execution.</para>
</listitem>
<listitem>
<para>CONTINUABLE: Character indicating whether or not the execution
is currently able to continue. 'Y' for yes and 'N' for no.</para>
<para>FILTER_COUNT: The number of items filtered out of this
execution.</para>
</listitem>
<listitem>
<para>WRITE_COUNT: The number of items written during this
execution.</para>
</listitem>
<listitem>
<para>READ_SKIP_COUNT: The number of items skipped on read during this
execution.</para>
</listitem>
<listitem>
<para>WRITE_SKIP_COUNT: The number of items skipped on write during
this execution.</para>
</listitem>
<listitem>
<para>PROCESS_SKIP_COUNT: The number of items skipped during
processing during this execution.</para>
</listitem>
<listitem>
<para>ROLLBACK_COUNT: The number of rollbacks during this
execution.</para>
</listitem>
<listitem>
@@ -372,88 +415,94 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
description of how the job exited. In the case of failure, this might
include as much of the stack trace as is possible.</para>
</listitem>
<listitem>
<para>LAST_UPDATED: Timestamp representing the last time this
execution was persisted.</para>
</listitem>
</itemizedlist>
</section>
<section>
<title>BATCH_EXECUTION_CONTEXT</title>
<title>BATCH_JOB_EXECUTION_CONTEXT</title>
<para>The BATCH_STEP_EXECUTION_CONTEXT table holds all information
relevant to an <classname>ExecutionContext</classname>. There is exactly
one <classname>ExecutionContext</classname> per
<classname>StepExecution</classname>, and it contains all user defined
key/value pairs that need to persisted for a particular job run. This data
is typically state that must be retrieved back after a failure so that a
<classname>JobInstance</classname> can 'start from where it left off'. As
with the BATCH_JOB_PARAMS table, this table has been denormalized and uses
a column to determine the type:</para>
<para>The BATCH_JOB_EXECUTION_CONTEXT table holds all information relevant
to an <classname>Job</classname>'s
<classname>ExecutionContext</classname>. There is exactly one
<classname>ExecutionContext</classname> per
<classname>StepExecution</classname>, and it contains all of the job-level
data that is needed for a particular job execution. This data typically
represents the state that must be retrieved after a failure so that a
<classname>JobInstance</classname> can 'start from where it left
off'.</para>
<programlisting>CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT (
EXECUTION_ID BIGINT NOT NULL ,
DISCRIMINATOR VARCHAR2(1) NOT NULL,
TYPE_CD VARCHAR(6) NOT NULL ,
KEY_NAME VARCHAR(1000) NOT NULL ,
STRING_VAL VARCHAR(1000) ,
DATE_VAL TIMESTAMP DEFAULT NULL ,
LONG_VAL VARCHAR(10) ,
DOUBLE_VAL DOUBLE PRECISION ,
OBJECT_VAL BLOB,
<programlisting>CREATE TABLE BATCH_JOB_EXECUTION_CONTEXT (
JOB_EXECUTION_ID BIGINT PRIMARY KEY,
SHORT_CONTEXT VARCHAR(2500) NOT NULL,
SERIALIZED_CONTEXT CLOB,
constraint JOB_EXEC_CTX_FK foreign key (JOB_EXECUTION_ID)
references BATCH_JOB_EXECUTION(JOB_EXECUTION_ID)
) ;</programlisting>
<para>Below are descriptions for each column:</para>
<itemizedlist>
<listitem>
<para>EXECUTION_ID: Foreign key representing the
<classname>StepExecution</classname> or
<para>JOB_EXECUTION_ID: Foreign key representing the
<classname>JobExecution</classname> to which the context belongs.
There may be more than one row associated to a given execution.</para>
</listitem>
<listitem>
<para>DISCRIMINATOR: Character indicating whether or not the entry is
job or step scoped. (i.e. does it belong to the JobExecution or
StepExecution)</para>
<para>SHORT_CONTEXT: A string version of the
SERIALIZED_CONTEXT.</para>
</listitem>
<listitem>
<para>TYPE_CD: String representation of the type of value stored,
which can be either a character string, date, long, or double. Because
the type must be known, it cannot be null.</para>
</listitem>
<listitem>
<para>KEY_NAME: The Parameter key.</para>
</listitem>
<listitem>
<para>STRING_VAL: Parameter value, if the type is string.</para>
</listitem>
<listitem>
<para>DATE_VAL: Parameter value, if the type is date.</para>
</listitem>
<listitem>
<para>LONG_VAL: Parameter value, if the type is a long.</para>
</listitem>
<listitem>
<para>DOUBLE_VAL: Parameter value, if the type is double.</para>
</listitem>
<listitem>
<para>OBJECT_VAL: Parameter value, if the type is a blob.</para>
<para>SERIALIZED_CONTEXT: The entire context, serialized.</para>
</listitem>
</itemizedlist>
</section>
<para>When an ExecutionContext is stored, values that are one of the well
known types above will be stored as their respective type. Any unknown
type will be serialized to a blob and stored in the OBJECT_VAL column. As
with BATCH_JOB_PARAMS, there is no primary key for this table. This is
simply because the framework has no use for one, and thus doesn't require
it. If a user so chooses, one may be added with a database generated key,
without causing any issues to the framework itself.</para>
<section>
<title>BATCH_STEP_EXECUTION_CONTEXT</title>
<para>The BATCH_STEP_EXECUTION_CONTEXT table holds all information
relevant to an <classname>Step</classname>'s
<classname>ExecutionContext</classname>. There is exactly one
<classname>ExecutionContext</classname> per
<classname>StepExecution</classname>, and it contains all of the data that
needs to persisted for a particular step execution. This data typically
represents the state that must be retrieved after a failure so that a
<classname>JobInstance</classname> can 'start from where it left
off'.</para>
<programlisting>CREATE TABLE BATCH_STEP_EXECUTION_CONTEXT (
STEP_EXECUTION_ID BIGINT PRIMARY KEY,
SHORT_CONTEXT VARCHAR(2500) NOT NULL,
SERIALIZED_CONTEXT CLOB,
constraint STEP_EXEC_CTX_FK foreign key (STEP_EXECUTION_ID)
references BATCH_STEP_EXECUTION(STEP_EXECUTION_ID)
) ;</programlisting>
<para>Below are descriptions for each column:</para>
<itemizedlist>
<listitem>
<para>STEP_EXECUTION_ID: Foreign key representing the
<classname>StepExecution</classname> to which the context belongs.
There may be more than one row associated to a given execution.</para>
</listitem>
<listitem>
<para>SHORT_CONTEXT: A string version of the
SERIALIZED_CONTEXT.</para>
</listitem>
<listitem>
<para>SERIALIZED_CONTEXT: The entire context, serialized.</para>
</listitem>
</itemizedlist>
</section>
<section>
@@ -463,23 +512,28 @@ INSERT INTO BATCH_JOB_SEQ values(0);</programlisting>
is run, it is common to create an archive strategy for the meta-data
tables. The tables themselves are designed to show a record of what
happened in the past, and generally won't affect the run of any job, with
a couple of notable exceptions:</para>
a couple of notable exceptions pertaining to restart:</para>
<itemizedlist>
<listitem>
<para>Restart: Because the ExecutionContext is persisted, removing any
entries from this table of jobs that haven't completed successfully,
will prevent them from starting at the correct point if run again.
Furthermore, if an entry for a JobInstance is removed without having
completed successfully, the framework will think that the job is new,
rather than a restart.</para>
<para>The framework will use the meta-data tables to determine if a
particular JobInstance has been run before. If it has been run, and
the job is not restartable, then an exception will be thrown. </para>
</listitem>
<listitem>
<para>Determining if an instance has been run: The framework will use
the meta-data tables to determine if a particular JobInstance has been
run before, and if it has an exception will be thrown.</para>
<para>If an entry for a JobInstance is removed without having
completed successfully, the framework will think that the job is new,
rather than a restart. </para>
</listitem>
<listitem>
<para>If a job is restarted, the framework will use any data that has
been persisted to the ExecutionContext to restore the Job's state.
Therefore, removing any entries from this table for jobs that haven't
completed successfully will prevent them from starting at the correct
point if run again.</para>
</listitem>
</itemizedlist>
</section>
</appendix>
</appendix>

70
docs/src/site/docbook/reference/step.xml
View File

@@ -1209,5 +1209,75 @@
</programlisting></para>
</section>
<section>
<title>Creating File Names at Runtime</title>
<para>Both the XML and Flat File examples above use the Spring
<classname>Resource</classname> abstraction to obtain the file to read
or write from. This works because <classname>Resource</classname> has a
<markup>getFile</markup> method, that returns a
<classname>java.io.File</classname>. Both XML and Flat File resources
can be configured using standard Spring constructs:</para>
<programlisting> &lt;bean id="flatFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"&gt;
&lt;property name="resource"
value="file://outputs/20070122.testStream.CustomerReportStep.TEMP.txt" /&gt;
&lt;/bean&gt;</programlisting>
<para>The above <classname>Resource</classname> will load the file from
the file system, at the location specificied. Note that absolute
locations have to start with a double slash ("//"). In most spring
applications, this solution is good enough because the names of these
are known at compile time. However, in batch scenarios, the file name
may need to be determined at runtime as a parameter to the job. This
could be solved using '-D' parameters, i.e. a system property:</para>
<programlisting>&lt;bean id="flatFileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"&gt;
&lt;property name="resource" value="${input.file.name}" /&gt;
&lt;/bean&gt;</programlisting>
<para>All that would be required for this solution to work would be a
system argument (-Dinput.file.name="file://file.txt"). (Note that
although a <classname>PropertyPlaceholderConfigurer</classname> can be
used here, it is not necessary if the system property is always set
because the <classname>ResourceEditor</classname> in Spring already
filters and does placeholder replacement on system properties.)</para>
<para>Often in a batch setting it is preferable to parameterize the file
name in the <classname>JobParameters</classname> of the job, instead of
through system properties, and access them that way. To allow for this,
Spring Batch provides the
<classname>StepExecutionResourceProxy</classname>. The proxy can use
either job name, step name, or any values from the
<classname>JobParameters</classname>, by surrounding them with %:</para>
<programlisting> &lt;bean id="inputFile"
class="org.springframework.batch.core.resource.StepExecutionResourceProxy" /&gt;
&lt;property name="filePattern" value="//%JOB_NAME%/%STEP_NAME%/%file.name%" /&gt;
&lt;/bean&gt;</programlisting>
<para>Assuming a job name of 'fooJob', and a step name of 'fooStep', and
the key-value pair of 'file.name="fileName.txt"' is in the
<classname>JobParameters</classname> the job is started with, the
following filename will be passed as the
<classname>Resource</classname>:
"<filename>//fooJob/fooStep/fileName.txt</filename>". It should be noted
that in order for the proxy to have access to the
<classname>StepExecution</classname>, it must be registered as a
<classname>StepListener</classname>:</para>
<programlisting> &lt;bean id="fooStep" parent="abstractStep"
p:itemReader-ref="itemReader"
p:itemWriter-ref="itemWriter"&gt;
&lt;property name="listeners" ref="inputFile" /&gt;
&lt;/bean&gt;</programlisting>
<para>The <classname>StepListener</classname> interface will be
discussed in more detail in Chapter 4. For now, it is sufficient to know
that the proxy must be registered.</para>
</section>
</section>
</chapter>

246
docs/src/site/docbook/reference/testing.xml
View File

@@ -9,180 +9,108 @@
documentation covers how to unit and integration test with Spring in great
detail, so it won't be repeated here. It is important, however, to think
about how to 'end to end' test a batch job, which is what this chapter will
focus on.</para>
focus on. The spring-batch-test project includes classes that will help
factillitate this end-to-end test approach.</para>
<section>
<title>Creating a Unit Test Class</title>
<para>In order for the unit test to run a batch job, the framework must
load the job's ApplicationContext. Two annotations are used to trigger
this:</para>
<itemizedlist>
<listitem>
<para><classname>@RunWith(SpringJUnit4ClassRunner.class)</classname>:
Indicates that the class should use Spring's JUnit facilities</para>
</listitem>
<listitem>
<para><classname>@ContextConfiguration(locations = {...})</classname>:
Indicates which xml files contain the ApplicationContext.</para>
</listitem>
</itemizedlist>
<programlisting>
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/simple-job-launcher-context.xml", "/jobs/skipSampleJob.xml" })
public class SkipSampleFunctionalTests extends AbstractJobTests { ... }
</programlisting>
</section>
<section>
<title>End To End Testing Batch Jobs</title>
<para>'End To End' testing can be defined as testing the complete run of a
batch job from beginning to end. If the job reads from a file, then writes
into the database, this type of testing ensures that any preconditions are
met (reference data, correct file, etc) and then runs the job, verifying
afterwards that all records that should be in the database are present and
correct. Below is an example from one of the Spring Batch sample jobs, the
'fixedLengthImportJob'. It reads from a flat file (in fixed length format)
and loads the records into the database. The following unit test code
assures it processes correctly:</para>
batch job from beginning to end. This allows for a test that sets up a
test condition, executes the job, and verifies the end result.</para>
<programlisting> //fixed-length file is expected on input
protected void validatePreConditions() throws Exception{
BufferedReader reader = null;
reader = new BufferedReader(new FileReader(fileLocator.getFile()));
String line;
while ((line = reader.readLine()) != null) {
assertEquals(LINE_LENGTH, line.length());
}
<para>In the example below, the batch job reads from the database and
writes to a flat file. The test method begins by setting up the database
with test data. It clears the CUSTOMER table and then inserts 10 new
records. The test then launches the <classname>Job </classname>using the
<methodname>launchJob()</methodname> method. The
<methodname>launchJob</methodname>() method is provided by the
<classname>AbstractJobTests</classname> parent class. Also provided by the
super class is <classname>launchJob(JobParameters)</classname>, which
allows the test to give particular parameters. The
<methodname>launchJob()</methodname> method returns the
<classname>JobExecution</classname> object which is useful for asserting
particular information about the <classname>Job</classname> run. In the
case below, the test verifies that the <classname>Job</classname> ended
with status "COMPLETED".</para>
<programlisting>
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/simple-job-launcher-context.xml", "/jobs/skipSampleJob.xml" })
public class SkipSampleFunctionalTests extends AbstractJobTests {
private SimpleJdbcTemplate simpleJdbcTemplate;
@Autowired
public void setDataSource(DataSource dataSource) {
this.simpleJdbcTemplate = new SimpleJdbcTemplate(dataSource);
}
@Transactional
@Test
public void testJob() throws Exception {
simpleJdbcTemplate.update("delete from CUSTOMER");
for (int i = 1; i &lt;= 10; i++) {
simpleJdbcTemplate.update("insert into CUSTOMER values (?, 0, ?, 100000)", i, "customer" + i);
}
JobExecution jobExecution = this.launchJob();
Assert.assertEquals("COMPLETED", jobExecution.getExitStatus());
}
}
//Check that records have been correctly written to database
protected void validatePostConditions() throws Exception {
inputSource.open(new ExecutionContext());
jdbcTemplate.query("SELECT ID, ISIN, QUANTITY, PRICE, CUSTOMER FROM trade ORDER BY id",
new RowCallbackHandler() {
public void processRow(ResultSet rs) throws SQLException {
Trade trade;
try {
trade = (Trade)inputSource.read();
}
catch (Exception e) {
throw new IllegalStateException(e.getMessage());
}
assertEquals(trade.getIsin(), rs.getString(2));
assertEquals(trade.getQuantity(),rs.getLong(3));
assertEquals(trade.getPrice(), rs.getBigDecimal(4));
assertEquals(trade.getCustomer(), rs.getString(5));
}});
assertNull(inputSource.read());
}</programlisting>
<para>In the first method, <methodname>validatePreConditions</methodname>,
the input file is checked to ensure it is correctly formatted. Because it
is common to add extra lines to the file to test additional use cases,
this test ensures that the fixed length lines are the length they should
be. If they are not, it is much preferred to fail in this phase, rather
than the job (correctly) failing during the run and causing needless
debugging.</para>
<para>In the second method, validatePostconditions, the database is
checked to ensure all data has been written correctly. This is arguably
the most important part of the test. In this case, it reads one line from
the file, and one row from the database, and checks each column one by one
for accuracy. It's important to not hard-code the data that should be
present in the database into the test class. Instead, use the input file
(bypassing the job) to check the output. This allows you to quickly add
additional test cases to your file without having to add them to code. The
same would be true for database to database jobs, or database to file
jobs. It is preferable to be able to add additional rows to the database
input without having to add them to the hard coded list in the test
class.</para>
</programlisting>
</section>
<section>
<title>Extending Unit Test frameworks</title>
<title>Validating Output Files</title>
<para>Because most unit testing of complete batch jobs will take place in
the development environment (i.e. eclipse) it's important to be able to
launch these tests in the same way you would launch any unit test. In the
following examples JUnit 4 will be used, but any testing framework could
be substituted. The Spring Batch samples contain many 'sample jobs' that
are unit tested using this technique. The most important step is being
able to launch the job within a unit test. This requires the use of the
<classname>JobLauncher</classname> interface that is discussed in chapters
2 and 4. A <classname>Job</classname> and
<classname>JobLauncher</classname> must be obtained from an
<classname>ApplicationContext</classname>, and then launched. The
following abstract class from Spring Batch Samples illustrates
this:</para>
<para>When a batch job writes to the database, it is easy to query the
database to verify that the output is as expected. However, if the batch
job writes to a file, it is equally important that the output be verified.
Spring Batch provides a class <classname>AssertFile</classname> to
facilitate the verification of output files. The method
<methodname>assertFileEquals</methodname> takes two
<classname>File</classname> objects (or two
<classname>Resource</classname> objects) and asserts, line by line, that
the two files have the same content. Therefore, it is possible to create a
file with the expected output and to compare it to the actual
result:</para>
<programlisting> public abstract class AbstractBatchLauncherTests implements ApplicationContextAware {
<programlisting>
private static final String EXPECTED_FILE = "src/main/resources/data/iosample/input/multiLine.txt";
private static final String OUTPUT_FILE = "target/test-outputs/multiLineOutput.txt";
JobLauncher launcher;
private Job job;
private JobParameters jobParameters = new JobParameters();
AssertFile.assertFileEquals(new FileSystemResource(EXPECTED_FILE), new FileSystemResource(OUTPUT_FILE));
@Test
public void testLaunchJob() throws Exception {
launcher.run(job, jobParameters);
}
@Autowired
public void setLauncher(JobLauncher bootstrap) {
this.launcher = bootstrap;
}
@Autowired
public void setJob(Job job) {
this.job = job;
}
}
</programlisting>
<note>
<para>Few additional convenience properties are left out from the real class definition for clarity.</para>
</note>
<para>Only two classes
are needed: The Job to be run, and the JobLauncher to run it. These properties
are declared to be autowired from the job's application context . Empty
<classname>JobParameters</classname> are used in the example above.
However, if the job requires specific parameters they could be coded in
subclasses with an abstract method, or using a factory bean in the
<classname>ApplicationContext</classname> for testing purposes. Because
none of the sample jobs require this, an empty
<classname>JobParameters</classname> is used. One simple JUnit test case
is present in the file, which actually launches the job. If any exceptions
are thrown or assertions fail, it will act the same way as any other unit
test and display as a failed test due to errors or assertion failure.
Because of the best practice for validation mentioned earlier in the
chapter, this class is extended further to allow for separate validation
before and after the job is run:</para>
<programlisting>
public abstract class AbstractValidatingBatchLauncherTests extends AbstractBatchLauncherTests {
@Test
public void testLaunchJob() throws Exception {
validatePreConditions();
super.testLaunchJob();
validatePostConditions();
}
/**
* Make sure input data meets expectations
*/
protected void validatePreConditions() throws Exception {}
/**
* Make sure job did what it was expected to do.
*/
protected abstract void validatePostConditions() throws Exception;
}
</programlisting>
<para>In the class above, the <methodname>testLaunchJob</methodname>
method is overridden to call the two abstract methods for validation.
Before actually running the job,
<methodname>validatePreConditions</methodname> is called (it should be
noted that it's not required), and then after the job completes
successfully, <methodname>validatePostConidtions</methodname> is
called.</para>
<para>Finally to create an executable test the abstract superclass needs to be subclassed.
Spring-specific annotations ensure the appropriate application context is loaded and
required properties are injected before executing the test. In this case the XML file name
is derived from the class name, so <filename>FixedLengthImportJobFunctionalTests-context.xml</filename>
(see the "Testing" chapter of Spring reference documentation for more details)</para>
<programlisting>
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration()
public class FixedLengthImportJobFunctionalTests extends AbstractValidatingBatchLauncherTests {...}
</programlisting>
<para></para>
</section>
</chapter>