425 lines
36 KiB
HTML
425 lines
36 KiB
HTML
<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE html><html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xmlns:m="http://www.w3.org/1998/Math/MathML" xmlns:pls="http://www.w3.org/2005/01/pronunciation-lexicon" xmlns:ssml="http://www.w3.org/2001/10/synthesis" xmlns:svg="http://www.w3.org/2000/svg"><head><title>Database</title><link rel="stylesheet" type="text/css" href="docbook-epub.css"/><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"/><link rel="prev" href="ch06s08.xhtml" title="Multi-File Input"/><link rel="next" href="ch06s10.xhtml" title="Reusing Existing Services"/></head><body><header/><section class="section" title="Database" epub:type="subchapter" id="database"><div class="titlepage"><div><div><h2 class="title" style="clear: both">Database</h2></div></div></div><p>Like most enterprise application styles, a database is the central
|
||
storage mechanism for batch. However, batch differs from other application
|
||
styles due to the sheer size of the datasets with which the system must
|
||
work. If a SQL statement returns 1 million rows, the result set probably
|
||
holds all returned results in memory until all rows have been read. Spring
|
||
Batch provides two types of solutions for this problem: Cursor and Paging
|
||
database ItemReaders.</p><section class="section" title="Cursor Based ItemReaders" epub:type="division" id="cursorBasedItemReaders"><div class="titlepage"><div><div><h3 class="title">Cursor Based ItemReaders</h3></div></div></div><p>Using a database cursor is generally the default approach of most
|
||
batch developers, because it is the database's solution to the problem
|
||
of 'streaming' relational data. The Java
|
||
<code class="classname">ResultSet</code> class is essentially an object
|
||
orientated mechanism for manipulating a cursor. A
|
||
<code class="classname">ResultSet</code> maintains a cursor to the current row
|
||
of data. Calling <code class="methodname">next</code> on a
|
||
<code class="classname">ResultSet</code> moves this cursor to the next row.
|
||
Spring Batch cursor based ItemReaders open the a cursor on
|
||
initialization, and move the cursor forward one row for every call to
|
||
<code class="methodname">read</code>, returning a mapped object that can be
|
||
used for processing. The <code class="methodname">close</code> method will then
|
||
be called to ensure all resources are freed up. The Spring core
|
||
<code class="classname">JdbcTemplate</code> gets around this problem by using
|
||
the callback pattern to completely map all rows in a
|
||
<code class="classname">ResultSet</code> and close before returning control back
|
||
to the method caller. However, in batch this must wait until the step is
|
||
complete. Below is a generic diagram of how a cursor based
|
||
<code class="classname">ItemReader</code> works, and while a SQL statement is
|
||
used as an example since it is so widely known, any technology could
|
||
implement the basic approach:</p><div style="text-align: center; " class="mediaobject"><img style="text-align: middle; " src="images/cursorExample.png" width="351"/></div><p>This example illustrates the basic pattern. Given a 'FOO' table,
|
||
which has three columns: ID, NAME, and BAR, select all rows with an ID
|
||
greater than 1 but less than 7. This puts the beginning of the cursor
|
||
(row 1) on ID 2. The result of this row should be a completely mapped
|
||
Foo object. Calling <code class="methodname">read</code>() again moves the
|
||
cursor to the next row, which is the Foo with an ID of 3. The results of
|
||
these reads will be written out after each
|
||
<code class="methodname">read</code>, thus allowing the objects to be garbage
|
||
collected (assuming no instance variables are maintaining references to
|
||
them).</p><section class="section" title="JdbcCursorItemReader" epub:type="division" id="JdbcCursorItemReader"><div class="titlepage"><div><div><h4 class="title">JdbcCursorItemReader</h4></div></div></div><p><code class="classname">JdbcCursorItemReader</code> is the Jdbc
|
||
implementation of the cursor based technique. It works directly with a
|
||
<code class="classname">ResultSet</code> and requires a SQL statement to run
|
||
against a connection obtained from a
|
||
<code class="classname">DataSource</code>. The following database schema will
|
||
be used as an example:</p><pre class="programlisting">CREATE TABLE CUSTOMER (
|
||
ID BIGINT IDENTITY PRIMARY KEY,
|
||
NAME VARCHAR(45),
|
||
CREDIT FLOAT
|
||
);</pre><p>Many people prefer to use a domain object for each row, so we'll
|
||
use an implementation of the <code class="classname">RowMapper</code>
|
||
interface to map a <code class="classname">CustomerCredit</code>
|
||
object:</p><pre class="programlisting">public class CustomerCreditRowMapper implements RowMapper {
|
||
|
||
public static final String ID_COLUMN = "id";
|
||
public static final String NAME_COLUMN = "name";
|
||
public static final String CREDIT_COLUMN = "credit";
|
||
|
||
public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
|
||
CustomerCredit customerCredit = new CustomerCredit();
|
||
|
||
customerCredit.setId(rs.getInt(ID_COLUMN));
|
||
customerCredit.setName(rs.getString(NAME_COLUMN));
|
||
customerCredit.setCredit(rs.getBigDecimal(CREDIT_COLUMN));
|
||
|
||
return customerCredit;
|
||
}
|
||
}</pre><p>Because <code class="classname">JdbcTemplate</code> is so familiar to
|
||
users of Spring, and the <code class="classname">JdbcCursorItemReader</code>
|
||
shares key interfaces with it, it is useful to see an example of how
|
||
to read in this data with <code class="classname">JdbcTemplate</code>, in
|
||
order to contrast it with the <code class="classname">ItemReader</code>. For
|
||
the purposes of this example, let's assume there are 1,000 rows in the
|
||
CUSTOMER database. The first example will be using
|
||
<code class="classname">JdbcTemplate</code>:</p><pre class="programlisting">//For simplicity sake, assume a dataSource has already been obtained
|
||
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
|
||
List customerCredits = jdbcTemplate.query("SELECT ID, NAME, CREDIT from CUSTOMER",
|
||
new CustomerCreditRowMapper());</pre><p>After running this code snippet the customerCredits list will
|
||
contain 1,000 <code class="classname">CustomerCredit</code> objects. In the
|
||
query method, a connection will be obtained from the
|
||
<code class="classname">DataSource</code>, the provided SQL will be run
|
||
against it, and the <code class="methodname">mapRow</code> method will be
|
||
called for each row in the <code class="classname">ResultSet</code>. Let's
|
||
contrast this with the approach of the
|
||
<code class="classname">JdbcCursorItemReader</code>:</p><pre class="programlisting">JdbcCursorItemReader itemReader = new JdbcCursorItemReader();
|
||
itemReader.setDataSource(dataSource);
|
||
itemReader.setSql("SELECT ID, NAME, CREDIT from CUSTOMER");
|
||
itemReader.setRowMapper(new CustomerCreditRowMapper());
|
||
int counter = 0;
|
||
ExecutionContext executionContext = new ExecutionContext();
|
||
itemReader.open(executionContext);
|
||
Object customerCredit = new Object();
|
||
while(customerCredit != null){
|
||
customerCredit = itemReader.read();
|
||
counter++;
|
||
}
|
||
itemReader.close(executionContext);</pre><p>After running this code snippet the counter will equal 1,000. If
|
||
the code above had put the returned customerCredit into a list, the
|
||
result would have been exactly the same as with the
|
||
<code class="classname">JdbcTemplate</code> example. However, the big
|
||
advantage of the <code class="classname">ItemReader</code> is that it allows
|
||
items to be 'streamed'. The <code class="methodname">read</code> method can
|
||
be called once, and the item written out via an
|
||
<code class="classname">ItemWriter</code>, and then the next item obtained via
|
||
<code class="methodname">read</code>. This allows item reading and writing to
|
||
be done in 'chunks' and committed periodically, which is the essence
|
||
of high performance batch processing. Furthermore, it is very easily
|
||
configured for injection into a Spring Batch
|
||
<code class="classname">Step</code>:</p><pre class="programlisting"><bean id="itemReader" class="org.spr...JdbcCursorItemReader">
|
||
<property name="dataSource" ref="dataSource"/>
|
||
<property name="sql" value="select ID, NAME, CREDIT from CUSTOMER"/>
|
||
<property name="rowMapper">
|
||
<bean class="org.springframework.batch.sample.domain.CustomerCreditRowMapper"/>
|
||
</property>
|
||
</bean></pre><section class="section" title="Additional Properties" epub:type="division" id="JdbcCursorItemReaderProperties"><div class="titlepage"><div><div><h5 class="title">Additional Properties</h5></div></div></div><p>Because there are so many varying options for opening a cursor
|
||
in Java, there are many properties on the
|
||
<code class="classname">JdbcCustorItemReader</code> that can be set:</p><div class="table" id="d5e2752"><div class="table-title">Table 6.2. JdbcCursorItemReader Properties</div><div class="table-contents"><table style="border-collapse: collapse; border-top: 0.5pt solid ; border-bottom: 0.5pt solid ; border-left: 0.5pt solid ; border-right: 0.5pt solid ; "><colgroup><col/><col/></colgroup><tbody><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">ignoreWarnings</td><td style="border-bottom: 0.5pt solid ; ">Determines whether or not SQLWarnings are logged or
|
||
cause an exception - default is true</td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">fetchSize</td><td style="border-bottom: 0.5pt solid ; ">Gives the Jdbc driver a hint as to the number of rows
|
||
that should be fetched from the database when more rows are
|
||
needed by the <code class="classname">ResultSet</code> object used
|
||
by the <code class="classname">ItemReader</code>. By default, no
|
||
hint is given.</td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">maxRows</td><td style="border-bottom: 0.5pt solid ; ">Sets the limit for the maximum number of rows the
|
||
underlying <code class="classname">ResultSet</code> can hold at any
|
||
one time.</td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">queryTimeout</td><td style="border-bottom: 0.5pt solid ; ">Sets the number of seconds the driver will wait for a
|
||
<code class="classname">Statement</code> object to execute to the
|
||
given number of seconds. If the limit is exceeded, a
|
||
<code class="classname">DataAccessEception</code> is thrown.
|
||
(Consult your driver vendor documentation for
|
||
details).</td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">verifyCursorPosition</td><td style="border-bottom: 0.5pt solid ; ">Because the same <code class="classname">ResultSet</code>
|
||
held by the <code class="classname">ItemReader</code> is passed to
|
||
the <code class="classname">RowMapper</code>, it is possible for
|
||
users to call <code class="methodname">ResultSet.next</code>()
|
||
themselves, which could cause issues with the reader's
|
||
internal count. Setting this value to true will cause an
|
||
exception to be thrown if the cursor position is not the
|
||
same after the <code class="classname">RowMapper</code> call as it
|
||
was before.</td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">saveState</td><td style="border-bottom: 0.5pt solid ; ">Indicates whether or not the reader's state should be
|
||
saved in the <code class="classname">ExecutionContext</code>
|
||
provided by
|
||
<code class="methodname">ItemStream#update</code>(<code class="classname">ExecutionContext</code>)
|
||
The default value is true.</td></tr><tr><td style="border-right: 0.5pt solid ; border-bottom: 0.5pt solid ; ">driverSupportsAbsolute</td><td style="border-bottom: 0.5pt solid ; ">Defaults to false. Indicates whether the Jdbc driver
|
||
supports setting the absolute row on a
|
||
<code class="classname">ResultSet</code>. It is recommended that
|
||
this is set to true for Jdbc drivers that supports
|
||
<code class="methodname">ResultSet.absolute</code>() as it may
|
||
improve performance, especially if a step fails while
|
||
working with a large data set.</td></tr><tr><td style="border-right: 0.5pt solid ; ">setUseSharedExtendedConnection</td><td>Defaults to false. Indicates whether the connection
|
||
used for the cursor should be used by all other processing
|
||
thus sharing the same transaction. If this is set to false,
|
||
which is the default, then the cursor will be opened using
|
||
its own connection and will not participate in any
|
||
transactions started for the rest of the step processing. If
|
||
you set this flag to true then you must wrap the
|
||
<code class="classname">DataSource</code> in an
|
||
<code class="classname">ExtendedConnectionDataSourceProxy</code> to
|
||
prevent the connection from being closed and released after
|
||
each commit. When you set this option to true then the
|
||
statement used to open the cursor will be created with both
|
||
'READ_ONLY' and 'HOLD_CUSORS_OVER_COMMIT' options. This
|
||
allows holding the cursor open over transaction start and
|
||
commits performed in the step processing. To use this
|
||
feature you need a database that supports this and a Jdbc
|
||
driver supporting Jdbc 3.0 or later.</td></tr></tbody></table></div></div></section></section><section class="section" title="HibernateCursorItemReader" epub:type="division" id="HibernateCursorItemReader"><div class="titlepage"><div><div><h4 class="title">HibernateCursorItemReader</h4></div></div></div><p>Just as normal Spring users make important decisions about
|
||
whether or not to use ORM solutions, which affect whether or not they
|
||
use a <code class="classname">JdbcTemplate</code> or a
|
||
<code class="classname">HibernateTemplate</code>, Spring Batch users have the
|
||
same options. <code class="classname">HibernateCursorItemReader</code> is the
|
||
Hibernate implementation of the cursor technique. Hibernate's usage in
|
||
batch has been fairly controversial. This has largely been because
|
||
Hibernate was originally developed to support online application
|
||
styles. However, that doesn't mean it can't be used for batch
|
||
processing. The easiest approach for solving this problem is to use a
|
||
<code class="classname">StatelessSession</code> rather than a standard
|
||
session. This removes all of the caching and dirty checking hibernate
|
||
employs that can cause issues in a batch scenario. For more
|
||
information on the differences between stateless and normal hibernate
|
||
sessions, refer to the documentation of your specific hibernate
|
||
release. The <code class="classname">HibernateCursorItemReader</code> allows
|
||
you to declare an HQL statement and pass in a
|
||
<code class="classname">SessionFactory</code>, which will pass back one item
|
||
per call to <code class="methodname">read</code> in the same basic fashion as
|
||
the <code class="classname">JdbcCursorItemReader</code>. Below is an example
|
||
configuration using the same 'customer credit' example as the JDBC
|
||
reader:</p><pre class="programlisting">HibernateCursorItemReader itemReader = new HibernateCursorItemReader();
|
||
itemReader.setQueryString("from CustomerCredit");
|
||
//For simplicity sake, assume sessionFactory already obtained.
|
||
itemReader.setSessionFactory(sessionFactory);
|
||
itemReader.setUseStatelessSession(true);
|
||
int counter = 0;
|
||
ExecutionContext executionContext = new ExecutionContext();
|
||
itemReader.open(executionContext);
|
||
Object customerCredit = new Object();
|
||
while(customerCredit != null){
|
||
customerCredit = itemReader.read();
|
||
counter++;
|
||
}
|
||
itemReader.close(executionContext);</pre><p>This configured <code class="classname">ItemReader</code> will return
|
||
<code class="classname">CustomerCredit</code> objects in the exact same manner
|
||
as described by the <code class="classname">JdbcCursorItemReader</code>,
|
||
assuming hibernate mapping files have been created correctly for the
|
||
Customer table. The 'useStatelessSession' property defaults to true,
|
||
but has been added here to draw attention to the ability to switch it
|
||
on or off. It is also worth noting that the fetchSize of the
|
||
underlying cursor can be set via the setFetchSize property. As with
|
||
<code class="classname">JdbcCursorItemReader</code>, configuration is
|
||
straightforward:</p><pre class="programlisting"><bean id="itemReader"
|
||
class="org.springframework.batch.item.database.HibernateCursorItemReader">
|
||
<property name="sessionFactory" ref="sessionFactory" />
|
||
<property name="queryString" value="from CustomerCredit" />
|
||
</bean></pre></section><section class="section" title="StoredProcedureItemReader" epub:type="division" id="StoredProcedureItemReader"><div class="titlepage"><div><div><h4 class="title">StoredProcedureItemReader</h4></div></div></div><p>Sometimes it is necessary to obtain the cursor data using a
|
||
stored procedure. The <code class="classname">StoredProcedureItemReader</code>
|
||
works like the <code class="classname">JdbcCursorItemReader</code> except that
|
||
instead of executing a query to obtain a cursor we execute a stored
|
||
procedure that returns a cursor. The stored procedure can return the
|
||
cursor in three different ways:</p><div class="orderedlist" epub:type="list"><ol class="orderedlist" type="1"><li class="listitem" epub:type="list-item"><p>as a returned ResultSet (used by SQL Server, Sybase, DB2,
|
||
Derby and MySQL)</p></li><li class="listitem" epub:type="list-item"><p>as a ref-cursor returned as an out parameter (used by Oracle
|
||
and PostgreSQL)</p></li><li class="listitem" epub:type="list-item"><p>as the return value of a stored function call</p></li></ol></div><p>Below is a basic example configuration using the same 'customer
|
||
credit' example as earlier:</p><pre class="programlisting"><bean id="reader" class="o.s.batch.item.database.StoredProcedureItemReader">
|
||
<property name="dataSource" ref="dataSource"/>
|
||
<property name="procedureName" value="sp_customer_credit"/>
|
||
<property name="rowMapper">
|
||
<bean class="org.springframework.batch.sample.domain.CustomerCreditRowMapper"/>
|
||
</property>
|
||
</bean>
|
||
</pre><p>This example relies on the stored procedure to provide a
|
||
ResultSet as a returned result (option 1 above). </p><p>If the stored procedure returned a ref-cursor (option 2) then we
|
||
would need to provide the position of the out parameter that is the
|
||
returned ref-cursor. Here is an example where the first parameter is
|
||
the returned ref-cursor:</p><pre class="programlisting"><bean id="reader" class="o.s.batch.item.database.StoredProcedureItemReader">
|
||
<property name="dataSource" ref="dataSource"/>
|
||
<property name="procedureName" value="sp_customer_credit"/>
|
||
<property name="refCursorPosition" value="1"/>
|
||
<property name="rowMapper">
|
||
<bean class="org.springframework.batch.sample.domain.CustomerCreditRowMapper"/>
|
||
</property>
|
||
</bean>
|
||
</pre><p>If the cursor was returned from a stored function (option 3) we
|
||
would need to set the property "<code class="varname">function</code>" to
|
||
<code class="literal">true</code>. It defaults to <code class="literal">false</code>. Here
|
||
is what that would look like:</p><pre class="programlisting"><bean id="reader" class="o.s.batch.item.database.StoredProcedureItemReader">
|
||
<property name="dataSource" ref="dataSource"/>
|
||
<property name="procedureName" value="sp_customer_credit"/>
|
||
<property name="function" value="true"/>
|
||
<property name="rowMapper">
|
||
<bean class="org.springframework.batch.sample.domain.CustomerCreditRowMapper"/>
|
||
</property>
|
||
</bean>
|
||
</pre><p>In all of these cases we need to define a
|
||
<code class="classname">RowMapper</code> as well as a
|
||
<code class="classname">DataSource</code> and the actual procedure
|
||
name.</p><p>If the stored procedure or function takes in parameter then they
|
||
must be declared and set via the parameters property. Here is an
|
||
example for Oracle that declares three parameters. The first one is
|
||
the out parameter that returns the ref-cursor, the second and third
|
||
are in parameters that takes a value of type INTEGER:</p><pre class="programlisting"><bean id="reader" class="o.s.batch.item.database.StoredProcedureItemReader">
|
||
<property name="dataSource" ref="dataSource"/>
|
||
<property name="procedureName" value="spring.cursor_func"/>
|
||
<property name="parameters">
|
||
<list>
|
||
<bean class="org.springframework.jdbc.core.SqlOutParameter">
|
||
<constructor-arg index="0" value="newid"/>
|
||
<constructor-arg index="1">
|
||
<util:constant static-field="oracle.jdbc.OracleTypes.CURSOR"/>
|
||
</constructor-arg>
|
||
</bean>
|
||
<bean class="org.springframework.jdbc.core.SqlParameter">
|
||
<constructor-arg index="0" value="amount"/>
|
||
<constructor-arg index="1">
|
||
<util:constant static-field="java.sql.Types.INTEGER"/>
|
||
</constructor-arg>
|
||
</bean>
|
||
<bean class="org.springframework.jdbc.core.SqlParameter">
|
||
<constructor-arg index="0" value="custid"/>
|
||
<constructor-arg index="1">
|
||
<util:constant static-field="java.sql.Types.INTEGER"/>
|
||
</constructor-arg>
|
||
</bean>
|
||
</list>
|
||
</property>
|
||
<property name="refCursorPosition" value="1"/>
|
||
<property name="rowMapper" ref="rowMapper"/>
|
||
<property name="preparedStatementSetter" ref="parameterSetter"/>
|
||
</bean></pre><p>In addition to the parameter declarations we need to specify a
|
||
<code class="classname">PreparedStatementSetter</code> implementation that
|
||
sets the parameter values for the call. This works the same as for the
|
||
<code class="classname">JdbcCursorItemReader</code> above. All the additional
|
||
properties listed in <a class="xref" href="ch06s09.xhtml#JdbcCursorItemReaderProperties" title="Additional Properties">the section called “Additional Properties”</a>
|
||
apply to the <code class="classname">StoredProcedureItemReader</code> as well.
|
||
</p></section></section><section class="section" title="Paging ItemReaders" epub:type="division" id="pagingItemReaders"><div class="titlepage"><div><div><h3 class="title">Paging ItemReaders</h3></div></div></div><p>An alternative to using a database cursor is executing multiple
|
||
queries where each query is bringing back a portion of the results. We
|
||
refer to this portion as a page. Each query that is executed must
|
||
specify the starting row number and the number of rows that we want
|
||
returned for the page.</p><section class="section" title="JdbcPagingItemReader" epub:type="division" id="JdbcPagingItemReader"><div class="titlepage"><div><div><h4 class="title">JdbcPagingItemReader</h4></div></div></div><p>One implementation of a paging <code class="classname">ItemReader</code>
|
||
is the <code class="classname">JdbcPagingItemReader</code>. The
|
||
<code class="classname">JdbcPagingItemReader</code> needs a
|
||
<code class="classname">PagingQueryProvider</code> responsible for providing
|
||
the SQL queries used to retrieve the rows making up a page. Since each
|
||
database has its own strategy for providing paging support, we need to
|
||
use a different <code class="classname">PagingQueryProvider</code> for each
|
||
supported database type. There is also the
|
||
<code class="classname">SqlPagingQueryProviderFactoryBean</code> that will
|
||
auto-detect the database that is being used and determine the
|
||
appropriate <code class="classname">PagingQueryProvider</code> implementation.
|
||
This simplifies the configuration and is the recommended best
|
||
practice.</p><p>The <code class="classname">SqlPagingQueryProviderFactoryBean</code>
|
||
requires that you specify a select clause and a from clause. You can
|
||
also provide an optional where clause. These clauses will be used to
|
||
build an SQL statement combined with the required sortKey.</p><p>After the reader has been opened, it will pass back one item per
|
||
call to <code class="methodname">read</code> in the same basic fashion as any
|
||
other <code class="classname">ItemReader</code>. The paging happens behind the
|
||
scenes when additional rows are needed.</p><p>Below is an example configuration using a similar 'customer
|
||
credit' example as the cursor based ItemReaders above:</p><pre class="programlisting"><bean id="itemReader" class="org.spr...JdbcPagingItemReader">
|
||
<property name="dataSource" ref="dataSource"/>
|
||
<property name="queryProvider">
|
||
<bean class="org.spr...SqlPagingQueryProviderFactoryBean">
|
||
<property name="selectClause" value="select id, name, credit"/>
|
||
<property name="fromClause" value="from customer"/>
|
||
<property name="whereClause" value="where status=:status"/>
|
||
<property name="sortKey" value="id"/>
|
||
</bean>
|
||
</property>
|
||
<property name="parameterValues">
|
||
<map>
|
||
<entry key="status" value="NEW"/>
|
||
</map>
|
||
</property>
|
||
<property name="pageSize" value="1000"/>
|
||
<property name="rowMapper" ref="customerMapper"/>
|
||
</bean></pre><p>This configured <code class="classname">ItemReader</code> will return
|
||
<code class="classname">CustomerCredit</code> objects using the
|
||
<code class="classname">RowMapper</code> that must be specified. The
|
||
'pageSize' property determines the number of entities read from the
|
||
database for each query execution.</p><p>The 'parameterValues' property can be used to specify a Map of
|
||
parameter values for the query. If you use named parameters in the
|
||
where clause the key for each entry should match the name of the named
|
||
parameter. If you use a traditional '?' placeholder then the key for
|
||
each entry should be the number of the placeholder, starting with
|
||
1.</p></section><section class="section" title="JpaPagingItemReader" epub:type="division" id="JpaPagingItemReader"><div class="titlepage"><div><div><h4 class="title">JpaPagingItemReader</h4></div></div></div><p>Another implementation of a paging
|
||
<code class="classname">ItemReader</code> is the
|
||
<code class="classname">JpaPagingItemReader</code>. JPA doesn't have a concept
|
||
similar to the Hibernate <code class="classname">StatelessSession</code> so we
|
||
have to use other features provided by the JPA specification. Since
|
||
JPA supports paging, this is a natural choice when it comes to using
|
||
JPA for batch processing. After each page is read, the entities will
|
||
become detached and the persistence context will be cleared in order
|
||
to allow the entities to be garbage collected once the page is
|
||
processed.</p><p>The <code class="classname">JpaPagingItemReader</code> allows you to
|
||
declare a JPQL statement and pass in a
|
||
<code class="classname">EntityManagerFactory</code>. It will then pass back
|
||
one item per call to <code class="methodname">read</code> in the same basic
|
||
fashion as any other <code class="classname">ItemReader</code>. The paging
|
||
happens behind the scenes when additional entities are needed. Below
|
||
is an example configuration using the same 'customer credit' example
|
||
as the JDBC reader above:</p><pre class="programlisting"><bean id="itemReader" class="org.spr...JpaPagingItemReader">
|
||
<property name="entityManagerFactory" ref="entityManagerFactory"/>
|
||
<property name="queryString" value="select c from CustomerCredit c"/>
|
||
<property name="pageSize" value="1000"/>
|
||
</bean></pre><p>This configured <code class="classname">ItemReader</code> will return
|
||
<code class="classname">CustomerCredit</code> objects in the exact same manner
|
||
as described by the <code class="classname">JdbcPagingItemReader</code> above,
|
||
assuming the Customer object has the correct JPA annotations or ORM
|
||
mapping file. The 'pageSize' property determines the number of
|
||
entities read from the database for each query execution.</p></section><section class="section" title="IbatisPagingItemReader" epub:type="division" id="IbatisPagingItemReader"><div class="titlepage"><div><div><h4 class="title">IbatisPagingItemReader</h4></div></div></div><div class="note" title="Note" epub:type="notice"><table style="border: 0; "><tr><td style="text-align: center; vertical-align: top; width: 25; " rowspan="2"><img alt="[Note]" src="images/note.png"/></td><th style="text-align: left; ">Note</th></tr><tr><td style="text-align: left; vertical-align: top; ">This reader is deprecated as of Spring Batch 3.0.</td></tr></table></div><p>If you use IBATIS for your data access then you can use the
|
||
<code class="classname">IbatisPagingItemReader</code> which, as the name
|
||
indicates, is an implementation of a paging
|
||
<code class="classname">ItemReader</code>. IBATIS doesn't have direct support
|
||
for reading rows in pages but by providing a couple of standard
|
||
variables you can add paging support to your IBATIS queries.</p><p>Here is an example of a configuration for a
|
||
<code class="classname">IbatisPagingItemReader</code> reading CustomerCredits
|
||
as in the examples above:</p><pre class="programlisting"><bean id="itemReader" class="org.spr...IbatisPagingItemReader">
|
||
<property name="sqlMapClient" ref="sqlMapClient"/>
|
||
<property name="queryId" value="getPagedCustomerCredits"/>
|
||
<property name="pageSize" value="1000"/>
|
||
</bean></pre><p>The <code class="classname">IbatisPagingItemReader</code> configuration
|
||
above references an IBATIS query called "getPagedCustomerCredits".
|
||
Here is an example of what that query should look like for
|
||
MySQL.</p><pre class="programlisting"><select id="getPagedCustomerCredits" resultMap="customerCreditResult">
|
||
select id, name, credit from customer order by id asc LIMIT #_skiprows#, #_pagesize#
|
||
</select></pre><p>The <code class="classname">_skiprows</code> and
|
||
<code class="classname">_pagesize</code> variables are provided by the
|
||
<code class="classname">IbatisPagingItemReader</code> and there is also a
|
||
<code class="classname">_page</code> variable that can be used if necessary.
|
||
The syntax for the paging queries varies with the database used. Here
|
||
is an example for Oracle (unfortunately we need to use CDATA for some
|
||
operators since this belongs in an XML document):</p><pre class="programlisting"><select id="getPagedCustomerCredits" resultMap="customerCreditResult">
|
||
select * from (
|
||
select * from (
|
||
select t.id, t.name, t.credit, ROWNUM ROWNUM_ from customer t order by id
|
||
)) where ROWNUM_ <![CDATA[ > ]]> ( #_page# * #_pagesize# )
|
||
) where ROWNUM <![CDATA[ <= ]]> #_pagesize#
|
||
</select></pre></section></section><section class="section" title="Database ItemWriters" epub:type="division" id="databaseItemWriters"><div class="titlepage"><div><div><h3 class="title">Database ItemWriters</h3></div></div></div><p>While both Flat Files and XML have specific ItemWriters, there is
|
||
no exact equivalent in the database world. This is because transactions
|
||
provide all the functionality that is needed. ItemWriters are necessary
|
||
for files because they must act as if they're transactional, keeping
|
||
track of written items and flushing or clearing at the appropriate
|
||
times. Databases have no need for this functionality, since the write is
|
||
already contained in a transaction. Users can create their own DAOs that
|
||
implement the <code class="classname">ItemWriter</code> interface or use one
|
||
from a custom <code class="classname">ItemWriter</code> that's written for
|
||
generic processing concerns, either way, they should work without any
|
||
issues. One thing to look out for is the performance and error handling
|
||
capabilities that are provided by batching the outputs. This is most
|
||
common when using hibernate as an <code class="classname">ItemWriter</code>, but
|
||
could have the same issues when using Jdbc batch mode. Batching database
|
||
output doesn't have any inherent flaws, assuming we are careful to flush
|
||
and there are no errors in the data. However, any errors while writing
|
||
out can cause confusion because there is no way to know which individual
|
||
item caused an exception, or even if any individual item was
|
||
responsible, as illustrated below:</p><div style="text-align: center; " class="mediaobject"><img style="text-align: middle; " src="images/errorOnFlush.png" width="513"/></div><p>If items are buffered before being written out, any
|
||
errors encountered will not be thrown until the buffer is flushed just
|
||
before a commit. For example, let's assume that 20 items will be written
|
||
per chunk, and the 15th item throws a DataIntegrityViolationException.
|
||
As far as the Step is concerned, all 20 item will be written out
|
||
successfully, since there's no way to know that an error will occur
|
||
until they are actually written out. Once
|
||
<code class="classname">Session#</code><code class="methodname">flush</code>() is
|
||
called, the buffer will be emptied and the exception will be hit. At
|
||
this point, there's nothing the <code class="classname">Step</code> can do, the
|
||
transaction must be rolled back. Normally, this exception might cause
|
||
the Item to be skipped (depending upon the skip/retry policies), and
|
||
then it won't be written out again. However, in the batched scenario,
|
||
there's no way for it to know which item caused the issue, the whole
|
||
buffer was being written out when the failure happened. The only way to
|
||
solve this issue is to flush after each item:</p><div style="text-align: center; " class="mediaobject"><img style="text-align: middle; " src="images/errorOnWrite.png" width="513"/></div><p>This is a common use case, especially when using Hibernate, and
|
||
the simple guideline for implementations of
|
||
<code class="classname">ItemWriter</code>, is to flush on each call to
|
||
<code class="methodname">write()</code>. Doing so allows for items to be
|
||
skipped reliably, with Spring Batch taking care internally of the
|
||
granularity of the calls to <code class="classname">ItemWriter</code> after an
|
||
error.</p></section></section><footer/></body></html> |