From 96a317c52fe6ae3d0c9d4c73a4049ff3b919da9d Mon Sep 17 00:00:00 2001 From: Jay Bryant Date: Fri, 6 Oct 2017 10:25:52 -0500 Subject: [PATCH] Editing pass for readersAndWriters.adoc I improved readability and consistency, fixed up errors, and asked a couple questions --- .../asciidoc/readersAndWriters.adoc | 1166 +++++++++-------- 1 file changed, 604 insertions(+), 562 deletions(-) diff --git a/spring-batch-docs/asciidoc/readersAndWriters.adoc b/spring-batch-docs/asciidoc/readersAndWriters.adoc index c4ddc2eb7..7f3f52afb 100644 --- a/spring-batch-docs/asciidoc/readersAndWriters.adoc +++ b/spring-batch-docs/asciidoc/readersAndWriters.adoc @@ -12,7 +12,7 @@ All batch processing can be described in its most simple form as reading in large amounts of data, performing some type of calculation or transformation, and writing the result out. Spring Batch provides three key interfaces to help perform bulk reading and writing: - `ItemReader`, `ItemProcessor` and + `ItemReader`, `ItemProcessor`, and `ItemWriter`. [[itemReader]] @@ -22,29 +22,29 @@ Although a simple concept, an `ItemReader` is the means for providing data from many different types of input. The most general examples include: -* Flat File- Flat File Item Readers read lines of data from a -flat file that typically describe records with fields of data +* Flat File: Flat-file item readers read lines of data from a +flat file that typically describes records with fields of data defined by fixed positions in the file or delimited by some special -character (e.g. Comma). +character (such as a comma). -* XML - XML `ItemReaders` process XML independently of +* XML: XML `ItemReaders` process XML independently of technologies used for parsing, mapping and validating objects. Input data allows for the validation of an XML file against an XSD schema. -* Database - A database resource is accessed to return +* Database: A database resource is accessed to return resultsets which can be mapped to objects for processing. The -default SQL `ItemReaders` invoke a `RowMapper` to +default SQL `ItemReader` implementations invoke a `RowMapper` to return objects, keep track of the current row if restart is required, store basic statistics, and provide some transaction -enhancements that will be explained later. +enhancements that are explained later. -There are many more possibilities, but we'll focus on the -basic ones for this chapter. A complete list of all available `ItemReaders` +There are many more possibilities, but we focus on the +basic ones for this chapter. A complete list of all available `ItemReader` implementations can be found in Appendix A. `ItemReader` is a basic interface for generic -input operations: +input operations, as shown in the following interface definition: [source, java] @@ -57,20 +57,20 @@ public interface ItemReader { ---- The read method defines the most essential -contract of the `ItemReader`; calling it returns one -Item or null if no more items are left. An item might represent a line in +contract of the `ItemReader`. Calling it returns one +item or `null` if no more items are left. An item might represent a line in a file, a row in a database, or an element in an XML file. It is generally -expected that these will be mapped to a usable domain object (i.e. Trade, -Foo, etc) but there is no requirement in the contract to do so. +expected that these are mapped to a usable domain object (such as `Trade`, +`Foo`, or others), but there is no requirement in the contract to do so. It is expected that implementations of the -`ItemReader` interface will be forward only. However, +`ItemReader` interface are forward only. However, if the underlying resource is transactional (such as a JMS queue) then -calling read may return the same logical item on subsequent calls in a +calling `read` may return the same logical item on subsequent calls in a rollback scenario. It is also worth noting that a lack of items to process -by an `ItemReader` will not cause an exception to be +by an `ItemReader` does not cause an exception to be thrown. For example, a database `ItemReader` that is -configured with a query that returns 0 results will simply return null on +configured with a query that returns 0 results returns `null` on the first invocation of read. [[itemWriter]] @@ -79,15 +79,15 @@ the first invocation of read. === ItemWriter `ItemWriter` is similar in functionality to an -`ItemReader`, but with inverse operations. Resources -still need to be located, opened and closed but they differ in that an +`ItemReader` but with inverse operations. Resources +still need to be located, opened, and closed but they differ in that an `ItemWriter` writes out, rather than reading in. In -the case of databases or queues these may be inserts, updates, or sends. +the case of databases or queues, these operations may be inserts, updates, or sends. The format of the serialization of the output is specific to each batch job. As with `ItemReader`, -`ItemWriter` is a fairly generic interface: +`ItemWriter` is a fairly generic interface, as shown in the following interface definition: [source, java] @@ -101,14 +101,14 @@ public interface ItemWriter { As with `read` on `ItemReader`, `write` provides -the basic contract of `ItemWriter`; it will attempt +the basic contract of `ItemWriter`. It attempts to write out the list of items passed in as long as it is open. Because it -is generally expected that items will be 'batched' together into a chunk +is generally expected that items are 'batched' together into a chunk and then output, the interface accepts a list of items, rather than an item by itself. After writing out the list, any flushing that may be necessary can be performed before returning from the write method. For example, if writing to a Hibernate DAO, multiple calls to write can be -made, one for each item. The writer can then call close on the hibernate +made, one for each item. The writer can then call `close` on the hibernate Session before returning. [[itemProcessor]] @@ -119,9 +119,9 @@ Session before returning. The `ItemReader` and `ItemWriter` interfaces are both very useful for their specific tasks, but what if you want to insert business logic before writing? One option for both reading and writing is to use the composite -pattern: create an `ItemWriter` that contains another -`ItemWriter`, or an `ItemReader` -that contains another `ItemReader`. For +pattern: Create an `ItemWriter` that contains another +`ItemWriter` or an `ItemReader` +that contains another `ItemReader`. The following code shows an example: @@ -146,17 +146,17 @@ public class CompositeItemWriter implements ItemWriter { } ---- -The class above contains another `ItemWriter` +The preceding class contains another `ItemWriter` to which it delegates after having provided some business logic. This pattern could easily be used for an `ItemReader` as well, perhaps to obtain more reference data based upon the input that was provided by the main `ItemReader`. It is also useful if you need to control the call to `write` yourself. However, if you only want to 'transform' the item passed in for writing -before it is actually written, there isn't much need to call -`write` yourself: you just want to modify the item. +before it is actually written, you need not +`write` yourself. You can just modify the item. For this scenario, Spring Batch provides the -`ItemProcessor` interface: +`ItemProcessor` interface, as shown in the following interface definition: [source, java] @@ -167,14 +167,14 @@ public interface ItemProcessor { } ---- -An `ItemProcessor` is very simple; given one +An `ItemProcessor` is simple. Given one object, transform it and return another. The provided object may or may not be of the same type. The point is that business logic may be applied -within process, and is completely up to the developer to create. An -`ItemProcessor` can be wired directly into a step, -For example, assuming an `ItemReader` provides a -class of type Foo, and it needs to be converted to type Bar before being -written out. An `ItemProcessor` can be written that +within the process, and it is completely up to the developer to create that logic. An +`ItemProcessor` can be wired directly into a step. +For example, assume an `ItemReader` provides a +class of type `Foo` and that it needs to be converted to type Bar before being +written out. The following example shows an `ItemProcessor` that performs the conversion: @@ -200,17 +200,17 @@ public class BarWriter implements ItemWriter{ } ---- -In the very simple example above, there is a class +In the preceding example, there is a class `Foo`, a class `Bar`, and a class `FooProcessor` that adheres to the `ItemProcessor` interface. The transformation is simple, but any type of transformation could be done here. The -`BarWriter` will be used to write out +`BarWriter` writes `Bar` objects, throwing an exception if any other -type is provided. Similarly, the `FooProcessor` will -throw an exception if anything but a `Foo` is +type is provided. Similarly, the `FooProcessor` +throws an exception if anything but a `Foo` is provided. The `FooProcessor` can then be injected -into a `Step`: +into a `Step`, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -254,11 +254,11 @@ public Step step1() { Performing a single transformation is useful in many scenarios, but what if you want to 'chain' together multiple -`ItemProcessors`? This can be accomplished using +`ItemProcessor` implementations? This can be accomplished using the composite pattern mentioned previously. To update the previous, -single transformation, example, `Foo` will be -transformed to `Bar`, which will be transformed to -`Foobar` and written out: +single transformation, example, `Foo` is +transformed to `Bar`, which is transformed to +`Foobar` and written out, as shown in the following example: [source, java] @@ -293,9 +293,9 @@ public class FoobarWriter implements ItemWriter{ } ---- -A `FooProcessor` and +A `FooProcessor` and a `BarProcessor` can be 'chained' together to give - the resultant `Foobar`: + the resultant `Foobar`, as shown in the following example: [source, java] @@ -376,8 +376,8 @@ public CompositeItemProcessor compositeProcessor() { One typical use for an item processor is to filter out records before they are passed to the `ItemWriter`. Filtering is an action -distinct from skipping; skipping indicates that a record is invalid -whereas filtering simply indicates that a record should not be +distinct from skipping. Skipping indicates that a record is invalid, +while filtering simply indicates that a record should not be written. For example, consider a batch job that reads a file containing @@ -385,16 +385,16 @@ three different types of records: records to insert, records to update, and records to delete. If record deletion is not supported by the system, then we would not want to send any "delete" records to the `ItemWriter`. But, since these records are not -actually bad records, we would want to filter them out, rather than -skip. As a result, the `ItemWriter` would receive only "insert" and +actually bad records, we would want to filter them out rather than +skip them. As a result, the `ItemWriter` would receive only "insert" and "update" records. -To filter a record, one simply returns "null" from the -`ItemProcessor`. The framework will detect that the -result is "null" and avoid adding that item to the list of records +To filter a record, you can return `null` from the +`ItemProcessor`. The framework detects that the +result is `null` and avoids adding that item to the list of records delivered to the `ItemWriter`. As usual, an -exception thrown from the `ItemProcessor` will -result in a skip. +exception thrown from the `ItemProcessor` +results in a skip. [[faultTolerant]] @@ -403,7 +403,7 @@ result in a skip. When a chunk is rolled back, items that have been cached during reading may be reprocessed. If a step is configured to - be fault tolerant (uses skip or retry processing typically), + be fault tolerant (typically by using skip or retry processing), any `ItemProcessor` used should be implemented in a way that is idempotent. Typically that would consist of performing no changes on the input item for the `ItemProcessor` and only updating the @@ -418,7 +418,7 @@ Both `ItemReaders` and but there is a common concern among both of them that necessitates another interface. In general, as part of the scope of a batch job, readers and writers need to be opened, closed, and require a mechanism for persisting -state: +state. The `ItemStream` interface serves that purpose, as shown in the following example: [source, java] @@ -435,20 +435,20 @@ public interface ItemStream { Before describing each method, we should mention the `ExecutionContext`. Clients of an `ItemReader` that also implement -ItemStream should call +`ItemStream` should call `open` before any calls to -`read` in order to open any resources such as files +`read`, in order to open any resources such as files or to obtain connections. A similar restriction applies to an `ItemWriter` that implements -ItemStream. As mentioned in Chapter 2, if expected +`ItemStream`. As mentioned in Chapter 2, if expected data is found in the `ExecutionContext`, it may be used to start the `ItemReader` or `ItemWriter` at a location other than its initial -state. Conversely, `close` will be called to ensure -that any resources allocated during open will be +state. Conversely, `close` is called to ensure +that any resources allocated during open are released safely. `update` is called primarily to ensure that any state currently being held is loaded into the provided -`ExecutionContext`. This method will be called before +`ExecutionContext`. This method is called before committing, to ensure that the current state is persisted in the database before commit. @@ -457,7 +457,7 @@ In the special case where the client of an the Spring Batch Core), an `ExecutionContext` is created for each StepExecution to allow users to store the state of a particular execution, with the expectation that it -will be returned if the same `JobInstance` is started +is returned if the same `JobInstance` is started again. For those familiar with Quartz, the semantics are very similar to a Quartz `JobDataMap`. @@ -467,16 +467,16 @@ Quartz `JobDataMap`. Note that the `CompositeItemWriter` is an example of the delegation pattern, which is common in Spring Batch. The - delegates themselves might implement callback interfaces `StepListener`. - If they do, and they are being used in conjunction with Spring Batch Core + delegates themselves might implement callback interfaces, such as `StepListener`. + If they do and if they are being used in conjunction with Spring Batch Core as part of a `Step` in a `Job`, then they almost certainly need to be registered manually with the `Step`. A reader, writer, or processor that is - directly wired into the `Step` will be registered automatically if it + directly wired into the `Step` gets registered automatically if it implements `ItemStream` or a - `StepListener` interface. But because the delegates + `StepListener` interface. However, because the delegates are not known to the `Step`, they need to be injected - as listeners or streams (or both if appropriate): + as listeners or streams (or both if appropriate), as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -547,7 +547,7 @@ One of the most common mechanisms for interchanging bulk data has always been the flat file. Unlike XML, which has an agreed upon standard for defining how it is structured (XSD), anyone reading a flat file must understand ahead of time exactly how the file is structured. In general, -all flat files fall into two types: Delimited and Fixed Length. Delimited +all flat files fall into two types: delimited and fixed length. Delimited files are those in which fields are separated by a delimiter, such as a comma. Fixed Length files have fields that are a set length. @@ -559,16 +559,16 @@ When working with flat files in Spring Batch, regardless of whether it is for input or output, one of the most important classes is the `FieldSet`. Many architectures and libraries contain abstractions for helping you read in from a file, but they -usually return a String or an array of Strings. This really only gets +usually return a `String` or an array of `String` objects. This really only gets you halfway there. A `FieldSet` is Spring Batch's abstraction for enabling the binding of fields from a file resource. It allows developers to work with file input in much the same way as they would work with database input. A `FieldSet` is -conceptually very similar to a Jdbc ResultSet. -`FieldSets` only require one argument, a String -array of tokens. Optionally, you can also configure in the names of the +conceptually similar to a JDBC `ResultSet`. +`FieldSets` only require one argument: a `String` +array of tokens. Optionally, you can also configure the names of the fields so that the fields may be accessed either by index or name as -patterned after ResultSet: +patterned after ResultSet, as shown in the following example: [source, java] @@ -582,7 +582,7 @@ boolean booleanValue = fs.readBoolean(2); There are many more options on the `FieldSet` interface, such as `Date`, long, -`BigDecimal`, etc. The biggest advantage of the +`BigDecimal`, and so on. The biggest advantage of the `FieldSet` is that it provides consistent parsing of flat file input. Rather than each batch job parsing differently in potentially unexpected ways, it can be consistent, both when handling @@ -595,32 +595,31 @@ conversions. A flat file is any type of file that contains at most two-dimensional (tabular) data. Reading flat files in the Spring Batch -framework is facilitated by the class +framework is facilitated by the class called `FlatFileItemReader`, which provides basic functionality for reading and parsing flat files. The two most important required dependencies of `FlatFileItemReader` are `Resource` and `LineMapper`. -The `LineMapper` interface will be +The `LineMapper` interface is explored more in the next sections. The resource property represents a Spring Core `Resource`. Documentation explaining how to create beans of this type can be found in link:$$http://docs.spring.io/spring/docs/3.2.x/spring-framework-reference/html/resources.html$$[Spring -Framework, Chapter 5.Resources]. Therefore, this -guide will not go into the details of creating -`Resource` objects. However, a simple example of a -file system resource can be found below: +Framework, Chapter 5. Resources]. Therefore, this +guide does not go into the details of creating +`Resource` objects beyond showing the following simple example: [source, java] ---- Resource resource = new FileSystemResource("resources/trades.csv"); ---- -In complex batch environments the directory structures are often - managed by the EAI infrastructure where drop zones for external - interfaces are established for moving files from ftp locations to batch +In complex batch environments, the directory structures are often + managed by the EAI infrastructure, where drop zones for external + interfaces are established for moving files from FTP locations to batch processing locations and vice versa. File moving utilities are beyond - the scope of the spring batch architecture but it is not unusual for + the scope of the Spring Batch architecture, but it is not unusual for batch job streams to include file moving utilities as steps in the job - stream. It is sufficient that the batch architecture only needs to know + stream. The batch architecture only needs to know how to locate the files to be processed. Spring Batch begins the process of feeding the data into the pipe from this starting point. However, link:$$http://projects.spring.io/spring-integration/$$[Spring @@ -628,30 +627,31 @@ In complex batch environments the directory structures are often services. The other properties in `FlatFileItemReader` -allow you to further specify how your data will be interpreted: `FlatFileItemReader` Properties +let you further specify how your data is interpreted, as described in the following table: +.`FlatFileItemReader` Properties [options="header"] |=============== |Property|Type|Description |comments|String[]|Specifies line prefixes that indicate - comment rows -|encoding|String|Specifies what text encoding to use - - default is "ISO-8859-1" + comment rows. +|encoding|String|Specifies what text encoding to use. The + default is "ISO-8859-1". |lineMapper|`LineMapper`|Converts a `String` to an `Object` representing the item. |linesToSkip|int|Number of lines to ignore at the top of - the file + the file. |recordSeparatorPolicy|RecordSeparatorPolicy|Used to determine where the line endings are and do things like continue over a line ending if inside a quoted string. |resource|`Resource`|The resource from which to read. -|skippedLinesCallback|LineCallbackHandler|Interface which passes the raw line - content of the lines in the file to be skipped. If linesToSkip - is set to 2, then this interface will be called twice. -|strict|boolean|In strict mode, the reader will throw an +|skippedLinesCallback|LineCallbackHandler|Interface that passes the raw line + content of the lines in the file to be skipped. If `linesToSkip` + is set to 2, then this interface is called twice. +|strict|boolean|In strict mode, the reader throws an exception on `ExecutionContext` if the input resource does not - exist. + exist. Otherwise, it logs the problem and continues. |=============== @@ -660,11 +660,10 @@ allow you to further specify how your data will be interpreted: `FlatFileItemRea ===== LineMapper -As with `RowMapper`, which takes a low -level construct such as ResultSet and returns -an Object, flat file processing requires the -same construct to convert a String line into an -Object: +As with `RowMapper`, which takes a low-level construct such as `ResultSet` and returns +an `Object`, flat file processing requires the +same construct to convert a `String` line into an +`Object`, as shown in the following interface definition: @@ -680,23 +679,23 @@ public interface LineMapper { The basic contract is that, given the current line and the line number with which it is associated, the mapper should return a resulting domain object. This is similar to -`RowMapper` in that each line is associated with +`RowMapper`, in that each line is associated with its line number, just as each row in a -ResultSet is tied to its row number. This +`ResultSet` is tied to its row number. This allows the line number to be tied to the resulting domain object for identity comparison or for more informative logging. However, unlike `RowMapper`, the `LineMapper` is given a raw line which, as discussed above, only gets you halfway there. The line must be tokenized into a `FieldSet`, which can then be -mapped to an object, as described below. +mapped to an object, as described later in this document. [[lineTokenizer]] ===== LineTokenizer -An abstraction for turning a line of input into a line into a +An abstraction for turning a line of input into a `FieldSet` is necessary because there can be many formats of flat file data that need to be converted to a `FieldSet`. In Spring Batch, this interface is @@ -715,25 +714,25 @@ public interface LineTokenizer { The contract of a `LineTokenizer` is such that, given a line of input (in theory the `String` could encompass more than one line), a -`FieldSet` representing the line will be +`FieldSet` representing the line is returned. This `FieldSet` can then be passed to a `FieldSetMapper`. Spring Batch contains the following `LineTokenizer` implementations: -* `DelmitedLineTokenizer` - Used for +* `DelmitedLineTokenizer`: Used for files where fields in a record are separated by a delimiter. The most common delimiter is a comma, but pipes or semicolons are often used as well. -* `FixedLengthTokenizer` - Used for files +* `FixedLengthTokenizer`: Used for files where fields in a record are each a 'fixed width'. The width of each field must be defined for each record type. -* `PatternMatchingCompositeLineTokenizer` -- Determines which among a list of +* `PatternMatchingCompositeLineTokenizer`: +Determines which among a list of `LineTokenizers` should be used on a particular line by checking against a pattern. @@ -744,11 +743,11 @@ particular line by checking against a pattern. The `FieldSetMapper` interface defines a single method, `mapFieldSet`, which takes a `FieldSet` object and maps its contents to an -object. This object may be a custom DTO, a domain object, or a simple +object. This object may be a custom DTO, a domain object, or an array, depending on the needs of the job. The `FieldSetMapper` is used in conjunction with the `LineTokenizer` to translate a line of data from -a resource into an object of the desired type: +a resource into an object of the desired type, as shown in the following interface definition: [source, java] @@ -775,8 +774,8 @@ required: . Read one line from the file. -. Pass the string line into the `LineTokenizer#tokenize()` method, in -order to retrieve a `FieldSet`. +. Pass the `String` line into the `LineTokenizer#tokenize()` method +to retrieve a `FieldSet`. . Pass the `FieldSet` returned from tokenizing to a `FieldSetMapper`, returning @@ -785,16 +784,16 @@ the result from the `ItemReader#read()` method. The two interfaces described above represent two separate tasks: -converting a line into a `FieldSet`, and mapping +converting a line into a `FieldSet` and mapping a `FieldSet` to a domain object. Because the input of a `LineTokenizer` matches the input of the `LineMapper` (a line), and the output of a `FieldSetMapper` matches the output of the `LineMapper`, a default implementation that uses -both a `LineTokenizer` and +both a `LineTokenizer` and a `FieldSetMapper` is provided. The -`DefaultLineMapper` represents the behavior most -users will need: +`DefaultLineMapper`, shown in the following class definition, represents the behavior most +users need: [source, java] @@ -822,7 +821,7 @@ public class DefaultLineMapper; implements LineMapper<>, InitializingBean { The above functionality is provided in a default implementation, rather than being built into the reader itself (as was done in -previous versions of the framework) in order to allow users greater +previous versions of the framework) to allow users greater flexibility in controlling the parsing process, especially if access to the raw line is needed. @@ -830,7 +829,7 @@ to the raw line is needed. ===== Simple Delimited File Reading Example -The following example will be used to illustrate this using an +The following example illustrates how to read a flat file with an actual domain scenario. This particular batch job reads in football players from the following file: @@ -843,10 +842,10 @@ ID,lastName,firstName,position,birthYear,debutYear "AberWa00,Abercrombie,Walter,rb,1959,1982", "AbraDa00,Abramowicz,Danny,wr,1945,1967", "AdamBo00,Adams,Bob,te,1946,1969", -"AdamCh00,Adams,Charlie,wr,1979,2003" +"AdamCh00,Adams,Charlie,wr,1979,2003" ---- -The contents of this file will be mapped to the following +The contents of this file are mapped to the following `Player` domain object: [source, java] @@ -871,10 +870,10 @@ public class Player implements Serializable { } ---- -In order to map a `FieldSet` into a +To map a `FieldSet` into a `Player` object, a `FieldSetMapper` that returns players needs to be - defined: + defined, as shown in the following example: [source, java] @@ -897,7 +896,7 @@ protected static class PlayerFieldSetMapper implements FieldSetMapper { The file can then be read by correctly constructing a `FlatFileItemReader` and calling - read: + `read`, as shown in the following example: [source, java] @@ -913,9 +912,9 @@ itemReader.open(new ExecutionContext()); Player player = itemReader.read(); ---- -Each call to read will return a new +Each call to `read` returns a new `Player` object from each line in the file. When the end of the file is - reached, null will be returned. + reached, `null` is returned. [[mappingFieldsByName]] @@ -923,17 +922,17 @@ Each call to read will return a new There is one additional piece of functionality that is allowed by both `DelimitedLineTokenizer` and -`FixedLengthTokenizer` that is similar in -function to a Jdbc `ResultSet`. The names of the +`FixedLengthTokenizer` and that is similar in +function to a JDBC `ResultSet`. The names of the fields can be injected into either of these `LineTokenizer` implementations to increase the readability of the mapping function. First, the column names of all -fields in the flat file are injected into the tokenizer: +fields in the flat file are injected into the tokenizer, as shown in the following example: [source, java] ---- -tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); +tokenizer.setNames(new String[] {"ID", "lastName","firstName","position","birthYear","debutYear"}); ---- A `FieldSetMapper` can use this information @@ -1009,12 +1008,12 @@ public Player player() { ---- For each entry in the `FieldSet`, the -mapper will look for a corresponding setter on a new instance of the +mapper looks for a corresponding setter on a new instance of the `Player` object (for this reason, prototype scope -is required) in the same way the Spring container will look for +is required) in the same way the Spring container looks for setters matching a property name. Each available field in the -`FieldSet` will be mapped, and the resultant -`Player` object will be returned, with no code +`FieldSet` is mapped, and the resultant +`Player` object is returned, with no code required. [[fixedLengthFileFormats]] @@ -1022,10 +1021,10 @@ required. ===== Fixed Length File Formats -So far only delimited files have been discussed in much detail, - however, they represent only half of the file reading picture. Many +So far, only delimited files have been discussed in much detail. + However, they represent only half of the file reading picture. Many organizations that use flat files use fixed length formats. An example - fixed length file is below: + fixed length file follows: ---- @@ -1039,19 +1038,19 @@ UK21341EAH4521535.11customer5 While this looks like one large field, it actually represent 4 distinct fields: -. ISIN: Unique identifier for the item being order - 12 characters long. +. ISIN: Unique identifier for the item being ordered - 12 characters long. -. Quantity: Number of this item being ordered - 3 characters long. +. Quantity: Number of the item being ordered - 3 characters long. . Price: Price of the item - 5 characters long. -. Customer: Id of the customer ordering the item - 9 characters long. +. Customer: ID of the customer ordering the item - 9 characters long. When configuring the `FixedLengthLineTokenizer`, each of these lengths -must be provided in the form of ranges: +must be provided in the form of ranges, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -1064,6 +1063,14 @@ must be provided in the form of ranges: ---- [role="xmlContent"] +Because the `FixedLengthLineTokenizer` uses +the same `LineTokenizer` interface as discussed +above, it returns the same `FieldSet` as if a +delimiter had been used. This allows the same approaches to be used in +handling its output, such as using the +`BeanWrapperFieldSetMapper`. + + [NOTE] ==== Supporting the above syntax for ranges requires that a @@ -1117,19 +1124,19 @@ LINEB;2134776319DEF422.99M005LI ---- In this file we have three types of records, "USER", "LINEA", -and "LINEB". A "USER" line corresponds to a User object. "LINEA" and -"LINEB" both correspond to Line objects, though a "LINEA" has more +and "LINEB". A "USER" line corresponds to a `User` object. "LINEA" and +"LINEB" both correspond to `Line` objects, though a "LINEA" has more information than a "LINEB". -The `ItemReader` will read each line +The `ItemReader` reads each line individually, but we must specify different `LineTokenizer` and `FieldSetMapper` objects so that the -`ItemWriter` will receive the correct items. The +`ItemWriter` receives the correct items. The `PatternMatchingCompositeLineMapper` makes this easy by allowing maps of patterns to `LineTokenizers` and patterns to -`FieldSetMappers` to be configured: +`FieldSetMappers` to be configured, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -1178,7 +1185,7 @@ public PatternMatchingCompositeLineMapper orderFileLineMapper() { ---- In this example, "LINEA" and "LINEB" have separate -`LineTokenizers` but they both use the same +`LineTokenizers`, but they both use the same `FieldSetMapper`. The `PatternMatchingCompositeLineMapper` @@ -1186,16 +1193,16 @@ makes use of the `PatternMatcher's` match method in order to select the correct delegate for each line. The `PatternMatcher` allows for two wildcard characters with special meaning: the question -mark ("?") will match exactly one character, while the asterisk ("\*") -will match zero or more characters. Note that in the configuration +mark ("?") matches exactly one character, while the asterisk ("\*") +matches zero or more characters. Note that, in the configuration above, all patterns end with an asterisk, making them effectively -prefixes to lines. The `PatternMatcher` will -always match the most specific pattern possible, regardless of the +prefixes to lines. The `PatternMatcher` +always matches the most specific pattern possible, regardless of the order in the configuration. So if "LINE*" and "LINEA*" were both listed as patterns, "LINEA" would match pattern "LINEA*", while "LINEB" would match pattern "LINE*". Additionally, a single asterisk ("*") can serve as a default by matching any line not matched by any -other pattern. +other pattern, as shown in the following example. .XML Configuration [source, xml, role="xmlContent"] @@ -1227,15 +1234,15 @@ It is also common for a flat file to contain records that each There are many scenarios when tokenizing a line may cause exceptions to be thrown. Many flat files are imperfect and contain -records that aren't formatted correctly. Many users choose to skip -these erroneous lines, logging out the issue, original line, and line +incorrectly formatted records. Many users choose to skip +these erroneous lines while logging the issue, the original line, and the line number. These logs can later be inspected manually or by another batch job. For this reason, Spring Batch provides a hierarchy of exceptions for handling parse exceptions: `FlatFileParseException` and `FlatFileFormatException`. `FlatFileParseException` is thrown by the `FlatFileItemReader` when any errors are encountered while trying to read a file. `FlatFileFormatException` is thrown by -implementations of the `LineTokenizer` interface, +implementations of the `LineTokenizer` interface and indicates a more specific error encountered while tokenizing. @@ -1247,11 +1254,11 @@ Both `DelimitedLineTokenizer` and `FixedLengthLineTokenizer` have the ability to specify column names that can be used for creating a `FieldSet`. However, if the number of column -names doesn't match the number of columns found while tokenizing a -line the `FieldSet` can't be created, and a +names does not match the number of columns found while tokenizing a +line, the `FieldSet` can't be created, and an `IncorrectTokenCountException` is thrown, which contains the number of tokens encountered, and the number -expected: +expected, as shown in the following example: [source, java] @@ -1267,7 +1274,7 @@ catch(IncorrectTokenCountException e){ } ---- -Because the tokenizer was configured with 4 column names, but +Because the tokenizer was configured with 4 column names but only 3 tokens were found in the file, an `IncorrectTokenCountException` was thrown. @@ -1276,11 +1283,11 @@ thrown. ====== IncorrectLineLengthException -Files formatted in a fixed length format have additional +Files formatted in a fixed-length format have additional requirements when parsing because, unlike a delimited format, each column must strictly adhere to its predefined width. If the total -line length doesn't add up to the widest value of this column, an -exception is thrown: +line length does not equal the widest value of this column, an +exception is thrown, as shown in the following example: [source, java] @@ -1299,16 +1306,16 @@ catch (IncorrectLineLengthException ex) { ---- The configured ranges for the tokenizer above are: 1-5, 6-10, -and 11-15, thus the total length of the line expected is 15. -However, in this case a line of length 5 was passed in, causing an +and 11-15. Consequently, the total length of the line is 15. +However, in the preceding example, a line of length 5 was passed in, causing an `IncorrectLineLengthException` to be thrown. Throwing an exception here rather than only mapping the first column -allows the processing of the line to fail earlier, and with more -information than it would if it failed while trying to read in +allows the processing of the line to fail earlier and with more +information than it would contain if it failed while trying to read in column 2 in a `FieldSetMapper`. However, there -are scenarios where the length of the line isn't always constant. +are scenarios where the length of the line is not always constant. For this reason, validation of line length can be turned off via the -'strict' property: +'strict' property, as shown in the following example: [source, java] ---- @@ -1319,11 +1326,11 @@ assertEquals("12345", tokens.readString(0)); assertEquals("", tokens.readString(1)); ---- -The above example is almost identical to the one before it, +The preceding example is almost identical to the one before it, except that tokenizer.setStrict(false) was called. This setting tells the tokenizer to not enforce line lengths when tokenizing the line. A `FieldSet` is now correctly created and -returned. However, it will only contain empty tokens for the +returned. However, it contains only empty tokens for the remaining values. [[flatFileItemWriter]] @@ -1331,8 +1338,8 @@ remaining values. ==== FlatFileItemWriter Writing out to flat files has the same problems and issues that -reading in from a file must overcome. A step must be able to write out -in either delimited or fixed length formats in a transactional +reading in from a file must overcome. A step must be able to write +either delimited or fixed length formats in a transactional manner. [[lineAggregator]] @@ -1341,10 +1348,10 @@ manner. Just as the `LineTokenizer` interface is necessary to take an item and turn it into a -String, file writing must have a way to +`String`, file writing must have a way to aggregate multiple fields into a single string for writing to a file. -In Spring Batch this is the -`LineAggregator`: +In Spring Batch, this is the +`LineAggregator`, shown in the following interface definition: [source, java] @@ -1356,7 +1363,7 @@ public interface LineAggregator { } ---- -The `LineAggregator` is the opposite of a `LineTokenizer`. `LineTokenizer` takes a +The `LineAggregator` is the logical opposite of `LineTokenizer`. `LineTokenizer` takes a `String` and returns a `FieldSet`, whereas `LineAggregator` takes an `item` and returns a `String`. @@ -1366,8 +1373,8 @@ The `LineAggregator` is the opposite of a `LineTokenizer`. `LineTokenizer` take The most basic implementation of the `LineAggregator` interface is the `PassThroughLineAggregator`, which - simply assumes that the object is already a string, or that its - string representation is acceptable for writing: + assumes that the object is already a string or that its + string representation is acceptable for writing, as shown in the following code: [source, java] @@ -1380,8 +1387,8 @@ public class PassThroughLineAggregator implements LineAggregator { } ---- -The above implementation is useful if direct control of -creating the string is required, but the advantages of a +The preceding implementation is useful if direct control of +creating the string is required but the advantages of a `FlatFileItemWriter`, such as transaction and restart support, are necessary. @@ -1394,10 +1401,10 @@ Now that the `LineAggregator` interface and its most basic implementation, . The object to be written is passed to the `LineAggregator` in order to obtain a -String. +`String`. -. The returned String is written to the configured file. +. The returned `String` is written to the configured file. The following excerpt from the `FlatFileItemWriter` expresses this in code: @@ -1409,7 +1416,7 @@ public void write(T item) throws Exception { } ---- -A simple configuration would look like the following: +A simple configuration might look like the following: .XML Configuration [source, xml, role="xmlContent"] @@ -1439,34 +1446,34 @@ public FlatFileItemWriter itemWriter() { ===== FieldExtractor -The above example may be useful for the most basic uses of a +The preceding example may be useful for the most basic uses of a writing to a file. However, most users of the - `FlatFileItemWriter` will have a domain object - that needs to be written out, and thus must be converted into a line. + `FlatFileItemWriter` have a domain object + that needs to be written out and, thus, must be converted into a line. In file reading, the following was required: . Read one line from the file. -. Pass the string line into the `LineTokenizer#tokenize()` method, in -order to retrieve a `FieldSet` +. Pass the line into the `LineTokenizer#tokenize()` method, in +order to retrieve a `FieldSet`. . Pass the `FieldSet` returned from tokenizing to a `FieldSetMapper`, returning -the result from the `ItemReader#read()` method +the result from the `ItemReader#read()` method. File writing has similar, but inverse steps: -. Pass the item to be written to the writer +. Pass the item to be written to the writer. -. convert the fields on the item into an array +. Convert the fields on the item into an array. -. aggregate the resulting array into a line +. Aggregate the resulting array into a line. Because there is no way for the framework to know which fields from the object need to be written out, a `FieldExtractor` must be written to accomplish -the task of turning the item into an array: +the task of turning the item into an array, as shown in the following interface definition: [source, java] @@ -1481,7 +1488,7 @@ public interface FieldExtractor { Implementations of the `FieldExtractor` interface should create an array from the fields of the provided object, which can then be written out with a delimiter between the -elements, or as part of a field-width line. +elements or as part of a fixed-width line. [[PassThroughFieldExtractor]] @@ -1490,12 +1497,12 @@ elements, or as part of a field-width line. There are many cases where a collection, such as an array, `Collection`, or `FieldSet`, needs to be written out. -"Extracting" an array from a one of these collection types is very -straightforward: simply convert the collection to an array. +"Extracting" an array from one of these collection types is very +straightforward. To do so, convert the collection to an array. Therefore, the `PassThroughFieldExtractor` -should be used in this scenario. It should be noted, that if the +should be used in this scenario. It should be noted that, if the object passed in is not a type of collection, then the -`PassThroughFieldExtractor` will return an +`PassThroughFieldExtractor` returns an array containing solely the item to be extracted. [[BeanWrapperFieldExtractor]] @@ -1507,8 +1514,8 @@ As with the `BeanWrapperFieldSetMapper` described in the file reading section, it is often preferable to configure how to convert a domain object to an object array, rather than writing the conversion yourself. The -`BeanWrapperFieldExtractor` provides just this -type of functionality: +`BeanWrapperFieldExtractor` provides this +functionality, as shown in the following example: [source, java] @@ -1528,7 +1535,7 @@ assertEquals(last, values[1]); assertEquals(born, values[2]); ---- -This extractor implementation has only one required property, +This extractor implementation has only one required property: the names of the fields to map. Just as the `BeanWrapperFieldSetMapper` needs field names to map fields on the `FieldSet` to setters on @@ -1544,7 +1551,7 @@ array. The most basic flat file format is one in which all fields are separated by a delimiter. This can be accomplished using a -`DelimitedLineAggregator`. The example below +`DelimitedLineAggregator`. The following example writes out a simple domain object that represents a credit to a customer account: @@ -1563,7 +1570,7 @@ public class CustomerCredit { Because a domain object is being used, an implementation of the `FieldExtractor` interface must be provided, along with the delimiter to -use: +use, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -1604,7 +1611,7 @@ public FlatFileItemWriter itemWriter(Resource outputResource) throws Except } ---- -In this case, the +In the previous example, the `BeanWrapperFieldExtractor` described earlier in this chapter is used to turn the name and credit fields within `CustomerCredit` into an object array, which is @@ -1618,7 +1625,7 @@ then written out with commas between each field. Delimited is not the only type of flat file format. Many prefer to use a set width for each column to delineate between fields, which is usually referred to as 'fixed width'. Spring Batch supports this in -file writing via the `FormatterLineAggregator`. +file writing with the `FormatterLineAggregator`. Using the same `CustomerCredit` domain object described above, it can be configured as follows: @@ -1661,8 +1668,8 @@ public FlatFileItemWriter itemWriter(Resource outputResource) throws Exception { } ---- -Most of the above example should look familiar. However, the -value of the format property is new: +Most of the preceding example should look familiar. However, the +value of the format property is new and is shown in the following element: [source, xml, role="xmlContent"] @@ -1683,7 +1690,7 @@ The underlying implementation is built using the same `Formatter` is based on the `printf` functionality of the C programming language. Most details on how to configure a formatter can be found in -the javadoc of link:$$http://java.sun.com/j2se/1.5.0/docs/api/java/util/Formatter.html$$[Formatter]. +the Javadoc of link:$$https://docs.oracle.com/javase/8/docs/api/java/util/Formatter.html$$[Formatter]. [[handlingFileCreation]] @@ -1691,20 +1698,20 @@ the javadoc of link:$$http://java.sun.com/j2se/1.5.0/docs/api/java/util/Formatte `FlatFileItemReader` has a very simple relationship with file resources. When the reader is initialized, it -opens the file if it exists, and throws an exception if it does not. -File writing isn't quite so simple. At first glance it seems like a -similar straight forward contract should exist for -`FlatFileItemWriter`: if the file already exists, -throw an exception, and if it does not, create it and start writing. +opens the file (if it exists), and throws an exception if it does not. +File writing isn't quite so simple. At first glance, it seems like a +similar straightforward contract should exist for +`FlatFileItemWriter`: If the file already exists, +throw an exception, and, if it does not, create it and start writing. However, potentially restarting a `Job` can cause -issues. In normal restart scenarios, the contract is reversed: if the +issues. In normal restart scenarios, the contract is reversed: If the file exists, start writing to it from the last known good position, -and if it does not, throw an exception. However, what happens if the +and, if it does not, throw an exception. However, what happens if the file name for this job is always the same? In this case, you would want to delete the file if it exists, unless it's a restart. Because of this possibility, the `FlatFileItemWriter` contains the property, `shouldDeleteIfExists`. -Setting this property to true will cause an existing file with the +Setting this property to true causes an existing file with the same name to be deleted when the writer is opened. [[xmlReadingWriting]] @@ -1719,18 +1726,18 @@ objects as XML records. [NOTE] .Constraints on streaming XML ==== -The StAX API is used for I/O as other standard XML parsing APIs do +The StAX API is used for I/O, as other standard XML parsing APIs do not fit batch processing requirements (DOM loads the whole input into -memory at once and SAX controls the parsing process allowing the user -only to provide callbacks). +memory at once and SAX controls the parsing process by allowing the user +to provide only callbacks). ==== -Lets take a closer look how XML input and output works in Spring +We need to consider how XML input and output works in Spring Batch. First, there are a few concepts that vary from file reading and writing but are common across Spring Batch XML processing. With XML processing, instead of lines of records (`FieldSets`) that need to be tokenized, it is assumed an XML resource is a collection of 'fragments' -corresponding to individual records: +corresponding to individual records, as shown in the following image: .XML Input image::{batch-asciidoc}images/xmlinput.png[XML Input, scaledwidth="60%"] @@ -1745,14 +1752,14 @@ particular XML binding technology. Typical use is to delegate to link:$$http://d OXM], which provides uniform abstraction for the most popular OXM technologies. The dependency on Spring OXM is optional and you can choose to implement Spring Batch specific interfaces if desired. The -relationship to the technologies that OXM supports can be shown as the -following: +relationship to the technologies that OXM supports is shown in the +following image: .OXM Binding image::{batch-asciidoc}images/oxm-fragments.png[OXM Binding, scaledwidth="60%"] -Now with an introduction to OXM and how one can use XML fragments to -represent records, let's take a closer look at readers and writers. +With an introduction to OXM and how one can use XML fragments to +represent records, we can now more closely examine readers and writers. [[StaxEventItemReader]] @@ -1760,8 +1767,8 @@ represent records, let's take a closer look at readers and writers. The `StaxEventItemReader` configuration provides a typical setup for the processing of records from an XML input -stream. First, lets examine a set of XML records that the -`StaxEventItemReader` can process. +stream. First, consider the following set of XML records that the +`StaxEventItemReader` can process: [source, xml] @@ -1789,22 +1796,24 @@ stream. First, lets examine a set of XML records that the ---- -To be able to process the XML records the following is needed: +To be able to process the XML records, the following is needed: -* Root Element Name - Name of the root element of the fragment +* Root Element Name: The name of the root element of the fragment that constitutes the object to be mapped. The example configuration demonstrates this with the value of trade. -* Resource - Spring Resource that represents the file to be +* Resource: A Spring Resource that represents the file to be read. -* `Unmarshaller` - Unmarshalling +* `Unmarshaller`: An unmarshalling facility provided by Spring OXM for mapping the XML fragment to an object. - +The following example shows how to define a `StaxEventItemReader` that works with a root +element named trade, a resource of `data/iosample/input/input.xml`, and an unmarshaller +called `tradeMarshaller`. .XML Configuration [source, xml, role="xmlContent"] @@ -1831,14 +1840,14 @@ public StaxEventItemReader itemReader() { } ---- -Notice that in this example we have chosen to use an -`XStreamMarshaller` which accepts an alias passed +Note that, in this example, we have chosen to use an +`XStreamMarshaller`, which accepts an alias passed in as a map with the first key and value being the name of the fragment -(i.e. root element) and the object type to bind. Then, similar to a +(that is, a root element) and the object type to bind. Then, similar to a `FieldSet`, the names of the other elements that map to fields within the object type are described as key/value pairs in -the map. In the configuration file we can use a Spring configuration -utility to describe the required alias as follows: +the map. In the configuration file, we can use a Spring configuration +utility to describe the required alias, as follows: .XML Configuration [source, xml, role="xmlContent"] @@ -1874,16 +1883,16 @@ public XStreamMarshaller tradeMarshaller() { } ---- -On input the reader reads the XML resource until it recognizes -that a new fragment is about to start (by matching the tag name by -default). The reader creates a standalone XML document from the fragment -(or at least makes it appear so) and passes the document to a +On input, the reader reads the XML resource until it recognizes +that a new fragment is about to start. By default, the reader matches the element name to +recognize that a new fragment is about to start. The reader creates a standalone XML +document from the fragment and passes the document to a deserializer (typically a wrapper around a Spring OXM `Unmarshaller`) to map the XML to a Java object. -In summary, this procedure is analogous to the following scripted -Java code which uses the injection provided by the Spring +In summary, this procedure is analogous to the following +Java code, which uses the injection provided by the Spring configuration: [source, java] @@ -1923,15 +1932,15 @@ while (hasNext) { Output works symmetrically to input. The `StaxEventItemWriter` needs a -Resource, a marshaller, and a `rootTagName`. A Java +`Resource`, a marshaller, and a `rootTagName`. A Java object is passed to a marshaller (typically a standard Spring OXM Marshaller) which writes to a -Resource using a custom event writer that filters -the StartDocument and -EndDocument events produced for each fragment by -the OXM tools. We'll show this in an example using the -MarshallingEventWriterSerializer. The Spring -configuration for this setup looks as follows: +`Resource` by using a custom event writer that filters +the `StartDocument` and +`EndDocument` events produced for each fragment by +the OXM tools. The following example uses the +`MarshallingEventWriterSerializer`: +// TODO How does `MarshallingEventWriterSerializer` get involved? Because there's a property whose name is `marshaller`? .XML Configuration [source, xml, role="xmlContent"] @@ -1960,10 +1969,10 @@ public StaxEventItemWriter itemWriter(Resource outputResource) { } ---- -The configuration sets up the three required properties and -optionally sets the overwriteOutput=true, mentioned earlier in the +The preceding configuration sets up the three required properties and +sets the optional `overwriteOutput=true` attrbute, mentioned earlier in this chapter for specifying whether an existing file can be overwritten. It -should be noted the marshaller used for the writer is the exact same as +should be noted the marshaller used for the writer in the following example is the exact same as the one used in the reading example from earlier in the chapter: .XML Configuration @@ -2051,10 +2060,10 @@ It is a common requirement to process multiple files within a single file-1.txt file-2.txt ignored.txt ---- -file-1.txt and file-2.txt are formatted the same and for business - reasons should be processed together. The - MuliResourceItemReader can be used to read in both - files by using wildcards: +file-1.txt and file-2.txt are formatted the same and, for business + reasons, should be processed together. The + `MuliResourceItemReader` can be used to read in both + files by using wildcards, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -2078,8 +2087,8 @@ public MultiResourceItemReader multiResourceReader() { ---- The referenced delegate is a simple -`FlatFileItemReader`. The above configuration will -read input from both files, handling rollback and restart scenarios. It +`FlatFileItemReader`. The above configuration +reads input from both files, handling rollback and restart scenarios. It should be noted that, as with any `ItemReader`, adding extra input (in this case a file) could cause potential issues when restarting. It is recommended that batch jobs work with their own @@ -2095,12 +2104,14 @@ storage mechanism for batch. However, batch differs from other application styles due to the sheer size of the datasets with which the system must work. If a SQL statement returns 1 million rows, the result set probably holds all returned results in memory until all rows have been read. Spring -Batch provides two types of solutions for this problem: Cursor and Paging -database `ItemReaders`. +Batch provides two types of solutions for this problem: + +* <> +* <> [[cursorBasedItemReaders]] -==== Cursor Based ItemReaders +==== Cursor-based `ItemReader` Implementations Using a database cursor is generally the default approach of most batch developers, because it is the database's solution to the problem @@ -2110,32 +2121,32 @@ orientated mechanism for manipulating a cursor. A `ResultSet` maintains a cursor to the current row of data. Calling `next` on a `ResultSet` moves this cursor to the next row. -Spring Batch cursor based `ItemReaders` open the a cursor on -initialization, and move the cursor forward one row for every call to +The Spring Batch cursor-based `ItemReader` implementation opens a cursor on +initialization and moves the cursor forward one row for every call to `read`, returning a mapped object that can be -used for processing. The `close` method will then -be called to ensure all resources are freed up. The Spring core +used for processing. The `close` method is then +called to ensure all resources are freed up. The Spring core `JdbcTemplate` gets around this problem by using the callback pattern to completely map all rows in a `ResultSet` and close before returning control back -to the method caller. However, in batch this must wait until the step is -complete. Below is a generic diagram of how a cursor based -`ItemReader` works, and while a SQL statement is -used as an example since it is so widely known, any technology could -implement the basic approach: +to the method caller. However, in batch, this must wait until the step is +complete. The following image shows a generic diagram of how a cursor-based +`ItemReader` works. Note that, while the example uses SQL +(because SQL is so widely known), any technology could +implement the basic approach. .Cursor Example image::{batch-asciidoc}images/cursorExample.png[Cursor Example, scaledwidth="60%"] This example illustrates the basic pattern. Given a 'FOO' table, -which has three columns: ID, NAME, and BAR, select all rows with an ID +which has three columns: `ID`, `NAME`, and `BAR`, select all rows with an ID greater than 1 but less than 7. This puts the beginning of the cursor (row 1) on ID 2. The result of this row should be a completely mapped -Foo object. Calling `read()` again moves the -cursor to the next row, which is the Foo with an ID of 3. The results of -these reads will be written out after each -`read`, thus allowing the objects to be garbage +`Foo` object. Calling `read()` again moves the +cursor to the next row, which is the `Foo` with an ID of 3. The results of +these reads are written out after each +`read`, allowing the objects to be garbage collected (assuming no instance variables are maintaining references to them). @@ -2144,12 +2155,12 @@ them). ===== JdbcCursorItemReader -`JdbcCursorItemReader` is the Jdbc -implementation of the cursor based technique. It works directly with a -`ResultSet` and requires a SQL statement to run +`JdbcCursorItemReader` is the JDBC +implementation of the cursor-based technique. It works directly with a +`ResultSet` and requires an SQL statement to run against a connection obtained from a -DataSource. The following database schema will -be used as an example: +`DataSource`. The following database schema +is used as an example: [source, sql] @@ -2161,8 +2172,8 @@ CREATE TABLE CUSTOMER ( ); ---- -Many people prefer to use a domain object for each row, so we'll -use an implementation of the `RowMapper` +Many people prefer to use a domain object for each row, so the following example +uses an implementation of the `RowMapper` interface to map a `CustomerCredit` object: @@ -2187,13 +2198,12 @@ public class CustomerCreditRowMapper implements RowMapper { } ---- -Because `JdbcTemplate` is so familiar to -users of Spring, and the `JdbcCursorItemReader` -shares key interfaces with it, it is useful to see an example of how +Because `JdbcCursorItemReader` +shares key interfaces with `JdbcTemplate`, it is useful to see an example of how to read in this data with `JdbcTemplate`, in order to contrast it with the `ItemReader`. For -the purposes of this example, let's assume there are 1,000 rows in the -CUSTOMER database. The first example will be using +the purposes of this example, assume there are 1,000 rows in the +`CUSTOMER` database. The first example uses `JdbcTemplate`: @@ -2205,14 +2215,14 @@ List customerCredits = jdbcTemplate.query("SELECT ID, NAME, CREDIT from CUSTOMER new CustomerCreditRowMapper()); ---- -After running this code snippet the customerCredits list will -contain 1,000 `CustomerCredit` objects. In the -query method, a connection will be obtained from the -DataSource, the provided SQL will be run -against it, and the `mapRow` method will be -called for each row in the `ResultSet`. Let's -contrast this with the approach of the -`JdbcCursorItemReader`: +After running the preceding code snippet, the `customerCredits` list +contains 1,000 `CustomerCredit` objects. In the +query method, a connection is obtained from the +`DataSource`, the provided SQL is run +against it, and the `mapRow` method is +called for each row in the `ResultSet`. +Contrast this with the approach of the +`JdbcCursorItemReader`, shown in the following example: [source, java] @@ -2232,19 +2242,19 @@ while(customerCredit != null){ itemReader.close(executionContext); ---- -After running this code snippet the counter will equal 1,000. If -the code above had put the returned customerCredit into a list, the +After running the preceding code snippet, the counter equals 1,000. If +the code above had put the returned `customerCredit` into a list, the result would have been exactly the same as with the `JdbcTemplate` example. However, the big advantage of the `ItemReader` is that it allows items to be 'streamed'. The `read` method can -be called once, and the item written out via an -`ItemWriter`, and then the next item obtained via +be called once, the item can be written out by an +`ItemWriter`, and then the next item can be obtained with `read`. This allows item reading and writing to be done in 'chunks' and committed periodically, which is the essence of high performance batch processing. Furthermore, it is very easily configured for injection into a Spring Batch -`Step`: +`Step`, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -2279,14 +2289,14 @@ public JdbcCursorItemReader itemReader() { Because there are so many varying options for opening a cursor in Java, there are many properties on the -JdbcCustorItemReader that can be set: +`JdbcCustorItemReader` that can be set, as described in the following table: .JdbcCursorItemReader Properties |=============== |ignoreWarnings|Determines whether or not SQLWarnings are logged or - cause an exception - default is true -|fetchSize|Gives the Jdbc driver a hint as to the number of rows + cause an exception. The default is `true` (meaning that warnings are logged). +|fetchSize|Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are needed by the `ResultSet` object used by the `ItemReader`. By default, no @@ -2294,9 +2304,8 @@ JdbcCustorItemReader that can be set: |maxRows|Sets the limit for the maximum number of rows the underlying `ResultSet` can hold at any one time. -|queryTimeout|Sets the number of seconds the driver will wait for a - `Statement` object to execute to the - given number of seconds. If the limit is exceeded, a +|queryTimeout|Sets the number of seconds the driver waits for a + `Statement` object to run. If the limit is exceeded, a `DataAccessException` is thrown. (Consult your driver vendor documentation for details). @@ -2305,7 +2314,7 @@ JdbcCustorItemReader that can be set: the `RowMapper`, it is possible for users to call `ResultSet.next()` themselves, which could cause issues with the reader's - internal count. Setting this value to true will cause an + internal count. Setting this value to `true` causes an exception to be thrown if the cursor position is not the same after the `RowMapper` call as it was before. @@ -2313,38 +2322,38 @@ JdbcCustorItemReader that can be set: saved in the `ExecutionContext` provided by `ItemStream#update(ExecutionContext)` - The default value is true. -|driverSupportsAbsolute|Defaults to false. Indicates whether the Jdbc driver + The default is `true`. +|driverSupportsAbsolute|Defaults to `false`. Indicates whether the JDBC driver supports setting the absolute row on a `ResultSet`. It is recommended that - this is set to true for Jdbc drivers that supports - `ResultSet`.absolute() as it may + this is set to `true` for JDBC drivers that support + `ResultSet`.absolute(), as it may improve performance, especially if a step fails while working with a large data set. -|setUseSharedExtendedConnection|Defaults to false. Indicates whether the connection - used for the cursor should be used by all other processing - thus sharing the same transaction. If this is set to false, - which is the default, then the cursor will be opened using - its own connection and will not participate in any +|setUseSharedExtendedConnection|Defaults to `false`. Indicates whether the connection + used for the cursor should be used by all other processing, + thus sharing the same transaction. If this is set to `false`, + then the cursor is opened with + its own connection and does not participate in any transactions started for the rest of the step processing. If - you set this flag to true then you must wrap the + you set this flag to `true` then you must wrap the DataSource in an `ExtendedConnectionDataSourceProxy` to prevent the connection from being closed and released after - each commit. When you set this option to true then the - statement used to open the cursor will be created with both - 'READ_ONLY' and 'HOLD_CUSORS_OVER_COMMIT' options. This + each commit. When you set this option to `true`, the + statement used to open the cursor is created with both + 'READ_ONLY' and 'HOLD_CURSORS_OVER_COMMIT' options. This allows holding the cursor open over transaction start and commits performed in the step processing. To use this - feature you need a database that supports this and a Jdbc - driver supporting Jdbc 3.0 or later. + feature, you need a database that supports this and a JDBC + driver supporting JDBC 3.0 or later. |=============== [[HibernateCursorItemReader]] -===== HibernateCursorItemReader +===== `HibernateCursorItemReader` Just as normal Spring users make important decisions about whether or not to use ORM solutions, which affect whether or not they @@ -2354,19 +2363,19 @@ same options. `HibernateCursorItemReader` is the Hibernate implementation of the cursor technique. Hibernate's usage in batch has been fairly controversial. This has largely been because Hibernate was originally developed to support online application -styles. However, that doesn't mean it can't be used for batch +styles. However, that does not mean it cannot be used for batch processing. The easiest approach for solving this problem is to use a `StatelessSession` rather than a standard -session. This removes all of the caching and dirty checking hibernate -employs that can cause issues in a batch scenario. For more +session. This removes all of the caching and dirty checking Hibernate +employs and that can cause issues in a batch scenario. For more information on the differences between stateless and normal hibernate sessions, refer to the documentation of your specific hibernate -release. The `HibernateCursorItemReader` allows -you to declare an HQL statement and pass in a +release. The `HibernateCursorItemReader` lets +you declare an HQL statement and pass in a `SessionFactory`, which will pass back one item per call to read in the same basic fashion as -the `JdbcCursorItemReader`. Below is an example -configuration using the same 'customer credit' example as the JDBC +the `JdbcCursorItemReader`. The following example +configuration uses the same 'customer credit' example as the JDBC reader: @@ -2388,16 +2397,16 @@ while(customerCredit != null){ itemReader.close(executionContext); ---- -This configured `ItemReader` will return +This configured `ItemReader` returns `CustomerCredit` objects in the exact same manner as described by the `JdbcCursorItemReader`, assuming hibernate mapping files have been created correctly for the -Customer table. The 'useStatelessSession' property defaults to true, +`Customer` table. The 'useStatelessSession' property defaults to true but has been added here to draw attention to the ability to switch it -on or off. It is also worth noting that the fetchSize of the -underlying cursor can be set via the setFetchSize property. As with +on or off. It is also worth noting that the fetch size of the +underlying cursor can be set via the `setFetchSize` property. As with `JdbcCursorItemReader`, configuration is -straightforward: +straightforward, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -2424,24 +2433,24 @@ public HibernateCursorItemReader itemReader(SessionFactory sessionFactory) { [[StoredProcedureItemReader]] -===== StoredProcedureItemReader +===== `StoredProcedureItemReader` -Sometimes it is necessary to obtain the cursor data using a +Sometimes it is necessary to obtain the cursor data by using a stored procedure. The `StoredProcedureItemReader` -works like the `JdbcCursorItemReader` except that -instead of executing a query to obtain a cursor we execute a stored +works like the `JdbcCursorItemReader`, except that, +instead of running a query to obtain a cursor, it runs a stored procedure that returns a cursor. The stored procedure can return the cursor in three different ways: -. as a returned `ResultSet` (used by SQL Server, Sybase, DB2, Derby and MySQL) +* As a returned `ResultSet` (used by SQL Server, Sybase, DB2, Derby, and MySQL). -. as a ref-cursor returned as an out parameter (used by Oracle and PostgreSQL) +* As a ref-cursor returned as an out parameter (used by Oracle and PostgreSQL). -. as the return value of a stored function call +* As the return value of a stored function call. -Below is a basic example configuration using the same 'customer -credit' example as earlier: +The following example configuration uses the same 'customer +credit' example as earlier examples: .XML Configuration [source, xml, role="xmlContent"] @@ -2471,13 +2480,13 @@ public StoredProcedureItemReader reader(DataSource dataSource) { ---- //TODO: Fix the above config to use a builder once we have one for it. -This example relies on the stored procedure to provide a -`ResultSet` as a returned result (option 1 above). +The preceding example relies on the stored procedure to provide a +`ResultSet` as a returned result (option 1 from earlier). -If the stored procedure returned a ref-cursor (option 2) then we +If the stored procedure returned a `ref-cursor` (option 2), then we would need to provide the position of the out parameter that is the -returned ref-cursor. Here is an example where the first parameter is -the returned ref-cursor: +returned `ref-cursor`. The following example shows how to work with the first parameter being +a ref-cursor: .XML Configuration [source, xml, role="xmlContent"] @@ -2508,10 +2517,10 @@ public StoredProcedureItemReader reader(DataSource dataSource) { } ---- -If the cursor was returned from a stored function (option 3) we +If the cursor was returned from a stored function (option 3), we would need to set the property "[maroon]#function#" to -`true`. It defaults to `false`. Here -is what that would look like: +`true`. It defaults to `false`. The following example +shows what that would look like: .XML Configuration [source, xml, role="xmlContent"] @@ -2542,16 +2551,16 @@ public StoredProcedureItemReader reader(DataSource dataSource) { } ---- -In all of these cases we need to define a +In all of these cases, we need to define a `RowMapper` as well as a - DataSource and the actual procedure + `DataSource` and the actual procedure name. -If the stored procedure or function takes in parameter then they - must be declared and set via the parameters property. Here is an - example for Oracle that declares three parameters. The first one is - the out parameter that returns the ref-cursor, the second and third - are in parameters that takes a value of type INTEGER: +If the stored procedure or function takes in parameters, then they + must be declared and set via the `parameters` property. The following + example, for Oracle, declares three parameters. The first one is + the out parameter that returns the ref-cursor, and the second and third + are in parameters that takes a value of type `INTEGER`. .XML Configuration [source, xml, role="xmlContent"] @@ -2610,7 +2619,7 @@ public StoredProcedureItemReader reader(DataSource dataSource) { } ---- -In addition to the parameter declarations we need to specify a +In addition to the parameter declarations, we need to specify a `PreparedStatementSetter` implementation that sets the parameter values for the call. This works the same as for the `JdbcCursorItemReader` above. All the additional @@ -2619,13 +2628,13 @@ apply to the `StoredProcedureItemReader` as well. [[pagingItemReaders]] -==== Paging ItemReaders +==== Paging `ItemReader` Implementations -An alternative to using a database cursor is executing multiple - queries where each query is bringing back a portion of the results. We - refer to this portion as a page. Each query that is executed must +An alternative to using a database cursor is running multiple + queries where each query fetches a portion of the results. We + refer to this portion as a page. Each query must specify the starting row number and the number of rows that we want - returned for the page. + returned in the page. [[JdbcPagingItemReader]] @@ -2646,17 +2655,17 @@ One implementation of a paging `ItemReader` practice. The `SqlPagingQueryProviderFactoryBean` - requires that you specify a select clause and a from clause. You can - also provide an optional where clause. These clauses will be used to - build an SQL statement combined with the required sortKey. + requires that you specify a `select` clause and a `from` clause. You can + also provide an optional `where` clause. These clauses and the required `sortKey` are used to + build an SQL statement. -After the reader has been opened, it will pass back one item per +After the reader has been opened, it passes back one item per call to `read` in the same basic fashion as any other `ItemReader`. The paging happens behind the scenes when additional rows are needed. -Below is an example configuration using a similar 'customer - credit' example as the cursor based `ItemReaders` above: +The following example configuration uses a similar 'customer + credit' example as the cursor based `ItemReaders` shown previously: .XML Configuration [source, xml, role="xmlContent"] @@ -2712,16 +2721,16 @@ public SqlPagingQueryProviderFactoryBean queryProvider() { } ---- -This configured `ItemReader` will return +This configured `ItemReader` returns `CustomerCredit` objects using the -`RowMapper` that must be specified. The +`RowMapper`, which must be specified. The 'pageSize' property determines the number of entities read from the -database for each query execution. +database for each query run. -The 'parameterValues' property can be used to specify a Map of +The 'parameterValues' property can be used to specify a `Map` of parameter values for the query. If you use named parameters in the -where clause the key for each entry should match the name of the named -parameter. If you use a traditional '?' placeholder then the key for +`where` clause, the key for each entry should match the name of the named +parameter. If you use a traditional '?' placeholder, then the key for each entry should be the number of the placeholder, starting with 1. @@ -2731,23 +2740,23 @@ each entry should be the number of the placeholder, starting with Another implementation of a paging `ItemReader` is the -`JpaPagingItemReader`. JPA doesn't have a concept -similar to the Hibernate `StatelessSession` so we +`JpaPagingItemReader`. JPA does not have a concept +similar to the Hibernate `StatelessSession`, so we have to use other features provided by the JPA specification. Since JPA supports paging, this is a natural choice when it comes to using -JPA for batch processing. After each page is read, the entities will -become detached and the persistence context will be cleared in order +JPA for batch processing. After each page is read, the entities +become detached and the persistence context is cleared, to allow the entities to be garbage collected once the page is processed. -The `JpaPagingItemReader` allows you to +The `JpaPagingItemReader` lets you declare a JPQL statement and pass in a -`EntityManagerFactory`. It will then pass back +`EntityManagerFactory`. It then passes back one item per call to read in the same basic fashion as any other `ItemReader`. The paging -happens behind the scenes when additional entities are needed. Below -is an example configuration using the same 'customer credit' example -as the JDBC reader above: +happens behind the scenes when additional entities are needed. The following +example configuration uses the same 'customer credit' example +as the JDBC reader shown previously: .XML Configuration [source, xml, role="xmlContent"] @@ -2773,10 +2782,10 @@ public JpaPagingItemReader itemReader() { } ---- -This configured `ItemReader` will return +This configured `ItemReader` returns `CustomerCredit` objects in the exact same manner -as described by the `JdbcPagingItemReader` above, -assuming the Customer object has the correct JPA annotations or ORM +as described for the `JdbcPagingItemReader` above, +assuming the `Customer` object has the correct JPA annotations or ORM mapping file. The 'pageSize' property determines the number of entities read from the database for each query execution. @@ -2785,55 +2794,54 @@ entities read from the database for each query execution. ==== Database ItemWriters -While both Flat Files and XML have specific `ItemWriters`, there is +While both flat files and XML have specific `ItemWriters`, there is no exact equivalent in the database world. This is because transactions -provide all the functionality that is needed. `ItemWriters` are necessary +provide all the needed functionality. `ItemWriters` are necessary for files because they must act as if they're transactional, keeping track of written items and flushing or clearing at the appropriate times. Databases have no need for this functionality, since the write is already contained in a transaction. Users can create their own DAOs that implement the `ItemWriter` interface or use one from a custom `ItemWriter` that's written for -generic processing concerns, either way, they should work without any +generic processing concerns. Either way, they should work without any issues. One thing to look out for is the performance and error handling capabilities that are provided by batching the outputs. This is most -common when using hibernate as an `ItemWriter`, but -could have the same issues when using Jdbc batch mode. Batching database -output doesn't have any inherent flaws, assuming we are careful to flush +common when using hibernate as an `ItemWriter` but +could have the same issues when using JDBC batch mode. Batching database +output does not have any inherent flaws, assuming we are careful to flush and there are no errors in the data. However, any errors while writing -out can cause confusion because there is no way to know which individual -item caused an exception, or even if any individual item was -responsible, as illustrated below: +can cause confusion, because there is no way to know which individual +item caused an exception or even if any individual item was +responsible, as illustrated in the following image: .Error On Flush image::{batch-asciidoc}images/errorOnFlush.png[Error On Flush, scaledwidth="60%"] -If items are buffered before being written out, any -errors encountered will not be thrown until the buffer is flushed just -before a commit. For example, let's assume that 20 items will be written -per chunk, and the 15th item throws a DataIntegrityViolationException. -As far as the `Step` is concerned, all 20 item will be written out -successfully, since there's no way to know that an error will occur -until they are actually written out. Once -Session#flush() is -called, the buffer will be emptied and the exception will be hit. At -this point, there's nothing the `Step` can do, the +If items are buffered before being written, any +errors are not thrown until the buffer is flushed just +before a commit. For example, assume that 20 items are written +per chunk, and the 15th item throws a `DataIntegrityViolationException`. +As far as the `Step` is concerned, all 20 item are written +successfully, since there is no way to know that an error occurs +until they are actually written. Once +`Session#flush()` is +called, the buffer is emptied and the exception is hit. At +this point, there is nothing the `Step` can do. The transaction must be rolled back. Normally, this exception might cause -the Item to be skipped (depending upon the skip/retry policies), and -then it won't be written out again. However, in the batched scenario, -there's no way for it to know which item caused the issue, the whole -buffer was being written out when the failure happened. The only way to -solve this issue is to flush after each item: +the item to be skipped (depending upon the skip/retry policies), and +then it is not written again. However, in the batched scenario, +there is no way to know which item caused the issue. The whole +buffer was being written when the failure happened. The only way to +solve this issue is to flush after each item, as shown in the following image: .Error On Write image::{batch-asciidoc}images/errorOnWrite.png[Error On Write, scaledwidth="60%"] -http://docs.spring.io/spring-batch/reference/html/images/errorOnWrite.png.pagespeed.ce.SKTuwx-ca1.png This is a common use case, especially when using Hibernate, and -the simple guideline for implementations of `ItemWriter`, is to flush on each call to +the simple guideline for implementations of `ItemWriter` is to flush on each call to `write()`. Doing so allows for items to be -skipped reliably, with Spring Batch taking care internally of the +skipped reliably, with Spring Batch internally taking care of the granularity of the calls to `ItemWriter` after an error. @@ -2851,14 +2859,14 @@ batch jobs. The Spring container itself makes this fairly easy by allowing any necessary class to be injected. However, there may be cases where the existing service needs to act as an `ItemReader` or `ItemWriter`, either to satisfy the dependency of -another Spring Batch class, or because it truly is the main +another Spring Batch class or because it truly is the main `ItemReader` for a step. It is fairly trivial to write an adaptor class for each service that needs wrapping, but because it is such a common concern, Spring Batch provides implementations: `ItemReaderAdapter` and `ItemWriterAdapter`. Both classes implement the -standard Spring method invoking the delegate pattern and are fairly simple -to set up. Below is an example of the reader: +standard Spring method by invoking the delegate pattern and are fairly simple +to set up. The following example uses the `ItemReaderAdapter`: .XML Configuration [source, xml, role="xmlContent"] @@ -2890,14 +2898,13 @@ public FooService fooService() { } ---- -One important point to note is that the contract of the targetMethod -must be the same as the contract for `read`: when -exhausted it will return null, otherwise an `Object`. -Anything else will prevent the framework from knowing when processing +One important point to note is that the contract of the `targetMethod` +must be the same as the contract for `read`: When +exhausted, it return nulls. Otherwise, it returns an `Object`. +Anything else prevents the framework from knowing when processing should end, either causing an infinite loop or incorrect failure, depending upon the implementation of the -`ItemWriter`. The `ItemWriter` -implementation is equally as simple: +`ItemWriter`. The following example uses the `ItemWriterAdapter`: .XML Configuration [source, xml, role="xmlContent"] @@ -2935,21 +2942,21 @@ public FooService fooService() { === Validating Input During the course of this chapter, multiple approaches to parsing -input have been discussed. Each major implementation will throw an +input have been discussed. Each major implementation throws an exception if it is not 'well-formed'. The -`FixedLengthTokenizer` will throw an exception if a +`FixedLengthTokenizer` throws an exception if a range of data is missing. Similarly, attempting to access an index in a `RowMapper` of `FieldSetMapper` -that doesn't exist or is in a different format than the one expected will -cause an exception to be thrown. All of these types of exceptions will be -thrown before read returns. However, they don't +that does not exist or is in a different format than the one expected +causes an exception to be thrown. All of these types of exceptions are +thrown before `read` returns. However, they do not address the issue of whether or not the returned item is valid. For example, if one of the fields is an age, it obviously cannot be negative. -It will parse correctly, because it existed and is a number, but it won't -cause an exception. Since there are already a plethora of Validation -frameworks, Spring Batch does not attempt to provide yet another, but -rather provides a very simple interface that can be implemented by any -number of frameworks: +It may parse correctly, because it exists and is a number, but it does not +cause an exception. Since there are already a plethora of validation +frameworks, Spring Batch does not attempt to provide yet another. +Rather, it provides a simple interface, called `Validator`, that can be implemented by any +number of frameworks, as shown in the following interface definition: [source, java] @@ -2962,9 +2969,9 @@ public interface Validator { ---- The contract is that the `validate` method -will throw an exception if the object is invalid, and return normally if +throws an exception if the object is invalid and returns normally if it is valid. Spring Batch provides an out of the box -`ItemProcessor`: +`ValidatingItemProcessor`, as shown in the following bean definition: .XML Configuration [source, xml, role="xmlContent"] @@ -3014,14 +3021,14 @@ committed. However, this may not always be the desired behavior. For example, many developers choose to make their database readers 'rerunnable' by using a process indicator. An extra column is added to the input data to indicate whether or not it has been processed. When a -particular record is being read (or written out) the processed flag is -flipped from false to true. The SQL statement can then contain an extra -statement in the where clause, such as "where PROCESSED_IND = false", -thereby ensuring that only unprocessed records will be returned in the +particular record is being read (or written) the processed flag is +flipped from `false` to `true`. The SQL statement can then contain an extra +statement in the `where` clause, such as `where PROCESSED_IND = false`, +thereby ensuring that only unprocessed records are returned in the case of a restart. In this scenario, it is preferable to not store any -state, such as the current row number, since it will be irrelevant upon +state, such as the current row number, since it is irrelevant upon restart. For this reason, all readers and writers include the 'saveState' -property: +property, as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -3065,7 +3072,7 @@ public JdbcCursorItemReader playerSummarizationSource(DataSource dataSource) { } ---- -The `ItemReader` configured above will not make +The `ItemReader` configured above does not make any entries in the `ExecutionContext` for any executions in which it participates. @@ -3073,26 +3080,26 @@ executions in which it participates. === Creating Custom ItemReaders and ItemWriters -So far in this chapter the basic contracts that exist for reading -and writing in Spring Batch and some common implementations have been -discussed. However, these are all fairly generic, and there are many -potential scenarios that may not be covered by out of the box -implementations. This section will show, using a simple example, how to +So far, this chapter has discussed the basic contracts of reading +and writing in Spring Batch and some common implementations +for doing so. However, these are all fairly generic, and there are many +potential scenarios that may not be covered by out-of-the-box +implementations. This section shows, by using a simple example, how to create a custom `ItemReader` and `ItemWriter` implementation and implement their -contracts correctly. The `ItemReader` will also -implement `ItemStream`, in order to illustrate how to +contracts correctly. The `ItemReader` also +implements `ItemStream`, in order to illustrate how to make a reader or writer restartable. [[customReader]] ==== Custom ItemReader Example -For the purpose of this example, a simple +For the purpose of this example, we create a simple `ItemReader` implementation that reads from a -provided list will be created. We'll start out by implementing the most +provided list. We start by implementing the most basic contract of `ItemReader`, -`read`: +the `read` method, as shown in the following code: [source, java] @@ -3116,10 +3123,10 @@ public class CustomItemReader implements ItemReader{ } ---- -This very simple class takes a list of items, and returns them one +The preceding class takes a list of items and returns them one at a time, removing each from the list. When the list is empty, it -returns null, thus satisfying the most basic requirements of an -`ItemReader`, as illustrated below: +returns `null`, thus satisfying the most basic requirements of an +`ItemReader`, as illustrated in the following test code: [source, java] @@ -3141,17 +3148,17 @@ assertNull(itemReader.read()); ===== Making the ItemReader Restartable -The final challenge now is to make the -`ItemReader` restartable. Currently, if the power -goes out, and processing begins again, the +The final challenge is to make the +`ItemReader` restartable. Currently, if +processing is interrupted and begins again, the `ItemReader` must start at the beginning. This is actually valid in many scenarios, but it is sometimes preferable that -a batch job starts where it left off. The key discriminant is often +a batch job restarts where it left off. The key discriminant is often whether the reader is stateful or stateless. A stateless reader does not need to worry about restartability, but a stateful one has to try -and reconstitute its last known state on restart. For this reason, we +to reconstitute its last known state on restart. For this reason, we recommend that you keep custom readers stateless if possible, so you -don't have to worry about restartability. +need not worry about restartability. If you do need to store state, then the `ItemStream` interface should be used: @@ -3198,7 +3205,7 @@ public class CustomItemReader implements ItemReader, ItemStream { On each call to the `ItemStream` `update` method, the current index of the -`ItemReader` will be stored in the provided +`ItemReader` is stored in the provided `ExecutionContext` with a key of 'current.index'. When the `ItemStream` `open` method is called, the `ExecutionContext` is @@ -3227,7 +3234,7 @@ assertEquals("2", itemReader.read()); Most `ItemReaders` have much more sophisticated restart logic. The `JdbcCursorItemReader`, for example, stores the -row id of the last processed row in the Cursor. +row ID of the last processed row in the Cursor. It is also worth noting that the key used within the `ExecutionContext` should not be trivial. That is @@ -3237,12 +3244,12 @@ all `ItemStreams` within a with the class name should be enough to guarantee uniqueness. However, in the rare cases where two of the same type of `ItemStream` are used in the same step (which can -happen if two files are need for output) then a more unique name will -be needed. For this reason, many of the Spring Batch +happen if two files are need for output), a more unique name +is needed. For this reason, many of the Spring Batch `ItemReader` and `ItemWriter` implementations have a -`setName()` property that allows this key name -to be overridden. +`setName()` property that lets this key name +be overridden. [[customWriter]] @@ -3250,11 +3257,11 @@ to be overridden. ==== Custom ItemWriter Example Implementing a Custom `ItemWriter` is similar -in many ways to the `ItemReader` example above, but +in many ways to the `ItemReader` example above but differs in enough ways as to warrant its own example. However, adding -restartability is essentially the same, so it won't be covered in this +restartability is essentially the same, so it is not covered in this example. As with the `ItemReader` example, a -`List` will be used in order to keep the example as +`List` is used in order to keep the example as simple as possible: @@ -3279,10 +3286,10 @@ public class CustomItemWriter implements ItemWriter { ===== Making the ItemWriter Restartable -To make the `ItemWriter` restartable we would follow the same +To make the `ItemWriter` restartable, we would follow the same process as for the `ItemReader`, adding and implementing the `ItemStream` interface to -synchronize the execution context. In the example we might have to +synchronize the execution context. In the example, we might have to count the number of items processed and add that as a footer record. If we needed to do that, we could implement `ItemStream` in our @@ -3291,30 +3298,39 @@ reconstituted from the execution context if the stream was re-opened. In many realistic cases, custom `ItemWriters` also delegate to -another writer that itself is restartable (e.g. when writing to a -file), or else it writes to a transactional resource so doesn't need -to be restartable because it is stateless. When you have a stateful -writer you should probably also be sure to implement +another writer that itself is restartable (for example, when writing to a +file), or else it writes to a transactional resource and so does not need +to be restartable, because it is stateless. When you have a stateful +writer you should probably be sure to implement `ItemStream` as well as `ItemWriter`. Remember also that the client of the writer needs to be aware of the `ItemStream`, so you may need to register it as a stream in the configuration -xml. +XML. [[itemReaderAndWriterImplementations]] === Item Reader and Writer Implementations -In this section we will introduce you to Reader and Writers that have not +In this section, we will introduce you to readers and writers that have not already been discussed in the previous sections. [[decorators]] ==== Decorators -In some cases a user needs a specialized behavior to be appended to a -pre-existing ItemReader. Spring Batch offers some out of the box decorators -that can add additional behavior to to your `ItemReaders` and `ItemWriters`. +In some cases, a user needs specialized behavior to be appended to a +pre-existing `ItemReader`. Spring Batch offers some out of the box decorators +that can add additional behavior to to your `ItemReader` and `ItemWriter` implementations. + +Spring Batch includes the following decorators: + +* <> +* <> +* <> +* <> +* <> + [[synchronizedItemStreamReader]] -===== SynchronizedItemStreamReader +===== `SynchronizedItemStreamReader` When using a `ItemReader` that is not thread safe, Spring Batch offers the `SynchronizedItemStreamReader` decorator that can be used to make the `ItemReader` thread safe. Spring Batch provides a @@ -3322,127 +3338,162 @@ When using a `ItemReader` that is not thread safe, Spring Batch offers the `SynchronizedItemStreamReader`. [[singleItemPeekableItemReader]] -===== SingleItemPeekableItemReader -A decorator that adds a peek method to an `ItemReader`. -This peek method allow the user to peek one item ahead. Repeated calls to the -peek will return the same item, and this will be the next item returned from read method. +===== `SingleItemPeekableItemReader` +Spring Batch includes a decorator that adds a peek method to an `ItemReader`. +This peek method lets the user peek one item ahead. Repeated calls to the +peek returns the same item, and this is the next item returned from the `read` method. Spring Batch provides a `SingleItemPeekableItemReaderBuilder` to construct an instance of the `SingleItemPeekableItemReader`. -NOTE: SingleItemPeekableItemReader's peek method is not thread-safe: -it wouldn't be possible to honour the peek in multiple threads because only one +NOTE: SingleItemPeekableItemReader's peek method is not thread-safe, because +it would not be possible to honor the peek in multiple threads. Only one of the threads that peeked would get that item in the next call to read. [[multiResourceItemWriter]] -===== MultiResourceItemWriter +===== `MultiResourceItemWriter` The MultiResourceItemWriter wraps a `ResourceAwareItemWriterItemStream` and -creates a new output resource when the count of items written in current +creates a new output resource when the count of items written in the current resource exceeds the `itemCountLimitPerResource`. Spring Batch provides a `MultiResourceItemWriterBuilder` to construct an instance of the `MultiResourceItemWriter`. +[[classifierCompositeItemWriter]] +===== `ClassifierCompositeItemWriter` +The `ClassifierCompositeItemWriter` calls one of a collection of `ItemWriter` implementations for +each item, based on a router pattern implemented through the provided +`Classifier`. The implementation is thread-safe if all delegates are thread-safe. +Spring Batch provides a `ClassifierCompositeItemWriterBuilder` to construct an +instance of the `ClassifierCompositeItemWriter`. + +[[classifierCompositeItemProcessor]] +===== ClassifierCompositeItemProcessor +The `ClassifierCompositeItemProcessor` is an `ItemProcessor` that calls one of a collection +of `ItemProcessor` implemenations, based on a router pattern implemented through the provided +`Classifier`. +Spring Batch provides a `ClassifierCompositeItemProcessorBuilder` to construct an +instance of the `ClassifierCompositeItemProcessor`. + [[messagingReadersAndWriters]] ==== Messaging Readers And Writers -Spring Batch offers readers and writers to some commonly used messaging systems. +Spring Batch offers the following readers and writers for commonly used messaging systems: + +* <> +* <> +* <> +* <> [[amqpItemReader]] -===== AmqpItemReader -The `AmqpItemReader` is an `ItemReader` that uses an `AmqpTemplate` to receive and/or +===== `AmqpItemReader` +The `AmqpItemReader` is an `ItemReader` that uses an `AmqpTemplate` to receive or convert messages from an exchange. Spring Batch provides a `AmqpItemReaderBuilder` to construct an instance of the `AmqpItemReader`. [[amqpItemWriter]] -===== AmqpItemWriter +===== `AmqpItemWriter` The `AmqpItemWriter` is an `ItemWriter` that uses an `AmqpTemplate` to -send messages to an AMQP exchange. Messages will be sent to the nameless exchange if -not specified on the provided `AmqpTemplate`. Spring Batch provides a `AmqpItemWriterBuilder` +send messages to an AMQP exchange. Messages are sent to the nameless exchange if the name +not specified in the provided `AmqpTemplate`. Spring Batch provides an `AmqpItemWriterBuilder` to construct an instance of the `AmqpItemWriter`. [[jmsItemReader]] -===== JmsItemReader -The `JmsItemReader` is an `ItemReader` for JMS that uses an `JmsTemplate`. The template -should have a default destination, which will be used to provide items +===== `JmsItemReader` +The `JmsItemReader` is an `ItemReader` for JMS that uses a `JmsTemplate`. The template +should have a default destination, which is used to provide items for the `read()` method. Spring Batch provides a `JmsItemReaderBuilder` to construct an instance of the `JmsItemReader`. [[jmsItemWriter]] -===== JmsItemWriter -The `JmsItemWriter` is an `ItemWriter` for JMS that uses an `JmsTemplate`. The template -should have a default destination, which will be used to send items in `write(List)`. +===== `JmsItemWriter` +The `JmsItemWriter` is an `ItemWriter` for JMS that uses a `JmsTemplate`. The template +should have a default destination, which is used to send items in `write(List)`. Spring Batch provides a `JmsItemWriterBuilder` to construct an instance of the `JmsItemWriter`. [[databaseReaders]] ==== Database Readers -Here are the database readers offered by Spring Batch out of the box. +Spring Batch offers the following database readers: + +* <> +* <> +* <> +* <> +* <> [[Neo4jItemReader]] -===== Neo4jItemReader -The `Neo4jItemReader` is an `ItemReader` that reads objects from the graph database Neo4j via a paging technique. +===== `Neo4jItemReader` +The `Neo4jItemReader` is an `ItemReader` that reads objects from the graph database Neo4j by using a paging technique. Spring Batch provides a `Neo4jItemReaderBuilder` to construct an instance of the `Neo4jItemReader`. [[mongoItemReader]] -===== MongoItemReader -The `MongoItemReader` is an `ItemReader` that reads documents from MongoDB via a paging technique. +===== `MongoItemReader` +The `MongoItemReader` is an `ItemReader` that reads documents from MongoDB by using a paging technique. Spring Batch provides a `MongoItemReaderBuilder` to construct an instance of the `MongoItemReader`. [[hibernateCursorItemReader]] -===== HibernateCursorItemReader +===== `HibernateCursorItemReader` The `HibernateCursorItemReader` is an `ItemStreamReader` for reading database -records built on top of Hibernate. It executes the HQL query, then when initialized, -iterates over the result set as the `read()` method is called, returning an object +records built on top of Hibernate. It executes the HQL query and then, when initialized, +iterates over the result set as the `read()` method is called, successively returning an object corresponding to the current row. Spring Batch provides a `HibernateCursorItemReaderBuilder` to construct an instance of the `HibernateCursorItemReader`. [[hibernatePagingItemReader]] -===== HibernatePagingItemReader +===== `HibernatePagingItemReader` The `HibernatePagingItemReader` is an `ItemReader` for reading database records built on top of Hibernate and reading only up to a fixed number of items at a time. Spring Batch provides a `HibernatePagingItemReaderBuilder` to construct an instance of the `HibernatePagingItemReader`. [[repositoryItemReader]] -===== RepositoryItemReader -The `RepositoryItemReader` is an `ItemReader` that reads records utilizing +===== `RepositoryItemReader` +The `RepositoryItemReader` is an `ItemReader` that reads records by using a `PagingAndSortingRepository`. Spring Batch provides a `RepositoryItemReaderBuilder` to construct an instance of the `RepositoryItemReader`. [[databaseWriters]] ==== Database Writers -Here are the database writers offered by Spring Batch out of the box. +Spring Batch offers the following database writers: + +* <> +* <> +* <> +* <> +* <> +* <> +* <> [[neo4jItemWriter]] -===== Neo4jItemWriter +===== `Neo4jItemWriter` The `Neo4jItemWriter` is an `ItemWriter` implementation that writes to a Neo4j database. Spring Batch provides a `Neo4jItemWriterBuilder` to construct an instance of the `Neo4jItemWriter`. [[mongoItemWriter]] -===== MongoItemWriter +===== `MongoItemWriter` The `MongoItemWriter` is an `ItemWriter` implementation that writes to a MongoDB store using an implementation of Spring Data's `MongoOperations`. Spring Batch provides a `MongoItemWriterBuilder` to construct an instance of the `MongoItemWriter`. [[repositoryItemWriter]] -===== RepositoryItemWriter +===== `RepositoryItemWriter` The `RepositoryItemWriter` is an `ItemWriter` wrapper for a `CrudRepository` from Spring Data. Spring Batch provides a `RepositoryItemWriterBuilder` to construct an instance of the `RepositoryItemWriter`. [[hibernateItemWriter]] -===== HibernateItemWriter +===== `HibernateItemWriter` The `HibernateItemWriter` is an `ItemWriter` that uses a Hibernate session to save or update entities that are not part of the current Hibernate session. Spring Batch provides a `HibernateItemWriterBuilder` to construct an instance of the `HibernateItemWriter`. [[jdbcBatchItemWriter]] -===== JdbcBatchItemWriter +===== `JdbcBatchItemWriter` The `JdbcBatchItemWriter` is an `ItemWriter` that uses the batching features from `NamedParameterJdbcTemplate` to execute a batch of statements for all items provided. @@ -3450,71 +3501,62 @@ Spring Batch provides a `JdbcBatchItemWriterBuilder` to construct an instance of the `JdbcBatchItemWriter`. [[jpaItemWriter]] -===== JpaItemWriter -The `JpaItemWriter` is an `ItemWriter` that is using a JPA EntityManagerFactory -to merge any Entities that aren't part of the persistence context. +===== `JpaItemWriter` +The `JpaItemWriter` is an `ItemWriter` that uses a JPA `EntityManagerFactory` +to merge any entities that are not part of the persistence context. Spring Batch provides a `JpaItemWriterBuilder` to construct an instance of the `JpaItemWriter`. +[[gemfireItemWriter]] +===== `GemfireItemWriter` +The `GemfireItemWriter` is an `ItemWriter` that uses a `GemfireTemplate` that +stores items in GemFire as key/value pairs. +Spring Batch provides a `GemfireItemWriterBuilder` to construct an +instance of the `GemfireItemWriter`. + [[specializedReaders]] ==== Specialized Readers -Here are some specialized readers offered by Spring Batch out of the box. +Spring Batch offers the following specialized readers: + +* <> +* <> [[ldifReader]] -===== LdifReader -The `LdifReader` reads LDIF (LDAP Data Interchange Format) records from a `Resource`, parses them +===== `LdifReader` +The `LdifReader` reads LDIF (LDAP Data Interchange Format) records from a `Resource`, parses them, and returns a `LdapAttribute` object for each `read` executed. Spring Batch provides a `LdifReaderBuilder` to construct an instance of the `LdifReader`. [[mappingLdifReader]] -===== MappingLdifReader +===== `MappingLdifReader` The `MappingLdifReader` reads LDIF (LDAP Data Interchange Format) records from a `Resource`, parses them then maps each LDIF record to a POJO (Plain Old Java Object). Each read returns a POJO. Spring Batch provides a `MappingLdifReaderBuilder` to construct an instance of the `MappingLdifReader`. [[specializedWriters]] ==== Specialized Writers -Here are some specialized writers offered by Spring Batch out of the box. +Spring Batch offers the following specialized writers: -[[classifierCompositeItemWriter]] -===== ClassifierCompositeItemWriter -The `ClassifierCompositeItemWriter` calls one of a collection of ItemWriters for -each item, based on a router pattern implemented through the provided -`Classifier`. The implementation is thread-safe if all delegates are thread-safe. -Spring Batch provides a `ClassifierCompositeItemWriterBuilder` to construct an -instance of the `ClassifierCompositeItemWriter`. +* <> [[simpleMailMessageItemWriter]] -===== SimpleMailMessageItemWriter -The SimpleMailMessageItemWriter is an `ItemWriter` that can send mail messages. +===== `SimpleMailMessageItemWriter` +The `SimpleMailMessageItemWriter` is an `ItemWriter` that can send mail messages. It delegates the actual sending of messages to an instance of `MailSender`. Spring Batch provides a `SimpleMailMessageItemWriterBuilder` to construct an instance of the `SimpleMailMessageItemWriter`. -[[gemfireItemWriter]] -===== GemfireItemWriter -The GemfireItemWriter is an `ItemWriter` that uses an `GemfireTemplate` that -stores items in GemFire as key/value pairs. -Spring Batch provides a `GemfireItemWriterBuilder` to construct an -instance of the `GemfireItemWriter`. - [[specializedProcessors]] ==== Specialized Processors -Here are some specialized processors offered by Spring Batch out of the box. +Spring Batch offers the following specialized processors: + +* <> [[scriptItemProcessor]] -===== ScriptItemProcessor +===== `ScriptItemProcessor` The `ScriptItemProcessor` is an `ItemProcessor` that passes the current item to process to the provided script and the result of the script is returned by the processor. Spring Batch provides a `ScriptItemProcessorBuilder` to construct an instance of the `ScriptItemProcessor`. - -[[classifierCompositeItemProcessor]] -===== ClassifierCompositeItemProcessor -The `ClassifierCompositeItemProcessor` is an `ItemProcessor` that calls one of a collection -of ItemProcessors, based on a router pattern implemented through the provided -`Classifier`. -Spring Batch provides a `ClassifierCompositeItemProcessorBuilder` to construct an -instance of the `ClassifierCompositeItemProcessor`.