Spring/spring-batch
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update documentation for version 4.1.0

Browse Source
This commit is contained in:
Mahmoud Ben Hassine
2018-06-05 11:26:41 +02:00
committed by Michael Minella
parent 547533fab0
commit 1046862101
3 changed files with 228 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
spring-batch-docs/asciidoc/index.adoc
View File

@@ -10,7 +10,7 @@ The reference documentation is divided into several sections:
[horizontal]
<<spring-batch-intro.adoc#spring-batch-intro,Spring Batch Introduction>> :: Background, usage
scenarios and general guidelines.
<<whatsnew.adoc#whatsNew,What's new in Spring Batch 4.0>> :: New features introduced in version 4.0.
<<whatsnew.adoc#whatsNew,What's new in Spring Batch 4.1>> :: New features introduced in version 4.1.
<<domain.adoc#domainLanguageOfBatch,The Domain Language of Batch>> :: Core concepts and abstractions
of the Batch domain language.
<<job.adoc#configureJob,Configuring and Running a Job>> :: Job configuration, execution and

117
spring-batch-docs/asciidoc/readersAndWriters.adoc
View File

@@ -15,7 +15,7 @@ out. Spring Batch provides three key interfaces to help perform bulk reading and
`ItemReader`, `ItemProcessor`, and `ItemWriter`.
[[itemReader]]
=== ItemReader
=== `ItemReader`
Although a simple concept, an `ItemReader` is the means for providing data from many
different types of input. The most general examples include:
@@ -63,7 +63,7 @@ exception to be thrown. For example, a database `ItemReader` that is configured
query that returns 0 results returns `null` on the first invocation of read.
[[itemWriter]]
=== ItemWriter
=== `ItemWriter`
`ItemWriter` is similar in functionality to an `ItemReader` but with inverse operations.
Resources still need to be located, opened, and closed but they differ in that an
@@ -93,7 +93,7 @@ one for each item. The writer can then call `flush` on the hibernate session bef
returning.
[[itemProcessor]]
=== ItemProcessor
=== `ItemProcessor`
The `ItemReader` and `ItemWriter` interfaces are both very useful for their specific
tasks, but what if you want to insert business logic before writing? One option for both
@@ -358,7 +358,7 @@ the `ItemProcessor` and only updating the
instance that is the result.
[[itemStream]]
=== ItemStream
=== `ItemStream`
Both `ItemReaders` and `ItemWriters` serve their individual purposes well, but there is a
common concern among both of them that necessitates another interface. In general, as
@@ -480,7 +480,7 @@ Delimited files are those in which fields are separated by a delimiter, such as
Fixed Length files have fields that are a set length.
[[fieldSet]]
==== The FieldSet
==== The `FieldSet`
When working with flat files in Spring Batch, regardless of whether it is for input or
output, one of the most important classes is the `FieldSet`. Many architectures and
@@ -510,7 +510,7 @@ potentially unexpected ways, it can be consistent, both when handling errors cau
format exception, or when doing simple data conversions.
[[flatFileItemReader]]
==== FlatFileItemReader
==== `FlatFileItemReader`
A flat file is any type of file that contains at most two-dimensional (tabular) data.
Reading flat files in the Spring Batch framework is facilitated by the class called
@@ -560,7 +560,7 @@ the input resource does not exist. Otherwise, it logs the problem and continues.
|===============
[[lineMapper]]
===== LineMapper
===== `LineMapper`
As with `RowMapper`, which takes a low-level construct such as `ResultSet` and returns
an `Object`, flat file processing requires the same construct to convert a `String` line
@@ -585,7 +585,7 @@ gets you halfway there. The line must be tokenized into a `FieldSet`, which can
mapped to an object, as described later in this document.
[[lineTokenizer]]
===== LineTokenizer
===== `LineTokenizer`
An abstraction for turning a line of input into a `FieldSet` is necessary because there
can be many formats of flat file data that need to be converted to a `FieldSet`. In
@@ -614,7 +614,7 @@ width". The width of each field must be defined for each record type.
tokenizers should be used on a particular line by checking against a pattern.
[[fieldSetMapper]]
===== FieldSetMapper
===== `FieldSetMapper`
The `FieldSetMapper` interface defines a single method, `mapFieldSet`, which takes a
`FieldSet` object and maps its contents to an object. This object may be a custom DTO, a
@@ -634,7 +634,7 @@ public interface FieldSetMapper<T> {
The pattern used is the same as the `RowMapper` used by `JdbcTemplate`.
[[defaultLineMapper]]
===== DefaultLineMapper
===== `DefaultLineMapper`
Now that the basic interfaces for reading in flat files have been defined, it becomes
clear that three basic steps are required:
@@ -1039,7 +1039,7 @@ file. `FlatFileFormatException` is thrown by implementations of the `LineTokeniz
interface and indicates a more specific error encountered while tokenizing.
[[incorrectTokenCountException]]
====== IncorrectTokenCountException
====== `IncorrectTokenCountException`
Both `DelimitedLineTokenizer` and `FixedLengthLineTokenizer` have the ability to specify
column names that can be used for creating a `FieldSet`. However, if the number of column
@@ -1064,7 +1064,7 @@ Because the tokenizer was configured with 4 column names but only 3 tokens were
the file, an `IncorrectTokenCountException` was thrown.
[[incorrectLineLengthException]]
====== IncorrectLineLengthException
====== `IncorrectLineLengthException`
Files formatted in a fixed-length format have additional requirements when parsing
because, unlike a delimited format, each column must strictly adhere to its predefined
@@ -1110,14 +1110,14 @@ line lengths when tokenizing the line. A `FieldSet` is now correctly created and
returned. However, it contains only empty tokens for the remaining values.
[[flatFileItemWriter]]
==== FlatFileItemWriter
==== `FlatFileItemWriter`
Writing out to flat files has the same problems and issues that reading in from a file
must overcome. A step must be able to write either delimited or fixed length formats in a
transactional manner.
[[lineAggregator]]
===== LineAggregator
===== `LineAggregator`
Just as the `LineTokenizer` interface is necessary to take an item and turn it into a
`String`, file writing must have a way to aggregate multiple fields into a single string
@@ -1138,7 +1138,7 @@ The `LineAggregator` is the logical opposite of `LineTokenizer`. `LineTokenizer
`String`.
[[PassThroughLineAggregator]]
====== PassThroughLineAggregator
====== `PassThroughLineAggregator`
The most basic implementation of the `LineAggregator` interface is the
`PassThroughLineAggregator`, which assumes that the object is already a string or that
@@ -1205,7 +1205,7 @@ public FlatFileItemWriter itemWriter() {
----
[[FieldExtractor]]
===== FieldExtractor
===== `FieldExtractor`
The preceding example may be useful for the most basic uses of a writing to a file.
However, most users of the `FlatFileItemWriter` have a domain object that needs to be
@@ -1242,7 +1242,7 @@ of the provided object, which can then be written out with a delimiter between t
elements or as part of a fixed-width line.
[[PassThroughFieldExtractor]]
====== PassThroughFieldExtractor
====== `PassThroughFieldExtractor`
There are many cases where a collection, such as an array, `Collection`, or `FieldSet`,
needs to be written out. "Extracting" an array from one of these collection types is very
@@ -1252,7 +1252,7 @@ the object passed in is not a type of collection, then the `PassThroughFieldExtr
returns an array containing solely the item to be extracted.
[[BeanWrapperFieldExtractor]]
====== BeanWrapperFieldExtractor
====== `BeanWrapperFieldExtractor`
As with the `BeanWrapperFieldSetMapper` described in the file reading section, it is
often preferable to configure how to convert a domain object to an object array, rather
@@ -1474,7 +1474,7 @@ With an introduction to OXM and how one can use XML fragments to represent recor
can now more closely examine readers and writers.
[[StaxEventItemReader]]
==== StaxEventItemReader
==== `StaxEventItemReader`
The `StaxEventItemReader` configuration provides a typical setup for the processing of
records from an XML input stream. First, consider the following set of XML records that
@@ -1509,8 +1509,7 @@ To be able to process the XML records, the following is needed:
* Root Element Name: The name of the root element of the fragment that constitutes the
object to be mapped. The example configuration demonstrates this with the value of trade.
* Resource: A Spring Resource that represents the file to be
read.
* Resource: A Spring Resource that represents the file to read.
* `Unmarshaller`: An unmarshalling facility provided by Spring OXM for mapping the XML
fragment to an object.
@@ -1631,7 +1630,7 @@ while (hasNext) {
----
[[StaxEventItemWriter]]
==== StaxEventItemWriter
==== `StaxEventItemWriter`
Output works symmetrically to input. The `StaxEventItemWriter` needs a `Resource`, a
marshaller, and a `rootTagName`. A Java object is passed to a marshaller (typically a
@@ -1748,6 +1747,64 @@ trade.setCustomer("Customer1");
staxItemWriter.write(trade);
----
[[jsonReadingWriting]]
=== JSON Item Readers
Spring Batch provides support for reading JSON resources in the following format:
[source, json]
----
[
{
"isin": "123",
"quantity": 1,
"price": 1.2,
"customer": "foo"
},
{
"isin": "456",
"quantity": 2,
"price": 1.4,
"customer": "bar"
}
]
----
It is assumed that the JSON resource is an array of JSON objects corresponding to
individual items. Spring Batch is not tied to any particular JSON library.
[[JsonItemReader]]
==== `JsonItemReader`
The `JsonItemReader` delegates JSON parsing and binding to implementations of the
`org.springframework.batch.item.json.JsonObjectReader` interface. This interface
is intended to be implemented by using a streaming API to read JSON objects
in chunks. Two implementations are currently provided:
* link:$$https://github.com/FasterXML/jackson$$[Jackson] through the `org.springframework.batch.item.json.JacksonJsonObjectReader`
* link:$$https://github.com/google/gson$$[Gson] through the `org.springframework.batch.item.json.GsonJsonObjectReader`
To be able to process JSON records, the following is needed:
* `Resource`: A Spring Resource that represents the JSON file to read.
* `JsonObjectReader`: A JSON object reader to parse and bind JSON objects to items
The following example shows how to define a `JsonItemReader` that works with the
previous JSON resource `org/springframework/batch/item/json/trades.json` and a
`JsonObjectReader` based on Jackson:
[source, java]
----
@Bean
public JsonItemReader<Trade> jsonItemReader() {
return new JsonItemReaderBuilder<Trade>()
.jsonObjectReader(new JacksonJsonObjectReader<>(Trade.class))
.resource(new ClassPathResource("trades.json"))
.name("tradeJsonItemReader")
.build();
}
----
[[multiFileInput]]
=== Multi-File Input
@@ -1835,7 +1892,7 @@ which is the `Foo` with an ID of 3. The results of these reads are written out a
maintaining references to them).
[[JdbcCursorItemReader]]
===== JdbcCursorItemReader
===== `JdbcCursorItemReader`
`JdbcCursorItemReader` is the JDBC implementation of the cursor-based technique. It works
directly with a `ResultSet` and requires an SQL statement to run against a connection
@@ -2250,7 +2307,7 @@ fetches a portion of the results. We refer to this portion as a page. Each query
specify the starting row number and the number of rows that we want returned in the page.
[[JdbcPagingItemReader]]
===== JdbcPagingItemReader
===== `JdbcPagingItemReader`
One implementation of a paging `ItemReader` is the `JdbcPagingItemReader`. The
`JdbcPagingItemReader` needs a `PagingQueryProvider` responsible for providing the SQL
@@ -2339,7 +2396,7 @@ match the name of the named parameter. If you use a traditional '?' placeholder,
key for each entry should be the number of the placeholder, starting with 1.
[[JpaPagingItemReader]]
===== JpaPagingItemReader
===== `JpaPagingItemReader`
Another implementation of a paging `ItemReader` is the `JpaPagingItemReader`. JPA does
not have a concept similar to the Hibernate `StatelessSession`, so we have to use other
@@ -2645,7 +2702,7 @@ implementations. This section shows, by using a simple example, how to create a
writer restartable.
[[customReader]]
==== Custom ItemReader Example
==== Custom `ItemReader` Example
For the purpose of this example, we create a simple `ItemReader` implementation that
reads from a provided list. We start by implementing the most basic contract of
@@ -2691,7 +2748,7 @@ assertNull(itemReader.read());
----
[[restartableReader]]
===== Making the ItemReader Restartable
===== Making the `ItemReader` Restartable
The final challenge is to make the `ItemReader` restartable. Currently, if processing is
interrupted and begins again, the `ItemReader` must start at the beginning. This is
@@ -2780,7 +2837,7 @@ output), a more unique name is needed. For this reason, many of the Spring Batch
key name be overridden.
[[customWriter]]
==== Custom ItemWriter Example
==== Custom `ItemWriter` Example
Implementing a Custom `ItemWriter` is similar in many ways to the `ItemReader` example
above but differs in enough ways as to warrant its own example. However, adding
@@ -2805,7 +2862,7 @@ public class CustomItemWriter<T> implements ItemWriter<T> {
----
[[restartableWriter]]
===== Making the ItemWriter Restartable
===== Making the `ItemWriter` Restartable
To make the `ItemWriter` restartable, we would follow the same process as for the
`ItemReader`, adding and implementing the `ItemStream` interface to synchronize the
@@ -2876,7 +2933,7 @@ Batch provides a `ClassifierCompositeItemWriterBuilder` to construct an instance
`ClassifierCompositeItemWriter`.
[[classifierCompositeItemProcessor]]
===== ClassifierCompositeItemProcessor
===== `ClassifierCompositeItemProcessor`
The `ClassifierCompositeItemProcessor` is an `ItemProcessor` that calls one of a
collection of `ItemProcessor` implementations, based on a router pattern implemented
through the provided `Classifier`. Spring Batch provides a

192
spring-batch-docs/asciidoc/whatsnew.adoc
View File

@@ -4,69 +4,157 @@
[[whatsNew]]
== What's New in Spring Batch 4.0
== What's New in Spring Batch 4.1
The Spring Batch 4.0 release has three major themes:
The Spring Batch 4.1 release adds the following features:
* A new `@SpringBatchTest` annotation to simplify testing batch components
* A new `@EnableBatchIntegration` annotation to simplify remote chunking configuration
* A new `JsonItemReader` to support the JSON format
[[whatsNewTesting]]
=== `@SpringBatchTest` Annotation
Spring Batch provides some nice utility classes (such as the `JobLauncherTestUtils` and
`JobRepositoryTestUtils`) and test execution listeners (`StepScopeTestExecutionListener`
and `JobScopeTestExecutionListener`) to test batch components. However, in order
to use these utilities, you must configure them explicitly. This release introduces
a new annotation named `@SpringBatchTest` that automatically adds utility beans and
listeners to the test context and makes them available for autowiring,
as the following example shows:
[source, java]
----
@RunWith(SpringRunner.class)
@SpringBatchTest
@ContextConfiguration(classes = {JobConfiguration.class})
public class JobTest {
@Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
@Autowired
private JobRepositoryTestUtils jobRepositoryTestUtils;
* Java 8 Requirement
* Dependencies Re-baseline
* Builders for ItemReaders and ItemWriters
@Before
public void clearMetadata() {
jobRepositoryTestUtils.removeJobExecutions();
}
[[whatsNewJava]]
@Test
public void testJob() throws Exception {
// given
JobParameters jobParameters =
jobLauncherTestUtils.getUniqueJobParameters();
// when
JobExecution jobExecution =
jobLauncherTestUtils.launchJob(jobParameters);
=== Java 8 Requirement
// then
Assert.assertEquals(ExitStatus.COMPLETED,
jobExecution.getExitStatus());
}
Spring Batch has historically followed Spring Framework's baselines for both java version
and third party dependencies. With Spring Batch 4, the Spring Framework version is being
upgraded to Spring Framework 5. As a result, the Java version requirement for Spring
Batch is also increasing to Java 8. This should be mainly an internal change, in that
most of the framework has supported things like functional interfaces and lambdas since
they were released.
}
----
For more details about this new annotation, see the
<<testing.adoc#creatingUnitTestClass,Unit Testing>> chapter.
[[whatsNewDependencies]]
=== Dependencies Re-baseline
[[whatsNewIntegration]]
=== `@EnableBatchIntegration` Annotation
In order to continue to integrate with supported versions of the third party libraries
Spring Batch uses, Spring Batch 4 is updating the dependencies across the board. The new
dependency versions align with Spring Framework 5.
Setting up a remote chunking job requires the definition of a number of beans:
* A connection factory to acquire connections from the messaging middleware (JMS, AMQP, and others)
* A `MessagingTemplate` to send requests from the master to the workers and back again
* An input channel and an output channel for Spring Integration to get messages from the messaging middleware
* A special item writer (`ChunkMessageChannelItemWriter`) on the master side that knows how to send chunks of data to workers for processing and writing
* A message listener (`ChunkProcessorChunkHandler`) on the worker side to receive data from the master
[[whatsNewBuilders]]
=== Provide builders for the ItemReaders, ItemProcessors, and ItemWriters
This can be a bit daunting at first glance. This release introduces a new annotation
named `@EnableBatchIntegration` as well as new APIs (`RemoteChunkingMasterStepBuilder`
and `RemoteChunkingWorkerBuilder`) to simplify the configuration. The following
example shows how to use the new annotation and APIs:
Spring Batch 4 is providing a collection of builders for all of the `ItemReader`
implementations and `ItemWriter` implementations that come with the framework. As of
this release, builders for the following components (as well as some related utility
builders) are available:
[source, java]
----
@Configuration
@EnableBatchProcessing
@EnableBatchIntegration
public class RemoteChunkingAppConfig {
@Autowired
private RemoteChunkingMasterStepBuilderFactory masterStepBuilderFactory;
@Autowired
private RemoteChunkingWorkerBuilder workerBuilder;
@Bean
public TaskletStep masterStep() {
return this.masterStepBuilderFactory
.get("masterStep")
.chunk(100)
.reader(itemReader())
.outputChannel(outgoingRequestsToWorkers())
.inputChannel(incomingRepliesFromWorkers())
.build();
}
@Bean
public IntegrationFlow worker() {
return this.workerBuilder
.itemProcessor(itemProcessor())
.itemWriter(itemWriter())
.inputChannel(incomingRequestsFromMaster())
.outputChannel(outgoingRepliesToMaster())
.build();
}
// Middleware beans setup omitted
}
----
This new annotation and builders take care of the heavy lifting of configuring
infrastructure beans. You can now easily configure a master step as well as
a Spring Integration flow on the worker side. You can find a remote chunking sample
that uses these new APIs in the
link:$$https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples#remote-chunking-sample$$[samples module]
as well as more details in the <<spring-batch-integration.adoc#remote-chunking,Spring Batch Integration>> chapter.
[[whatsNewJson]]
=== JSON support
Spring Batch 4.1 adds support for the JSON format. This release introduces a new
item reader that can read a JSON resource in the following format:
[source, json]
----
[
{
"isin": "123",
"quantity": 1,
"price": 1.2,
"customer": "foo"
},
{
"isin": "456",
"quantity": 2,
"price": 1.4,
"customer": "bar"
}
]
----
Similar to the `StaxEventItemReader` for XML, the new `JsonItemReader` uses streaming
APIs to read JSON objects in chunks. Spring Batch supports two libraries:
* link:$$https://github.com/FasterXML/jackson$$[Jackson]
* link:$$https://github.com/google/gson$$[Gson]
To add other libraries, you can implement the `JsonObjectReader` interface.
For more details about JSON support, see the
<<readersAndWriters.adoc#jsonReadingWriting,ItemReaders and ItemWriters>> chapter.
* `AmqpItemReader` - `AmqpItemReaderBuilder`
* `ClassifierCompositeItemProcessor` - `ClassifierCompositeItemProcessorBuilder`
* `ClassifierCompositeItemWriter` - `ClassifierCompositeItemWriterBuilder`
* `CompositeItemWriter` - `CompositeItemWriterBuilder`
* `FlatFileItemReader` - `FlatFileItemReaderBuilder`
* `FlatFileItemWriter` - `FlatFileItemWriterBuilder`
* `GemfireItemWriter` - `GemfireItemWriterBuilder`
* `HibernateCursorItemReader` - `HibernateCursorItemReaderBuilder`
* `HibernateItemWriter` - `HibernateItemWriterBuilder`
* `HibernatePagingItemReader` - `HibernatePagingItemReaderBuilder`
* `JdbcBatchItemWriter` - `JdbcBatchItemWriterBuilder`
* `JdbcCursorItemReader` - `JdbcCursorItemReaderBuilder`
* `JdbcPagingItemReader` - `JdbcPagingItemReaderBuilder`
* `JmsItemReader` - `JmsItemReaderBuilder`
* `JmsItemWriter` - `JmsItemWriterBuilder`
* `JpaPagingItemReader` - `JpaPagingItemReaderBuilder`
* `MongoItemReader` - `MongoItemReaderBuilder`
* `MultiResourceItemReader` - `MultiResourceItemReaderBuilder`
* `MultiResourceItemWriter` - `MultiResourceItemWriterBuilder`
* `Neo4jItemWriter` - `Neo4jItemWriterBuilder`
* `RepositoryItemReader` - `RepositoryItemReaderBuilder`
* `RepositoryItemWriter` - `RepositoryItemWriterBuilder`
* `ScriptItemProcessor` - `ScriptItemProcessorBuilder`
* `SimpleMailMessageItemWriter` - `SimpleMailMessageItemWriterBuilder`
* `SingleItemPeekableItemReader` - `SingleItemPeekableItemReaderBuilder`
* `StaxEventItemReader` - `StaxEventItemReaderBuilder`
* `StaxEventItemWriter` - `StaxEventItemWriterBuilder`
* `SynchronizedItemStreamReader` - `SynchronizedItemStreamReaderBuilder`