diff --git a/spring-batch-docs/asciidoc/appendix.adoc b/spring-batch-docs/asciidoc/appendix.adoc index f05d1331f..5b4a78cc2 100644 --- a/spring-batch-docs/asciidoc/appendix.adoc +++ b/spring-batch-docs/asciidoc/appendix.adoc @@ -78,7 +78,7 @@ |AmqpItemWriter|Given a Spring `AmqpTemplate`, it provides for a synchronous `send` method. The `convertAndSend(Object)` method lets you send POJO objects. -|CompositeItemWriter|Passes an item to the `process` method of each +|CompositeItemWriter|Passes an item to the `write` method of each in an injected `List` of `ItemWriter` objects. |FlatFileItemWriter|Writes to a flat file. Includes `ItemStream` and Skippable functionality. See link:readersAndWriters.html#flatFileItemWriter[`FlatFileItemWriter`]. @@ -116,7 +116,7 @@ names. |RepositoryItemWriter|Given a Spring Data `CrudRepository` implementation, items are saved through the method specified in the configuration. -|StaxEventItemWriter|Uses an `ObjectToXmlSerializer` implementation to +|StaxEventItemWriter|Uses a `Marshaller` implementation to convert each item to XML and then writes it to an XML file using StAX. diff --git a/spring-batch-docs/asciidoc/common-patterns.adoc b/spring-batch-docs/asciidoc/common-patterns.adoc index f2ad16dfb..088e04a1d 100644 --- a/spring-batch-docs/asciidoc/common-patterns.adoc +++ b/spring-batch-docs/asciidoc/common-patterns.adoc @@ -44,13 +44,13 @@ public class ItemFailureLoggerListener extends ItemListenerSupport { logger.error("Encountered error on read", e); } - public void onWriteError(Exception ex, Object item) { + public void onWriteError(Exception ex, List items) { logger.error("Encountered error on write", ex); } } ---- -Having implemented this listener it, must be registered with a step, as shown in the +Having implemented this listener, it must be registered with a step, as shown in the following example: .XML Configuration @@ -78,7 +78,7 @@ public Step simpleStep() { } ---- -Remember that, if your listener does anything in an `onError()` method, it must be inside +NOTE: if your listener does anything in an `onError()` method, it must be inside a transaction that is going to be rolled back. If you need to use a transactional resource, such as a database, inside an `onError()` method, consider adding a declarative transaction to that method (see Spring Core Reference Guide for details), and giving its @@ -98,12 +98,14 @@ in the following example: [source, java] ---- -public class PoisonPillItemWriter implements ItemWriter { +public class PoisonPillItemProcessor implements ItemProcessor { - public void write(T item) throws Exception { + @Override + public T process(T item) throws Exception { if (isPoisonPill(item)) { throw new PoisonPillException("Poison pill detected: " + item); - } + } + return item; } } ---- @@ -259,7 +261,7 @@ public class TradeItemWriter implements ItemWriter, private BigDecimal totalAmount = BigDecimal.ZERO; - public void write(List items) { + public void write(List items) throws Exception { BigDecimal chunkTotal = BigDecimal.ZERO; for (Trade trade : items) { chunkTotal = chunkTotal.add(trade.getAmount()); @@ -282,7 +284,7 @@ public class TradeItemWriter implements ItemWriter, This `TradeItemWriter` stores a `totalAmount` value that is increased with the `amount` from each `Trade` item written. After the last `Trade` is processed, the framework calls `writeFooter`, which puts the `totalAmount` into the file. Note that the `write` method -makes use of a temporary variable, `chunkTotalAmount`, that stores the total of the +makes use of a temporary variable, `chunkTotal`, that stores the total of the `Trade` amounts in the chunk. This is done to ensure that, if a skip occurs in the `write` method, the `totalAmount` is left unchanged. It is only at the end of the `write` method, once we are guaranteed that no exceptions are thrown, that we update the @@ -384,7 +386,7 @@ into a full 'Foo' object. An existing DAO can be used to query for the full obje on the key. [[multiLineRecords]] -==== Multi-Line Records +=== Multi-Line Records While it is usually the case with flat files that each record is confined to a single line, it is common that a file might have records spanning multiple lines with multiple @@ -419,9 +421,7 @@ do this, a custom `ItemReader` should be implemented as a wrapper for the - - - + @@ -445,10 +445,11 @@ public MultiLineTradeItemReader itemReader() { public FlatFileItemReader flatFileItemReader() { FlatFileItemReader reader = new FlatFileItemReaderBuilder() .name("flatFileItemReader") - .resource(new ClasspathResource("data/iosample/input/multiLine.txt")) + .resource(new ClassPathResource("data/iosample/input/multiLine.txt")) .lineTokenizer(orderFileTokenizer()) - .fieldSetMapper(new PassThroughFieldSetMapper()) + .fieldSetMapper(orderFieldSetMapper()) .build(); + return reader; } ---- @@ -490,7 +491,7 @@ public PatternMatchingCompositeLineTokenizer orderFileTokenizer() { tokenizers.put("NCU*", customerLineTokenizer()); tokenizers.put("BAD*", billingAddressLineTokenizer()); - tokenizer.setTokenizers(tokenizers()); + tokenizer.setTokenizers(tokenizers); return tokenizer; } @@ -578,11 +579,11 @@ exceptional. The `Step` is simply considered to have found no work and completes items read. All of the `ItemReader` implementations provided out of the box in Spring Batch default to this approach. This can lead to some confusion if nothing is written out even when input is present (which usually happens if a file was misnamed or some similar -issue arises) For this reason, the metadata itself should be inspected to determine how +issue arises). For this reason, the metadata itself should be inspected to determine how much work the framework found to be processed. However, what if finding no input is considered exceptional? In this case, programmatically checking the metadata for no items processed and causing failure is the best solution. Because this is a common use case, -Spring Batch provides a listener is provided with exactly this functionality, as shown in +Spring Batch provides a listener with exactly this functionality, as shown in the class definition for `NoWorkFoundStepExecutionListener`: [source, java] @@ -644,7 +645,7 @@ To make the data available to future `Steps`, it must be "promoted" to the `Job` with the keys related to the data in the `ExecutionContext` that must be promoted. It can also, optionally, be configured with a list of exit code patterns for which the promotion should occur (`COMPLETED` is the default). As with all listeners, it must be registered -on the`Step` as shown in the following example: +on the `Step` as shown in the following example: .XML Configuration [source, xml, role="xmlContent"] @@ -665,7 +666,11 @@ on the`Step` as shown in the following example: - + + + someKey + + ---- @@ -677,7 +682,6 @@ public Job job1() { return this.jobBuilderFactory.get("job1") .start(step1()) .next(step1()) - .end() .build(); } @@ -695,7 +699,7 @@ public Step step1() { public ExecutionContextPromotionListener promotionListener() { ExecutionContextPromotionListener listener = new ExecutionContextPromotionListener(); - listener.setKeys("someKey"); + listener.setKeys(new String[] {"someKey" }); return listener; } diff --git a/spring-batch-docs/asciidoc/domain.adoc b/spring-batch-docs/asciidoc/domain.adoc index dd57a4e73..842ce9647 100644 --- a/spring-batch-docs/asciidoc/domain.adoc +++ b/spring-batch-docs/asciidoc/domain.adoc @@ -170,8 +170,8 @@ image::{batch-asciidoc}images/job-stereotypes-parameters.png[Job Parameters, sca In the preceding example, where there are two instances, one for January 1st, and another for January 2nd, there is really only one `Job`, but it has two `JobParameter` objects: one that was started with a job parameter of 01-01-2017 and another that was started with -a parameter of 01-02-2017. Thus, the contract can be defined as: `JobInstance` = `Job` + -identifying `JobParameters`. This allows a developer to effectively control how a +a parameter of 01-02-2017. Thus, the contract can be defined as: `JobInstance` = `Job` + + identifying `JobParameters`. This allows a developer to effectively control how a `JobInstance` is defined, since they control what parameters are passed in. NOTE: Not all job parameters are required to contribute to the identification of a @@ -384,12 +384,12 @@ finishes successfully, the status is `BatchStatus.COMPLETED`. |startTime |A `java.util.Date` representing the current system time when the execution was started. -This field is empty if the job has yet to start. +This field is empty if the step has yet to start. |endTime |A `java.util.Date` representing the current system time when the execution finished, -regardless of whether or not it was successful. This field is empty if the job has yet to +regardless of whether or not it was successful. This field is empty if the step has yet to exit. |exitStatus @@ -456,7 +456,7 @@ the metadata tables would look like the following example: |EndOfDayJob |=== -.BATCH_JOB_PARAMS +.BATCH_JOB_EXECUTION_PARAMS |=== |JOB_INST_ID|TYPE_CD|KEY_NAME|DATE_VAL |1 @@ -587,7 +587,8 @@ When using java configuration, `@EnableBatchProcessing` annotation provides a public interface JobLauncher { public JobExecution run(Job job, JobParameters jobParameters) - throws JobExecutionAlreadyRunningException, JobRestartException; + throws JobExecutionAlreadyRunningException, JobRestartException, + JobInstanceAlreadyCompleteException, JobParametersInvalidException; } ---- It is expected that implementations obtain a valid `JobExecution` from the diff --git a/spring-batch-docs/asciidoc/job.adoc b/spring-batch-docs/asciidoc/job.adoc index 0cc78d5c3..4f71f060d 100644 --- a/spring-batch-docs/asciidoc/job.adoc +++ b/spring-batch-docs/asciidoc/job.adoc @@ -19,7 +19,7 @@ image::{batch-asciidoc}images/spring-batch-reference-model.png[Figure 2.1: Batch While the `Job` object may seem like a simple container for steps, there are many configuration options of which a -developers must be aware . Furthermore, there are many considerations for +developer must be aware. Furthermore, there are many considerations for how a `Job` will be run and how its meta-data will be stored during that run. This chapter will explain the various configuration options and runtime concerns of a `Job`. @@ -48,7 +48,7 @@ public Job footballJob() { [role="javaContent"] A `Job` (and typically any `Step` within it) requires a `JobRepository`. The -configuration of the `JobRepository` is handled via the <>. +configuration of the `JobRepository` is handled via the <>. [role="javaContent"] The above example illustrates a `Job` that consists of three `Step` instances. The job related @@ -58,8 +58,8 @@ declarative flow control (`Decision`) and externalization of flow definitions (` [role="xmlContent"] There are multiple implementations of the <> interface, however, the namespace abstracts away the differences in configuration. It has only three -required dependencies: a name, `JobRepository` , and -a list of `Step` s. +required dependencies: a name, a `JobRepository` , and +a list of ``Step``s. [source, xml, role="xmlContent"] ---- @@ -265,7 +265,7 @@ public Job footballJob() { } ---- -It should be noted that afterJob will be +It should be noted that `afterJob` will be called regardless of the success or failure of the Job. If success or failure needs to be determined it can be obtained from the `JobExecution`: @@ -342,7 +342,7 @@ endif::backend-pdf[] ==== JobParametersValidator A job declared in the XML namespace or using any subclass of - AbstractJob can optionally declare a validator for the job parameters at + `AbstractJob` can optionally declare a validator for the job parameters at runtime. This is useful when for instance you need to assert that a job is started with all its mandatory parameters. There is a `DefaultJobParametersValidator` that can be used to constrain combinations @@ -358,7 +358,7 @@ The configuration of a validator is supported through the XML namespace through ---- - + ---- @@ -402,7 +402,7 @@ XML namespace support is also available for configuration of a `JobParametersVal ---- - + ---- @@ -416,8 +416,8 @@ endif::backend-pdf[] === Java Config -Spring 3 brought the ability to configure applications via java instead - of XML. As of Spring Batch 2.2.0, batch jobs can be configured using the same +Spring 3 brought the ability to configure applications via java in addition to XML. + As of Spring Batch 2.2.0, batch jobs can be configured using the same java config. There are two components for the java based configuration: the `@EnableBatchProcessing` annotation and two builders. @@ -520,8 +520,8 @@ As described in earlier, the <> is used f domain objects within Spring Batch, such as `JobExecution` and `StepExecution`. It is required by many of the major - framework features, such as the JobLauncher, - Job, and `Step`. + framework features, such as the `JobLauncher`, + `Job`, and `Step`. [role="xmlContent"] The batch @@ -564,7 +564,7 @@ protected JobRepository createJobRepository() throws Exception { JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean(); factory.setDataSource(dataSource); factory.setTransactionManager(transactionManager); - factory.setIsolationLevelForCreate("SERIALIZABLE"); + factory.setIsolationLevelForCreate("ISOLATION_SERIALIZABLE"); factory.setTablePrefix("BATCH_"); factory.setMaxVarCharLength(1000); return factory.getObject(); @@ -617,7 +617,7 @@ protected JobRepository createJobRepository() throws Exception { JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean(); factory.setDataSource(dataSource); factory.setTransactionManager(transactionManager); - factory.setIsolationLevelForCreate("REPEATABLE_READ"); + factory.setIsolationLevelForCreate("ISOLATION_REPEATABLE_READ"); return factory.getObject(); } ---- @@ -744,7 +744,7 @@ protected JobRepository createJobRepository() throws Exception { JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean(); factory.setDataSource(dataSource); factory.setTransactionManager(transactionManager); - factory.setIsolationLevelForCreate("REPEATABLE_READ"); + factory.setIsolationLevelForCreate("ISOLATION_REPEATABLE_READ"); return factory.getObject(); } @@ -892,11 +892,11 @@ The `SimpleJobLauncher` can easily be ---- @Bean public JobLauncher jobLauncher() { - SimpleJobLauncher jobLauncher = new SimpleJobLauncher(); - jobLauncher.setJobRepository(jobRepository()); - jobLauncher.setTaskExecutor(new SimpleAsyncTaskExecutor()); - jobLauncher.afterPropertiesSet(); - return jobLauncher; + SimpleJobLauncher jobLauncher = new SimpleJobLauncher(); + jobLauncher.setJobRepository(jobRepository()); + jobLauncher.setTaskExecutor(new SimpleAsyncTaskExecutor()); + jobLauncher.afterPropertiesSet(); + return jobLauncher; } ---- @@ -951,11 +951,11 @@ Because the script launching the job must kick off a Java * Load the appropriate - ApplicationContext + `ApplicationContext` * Parse command line arguments into - JobParameters + `JobParameters` * Locate the appropriate job based on arguments @@ -971,7 +971,7 @@ All of these tasks are accomplished using only the arguments |=============== |jobPath|The location of the XML file that will be used to - create an ApplicationContext. This file + create an `ApplicationContext`. This file should contain everything needed to run the complete Job |jobName|The name of the job to be run. @@ -981,7 +981,7 @@ All of these tasks are accomplished using only the arguments These arguments must be passed in with the path first and the name second. All arguments after these are considered to be - JobParameters and must be in the format of 'name=value': + `JobParameters` and must be in the format of 'name=value': [source, role="xmlContent"] @@ -1000,10 +1000,10 @@ In most cases you would want to use a manifest to declare your main class in a jar, but for simplicity, the class was used directly. This example is using the same 'EndOfDay' example from the <>. The first argument is 'endOfDayJob.xml', which is the Spring - ApplicationContext containing the + `ApplicationContext` containing the Job. The second argument, 'endOfDay' represents the job name. The final argument, 'schedule.date(date)=2007/05/05' - will be converted into JobParameters. An + will be converted into `JobParameters`. An example of the XML configuration is below: @@ -1042,18 +1042,18 @@ public class EndOfDayJobConfiguration { private StepBuilderFactory stepBuilderFactory; @Bean - public Job endOfDay() { - return this.jobBuilderFactory.get("endOfDay") - .start(step1()) - .build(); - } + public Job endOfDay() { + return this.jobBuilderFactory.get("endOfDay") + .start(step1()) + .build(); + } - @Bean - public Step step1() { - return this.stepBuilderFactory.get("step1") - .tasklet((contribution, chunkContext) -> null) - .build(); - } + @Bean + public Step step1() { + return this.stepBuilderFactory.get("step1") + .tasklet((contribution, chunkContext) -> null) + .build(); + } } ---- endif::backend-html5[] @@ -1065,7 +1065,7 @@ In most cases you would want to use a manifest to declare your where your job is configured (either an XML file or a fully qualified class name). The second argument, 'endOfDay' represents the job name. The final argument, 'schedule.date(date)=2007/05/05' - will be converted into JobParameters. An + will be converted into `JobParameters`. An example of the configuration is below: .XML Configuration @@ -1116,7 +1116,7 @@ This example is overly simplistic, since there are many more serves to show the two main requirements of the `CommandLineJobRunner`: `Job` and - JobLauncher + `JobLauncher` [[exitCodes]] @@ -1137,15 +1137,15 @@ When launching a batch job from the command-line, an enterprise but it is important that a processing framework such as Spring Batch provide a way to return a numeric representation of the 'Exit Code' for a particular batch job. In Spring Batch this is encapsulated - within an ExitStatus, which is covered in more + within an `ExitStatus`, which is covered in more detail in Chapter 5. For the purposes of discussing exit codes, the only important thing to know is that an - ExitStatus has an exit code property that is + `ExitStatus` has an exit code property that is set by the framework (or the developer) and is returned as part of the `JobExecution` returned from the - JobLauncher. The + `JobLauncher`. The `CommandLineJobRunner` converts this string value - to a number using the ExitCodeMapper + to a number using the `ExitCodeMapper` interface: @@ -1159,25 +1159,25 @@ public interface ExitCodeMapper { ---- The essential contract of an - ExitCodeMapper is that, given a string exit + `ExitCodeMapper` is that, given a string exit code, a number representation will be returned. The default - implementation used by the job runner is the SimpleJvmExitCodeMapper + implementation used by the job runner is the `SimpleJvmExitCodeMapper` that returns 0 for completion, 1 for generic errors, and 2 for any job runner errors such as not being able to find a `Job` in the provided context. If anything more complex than the 3 values above is needed, then a custom - implementation of the ExitCodeMapper interface + implementation of the `ExitCodeMapper` interface must be supplied. Because the `CommandLineJobRunner` is the class that creates - an ApplicationContext, and thus cannot be + an `ApplicationContext`, and thus cannot be 'wired together', any values that need to be overwritten must be autowired. This means that if an implementation of - ExitCodeMapper is found within the BeanFactory, + `ExitCodeMapper` is found within the `BeanFactory`, it will be injected into the runner after the context is created. All that needs to be done to provide your own - ExitCodeMapper is to declare the implementation + `ExitCodeMapper` is to declare the implementation as a root level bean and ensure that it is part of the - ApplicationContext that is loaded by the + `ApplicationContext` that is loaded by the runner. [[runningJobsFromWebContainer]] @@ -1315,7 +1315,7 @@ public JobExplorer getJobExplorer() throws Exception { <>, it was mentioned that the table prefix of the `JobRepository` can be modified to allow for different versions or schemas. Because the - JobExplorer is working with the same tables, it + `JobExplorer` is working with the same tables, it too needs the ability to set a prefix: .XML Configuration @@ -1404,7 +1404,7 @@ public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor() { } ---- -Although it is not strictly necessary the post-processor in the +Although it is not strictly necessary, the post-processor in the example has been given an id so that it can be included in child contexts (e.g. as a parent bean definition) and cause all jobs created there to also be registered automatically. @@ -1420,10 +1420,10 @@ This is a lifecycle component that creates child contexts and have "natural" names. So for example, you can create a set of XML configuration files each having only one Job, but all having different definitions of an - ItemReader with the same bean name, e.g. + `ItemReader` with the same bean name, e.g. "reader". If all those files were imported into the same context, the reader definitions would clash and override one another, but with the - automatic regsistrar this is avoided. This makes it easier to + automatic registrar this is avoided. This makes it easier to integrate jobs contributed from separate modules of an application. @@ -1461,26 +1461,26 @@ public AutomaticJobRegistrar registrar() { ---- The registrar has two mandatory properties, one is an array of - ApplicationContextFactory (here created from a + `ApplicationContextFactory` (here created from a convenient factory bean), and the other is a - JobLoader. The JobLoader + `JobLoader`. The `JobLoader` is responsible for managing the lifecycle of the child contexts and - registering jobs in the JobRegistry. + registering jobs in the `JobRegistry`. -The ApplicationContextFactory is +The `ApplicationContextFactory` is responsible for creating the child context and the most common usage would be as above using a - ClassPathXmlApplicationContextFactory. One of + `ClassPathXmlApplicationContextFactory`. One of the features of this factory is that by default it copies some of the configuration down from the parent context to the child. So for instance you don't have to re-define the - PropertyPlaceholderConfigurer or AOP + `PropertyPlaceholderConfigurer` or AOP configuration in the child, if it should be the same as the parent. -The AutomaticJobRegistrar can be used in - conjunction with a JobRegistryBeanPostProcessor - if desired (as long as the DefaultJobLoader is +The `AutomaticJobRegistrar` can be used in + conjunction with a `JobRegistryBeanPostProcessor` + if desired (as long as the `DefaultJobLoader` is used as well). For instance this might be desirable if there are jobs defined in the main parent context as well as in the child locations. @@ -1490,14 +1490,14 @@ The AutomaticJobRegistrar can be used in ==== JobOperator -As previously discussed, the JobRepository +As previously discussed, the `JobRepository` provides CRUD operations on the meta-data, and the - JobExplorer provides read-only operations on the + `JobExplorer` provides read-only operations on the meta-data. However, those operations are most useful when used together to perform common monitoring tasks such as stopping, restarting, or summarizing a Job, as is commonly done by batch operators. Spring Batch - provides for these types of operations via the - JobOperator interface: + provides these types of operations via the + `JobOperator` interface: [source, java] @@ -1538,12 +1538,12 @@ public interface JobOperator { ---- The above operations represent methods from many different - interfaces, such as JobLauncher, - JobRepository, - JobExplorer, and - JobRegistry. For this reason, the provided - implementation of JobOperator, - SimpleJobOperator, has many dependencies: + interfaces, such as `JobLauncher`, + `JobRepository`, + `JobExplorer`, and + `JobRegistry`. For this reason, the provided + implementation of `JobOperator`, + `SimpleJobOperator`, has many dependencies: [source, xml, role="xmlContent"] @@ -1563,7 +1563,7 @@ The above operations represent methods from many different [source, java, role="javaContent"] ---- /** - * All injected dependcies for this bean are provided by the @EnableBatchProcessing + * All injected dependencies for this bean are provided by the @EnableBatchProcessing * infrastructure out of the box. */ @Bean @@ -1595,10 +1595,10 @@ If you set the table prefix on the job repository, don't forget to set it on the ==== JobParametersIncrementer -Most of the methods on JobOperator are +Most of the methods on `JobOperator` are self-explanatory, and more detailed explanations can be found on the link:$$http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/launch/JobOperator.html$$[javadoc of the interface]. However, the - startNextInstance method is worth noting. This + `startNextInstance` method is worth noting. This method will always start a new instance of a Job. This can be extremely useful if there are serious issues in a `JobExecution` and the Job @@ -1607,7 +1607,7 @@ Most of the methods on JobOperator are `JobParameters` object that will trigger a new `JobInstance` if the parameters are different from any previous set of parameters, the - startNextInstance method will use the + `startNextInstance` method will use the `JobParametersIncrementer` tied to the `Job` to force the `Job` to a new instance: @@ -1720,7 +1720,7 @@ endif::backend-pdf[] ==== Stopping a Job One of the most common use cases of - JobOperator is gracefully stopping a + `JobOperator` is gracefully stopping a Job: diff --git a/spring-batch-docs/asciidoc/jsr-352.adoc b/spring-batch-docs/asciidoc/jsr-352.adoc index 982b84429..36f39edd4 100644 --- a/spring-batch-docs/asciidoc/jsr-352.adoc +++ b/spring-batch-docs/asciidoc/jsr-352.adoc @@ -18,7 +18,7 @@ As of Spring Batch 3.0 support for JSR-352 has been fully implemented. This sect [[jsrGeneralNotes]] -=== General Notes Spring Batch and JSR-352 +=== General Notes about Spring Batch and JSR-352 Spring Batch and JSR-352 are structurally the same. They both have jobs that are made up of steps. They both have readers, processors, writers, and listeners. However, their interactions are subtly different. @@ -48,7 +48,7 @@ Spring Batch and JSR-352 are structurally the same. They both have jobs that ar All JSR-352 based jobs within Spring Batch consist of two application contexts. A parent context, that contains beans related to the infrastructure of Spring Batch such as the `JobRepository`, `PlatformTransactionManager`, etc and a child context that consists of the configuration - of the job to be run. The parent context is defined via the `baseContext.xml` provided + of the job to be run. The parent context is defined via the `jsrBaseContext.xml` provided by the framework. This context may be overridden via the `JSR-352-BASE-CONTEXT` system property. @@ -333,8 +333,8 @@ Supported operators: ---- The left hand side of the assignment is the expected value, the right hand side is the default value. In -this example, the result will resolve to a value of the system property file.separator as -#{jobParameters['unresolving.prop']} is assumed to not be resolvable. If neither expressions can be +this example, the result will resolve to a value of the system property `file.separator` as +`#{jobParameters['unresolving.prop']}` is assumed to not be resolvable. If neither expressions can be resolved, an empty String will be returned. Multiple conditions can be used, which are separated by a ';'. @@ -402,7 +402,7 @@ JSR-352 calls the process around the commit interval within a step "checkpointin implementing the `javax.batch.api.chunk.CheckpointAlgorithm` interface. This functionality is functionally the same as Spring Batch's custom completion policy. To use an implementation of `CheckpointAlgorithm`, configure your step with the custom - `checkpoint-policy` as shown below where fooCheckpointer refers to an + `checkpoint-policy` as shown below where `fooCheckpointer` refers to an implementation of `CheckpointAlgorithm`. @@ -427,7 +427,7 @@ JSR-352 calls the process around the commit interval within a step "checkpointin === Running a job The entrance to executing a JSR-352 based job is through the - `javax.batch.operations.JobOperator`. Spring Batch provides our own implementation to + `javax.batch.operations.JobOperator`. Spring Batch provides its own implementation of this interface (`org.springframework.batch.core.jsr.launch.JsrJobOperator`). This implementation is loaded via the `javax.batch.runtime.BatchRuntime`. Launching a JSR-352 based batch job is implemented as follows: @@ -488,7 +488,7 @@ JSR-352 defines two context objects that are used to interact with the meta-data `javax.batch.runtime.context.StepContext`. Both of these are available in any step level artifact (`Batchlet`, `ItemReader`, etc) with the `JobContext` being available to job level artifacts as well - (JobListener for example). + (`JobListener` for example). To obtain a reference to the `JobContext` or `StepContext` within the current scope, simply use the `@Inject` annotation: @@ -512,7 +512,7 @@ Using Spring's @Autowire is not supported for the injection of these contexts. In Spring Batch, the `JobContext` and `StepContext` wrap their corresponding execution objects (`JobExecution` and `StepExecution` respectively). Data stored via - `StepContext#persistent#setPersistentUserData(Serializable data)` is stored in the + `StepContext#setPersistentUserData(Serializable data)` is stored in the Spring Batch `StepExecution#executionContext`. [[jsrStepFlow]] @@ -614,6 +614,7 @@ Conceptually, partitioning in JSR-352 is the same as it is in Spring Batch. Met not get official `StepExecutions`. Because of that, calls to `JsrJobOperator#getStepExecutions(long jobExecutionId)` will only return the `StepExecution` for the master. + [NOTE] ==== The child `StepExecutions` still exist in the job repository and are available @@ -647,6 +648,6 @@ via the `JobExplorer` and Spring Batch Admin. Since all JSR-352 based jobs are executed asynchronously, it can be difficult to determine when a job has completed. To help with testing, Spring Batch provides the - `org.springframework.batch.core.jsr.JsrTestUtils`. This utility class provides the + `org.springframework.batch.test.JsrTestUtils`. This utility class provides the ability to start a job and restart a job and wait for it to complete. Once the job completes, the associated `JobExecution` is returned. diff --git a/spring-batch-docs/asciidoc/readersAndWriters.adoc b/spring-batch-docs/asciidoc/readersAndWriters.adoc index c39d7b884..195595da3 100644 --- a/spring-batch-docs/asciidoc/readersAndWriters.adoc +++ b/spring-batch-docs/asciidoc/readersAndWriters.adoc @@ -34,7 +34,8 @@ to return objects, keep track of the current row if restart is required, store b statistics, and provide some transaction enhancements that are explained later. There are many more possibilities, but we focus on the basic ones for this chapter. A -complete list of all available `ItemReader` implementations can be found in Appendix A. +complete list of all available `ItemReader` implementations can be found in +<>. `ItemReader` is a basic interface for generic input operations, as shown in the following interface definition: @@ -43,12 +44,12 @@ input operations, as shown in the following interface definition: ---- public interface ItemReader { - T read() throws Exception, UnexpectedInputException, ParseException; + T read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException; } ---- -The read method defines the most essential contract of the `ItemReader`. Calling it +The `read` method defines the most essential contract of the `ItemReader`. Calling it returns one item or `null` if no more items are left. An item might represent a line in a file, a row in a database, or an element in an XML file. It is generally expected that these are mapped to a usable domain object (such as `Trade`, `Foo`, or others), but there @@ -88,7 +89,7 @@ generally expected that items are 'batched' together into a chunk and then outpu interface accepts a list of items, rather than an item by itself. After writing out the list, any flushing that may be necessary can be performed before returning from the write method. For example, if writing to a Hibernate DAO, multiple calls to write can be made, -one for each item. The writer can then call `close` on the hibernate session before +one for each item. The writer can then call `flush` on the hibernate session before returning. [[itemProcessor]] @@ -112,7 +113,7 @@ public class CompositeItemWriter implements ItemWriter { public void write(List items) throws Exception { //Add business logic here - itemWriter.write(item); + itemWriter.write(items); } public void setDelegate(ItemWriter itemWriter){ @@ -142,7 +143,7 @@ An `ItemProcessor` is simple. Given one object, transform it and return another. provided object may or may not be of the same type. The point is that business logic may be applied within the process, and it is completely up to the developer to create that logic. An `ItemProcessor` can be wired directly into a step. For example, assume an -`ItemReader` provides a class of type `Foo` and that it needs to be converted to type Bar +`ItemReader` provides a class of type `Foo` and that it needs to be converted to type `Bar` before being written out. The following example shows an `ItemProcessor` that performs the conversion: @@ -227,7 +228,7 @@ public class Bar { public Bar(Foo foo) {} } -public class Foobar{ +public class Foobar { public Foobar(Bar bar) {} } @@ -238,14 +239,14 @@ public class FooProcessor implements ItemProcessor{ } } -public class BarProcessor implements ItemProcessor{ - public FooBar process(Bar bar) throws Exception { +public class BarProcessor implements ItemProcessor{ + public Foobar process(Bar bar) throws Exception { return new Foobar(bar); } } -public class FoobarWriter implements ItemWriter{ - public void write(List items) throws Exception { +public class FoobarWriter implements ItemWriter{ + public void write(List items) throws Exception { //write items } } @@ -274,7 +275,7 @@ Just as with the previous example, the composite processor can be configured int - @@ -326,7 +327,7 @@ public CompositeItemProcessor compositeProcessor() { } ---- -[[filiteringRecords]] +[[filteringRecords]] ==== Filtering Records One typical use for an item processor is to filter out records before they are passed to @@ -488,9 +489,9 @@ return a `String` or an array of `String` objects. This really only gets you hal there. A `FieldSet` is Spring Batch's abstraction for enabling the binding of fields from a file resource. It allows developers to work with file input in much the same way as they would work with database input. A `FieldSet` is conceptually similar to a JDBC -`ResultSet`. `FieldSets` only require one argument: a `String` array of tokens. +`ResultSet`. ``FieldSet``s only require one argument: a `String` array of tokens. Optionally, you can also configure the names of the fields so that the fields may be -accessed either by index or name as patterned after ResultSet, as shown in the following +accessed either by index or name as patterned after `ResultSet`, as shown in the following example: [source, java] @@ -533,7 +534,7 @@ from FTP locations to batch processing locations and vice versa. File moving uti are beyond the scope of the Spring Batch architecture, but it is not unusual for batch job streams to include file moving utilities as steps in the job stream. The batch architecture only needs to know how to locate the files to be processed. Spring Batch -begins the proces of feeding the data into the pipe from this starting point. However, +begins the process of feeding the data into the pipe from this starting point. However, link:$$http://projects.spring.io/spring-integration/$$[Spring Integration] provides many of these types of services. @@ -604,13 +605,13 @@ The contract of a `LineTokenizer` is such that, given a line of input (in theory returned. This `FieldSet` can then be passed to a `FieldSetMapper`. Spring Batch contains the following `LineTokenizer` implementations: -* `DelmitedLineTokenizer`: Used for files where fields in a record are separated by a +* `DelimitedLineTokenizer`: Used for files where fields in a record are separated by a delimiter. The most common delimiter is a comma, but pipes or semicolons are often used as well. * `FixedLengthTokenizer`: Used for files where fields in a record are each a "fixed width". The width of each field must be defined for each record type. * `PatternMatchingCompositeLineTokenizer`: Determines which among a list of -`LineTokenizers` should be used on a particular line by checking against a pattern. +``LineTokenizer``s should be used on a particular line by checking against a pattern. [[fieldSetMapper]] ===== FieldSetMapper @@ -625,7 +626,7 @@ into an object of the desired type, as shown in the following interface definiti ---- public interface FieldSetMapper { - T mapFieldSet(FieldSet fieldSet); + T mapFieldSet(FieldSet fieldSet) throws BindException; } ---- @@ -654,7 +655,7 @@ shown in the following class definition, represents the behavior most users need [source, java] ---- -public class DefaultLineMapper; implements LineMapper<>, InitializingBean { +public class DefaultLineMapper implements LineMapper<>, InitializingBean { private LineTokenizer tokenizer; @@ -668,7 +669,7 @@ public class DefaultLineMapper; implements LineMapper<>, InitializingBean { this.tokenizer = tokenizer; } - public void setFieldSetMapper(FieldSetMapper; fieldSetMapper) { + public void setFieldSetMapper(FieldSetMapper fieldSetMapper) { this.fieldSetMapper = fieldSetMapper; } } @@ -781,7 +782,7 @@ A `FieldSetMapper` can use this information as follows: [source, java] ---- - { +public class PlayerMapper implements FieldSetMapper { public Player mapFieldSet(FieldSet fs) { if(fs == null){ @@ -940,7 +941,7 @@ though a "LINEA" has more information than a "LINEB". The `ItemReader` reads each line individually, but we must specify different `LineTokenizer` and `FieldSetMapper` objects so that the `ItemWriter` receives the correct items. The `PatternMatchingCompositeLineMapper` makes this easy by allowing maps -of patterns to `LineTokenizers` and patterns to `FieldSetMappers` to be configured, as +of patterns to ``LineTokenizer``s and patterns to ``FieldSetMapper``s to be configured, as shown in the following example: .XML Configuration @@ -989,10 +990,10 @@ public PatternMatchingCompositeLineMapper orderFileLineMapper() { } ---- -In this example, "LINEA" and "LINEB" have separate `LineTokenizers`, but they both use +In this example, "LINEA" and "LINEB" have separate ``LineTokenizer``s, but they both use the same `FieldSetMapper`. -The `PatternMatchingCompositeLineMapper` makes use of the `PatternMatcher's` match method +The `PatternMatchingCompositeLineMapper` makes use of the ``PatternMatcher``'s match method in order to select the correct delegate for each line. The `PatternMatcher` allows for two wildcard characters with special meaning: the question mark ("?") matches exactly one character, while the asterisk ("\*") matches zero or more characters. Note that, in the @@ -1022,7 +1023,8 @@ alone. It is also common for a flat file to contain records that each span multiple lines. To handle this situation, a more complex strategy is required. A demonstration of this -common pattern can be found in <>. +common pattern can be found in the + link:$$https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples#multiline$$[multiLineRecords] sample. [[exceptionHandlingInFlatFiles]] ===== Exception Handling in Flat Files @@ -1104,7 +1106,7 @@ assertEquals("", tokens.readString(1)); ---- The preceding example is almost identical to the one before it, except that -tokenizer.setStrict(false) was called. This setting tells the tokenizer to not enforce +`tokenizer.setStrict(false)` was called. This setting tells the tokenizer to not enforce line lengths when tokenizing the line. A `FieldSet` is now correctly created and returned. However, it contains only empty tokens for the remaining values. @@ -1196,10 +1198,10 @@ A simple configuration might look like the following: @Bean public FlatFileItemWriter itemWriter() { return new FlatFileItemWriterBuilder() - .name("itemWriter") - .resource(new FileSystemResource("file:target/test-outputs/output.txt")) - .lineAggregator(new PassThroughLineAggregator<>()) - .build(); + .name("itemWriter") + .resource(new FileSystemResource("target/test-outputs/output.txt")) + .lineAggregator(new PassThroughLineAggregator<>()) + .build(); } ---- @@ -1326,17 +1328,17 @@ example: [source, java, role="javaContent"] ---- @Bean -public FlatFileItemWriter itemWriter(Resource outputResource) throws Exception { - BeanWrapperFieldExtractor fieldExtractor = new BeanWrapperFieldExtractor<>(); +public FlatFileItemWriter itemWriter(Resource outputResource) throws Exception { + BeanWrapperFieldExtractor fieldExtractor = new BeanWrapperFieldExtractor<>(); fieldExtractor.setNames(new String[] {"name", "credit"}); fieldExtractor.afterPropertiesSet(); - DelimitedLineAggregator lineAggregator = new DelimitedLineAggregator<>(); + DelimitedLineAggregator lineAggregator = new DelimitedLineAggregator<>(); lineAggregator.setDelimiter(","); lineAggregator.setFieldExtractor(fieldExtractor); - return new FlatFileItemWriterBuilder() - .name("foo") + return new FlatFileItemWriterBuilder() + .name("customerCreditWriter") .resource(outputResource) .lineAggregator(lineAggregator) .build(); @@ -1377,17 +1379,17 @@ same `CustomerCredit` domain object described above, it can be configured as fol [source, java, role="javaContent"] ---- @Bean -public FlatFileItemWriter itemWriter(Resource outputResource) throws Exception { - BeanWrapperFieldExtractor fieldExtractor = new BeanWrapperFieldExtractor<>(); +public FlatFileItemWriter itemWriter(Resource outputResource) throws Exception { + BeanWrapperFieldExtractor fieldExtractor = new BeanWrapperFieldExtractor<>(); fieldExtractor.setNames(new String[] {"name", "credit"}); fieldExtractor.afterPropertiesSet(); - FormatterLineAggregator lineAggregator = new FormatterLineAggregator<>(); + FormatterLineAggregator lineAggregator = new FormatterLineAggregator<>(); lineAggregator.setFormat("%-9s%-2.0f"); lineAggregator.setFieldExtractor(fieldExtractor); - return new FlatFileItemWriterBuilder() - .name("foo") + return new FlatFileItemWriterBuilder() + .name("customerCreditWriter") .resource(outputResource) .lineAggregator(lineAggregator) .build(); @@ -1405,7 +1407,7 @@ property is new and is shown in the following element: [source, java, role="javaContent"] ---- ... -FormatterLineAggregator lineAggregator = new FormatterLineAggregator<>(); +FormatterLineAggregator lineAggregator = new FormatterLineAggregator<>(); lineAggregator.setFormat("%-9s%-2.0f"); ... ---- @@ -1449,7 +1451,7 @@ the parsing process by allowing the user to provide only callbacks). We need to consider how XML input and output works in Spring Batch. First, there are a few concepts that vary from file reading and writing but are common across Spring Batch -XML processing. With XML processing, instead of lines of records (`FieldSets`) that need +XML processing. With XML processing, instead of lines of records (``FieldSet``s) that need to be tokenized, it is assumed an XML resource is a collection of 'fragments' corresponding to individual records, as shown in the following image: @@ -1460,7 +1462,7 @@ The 'trade' tag is defined as the 'root element' in the scenario above. Everythi between '<trade>' and '</trade>' is considered one 'fragment'. Spring Batch uses Object/XML Mapping (OXM) to bind fragments to objects. However, Spring Batch is not tied to any particular XML binding technology. Typical use is to delegate to -link:$$http://docs.spring.io/spring-ws/site/reference/html/oxm.html$$[Spring OXM], which +link:$$https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#oxm$$[Spring OXM], which provides uniform abstraction for the most popular OXM technologies. The dependency on Spring OXM is optional and you can choose to implement Spring Batch specific interfaces if desired. The relationship to the technologies that OXM supports is shown in the @@ -1514,7 +1516,7 @@ read. fragment to an object. The following example shows how to define a `StaxEventItemReader` that works with a root -element named `trade`, a resource of `data/iosample/input/input.xml`, and an unmarshaller +element named `trade`, a resource of `org/springframework/batch/item/xml/domain/trades.xml`, and an unmarshaller called `tradeMarshaller`. .XML Configuration @@ -1522,7 +1524,7 @@ called `tradeMarshaller`. ---- - + ---- @@ -1532,9 +1534,9 @@ called `tradeMarshaller`. ---- @Bean public StaxEventItemReader itemReader() { - return new StaxEventItemReaderBuilder() + return new StaxEventItemReaderBuilder() .name("itemReader") - .resource(new FileSystemResource("data/iosample/input/input.xml")) + .resource(new FileSystemResource("org/springframework/batch/item/xml/domain/trades.xml")) .addFragmentRootElements("trade") .unmarshaller(tradeMarshaller()) .build(); @@ -1557,9 +1559,11 @@ utility to describe the required alias, as follows: + value="org.springframework.batch.sample.domain.trade.Trade" /> - + + + @@ -1573,7 +1577,9 @@ public XStreamMarshaller tradeMarshaller() { Map aliases = new HashMap<>(); aliases.put("trade", Trade.class); aliases.put("price", BigDecimal.class); - aliases.put("name", String.class); + aliases.put("isin", String.class); + aliases.put("customer", String.class); + aliases.put("quantity", Long.class); XStreamMarshaller marshaller = new XStreamMarshaller(); @@ -1594,13 +1600,15 @@ injection provided by the Spring configuration: [source, java] ---- -StaxEventItemReader xmlStaxEventItemReader = new StaxEventItemReader() -Resource resource = new ByteArrayResource(xmlResource.getBytes()) +StaxEventItemReader xmlStaxEventItemReader = new StaxEventItemReader<>(); +Resource resource = new ByteArrayResource(xmlResource.getBytes()); Map aliases = new HashMap(); -aliases.put("trade","org.springframework.batch.sample.domain.Trade"); +aliases.put("trade","org.springframework.batch.sample.domain.trade.Trade"); aliases.put("price","java.math.BigDecimal"); aliases.put("customer","java.lang.String"); +aliases.put("isin","java.lang.String"); +aliases.put("quantity","java.lang.Long"); XStreamMarshaller unmarshaller = new XStreamMarshaller(); unmarshaller.setAliases(aliases); xmlStaxEventItemReader.setUnmarshaller(unmarshaller); @@ -1608,17 +1616,17 @@ xmlStaxEventItemReader.setResource(resource); xmlStaxEventItemReader.setFragmentRootElementName("trade"); xmlStaxEventItemReader.open(new ExecutionContext()); -boolean hasNext = true +boolean hasNext = true; -CustomerCredit credit = null; +Trade trade = null; while (hasNext) { - credit = xmlStaxEventItemReader.read(); - if (credit == null) { + trade = xmlStaxEventItemReader.read(); + if (trade == null) { hasNext = false; } else { - System.out.println(credit); + System.out.println(trade); } } ---- @@ -1631,17 +1639,15 @@ marshaller, and a `rootTagName`. A Java object is passed to a marshaller (typica standard Spring OXM Marshaller) which writes to a `Resource` by using a custom event writer that filters the `StartDocument` and `EndDocument` events produced for each fragment by the OXM tools. The following example uses the -`MarshallingEventWriterSerializer`: -// TODO How does `MarshallingEventWriterSerializer` get involved? Because there's a -// property whose name is `marshaller`? +`StaxEventItemWriter`: .XML Configuration [source, xml, role="xmlContent"] ---- - - + + ---- @@ -1651,11 +1657,11 @@ fragment by the OXM tools. The following example uses the ---- @Bean public StaxEventItemWriter itemWriter(Resource outputResource) { - return new StaxEventItemWriterBuilder() - .name("fooWriter") - .marshaller(customerCreditMarshaller()) + return new StaxEventItemWriterBuilder() + .name("tradesWriter") + .marshaller(tradeMarshaller()) .resource(outputResource) - .rootTagName("customers") + .rootTagName("trade") .overwriteOutput(true) .build(); @@ -1663,7 +1669,7 @@ public StaxEventItemWriter itemWriter(Resource outputResource) { ---- The preceding configuration sets up the three required properties and sets the optional -`overwriteOutput=true` attrbute, mentioned earlier in this chapter for specifying whether +`overwriteOutput=true` attribute, mentioned earlier in this chapter for specifying whether an existing file can be overwritten. It should be noted the marshaller used for the writer in the following example is the exact same as the one used in the reading example from earlier in the chapter: @@ -1676,9 +1682,11 @@ from earlier in the chapter: - - + value="org.springframework.batch.sample.domain.trade.Trade" /> + + + + @@ -1692,9 +1700,11 @@ public XStreamMarshaller customerCreditMarshaller() { XStreamMarshaller marshaller = new XStreamMarshaller(); Map aliases = new HashMap<>(); - aliases.put("customer", CustomerCredit.class); - aliases.put("credit", BigDecimal.class); - aliases.put("name", String.class); + aliases.put("trade", Trade.class); + aliases.put("price", BigDecimal.class); + aliases.put("isin", String.class); + aliases.put("customer", String.class); + aliases.put("quantity", Long.class); marshaller.setAliases(aliases); @@ -1710,18 +1720,20 @@ discussed, demonstrating the programmatic setup of the required properties: FileSystemResource resource = new FileSystemResource("data/outputFile.xml") Map aliases = new HashMap(); -aliases.put("customer","org.springframework.batch.sample.domain.CustomerCredit"); -aliases.put("credit","java.math.BigDecimal"); -aliases.put("name","java.lang.String"); +aliases.put("trade","org.springframework.batch.sample.domain.trade.Trade"); +aliases.put("price","java.math.BigDecimal"); +aliases.put("customer","java.lang.String"); +aliases.put("isin","java.lang.String"); +aliases.put("quantity","java.lang.Long"); Marshaller marshaller = new XStreamMarshaller(); marshaller.setAliases(aliases); StaxEventItemWriter staxItemWriter = - new StaxEventItemWriterBuilder() - .name("creditWriter") + new StaxEventItemWriterBuilder() + .name("tradesWriter") .marshaller(marshaller) .resource(resource) - .rootTagName("trades") + .rootTagName("trade") .overwriteOutput(true) .build(); @@ -1729,9 +1741,11 @@ staxItemWriter.afterPropertiesSet(); ExecutionContext executionContext = new ExecutionContext(); staxItemWriter.open(executionContext); -CustomerCredit Credit = new CustomerCredit(); +Trade trade = new Trade(); trade.setPrice(11.39); -credit.setName("Customer1"); +trade.setIsin("XYZ0001"); +trade.setQuantity(5L); +trade.setCustomer("Customer1"); staxItemWriter.write(trade); ---- @@ -1746,8 +1760,8 @@ input for both XML and flat file processing. Consider the following files in a d file-1.txt file-2.txt ignored.txt ---- -file-1.txt and file-2.txt are formatted the same and, for business reasons, should be -processed together. The `MuliResourceItemReader` can be used to read in both files by +`file-1.txt` and `file-2.txt` are formatted the same and, for business reasons, should be +processed together. The `MultiResourceItemReader` can be used to read in both files by using wildcards, as shown in the following example: .XML Configuration @@ -1764,9 +1778,9 @@ using wildcards, as shown in the following example: ---- @Bean public MultiResourceItemReader multiResourceReader() { - return new MultiResourceItemReaderBuilder() + return new MultiResourceItemReaderBuilder() .delegate(flatFileItemReader()) - .resources(resources()) + .resources(resources()) .build(); } ---- @@ -1777,6 +1791,9 @@ as with any `ItemReader`, adding extra input (in this case a file) could cause p issues when restarting. It is recommended that batch jobs work with their own individual directories until completed successfully. +NOTE: Input resources are ordered using `MultiResourceItemReader#setComparator(Comparator)` + to make sure resource ordering is preserved between job runs in restart scenario. + [[database]] === Database @@ -1839,13 +1856,13 @@ implementation of the `RowMapper` interface to map a `CustomerCredit` object: [source, java] ---- -public class CustomerCreditRowMapper implements RowMapper { +public class CustomerCreditRowMapper implements RowMapper { public static final String ID_COLUMN = "id"; public static final String NAME_COLUMN = "name"; public static final String CREDIT_COLUMN = "credit"; - public Object mapRow(ResultSet rs, int rowNum) throws SQLException { + public CustomerCredit mapRow(ResultSet rs, int rowNum) throws SQLException { CustomerCredit customerCredit = new CustomerCredit(); customerCredit.setId(rs.getInt(ID_COLUMN)); @@ -1890,7 +1907,7 @@ while(customerCredit != null){ customerCredit = itemReader.read(); counter++; } -itemReader.close(executionContext); +itemReader.close(); ---- After running the preceding code snippet, the counter equals 1,000. If the code above had @@ -1934,7 +1951,7 @@ public JdbcCursorItemReader itemReader() { ====== Additional Properties Because there are so many varying options for opening a cursor in Java, there are many -properties on the `JdbcCustorItemReader` that can be set, as described in the following +properties on the `JdbcCursorItemReader` that can be set, as described in the following table: .JdbcCursorItemReader Properties @@ -1956,13 +1973,13 @@ could cause issues with the reader's internal count. Setting this value to `true an exception to be thrown if the cursor position is not the same after the `RowMapper` call as it was before. |saveState|Indicates whether or not the reader's state should be saved in the -`ExecutionContext` provided by `ItemStream#update(ExecutionContext)` The default is +`ExecutionContext` provided by `ItemStream#update(ExecutionContext)`. The default is `true`. -|driverSupportsAbsolute|Defaults to `false`. Indicates whether the JDBC driver supports +|driverSupportsAbsolute|Indicates whether the JDBC driver supports setting the absolute row on a `ResultSet`. It is recommended that this is set to `true` -for JDBC drivers that support `ResultSet`.absolute(), as it may improve performance, -especially if a step fails while working with a large data set. -|setUseSharedExtendedConnection|Defaults to `false`. Indicates whether the connection +for JDBC drivers that support `ResultSet.absolute()`, as it may improve performance, +especially if a step fails while working with a large data set. Defaults to `false`. +|setUseSharedExtendedConnection| Indicates whether the connection used for the cursor should be used by all other processing, thus sharing the same transaction. If this is set to `false`, then the cursor is opened with its own connection and does not participate in any transactions started for the rest of the step processing. @@ -1972,7 +1989,7 @@ released after each commit. When you set this option to `true`, the statement us open the cursor is created with both 'READ_ONLY' and 'HOLD_CURSORS_OVER_COMMIT' options. This allows holding the cursor open over transaction start and commits performed in the step processing. To use this feature, you need a database that supports this and a JDBC -driver supporting JDBC 3.0 or later. +driver supporting JDBC 3.0 or later. Defaults to `false`. |=============== [[HibernateCursorItemReader]] @@ -2009,7 +2026,7 @@ while(customerCredit != null){ customerCredit = itemReader.read(); counter++; } -itemReader.close(executionContext); +itemReader.close(); ---- This configured `ItemReader` returns `CustomerCredit` objects in the exact same manner @@ -2035,7 +2052,7 @@ straightforward, as shown in the following example: ---- @Bean public HibernateCursorItemReader itemReader(SessionFactory sessionFactory) { - return new HibernateCursorItemReaderBuilder() + return new HibernateCursorItemReaderBuilder() .name("creditReader") .sessionFactory(sessionFactory) .queryString("from CustomerCredit") @@ -2290,12 +2307,12 @@ public JdbcPagingItemReader itemReader(DataSource dataSource) { Map parameterValues = new HashMap<>(); parameterValues.put("status", "NEW"); - return new JdbcPagingItemReaderBuilder() + return new JdbcPagingItemReaderBuilder() .name("creditReader") .dataSource(dataSource) .queryProvider(queryProvider()) .parameterValues(parameterValues) - .rowMapper(customerMapper()) + .rowMapper(customerCreditMapper()) .pageSize(1000) .build(); } @@ -2354,7 +2371,7 @@ example as the JDBC reader shown previously: @Bean public JpaPagingItemReader itemReader() { return new JpaPagingItemReaderBuilder() - .name("credit") + .name("creditReader") .entityManagerFactory(entityManagerFactory()) .queryString("select c from CustomerCredit c") .pageSize(1000) @@ -2363,16 +2380,16 @@ public JpaPagingItemReader itemReader() { ---- This configured `ItemReader` returns `CustomerCredit` objects in the exact same manner as -described for the `JdbcPagingItemReader` above, assuming the `Customer` object has the +described for the `JdbcPagingItemReader` above, assuming the `CustomerCredit` object has the correct JPA annotations or ORM mapping file. The 'pageSize' property determines the number of entities read from the database for each query execution. [[databaseItemWriters]] ==== Database ItemWriters -While both flat files and XML have specific `ItemWriters`, there is no exact equivalent +While both flat files and XML have specific ``ItemWriter``s, there is no exact equivalent in the database world. This is because transactions provide all the needed functionality. -`ItemWriters` are necessary for files because they must act as if they're transactional, +``ItemWriter``s are necessary for files because they must act as if they're transactional, keeping track of written items and flushing or clearing at the appropriate times. Databases have no need for this functionality, since the write is already contained in a transaction. Users can create their own DAOs that implement the `ItemWriter` interface or @@ -2420,7 +2437,7 @@ their batch jobs. The Spring container itself makes this fairly easy by allowing necessary class to be injected. However, there may be cases where the existing service needs to act as an `ItemReader` or `ItemWriter`, either to satisfy the dependency of another Spring Batch class or because it truly is the main `ItemReader` for a step. It is -fairly trivial to write an adaptor class for each service that needs wrapping, but +fairly trivial to write an adapter class for each service that needs wrapping, but because it is such a common concern, Spring Batch provides implementations: `ItemReaderAdapter` and `ItemWriterAdapter`. Both classes implement the standard Spring method by invoking the delegate pattern and are fairly simple to set up. The following @@ -2457,7 +2474,7 @@ public FooService fooService() { ---- One important point to note is that the contract of the `targetMethod` must be the same -as the contract for `read`: When exhausted, it return nulls. Otherwise, it returns an +as the contract for `read`: When exhausted, it returns `null`. Otherwise, it returns an `Object`. Anything else prevents the framework from knowing when processing should end, either causing an infinite loop or incorrect failure, depending upon the implementation of the `ItemWriter`. The following example uses the `ItemWriterAdapter`: @@ -2510,9 +2527,9 @@ frameworks, as shown in the following interface definition: [source, java] ---- -public interface Validator { +public interface Validator { - void validate(Object value) throws ValidationException; + void validate(T value) throws ValidationException; } ---- @@ -2646,7 +2663,7 @@ public class CustomItemReader implements ItemReader{ } public T read() throws Exception, UnexpectedInputException, - NoWorkFoundException, ParseException { + NonTransientResourceException, ParseException { if (!items.isEmpty()) { return items.remove(0); @@ -2701,7 +2718,7 @@ public class CustomItemReader implements ItemReader, ItemStream { } public T read() throws Exception, UnexpectedInputException, - ParseException { + ParseException, NonTransientResourceException { if (currentIndex < items.size()) { return items.get(currentIndex++); @@ -2758,7 +2775,7 @@ It is also worth noting that the key used within the `ExecutionContext` should n trivial. That is because the same `ExecutionContext` is used for all `ItemStreams` within a `Step`. In most cases, simply prepending the key with the class name should be enough to guarantee uniqueness. However, in the rare cases where two of the same type of -`ItemStream` are used in the same step (which can happen if two files are need for +`ItemStream` are used in the same step (which can happen if two files are needed for output), a more unique name is needed. For this reason, many of the Spring Batch `ItemReader` and `ItemWriter` implementations have a `setName()` property that lets this key name be overridden. @@ -2846,7 +2863,7 @@ would get that item in the next call to read. [[multiResourceItemWriter]] ===== `MultiResourceItemWriter` -The MultiResourceItemWriter wraps a `ResourceAwareItemWriterItemStream` and creates a new +The `MultiResourceItemWriter` wraps a `ResourceAwareItemWriterItemStream` and creates a new output resource when the count of items written in the current resource exceeds the `itemCountLimitPerResource`. Spring Batch provides a `MultiResourceItemWriterBuilder` to construct an instance of the `MultiResourceItemWriter`. diff --git a/spring-batch-docs/asciidoc/retry.adoc b/spring-batch-docs/asciidoc/retry.adoc index a74e50e96..13acb1f82 100644 --- a/spring-batch-docs/asciidoc/retry.adoc +++ b/spring-batch-docs/asciidoc/retry.adoc @@ -26,7 +26,7 @@ To make processing more robust and less prone to failure, it sometimes [NOTE] ==== The retry functionality was pulled out of Spring Batch as of 2.2.0. - It is now part of a new library, Spring Retry. +It is now part of a new library, https://github.com/spring-projects/spring-retry[Spring Retry]. ==== @@ -39,16 +39,16 @@ To automate retry ---- public interface RetryOperations { - T execute(RetryCallback retryCallback) throws Exception; + T execute(RetryCallback retryCallback) throws E; - T execute(RetryCallback retryCallback, RecoveryCallback recoveryCallback) - throws Exception; + T execute(RetryCallback retryCallback, RecoveryCallback recoveryCallback) + throws E; - T execute(RetryCallback retryCallback, RetryState retryState) - throws Exception, ExhaustedRetryException; + T execute(RetryCallback retryCallback, RetryState retryState) + throws E, ExhaustedRetryException; - T execute(RetryCallback retryCallback, RecoveryCallback recoveryCallback, - RetryState retryState) throws Exception; + T execute(RetryCallback retryCallback, RecoveryCallback recoveryCallback, + RetryState retryState) throws E; } ---- @@ -59,9 +59,9 @@ The basic callback is a simple interface that lets you [source, java] ---- -public interface RetryCallback { +public interface RetryCallback { - T doWithRetry(RetryContext context) throws Throwable; + T doWithRetry(RetryContext context) throws E; } ---- @@ -368,11 +368,11 @@ The following code shows the interface definition for `RetryListener`: ---- public interface RetryListener { - void open(RetryContext context, RetryCallback callback); + boolean open(RetryContext context, RetryCallback callback); - void onError(RetryContext context, RetryCallback callback, Throwable e); + void onError(RetryContext context, RetryCallback callback, Throwable throwable); - void close(RetryContext context, RetryCallback callback, Throwable e); + void close(RetryContext context, RetryCallback callback, Throwable throwable); } ---- @@ -401,7 +401,7 @@ Sometimes, there is some business processing that you know you want The `RetryOperationsInterceptor` executes the intercepted method and retries on failure according to the `RetryPolicy` in the provided - `RepeatTemplate`. + `RetryTemplate`. [role="xmlContent"] The following example shows a declarative retry that uses the Spring AOP diff --git a/spring-batch-docs/asciidoc/scalability.adoc b/spring-batch-docs/asciidoc/scalability.adoc index cf9700d4e..cb316bd82 100644 --- a/spring-batch-docs/asciidoc/scalability.adoc +++ b/spring-batch-docs/asciidoc/scalability.adoc @@ -74,7 +74,7 @@ public Step sampleStep(TaskExecutor taskExecutor) { } ---- -In this example, the taskExecutor is a reference to another bean definition that +In this example, the `taskExecutor` is a reference to another bean definition that implements the `TaskExecutor` interface. https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/core/task/TaskExecutor.html[`TaskExecutor`] is a standard Spring interface, so consult the Spring User Guide for details of available @@ -130,7 +130,7 @@ writers from Spring Batch are not designed for multi-threaded use. It is, howeve possible to work with stateless or thread safe readers and writers, and there is a sample (called `parallelJob`) in the https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples[Spring -Batch Samples] that show the use of a process indicator (see +Batch Samples] that shows the use of a process indicator (see <>) to keep track of items that have been processed in a database input table. @@ -138,10 +138,10 @@ Spring Batch provides some implementations of `ItemWriter` and `ItemReader`. Us they say in the Javadoc if they are thread safe or not or what you have to do to avoid problems in a concurrent environment. If there is no information in the Javadoc, you can check the implementation to see if there is any state. If a reader is not thread safe, -it may still be efficient to use it in your own synchronizing delegator. You can -synchronize the call to `read()` and as long as the processing and writing is the most -expensive part of the chunk, your step may still complete much faster than it would in a -single threaded configuration. +you can decorate it with the built-in `SynchronizedItemStreamReader` or use it in your own +synchronizing delegator. You can synchronize the call to `read()` and as long as the +processing and writing is the most expensive part of the chunk, your step may still +complete much faster than it would in a single threaded configuration. [[scalabilityParallelSteps]] @@ -254,7 +254,7 @@ and work is shared through the middleware, so that, if the listeners are all eag consumers, then load balancing is automatic. The middleware has to be durable, with guaranteed delivery and a single consumer for each -message. JMS is the obvious candidate, but other options (such as Java Spaces exist in +message. JMS is the obvious candidate, but other options (such as Java Spaces) exist in the grid computing and shared memory product space. See the section on @@ -279,7 +279,7 @@ instances of a `Step`, which could in fact take the place of the master, resulti same outcome for the `Job`. The slaves are typically going to be remote services but could also be local threads of execution. The messages sent by the master to the slaves in this pattern do not need to be durable or have guaranteed delivery. Spring Batch -metadata in the JobRepository ensures that each slave is executed once and only once for +metadata in the `JobRepository` ensures that each slave is executed once and only once for each `Job` execution. The SPI in Spring Batch consists of a special implementation of `Step` (called the diff --git a/spring-batch-docs/asciidoc/schema-appendix.adoc b/spring-batch-docs/asciidoc/schema-appendix.adoc index 2d05208a0..7620e42e6 100644 --- a/spring-batch-docs/asciidoc/schema-appendix.adoc +++ b/spring-batch-docs/asciidoc/schema-appendix.adoc @@ -282,7 +282,7 @@ possible. * `LAST_UPDATED`: Timestamp representing the last time this execution was persisted. [[metaDataBatchJobExecutionContext]] -=== BATCH_JOB_EXECUTION_CONTEXT +=== `BATCH_JOB_EXECUTION_CONTEXT` The `BATCH_JOB_EXECUTION_CONTEXT` table holds all information relevant to the `ExecutionContext` of a `Job`. There is exactly one `Job` `ExecutionContext` per diff --git a/spring-batch-docs/asciidoc/spring-batch-integration.adoc b/spring-batch-docs/asciidoc/spring-batch-integration.adoc index 92a2734c3..7ccfa0bfa 100644 --- a/spring-batch-docs/asciidoc/spring-batch-integration.adoc +++ b/spring-batch-docs/asciidoc/spring-batch-integration.adoc @@ -17,7 +17,7 @@ endif::onlyonetoggle[] Many users of Spring Batch may encounter requirements that are outside the scope of Spring Batch but that may be efficiently and concisely implemented by using Spring Integration. Conversely, Spring -Batch users may encounter Spring Batch requirements and need a way +Integration users may encounter Spring Batch requirements and need a way to efficiently integrate both frameworks. In this context, several patterns and use-cases emerge, and Spring Batch Integration addresses those requirements. @@ -129,7 +129,7 @@ When starting batch jobs by using the core Spring Batch API, you basically have 2 options: * From the command line, with the `CommandLineJobRunner` -* Programatically, with either `JobOperator.start()` or `JobLauncher.run()` +* Programmatically, with either `JobOperator.start()` or `JobLauncher.run()` @@ -253,7 +253,7 @@ immediately. Users can then take the `id` of using the `JobExplorer`. For more information, please refer to the Spring Batch reference documentation on -link:$$http://docs.spring.io/spring-batch/reference/html/configureJob.html#queryingRepository$$[Querying the Repository]. +<>. @@ -444,7 +444,7 @@ public JobLaunchingGateway sampleJobLaunchingGateway() { As Spring Batch jobs can run for long times, providing progress information is often critical. For example, stake-holders may want -to be notified if a some or all parts of a batch job have failed. +to be notified if some or all parts of a batch job have failed. Spring Batch provides support for this information being gathered through: @@ -629,7 +629,7 @@ The following example shows how to configure the `AsyncItemWriter`: [source, java, role="javaContent"] ---- @Bean -public AsyncItemWriter processor(ItemWriter itemWriter) { +public AsyncItemWriter writer(ItemWriter itemWriter) { AsyncItemWriter asyncItemWriter = new AsyncItemWriter(); asyncItemWriter.setDelegate(itemWriter); return asyncItemWriter; @@ -976,7 +976,7 @@ and AMQP) being used to communicate with the remote workers. The section of the "Scalability" chapter that addresses -link:$$http://docs.spring.io/spring-batch/reference/html/scalability.html#partitioning$$[remote partitioning] provides an overview of the concepts and +<> provides an overview of the concepts and components needed to configure remote partitioning and shows an example of using the default `TaskExecutorPartitionHandler` to partition diff --git a/spring-batch-docs/asciidoc/spring-batch-intro.adoc b/spring-batch-docs/asciidoc/spring-batch-intro.adoc index cf0b5b89f..23619728a 100644 --- a/spring-batch-docs/asciidoc/spring-batch-intro.adoc +++ b/spring-batch-docs/asciidoc/spring-batch-intro.adoc @@ -6,8 +6,6 @@ == Spring Batch Introduction -include::toggle.adoc[] - Many applications within the enterprise domain require bulk processing to perform business operations in mission critical environments. These business operations include: @@ -143,36 +141,47 @@ which is its own library). The following key principles, guidelines, and general considerations should be considered when building a batch solution. + * Remember that a batch architecture typically affects on-line architecture and vice versa. Design with both architectures and environments in mind using common building blocks when possible. + * Simplify as much as possible and avoid building complex logical structures in single batch applications. + * Keep the processing and storage of data physically close together (in other words, keep your data where your processing occurs). + * Minimize system resource use, especially I/O. Perform as many operations as possible in internal memory. + * Review application I/O (analyze SQL statements) to ensure that unnecessary physical I/O -is avoided. In particular, the following four common flaws need to be looked for: +is avoided. In particular, the following four common flaws need to be looked for: ** Reading data for every transaction when the data could be read once and cached or kept in the working storage. ** Rereading data for a transaction where the data was read earlier in the same transaction. ** Causing unnecessary table or index scans. ** Not specifying key values in the WHERE clause of an SQL statement. + * Do not do things twice in a batch run. For instance, if you need data summarization for reporting purposes, you should (if possible) increment stored totals when data is being initially processed, so your reporting application does not have to reprocess the same data. + * Allocate enough memory at the beginning of a batch application to avoid time-consuming reallocation during the process. + * Always assume the worst with regard to data integrity. Insert adequate checks and record validation to maintain data integrity. + * Implement checksums for internal validation where possible. For example, flat files should have a trailer record telling the total of records in the file and an aggregate of the key fields. + * Plan and execute stress tests as early as possible in a production-like environment with realistic data volumes. + * In large batch systems, backups can be challenging, especially if the system is running concurrent with on-line on a 24-7 basis. Database backups are typically well taken care of in the on-line design, but file backups should be considered to be just as important. @@ -304,7 +313,7 @@ record contention and therefore either a physical or logical lock needs to be ob retrieval time. One type of pessimistic logical locking uses a dedicated lock-column in the database table. When an application retrieves the row for update, it sets a flag in the lock column. With the flag in place, other applications attempting to retrieve the -same row logically fail. When the application that set the flag updates the row, it also +same row logically fail. When the application that sets the flag updates the row, it also clears the flag, enabling the row to be retrieved by other applications. Please note that the integrity of data must be maintained also between the initial fetch and the setting of the flag, for example by using db locks (such as `SELECT FOR UPDATE`). Note also that diff --git a/spring-batch-docs/asciidoc/step.adoc b/spring-batch-docs/asciidoc/step.adoc index d135ae304..a49078f2f 100644 --- a/spring-batch-docs/asciidoc/step.adoc +++ b/spring-batch-docs/asciidoc/step.adoc @@ -110,7 +110,7 @@ The configuration above includes the only required dependencies to create a item step: * `reader`: The `ItemReader` that provides items for processing. -* `writer`: The ItemWriter that processes the items provided by the`ItemReader`. +* `writer`: The `ItemWriter` that processes the items provided by the `ItemReader`. [role="xmlContent"] * `transaction-manager`: Spring's `PlatformTransactionManager` that begins and commits @@ -155,7 +155,7 @@ The configuration above includes the only required dependencies to create a item step: * `reader`: The `ItemReader` that provides items for processing. -* `writer`: The ItemWriter that processes the items provided by the `ItemReader`. +* `writer`: The `ItemWriter` that processes the items provided by the `ItemReader`. * `transaction-manager`/`transactionManager`: Spring's `PlatformTransactionManager` that begins and commits transactions during processing. * `job-repository`/`repository`: The `JobRepository` that periodically stores the @@ -206,6 +206,7 @@ overridden by the "concreteStep1" `Step`, as shown in the following example: [role="xmlContent"] The `id` attribute is still required on the step within the job element. This is for two reasons: + * The `id` is used as the step name when persisting the `StepExecution`. If the same standalone step is referenced in more than one step in the job, an error occurs. @@ -227,7 +228,7 @@ defined without these properties, then the `abstract` attribute should be used. [role="xmlContent"] In the following example, the `Step` `abstractParentStep` would not be instantiated if it were not declared to be abstract. The `Step`, "concreteStep2", has 'itemReader', -'itemWriter', and commitInterval=10. +'itemWriter', and commit-interval=10. [source, xml, role="xmlContent"] ---- @@ -249,15 +250,15 @@ were not declared to be abstract. The `Step`, "concreteStep2", has 'itemReader', ===== Merging Lists [role="xmlContent"] -Some of the configurable elements on `Steps` are lists. The `` element, for -instance. If both the parent and child `Steps` declare a `` element, then the +Some of the configurable elements on `Steps` are lists, such as the `` element. +If both the parent and child `Steps` declare a `` element, then the child's list overrides the parent's. In order to allow a child to add additional listeners to the list defined by the parent, every list element has a `merge` attribute. If the element specifies that `merge="true"`, then the child's list is combined with the parent's instead of overriding it. [role="xmlContent"] -In the following example, the `Step`, "concreteStep3", is created with two listeners: +In the following example, the `Step` "concreteStep3" is created with two listeners: `listenerOne` and `listenerTwo`: [source, xml, role="xmlContent"] @@ -316,7 +317,7 @@ public Job sampleJob() { @Bean public Step step1() { - return this.stepBuilderFactory.get("step1" + return this.stepBuilderFactory.get("step1") .chunk(10) .reader(itemReader()) .writer(itemWriter()) @@ -447,7 +448,7 @@ public Job footballJob() { return this.jobBuilderFactory.get("footballJob") .start(playerLoad()) .next(gameLoad()) - .next(playerSumarization()) + .next(playerSummarization()) .end() .build(); } @@ -484,19 +485,19 @@ public Step playerSummarization() { The preceding example configuration is for a job that loads in information about football games and summarizes them. It contains three steps: `playerLoad`, `gameLoad`, and -`playerSummarization`. The playerLoad `Step` loads player information from a flat file, -while the gameLoad `Step` does the same for games. The final `Step`, +`playerSummarization`. The `playerLoad` step loads player information from a flat file, +while the `gameLoad` step does the same for games. The final step, `playerSummarization`, then summarizes the statistics for each player, based upon the -provided games. It is assumed that the file loaded by 'playerLoad' must be loaded only -once, but that 'gameLoad' can load any games found within a particular directory, +provided games. It is assumed that the file loaded by `playerLoad` must be loaded only +once, but that `gameLoad` can load any games found within a particular directory, deleting them after they have been successfully loaded into the database. As a result, -the playerLoad `Step` contains no additional configuration. It can be started any number -of times, and, if complete, is skipped. The 'gameLoad' `Step`, however, needs to be run +the `playerLoad` step contains no additional configuration. It can be started any number +of times, and, if complete, is skipped. The `gameLoad` step, however, needs to be run every time in case extra files have been added since it last ran. It has 'allow-start-if-complete' set to 'true' in order to always be started. (It is assumed that the database tables games are loaded into has a process indicator on it, to ensure -new games can be properly found by the summarization step). The summarization `Step`, -which is the most important in the `Job`, is configured to have a start limit of 3. This +new games can be properly found by the summarization step). The summarization step, +which is the most important in the job, is configured to have a start limit of 3. This is useful because if the step continually fails, a new exit code is returned to the operators that control job execution, and it can not start again until manual intervention has taken place. @@ -507,7 +508,7 @@ This job provides an example for this document and is not the same as the `footb found in the samples project. ==== -The remainder of this section describes what happens for each of three runs of the +The remainder of this section describes what happens for each of the three runs of the `footballJob` example. Run 1: @@ -529,13 +530,14 @@ processed). process indicator) and fails again after 30 minutes. Run 3: + . `playerLoad` does not run, since it has already completed successfully, and `allow-start-if-complete` is 'false' (the default). . `gameLoad` runs again and processes another 2 files, loading their contents into the 'GAMES' table as well (with a process indicator indicating they have yet to be processed). . `playerSummarization` is not started and the job is immediately killed, since this is -the third execution of playerSummarization, and its limit is only 2. Either the limit +the third execution of `playerSummarization`, and its limit is only 2. Either the limit must be raised or the `Job` must be executed as a new `JobInstance`. [[configuringSkip]] @@ -573,10 +575,10 @@ The following example shows an example of using a skip limit: @Bean public Step step1() { return this.stepBuilderFactory.get("step1") - .faultTolerant() .chunk(10) .reader(flatFileItemReader()) .writer(itemWriter()) + .faultTolerant() .skipLimit(10) .skip(FlatFileParseException.class) .build(); @@ -619,10 +621,10 @@ example: @Bean public Step step1() { return this.stepBuilderFactory.get("step1") - .faultTolerant() .chunk(10) .reader(flatFileItemReader()) .writer(itemWriter()) + .faultTolerant() .skipLimit(10) .skip(Exception.class) .noSkip(FileNotFoundException.class) @@ -637,7 +639,7 @@ exception classes to be all `Exceptions` __except__ `FileNotFoundException`. Any exception classes is fatal if encountered (that is, they are not skipped). For any exception encountered, the skippability is determined by the nearest superclass -in the class hierarchy. Any unclassifed exception is treated as 'fatal'. +in the class hierarchy. Any unclassified exception is treated as 'fatal'. ifdef::backend-html5[] [role="xmlContent"] @@ -660,7 +662,7 @@ not all exceptions are deterministic. If a `FlatFileParseException` is encounter reading, it is always thrown for that record. Resetting the `ItemReader` does not help. However, for other exceptions, such as a `DeadlockLoserDataAccessException`, which indicates that the current process has attempted to update a record that another process -holds a lock on. Waiting and trying again might result in success. In this case, retry +holds a lock on, waiting and trying again might result in success. In this case, retry should be configured as follows: [source, xml, role="xmlContent"] @@ -682,11 +684,11 @@ should be configured as follows: @Bean public Step step1() { return this.stepBuilderFactory.get("step1") - .faultTolerant() .chunk(2) - .retryLimit(3) .reader(itemReader()) .writer(itemWriter()) + .faultTolerant() + .retryLimit(3) .retry(DeadlockLoserDataAccessException.class) .build(); } @@ -694,7 +696,7 @@ public Step step1() { The `Step` allows a limit for the number of times an individual item can be retried and a list of exceptions that are 'retryable'. More details on how retry works can be found in -<>. +<>. [[controllingRollback]] ==== Controlling Rollback @@ -726,10 +728,10 @@ cause rollback, as shown in the following example: @Bean public Step step1() { return this.stepBuilderFactory.get("step1") - .faultTolerant() .chunk(2) .reader(itemReader()) .writer(itemWriter()) + .faultTolerant() .noRollback(ValidationException.class) .build(); } @@ -739,7 +741,7 @@ public Step step1() { ===== Transactional Readers The basic contract of the `ItemReader` is that it is forward only. The step buffers -reader input, so that, in the case of a rollback, the items do not need to be re-read +reader input, so that in the case of a rollback, the items do not need to be re-read from the reader. However, there are certain scenarios in which the reader is built on top of a transactional resource, such as a JMS queue. In this case, since the queue is tied to the transaction that is rolled back, the messages that have been pulled from the @@ -766,7 +768,7 @@ public Step step1() { .chunk(2) .reader(itemReader()) .writer(itemWriter()) - .readerIsTransactionalQueue(true) + .readerIsTransactionalQueue() .build(); } ---- @@ -800,8 +802,8 @@ core documentation]. The following example sets the `isolation`, `propagation`, @Bean public Step step1() { DefaultTransactionAttribute attribute = new DefaultTransactionAttribute(); - attribute.setPropagationBehavior(Propagation.REQUIRED); - attribute.setIsolationLevel(Isolation.DEFAULT); + attribute.setPropagationBehavior(Propagation.REQUIRED.value()); + attribute.setIsolationLevel(Isolation.DEFAULT.value()); attribute.setTimeout(30); return this.stepBuilderFactory.get("step1") @@ -817,8 +819,8 @@ public Step step1() { ==== Registering `ItemStream` with a `Step` The step has to take care of `ItemStream` callbacks at the necessary points in its -lifecycle. (For more information on the `ItemStream` interface, see -<>) This is vital if a step fails and might +lifecycle (For more information on the `ItemStream` interface, see +<>). This is vital if a step fails and might need to be restarted, because the `ItemStream` interface is where the step gets the information it needs about persistent state between executions. @@ -898,7 +900,7 @@ event of a failure. Just as with the `Job`, there are many events during the execution of a `Step` where a user may need to perform some functionality. For example, in order to write out to a flat file that requires a footer, the `ItemWriter` needs to be notified when the `Step` has -been completed, so that the footer can written. This can be accomplished with one of many +been completed, so that the footer can be written. This can be accomplished with one of many `Step` scoped listeners. Any class that implements one of the extensions of `StepListener` (but not that interface @@ -989,8 +991,9 @@ successfully, as shown in the following interface definition: ---- public interface ChunkListener extends StepListener { - void beforeChunk(); - void afterChunk(); + void beforeChunk(ChunkContext context); + void afterChunk(ChunkContext context); + void afterChunkError(ChunkContext context); } ---- @@ -1003,6 +1006,7 @@ The annotations corresponding to this interface are: * `@BeforeChunk` * `@AfterChunk` +* `@AfterChunkError` A `ChunkListener` can be applied when there is no chunk declaration. The `TaskletStep` is responsible for calling the `ChunkListener`, so it applies to a non-item-oriented tasklet @@ -1059,7 +1063,7 @@ The `beforeProcess` method is called before `process` on the `ItemProcessor` and handed the item that is to be processed. The `afterProcess` method is called after the item has been successfully processed. If there was an error while processing, the `onProcessError` method is called. The exception encountered and the item that was -attempted to be processed is provided, so that they can be logged. +attempted to be processed are provided, so that they can be logged. The annotations corresponding to this interface are: @@ -1087,7 +1091,7 @@ public interface ItemWriteListener extends StepListener { The `beforeWrite` method is called before `write` on the `ItemWriter` and is handed the list of items that is written. The `afterWrite` method is called after the item has been successfully written. If there was an error while writing, the `onWriteError` method is -called. The exception encountered and the item that was attempted to be written is +called. The exception encountered and the item that was attempted to be written are provided, so that they can be logged. The annotations corresponding to this interface are: @@ -1248,8 +1252,8 @@ following example: public MethodInvokingTaskletAdapter myTasklet() { MethodInvokingTaskletAdapter adapter = new MethodInvokingTaskletAdapter(); - adapter.setTarkgetObject(fooDao()); - adapter.setTargetMethod(updateFoo); + adapter.setTargetObject(fooDao()); + adapter.setTargetMethod("updateFoo"); return adapter; } @@ -1329,8 +1333,7 @@ reference the `Tasklet` from the `Step`: public Job taskletJob() { return this.jobBuilderFactory.get("taskletJob") .start(deleteFilesInDir()) - .end() - .build; + .build(); } @Bean @@ -1428,7 +1431,7 @@ In order to handle more complex scenarios, the Spring Batch namespace allows tra elements to be defined within the step element. One such transition is the `next` element. Like the `next` attribute, the `next` element tells the `Job` which `Step` to execute next. However, unlike the attribute, any number of `next` elements are allowed on -a given `Step`, and there is no default behavior the case of failure. This means that, if +a given `Step`, and there is no default behavior in the case of failure. This means that, if transition elements are used, then all of the behavior for the `Step` transitions must be defined explicitly. Note also that a single step cannot have both a `next` attribute and a `transition` element. @@ -1530,7 +1533,7 @@ preceding XML configuration example references the exit code of `ExitStatus`. When using Java configuration, the 'on' method shown in the preceding Java configuration example references the exit code of `ExitStatus`. -In English, it says: "go to stepB if the exit code is `FAILED`". By default, the exit +In English, it says: "go to stepB if the exit code is `FAILED` ". By default, the exit code is always the same as the `BatchStatus` for the `Step`, which is why the entry above works. However, what if the exit code needs to be different? A good example comes from the skip sample job within the samples project: @@ -1617,7 +1620,7 @@ public Job job() { } ---- -If no transitions are defined for a `Step`, then the `Job`'s statuses is defined as +If no transitions are defined for a `Step`, then the ``Job``'s statuses is defined as follows: * If the `Step` ends with `ExitStatus` FAILED, then the `BatchStatus` and `ExitStatus` of @@ -1689,7 +1692,7 @@ public Job job() { ===== Failing a Step Configuring a step to fail at a given point instructs a `Job` to stop with a -`BatchStatus` of `FAILED`. Unlike end, the failing a `Job` does not prevent the `Job` +`BatchStatus` of `FAILED`. Unlike end, the failure of a `Job` does not prevent the `Job` from being restarted. [role="xmlContent"] @@ -1745,7 +1748,7 @@ the step where execution should pick up when the "Job is restarted". When using java configuration, the `stopAndRestart` method requires a 'restart' attribute that specifies the step where execution should pick up when the "Job is restarted". -In the following scenario, if `step1` finishes with `COMPLETE`, then the job then stops. +In the following scenario, if `step1` finishes with `COMPLETE`, then the job stops. Once it is restarted, execution begins on `step2`. [source, xml, role="xmlContent"] @@ -1779,12 +1782,14 @@ in the decision, as shown in the following example: ---- public class MyDecider implements JobExecutionDecider { public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) { - if (someCondition) { - return "FAILED"; + String status; + if (someCondition()) { + status = "FAILED"; } else { - return "COMPLETED"; + status = "COMPLETED"; } + return new FlowExecutionStatus(status); } } ---- @@ -1832,15 +1837,15 @@ public Job job() { [[split-flows]] ==== Split Flows -Every scenario described so far has involved a `Job` that executes its `Steps` one at a -time in a linear fashion. In addition to this typical style, the Spring Batch also allows +Every scenario described so far has involved a `Job` that executes its ``Step``s one at a +time in a linear fashion. In addition to this typical style, Spring Batch also allows for a job to be configured with parallel flows. [role="xmlContent"] The XML namespace allows you to use the 'split' element. As the following example shows, the 'split' element contains one or more 'flow' elements, where entire separate flows can be defined. A 'split' element may also contain any of the previously discussed transition -elements, such as the 'next' attribute or the 'next', 'end', 'fail', or 'pause' elements. +elements, such as the 'next' attribute or the 'next', 'end' or 'fail' elements. [source, xml, role="xmlContent"] ---- @@ -1858,7 +1863,7 @@ elements, such as the 'next' attribute or the 'next', 'end', 'fail', or 'pause' [role="javaContent"] Java based configuration lets you configure splits through the provided builders. As the -following example shows, the 'split' element contains one or more 'flow' elements, where +following example shows, the 'split' element contains one or more 'flow' elements, where entire separate flows can be defined. A 'split' element may also contain any of the previously discussed transition elements, such as the 'next' attribute or the 'next', 'end', 'fail', or 'pause' elements. @@ -1871,7 +1876,7 @@ public Job job() { .start(step1()) .next(step2()) .build(); - Flow flow2 = new FlowBuilder("flow1") + Flow flow2 = new FlowBuilder("flow2") .start(step3()) .build(); @@ -1892,7 +1897,7 @@ Part of the flow in a job can be externalized as a separate bean definition and re-used. There are two ways to do so. The first is to simply declare the flow as a reference to one defined elsewhere, as shown in the following example: -.XML Confguration +.XML Configuration [source, xml, role="xmlContent"] ---- @@ -1906,7 +1911,7 @@ reference to one defined elsewhere, as shown in the following example: ---- -.Java Confguration +.Java Configuration [source, java, role="javaContent"] ---- @Bean @@ -1965,9 +1970,8 @@ The following Java snippet shows an example of a `JobStep`: ---- @Bean public Job jobStepJob() { - return this.jobBuilderFactor.get("jobStepJob") + return this.jobBuilderFactory.get("jobStepJob") .start(jobStepJobStep1(null)) - .end() .build(); } @@ -1991,7 +1995,7 @@ public Job job() { public DefaultJobParametersExtractor jobParametersExtractor() { DefaultJobParametersExtractor extractor = new DefaultJobParametersExtractor(); - extractor.setKeys("input.file"); + extractor.setKeys(new String[]{"input.file"}); return extractor; } @@ -2018,7 +2022,7 @@ constructs, as shown in the following example: + value="file://outputs/file.txt" /> ---- @@ -2026,10 +2030,10 @@ constructs, as shown in the following example: [source, java, role="javaContent"] ---- @Bean -public FlatFileItemReaer flatFileItemReader() { +public FlatFileItemReader flatFileItemReader() { FlatFileItemReader reader = new FlatFileItemReaderBuilder() .name("flatFileItemReader") - .resource(new FileSystemResource("file://outputs/20070122.testStream.CustomerReportStep.TEMP.txt")) + .resource(new FileSystemResource("file://outputs/file.txt")) ... } ---- @@ -2060,7 +2064,7 @@ The following Java snippet shows how to read a file name from a property: [source, java, role="javaContent"] ---- @Bean -public FlatFileItemReaer flatFileItemReader(@Value("${input.file.name}") String name) { +public FlatFileItemReader flatFileItemReader(@Value("${input.file.name}") String name) { return new FlatFileItemReaderBuilder() .name("flatFileItemReader") .resource(new FileSystemResource(name)) @@ -2069,12 +2073,13 @@ public FlatFileItemReaer flatFileItemReader(@Value("${input.file.name}") String ---- All that would be required for this solution to work would be a system argument (such as -`-Dinput.file.name="file://file.txt"`). (Note that, although a -`PropertyPlaceholderConfigurer` can be used here, it is not necessary if the system -property is always set because the `ResourceEditor` in Spring already filters and does -placeholder replacement on system properties.) +`-Dinput.file.name="file://outputs/file.txt"`). -Often, in a batch setting, it is preferable to parameterize the file name in the +NOTE: Although a `PropertyPlaceholderConfigurer` can be used here, it is not +necessary if the system property is always set because the `ResourceEditor` in Spring +already filters and does placeholder replacement on system properties. + +Often, in a batch setting, it is preferable to parametrize the file name in the `JobParameters` of the job, instead of through system properties, and access them that way. To accomplish this, Spring Batch allows for the late binding of various `Job` and `Step` attributes, as shown in the following snippet: @@ -2123,7 +2128,7 @@ the same way, as shown in the following examples: ---- .Java Configuration -[source, java, role="javalContent"] +[source, java, role="javaContent"] ---- @StepScope @Bean @@ -2217,7 +2222,7 @@ accessible from the `JobContext` using `#{..}` placeholders. Using this feature, properties can be pulled from the job or job execution context and the job parameters, as shown in the following examples: -.XML Configurtation +.XML Configuration [source, xml, role="xmlContent"] ---- @@ -2233,7 +2238,7 @@ shown in the following examples: ---- -.Java Configurtation +.Java Configuration [source, java, role="javaContent"] ---- @JobScope @@ -2271,9 +2276,9 @@ The following example uses the `batch` namespace: xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="..."> - - ... - + +... + ---- The following example includes a bean that explicitly defines the `JobScope`: diff --git a/spring-batch-docs/asciidoc/testing.adoc b/spring-batch-docs/asciidoc/testing.adoc index e22a43767..110ea973c 100644 --- a/spring-batch-docs/asciidoc/testing.adoc +++ b/spring-batch-docs/asciidoc/testing.adoc @@ -24,7 +24,7 @@ As with other application styles, it is extremely important to === Creating a Unit Test Class In order for the unit test to run a batch job, the framework must - load the job's ApplicationContext. Two annotations are used to trigger + load the job's `ApplicationContext`. Two annotations are used to trigger this behavior: @@ -153,12 +153,12 @@ public class SkipSampleFunctionalTests { For complex batch jobs, test cases in the end-to-end testing approach may become unmanageable. It these cases, it may be more useful to have test cases to test individual steps on their own. The - `AbstractJobTests` class contains a method called + `JobLauncherTestUtils` class contains a method called `launchStep`, which takes a step name and runs just that particular `Step`. This approach allows for more targeted tests letting the test set up data for only that step and to validate its results directly. The following example shows how to use the - `launchStep` method to load a `Step` by name + `launchStep` method to load a `Step` by name: [source, java] @@ -289,9 +289,9 @@ public class NoWorkFoundStepExecutionListener extends StepExecutionListenerSuppo public ExitStatus afterStep(StepExecution stepExecution) { if (stepExecution.getReadCount() == 0) { - throw new NoWorkFoundException("Step has not processed any items"); + return ExitStatus.FAILED; } - return stepExecution.getExitStatus(); + return null; } } ---- @@ -309,20 +309,17 @@ The preceding listener example is provided by the framework and checks a private NoWorkFoundStepExecutionListener tested = new NoWorkFoundStepExecutionListener(); @Test -public void testAfterStep() { +public void noWork() { StepExecution stepExecution = new StepExecution("NoProcessingStep", new JobExecution(new JobInstance(1L, new JobParameters(), "NoProcessingJob"))); + stepExecution.setExitStatus(ExitStatus.COMPLETED); stepExecution.setReadCount(0); - try { - tested.afterStep(stepExecution); - fail(); - } catch (NoWorkFoundException e) { - assertEquals("Step has not processed any items", e.getMessage()); - } -} + ExitStatus exitStatus = tested.afterStep(stepExecution); + assertEquals(ExitStatus.FAILED.getExitCode(), exitStatus.getExitCode()); +} ---- Because the Spring Batch domain model follows good object-oriented @@ -346,15 +343,12 @@ private NoWorkFoundStepExecutionListener tested = new NoWorkFoundStepExecutionLi public void testAfterStep() { StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution(); + stepExecution.setExitStatus(ExitStatus.COMPLETED); stepExecution.setReadCount(0); - try { - tested.afterStep(stepExecution); - fail(); - } catch (NoWorkFoundException e) { - assertEquals("Step has not processed any items", e.getMessage()); - } -} + ExitStatus exitStatus = tested.afterStep(stepExecution); + assertEquals(ExitStatus.FAILED.getExitCode(), exitStatus.getExitCode()); +} ---- The preceding method for creating a simple