From 1eed5a30c3586bb5e5eedd7930ac1e62ebef6cef Mon Sep 17 00:00:00 2001 From: Dimitrios Liapis Date: Sun, 19 Apr 2020 23:19:48 +0200 Subject: [PATCH] Refactor Item processor documentation Issue #3644 --- spring-batch-docs/asciidoc/index-single.adoc | 2 + spring-batch-docs/asciidoc/index.adoc | 1 + spring-batch-docs/asciidoc/processor.adoc | 379 +++++++++++++++++ .../asciidoc/readersAndWriters.adoc | 388 ------------------ 4 files changed, 382 insertions(+), 388 deletions(-) create mode 100644 spring-batch-docs/asciidoc/processor.adoc diff --git a/spring-batch-docs/asciidoc/index-single.adoc b/spring-batch-docs/asciidoc/index-single.adoc index 081f1fe01..b1f568c08 100644 --- a/spring-batch-docs/asciidoc/index-single.adoc +++ b/spring-batch-docs/asciidoc/index-single.adoc @@ -20,6 +20,8 @@ include::step.adoc[] include::readersAndWriters.adoc[] +include::processor.adoc[] + include::scalability.adoc[] include::repeat.adoc[] diff --git a/spring-batch-docs/asciidoc/index.adoc b/spring-batch-docs/asciidoc/index.adoc index e14a586ed..4b13846cd 100644 --- a/spring-batch-docs/asciidoc/index.adoc +++ b/spring-batch-docs/asciidoc/index.adoc @@ -19,6 +19,7 @@ administration. controlling step flow. <> :: Item readers and writers interfaces and how to use them. +<> :: Item processor interface and how to use it. <> :: Multi-threaded steps, parallel steps, remote chunking and partitioning. <> :: Completion policies and exception handling of repetitive actions. diff --git a/spring-batch-docs/asciidoc/processor.adoc b/spring-batch-docs/asciidoc/processor.adoc new file mode 100644 index 000000000..898aeb95c --- /dev/null +++ b/spring-batch-docs/asciidoc/processor.adoc @@ -0,0 +1,379 @@ +:batch-asciidoc: ./ +:toc: left +:toclevels: 4 + +[[processor]] +== ItemProcessor + +ifndef::onlyonetoggle[] +include::toggle.adoc[] +endif::onlyonetoggle[] + +The `ItemReader` and `ItemWriter` interfaces are both very useful for their specific +tasks, but what if you want to insert business logic before writing? One option for both +reading and writing is to use the composite pattern: Create an `ItemWriter` that contains +another `ItemWriter` or an `ItemReader` that contains another `ItemReader`. The following +code shows an example: + +[source, java] +---- +public class CompositeItemWriter implements ItemWriter { + + ItemWriter itemWriter; + + public CompositeItemWriter(ItemWriter itemWriter) { + this.itemWriter = itemWriter; + } + + public void write(List items) throws Exception { + //Add business logic here + itemWriter.write(items); + } + + public void setDelegate(ItemWriter itemWriter){ + this.itemWriter = itemWriter; + } +} +---- + +The preceding class contains another `ItemWriter` to which it delegates after having +provided some business logic. This pattern could easily be used for an `ItemReader` as +well, perhaps to obtain more reference data based upon the input that was provided by the +main `ItemReader`. It is also useful if you need to control the call to `write` yourself. +However, if you only want to 'transform' the item passed in for writing before it is +actually written, you need not `write` yourself. You can just modify the item. For this +scenario, Spring Batch provides the `ItemProcessor` interface, as shown in the following +interface definition: + +[source, java] +---- +public interface ItemProcessor { + + O process(I item) throws Exception; +} +---- + +An `ItemProcessor` is simple. Given one object, transform it and return another. The +provided object may or may not be of the same type. The point is that business logic may +be applied within the process, and it is completely up to the developer to create that +logic. An `ItemProcessor` can be wired directly into a step. For example, assume an +`ItemReader` provides a class of type `Foo` and that it needs to be converted to type `Bar` +before being written out. The following example shows an `ItemProcessor` that performs +the conversion: + +[source, java] +---- +public class Foo {} + +public class Bar { + public Bar(Foo foo) {} +} + +public class FooProcessor implements ItemProcessor { + public Bar process(Foo foo) throws Exception { + //Perform simple transformation, convert a Foo to a Bar + return new Bar(foo); + } +} + +public class BarWriter implements ItemWriter { + public void write(List bars) throws Exception { + //write bars + } +} +---- + +In the preceding example, there is a class `Foo`, a class `Bar`, and a class +`FooProcessor` that adheres to the `ItemProcessor` interface. The transformation is +simple, but any type of transformation could be done here. The `BarWriter` writes `Bar` +objects, throwing an exception if any other type is provided. Similarly, the +`FooProcessor` throws an exception if anything but a `Foo` is provided. The +`FooProcessor` can then be injected into a `Step`, as shown in the following example: + +.XML Configuration +[source, xml, role="xmlContent"] +---- + + + + + + + +---- + +.Java Configuration +[source, java, role="javaContent"] +---- +@Bean +public Job ioSampleJob() { + return this.jobBuilderFactory.get("ioSampleJob") + .start(step1()) + .end() + .build(); +} + +@Bean +public Step step1() { + return this.stepBuilderFactory.get("step1") + .chunk(2) + .reader(fooReader()) + .processor(fooProcessor()) + .writer(barWriter()) + .build(); +} +---- + +A difference between `ItemProcessor` and `ItemReader` or `ItemWriter` is that an `ItemProcessor` +is optional for a `Step`. + +[[chainingItemProcessors]] +==== Chaining ItemProcessors + +Performing a single transformation is useful in many scenarios, but what if you want to +'chain' together multiple `ItemProcessor` implementations? This can be accomplished using +the composite pattern mentioned previously. To update the previous, single +transformation, example, `Foo` is transformed to `Bar`, which is transformed to `Foobar` +and written out, as shown in the following example: + +[source, java] +---- +public class Foo {} + +public class Bar { + public Bar(Foo foo) {} +} + +public class Foobar { + public Foobar(Bar bar) {} +} + +public class FooProcessor implements ItemProcessor { + public Bar process(Foo foo) throws Exception { + //Perform simple transformation, convert a Foo to a Bar + return new Bar(foo); + } +} + +public class BarProcessor implements ItemProcessor { + public Foobar process(Bar bar) throws Exception { + return new Foobar(bar); + } +} + +public class FoobarWriter implements ItemWriter{ + public void write(List items) throws Exception { + //write items + } +} +---- + +A `FooProcessor` and a `BarProcessor` can be 'chained' together to give the resultant +`Foobar`, as shown in the following example: + + +[source, java] +---- +CompositeItemProcessor compositeProcessor = + new CompositeItemProcessor(); +List itemProcessors = new ArrayList(); +itemProcessors.add(new FooProcessor()); +itemProcessors.add(new BarProcessor()); +compositeProcessor.setDelegates(itemProcessors); +---- + +Just as with the previous example, the composite processor can be configured into the +`Step`: + +.XML Configuration +[source, xml, role="xmlContent"] +---- + + + + + + + + + + + + + + + + +---- + +.Java Configuration +[source, java, role="javaContent"] +---- +@Bean +public Job ioSampleJob() { + return this.jobBuilderFactory.get("ioSampleJob") + .start(step1()) + .end() + .build(); +} + +@Bean +public Step step1() { + return this.stepBuilderFactory.get("step1") + .chunk(2) + .reader(fooReader()) + .processor(compositeProcessor()) + .writer(foobarWriter()) + .build(); +} + +@Bean +public CompositeItemProcessor compositeProcessor() { + List delegates = new ArrayList<>(2); + delegates.add(new FooProcessor()); + delegates.add(new BarProcessor()); + + CompositeItemProcessor processor = new CompositeItemProcessor(); + + processor.setDelegates(delegates); + + return processor; +} +---- + +[[filteringRecords]] +==== Filtering Records + +One typical use for an item processor is to filter out records before they are passed to +the `ItemWriter`. Filtering is an action distinct from skipping. Skipping indicates that +a record is invalid, while filtering simply indicates that a record should not be +written. + +For example, consider a batch job that reads a file containing three different types of +records: records to insert, records to update, and records to delete. If record deletion +is not supported by the system, then we would not want to send any "delete" records to +the `ItemWriter`. But, since these records are not actually bad records, we would want to +filter them out rather than skip them. As a result, the `ItemWriter` would receive only +"insert" and "update" records. + +To filter a record, you can return `null` from the `ItemProcessor`. The framework detects +that the result is `null` and avoids adding that item to the list of records delivered to +the `ItemWriter`. As usual, an exception thrown from the `ItemProcessor` results in a +skip. + +[[validatingInput]] +==== Validating Input + +In the <> chapter, multiple approaches to parsing input have been +discussed. Each major implementation throws an exception if it is not 'well-formed'. The +`FixedLengthTokenizer` throws an exception if a range of data is missing. Similarly, +attempting to access an index in a `RowMapper` or `FieldSetMapper` that does not exist or +is in a different format than the one expected causes an exception to be thrown. All of +these types of exceptions are thrown before `read` returns. However, they do not address +the issue of whether or not the returned item is valid. For example, if one of the fields +is an age, it obviously cannot be negative. It may parse correctly, because it exists and +is a number, but it does not cause an exception. Since there are already a plethora of +validation frameworks, Spring Batch does not attempt to provide yet another. Rather, it +provides a simple interface, called `Validator`, that can be implemented by any number of +frameworks, as shown in the following interface definition: + +[source, java] +---- +public interface Validator { + + void validate(T value) throws ValidationException; + +} +---- + +The contract is that the `validate` method throws an exception if the object is invalid +and returns normally if it is valid. Spring Batch provides an out of the box +`ValidatingItemProcessor`, as shown in the following bean definition: + +.XML Configuration +[source, xml, role="xmlContent"] +---- + + + + + + + + + +---- + +.Java Configuration +[source, java, role="javaContent"] +---- +@Bean +public ValidatingItemProcessor itemProcessor() { + ValidatingItemProcessor processor = new ValidatingItemProcessor(); + + processor.setValidator(validator()); + + return processor; +} + +@Bean +public SpringValidator validator() { + SpringValidator validator = new SpringValidator(); + + validator.setValidator(new TradeValidator()); + + return validator; +} +---- + +You can also use the `BeanValidatingItemProcessor` to validate items annotated with +the Bean Validation API (JSR-303) annotations. For example, given the following type `Person`: + +[source, java] +---- +class Person { + + @NotEmpty + private String name; + + public Person(String name) { + this.name = name; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + +} +---- + +you can validate items by declaring a `BeanValidatingItemProcessor` bean in your +application context and register it as a processor in your chunk-oriented step: + +[source, java] +---- +@Bean +public BeanValidatingItemProcessor beanValidatingItemProcessor() throws Exception { + BeanValidatingItemProcessor beanValidatingItemProcessor = new BeanValidatingItemProcessor<>(); + beanValidatingItemProcessor.setFilter(true); + + return beanValidatingItemProcessor; +} +---- + +[[faultTolerant]] +==== Fault Tolerance + +When a chunk is rolled back, items that have been cached during reading may be +reprocessed. If a step is configured to be fault tolerant (typically by using skip or +retry processing), any `ItemProcessor` used should be implemented in a way that is +idempotent. Typically that would consist of performing no changes on the input item for +the `ItemProcessor` and only updating the +instance that is the result. diff --git a/spring-batch-docs/asciidoc/readersAndWriters.adoc b/spring-batch-docs/asciidoc/readersAndWriters.adoc index 2654461a9..ed204efae 100644 --- a/spring-batch-docs/asciidoc/readersAndWriters.adoc +++ b/spring-batch-docs/asciidoc/readersAndWriters.adoc @@ -92,284 +92,6 @@ method. For example, if writing to a Hibernate DAO, multiple calls to write can one for each item. The writer can then call `flush` on the hibernate session before returning. -[[itemProcessor]] -=== `ItemProcessor` - -The `ItemReader` and `ItemWriter` interfaces are both very useful for their specific -tasks, but what if you want to insert business logic before writing? One option for both -reading and writing is to use the composite pattern: Create an `ItemWriter` that contains -another `ItemWriter` or an `ItemReader` that contains another `ItemReader`. The following -code shows an example: - -[source, java] ----- -public class CompositeItemWriter implements ItemWriter { - - ItemWriter itemWriter; - - public CompositeItemWriter(ItemWriter itemWriter) { - this.itemWriter = itemWriter; - } - - public void write(List items) throws Exception { - //Add business logic here - itemWriter.write(items); - } - - public void setDelegate(ItemWriter itemWriter){ - this.itemWriter = itemWriter; - } -} ----- - -The preceding class contains another `ItemWriter` to which it delegates after having -provided some business logic. This pattern could easily be used for an `ItemReader` as -well, perhaps to obtain more reference data based upon the input that was provided by the -main `ItemReader`. It is also useful if you need to control the call to `write` yourself. -However, if you only want to 'transform' the item passed in for writing before it is -actually written, you need not `write` yourself. You can just modify the item. For this -scenario, Spring Batch provides the `ItemProcessor` interface, as shown in the following -interface definition: - -[source, java] ----- -public interface ItemProcessor { - - O process(I item) throws Exception; -} ----- - -An `ItemProcessor` is simple. Given one object, transform it and return another. The -provided object may or may not be of the same type. The point is that business logic may -be applied within the process, and it is completely up to the developer to create that -logic. An `ItemProcessor` can be wired directly into a step. For example, assume an -`ItemReader` provides a class of type `Foo` and that it needs to be converted to type `Bar` -before being written out. The following example shows an `ItemProcessor` that performs -the conversion: - -[source, java] ----- -public class Foo {} - -public class Bar { - public Bar(Foo foo) {} -} - -public class FooProcessor implements ItemProcessor { - public Bar process(Foo foo) throws Exception { - //Perform simple transformation, convert a Foo to a Bar - return new Bar(foo); - } -} - -public class BarWriter implements ItemWriter { - public void write(List bars) throws Exception { - //write bars - } -} ----- - -In the preceding example, there is a class `Foo`, a class `Bar`, and a class -`FooProcessor` that adheres to the `ItemProcessor` interface. The transformation is -simple, but any type of transformation could be done here. The `BarWriter` writes `Bar` -objects, throwing an exception if any other type is provided. Similarly, the -`FooProcessor` throws an exception if anything but a `Foo` is provided. The -`FooProcessor` can then be injected into a `Step`. - -[role="xmlContent"] -The following example shows how to inject the `FooProcessor` into a step in XML: - -.XML Configuration -[source, xml, role="xmlContent"] ----- - - - - - - - ----- - -[role="javaContent"] -The following example shows how to inject the `FooProcessor` into a step in Java: - -.Java Configuration -[source, java, role="javaContent"] ----- -@Bean -public Job ioSampleJob() { - return this.jobBuilderFactory.get("ioSampleJOb") - .start(step1()) - .end() - .build(); -} - -@Bean -public Step step1() { - return this.stepBuilderFactory.get("step1") - .chunk(2) - .reader(fooReader()) - .processor(fooProcessor()) - .writer(barWriter()) - .build(); -} ----- - -[[chainingItemProcessors]] -==== Chaining ItemProcessors - -Performing a single transformation is useful in many scenarios, but what if you want to -'chain' together multiple `ItemProcessor` implementations? This can be accomplished using -the composite pattern mentioned previously. To update the previous, single -transformation, example, `Foo` is transformed to `Bar`, which is transformed to `Foobar` -and written out, as shown in the following example: - -[source, java] ----- -public class Foo {} - -public class Bar { - public Bar(Foo foo) {} -} - -public class Foobar { - public Foobar(Bar bar) {} -} - -public class FooProcessor implements ItemProcessor { - public Bar process(Foo foo) throws Exception { - //Perform simple transformation, convert a Foo to a Bar - return new Bar(foo); - } -} - -public class BarProcessor implements ItemProcessor { - public Foobar process(Bar bar) throws Exception { - return new Foobar(bar); - } -} - -public class FoobarWriter implements ItemWriter{ - public void write(List items) throws Exception { - //write items - } -} ----- - -A `FooProcessor` and a `BarProcessor` can be 'chained' together to give the resultant -`Foobar`, as shown in the following example: - - -[source, java] ----- -CompositeItemProcessor compositeProcessor = - new CompositeItemProcessor(); -List itemProcessors = new ArrayList(); -itemProcessors.add(new FooProcessor()); -itemProcessors.add(new BarProcessor()); -compositeProcessor.setDelegates(itemProcessors); ----- - -Just as with the previous example, the composite processor can be configured into the -`Step`. - -[role="xmlContent"] -The following example shows how to configure the composite processor into the step in XML: - -.XML Configuration -[source, xml, role="xmlContent"] ----- - - - - - - - - - - - - - - - - ----- - -[role="javaContent"] -The following example shows how to configure the composite processor into the step in -Java: - -.Java Configuration -[source, java, role="javaContent"] ----- -@Bean -public Job ioSampleJob() { - return this.jobBuilderFactory.get("ioSampleJob") - .start(step1()) - .end() - .build(); -} - -@Bean -public Step step1() { - return this.stepBuilderFactory.get("step1") - .chunk(2) - .reader(fooReader()) - .processor(compositeProcessor()) - .writer(foobarWriter()) - .build(); -} - -@Bean -public CompositeItemProcessor compositeProcessor() { - List delegates = new ArrayList<>(2); - delegates.add(new FooProcessor()); - delegates.add(new BarProcessor()); - - CompositeItemProcessor processor = new CompositeItemProcessor(); - - processor.setDelegates(delegates); - - return processor; -} ----- - -[[filteringRecords]] -==== Filtering Records - -One typical use for an item processor is to filter out records before they are passed to -the `ItemWriter`. Filtering is an action distinct from skipping. Skipping indicates that -a record is invalid, while filtering simply indicates that a record should not be -written. - -For example, consider a batch job that reads a file containing three different types of -records: records to insert, records to update, and records to delete. If record deletion -is not supported by the system, then we would not want to send any "delete" records to -the `ItemWriter`. But, since these records are not actually bad records, we would want to -filter them out rather than skip them. As a result, the `ItemWriter` would receive only -"insert" and "update" records. - -To filter a record, you can return `null` from the `ItemProcessor`. The framework detects -that the result is `null` and avoids adding that item to the list of records delivered to -the `ItemWriter`. As usual, an exception thrown from the `ItemProcessor` results in a -skip. - -[[faultTolerant]] -==== Fault Tolerance - -When a chunk is rolled back, items that have been cached during reading may be -reprocessed. If a step is configured to be fault tolerant (typically by using skip or -retry processing), any `ItemProcessor` used should be implemented in a way that is -idempotent. Typically that would consist of performing no changes on the input item for -the `ItemProcessor` and only updating the -instance that is the result. - [[itemStream]] === `ItemStream` @@ -2784,116 +2506,6 @@ public FooService fooService() { } ---- -[[validatingInput]] -=== Validating Input - -During the course of this chapter, multiple approaches to parsing input have been -discussed. Each major implementation throws an exception if it is not 'well-formed'. The -`FixedLengthTokenizer` throws an exception if a range of data is missing. Similarly, -attempting to access an index in a `RowMapper` or `FieldSetMapper` that does not exist or -is in a different format than the one expected causes an exception to be thrown. All of -these types of exceptions are thrown before `read` returns. However, they do not address -the issue of whether or not the returned item is valid. For example, if one of the fields -is an age, it obviously cannot be negative. It may parse correctly, because it exists and -is a number, but it does not cause an exception. Since there are already a plethora of -validation frameworks, Spring Batch does not attempt to provide yet another. Rather, it -provides a simple interface, called `Validator`, that can be implemented by any number of -frameworks, as shown in the following interface definition: - -[source, java] ----- -public interface Validator { - - void validate(T value) throws ValidationException; - -} ----- - -The contract is that the `validate` method throws an exception if the object is invalid -and returns normally if it is valid. Spring Batch provides an out of the box -`ValidatingItemProcessor`. - -[role="xmlContent"] -The following bean definition shows how to configure a `ValidatingItemProcessor` in XML: - -.XML Configuration -[source, xml, role="xmlContent"] ----- - - - - - - - - - ----- - -[role="javaContent"] -The following bean definition shows how to configure a `ValidatingItemProcessor` in Java: - -.Java Configuration -[source, java, role="javaContent"] ----- -@Bean -public ValidatingItemProcessor itemProcessor() { - ValidatingItemProcessor processor = new ValidatingItemProcessor(); - - processor.setValidator(validator()); - - return processor; -} - -@Bean -public SpringValidator validator() { - SpringValidator validator = new SpringValidator(); - - validator.setValidator(new TradeValidator()); - - return validator; -} ----- - -You can also use the `BeanValidatingItemProcessor` to validate items annotated with -the Bean Validation API (JSR-303) annotations. For example, given the following type `Person`: - -[source, java] ----- -class Person { - - @NotEmpty - private String name; - - public Person(String name) { - this.name = name; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - -} ----- - -you can validate items by declaring a `BeanValidatingItemProcessor` bean in your -application context and register it as a processor in your chunk-oriented step: - -[source, java] ----- -@Bean -public BeanValidatingItemProcessor beanValidatingItemProcessor() throws Exception { - BeanValidatingItemProcessor beanValidatingItemProcessor = new BeanValidatingItemProcessor<>(); - beanValidatingItemProcessor.setFilter(true); - - return beanValidatingItemProcessor; -} ----- - [[process-indicator]] === Preventing State Persistence