Add builders to simplify remote partitioning setup

Resolves BATCH-2687

* use hard coded constants in docs
* change visibility of method `beanFactory` to public in builders
This commit is contained in:
Mahmoud Ben Hassine
2018-07-05 22:21:48 +02:00
committed by Michael Minella
parent 9f2b752f3f
commit 39676a45c4
16 changed files with 1351 additions and 185 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 362 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 376 KiB

View File

@@ -945,7 +945,7 @@ two beans that can be autowired in the application context:
* `RemoteChunkingMasterStepBuilderFactory`: used to configure the master step
* `RemoteChunkingWorkerBuilder`: used to configure the remote worker integration flow
These APIs will take care of configuring a number of components as described in the following diagram:
These APIs take care of configuring a number of components as described in the following diagram:
.Remote Chunking Configuration
image::{batch-asciidoc}images/remote-chunking-config.png[Remote Chunking Configuration, scaledwidth="80%"]
@@ -1261,3 +1261,92 @@ You must also ensure that the partition `handler` attribute maps to the `partiti
You can find a complete example of a remote partitioning job
link:$$https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples#remote-partitioning-sample$$[here].
The `@EnableBatchIntegration` annotation that can be used to simplify a remote
partitioning setup. This annotation provides two beans useful for remote partitioning:
* `RemotePartitioningMasterStepBuilderFactory`: used to configure the master step
* `RemotePartitioningWorkerStepBuilderFactory`: used to configure the worker step
These APIs take care of configuring a number of components as described in the following diagram:
.Remote Partitioning Configuration (with job repository polling)
image::{batch-asciidoc}images/remote-partitioning-polling-config.png[Remote Partitioning Configuration (with job repository polling), scaledwidth="80%"]
.Remote Partitioning Configuration (with replies aggregation)
image::{batch-asciidoc}images/remote-partitioning-aggregation-config.png[Remote Partitioning Configuration (with replies aggregation), scaledwidth="80%"]
On the master side, the `RemotePartitioningMasterStepBuilderFactory` allows you to
configure a master step by declaring:
* the `Partitioner` used to partition data
* the output channel ("Outgoing requests") to send requests to workers
* the input channel ("Incoming replies") to receive replies from workers (when configuring replies aggregation)
* the poll interval and timeout parameters (when configuring job repository polling)
The `MessageChannelPartitionHandler` and the `MessagingTemplate` are not needed to be explicitly configured
(Those can still be explicitly configured if required).
On the worker side, the `RemotePartitioningWorkerStepBuilderFactory` allows you to configure a worker to:
* listen to requests sent by the master on the input channel ("Incoming requests")
* call the `handle` method of `StepExecutionRequestHandler` for each request
* send replies on the output channel ("Outgoing replies") to the master
There is no need to explicitly configure the `StepExecutionRequestHandler` (which can be explicitly configured if required).
The following example shows how to use these APIs:
[source, java]
----
@Configuration
@EnableBatchProcessing
@EnableBatchIntegration
public class RemotePartitioningJobConfiguration {
@Configuration
public static class MasterConfiguration {
@Autowired
private RemotePartitioningMasterStepBuilderFactory masterStepBuilderFactory;
@Bean
public Step masterStep() {
return this.masterStepBuilderFactory
.get("masterStep")
.partitioner("workerStep", partitioner())
.gridSize(10)
.outputChannel(outgoingRequestsToWorkers())
.inputChannel(incomingRepliesFromWorkers())
.build();
}
// Middleware beans setup omitted
}
@Configuration
public static class WorkerConfiguration {
@Autowired
private RemotePartitioningWorkerStepBuilderFactory workerStepBuilderFactory;
@Bean
public Step workerStep() {
return this.workerStepBuilderFactory
.get("workerStep")
.inputChannel(incomingRequestsFromMaster())
.outputChannel(outgoingRepliesToMaster())
.chunk(100)
.reader(itemReader())
.processor(itemProcessor())
.writer(itemWriter())
.build();
}
// Middleware beans setup omitted
}
}
----

View File

@@ -9,7 +9,7 @@
The Spring Batch 4.1 release adds the following features:
* A new `@SpringBatchTest` annotation to simplify testing batch components
* A new `@EnableBatchIntegration` annotation to simplify remote chunking configuration
* A new `@EnableBatchIntegration` annotation to simplify remote chunking and partitioning configuration
* A new `JsonItemReader` and `JsonFileItemWriter` to support the JSON format
* Add support for validating items with the Bean Validation API
@@ -125,6 +125,55 @@ that uses these new APIs in the
link:$$https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples#remote-chunking-sample$$[samples module]
as well as more details in the <<spring-batch-integration.adoc#remote-chunking,Spring Batch Integration>> chapter.
Just like the remote chunking configuration simplification, this version also
introduces new APIs to simplify a remote partitioning setup:
`RemotePartitioningMasterStepBuilder` and `RemotePartitioningWorkerStepBuilder`.
Those can be autowired in your configuration class if the
`@EnableBatchIntegration` is present as shown in the following example:
[source, java]
----
@Configuration
@EnableBatchProcessing
@EnableBatchIntegration
public class RemotePartitioningAppConfig {
@Autowired
private RemotePartitioningMasterStepBuilderFactory masterStepBuilderFactory;
@Autowired
private RemotePartitioningWorkerStepBuilderFactory workerStepBuilderFactory;
@Bean
public Step masterStep() {
return this.masterStepBuilderFactory
.get("masterStep")
.partitioner("workerStep", partitioner())
.gridSize(10)
.outputChannel(outgoingRequestsToWorkers())
.inputChannel(incomingRepliesFromWorkers())
.build();
}
@Bean
public Step workerStep() {
return this.workerStepBuilderFactory
.get("workerStep")
.inputChannel(incomingRequestsFromMaster())
.outputChannel(outgoingRepliesToMaster())
.chunk(100)
.reader(itemReader())
.processor(itemProcessor())
.writer(itemWriter())
.build();
}
// Middleware beans setup omitted
}
----
You can find more details about these new APIs in the <<spring-batch-integration.adoc#remote-partitioning,Spring Batch Integration>> chapter.
[[whatsNewJson]]
=== JSON support