Updated to have java config examples
This commit updates the scalability.adoc to have java configuration examples for all but the last snip in the doc.
This commit is contained in:
Chris Kirk
committed by
Michael Minella
Michael Minella
parent
38d6e7d496
commit
c3814cc27d
@@ -6,6 +6,8 @@
|
||||
|
||||
== Scaling and Parallel Processing
|
||||
|
||||
include::toggle.adoc[]
|
||||
|
||||
Many batch processing problems can be solved with single threaded, single process jobs,
|
||||
so it is always a good idea to properly check if that meets your needs before thinking
|
||||
about more complex implementations. Measure the performance of a realistic job and see if
|
||||
@@ -34,16 +36,42 @@ First, we review the single-process options. Then we review the multi-process op
|
||||
=== Multi-threaded Step
|
||||
|
||||
The simplest way to start parallel processing is to add a `TaskExecutor` to your Step
|
||||
configuration. For example, you might add an attribute of the `tasklet`, as shown in the
|
||||
configuration.
|
||||
|
||||
[role="xmlContent"]
|
||||
For example, you might add an attribute of the `tasklet`, as shown in the
|
||||
following example:
|
||||
|
||||
[source, xml]
|
||||
[source, xml, role="xmlContent"]
|
||||
----
|
||||
<step id="loading">
|
||||
<tasklet task-executor="taskExecutor">...</tasklet>
|
||||
</step>
|
||||
----
|
||||
|
||||
[role="javaContent"]
|
||||
When using java configuration, a `TaskExecutor` can be added to the step
|
||||
as shown in the following example:
|
||||
|
||||
.Java Configuration
|
||||
[source, java, role="javaContent"]
|
||||
----
|
||||
@Bean
|
||||
public TaskExecutor taskExecutor(){
|
||||
return new SimpleAsyncTaskExecutor("spring_batch");
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Step sampleStep(TaskExecutor taskExecutor) {
|
||||
return this.stepBuilderFactory.get("sampleStep")
|
||||
.<String, String>chunk(10)
|
||||
.reader(itemReader())
|
||||
.writer(itemWriter())
|
||||
.taskExecutor(taskExecutor())
|
||||
.build();
|
||||
}
|
||||
----
|
||||
|
||||
In this example, the taskExecutor is a reference to another bean definition that
|
||||
implements the `TaskExecutor` interface.
|
||||
https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/core/task/TaskExecutor.html[`TaskExecutor`]
|
||||
@@ -57,10 +85,12 @@ Note that this means there is no fixed order for the items to be processed, and
|
||||
might contain items that are non-consecutive compared to the single-threaded case. In
|
||||
addition to any limits placed by the task executor (such as whether it is backed by a
|
||||
thread pool), there is a throttle limit in the tasklet configuration which defaults to 4.
|
||||
You may need to increase this to ensure that a thread pool is fully utilized, as shown in
|
||||
the following example:
|
||||
You may need to increase this to ensure that a thread pool is fully utilized.
|
||||
|
||||
[source, xml]
|
||||
[role="xmlContent"]
|
||||
For example you might increase threads, as shown in the following example:
|
||||
|
||||
[source, xml, role="xmlContent"]
|
||||
----
|
||||
<step id="loading"> <tasklet
|
||||
task-executor="taskExecutor"
|
||||
@@ -68,6 +98,21 @@ the following example:
|
||||
</step>
|
||||
----
|
||||
|
||||
[role="javaContent"]
|
||||
When using java configuration, threads can be increased on the `TaskExecutor`,
|
||||
as shown in the following example:
|
||||
|
||||
.Java Configuration
|
||||
[source, java, role="javaContent"]
|
||||
----
|
||||
@Bean
|
||||
public TaskExecutor taskExecutor(){
|
||||
SimpleAsyncTaskExecutor asyncTaskExecutor=new SimpleAsyncTaskExecutor("spring_batch");
|
||||
asyncTaskExecutor.setConcurrencyLimit(20);
|
||||
return asyncTaskExecutor;
|
||||
}
|
||||
----
|
||||
|
||||
Note also that there may be limits placed on concurrency by any pooled resources used in
|
||||
your step, such as a `DataSource`. Be sure to make the pool in those resources at least
|
||||
as large as the desired number of concurrent threads in the step.
|
||||
@@ -100,11 +145,13 @@ single threaded configuration.
|
||||
|
||||
As long as the application logic that needs to be parallelized can be split into distinct
|
||||
responsibilities and assigned to individual steps, then it can be parallelized in a
|
||||
single process. Parallel Step execution is easy to configure and use, for example, to
|
||||
execute steps `(step1,step2)` in parallel with `step3`, as shown in the following
|
||||
example:
|
||||
single process. Parallel Step execution is easy to configure and use.
|
||||
|
||||
[source, xml]
|
||||
[role="xmlContent"]
|
||||
For example, executing steps `(step1,step2)` in parallel with `step3` is straightforward,
|
||||
as shown in the following example:
|
||||
|
||||
[source, xml, role="xmlContent"]
|
||||
----
|
||||
<job id="job1">
|
||||
<split id="split1" task-executor="taskExecutor" next="step4">
|
||||
@@ -122,6 +169,51 @@ example:
|
||||
<beans:bean id="taskExecutor" class="org.spr...SimpleAsyncTaskExecutor"/>
|
||||
----
|
||||
|
||||
[role="javaContent"]
|
||||
When using java configuration, executing steps `(step1,step2)` in parallel with `step3`
|
||||
is straightforward, as shown in the following example:
|
||||
|
||||
.Java Configuration
|
||||
[source, java, role="javaContent"]
|
||||
----
|
||||
@Bean
|
||||
public Job job() {
|
||||
return jobBuilderFactory.get("job")
|
||||
.start(splitFlow())
|
||||
.next(step4())
|
||||
.build() //builds FlowJobBuilder instance
|
||||
.build(); //builds Job instance
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Flow splitFlow() {
|
||||
return new FlowBuilder<SimpleFlow>("splitFlow")
|
||||
.split(taskExecutor())
|
||||
.add(flow1(), flow2())
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Flow flow1() {
|
||||
return new FlowBuilder<SimpleFlow>("flow1")
|
||||
.start(step1())
|
||||
.next(step2())
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Flow flow2() {
|
||||
return new FlowBuilder<SimpleFlow>("flow2")
|
||||
.start(step3())
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public TaskExecutor taskExecutor(){
|
||||
return new SimpleAsyncTaskExecutor("spring_batch");
|
||||
}
|
||||
----
|
||||
|
||||
The configurable "task-executor" attribute is used to specify which `TaskExecutor`
|
||||
implementation should be used to execute the individual flows. The default is
|
||||
`SyncTaskExecutor`, but an asynchronous `TaskExecutor` is required to run the steps in
|
||||
@@ -191,9 +283,12 @@ image::{batch-asciidoc}images/partitioning-spi.png[Partitioning SPI, scaledwidth
|
||||
|
||||
The `Step` on the right in this case is the "remote" slave, so, potentially, there are
|
||||
many objects and or processes playing this role, and the `PartitionStep` is shown driving
|
||||
the execution. The following example shows the `PartitionStep` configuration:
|
||||
the execution.
|
||||
|
||||
[source, xml]
|
||||
[role="xmlContent"]
|
||||
The following example shows the `PartitionStep` configuration:
|
||||
|
||||
[source, xml, role="xmlContent"]
|
||||
----
|
||||
<step id="step1.master">
|
||||
<partition step="step1" partitioner="partitioner">
|
||||
@@ -202,6 +297,23 @@ the execution. The following example shows the `PartitionStep` configuration:
|
||||
</step>
|
||||
----
|
||||
|
||||
[role="javaContent"]
|
||||
The following example shows the `PartitionStep` configuration using java configuration:
|
||||
|
||||
.Java Configuration
|
||||
[source, java, role="javaContent"]
|
||||
----
|
||||
@Bean
|
||||
public Step step1Master() {
|
||||
return stepBuilderFactory.get("step1.master")
|
||||
.<String, String>partitioner("step1", partitioner())
|
||||
.step(step1())
|
||||
.gridSize(10)
|
||||
.taskExecutor(taskExecutor())
|
||||
.build();
|
||||
}
|
||||
----
|
||||
|
||||
Similar to the multi-threaded step's `throttle-limit` attribute, the `grid-size`
|
||||
attribute prevents the task executor from being saturated with requests from a single
|
||||
step.
|
||||
@@ -237,11 +349,14 @@ or remoting fabrics.
|
||||
Spring Batch does, however, provide a useful implementation of `PartitionHandler` that
|
||||
executes `Step` instances locally in separate threads of execution, using the
|
||||
`TaskExecutor` strategy from Spring. The implementation is called
|
||||
`TaskExecutorPartitionHandler`, and it is the default for a step configured with the XML
|
||||
`TaskExecutorPartitionHandler`.
|
||||
|
||||
[role="xmlContent"]
|
||||
The `TaskExecutorPartitionHandler` is the default for a step configured with the XML
|
||||
namespace shown previously. It can also be configured explicitly, as shown in the
|
||||
following example:
|
||||
|
||||
[source, xml]
|
||||
[source, xml, role="xmlContent"]
|
||||
----
|
||||
<step id="step1.master">
|
||||
<partition step="step1" handler="handler"/>
|
||||
@@ -254,6 +369,31 @@ following example:
|
||||
</bean>
|
||||
----
|
||||
|
||||
[role="javaContent"]
|
||||
The `TaskExecutorPartitionHandler` can be configured explicitly within java configuration,
|
||||
as shown in the following example:
|
||||
|
||||
.Java Configuration
|
||||
[source, java, role="javaContent"]
|
||||
----
|
||||
@Bean
|
||||
public Step step1Master() {
|
||||
return stepBuilderFactory.get("step1.master")
|
||||
.partitioner("step1", partitioner())
|
||||
.partitionHandler(partitionHandler())
|
||||
.build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public PartitionHandler partitionHandler() {
|
||||
TaskExecutorPartitionHandler retVal = new TaskExecutorPartitionHandler();
|
||||
retVal.setTaskExecutor(taskExecutor());
|
||||
retVal.setStep(step1());
|
||||
retVal.setGridSize(10);
|
||||
return retVal;
|
||||
}
|
||||
----
|
||||
|
||||
The `gridSize` attribute determines the number of separate step executions to create, so
|
||||
it can be matched to the size of the thread pool in the `TaskExecutor`. Alternatively, it
|
||||
can be set to be larger than the number of threads available, which makes the blocks of
|
||||
|
||||
Reference in New Issue
Block a user