Before this commit, a null job parameter of type long
or double was retrieved as 0L or 0.0. This caused
JobOperator.restart to create a new job instance
instead of restarting the previous failed execution.
This commit fixes how null job parameters of type
long or double are retrieved from the database to
correctly restart the same job instance.
Issue #4087
This commit adds DEFAULT to BATCH_JOB_EXECUTION#JOB_CONFIGURATION_LOCATION
for DB2. This fixes a SqlSyntaxErrorException with DB2 v9.
Credits to @omheni for the fix, see #4002.
Resolves#4001
Before this commit, SimpleJobExplorer#getLastJobExecution returned
the last job execution without fetching its step executions and
their execution contexts.
This commit fixes the implementation to load the entire object
graph as done in SimpleJobExplorer#getJobExecution.
Resolves#3943#3944
Override SimpleStepBuilder.faultTolerant() in FaultTolerantStepBuilder
to prevent creation of a new FaultTolerantStepBuilder when calling
faultTolerant() on an existing FaultTolerantStepBuilder. Otherwise
configuration, like chunkListeners, are lost.
Issue #3840
Before this commit, the logic that validates if a job
instance is complete or not was based only on the size
of previous parameters, without checking if they are
identifying or not.
This commit introduces a change that uses only
identifying job parameters to identify a job instance
and validate its previous executions.
Issue #1221
Guarded the logging statements which do concatenation of strings
or calling toString on objects. When a log level isn't enabled
this still would produce garbage that would need to be collected.
Guarded all logging up to info, warn and error can be assumed to be
enabled on a production system.
Some tests in PartitionParserTests were
failing intermittently due to the usage
of non-synchronized shared state between
concurrent threads.
This commit updates the test code to use
`AtomicInteger` instead of `int` for the
state shared between concurrent threads.
(cherry picked from commit 98fba4a03b)
This test was failing intermittently due to an incorrect
way of waiting for a job execution to finish, which is:
```
while(jobExecution.isRunning()) {
// wait for async launched job to complete execution
}
```
`JobExecution#isRunning()` is based on the status
of the job execution in memory which might not be
persisted yet. Here is an excerpt from the Javadoc:
```
Test if this JobExecution indicates that it is
running. It should be noted that this does not
necessarily mean that it has been persisted as
such yet.
```
That's why in the case where `isRunning` returns
false and the JobExecution is not persisted yet
(which is still in a running status in db),
the second attempt of re-running the job fails
with a `JobExecutionAlreadyRunningException`.
This commit fixes the loop by continuously
checking the status of the Job execution
in the job repository until it reaches one
of the end statuses.
Issue #1121
(cherry picked from commit cf2642223d)
JsrSplitParsingTests#test and DecisionStepTests#testDecisionAfterSplit
were failing intermittently with:
```
org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
at org.springframework.batch.core.jsr.job.flow.JsrFlowJob.doExecute(JsrFlowJob.java:88) ~[main/:?]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) [main/:?]
at org.springframework.batch.core.jsr.launch.JsrJobOperator$2.run(JsrJobOperator.java:674) [main/:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: Ended flow=flow1.step1 at state=flow1.step1.flow1.step1 with exception
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:178) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
Caused by: org.springframework.dao.ConcurrencyFailureException: PreparedStatementCallback; SQL [INSERT INTO BATCH_STEP_EXECUTION_CONTEXT (SHORT_CONTEXT, SERIALIZED_CONTEXT, STEP_EXECUTION_ID) VALUES(?, ?, ?)]; transaction rollback: serialization failure; nested exception is java.sql.SQLTransactionRollbackException: transaction rollback: serialization failure
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:73) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236) ~[main/:?]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveExecutionContext(JdbcExecutionContextDao.java:189) ~[main/:?]
at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:177) ~[main/:?]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.0.jar:5.3.0]
at com.sun.proxy.$Proxy41.add(Unknown Source) ~[?:?]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) ~[main/:?]
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
Caused by: java.sql.SQLTransactionRollbackException: transaction rollback: serialization failure
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236) ~[main/:?]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveExecutionContext(JdbcExecutionContextDao.java:189) ~[main/:?]
at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:177) ~[main/:?]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.0.jar:5.3.0]
at com.sun.proxy.$Proxy41.add(Unknown Source) ~[?:?]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) ~[main/:?]
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
Caused by: org.hsqldb.HsqlException: transaction rollback: serialization failure
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.Session.handleAbortTransaction(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236) ~[main/:?]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveExecutionContext(JdbcExecutionContextDao.java:189) ~[main/:?]
at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:177) ~[main/:?]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.0.jar:5.3.0]
at com.sun.proxy.$Proxy41.add(Unknown Source) ~[?:?]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) ~[main/:?]
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
```
This failure is due to a concurrency issue when executing
the split flow with the default SimpleAsyncTaskExecutor.
Several attempts have been made to fix this issue with no success:
* use `@DirtiesContext` annotation
* use a separate db for each test
* use `READ_COMMITTED` isolation level in the job repository
* downgrade hsqldb from v2.5.1 to v2.4.1 (since v2.5.1 has
introduced several changes in the MVCC mode)
The issue seems to be related to how the in-memory database
is shared between tests *and* how the test context is cached
(ie the combination of both).
This commit moves these tests to a separate test class which
seem to fix the issue.
(cherry picked from commit 1090731302)
Before this commit, the RunIdIncrementer was failing
with a ClassCastException if the run.id parameter is
not passed as a Long.
This commit makes the RunIdIncrementer more liberal
in what it accepts by trying to parse the parameter
to a Long.
Resolves#3799
This commit deprecates the Map-based JobRepository
and JobExplorer factory beans with the associated
DAOs in favor of using the JDBC-based implementations
with an in-memory database.
Resolves#3780
Calling jobExecution#stop manually is prone to
forgetting to save the job execution in the job
repository.
The way to request a job execution to stop is
by using JobOperator#stop or by using the
CommandLineJobRunner with the "-stop" option.
Both will correctly set the status to STOPPING
and update the job execution in the repository.
Resolves#1605
Before this commit, metrics were not collected in a fault-tolerant step.
This commit updates the FaultTolerantChunkProcessor to collect metrics.
For the record, chunk scanning is not covered for two reasons:
1. When scanning a chunk, there is a single item in each write operation,
so it would be incorrect to report a metric called "chunk.write" for a
single item. We could argue that it is a singleton chunk, but still..
If we want to time scanned (aka individual) items, we need a more fine
grained timer called "scanned.item.write" for example.
2. The end result can be confusing and might distort the overall metrics
view in case of errors (because of the noisy metrics of additional transactions
for individual items).
As a reminder, the goal of the "chunk.write" metric is to give an overview
of the write operation time of the whole chunk and not to time each item
individually (this could be done using an `ItemWriteListener` if needed).
Resolves#3664
