Due to GenericRecord being checked before SpecificRecordBase,
GenericDatumWriter was used incorrectly for types that extend
SpecificRecordBase.
Resolves#3859
The sample that shows how to provide a custom transaction manager
via a BatchConfigurer has been updated to use the constructor
of `DefaultBatchConfigurer` that takes a DataSource.
Issue #3888
Override SimpleStepBuilder.faultTolerant() in FaultTolerantStepBuilder
to prevent creation of a new FaultTolerantStepBuilder when calling
faultTolerant() on an existing FaultTolerantStepBuilder. Otherwise
configuration, like chunkListeners, are lost.
Issue #3840
Before this commit, the logic that validates if a job
instance is complete or not was based only on the size
of previous parameters, without checking if they are
identifying or not.
This commit introduces a change that uses only
identifying job parameters to identify a job instance
and validate its previous executions.
Issue #1221
Both the PatternMatcher and JdbcParameterUtils use String.toCharArray for
processing. This will create a new char[] which needs to be collected. The
same code has been rewritten to use String.charAt to reduce the overhead.
Additionally unneeded object creation in the PatternMatcher has been removed
as well, to reduce further objects needed to be gc'ed.
StringBuilder by default has a size of 16, however there
are some places that already create larger strings already.
For efficiency and garbage reduction it would be better to have
larger sizes to begin with.
Guarded the logging statements which do concatenation of strings
or calling toString on objects. When a log level isn't enabled
this still would produce garbage that would need to be collected.
Guarded all logging up to info, warn and error can be assumed to be
enabled on a production system.
Currently when reading/writing JSON and using the reader/marshaller provided
and you want a custom Gson or ObjectMapper instance it still creates the not
needed instance. Move the construction to a constructor and provide a constructor
to directly pass in the pre-configured Gson or ObjectMapper instance.
The same approach is used in Spring itself where Gson or ObjectMapper instances
can be passed in.
Some tests in PartitionParserTests were
failing intermittently due to the usage
of non-synchronized shared state between
concurrent threads.
This commit updates the test code to use
`AtomicInteger` instead of `int` for the
state shared between concurrent threads.
(cherry picked from commit 98fba4a03b)
Tests in this class fail intermittently because
they send messages to Kafka in an asynchronous
way and assert on the results immediately without
waiting for the send operation to complete.
This commit updates the tests to wait for
send results before asserting on them
(similar to a140a9f5).
(cherry picked from commit 5826111b85)
This test was failing intermittently due to an incorrect
way of waiting for a job execution to finish, which is:
```
while(jobExecution.isRunning()) {
// wait for async launched job to complete execution
}
```
`JobExecution#isRunning()` is based on the status
of the job execution in memory which might not be
persisted yet. Here is an excerpt from the Javadoc:
```
Test if this JobExecution indicates that it is
running. It should be noted that this does not
necessarily mean that it has been persisted as
such yet.
```
That's why in the case where `isRunning` returns
false and the JobExecution is not persisted yet
(which is still in a running status in db),
the second attempt of re-running the job fails
with a `JobExecutionAlreadyRunningException`.
This commit fixes the loop by continuously
checking the status of the Job execution
in the job repository until it reaches one
of the end statuses.
Issue #1121
(cherry picked from commit cf2642223d)
JsrSplitParsingTests#test and DecisionStepTests#testDecisionAfterSplit
were failing intermittently with:
```
org.springframework.batch.core.JobExecutionException: Flow execution ended unexpectedly
at org.springframework.batch.core.jsr.job.flow.JsrFlowJob.doExecute(JsrFlowJob.java:88) ~[main/:?]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) [main/:?]
at org.springframework.batch.core.jsr.launch.JsrJobOperator$2.run(JsrJobOperator.java:674) [main/:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.springframework.batch.core.job.flow.FlowExecutionException: Ended flow=flow1.step1 at state=flow1.step1.flow1.step1 with exception
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:178) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
Caused by: org.springframework.dao.ConcurrencyFailureException: PreparedStatementCallback; SQL [INSERT INTO BATCH_STEP_EXECUTION_CONTEXT (SHORT_CONTEXT, SERIALIZED_CONTEXT, STEP_EXECUTION_ID) VALUES(?, ?, ?)]; transaction rollback: serialization failure; nested exception is java.sql.SQLTransactionRollbackException: transaction rollback: serialization failure
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:73) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236) ~[main/:?]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveExecutionContext(JdbcExecutionContextDao.java:189) ~[main/:?]
at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:177) ~[main/:?]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.0.jar:5.3.0]
at com.sun.proxy.$Proxy41.add(Unknown Source) ~[?:?]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) ~[main/:?]
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
Caused by: java.sql.SQLTransactionRollbackException: transaction rollback: serialization failure
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236) ~[main/:?]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveExecutionContext(JdbcExecutionContextDao.java:189) ~[main/:?]
at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:177) ~[main/:?]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.0.jar:5.3.0]
at com.sun.proxy.$Proxy41.add(Unknown Source) ~[?:?]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) ~[main/:?]
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
Caused by: org.hsqldb.HsqlException: transaction rollback: serialization failure
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.Session.handleAbortTransaction(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.Session.executeCompiledStatement(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source) ~[hsqldb-2.5.1.jar:2.5.1]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) ~[spring-jdbc-5.3.0.jar:5.3.0]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236) ~[main/:?]
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.saveExecutionContext(JdbcExecutionContextDao.java:189) ~[main/:?]
at org.springframework.batch.core.repository.support.SimpleJobRepository.add(SimpleJobRepository.java:177) ~[main/:?]
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:371) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:134) ~[spring-tx-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.0.jar:5.3.0]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.0.jar:5.3.0]
at com.sun.proxy.$Proxy41.add(Unknown Source) ~[?:?]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) ~[main/:?]
at org.springframework.batch.core.jsr.job.flow.support.state.JsrStepState.handle(JsrStepState.java:53) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[main/:?]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[main/:?]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[main/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
... 1 more
```
This failure is due to a concurrency issue when executing
the split flow with the default SimpleAsyncTaskExecutor.
Several attempts have been made to fix this issue with no success:
* use `@DirtiesContext` annotation
* use a separate db for each test
* use `READ_COMMITTED` isolation level in the job repository
* downgrade hsqldb from v2.5.1 to v2.4.1 (since v2.5.1 has
introduced several changes in the MVCC mode)
The issue seems to be related to how the in-memory database
is shared between tests *and* how the test context is cached
(ie the combination of both).
This commit moves these tests to a separate test class which
seem to fix the issue.
(cherry picked from commit 1090731302)
