From 29c4b77e4ac0bdb025ea2530a4b3c6fd54402dae Mon Sep 17 00:00:00 2001 From: dsyer Date: Tue, 29 Jul 2008 17:09:05 +0000 Subject: [PATCH] copy samples docos over from trunk --- spring-batch-samples/src/site/apt/index.apt | 671 ++++++++++++++------ 1 file changed, 478 insertions(+), 193 deletions(-) diff --git a/spring-batch-samples/src/site/apt/index.apt b/spring-batch-samples/src/site/apt/index.apt index 1d1d97c6e..16b5608bb 100644 --- a/spring-batch-samples/src/site/apt/index.apt +++ b/spring-batch-samples/src/site/apt/index.apt @@ -46,37 +46,45 @@ Spring Batch Samples Here is a list of samples with checks to indicate which features each one demonstrates: *----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+ -|<> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> +|<> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | <> | *---- -adhocLoopJob | | | | | | | | | | | | | | | x | | +{{adhocLoop}} | | | | | | | | | | | | | | |x | | | | *---- -beanWrapperMapperSample | | x | | | | | | | | | x | | | x | | x | +{{batchUpdate}} | | | | | |x | | | | | | | | | | | |x | *---- -compositeProcessorSample | | | | | | | | x | | | x | | | | | x | +{{beanWrapperMapperSample}} | |x | | | | | | | | |x | | |x | |x | | | *---- -delegatingJob | | | | | | | | | | | | | | | | | x +{{compositeItemWriterSample}}| | | | | | | |x | | |x | | | | |x | | | *---- -fixedLengthImportJob | | x | | | | | | | | | x | | | | | x | +{{delegating}} | | | | | | | | | | | | | | | | |x | | *---- -hibernateJob | | | | | | x | | | | | x | | | | | | +{{fixedLengthImport}} | |x | | | | | | | | |x | | | | |x | | | *---- -ibatisJob | | | | | x | | | | | | x | | | | | | +{{football}} |x | | | | |x | | | | |x | | | | | | | | *---- -infiniteLoopJob | | | | | | | | | | | | | | | | | +{{hibernate}} | | | | | |x | | | | |x | | | | | | |x | *---- -multilineJob | | x | | x | | | | | | | | | | | | | +{{ibatis}} | | | | |x | | | | | |x | | | | | | | | *---- -multilineOrderJob | x | | | x | | | | x | | x | | | | | | | +{{multiline}} | |x | |x | | | | | | | | | | | | | | | *---- -fotballlJob | x | | | | | x | | | | | x | | | | | | +{{multilineOrder}} |x | | |x | | | |x | |x | | | | | | | | | *---- -restartSample | | x | | | | | | | | | x | | x | | | x | +{{parallel}} | |x | | | | | | | | |x | | | | |x | | | *---- -simpleTaskletJob | | | | | | | | | | | | | | | | | +{{quartzSample}} |x | | | | |x | | | | |x | | | |x | | | | *---- -tradeJob | x | | | | x | | x | | | | x | | | | | x | +{{restartSample}} | |x | | | | | | | | |x | |x | | |x | | | *---- -xmlStaxJob | | | x | | | | | | x | | | | | | | | +{{retrySample}} | | | | | | | | | | | | | | | |x | | | +*---- +{{skipSample}} |x | | | |x | |x | | | |x |x | | | |x | | | +*---- +{{tasklet}} | | | | | | | | | | | | | | | | | | | +*---- +{{trade}} |x | | | |x | |x | | | |x | | | | |x | | | +*---- +{{xmlStax}} | | |x | | | | | |x | | | | | | | | | | *---- * Common Sample Source Structures @@ -102,84 +110,93 @@ xmlStaxJob | | | x | | | | | | x | | | | | | | | Each job consists of several steps, these steps are defined in steps property. -** Tasklet Job +* Adhoc Loop and JMX Demo ({adhocLoop}) - The goal is to show the simplest use of the batch framework with a - single job with a single step, which cleans up a directory and runs - a system command. + This job is simply an infinite loop. It runs forever so it is + useful for testing features to do with stopping and starting jobs. + It is used, for instance, as one of the jobs that can be run from + JMX using the Eclipse launch configuration "jmxLauncher". - This job is defined by <<>> file. The - <<>> itself is defined by the bean definition with - <<>>. In this example we have two steps. + The JMX launcher uses an additional XML configuration file + (adhoc-job-launcher-context.xml) to set up a <<>> for + running jobs asynchronously (i.e. in a background thread). This + follows the same pattern as the {{{quartzSample}Quartz sample}}, so + see that section for more details of the <<>> + configuration. - * The first step defines a tasklet that is responsible for - clearing out a directory though a custom <<>>. Each - tasklet has an <<>> method which is called by the - step. All processing of business data should be handled by this - method. + The rest of the configuration for this demo consists of exposing + some components from the application context as JMX managed beans. + The <<>> is exposed as a stripped down interface + <<>>, so that it can be controlled from a + remote client (such as JConsole from the JDK) which does not have + Spring Batch on the classpath. See the Spring Core Reference Guide + for more details on how to customise the JMX configuration. - * The second step uses another tasklet to execute a system (OS) - command line. +* Batch Update ({batchUpdate}) - taskletJob.xml + The purpose of this sample is to show to usage of the + <<>> to make efficient updates to a + database table. - You can visualize the Spring configuration of a job through - Spring-IDE. See {{{http://springide.org/blog/}Spring IDE}}. The - source view of the configuration is as follows: + The <<>> accepts a special form of + <<>> as a (mandatory) dependency. This is + responsible for copying fields from the item to be written to a + <<>> matching the SQL query that has been + injected. The implementation of the + <<>> shows best + practice of keeping all the information needed for the execution in + one place, since it contains a static constant value (<<>>) + which is used to configure the query for the writer. -+--- - - - - - - - - - - - - - - - - - - - - - +* BeanWrapperMapper Sample ({beanWrapperMapperSample}) - + This sample shows the use of automatic mapping from fields in a file + to a domain object. The <<>> and <<>> objects needed + by the job are created from the Spring configuration using prototype + beans, and then their properties are set using the + <<>>, which sets properties of the + prototype according to the field names in the file. - - - -+--- + Nested property paths are resolved in the same way as normal Spring + binding occurs, but with a little extra leeway in terms of spelling + and capitalisation. Thus for instance, the <<>> object has a + property called <<>> (lower case), but the file has been + configured to have a column name <<>> (upper case), and + the mapper will accept the values happily. Underscores instead of + camel-casing (e.g. <<>> instead of <<>>) + also work. - For simplicity we are only displaying the job configuration itself - and leaving out the details of the supporting batch execution - environment configuration. +* Composite ItemWriter Sample ({compositeItemWriterSample}) -** Fixed Length Import Job + This shows a common use case using a composite pattern, composing + instances of other framework readers or writers. It is also quite + common for business-specific readers or writers to wrap + off-the-shelf components in a similar way. + + In this job the composite pattern is used just to make duplicate + copies of the output data. The delegates for the + <<>> have to be separately registered as + streams in the <<>> where they are used, in order for the step + to be restartable. This is a common feature of all delegate + patterns. + +* Delegating Sample ({delegating}) + + This sample shows the delegate pattern again, and also the + <<>> which is used to adapt a POJO to the + <<>> interface. + +* Fixed Length Import Job ({fixedLengthImport}) The goal is to demonstrate a typical scenario of importing data from a fixed-length file to database - This job shows a more typical scenario, when reading - input data and processing the data is cleanly separated. The data - provider is responsible for reading input and mapping each record to - a domain object, which is then passed to the module processor. The - module processor handles the processing of the domain objects, in - this case it only writes them to database. - - fixedLengthImportJob.xml - - file with fixed row structure + This job shows a typical scenario, when reading input data and + processing the data is cleanly separated. The data provider is + responsible for reading input and mapping each record to a domain + object, which is then passed to the module processor. The module + processor handles the processing of the domain objects, in this case + it only writes them to database. In this example we are using a simple fixed length record structure that can be found in the project at @@ -214,109 +231,7 @@ xmlStaxJob | | | x | | | | | | x | | | | | | | | object -* Multiline Order Job - - The goal is to demostrate how to handle a more complex file input - format, where a record meant for processing inludes nested records - and spans multiple lines - - multilineOrderJob.xml - - file with multiline records. OrderDataProvider is - an example of a non-default programmatic data provider. It reads - input until it detects that the multiline record has finished and - encapsulates the record in a single domain object. - - file with multiline records. The concrete - <<>> passes the object to a an injected 'delegate - writer' which in this case writes the output to a file. The writer - in this case demonstrates how to write multiline output using a - custom aggregator transformer. - -* Quartz Sample - - The goal is to demonstrate how to schedule job execution using - Quartz scheduler. In this case there is no unit test to launch the - sample because it just re-uses the football job. There is a main - method in <<>> and an Eclipse launch - configuration which runs it with empty arguments. The main method - is very basic - it is intended only as a guide to how Quartz might - be used in principle. - - <<>>, also re-uses - <<>> - - The configuration declares a <<>> bean. The launcher - bean is different from the other samples only in that it uses an - asynchronous task executor, so that the jobs are launched in a - separate thread to the main method: - -+--- - - - - - - -+--- - - Also, a Quartz <<>> is defined using a Spring - <<>> as a convenience. - -+-- - - - - - - - - -+-- - - Finally, a trigger with a scheduler is defined that will launch the - job detail every 10 seconds: - -+--- - - - - - - - - -+--- - - The job is thus scheduled to run every 10 seconds. In fact it - should be successful on the first attempt, so the second and - subsequent attempts should through a - <<>>. In a production system, - the job detail would probably be modified to account for this - exception (e.g. catch it and re-submit with a new set of job - parameters). The point here is that Spring Batch guarantees that - the job execution is idempotent - you can never inadvertently - process the same data twice. - -* Trade Job - - The goal is to show a reasonably complex scenario, that would - resemble the real-life usage of the framework. - - This job has 3 steps. First, data about trades are - imported from a file to database. Second, the trades are read from - the database and credit on customer accounts is decreased - appropriately. Last, a report about customers is exported to a file. - - <<>> - the job definition, - <<>> - input and output configuration - - This job has 3 steps. First, data about trades is - imported from a file to database. Second, the data about trades is - read from the database and credit on customer accounts is decreased - appropriately. Last, a report about customers is exported to a file. - -* Football Job +* Football Job ({football}) This is a (American) Football statistics loading job. We gave it the id of <<>> in our configuration file. Before diving @@ -363,10 +278,10 @@ AbduKa00,1996,mia,15,nyg,0,0,0,0,0,17,96,,14,0 receptions, rushes, and total touchdowns. Our example batch job is going to load both files into a database, - and then combine each to summarize how each player performed for a + and then combine each to summarise how each player performed for a particular year. Although this example is fairly trivial, it shows multiple types of input, and the general style is a common batch - scenario. That is, summarizing a very large dataset so that it can + scenario. That is, summarising a very large dataset so that it can be more easily manipulated or viewed by an online web-based application. In an enterprise solution the third step, the reporting step, could be implemented through the use of Eclipse BIRT or one of @@ -425,10 +340,10 @@ AbduKa00,1996,mia,15,nyg,0,0,0,0,0,17,96,,14,0 ref="playerFileItemReader" /> + class="org.springframework.batch.sample.domain.football.internal.internal.PlayerItemWriter"> + class="org.springframework.batch.sample.domain.football.internal.internal.JdbcPlayerDao"> @@ -448,7 +363,7 @@ AbduKa00,1996,mia,15,nyg,0,0,0,0,0,17,96,,14,0 the developer can remain solely concerned with their business logic. - * – the item reader is the source of the information + * – the item reader is the source of the information pipe. At the most basic level input is read in from an input source, parsed into a domain object and returned. In this way, the good batch architecture practice of ensuring all data has been @@ -486,7 +401,7 @@ AbduKa00,1996,mia,15,nyg,0,0,0,0,0,17,96,,14,0 Following the general flow of the batch job, the next step is to describe how each line of the file will be parsed from its string representation into a domain object. The first thing the provider - will need is an InputSource, which is provided as part of the Spring + will need is an <<>>, which is provided as part of the Spring Batch infrastructure. Because the input is flat-file based, a <<>> is used: @@ -504,7 +419,7 @@ AbduKa00,1996,mia,15,nyg,0,0,0,0,0,17,96,,14,0 + class="org.springframework.batch.sample.domain.football.internal.internal.PlayerFieldSetMapper" /> +--- @@ -525,7 +440,7 @@ AbduKa00,1996,mia,15,nyg,0,0,0,0,0,17,96,,14,0 <<>> interface. There is a single method, <<>>, which maps <<
>>s the same way that developers are comfortable mapping <<>>s into Java - <<>>s, either by index or fieldname. This behavior is by + <<>>s, either by index or field name. This behaviour is by intention and design similar to the <<>> passed into a <<>>. You can see this below: @@ -581,7 +496,7 @@ public class + class="org.springframework.batch.sample.domain.football.internal.internal.PlayerSummaryMapper" /> @@ -596,7 +511,7 @@ games.player_id group by games.player_id, games.year_no +---- - The SqlCursorInputSource has three dependences: + The <<>> has three dependences: * A <<>> @@ -613,5 +528,375 @@ games.player_id group by games.player_id, games.year_no sample application you can execute the JUnit test <<>>, and you'll see an output showing each of the records as they are processed. Please keep in mind that - AoP is used to wrap the <<> and output each record as it + AoP is used to wrap the <<>> and output each record as it is processed to the logger, which may impact performance. + +* Hibernate {hibernate} + + The purpose of this sample is to show a typical usage of Hibernate + as an ORM tool in the input and output of a job. + + The job uses a <<>> for the input, where + a simple HQL query is used to supply items. It also uses a + non-framework <<>> wrapping a DAO, which perhaps was + written as part of an online system. + + + The output reliability and robustness are improved by the use of the + <<>> from the framework. One of its roles + is to buffer items and flush them explicitly, rather than implicitly + on a transaction boundary (which would be the default). This + "write-behind" behaviour is provided by Hibernate implicitly, but we + need to take control of it so that the skip and retry features + provided by Spring Batch can work effectively. Thus the other role + of the <<>> is to watch out for failures + and flush aggressively when an item is seen from a previously failed + chunk. In this way there will always be a failure at some point + immediately after the bad item was written, and the item is then + easily identifiable. + +* Ibatis ({ibatis}) + + The goal of this sample is to show the use of Ibatis as a query + mapping tool. Its features are similar to the Hibernate sample, but + it uses Ibatis to drive its input and output. + +* Multiline ({multiline}) + + The goal of this sample is to show some common tricks with multiline + records in file input jobs. + + The input file in this case consists of two groups of trades + delimited by special lines in a file (BEGIN and END): + ++--- +BEGIN +UK21341EAH4597898.34customer1 +UK21341EAH4611218.12customer2 +END +BEGIN +UK21341EAH4724512.78customer2 +UK21341EAH4810809.25customer3 +UK21341EAH4985423.39customer4 +END ++--- + + The goal of the job is to operate on the two groups, so the item + type is naturally <<>>>. To get these items delivered + from an item reader we employ two components from Spring Batch: the + <<>> and the + <<>>. The latter is + responsible for recognising the difference between the trade data + and the delimiter records. The former is responsible for + aggregating the trades from each group into a <<>> and handing + out the list from its <<>> method. To help these components + perform their responsibilities we also provide some business + knowledge about the data in the form of a <<>> + (<<>>). The <<>> checks + its input for the delimiter fields (BEGIN, END) and if it detects + them, returns the special tokens that <<>> + needs. Otherwise it maps the input into a <<>> object. + +* Multiline Order Job ({multilineOrder}) + + The goal is to demonstrate how to handle a more complex file input + format, where a record meant for processing includes nested records + and spans multiple lines + + The input source is file with multiline records. + <<>> is an example of a non-default programmatic + item reader. It reads input until it detects that the multiline + record has finished and encapsulates the record in a single domain + object. + + The output target is a file with multiline records. The concrete + <<>> passes the object to a an injected 'delegate + writer' which in this case writes the output to a file. The writer + in this case demonstrates how to write multiline output using a + custom aggregator transformer. + +* Parallel Sample ({parallel}) + + The purpose of this sample is to show multi-threaded step execution + using the Process Indicator pattern. + + The job reads data from the same file as the + {{{fixedLengthImport}Fixed Length Import}} sample, but instead of + writing it out directly it goes through a staging table, and the + staging table is read in a multi-threaded step. Note that for such + a simple example where the item processing was not expensive, there + is unlikely to be much if any benefit in using a multi-threaded + step. + + Multi-threaded step execution is easy to configure using Spring + Batch, but there are some limitations. Most of the out-of-the-box + <<>> and <<>> implementations are not + designed to work in this scenario because they need to be + restartable and they are also stateful. There should be no surprise + about this, and reading a file (for instance) is usually fast enough + that multi-threading that part of the process is not likely to + provide much benefit, compared to the cost of managing the state. + + The best strategy to cope with restart state from multiple + concurrent threads depends on the kind of input source involved: + + * For file-based input (and output) restart sate is practically + impossible to manage. Spring Batch does not provide any features + or samples to help with this use case. + + * With message middleware input it is trivial to manage restarts, + since there is no state to store (if a transaction rolls back the + messages are returned to the destination they came from). + + * With database input state management is still necessary, but it + isn't particularly difficult. The easiest thing to do is rely on + a Process Indicator in the input data, which is a column in the + data indicating for each row if it has been processed or not. The + flag is updated inside the batch transaction, and then in the case + of a failure the updates are lost, and the records will show as + un-processed on a restart. + + This last strategy is implemented in the <<>>. + Its companion, the <<>> is responsible for + setting up the data in a staging table which contains the process + indicator. The reader is then driven by a simple SQL query that + includes a where clause for the processed flag, i.e. + ++--- +SELECT ID FROM BATCH_STAGING WHERE JOB_ID=? AND PROCESSED=? ORDER BY ID ++--- + + It is then responsible for updating the processed flag (which + happens inside the main step transaction). + +* Quartz Sample ({quartz}) + + The goal is to demonstrate how to schedule job execution using + Quartz scheduler. In this case there is no unit test to launch the + sample because it just re-uses the football job. There is a main + method in <<>> and an Eclipse launch + configuration which runs it with arguments to pick up the football + job. + + The additional XML configuration for this job is in + <<>>, and it also re-uses + <<>> + + The configuration declares a <<>> bean. The launcher + bean is different from the other samples only in that it uses an + asynchronous task executor, so that the jobs are launched in a + separate thread to the main method: + ++--- + + + + + + ++--- + + Also, a Quartz <<>> is defined using a Spring + <<>> as a convenience. + ++-- + + + + + + + + + + + ++-- + + Finally, a trigger with a scheduler is defined that will launch the + job detail every 10 seconds: + ++--- + + + + + + + + ++--- + + The job is thus scheduled to run every 10 seconds. In fact it + should be successful on the first attempt, so the second and + subsequent attempts should through a + <<>>. In a production system, + the job detail would probably be modified to account for this + exception (e.g. catch it and re-submit with a new set of job + parameters). The point here is that Spring Batch guarantees that + the job execution is idempotent - you can never inadvertently + process the same data twice. + +* Restart Sample ({restartSample}) + + The goal of this sample is to show how a job can be restarted after + a failure and continue processing where it left off. + + To simulate a failure we "fake" a failure on the fourth record + though the use of a sample component + <<>>. This is a stateful reader + that counts how many records it has processed and throws a planned + exception in a specified place. Since we re-use the same instance + when we restart the job it will not fail the second time. + +* Retry Sample ({retrySample}) + + The purpose of this sample is to show how to use the automatic retry + capabilities of Spring Batch. + + The retry is configured in the step through the + <<>>: + ++--- + + ... + + + ++--- + + Failed items will cause a rollback for all <<>> types, up + to a limit of 3 attempts. On the 4th attempt, the failed item would + be skipped, and there would be a callback to a + <<>> if one was provided (via the "listeners" + property of the step factory bean). + + An <<>> is provided that will generate unique + <<>> data by just incrementing a counter. Note that it uses + the counter in its <<>> and <<>> methods so that + the same content is returned after a rollback. The same content is + returned, but the instance of <<>> is different, which means + that the implementation of <<>> in the <<>> object + is important. This is because to identify a failed item on retry + (so that the number of attempts can be counted) the framework by + default uses <<>> to compare the recently failed + item with a cache of previously failed items. Without implementing + a field-based <<>> method for the domain object, our job + will spin round the retry for potentially quite a long time before + failing because the default implementation of <<>> is + based on object reference, not on field content. + +* Skip Sample ({skipSample}) + + The purpose of this sample is to show how to use the skip features + of Spring Batch. Since skip is really just a special case of retry + (with limit 0), the details are quite similar to the {{{retrySample}Retry + Sample}}, but the use case is less artificial, since it + is based on the {{{trade}Trade Sample}}. + + The failure condition is still artificial, since it is triggered by + a special <<>> wrapper (<<>>). + The plan is that a certain item (the third) will fail business + validation in the writer, and the system can then respond by + skipping it. We also configure the step so that it will not roll + back on the validation exception, since we know that it didn't + invalidate the transaction, only the item. This is done through the + transaction attribute: + ++--- + + + + + .... + ++--- + + The format for the transaction attribute specification is given in + the Spring Core documentation (e.g. see the Javadocs for + {{{http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/transaction/interceptor/TransactionAttributeEditor.html}TransactionAttributeEditor}}). + +* Tasklet Job ({tasklet}) + + The goal is to show the simplest use of the batch framework with a + single job with a single step, which cleans up a directory and runs + a system command. + + The + <<>> itself is defined by the bean definition with + <<>>. In this example we have two steps. + + * The first step defines a tasklet that is responsible for + clearing out a directory though a custom <<>>. Each + tasklet has an <<>> method which is called by the + step. All processing of business data should be handled by this + method. + + * The second step uses another tasklet to execute a system (OS) + command line. + + You can visualise the Spring configuration of a job through + Spring-IDE. See {{{http://springide.org/blog/}Spring IDE}}. The + source view of the configuration is as follows: + ++--- + + + + + + + + + + + + + + + + + + + + + + + + + + + ++--- + + For simplicity we are only displaying the job configuration itself + and leaving out the details of the supporting batch execution + environment configuration. + +* Trade Job ({trade}) + + The goal is to show a reasonably complex scenario, that would + resemble the real-life usage of the framework. + + This job has 3 steps. First, data about trades are imported from a + file to database. Second, the trades are read from the database and + credit on customer accounts is decreased appropriately. Last, a + report about customers is exported to a file. + +* XML Input Output ({xmlStax}) + + The goal here is to show the use of XML input and output through + streaming and Spring OXM marshallers and unmarshallers. + + The job has a single step that copies <<>> data from one XML + file to another. It uses XStream for the object XML conversion, + because this is simple to configure for basic use cases like this + one. See + {{{http://static.springframework.org/spring-ws/sites/1.5/reference/html/oxm.html}Spring + OXM documentation}} for details of other options.