diff --git a/src/reference/docbook/feed.xml b/src/reference/docbook/feed.xml index cb02137b39..a2944a3a8b 100644 --- a/src/reference/docbook/feed.xml +++ b/src/reference/docbook/feed.xml @@ -62,55 +62,12 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/feed Spring Integration provides a convenient mechanism to eliminate the need to worry about duplicate entries. Each feed entry will have a published date field. Every time a new Message is generated and sent, Spring Integration will store the value of the latest published date in an instance of the - org.springframework.integration.metadata.MetadataStore strategy. The MetadataStore interface is - designed to store various types of generic meta-data (e.g., published date of the last feed entry that has been processed) - to help components such as this Feed adapter deal with duplicates. + MetadataStore strategy (). - - The default rule for locating this metadata store is as follows: - Spring Integration will look for a bean of type - org.springframework.integration.metadata.MetadataStore in - the ApplicationContext. If one is found then it will be used, otherwise - it will create a new instance of SimpleMetadataStore - which is an in-memory implementation that will only persist metadata within - the lifecycle of the currently running Application Context. This means - that upon restart you may end up with duplicate entries. - - - If you need to persist metadata between Application Context restarts, two - persistent MetadataStores are available: - - - PropertiesPersistingMetadataStore - RedisMetadataStore - - - The PropertiesPersistingMetadataStore is backed by - a properties file and a - PropertiesPersister. - - ]]> - - As of Spring Integration 3.0 a Redis-based - MetadataStore is also available. For - more information regarding the RedisMetadataStore - see . - - - Be careful when using the same Redis instancce across multiple application - contexts as separate Feed adapters may accidentally use the same persisted - key. - - - Alternatively, you could provide your own implementation of the - MetadataStore interface (e.g. JdbcMetadataStore) - and configure it as bean in the Application Context. - - - The key used to persist the latest published date is the value of the (required) - id attribute of the Feed Inbound Channel Adapter component plus the feedUrl - from the adapter's configuration. - + + The key used to persist the latest published date is the value of the (required) + id attribute of the Feed Inbound Channel Adapter component plus the feedUrl + from the adapter's configuration. + diff --git a/src/reference/docbook/file.xml b/src/reference/docbook/file.xml index 515d3f0a00..3c18d47378 100644 --- a/src/reference/docbook/file.xml +++ b/src/reference/docbook/file.xml @@ -39,9 +39,8 @@ The AcceptOnceFileListFilter stores its state in memory. If you wish the state to survive a system restart, consider using the FileSystemPersistentAcceptOnceFileListFilter instead. This filter stores - the accepted file names in a MetadataStore. The framework supplies - several store implementations (such as Redis), or you can provide your own. This filter matches on - the filename and modified time. + the accepted file names in a MetadataStore strategy (). + This filter matches on the filename and modified time. AcceptOnceFileListFilter stores its state in memory. If you wish the state to survive a system restart, consider using the FtpPersistentAcceptOnceFileListFilter instead. This filter stores - the accepted file names in a MetadataStore. The framework supplies - several store implementations (such as Redis), or you can provide your own. This filter matches on - the filename and the remote modified time. - + the accepted file names in an instance of the + MetadataStore strategy (). + This filter matches on the filename and the remote modified time. + Beginning with version 3.0, you can also specify a filter used to filter the files locally, once they have @@ -208,8 +208,8 @@ protected void postProcessClientBeforeConnect(T client) throws IOException { The AcceptOnceFileListFilter stores its state in memory. If you wish the state to survive a system restart, consider using the FileSystemPersistentAcceptOnceFileListFilter as a local filter instead. This filter stores - the accepted file names in a MetadataStore. The framework supplies - several store implementations (such as Redis), or you can provide your own. + the accepted file names in an instance of the + MetadataStore strategy (). This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a re-synchronized file from being processed, you should use the preserve-timestamp attribute discussed above. diff --git a/src/reference/docbook/meta-data-store.xml b/src/reference/docbook/meta-data-store.xml new file mode 100644 index 0000000000..f242f3c0da --- /dev/null +++ b/src/reference/docbook/meta-data-store.xml @@ -0,0 +1,68 @@ + +
+ Metadata Store + + Many external systems, services or resources aren't transactional (Twitter, RSS, file system etc.) + and there is no any ability to mark the data as read. Or there is just need to implement the + Enterprise Integration Pattern Idempotent Receiver + in some integration solutions. To achieve this goal and store some previous state of the Endpoint before the next + interaction with external system, or deal with the next Message, Spring Integration provides the Metadata Store + component being an implementation of the org.springframework.integration.metadata.MetadataStore + interface with a general key-value contract. + + + The Metadata Store is designed to store various types of generic meta-data + (e.g., published date of the last feed entry that has been processed) to help components such as the Feed adapter deal with duplicates. + If a component is not directly provided with a reference to a MetadataStore, + the algorithm for locating a metadata store is as follows: First, look for a bean with id + metadataStore in the ApplicationContext. If one is found then it will be used, otherwise + it will create a new instance of SimpleMetadataStore which is an in-memory implementation + that will only persist metadata within the lifecycle of the currently running Application Context. This means + that upon restart you may end up with duplicate entries. + + + If you need to persist metadata between Application Context restarts, two + persistent MetadataStores are provided by the framework: + + + PropertiesPersistingMetadataStore + + + + The PropertiesPersistingMetadataStore is backed by a properties file and a + PropertiesPersister. + + ]]> + + Alternatively, you can provide your own implementation of the + MetadataStore interface (e.g. JdbcMetadataStore) + and configure it as a bean in the Application Context. + +
+ Idempotent Receiver + + The Metadata Store is useful for implementating the + EIP Idempotent Receiver pattern, when + there is need to filter an incoming Message if it has already been processed, and just discard + it or perform some other logic on discarding. The following configuration is an example of how to do this: + + + + + + + +]]> + + The value of the idempotent entry may be some expiration date, after which that entry should + be removed from Metadata Store by some scheduled reaper. + +
+ +
diff --git a/src/reference/docbook/redis.xml b/src/reference/docbook/redis.xml index e3c71c2be7..4fa5ea345c 100644 --- a/src/reference/docbook/redis.xml +++ b/src/reference/docbook/redis.xml @@ -424,15 +424,18 @@ rt.setConnectionFactory(redisConnectionFactory);]]> Redis Metadata Store As of Spring Integration 3.0 a new Redis-based - MetadataStore - implementation is available. The RedisMetadataStore can + MetadataStore + () implementation is available. The RedisMetadataStore can be used to maintain state of a MetadataStore across application restarts. This new MetadataStore implementation can be used with adapters such as: - Twitter Inbound Adapters - Feed Inbound Channel Adapter + + + + + In order to instruct these adapters to use the new RedisMetadataStore @@ -444,11 +447,16 @@ rt.setConnectionFactory(redisConnectionFactory);]]> ]]> - - Be careful when using the same Redis instancce across multiple application - contexts as separate adapters may accidentally use the same persisted - key. - + + The RedisMetadataStore is backed by + RedisProperties and interaction with it uses + BoundHashOperations, which, in turn, requires a key for the entire + Properties store. In the case of the MetadataStore, this + key plays the role of a region, which is useful in distributed environment, + when several applications use the same Redis server. By default this key has the value MetaData. +
RedisStore Inbound Channel Adapter diff --git a/src/reference/docbook/sftp.xml b/src/reference/docbook/sftp.xml index 7952a3d15e..63fc8aa1d5 100644 --- a/src/reference/docbook/sftp.xml +++ b/src/reference/docbook/sftp.xml @@ -317,10 +317,10 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/sftp The AcceptOnceFileListFilter stores its state in memory. If you wish the state to survive a system restart, consider using the SftpPersistentAcceptOnceFileListFilter instead. This filter stores - the accepted file names in a MetadataStore. The framework supplies - several store implementations (such as Redis), or you can provide your own. This filter matches on - the filename and the remote modified time. - + the accepted file names in an instance of the + MetadataStore strategy (). + This filter matches on the filename and the remote modified time. + Beginning with version 3.0, you can also specify a filter used to filter the files locally, once they have @@ -333,8 +333,8 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/sftp The AcceptOnceFileListFilter stores its state in memory. If you wish the state to survive a system restart, consider using the FileSystemPersistentAcceptOnceFileListFilter as a local filter instead. This filter stores - the accepted file names in a MetadataStore. The framework supplies - several store implementations (such as Redis), or you can provide your own. + the accepted file names in an instance of the + MetadataStore strategy (). This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a re-synchronized file from being processed, you should use the preserve-timestamp attribute discussed above. diff --git a/src/reference/docbook/system-management.xml b/src/reference/docbook/system-management.xml index dacb6a1ea6..5cda05805b 100644 --- a/src/reference/docbook/system-management.xml +++ b/src/reference/docbook/system-management.xml @@ -3,10 +3,11 @@ xmlns:xi="http://www.w3.org/2001/XInclude" xmlns:xlink="http://www.w3.org/1999/xlink"> System Management - + + diff --git a/src/reference/docbook/twitter.xml b/src/reference/docbook/twitter.xml index fe72a3a45f..0555ce44e5 100644 --- a/src/reference/docbook/twitter.xml +++ b/src/reference/docbook/twitter.xml @@ -148,34 +148,8 @@ twitter.oauth.accessTokenSecret=AbRxUAvyNCtqQtxFK8w5ZMtMj20KFhB6o]]>org.springframework.integration.metadata.MetadataStore which is a - strategy interface designed for storing various types of metadata (e.g., last retrieved tweet in this case). That strategy helps components such as - these Twitter adapters avoid duplicates. By default, Spring Integration will look for a bean of type - org.springframework.integration.metadata.MetadataStore in the ApplicationContext. Alternatively, - you can configure an explicit MetadataStore on the adapter. - If there is no explicit or default store, the adapter will create a new instance of SimpleMetadataStore - which is a simple in-memory implementation that will only persist metadata within the lifecycle of the currently running application context. - That means upon restart you may end up with duplicate entries. If you need to persist metadata between Application Context - restarts, you may use the PropertiesPersistingMetadataStore (which is backed by a properties file, and a persister - strategy), or you may create your own custom implementation of the MetadataStore interface (e.g., JdbcMetadatStore) - and configure it as a bean named 'metadataStore' within the Application Context. - - - As of Spring Integration 3.0 a Redis-based - MetadataStore is available. The - RedisMetadataStore allows you to maintain persisted - metadata across Application Context restarts. For more information see . - - - Be careful when using the same Redis instance across multiple application - contexts as separate Twitter adapters may accidentally use the same persisted - key. - - -]]> - - If the MetadataStore is persistent, during initialization, any Inbound Twitter Adapter (see below) - will retrieve the latest tweet id that has already been sent by the adapter. + The latest Tweet id will be stored in an instance of the org.springframework.integration.metadata.MetadataStore + strategy (e.g. last retrieved tweet in this case). For more information see . The key used to persist the latest twitter id is the value of the (required)