Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-3148: Add MetadataStore Chapter

Browse Source 
JIRA: https://jira.springsource.org/browse/INT-3148

* Add a general chapter about `MetadataStore`
* Add cross-links between chapters on the matter of `MetadataStore`

Doc Polishing
This commit is contained in:
Artem Bilan
2013-12-12 13:56:09 +02:00
committed by Gary Russell
parent ef756bfd6d
commit 66f714e33a
8 changed files with 109 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
src/reference/docbook/feed.xml
View File

@@ -62,55 +62,12 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
Spring Integration provides a convenient mechanism to eliminate the need to worry about duplicate entries.
Each feed entry will have a <emphasis>published date</emphasis> field. Every time a new Message is generated and sent,
Spring Integration will store the value of the latest <emphasis>published date</emphasis> in an instance of the
<classname>org.springframework.integration.metadata.MetadataStore</classname> strategy. The MetadataStore interface is
designed to store various types of generic meta-data (e.g., published date of the last feed entry that has been processed)
to help components such as this Feed adapter deal with duplicates.
<interfacename>MetadataStore</interfacename> strategy (<xref linkend="metadata-store"/>).
</para>
<para>
The default rule for locating this metadata store is as follows:
<emphasis>Spring Integration</emphasis> will look for a bean of type
<classname>org.springframework.integration.metadata.MetadataStore</classname> in
the ApplicationContext. If one is found then it will be used, otherwise
it will create a new instance of <classname>SimpleMetadataStore</classname>
which is an in-memory implementation that will only persist metadata within
the lifecycle of the currently running Application Context. This means
that upon restart you may end up with duplicate entries.
</para>
<para>
If you need to persist metadata between Application Context restarts, two
persistent <interfacename>MetadataStores</interfacename> are available:
</para>
<itemizedlist>
<listitem>PropertiesPersistingMetadataStore</listitem>
<listitem>RedisMetadataStore</listitem>
</itemizedlist>
<para>
The <classname>PropertiesPersistingMetadataStore</classname> is backed by
a properties file and a
<interfacename><ulink url="http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/util/PropertiesPersister.html">PropertiesPersister</ulink></interfacename>.
</para>
<programlisting language="xml"><![CDATA[<bean id="metadataStore"
class="org.springframework.integration.store.PropertiesPersistingMetadataStore"/>]]></programlisting>
<para>
As of <emphasis>Spring Integration 3.0</emphasis> a Redis-based
<interfacename>MetadataStore</interfacename> is also available. For
more information regarding the <classname>RedisMetadataStore</classname>
see <xref linkend="redis-metadata-store" />.
</para>
<warning>
Be careful when using the same Redis instancce across multiple application
contexts as separate Feed adapters may accidentally use the same persisted
key.
</warning>
<para>
Alternatively, you could provide your own implementation of the
<interfacename>MetadataStore</interfacename> interface (e.g. JdbcMetadataStore)
and configure it as bean in the Application Context.
</para>
<note>
The key used to persist the latest <emphasis>published date</emphasis> is the value of the (required)
<code>id</code> attribute of the Feed Inbound Channel Adapter component plus the <code>feedUrl</code>
from the adapter's configuration.
</note>
<note>
The key used to persist the latest <emphasis>published date</emphasis> is the value of the (required)
<code>id</code> attribute of the Feed Inbound Channel Adapter component plus the <code>feedUrl</code>
from the adapter's configuration.
</note>
</section>
</chapter>

5
src/reference/docbook/file.xml
View File

@@ -39,9 +39,8 @@
The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
state to survive a system restart, consider using the
<classname>FileSystemPersistentAcceptOnceFileListFilter</classname> instead. This filter stores
the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
several store implementations (such as Redis), or you can provide your own. This filter matches on
the filename and modified time.
the accepted file names in a <interfacename>MetadataStore</interfacename> strategy (<xref linkend="metadata-store"/>).
This filter matches on the filename and modified time.
</note>
<programlisting language="xml"><![CDATA[<bean id="pollableFileSource"
class="org.springframework.integration.file.FileReadingMessageSource"

12
src/reference/docbook/ftp.xml
View File

@@ -192,10 +192,10 @@ protected void postProcessClientBeforeConnect(T client) throws IOException {
The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
state to survive a system restart, consider using the
<classname>FtpPersistentAcceptOnceFileListFilter</classname> instead. This filter stores
the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
several store implementations (such as Redis), or you can provide your own. This filter matches on
the filename and the remote modified time.
</para>
the accepted file names in an instance of the
<interfacename>MetadataStore</interfacename> strategy (<xref linkend="metadata-store"/>).
This filter matches on the filename and the remote modified time.
</para>
<note>
<para>
Beginning with <emphasis>version 3.0</emphasis>, you can also specify a filter used to filter the files locally, once they have
@@ -208,8 +208,8 @@ protected void postProcessClientBeforeConnect(T client) throws IOException {
The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
state to survive a system restart, consider using the
<classname>FileSystemPersistentAcceptOnceFileListFilter</classname> as a local filter instead. This filter stores
the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
several store implementations (such as Redis), or you can provide your own.
the accepted file names in an instance of the
<interfacename>MetadataStore</interfacename> strategy (<xref linkend="metadata-store"/>).
<important>
This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a
re-synchronized file from being processed, you should use the <code>preserve-timestamp</code> attribute discussed above.

68
src/reference/docbook/meta-data-store.xml Normal file
View File

@@ -0,0 +1,68 @@
<?xml version="1.0" encoding="UTF-8"?>
<section xmlns="http://docbook.org/ns/docbook" version="5.0" xml:id="metadata-store"
xmlns:xlink="http://www.w3.org/1999/xlink">
<title>Metadata Store</title>
<para>
Many external systems, services or resources aren't transactional (Twitter, RSS, file system etc.)
and there is no any ability to mark the data as read. Or there is just need to implement the
Enterprise Integration Pattern <ulink url="http://eaipatterns.com/IdempotentReceiver.html">Idempotent Receiver</ulink>
in some integration solutions. To achieve this goal and store some previous state of the Endpoint before the next
interaction with external system, or deal with the next Message, Spring Integration provides the <emphasis>Metadata Store</emphasis>
component being an implementation of the <interfacename>org.springframework.integration.metadata.MetadataStore</interfacename>
interface with a general <emphasis>key-value</emphasis> contract.
</para>
<para>
The <emphasis>Metadata Store</emphasis> is designed to store various types of generic meta-data
(e.g., published date of the last feed entry that has been processed) to help components such as the Feed adapter deal with duplicates.
If a component is not directly provided with a reference to a <interfacename>MetadataStore</interfacename>,
the algorithm for locating a metadata store is as follows: First, look for a bean with id
<code>metadataStore</code> in the ApplicationContext. If one is found then it will be used, otherwise
it will create a new instance of <classname>SimpleMetadataStore</classname> which is an in-memory implementation
that will only persist metadata within the lifecycle of the currently running Application Context. This means
that upon restart you may end up with duplicate entries.
</para>
<para>
If you need to persist metadata between Application Context restarts, two
persistent <interfacename>MetadataStores</interfacename> are provided by the framework:
</para>
<itemizedlist>
<listitem>PropertiesPersistingMetadataStore</listitem>
<listitem><xref linkend="redis-metadata-store"/></listitem>
</itemizedlist>
<para>
The <classname>PropertiesPersistingMetadataStore</classname> is backed by a properties file and a
<interfacename><ulink url="http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/util/PropertiesPersister.html">PropertiesPersister</ulink></interfacename>.
</para>
<programlisting language="xml"><![CDATA[<bean id="metadataStore"
class="org.springframework.integration.store.PropertiesPersistingMetadataStore"/>]]></programlisting>
<para>
Alternatively, you can provide your own implementation of the
<interfacename>MetadataStore</interfacename> interface (e.g. JdbcMetadataStore)
and configure it as a bean in the Application Context.
</para>
<section id="idempotent-receiver">
<title>Idempotent Receiver</title>
<para>
The <emphasis>Metadata Store</emphasis> is useful for implementating the
EIP <ulink url="http://eaipatterns.com/IdempotentReceiver.html">Idempotent Receiver</ulink> pattern, when
there is need to <emphasis>filter</emphasis> an incoming Message if it has already been processed, and just discard
it or perform some other logic on discarding. The following configuration is an example of how to do this:
</para>
<programlisting language="xml"><![CDATA[<int:filter input-channel="serviceChannel"
output-channel="idempotentServiceChannel"
discard-channel="discardChannel"
expression="@metadataStore.get(headers.businessKey) == null"/>
<int:publish-subscribe-channel id="idempotentServiceChannel"/>
<int:outbound-channel-adapter channel="idempotentServiceChannel"
expression="@metadataStore.put(headers.businessKey, '')"/>
<int:service-activator input-channel="idempotentServiceChannel" ref="service"/>]]></programlisting>
<para>
The <code>value</code> of the idempotent entry may be some expiration date, after which that entry should
be removed from <emphasis>Metadata Store</emphasis> by some scheduled reaper.
</para>
</section>
</section>

26
src/reference/docbook/redis.xml
View File

@@ -424,15 +424,18 @@ rt.setConnectionFactory(redisConnectionFactory);]]></programlisting>
<title>Redis Metadata Store</title>
<para>
As of <emphasis>Spring Integration 3.0</emphasis> a new Redis-based
<interfacename><ulink url="http://docs.spring.io/spring-integration/docs/latest-ga/api/org/springframework/integration/store/MetadataStore.html">MetadataStore</ulink></interfacename>
implementation is available. The <classname>RedisMetadataStore</classname> can
<interfacename><ulink url="http://docs.spring.io/spring-integration/docs/latest-ga/api/org/springframework/integration/metadata/MetadataStore.html">MetadataStore</ulink></interfacename>
(<xref linkend="metadata-store"/>) implementation is available. The <classname>RedisMetadataStore</classname> can
be used to maintain state of a <interfacename>MetadataStore</interfacename>
across application restarts. This new <interfacename>MetadataStore</interfacename>
implementation can be used with adapters such as:
</para>
<itemizedlist>
<listitem>Twitter Inbound Adapters</listitem>
<listitem>Feed Inbound Channel Adapter</listitem>
<listitem><xref linkend="twitter-inbound"/></listitem>
<listitem><xref linkend="feed-inbound-channel-adapter"/></listitem>
<listitem><xref linkend="file-reading"/></listitem>
<listitem><xref linkend="ftp-inbound"/></listitem>
<listitem><xref linkend="sftp-inbound"/></listitem>
</itemizedlist>
<para>
In order to instruct these adapters to use the new <classname>RedisMetadataStore</classname>
@@ -444,11 +447,16 @@ rt.setConnectionFactory(redisConnectionFactory);]]></programlisting>
<programlisting language="xml"><![CDATA[<bean name="metadataStore" class="o.s.i.redis.store.metadata.RedisMetadataStore">
<constructor-arg name="connectionFactory" ref="redisConnectionFactory"/>
</bean>]]></programlisting>
<warning>
Be careful when using the same Redis instancce across multiple application
contexts as separate adapters may accidentally use the same persisted
key.
</warning>
<para>
The <classname>RedisMetadataStore</classname> is backed by
<ulink url="http://docs.spring.io/spring-data/data-redis/docs/current/api/org/springframework/data/redis/support/collections/RedisProperties.html"
><classname>RedisProperties</classname></ulink> and interaction with it uses
<ulink url="http://docs.spring.io/spring-data/data-redis/docs/current/api/org/springframework/data/redis/core/BoundHashOperations.html"
><classname>BoundHashOperations</classname></ulink>, which, in turn, requires a <code>key</code> for the entire
<classname>Properties</classname> store. In the case of the <interfacename>MetadataStore</interfacename>, this
<code>key</code> plays the role of a <emphasis>region</emphasis>, which is useful in distributed environment,
when several applications use the same Redis server. By default this <code>key</code> has the value <code>MetaData</code>.
</para>
</section>
<section id="redis-store-inbound-channel-adapter">
<title>RedisStore Inbound Channel Adapter</title>

12
src/reference/docbook/sftp.xml
View File

@@ -317,10 +317,10 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/sftp
The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
state to survive a system restart, consider using the
<classname>SftpPersistentAcceptOnceFileListFilter</classname> instead. This filter stores
the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
several store implementations (such as Redis), or you can provide your own. This filter matches on
the filename and the remote modified time.
</para>
the accepted file names in an instance of the
<interfacename>MetadataStore</interfacename> strategy (<xref linkend="metadata-store"/>).
This filter matches on the filename and the remote modified time.
</para>
<note>
<para>
Beginning with <emphasis>version 3.0</emphasis>, you can also specify a filter used to filter the files locally, once they have
@@ -333,8 +333,8 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/sftp
The <classname>AcceptOnceFileListFilter</classname> stores its state in memory. If you wish the
state to survive a system restart, consider using the
<classname>FileSystemPersistentAcceptOnceFileListFilter</classname> as a local filter instead. This filter stores
the accepted file names in a <interfacename>MetadataStore</interfacename>. The framework supplies
several store implementations (such as Redis), or you can provide your own.
the accepted file names in an instance of the
<interfacename>MetadataStore</interfacename> strategy (<xref linkend="metadata-store"/>).
<important>
This filter compares the filename and modified timestamp. If you wish to use this technique to avoid a
re-synchronized file from being processed, you should use the <code>preserve-timestamp</code> attribute discussed above.

3
src/reference/docbook/system-management.xml
View File

@@ -3,10 +3,11 @@
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:xlink="http://www.w3.org/1999/xlink">
<title>System Management</title>
<xi:include href="./jmx.xml"/>
<xi:include href="./message-history.xml"/>
<xi:include href="./message-store.xml"/>
<xi:include href="./meta-data-store.xml"/>
<xi:include href="./control-bus.xml"/>
<xi:include href="./shutdown.xml"/>

30
src/reference/docbook/twitter.xml
View File

@@ -148,34 +148,8 @@ twitter.oauth.accessTokenSecret=AbRxUAvyNCtqQtxFK8w5ZMtMj20KFhB6o]]></programlis
criteria you'll end up with the same set of tweets unless some other new tweet that matches your search criteria was posted
in between your searches. In that situation you'll get all the tweets you had before plus the new one. But what you really
want is only the new tweet(s). Spring Integration provides an elegant mechanism for handling these situations.
The latest Tweet id will be stored in an instance of the <classname>org.springframework.integration.metadata.MetadataStore</classname> which is a
strategy interface designed for storing various types of metadata (e.g., last retrieved tweet in this case). That strategy helps components such as
these Twitter adapters avoid duplicates. By default, Spring Integration will look for a bean of type
<classname>org.springframework.integration.metadata.MetadataStore</classname> in the ApplicationContext. Alternatively,
you can configure an explicit <classname>MetadataStore</classname> on the adapter.
If there is no explicit or default store, the adapter will create a new instance of <classname>SimpleMetadataStore</classname>
which is a simple in-memory implementation that will only persist metadata within the lifecycle of the currently running application context.
That means upon restart you may end up with duplicate entries. If you need to persist metadata between Application Context
restarts, you may use the <classname>PropertiesPersistingMetadataStore</classname> (which is backed by a properties file, and a persister
strategy), or you may create your own custom implementation of the <classname>MetadataStore</classname> interface (e.g., JdbcMetadatStore)
and configure it as a bean named 'metadataStore' within the Application Context.
</para>
<para>
As of <emphasis>Spring Integration 3.0</emphasis> a Redis-based
<interfacename>MetadataStore</interfacename> is available. The
<classname>RedisMetadataStore</classname> allows you to maintain persisted
metadata across Application Context restarts. For more information see <xref linkend="redis-metadata-store" />.
</para>
<warning>
Be careful when using the same Redis instance across multiple application
contexts as separate Twitter adapters may accidentally use the same persisted
key.
</warning>
<programlisting language="xml"><![CDATA[<bean id="metadataStore" class="o.s.i.store.PropertiesPersistingMetadataStore"/>
]]></programlisting>
<para>
If the <classname>MetadataStore</classname> is persistent, during initialization, any Inbound Twitter Adapter (see below)
will retrieve the latest tweet id that has already been sent by the adapter.
The latest Tweet id will be stored in an instance of the <classname>org.springframework.integration.metadata.MetadataStore</classname>
strategy (e.g. last retrieved tweet in this case). For more information see <xref linkend="metadata-store"/>.
</para>
<note>
The key used to persist the latest <emphasis>twitter id</emphasis> is the value of the (required)