Spring/spring-integration
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

INT-3085 Add a Redis-backed MetadataStore

Browse Source 
* Add tests
* Add documentation

INT-3085 Code review changes

INT-3085 Add Twitter Integration Test

JIRA: https://jira.springsource.org/browse/INT-3085
This commit is contained in:
Gunnar Hillert
2013-10-04 11:00:57 +03:00
committed by Artem Bilan
parent bd2cde4202
commit 1fa9b92c0b
10 changed files with 533 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
src/reference/docbook/feed.xml
View File

@@ -3,33 +3,33 @@
xmlns:xlink="http://www.w3.org/1999/xlink">
<title>Feed Adapter</title>
<para>
Spring Integration provides support for Syndication via Feed Adapters
Spring Integration provides support for Syndication via Feed Adapters
</para>
<section id="feed-intro">
<title>Introduction</title>
<para>
Web syndication is a form of publishing material such as news stories, press releases, blog posts, and
Web syndication is a form of publishing material such as news stories, press releases, blog posts, and
other items typically available on a website but also made available in a feed format such as RSS or ATOM.
</para>
<para>
Spring integration provides support for Web Syndication via its 'feed' adapter and provides convenient
namespace-based configuration for it.
Spring integration provides support for Web Syndication via its 'feed' adapter and provides convenient
namespace-based configuration for it.
To configure the 'feed' namespace, include the following elements within the headers of your XML configuration file:
<programlisting language="xml"><![CDATA[xmlns:int-feed="http://www.springframework.org/schema/integration/feed"
xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
http://www.springframework.org/schema/integration/feed/spring-integration-feed.xsd"]]></programlisting>
</para>
</section>
<section>
<section id="feed-inbound-channel-adapter">
<title>Feed Inbound Channel Adapter</title>
<para>
The only adapter that is really needed to provide support for retrieving feeds is an <emphasis>inbound channel adapter</emphasis>.
This allows you to subscribe to a particular URL. Below is an example configuration:
<programlisting language="xml"><![CDATA[<int-feed:inbound-channel-adapter id="feedAdapter"
channel="feedChannel"
<programlisting language="xml"><![CDATA[<int-feed:inbound-channel-adapter id="feedAdapter"
channel="feedChannel"
url="http://feeds.bbci.co.uk/news/rss.xml">
<int:poller fixed-rate="10000" max-messages-per-poll="100" />
</int-feed:inbound-channel-adapter>]]></programlisting>
@@ -38,46 +38,74 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
</para>
<para>
As news items are retrieved they will be converted to Messages and sent to a channel identified by the <code>channel</code> attribute.
The payload of each message will be a <classname>com.sun.syndication.feed.synd.SyndEntry</classname> instance. That encapsulates
The payload of each message will be a <classname>com.sun.syndication.feed.synd.SyndEntry</classname> instance. That encapsulates
various data about a news item (content, dates, authors, etc.).
</para>
<para>
You can also see that the <emphasis>Inbound Feed Channel Adapter</emphasis> is a Polling Consumer. That means you have to
You can also see that the <emphasis>Inbound Feed Channel Adapter</emphasis> is a Polling Consumer. That means you have to
provide a poller configuration. However, one important thing you must understand with regard to Feeds is that its inner-workings
are slightly different then most other poling consumers. When an Inbound Feed adapter is started, it does the first poll and
receives a <classname>com.sun.syndication.feed.synd.SyndEntryFeed</classname> instance. That is an object that contains multiple
<classname>SyndEntry</classname> objects. Each entry is stored in the local entry queue and is released based on
the value in the <code>max-messages-per-poll</code> attribute such that each Message will contain a single entry.
If during retrieval of the entries from the entry queue the queue had become empty, the adapter will attempt to update
are slightly different then most other poling consumers. When an Inbound Feed adapter is started, it does the first poll and
receives a <classname>com.sun.syndication.feed.synd.SyndEntryFeed</classname> instance. That is an object that contains multiple
<classname>SyndEntry</classname> objects. Each entry is stored in the local entry queue and is released based on
the value in the <code>max-messages-per-poll</code> attribute such that each Message will contain a single entry.
If during retrieval of the entries from the entry queue the queue had become empty, the adapter will attempt to update
the Feed thereby populating the queue with more entries (SyndEntry instances) if available. Otherwise the next attempt to
poll for a feed will be determined by the trigger of the poller (e.g., every 10 seconds in the above configuration).
</para>
<para>
<emphasis>Duplicate Entries</emphasis>
</para>
<para>
Polling for a Feed might result in entries that have already been processed
("I already read that news item, why are you showing it to me again?").
("I already read that news item, why are you showing it to me again?").
Spring Integration provides a convenient mechanism to eliminate the need to worry about duplicate entries.
Each feed entry will have a <emphasis>published date</emphasis> field. Every time a new Message is generated and sent,
Each feed entry will have a <emphasis>published date</emphasis> field. Every time a new Message is generated and sent,
Spring Integration will store the value of the latest <emphasis>published date</emphasis> in an instance of the
<classname>org.springframework.integration.store.MetadataStore</classname> strategy. The MetadataStore interface is
designed to store various types of generic meta-data (e.g., published date of the last feed entry that has been processed)
to help components such as this Feed adapter deal with duplicates.
to help components such as this Feed adapter deal with duplicates.
</para>
<para>
The default rule for locating this metadata store is as follows: Spring Integration will look for a bean of type
<classname>org.springframework.integration.store.MetadataStore</classname> in the ApplicationContext. If one is found then it will be used,
otherwise it will create a new instance of <classname>SimpleMetadataStore</classname> which is an in-memory implementation that
will only persist metadata within the lifecycle of the currently running Application Context. This means that upon restart you may
end up with duplicate entries. If you need to persist metadata between Application Context restarts, you may use the
<classname>PropertiesPersistingMetadataStore</classname> which is backed by a properties file and a properties-persister.
Alternatively, you could provide your own implementation of the <classname>MetadataStore</classname> interface
(e.g. JdbcMetadataStore) and configure it as bean in the Application Context.
<programlisting language="xml"><![CDATA[<bean id="metadataStore"
<para>
The default rule for locating this metadata store is as follows:
<emphasis>Spring Integration</emphasis> will look for a bean of type
<classname>org.springframework.integration.store.MetadataStore</classname> in
the ApplicationContext. If one is found then it will be used, otherwise
it will create a new instance of <classname>SimpleMetadataStore</classname>
which is an in-memory implementation that will only persist metadata within
the lifecycle of the currently running Application Context. This means
that upon restart you may end up with duplicate entries.
</para>
<para>
If you need to persist metadata between Application Context restarts, two
persistent <interfacename>MetadataStores</interfacename> are available:
</para>
<itemizedlist>
<listitem>PropertiesPersistingMetadataStore</listitem>
<listitem>RedisMetadataStore</listitem>
</itemizedlist>
<para>
The <classname>PropertiesPersistingMetadataStore</classname> is backed by
a properties file and a
<interfacename><ulink url="http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/util/PropertiesPersister.html">PropertiesPersister</ulink></interfacename>.
</para>
<programlisting language="xml"><![CDATA[<bean id="metadataStore"
class="org.springframework.integration.store.PropertiesPersistingMetadataStore"/>]]></programlisting>
</para>
</section>
<para>
As of <emphasis>Spring Integration 3.0</emphasis> a Redis-based
<interfacename>MetadataStore</interfacename> is also available. For
more information regarding the <classname>RedisMetadataStore</classname>
see <xref linkend="redis-metadata-store" />.
</para>
<warning>
Be careful when using the same Redis instancce across multiple application
contexts as separate Feed adapters may accidentally use the same persisted
key.
</warning>
<para>
Alternatively, you could provide your own implementation of the
<interfacename>MetadataStore</interfacename> interface (e.g. JdbcMetadataStore)
and configure it as bean in the Application Context.
</para>
</section>
</chapter>

31
src/reference/docbook/redis.xml
View File

@@ -223,7 +223,36 @@ rt.setConnectionFactory(redisConnectionFactory);]]></programlisting>
the <code>valueSerializer</code> property of the <classname>RedisMessageStore</classname>.
</para>
</section>
<section id="redis-metadata-store">
<title>Redis Metadata Store</title>
<para>
As of <emphasis>Spring Integration 3.0</emphasis> a new Redis-based
<interfacename><ulink url="http://docs.spring.io/spring-integration/docs/latest-ga/api/org/springframework/integration/store/MetadataStore.html">MetadataStore</ulink></interfacename>
implementation is available. The <classname>RedisMetadataStore</classname> can
be used to maintain state of a <interfacename>MetadataStore</interfacename>
across application restarts. This new <interfacename>MetadataStore</interfacename>
implementation can be used with adapters such as:
</para>
<itemizedlist>
<listitem>Twitter Inbound Adapters</listitem>
<listitem>Feed Inbound Channel Adapter</listitem>
</itemizedlist>
<para>
In order to instruct these adapters to use the new <classname>RedisMetadataStore</classname>
simply declare a Spring bean using the bean name <emphasis role="bold">metadataStore</emphasis>.
The <emphasis>Twitter Inbound Channel Adapter</emphasis> and the
<emphasis>Feed Inbound Channel Adapter</emphasis> will both automatically
pick up and use the declared <classname>RedisMetadataStore</classname>.
</para>
<programlisting language="xml"><![CDATA[<bean name="metadataStore" class="o.s.i.redis.store.metadata.RedisMetadataStore">
<constructor-arg name="connectionFactory" ref="redisConnectionFactory"/>
</bean>]]></programlisting>
<warning>
Be careful when using the same Redis instancce across multiple application
contexts as separate adapters may accidentally use the same persisted
key.
</warning>
</section>
<section id="redis-store-inbound-channel-adapter">
<title>RedisStore Inbound Channel Adapter</title>

13
src/reference/docbook/twitter.xml
View File

@@ -151,11 +151,22 @@ twitter.oauth.accessTokenSecret=AbRxUAvyNCtqQtxFK8w5ZMtMj20KFhB6o]]></programlis
restarts, you may use the <classname>PropertiesPersistingMetadataStore</classname> (which is backed by a properties file, and a persister
strategy), or you may create your own custom implementation of the <classname>MetadataStore</classname> interface (e.g., JdbcMetadatStore)
and configure it as a bean named 'metadataStore' within the Application Context.
</para>
<para>
As of <emphasis>Spring Integration 3.0</emphasis> a Redis-based
<interfacename>MetadataStore</interfacename> is available. The
<classname>RedisMetadataStore</classname> allows you to maintain persisted
metadata across Application Context restarts. For more information see <xref linkend="redis-metadata-store" />.
</para>
<warning>
Be careful when using the same Redis instance across multiple application
contexts as separate Twitter adapters may accidentally use the same persisted
key.
</warning>
<programlisting language="xml"><![CDATA[<bean id="metadataStore" class="o.s.i.store.PropertiesPersistingMetadataStore"/>
]]></programlisting>
The Poller that is configured as part of any Inbound Twitter Adapter (see below) will simply poll from this MetadataStore to determine the latest tweet
received.
</para>
<section id="inbound-twitter-update">
<title>Inbound Message Channel Adapter</title>
<para>

18
src/reference/docbook/whats-new.xml
View File

@@ -132,6 +132,24 @@
For more information see <xref linkend="http-namespace"/>.
</para>
</section>
<section id="3.0-redis-meta-data-store">
<title>Redis Metadata Store</title>
<para>
A new Redis-based
<interfacename><ulink url="http://docs.spring.io/spring-integration/docs/latest-ga/api/org/springframework/integration/store/MetadataStore.html">MetadataStore</ulink></interfacename>
implementation was added. The <classname>RedisMetadataStore</classname> can
be used to maintain state of a <interfacename>MetadataStore</interfacename>
across application restarts. This new <interfacename>MetadataStore</interfacename>
implementation can be used with adapters such as:
</para>
<itemizedlist>
<listitem>Twitter Inbound Adapters</listitem>
<listitem>Feed Inbound Channel Adapter</listitem>
</itemizedlist>
<para>
For more information see <xref linkend="redis-metadata-store" />.
</para>
</section>
</section>
<section id="3.0-general">