INT-3147: (3167,3173,1941) Improve MetadataStore

Previously, `MetadataStore` couldn't be configured for Twitter Adapters
- only a global one could be used.
The `metadataKey` was generated automatically with a 'difficult' value.

* Register all `MessageSource` for `SourcePollingChannelAdapter`
as beans with id based on adapter id and prefix '.source' (INT-3147)
* Polishing parser to get rid of explicit `MessageSource` beans. (INT-3147)
* Make Feed and Twitter adapters `id` attribute as required -
now it presents a `metadataKey` for `MetadataStore` (INT-3147)
* Add to Twitter adapters a reference attribute for `MetadataStore` (INT-3173)
* Add Twitter adapters `poll-skip-period` attribute (INT-3167)
* Add and implement `MetadataStore#remove` (INT-1941)
* Make `MetadataStore` as `@ManagedResource` (INT-1941)
* Polishing tests

JIRAs:
https://jira.springsource.org/browse/INT-3147
https://jira.springsource.org/browse/INT-3167
https://jira.springsource.org/browse/INT-3173
https://jira.springsource.org/browse/INT-1941

INT-3147: Polishing and fixes

* add domain suffix to `metadataKey`
* change contract of `MetadataStore.remove`
* remove timeout window from `AbstractTwitterMessageSource`
* polishing and fix `SearchReceivingMessageSourceWithRedisTests`

INT-3147: Rebasing and polishing

INT-3147: fix 'metadata' package tangle

INT-3147 Doc Polishing
This commit is contained in:
Artem Bilan
2013-10-03 11:27:36 -04:00
committed by Gary Russell
parent 1fb838dd1a
commit 5be8ef3fd8
40 changed files with 393 additions and 271 deletions

View File

@@ -62,14 +62,14 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
Spring Integration provides a convenient mechanism to eliminate the need to worry about duplicate entries.
Each feed entry will have a <emphasis>published date</emphasis> field. Every time a new Message is generated and sent,
Spring Integration will store the value of the latest <emphasis>published date</emphasis> in an instance of the
<classname>org.springframework.integration.store.MetadataStore</classname> strategy. The MetadataStore interface is
<classname>org.springframework.integration.metadata.MetadataStore</classname> strategy. The MetadataStore interface is
designed to store various types of generic meta-data (e.g., published date of the last feed entry that has been processed)
to help components such as this Feed adapter deal with duplicates.
</para>
<para>
The default rule for locating this metadata store is as follows:
<emphasis>Spring Integration</emphasis> will look for a bean of type
<classname>org.springframework.integration.store.MetadataStore</classname> in
<classname>org.springframework.integration.metadata.MetadataStore</classname> in
the ApplicationContext. If one is found then it will be used, otherwise
it will create a new instance of <classname>SimpleMetadataStore</classname>
which is an in-memory implementation that will only persist metadata within
@@ -107,5 +107,10 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
<interfacename>MetadataStore</interfacename> interface (e.g. JdbcMetadataStore)
and configure it as bean in the Application Context.
</para>
<note>
The key used to persist the latest <emphasis>published date</emphasis> is the value of the (required)
<code>id</code> attribute of the Feed Inbound Channel Adapter component plus the <code>feedUrl</code>
from the adapter's configuration.
</note>
</section>
</chapter>

View File

@@ -141,11 +141,12 @@ twitter.oauth.accessTokenSecret=AbRxUAvyNCtqQtxFK8w5ZMtMj20KFhB6o]]></programlis
criteria you'll end up with the same set of tweets unless some other new tweet that matches your search criteria was posted
in between your searches. In that situation you'll get all the tweets you had before plus the new one. But what you really
want is only the new tweet(s). Spring Integration provides an elegant mechanism for handling these situations.
The latest Tweet timestamp will be stored in an instance of the <classname>org.springframework.integration.store.MetadataStore</classname> which is a
The latest Tweet id will be stored in an instance of the <classname>org.springframework.integration.metadata.MetadataStore</classname> which is a
strategy interface designed for storing various types of metadata (e.g., last retrieved tweet in this case). That strategy helps components such as
these Twitter adapters avoid duplicates. By default, Spring Integration will look for a bean of type
<classname>org.springframework.integration.store.MetadataStore</classname> in the ApplicationContext.
If one is found then it will be used, otherwise it will create a new instance of <classname>SimpleMetadataStore</classname>
<classname>org.springframework.integration.metadata.MetadataStore</classname> in the ApplicationContext. Alternatively,
you can configure an explicit <classname>MetadataStore</classname> on the adapter.
If there is no explicit or default store, the adapter will create a new instance of <classname>SimpleMetadataStore</classname>
which is a simple in-memory implementation that will only persist metadata within the lifecycle of the currently running application context.
That means upon restart you may end up with duplicate entries. If you need to persist metadata between Application Context
restarts, you may use the <classname>PropertiesPersistingMetadataStore</classname> (which is backed by a properties file, and a persister
@@ -165,8 +166,16 @@ twitter.oauth.accessTokenSecret=AbRxUAvyNCtqQtxFK8w5ZMtMj20KFhB6o]]></programlis
</warning>
<programlisting language="xml"><![CDATA[<bean id="metadataStore" class="o.s.i.store.PropertiesPersistingMetadataStore"/>
]]></programlisting>
The Poller that is configured as part of any Inbound Twitter Adapter (see below) will simply poll from this MetadataStore to determine the latest tweet
received.
<para>
If the <classname>MetadataStore</classname> is persistent, during initialization, any Inbound Twitter Adapter (see below)
will retrieve the latest tweet id that has already been sent by the adapter.
</para>
<note>
The key used to persist the latest <emphasis>twitter id</emphasis> is the value of the (required)
<code>id</code> attribute of the Twitter Inbound Channel Adapter component plus the <code>profileId</code>
of the Twitter user.
</note>
<section id="inbound-twitter-update">
<title>Inbound Message Channel Adapter</title>
<para>