INT-3147: (3167,3173,1941) Improve MetadataStore

Previously, `MetadataStore` couldn't be configured for Twitter Adapters
- only a global one could be used.
The `metadataKey` was generated automatically with a 'difficult' value.

* Register all `MessageSource` for `SourcePollingChannelAdapter`
as beans with id based on adapter id and prefix '.source' (INT-3147)
* Polishing parser to get rid of explicit `MessageSource` beans. (INT-3147)
* Make Feed and Twitter adapters `id` attribute as required -
now it presents a `metadataKey` for `MetadataStore` (INT-3147)
* Add to Twitter adapters a reference attribute for `MetadataStore` (INT-3173)
* Add Twitter adapters `poll-skip-period` attribute (INT-3167)
* Add and implement `MetadataStore#remove` (INT-1941)
* Make `MetadataStore` as `@ManagedResource` (INT-1941)
* Polishing tests

JIRAs:
https://jira.springsource.org/browse/INT-3147
https://jira.springsource.org/browse/INT-3167
https://jira.springsource.org/browse/INT-3173
https://jira.springsource.org/browse/INT-1941

INT-3147: Polishing and fixes

* add domain suffix to `metadataKey`
* change contract of `MetadataStore.remove`
* remove timeout window from `AbstractTwitterMessageSource`
* polishing and fix `SearchReceivingMessageSourceWithRedisTests`

INT-3147: Rebasing and polishing

INT-3147: fix 'metadata' package tangle

INT-3147 Doc Polishing
This commit is contained in:
Artem Bilan
2013-10-03 11:27:36 -04:00
committed by Gary Russell
parent 1fb838dd1a
commit 5be8ef3fd8
40 changed files with 393 additions and 271 deletions

View File

@@ -19,6 +19,7 @@ package org.springframework.integration.config.xml;
import org.w3c.dom.Element;
import org.springframework.beans.BeanMetadataElement;
import org.springframework.beans.factory.config.BeanDefinition;
import org.springframework.beans.factory.support.AbstractBeanDefinition;
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.xml.ParserContext;
@@ -31,18 +32,25 @@ import org.springframework.util.xml.DomUtils;
* @author Mark Fisher
* @author Gary Russell
* @author Oleg Zhurakousky
* @author Artem Bilan
*/
public abstract class AbstractPollingInboundChannelAdapterParser extends AbstractChannelAdapterParser {
@Override
@SuppressWarnings("unchecked")
protected AbstractBeanDefinition doParse(Element element, ParserContext parserContext, String channelName) {
BeanMetadataElement source = this.parseSource(element, parserContext);
if (source == null) {
parserContext.getReaderContext().error("failed to parse source", element);
}
String channelAdapterId = this.resolveId(element, (AbstractBeanDefinition) source, parserContext);
String sourceBeanName = channelAdapterId + ".source";
parserContext.getRegistry().registerBeanDefinition(sourceBeanName, (BeanDefinition) source);
BeanDefinitionBuilder adapterBuilder = BeanDefinitionBuilder
.genericBeanDefinition(SourcePollingChannelAdapterFactoryBean.class);
adapterBuilder.addPropertyValue("source", source);
adapterBuilder.addPropertyReference("source", sourceBeanName);
adapterBuilder.addPropertyReference("outputChannel", channelName);
IntegrationNamespaceUtils.setValueIfAttributeDefined(adapterBuilder, element, "send-timeout");
Element pollerElement = DomUtils.getChildElementByTagName(element, "poller");

View File

@@ -20,7 +20,7 @@ import org.springframework.beans.factory.BeanFactory;
import org.springframework.core.convert.ConversionService;
import org.springframework.expression.spel.support.StandardEvaluationContext;
import org.springframework.integration.MessageChannel;
import org.springframework.integration.store.metadata.MetadataStore;
import org.springframework.integration.metadata.MetadataStore;
import org.springframework.scheduling.TaskScheduler;
import org.springframework.util.Assert;

View File

@@ -14,7 +14,10 @@
* limitations under the License.
*/
package org.springframework.integration.store.metadata;
package org.springframework.integration.metadata;
import org.springframework.jmx.export.annotation.ManagedAttribute;
import org.springframework.jmx.export.annotation.ManagedResource;
/**
* Strategy interface for storing metadata from certain adapters
@@ -25,6 +28,7 @@ package org.springframework.integration.store.metadata;
* @author Mark Fisher
* @since 2.0
*/
@ManagedResource
public interface MetadataStore {
/**
@@ -35,6 +39,15 @@ public interface MetadataStore {
/**
* Reads a value for the given key from this MetadataStore.
*/
@ManagedAttribute
String get(String key);
/**
* Remove a value for the given key from this MetadataStore.
* return the previous value associated with <tt>key</tt>, or
* <tt>null</tt> if there was no mapping for <tt>key</tt>.
*/
@ManagedAttribute
String remove(String key);
}

View File

@@ -14,7 +14,7 @@
* limitations under the License.
*/
package org.springframework.integration.store.metadata;
package org.springframework.integration.metadata;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
@@ -86,6 +86,12 @@ public class PropertiesPersistingMetadataStore implements MetadataStore, Initial
return this.metadata.getProperty(key);
}
@Override
@SuppressWarnings("uchecked")
public String remove(String key) {
return (String) this.metadata.remove(key);
}
public void destroy() throws Exception {
this.saveMetadata();
}

View File

@@ -11,7 +11,7 @@
* specific language governing permissions and limitations under the License.
*/
package org.springframework.integration.store.metadata;
package org.springframework.integration.metadata;
import java.util.HashMap;
import java.util.Map;
@@ -37,4 +37,9 @@ public class SimpleMetadataStore implements MetadataStore {
return this.metadata.get(key);
}
@Override
public String remove(String key) {
return metadata.remove(key);
}
}

View File

@@ -1,4 +1,4 @@
/**
* Provides classes supporting metadata stores.
*/
package org.springframework.integration.store.metadata;
package org.springframework.integration.metadata;

View File

@@ -14,7 +14,7 @@
* limitations under the License.
*/
package org.springframework.integration.store.metadata;
package org.springframework.integration.metadata;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
@@ -27,7 +27,6 @@ import org.junit.Test;
import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.support.PropertiesLoaderUtils;
import org.springframework.integration.store.metadata.PropertiesPersistingMetadataStore;
/**
* @author Oleg Zhurakousky

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2010 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -23,6 +23,7 @@ import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.xml.ParserContext;
import org.springframework.integration.config.xml.AbstractPollingInboundChannelAdapterParser;
import org.springframework.integration.config.xml.IntegrationNamespaceUtils;
import org.springframework.integration.feed.inbound.FeedEntryMessageSource;
import org.springframework.util.StringUtils;
/**
@@ -31,20 +32,23 @@ import org.springframework.util.StringUtils;
* @author Josh Long
* @author Oleg Zhurakousky
* @author Mark Fisher
* @author Gunnar Hillert
* @author Artem Bilan
* @since 2.0
*/
public class FeedInboundChannelAdapterParser extends AbstractPollingInboundChannelAdapterParser {
@Override
protected BeanMetadataElement parseSource(final Element element, final ParserContext parserContext) {
BeanDefinitionBuilder sourceBuilder = BeanDefinitionBuilder.genericBeanDefinition(
"org.springframework.integration.feed.inbound.FeedEntryMessageSource");
BeanDefinitionBuilder sourceBuilder = BeanDefinitionBuilder.genericBeanDefinition(FeedEntryMessageSource.class);
sourceBuilder.addConstructorArgValue(element.getAttribute("url"));
sourceBuilder.addConstructorArgValue(element.getAttribute(ID_ATTRIBUTE));
String feedFetcherRef = element.getAttribute("feed-fetcher");
if (StringUtils.hasText(feedFetcherRef)) {
sourceBuilder.addConstructorArgReference(feedFetcherRef);
}
IntegrationNamespaceUtils.setReferenceIfAttributeDefined(sourceBuilder, element, "metadata-store");
return sourceBuilder.getBeanDefinition();
}

View File

@@ -30,8 +30,8 @@ import org.springframework.integration.MessagingException;
import org.springframework.integration.context.IntegrationContextUtils;
import org.springframework.integration.context.IntegrationObjectSupport;
import org.springframework.integration.core.MessageSource;
import org.springframework.integration.store.metadata.MetadataStore;
import org.springframework.integration.store.metadata.SimpleMetadataStore;
import org.springframework.integration.metadata.MetadataStore;
import org.springframework.integration.metadata.SimpleMetadataStore;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.util.Assert;
import org.springframework.util.CollectionUtils;
@@ -52,6 +52,7 @@ import com.sun.syndication.fetcher.impl.HttpURLFeedFetcher;
* @author Josh Long
* @author Mario Gray
* @author Oleg Zhurakousky
* @author Artem Bilan
* @since 2.0
*/
public class FeedEntryMessageSource extends IntegrationObjectSupport implements MessageSource<SyndEntry> {
@@ -62,7 +63,7 @@ public class FeedEntryMessageSource extends IntegrationObjectSupport implements
private final Queue<SyndEntry> entries = new ConcurrentLinkedQueue<SyndEntry>();
private volatile String metadataKey;
private final String metadataKey;
private volatile MetadataStore metadataStore;
@@ -82,17 +83,19 @@ public class FeedEntryMessageSource extends IntegrationObjectSupport implements
* If the feed URL has a protocol other than http*, consider providing a custom implementation of the
* {@link FeedFetcher} via the alternate constructor.
*/
public FeedEntryMessageSource(URL feedUrl) {
this(feedUrl, new HttpURLFeedFetcher(HashMapFeedInfoCache.getInstance()));
public FeedEntryMessageSource(URL feedUrl, String metadataKey) {
this(feedUrl, metadataKey, new HttpURLFeedFetcher(HashMapFeedInfoCache.getInstance()));
}
/**
* Creates a FeedEntryMessageSource that will use the provided FeedFetcher to read from the given feed URL.
*/
public FeedEntryMessageSource(URL feedUrl, FeedFetcher feedFetcher) {
public FeedEntryMessageSource(URL feedUrl, String metadataKey, FeedFetcher feedFetcher) {
Assert.notNull(feedUrl, "feedUrl must not be null");
Assert.notNull(metadataKey, "metadataKey must not be null");
Assert.notNull(feedFetcher, "feedFetcher must not be null");
this.feedUrl = feedUrl;
this.metadataKey = metadataKey + "." + this.feedUrl;
this.feedFetcher = feedFetcher;
}
@@ -130,18 +133,7 @@ public class FeedEntryMessageSource extends IntegrationObjectSupport implements
this.metadataStore = new SimpleMetadataStore();
}
}
StringBuilder metadataKeyBuilder = new StringBuilder();
if (StringUtils.hasText(this.getComponentType())) {
metadataKeyBuilder.append(this.getComponentType() + ".");
}
if (StringUtils.hasText(this.getComponentName())) {
metadataKeyBuilder.append(this.getComponentName() + ".");
}
else if (logger.isWarnEnabled()) {
logger.warn("FeedEntryMessageSource has no name. MetadataStore key might not be unique.");
}
metadataKeyBuilder.append(this.feedUrl);
this.metadataKey = metadataKeyBuilder.toString();
String lastTimeValue = this.metadataStore.get(this.metadataKey);
if (StringUtils.hasText(lastTimeValue)) {
this.lastTime = Long.parseLong(lastTimeValue);
@@ -237,6 +229,7 @@ public class FeedEntryMessageSource extends IntegrationObjectSupport implements
}
return (date2 == null) ? 1 : 0;
}
}
@@ -258,6 +251,7 @@ public class FeedEntryMessageSource extends IntegrationObjectSupport implements
}
}
}
}
}

View File

@@ -22,7 +22,28 @@
<xsd:sequence>
<xsd:element ref="integration:poller" minOccurs="0" maxOccurs="1" />
</xsd:sequence>
<xsd:attributeGroup ref="integration:channelAdapterAttributes"/>
<xsd:attribute name="id" type="xsd:string" use="required">
<xsd:annotation>
<xsd:documentation>
The bean id of this Polling Endpoint; the MessageSource is also registered with this id
plus a suffix '.source'; also used as the
MetaDataStore key with suffix '.' + feedUrl - The URL for an RSS or ATOM feed.
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
<xsd:attribute name="channel" type="xsd:string">
<xsd:annotation>
<xsd:appinfo>
<tool:annotation kind="ref">
<tool:expected-type type="org.springframework.integration.MessageChannel" />
</tool:annotation>
</xsd:appinfo>
<xsd:documentation>
Identifies the channel attached to this adapter, to which messages will be sent.
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
<xsd:attributeGroup ref="integration:smartLifeCycleAttributeGroup"/>
<xsd:attribute name="url" type="xsd:string" use="required">
<xsd:annotation>
<xsd:documentation>
@@ -54,7 +75,7 @@
</xsd:documentation>
<xsd:appinfo>
<tool:annotation kind="ref">
<tool:expected-type type="org.springframework.integration.store.MetadataStore" />
<tool:expected-type type="org.springframework.integration.metadata.MetadataStore" />
</tool:annotation>
</xsd:appinfo>
</xsd:annotation>

View File

@@ -21,6 +21,6 @@
<bean id="fileUrlFeedFetcher" class="org.springframework.integration.feed.inbound.FileUrlFeedFetcher"/>
<bean id="metadataStore" class="org.springframework.integration.store.metadata.PropertiesPersistingMetadataStore"/>
<bean id="metadataStore" class="org.springframework.integration.metadata.PropertiesPersistingMetadataStore"/>
</beans>

View File

@@ -1,21 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-feed="http://www.springframework.org/schema/integration/feed"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/integration/feed http://www.springframework.org/schema/integration/feed/spring-integration-feed.xsd">
<int-feed:inbound-channel-adapter channel="feedChannelUsage"
url="file:src/test/java/org/springframework/integration/feed/sample.rss"
feed-fetcher="fileUrlFeedFetcher">
<int:poller fixed-rate="10000" max-messages-per-poll="100"/>
</int-feed:inbound-channel-adapter>
<int:service-activator id="sampleActivator" input-channel="feedChannelUsage">
<bean class="org.springframework.integration.feed.config.FeedInboundChannelAdapterParserTests$SampleServiceNoHistory" />
</int:service-activator>
<bean id="fileUrlFeedFetcher" class="org.springframework.integration.feed.inbound.FileUrlFeedFetcher"/>
</beans>

View File

@@ -33,6 +33,7 @@ import org.junit.Before;
import org.junit.Ignore;
import org.junit.Test;
import org.mockito.Mockito;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.springframework.integration.Message;
@@ -43,7 +44,7 @@ import org.springframework.integration.core.MessageHandler;
import org.springframework.integration.endpoint.SourcePollingChannelAdapter;
import org.springframework.integration.feed.inbound.FeedEntryMessageSource;
import org.springframework.integration.history.MessageHistory;
import org.springframework.integration.store.metadata.MetadataStore;
import org.springframework.integration.metadata.MetadataStore;
import org.springframework.integration.test.util.TestUtils;
import com.sun.syndication.feed.synd.SyndEntry;
@@ -83,6 +84,7 @@ public class FeedInboundChannelAdapterParserTests {
context.destroy();
}
public void validateSuccessfulHttpConfigurationWithCustomMetadataStore() {
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(
"FeedInboundChannelAdapterParserTests-http-context.xml", this.getClass());
@@ -98,7 +100,7 @@ public class FeedInboundChannelAdapterParserTests {
@Test
public void validateSuccessfulNewsRetrievalWithFileUrlAndMessageHistory() throws Exception {
File persisterFile = new File(System.getProperty("java.io.tmpdir") + "/spring-integration/", "message-store.properties");
File persisterFile = new File(System.getProperty("java.io.tmpdir") + "/spring-integration/", "metadata-store.properties");
if (persisterFile.exists()) {
persisterFile.delete();
}
@@ -120,26 +122,6 @@ public class FeedInboundChannelAdapterParserTests {
context.destroy();
}
@Test
public void validateSuccessfulNewsRetrievalWithFileUrlNoPersistentIdentifier() throws Exception{
//Test file samples.rss has 3 news items
latch = spy(new CountDownLatch(3));
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(
"FeedInboundChannelAdapterParserTests-file-usage-noid-context.xml", this.getClass());
latch.await(5, TimeUnit.SECONDS);
verify(latch, times(3)).countDown();
context.destroy();
// since we are not deleting the persister file
// in this iteration no new feeds will be received and the latch will timeout
latch = spy(new CountDownLatch(3));
context = new ClassPathXmlApplicationContext(
"FeedInboundChannelAdapterParserTests-file-usage-noid-context.xml", this.getClass());
latch.await(5, TimeUnit.SECONDS);
verify(latch, times(3)).countDown();
context.destroy();
}
@Test
@Ignore // goes against the real feed
public void validateSuccessfulNewsRetrievalWithHttpUrl() throws Exception{
@@ -204,6 +186,11 @@ public class FeedInboundChannelAdapterParserTests {
public String get(String key) {
return null;
}
@Override
public String remove(String key) {
return null;
}
}
}

View File

@@ -26,7 +26,7 @@ import java.net.URL;
import org.junit.Before;
import org.junit.Test;
import org.springframework.integration.Message;
import org.springframework.integration.store.metadata.PropertiesPersistingMetadataStore;
import org.springframework.integration.metadata.PropertiesPersistingMetadataStore;
import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.fetcher.FeedFetcher;
@@ -52,14 +52,14 @@ public class FeedEntryMessageSourceTests {
@Test(expected=IllegalArgumentException.class)
public void testFailureWhenNotInitialized() throws Exception {
URL url = new URL("file:src/test/java/org/springframework/integration/feed/sample.rss");
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url);
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url, "foo");
feedEntrySource.receive();
}
@Test
public void testReceiveFeedWithNoEntries() throws Exception {
URL url = new URL("file:src/test/java/org/springframework/integration/feed/empty.rss");
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url, this.feedFetcher);
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url, "foo", this.feedFetcher);
feedEntrySource.setBeanName("feedReader");
feedEntrySource.afterPropertiesSet();
assertNull(feedEntrySource.receive());
@@ -68,7 +68,7 @@ public class FeedEntryMessageSourceTests {
@Test
public void testReceiveFeedWithEntriesSorted() throws Exception {
URL url = new URL("file:src/test/java/org/springframework/integration/feed/sample.rss");
FeedEntryMessageSource source = new FeedEntryMessageSource(url, this.feedFetcher);
FeedEntryMessageSource source = new FeedEntryMessageSource(url, "foo", this.feedFetcher);
source.setComponentName("feedReader");
source.afterPropertiesSet();
Message<SyndEntry> message1 = source.receive();
@@ -87,7 +87,7 @@ public class FeedEntryMessageSourceTests {
@Test
public void testReceiveFeedWithRealEntriesAndRepeatWithPersistentMetadataStore() throws Exception {
URL url = new URL("file:src/test/java/org/springframework/integration/feed/sample.rss");
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url, this.feedFetcher);
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url, "foo", this.feedFetcher);
feedEntrySource.setBeanName("feedReader");
PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
metadataStore.afterPropertiesSet();
@@ -111,7 +111,7 @@ public class FeedEntryMessageSourceTests {
metadataStore.afterPropertiesSet();
// now test that what's been read is no longer retrieved
feedEntrySource = new FeedEntryMessageSource(url, this.feedFetcher);
feedEntrySource = new FeedEntryMessageSource(url, "foo", this.feedFetcher);
feedEntrySource.setBeanName("feedReader");
metadataStore = new PropertiesPersistingMetadataStore();
metadataStore.afterPropertiesSet();
@@ -127,7 +127,7 @@ public class FeedEntryMessageSourceTests {
@Test
public void testReceiveFeedWithRealEntriesAndRepeatNoPersistentMetadataStore() throws Exception {
URL url = new URL("file:src/test/java/org/springframework/integration/feed/sample.rss");
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url, this.feedFetcher);
FeedEntryMessageSource feedEntrySource = new FeedEntryMessageSource(url, "foo", this.feedFetcher);
feedEntrySource.setBeanName("feedReader");
feedEntrySource.afterPropertiesSet();
SyndEntry entry1 = feedEntrySource.receive().getPayload();
@@ -146,7 +146,7 @@ public class FeedEntryMessageSourceTests {
// UNLIKE the previous test
// now test that what's been read is read AGAIN
feedEntrySource = new FeedEntryMessageSource(url, this.feedFetcher);
feedEntrySource = new FeedEntryMessageSource(url, "foo", this.feedFetcher);
feedEntrySource.setBeanName("feedReader");
feedEntrySource.afterPropertiesSet();
entry1 = feedEntrySource.receive().getPayload();

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2012 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -16,9 +16,10 @@
package org.springframework.integration.file.config;
import org.w3c.dom.Element;
import org.springframework.beans.BeanMetadataElement;
import org.springframework.beans.factory.config.BeanDefinition;
import org.springframework.beans.factory.config.RuntimeBeanReference;
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.support.BeanDefinitionReaderUtils;
import org.springframework.beans.factory.xml.ParserContext;
@@ -27,7 +28,6 @@ import org.springframework.integration.config.xml.IntegrationNamespaceUtils;
import org.springframework.integration.file.locking.NioFileLocker;
import org.springframework.util.StringUtils;
import org.springframework.util.xml.DomUtils;
import org.w3c.dom.Element;
/**
* Parser for the &lt;inbound-channel-adapter&gt; element of the 'file' namespace.
@@ -53,9 +53,8 @@ public class FileInboundChannelAdapterParser extends AbstractPollingInboundChann
builder.addPropertyReference("locker", lockerBeanName);
}
builder.addPropertyReference("filter", filterBeanName);
String beanName = BeanDefinitionReaderUtils.registerWithGeneratedName(
builder.getBeanDefinition(), parserContext.getRegistry());
return new RuntimeBeanReference(beanName);
return builder.getBeanDefinition();
}
private String registerLocker(Element element, ParserContext parserContext) {

View File

@@ -60,9 +60,8 @@ public class FtpParserInboundTests {
fail("BeansException expected.");
}
catch (BeansException e) {
assertThat(e, Matchers.instanceOf(BeanCreationException.class));
Throwable cause = e.getCause();
assertThat(cause, Matchers.instanceOf(BeanCreationException.class));
cause = cause.getCause();
assertThat(cause, Matchers.instanceOf(MessagingException.class));
cause = cause.getCause();
assertThat(cause, Matchers.instanceOf(FileNotFoundException.class));

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2010 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -19,18 +19,15 @@ package org.springframework.integration.jms.config;
import org.w3c.dom.Element;
import org.springframework.beans.BeanMetadataElement;
import org.springframework.beans.factory.config.BeanDefinition;
import org.springframework.beans.factory.parsing.BeanComponentDefinition;
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.support.BeanDefinitionReaderUtils;
import org.springframework.beans.factory.xml.ParserContext;
import org.springframework.integration.config.xml.AbstractPollingInboundChannelAdapterParser;
import org.springframework.integration.config.xml.IntegrationNamespaceUtils;
import org.springframework.util.StringUtils;
/**
* Parser for the &lt;inbound-channel-adapter/&gt; element of the 'jms' namespace.
*
* Parser for the &lt;inbound-channel-adapter/&gt; element of the 'jms' namespace.
*
* @author Mark Fisher
*/
public class JmsInboundChannelAdapterParser extends AbstractPollingInboundChannelAdapterParser {
@@ -87,9 +84,7 @@ public class JmsInboundChannelAdapterParser extends AbstractPollingInboundChanne
builder.addPropertyReference(JmsAdapterParserUtils.HEADER_MAPPER_PROPERTY, headerMapper);
}
IntegrationNamespaceUtils.setValueIfAttributeDefined(builder, element, "selector", "messageSelector");
BeanDefinition beanDefinition = builder.getBeanDefinition();
String beanName = BeanDefinitionReaderUtils.generateBeanName(beanDefinition, parserContext.getRegistry());
return new BeanComponentDefinition(beanDefinition, beanName);
return builder.getBeanDefinition();
}
}

View File

@@ -13,18 +13,18 @@
http://www.springframework.org/schema/integration/jmx/spring-integration-jmx.xsd">
<context:mbean-server id="mbs" />
<context:mbean-export server="mbs" default-domain="test.PollingAdapterMBean"/>
<int:channel id="testChannel" />
<int:inbound-channel-adapter channel="testChannel" method="get">
<int:inbound-channel-adapter id="adapter" channel="testChannel" method="get">
<int:poller fixed-rate="5000" max-messages-per-poll="1"/>
<bean class="org.springframework.integration.jmx.config.PollingAdapterMBeanTests$Source"/>
</int:inbound-channel-adapter>
<int:logging-channel-adapter channel="testChannel"/>
<jmx:mbean-export id="integrationMbeanExporter" server="mbs" default-domain="test.PollingAdapterMBean"/>
</beans>

View File

@@ -1,11 +1,11 @@
/*
* Copyright 2002-2010 the original author or authors.
*
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
*
* http://www.apache.org/licenses/LICENSE-2.0
*
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
@@ -36,7 +36,7 @@ public class PollingAdapterMBeanTests {
@Autowired
private MBeanServer server;
@Test
public void testMessageSourceMBeanExists() throws Exception {
// System.err.println(server.queryNames(new ObjectName("*:type=MessageSource,*"), null));

View File

@@ -15,13 +15,11 @@
*/
package org.springframework.integration.mongodb.config;
import org.springframework.beans.factory.config.BeanDefinition;
import org.w3c.dom.Element;
import org.springframework.beans.BeanMetadataElement;
import org.springframework.beans.factory.config.RuntimeBeanReference;
import org.springframework.beans.factory.config.BeanDefinition;
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.support.BeanDefinitionReaderUtils;
import org.springframework.beans.factory.xml.ParserContext;
import org.springframework.integration.config.xml.AbstractPollingInboundChannelAdapterParser;
import org.springframework.integration.config.xml.IntegrationNamespaceUtils;
@@ -52,8 +50,7 @@ public class MongoDbInboundChannelAdapterParser extends AbstractPollingInboundCh
IntegrationNamespaceUtils.setValueIfAttributeDefined(builder, element, "entity-class");
IntegrationNamespaceUtils.setValueIfAttributeDefined(builder, element, "expect-single-result");
String beanName = BeanDefinitionReaderUtils.registerWithGeneratedName(
builder.getBeanDefinition(), parserContext.getRegistry());
return new RuntimeBeanReference(beanName);
return builder.getBeanDefinition();
}
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2012 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -20,9 +20,7 @@ import org.w3c.dom.Element;
import org.springframework.beans.BeanMetadataElement;
import org.springframework.beans.factory.config.BeanDefinition;
import org.springframework.beans.factory.config.RuntimeBeanReference;
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.support.BeanDefinitionReaderUtils;
import org.springframework.beans.factory.xml.ParserContext;
import org.springframework.integration.config.xml.AbstractPollingInboundChannelAdapterParser;
import org.springframework.integration.config.xml.IntegrationNamespaceUtils;
@@ -62,9 +60,8 @@ public class RedisStoreInboundChannelAdapterParser extends AbstractPollingInboun
parserContext, element, atLeastOneRequired);
builder.addConstructorArgValue(expressionDef);
IntegrationNamespaceUtils.setValueIfAttributeDefined(builder, element, "collection-type");
String beanName = BeanDefinitionReaderUtils.registerWithGeneratedName(
builder.getBeanDefinition(), parserContext.getRegistry());
return new RuntimeBeanReference(beanName);
return builder.getBeanDefinition();
}
}

View File

@@ -11,13 +11,13 @@
* specific language governing permissions and limitations under the License.
*/
package org.springframework.integration.redis.store.metadata;
package org.springframework.integration.redis.metadata;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.BoundValueOperations;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.integration.store.metadata.MetadataStore;
import org.springframework.integration.metadata.MetadataStore;
import org.springframework.util.Assert;
/**
@@ -65,4 +65,13 @@ public class RedisMetadataStore implements MetadataStore {
BoundValueOperations<String, String> ops = this.redisTemplate.boundValueOps(key);
return ops.get();
}
@Override
public String remove(String key) {
Assert.notNull(key, "'key' must not be null.");
String value = this.get(key);
this.redisTemplate.delete(key);
return value;
}
}

View File

@@ -0,0 +1,5 @@
/**
* Provides support for Redis-based
* {@link org.springframework.integration.metadata.MetadataStore}s.
*/
package org.springframework.integration.redis.metadata;

View File

@@ -1,5 +0,0 @@
/**
* Provides support for Redis-based
* {@link org.springframework.integration.store.metadata.MetadataStore}s.
*/
package org.springframework.integration.redis.store.metadata;

View File

@@ -13,7 +13,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.integration.redis.store.metadata;
package org.springframework.integration.redis.metadata;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
@@ -29,6 +29,7 @@ import org.springframework.integration.redis.rules.RedisAvailableTests;
/**
* @author Gunnar Hillert
* @author Artem Bilan
* @since 3.0
*
*/
@@ -143,4 +144,20 @@ public class RedisMetadataStoreTests extends RedisAvailableTests {
fail("Expected an IllegalArgumentException to be thrown.");
}
@Test
@RedisAvailable
public void testRemoveFromMetadataStore(){
RedisConnectionFactory jcf = this.getConnectionFactoryForTest();
RedisMetadataStore metadataStore = new RedisMetadataStore(jcf);
String testKey = "RedisMetadataStoreTests-Remove";
String testValue = "Integration";
metadataStore.put(testKey, testValue);
assertEquals(testValue, metadataStore.remove(testKey));
assertNull(metadataStore.remove(testKey));
}
}

View File

@@ -32,7 +32,7 @@ import org.springframework.util.StringUtils;
/**
* Parser for inbound Twitter Channel Adapters.
*
*
* @author Oleg Zhurakousky
* @since 2.0
*/
@@ -50,7 +50,10 @@ public class TwitterInboundChannelAdapterParser extends AbstractPollingInboundCh
BeanDefinitionBuilder templateBuilder = BeanDefinitionBuilder.genericBeanDefinition(TwitterTemplate.class);
builder.addConstructorArgValue(templateBuilder.getBeanDefinition());
}
builder.addConstructorArgValue(element.getAttribute(ID_ATTRIBUTE));
IntegrationNamespaceUtils.setValueIfAttributeDefined(builder, element, "query");
IntegrationNamespaceUtils.setReferenceIfAttributeDefined(builder, element, "metadata-store");
return builder.getBeanDefinition();
}

View File

@@ -28,9 +28,11 @@ import org.springframework.integration.MessagingException;
import org.springframework.integration.context.IntegrationContextUtils;
import org.springframework.integration.context.IntegrationObjectSupport;
import org.springframework.integration.core.MessageSource;
import org.springframework.integration.store.metadata.MetadataStore;
import org.springframework.integration.store.metadata.SimpleMetadataStore;
import org.springframework.integration.metadata.MetadataStore;
import org.springframework.integration.metadata.SimpleMetadataStore;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.jmx.export.annotation.ManagedAttribute;
import org.springframework.jmx.export.annotation.ManagedOperation;
import org.springframework.social.twitter.api.DirectMessage;
import org.springframework.social.twitter.api.Tweet;
import org.springframework.social.twitter.api.Twitter;
@@ -44,24 +46,29 @@ import org.springframework.util.StringUtils;
* messages when using the Twitter API. This class also handles keeping track of
* the latest inbound message it has received and avoiding, where possible,
* redelivery of duplicate messages. This functionality is enabled using the
* {@link org.springframework.integration.store.MetadataStore} strategy.
* {@link org.springframework.integration.metadata.MetadataStore} strategy.
*
* @author Josh Long
* @author Oleg Zhurakousky
* @author Mark Fisher
* @author Gunnar Hillert
* @author Artem Bilan
*
* @since 2.0
*/
@SuppressWarnings("rawtypes")
abstract class AbstractTwitterMessageSource<T> extends IntegrationObjectSupport implements MessageSource {
private volatile long lastPollForTweet;
private final Twitter twitter;
private final TweetComparator tweetComparator = new TweetComparator();
private final Object lastEnqueuedIdMonitor = new Object();
private final String metadataKey;
private volatile MetadataStore metadataStore;
private volatile String metadataKey;
private final Queue<T> tweets = new LinkedBlockingQueue<T>();
private volatile int prefetchThreshold = 0;
@@ -70,25 +77,36 @@ abstract class AbstractTwitterMessageSource<T> extends IntegrationObjectSupport
private volatile long lastProcessedId = -1;
private final Twitter twitter;
private final TweetComparator tweetComparator = new TweetComparator();
private final Object lastEnqueuedIdMonitor = new Object();
public AbstractTwitterMessageSource(Twitter twitter) {
public AbstractTwitterMessageSource(Twitter twitter, String metadataKey) {
Assert.notNull(twitter, "twitter must not be null");
Assert.notNull(metadataKey, "metadataKey must not be null");
this.twitter = twitter;
if (this.twitter.isAuthorized()){
UserOperations userOperations = this.twitter.userOperations();
String profileId = String.valueOf(userOperations.getProfileId());
if (profileId != null) {
metadataKey += "." + profileId;
}
}
this.metadataKey = metadataKey;
}
public void setMetadataStore(MetadataStore metadataStore) {
this.metadataStore = metadataStore;
}
public void setPrefetchThreshold(int prefetchThreshold) {
this.prefetchThreshold = prefetchThreshold;
}
protected Twitter getTwitter() {
return this.twitter;
}
@Override
protected void onInit() throws Exception{
protected void onInit() throws Exception {
super.onInit();
if (this.metadataStore == null) {
// first try to look for a 'metadataStore' in the context
@@ -100,26 +118,7 @@ abstract class AbstractTwitterMessageSource<T> extends IntegrationObjectSupport
this.metadataStore = new SimpleMetadataStore();
}
}
StringBuilder metadataKeyBuilder = new StringBuilder();
if (StringUtils.hasText(this.getComponentType())) {
metadataKeyBuilder.append(this.getComponentType());
}
if (StringUtils.hasText(this.getComponentName())) {
metadataKeyBuilder.append("." + this.getComponentName());
}
else if (logger.isWarnEnabled()) {
logger.warn(this.getClass().getSimpleName() + " has no name. MetadataStore key might not be unique.");
}
if (this.twitter.isAuthorized()){
UserOperations userOperations = this.twitter.userOperations();
String profileId = String.valueOf(userOperations.getProfileId());
if (profileId != null) {
metadataKeyBuilder.append("." + profileId);
}
}
this.metadataKey = metadataKeyBuilder.toString();
String lastId = this.metadataStore.get(this.metadataKey);
// initialize the last status ID from the metadataStore
if (StringUtils.hasText(lastId)) {
@@ -132,16 +131,10 @@ abstract class AbstractTwitterMessageSource<T> extends IntegrationObjectSupport
public Message<?> receive() {
T tweet = this.tweets.poll();
if (tweet == null) {
long currentTime = System.currentTimeMillis();
long elapsedTime = currentTime - this.lastPollForTweet;
if (elapsedTime < 15000) {
// need to wait longer
return null;
}
this.refreshTweetQueueIfNecessary();
tweet = this.tweets.poll();
this.lastPollForTweet = currentTime;
}
if (tweet != null) {
this.lastProcessedId = this.getIdForTweet(tweet);
this.metadataStore.put(this.metadataKey, String.valueOf(this.lastProcessedId));
@@ -204,6 +197,27 @@ abstract class AbstractTwitterMessageSource<T> extends IntegrationObjectSupport
}
/**
* Remove the metadata key and the corresponding value from the Metadata Store.
*/
@ManagedOperation(description="Remove the metadata key and the corresponding value from the Metadata Store.")
void resetMetadataStore() {
synchronized(this) {
this.metadataStore.remove(this.metadataKey);
this.lastProcessedId = -1L;
this.lastEnqueuedId = -1L;
}
}
/**
*
* @return {@code -1} if lastProcessedId is not set, yet.
*/
@ManagedAttribute
public long getLastProcessedId() {
return this.lastProcessedId;
}
private class TweetComparator implements Comparator<T> {
public int compare(T tweet1, T tweet2) {
@@ -230,6 +244,7 @@ abstract class AbstractTwitterMessageSource<T> extends IntegrationObjectSupport
throw new IllegalArgumentException("Uncomparable Twitter objects: " + tweet1 + " and " + tweet2);
}
}
}
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2011 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -31,14 +31,13 @@ import org.springframework.social.twitter.api.Twitter;
*/
public class DirectMessageReceivingMessageSource extends AbstractTwitterMessageSource<DirectMessage> {
public DirectMessageReceivingMessageSource(Twitter twitter) {
super(twitter);
public DirectMessageReceivingMessageSource(Twitter twitter, String metadataKey) {
super(twitter, metadataKey);
}
@Override
public String getComponentType() {
return "twitter:dm-inbound-channel-adapter";
return "twitter:dm-inbound-channel-adapter";
}
@Override

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2011 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -30,11 +30,10 @@ import org.springframework.social.twitter.api.Twitter;
*/
public class MentionsReceivingMessageSource extends AbstractTwitterMessageSource<Tweet> {
public MentionsReceivingMessageSource(Twitter twitter) {
super(twitter);
public MentionsReceivingMessageSource(Twitter twitter, String metadataKey) {
super(twitter, metadataKey);
}
@Override
public String getComponentType() {
return "twitter:mentions-inbound-channel-adapter";

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2011 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -35,11 +35,10 @@ public class SearchReceivingMessageSource extends AbstractTwitterMessageSource<T
private volatile String query;
public SearchReceivingMessageSource(Twitter twitter) {
super(twitter);
public SearchReceivingMessageSource(Twitter twitter, String metadataKey) {
super(twitter, metadataKey);
}
public void setQuery(String query) {
Assert.hasText(query, "'query' must not be null");
this.query = query;
@@ -47,7 +46,7 @@ public class SearchReceivingMessageSource extends AbstractTwitterMessageSource<T
@Override
public String getComponentType() {
return "twitter:search-inbound-channel-adapter";
return "twitter:search-inbound-channel-adapter";
}
@Override

View File

@@ -22,7 +22,7 @@ import org.springframework.social.twitter.api.Tweet;
import org.springframework.social.twitter.api.Twitter;
/**
* This {@link org.springframework.integration.core.MessageSource} lets Spring Integration consume
* This {@link org.springframework.integration.core.MessageSource} lets Spring Integration consume
* given account's timeline as messages. It has support for dynamic throttling of API requests.
*
* @author Josh Long
@@ -31,14 +31,13 @@ import org.springframework.social.twitter.api.Twitter;
*/
public class TimelineReceivingMessageSource extends AbstractTwitterMessageSource<Tweet> {
public TimelineReceivingMessageSource(Twitter twitter) {
super(twitter);
public TimelineReceivingMessageSource(Twitter twitter, String metadataKey) {
super(twitter, metadataKey);
}
@Override
public String getComponentType() {
return "twitter:inbound-channel-adapter";
return "twitter:inbound-channel-adapter";
}
@Override

View File

@@ -123,7 +123,28 @@
<xsd:sequence>
<xsd:element ref="integration:poller" minOccurs="0" maxOccurs="1" />
</xsd:sequence>
<xsd:attributeGroup ref="integration:channelAdapterAttributes"/>
<xsd:attribute name="id" type="xsd:string" use="required">
<xsd:annotation>
<xsd:documentation>
The bean id of this Polling Endpoint; the MessageSource is also registered with this id
plus a suffix '.source'; also used as the
MetaDataStore key with suffix '.' + the profileId from the authorized Twitter user.
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
<xsd:attribute name="channel" type="xsd:string">
<xsd:annotation>
<xsd:appinfo>
<tool:annotation kind="ref">
<tool:expected-type type="org.springframework.integration.MessageChannel" />
</tool:annotation>
</xsd:appinfo>
<xsd:documentation>
Identifies the channel the attached to this adapter, to which messages will be sent.
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
<xsd:attributeGroup ref="integration:smartLifeCycleAttributeGroup"/>
<xsd:attribute name="twitter-template" type="xsd:string">
<xsd:annotation>
<xsd:appinfo>
@@ -136,6 +157,21 @@
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
<xsd:attribute name="metadata-store" use="optional" type="xsd:string">
<xsd:annotation>
<xsd:documentation>
Reference to a MetadataStore instance for storing metadata associated with
the retrieved feeds. If the implementation is persistent, it can help to
prevent duplicates between restarts. If shared, it can help coordinate multiple
instances of an adapter across different processes.
</xsd:documentation>
<xsd:appinfo>
<tool:annotation kind="ref">
<tool:expected-type type="org.springframework.integration.metadata.MetadataStore" />
</tool:annotation>
</xsd:appinfo>
</xsd:annotation>
</xsd:attribute>
</xsd:complexType>
<xsd:complexType name="outbound-twitter-type">

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2002-2011 the original author or authors.
* Copyright 2002-2013 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -16,7 +16,10 @@
package org.springframework.integration.twitter.config;
import static org.junit.Assert.assertNotNull;
import org.junit.Test;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.springframework.integration.endpoint.SourcePollingChannelAdapter;
@@ -25,16 +28,13 @@ import org.springframework.integration.twitter.inbound.DirectMessageReceivingMes
import org.springframework.integration.twitter.inbound.MentionsReceivingMessageSource;
import org.springframework.integration.twitter.inbound.TimelineReceivingMessageSource;
import static org.junit.Assert.assertNotNull;
/**
* @author Oleg Zhurakousky
* @author Gunnar Hillert
*/
public class TestReceivingMessageSourceParserTests {
@Test
public void testReceivingAdapterConfigurationAutoStartup(){
ApplicationContext ac = new ClassPathXmlApplicationContext("TestReceivingMessageSourceParser-context.xml", this.getClass());
@@ -47,10 +47,24 @@ public class TestReceivingMessageSourceParserTests {
assertNotNull(dms);
spca = ac.getBean("updateAdapter", SourcePollingChannelAdapter.class);
spca = ac.getBean("updateAdapter", SourcePollingChannelAdapter.class);
TimelineReceivingMessageSource tms = TestUtils.getPropertyValue(spca, "source", TimelineReceivingMessageSource.class);
assertNotNull(tms);
}
@Test
public void testThatMessageSourcesAreRegisteredAsBeans(){
ApplicationContext ac = new ClassPathXmlApplicationContext("TestReceivingMessageSourceParser-context.xml", this.getClass());
MentionsReceivingMessageSource ms = ac.getBean("mentionAdapter.source", MentionsReceivingMessageSource.class);
assertNotNull(ms);
DirectMessageReceivingMessageSource dms = ac.getBean("dmAdapter.source", DirectMessageReceivingMessageSource.class);
assertNotNull(dms);
TimelineReceivingMessageSource tms = ac.getBean("updateAdapter.source", TimelineReceivingMessageSource.class);
assertNotNull(tms);
}
}

View File

@@ -31,7 +31,7 @@ import org.springframework.social.twitter.api.impl.TwitterTemplate;
*/
public class DirectMessageReceivingMessageSourceTests {
@SuppressWarnings("unchecked")
@Test @Ignore
public void demoReceiveDm() throws Exception{
@@ -40,11 +40,11 @@ public class DirectMessageReceivingMessageSourceTests {
pf.afterPropertiesSet();
Properties prop = pf.getObject();
System.out.println(prop);
TwitterTemplate template = new TwitterTemplate(prop.getProperty("z_oleg.oauth.consumerKey"),
prop.getProperty("z_oleg.oauth.consumerSecret"),
prop.getProperty("z_oleg.oauth.accessToken"),
TwitterTemplate template = new TwitterTemplate(prop.getProperty("z_oleg.oauth.consumerKey"),
prop.getProperty("z_oleg.oauth.consumerSecret"),
prop.getProperty("z_oleg.oauth.accessToken"),
prop.getProperty("z_oleg.oauth.accessTokenSecret"));
DirectMessageReceivingMessageSource tSource = new DirectMessageReceivingMessageSource(template);
DirectMessageReceivingMessageSource tSource = new DirectMessageReceivingMessageSource(template, "foo");
tSource.afterPropertiesSet();
for (int i = 0; i < 50; i++) {
Message<DirectMessage> message = (Message<DirectMessage>) tSource.receive();

View File

@@ -35,7 +35,7 @@ import org.junit.Test;
import org.springframework.beans.factory.config.PropertiesFactoryBean;
import org.springframework.core.io.ClassPathResource;
import org.springframework.integration.Message;
import org.springframework.integration.store.metadata.SimpleMetadataStore;
import org.springframework.integration.metadata.SimpleMetadataStore;
import org.springframework.integration.test.util.TestUtils;
import org.springframework.social.twitter.api.SearchMetadata;
import org.springframework.social.twitter.api.SearchOperations;
@@ -66,7 +66,7 @@ public class SearchReceivingMessageSourceTests {
prop.getProperty("z_oleg.oauth.consumerSecret"),
prop.getProperty("z_oleg.oauth.accessToken"),
prop.getProperty("z_oleg.oauth.accessTokenSecret"));
SearchReceivingMessageSource tSource = new SearchReceivingMessageSource(template);
SearchReceivingMessageSource tSource = new SearchReceivingMessageSource(template, "foo");
tSource.setQuery(SEARCH_QUERY);
tSource.afterPropertiesSet();
for (int i = 0; i < 50; i++) {
@@ -84,14 +84,14 @@ public class SearchReceivingMessageSourceTests {
@Test
public void testSearchReceivingMessageSourceInit() {
final SearchReceivingMessageSource messageSource = new SearchReceivingMessageSource(new TwitterTemplate("test"));
final SearchReceivingMessageSource messageSource = new SearchReceivingMessageSource(new TwitterTemplate("test"), "foo");
messageSource.setComponentName("twitterSearchMessageSource");
final Object metadataStore = TestUtils.getPropertyValue(messageSource, "metadataStore");
final Object metadataKey = TestUtils.getPropertyValue(messageSource, "metadataKey");
assertNull(metadataStore);
assertNull(metadataKey);
assertNotNull(metadataKey);
messageSource.afterPropertiesSet();
@@ -101,7 +101,7 @@ public class SearchReceivingMessageSourceTests {
assertNotNull(metadataStoreInitialized);
assertTrue(metadataStoreInitialized instanceof SimpleMetadataStore);
assertNotNull(metadataKeyInitialized);
assertEquals("twitter:search-inbound-channel-adapter.twitterSearchMessageSource", metadataKeyInitialized);
assertEquals("foo", metadataKeyInitialized);
final Twitter twitter = TestUtils.getPropertyValue(messageSource, "twitter", Twitter.class);
@@ -123,7 +123,7 @@ public class SearchReceivingMessageSourceTests {
when(twitterTemplate.searchOperations()).thenReturn(so);
when(twitterTemplate.searchOperations().search(SEARCH_QUERY, 20, 0, 0)).thenReturn(null);
final SearchReceivingMessageSource messageSource = new SearchReceivingMessageSource(twitterTemplate);
final SearchReceivingMessageSource messageSource = new SearchReceivingMessageSource(twitterTemplate, "foo");
messageSource.setQuery(SEARCH_QUERY);
final String setQuery = TestUtils.getPropertyValue(messageSource, "query", String.class);
@@ -165,7 +165,7 @@ public class SearchReceivingMessageSourceTests {
SearchParameters params = new SearchParameters(SEARCH_QUERY).count(20).sinceId(0);
when(twitterTemplate.searchOperations().search(params)).thenReturn(results);
final SearchReceivingMessageSource messageSource = new SearchReceivingMessageSource(twitterTemplate);
final SearchReceivingMessageSource messageSource = new SearchReceivingMessageSource(twitterTemplate, "foo");
messageSource.setQuery(SEARCH_QUERY);

View File

@@ -1,27 +1,27 @@
<?xml version="1.0" encoding="UTF-8"?>
<beans
xmlns="http://www.springframework.org/schema/beans"
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-twitter="http://www.springframework.org/schema/integration/twitter"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/integration/twitter http://www.springframework.org/schema/integration/twitter/spring-integration-twitter.xsd">
<int:channel id="inbound_twitter"/>
<int:channel id="inbound_twitter">
<int:queue/>
</int:channel>
<int-twitter:search-inbound-channel-adapter id="twitterSearchAdapter"
query="springintegration"
twitter-template="twitterTemplate"
channel="inbound_twitter"
metadata-store="redisMetadataStore"
auto-startup="false">
<int:poller fixed-rate="5000" max-messages-per-poll="3"/>
<int:poller fixed-rate="100" max-messages-per-poll="3"/>
</int-twitter:search-inbound-channel-adapter>
<bean id="metadataStore" class="org.springframework.integration.redis.store.metadata.RedisMetadataStore">
<bean id="redisMetadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
<constructor-arg name="connectionFactory" ref="redisConnectionFactory"/>
</bean>

View File

@@ -19,6 +19,7 @@ package org.springframework.integration.twitter.inbound;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertSame;
import static org.junit.Assert.assertTrue;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.mock;
@@ -30,47 +31,61 @@ import java.util.List;
import org.junit.Before;
import org.junit.Test;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.ImportResource;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.integration.Message;
import org.springframework.integration.core.PollableChannel;
import org.springframework.integration.endpoint.SourcePollingChannelAdapter;
import org.springframework.integration.redis.rules.RedisAvailable;
import org.springframework.integration.redis.rules.RedisAvailableTests;
import org.springframework.integration.redis.store.metadata.RedisMetadataStore;
import org.springframework.integration.store.metadata.MetadataStore;
import org.springframework.integration.redis.metadata.RedisMetadataStore;
import org.springframework.integration.metadata.MetadataStore;
import org.springframework.integration.test.util.TestUtils;
import org.springframework.social.twitter.api.SearchMetadata;
import org.springframework.social.twitter.api.SearchOperations;
import org.springframework.social.twitter.api.SearchResults;
import org.springframework.social.twitter.api.Tweet;
import org.springframework.social.twitter.api.UserOperations;
import org.springframework.social.twitter.api.impl.SearchParameters;
import org.springframework.social.twitter.api.impl.TwitterTemplate;
/**
* @author Gunnar Hillert
* @author Artem Bilan
* @since 3.0
*/
public class SearchReceivingMessageSourceWithRedisTests extends RedisAvailableTests {
private SourcePollingChannelAdapter twitterSearchAdapter;
private RedisConnectionFactory redisConnectionFactory;
private StringRedisTemplate redisTemplate;
private AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
private AbstractTwitterMessageSource twitterMessageSource;
private MetadataStore metadataStore;
private String metadataKey;
private PollableChannel tweets;
@Before
public void setup() {
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
context.register(SearchReceivingMessageSourceWithRedisTestsConfig.class);
context.registerShutdownHook();
context.refresh();
this.redisConnectionFactory = context.getBean(RedisConnectionFactory.class);
this.twitterSearchAdapter = context.getBean(SourcePollingChannelAdapter.class);
this.redisTemplate = new StringRedisTemplate(redisConnectionFactory);
this.twitterMessageSource = context.getBean(AbstractTwitterMessageSource.class);
this.metadataStore = context.getBean(MetadataStore.class);
this.tweets = context.getBean("inbound_twitter", PollableChannel.class);
this.metadataKey = TestUtils.getPropertyValue(twitterSearchAdapter, "source.metadataKey", String.class);
// There is need to set a value, not 'remove' and re-init 'twitterMessageSource'
this.metadataStore.put(metadataKey, "-1");
this.twitterMessageSource.afterPropertiesSet();
}
/**
@@ -80,51 +95,43 @@ public class SearchReceivingMessageSourceWithRedisTests extends RedisAvailableTe
@Test
@RedisAvailable
public void testPollForTweetsThreeResultsWithRedisMetadataStore() throws Exception {
final MetadataStore metadataStore = TestUtils.getPropertyValue(twitterSearchAdapter, "source.metadataStore", MetadataStore.class);
MetadataStore metadataStore = TestUtils.getPropertyValue(this.twitterSearchAdapter, "source.metadataStore", MetadataStore.class);
assertTrue("Exptected metadataStore to be an instance of RedisMetadataStore", metadataStore instanceof RedisMetadataStore);
assertSame(this.metadataStore, metadataStore);
/*
* The metadataKey is automatically generated. To ensure that we use the
* the correct key, we retrieve it from the adapter.
*/
final String metadataKey = TestUtils.getPropertyValue(twitterSearchAdapter, "source.metadataKey", String.class);
assertEquals("twitterSearchAdapter.74", metadataKey);
/*
* As we had to retrieve the metadataKey from the adapter. The metdataStore
* was already invoked and the id retrieved from Redis before we had a chance
* to reset possibly pre-existing values.
*
* Rather than deleting the value, we have to set a value, because "null" values
* returned from the MetadataStore are ignored by the onInit() method in
* the AbstractTwitterMessageSource. */
redisTemplate.opsForValue().set(metadataKey, "-1");
assertEquals("-1", redisTemplate.opsForValue().get(metadataKey));
this.twitterSearchAdapter.start();
final SearchReceivingMessageSource source = TestUtils.getPropertyValue(twitterSearchAdapter, "source", SearchReceivingMessageSource.class);
/* We need to call onInit() in order to update the id from the metadataStore. */
source.onInit();
final Message<?> message1 = source.receive();
final Message<?> message2 = source.receive();
final Message<?> message3 = source.receive();
assertNotNull(this.tweets.receive(10000));
assertNotNull(this.tweets.receive(1000));
assertNotNull(this.tweets.receive(1000));
/* We received 3 messages so far. When invoking receive() again the search
* will return again the 3 test Tweets but as we already processed them
* no message (null) is returned. */
final Message<?> message4 = source.receive();
assertNull(this.tweets.receive(0));
assertNotNull(message1);
assertNotNull(message2);
assertNotNull(message3);
assertNull(message4);
final String persistedMetadataStoreValue = redisTemplate.opsForValue().get(metadataKey);
String persistedMetadataStoreValue = this.metadataStore.get(metadataKey);
assertNotNull(persistedMetadataStoreValue);
assertEquals("3", redisTemplate.opsForValue().get(metadataKey));
assertEquals("3", persistedMetadataStoreValue);
redisTemplate.delete(metadataKey);
this.twitterSearchAdapter.stop();
this.metadataStore.put(metadataKey, "1");
this.twitterMessageSource.afterPropertiesSet();
this.twitterSearchAdapter.start();
assertNotNull(this.tweets.receive(1000));
assertNotNull(this.tweets.receive(1000));
assertNull(this.tweets.receive(0));
persistedMetadataStoreValue = this.metadataStore.get(metadataKey);
assertNotNull(persistedMetadataStoreValue);
assertEquals("3", persistedMetadataStoreValue);
}
@Configuration
@@ -152,7 +159,15 @@ public class SearchReceivingMessageSourceWithRedisTests extends RedisAvailableTe
when(twitterTemplate.searchOperations()).thenReturn(so);
when(twitterTemplate.searchOperations().search(any(SearchParameters.class))).thenReturn(results);
when(twitterTemplate.isAuthorized()).thenReturn(true);
final UserOperations userOperations = mock(UserOperations.class);
when(twitterTemplate.userOperations()).thenReturn(userOperations);
when(userOperations.getProfileId()).thenReturn(74L);
return twitterTemplate;
}
}
}

View File

@@ -32,7 +32,7 @@ import org.springframework.social.twitter.api.impl.TwitterTemplate;
*/
public class TimelineReceivingMessageSourceTests {
@SuppressWarnings("unchecked")
@Test @Ignore
public void demoReceiveTimeline() throws Exception{
@@ -41,11 +41,11 @@ public class TimelineReceivingMessageSourceTests {
pf.afterPropertiesSet();
Properties prop = pf.getObject();
System.out.println(prop);
TwitterTemplate template = new TwitterTemplate(prop.getProperty("z_oleg.oauth.consumerKey"),
prop.getProperty("z_oleg.oauth.consumerSecret"),
prop.getProperty("z_oleg.oauth.accessToken"),
TwitterTemplate template = new TwitterTemplate(prop.getProperty("z_oleg.oauth.consumerKey"),
prop.getProperty("z_oleg.oauth.consumerSecret"),
prop.getProperty("z_oleg.oauth.accessToken"),
prop.getProperty("z_oleg.oauth.accessTokenSecret"));
TimelineReceivingMessageSource tSource = new TimelineReceivingMessageSource(template);
TimelineReceivingMessageSource tSource = new TimelineReceivingMessageSource(template, "foo");
tSource.afterPropertiesSet();
for (int i = 0; i < 50; i++) {
Message<Tweet> message = (Message<Tweet>) tSource.receive();

View File

@@ -62,14 +62,14 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
Spring Integration provides a convenient mechanism to eliminate the need to worry about duplicate entries.
Each feed entry will have a <emphasis>published date</emphasis> field. Every time a new Message is generated and sent,
Spring Integration will store the value of the latest <emphasis>published date</emphasis> in an instance of the
<classname>org.springframework.integration.store.MetadataStore</classname> strategy. The MetadataStore interface is
<classname>org.springframework.integration.metadata.MetadataStore</classname> strategy. The MetadataStore interface is
designed to store various types of generic meta-data (e.g., published date of the last feed entry that has been processed)
to help components such as this Feed adapter deal with duplicates.
</para>
<para>
The default rule for locating this metadata store is as follows:
<emphasis>Spring Integration</emphasis> will look for a bean of type
<classname>org.springframework.integration.store.MetadataStore</classname> in
<classname>org.springframework.integration.metadata.MetadataStore</classname> in
the ApplicationContext. If one is found then it will be used, otherwise
it will create a new instance of <classname>SimpleMetadataStore</classname>
which is an in-memory implementation that will only persist metadata within
@@ -107,5 +107,10 @@ xsi:schemaLocation="http://www.springframework.org/schema/integration/feed
<interfacename>MetadataStore</interfacename> interface (e.g. JdbcMetadataStore)
and configure it as bean in the Application Context.
</para>
<note>
The key used to persist the latest <emphasis>published date</emphasis> is the value of the (required)
<code>id</code> attribute of the Feed Inbound Channel Adapter component plus the <code>feedUrl</code>
from the adapter's configuration.
</note>
</section>
</chapter>

View File

@@ -141,11 +141,12 @@ twitter.oauth.accessTokenSecret=AbRxUAvyNCtqQtxFK8w5ZMtMj20KFhB6o]]></programlis
criteria you'll end up with the same set of tweets unless some other new tweet that matches your search criteria was posted
in between your searches. In that situation you'll get all the tweets you had before plus the new one. But what you really
want is only the new tweet(s). Spring Integration provides an elegant mechanism for handling these situations.
The latest Tweet timestamp will be stored in an instance of the <classname>org.springframework.integration.store.MetadataStore</classname> which is a
The latest Tweet id will be stored in an instance of the <classname>org.springframework.integration.metadata.MetadataStore</classname> which is a
strategy interface designed for storing various types of metadata (e.g., last retrieved tweet in this case). That strategy helps components such as
these Twitter adapters avoid duplicates. By default, Spring Integration will look for a bean of type
<classname>org.springframework.integration.store.MetadataStore</classname> in the ApplicationContext.
If one is found then it will be used, otherwise it will create a new instance of <classname>SimpleMetadataStore</classname>
<classname>org.springframework.integration.metadata.MetadataStore</classname> in the ApplicationContext. Alternatively,
you can configure an explicit <classname>MetadataStore</classname> on the adapter.
If there is no explicit or default store, the adapter will create a new instance of <classname>SimpleMetadataStore</classname>
which is a simple in-memory implementation that will only persist metadata within the lifecycle of the currently running application context.
That means upon restart you may end up with duplicate entries. If you need to persist metadata between Application Context
restarts, you may use the <classname>PropertiesPersistingMetadataStore</classname> (which is backed by a properties file, and a persister
@@ -165,8 +166,16 @@ twitter.oauth.accessTokenSecret=AbRxUAvyNCtqQtxFK8w5ZMtMj20KFhB6o]]></programlis
</warning>
<programlisting language="xml"><![CDATA[<bean id="metadataStore" class="o.s.i.store.PropertiesPersistingMetadataStore"/>
]]></programlisting>
The Poller that is configured as part of any Inbound Twitter Adapter (see below) will simply poll from this MetadataStore to determine the latest tweet
received.
<para>
If the <classname>MetadataStore</classname> is persistent, during initialization, any Inbound Twitter Adapter (see below)
will retrieve the latest tweet id that has already been sent by the adapter.
</para>
<note>
The key used to persist the latest <emphasis>twitter id</emphasis> is the value of the (required)
<code>id</code> attribute of the Twitter Inbound Channel Adapter component plus the <code>profileId</code>
of the Twitter user.
</note>
<section id="inbound-twitter-update">
<title>Inbound Message Channel Adapter</title>
<para>