Twitter functional apps
* Streaming source * Add twitter search and messasge sources * Add more twitter source, sink and processors * Add IT tests for twitter update sink * Add IT tests for twitter message sink * Add IT tests for twitter trend processor * twitter suppliers readme * twitter consumers readme * twitter functions readme * Disable TwitterStreamSourceTests
This commit is contained in:
Christian Tzolov
committed by
Soby Chacko
Soby Chacko
parent
9bf8dcc4e7
commit
873f17414e
11
applications/processor/bridge-processor/README.adoc.tmp
Normal file
11
applications/processor/bridge-processor/README.adoc.tmp
Normal file
@@ -0,0 +1,11 @@
|
||||
//tag::ref-doc[]
|
||||
= Bridge Processor
|
||||
|
||||
A processor that bridges the input and ouput by simply passing the incoming payload to the outbound.
|
||||
|
||||
=== Payload
|
||||
|
||||
Any
|
||||
|
||||
//end::ref-doc[]
|
||||
null
|
||||
@@ -18,6 +18,7 @@
|
||||
<module>splitter-processor</module>
|
||||
<module>transform-processor</module>
|
||||
<module>script-processor</module>
|
||||
<module>twitter-trend-processor</module>
|
||||
</modules>
|
||||
|
||||
</project>
|
||||
|
||||
36
applications/processor/twitter-trend-processor/README.adoc
Normal file
36
applications/processor/twitter-trend-processor/README.adoc
Normal file
@@ -0,0 +1,36 @@
|
||||
//tag::ref-doc[]
|
||||
= Twitter Trend and Trend Locations Processor
|
||||
|
||||
Processor that can return either trending topic or the Locations of the trending topics.
|
||||
The `twitter.trend.trend-query-type` property allow to select the query type.
|
||||
|
||||
== Retrieve trending topic in a location (optionally)
|
||||
|
||||
For this mode set `twitter.trend.trend-query-type` to `trend`.
|
||||
|
||||
Processor based on https://developer.twitter.com/en/docs/trends/trends-for-location/api-reference/get-trends-place[Trends API].
|
||||
Returns the https://help.twitter.com/en/using-twitter/twitter-trending-faqs[trending topics] near a specific latitude, longitude location.
|
||||
|
||||
== Retrieve trend Locations
|
||||
|
||||
For this mode set `twitter.trend.trend-query-type` to `trendLocation`.
|
||||
|
||||
Retrieve a full or nearby locations list of trending topics by location.
|
||||
|
||||
If the `latitude`, `longitude` parameters are NOT provided the processor performs the https://developer.twitter.com/en/docs/trends/locations-with-trending-topics/api-reference/get-trends-available[Trends Available API] and returns the locations that Twitter has trending topic information for.
|
||||
|
||||
If the `latitude`, `longitude` parameters are provided the processor performs the https://developer.twitter.com/en/docs/trends/locations-with-trending-topics/api-reference/get-trends-closest[Trends Closest API] and returns the locations that Twitter has trending topic information for, closest to a specified location.
|
||||
|
||||
Response is an array of `locations` that encode the location's WOEID and some other human-readable information such as a canonical name and country the location belongs in.
|
||||
|
||||
== Options
|
||||
|
||||
//tag::configuration-properties[]
|
||||
$$twitter.trend.closest.lat$$:: $$If provided with a long parameter the available trend locations will be sorted by distance, nearest to furthest, to the co-ordinate pair. The valid ranges for longitude is -180.0 to +180.0 (West is negative, East is positive) inclusive.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.trend.closest.lon$$:: $$If provided with a lat parameter the available trend locations will be sorted by distance, nearest to furthest, to the co-ordinate pair. The valid ranges for longitude is -180.0 to +180.0 (West is negative, East is positive) inclusive.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.trend.location-id$$:: $$The Yahoo! Where On Earth ID of the location to return trending information for. Global information is available by using 1 as the WOEID.$$ *($$Expression$$, default: `$$payload$$`)*
|
||||
$$twitter.trend.trend-query-type$$:: $$<documentation missing>$$ *($$TrendQueryType$$, default: `$$<none>$$`, possible values: `trend`,`trendLocation`)*
|
||||
//end::configuration-properties[]
|
||||
|
||||
//end::ref-doc[]
|
||||
|
||||
103
applications/processor/twitter-trend-processor/pom.xml
Normal file
103
applications/processor/twitter-trend-processor/pom.xml
Normal file
@@ -0,0 +1,103 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-trend-processor</artifactId>
|
||||
<name>twitter-trend-processor</name>
|
||||
<description>twitter trend processor apps</description>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.stream.app</groupId>
|
||||
<artifactId>stream-applications-core</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-function</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<groupId>org.junit.vintage</groupId>
|
||||
<artifactId>junit-vintage-engine</artifactId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-json</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud</groupId>
|
||||
<artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud.stream.app.plugin</groupId>
|
||||
<artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<generatedApp>
|
||||
<name>twitter-trend</name>
|
||||
<type>processor</type>
|
||||
<version>${project.version}</version>
|
||||
<configClass>org.springframework.cloud.fn.twitter.trend.TwitterTrendFunctionConfiguration.class</configClass>
|
||||
<functionDefinition>trendOrTrendLocationsFunction</functionDefinition>
|
||||
</generatedApp>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-function</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
<id>spring-snapshots</id>
|
||||
<name>Spring Snapshots</name>
|
||||
<url>https://repo.spring.io/libs-snapshot-local</url>
|
||||
</repository>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>false</enabled>
|
||||
</snapshots>
|
||||
<id>spring-milestones</id>
|
||||
<name>Spring Milestones</name>
|
||||
<url>https://repo.spring.io/libs-milestone-local</url>
|
||||
</repository>
|
||||
</repositories>
|
||||
</project>
|
||||
@@ -0,0 +1,3 @@
|
||||
configuration-properties.classes=org.springframework.cloud.fn.twitter.trend.TwitterTrendFunctionProperties, \
|
||||
org.springframework.cloud.fn.twitter.trend.TwitterTrendFunctionProperties$Closest
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.processor.twitter.trend;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.function.Function;
|
||||
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTestUtils {
|
||||
|
||||
public Function<ConfigurationBuilder, ConfigurationBuilder> mockTwitterUrls(String baseUrl) {
|
||||
return configBuilder -> {
|
||||
configBuilder.setRestBaseURL(baseUrl + "/");
|
||||
configBuilder.setStreamBaseURL(baseUrl + "/stream/");
|
||||
configBuilder.setUserStreamBaseURL(baseUrl + "/user/");
|
||||
configBuilder.setSiteStreamBaseURL(baseUrl + "/site/");
|
||||
configBuilder.setUploadBaseURL(baseUrl + "/upload/");
|
||||
|
||||
configBuilder.setOAuthAccessTokenURL(baseUrl + "/oauth/access_token");
|
||||
configBuilder.setOAuthAuthenticationURL(baseUrl + "/oauth/authenticate");
|
||||
configBuilder.setOAuthAuthorizationURL(baseUrl + "/oauth/authorize");
|
||||
configBuilder.setOAuthRequestTokenURL(baseUrl + "/oauth/request_token");
|
||||
configBuilder.setOAuth2TokenURL(baseUrl + "/oauth2/token");
|
||||
configBuilder.setOAuth2InvalidateTokenURL(baseUrl + "/oauth2/invalidate_token");
|
||||
|
||||
return configBuilder;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load Spring Resource as String.
|
||||
* @param resourcePath Resource path (accepts file:// , classpath:// and http:// uri schemas)
|
||||
* @return Returns text (UTF8) representation of the resource pointed by the resourcePath
|
||||
*/
|
||||
public static String asString(String resourcePath) {
|
||||
try {
|
||||
return StreamUtils.copyToString(new DefaultResourceLoader().getResource(resourcePath).getInputStream(),
|
||||
Charset.forName("UTF-8"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Can not load resource:" + resourcePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.processor.twitter.trend;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.Header;
|
||||
import org.mockserver.model.HttpRequest;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.cloud.fn.twitter.trend.TwitterTrendFunctionConfiguration;
|
||||
import org.springframework.cloud.stream.binder.test.InputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.OutputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.messaging.support.GenericMessage;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.exactly;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTrendLocationProcessorIntegrationTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
private static HttpRequest availableTrendsRequest;
|
||||
private static HttpRequest closestTrendsRequest;
|
||||
|
||||
@BeforeEach
|
||||
public void startServer() {
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
|
||||
availableTrendsRequest = setExpectation(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/trends/available.json"));
|
||||
|
||||
closestTrendsRequest = setExpectation(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/trends/closest.json")
|
||||
.withQueryStringParameter("lat", "52.379189")
|
||||
.withQueryStringParameter("long", "4.899431"));
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
public void stopServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testTwitterAvailableTrends() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TwitterTrendProcessorIntegrationTests.TestTwitterTrendProcessorApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=trendOrTrendLocationsFunction",
|
||||
|
||||
"--twitter.trend.trendQueryType=trendLocation",
|
||||
"--twitter.connection.rawJson=false",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination input = context.getBean(InputDestination.class);
|
||||
OutputDestination output = context.getBean(OutputDestination.class);
|
||||
assertThat(input).isNotNull();
|
||||
assertThat(output).isNotNull();
|
||||
|
||||
input.send(new GenericMessage<>("hello".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
Message<byte[]> outputMessage = output.receive(Duration.ofSeconds(300).toMillis());
|
||||
assertThat(outputMessage).isNotNull();
|
||||
|
||||
mockClient.verify(availableTrendsRequest, once());
|
||||
assertThat(outputMessage);
|
||||
|
||||
String payload = new String(outputMessage.getPayload());
|
||||
assertThat(payload).containsSequence("countryName");
|
||||
assertThat(payload).contains("placeCode");
|
||||
assertThat(payload).doesNotContain("placeType");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testTwitterAvailableTrendsTwitterJson() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TwitterTrendProcessorIntegrationTests.TestTwitterTrendProcessorApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=trendOrTrendLocationsFunction",
|
||||
|
||||
"--twitter.trend.trendQueryType=trendLocation",
|
||||
"--twitter.connection.rawJson=true",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination input = context.getBean(InputDestination.class);
|
||||
OutputDestination output = context.getBean(OutputDestination.class);
|
||||
assertThat(input).isNotNull();
|
||||
assertThat(output).isNotNull();
|
||||
|
||||
input.send(new GenericMessage<>("hello".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
Message<byte[]> outputMessage = output.receive(Duration.ofSeconds(300).toMillis());
|
||||
assertThat(outputMessage).isNotNull();
|
||||
|
||||
mockClient.verify(availableTrendsRequest, once());
|
||||
assertThat(outputMessage).isNotNull();
|
||||
|
||||
String payload = new String(outputMessage.getPayload());
|
||||
|
||||
assertThat(payload).contains("placeType");
|
||||
assertThat(payload).doesNotContain("placeCode");
|
||||
assertThat(payload).doesNotContain("countryName");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testTwitterClosestTrends() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TwitterTrendProcessorIntegrationTests.TestTwitterTrendProcessorApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=trendOrTrendLocationsFunction",
|
||||
|
||||
"--twitter.trend.trendQueryType=trendLocation",
|
||||
"--twitter.connection.rawJson=true",
|
||||
"--twitter.trend.closest.lat='52.379189'",
|
||||
"--twitter.trend.closest.lon='4.899431'",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination input = context.getBean(InputDestination.class);
|
||||
OutputDestination output = context.getBean(OutputDestination.class);
|
||||
assertThat(input).isNotNull();
|
||||
assertThat(output).isNotNull();
|
||||
|
||||
input.send(new GenericMessage<>("hello".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
Message<byte[]> outputMessage = output.receive(Duration.ofSeconds(300).toMillis());
|
||||
assertThat(outputMessage).isNotNull();
|
||||
|
||||
mockClient.verify(closestTrendsRequest, once());
|
||||
assertThat(outputMessage).isNotNull();
|
||||
}
|
||||
}
|
||||
|
||||
public static HttpRequest setExpectation(HttpRequest request) {
|
||||
mockClient
|
||||
.when(request, exactly(1))
|
||||
.respond(response()
|
||||
.withStatusCode(200)
|
||||
.withHeaders(
|
||||
new Header("Content-Type", "application/json; charset=utf-8"),
|
||||
new Header("Cache-Control", "public, max-age=86400"))
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/trend_locations.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 1)
|
||||
);
|
||||
return request;
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterTrendFunctionConfiguration.class)
|
||||
public static class TestTwitterTrendLocationProcessorApplication {
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.processor.twitter.trend;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.Header;
|
||||
import org.mockserver.model.HttpRequest;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.cloud.fn.twitter.trend.TwitterTrendFunctionConfiguration;
|
||||
import org.springframework.cloud.stream.binder.test.InputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.OutputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.messaging.support.GenericMessage;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.exactly;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTrendProcessorIntegrationTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
private static HttpRequest trendsRequest;
|
||||
|
||||
@BeforeAll
|
||||
public static void startServer() {
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
|
||||
trendsRequest = setExpectation(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/trends/place.json")
|
||||
.withQueryStringParameter("id", "2972"));
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
public static void stopServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testTwitterTrendPayload() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterTrendProcessorApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=trendOrTrendLocationsFunction",
|
||||
|
||||
"--twitter.trend.locationId='2972'",
|
||||
"--twitter.connection.rawJson=true",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination input = context.getBean(InputDestination.class);
|
||||
OutputDestination output = context.getBean(OutputDestination.class);
|
||||
assertThat(input).isNotNull();
|
||||
assertThat(output).isNotNull();
|
||||
|
||||
input.send(new GenericMessage<>("Hello".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
Message<byte[]> outputMessage = output.receive(Duration.ofSeconds(300).toMillis());
|
||||
assertThat(outputMessage).isNotNull();
|
||||
|
||||
mockClient.verify(trendsRequest, once());
|
||||
|
||||
//Resource trendsResource = new DefaultResourceLoader().getResource("classpath:/response/trends.json");
|
||||
//String expected = new String(StreamUtils.copyToByteArray(trendsResource.getInputStream()), StandardCharsets.UTF_8).trim();
|
||||
//String actual = new String(outputMessage.getPayload(), StandardCharsets.UTF_8);
|
||||
//JSONAssert.assertEquals(expected, actual, JSONCompareMode.LENIENT);
|
||||
}
|
||||
}
|
||||
|
||||
public static HttpRequest setExpectation(HttpRequest request) {
|
||||
mockClient
|
||||
.when(request, exactly(1))
|
||||
.respond(response()
|
||||
.withStatusCode(200)
|
||||
.withHeaders(
|
||||
new Header("Content-Type", "application/json; charset=utf-8"),
|
||||
new Header("Cache-Control", "public, max-age=86400"))
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/trends.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 1)
|
||||
);
|
||||
return request;
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterTrendFunctionConfiguration.class)
|
||||
public static class TestTwitterTrendProcessorApplication {
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -28,5 +28,7 @@
|
||||
<module>tcp-sink</module>
|
||||
<module>throughput-sink</module>
|
||||
<module>websocket-sink</module>
|
||||
<module>twitter-update-sink</module>
|
||||
<module>twitter-message-sink</module>
|
||||
</modules>
|
||||
</project>
|
||||
|
||||
51
applications/sink/twitter-message-sink/README.adoc
Normal file
51
applications/sink/twitter-message-sink/README.adoc
Normal file
@@ -0,0 +1,51 @@
|
||||
//tag::ref-doc[]
|
||||
= Twitter Message Sink
|
||||
|
||||
Send Direct Messages to a specified user from the authenticating user.
|
||||
Requires a JSON POST body and `Content-Type` header to be set to `application/json`.
|
||||
|
||||
NOTE: When a message is received from a user you may send up to 5 messages in response within a 24 hour window.
|
||||
Each message received resets the 24 hour window and the 5 allotted messages.
|
||||
Sending a 6th message within a 24 hour window or sending a message outside of a 24 hour window will count towards rate-limiting.
|
||||
This behavior only applies when using the POST direct_messages/events/new endpoint.
|
||||
|
||||
SpEL expressions are used to compute the request parameters from the input message.
|
||||
|
||||
== Options
|
||||
TIP: Use single quotes (`'`) to wrap the literal values of the `SpEL` expression properties.
|
||||
For example to set a fixed message text use `text='Fixed Text'`.
|
||||
For fixed target userId use `userId='666'`.
|
||||
|
||||
//tag::configuration-properties[]
|
||||
$$twitter.message.update.media-id$$:: $$A media id to associate with the message. A Direct Message may only reference a single media id.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.message.update.screen-name$$:: $$The screen name of the user to whom send the direct message.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.message.update.text$$:: $$The direct message text. URL encode as necessary. Max length of 10,000 characters.$$ *($$Expression$$, default: `$$payload$$`)*
|
||||
$$twitter.message.update.user-id$$:: $$The user id of the user to whom send the direct message.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
//end::configuration-properties[]
|
||||
|
||||
//end::ref-doc[]
|
||||
|
||||
|
||||
== Examples
|
||||
|
||||
```
|
||||
java -jar twitter-message-sink.jar
|
||||
|
||||
--twitter.message.update.userId=headers['user']
|
||||
--twitter.message.update.text=payload.concat(\" with suffix \")
|
||||
|
||||
--twitter.connection.consumerKey= ...
|
||||
--twitter.connection.consumerSecret= ...
|
||||
--twitter.connection.accessToken= ...
|
||||
--twitter.connection.accessTokenSecret= ...
|
||||
|
||||
--
|
||||
|
||||
```
|
||||
|
||||
And here is a example pipeline that uses twitter-message:
|
||||
|
||||
```
|
||||
twitter-message-stream= TODO
|
||||
```
|
||||
|
||||
103
applications/sink/twitter-message-sink/pom.xml
Normal file
103
applications/sink/twitter-message-sink/pom.xml
Normal file
@@ -0,0 +1,103 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-message-sink</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<name>twitter-message-sink</name>
|
||||
<description>twitter message sink apps</description>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.stream.app</groupId>
|
||||
<artifactId>stream-applications-core</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-consumer</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.awaitility</groupId>
|
||||
<artifactId>awaitility</artifactId>
|
||||
<scope>test</scope>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud</groupId>
|
||||
<artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud.stream.app.plugin</groupId>
|
||||
<artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<generatedApp>
|
||||
<name>twitter-message</name>
|
||||
<type>sink</type>
|
||||
<version>${project.version}</version>
|
||||
<configClass>org.springframework.cloud.fn.consumer.twitter.message.TwitterMessageConsumerConfiguration.class</configClass>
|
||||
<functionDefinition>byteArrayTextToString|sendDirectMessageConsumer</functionDefinition>
|
||||
</generatedApp>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-consumer</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</configuration>
|
||||
</plugin>
|
||||
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
<id>spring-snapshots</id>
|
||||
<name>Spring Snapshots</name>
|
||||
<url>https://repo.spring.io/libs-snapshot-local</url>
|
||||
</repository>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>false</enabled>
|
||||
</snapshots>
|
||||
<id>spring-milestones</id>
|
||||
<name>Spring Milestones</name>
|
||||
<url>https://repo.spring.io/libs-milestone-local</url>
|
||||
</repository>
|
||||
</repositories>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,2 @@
|
||||
configuration-properties.classes=org.springframework.cloud.fn.consumer.twitter.message.TwitterMessageConsumerProperties
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.sink.twitter.message;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.Collections;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.StringBody;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.cloud.fn.consumer.twitter.message.TwitterMessageConsumerConfiguration;
|
||||
import org.springframework.cloud.stream.binder.test.InputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.messaging.support.GenericMessage;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.unlimited;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterMessageSinkIntegrationTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
|
||||
@BeforeEach
|
||||
public void startMockServer() {
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
|
||||
mockClient
|
||||
.when(
|
||||
request()
|
||||
.withMethod("GET")
|
||||
.withPath("/users/show.json")
|
||||
.withQueryStringParameter("screen_name", "user666")
|
||||
.withQueryStringParameter("include_entities", "true")
|
||||
.withQueryStringParameter("include_ext_alt_text", "true")
|
||||
.withQueryStringParameter("tweet_mode", "extended"),
|
||||
unlimited())
|
||||
.respond(
|
||||
response()
|
||||
.withStatusCode(200)
|
||||
.withHeader("Content-Type", "application/json; charset=utf-8")
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/user_1075751718749659136.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 1));
|
||||
|
||||
mockClient
|
||||
.when(
|
||||
request()
|
||||
.withMethod("POST")
|
||||
.withPath("/direct_messages/events/new.json"),
|
||||
unlimited())
|
||||
.respond(
|
||||
response()
|
||||
.withStatusCode(200)
|
||||
.withHeader("Content-Type", "application/json; charset=utf-8")
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/test_direct_message.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 1));
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
public void stopMockServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void directMessageScreenName() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterMessageSinkApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=byteArrayTextToString|sendDirectMessageConsumer",
|
||||
|
||||
"--twitter.message.update.screenName='user666'",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination source = context.getBean(InputDestination.class);
|
||||
assertThat(source).isNotNull();
|
||||
|
||||
source.send(new GenericMessage<>("hello".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/users/show.json")
|
||||
.withQueryStringParameter("screen_name", "user666")
|
||||
.withQueryStringParameter("include_entities", "true")
|
||||
.withQueryStringParameter("include_ext_alt_text", "true")
|
||||
.withQueryStringParameter("tweet_mode", "extended"),
|
||||
once());
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/direct_messages/events/new.json")
|
||||
.withBody(new StringBody("{\"event\":{\"type\":\"message_create\"," +
|
||||
"\"message_create\":{\"target\":{\"recipient_id\":1075751718749659136}," +
|
||||
"\"message_data\":{\"text\":\"hello\"}}}}")),
|
||||
once());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void directMessageUserId() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterMessageSinkApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=byteArrayTextToString|sendDirectMessageConsumer",
|
||||
|
||||
"--twitter.message.update.userId='1075751718749659136'",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination source = context.getBean(InputDestination.class);
|
||||
assertThat(source).isNotNull();
|
||||
|
||||
source.send(new GenericMessage<>("hello".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/direct_messages/events/new.json")
|
||||
.withBody(new StringBody("{\"event\":{\"type\":\"message_create\"," +
|
||||
"\"message_create\":{\"target\":{\"recipient_id\":1075751718749659136}," +
|
||||
"\"message_data\":{\"text\":\"hello\"}}}}")),
|
||||
once());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void directMessageDefaults() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterMessageSinkApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=byteArrayTextToString|sendDirectMessageConsumer",
|
||||
|
||||
"--twitter.message.update.userId=headers['user']",
|
||||
"--twitter.message.update.text=payload.concat(\" with suffix \")",
|
||||
"--twitter.message.update.mediaId='666'",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination source = context.getBean(InputDestination.class);
|
||||
assertThat(source).isNotNull();
|
||||
|
||||
Map<String, String> headers = Collections.singletonMap("user", "1075751718749659136");
|
||||
source.send(new GenericMessage("hello".getBytes(StandardCharsets.UTF_8), headers));
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/direct_messages/events/new.json")
|
||||
.withBody(new StringBody("{\"event\":{\"type\":\"message_create\",\"message_create\":" +
|
||||
"{\"target\":{\"recipient_id\":1075751718749659136},\"message_data\":" +
|
||||
"{\"text\":\"hello with suffix \",\"attachment\":{\"type\":\"media\",\"media\":" +
|
||||
"{\"id\":666}}}}}}")),
|
||||
once());
|
||||
}
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterMessageConsumerConfiguration.class)
|
||||
public static class TestTwitterMessageSinkApplication {
|
||||
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.sink.twitter.message;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.function.Function;
|
||||
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTestUtils {
|
||||
|
||||
public Function<ConfigurationBuilder, ConfigurationBuilder> mockTwitterUrls(String baseUrl) {
|
||||
return configBuilder -> {
|
||||
configBuilder.setRestBaseURL(baseUrl + "/");
|
||||
configBuilder.setStreamBaseURL(baseUrl + "/stream/");
|
||||
configBuilder.setUserStreamBaseURL(baseUrl + "/user/");
|
||||
configBuilder.setSiteStreamBaseURL(baseUrl + "/site/");
|
||||
configBuilder.setUploadBaseURL(baseUrl + "/upload/");
|
||||
|
||||
configBuilder.setOAuthAccessTokenURL(baseUrl + "/oauth/access_token");
|
||||
configBuilder.setOAuthAuthenticationURL(baseUrl + "/oauth/authenticate");
|
||||
configBuilder.setOAuthAuthorizationURL(baseUrl + "/oauth/authorize");
|
||||
configBuilder.setOAuthRequestTokenURL(baseUrl + "/oauth/request_token");
|
||||
configBuilder.setOAuth2TokenURL(baseUrl + "/oauth2/token");
|
||||
configBuilder.setOAuth2InvalidateTokenURL(baseUrl + "/oauth2/invalidate_token");
|
||||
|
||||
return configBuilder;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load Spring Resource as String.
|
||||
* @param resourcePath Resource path (accepts file:// , classpath:// and http:// uri schemas)
|
||||
* @return Returns text (UTF8) representation of the resource pointed by the resourcePath
|
||||
*/
|
||||
public static String asString(String resourcePath) {
|
||||
try {
|
||||
return StreamUtils.copyToString(new DefaultResourceLoader().getResource(resourcePath).getInputStream(),
|
||||
Charset.forName("UTF-8"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Can not load resource:" + resourcePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"created_at":"Thu Dec 20 13:56:10 +0000 2018","id":1075751718749659136,"id_str":"1075751718749659136","text":"Codementor: Kubernetes for Python Developers: Part 1\n#100daysofcode #python https:\/\/t.co\/VviTgpFpge","source":"\u003ca href=\"https:\/\/ifttt.com\" rel=\"nofollow\"\u003eIFTTT\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":859252650512072704,"id_str":"859252650512072704","name":"Freelancing|WebDev","screen_name":"FreelanceForBTC","location":"THE NET","url":"http:\/\/bit.ly\/BTCFREELANCING","description":"This twitter is designed to give the best information from web development to Blockchain.\n|YouTube's |Jobs |learning contented","translator_type":"none","protected":false,"verified":false,"followers_count":146,"friends_count":118,"listed_count":4,"favourites_count":445,"statuses_count":7528,"created_at":"Tue May 02 03:46:10 +0000 2017","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/859252650512072704\/1544326203","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"100daysofcode","indices":[53,67]},{"text":"python","indices":[68,75]}],"urls":[{"url":"https:\/\/t.co\/VviTgpFpge","expanded_url":"https:\/\/ift.tt\/2GxdMZF","display_url":"ift.tt\/2GxdMZF","indices":[76,99]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"ca","timestamp_ms":"1545314170907"}
|
||||
@@ -0,0 +1 @@
|
||||
{"created_at":"Thu Dec 20 13:56:10 +0000 2018","id":1075751718749659136,"id_str":"1075751718749659136","text":"Codementor: Kubernetes for Python Developers: Part 1\n#100daysofcode #python https:\/\/t.co\/VviTgpFpge","source":"\u003ca href=\"https:\/\/ifttt.com\" rel=\"nofollow\"\u003eIFTTT\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":859252650512072704,"id_str":"859252650512072704","name":"Freelancing|WebDev","screen_name":"FreelanceForBTC","location":"THE NET","url":"http:\/\/bit.ly\/BTCFREELANCING","description":"This twitter is designed to give the best information from web development to Blockchain.\n|YouTube's |Jobs |learning contented","translator_type":"none","protected":false,"verified":false,"followers_count":146,"friends_count":118,"listed_count":4,"favourites_count":445,"statuses_count":7528,"created_at":"Tue May 02 03:46:10 +0000 2017","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/859252650512072704\/1544326203","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"100daysofcode","indices":[53,67]},{"text":"python","indices":[68,75]}],"urls":[{"url":"https:\/\/t.co\/VviTgpFpge","expanded_url":"https:\/\/ift.tt\/2GxdMZF","display_url":"ift.tt\/2GxdMZF","indices":[76,99]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"ca","timestamp_ms":"1545314170907"}
|
||||
62
applications/sink/twitter-update-sink/README.adoc
Normal file
62
applications/sink/twitter-update-sink/README.adoc
Normal file
@@ -0,0 +1,62 @@
|
||||
//tag::ref-doc[]
|
||||
= Twitter Update Sink
|
||||
|
||||
Updates the authenticating user's current text (e.g Tweeting).
|
||||
|
||||
NOTE: For each update attempt, the update text is compared with the authenticating user's recent Tweets.
|
||||
Any attempt that would result in duplication will be blocked, resulting in a 403 error.
|
||||
A user cannot submit the same text twice in a row.
|
||||
|
||||
While not rate limited by the API, a user is limited in the number of Tweets they can create at a time.
|
||||
The update limit for standard API is 300 in 3 hours windows.
|
||||
If the number of updates posted by the user reaches the current allowed limit this method will return an HTTP 403 error.
|
||||
|
||||
You can find details for the Update API here: https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/post-statuses-update
|
||||
|
||||
== Options
|
||||
|
||||
//tag::configuration-properties[]
|
||||
$$twitter.update.attachment-url$$:: $$(SpEL expression) In order for a URL to not be counted in the text body of an extended Tweet, provide a URL as a Tweet attachment. This URL must be a Tweet permalink, or Direct Message deep link. Arbitrary, non-Twitter URLs must remain in the text text. URLs passed to the attachment_url parameter not matching either a Tweet permalink or Direct Message deep link will fail at Tweet creation and cause an exception.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.update.display-coordinates$$:: $$(SpEL expression) Whether or not to put a pin on the exact coordinates a Tweet has been sent from.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.update.in-reply-to-status-id$$:: $$(SpEL expression) The ID of an existing text that the update is in reply to. Note: This parameter will be ignored unless the author of the Tweet this parameter references is mentioned within the text text. Therefore, you must include @username, where username is the author of the referenced Tweet, within the update. When inReplyToStatusId is set the auto_populate_reply_metadata is automatically set as well. Later ensures that leading @mentions will be looked up from the original Tweet, and added to the new Tweet from there. This wil append @mentions into the metadata of an extended Tweet as a reply chain grows, until the limit on @mentions is reached. In cases where the original Tweet has been deleted, the reply will fail.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.update.location.lat$$:: $$The latitude of the location this Tweet refers to. This parameter will be ignored unless it is inside the range -90.0 to +90.0 (North is positive) inclusive. It will also be ignored if there is no corresponding long parameter.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.update.location.lon$$:: $$The longitude of the location this Tweet refers to. The valid ranges for longitude are -180.0 to +180.0 (East is positive) inclusive. This parameter will be ignored if outside that range, if it is not a number, if geo_enabled is disabled, or if there no corresponding lat parameter.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.update.media-ids$$:: $$(SpEL expression) A comma-delimited list of media_ids to associate with the Tweet. You may include up to 4 photos or 1 animated GIF or 1 video in a Tweet. See Uploading Media for further details on uploading media.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.update.place-id$$:: $$(SpEL expression) A place in the world.$$ *($$Expression$$, default: `$$<none>$$`)*
|
||||
$$twitter.update.text$$:: $$(SpEL expression) The text of the text update. URL encode as necessary. t.co link wrapping will affect character counts. Defaults to message's payload$$ *($$Expression$$, default: `$$payload$$`)*
|
||||
//end::configuration-properties[]
|
||||
|
||||
//end::ref-doc[]
|
||||
|
||||
== Configuration
|
||||
|
||||
`TwitterUpdateConsumerConfiguration` exposes 2 composable functions:
|
||||
|
||||
* `Function<Message<?>, StatusUpdate> toStatusUpdateQuery(TwitterUpdateConsumerProperties updateProperties)` - Converts input message into `StatusUpdate` query object.
|
||||
* `Consumer<StatusUpdate> updateStatus(Twitter twitter)` - Sends the input `StatusUpdate` argument as Twitter text update.
|
||||
|
||||
Use `@Import(TwitterUpdateConsumerConfiguration.class)` to compose those functions.
|
||||
|
||||
By default the `twitter-update` implements the following composite function chain:
|
||||
`spring.cloud.stream.function.definition=byteArrayTextToString|toStatusUpdateQuery|updateStatus` or (`byteArrayTextToString|twitterStatusUpdateConsumer`)
|
||||
|
||||
|
||||
== Examples
|
||||
|
||||
```
|
||||
java -jar twitter-update-sink.jar
|
||||
|
||||
--twitter.connection.consumerKey= ...
|
||||
--twitter.connection.consumerSecret= ...
|
||||
--twitter.connection.accessToken= ...
|
||||
--twitter.connection.accessTokenSecret= ...
|
||||
```
|
||||
|
||||
and send message to the `input` channel.
|
||||
|
||||
And here is a example pipeline that uses twitter-update:
|
||||
|
||||
```
|
||||
twitter-update-stream= time | twitter-update --twitter.connection.consumerKey= ... --twitter.connection.consumerSecret= ... --twitter.connection.accessToken= ... --twitter.connection.accessTokenSecret= ...
|
||||
```
|
||||
|
||||
103
applications/sink/twitter-update-sink/pom.xml
Normal file
103
applications/sink/twitter-update-sink/pom.xml
Normal file
@@ -0,0 +1,103 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-update-sink</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<name>twitter-update-sink</name>
|
||||
<description>twitter update sink apps</description>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.stream.app</groupId>
|
||||
<artifactId>stream-applications-core</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-consumer</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.awaitility</groupId>
|
||||
<artifactId>awaitility</artifactId>
|
||||
<scope>test</scope>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud</groupId>
|
||||
<artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud.stream.app.plugin</groupId>
|
||||
<artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<generatedApp>
|
||||
<name>twitter-update</name>
|
||||
<type>sink</type>
|
||||
<version>${project.version}</version>
|
||||
<configClass>org.springframework.cloud.fn.consumer.twitter.status.update.TwitterUpdateConsumerConfiguration.class</configClass>
|
||||
<functionDefinition>byteArrayTextToString|twitterStatusUpdateConsumer</functionDefinition>
|
||||
</generatedApp>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-consumer</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</configuration>
|
||||
</plugin>
|
||||
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
<id>spring-snapshots</id>
|
||||
<name>Spring Snapshots</name>
|
||||
<url>https://repo.spring.io/libs-snapshot-local</url>
|
||||
</repository>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>false</enabled>
|
||||
</snapshots>
|
||||
<id>spring-milestones</id>
|
||||
<name>Spring Milestones</name>
|
||||
<url>https://repo.spring.io/libs-milestone-local</url>
|
||||
</repository>
|
||||
</repositories>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,2 @@
|
||||
configuration-properties.classes=org.springframework.cloud.fn.consumer.twitter.status.update.TwitterUpdateConsumerProperties, \
|
||||
org.springframework.cloud.fn.consumer.twitter.status.update.TwitterUpdateConsumerProperties$Location
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.sink.twitter.update;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.function.Function;
|
||||
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTestUtils {
|
||||
|
||||
public Function<ConfigurationBuilder, ConfigurationBuilder> mockTwitterUrls(String baseUrl) {
|
||||
return configBuilder -> {
|
||||
configBuilder.setRestBaseURL(baseUrl + "/");
|
||||
configBuilder.setStreamBaseURL(baseUrl + "/stream/");
|
||||
configBuilder.setUserStreamBaseURL(baseUrl + "/user/");
|
||||
configBuilder.setSiteStreamBaseURL(baseUrl + "/site/");
|
||||
configBuilder.setUploadBaseURL(baseUrl + "/upload/");
|
||||
|
||||
configBuilder.setOAuthAccessTokenURL(baseUrl + "/oauth/access_token");
|
||||
configBuilder.setOAuthAuthenticationURL(baseUrl + "/oauth/authenticate");
|
||||
configBuilder.setOAuthAuthorizationURL(baseUrl + "/oauth/authorize");
|
||||
configBuilder.setOAuthRequestTokenURL(baseUrl + "/oauth/request_token");
|
||||
configBuilder.setOAuth2TokenURL(baseUrl + "/oauth2/token");
|
||||
configBuilder.setOAuth2InvalidateTokenURL(baseUrl + "/oauth2/invalidate_token");
|
||||
|
||||
return configBuilder;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load Spring Resource as String.
|
||||
* @param resourcePath Resource path (accepts file:// , classpath:// and http:// uri schemas)
|
||||
* @return Returns text (UTF8) representation of the resource pointed by the resourcePath
|
||||
*/
|
||||
public static String asString(String resourcePath) {
|
||||
try {
|
||||
return StreamUtils.copyToString(new DefaultResourceLoader().getResource(resourcePath).getInputStream(),
|
||||
Charset.forName("UTF-8"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Can not load resource:" + resourcePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.sink.twitter.update;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.StringBody;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.cloud.fn.consumer.twitter.status.update.TwitterUpdateConsumerConfiguration;
|
||||
import org.springframework.cloud.fn.consumer.twitter.status.update.TwitterUpdateConsumerProperties;
|
||||
import org.springframework.cloud.stream.binder.test.InputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.messaging.support.GenericMessage;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.unlimited;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterUpdateSinkIntegrationTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
|
||||
@BeforeAll
|
||||
public static void startMockServer() {
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
mockClient
|
||||
.when(
|
||||
request().withMethod("POST").withPath("/statuses/update.json"),
|
||||
unlimited())
|
||||
.respond(response()
|
||||
.withStatusCode(200)
|
||||
.withHeader("Content-Type", "application/json; charset=utf-8")
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/update_test_1.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 1));
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
public static void stopMockServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testUpdateStatus() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterUpdateSinkApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=byteArrayTextToString|twitterStatusUpdateConsumer",
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination source = context.getBean(InputDestination.class);
|
||||
assertThat(source).isNotNull();
|
||||
|
||||
source.send(new GenericMessage<>("Test Update 678".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/statuses/update.json")
|
||||
.withBody(new StringBody("status=Test%20Update%20678" +
|
||||
"&include_entities=true" +
|
||||
"&include_ext_alt_text=true" +
|
||||
"&tweet_mode=extended")),
|
||||
once());
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void updateWithPayloadExpression() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterUpdateSinkApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=byteArrayTextToString|twitterStatusUpdateConsumer",
|
||||
"--twitter.update.text=payload.toUpperCase().concat(\" With Suffix\")",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
InputDestination source = context.getBean(InputDestination.class);
|
||||
assertThat(source).isNotNull();
|
||||
|
||||
source.send(new GenericMessage<>("1 Expression Test".getBytes(StandardCharsets.UTF_8)));
|
||||
source.send(new GenericMessage<>("2 Expression Test".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/statuses/update.json")
|
||||
.withBody(new StringBody("status=1%20EXPRESSION%20TEST%20With%20Suffix" +
|
||||
"&include_entities=true" +
|
||||
"&include_ext_alt_text=true" +
|
||||
"&tweet_mode=extended")),
|
||||
once());
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/statuses/update.json")
|
||||
.withBody(new StringBody("status=2%20EXPRESSION%20TEST%20With%20Suffix" +
|
||||
"&include_entities=true" +
|
||||
"&include_ext_alt_text=true" +
|
||||
"&tweet_mode=extended")),
|
||||
once());
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void updateWithAllParams() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterUpdateSinkApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=byteArrayTextToString|twitterStatusUpdateConsumer",
|
||||
|
||||
"--twitter.update.attachmentUrl='http://attachementUrl'",
|
||||
"--twitter.update.placeId='myPlaceId'",
|
||||
"--twitter.update.inReplyToStatusId='666666'",
|
||||
"--twitter.update.displayCoordinates='true'",
|
||||
"--twitter.update.mediaIds='471592142565957632, 471592142565957633'",
|
||||
"--twitter.update.location.lat='37.78217'",
|
||||
"--twitter.update.location.lon='-122.40062'",
|
||||
|
||||
"--twitter.connection.consumerKey=myConsumerKey",
|
||||
"--twitter.connection.consumerSecret=myConsumerSecret",
|
||||
"--twitter.connection.accessToken=myAccessToken",
|
||||
"--twitter.connection.accessTokenSecret=myAccessTokenSecret")) {
|
||||
|
||||
TwitterConnectionProperties twitterConnectionProperties = context.getBean(TwitterConnectionProperties.class);
|
||||
assertThat(twitterConnectionProperties.getConsumerKey()).isEqualTo("myConsumerKey");
|
||||
assertThat(twitterConnectionProperties.getConsumerSecret()).isEqualTo("myConsumerSecret");
|
||||
assertThat(twitterConnectionProperties.getAccessToken()).isEqualTo("myAccessToken");
|
||||
assertThat(twitterConnectionProperties.getAccessTokenSecret()).isEqualTo("myAccessTokenSecret");
|
||||
|
||||
TwitterUpdateConsumerProperties twitterUpdateConsumerProperties = context.getBean(TwitterUpdateConsumerProperties.class);
|
||||
assertThat(twitterUpdateConsumerProperties.getAttachmentUrl().getValue()).isEqualTo("http://attachementUrl");
|
||||
assertThat(twitterUpdateConsumerProperties.getPlaceId().getValue()).isEqualTo("myPlaceId");
|
||||
assertThat(twitterUpdateConsumerProperties.getInReplyToStatusId().getValue()).isEqualTo("666666");
|
||||
assertThat(twitterUpdateConsumerProperties.getDisplayCoordinates().getValue()).isEqualTo("true");
|
||||
assertThat(twitterUpdateConsumerProperties.getMediaIds().getValue()).isEqualTo("471592142565957632, 471592142565957633");
|
||||
assertThat(twitterUpdateConsumerProperties.getLocation().getLat().getValue()).isEqualTo("37.78217");
|
||||
assertThat(twitterUpdateConsumerProperties.getLocation().getLon().getValue()).isEqualTo("-122.40062");
|
||||
|
||||
InputDestination source = context.getBean(InputDestination.class);
|
||||
assertThat(source).isNotNull();
|
||||
|
||||
source.send(new GenericMessage<>("Test Tweet".getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
mockClient.verify(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/statuses/update.json")
|
||||
.withBody(new StringBody("status=Test%20Tweet" +
|
||||
"&in_reply_to_status_id=666666" +
|
||||
"&lat=37.78217&long=-122.40062" +
|
||||
"&place_id=myPlaceId" +
|
||||
"&media_ids=471592142565957632%2C471592142565957633" +
|
||||
"&auto_populate_reply_metadata=true" +
|
||||
"&attachment_url=http%3A%2F%2FattachementUrl" +
|
||||
"&include_entities=true" +
|
||||
"&include_ext_alt_text=true" +
|
||||
"&tweet_mode=extended")),
|
||||
once());
|
||||
}
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterUpdateConsumerConfiguration.class)
|
||||
public static class TestTwitterUpdateSinkApplication {
|
||||
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"created_at":"Thu Dec 20 13:56:10 +0000 2018","id":1075751718749659136,"id_str":"1075751718749659136","text":"Codementor: Kubernetes for Python Developers: Part 1\n#100daysofcode #python https:\/\/t.co\/VviTgpFpge","source":"\u003ca href=\"https:\/\/ifttt.com\" rel=\"nofollow\"\u003eIFTTT\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":859252650512072704,"id_str":"859252650512072704","name":"Freelancing|WebDev","screen_name":"FreelanceForBTC","location":"THE NET","url":"http:\/\/bit.ly\/BTCFREELANCING","description":"This twitter is designed to give the best information from web development to Blockchain.\n|YouTube's |Jobs |learning contented","translator_type":"none","protected":false,"verified":false,"followers_count":146,"friends_count":118,"listed_count":4,"favourites_count":445,"statuses_count":7528,"created_at":"Tue May 02 03:46:10 +0000 2017","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/859252650512072704\/1544326203","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"100daysofcode","indices":[53,67]},{"text":"python","indices":[68,75]}],"urls":[{"url":"https:\/\/t.co\/VviTgpFpge","expanded_url":"https:\/\/ift.tt\/2GxdMZF","display_url":"ift.tt\/2GxdMZF","indices":[76,99]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"ca","timestamp_ms":"1545314170907"}
|
||||
@@ -24,5 +24,8 @@
|
||||
<module>rabbit-source</module>
|
||||
<module>websocket-source</module>
|
||||
<module>s3-source</module>
|
||||
<module>twitter-stream-source</module>
|
||||
<module>twitter-search-source</module>
|
||||
<module>twitter-message-source</module>
|
||||
</modules>
|
||||
</project>
|
||||
|
||||
30
applications/source/twitter-message-source/README.adoc
Normal file
30
applications/source/twitter-message-source/README.adoc
Normal file
@@ -0,0 +1,30 @@
|
||||
//tag::ref-doc[]
|
||||
= Twitter Message Source
|
||||
|
||||
Repeatedly retrieves the direct messages (both sent and received) within the last 30 days, sorted in reverse-chronological order.
|
||||
The relieved messages are cached (in a `MetadataStore` cache) to prevent duplications.
|
||||
By default an in-memory `SimpleMetadataStore` is used.
|
||||
|
||||
The `twitter.message.source.count` controls the number or returned messages.
|
||||
|
||||
The `spring.cloud.stream.poller` properties control the message poll interval.
|
||||
Must be aligned with used APIs rate limit
|
||||
|
||||
== Options
|
||||
|
||||
//tag::configuration-properties[]
|
||||
$$spring.cloud.stream.poller.cron$$:: $$Cron expression value for the Cron Trigger.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$spring.cloud.stream.poller.fixed-delay$$:: $$Fixed delay for default poller.$$ *($$Long$$, default: `$$1000$$`)*
|
||||
$$spring.cloud.stream.poller.initial-delay$$:: $$Initial delay for periodic triggers.$$ *($$Integer$$, default: `$$0$$`)*
|
||||
$$spring.cloud.stream.poller.max-messages-per-poll$$:: $$Maximum messages per poll for the default poller.$$ *($$Long$$, default: `$$1$$`)*
|
||||
$$twitter.connection.access-token$$:: $$Your Twitter token.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.access-token-secret$$:: $$Your Twitter token secret.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.consumer-key$$:: $$Your Twitter key.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.consumer-secret$$:: $$Your Twitter secret.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.debug-enabled$$:: $$Enables Twitter4J debug mode.$$ *($$Boolean$$, default: `$$false$$`)*
|
||||
$$twitter.connection.raw-json$$:: $$Enable caching the original (raw) JSON objects as returned by the Twitter APIs. When set to False the result will use the Twitter4J's json representations. When set to True the result will use the original Twitter APISs json representations.$$ *($$Boolean$$, default: `$$true$$`)*
|
||||
$$twitter.message.source.count$$:: $$Max number of events to be returned. 20 default. 50 max.$$ *($$Integer$$, default: `$$20$$`)*
|
||||
//end::configuration-properties[]
|
||||
|
||||
//end::ref-doc[]
|
||||
|
||||
97
applications/source/twitter-message-source/pom.xml
Normal file
97
applications/source/twitter-message-source/pom.xml
Normal file
@@ -0,0 +1,97 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-message-source</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<name>twitter-message-source</name>
|
||||
<description>twitter message source apps</description>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.stream.app</groupId>
|
||||
<artifactId>stream-applications-core</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-supplier</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>function-test-support</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud</groupId>
|
||||
<artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud.stream.app.plugin</groupId>
|
||||
<artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<generatedApp>
|
||||
<name>twitter-message</name>
|
||||
<type>source</type>
|
||||
<version>${project.version}</version>
|
||||
<configClass>
|
||||
org.springframework.cloud.fn.supplier.twitter.message.TwitterMessageSupplierConfiguration.class
|
||||
</configClass>
|
||||
<functionDefinition>twitterMessageSupplier</functionDefinition>
|
||||
</generatedApp>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-supplier</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
<id>spring-snapshots</id>
|
||||
<name>Spring Snapshots</name>
|
||||
<url>https://repo.spring.io/libs-snapshot-local</url>
|
||||
</repository>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>false</enabled>
|
||||
</snapshots>
|
||||
<id>spring-milestones</id>
|
||||
<name>Spring Milestones</name>
|
||||
<url>https://repo.spring.io/libs-milestone-local</url>
|
||||
</repository>
|
||||
</repositories>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,4 @@
|
||||
configuration-properties.classes=org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties,\
|
||||
org.springframework.cloud.fn.supplier.twitter.message.TwitterMessageSupplierProperties, \
|
||||
org.springframework.cloud.stream.config.DefaultPollerProperties
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.source.twitter.message;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.Header;
|
||||
import org.mockserver.model.HttpRequest;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.cloud.fn.supplier.twitter.message.TwitterMessageSupplierConfiguration;
|
||||
import org.springframework.cloud.fn.supplier.twitter.message.TwitterMessageSupplierProperties;
|
||||
import org.springframework.cloud.stream.binder.test.OutputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
|
||||
import org.springframework.cloud.stream.config.DefaultPollerProperties;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.messaging.Message;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.exactly;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterMessageSourceIntegrationTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
private static HttpRequest messageRequest;
|
||||
|
||||
|
||||
@BeforeAll
|
||||
public static void startServer() {
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
|
||||
messageRequest = setExpectation(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/direct_messages/events/list.json")
|
||||
.withQueryStringParameter("count", "15"));
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
public static void stopServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void twitterMessageSourceTests() throws JsonProcessingException {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration
|
||||
.getCompleteConfiguration(TestTwitterMessageSourceApplication.class))
|
||||
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=twitterMessageSupplier",
|
||||
|
||||
"--twitter.connection.consumerKey=consumerKey666",
|
||||
"--twitter.connection.consumerSecret=consumerSecret666",
|
||||
"--twitter.connection.accessToken=accessToken666",
|
||||
"--twitter.connection.accessTokenSecret=accessTokenSecret666",
|
||||
|
||||
"--twitter.message.source.count=15",
|
||||
"--spring.cloud.stream.poller.fixed-delay=3000")) {
|
||||
|
||||
TwitterConnectionProperties twitterConnectionProperties = context.getBean(TwitterConnectionProperties.class);
|
||||
assertThat(twitterConnectionProperties.getConsumerKey()).isEqualTo("consumerKey666");
|
||||
assertThat(twitterConnectionProperties.getConsumerSecret()).isEqualTo("consumerSecret666");
|
||||
assertThat(twitterConnectionProperties.getAccessToken()).isEqualTo("accessToken666");
|
||||
assertThat(twitterConnectionProperties.getAccessTokenSecret()).isEqualTo("accessTokenSecret666");
|
||||
|
||||
DefaultPollerProperties defaultPollerProperties = context.getBean(DefaultPollerProperties.class);
|
||||
assertThat(defaultPollerProperties.getFixedDelay()).isEqualTo(3000);
|
||||
|
||||
TwitterMessageSupplierProperties twitterMessageSupplierProperties = context.getBean(TwitterMessageSupplierProperties.class);
|
||||
assertThat(twitterMessageSupplierProperties.getCount()).isEqualTo(15);
|
||||
|
||||
OutputDestination outputDestination = context.getBean(OutputDestination.class);
|
||||
// Using local region here
|
||||
Message<byte[]> message = outputDestination.receive(Duration.ofSeconds(300).toMillis());
|
||||
assertThat(message).isNotNull();
|
||||
String payload = new String(message.getPayload());
|
||||
|
||||
List tweets = new ObjectMapper().readValue(payload, List.class);
|
||||
assertThat(tweets).hasSize(4);
|
||||
mockClient.verify(messageRequest, once());
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpRequest setExpectation(HttpRequest request) {
|
||||
mockClient.when(request, exactly(1))
|
||||
.respond(
|
||||
response()
|
||||
.withStatusCode(200)
|
||||
.withHeaders(
|
||||
new Header("Content-Type", "application/json; charset=utf-8"),
|
||||
new Header("Cache-Control", "public, max-age=86400"))
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/messages.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 10));
|
||||
return request;
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterMessageSupplierConfiguration.class)
|
||||
public static class TestTwitterMessageSourceApplication {
|
||||
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.source.twitter.message;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.function.Function;
|
||||
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTestUtils {
|
||||
|
||||
public Function<ConfigurationBuilder, ConfigurationBuilder> mockTwitterUrls(String baseUrl) {
|
||||
return configBuilder -> {
|
||||
configBuilder.setRestBaseURL(baseUrl + "/");
|
||||
configBuilder.setStreamBaseURL(baseUrl + "/stream/");
|
||||
configBuilder.setUserStreamBaseURL(baseUrl + "/user/");
|
||||
configBuilder.setSiteStreamBaseURL(baseUrl + "/site/");
|
||||
configBuilder.setUploadBaseURL(baseUrl + "/upload/");
|
||||
|
||||
configBuilder.setOAuthAccessTokenURL(baseUrl + "/oauth/access_token");
|
||||
configBuilder.setOAuthAuthenticationURL(baseUrl + "/oauth/authenticate");
|
||||
configBuilder.setOAuthAuthorizationURL(baseUrl + "/oauth/authorize");
|
||||
configBuilder.setOAuthRequestTokenURL(baseUrl + "/oauth/request_token");
|
||||
configBuilder.setOAuth2TokenURL(baseUrl + "/oauth2/token");
|
||||
configBuilder.setOAuth2InvalidateTokenURL(baseUrl + "/oauth2/invalidate_token");
|
||||
|
||||
return configBuilder;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load Spring Resource as String.
|
||||
* @param resourcePath Resource path (accepts file:// , classpath:// and http:// uri schemas)
|
||||
* @return Returns text (UTF8) representation of the resource pointed by the resourcePath
|
||||
*/
|
||||
public static String asString(String resourcePath) {
|
||||
try {
|
||||
return StreamUtils.copyToString(new DefaultResourceLoader().getResource(resourcePath).getInputStream(),
|
||||
Charset.forName("UTF-8"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Can not load resource:" + resourcePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"events":[{"type":"message_create","id":"1073991611720351749","created_timestamp":"1544894528690","message_create":{"target":{"recipient_id":"1073883577107038208"},"sender_id":"10219792","message_data":{"text":"Test6","entities":{"hashtags":[],"symbols":[],"user_mentions":[],"urls":[]}}}},{"type":"message_create","id":"1073922523325218821","created_timestamp":"1544878056733","message_create":{"target":{"recipient_id":"10219792"},"sender_id":"1073883577107038208","source_app_id":"268278","message_data":{"text":"Here again","entities":{"hashtags":[],"symbols":[],"user_mentions":[],"urls":[]}}}},{"type":"message_create","id":"1073885014805356549","created_timestamp":"1544869114005","message_create":{"target":{"recipient_id":"1073883577107038208"},"sender_id":"10219792","message_data":{"text":"Test Back","entities":{"hashtags":[],"symbols":[],"user_mentions":[],"urls":[]}}}},{"type":"message_create","id":"1073884892927275012","created_timestamp":"1544869084947","message_create":{"target":{"recipient_id":"10219792"},"sender_id":"1073883577107038208","source_app_id":"268278","message_data":{"text":"Test","entities":{"hashtags":[],"symbols":[],"user_mentions":[],"urls":[]}}}}],"apps":{"268278":{"id":"268278","name":"Twitter Web Client","url":"http:\/\/twitter.com"}}}
|
||||
64
applications/source/twitter-search-source/README.adoc
Normal file
64
applications/source/twitter-search-source/README.adoc
Normal file
@@ -0,0 +1,64 @@
|
||||
//tag::ref-doc[]
|
||||
= Twitter Search Source
|
||||
|
||||
The Twitter's https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets.html[Standard search API] (search/tweets) allows simple queries against the indices of recent or popular Tweets. This `Source` provides continuous searches against a sampling of recent Tweets published in the past 7 days. Part of the 'public' set of APIs.
|
||||
|
||||
Returns a collection of relevant Tweets matching a specified query.
|
||||
|
||||
Use the `spring.cloud.stream.poller` properties to control the interval between consecutive search requests. Rate Limit - 180 requests per 30 min. window (e.g. ~6 r/m, ~ 1 req / 10 sec.)
|
||||
|
||||
The `twitter.search` query properties allows querying by keywords and filter the result by time and geolocation.
|
||||
|
||||
The `twitter.search.count` and `twitter.search.page` control the result pagination in accordance with to the Search API.
|
||||
|
||||
Note: Twitter's search service and, by extension, the Search API is not meant to be an exhaustive source of Tweets. Not all Tweets will be indexed or made available via the search interface.
|
||||
|
||||
== Options
|
||||
|
||||
//tag::configuration-properties[]
|
||||
$$spring.cloud.stream.poller.cron$$:: $$Cron expression value for the Cron Trigger.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$spring.cloud.stream.poller.fixed-delay$$:: $$Fixed delay for default poller.$$ *($$Long$$, default: `$$1000$$`)*
|
||||
$$spring.cloud.stream.poller.initial-delay$$:: $$Initial delay for periodic triggers.$$ *($$Integer$$, default: `$$0$$`)*
|
||||
$$spring.cloud.stream.poller.max-messages-per-poll$$:: $$Maximum messages per poll for the default poller.$$ *($$Long$$, default: `$$1$$`)*
|
||||
$$twitter.connection.access-token$$:: $$Your Twitter token.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.access-token-secret$$:: $$Your Twitter token secret.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.consumer-key$$:: $$Your Twitter key.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.consumer-secret$$:: $$Your Twitter secret.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.debug-enabled$$:: $$Enables Twitter4J debug mode.$$ *($$Boolean$$, default: `$$false$$`)*
|
||||
$$twitter.connection.raw-json$$:: $$Enable caching the original (raw) JSON objects as returned by the Twitter APIs. When set to False the result will use the Twitter4J's json representations. When set to True the result will use the original Twitter APISs json representations.$$ *($$Boolean$$, default: `$$true$$`)*
|
||||
$$twitter.search.count$$:: $$Number of tweets to return per page (e.g. per single request), up to a max of 100.$$ *($$Integer$$, default: `$$100$$`)*
|
||||
$$twitter.search.geocode.latitude$$:: $$User's latitude.$$ *($$Double$$, default: `$$-1$$`)*
|
||||
$$twitter.search.geocode.longitude$$:: $$User's longitude.$$ *($$Double$$, default: `$$-1$$`)*
|
||||
$$twitter.search.geocode.radius$$:: $$Radius (in kilometers) around the (latitude, longitude) point.$$ *($$Double$$, default: `$$-1$$`)*
|
||||
$$twitter.search.lang$$:: $$Restricts searched tweets to the given language, given by an <a href="http://en.wikipedia.org/wiki/ISO_639-1">ISO 639-1 code</a>.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.search.page$$:: $$Number of pages (e.g. requests) to search backwards (from most recent to the oldest tweets) before start the search from the most recent tweets again. The total amount of tweets searched backwards is (page * count)$$ *($$Integer$$, default: `$$3$$`)*
|
||||
$$twitter.search.query$$:: $$Search tweets by search query string.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.search.restart-from-most-recent-on-empty-response$$:: $$Restart search from the most recent tweets on empty response. Applied only after the first restart (e.g. when since_id != UNBOUNDED)$$ *($$Boolean$$, default: `$$false$$`)*
|
||||
$$twitter.search.result-type$$:: $$Specifies what type of search results you would prefer to receive. The current default is "mixed." Valid values include: mixed : Include both popular and real time results in the response. recent : return only the most recent results in the response popular : return only the most popular results in the response$$ *($$ResultType$$, default: `$$<none>$$`, possible values: `popular`,`mixed`,`recent`)*
|
||||
$$twitter.search.since$$:: $$If specified, returns tweets with since the given date. Date should be formatted as YYYY-MM-DD.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
//end::configuration-properties[]
|
||||
|
||||
//end::ref-doc[]
|
||||
|
||||
|
||||
== Examples
|
||||
|
||||
```
|
||||
java -jar twitter-search-source.jar
|
||||
--twitter.connection.consumerKey= ...
|
||||
--twitter.connection.consumerSecret= ...
|
||||
--twitter.connection.accessToken= ...
|
||||
--twitter.connection.accessTokenSecret= ...
|
||||
|
||||
--twitter.search.query=Amsterdam
|
||||
--twitter.search.count=30
|
||||
--twitter.search.page=3
|
||||
```
|
||||
|
||||
And here is an example pipeline that uses twitter-search:
|
||||
|
||||
```
|
||||
twitter-search-stream= twitter-search --twitter.connection.consumerKey= ... --twitter.connection.consumerSecret= ... --twitter.connection.accessToken= ... --twitter.connection.accessTokenSecret= ... --twitter.search.query=Amsterdam --twitter.search.count=30 --twitter.search.page=3
|
||||
|
||||
```
|
||||
|
||||
94
applications/source/twitter-search-source/pom.xml
Normal file
94
applications/source/twitter-search-source/pom.xml
Normal file
@@ -0,0 +1,94 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-search-source</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<name>twitter-search-source</name>
|
||||
<description>twitter search source apps</description>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.stream.app</groupId>
|
||||
<artifactId>stream-applications-core</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-supplier</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>function-test-support</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud</groupId>
|
||||
<artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud.stream.app.plugin</groupId>
|
||||
<artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<generatedApp>
|
||||
<name>twitter-search</name>
|
||||
<type>source</type>
|
||||
<version>${project.version}</version>
|
||||
<configClass>org.springframework.cloud.fn.supplier.twitter.status.search.TwitterSearchSupplierConfiguration.class</configClass>
|
||||
<functionDefinition>twitterSearchSupplier</functionDefinition>
|
||||
</generatedApp>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-supplier</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
<id>spring-snapshots</id>
|
||||
<name>Spring Snapshots</name>
|
||||
<url>https://repo.spring.io/libs-snapshot-local</url>
|
||||
</repository>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>false</enabled>
|
||||
</snapshots>
|
||||
<id>spring-milestones</id>
|
||||
<name>Spring Milestones</name>
|
||||
<url>https://repo.spring.io/libs-milestone-local</url>
|
||||
</repository>
|
||||
</repositories>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,5 @@
|
||||
configuration-properties.classes=org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties,\
|
||||
org.springframework.cloud.fn.supplier.twitter.status.search.TwitterSearchSupplierProperties, \
|
||||
org.springframework.cloud.fn.supplier.twitter.status.search.TwitterSearchSupplierProperties$Geocode, \
|
||||
org.springframework.cloud.stream.config.DefaultPollerProperties
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.source.twitter.search;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.Header;
|
||||
import org.mockserver.model.HttpRequest;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.cloud.fn.supplier.twitter.status.search.TwitterSearchSupplierConfiguration;
|
||||
import org.springframework.cloud.fn.supplier.twitter.status.search.TwitterSearchSupplierProperties;
|
||||
import org.springframework.cloud.stream.binder.test.OutputDestination;
|
||||
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
|
||||
import org.springframework.cloud.stream.config.DefaultPollerProperties;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.messaging.Message;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.exactly;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterSearchSourceIntegrationTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
|
||||
private static HttpRequest searchVratsaRequest;
|
||||
|
||||
private static HttpRequest searchAmsterdamRequest;
|
||||
|
||||
@BeforeAll
|
||||
public static void startServer() {
|
||||
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
|
||||
searchVratsaRequest = setExpectation(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/search/tweets.json")
|
||||
.withQueryStringParameter("q", "Vratsa")
|
||||
.withQueryStringParameter("count", "3"));
|
||||
|
||||
searchAmsterdamRequest = setExpectation(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/search/tweets.json")
|
||||
.withQueryStringParameter("q", "Amsterdam")
|
||||
.withQueryStringParameter("count", "3")
|
||||
.withQueryStringParameter("result_type", "popular")
|
||||
.withQueryStringParameter("geocode", "52.1,4.8,10.0km")
|
||||
.withQueryStringParameter("since", "2018-01-01")
|
||||
.withQueryStringParameter("lang", "en"));
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
public static void stopServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void twitterSearchTests() throws JsonProcessingException {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration
|
||||
.getCompleteConfiguration(TestTwitterSearchSourceApplication.class))
|
||||
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=twitterSearchSupplier",
|
||||
"--twitter.connection.consumerKey=consumerKey666",
|
||||
"--twitter.connection.consumerSecret=consumerSecret666",
|
||||
"--twitter.connection.accessToken=accessToken666",
|
||||
"--twitter.connection.accessTokenSecret=accessTokenSecret666",
|
||||
"--twitter.search.query=Vratsa",
|
||||
"--twitter.search.count=3",
|
||||
"--twitter.search.page=3")) {
|
||||
|
||||
OutputDestination outputDestination = context.getBean(OutputDestination.class);
|
||||
// Using local region here
|
||||
Message<byte[]> message = outputDestination.receive(Duration.ofSeconds(300).toMillis());
|
||||
assertThat(message).isNotNull();
|
||||
String payload = new String(message.getPayload());
|
||||
|
||||
List tweets = new ObjectMapper().readValue(payload, List.class);
|
||||
assertThat(tweets).hasSize(3);
|
||||
mockClient.verify(searchVratsaRequest, once());
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void twitterSearchTestsAmsterdam() throws JsonProcessingException {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration
|
||||
.getCompleteConfiguration(TestTwitterSearchSourceApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=twitterSearchSupplier",
|
||||
"--twitter.connection.consumerKey=consumerKey666",
|
||||
"--twitter.connection.consumerSecret=consumerSecret666",
|
||||
"--twitter.connection.accessToken=accessToken666",
|
||||
"--twitter.connection.accessTokenSecret=accessTokenSecret666",
|
||||
"--twitter.search.query=Amsterdam",
|
||||
"--twitter.search.count=3",
|
||||
"--twitter.search.page=3",
|
||||
"--twitter.search.lang=en",
|
||||
"--twitter.search.geocode.latitude=52.1",
|
||||
"--twitter.search.geocode.longitude=4.8",
|
||||
"--twitter.search.geocode.radius=10",
|
||||
"--twitter.search.since=2018-01-01",
|
||||
"--twitter.search.resultType=popular",
|
||||
"--spring.cloud.stream.poller.fixed-delay=10000")) {
|
||||
|
||||
TwitterConnectionProperties twitterConnectionProperties = context.getBean(TwitterConnectionProperties.class);
|
||||
assertThat(twitterConnectionProperties.getConsumerKey()).isEqualTo("consumerKey666");
|
||||
assertThat(twitterConnectionProperties.getConsumerSecret()).isEqualTo("consumerSecret666");
|
||||
assertThat(twitterConnectionProperties.getAccessToken()).isEqualTo("accessToken666");
|
||||
assertThat(twitterConnectionProperties.getAccessTokenSecret()).isEqualTo("accessTokenSecret666");
|
||||
|
||||
DefaultPollerProperties defaultPollerProperties = context.getBean(DefaultPollerProperties.class);
|
||||
assertThat(defaultPollerProperties.getFixedDelay()).isEqualTo(10000);
|
||||
|
||||
TwitterSearchSupplierProperties searchSupplierProperties =
|
||||
context.getBean(TwitterSearchSupplierProperties.class);
|
||||
|
||||
assertThat(searchSupplierProperties.getQuery()).isEqualTo("Amsterdam");
|
||||
assertThat(searchSupplierProperties.getCount()).isEqualTo(3);
|
||||
assertThat(searchSupplierProperties.getPage()).isEqualTo(3);
|
||||
assertThat(searchSupplierProperties.getLang()).isEqualTo("en");
|
||||
assertThat(searchSupplierProperties.getGeocode().getLatitude()).isEqualTo(52.1D);
|
||||
assertThat(searchSupplierProperties.getGeocode().getLongitude()).isEqualTo(4.8D);
|
||||
assertThat(searchSupplierProperties.getGeocode().getRadius()).isEqualTo(10D);
|
||||
assertThat(searchSupplierProperties.getSince()).isEqualTo("2018-01-01");
|
||||
|
||||
OutputDestination outputDestination = context.getBean(OutputDestination.class);
|
||||
// Using local region here
|
||||
Message<byte[]> message = outputDestination.receive(Duration.ofSeconds(300).toMillis());
|
||||
assertThat(message).isNotNull();
|
||||
String payload = new String(message.getPayload());
|
||||
|
||||
List tweets = new ObjectMapper().readValue(payload, List.class);
|
||||
assertThat(tweets).hasSize(3);
|
||||
mockClient.verify(searchAmsterdamRequest, once());
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
public static HttpRequest setExpectation(HttpRequest request) {
|
||||
mockClient
|
||||
.when(request, exactly(1))
|
||||
.respond(response()
|
||||
.withStatusCode(200)
|
||||
.withHeaders(
|
||||
new Header("Content-Type", "application/json; charset=utf-8"),
|
||||
new Header("Cache-Control", "public, max-age=86400"))
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/search_3.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 1)
|
||||
);
|
||||
return request;
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterSearchSupplierConfiguration.class)
|
||||
public static class TestTwitterSearchSourceApplication {
|
||||
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.source.twitter.search;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.function.Function;
|
||||
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTestUtils {
|
||||
|
||||
public Function<ConfigurationBuilder, ConfigurationBuilder> mockTwitterUrls(String baseUrl) {
|
||||
return configBuilder -> {
|
||||
configBuilder.setRestBaseURL(baseUrl + "/");
|
||||
configBuilder.setStreamBaseURL(baseUrl + "/stream/");
|
||||
configBuilder.setUserStreamBaseURL(baseUrl + "/user/");
|
||||
configBuilder.setSiteStreamBaseURL(baseUrl + "/site/");
|
||||
configBuilder.setUploadBaseURL(baseUrl + "/upload/");
|
||||
|
||||
configBuilder.setOAuthAccessTokenURL(baseUrl + "/oauth/access_token");
|
||||
configBuilder.setOAuthAuthenticationURL(baseUrl + "/oauth/authenticate");
|
||||
configBuilder.setOAuthAuthorizationURL(baseUrl + "/oauth/authorize");
|
||||
configBuilder.setOAuthRequestTokenURL(baseUrl + "/oauth/request_token");
|
||||
configBuilder.setOAuth2TokenURL(baseUrl + "/oauth2/token");
|
||||
configBuilder.setOAuth2InvalidateTokenURL(baseUrl + "/oauth2/invalidate_token");
|
||||
|
||||
return configBuilder;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load Spring Resource as String.
|
||||
* @param resourcePath Resource path (accepts file:// , classpath:// and http:// uri schemas)
|
||||
* @return Returns text (UTF8) representation of the resource pointed by the resourcePath
|
||||
*/
|
||||
public static String asString(String resourcePath) {
|
||||
try {
|
||||
return StreamUtils.copyToString(new DefaultResourceLoader().getResource(resourcePath).getInputStream(),
|
||||
Charset.forName("UTF-8"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Can not load resource:" + resourcePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,447 @@
|
||||
{
|
||||
"statuses": [
|
||||
{
|
||||
"created_at": "Sun Jun 21 05:44:20 +0000 2020",
|
||||
"id": 1274578883334078468,
|
||||
"id_str": "1274578883334078468",
|
||||
"text": "\u0423\u0441\u043c\u0438\u0432\u043a\u0438 \u043e\u0442 \u0441\u0442\u0430\u0440\u0438\u0442\u0435 \u0424\u0423\u0422\u0411\u041e\u041b\u041d\u0418 \u043b\u0435\u043d\u0442\u0438: \u041f\u0443\u0431\u043b\u0438\u043a\u0430\u0442\u0430 \u0432\u044a\u0432 \u0412\u0440\u0430\u0446\u0430 \u0435 \u0433\u043b\u0435\u0434\u0430\u043b\u0430 \u0438 \u0437\u0440\u0435\u043b\u0438\u0449\u0430 \u043e\u0442 \u0447\u0443\u0436\u0434\u0435\u043d\u0446\u0438 : https:\/\/t.co\/odbEXBdbGT https:\/\/t.co\/ODL3TbZdIW",
|
||||
"truncated": false,
|
||||
"entities": {
|
||||
"hashtags": [],
|
||||
"symbols": [],
|
||||
"user_mentions": [],
|
||||
"urls": [
|
||||
{
|
||||
"url": "https:\/\/t.co\/odbEXBdbGT",
|
||||
"expanded_url": "http:\/\/Konkurent.bg",
|
||||
"display_url": "Konkurent.bg",
|
||||
"indices": [
|
||||
89,
|
||||
112
|
||||
]
|
||||
},
|
||||
{
|
||||
"url": "https:\/\/t.co\/ODL3TbZdIW",
|
||||
"expanded_url": "https:\/\/www.konkurent.bg\/news\/15913509855903\/usmivki-ot-starite-futbolni-lenti-publikata-vav-vratsa-e-gledala-i-zrelishta-ot-chuzhdentsi#.Xu7zsZ0PotM.twitter",
|
||||
"display_url": "konkurent.bg\/news\/159135098\u2026",
|
||||
"indices": [
|
||||
113,
|
||||
136
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"metadata": {
|
||||
"iso_language_code": "bg",
|
||||
"result_type": "recent"
|
||||
},
|
||||
"source": "\u003ca href=\"https:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003eTwitter Web App\u003c\/a\u003e",
|
||||
"in_reply_to_status_id": null,
|
||||
"in_reply_to_status_id_str": null,
|
||||
"in_reply_to_user_id": null,
|
||||
"in_reply_to_user_id_str": null,
|
||||
"in_reply_to_screen_name": null,
|
||||
"user": {
|
||||
"id": 2793798645,
|
||||
"id_str": "2793798645",
|
||||
"name": "konkurent.bg",
|
||||
"screen_name": "KonkurentBG",
|
||||
"location": "\u0421\u0435\u0432\u0435\u0440\u043e\u0437\u0430\u043f\u0430\u0434\u043d\u0430 \u0411\u044a\u043b\u0433\u0430\u0440\u0438\u044f",
|
||||
"description": "\u201e\u041a\u043e\u043d\u043a\u0443\u0440\u0435\u043d\u0442\u201d \u0435 \u043d\u0430\u0439-\u0433\u043e\u043b\u0435\u043c\u0438\u044f\u0442 \u0438 \u0432\u043b\u0438\u044f\u0442\u0435\u043b\u0435\u043d \u0440\u0435\u0433\u0438\u043e\u043d\u0430\u043b\u0435\u043d \u0432\u0441\u0435\u043a\u0438\u0434\u043d\u0435\u0432\u043d\u0438\u043a \u0432 \u0421\u0435\u0432\u0435\u0440\u043e\u0437\u0430\u043f\u0430\u0434\u043d\u0430 \u0411\u044a\u043b\u0433\u0430\u0440\u0438\u044f. \u0426\u0435\u043d\u0442\u0440\u0430\u043b\u043d\u0438\u044f\u0442 \u043e\u0444\u0438\u0441 \u0438 \u0441\u0435\u0434\u0430\u043b\u0438\u0449\u0435\u0442\u043e \u043d\u0430 \u0434\u0440\u0443\u0436\u0435\u0441\u0442\u0432\u043e\u0442\u043e \u0441\u0430 \u043f\u043e\u0437\u0438\u0446\u0438\u043e\u043d\u0438\u0440\u0430\u043d\u0438 \u0432\u044a\u0432 \u0412\u0440\u0430\u0446\u0430",
|
||||
"url": "http:\/\/t.co\/VMbcih28OB",
|
||||
"entities": {
|
||||
"url": {
|
||||
"urls": [
|
||||
{
|
||||
"url": "http:\/\/t.co\/VMbcih28OB",
|
||||
"expanded_url": "http:\/\/konkurent.bg\/",
|
||||
"display_url": "konkurent.bg",
|
||||
"indices": [
|
||||
0,
|
||||
22
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": {
|
||||
"urls": []
|
||||
}
|
||||
},
|
||||
"protected": false,
|
||||
"followers_count": 1522,
|
||||
"friends_count": 1867,
|
||||
"listed_count": 10,
|
||||
"created_at": "Tue Sep 30 07:35:14 +0000 2014",
|
||||
"favourites_count": 32,
|
||||
"utc_offset": null,
|
||||
"time_zone": null,
|
||||
"geo_enabled": false,
|
||||
"verified": false,
|
||||
"statuses_count": 121700,
|
||||
"lang": null,
|
||||
"contributors_enabled": false,
|
||||
"is_translator": false,
|
||||
"is_translation_enabled": false,
|
||||
"profile_background_color": "C0DEED",
|
||||
"profile_background_image_url": "http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_image_url_https": "https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_tile": true,
|
||||
"profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/608300958607671297\/vNObwz_B_normal.jpg",
|
||||
"profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/608300958607671297\/vNObwz_B_normal.jpg",
|
||||
"profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/2793798645\/1433865279",
|
||||
"profile_link_color": "0084B4",
|
||||
"profile_sidebar_border_color": "FFFFFF",
|
||||
"profile_sidebar_fill_color": "DDEEF6",
|
||||
"profile_text_color": "333333",
|
||||
"profile_use_background_image": true,
|
||||
"has_extended_profile": false,
|
||||
"default_profile": false,
|
||||
"default_profile_image": false,
|
||||
"following": null,
|
||||
"follow_request_sent": null,
|
||||
"notifications": null,
|
||||
"translator_type": "none"
|
||||
},
|
||||
"geo": null,
|
||||
"coordinates": null,
|
||||
"place": null,
|
||||
"contributors": null,
|
||||
"is_quote_status": false,
|
||||
"retweet_count": 0,
|
||||
"favorite_count": 0,
|
||||
"favorited": false,
|
||||
"retweeted": false,
|
||||
"possibly_sensitive": false,
|
||||
"lang": "bg"
|
||||
},
|
||||
{
|
||||
"created_at": "Sun Jun 21 04:37:39 +0000 2020",
|
||||
"id": 1274562101856567297,
|
||||
"id_str": "1274562101856567297",
|
||||
"text": "RT @scparametro2016: \ud83c\udde7\ud83c\uddec #EfbetLeague \u2502 Grupo do Rebaixamento \u2502 Grupo B \u2502 27\u00aa Rodada\n\nBotev Plovdiv 3x2 Botev Vratsa\n\n21\/06 - 13h\nArda x Dun\u2026",
|
||||
"truncated": false,
|
||||
"entities": {
|
||||
"hashtags": [
|
||||
{
|
||||
"text": "EfbetLeague",
|
||||
"indices": [
|
||||
24,
|
||||
36
|
||||
]
|
||||
}
|
||||
],
|
||||
"symbols": [],
|
||||
"user_mentions": [
|
||||
{
|
||||
"screen_name": "scparametro2016",
|
||||
"name": "SC Par\u00e2metro (de \ud83c\udfe0) \u270a\ud83c\udffd",
|
||||
"id": 1093567069201842178,
|
||||
"id_str": "1093567069201842178",
|
||||
"indices": [
|
||||
3,
|
||||
19
|
||||
]
|
||||
}
|
||||
],
|
||||
"urls": []
|
||||
},
|
||||
"metadata": {
|
||||
"iso_language_code": "und",
|
||||
"result_type": "recent"
|
||||
},
|
||||
"source": "\u003ca href=\"https:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003eTwitter Web App\u003c\/a\u003e",
|
||||
"in_reply_to_status_id": null,
|
||||
"in_reply_to_status_id_str": null,
|
||||
"in_reply_to_user_id": null,
|
||||
"in_reply_to_user_id_str": null,
|
||||
"in_reply_to_screen_name": null,
|
||||
"user": {
|
||||
"id": 227357431,
|
||||
"id_str": "227357431",
|
||||
"name": "Igor Sausmikat",
|
||||
"screen_name": "igorsausmikat",
|
||||
"location": "Bras\u00edlia-DF\/SP",
|
||||
"description": "Ex um monte de @, tinha um blog, mas elogio, critico, sugiro sempre que necess\u00e1rio. No mais? S\u00f3 conhecendo melhor.... e ac\u00e1 estamos donde tenemos que estar!!",
|
||||
"url": "https:\/\/t.co\/dyjPEcCeQl",
|
||||
"entities": {
|
||||
"url": {
|
||||
"urls": [
|
||||
{
|
||||
"url": "https:\/\/t.co\/dyjPEcCeQl",
|
||||
"expanded_url": "https:\/\/www.instagram.com\/igorsausmikat\/",
|
||||
"display_url": "instagram.com\/igorsausmikat\/",
|
||||
"indices": [
|
||||
0,
|
||||
23
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": {
|
||||
"urls": []
|
||||
}
|
||||
},
|
||||
"protected": false,
|
||||
"followers_count": 788,
|
||||
"friends_count": 4989,
|
||||
"listed_count": 8,
|
||||
"created_at": "Thu Dec 16 16:41:42 +0000 2010",
|
||||
"favourites_count": 130,
|
||||
"utc_offset": null,
|
||||
"time_zone": null,
|
||||
"geo_enabled": true,
|
||||
"verified": false,
|
||||
"statuses_count": 349115,
|
||||
"lang": null,
|
||||
"contributors_enabled": false,
|
||||
"is_translator": false,
|
||||
"is_translation_enabled": false,
|
||||
"profile_background_color": "C0DEED",
|
||||
"profile_background_image_url": "http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_image_url_https": "https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_tile": false,
|
||||
"profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/1221347767966162944\/zEpQD_Gm_normal.jpg",
|
||||
"profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/1221347767966162944\/zEpQD_Gm_normal.jpg",
|
||||
"profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/227357431\/1546748487",
|
||||
"profile_link_color": "1DA1F2",
|
||||
"profile_sidebar_border_color": "C0DEED",
|
||||
"profile_sidebar_fill_color": "DDEEF6",
|
||||
"profile_text_color": "333333",
|
||||
"profile_use_background_image": true,
|
||||
"has_extended_profile": true,
|
||||
"default_profile": true,
|
||||
"default_profile_image": false,
|
||||
"following": null,
|
||||
"follow_request_sent": null,
|
||||
"notifications": null,
|
||||
"translator_type": "none"
|
||||
},
|
||||
"geo": null,
|
||||
"coordinates": null,
|
||||
"place": null,
|
||||
"contributors": null,
|
||||
"retweeted_status": {
|
||||
"created_at": "Sat Jun 20 20:29:55 +0000 2020",
|
||||
"id": 1274439357991182336,
|
||||
"id_str": "1274439357991182336",
|
||||
"text": "\ud83c\udde7\ud83c\uddec #EfbetLeague \u2502 Grupo do Rebaixamento \u2502 Grupo B \u2502 27\u00aa Rodada\n\nBotev Plovdiv 3x2 Botev Vratsa\n\n21\/06 - 13h\nArda x\u2026 https:\/\/t.co\/7cNRbwTQTb",
|
||||
"truncated": true,
|
||||
"entities": {
|
||||
"hashtags": [
|
||||
{
|
||||
"text": "EfbetLeague",
|
||||
"indices": [
|
||||
3,
|
||||
15
|
||||
]
|
||||
}
|
||||
],
|
||||
"symbols": [],
|
||||
"user_mentions": [],
|
||||
"urls": [
|
||||
{
|
||||
"url": "https:\/\/t.co\/7cNRbwTQTb",
|
||||
"expanded_url": "https:\/\/twitter.com\/i\/web\/status\/1274439357991182336",
|
||||
"display_url": "twitter.com\/i\/web\/status\/1\u2026",
|
||||
"indices": [
|
||||
116,
|
||||
139
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"metadata": {
|
||||
"iso_language_code": "und",
|
||||
"result_type": "recent"
|
||||
},
|
||||
"source": "\u003ca href=\"https:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003eTwitter Web App\u003c\/a\u003e",
|
||||
"in_reply_to_status_id": null,
|
||||
"in_reply_to_status_id_str": null,
|
||||
"in_reply_to_user_id": null,
|
||||
"in_reply_to_user_id_str": null,
|
||||
"in_reply_to_screen_name": null,
|
||||
"user": {
|
||||
"id": 1093567069201842178,
|
||||
"id_str": "1093567069201842178",
|
||||
"name": "SC Par\u00e2metro (de \ud83c\udfe0) \u270a\ud83c\udffd",
|
||||
"screen_name": "scparametro2016",
|
||||
"location": "Mogi das Cruzes, Brasil",
|
||||
"description": "Cobertura das principais divis\u00f5es masculinas e femininas com bola rolando no mundo!",
|
||||
"url": "https:\/\/t.co\/yOzONBTTCo",
|
||||
"entities": {
|
||||
"url": {
|
||||
"urls": [
|
||||
{
|
||||
"url": "https:\/\/t.co\/yOzONBTTCo",
|
||||
"expanded_url": "https:\/\/www.facebook.com\/scparametro2016\/",
|
||||
"display_url": "facebook.com\/scparametro201\u2026",
|
||||
"indices": [
|
||||
0,
|
||||
23
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": {
|
||||
"urls": []
|
||||
}
|
||||
},
|
||||
"protected": false,
|
||||
"followers_count": 3608,
|
||||
"friends_count": 2336,
|
||||
"listed_count": 11,
|
||||
"created_at": "Thu Feb 07 17:48:01 +0000 2019",
|
||||
"favourites_count": 9961,
|
||||
"utc_offset": null,
|
||||
"time_zone": null,
|
||||
"geo_enabled": false,
|
||||
"verified": false,
|
||||
"statuses_count": 24226,
|
||||
"lang": null,
|
||||
"contributors_enabled": false,
|
||||
"is_translator": false,
|
||||
"is_translation_enabled": false,
|
||||
"profile_background_color": "000000",
|
||||
"profile_background_image_url": "http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_image_url_https": "https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_tile": false,
|
||||
"profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/1267574634523824128\/xreXNzJZ_normal.jpg",
|
||||
"profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/1267574634523824128\/xreXNzJZ_normal.jpg",
|
||||
"profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/1093567069201842178\/1584658723",
|
||||
"profile_link_color": "000080",
|
||||
"profile_sidebar_border_color": "000000",
|
||||
"profile_sidebar_fill_color": "000000",
|
||||
"profile_text_color": "000000",
|
||||
"profile_use_background_image": false,
|
||||
"has_extended_profile": true,
|
||||
"default_profile": false,
|
||||
"default_profile_image": false,
|
||||
"following": null,
|
||||
"follow_request_sent": null,
|
||||
"notifications": null,
|
||||
"translator_type": "none"
|
||||
},
|
||||
"geo": null,
|
||||
"coordinates": null,
|
||||
"place": null,
|
||||
"contributors": null,
|
||||
"is_quote_status": false,
|
||||
"retweet_count": 2,
|
||||
"favorite_count": 7,
|
||||
"favorited": false,
|
||||
"retweeted": false,
|
||||
"possibly_sensitive": false,
|
||||
"lang": "und"
|
||||
},
|
||||
"is_quote_status": false,
|
||||
"retweet_count": 2,
|
||||
"favorite_count": 0,
|
||||
"favorited": false,
|
||||
"retweeted": false,
|
||||
"lang": "und"
|
||||
},
|
||||
{
|
||||
"created_at": "Sun Jun 21 02:26:46 +0000 2020",
|
||||
"id": 1274529164138287107,
|
||||
"id_str": "1274529164138287107",
|
||||
"text": "Campeonato B\u00falgaro - 27\u00b0 Rodada - Playoff de Rebaixamento - Grupo B\n\nBotev Plovdiv 3 x 2 Botev Vratsa\u2026 https:\/\/t.co\/TxFgoqIfGF",
|
||||
"truncated": true,
|
||||
"entities": {
|
||||
"hashtags": [],
|
||||
"symbols": [],
|
||||
"user_mentions": [],
|
||||
"urls": [
|
||||
{
|
||||
"url": "https:\/\/t.co\/TxFgoqIfGF",
|
||||
"expanded_url": "https:\/\/twitter.com\/i\/web\/status\/1274529164138287107",
|
||||
"display_url": "twitter.com\/i\/web\/status\/1\u2026",
|
||||
"indices": [
|
||||
104,
|
||||
127
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"metadata": {
|
||||
"iso_language_code": "pt",
|
||||
"result_type": "recent"
|
||||
},
|
||||
"source": "\u003ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003eTwitter for Android\u003c\/a\u003e",
|
||||
"in_reply_to_status_id": null,
|
||||
"in_reply_to_status_id_str": null,
|
||||
"in_reply_to_user_id": null,
|
||||
"in_reply_to_user_id_str": null,
|
||||
"in_reply_to_screen_name": null,
|
||||
"user": {
|
||||
"id": 1110343004768874496,
|
||||
"id_str": "1110343004768874496",
|
||||
"name": "Loucos por futebol nacional e Internacional \u26bd",
|
||||
"screen_name": "EsportePor",
|
||||
"location": "Rio de Janeiro, Brasil",
|
||||
"description": "Canal sobre futebol mundial, not\u00edcias, resultados, classifica\u00e7\u00e3o e goleadores.",
|
||||
"url": null,
|
||||
"entities": {
|
||||
"description": {
|
||||
"urls": []
|
||||
}
|
||||
},
|
||||
"protected": false,
|
||||
"followers_count": 664,
|
||||
"friends_count": 891,
|
||||
"listed_count": 4,
|
||||
"created_at": "Tue Mar 26 00:49:36 +0000 2019",
|
||||
"favourites_count": 2990,
|
||||
"utc_offset": null,
|
||||
"time_zone": null,
|
||||
"geo_enabled": false,
|
||||
"verified": false,
|
||||
"statuses_count": 7437,
|
||||
"lang": null,
|
||||
"contributors_enabled": false,
|
||||
"is_translator": false,
|
||||
"is_translation_enabled": false,
|
||||
"profile_background_color": "000000",
|
||||
"profile_background_image_url": "http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_image_url_https": "https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
|
||||
"profile_background_tile": false,
|
||||
"profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/1110343459523710981\/mfUCJj_N_normal.jpg",
|
||||
"profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/1110343459523710981\/mfUCJj_N_normal.jpg",
|
||||
"profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/1110343004768874496\/1553561950",
|
||||
"profile_link_color": "1B95E0",
|
||||
"profile_sidebar_border_color": "000000",
|
||||
"profile_sidebar_fill_color": "000000",
|
||||
"profile_text_color": "000000",
|
||||
"profile_use_background_image": false,
|
||||
"has_extended_profile": false,
|
||||
"default_profile": false,
|
||||
"default_profile_image": false,
|
||||
"following": null,
|
||||
"follow_request_sent": null,
|
||||
"notifications": null,
|
||||
"translator_type": "none"
|
||||
},
|
||||
"geo": null,
|
||||
"coordinates": null,
|
||||
"place": null,
|
||||
"contributors": null,
|
||||
"is_quote_status": false,
|
||||
"retweet_count": 0,
|
||||
"favorite_count": 0,
|
||||
"favorited": false,
|
||||
"retweeted": false,
|
||||
"possibly_sensitive": false,
|
||||
"lang": "pt"
|
||||
}
|
||||
],
|
||||
"search_metadata": {
|
||||
"completed_in": 0.028,
|
||||
"max_id": 1274578883334078468,
|
||||
"max_id_str": "1274578883334078468",
|
||||
"next_results": "?max_id=1274529164138287106&q=Vratsa&count=3&include_entities=1&result_type=mixed",
|
||||
"query": "Vratsa",
|
||||
"refresh_url": "?since_id=1274578883334078468&q=Vratsa&result_type=mixed&include_entities=1",
|
||||
"count": 3,
|
||||
"since_id": 0,
|
||||
"since_id_str": "0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"created_at":"Thu Dec 20 13:56:10 +0000 2018","id":1075751718749659136,"id_str":"1075751718749659136","text":"Codementor: Kubernetes for Python Developers: Part 1\n#100daysofcode #python https:\/\/t.co\/VviTgpFpge","source":"\u003ca href=\"https:\/\/ifttt.com\" rel=\"nofollow\"\u003eIFTTT\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":859252650512072704,"id_str":"859252650512072704","name":"Freelancing|WebDev","screen_name":"FreelanceForBTC","location":"THE NET","url":"http:\/\/bit.ly\/BTCFREELANCING","description":"This twitter is designed to give the best information from web development to Blockchain.\n|YouTube's |Jobs |learning contented","translator_type":"none","protected":false,"verified":false,"followers_count":146,"friends_count":118,"listed_count":4,"favourites_count":445,"statuses_count":7528,"created_at":"Tue May 02 03:46:10 +0000 2017","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/859252650512072704\/1544326203","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"100daysofcode","indices":[53,67]},{"text":"python","indices":[68,75]}],"urls":[{"url":"https:\/\/t.co\/VviTgpFpge","expanded_url":"https:\/\/ift.tt\/2GxdMZF","display_url":"ift.tt\/2GxdMZF","indices":[76,99]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"ca","timestamp_ms":"1545314170907"}
|
||||
43
applications/source/twitter-stream-source/README.adoc
Normal file
43
applications/source/twitter-stream-source/README.adoc
Normal file
@@ -0,0 +1,43 @@
|
||||
//tag::ref-doc[]
|
||||
= Twitter Stream Source
|
||||
|
||||
Real-time Tweet streaming https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html[Filter] and https://developer.twitter.com/en/docs/tweets/sample-realtime/overview/GET_statuse_sample[Sample] APIs support.
|
||||
|
||||
* The `Filter API` returns public statuses that match one or more filter predicates.
|
||||
Multiple parameters allows using a single connection to the Streaming API.
|
||||
TIP: The `track`, `follow`, and `locations` fields are combined with an *OR* operator!
|
||||
Queries with `track=foo` and `follow=1234` returns Tweets matching `foo` *OR* created by user `1234`.
|
||||
|
||||
* The `Sample API` returns a small random sample of all public statuses.
|
||||
The Tweets returned by the default access level are the same, so if two different clients connect to this endpoint, they will see the same Tweets.
|
||||
|
||||
The default access level allows up to 400 track keywords, 5,000 follow user Ids and 25 0.1-360 degree location boxes.
|
||||
|
||||
== Options
|
||||
|
||||
//tag::configuration-properties[]
|
||||
$$twitter.connection.access-token$$:: $$Your Twitter token.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.access-token-secret$$:: $$Your Twitter token secret.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.consumer-key$$:: $$Your Twitter key.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.consumer-secret$$:: $$Your Twitter secret.$$ *($$String$$, default: `$$<none>$$`)*
|
||||
$$twitter.connection.debug-enabled$$:: $$Enables Twitter4J debug mode.$$ *($$Boolean$$, default: `$$false$$`)*
|
||||
$$twitter.connection.raw-json$$:: $$Enable caching the original (raw) JSON objects as returned by the Twitter APIs. When set to False the result will use the Twitter4J's json representations. When set to True the result will use the original Twitter APISs json representations.$$ *($$Boolean$$, default: `$$true$$`)*
|
||||
$$twitter.stream.filter.count$$:: $$Indicates the number of previous statuses to stream before transitioning to the live stream.$$ *($$Integer$$, default: `$$0$$`)*
|
||||
$$twitter.stream.filter.filter-level$$:: $$The filter level limits what tweets appear in the stream to those with a minimum filterLevel attribute value. One of either none, low, or medium.$$ *($$FilterLevel$$, default: `$$<none>$$`)*
|
||||
$$twitter.stream.filter.follow$$:: $$Specifies the users, by ID, to receive public tweets from.$$ *($$List<Long>$$, default: `$$<none>$$`)*
|
||||
$$twitter.stream.filter.language$$:: $$Specifies the tweets language of the stream.$$ *($$List<String>$$, default: `$$<none>$$`)*
|
||||
$$twitter.stream.filter.locations$$:: $$Locations to track. Internally represented as 2D array. Bounding box is invalid: 52.38, 4.90, 51.51, -0.12. The first pair must be the SW corner of the box$$ *($$List<BoundingBox>$$, default: `$$<none>$$`)*
|
||||
$$twitter.stream.filter.track$$:: $$Specifies keywords to track.$$ *($$List<String>$$, default: `$$<none>$$`)*
|
||||
$$twitter.stream.type$$:: $$<documentation missing>$$ *($$StreamType$$, default: `$$<none>$$`, possible values: `sample`,`filter`,`firehose`,`link`)*
|
||||
//end::configuration-properties[]
|
||||
|
||||
//end::ref-doc[]
|
||||
|
||||
|
||||
== Examples
|
||||
|
||||
```
|
||||
java -jar twitter-stream-source.jar --twitter.connection.accessTokenSecret=xxx --twitter.connection.accessToken=xxx --twitter.connection.consumerKey=xxx --twitter.connection.consumerSecret=xxx
|
||||
```
|
||||
|
||||
|
||||
93
applications/source/twitter-stream-source/pom.xml
Normal file
93
applications/source/twitter-stream-source/pom.xml
Normal file
@@ -0,0 +1,93 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-stream-source</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<name>twitter-stream-source</name>
|
||||
<description>twitter stream source apps</description>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.stream.app</groupId>
|
||||
<artifactId>stream-applications-core</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-supplier</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>function-test-support</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud</groupId>
|
||||
<artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.springframework.cloud.stream.app.plugin</groupId>
|
||||
<artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<generatedApp>
|
||||
<name>twitter-stream</name>
|
||||
<type>source</type>
|
||||
<version>${project.version}</version>
|
||||
<configClass>org.springframework.cloud.fn.supplier.twitter.status.stream.TwitterStreamSupplierConfiguration.class</configClass>
|
||||
<functionDefinition>twitterStreamSupplier</functionDefinition>
|
||||
</generatedApp>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-supplier</artifactId>
|
||||
<version>${java-functions.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
<repositories>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>true</enabled>
|
||||
</snapshots>
|
||||
<id>spring-snapshots</id>
|
||||
<name>Spring Snapshots</name>
|
||||
<url>https://repo.spring.io/libs-snapshot-local</url>
|
||||
</repository>
|
||||
<repository>
|
||||
<snapshots>
|
||||
<enabled>false</enabled>
|
||||
</snapshots>
|
||||
<id>spring-milestones</id>
|
||||
<name>Spring Milestones</name>
|
||||
<url>https://repo.spring.io/libs-milestone-local</url>
|
||||
</repository>
|
||||
</repositories>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,5 @@
|
||||
configuration-properties.classes=org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties,\
|
||||
org.springframework.cloud.fn.supplier.twitter.status.stream.TwitterStreamSupplierProperties, \
|
||||
org.springframework.cloud.fn.supplier.twitter.status.stream.TwitterStreamSupplierProperties$Filter, \
|
||||
org.springframework.cloud.fn.supplier.twitter.status.stream.TwitterStreamSupplierProperties$Filter$BoundingBox, \
|
||||
org.springframework.cloud.fn.supplier.twitter.status.stream.TwitterStreamSupplierProperties$Filter$Geocode
|
||||
@@ -0,0 +1,164 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.source.twitter.stream;
|
||||
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Disabled;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.Header;
|
||||
import org.mockserver.model.HttpRequest;
|
||||
import org.mockserver.model.StringBody;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.WebApplicationType;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.builder.SpringApplicationBuilder;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.cloud.fn.supplier.twitter.status.stream.TwitterStreamSupplierConfiguration;
|
||||
import org.springframework.cloud.fn.supplier.twitter.status.stream.TwitterStreamSupplierProperties;
|
||||
import org.springframework.cloud.stream.binder.test.TestChannelBinderConfiguration;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.exactly;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
public class TwitterStreamSourceTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
private static HttpRequest streamFilterRequest;
|
||||
private static HttpRequest streamSampleRequest;
|
||||
private static HttpRequest streamFirehoseRequest;
|
||||
private static HttpRequest streamLinknsRequest;
|
||||
|
||||
@BeforeAll
|
||||
public static void startServer() {
|
||||
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
|
||||
streamFilterRequest = mockClientRecordRequest(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/stream/statuses/filter.json")
|
||||
.withBody(new StringBody("count=0&track=Java%2CPython&stall_warnings=true")));
|
||||
|
||||
streamSampleRequest = mockClientRecordRequest(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/stream/statuses/sample.json"));
|
||||
|
||||
streamLinknsRequest = mockClientRecordRequest(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/stream/statuses/links.json"));
|
||||
//.withBody(new StringBody("count=0&stall_warnings=true")));
|
||||
|
||||
streamFirehoseRequest = mockClientRecordRequest(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/stream/statuses/firehose.json")
|
||||
.withBody(new StringBody("count=0&stall_warnings=true")));
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
public static void stopServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
@Test
|
||||
@Disabled
|
||||
public void testSourceFromSupplier() {
|
||||
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(
|
||||
TestChannelBinderConfiguration.getCompleteConfiguration(TestTwitterStreamSourceApplication.class))
|
||||
.web(WebApplicationType.NONE)
|
||||
.run("--spring.cloud.stream.function.definition=twitterStreamSupplier",
|
||||
|
||||
"--twitter.connection.consumerKey=consumerKey666",
|
||||
"--twitter.connection.consumerSecret=consumerSecret666",
|
||||
"--twitter.connection.accessToken=accessToken666",
|
||||
"--twitter.connection.accessTokenSecret=accessTokenSecret666",
|
||||
|
||||
"--twitter.stream.type=filter",
|
||||
"--twitter.stream.filter.track=Java,Python",
|
||||
"--twitter.stream.filter.count=3")) {
|
||||
|
||||
TwitterConnectionProperties twitterConnectionProperties = context.getBean(TwitterConnectionProperties.class);
|
||||
assertThat(twitterConnectionProperties.getConsumerKey()).isEqualTo("consumerKey666");
|
||||
assertThat(twitterConnectionProperties.getConsumerSecret()).isEqualTo("consumerSecret666");
|
||||
assertThat(twitterConnectionProperties.getAccessToken()).isEqualTo("accessToken666");
|
||||
assertThat(twitterConnectionProperties.getAccessTokenSecret()).isEqualTo("accessTokenSecret666");
|
||||
|
||||
TwitterStreamSupplierProperties twitterStreamSupplierProperties =
|
||||
context.getBean(TwitterStreamSupplierProperties.class);
|
||||
assertThat(twitterStreamSupplierProperties.getType()).isEqualTo(TwitterStreamSupplierProperties.StreamType.filter);
|
||||
assertThat(twitterStreamSupplierProperties.getFilter().getTrack()).contains("Java", "Python");
|
||||
|
||||
//OutputDestination target = context.getBean(OutputDestination.class);
|
||||
//Message<byte[]> sourceMessage = target.receive(10000);
|
||||
//final String actual = new String(sourceMessage.getPayload());
|
||||
|
||||
mockClient.verify(streamFilterRequest, once());
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpRequest mockClientRecordRequest(HttpRequest request) {
|
||||
mockClient.when(request, exactly(1))
|
||||
.respond(response()
|
||||
.withStatusCode(200)
|
||||
.withHeaders(
|
||||
new Header("Content-Type", "application/json; charset=utf-8"),
|
||||
new Header("Cache-Control", "public, max-age=86400"))
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/stream_test_1.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 10));
|
||||
return request;
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterStreamSupplierConfiguration.class)
|
||||
public static class TestTwitterStreamSourceApplication {
|
||||
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.stream.app.source.twitter.stream;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.function.Function;
|
||||
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTestUtils {
|
||||
|
||||
public Function<ConfigurationBuilder, ConfigurationBuilder> mockTwitterUrls(String baseUrl) {
|
||||
return configBuilder -> {
|
||||
configBuilder.setRestBaseURL(baseUrl + "/");
|
||||
configBuilder.setStreamBaseURL(baseUrl + "/stream/");
|
||||
configBuilder.setUserStreamBaseURL(baseUrl + "/user/");
|
||||
configBuilder.setSiteStreamBaseURL(baseUrl + "/site/");
|
||||
configBuilder.setUploadBaseURL(baseUrl + "/upload/");
|
||||
|
||||
configBuilder.setOAuthAccessTokenURL(baseUrl + "/oauth/access_token");
|
||||
configBuilder.setOAuthAuthenticationURL(baseUrl + "/oauth/authenticate");
|
||||
configBuilder.setOAuthAuthorizationURL(baseUrl + "/oauth/authorize");
|
||||
configBuilder.setOAuthRequestTokenURL(baseUrl + "/oauth/request_token");
|
||||
configBuilder.setOAuth2TokenURL(baseUrl + "/oauth2/token");
|
||||
configBuilder.setOAuth2InvalidateTokenURL(baseUrl + "/oauth2/invalidate_token");
|
||||
|
||||
return configBuilder;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load Spring Resource as String.
|
||||
* @param resourcePath Resource path (accepts file:// , classpath:// and http:// uri schemas)
|
||||
* @return Returns text (UTF8) representation of the resource pointed by the resourcePath
|
||||
*/
|
||||
public static String asString(String resourcePath) {
|
||||
try {
|
||||
return StreamUtils.copyToString(new DefaultResourceLoader().getResource(resourcePath).getInputStream(),
|
||||
Charset.forName("UTF-8"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Can not load resource:" + resourcePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"created_at":"Thu Dec 20 13:56:10 +0000 2018","id":1075751718749659136,"id_str":"1075751718749659136","text":"Codementor: Kubernetes for Python Developers: Part 1\n#100daysofcode #python https:\/\/t.co\/VviTgpFpge","source":"\u003ca href=\"https:\/\/ifttt.com\" rel=\"nofollow\"\u003eIFTTT\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":859252650512072704,"id_str":"859252650512072704","name":"Freelancing|WebDev","screen_name":"FreelanceForBTC","location":"THE NET","url":"http:\/\/bit.ly\/BTCFREELANCING","description":"This twitter is designed to give the best information from web development to Blockchain.\n|YouTube's |Jobs |learning contented","translator_type":"none","protected":false,"verified":false,"followers_count":146,"friends_count":118,"listed_count":4,"favourites_count":445,"statuses_count":7528,"created_at":"Tue May 02 03:46:10 +0000 2017","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/859252650512072704\/1544326203","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"100daysofcode","indices":[53,67]},{"text":"python","indices":[68,75]}],"urls":[{"url":"https:\/\/t.co\/VviTgpFpge","expanded_url":"https:\/\/ift.tt\/2GxdMZF","display_url":"ift.tt\/2GxdMZF","indices":[76,99]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"ca","timestamp_ms":"1545314170907"}
|
||||
@@ -50,6 +50,7 @@
|
||||
<spring-cloud-build.version>3.0.0-SNAPSHOT</spring-cloud-build.version>
|
||||
|
||||
<maven-javadoc-plugin.version>3.1.1</maven-javadoc-plugin.version>
|
||||
<mockserver.version>5.10</mockserver.version>
|
||||
</properties>
|
||||
|
||||
<modules>
|
||||
@@ -69,6 +70,16 @@
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>${mockserver.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>${mockserver.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
|
||||
56
functions/common/twitter-common/pom.xml
Normal file
56
functions/common/twitter-common/pom.xml
Normal file
@@ -0,0 +1,56 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-common</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<name>twitter-common</name>
|
||||
<description>Twitter common</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>spring-functions-parent</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../spring-functions-parent</relativePath>
|
||||
</parent>
|
||||
|
||||
<properties>
|
||||
<twitter4j.version>4.0.7</twitter4j.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.twitter4j</groupId>
|
||||
<artifactId>twitter4j-stream</artifactId>
|
||||
<version>${twitter4j.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-json</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-validation</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.integration</groupId>
|
||||
<artifactId>spring-integration-ip</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-integration</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-configuration-processor</artifactId>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,37 @@
|
||||
/*
|
||||
* Copyright 2015-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.common.twitter;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class Cursor {
|
||||
private long cursor = -1;
|
||||
|
||||
public long getCursor() {
|
||||
return cursor;
|
||||
}
|
||||
|
||||
public void updateCursor(long newCursor) {
|
||||
this.cursor = (newCursor > 0) ? newCursor : -1;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "Cursor{cursor=" + cursor + '}';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
/*
|
||||
* Copyright 2015-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.common.twitter;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.boot.autoconfigure.condition.NoneNestedConditions;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class OnMissingStreamFunctionDefinitionCondition extends NoneNestedConditions {
|
||||
|
||||
public OnMissingStreamFunctionDefinitionCondition() {
|
||||
super(ConfigurationPhase.REGISTER_BEAN);
|
||||
}
|
||||
|
||||
@ConditionalOnProperty(name = "spring.cloud.stream.function.definition")
|
||||
static class OnFunctionDslProperty {
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
/*
|
||||
* Copyright 2015-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.common.twitter;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.function.Function;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterFactory;
|
||||
import twitter4j.TwitterObjectFactory;
|
||||
import twitter4j.TwitterStream;
|
||||
import twitter4j.TwitterStreamFactory;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.messaging.MessageHeaders;
|
||||
import org.springframework.messaging.support.MessageBuilder;
|
||||
import org.springframework.util.MimeTypeUtils;
|
||||
|
||||
//import org.springframework.cloud.stream.config.BindingProperties;
|
||||
// import org.springframework.integration.support.MutableMessage;
|
||||
|
||||
/**
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@Configuration
|
||||
@EnableConfigurationProperties({ TwitterConnectionProperties.class })
|
||||
public class TwitterConnectionConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterConnectionConfiguration.class);
|
||||
|
||||
@Bean
|
||||
public twitter4j.conf.Configuration twitterConfiguration(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
return toConfigurationBuilder.apply(properties).build();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Twitter twitter(twitter4j.conf.Configuration configuration) {
|
||||
return new TwitterFactory(configuration).getInstance();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public TwitterStream twitterStream(twitter4j.conf.Configuration configuration) {
|
||||
return new TwitterStreamFactory(configuration).getInstance();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder() {
|
||||
return properties -> new ConfigurationBuilder()
|
||||
.setJSONStoreEnabled(properties.isRawJson())
|
||||
.setDebugEnabled(properties.isDebugEnabled())
|
||||
.setOAuthConsumerKey(properties.getConsumerKey())
|
||||
.setOAuthConsumerSecret(properties.getConsumerSecret())
|
||||
.setOAuthAccessToken(properties.getAccessToken())
|
||||
.setOAuthAccessTokenSecret(properties.getAccessTokenSecret());
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<Object, Message<byte[]>> json(ObjectMapper mapper) {
|
||||
return objects -> {
|
||||
try {
|
||||
String json = mapper.writeValueAsString(objects);
|
||||
|
||||
return MessageBuilder
|
||||
.withPayload(json.getBytes())
|
||||
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON_VALUE)
|
||||
.build();
|
||||
}
|
||||
catch (JsonProcessingException e) {
|
||||
logger.error("Status to JSON conversion error!", e);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieves the raw JSON form of the provided object.
|
||||
*
|
||||
* Note that raw JSON forms can be retrieved only from the same thread invoked the last method
|
||||
* call and will become inaccessible once another method call.
|
||||
*
|
||||
* @return Function that can retrieve the raw JSON object from the objects returned by the Twitter4J's APIs.
|
||||
*/
|
||||
@Bean
|
||||
public Function<Object, Object> rawJsonExtractor() {
|
||||
return response -> {
|
||||
if (response instanceof List) {
|
||||
List responses = (List) response;
|
||||
List<String> rawJsonList = new ArrayList<>();
|
||||
for (Object object : responses) {
|
||||
rawJsonList.add(TwitterObjectFactory.getRawJSON(object));
|
||||
}
|
||||
return rawJsonList;
|
||||
}
|
||||
else {
|
||||
return TwitterObjectFactory.getRawJSON(response);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<Object, Message<byte[]>> managedJson(TwitterConnectionProperties properties,
|
||||
Function<Object, Object> rawJsonExtractor, Function<Object, Message<byte[]>> json) {
|
||||
return list -> (properties.isRawJson()) ? rawJsonExtractor.andThen(json).apply(list) : json.apply(list);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
/*
|
||||
* Copyright 2015-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.common.twitter;
|
||||
|
||||
import javax.validation.constraints.NotEmpty;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.connection")
|
||||
@Validated
|
||||
public class TwitterConnectionProperties {
|
||||
|
||||
/**
|
||||
* Your Twitter key.
|
||||
*/
|
||||
@NotEmpty
|
||||
private String consumerKey;
|
||||
|
||||
/**
|
||||
* Your Twitter secret.
|
||||
*/
|
||||
@NotEmpty
|
||||
private String consumerSecret;
|
||||
|
||||
/**
|
||||
* Your Twitter token.
|
||||
*/
|
||||
@NotEmpty
|
||||
private String accessToken;
|
||||
|
||||
/**
|
||||
* Your Twitter token secret.
|
||||
*/
|
||||
@NotEmpty
|
||||
private String accessTokenSecret;
|
||||
|
||||
/**
|
||||
* Enables Twitter4J debug mode.
|
||||
*/
|
||||
private boolean debugEnabled = false;
|
||||
|
||||
/**
|
||||
* Enable caching the original (raw) JSON objects as returned by the Twitter APIs.
|
||||
* When set to False the result will use the Twitter4J's json representations.
|
||||
* When set to True the result will use the original Twitter APISs json representations.
|
||||
*/
|
||||
private boolean rawJson = true;
|
||||
|
||||
public String getConsumerKey() {
|
||||
return consumerKey;
|
||||
}
|
||||
|
||||
public void setConsumerKey(String consumerKey) {
|
||||
this.consumerKey = consumerKey;
|
||||
}
|
||||
|
||||
public String getConsumerSecret() {
|
||||
return consumerSecret;
|
||||
}
|
||||
|
||||
public void setConsumerSecret(String consumerSecret) {
|
||||
this.consumerSecret = consumerSecret;
|
||||
}
|
||||
|
||||
public String getAccessToken() {
|
||||
return accessToken;
|
||||
}
|
||||
|
||||
public void setAccessToken(String accessToken) {
|
||||
this.accessToken = accessToken;
|
||||
}
|
||||
|
||||
public String getAccessTokenSecret() {
|
||||
return accessTokenSecret;
|
||||
}
|
||||
|
||||
public void setAccessTokenSecret(String accessTokenSecret) {
|
||||
this.accessTokenSecret = accessTokenSecret;
|
||||
}
|
||||
|
||||
public boolean isDebugEnabled() {
|
||||
return debugEnabled;
|
||||
}
|
||||
|
||||
public void setDebugEnabled(boolean debugEnabled) {
|
||||
this.debugEnabled = debugEnabled;
|
||||
}
|
||||
|
||||
public boolean isRawJson() {
|
||||
return this.rawJson;
|
||||
}
|
||||
|
||||
public void setRawJson(boolean rawJson) {
|
||||
this.rawJson = rawJson;
|
||||
}
|
||||
}
|
||||
110
functions/consumer/twitter-consumer/README.adoc
Normal file
110
functions/consumer/twitter-consumer/README.adoc
Normal file
@@ -0,0 +1,110 @@
|
||||
# Twitter Consumers
|
||||
|
||||
|
||||
## 1. Twitter Status Update Consumer.
|
||||
|
||||
Updates the authenticating user's current text (e.g Tweeting).
|
||||
|
||||
NOTE: For each update attempt, the update text is compared with the authenticating user's recent Tweets.
|
||||
Any attempt that would result in duplication will be blocked, resulting in a 403 error.
|
||||
A user cannot submit the same text twice in a row.
|
||||
|
||||
While not rate limited by the API, a user is limited in the number of Tweets they can create at a time.
|
||||
The update limit for standard API is 300 in 3 hours windows.
|
||||
If the number of updates posted by the user reaches the current allowed limit this method will return an HTTP 403 error.
|
||||
|
||||
You can find details for the Update API here: https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/post-statuses-update
|
||||
|
||||
|
||||
### 1.1 Beans for injection
|
||||
|
||||
You can import `TwitterUpdateConsumerConfiguration` in the application and then inject the following beans.
|
||||
|
||||
- `Consumer<StatusUpdate> updateStatus` - if you have an `StatusUpdate` instance you can use the `updateStatus` to apply it.
|
||||
|
||||
- `Function<Message<?>, StatusUpdate> toStatusUpdateQuery` - function that converts a `Message<?>` text into a `StatusUpdate` instance using the `TwitterUpdateConsumerProperties` properties.
|
||||
|
||||
- `Consumer<Message<?>> twitterStatusUpdateConsumer` - composes `toStatusUpdateQuery` and `updateStatus` to update the twitter status from Message text.
|
||||
|
||||
Note: the Message content is expected to be in text format. Consider using the `byteArrayTextToString` utility `Function`.
|
||||
|
||||
You can use `twitterStatusUpdateConsumer` as a qualifier when injecting.
|
||||
|
||||
### 1.2 Configuration Options
|
||||
|
||||
All configuration properties are prefixed with `twitter.update`.
|
||||
|
||||
For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/consumer/twitter/status/update/TwitterUpdateConsumerProperties.java[TwitterUpdateConsumerProperties].
|
||||
|
||||
The twitter function makes uses of link:../spel-function/README.adoc[SpEL function].
|
||||
|
||||
### 1.3 Other usage
|
||||
|
||||
See this https://github.com/spring-cloud/stream-applications/blob/master/applications/sink/twitter-update-sink/README.adoc[README] where this consumer is used to create a Spring Cloud Stream application where it makes a Twitter Update sink.
|
||||
|
||||
## 2. Twitter Message Consumer.
|
||||
|
||||
Send Direct Messages to a specified user from the authenticating user.
|
||||
Requires a JSON POST body and `Content-Type` header to be set to `application/json`.
|
||||
|
||||
NOTE: When a message is received from a user you may send up to 5 messages in response within a 24 hour window.
|
||||
Each message received resets the 24 hour window and the 5 allotted messages.
|
||||
Sending a 6th message within a 24 hour window or sending a message outside of a 24 hour window will count towards rate-limiting.
|
||||
This behavior only applies when using the POST direct_messages/events/new endpoint.
|
||||
|
||||
SpEL expressions are used to compute the request parameters from the input message.
|
||||
|
||||
### 2.1 Beans for injection
|
||||
|
||||
You can import `TwitterMessageConsumerConfiguration` in the application and then inject the following bean.
|
||||
|
||||
- `Consumer<Message<?>> sendDirectMessageConsumer`
|
||||
|
||||
Note: the Message content is expected to be in text format. Consider using the `byteArrayTextToString` utility `Function`.
|
||||
|
||||
You can use `twitterStatusUpdateConsumer` as a qualifier when injecting.
|
||||
|
||||
### 2.2 Configuration Options
|
||||
|
||||
All configuration properties are prefixed with `twitter.message.update`.
|
||||
|
||||
For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/consumer/twitter/message/TwitterMessageConsumerProperties.java[TwitterMessageConsumerProperties].
|
||||
|
||||
The twitter function makes uses of link:../spel-function/README.adoc[SpEL function].
|
||||
|
||||
### 2.3 Other usage
|
||||
|
||||
See this https://github.com/spring-cloud/stream-applications/blob/master/applications/sink/twitter-message-sink/README.adoc[README] where this consumer is used to create a Spring Cloud Stream application where it makes a Twitter Message sink.
|
||||
|
||||
## 3. Twitter Friendship Consumer.
|
||||
|
||||
Allows creating `follow`, `unfollow` and `update` relationships with specified `userId` or `screenName`.
|
||||
The `twitter.friendships.sink.type` property allows to select the desired friendship operation.
|
||||
|
||||
* https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/post-friendships-create[Friendships Create API] - Allows the authenticating user to follow (friend) the user specified in the ID parameter.
|
||||
Actions taken in this method are asynchronous.
|
||||
Changes will be eventually consistent.
|
||||
* https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/post-friendships-update[Friendships Update API] - Enable or disable Retweets and device notifications from the specified user.
|
||||
* https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/post-friendships-destroy[Friendships Destroy API] - Allows the authenticating user to unfollow the user specified in the ID parameter.
|
||||
|
||||
SpEL expressions are used to compute the request parameters from the input message.
|
||||
Every operation type has its own parameters.
|
||||
|
||||
### 3.1 Beans for injection
|
||||
|
||||
You can import `TwitterFriendshipsConsumerConfiguration` in the application and then inject the following bean.
|
||||
|
||||
- `Consumer<Message<?>> friendshipConsumer`
|
||||
|
||||
Note: the Message content is expected to be in text format. Consider using the `byteArrayTextToString` utility `Function`.
|
||||
|
||||
You can use `friendshipConsumer` as a qualifier when injecting.
|
||||
|
||||
### 3.2 Configuration Options
|
||||
|
||||
All configuration properties are prefixed with `twitter.friendships.update`.
|
||||
|
||||
For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/consumer/twitter/friendship/TwitterFriendshipsConsumerProperties.java[TwitterFriendshipsConsumerProperties].
|
||||
|
||||
The twitter function makes uses of link:../spel-function/README.adoc[SpEL function].
|
||||
|
||||
57
functions/consumer/twitter-consumer/pom.xml
Normal file
57
functions/consumer/twitter-consumer/pom.xml
Normal file
@@ -0,0 +1,57 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-consumer</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<name>twitter-consumer</name>
|
||||
<description>twitter consumer</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>spring-functions-parent</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../spring-functions-parent</relativePath>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-common</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>payload-converter-function</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-integration</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-validation</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-configuration-processor</artifactId>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>io.projectreactor</groupId>
|
||||
<artifactId>reactor-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.integration</groupId>
|
||||
<artifactId>spring-integration-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,96 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.consumer.twitter.friendship;
|
||||
|
||||
import java.util.function.Consumer;
|
||||
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.messaging.Message;
|
||||
|
||||
/**
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@Configuration
|
||||
@EnableConfigurationProperties(TwitterFriendshipsConsumerProperties.class)
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterFriendshipsConsumerConfiguration {
|
||||
|
||||
@Bean
|
||||
@SuppressWarnings("Duplicates")
|
||||
public Consumer<Message<?>> friendshipConsumer(TwitterFriendshipsConsumerProperties properties, Twitter twitter) {
|
||||
|
||||
return message -> {
|
||||
try {
|
||||
TwitterFriendshipsConsumerProperties.OperationType type =
|
||||
properties.getType().getValue(message, TwitterFriendshipsConsumerProperties.OperationType.class);
|
||||
//TwitterFriendshipsSinkProperties.OperationType type = TwitterFriendshipsSinkProperties.OperationType.create;
|
||||
if (properties.getUserId() != null) {
|
||||
Long userId = properties.getUserId().getValue(message, long.class);
|
||||
switch (type) {
|
||||
case create:
|
||||
boolean follow = properties.getCreate().getFollow().getValue(message, boolean.class);
|
||||
twitter.createFriendship(userId, follow);
|
||||
return;
|
||||
|
||||
case update:
|
||||
boolean enableDeviceNotification = properties.getUpdate().getDevice().getValue(message, boolean.class);
|
||||
boolean retweets = properties.getUpdate().getRetweets().getValue(message, boolean.class);
|
||||
twitter.updateFriendship(userId, enableDeviceNotification, retweets);
|
||||
return;
|
||||
|
||||
case destroy:
|
||||
twitter.destroyFriendship(userId);
|
||||
return;
|
||||
}
|
||||
}
|
||||
else if (properties.getScreenName() != null) {
|
||||
String screenName = properties.getScreenName().getValue(message, String.class);
|
||||
switch (type) {
|
||||
case create:
|
||||
boolean follow = properties.getCreate().getFollow().getValue(message, boolean.class);
|
||||
twitter.createFriendship(screenName, follow);
|
||||
return;
|
||||
|
||||
case update:
|
||||
boolean enableDeviceNotification = properties.getUpdate().getDevice().getValue(message, boolean.class);
|
||||
boolean retweets = properties.getUpdate().getRetweets().getValue(message, boolean.class);
|
||||
twitter.updateFriendship(screenName, enableDeviceNotification, retweets);
|
||||
return;
|
||||
|
||||
case destroy:
|
||||
twitter.destroyFriendship(screenName);
|
||||
return;
|
||||
}
|
||||
}
|
||||
else {
|
||||
throw new IllegalStateException("Either ScreenName or UserID must be set");
|
||||
}
|
||||
}
|
||||
catch (TwitterException te) {
|
||||
throw new IllegalStateException("Twitter API error!", te);
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.consumer.twitter.friendship;
|
||||
|
||||
import javax.validation.constraints.AssertTrue;
|
||||
import javax.validation.constraints.NotNull;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.expression.Expression;
|
||||
import org.springframework.expression.spel.standard.SpelExpressionParser;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@Component
|
||||
@ConfigurationProperties("twitter.friendships.update")
|
||||
@Validated
|
||||
public class TwitterFriendshipsConsumerProperties {
|
||||
|
||||
public enum OperationType {
|
||||
/** Friendship operation types. */
|
||||
create, update, destroy
|
||||
}
|
||||
|
||||
/**
|
||||
* The screen name of the user to follow (String).
|
||||
*/
|
||||
private Expression screenName;
|
||||
|
||||
/**
|
||||
* The ID of the user to follow (Integer).
|
||||
*/
|
||||
private Expression userId;
|
||||
|
||||
/**
|
||||
* Type of Friendships request.
|
||||
*/
|
||||
private Expression type = new SpelExpressionParser().parseExpression("'create'");
|
||||
|
||||
/**
|
||||
* Additional properties for the Friendships create requests.
|
||||
*/
|
||||
private Create create = new Create();
|
||||
|
||||
/**
|
||||
* Additional properties for the Friendships update requests.
|
||||
*/
|
||||
private Update update = new Update();
|
||||
|
||||
public Expression getScreenName() {
|
||||
return screenName;
|
||||
}
|
||||
|
||||
public void setScreenName(Expression screenName) {
|
||||
this.screenName = screenName;
|
||||
}
|
||||
|
||||
public Expression getUserId() {
|
||||
return userId;
|
||||
}
|
||||
|
||||
public void setUserId(Expression userId) {
|
||||
this.userId = userId;
|
||||
}
|
||||
|
||||
public Expression getType() {
|
||||
return type;
|
||||
}
|
||||
|
||||
public void setType(Expression type) {
|
||||
this.type = type;
|
||||
}
|
||||
|
||||
public Create getCreate() {
|
||||
return create;
|
||||
}
|
||||
|
||||
public Update getUpdate() {
|
||||
return update;
|
||||
}
|
||||
|
||||
@AssertTrue(message = "Either userId or screenName must be provided")
|
||||
public boolean isUserProvided() {
|
||||
return this.userId != null || this.screenName != null;
|
||||
}
|
||||
|
||||
public static class Create {
|
||||
/**
|
||||
* The ID of the user to follow (boolean).
|
||||
*/
|
||||
@NotNull
|
||||
private Expression follow = new SpelExpressionParser().parseExpression("'true'");
|
||||
|
||||
public Expression getFollow() {
|
||||
return follow;
|
||||
}
|
||||
|
||||
public void setFollow(Expression follow) {
|
||||
this.follow = follow;
|
||||
}
|
||||
}
|
||||
|
||||
public static class Update {
|
||||
/**
|
||||
* Enable/disable device notifications from the target user.
|
||||
*/
|
||||
@NotNull
|
||||
private Expression device = new SpelExpressionParser().parseExpression("'true'");
|
||||
|
||||
/**
|
||||
* Enable/disable Retweets from the target user.
|
||||
*/
|
||||
@NotNull
|
||||
private Expression retweets = new SpelExpressionParser().parseExpression("'true'");
|
||||
|
||||
public Expression getDevice() {
|
||||
return device;
|
||||
}
|
||||
|
||||
public void setDevice(Expression device) {
|
||||
this.device = device;
|
||||
}
|
||||
|
||||
public Expression getRetweets() {
|
||||
return retweets;
|
||||
}
|
||||
|
||||
public void setRetweets(Expression retweets) {
|
||||
this.retweets = retweets;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.consumer.twitter.message;
|
||||
|
||||
import java.util.function.Consumer;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.messaging.Message;
|
||||
|
||||
/**
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@Configuration
|
||||
@EnableConfigurationProperties(TwitterMessageConsumerProperties.class)
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterMessageConsumerConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterMessageConsumerConfiguration.class);
|
||||
|
||||
@Bean
|
||||
public Consumer<Message<?>> sendDirectMessageConsumer(TwitterMessageConsumerProperties messageProperties, Twitter twitter) {
|
||||
return message -> {
|
||||
try {
|
||||
String messageText = messageProperties.getText().getValue(message, String.class);
|
||||
|
||||
if (messageProperties.getUserId() != null) {
|
||||
Long userId = messageProperties.getUserId().getValue(message, long.class);
|
||||
if (messageProperties.getMediaId() != null) {
|
||||
Long mediaId = messageProperties.getMediaId().getValue(message, long.class);
|
||||
twitter.sendDirectMessage(userId, messageText, mediaId);
|
||||
}
|
||||
twitter.sendDirectMessage(userId, messageText);
|
||||
}
|
||||
else if (messageProperties.getScreenName() != null) {
|
||||
String screenName = messageProperties.getScreenName().getValue(message, String.class);
|
||||
twitter.sendDirectMessage(screenName, messageText);
|
||||
}
|
||||
else {
|
||||
throw new RuntimeException("Either the UserId or screenName must be set");
|
||||
}
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Failed to process message:" + message, e);
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.consumer.twitter.message;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.expression.Expression;
|
||||
import org.springframework.expression.spel.standard.SpelExpressionParser;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.message.update")
|
||||
@Validated
|
||||
public class TwitterMessageConsumerProperties {
|
||||
|
||||
private static final Expression DEFAULT_EXPRESSION = new SpelExpressionParser().parseExpression("payload");
|
||||
|
||||
/**
|
||||
* The direct message text. URL encode as necessary. Max length of 10,000 characters.
|
||||
*/
|
||||
private Expression text = DEFAULT_EXPRESSION;
|
||||
|
||||
/**
|
||||
* The screen name of the user to whom send the direct message.
|
||||
*/
|
||||
private Expression screenName;
|
||||
|
||||
/**
|
||||
* The user id of the user to whom send the direct message.
|
||||
*/
|
||||
private Expression userId;
|
||||
|
||||
/**
|
||||
* A media id to associate with the message. A Direct Message may only reference a single media id.
|
||||
*/
|
||||
private Expression mediaId;
|
||||
|
||||
public Expression getUserId() {
|
||||
return userId;
|
||||
}
|
||||
|
||||
public void setUserId(Expression userId) {
|
||||
this.userId = userId;
|
||||
}
|
||||
|
||||
public void setText(Expression text) {
|
||||
this.text = text;
|
||||
}
|
||||
|
||||
public Expression getText() {
|
||||
return text;
|
||||
}
|
||||
|
||||
public Expression getScreenName() {
|
||||
return screenName;
|
||||
}
|
||||
|
||||
public void setScreenName(Expression screenName) {
|
||||
this.screenName = screenName;
|
||||
}
|
||||
|
||||
public Expression getMediaId() {
|
||||
return mediaId;
|
||||
}
|
||||
|
||||
public void setMediaId(Expression mediaId) {
|
||||
this.mediaId = mediaId;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.consumer.twitter.status.update;
|
||||
|
||||
import java.util.Properties;
|
||||
import java.util.function.Consumer;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import twitter4j.GeoLocation;
|
||||
import twitter4j.Status;
|
||||
import twitter4j.StatusUpdate;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.core.env.ConfigurableEnvironment;
|
||||
import org.springframework.core.env.PropertiesPropertySource;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.util.Assert;
|
||||
|
||||
/**
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@Configuration
|
||||
@EnableConfigurationProperties(TwitterUpdateConsumerProperties.class)
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterUpdateConsumerConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterUpdateConsumerConfiguration.class);
|
||||
|
||||
@Autowired
|
||||
public void setInfoProperties(ConfigurableEnvironment env) {
|
||||
Properties props = new Properties();
|
||||
props.put("spring.cloud.stream.function.definition", "toText|upper|sink");
|
||||
env.getPropertySources().addFirst(new PropertiesPropertySource("function-dsl-props", props));
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Consumer<StatusUpdate> updateStatus(Twitter twitter) {
|
||||
return statusUpdate -> {
|
||||
try {
|
||||
Status status = twitter.updateStatus(statusUpdate);
|
||||
|
||||
if (logger.isDebugEnabled()) {
|
||||
logger.debug(status);
|
||||
}
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Failed apply update status: " + statusUpdate, e);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<Message<?>, StatusUpdate> toStatusUpdateQuery(TwitterUpdateConsumerProperties updateProperties) {
|
||||
|
||||
return message -> {
|
||||
|
||||
String updateText = updateProperties.getText().getValue(message, String.class);
|
||||
|
||||
StatusUpdate statusUpdate = new StatusUpdate(updateText);
|
||||
|
||||
if (updateProperties.getAttachmentUrl() != null) {
|
||||
statusUpdate.setAttachmentUrl(updateProperties.getAttachmentUrl().getValue(message, String.class));
|
||||
}
|
||||
|
||||
if (updateProperties.getPlaceId() != null) {
|
||||
statusUpdate.setPlaceId(updateProperties.getPlaceId().getValue(message, String.class));
|
||||
}
|
||||
|
||||
if (updateProperties.getInReplyToStatusId() != null) {
|
||||
statusUpdate.setInReplyToStatusId(updateProperties.getInReplyToStatusId().getValue(message, int.class));
|
||||
statusUpdate.setAutoPopulateReplyMetadata(true);
|
||||
}
|
||||
|
||||
if (updateProperties.getDisplayCoordinates() != null) {
|
||||
statusUpdate.setDisplayCoordinates(
|
||||
updateProperties.getDisplayCoordinates().getValue(message, boolean.class));
|
||||
}
|
||||
|
||||
if (updateProperties.getMediaIds() != null) {
|
||||
long[] mediaIds = updateProperties.getMediaIds().getValue(message, long[].class);
|
||||
statusUpdate.setMediaIds(mediaIds);
|
||||
}
|
||||
|
||||
if (updateProperties.getLocation().getLat() != null) {
|
||||
Assert.notNull(updateProperties.getLocation().getLon(),
|
||||
"If the latitude is set then the longitude must be set too");
|
||||
double lat = updateProperties.getLocation().getLat().getValue(message, Double.class);
|
||||
double lon = updateProperties.getLocation().getLon().getValue(message, Double.class);
|
||||
statusUpdate.setLocation(new GeoLocation(lat, lon));
|
||||
}
|
||||
|
||||
return statusUpdate;
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Consumer<Message<?>> twitterStatusUpdateConsumer(Function<Message<?>, StatusUpdate> statusUpdateQuery,
|
||||
Consumer<StatusUpdate> updateStatus) {
|
||||
return message -> updateStatus.accept(statusUpdateQuery.apply(message));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.consumer.twitter.status.update;
|
||||
|
||||
import javax.validation.constraints.AssertTrue;
|
||||
import javax.validation.constraints.NotNull;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.expression.Expression;
|
||||
import org.springframework.expression.spel.standard.SpelExpressionParser;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.update")
|
||||
@Validated
|
||||
public class TwitterUpdateConsumerProperties {
|
||||
|
||||
private static final Expression DEFAULT_EXPRESSION = new SpelExpressionParser().parseExpression("payload");
|
||||
|
||||
/**
|
||||
* (SpEL expression) The text of the text update. URL encode as necessary. t.co link wrapping will
|
||||
* affect character counts. Defaults to message's payload
|
||||
*/
|
||||
@NotNull
|
||||
private Expression text = DEFAULT_EXPRESSION;
|
||||
|
||||
/**
|
||||
* (SpEL expression) In order for a URL to not be counted in the text body of an extended Tweet, provide a URL as a Tweet attachment.
|
||||
* This URL must be a Tweet permalink, or Direct Message deep link. Arbitrary, non-Twitter URLs must remain in
|
||||
* the text text. URLs passed to the attachment_url parameter not matching either a Tweet permalink or Direct
|
||||
* Message deep link will fail at Tweet creation and cause an exception.
|
||||
*/
|
||||
private Expression attachmentUrl;
|
||||
|
||||
/**
|
||||
* (SpEL expression) A place in the world.
|
||||
*/
|
||||
private Expression placeId;
|
||||
|
||||
/**
|
||||
* (SpEL expression) The ID of an existing text that the update is in reply to. Note: This parameter will be ignored unless the
|
||||
* author of the Tweet this parameter references is mentioned within the text text. Therefore, you must
|
||||
* include @username, where username is the author of the referenced Tweet, within the update.
|
||||
*
|
||||
* When inReplyToStatusId is set the auto_populate_reply_metadata is automatically set as well. Later ensures
|
||||
* that leading @mentions will be looked up from the original Tweet, and added to the new Tweet from there.
|
||||
* This wil append @mentions into the metadata of an extended Tweet as a reply chain grows, until the limit
|
||||
* on @mentions is reached. In cases where the original Tweet has been deleted,
|
||||
* the reply will fail.
|
||||
*/
|
||||
private Expression inReplyToStatusId;
|
||||
|
||||
/**
|
||||
* (SpEL expression) Whether or not to put a pin on the exact coordinates a Tweet has been sent from.
|
||||
*/
|
||||
private Expression displayCoordinates;
|
||||
|
||||
/**
|
||||
* (SpEL expression) A comma-delimited list of media_ids to associate with the Tweet. You may include up to 4 photos or 1 animated
|
||||
* GIF or 1 video in a Tweet. See Uploading Media for further details on uploading media.
|
||||
*/
|
||||
private Expression mediaIds;
|
||||
|
||||
/**
|
||||
* (SpEL expression) The location this Tweet refers to. Ignored if geo_enabled for the user is false!
|
||||
*/
|
||||
private Location location = new Location();
|
||||
|
||||
public Expression getText() {
|
||||
return text;
|
||||
}
|
||||
|
||||
public void setText(Expression text) {
|
||||
this.text = text;
|
||||
}
|
||||
|
||||
public Expression getAttachmentUrl() {
|
||||
return attachmentUrl;
|
||||
}
|
||||
|
||||
public void setAttachmentUrl(Expression attachmentUrl) {
|
||||
this.attachmentUrl = attachmentUrl;
|
||||
}
|
||||
|
||||
public Expression getPlaceId() {
|
||||
return placeId;
|
||||
}
|
||||
|
||||
public void setPlaceId(Expression placeId) {
|
||||
this.placeId = placeId;
|
||||
}
|
||||
|
||||
public Expression getInReplyToStatusId() {
|
||||
return inReplyToStatusId;
|
||||
}
|
||||
|
||||
public void setInReplyToStatusId(Expression inReplyToStatusId) {
|
||||
this.inReplyToStatusId = inReplyToStatusId;
|
||||
}
|
||||
|
||||
public Expression getDisplayCoordinates() {
|
||||
return displayCoordinates;
|
||||
}
|
||||
|
||||
public void setDisplayCoordinates(Expression displayCoordinates) {
|
||||
this.displayCoordinates = displayCoordinates;
|
||||
}
|
||||
|
||||
public Expression getMediaIds() {
|
||||
return mediaIds;
|
||||
}
|
||||
|
||||
public void setMediaIds(Expression mediaIds) {
|
||||
this.mediaIds = mediaIds;
|
||||
}
|
||||
|
||||
public Location getLocation() {
|
||||
return location;
|
||||
}
|
||||
|
||||
@AssertTrue(message = "Lat and Long must be set together or both not being set")
|
||||
public boolean validateLatLon() {
|
||||
return (this.getLocation().getLat() != null && this.getLocation().getLon() != null)
|
||||
|| (this.getLocation().getLat() == null && this.getLocation().getLon() == null);
|
||||
}
|
||||
|
||||
public static class Location {
|
||||
/**
|
||||
* The latitude of the location this Tweet refers to. This parameter will be ignored unless it is inside the range
|
||||
* -90.0 to +90.0 (North is positive) inclusive. It will also be ignored if there is no corresponding long parameter.
|
||||
*/
|
||||
private Expression lat;
|
||||
|
||||
/**
|
||||
* The longitude of the location this Tweet refers to. The valid ranges for longitude are -180.0 to +180.0 (East
|
||||
* is positive) inclusive. This parameter will be ignored if outside that range, if it is not a number,
|
||||
* if geo_enabled is disabled, or if there no corresponding lat parameter.
|
||||
*/
|
||||
private Expression lon;
|
||||
|
||||
public Expression getLat() {
|
||||
return lat;
|
||||
}
|
||||
|
||||
public void setLat(Expression lat) {
|
||||
this.lat = lat;
|
||||
}
|
||||
|
||||
public Expression getLon() {
|
||||
return lon;
|
||||
}
|
||||
|
||||
public void setLon(Expression lon) {
|
||||
this.lon = lon;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.consumer.twitter.status.update;
|
||||
|
||||
import java.util.function.Consumer;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.junit.Test;
|
||||
import twitter4j.StatusUpdate;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
|
||||
import org.springframework.expression.Expression;
|
||||
import org.springframework.expression.ExpressionParser;
|
||||
import org.springframework.expression.spel.standard.SpelExpressionParser;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.messaging.support.GenericMessage;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.verify;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterUpdateSinkFunctionConfigurationTests {
|
||||
|
||||
@Test
|
||||
public void testStatusUpdateConsumer() throws TwitterException {
|
||||
Twitter twitter = mock(Twitter.class);
|
||||
|
||||
Consumer<StatusUpdate> statusUpdateConsumer =
|
||||
new TwitterUpdateConsumerConfiguration().updateStatus(twitter);
|
||||
|
||||
StatusUpdate statusUpdateQuery = new StatusUpdate("Hello World");
|
||||
statusUpdateConsumer.accept(statusUpdateQuery);
|
||||
verify(twitter).updateStatus(eq(statusUpdateQuery));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testToStatusUpdateQueryFunction() {
|
||||
TwitterUpdateConsumerProperties properties = new TwitterUpdateConsumerProperties();
|
||||
|
||||
properties.setAttachmentUrl(expression("'attachmentUrl'"));
|
||||
properties.setPlaceId(expression("'myPlaceId'"));
|
||||
properties.setInReplyToStatusId(expression("'666666'"));
|
||||
properties.setDisplayCoordinates(expression("'true'"));
|
||||
properties.setMediaIds(expression("'471592142565957632, 471592142565957633'"));
|
||||
properties.getLocation().setLat(expression("'37.78217'"));
|
||||
properties.getLocation().setLon(expression("'-122.40062'"));
|
||||
|
||||
Function<Message<?>, StatusUpdate> toStatusUpdateQueryFunction =
|
||||
new TwitterUpdateConsumerConfiguration().toStatusUpdateQuery(properties);
|
||||
|
||||
StatusUpdate result = toStatusUpdateQueryFunction.apply(new GenericMessage<>("Hello World"));
|
||||
|
||||
assertThat(result).isNotNull();
|
||||
assertThat(result.getStatus()).isEqualTo("Hello World");
|
||||
assertThat(result.getAttachmentUrl()).isEqualTo("attachmentUrl");
|
||||
assertThat(result.getPlaceId()).isEqualTo("myPlaceId");
|
||||
assertThat(result.getInReplyToStatusId()).isEqualTo(666666L);
|
||||
assertThat(result.isDisplayCoordinates()).isTrue();
|
||||
assertThat(result.getLocation().getLatitude()).isEqualTo(37.78217);
|
||||
assertThat(result.getLocation().getLongitude()).isEqualTo(-122.40062);
|
||||
}
|
||||
|
||||
private Expression expression(String expressionString) {
|
||||
ExpressionParser parser = new SpelExpressionParser();
|
||||
return parser.parseExpression(expressionString);
|
||||
}
|
||||
}
|
||||
30
functions/function/twitter-function/README.adoc
Normal file
30
functions/function/twitter-function/README.adoc
Normal file
@@ -0,0 +1,30 @@
|
||||
# Twitter Functions
|
||||
|
||||
This module provides couple of twitter functions that can be reused and composed in other applications.
|
||||
|
||||
## Twitter Trend Function
|
||||
|
||||
Functions can return either Trends topics or the Locations of the trending topics. The `twitter.trend.trend-query-type` property allows choosing between both types.
|
||||
|
||||
* Trends - `twitter.trend.trend-query-type` is set to `trend`. Leverages the https://developer.twitter.com/en/docs/trends/trends-for-location/api-reference/get-trends-place[Trends API] to return the https://help.twitter.com/en/using-twitter/twitter-trending-faqs[trending topics] near a specific latitude, longitude location.
|
||||
|
||||
* Trend Locations - the `twitter.trend.trend-query-type` is set `trendLocation`. Retrieve a full or nearby locations list of trending topics by location. If the `latitude`, `longitude` parameters are NOT provided the processor performs the https://developer.twitter.com/en/docs/trends/locations-with-trending-topics/api-reference/get-trends-available[Trends Available API] and returns the locations that Twitter has trending topic information for.
|
||||
If the `latitude`, `longitude` parameters are provided the processor performs the https://developer.twitter.com/en/docs/trends/locations-with-trending-topics/api-reference/get-trends-closest[Trends Closest API] and returns the locations that Twitter has trending topic information for, closest to a specified location.
|
||||
|
||||
Response is an array of `locations` that encode the location's WOEID and some other human-readable information such as a canonical name and country the location belongs in.
|
||||
|
||||
### Beans for injection
|
||||
|
||||
You can import the `TwitterTrendFunctionConfiguration` in a Spring Boot application and then inject the following bean.
|
||||
|
||||
`filterFunction`
|
||||
|
||||
You can use `Function<Message<?>, Message<byte[]>> trendOrTrendLocationsFunction` as a qualifier when injecting.
|
||||
|
||||
Once injected, you can use the `apply` method of the `Function` to invoke it and get the result.
|
||||
|
||||
### Configuration Options
|
||||
|
||||
|
||||
### Other usage
|
||||
|
||||
46
functions/function/twitter-function/pom.xml
Normal file
46
functions/function/twitter-function/pom.xml
Normal file
@@ -0,0 +1,46 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-function</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<name>twitter-function</name>
|
||||
<description>twitter functions</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>spring-functions-parent</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../spring-functions-parent</relativePath>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-common</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-integration</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<groupId>org.junit.vintage</groupId>
|
||||
<artifactId>junit-vintage-engine</artifactId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-configuration-processor</artifactId>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,91 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.twitter.trend;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.function.Function;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import twitter4j.GeoLocation;
|
||||
import twitter4j.Location;
|
||||
import twitter4j.Trends;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.messaging.Message;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@Configuration
|
||||
@EnableConfigurationProperties(TwitterTrendFunctionProperties.class)
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterTrendFunctionConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterTrendFunctionConfiguration.class);
|
||||
|
||||
@Bean
|
||||
public Function<Message<?>, Trends> trend(TwitterTrendFunctionProperties properties, Twitter twitter) {
|
||||
return message -> {
|
||||
try {
|
||||
int woeid = properties.getLocationId().getValue(message, int.class);
|
||||
return twitter.getPlaceTrends(woeid);
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Twitter API error!", e);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<Message<?>, List<Location>> closestOrAvailableTrends(
|
||||
TwitterTrendFunctionProperties properties, Twitter twitter) {
|
||||
return message -> {
|
||||
try {
|
||||
if (properties.getClosest().getLat() != null && properties.getClosest().getLon() != null) {
|
||||
double lat = properties.getClosest().getLat().getValue(message, double.class);
|
||||
double lon = properties.getClosest().getLon().getValue(message, double.class);
|
||||
return twitter.getClosestTrends(new GeoLocation(lat, lon));
|
||||
}
|
||||
else {
|
||||
return twitter.getAvailableTrends();
|
||||
}
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Twitter API error!", e);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<Message<?>, Message<byte[]>> trendOrTrendLocationsFunction(
|
||||
Function<Object, Message<byte[]>> managedJson, Function<Message<?>, Trends> trend,
|
||||
TwitterTrendFunctionProperties properties, Function<Message<?>,
|
||||
List<Location>> closestOrAvailableTrends) {
|
||||
|
||||
return (properties.getTrendQueryType() == TwitterTrendFunctionProperties.TrendQueryType.trend) ?
|
||||
trend.andThen(managedJson) : closestOrAvailableTrends.andThen(managedJson);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.twitter.trend;
|
||||
|
||||
import javax.validation.constraints.NotNull;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.expression.Expression;
|
||||
import org.springframework.expression.spel.standard.SpelExpressionParser;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.trend")
|
||||
@Validated
|
||||
public class TwitterTrendFunctionProperties {
|
||||
|
||||
private static final Expression DEFAULT_EXPRESSION = new SpelExpressionParser().parseExpression("payload");
|
||||
|
||||
enum TrendQueryType {
|
||||
/** Retrieve trending places. */
|
||||
trend,
|
||||
/** Retrieve the Locations of trending places. */
|
||||
trendLocation
|
||||
}
|
||||
|
||||
private TrendQueryType trendQueryType = TrendQueryType.trend;
|
||||
|
||||
public TrendQueryType getTrendQueryType() {
|
||||
return trendQueryType;
|
||||
}
|
||||
|
||||
public void setTrendQueryType(TrendQueryType trendQueryType) {
|
||||
this.trendQueryType = trendQueryType;
|
||||
}
|
||||
|
||||
/**
|
||||
* The Yahoo! Where On Earth ID of the location to return trending information for.
|
||||
* Global information is available by using 1 as the WOEID.
|
||||
*/
|
||||
@NotNull
|
||||
private Expression locationId = DEFAULT_EXPRESSION;
|
||||
|
||||
public Expression getLocationId() {
|
||||
return locationId;
|
||||
}
|
||||
|
||||
public void setLocationId(Expression locationId) {
|
||||
this.locationId = locationId;
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
private Closest closest = new Closest();
|
||||
|
||||
public Closest getClosest() {
|
||||
return closest;
|
||||
}
|
||||
|
||||
public static class Closest {
|
||||
/**
|
||||
* If provided with a long parameter the available trend locations will be sorted by distance, nearest
|
||||
* to furthest, to the co-ordinate pair.
|
||||
* The valid ranges for longitude is -180.0 to +180.0 (West is negative, East is positive) inclusive.
|
||||
*/
|
||||
private Expression lat;
|
||||
|
||||
/**
|
||||
* If provided with a lat parameter the available trend locations will be sorted by distance, nearest to
|
||||
* furthest, to the co-ordinate pair. The valid ranges for longitude is -180.0 to +180.0 (West is negative,
|
||||
* East is positive) inclusive.
|
||||
*/
|
||||
private Expression lon;
|
||||
|
||||
public Expression getLat() {
|
||||
return lat;
|
||||
}
|
||||
|
||||
public void setLat(Expression lat) {
|
||||
this.lat = lat;
|
||||
}
|
||||
|
||||
public Expression getLon() {
|
||||
return lon;
|
||||
}
|
||||
|
||||
public void setLon(Expression lon) {
|
||||
this.lon = lon;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -50,6 +50,7 @@
|
||||
<module>common/metadata-store-common</module>
|
||||
<module>common/mqtt-common</module>
|
||||
<module>common/tcp-common</module>
|
||||
<module>common/twitter-common</module>
|
||||
|
||||
|
||||
<module>consumer/cassandra-consumer</module>
|
||||
@@ -67,6 +68,7 @@
|
||||
<module>consumer/tcp-consumer</module>
|
||||
<module>consumer/websocket-consumer</module>
|
||||
<module>consumer/s3-consumer</module>
|
||||
<module>consumer/twitter-consumer</module>
|
||||
|
||||
<module>function/filter-function</module>
|
||||
<module>function/header-enricher-function</module>
|
||||
@@ -76,6 +78,7 @@
|
||||
<module>function/splitter-function</module>
|
||||
<module>function/tasklauncher-function</module>
|
||||
<module>function/task-launch-request-function</module>
|
||||
<module>function/twitter-function</module>
|
||||
|
||||
<module>supplier/file-supplier</module>
|
||||
<module>supplier/ftp-supplier</module>
|
||||
@@ -90,6 +93,7 @@
|
||||
<module>supplier/rabbit-supplier</module>
|
||||
<module>supplier/websocket-supplier</module>
|
||||
<module>supplier/s3-supplier</module>
|
||||
<module>supplier/twitter-supplier</module>
|
||||
|
||||
<module>spring-functions-parent</module>
|
||||
</modules>
|
||||
|
||||
143
functions/supplier/twitter-supplier/README.adoc
Normal file
143
functions/supplier/twitter-supplier/README.adoc
Normal file
@@ -0,0 +1,143 @@
|
||||
# Twitter Suppliers
|
||||
|
||||
This module provides a Twitter Status, Message, Friendship suppliers that can be reused and composed in various applications.
|
||||
|
||||
`java.util.function.Supplier`
|
||||
|
||||
|
||||
## 1. Twitter Status Search
|
||||
|
||||
The Twitter's https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets.html[Standard search API] (search/tweets) allows simple queries against the indices of recent or popular Tweets. This `Source` provides continuous searches against a sampling of recent Tweets published in the past 7 days. Part of the 'public' set of APIs.
|
||||
|
||||
Returns a collection of relevant Tweets matching a specified query.
|
||||
|
||||
### 1.1 Beans for injection
|
||||
|
||||
You can import the `TwitterSearchSupplierConfiguration` in the application and then inject the following bean.
|
||||
|
||||
`twitterSearchSupplier`
|
||||
|
||||
You need to inject this as `Supplier<Message<byte[]>>`.
|
||||
|
||||
You can use `twitterSearchSupplier` as a qualifier when injecting.
|
||||
|
||||
Once injected, you can use the `get` method of the `Supplier` to invoke it.
|
||||
|
||||
### 1.2 Configuration Options
|
||||
|
||||
The configuration properties prefixed with `twitter.search`.
|
||||
There are also properties that need to be used with the prefix `twitter.connection`.
|
||||
|
||||
The `spring.cloud.stream.poller` properties control the interval between consecutive search requests. Rate Limit - 180 requests per 30 min. window (e.g. ~6 r/m, ~ 1 req / 10 sec.)
|
||||
|
||||
For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/supplier/twitter/search/stream/TwitterSearchSupplierProperties.java[TwitterSearchSupplierProperties].
|
||||
See link:src/main/java/org/springframework/cloud/fn/common/twitter/TwitterConnectionProperties.java[TwitterConnectionProperties] and https://github.com/spring-cloud/spring-cloud-stream/blob/master/spring-cloud-stream/src/main/java/org/springframework/cloud/stream/config/DefaultPollerProperties.java[DefaultPollerProperties]
|
||||
|
||||
### 1.3 Other usage
|
||||
|
||||
See this https://github.com/spring-cloud/stream-applications/blob/master/applications/source/twitter-search-source/README.adoc[README] where this supplier is used to create a Spring Cloud Stream application where it makes a Twitter Search Source.
|
||||
|
||||
## 2. Twitter Status Real-time Retrieval
|
||||
|
||||
Provides real-time, Tweet streaming based on the https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html[Filter] and https://developer.twitter.com/en/docs/tweets/sample-realtime/overview/GET_statuse_sample[Sample] APIs.
|
||||
The `Filter API` flavor returns public statuses that match one or more filter predicates.
|
||||
The `Sample API` flavor returns a small random sample of all public statuses.
|
||||
|
||||
This supplier gives you a reactive stream of tweets from the configured connection as the supplier has a signature of `Supplier<Flux<Message<?>>>`.
|
||||
Users have to subscribe to this `Flux` and receive the data.
|
||||
|
||||
The default access level allows up to 400 track keywords, 5,000 follow user Ids and 25 0.1-360 degree location boxes.
|
||||
|
||||
### 2.1 Beans for injection
|
||||
|
||||
You can import the `TwitterStreamSupplierConfiguration` in the application and then inject the following bean.
|
||||
|
||||
`twitterStreamSupplier`
|
||||
|
||||
You need to inject this as `Supplier<Flux<Message<?>>>`.
|
||||
|
||||
You can use `twitterStreamSupplier` as a qualifier when injecting.
|
||||
|
||||
Once injected, you can use the `get` method of the `Supplier` to invoke it and then subscribe to the returned `Flux`.
|
||||
|
||||
### 2.2 Configuration Options
|
||||
|
||||
All configuration properties are prefixed with `twitter.stream`.
|
||||
There are also properties that need to be used with the prefix `twitter.connection`.
|
||||
|
||||
For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/supplier/twitter/status/stream/TwitterStreamSupplierProperties.java[TwitterStreamSupplierProperties].
|
||||
See link:src/main/java/org/springframework/cloud/fn/common/twitter/TwitterConnectionProperties.java[TwitterConnectionProperties] also.
|
||||
|
||||
|
||||
### 2.3 Other usage
|
||||
|
||||
See this https://github.com/spring-cloud/stream-applications/blob/master/applications/source/twitter-stream-source/README.adoc[README] where this supplier is used to create a Spring Cloud Stream application where it makes a Twitter Stream Source.
|
||||
|
||||
## 3. Twitter Direct Message Supplier
|
||||
|
||||
Repeatedly retrieves the direct messages (both sent and received) within the last 30 days, sorted in reverse-chronological order.
|
||||
The relieved messages are cached (in a `MetadataStore` cache) to prevent duplications.
|
||||
By default an in-memory `SimpleMetadataStore` is used.
|
||||
|
||||
The `twitter.message.source.count` controls the number or returned messages.
|
||||
|
||||
The `spring.cloud.stream.poller` properties control the message poll interval.
|
||||
Must be aligned with used APIs rate limit
|
||||
|
||||
### 3.1 Beans for injection
|
||||
|
||||
You can import the `TwitterMessageSupplierConfiguration` in the application and then inject the following bean.
|
||||
|
||||
`twitterMessageSupplier`
|
||||
|
||||
You need to inject this as `Supplier<Message<byte[]>>`.
|
||||
|
||||
You can use `twitterMessageSupplier` as a qualifier when injecting.
|
||||
|
||||
Once injected, you can use the `get` method of the `Supplier` to invoke it.
|
||||
|
||||
### 3.2 Configuration Options
|
||||
|
||||
The configuration properties prefixed with `twitter.search`.
|
||||
There are also properties that need to be used with the prefix `twitter.connection`.
|
||||
|
||||
The `spring.cloud.stream.poller` properties control the interval between consecutive search requests. Rate Limit - 180 requests per 30 min. window (e.g. ~6 r/m, ~ 1 req / 10 sec.)
|
||||
|
||||
For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/supplier/twitter/message/TwitterMessageSupplierProperties.java[TwitterMessageSupplierProperties].
|
||||
See link:src/main/java/org/springframework/cloud/fn/common/twitter/TwitterConnectionProperties.java[TwitterConnectionProperties] and https://github.com/spring-cloud/spring-cloud-stream/blob/master/spring-cloud-stream/src/main/java/org/springframework/cloud/stream/config/DefaultPollerProperties.java[DefaultPollerProperties]
|
||||
|
||||
### 3.3 Other usage
|
||||
|
||||
See this https://github.com/spring-cloud/stream-applications/blob/master/applications/source/twitter-message-source/README.adoc[README] where this supplier is used to create a Spring Cloud Stream application where it makes a Twitter Message Source.
|
||||
|
||||
## 4. Twitter Friendships Supplier
|
||||
|
||||
Returns a cursored collection of user objects either for the https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-list[users following the specified user] (`followers`) or for https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-list[every user the specified user is following] (`friends`).
|
||||
|
||||
The `twitter.friendships.source.type` property allow to select between both types.
|
||||
|
||||
TIP: Rate limit: 15 requests per 30 min window. ~ 1 req/ 2 min
|
||||
|
||||
### 4.1 Beans for injection
|
||||
|
||||
You can import the `TwitterFriendshipsSupplierConfiguration` in the application and then inject one the following beans.
|
||||
|
||||
- `followersSupplier` (only if `twitter.friendships.source.type=followers` ) - retrieves the https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-list[users following the specified user] (`followers`)
|
||||
|
||||
- `friendsSupplier` (only if `twitter.friendships.source.type=friends`) - retrieves the for https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-list[every user the specified user is following] (`friends`).
|
||||
|
||||
Both suppliers expose `Supplier<List<User>>`.
|
||||
|
||||
- `deduplicatedFriendsJsonSupplier` - retrieves either the followers, or the friends collection (controlled by the `twitter.friendships.source.type`) property, .
|
||||
encoded as JSON `Message` payloads. You need to inject this as `Supplier<Message<byte[]>>`.
|
||||
|
||||
|
||||
### 4.2 Configuration Options
|
||||
|
||||
The configuration properties prefixed with `twitter.friendships.source`.
|
||||
There are also properties that need to be used with the prefix `twitter.connection`.
|
||||
|
||||
The `spring.cloud.stream.poller` properties control the interval between consecutive search requests.
|
||||
|
||||
For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/supplier/twitter/friendships/TwitterFriendshipsSupplierProperties.java[TwitterFriendshipsSupplierProperties].
|
||||
See link:src/main/java/org/springframework/cloud/fn/common/twitter/TwitterConnectionProperties.java[TwitterConnectionProperties] and https://github.com/spring-cloud/spring-cloud-stream/blob/master/spring-cloud-stream/src/main/java/org/springframework/cloud/stream/config/DefaultPollerProperties.java[DefaultPollerProperties]
|
||||
80
functions/supplier/twitter-supplier/pom.xml
Normal file
80
functions/supplier/twitter-supplier/pom.xml
Normal file
@@ -0,0 +1,80 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>twitter-supplier</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<name>twitter-supplier</name>
|
||||
<description>twitter suppliers</description>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>spring-functions-parent</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../spring-functions-parent</relativePath>
|
||||
</parent>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.cloud.fn</groupId>
|
||||
<artifactId>twitter-common</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-integration</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.integration</groupId>
|
||||
<artifactId>spring-integration-jms</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>javax.jms</groupId>
|
||||
<artifactId>javax.jms-api</artifactId>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-validation</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-configuration-processor</artifactId>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.activemq</groupId>
|
||||
<artifactId>activemq-broker</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>io.projectreactor</groupId>
|
||||
<artifactId>reactor-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.integration</groupId>
|
||||
<artifactId>spring-integration-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-netty</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.mock-server</groupId>
|
||||
<artifactId>mockserver-client-java</artifactId>
|
||||
<version>5.10</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
</project>
|
||||
@@ -0,0 +1,150 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.friendships;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import twitter4j.PagableResponseList;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
import twitter4j.User;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.Cursor;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.integration.metadata.MetadataStore;
|
||||
import org.springframework.integration.metadata.SimpleMetadataStore;
|
||||
import org.springframework.messaging.Message;
|
||||
|
||||
/**
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@Configuration
|
||||
@EnableConfigurationProperties(TwitterFriendshipsSupplierProperties.class)
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterFriendshipsSupplierConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterFriendshipsSupplierConfiguration.class);
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
public MetadataStore metadataStore() {
|
||||
return new SimpleMetadataStore();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
public Cursor cursor() {
|
||||
return new Cursor();
|
||||
}
|
||||
|
||||
|
||||
@Bean
|
||||
@ConditionalOnProperty(name = "twitter.friendships.source.type", havingValue = "followers")
|
||||
public Supplier<List<User>> followersSupplier(TwitterFriendshipsSupplierProperties properties,
|
||||
Twitter twitter, Cursor cursorState) {
|
||||
return () -> {
|
||||
try {
|
||||
PagableResponseList<User> users;
|
||||
if (properties.getUserId() != null) {
|
||||
users = twitter.getFollowersList(properties.getUserId(), cursorState.getCursor(),
|
||||
properties.getCount(), properties.isSkipStatus(), properties.isIncludeUserEntities());
|
||||
}
|
||||
else { // by ScreenName
|
||||
users = twitter.getFollowersList(properties.getScreenName(), cursorState.getCursor(),
|
||||
properties.getCount(), properties.isSkipStatus(), properties.isIncludeUserEntities());
|
||||
}
|
||||
|
||||
if (users != null) {
|
||||
cursorState.updateCursor(users.getNextCursor());
|
||||
return users;
|
||||
}
|
||||
|
||||
logger.error(String.format("NULL users response for properties: %s and cursor: %s!", properties, cursorState));
|
||||
cursorState.updateCursor(-1);
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Twitter API error:", e);
|
||||
}
|
||||
|
||||
return new ArrayList<>();
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnProperty(name = "twitter.friendships.source.type", havingValue = "friends")
|
||||
public Supplier<List<User>> friendsSupplier(TwitterFriendshipsSupplierProperties properties,
|
||||
Twitter twitter, Cursor cursorState) {
|
||||
return () -> {
|
||||
try {
|
||||
PagableResponseList<User> users;
|
||||
if (properties.getUserId() != null) {
|
||||
users = twitter.getFriendsList(properties.getUserId(), cursorState.getCursor(),
|
||||
properties.getCount(), properties.isSkipStatus(), properties.isIncludeUserEntities());
|
||||
}
|
||||
else { // by ScreenName
|
||||
users = twitter.getFriendsList(properties.getScreenName(), cursorState.getCursor(),
|
||||
properties.getCount(), properties.isSkipStatus(), properties.isIncludeUserEntities());
|
||||
}
|
||||
|
||||
if (users != null) {
|
||||
cursorState.updateCursor(users.getNextCursor());
|
||||
return users;
|
||||
}
|
||||
|
||||
logger.error(String.format("NULL users response for properties: %s and cursor: %s!", properties, cursorState));
|
||||
cursorState.updateCursor(-1);
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Twitter API error:", e);
|
||||
}
|
||||
|
||||
return new ArrayList<>();
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<List<User>, List<User>> userDeduplicate(MetadataStore metadataStore) {
|
||||
return users -> {
|
||||
List<User> uniqueUsers = new ArrayList<>();
|
||||
for (User user : users) {
|
||||
if (metadataStore.get(user.getId() + "") == null) {
|
||||
metadataStore.put(user.getId() + "", user.getName());
|
||||
uniqueUsers.add(user);
|
||||
}
|
||||
}
|
||||
return uniqueUsers;
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Supplier<Message<byte[]>> deduplicatedFriendsJsonSupplier(Function<List<User>, List<User>> userDeduplication,
|
||||
Supplier<List<User>> userRetriever, Function<Object, Message<byte[]>> managedJson) {
|
||||
return () -> userDeduplication.andThen(managedJson).apply(userRetriever.get());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.friendships;
|
||||
|
||||
import javax.validation.constraints.AssertTrue;
|
||||
import javax.validation.constraints.Max;
|
||||
import javax.validation.constraints.NotNull;
|
||||
import javax.validation.constraints.Positive;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.friendships.source")
|
||||
@Validated
|
||||
public class TwitterFriendshipsSupplierProperties {
|
||||
|
||||
public enum FriendshipsRequestType {
|
||||
/** Friendship query types. */
|
||||
followers, friends
|
||||
}
|
||||
|
||||
/**
|
||||
* Selects between followers or friends APIs.
|
||||
*/
|
||||
@NotNull
|
||||
private TwitterFriendshipsSupplierProperties.FriendshipsRequestType type = FriendshipsRequestType.followers;
|
||||
|
||||
/**
|
||||
* The screen name of the user for whom to return results.
|
||||
*/
|
||||
private String screenName;
|
||||
|
||||
/**
|
||||
* The ID of the user for whom to return results.
|
||||
*/
|
||||
private Long userId;
|
||||
|
||||
/**
|
||||
* The number of users to return per page, up to a maximum of 200. Defaults to 20.
|
||||
*/
|
||||
@Positive
|
||||
@Max(200)
|
||||
private int count = 200;
|
||||
|
||||
/**
|
||||
* When set to true, statuses will not be included in the returned user objects.
|
||||
*/
|
||||
private boolean skipStatus = false;
|
||||
|
||||
/**
|
||||
* The user object entities node will be disincluded when set to false.
|
||||
*/
|
||||
private boolean includeUserEntities = true;
|
||||
|
||||
/**
|
||||
* API request poll interval in milliseconds. Must be aligned with used APIs rate limits (~ 1 req/ 2 min).
|
||||
*/
|
||||
private int pollInterval = 121000;
|
||||
|
||||
public FriendshipsRequestType getType() {
|
||||
return type;
|
||||
}
|
||||
|
||||
public void setType(FriendshipsRequestType type) {
|
||||
this.type = type;
|
||||
}
|
||||
|
||||
public String getScreenName() {
|
||||
return screenName;
|
||||
}
|
||||
|
||||
public void setScreenName(String screenName) {
|
||||
this.screenName = screenName;
|
||||
}
|
||||
|
||||
public Long getUserId() {
|
||||
return userId;
|
||||
}
|
||||
|
||||
public void setUserId(Long userId) {
|
||||
this.userId = userId;
|
||||
}
|
||||
|
||||
public int getCount() {
|
||||
return count;
|
||||
}
|
||||
|
||||
public void setCount(int count) {
|
||||
this.count = count;
|
||||
}
|
||||
|
||||
public boolean isSkipStatus() {
|
||||
return skipStatus;
|
||||
}
|
||||
|
||||
public void setSkipStatus(boolean skipStatus) {
|
||||
this.skipStatus = skipStatus;
|
||||
}
|
||||
|
||||
public boolean isIncludeUserEntities() {
|
||||
return includeUserEntities;
|
||||
}
|
||||
|
||||
public void setIncludeUserEntities(boolean includeUserEntities) {
|
||||
this.includeUserEntities = includeUserEntities;
|
||||
}
|
||||
|
||||
public int getPollInterval() {
|
||||
return pollInterval;
|
||||
}
|
||||
|
||||
public void setPollInterval(int pollInterval) {
|
||||
this.pollInterval = pollInterval;
|
||||
}
|
||||
|
||||
@AssertTrue(message = "Either userId or screenName must be provided")
|
||||
public boolean isUserProvided() {
|
||||
return this.userId != null || this.screenName != null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "TwitterFriendshipsSourceProperties{" +
|
||||
"type=" + type +
|
||||
", screenName='" + screenName + '\'' +
|
||||
", userId=" + userId +
|
||||
", count=" + count +
|
||||
", skipStatus=" + skipStatus +
|
||||
", includeUserEntities=" + includeUserEntities +
|
||||
", pollInterval=" + pollInterval +
|
||||
'}';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.message;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import twitter4j.DirectMessage;
|
||||
import twitter4j.DirectMessageList;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.integration.metadata.MetadataStore;
|
||||
import org.springframework.integration.metadata.SimpleMetadataStore;
|
||||
import org.springframework.messaging.Message;
|
||||
|
||||
/**
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@EnableConfigurationProperties({ TwitterMessageSupplierProperties.class })
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterMessageSupplierConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterMessageSupplierConfiguration.class);
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
public MetadataStore metadataStore() {
|
||||
return new SimpleMetadataStore();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
public MessageCursor cursor() {
|
||||
return new MessageCursor();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Supplier<List<DirectMessage>> directMessagesSupplier(TwitterMessageSupplierProperties properties,
|
||||
Twitter twitter, MessageCursor cursorState) {
|
||||
return () -> {
|
||||
try {
|
||||
String cs = cursorState.getCursor();
|
||||
DirectMessageList messages = (cursorState.getCursor() == null) ?
|
||||
twitter.getDirectMessages(properties.getCount()) :
|
||||
twitter.getDirectMessages(properties.getCount(), cursorState.getCursor());
|
||||
|
||||
if (messages != null) {
|
||||
cursorState.updateCursor(messages.getNextCursor());
|
||||
return messages;
|
||||
}
|
||||
|
||||
logger.error(String.format("NULL messages response for properties: %s and cursor: %s!", properties, cursorState));
|
||||
cursorState.updateCursor(null);
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Twitter API error:", e);
|
||||
}
|
||||
|
||||
return new ArrayList<>();
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Function<List<DirectMessage>, List<DirectMessage>> messageDeduplicate(MetadataStore metadataStore) {
|
||||
return messages -> {
|
||||
List<DirectMessage> uniqueMessages = new ArrayList<>();
|
||||
for (DirectMessage message : messages) {
|
||||
if (metadataStore.get(message.getId() + "") == null) {
|
||||
metadataStore.put(message.getId() + "", message.getCreatedAt() + "");
|
||||
uniqueMessages.add(message);
|
||||
}
|
||||
}
|
||||
return uniqueMessages;
|
||||
};
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Supplier<Message<byte[]>> twitterMessageSupplier(Function<List<DirectMessage>, List<DirectMessage>> messageDeduplicate,
|
||||
Function<Object, Message<byte[]>> managedJson, Supplier<List<DirectMessage>> directMessagesSupplier) {
|
||||
return () -> messageDeduplicate.andThen(managedJson).apply(directMessagesSupplier.get());
|
||||
}
|
||||
|
||||
public static class MessageCursor {
|
||||
private String cursor = null;
|
||||
|
||||
public String getCursor() {
|
||||
return cursor;
|
||||
}
|
||||
|
||||
public void updateCursor(String newCursor) {
|
||||
this.cursor = newCursor;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "Cursor{" +
|
||||
"cursor=" + cursor +
|
||||
'}';
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.message;
|
||||
|
||||
import javax.validation.constraints.Max;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.message.source")
|
||||
@Validated
|
||||
public class TwitterMessageSupplierProperties {
|
||||
|
||||
/**
|
||||
* Max number of events to be returned. 20 default. 50 max.
|
||||
*/
|
||||
@Max(50)
|
||||
private int count = 20;
|
||||
|
||||
public int getCount() {
|
||||
return count;
|
||||
}
|
||||
|
||||
public void setCount(int count) {
|
||||
this.count = count;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.search;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import twitter4j.Status;
|
||||
|
||||
import org.springframework.util.Assert;
|
||||
|
||||
/**
|
||||
* Searched tweets are ordered from top to bottom by their IDs. The higher the ID, more recent the tweet is.
|
||||
*
|
||||
* The search goes backwards - from most recent to the oldest tweets and it uses the sinceId and the maxId to retrieve
|
||||
* only tweets with IDs in the [sinceId, maxId) range. The -1 stands for unbounded sinceId or maxId.
|
||||
*
|
||||
* The first `pageCount` number requests are performed backwards, leaving the bottom boundary (sinceId) unbounded and
|
||||
* adjusting the upper boundary (maxId) to the lowest tweet ID received. This ensures that no already processed
|
||||
* tweets are returned.
|
||||
*
|
||||
* The pageCounter is used the to count the number of pages retrieved in the pageCount range. Is starts from pageCount
|
||||
* and goes backward until 0. On 0 the pageCounter is reset back to pageCount.
|
||||
*
|
||||
* After performing pageCount number requests (e.g. pageCounter = 0), we start new iteration of searches from the top,
|
||||
* most recent tweets but now the bottom boundary (sinceId) is adjusted to the max ID received so far. That means that
|
||||
* only the newly added tweets will be processed
|
||||
*
|
||||
* Search pagination with max_id and since_id: https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines.html
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class SearchPagination {
|
||||
|
||||
/**
|
||||
* Wildcard used as refer as infinity.
|
||||
*/
|
||||
public static final long UNBOUNDED = -1;
|
||||
|
||||
/**
|
||||
* Number of pages to search in history before start form the top again.
|
||||
*
|
||||
* Note that the search goes backwards - from most recent to the oldest tweets.
|
||||
* (eg. maxId == To max ID , sinceId == From min ID)
|
||||
*/
|
||||
private final int pageCount;
|
||||
|
||||
/**
|
||||
* Min tweet ID (e.g. from ID) to be included in the search result
|
||||
*/
|
||||
private long sinceId;
|
||||
|
||||
/**
|
||||
* Keep the max tweet ID in the last pageCount search requests.
|
||||
*/
|
||||
private long pageMaxId;
|
||||
|
||||
/**
|
||||
* Max tweet ID (e.g. To (ID - 1) )to be included in the search result
|
||||
*/
|
||||
private long maxId;
|
||||
|
||||
/**
|
||||
* Current page. Goes backwards from (pageCount-1) to 0.
|
||||
*/
|
||||
private int pageCounter;
|
||||
|
||||
/**
|
||||
* When set it.
|
||||
*/
|
||||
boolean searchBackwardsUntilEmptyResponse = false;
|
||||
|
||||
public SearchPagination(int pageCount, boolean searchBackwardsUntilEmptyResponse) {
|
||||
|
||||
Assert.isTrue(pageCount > 0, "At least one page needs to be set but was: " + pageCount);
|
||||
|
||||
this.searchBackwardsUntilEmptyResponse = searchBackwardsUntilEmptyResponse;
|
||||
this.pageCount = pageCount;
|
||||
this.sinceId = UNBOUNDED; // == From ID
|
||||
this.pageMaxId = UNBOUNDED; // == From ID
|
||||
this.maxId = UNBOUNDED; // == To (ID - 1)
|
||||
this.pageCounter = pageCount - 1;
|
||||
}
|
||||
|
||||
public long getSinceId() {
|
||||
return sinceId;
|
||||
}
|
||||
|
||||
public long getMaxId() {
|
||||
return maxId;
|
||||
}
|
||||
|
||||
public long getPageMaxId() {
|
||||
return pageMaxId;
|
||||
}
|
||||
|
||||
public int getPageCounter() {
|
||||
return pageCounter;
|
||||
}
|
||||
|
||||
public void update(List<Status> tweets) {
|
||||
|
||||
tweets.stream().mapToLong(t -> t.getId()).min()
|
||||
.ifPresent(tweetsMinId -> {
|
||||
this.maxId = tweetsMinId - 1;
|
||||
});
|
||||
|
||||
tweets.stream().mapToLong(t -> t.getId()).max()
|
||||
.ifPresent(tweetsMaxId -> {
|
||||
Assert.isTrue(this.sinceId <= tweetsMaxId,
|
||||
String.format("MAX_ID (%s) must be bigger then current SINCE_ID(%s)",
|
||||
tweetsMaxId, this.sinceId));
|
||||
this.pageMaxId = Math.max(this.pageMaxId, tweetsMaxId);
|
||||
});
|
||||
|
||||
this.countDown(tweets.size());
|
||||
}
|
||||
|
||||
private void countDown(int responseSize) {
|
||||
|
||||
if (this.sinceId == UNBOUNDED) { // == first pass before reset
|
||||
if (this.pageCounter <= 0) {
|
||||
this.restartSearchFromMostRecent();
|
||||
}
|
||||
}
|
||||
else {
|
||||
if (this.searchBackwardsUntilEmptyResponse) {
|
||||
if (responseSize == 0) {
|
||||
this.restartSearchFromMostRecent();
|
||||
}
|
||||
}
|
||||
else if (this.pageCounter <= 0) {
|
||||
this.restartSearchFromMostRecent();
|
||||
}
|
||||
}
|
||||
|
||||
this.pageCounter--;
|
||||
}
|
||||
|
||||
private void restartSearchFromMostRecent() {
|
||||
this.pageCounter = this.pageCount;
|
||||
|
||||
this.sinceId = Math.max(this.sinceId, Math.max(this.maxId + 1, this.pageMaxId));
|
||||
|
||||
this.maxId = UNBOUNDED;
|
||||
this.pageMaxId = UNBOUNDED;
|
||||
}
|
||||
|
||||
public String status() {
|
||||
return String.format("MaxId: %s, SinceId: %s, Page Counter# %s, pageMaxId: %s",
|
||||
this.maxId, this.sinceId, this.pageCounter, this.pageMaxId);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.search;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import twitter4j.GeoLocation;
|
||||
import twitter4j.Query;
|
||||
import twitter4j.QueryResult;
|
||||
import twitter4j.Status;
|
||||
import twitter4j.Twitter;
|
||||
import twitter4j.TwitterException;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.util.Assert;
|
||||
import org.springframework.util.StringUtils;
|
||||
|
||||
/**
|
||||
* Search pagination with max_id and since_id: https://developer.twitter.com/en/docs/tweets/timelines/guides/working-with-timelines.html .
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
|
||||
@EnableConfigurationProperties({ TwitterSearchSupplierProperties.class })
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterSearchSupplierConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterSearchSupplierConfiguration.class);
|
||||
|
||||
@Autowired
|
||||
private TwitterSearchSupplierProperties searchProperties;
|
||||
|
||||
@Autowired
|
||||
private Twitter twitter;
|
||||
|
||||
@Autowired
|
||||
private SearchPagination searchPage;
|
||||
|
||||
@Autowired
|
||||
private Function<Object, Message<byte[]>> json;
|
||||
|
||||
@Bean
|
||||
public SearchPagination searchPage() {
|
||||
return new SearchPagination(
|
||||
this.searchProperties.getPage(),
|
||||
this.searchProperties.isRestartFromMostRecentOnEmptyResponse());
|
||||
}
|
||||
|
||||
|
||||
@Bean
|
||||
public Supplier<Message<byte[]>> twitterSearchSupplier() {
|
||||
return () -> {
|
||||
try {
|
||||
Query query = toQuery(this.searchProperties, this.searchPage);
|
||||
|
||||
QueryResult result = this.twitter.search(query);
|
||||
|
||||
List<Status> tweets = result.getTweets();
|
||||
|
||||
logger.info(String.format("%s, size: %s", this.searchPage.status(), tweets.size()));
|
||||
|
||||
this.searchPage.update(tweets);
|
||||
|
||||
return this.json.apply(tweets);
|
||||
}
|
||||
catch (TwitterException e) {
|
||||
logger.error("Twitter error", e);
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
}
|
||||
|
||||
private Query toQuery(TwitterSearchSupplierProperties searchProperties, SearchPagination pagination) {
|
||||
|
||||
Query query = new Query();
|
||||
if (searchProperties.getCount() > 0) {
|
||||
query.count(searchProperties.getCount());
|
||||
}
|
||||
if (StringUtils.hasText(searchProperties.getQuery())) {
|
||||
query.setQuery(searchProperties.getQuery());
|
||||
}
|
||||
if (StringUtils.hasText(searchProperties.getLang())) {
|
||||
query.setLang(searchProperties.getLang());
|
||||
}
|
||||
if (StringUtils.hasText(searchProperties.getSince())) {
|
||||
query.setSince(searchProperties.getSince());
|
||||
}
|
||||
if (searchProperties.getGeocode().isValid()) {
|
||||
query.setGeoCode(
|
||||
new GeoLocation(
|
||||
searchProperties.getGeocode().getLatitude(),
|
||||
searchProperties.getGeocode().getLongitude()),
|
||||
searchProperties.getGeocode().getRadius(),
|
||||
Query.KILOMETERS);
|
||||
}
|
||||
|
||||
if (searchProperties.getResultType() != Query.ResultType.mixed) {
|
||||
query.setResultType(searchProperties.getResultType());
|
||||
}
|
||||
|
||||
if (pagination.getSinceId() > 0) {
|
||||
query.setSinceId(pagination.getSinceId());
|
||||
}
|
||||
if (pagination.getMaxId() > 0) {
|
||||
query.setMaxId(pagination.getMaxId());
|
||||
|
||||
Assert.isTrue(pagination.getMaxId() >= (pagination.getSinceId() - 1),
|
||||
String.format("For non empty MAX_ID, The MAX_ID (%s) must always be bigger than [SINCE_ID -1](%s)",
|
||||
pagination.getMaxId(), (pagination.getSinceId() - 1)));
|
||||
}
|
||||
|
||||
return query;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.search;
|
||||
|
||||
import javax.validation.constraints.Max;
|
||||
import javax.validation.constraints.NotEmpty;
|
||||
import javax.validation.constraints.NotNull;
|
||||
import javax.validation.constraints.Pattern;
|
||||
import javax.validation.constraints.Positive;
|
||||
|
||||
import twitter4j.Query;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.search")
|
||||
@Validated
|
||||
public class TwitterSearchSupplierProperties {
|
||||
|
||||
/**
|
||||
* Search tweets by search query string.
|
||||
*/
|
||||
@NotNull
|
||||
@NotEmpty
|
||||
private String query;
|
||||
|
||||
/**
|
||||
* Number of pages (e.g. requests) to search backwards (from most recent to the oldest tweets) before start
|
||||
* the search from the most recent tweets again.
|
||||
* The total amount of tweets searched backwards is (page * count)
|
||||
*/
|
||||
@Positive
|
||||
private int page = 3;
|
||||
|
||||
/**
|
||||
* Number of tweets to return per page (e.g. per single request), up to a max of 100.
|
||||
*/
|
||||
@Positive
|
||||
@Max(100)
|
||||
private int count = 100;
|
||||
|
||||
/**
|
||||
* Restricts searched tweets to the given language, given by an <a href="http://en.wikipedia.org/wiki/ISO_639-1">ISO 639-1 code</a>.
|
||||
*/
|
||||
private String lang = null;
|
||||
|
||||
/**
|
||||
* If specified, returns tweets with since the given date. Date should be formatted as YYYY-MM-DD.
|
||||
*/
|
||||
@Pattern(regexp = "^\\d{4}-\\d{2}-\\d{2}$")
|
||||
private String since = null;
|
||||
|
||||
/**
|
||||
* If specified, returns tweets by users located within a given radius (in Km) of the given latitude/longitude,
|
||||
* where the user's location is taken from their Twitter profile.
|
||||
* Should be formatted as
|
||||
*/
|
||||
private Geocode geocode = new Geocode();
|
||||
|
||||
/**
|
||||
* Specifies what type of search results you would prefer to receive.
|
||||
* The current default is "mixed." Valid values include:
|
||||
* mixed : Include both popular and real time results in the response.
|
||||
* recent : return only the most recent results in the response
|
||||
* popular : return only the most popular results in the response
|
||||
*/
|
||||
@NotNull
|
||||
private Query.ResultType resultType = Query.ResultType.mixed;
|
||||
|
||||
/**
|
||||
* Restart search from the most recent tweets on empty response.
|
||||
* Applied only after the first restart (e.g. when since_id != UNBOUNDED)
|
||||
*/
|
||||
private boolean restartFromMostRecentOnEmptyResponse = false;
|
||||
|
||||
public String getQuery() {
|
||||
return query;
|
||||
}
|
||||
|
||||
public void setQuery(String query) {
|
||||
this.query = query;
|
||||
}
|
||||
|
||||
public String getLang() {
|
||||
return lang;
|
||||
}
|
||||
|
||||
public void setLang(String lang) {
|
||||
this.lang = lang;
|
||||
}
|
||||
|
||||
public int getPage() {
|
||||
return page;
|
||||
}
|
||||
|
||||
public void setPage(int page) {
|
||||
this.page = page;
|
||||
}
|
||||
|
||||
public int getCount() {
|
||||
return count;
|
||||
}
|
||||
|
||||
public void setCount(int count) {
|
||||
this.count = count;
|
||||
}
|
||||
|
||||
public String getSince() {
|
||||
return since;
|
||||
}
|
||||
|
||||
public void setSince(String since) {
|
||||
this.since = since;
|
||||
}
|
||||
|
||||
public Geocode getGeocode() {
|
||||
return geocode;
|
||||
}
|
||||
|
||||
public Query.ResultType getResultType() {
|
||||
return resultType;
|
||||
}
|
||||
|
||||
public void setResultType(Query.ResultType resultType) {
|
||||
this.resultType = resultType;
|
||||
}
|
||||
|
||||
public boolean isRestartFromMostRecentOnEmptyResponse() {
|
||||
return restartFromMostRecentOnEmptyResponse;
|
||||
}
|
||||
|
||||
public void setRestartFromMostRecentOnEmptyResponse(boolean restartFromMostRecentOnEmptyResponse) {
|
||||
this.restartFromMostRecentOnEmptyResponse = restartFromMostRecentOnEmptyResponse;
|
||||
}
|
||||
|
||||
public static class Geocode {
|
||||
|
||||
/**
|
||||
* User's latitude.
|
||||
*/
|
||||
private double latitude = -1;
|
||||
|
||||
/**
|
||||
* User's longitude.
|
||||
*/
|
||||
private double longitude = -1;
|
||||
|
||||
/**
|
||||
* Radius (in kilometers) around the (latitude, longitude) point.
|
||||
*/
|
||||
private double radius = -1;
|
||||
|
||||
public double getLatitude() {
|
||||
return latitude;
|
||||
}
|
||||
|
||||
public void setLatitude(double latitude) {
|
||||
this.latitude = latitude;
|
||||
}
|
||||
|
||||
public double getLongitude() {
|
||||
return longitude;
|
||||
}
|
||||
|
||||
public void setLongitude(double longitude) {
|
||||
this.longitude = longitude;
|
||||
}
|
||||
|
||||
public double getRadius() {
|
||||
return radius;
|
||||
}
|
||||
|
||||
public void setRadius(double radius) {
|
||||
this.radius = radius;
|
||||
}
|
||||
|
||||
public boolean isValid() {
|
||||
return this.radius > 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.stream;
|
||||
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import reactor.core.publisher.Flux;
|
||||
import twitter4j.StallWarning;
|
||||
import twitter4j.Status;
|
||||
import twitter4j.StatusDeletionNotice;
|
||||
import twitter4j.StatusListener;
|
||||
import twitter4j.TwitterStream;
|
||||
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionConfiguration;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.integration.channel.FluxMessageChannel;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.messaging.MessageHeaders;
|
||||
import org.springframework.messaging.support.MessageBuilder;
|
||||
import org.springframework.util.MimeTypeUtils;
|
||||
|
||||
/**
|
||||
*
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
|
||||
@EnableConfigurationProperties({ TwitterStreamSupplierProperties.class, TwitterConnectionProperties.class })
|
||||
@Import(TwitterConnectionConfiguration.class)
|
||||
public class TwitterStreamSupplierConfiguration {
|
||||
|
||||
private static final Log logger = LogFactory.getLog(TwitterStreamSupplierConfiguration.class);
|
||||
|
||||
@Bean
|
||||
public FluxMessageChannel output() {
|
||||
return new FluxMessageChannel();
|
||||
}
|
||||
|
||||
@Bean
|
||||
public StatusListener twitterStatusListener(FluxMessageChannel output, TwitterStream twitterStream,
|
||||
ObjectMapper objectMapper) {
|
||||
|
||||
StatusListener statusListener = new StatusListener() {
|
||||
|
||||
@Override
|
||||
public void onException(Exception e) {
|
||||
logger.error("Status Error: ", e);
|
||||
throw new RuntimeException("Status Error: ", e);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onDeletionNotice(StatusDeletionNotice arg) {
|
||||
logger.info("StatusDeletionNotice: " + arg);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onScrubGeo(long userId, long upToStatusId) {
|
||||
logger.info("onScrubGeo: " + userId + ", " + upToStatusId);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onStallWarning(StallWarning warning) {
|
||||
logger.warn("Stall Warning: " + warning);
|
||||
throw new RuntimeException("Stall Warning: " + warning);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onStatus(Status status) {
|
||||
|
||||
try {
|
||||
String json = objectMapper.writeValueAsString(status);
|
||||
Message<byte[]> message = MessageBuilder.withPayload(json.getBytes())
|
||||
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON_VALUE)
|
||||
.build();
|
||||
output.send(message);
|
||||
// System.out.println(json);
|
||||
}
|
||||
catch (JsonProcessingException e) {
|
||||
logger.error("Status to JSON conversion error!", e);
|
||||
throw new RuntimeException("Status to JSON conversion error!", e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void onTrackLimitationNotice(int numberOfLimitedStatuses) {
|
||||
logger.warn("Track Limitation Notice: " + numberOfLimitedStatuses);
|
||||
}
|
||||
};
|
||||
|
||||
twitterStream.addListener(statusListener);
|
||||
|
||||
return statusListener;
|
||||
}
|
||||
|
||||
@Bean
|
||||
public Supplier<Flux<Message<?>>> twitterStreamSupplier(TwitterStream twitterStream,
|
||||
FluxMessageChannel output, TwitterStreamSupplierProperties streamProperties) {
|
||||
|
||||
return () -> Flux.from(output)
|
||||
.doOnSubscribe(subscription -> {
|
||||
try {
|
||||
switch (streamProperties.getType()) {
|
||||
|
||||
case filter:
|
||||
twitterStream.filter(streamProperties.getFilter().toFilterQuery());
|
||||
return;
|
||||
|
||||
case sample:
|
||||
twitterStream.sample();
|
||||
return;
|
||||
|
||||
case firehose:
|
||||
twitterStream.firehose(streamProperties.getFilter().getCount());
|
||||
return;
|
||||
|
||||
case link:
|
||||
twitterStream.links(streamProperties.getFilter().getCount());
|
||||
return;
|
||||
default:
|
||||
throw new IllegalArgumentException("Unknown stream type:" + streamProperties.getType());
|
||||
}
|
||||
}
|
||||
catch (Exception e) {
|
||||
this.logger.error("Filter is not property set");
|
||||
}
|
||||
})
|
||||
.doAfterTerminate(() -> {
|
||||
this.logger.info("Proactive cancel for twitter stream");
|
||||
twitterStream.shutdown();
|
||||
})
|
||||
.doOnError(throwable -> {
|
||||
this.logger.error(throwable.getMessage(), throwable);
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,267 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.stream;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import twitter4j.FilterQuery;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.util.CollectionUtils;
|
||||
import org.springframework.validation.annotation.Validated;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@ConfigurationProperties("twitter.stream")
|
||||
@Validated
|
||||
public class TwitterStreamSupplierProperties {
|
||||
|
||||
public enum StreamType {
|
||||
/** Starts listening on random sample of all public statuses. The default access level provides a small
|
||||
* proportion of the Firehose. */
|
||||
sample,
|
||||
|
||||
/** Start consuming public statuses that match one or more filter predicates. At least one predicate parameter,
|
||||
* follow, locations, or track must be specified. Multiple parameters may be specified which allows most
|
||||
* clients to use a single connection to the Streaming API. Placing long parameters in the URL may cause the
|
||||
* request to be rejected for excessive URL length.<br>
|
||||
* The default access level allows up to 200 track keywords, 400 follow userids and 10 1-degree location boxes.
|
||||
* Increased access levels allow 80,000 follow userids ("shadow" role), 400,000 follow userids ("birddog" role),
|
||||
* 10,000 track keywords ("restricted track" role), 200,000 track keywords ("partner track" role),
|
||||
* and 200 10-degree location boxes ("locRestricted" role). Increased track access levels also pass a higher
|
||||
* proportion of statuses before limiting the stream.*/
|
||||
filter,
|
||||
|
||||
/** tarts listening on all public statuses. Available only to approved parties and requires a signed agreement
|
||||
* to access. */
|
||||
firehose,
|
||||
|
||||
/** Starts listening on all public statuses containing links.
|
||||
* Available only to approved parties and requires a signed agreement to access.*/
|
||||
link
|
||||
}
|
||||
|
||||
private StreamType type = StreamType.sample;
|
||||
|
||||
private Filter filter = new Filter();
|
||||
|
||||
public Filter getFilter() {
|
||||
return filter;
|
||||
}
|
||||
|
||||
public StreamType getType() {
|
||||
return type;
|
||||
}
|
||||
|
||||
public void setType(StreamType type) {
|
||||
this.type = type;
|
||||
}
|
||||
|
||||
public static class Filter {
|
||||
|
||||
public enum FilterLevel {
|
||||
/** filter level. */
|
||||
all, none, low, medium
|
||||
}
|
||||
|
||||
/**
|
||||
* Indicates the number of previous statuses to stream before transitioning to the live stream.
|
||||
*/
|
||||
private int count = 0;
|
||||
|
||||
/**
|
||||
* Specifies the users, by ID, to receive public tweets from.
|
||||
*/
|
||||
private List<Long> follow;
|
||||
|
||||
/**
|
||||
* Specifies keywords to track.
|
||||
*/
|
||||
private List<String> track;
|
||||
|
||||
|
||||
/**
|
||||
* Locations to track. Internally represented as 2D array.
|
||||
* Bounding box is invalid: 52.38, 4.90, 51.51, -0.12. The first pair must be the SW corner of the box
|
||||
*/
|
||||
private List<BoundingBox> locations = new ArrayList<>();
|
||||
|
||||
|
||||
/**
|
||||
* Specifies the tweets language of the stream.
|
||||
*/
|
||||
private List<String> language;
|
||||
|
||||
/**
|
||||
* The filter level limits what tweets appear in the stream to those with a minimum filterLevel attribute value.
|
||||
* One of either none, low, or medium.
|
||||
*/
|
||||
private FilterLevel filterLevel = FilterLevel.all;
|
||||
|
||||
public int getCount() {
|
||||
return count;
|
||||
}
|
||||
|
||||
public void setCount(int count) {
|
||||
this.count = count;
|
||||
}
|
||||
|
||||
public List<Long> getFollow() {
|
||||
return follow;
|
||||
}
|
||||
|
||||
public void setFollow(List<Long> follow) {
|
||||
this.follow = follow;
|
||||
}
|
||||
|
||||
public List<String> getTrack() {
|
||||
return track;
|
||||
}
|
||||
|
||||
public void setTrack(List<String> track) {
|
||||
this.track = track;
|
||||
}
|
||||
|
||||
public List<String> getLanguage() {
|
||||
return language;
|
||||
}
|
||||
|
||||
public void setLanguage(List<String> language) {
|
||||
this.language = language;
|
||||
}
|
||||
|
||||
public List<BoundingBox> getLocations() {
|
||||
return locations;
|
||||
}
|
||||
|
||||
public FilterLevel getFilterLevel() {
|
||||
return filterLevel;
|
||||
}
|
||||
|
||||
public void setFilterLevel(FilterLevel filterLevel) {
|
||||
this.filterLevel = filterLevel;
|
||||
}
|
||||
|
||||
public FilterQuery toFilterQuery() {
|
||||
|
||||
FilterQuery filterQuery = new FilterQuery();
|
||||
|
||||
filterQuery.count(this.count);
|
||||
|
||||
if (!CollectionUtils.isEmpty(this.track)) {
|
||||
filterQuery.track(String.join(",", this.track));
|
||||
}
|
||||
|
||||
if (!CollectionUtils.isEmpty(this.follow)) {
|
||||
long[] followIds = new long[this.follow.size()];
|
||||
for (int i = 0; i < this.follow.size(); i++) {
|
||||
followIds[i] = this.follow.get(i);
|
||||
}
|
||||
filterQuery.follow(followIds);
|
||||
}
|
||||
|
||||
if (!CollectionUtils.isEmpty(this.language)) {
|
||||
filterQuery.language(this.language.toArray(new String[this.language.size()]));
|
||||
}
|
||||
|
||||
if (!CollectionUtils.isEmpty(this.locations)) {
|
||||
double[][] bboxLocations = new double[this.locations.size() * 2][2];
|
||||
for (int i = 0; i < this.locations.size(); i = i + 2) {
|
||||
//SW lat, lon
|
||||
bboxLocations[i][0] = this.locations.get(i).getSw().getLat();
|
||||
bboxLocations[i][1] = this.locations.get(i).getSw().getLon();
|
||||
//NE lat, lon
|
||||
bboxLocations[i + 1][0] = this.locations.get(i).getNe().getLat();
|
||||
bboxLocations[i + 1][1] = this.locations.get(i).getNe().getLon();
|
||||
}
|
||||
filterQuery.locations(bboxLocations);
|
||||
}
|
||||
|
||||
if (this.filterLevel != FilterLevel.all) {
|
||||
filterQuery.filterLevel(this.filterLevel.name());
|
||||
}
|
||||
|
||||
return filterQuery;
|
||||
}
|
||||
|
||||
public boolean isValid() {
|
||||
return count > 0 || !CollectionUtils.isEmpty(this.track) || !CollectionUtils.isEmpty(this.follow)
|
||||
|| !CollectionUtils.isEmpty(this.language) || this.filterLevel != FilterLevel.all;
|
||||
}
|
||||
|
||||
public static class BoundingBox {
|
||||
|
||||
/**
|
||||
* Bounding Box's South-West point (e.g. bottom-left).
|
||||
*/
|
||||
private Geocode sw;
|
||||
|
||||
/**
|
||||
* Bounding Box's North-East point (e.g. top-right).
|
||||
*/
|
||||
private Geocode ne;
|
||||
|
||||
public Geocode getSw() {
|
||||
return sw;
|
||||
}
|
||||
|
||||
public void setSw(Geocode sw) {
|
||||
this.sw = sw;
|
||||
}
|
||||
|
||||
public Geocode getNe() {
|
||||
return ne;
|
||||
}
|
||||
|
||||
public void setNe(Geocode ne) {
|
||||
this.ne = ne;
|
||||
}
|
||||
}
|
||||
|
||||
public static class Geocode {
|
||||
|
||||
/**
|
||||
* latitude.
|
||||
*/
|
||||
private double lat = -1;
|
||||
|
||||
/**
|
||||
* longitude.
|
||||
*/
|
||||
private double lon = -1;
|
||||
|
||||
|
||||
public double getLat() {
|
||||
return lat;
|
||||
}
|
||||
|
||||
public void setLat(double lat) {
|
||||
this.lat = lat;
|
||||
}
|
||||
|
||||
public double getLon() {
|
||||
return lon;
|
||||
}
|
||||
|
||||
public void setLon(double lon) {
|
||||
this.lon = lon;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.search;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Date;
|
||||
import java.util.List;
|
||||
|
||||
import org.junit.Test;
|
||||
import twitter4j.GeoLocation;
|
||||
import twitter4j.HashtagEntity;
|
||||
import twitter4j.MediaEntity;
|
||||
import twitter4j.Place;
|
||||
import twitter4j.RateLimitStatus;
|
||||
import twitter4j.Scopes;
|
||||
import twitter4j.Status;
|
||||
import twitter4j.SymbolEntity;
|
||||
import twitter4j.URLEntity;
|
||||
import twitter4j.User;
|
||||
import twitter4j.UserMentionEntity;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.springframework.cloud.fn.supplier.twitter.status.search.SearchPagination.UNBOUNDED;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class SearchPaginationTests {
|
||||
|
||||
@Test
|
||||
public void tests1() {
|
||||
|
||||
int maxCountPerRequest = 5;
|
||||
|
||||
int count = 10;
|
||||
|
||||
int pageCount = count / maxCountPerRequest;
|
||||
|
||||
assertThat(pageCount).isEqualTo(2);
|
||||
|
||||
SearchPagination pagination = new SearchPagination(pageCount, false);
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 1);
|
||||
|
||||
pagination.update(tweets(10, 8, 1));
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(8L - 1); // = min - 1
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(10L);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 2);
|
||||
|
||||
pagination.update(tweets(7, 4, 1));
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(10L);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 1);
|
||||
|
||||
pagination.update(tweets(20, 14, 1));
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(10L);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(14L - 1);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(20L);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 2);
|
||||
|
||||
pagination.update(tweets(13, 8, 1));
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(20L);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void tests2() {
|
||||
|
||||
int maxCountPerRequest = 5;
|
||||
|
||||
int count = 10;
|
||||
|
||||
int pageCount = count / maxCountPerRequest;
|
||||
|
||||
assertThat(pageCount).isEqualTo(2);
|
||||
|
||||
SearchPagination pagination = new SearchPagination(pageCount, true);
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 1);
|
||||
|
||||
pagination.update(tweets(10, 8, 1));
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(8L - 1); // = min - 1
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(10L);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 2);
|
||||
|
||||
pagination.update(tweets(7, 4, 1));
|
||||
|
||||
// Restart from Most Recent due to pageCounter == 0 while no reset have been performed so ar
|
||||
assertThat(pagination.getSinceId()).isEqualTo(10L);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 1);
|
||||
|
||||
pagination.update(tweets(20, 14, 1));
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(10L);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(14L - 1);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(20L);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 2);
|
||||
|
||||
pagination.update(tweets(13, 8, 1));
|
||||
|
||||
assertThat(pagination.getSinceId()).isEqualTo(10L);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(8L - 1);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(20L);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 3);
|
||||
|
||||
pagination.update(new ArrayList<>()); // EMPTY
|
||||
|
||||
// Restart from Most Recent on Empty Response
|
||||
assertThat(pagination.getSinceId()).isEqualTo(20L);
|
||||
assertThat(pagination.getMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageMaxId()).isEqualTo(UNBOUNDED);
|
||||
assertThat(pagination.getPageCounter()).isEqualTo(pageCount - 1);
|
||||
}
|
||||
|
||||
public List<Status> tweets(int to, int from, int step) {
|
||||
List<Status> tweets = new ArrayList<>(to - from);
|
||||
for (int i = to; i >= from; i = i - step) {
|
||||
tweets.add(new MyStatus(i));
|
||||
}
|
||||
|
||||
return tweets;
|
||||
}
|
||||
|
||||
public static class MyStatus implements Status {
|
||||
|
||||
private long id;
|
||||
|
||||
public MyStatus(long id) {
|
||||
this.id = id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Date getCreatedAt() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public long getId() {
|
||||
return this.id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getText() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getDisplayTextRangeStart() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getDisplayTextRangeEnd() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getSource() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isTruncated() {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public long getInReplyToStatusId() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public long getInReplyToUserId() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getInReplyToScreenName() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public GeoLocation getGeoLocation() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Place getPlace() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isFavorited() {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isRetweeted() {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getFavoriteCount() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public User getUser() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isRetweet() {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Status getRetweetedStatus() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public long[] getContributors() {
|
||||
return new long[0];
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getRetweetCount() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isRetweetedByMe() {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public long getCurrentUserRetweetId() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isPossiblySensitive() {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getLang() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Scopes getScopes() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String[] getWithheldInCountries() {
|
||||
return new String[0];
|
||||
}
|
||||
|
||||
@Override
|
||||
public long getQuotedStatusId() {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Status getQuotedStatus() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public URLEntity getQuotedStatusPermalink() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int compareTo(Status o) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public UserMentionEntity[] getUserMentionEntities() {
|
||||
return new UserMentionEntity[0];
|
||||
}
|
||||
|
||||
@Override
|
||||
public URLEntity[] getURLEntities() {
|
||||
return new URLEntity[0];
|
||||
}
|
||||
|
||||
@Override
|
||||
public HashtagEntity[] getHashtagEntities() {
|
||||
return new HashtagEntity[0];
|
||||
}
|
||||
|
||||
@Override
|
||||
public MediaEntity[] getMediaEntities() {
|
||||
return new MediaEntity[0];
|
||||
}
|
||||
|
||||
@Override
|
||||
public SymbolEntity[] getSymbolEntities() {
|
||||
return new SymbolEntity[0];
|
||||
}
|
||||
|
||||
@Override
|
||||
public RateLimitStatus getRateLimitStatus() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getAccessLevel() {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.stream;
|
||||
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import org.junit.AfterClass;
|
||||
import org.junit.BeforeClass;
|
||||
import org.junit.Test;
|
||||
import org.junit.runner.RunWith;
|
||||
import org.mockserver.client.MockServerClient;
|
||||
import org.mockserver.integration.ClientAndServer;
|
||||
import org.mockserver.model.Header;
|
||||
import org.mockserver.model.HttpRequest;
|
||||
import org.mockserver.model.StringBody;
|
||||
import reactor.core.publisher.Flux;
|
||||
import reactor.test.StepVerifier;
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.cloud.fn.common.twitter.TwitterConnectionProperties;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.messaging.Message;
|
||||
import org.springframework.test.annotation.DirtiesContext;
|
||||
import org.springframework.test.context.TestPropertySource;
|
||||
import org.springframework.test.context.junit4.SpringRunner;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockserver.matchers.Times.exactly;
|
||||
import static org.mockserver.model.HttpRequest.request;
|
||||
import static org.mockserver.model.HttpResponse.response;
|
||||
import static org.mockserver.verify.VerificationTimes.once;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
@RunWith(SpringRunner.class)
|
||||
@SpringBootTest(
|
||||
webEnvironment = SpringBootTest.WebEnvironment.NONE,
|
||||
properties = {
|
||||
"twitter.connection.consumerKey=consumerKey666",
|
||||
"twitter.connection.consumerSecret=consumerSecret666",
|
||||
"twitter.connection.accessToken=accessToken666",
|
||||
"twitter.connection.accessTokenSecret=accessTokenSecret666"
|
||||
})
|
||||
@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS)
|
||||
public abstract class TwitterStreamSupplierTests {
|
||||
|
||||
private static final String MOCK_SERVER_IP = "127.0.0.1";
|
||||
|
||||
private static final Integer MOCK_SERVER_PORT = 1080;
|
||||
|
||||
private static ClientAndServer mockServer;
|
||||
|
||||
private static MockServerClient mockClient;
|
||||
private static HttpRequest streamFilterRequest;
|
||||
private static HttpRequest streamSampleRequest;
|
||||
private static HttpRequest streamFirehoseRequest;
|
||||
|
||||
@Autowired
|
||||
protected Supplier<Flux<Message<?>>> twitterStreamSupplier;
|
||||
|
||||
@BeforeClass
|
||||
public static void startServer() {
|
||||
|
||||
mockServer = ClientAndServer.startClientAndServer(MOCK_SERVER_PORT);
|
||||
|
||||
mockClient = new MockServerClient(MOCK_SERVER_IP, MOCK_SERVER_PORT);
|
||||
|
||||
streamFilterRequest = mockClientRecordRequest(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/stream/statuses/filter.json")
|
||||
.withBody(new StringBody("count=0&track=Java%2CPython&stall_warnings=true")));
|
||||
|
||||
streamSampleRequest = mockClientRecordRequest(request()
|
||||
.withMethod("GET")
|
||||
.withPath("/stream/statuses/sample.json"));
|
||||
|
||||
streamFirehoseRequest = mockClientRecordRequest(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/stream/statuses/links.json")
|
||||
.withBody(new StringBody("count=0&stall_warnings=true")));
|
||||
|
||||
streamFirehoseRequest = mockClientRecordRequest(request()
|
||||
.withMethod("POST")
|
||||
.withPath("/stream/statuses/firehose.json")
|
||||
.withBody(new StringBody("count=0&stall_warnings=true")));
|
||||
}
|
||||
|
||||
@AfterClass
|
||||
public static void stopServer() {
|
||||
mockServer.stop();
|
||||
}
|
||||
|
||||
private static HttpRequest mockClientRecordRequest(HttpRequest request) {
|
||||
mockClient.when(request, /*unlimited())*/ exactly(1))
|
||||
.respond(
|
||||
response()
|
||||
.withStatusCode(200)
|
||||
.withHeaders(
|
||||
new Header("Content-Type", "application/json; charset=utf-8"),
|
||||
new Header("Cache-Control", "public, max-age=86400"))
|
||||
.withBody(TwitterTestUtils.asString("classpath:/response/stream_test_1.json"))
|
||||
.withDelay(TimeUnit.SECONDS, 10));
|
||||
return request;
|
||||
}
|
||||
|
||||
@TestPropertySource(properties = {
|
||||
"twitter.stream.type=sample"
|
||||
})
|
||||
public static class TwitterStreamSampleTests extends TwitterStreamSupplierTests {
|
||||
|
||||
@Test
|
||||
public void testOne() {
|
||||
final Flux<Message<?>> messageFlux = twitterStreamSupplier.get();
|
||||
|
||||
final StepVerifier stepVerifier = StepVerifier.create(messageFlux)
|
||||
.assertNext((message) -> assertThat(new String((byte[]) message.getPayload()))
|
||||
.contains("\"id\":1075751718749659136"))
|
||||
.thenCancel()
|
||||
.verifyLater();
|
||||
|
||||
stepVerifier.verify();
|
||||
|
||||
mockClient.verify(streamSampleRequest, once());
|
||||
}
|
||||
}
|
||||
|
||||
@TestPropertySource(properties = {
|
||||
"twitter.stream.type=filter",
|
||||
"twitter.stream.filter.track=Java,Python"
|
||||
})
|
||||
public static class TwitterStreamFilterTests extends TwitterStreamSupplierTests {
|
||||
|
||||
@Test
|
||||
public void testOne() {
|
||||
final Flux<Message<?>> messageFlux = twitterStreamSupplier.get();
|
||||
|
||||
final StepVerifier stepVerifier = StepVerifier.create(messageFlux)
|
||||
.assertNext((message) -> assertThat(new String((byte[]) message.getPayload()))
|
||||
.contains("\"id\":1075751718749659136"))
|
||||
.thenCancel()
|
||||
.verifyLater();
|
||||
|
||||
stepVerifier.verify();
|
||||
|
||||
mockClient.verify(streamFilterRequest, once());
|
||||
}
|
||||
}
|
||||
|
||||
@TestPropertySource(properties = {
|
||||
"twitter.stream.type=firehose"
|
||||
})
|
||||
public static class TwitterStreamFirehoseTests extends TwitterStreamSupplierTests {
|
||||
|
||||
@Test
|
||||
public void testOne() {
|
||||
final Flux<Message<?>> messageFlux = twitterStreamSupplier.get();
|
||||
|
||||
final StepVerifier stepVerifier = StepVerifier.create(messageFlux)
|
||||
.assertNext((message) -> assertThat(new String((byte[]) message.getPayload()))
|
||||
.contains("\"id\":1075751718749659136"))
|
||||
.thenCancel()
|
||||
.verifyLater();
|
||||
|
||||
stepVerifier.verify();
|
||||
|
||||
mockClient.verify(streamFirehoseRequest, once());
|
||||
}
|
||||
}
|
||||
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(TwitterStreamSupplierConfiguration.class)
|
||||
public static class TestTwitterStreamSourceApplication {
|
||||
|
||||
@Bean
|
||||
@Primary
|
||||
public twitter4j.conf.Configuration twitterConfiguration2(TwitterConnectionProperties properties,
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> toConfigurationBuilder) {
|
||||
|
||||
Function<TwitterConnectionProperties, ConfigurationBuilder> mockedConfiguration =
|
||||
toConfigurationBuilder.andThen(
|
||||
new TwitterTestUtils().mockTwitterUrls(
|
||||
String.format("http://%s:%s", MOCK_SERVER_IP, MOCK_SERVER_PORT)));
|
||||
|
||||
return mockedConfiguration.apply(properties).build();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright 2020-2020 the original author or authors.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.springframework.cloud.fn.supplier.twitter.status.stream;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.Charset;
|
||||
import java.util.function.Function;
|
||||
|
||||
import twitter4j.conf.ConfigurationBuilder;
|
||||
|
||||
import org.springframework.core.io.DefaultResourceLoader;
|
||||
import org.springframework.util.StreamUtils;
|
||||
|
||||
/**
|
||||
* @author Christian Tzolov
|
||||
*/
|
||||
public class TwitterTestUtils {
|
||||
|
||||
public Function<ConfigurationBuilder, ConfigurationBuilder> mockTwitterUrls(String baseUrl) {
|
||||
return configBuilder -> {
|
||||
configBuilder.setRestBaseURL(baseUrl + "/");
|
||||
configBuilder.setStreamBaseURL(baseUrl + "/stream/");
|
||||
configBuilder.setUserStreamBaseURL(baseUrl + "/user/");
|
||||
configBuilder.setSiteStreamBaseURL(baseUrl + "/site/");
|
||||
configBuilder.setUploadBaseURL(baseUrl + "/upload/");
|
||||
|
||||
configBuilder.setOAuthAccessTokenURL(baseUrl + "/oauth/access_token");
|
||||
configBuilder.setOAuthAuthenticationURL(baseUrl + "/oauth/authenticate");
|
||||
configBuilder.setOAuthAuthorizationURL(baseUrl + "/oauth/authorize");
|
||||
configBuilder.setOAuthRequestTokenURL(baseUrl + "/oauth/request_token");
|
||||
configBuilder.setOAuth2TokenURL(baseUrl + "/oauth2/token");
|
||||
configBuilder.setOAuth2InvalidateTokenURL(baseUrl + "/oauth2/invalidate_token");
|
||||
|
||||
return configBuilder;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load Spring Resource as String.
|
||||
* @param resourcePath Resource path (accepts file:// , classpath:// and http:// uri schemas)
|
||||
* @return Returns text (UTF8) representation of the resource pointed by the resourcePath
|
||||
*/
|
||||
public static String asString(String resourcePath) {
|
||||
try {
|
||||
return StreamUtils.copyToString(new DefaultResourceLoader().getResource(resourcePath).getInputStream(),
|
||||
Charset.forName("UTF-8"));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("Can not load resource:" + resourcePath, e);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"created_at":"Thu Dec 20 13:56:10 +0000 2018","id":1075751718749659136,"id_str":"1075751718749659136","text":"Codementor: Kubernetes for Python Developers: Part 1\n#100daysofcode #python https:\/\/t.co\/VviTgpFpge","source":"\u003ca href=\"https:\/\/ifttt.com\" rel=\"nofollow\"\u003eIFTTT\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":859252650512072704,"id_str":"859252650512072704","name":"Freelancing|WebDev","screen_name":"FreelanceForBTC","location":"THE NET","url":"http:\/\/bit.ly\/BTCFREELANCING","description":"This twitter is designed to give the best information from web development to Blockchain.\n|YouTube's |Jobs |learning contented","translator_type":"none","protected":false,"verified":false,"followers_count":146,"friends_count":118,"listed_count":4,"favourites_count":445,"statuses_count":7528,"created_at":"Tue May 02 03:46:10 +0000 2017","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1071607798771793920\/UegrCs84_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/859252650512072704\/1544326203","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"100daysofcode","indices":[53,67]},{"text":"python","indices":[68,75]}],"urls":[{"url":"https:\/\/t.co\/VviTgpFpge","expanded_url":"https:\/\/ift.tt\/2GxdMZF","display_url":"ift.tt\/2GxdMZF","indices":[76,99]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"ca","timestamp_ms":"1545314170907"}
|
||||
|
||||
Reference in New Issue
Block a user