diff --git a/src/main/asciidoc/analytics/twitter-analytics/main.adoc b/src/main/asciidoc/analytics/twitter-analytics/main.adoc index e17ade7..40d9cb8 100644 --- a/src/main/asciidoc/analytics/twitter-analytics/main.adoc +++ b/src/main/asciidoc/analytics/twitter-analytics/main.adoc @@ -3,7 +3,8 @@ :docs_dir: ../.. === Twitter Analytics -In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_ and compute simple analytics over data-in-transit using _Field-Value-Counter_. +In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_, compute analytics over data-in-transit using https://github.com/spring-cloud-stream-app-starters/analytics[Analytics-Counter]. +Use Prometheus for storing and data aggregation analysis and Grafana for visualizing the computed data. We will take you through the steps to configure Spring Cloud Data Flow's `Local` server. @@ -13,7 +14,15 @@ We will take you through the steps to configure Spring Cloud Data Flow's `Local` include::{docs_dir}/shell.adoc[] * A running local Data Flow Server include::{docs_dir}/local-server.adoc[] -* Running instance of link:http://redis.io/[Redis] +Make sure to add the following properties when starting the Data Flow server: +``` +--spring.cloud.dataflow.applicationProperties.stream.management.metrics.export.prometheus.enabled=true +--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.streamapp.security.enabled=false +--spring.cloud.dataflow.applicationProperties.stream.management.endpoints.web.exposure.include=prometheus,info,health +--spring.cloud.dataflow.grafana-info.url=http://localhost:3000 +``` +* Running instance of link:http://docs.spring.io/spring-cloud-dataflow/docs/2.0.0.BUILD-SNAPSHOT/reference/htmlsingle/#streams-monitoring-local-prometheus[Prometheus, Service Discovery and Grafana]. +Follow the http://docs.spring.io/spring-cloud-dataflow/docs/2.0.0.BUILD-SNAPSHOT/reference/htmlsingle/#streams-monitoring-local-prometheus[instructions] to start those services in Docker containers. * Running instance of link:http://kafka.apache.org/downloads.html[Kafka] * Twitter credentials from link:https://apps.twitter.com/[Twitter Developers] site @@ -31,119 +40,102 @@ dataflow:>app import --uri {app-import-kafka-maven} . Create and deploy the following streams + +image::scdf-tweets-analysis-architecture.png[Twitter Analytics Visualization, scaledwidth="100%"] +The `tweets` stream subscribes to the provided twitter account, reads the incoming JSON tweets and logs their content to the log. ++ ``` dataflow:>stream create tweets --definition "twitterstream --consumerKey= --consumerSecret= --accessToken= --accessTokenSecret= | log" Created new stream 'tweets' ``` +The received https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/intro-to-tweet-json.html[tweet messages] have a format similar to this: ++ +[source,json] +---- +{ + "created_at": "Thu Apr 06 15:24:15 +0000 2017", + "id_str": "850006245121695744", + "text": "Today we are sharing our vision for the future of the Twitter API platform!", + "user": { + "id": 2244994945, + "name": "Twitter Dev", + "screen_name": "TwitterDev", + "lang": "en" + }, + "place": {}, + "entities": { + "hashtags": [ + { + "text": "documentation", + "indices": [211, 225] + }, + { + "text": "GeoTagged", + "indices": [239, 249] + } + ], + .... + } +} +---- ++ +The https://github.com/json-path/JsonPath[JsonPath] SpEL expressions can help to extract the attributes to be analysed. +For example the `#jsonPath(payload,'$..lang')` expression extracts all values of the `lang` attributes in the tweet. +The https://github.com/spring-cloud-stream-app-starters/analytics/tree/master/spring-cloud-starter-stream-sink-counter[Analytics Counter Sink] maps the extracted values to custom https://micrometer.io/docs/concepts#_meters[Micrometer tags/dimensions] attached to every measurement send. +The `tweetlang` stream created below, extracts and counts the languages found in the tweets. +The counter, named `language`, applies the `--counter.tag.expression.lang=#jsonPath(payload,'$..lang')` to extract the language values and map them to a Micrometer tag named: `lang`. +This counter generates the `language_total` time-series send to Prometheus. + ``` -dataflow:>stream create tweetlang --definition ":tweets.twitterstream > field-value-counter --fieldName=lang --name=language" --deploy +dataflow:>stream create tweetlang --definition ":tweets.twitterstream > counter --name=language --counter.tag.expression.lang=#jsonPath(payload,'$..lang')" --deploy Created and deployed new stream 'tweetlang' ``` + +Similarly, we can use the `#jsonPath(payload,'$.entities.hashtags[*].text')` expression to extract and count the hastags in the incoming tweets. +The following stream uses the counter-sink to compute real-time counts (named as `hashtags`) and the `htag` attribute in `counter.tag.expression.htag` indicate to Micrometer in what tag to hold the extracted hashtag values from the incoming tweets. ++ ``` -dataflow:>stream create tagcount --definition ":tweets.twitterstream > field-value-counter --fieldName=entities.hashtags.text --name=hashtags" --deploy +dataflow:>stream create tagcount --definition ":tweets.twitterstream > counter --name=hashtags --counter.tag.expression.htag=#jsonPath(payload,'$.entities.hashtags[*].text')" --deploy Created and deployed new stream 'tagcount' ``` + +Now we can deploy the `tweets` stream to start tweet analysis. ++ ``` dataflow:>stream deploy tweets Deployed stream 'tweets' ``` + NOTE: To get a consumerKey and consumerSecret you need to register a twitter application. If you don’t already have one set up, you can create an app at the link:https://apps.twitter.com/[Twitter Developers] site to get these credentials. The tokens ``, ``, ``, and `` are required to be replaced with your account credentials. - + . Verify the streams are successfully deployed. Where: (1) is the primary pipeline; (2) and (3) are tapping the primary pipeline with the DSL syntax `.