SpringCloud/spring-cloud-dataflow-samples
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add scfn + scdf integration sample

Browse Source 
- add a new folder for scfn + scdf integration
- include quick-link from the root README
- replace spring-cloud-stream-modules with spring-cloud-stream-app-starters
- adjust text content for better continuity
- replace older bit.ly's with bacon release-train

Add function-runner repo link for clarity
This commit is contained in:
Sabby Anandan
2017-10-17 13:59:46 -07:00
committed by Soby Chacko
parent 84d872204b
commit 5d54cb280d
6 changed files with 202 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.adoc
View File

@@ -18,6 +18,12 @@ A data pipeline demonstration that consumes data from an `http` endpoint and wri
A simple Spring Batch job running as short-lived task in Cloud Foundry. A simple Spring Batch job running as short-lived task in Cloud Foundry.
## Functions
### link:functions/README.adoc[functions-in-scdf]
A simple demonstration of Spring Cloud Function and Spring Cloud Data Flow integration.
## Analytics ## Analytics
### link:analytics/twitter-analytics/README.adoc[twitter-analytics] ### link:analytics/twitter-analytics/README.adoc[twitter-analytics]
@@ -28,4 +34,4 @@ A data pipeline demonstration that consumes data from twitter-firehose using `tw
### link:datascience/species-prediction/README.adoc[species-prediction] ### link:datascience/species-prediction/README.adoc[species-prediction]
A simple demonstration to walkthrough the steps to compute real-time predictions using https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] data microservice application. A simple demonstration to walkthrough the steps to compute real-time predictions using https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] application.

10
analytics/twitter-analytics/README.adoc
View File

@@ -3,13 +3,13 @@
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_ and compute simple analytics over data-in-transit using _Field-Value-Counter_. In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_ and compute simple analytics over data-in-transit using _Field-Value-Counter_.
We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `Local` server, a Spring Boot application. We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `Local` server.
== Using Local Server == Using Local Server
=== Prerequisites === Prerequisites
In order to get started, make sure that you have the following components: Make sure that you have the following components:
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] * Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
* Running instance of link:http://redis.io/[Redis] * Running instance of link:http://redis.io/[Redis]
@@ -85,7 +85,7 @@ NOTE: To get a consumerKey and consumerSecret you need to register a twitter app
dataflow:>stream list dataflow:>stream list
``` ```
+ +
. Notice that `tweetlang.field-value-counter`, `tagcount.field-value-counter`, `tweets.log` and `tweets.twitterstream` link:https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] applications are running as Spring Boot applications within the `local-server`. . Notice that `tweetlang.field-value-counter`, `tagcount.field-value-counter`, `tweets.log` and `tweets.twitterstream` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as Spring Boot applications within the `local-server`.
+ +
``` ```
@@ -165,5 +165,5 @@ image::images/twitter_analytics.png[Twitter Analytics Visualization]
In this sample, you have learned: In this sample, you have learned:
* How to use Spring Cloud Data Flow's `Local` server * How to use Spring Cloud Data Flow's `Local` server
* How to use Spring Cloud Data Flow's `shell` * How to use Spring Cloud Data Flow's `shell` application
* How to create streaming data pipeline to compute simple analytics using `Twitter Stream` and `Field Value Counter` data microservices * How to create streaming data pipeline to compute simple analytics using `Twitter Stream` and `Field Value Counter` applications

8
datascience/species-prediction/README.adoc
View File

@@ -9,7 +9,7 @@ We will begin by discussing the steps to prep, configure and operationalize Spri
=== Prerequisites === Prerequisites
In order to get started, make sure that you have the following components: Make sure that you have the following components:
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] * Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka] * Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
@@ -54,7 +54,7 @@ dataflow:>version
+ +
``` ```
dataflow:>app import --uri http://bit.ly/stream-applications-kafka-maven dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
``` ```
+ +
@@ -76,7 +76,7 @@ NOTE: The built-in `pmml` processor will load the given PMML model definition an
dataflow:>stream list dataflow:>stream list
``` ```
+ +
. Notice that `pmmlTest.http`, `pmmlTest.pmml`, and `pmmlTest.log` link:https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] applications are running within the `local-server`. . Notice that `pmmlTest.http`, `pmmlTest.pmml`, and `pmmlTest.log` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running within the `local-server`.
+ +
``` ```
@@ -162,5 +162,5 @@ NOTE: `petalWidth` value changed from `1.5` to `1.8`
In this sample, you have learned: In this sample, you have learned:
* How to use Spring Cloud Data Flow's `Local` server * How to use Spring Cloud Data Flow's `Local` server
* How to use Spring Cloud Data Flow's `shell` * How to use Spring Cloud Data Flow's `shell` application
* How to use `pmml` processor to compute real-time predictions * How to use `pmml` processor to compute real-time predictions

155
functions/README.adoc Normal file
View File

@@ -0,0 +1,155 @@
:sectnums:
= Functions in Spring Cloud Data Flow
In this sample, you will learn how to use https://github.com/spring-cloud/spring-cloud-function[Spring Cloud Function] based streaming applications in Spring Cloud Data Flow. To learn more about Spring Cloud Function, check out the http://cloud.spring.io/spring-cloud-function/[project page].
== Prerequisites
Make sure that you have the following components:
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
* Local build of link:https://github.com/spring-cloud/spring-cloud-function[Spring Cloud Function]
* Running instance of RabbitMQ
* General understanding of the out-of-the-box https://github.com/spring-cloud-stream-app-starters/function/blob/master/spring-cloud-starter-stream-app-function/README.adoc[function-runner] application
=== Running the Sample Locally
. Launch the `Local-server`
+
```
$ cd <PATH/TO/SPRING-CLOUD-DATAFLOW>
$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-<VERSION>.jar
```
+
. Connect the `shell`
+
[source,bash]
----
$ cd <PATH/TO/SPRING-CLOUD-DATAFLOW>
$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-<VERSION>.jar
____ ____ _ __
/ ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| |
\___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` |
___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| |
|____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_|
____ |_| _ __|___/ __________
| _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \
| | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \
| |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / /
|____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/
<VERSION>
Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
dataflow:>version
<VERSION>
----
+
. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the RabbitMQ-binder variant of the out-of-the-box applications
+
```
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven
```
+
. Register the out-of-the-box https://github.com/spring-cloud-stream-app-starters/function/blob/master/spring-cloud-starter-stream-app-function/README.adoc[function-runner] application (_we will use the `1.0.0.BUILD-SNAPSHOT` built by the Spring CI system_)
+
```
dataflow:>app register --name function-runner --type processor --uri http://repo.spring.io/libs-snapshot/org/springframework/cloud/stream/app/function-app-rabbit/1.0.0.BUILD-SNAPSHOT/function-app-rabbit-1.0.0.BUILD-SNAPSHOT.jar --metadata-uri http://repo.spring.io/libs-snapshot/org/springframework/cloud/stream/app/function-app-rabbit/1.0.0.BUILD-SNAPSHOT/function-app-rabbit-1.0.0.BUILD-SNAPSHOT-metadata.jar
```
+
. Create and deploy the following stream
+
```
dataflow:>stream create foo --definition "http --server.port=9001 | function-runner --function.className=com.example.functions.CharCounter --function.location=file:///<PATH/TO/SPRING-CLOUD-FUNCTION>/spring-cloud-function-samples/function-sample/target/spring-cloud-function-sample-1.0.0.BUILD-SNAPSHOT.jar | log" --deploy
```
NOTE: Replace the `<PATH/TO/SPRING-CLOUD-FUNCTION>` with the correct path.
+
NOTE: The source core of `CharCounter` function is in Spring cloud Function's https://github.com/spring-cloud/spring-cloud-function/blob/master/spring-cloud-function-samples/function-sample/src/main/java/com/example/functions/CharCounter.java[samples repo].
+
+
. Verify the stream is successfully deployed.
+
[source,bash,options="nowrap"]
----
dataflow:>stream list
╔══════╤══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╗
║Stream│ Stream Definition │ Status ║
║ Name │ │ ║
╠══════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╣
║foo │http --server.port=9001 | function-runner --function.className=com.example.functions.CharCounter │All apps ║
║ │--function.location=file:///<PATH/TO/SPRING-CLOUD-FUNCTION>/spring-cloud-function-samples/function-sample/target/spring-cloud-function-sample-1.0.0.BUILD-<SNAPSHOT class="jar"></SNAPSHOT> │have been ║
║ │| log │successfully║
║ │ │deployed ║
╚══════╧══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╝
----
+
. Notice that `foo-http`, `foo-function-runner`, and `foo-log` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as Spring Boot applications and the log locations will be printed in the Local-server console.
+
[source,bash,options="nowrap"]
----
....
....
2017-10-17 11:43:03.714 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.s.s.AppDeployerStreamDeployer : Deploying application named [log] as part of stream named [foo] with resource URI [maven://org.springframework.cloud.stream.app:log-sink-rabbit:jar:1.2.0.RELEASE]
2017-10-17 11:43:04.379 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId foo.log instance 0.
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265783715/foo.log
2017-10-17 11:43:04.380 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.s.s.AppDeployerStreamDeployer : Deploying application named [function-runner] as part of stream named [foo] with resource URI [file:/var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/deployer-resource-cache8941581850579153886/http-c73a62adae0abd7ec0dee91d891575709f02f8c9]
2017-10-17 11:43:04.384 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId foo.function-runner instance 0.
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265784380/foo.function-runner
2017-10-17 11:43:04.385 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.s.s.AppDeployerStreamDeployer : Deploying application named [http] as part of stream named [foo] with resource URI [maven://org.springframework.cloud.stream.app:http-source-rabbit:jar:1.2.0.RELEASE]
2017-10-17 11:43:04.391 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId foo.http instance 0.
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265784385/foo.http
....
....
----
+
. Post sample data pointing to the `http` endpoint: `http://localhost:9001` [`9001` is the `server.port` we specified for the `http` source in this case]
+
```
dataflow:>http post --target http://localhost:9001 --data "hello world"
> POST (text/plain) http://localhost:9001 hello world
> 202 ACCEPTED
dataflow:>http post --target http://localhost:9001 --data "hmm, yeah, it works now!"
> POST (text/plain) http://localhost:9001 hmm, yeah, it works now!
> 202 ACCEPTED
```
+
. Tail the log-sink's standard-out logs to see the character counts
+
[source,bash,options="nowrap"]
----
$ tail -f /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265783715/foo.log/stdout_0.log
....
....
....
....
2017-10-17 11:45:39.363 INFO 19193 --- [on-runner.foo-1] log-sink : 11
2017-10-17 11:46:40.997 INFO 19193 --- [on-runner.foo-1] log-sink : 24
....
....
----
== Summary
In this sample, you have learned:
* How to use Spring Cloud Data Flow's `Local` server
* How to use Spring Cloud Data Flow's `shell` application
* How to use the out-of-the-box `function-runner` application in Spring Cloud Data Flow

10
streaming/http-to-cassandra/README.adoc
View File

@@ -9,7 +9,7 @@ We will begin by discussing the steps to prep, configure and operationalize Spri
=== Prerequisites === Prerequisites
In order to get started, make sure that you have the following components: Make sure that you have the following components:
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] * Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka] * Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
@@ -68,7 +68,7 @@ dataflow:>version
+ +
``` ```
dataflow:>app import --uri http://bit.ly/stream-applications-kafka-maven dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
``` ```
+ +
@@ -88,7 +88,7 @@ NOTE: If Cassandra isn't running on default port on `localhost` or if you need u
dataflow:>stream list dataflow:>stream list
``` ```
+ +
. Notice that `cassandrastream-http` and `cassandrastream-cassandra` link:https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as Spring Boot applications within the `server` as a collocated process. . Notice that `cassandrastream-http` and `cassandrastream-cassandra` link:https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the `server` as a collocated process.
+ +
``` ```
@@ -213,7 +213,7 @@ dataflow:>version
+ +
``` ```
dataflow:>app import --uri http://bit.ly/stream-applications-rabbit-maven dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven
``` ```
+ +
@@ -232,7 +232,7 @@ Created and deployed new stream 'cassandrastream'
dataflow:>stream list dataflow:>stream list
``` ```
+ +
. Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry . Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry
+ +
``` ```

10
streaming/http-to-mysql/README.adoc
View File

@@ -9,7 +9,7 @@ We will begin by discussing the steps to prep, configure and operationalize Spri
=== Prerequisites === Prerequisites
In order to get started, make sure that you have the following components: Make sure that you have the following components:
* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] * Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka] * Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
@@ -68,7 +68,7 @@ dataflow:>version
+ +
``` ```
dataflow:>app import --uri http://bit.ly/1-0-4-GA-stream-applications-kafka-maven dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
``` ```
+ +
@@ -88,7 +88,7 @@ NOTE: If MySQL isn't running on default port on `localhost` or if you need usern
dataflow:>stream list dataflow:>stream list
``` ```
+ +
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as Spring Boot applications within the Local `server` as collocated processes. . Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the Local `server` as collocated processes.
+ +
``` ```
@@ -209,7 +209,7 @@ dataflow:>version
+ +
``` ```
dataflow:>app import --uri http://bit.ly/1-0-4-GA-stream-applications-rabbit-maven dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven
``` ```
+ +
@@ -234,7 +234,7 @@ NOTE: By supplying `mysql` property through `app.jdbc.spring.cloud.deployer.clou
dataflow:>stream list dataflow:>stream list
``` ```
+ +
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry . Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry
+ +
``` ```