Add scfn + scdf integration sample
- add a new folder for scfn + scdf integration - include quick-link from the root README - replace spring-cloud-stream-modules with spring-cloud-stream-app-starters - adjust text content for better continuity - replace older bit.ly's with bacon release-train Add function-runner repo link for clarity
This commit is contained in:
Sabby Anandan
committed by
Soby Chacko
Soby Chacko
parent
84d872204b
commit
5d54cb280d
@@ -18,6 +18,12 @@ A data pipeline demonstration that consumes data from an `http` endpoint and wri
|
||||
|
||||
A simple Spring Batch job running as short-lived task in Cloud Foundry.
|
||||
|
||||
## Functions
|
||||
|
||||
### link:functions/README.adoc[functions-in-scdf]
|
||||
|
||||
A simple demonstration of Spring Cloud Function and Spring Cloud Data Flow integration.
|
||||
|
||||
## Analytics
|
||||
|
||||
### link:analytics/twitter-analytics/README.adoc[twitter-analytics]
|
||||
@@ -28,4 +34,4 @@ A data pipeline demonstration that consumes data from twitter-firehose using `tw
|
||||
|
||||
### link:datascience/species-prediction/README.adoc[species-prediction]
|
||||
|
||||
A simple demonstration to walkthrough the steps to compute real-time predictions using https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] data microservice application.
|
||||
A simple demonstration to walkthrough the steps to compute real-time predictions using https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] application.
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
:sectnums:
|
||||
= Twitter Analytics
|
||||
|
||||
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_ and compute simple analytics over data-in-transit using _Field-Value-Counter_.
|
||||
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_ and compute simple analytics over data-in-transit using _Field-Value-Counter_.
|
||||
|
||||
We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `Local` server, a Spring Boot application.
|
||||
We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `Local` server.
|
||||
|
||||
== Using Local Server
|
||||
|
||||
=== Prerequisites
|
||||
|
||||
In order to get started, make sure that you have the following components:
|
||||
Make sure that you have the following components:
|
||||
|
||||
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
|
||||
* Running instance of link:http://redis.io/[Redis]
|
||||
@@ -56,7 +56,7 @@ dataflow:>version
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
|
||||
```
|
||||
|
||||
+
|
||||
@@ -75,7 +75,7 @@ Created and deployed new stream 'tagcount'
|
||||
(4) dataflow:>stream deploy tweets
|
||||
Deployed stream 'tweets'
|
||||
```
|
||||
NOTE: To get a consumerKey and consumerSecret you need to register a twitter application. If you don’t already have one set up, you can create an app at the link:https://apps.twitter.com/[Twitter Developers] site to get these credentials. The tokens `<CONSUMER_KEY>`, `<CONSUMER_SECRET>`, `<CONSUMER_SECRET>`, and `<ACCESS_TOKEN_SECRET>` are required to be replaced with your account credentials.
|
||||
NOTE: To get a consumerKey and consumerSecret you need to register a twitter application. If you don’t already have one set up, you can create an app at the link:https://apps.twitter.com/[Twitter Developers] site to get these credentials. The tokens `<CONSUMER_KEY>`, `<CONSUMER_SECRET>`, `<CONSUMER_SECRET>`, and `<ACCESS_TOKEN_SECRET>` are required to be replaced with your account credentials.
|
||||
|
||||
+
|
||||
. Verify the streams are successfully deployed. Where: (1) is the primary pipeline; (2) and (3) are tapping the primary pipeline with the DSL syntax `<stream-name>.<label/app name>` [e.x. `:tweets.twitterstream`]; and (4) is the final deployment of primary pipeline
|
||||
@@ -85,7 +85,7 @@ NOTE: To get a consumerKey and consumerSecret you need to register a twitter app
|
||||
dataflow:>stream list
|
||||
```
|
||||
+
|
||||
. Notice that `tweetlang.field-value-counter`, `tagcount.field-value-counter`, `tweets.log` and `tweets.twitterstream` link:https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] applications are running as Spring Boot applications within the `local-server`.
|
||||
. Notice that `tweetlang.field-value-counter`, `tagcount.field-value-counter`, `tweets.log` and `tweets.twitterstream` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as Spring Boot applications within the `local-server`.
|
||||
+
|
||||
|
||||
```
|
||||
@@ -111,7 +111,7 @@ dataflow:>field-value-counter list
|
||||
╚════════════════════════╝
|
||||
```
|
||||
+
|
||||
. Verify you can query individual `field-value-counter` results successfully
|
||||
. Verify you can query individual `field-value-counter` results successfully
|
||||
+
|
||||
```
|
||||
dataflow:>field-value-counter display hashtags
|
||||
@@ -151,19 +151,19 @@ Displaying values for field value counter 'language'
|
||||
|
||||
- For real-time updates on `language` tags, select:
|
||||
.. Metric Type as `Field-Value-Counters`
|
||||
.. Stream as `language`
|
||||
.. Stream as `language`
|
||||
.. Visualization as `Bubble-Chart` or `Pie-Chart`
|
||||
- For real-time updates on `hashtags` tags, select:
|
||||
.. Metric Type as `Field-Value-Counters`
|
||||
.. Stream as `hashtags`
|
||||
.. Stream as `hashtags`
|
||||
.. Visualization as `Bubble-Chart` or `Pie-Chart`
|
||||
|
||||
image::images/twitter_analytics.png[Twitter Analytics Visualization]
|
||||
|
||||
== Summary
|
||||
== Summary
|
||||
|
||||
In this sample, you have learned:
|
||||
|
||||
* How to use Spring Cloud Data Flow's `Local` server
|
||||
* How to use Spring Cloud Data Flow's `shell`
|
||||
* How to create streaming data pipeline to compute simple analytics using `Twitter Stream` and `Field Value Counter` data microservices
|
||||
* How to use Spring Cloud Data Flow's `shell` application
|
||||
* How to create streaming data pipeline to compute simple analytics using `Twitter Stream` and `Field Value Counter` applications
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
:sectnums:
|
||||
= Species Prediction
|
||||
|
||||
In this demonstration, you will learn how to use https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] model in the context of streaming data pipeline orchestrated by http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow].
|
||||
In this demonstration, you will learn how to use https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] model in the context of streaming data pipeline orchestrated by http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow].
|
||||
|
||||
We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `Local` server, a Spring Boot application.
|
||||
|
||||
@@ -9,7 +9,7 @@ We will begin by discussing the steps to prep, configure and operationalize Spri
|
||||
|
||||
=== Prerequisites
|
||||
|
||||
In order to get started, make sure that you have the following components:
|
||||
Make sure that you have the following components:
|
||||
|
||||
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
|
||||
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
|
||||
@@ -54,7 +54,7 @@ dataflow:>version
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app import --uri http://bit.ly/stream-applications-kafka-maven
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
|
||||
```
|
||||
|
||||
+
|
||||
@@ -66,7 +66,7 @@ dataflow:>stream create --name pmmlTest --definition "http --server.port=9001 |
|
||||
Created and deployed new stream 'pmmlTest'
|
||||
|
||||
```
|
||||
NOTE: The built-in `pmml` processor will load the given PMML model definition and create an internal object representation that can be evaluated quickly. When the stream receives the data, it will be used as the input for the evaluation of the analytical model `iris-flower-classifier-1` contained in the PMML document. The result of this evaluation is a new field `predictedSpecies` that was created from the `pmml` processor by applying a classifier that uses the naiveBayes algorithm.
|
||||
NOTE: The built-in `pmml` processor will load the given PMML model definition and create an internal object representation that can be evaluated quickly. When the stream receives the data, it will be used as the input for the evaluation of the analytical model `iris-flower-classifier-1` contained in the PMML document. The result of this evaluation is a new field `predictedSpecies` that was created from the `pmml` processor by applying a classifier that uses the naiveBayes algorithm.
|
||||
|
||||
+
|
||||
. Verify the stream is successfully deployed
|
||||
@@ -76,7 +76,7 @@ NOTE: The built-in `pmml` processor will load the given PMML model definition an
|
||||
dataflow:>stream list
|
||||
```
|
||||
+
|
||||
. Notice that `pmmlTest.http`, `pmmlTest.pmml`, and `pmmlTest.log` link:https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] applications are running within the `local-server`.
|
||||
. Notice that `pmmlTest.http`, `pmmlTest.pmml`, and `pmmlTest.log` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running within the `local-server`.
|
||||
+
|
||||
|
||||
```
|
||||
@@ -96,7 +96,7 @@ dataflow:>http post --target http://localhost:9001 --contentType application/jso
|
||||
> 202 ACCEPTED
|
||||
```
|
||||
+
|
||||
. Verify the predicted outcome by tailing `<PATH/TO/LOGAPP/pmmlTest.log/stdout_0.log` file. The `predictedSpecies` in this case is `versicolor`.
|
||||
. Verify the predicted outcome by tailing `<PATH/TO/LOGAPP/pmmlTest.log/stdout_0.log` file. The `predictedSpecies` in this case is `versicolor`.
|
||||
+
|
||||
```
|
||||
{
|
||||
@@ -121,7 +121,7 @@ dataflow:>http post --target http://localhost:9001 --contentType application/jso
|
||||
```
|
||||
|
||||
+
|
||||
. Let's post with a slight variation in data.
|
||||
. Let's post with a slight variation in data.
|
||||
+
|
||||
|
||||
```
|
||||
@@ -157,10 +157,10 @@ NOTE: `petalWidth` value changed from `1.5` to `1.8`
|
||||
}
|
||||
```
|
||||
|
||||
== Summary
|
||||
== Summary
|
||||
|
||||
In this sample, you have learned:
|
||||
|
||||
* How to use Spring Cloud Data Flow's `Local` server
|
||||
* How to use Spring Cloud Data Flow's `shell`
|
||||
* How to use Spring Cloud Data Flow's `shell` application
|
||||
* How to use `pmml` processor to compute real-time predictions
|
||||
|
||||
155
functions/README.adoc
Normal file
155
functions/README.adoc
Normal file
@@ -0,0 +1,155 @@
|
||||
:sectnums:
|
||||
= Functions in Spring Cloud Data Flow
|
||||
|
||||
In this sample, you will learn how to use https://github.com/spring-cloud/spring-cloud-function[Spring Cloud Function] based streaming applications in Spring Cloud Data Flow. To learn more about Spring Cloud Function, check out the http://cloud.spring.io/spring-cloud-function/[project page].
|
||||
|
||||
== Prerequisites
|
||||
|
||||
Make sure that you have the following components:
|
||||
|
||||
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
|
||||
* Local build of link:https://github.com/spring-cloud/spring-cloud-function[Spring Cloud Function]
|
||||
* Running instance of RabbitMQ
|
||||
* General understanding of the out-of-the-box https://github.com/spring-cloud-stream-app-starters/function/blob/master/spring-cloud-starter-stream-app-function/README.adoc[function-runner] application
|
||||
|
||||
=== Running the Sample Locally
|
||||
|
||||
. Launch the `Local-server`
|
||||
+
|
||||
```
|
||||
$ cd <PATH/TO/SPRING-CLOUD-DATAFLOW>
|
||||
$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-<VERSION>.jar
|
||||
|
||||
```
|
||||
+
|
||||
|
||||
. Connect the `shell`
|
||||
+
|
||||
|
||||
[source,bash]
|
||||
----
|
||||
$ cd <PATH/TO/SPRING-CLOUD-DATAFLOW>
|
||||
$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-<VERSION>.jar
|
||||
|
||||
____ ____ _ __
|
||||
/ ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| |
|
||||
\___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` |
|
||||
___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| |
|
||||
|____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_|
|
||||
____ |_| _ __|___/ __________
|
||||
| _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \
|
||||
| | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \
|
||||
| |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / /
|
||||
|____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/
|
||||
|
||||
<VERSION>
|
||||
|
||||
Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
|
||||
dataflow:>version
|
||||
<VERSION>
|
||||
----
|
||||
|
||||
+
|
||||
. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the RabbitMQ-binder variant of the out-of-the-box applications
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven
|
||||
```
|
||||
|
||||
+
|
||||
. Register the out-of-the-box https://github.com/spring-cloud-stream-app-starters/function/blob/master/spring-cloud-starter-stream-app-function/README.adoc[function-runner] application (_we will use the `1.0.0.BUILD-SNAPSHOT` built by the Spring CI system_)
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app register --name function-runner --type processor --uri http://repo.spring.io/libs-snapshot/org/springframework/cloud/stream/app/function-app-rabbit/1.0.0.BUILD-SNAPSHOT/function-app-rabbit-1.0.0.BUILD-SNAPSHOT.jar --metadata-uri http://repo.spring.io/libs-snapshot/org/springframework/cloud/stream/app/function-app-rabbit/1.0.0.BUILD-SNAPSHOT/function-app-rabbit-1.0.0.BUILD-SNAPSHOT-metadata.jar
|
||||
```
|
||||
|
||||
+
|
||||
. Create and deploy the following stream
|
||||
+
|
||||
```
|
||||
dataflow:>stream create foo --definition "http --server.port=9001 | function-runner --function.className=com.example.functions.CharCounter --function.location=file:///<PATH/TO/SPRING-CLOUD-FUNCTION>/spring-cloud-function-samples/function-sample/target/spring-cloud-function-sample-1.0.0.BUILD-SNAPSHOT.jar | log" --deploy
|
||||
|
||||
```
|
||||
NOTE: Replace the `<PATH/TO/SPRING-CLOUD-FUNCTION>` with the correct path.
|
||||
|
||||
+
|
||||
NOTE: The source core of `CharCounter` function is in Spring cloud Function's https://github.com/spring-cloud/spring-cloud-function/blob/master/spring-cloud-function-samples/function-sample/src/main/java/com/example/functions/CharCounter.java[samples repo].
|
||||
+
|
||||
|
||||
+
|
||||
. Verify the stream is successfully deployed.
|
||||
+
|
||||
[source,bash,options="nowrap"]
|
||||
----
|
||||
dataflow:>stream list
|
||||
╔══════╤══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╗
|
||||
║Stream│ Stream Definition │ Status ║
|
||||
║ Name │ │ ║
|
||||
╠══════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╣
|
||||
║foo │http --server.port=9001 | function-runner --function.className=com.example.functions.CharCounter │All apps ║
|
||||
║ │--function.location=file:///<PATH/TO/SPRING-CLOUD-FUNCTION>/spring-cloud-function-samples/function-sample/target/spring-cloud-function-sample-1.0.0.BUILD-<SNAPSHOT class="jar"></SNAPSHOT> │have been ║
|
||||
║ │| log │successfully║
|
||||
║ │ │deployed ║
|
||||
╚══════╧══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╝
|
||||
----
|
||||
|
||||
+
|
||||
. Notice that `foo-http`, `foo-function-runner`, and `foo-log` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as Spring Boot applications and the log locations will be printed in the Local-server console.
|
||||
+
|
||||
[source,bash,options="nowrap"]
|
||||
----
|
||||
....
|
||||
....
|
||||
2017-10-17 11:43:03.714 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.s.s.AppDeployerStreamDeployer : Deploying application named [log] as part of stream named [foo] with resource URI [maven://org.springframework.cloud.stream.app:log-sink-rabbit:jar:1.2.0.RELEASE]
|
||||
2017-10-17 11:43:04.379 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId foo.log instance 0.
|
||||
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265783715/foo.log
|
||||
2017-10-17 11:43:04.380 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.s.s.AppDeployerStreamDeployer : Deploying application named [function-runner] as part of stream named [foo] with resource URI [file:/var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/deployer-resource-cache8941581850579153886/http-c73a62adae0abd7ec0dee91d891575709f02f8c9]
|
||||
2017-10-17 11:43:04.384 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId foo.function-runner instance 0.
|
||||
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265784380/foo.function-runner
|
||||
2017-10-17 11:43:04.385 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.s.s.AppDeployerStreamDeployer : Deploying application named [http] as part of stream named [foo] with resource URI [maven://org.springframework.cloud.stream.app:http-source-rabbit:jar:1.2.0.RELEASE]
|
||||
2017-10-17 11:43:04.391 INFO 18409 --- [nio-9393-exec-7] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId foo.http instance 0.
|
||||
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265784385/foo.http
|
||||
....
|
||||
....
|
||||
----
|
||||
|
||||
+
|
||||
. Post sample data pointing to the `http` endpoint: `http://localhost:9001` [`9001` is the `server.port` we specified for the `http` source in this case]
|
||||
+
|
||||
```
|
||||
dataflow:>http post --target http://localhost:9001 --data "hello world"
|
||||
> POST (text/plain) http://localhost:9001 hello world
|
||||
> 202 ACCEPTED
|
||||
|
||||
|
||||
dataflow:>http post --target http://localhost:9001 --data "hmm, yeah, it works now!"
|
||||
> POST (text/plain) http://localhost:9001 hmm, yeah, it works now!
|
||||
> 202 ACCEPTED
|
||||
```
|
||||
|
||||
+
|
||||
. Tail the log-sink's standard-out logs to see the character counts
|
||||
+
|
||||
[source,bash,options="nowrap"]
|
||||
----
|
||||
$ tail -f /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow-6549025456609489200/foo-1508265783715/foo.log/stdout_0.log
|
||||
|
||||
....
|
||||
....
|
||||
....
|
||||
....
|
||||
2017-10-17 11:45:39.363 INFO 19193 --- [on-runner.foo-1] log-sink : 11
|
||||
2017-10-17 11:46:40.997 INFO 19193 --- [on-runner.foo-1] log-sink : 24
|
||||
....
|
||||
....
|
||||
----
|
||||
|
||||
== Summary
|
||||
|
||||
In this sample, you have learned:
|
||||
|
||||
* How to use Spring Cloud Data Flow's `Local` server
|
||||
* How to use Spring Cloud Data Flow's `shell` application
|
||||
* How to use the out-of-the-box `function-runner` application in Spring Cloud Data Flow
|
||||
@@ -1,7 +1,7 @@
|
||||
:sectnums:
|
||||
= HTTP to Cassandra Demo
|
||||
|
||||
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an _HTTP_ endpoint and write the payload to a _Cassandra_ database.
|
||||
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an _HTTP_ endpoint and write the payload to a _Cassandra_ database.
|
||||
|
||||
We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `server` Spring Boot application. We will deploy the `server` using https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-server-local[Local] as well as https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry] SPIs (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _dev-sandbox_ and _cloud-native_ platform capabilities respectively.
|
||||
|
||||
@@ -9,12 +9,12 @@ We will begin by discussing the steps to prep, configure and operationalize Spri
|
||||
|
||||
=== Prerequisites
|
||||
|
||||
In order to get started, make sure that you have the following components:
|
||||
Make sure that you have the following components:
|
||||
|
||||
* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
|
||||
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
|
||||
* Running instance of link:http://cassandra.apache.org/[Apache Cassandra]
|
||||
* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using.
|
||||
* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using.
|
||||
* Create a keyspace and a `book` table in Cassandra using:
|
||||
+
|
||||
```
|
||||
@@ -68,7 +68,7 @@ dataflow:>version
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app import --uri http://bit.ly/stream-applications-kafka-maven
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
|
||||
```
|
||||
|
||||
+
|
||||
@@ -88,7 +88,7 @@ NOTE: If Cassandra isn't running on default port on `localhost` or if you need u
|
||||
dataflow:>stream list
|
||||
```
|
||||
+
|
||||
. Notice that `cassandrastream-http` and `cassandrastream-cassandra` link:https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as Spring Boot applications within the `server` as a collocated process.
|
||||
. Notice that `cassandrastream-http` and `cassandrastream-cassandra` link:https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the `server` as a collocated process.
|
||||
+
|
||||
|
||||
```
|
||||
@@ -127,7 +127,7 @@ In order to get started, make sure that you have the following components:
|
||||
* Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry Server]
|
||||
* Running instance of `rabbit` in Cloud Foundry
|
||||
* Running instance of `cassandra` in Cloud Foundry or from another Cloud provider
|
||||
* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using.
|
||||
* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using.
|
||||
* Create a `book` table in your Cassandra keyspace using:
|
||||
+
|
||||
```
|
||||
@@ -174,7 +174,7 @@ dataflow-server started 1/1 1G 1G dataflow-server.a
|
||||
+
|
||||
. Notice that `dataflow-server` application is started and ready for interaction via `http://dataflow-server.app.io` endpoint
|
||||
|
||||
. Connect to Spring Cloud Data Flow's `shell`.
|
||||
. Connect to Spring Cloud Data Flow's `shell`.
|
||||
+
|
||||
|
||||
```
|
||||
@@ -213,7 +213,7 @@ dataflow:>version
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app import --uri http://bit.ly/stream-applications-rabbit-maven
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven
|
||||
```
|
||||
|
||||
+
|
||||
@@ -232,7 +232,7 @@ Created and deployed new stream 'cassandrastream'
|
||||
dataflow:>stream list
|
||||
```
|
||||
+
|
||||
. Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry
|
||||
. Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry
|
||||
+
|
||||
|
||||
```
|
||||
@@ -285,7 +285,7 @@ dataflow-server started 1/1 1G 1G datafl
|
||||
. That's it; you're done!
|
||||
|
||||
:!sectnums:
|
||||
== Summary
|
||||
== Summary
|
||||
|
||||
In this sample, you have learned:
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
:sectnums:
|
||||
= HTTP to MySQL Demo
|
||||
|
||||
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to MySQL database using `jdbc` sink.
|
||||
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to MySQL database using `jdbc` sink.
|
||||
|
||||
We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `server` Spring Boot application. We will deploy the `server` using https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-server-local[Local] as well as https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry] SPIs (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _dev-sandbox_ and _cloud-native_ platform capabilities, respectively.
|
||||
|
||||
@@ -9,7 +9,7 @@ We will begin by discussing the steps to prep, configure and operationalize Spri
|
||||
|
||||
=== Prerequisites
|
||||
|
||||
In order to get started, make sure that you have the following components:
|
||||
Make sure that you have the following components:
|
||||
|
||||
* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
|
||||
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
|
||||
@@ -28,7 +28,7 @@ CREATE TABLE names
|
||||
|
||||
=== Running the Sample Locally
|
||||
|
||||
. Launch the locally built `server`
|
||||
. Launch the locally built `server`
|
||||
+
|
||||
|
||||
```
|
||||
@@ -68,7 +68,7 @@ dataflow:>version
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app import --uri http://bit.ly/1-0-4-GA-stream-applications-kafka-maven
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven
|
||||
```
|
||||
|
||||
+
|
||||
@@ -88,7 +88,7 @@ NOTE: If MySQL isn't running on default port on `localhost` or if you need usern
|
||||
dataflow:>stream list
|
||||
```
|
||||
+
|
||||
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as Spring Boot applications within the Local `server` as collocated processes.
|
||||
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the Local `server` as collocated processes.
|
||||
+
|
||||
|
||||
```
|
||||
@@ -170,7 +170,7 @@ dataflow-server started 1/1 1G 1G dataflow-serv
|
||||
+
|
||||
. Notice that `dataflow-server` application is started and ready for interaction via `http://dataflow-server.app.io` endpoint
|
||||
|
||||
. Connect to Spring Cloud Data Flow's `shell`
|
||||
. Connect to Spring Cloud Data Flow's `shell`
|
||||
+
|
||||
|
||||
```
|
||||
@@ -209,7 +209,7 @@ dataflow:>version
|
||||
+
|
||||
|
||||
```
|
||||
dataflow:>app import --uri http://bit.ly/1-0-4-GA-stream-applications-rabbit-maven
|
||||
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven
|
||||
```
|
||||
|
||||
+
|
||||
@@ -234,7 +234,7 @@ NOTE: By supplying `mysql` property through `app.jdbc.spring.cloud.deployer.clou
|
||||
dataflow:>stream list
|
||||
```
|
||||
+
|
||||
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry
|
||||
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry
|
||||
+
|
||||
|
||||
```
|
||||
@@ -287,7 +287,7 @@ dataflow-server started 1/1 1G 1G datafl
|
||||
. That's it; you're done!
|
||||
|
||||
:!sectnums:
|
||||
== Summary
|
||||
== Summary
|
||||
|
||||
In this sample, you have learned:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user