SpringCloud/spring-cloud-dataflow-samples
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Files
bbf1cddcdebc762a2636ed1d5fffe26090a4519f
spring-cloud-dataflow-samples/streaming/http-to-mysql/README.adoc
Sabby Anandan cb052f9b29 Update stream definition to deploy with mysql 
- This commit eliminates the necessity to supply DB credentials in the stream definition. We could have it automatically bind/negotiate (_with mysql_) through the deployment manifest. 
- Changes to bit.ly links included
2016-11-11 14:03:23 -08:00

298 lines
11 KiB
Plaintext
Raw Blame History

:sectnums:
= HTTP to MySQL Demo
In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to MySQL database using `jdbc` sink.
We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `server` Spring Boot application. We will deploy the `server` using https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-server-local[Local] as well as https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry] SPIs (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _dev-sandbox_ and _cloud-native_ platform capabilities, respectively.
== Using Local Server
=== Prerequisites
In order to get started, make sure that you have the following components:
* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
* Running instance of link:http://www.mysql.com/[MySQL]
* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] or link:https://www.dbvis.com/[DbVisualizer]
* Create the `test` database with a `names` table (in MySQL) using:
+
```
CREATE DATABASE test;
USE test;
CREATE TABLE names
(
name varchar(255)
);
```
=== Running the Sample Locally
. Launch the locally built `server`
+
```
$ cd <PATH/TO/SPRING-CLOUD-DATAFLOW>
$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-<VERSION>.jar
```
+
. Connect to Spring Cloud Data Flow's `shell`
+
```
$ cd <PATH/TO/SPRING-CLOUD-DATAFLOW>
$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-<VERSION>.jar
____ ____ _ __
/ ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| |
\___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` |
___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| |
|____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_|
____ |_| _ __|___/ __________
| _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \
| | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \
| |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / /
|____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/
<VERSION>
Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
dataflow:>version
<VERSION>
```
+
. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] Kafka binder variant of out-of-the-box applications
+
```
dataflow:>app import --uri http://bit.ly/1-0-4-GA-stream-applications-kafka-maven
```
+
. Create the stream
+
```
dataflow:>stream create --name mysqlstream --definition "http --server.port=8787 | jdbc --tableName=names --columns=name --spring.datasource.driver-class-name=org.mariadb.jdbc.Driver --spring.datasource.url='jdbc:mysql://localhost:3306/test'" --deploy
Created and deployed new stream 'mysqlstream'
```
NOTE: If MySQL isn't running on default port on `localhost` or if you need username and password to connect, use one of the following options to specify the necessary connection parameters: `--spring.datasource.url='jdbc:mysql://<HOST>:<PORT>/<NAME>' --spring.datasource.username=<USERNAME> --spring.datasource.password=<PASSWORD>`
+
. Verify the stream is successfully deployed
+
```
dataflow:>stream list
```
+
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as Spring Boot applications within the Local `server` as collocated processes.
+
```
2016-05-03 09:29:55.918 INFO 65162 --- [nio-9393-exec-3] o.s.c.d.spi.local.LocalAppDeployer : deploying app mysqlstream.jdbc instance 0
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-dataflow-6850863945840320040/mysqlstream1-1462292995903/mysqlstream.jdbc
2016-05-03 09:29:55.939 INFO 65162 --- [nio-9393-exec-3] o.s.c.d.spi.local.LocalAppDeployer : deploying app mysqlstream.http instance 0
Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-dataflow-6850863945840320040/mysqlstream-1462292995934/mysqlstream.http
```
. Post sample data pointing to the `http` endpoint: `http://localhost:8787` [`8787` is the `server.port` we specified for the `http` source in this case]
+
```
dataflow:>http post --contentType 'application/json' --target http://localhost:8787 --data "{\"name\": \"Foo\"}"
> POST (application/json;charset=UTF-8) http://localhost:8787 {"name": "Spring Boot"}
> 202 ACCEPTED
```
+
. Connect to the MySQL instance and query the table `test.names` to list the new rows:
+
```
select * from test.names;
```
+
. That's it; you're done!
== Using Cloud Foundry Server
=== Prerequisites
In order to get started, make sure that you have the following components:
* Cloud Foundry instance
* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow]
* Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry Server]
* Running instance of `rabbit` in Cloud Foundry
* Running instance of `mysql` in Cloud Foundry
* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] or link:https://www.dbvis.com/[DbVisualizer]
* Create the `names` table (in MySQL) using:
+
```
CREATE TABLE names
(
name varchar(255)
);
```
=== Running the Sample in Cloud Foundry
. Verify that CF instance is reachable
+
```
$ cf api
API endpoint: https://api.system.io (API version: 2.43.0)
$ cf apps
Getting apps in org user-dataflow / space development as user...
OK
No apps found
```
+
. Follow the instructions to deploy Spring Cloud Data Flow's `server` from https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry/blob/master/README.adoc[Cloud Foundry Server] repo
+
. Once you complete step#3 from https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry/blob/master/README.adoc[Cloud Foundry Server] instructions, you'll be able to list the newly deployed `dataflow-server` application in Cloud Foundry
+
```
$ cf apps
Getting apps in org user-dataflow / space development as user...
OK
name requested state instances memory disk urls
dataflow-server started 1/1 1G 1G dataflow-server.app.io
```
+
. Notice that `dataflow-server` application is started and ready for interaction via `http://dataflow-server.app.io` endpoint
. Connect to Spring Cloud Data Flow's `shell`
+
```
$ cd <PATH/TO/SPRING-CLOUD-DATAFLOW>
$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-<VERSION>.jar
____ ____ _ __
/ ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| |
\___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` |
___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| |
|____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_|
____ |_| _ __|___/ __________
| _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \
| | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \
| |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / /
|____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/
<VERSION>
Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
server-unknown:>
```
+
. Connect the `shell` with `server` running at `http://dataflow-server.app.io`
+
```
server-unknown:>dataflow config server http://dataflow-server.app.io
Successfully targeted http://dataflow-server.app.io
dataflow:>version
<VERSION>
```
+
. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] RabbitMQ binder variant of out-of-the-box applications
+
```
dataflow:>app import --uri http://bit.ly/1-0-4-GA-stream-applications-rabbit-maven
```
+
. Create the stream
+
```
dataflow:>stream create --name mysqlstream --definition "http | jdbc --tableName=names --columns=name"
Created new stream 'mysqlstream'
dataflow:>stream deploy --name mysqlstream --properties "app.jdbc.spring.cloud.deployer.cloudfoundry.services=mysql"
Deployed stream 'mysqlstream'
```
+
NOTE: By supplying `mysql` property through `app.jdbc.spring.cloud.deployer.cloudfoundry.services` token, we are deploying the stream with `jdbc-sink` to automatically bind to `mysql` service and only this application in the stream gets the service binding. This also eliminates the requirement to supply `datasource` credentials in stream definition.
+
. Verify the stream is successfully deployed
+
```
dataflow:>stream list
```
+
. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry
+
```
$ cf apps
Getting apps in org user-dataflow / space development as user...
OK
name requested state instances memory disk urls
mysqlstream-http started 1/1 1G 1G mysqlstream-http.app.io
mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.io
dataflow-server started 1/1 1G 1G dataflow-server.app.io
```
+
. Lookup the `url` for `mysqlstream-http` application from the list above. Post sample data pointing to the `http` endpoint: `<YOUR-mysqlstream-http-APP-URL>`
+
```
http post --contentType 'application/json' --target http://mysqlstream-http.app.io --data "{\"name\": \"Bar"}"
> POST (application/json;charset=UTF-8) http://mysqlstream-http.app.io {"name": "Bar"}
> 202 ACCEPTED
```
+
. Connect to the MySQL instance and query the table `names` to list the new rows:
+
```
select * from names;
```
+
. Now, let's take advantage of Pivotal Cloud Foundry's platform capability. Let's scale the `mysqlstream-http` application from 1 to 3 instances
+
```
$ cf scale mysqlstream-http -i 3
Scaling app mysqlstream-http in org user-dataflow / space development as user...
OK
```
+
. Verify App instances (3/3) running successfully
+
```
$ cf apps
Getting apps in org user-dataflow / space development as user...
OK
name requested state instances memory disk urls
mysqlstream-http started 3/3 1G 1G mysqlstream-http.app.io
mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.io
dataflow-server started 1/1 1G 1G dataflow-server.app.io
```
+
. That's it; you're done!
:!sectnums:
== Summary
In this sample, you have learned:
* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers
* How to use Spring Cloud Data Flow's `shell`
* How to create streaming data pipeline to connect and write to `MySQL`
* How to scale data microservice applications on `Pivotal Cloud Foundry`
Reference in New Issue View Git Blame Copy Permalink