diff --git a/README.adoc b/README.adoc index 9acb3b4..046c7f3 100644 --- a/README.adoc +++ b/README.adoc @@ -1 +1,17 @@ # spring-cloud-dataflow-samples + +This repository provides sample starter applications and code for use with the Spring Cloud Data Flow project. The following samples are available: + +## Streaming + +### http-cassandra + +A data pipeline demonstration that consumes data from an `http` endpoint, `transforms` while the data is in-transit, and writes the transformed payload to `cassandra` database. + +### http-mysql + +A data pipeline demonstration that consumes data from an `http` endpoint and writes the transformed payload to MySQL database using the `jdbc` sink. + +## Task / Batch + +## Data Science diff --git a/http-to-cassandra/README.adoc b/http-to-cassandra/README.adoc new file mode 100644 index 0000000..12792eb --- /dev/null +++ b/http-to-cassandra/README.adoc @@ -0,0 +1,168 @@ +# HTTP to Cassandra Demo + +In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint, `transform` while the data is in-transit, and write the transformed payload to `cassandra` database. + +We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `admin` Spring Boot application. We will deploy `admin` using the https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud Foundry] SPI (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _cloud-native_ platform capabilities natively. + +# Prerequisites for Pivotal Cloud Foundry + +In order to get started, make sure that you have the following components: + +* Cloud Foundry instance +* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] +* Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud-Foundry-Admin] +* The `Book` https://github.com/trisberg/spring-cloud-demo/tree/master/book[domain object] (maven artifact) +* Redis instance +* Cassandra instance + +## Running the Sample in Cloud Foundry + +. Verify that CF instance is reachable ++ + +``` +→ cf api +API endpoint: https://api.system.navy.springapps.io (API version: 2.43.0) + +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +No apps found +``` ++ +. Follow the instructions to deploy Spring Cloud Data Flow's `admin` from https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry/blob/master/README.adoc[CF SPI] repo + ++ +. Once you complete step#3 from https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry/blob/master/README.adoc[CF SPI] instructions, you'll be able to list the newly deployed `s-c-dataflow-admin` application in Cloud Foundry ++ + +``` +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +name requested state instances memory disk urls +s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io +``` + ++ +. Notice that `s-c-dataflow-admin` application is started and ready for interaction via `http://s-c-dataflow-admin.app.navy.springapps.io` endpoint + +. Connect to Spring Cloud Data Flow's `shell`. ++ + +``` +→ cd +→ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar + + ____ ____ _ __ + / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | + \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | + ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | + |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| + ____ |_| _ __|___/ __________ + | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ + | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ + | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / + |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ + +1.0.0.BUILD-SNAPSHOT + +Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". +server-unknown:> +``` ++ +. Connect the `shell` with `admin` running at `http://s-c-dataflow-admin.app.navy.springapps.io` ++ + +``` +server-unknown:>admin config server http://s-c-dataflow-admin.app.navy.springapps.io +Successfully targeted http://s-c-dataflow-admin.app.navy.springapps.io +dataflow:>version +1.0.0.BUILD-SNAPSHOT +``` ++ +. Create the stream ++ + +``` +dataflow:>stream create cassandrastream --definition "http --spring.cloud.stream.bindings.output.contentType='application/json' | transform --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT,com.datastax.cassandra:cassandra-driver-core:2.1.5' --expression='\"new demo.domain.Book(T(com.datastax.driver.core.utils.UUIDs).timeBased(), payload.title, payload.author)\"' | cassandra --spring.cassandra.username='' --spring.cassandra.password='' --spring.cassandra.port= --spring.cassandra.contactPoints= --spring.cassandra.keyspace='' --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT' --spring.cassandra.entity-base-packages=demo.domain --spring.cassandra.schemaAction=create --queryType=INSERT" --deploy + +Created and deployed new stream 'cassandrastream' +``` ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `cassandrastream-http`, `cassandrastream-transform`, and `cassandrastream-cassandra` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry ++ + +``` +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +name requested state instances memory disk urls +cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.navy.springapps.io +cassandrastream-http started 1/1 1G 1G cassandrastream-http.app.navy.springapps.io +cassandrastream-transform started 1/1 1G 1G cassandrastream-transform.app.navy.springapps.io +s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io +``` ++ +. Lookup the `url` for `cassandrastream-http` application from the list above. Post sample data pointing to the `http` endpoint: `/messages` ++ +``` +dataflow:>script --file /spring-cloud-dataflow-samples/http-to-cassandra/data.txt + +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 1\", \"author\": \"Spring Boot\"}" --target http://cassandrastream-http.app.navy.springapps.io/messages +> POST (application/json;charset=UTF-8) http://cassandrastream-http.app.navy.springapps.io/messages {"title": "The Art of JAR Files 1", "author": "Spring Boot"} +> 202 ACCEPTED +... +... +... +``` ++ +. Use a database utility tool such as http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. Query the table `book` to list all the 10 rows ++ +``` +select * from book; +``` +image:img/cassandra_table_results.png[Table Results] + ++ +. Now, let's try to take advantage of Pivotal Cloud Foundry's platform capability. Let's scale the `cassandrastream-http` application from 1 to 3 instances ++ +``` +→ cf scale cassandrastream-http -i 3 +Scaling app cassandrastream-http in org sabby-dataflow / space development as sabby... +OK +``` ++ +. Verify App instances (3/3) running successfully ++ +``` +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +name requested state instances memory disk urls +cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.navy.springapps.io +cassandrastream-http started 3/3 1G 1G cassandrastream-http.app.navy.springapps.io +cassandrastream-transform started 1/1 1G 1G cassandrastream-transform.app.navy.springapps.io +s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io +``` ++ +. That's it; you're done! + +# Summary + +In this sample, you have learned: + +* How to use Spring Cloud Data Flow in Pivotal Cloud Foundry +* How to use Spring Cloud Data Flow's `shell` +* How to create streaming data pipeline to connect and write to Cassandra using Spring Cloud Data Flow +* How to scale data microservice applications in isolation \ No newline at end of file diff --git a/http-to-cassandra/data.txt b/http-to-cassandra/data.txt new file mode 100644 index 0000000..e7ce749 --- /dev/null +++ b/http-to-cassandra/data.txt @@ -0,0 +1,10 @@ +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 1\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 2\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 3\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 4\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 5\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 6\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 7\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 8\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 9\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages +http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files 10\", \"author\": \"Spring Boot\"}" --target http://YOUR-cassandrastream-http-APP-URL/messages \ No newline at end of file diff --git a/http-to-cassandra/img/cassandra_table_results.png b/http-to-cassandra/img/cassandra_table_results.png new file mode 100644 index 0000000..ee14efb Binary files /dev/null and b/http-to-cassandra/img/cassandra_table_results.png differ diff --git a/http-to-mysql/README.adoc b/http-to-mysql/README.adoc new file mode 100644 index 0000000..ccd5681 --- /dev/null +++ b/http-to-mysql/README.adoc @@ -0,0 +1,165 @@ +# HTTP to MySQL Demo + +In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to MySQL database through `jdbc` sink. + +We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `admin` Spring Boot application. We will deploy `admin` using the https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud Foundry] SPI (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _cloud-native_ platform capabilities natively. + +# Prerequisites for Pivotal Cloud Foundry + +In order to get started, make sure that you have the following components: + +* Cloud Foundry instance +* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] +* Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud-Foundry-Admin] +* The `Book` https://github.com/trisberg/spring-cloud-demo/tree/master/book[domain object] (maven artifact) +* Redis instance +* MySQL instance + +## Running the Sample in Cloud Foundry + +. Verify that CF instance is reachable ++ + +``` +→ cf api +API endpoint: https://api.system.navy.springapps.io (API version: 2.43.0) + +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +No apps found +``` ++ +. Follow the instructions to deploy Spring Cloud Data Flow's `admin` from https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry/blob/master/README.adoc[CF SPI] repo + ++ +. Once you complete step#3 from https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry/blob/master/README.adoc[CF SPI] instructions, you'll be able to list the newly deployed `s-c-dataflow-admin` application in Cloud Foundry ++ + +``` +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +name requested state instances memory disk urls +s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io +``` + ++ +. Notice that `s-c-dataflow-admin` application is started and ready for interaction via `http://s-c-dataflow-admin.app.navy.springapps.io` endpoint + +. Connect to Spring Cloud Data Flow's `shell`. ++ + +``` +→ cd +→ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar + + ____ ____ _ __ + / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | + \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | + ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | + |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| + ____ |_| _ __|___/ __________ + | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ + | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ + | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / + |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ + +1.0.0.BUILD-SNAPSHOT + +Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". +server-unknown:> +``` ++ +. Connect the `shell` with `admin` running at `http://s-c-dataflow-admin.app.navy.springapps.io` ++ + +``` +server-unknown:>admin config server http://s-c-dataflow-admin.app.navy.springapps.io +Successfully targeted http://s-c-dataflow-admin.app.navy.springapps.io +dataflow:>version +1.0.0.BUILD-SNAPSHOT +``` ++ +. Create the stream ++ + +``` +dataflow:>stream create --name mysqlstream --definition "http | jdbc --includes='mysql:mysql-connector-java:5.1.37' --spring.datasource.url='jdbc:mysql://:/' --spring.datasource.username= --spring.datasource.password= --tableName=names --columns=name --spring.datasource.driverClassName=com.mysql.jdbc.Driver --initialize=true" --deploy + +Created and deployed new stream 'mysqlstream' +``` ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry ++ + +``` +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +name requested state instances memory disk urls +mysqlstream-http started 1/1 1G 1G mysqlstream-http.app.navy.springapps.io +mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.navy.springapps.io +s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io +``` ++ +. Lookup the `url` for `mysqlstream-http` application from the list above. Post sample data pointing to the `http` endpoint: `/messages` ++ +``` +dataflow:>script --file /spring-cloud-dataflow-samples/http-to-mysql/data.txt +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 1\"}" +> POST (application/json;charset=UTF-8) http://mysqlstream-http.app.navy.springapps.io/messages {"name": "Spring Boot 1"} +> 202 ACCEPTED +... +... +... +``` ++ +. Use a database utility tool such as http://dbeaver.jkiss.org/[DBeaver] to connect to the MySQL instance. Query the table `book` to list all the 10 rows ++ +``` +select * from book; +``` +image:img/mysql_table_results.png[Table Results] + ++ +. Now, let's try to take advantage of Pivotal Cloud Foundry's platform capability. Let's scale the `mysqlstream-http` application from 1 to 3 instances ++ +``` +→ cf scale mysqlstream-http -i 3 +Scaling app mysqlstream-http in org sabby-dataflow / space development as sabby... +OK +``` ++ +. Verify App instances (3/3) running successfully ++ +``` +→ cf apps +Getting apps in org sabby-dataflow / space development as sabby... +OK + +name requested state instances memory disk urls +mysqlstream-http started 3/3 1G 1G mysqlstream-http.app.navy.springapps.io +mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.navy.springapps.io +s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io +``` ++ +. That's it; you're done! + +# Summary + +In this sample, you have learned: + +* How to use Spring Cloud Data Flow in Pivotal Cloud Foundry +* How to use Spring Cloud Data Flow's `shell` +* How to create streaming data pipeline to connect and write to MySQL using Spring Cloud Data Flow +* How to scale data microservice applications in isolation \ No newline at end of file diff --git a/http-to-mysql/data.txt b/http-to-mysql/data.txt new file mode 100644 index 0000000..18b79f9 --- /dev/null +++ b/http-to-mysql/data.txt @@ -0,0 +1,10 @@ +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 1\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 2\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 3\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 4\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 5\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 6\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 7\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 8\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 9\"}" +http post --contentType 'application/json' --target http://mysqlstream-http.app.navy.springapps.io/messages --data "{\"name\": \"Spring Boot 10\"}" diff --git a/http-to-mysql/img/mysql_table_results.png b/http-to-mysql/img/mysql_table_results.png new file mode 100644 index 0000000..39ea338 Binary files /dev/null and b/http-to-mysql/img/mysql_table_results.png differ