diff --git a/streaming/http-to-cassandra/README.adoc b/streaming/http-to-cassandra/README.adoc index 12792eb..2fe8ac8 100644 --- a/streaming/http-to-cassandra/README.adoc +++ b/streaming/http-to-cassandra/README.adoc @@ -1,21 +1,120 @@ # HTTP to Cassandra Demo -In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint, `transform` while the data is in-transit, and write the transformed payload to `cassandra` database. +In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write the payload to `cassandra` database. -We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `admin` Spring Boot application. We will deploy `admin` using the https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud Foundry] SPI (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _cloud-native_ platform capabilities natively. +We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `admin` Spring Boot application. We will deploy `admin` using https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-admin-local[Local] and as well as https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud Foundry] SPIs (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _dev-sandbox_ and _cloud-native_ platform capabilities respectively. -# Prerequisites for Pivotal Cloud Foundry +# 1: Prerequisites for Local SPI + +In order to get started, make sure that you have the following components: + +* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] +* Running instance of `redis` +* Running instance of `cassandra` +* Create the `book` table (in `cassandra`) using `create-table.sql` + +## 1.1: Running the Sample Locally + +. Launch the locally built `admin` application ++ + +``` +→ cd +→ java -jar spring-cloud-dataflow-admin-local/target/spring-cloud-dataflow-admin-local-1.0.0.BUILD-SNAPSHOT.jar + +``` ++ + +. Connect to Spring Cloud Data Flow's `shell` ++ + +``` +→ cd +→ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar + + ____ ____ _ __ + / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | + \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | + ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | + |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| + ____ |_| _ __|___/ __________ + | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ + | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ + | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / + |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ + +1.0.0.BUILD-SNAPSHOT + +Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". +dataflow:>version +1.0.0.BUILD-SNAPSHOT +``` + ++ +. Create the stream ++ + +``` +dataflow:>stream create cassandrastream --definition "http --spring.cloud.stream.bindings.output.contentType='application/json' | cassandra --ingestQuery='insert into book (isbn, title, author) values (uuid(), ?, ?)' --spring.cassandra.username='' --spring.cassandra.password='' --spring.cassandra.port= --spring.cassandra.contactPoints= --spring.cassandra.keyspace='' --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT' --spring.cassandra.entity-base-packages=demo.domain --queryType=INSERT" --deploy + +Created and deployed new stream 'cassandrastream' +``` ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as Spring Boot applications within the `admin` as a collocated process. ++ + +``` +2015-12-15 15:52:31.576 INFO 18337 --- [nio-9393-exec-1] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:cassandra-sink:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/cassandrastream.cassandra +2015-12-15 15:52:31.583 INFO 18337 --- [nio-9393-exec-1] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:http-source:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/cassandrastream.http +``` ++ +. Lookup the `port` for `cassandrastream-http` application from the logs [i.e: `/var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/cassandrastream.http`]. ++ + +``` +2015-12-15 15:52:42.896 INFO 18341 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 14491 (http) +2015-12-15 15:52:42.900 INFO 18341 --- [ main] o.s.c.s.m.http.HttpSourceApplication : Started HttpSourceApplication in 6.494 seconds (JVM running for 11.291) +``` ++ + +. Post sample data pointing to the `http` endpoint: `http://localhost:14491/messages` [i.e: **14491** in this case] ++ +``` +dataflow:>http post --contentType 'application/json' --data "{\"title\": \"The Art of JAR Files\", \"author\": \"Spring Boot\"}" --target http://localhost:14491/messages +> POST (application/json;charset=UTF-8) http://localhost:14491/messages {"title": "The Art of JAR Files", "author": "Spring Boot"} +> 202 ACCEPTED +``` ++ +. Use a database utility tool such as http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. Query the table `book` to list the persisted records ++ +``` +select * from book; +``` + ++ +. That's it; you're done! + +# 2: Prerequisites for Pivotal Cloud Foundry SPI In order to get started, make sure that you have the following components: * Cloud Foundry instance * Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] * Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud-Foundry-Admin] -* The `Book` https://github.com/trisberg/spring-cloud-demo/tree/master/book[domain object] (maven artifact) -* Redis instance -* Cassandra instance +* Running instance of `redis` in Cloud Foundry +* Running instance of `cassandra` +* Create the `book` table (in `cassandra`) using `create-table.sql` -## Running the Sample in Cloud Foundry + +## 2.1: Running the Sample in Cloud Foundry . Verify that CF instance is reachable + @@ -87,7 +186,7 @@ dataflow:>version + ``` -dataflow:>stream create cassandrastream --definition "http --spring.cloud.stream.bindings.output.contentType='application/json' | transform --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT,com.datastax.cassandra:cassandra-driver-core:2.1.5' --expression='\"new demo.domain.Book(T(com.datastax.driver.core.utils.UUIDs).timeBased(), payload.title, payload.author)\"' | cassandra --spring.cassandra.username='' --spring.cassandra.password='' --spring.cassandra.port= --spring.cassandra.contactPoints= --spring.cassandra.keyspace='' --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT' --spring.cassandra.entity-base-packages=demo.domain --spring.cassandra.schemaAction=create --queryType=INSERT" --deploy +dataflow:>stream create cassandrastream --definition "http --spring.cloud.stream.bindings.output.contentType='application/json' | cassandra --ingestQuery='insert into book (isbn, title, author) values (uuid(), ?, ?)' --spring.cassandra.username='' --spring.cassandra.password='' --spring.cassandra.port= --spring.cassandra.contactPoints= --spring.cassandra.keyspace='' --includes='org.springframework.cloud.stream.demo:book:1.0.0.BUILD-SNAPSHOT' --spring.cassandra.entity-base-packages=demo.domain --queryType=INSERT" --deploy Created and deployed new stream 'cassandrastream' ``` @@ -98,7 +197,7 @@ Created and deployed new stream 'cassandrastream' dataflow:>stream list ``` + -. Notice that `cassandrastream-http`, `cassandrastream-transform`, and `cassandrastream-cassandra` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry +. Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as _cloud-native_ (microservice) applications in Cloud Foundry + ``` @@ -109,7 +208,6 @@ OK name requested state instances memory disk urls cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.navy.springapps.io cassandrastream-http started 1/1 1G 1G cassandrastream-http.app.navy.springapps.io -cassandrastream-transform started 1/1 1G 1G cassandrastream-transform.app.navy.springapps.io s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io ``` + @@ -152,17 +250,16 @@ OK name requested state instances memory disk urls cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.navy.springapps.io cassandrastream-http started 3/3 1G 1G cassandrastream-http.app.navy.springapps.io -cassandrastream-transform started 1/1 1G 1G cassandrastream-transform.app.navy.springapps.io s-c-dataflow-admin started 1/1 1G 1G s-c-dataflow-admin.app.navy.springapps.io ``` + . That's it; you're done! -# Summary +# 3: Summary In this sample, you have learned: -* How to use Spring Cloud Data Flow in Pivotal Cloud Foundry +* How to use Spring Cloud Data Flow in `Local` and `Pivotal Cloud Foundry` * How to use Spring Cloud Data Flow's `shell` -* How to create streaming data pipeline to connect and write to Cassandra using Spring Cloud Data Flow -* How to scale data microservice applications in isolation \ No newline at end of file +* How to create streaming data pipeline to connect and write to `Cassandra` +* How to scale data microservice applications on `Pivotal Cloud Foundry` \ No newline at end of file diff --git a/streaming/http-to-cassandra/create-table.cql b/streaming/http-to-cassandra/create-table.cql new file mode 100644 index 0000000..a3d305b --- /dev/null +++ b/streaming/http-to-cassandra/create-table.cql @@ -0,0 +1,10 @@ +DROP TABLE IF EXISTS book; + +CREATE TABLE book ( + isbn uuid PRIMARY KEY, + author text, + instock boolean, + pages int, + saledate timestamp, + title text +); \ No newline at end of file diff --git a/streaming/http-to-mysql/README.adoc b/streaming/http-to-mysql/README.adoc index ccd5681..fb7e3f7 100644 --- a/streaming/http-to-mysql/README.adoc +++ b/streaming/http-to-mysql/README.adoc @@ -2,20 +2,119 @@ In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to MySQL database through `jdbc` sink. -We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `admin` Spring Boot application. We will deploy `admin` using the https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud Foundry] SPI (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _cloud-native_ platform capabilities natively. +We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `admin` Spring Boot application. We will deploy `admin` using https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-admin-local[Local] and as well as https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud Foundry] SPIs (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _dev-sandbox_ and _cloud-native_ platform capabilities respectively. -# Prerequisites for Pivotal Cloud Foundry +# 1: Prerequisites for Local SPI + +In order to get started, make sure that you have the following components: + +* Cloud Foundry instance +* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] +* Running instance of `redis` +* Running instance of `mysql` +* Create the `names` table (in `mysql`) using `create-table.sql` + +## 1.1: Running the Sample Locally + +. Launch the locally built `admin` application ++ + +``` +→ cd +→ java -jar spring-cloud-dataflow-admin-local/target/spring-cloud-dataflow-admin-local-1.0.0.BUILD-SNAPSHOT.jar + +``` ++ + +. Connect to Spring Cloud Data Flow's `shell` ++ + +``` +→ cd +→ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar + + ____ ____ _ __ + / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | + \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | + ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | + |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| + ____ |_| _ __|___/ __________ + | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ + | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ + | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / + |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ + +1.0.0.BUILD-SNAPSHOT + +Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". +dataflow:>version +1.0.0.BUILD-SNAPSHOT +``` ++ +. Create the stream ++ + +``` +dataflow:>stream create --name mysqlstream --definition "http | jdbc --includes='mysql:mysql-connector-java:5.1.37' --spring.datasource.url='jdbc:mysql://:/' --spring.datasource.username= --spring.datasource.password= --tableName=names --columns=name --spring.datasource.driverClassName=com.mysql.jdbc.Driver --initialize=true" --deploy + +Created and deployed new stream 'mysqlstream' +``` ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud/spring-cloud-stream-modules/[Spring Cloud Stream] modules are running as Spring Boot applications within the `admin` as a collocated process. ++ + +``` +2015-12-15 16:38:46.795 INFO 18337 --- [nio-9393-exec-6] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:jdbc-sink:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/mysqlstream.jdbc +2015-12-15 16:38:46.798 INFO 18337 --- [nio-9393-exec-6] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:http-source:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/mysqlstream.http +``` + ++ +. Lookup the `port` for `mysqlstream-http` application from the logs [i.e: `/var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/mysqlstream.http`]. ++ + +``` +2015-12-15 16:38:55.557 INFO 19089 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 46073 (http) +2015-12-15 16:38:55.559 INFO 19089 --- [ main] o.s.c.s.m.http.HttpSourceApplication : Started HttpSourceApplication in 4.558 seconds (JVM running for 8.733) +``` ++ + +. Post sample data pointing to the `http` endpoint: `http://localhost:46073/messages` [i.e: **46073** in this case] + ++ +``` +dataflow:>http post --contentType 'application/json' --target http://localhost:46073/messages --data "{\"name\": \"Foo\"}" +> POST (application/json;charset=UTF-8) http://localhost:46073/messages {"name": "Spring Boot"} +> 202 ACCEPTED +``` ++ +. Use a database utility tool such as http://dbeaver.jkiss.org/[DBeaver] to connect to the MySQL instance. Query the table `names` to list all the 10 rows ++ +``` +select * from names; +``` ++ +. That's it; you're done! + +# 2: Prerequisites for Pivotal Cloud Foundry SPI In order to get started, make sure that you have the following components: * Cloud Foundry instance * Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] * Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-admin-cloudfoundry[Cloud-Foundry-Admin] -* The `Book` https://github.com/trisberg/spring-cloud-demo/tree/master/book[domain object] (maven artifact) -* Redis instance -* MySQL instance +* Running instance of `redis` in Cloud Foundry +* Running instance of `mysql` +* Create the `names` table (in `mysql`) using `create-table.sql` -## Running the Sample in Cloud Foundry +## 2.1: Running the Sample in Cloud Foundry . Verify that CF instance is reachable + @@ -124,10 +223,10 @@ http post --contentType 'application/json' --target http://mysqlstream-http.app. ... ``` + -. Use a database utility tool such as http://dbeaver.jkiss.org/[DBeaver] to connect to the MySQL instance. Query the table `book` to list all the 10 rows +. Use a database utility tool such as http://dbeaver.jkiss.org/[DBeaver] to connect to the MySQL instance. Query the table `names` to list all the 10 rows + ``` -select * from book; +select * from names; ``` image:img/mysql_table_results.png[Table Results] @@ -155,11 +254,11 @@ s-c-dataflow-admin started 1/1 1G 1G s-c-da + . That's it; you're done! -# Summary +# 3: Summary In this sample, you have learned: -* How to use Spring Cloud Data Flow in Pivotal Cloud Foundry +* How to use Spring Cloud Data Flow in `Local` and `Pivotal Cloud Foundry` * How to use Spring Cloud Data Flow's `shell` -* How to create streaming data pipeline to connect and write to MySQL using Spring Cloud Data Flow -* How to scale data microservice applications in isolation \ No newline at end of file +* How to create streaming data pipeline to connect and write to `MySQL` +* How to scale data microservice applications on `Pivotal Cloud Foundry` \ No newline at end of file diff --git a/streaming/http-to-mysql/create-table.sql b/streaming/http-to-mysql/create-table.sql new file mode 100644 index 0000000..f4c9465 --- /dev/null +++ b/streaming/http-to-mysql/create-table.sql @@ -0,0 +1,6 @@ +DROP TABLE IF EXISTS names; + +CREATE TABLE names +( + name varchar(255) +); \ No newline at end of file