diff --git a/.gitignore b/.gitignore index e43b0f9..f2564bc 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ .DS_Store +target/ diff --git a/README.adoc b/README.adoc index ddab21a..ce233ee 100644 --- a/README.adoc +++ b/README.adoc @@ -1,37 +1,9 @@ # Spring Cloud Data Flow Samples -This repository provides sample starter applications and code for use with the Spring Cloud Data Flow project. The following samples are available: +This repository provides various developer tutorials and samples for building data pipelines with Spring Cloud Data Flow. The samples are included in a document available in HTML and PDF. -## Streaming +To build the documents -### link:streaming/http-to-cassandra/README.adoc[http-cassandra] - -A data pipeline demonstration that consumes data from an `http` endpoint and writes the payload to Cassandra database using the `cassandra` sink application. - -### link:streaming/http-to-mysql/README.adoc[http-mysql] - -A data pipeline demonstration that consumes data from an `http` endpoint and writes the payload to MySQL database using the `jdbc` sink application. - -## Task / Batch - -### link:tasks/simple-batch-job/README.adoc[simple-batch-job] - -A simple Spring Batch job running as short-lived task in Cloud Foundry. - -## Functions - -### link:functions/README.adoc[functions-in-scdf] - -A simple demonstration of Spring Cloud Function and Spring Cloud Data Flow integration. - -## Analytics - -### link:analytics/twitter-analytics/README.adoc[twitter-analytics] - -A data pipeline demonstration that consumes data from twitter-firehose using `twitterstream` source application and computes simple analytics over data-in-transit with the help of `field-value-counter` sink application. - -## Data Science - -### link:datascience/species-prediction/README.adoc[species-prediction] - -A simple demonstration to walkthrough the steps to compute real-time predictions using https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] application. +``` +$mvn package +``` diff --git a/custom-apps/celsius-converter-processor/README.adoc b/custom-apps/celsius-converter-processor/README.adoc deleted file mode 100644 index 99d3f99..0000000 --- a/custom-apps/celsius-converter-processor/README.adoc +++ /dev/null @@ -1,172 +0,0 @@ -# Custom Spring Cloud Stream Processor -======================================= - -### Creating the project - -Today we're going to talk about how to create a custom Spring Cloud Stream application and running it on Spring Cloud Data Flow. -We're going to go through all the steps for making a simple processor that will allow you to convert temperature from Fahrenheit to Celsius. -We will be running the demo locally, but all the steps will work in a Cloud Foundry environment as well. -The first step is to create a new spring cloud stream project. -We can do that by going to http://start.spring.io/, which is the spring initializer. - -Choose the group as `demo.celsius.converter` and enter artifact name as celsius-converter-processor. -Next, we need to choose a message transport binding as a dependency for the custom app. -There are options for choosing ***rabbitmq*** or ***kafka*** as the message transport. -For this demo, we choose rabbit by typing rabbit in the search bar under "Search for dependencies" and then selecting "stream rabbit". - -Hit the generate project button and then open the new project in an IDE of your choice. - -### Developing the app - -Now we're at a point where we can actually create our custom app. In our Spring Cloud Stream application, the minimum configuration is going to require two Java class files. -* CelsiusConverterProcessorAplication.java -* CelsiusConverterProcessorConfiguration.java - -Spring Cloud Stream applications are stand alone spring boot applications that can be run on their own. -Spring Cloud Data Flow is the orchestration layer that ties these individual Spring Cloud Stream apps together. - -Since we named the project as celsius-converter-processor, the initializer created a class called CelsiusConverterProcessorAplication. -We are creating a transformer that takes a Fahrenheit input and converts it to Celsius. -We want to follow the same naming convention as the application file, so create a new file in the same directory called `CelsiusConverterProcessorConfiguration.java`. - -##### CelsiusConverterProcessorConfiguration.java -``` -@EnableBinding(Processor.class) -public class CelsiusConverterProcessorConfiguration { - - @Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT) - public int convertToCelsius(String payload) { - int fahrenheitTemperature = Integer.parseInt(payload); - return (farenheitTemperature-30)/2; - } -} -``` - -There are two important spring annotations that we introduced in the above code. -First we annotated the class with **@EnableBinding(Processor.class)**. -Second we created a method and annotated it with ***@Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)***. -By adding these two annotations we are basically classifying this stream app as a Processor(as opposed to a source or a sink). -This allows us to specify that the application is receiving input from upstream(Processor.input) and send the output data downstream(Processor.OUTPUT). - -The convertToCelsius method takes a `String` as input for Fahrenheit and then returns the converted Celsius as an integer. -This method is very simple, but that is also the beauty of this programming style. -We can add as much logic as we want to this method to enrich this processor. -As long as we annotate it properly and return valid output, it works as a proper Spring Cloud Stream processor. - -Once we're done putting together the app, move into the origin of the project directory and build a deployable jar with Maven, possibly on the command line. -``` -cd -mvn clean install - -Make sure you have a running instance of RabbitMQ https://www.rabbitmq.com/ - -java -jar target/celsius-converter-processor-0.0.1-SNAPSHOT.jar -``` - -If all goes well, we should have a running standalone Spring Boot Application. -Once we verify that the app is started and running without any errors, we can stop it. - -## Deploying Locally on Spring Cloud Data Flow - -#### Prerequisites - -In order to get started, make sure that you have the following components: - -* Local build of https://github.com/spring-cloud/spring-cloud-dataflow [Spring Cloud Data Flow] -* Running instance of RabbitMQ https://www.rabbitmq.com/ - - - Launch the locally built `server` - -``` -$ cd -$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-.jar -``` - -Connect to Spring Cloud Data Flow's `shell` - -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -dataflow:>version - -``` - - -[Register](https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app) RabbitMQ binder variant of out-of-the-box applications -``` -dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven -``` - -``` -app register --type processor --name convertToCelsius --uri --force -``` - -Create the stream. -We want to create a stream that uses the out of the box apps `http` and `log` and then connect them with our custom celsius converter app. - -``` -dataflow:>stream create --name convertToCelsiusStream --definition "http --port=9090 | convertToCelsius | log" --deploy - -Created and deployed new stream 'convertToCelsiusStream' -``` - -Verify the stream is successfully deployed - -``` -dataflow:>stream list -``` - -Verify that the apps have successfully deployed - -``` -dataflow:>runtime apps -``` - -Since we are running locally, note the file location of the logs. - - -``` -2016-09-27 10:03:11.988 INFO 95234 --- [nio-9393-exec-9] o.s.c.d.spi.local.LocalAppDeployer : deploying app convertToCelsiusStream.log instance 0 - Logs will be in /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-3236898888473815319/convertToCelsiusStream-1474984991968/convertToCelsiusStream.log -2016-09-27 10:03:12.397 INFO 95234 --- [nio-9393-exec-9] o.s.c.d.spi.local.LocalAppDeployer : deploying app convertToCelsiusStream.convertToCelsius instance 0 - Logs will be in /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-3236898888473815319/convertToCelsiusStream-1474984992392/convertToCelsiusStream.convertToCelsius -2016-09-27 10:03:14.445 INFO 95234 --- [nio-9393-exec-9] o.s.c.d.spi.local.LocalAppDeployer : deploying app convertToCelsiusStream.http instance 0 - Logs will be in /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-3236898888473815319/convertToCelsiusStream-1474984994440/convertToCelsiusStream.http -``` - - Post sample data pointing to the `http` endpoint: `http://localhost:9090` [`9090` is the `server.port` we specified for the `http` source in this case] - - -``` -dataflow:>http post --target http://localhost:9090 --data 76 -> POST (text/plain;Charset=UTF-8) http://localhost:9090 76 -> 202 ACCEPTED -``` - -Open the log file for the convertToCelsiusStream.log app to see the output of our stream -``` -tail -f /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-7563139704229890655/convertToCelsiusStream-1474990317406/convertToCelsiusStream.log/stdout_0.log -``` -You should see the temperature you posted converted to Celsius! -``` -2016-09-27 10:05:34.933 INFO 95616 --- [CelsiusStream-1] log.sink : 23 -``` - -In conclusion, we created a simple Spring Cloud Stream processor that converts the incoming Fahrenheit temperature data to Celsius. -We ran it first as a standalone Spring Boot microservice application and then took the same app to Spring Cloud Data Flow to orchestrate it to be part of a stream that contains other apps. diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..a7f2eff --- /dev/null +++ b/pom.xml @@ -0,0 +1,315 @@ + + + 4.0.0 + spring-cloud-dataflow-samples + org.springframework.cloud + 1.0.0.BUILD-SNAPSHOT + + Spring Cloud Data Flow Samples + + ${basedir}/.. + 1.2.3.RELEASE + + + + + + + org.asciidoctor + asciidoctor-maven-plugin + 1.5.3 + + + org.asciidoctor + asciidoctorj-pdf + 1.5.0-alpha.11 + + + + index.adoc + book + + true + warn + ${project.version} + ${project.artifactId} + ${version-type-lowercase} + ${spring-cloud-dataflow.version} + ${dataflow-version-type-lowercase} + ${dataflow-branch-or-tag} + https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/${dataflow-branch-or-tag}/spring-cloud-dataflow-docs/src/main/asciidoc + + + + ${org.springframework.cloud:spring-cloud-dataflow-core:jar.version} + ${org.springframework.cloud:spring-cloud-task-core:jar.version} + current-SNAPSHOT + current-SNAPSHOT + current-SNAPSHOT + current-SNAPSHOT + + + + + + generate-docbook + generate-resources + + process-asciidoc + + + docbook5 + + true + + + + + generate-html5 + prepare-package + + process-asciidoc + + + html5 + + + + + + + + + com.agilejava.docbkx + docbkx-maven-plugin + 2.0.15 + + ${basedir}/target/generated-docs + + 0 + index.xml + true + false + ${basedir}/src/main/docbook/xsl/pdf.xsl + 1 + 1 + + 1 + ${basedir}/src/main/docbook/xsl/xslthl-config.xml + + + + + + + + + net.sf.xslthl + xslthl + 2.1.0 + + + net.sf.docbook + docbook-xml + 5.0-all + resources + zip + runtime + + + + + html + + generate-html + + generate-resources + + ${basedir}/src/main/docbook/xsl/html-multipage.xsl + ${basedir}/target/docbook/html + true + + + + + + + + + + + + + + + + + + + + + + + + net.radai + grep-maven-plugin + 1.0 + + + + grep + + test + + + + target/generated-docs/index.html + Unresolved directive in .*\.adoc - include:: + true + + + + + + + + org.apache.maven.plugins + maven-antrun-plugin + 1.8 + + + ant-contrib + ant-contrib + 1.0b3 + + + ant + ant + + + + + org.apache.ant + ant-nodeps + 1.8.1 + + + org.tigris.antelope + antelopetasks + 3.2.10 + + + + + package-and-attach-docs-zip + package + + run + + + + + + + + + + + + + + + + setup-maven-properties + validate + + run + + + true + + + + + + + + + + + + + + + + + + org.codehaus.mojo + build-helper-maven-plugin + 1.9.1 + + + attach-zip + + attach-artifact + + + + + + ${project.build.directory}/${project.artifactId}-${project.version}.zip + + zip;zip.type=docs;zip.deployed=false + + + + + + + + + diff --git a/src/main/asciidoc/Guardfile b/src/main/asciidoc/Guardfile new file mode 100644 index 0000000..bdd4d72 --- /dev/null +++ b/src/main/asciidoc/Guardfile @@ -0,0 +1,20 @@ +require 'asciidoctor' +require 'erb' + +guard 'shell' do + watch(/.*\.adoc$/) {|m| + Asciidoctor.render_file('index.adoc', \ + :in_place => true, \ + :safe => Asciidoctor::SafeMode::UNSAFE, \ + :attributes=> { \ + 'source-highlighter' => 'prettify', \ + 'icons' => 'font', \ + 'linkcss'=> 'true', \ + 'copycss' => 'true', \ + 'doctype' => 'book'}) + } +end + +guard 'livereload' do + watch(%r{^.+\.(css|js|html)$}) +end diff --git a/analytics/twitter-analytics/README.adoc b/src/main/asciidoc/analytics/twitter-analytics/main.adoc similarity index 70% rename from analytics/twitter-analytics/README.adoc rename to src/main/asciidoc/analytics/twitter-analytics/main.adoc index 47198a3..d08121b 100644 --- a/analytics/twitter-analytics/README.adoc +++ b/src/main/asciidoc/analytics/twitter-analytics/main.adoc @@ -1,80 +1,55 @@ +[[spring-cloud-data-flow-samples-twitter-analytics-overview]] :sectnums: -= Twitter Analytics +:docs_dir: ../.. +=== Twitter Analytics -In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_ and compute simple analytics over data-in-transit using _Field-Value-Counter_. +In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from _TwitterStream_ and compute simple analytics over data-in-transit using _Field-Value-Counter_. -We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `Local` server. +We will take you through the steps to configure Spring Cloud Data Flow's `Local` server. -== Using Local Server +==== Prerequisites -=== Prerequisites - -Make sure that you have the following components: - -* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* A running local Data Flow Server +include::{docs_dir}/local-server.adoc[] * Running instance of link:http://redis.io/[Redis] * Running instance of link:http://kafka.apache.org/downloads.html[Kafka] * Twitter credentials from link:https://apps.twitter.com/[Twitter Developers] site -=== Running the Sample Locally +==== Building and Running the Demo -. Launch the `local-server` +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Kafka binder + -``` -$ cd -$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-.jar - -``` +include::{docs_dir}/maven-access.adoc[] + - -. Connect to Spring Cloud Data Flow's `shell` -+ -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -dataflow:>version - -``` - -+ -. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] Kafka binder variant of out-of-the-box applications -+ - ``` dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven ``` - + + . Create and deploy the following streams + ``` -(1) dataflow:>stream create tweets --definition "twitterstream --consumerKey= --consumerSecret= --accessToken= --accessTokenSecret= | log" +dataflow:>stream create tweets --definition "twitterstream --consumerKey= --consumerSecret= --accessToken= --accessTokenSecret= | log" Created new stream 'tweets' - -(2) dataflow:>stream create tweetlang --definition ":tweets.twitterstream > field-value-counter --fieldName=lang --name=language" --deploy +``` ++ +``` +dataflow:>stream create tweetlang --definition ":tweets.twitterstream > field-value-counter --fieldName=lang --name=language" --deploy Created and deployed new stream 'tweetlang' - -(3) dataflow:>stream create tagcount --definition ":tweets.twitterstream > field-value-counter --fieldName=entities.hashtags.text --name=hashtags" --deploy +``` ++ +``` +dataflow:>stream create tagcount --definition ":tweets.twitterstream > field-value-counter --fieldName=entities.hashtags.text --name=hashtags" --deploy Created and deployed new stream 'tagcount' - -(4) dataflow:>stream deploy tweets +``` ++ +``` +dataflow:>stream deploy tweets Deployed stream 'tweets' ``` ++ NOTE: To get a consumerKey and consumerSecret you need to register a twitter application. If you don’t already have one set up, you can create an app at the link:https://apps.twitter.com/[Twitter Developers] site to get these credentials. The tokens ``, ``, ``, and `` are required to be replaced with your account credentials. + @@ -88,7 +63,8 @@ dataflow:>stream list . Notice that `tweetlang.field-value-counter`, `tagcount.field-value-counter`, `tweets.log` and `tweets.twitterstream` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as Spring Boot applications within the `local-server`. + -``` +[source,console,options=nowrap] +---- 2016-02-16 11:43:26.174 INFO 10189 --- [nio-9393-exec-2] o.s.c.d.d.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:field-value-counter-sink:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-6990537012958280418/tweetlang-1455651806160/tweetlang.field-value-counter 2016-02-16 11:43:26.206 INFO 10189 --- [nio-9393-exec-3] o.s.c.d.d.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:field-value-counter-sink:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 @@ -97,11 +73,13 @@ dataflow:>stream list Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-6990537012958280418/tweets-1455651806800/tweets.log 2016-02-16 11:43:26.813 INFO 10189 --- [nio-9393-exec-4] o.s.c.d.d.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:twitterstream-source:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-6990537012958280418/tweets-1455651806800/tweets.twitterstream -``` +---- + . Verify that two `field-value-counter` with the names `hashtags` and `language` is listing successfully + -``` + +[source,console,options=nowrap] +---- dataflow:>field-value-counter list ╔════════════════════════╗ ║Field Value Counter name║ @@ -109,11 +87,12 @@ dataflow:>field-value-counter list ║hashtags ║ ║language ║ ╚════════════════════════╝ -``` +---- + . Verify you can query individual `field-value-counter` results successfully + -``` +[source,console,options=nowrap] +---- dataflow:>field-value-counter display hashtags Displaying values for field value counter 'hashtags' ╔══════════════════════════════════════╤═════╗ @@ -144,7 +123,7 @@ Displaying values for field value counter 'language' ║.. │ ...║ ╚═════╧═════╝ -``` +---- + . Go to `Dashboard` accessible at `http://localhost:9393/dashboard` and launch the `Analytics` tab. From the default `Dashboard` menu, select the following combinations to visualize real-time updates on `field-value-counter`. @@ -158,9 +137,9 @@ Displaying values for field value counter 'language' .. Stream as `hashtags` .. Visualization as `Bubble-Chart` or `Pie-Chart` -image::images/twitter_analytics.png[Twitter Analytics Visualization] +image::twitter_analytics.png[Twitter Analytics Visualization, scaledwidth="50%"] -== Summary +==== Summary In this sample, you have learned: diff --git a/src/main/asciidoc/cloudfoundry-server.adoc b/src/main/asciidoc/cloudfoundry-server.adoc new file mode 100644 index 0000000..6f5f0c9 --- /dev/null +++ b/src/main/asciidoc/cloudfoundry-server.adoc @@ -0,0 +1,66 @@ + +The Cloud Foundry Data Flow Server is Spring Boot application available for http://cloud.spring.io/spring-cloud-dataflow/#platform-implementations/[download] or you can https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[build] it yourself. +If you build it yourself, the executable jar will be in `spring-cloud-dataflow-server-cloudfoundry/target` + +NOTE: Although you can run the Data Flow Cloud Foundry Server locally and configure it to deploy to any Cloud Foundry instance, we will +deploy the server to Cloud Foundry as recommended. + +. Verify that CF instance is reachable (Your endpoint urls will be different from what is shown here). ++ + +``` +$ cf api +API endpoint: https://api.system.io (API version: ...) + +$ cf apps +Getting apps in org [your-org] / space [your-space] as user... +OK + +No apps found +``` +. Follow the instructions to deploy the https://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current/reference/htmlsingle[Spring Cloud Data Flow Cloud Foundry server]. Don't worry about creating a Redis service. We won't need it. If you are familiar with Cloud Foundry +application manifests, we recommend creating a manifest for the the Data Flow server as shown https://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current/reference/htmlsingle/#sample-manifest-template[here]. ++ +WARNING: As of this writing, there is a typo on the `SPRING_APPLICATION_JSON` entry in the sample manifest. `SPRING_APPLICATION_JSON` must be followed by `:` and The JSON string must be +wrapped in single quotes. Alternatively, you can replace that line with `MAVEN_REMOTE_REPOSITORIES_REPO1_URL: https://repo.spring.io/libs-snapshot`. If your Cloud Foundry installation is behind a firewall, you may need to install the stream apps used in this sample in your internal Maven repository and https://docs.spring.io/spring-cloud-dataflow/docs/1.3.0.M2/reference/htmlsingle/#getting-started-maven-configuration[configure] the server to access that repository. +. Once you have successfully executed `cf push`, verify the dataflow server is running ++ + +``` +$ cf apps +Getting apps in org [your-org] / space [your-space] as user... +OK + +name requested state instances memory disk urls +dataflow-server started 1/1 1G 1G dataflow-server.app.io +``` + +. Notice that the `dataflow-server` application is started and ready for interaction via the url endpoint + +. Connect the `shell` with `server` running on Cloud Foundry, e.g., `http://dataflow-server.app.io` ++ +``` +$ cd +$ java -jar spring-cloud-dataflow-shell-.jar + + ____ ____ _ __ + / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | + \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | + ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | + |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| + ____ |_| _ __|___/ __________ + | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ + | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ + | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / + |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ + + +Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". +server-unknown:> +``` ++ +``` +server-unknown:>dataflow config server http://dataflow-server.app.io +Successfully targeted http://dataflow-server.app.io +dataflow:> +``` diff --git a/src/main/asciidoc/coalesce.rb b/src/main/asciidoc/coalesce.rb new file mode 100644 index 0000000..829f6b5 --- /dev/null +++ b/src/main/asciidoc/coalesce.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby + +# This script coalesces the AsciiDoc content from a document master into a +# single output file. It does so by resolving all preprocessor directives in +# the document, and in any files which are included. The resolving of include +# directives is likely of most interest to users of this script. +# +# This script works by using Asciidoctor's PreprocessorReader to read and +# resolve all the lines in the specified input file. The script then writes the +# result to the output. +# +# The script only recognizes attributes passed in as options or those defined +# in the document header. It does not currently process attributes defined in +# other, arbitrary locations within the document. + +# TODO +# - add cli option to write attributes passed to cli to header of document +# - escape all preprocessor directives after lines are processed (these are preprocessor directives that were escaped in the input) + +require 'asciidoctor' +require 'optparse' + +options = { attributes: [], output: '-' } +OptionParser.new do |opts| + opts.banner = 'Usage: ruby asciidoc-coalescer.rb [OPTIONS] FILE' + opts.on('-a', '--attribute key[=value]', 'A document attribute to set in the form of key[=value]') do |a| + options[:attributes] << a + end + opts.on('-o', '--output FILE', 'Write output to FILE instead of stdout.') do |o| + options[:output] = o + end +end.parse! + +unless (source_file = ARGV.shift) + warn 'Please specify an AsciiDoc source file to coalesce.' + exit 1 +end + +unless (output_file = options[:output]) == '-' + if (output_file = File.expand_path output_file) == (File.expand_path source_file) + warn 'Source and output cannot be the same file.' + exit 1 + end +end + +# NOTE first, resolve attributes defined at the end of the document header +# QUESTION can we do this in a single load? +doc = Asciidoctor.load_file source_file, safe: :unsafe, header_only: true, attributes: options[:attributes] +# NOTE quick and dirty way to get the attributes set or unset by the document header +header_attr_names = (doc.instance_variable_get :@attributes_modified).to_a +header_attr_names.each {|k| doc.attributes[%(#{k}!)] = '' unless doc.attr? k } + +doc = Asciidoctor.load_file source_file, safe: :unsafe, parse: false, attributes: doc.attributes +# FIXME also escape ifdef, ifndef, ifeval and endif directives +# FIXME do this more carefully by reading line by line; if input differs by output by leading backslash, restore original line +lines = doc.reader.read.gsub(/^include::(?=.*\[\]$)/m, '\\include::').gsub(/^:docs_dir:.*/,'') + +if output_file == '-' + puts lines +else + File.open(output_file, 'w') {|f| f.write lines } +end diff --git a/datascience/species-prediction/README.adoc b/src/main/asciidoc/datascience/species-prediction/main.adoc similarity index 74% rename from datascience/species-prediction/README.adoc rename to src/main/asciidoc/datascience/species-prediction/main.adoc index b5bffeb..fee7a72 100644 --- a/datascience/species-prediction/README.adoc +++ b/src/main/asciidoc/datascience/species-prediction/main.adoc @@ -1,62 +1,28 @@ :sectnums: -= Species Prediction +:docs_dir: ../.. +=== Species Prediction In this demonstration, you will learn how to use https://en.wikipedia.org/wiki/Predictive_Model_Markup_Language[PMML] model in the context of streaming data pipeline orchestrated by http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow]. -We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `Local` server, a Spring Boot application. +We will present the steps to prep, configure and rub Spring Cloud Data Flow's `Local` server, a Spring Boot application. -== Using Local Server +==== Prerequisites -=== Prerequisites - -Make sure that you have the following components: - -* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* A running local Data Flow Server +include::{docs_dir}/local-server.adoc[] * Running instance of link:http://kafka.apache.org/downloads.html[Kafka] -=== Running the Sample Locally +==== Building and Running the Demo -. Launch the `local-server` +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Kafka binder + -``` -$ cd -$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-.jar - -``` +include::{docs_dir}/maven-access.adoc[] + - -. Connect to Spring Cloud Data Flow's `shell` -+ -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -dataflow:>version - -``` - -+ -. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] Kafka binder variant of out-of-the-box applications -+ - ``` dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven ``` - + . Create and deploy the following stream + @@ -79,16 +45,17 @@ dataflow:>stream list . Notice that `pmmlTest.http`, `pmmlTest.pmml`, and `pmmlTest.log` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running within the `local-server`. + -``` +[source,console,options=nowrap] +---- 2016-02-18 06:36:45.396 INFO 31194 --- [nio-9393-exec-1] o.s.c.d.d.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:log-sink:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-3038434123335455382/pmmlTest-1455806205386/pmmlTest.log 2016-02-18 06:36:45.402 INFO 31194 --- [nio-9393-exec-1] o.s.c.d.d.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:pmml-processor:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-3038434123335455382/pmmlTest-1455806205386/pmmlTest.pmml 2016-02-18 06:36:45.407 INFO 31194 --- [nio-9393-exec-1] o.s.c.d.d.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:http-source:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-3038434123335455382/pmmlTest-1455806205386/pmmlTest.http -``` +---- + -. Post sample data pointing to the `http` endpoint: `http://localhost:9001` [`9001` is the `server.port` we specified for the `http` source in this case] +. Post sample data to the `http` endpoint: `http://localhost:9001` (`9001` is the `port` we specified for the `http` source in this case) + ``` dataflow:>http post --target http://localhost:9001 --contentType application/json --data "{ \"sepalLength\": 6.4, \"sepalWidth\": 3.2, \"petalLength\":4.5, \"petalWidth\":1.5 }" @@ -157,7 +124,7 @@ NOTE: `petalWidth` value changed from `1.5` to `1.8` } ``` -== Summary +==== Summary In this sample, you have learned: diff --git a/src/main/asciidoc/docinfo.html b/src/main/asciidoc/docinfo.html new file mode 100644 index 0000000..49e6edf --- /dev/null +++ b/src/main/asciidoc/docinfo.html @@ -0,0 +1,76 @@ + + + + + + + + + diff --git a/functions/README.adoc b/src/main/asciidoc/functions/main.adoc similarity index 83% rename from functions/README.adoc rename to src/main/asciidoc/functions/main.adoc index 5bbeb2d..ba63c20 100644 --- a/functions/README.adoc +++ b/src/main/asciidoc/functions/main.adoc @@ -1,70 +1,34 @@ :sectnums: -= Functions in Spring Cloud Data Flow +:docs_dir: .. +=== Functions in Spring Cloud Data Flow In this sample, you will learn how to use https://github.com/spring-cloud/spring-cloud-function[Spring Cloud Function] based streaming applications in Spring Cloud Data Flow. To learn more about Spring Cloud Function, check out the http://cloud.spring.io/spring-cloud-function/[project page]. -== Prerequisites +==== Prerequisites -Make sure that you have the following components: - -* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] -* Local build of link:https://github.com/spring-cloud/spring-cloud-function[Spring Cloud Function] -* Running instance of RabbitMQ +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* A running local Data Flow Server +include::{docs_dir}/local-server.adoc[] +* A local build of link:https://github.com/spring-cloud/spring-cloud-function[Spring Cloud Function] +* A running instance of https://www.rabbitmq.com/[Rabbit MQ] * General understanding of the out-of-the-box https://github.com/spring-cloud-stream-app-starters/function/blob/master/spring-cloud-starter-stream-app-function/README.adoc[function-runner] application -=== Running the Sample Locally +==== Building and Running the Demo -. Launch the `Local-server` +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder + -``` -$ cd -$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-.jar - -``` +include::{docs_dir}/maven-access.adoc[] + - -. Connect the `shell` -+ - -[source,bash] ----- -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -dataflow:>version - ----- - -+ -. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the RabbitMQ-binder variant of the out-of-the-box applications -+ - ``` dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven ``` - + . Register the out-of-the-box https://github.com/spring-cloud-stream-app-starters/function/blob/master/spring-cloud-starter-stream-app-function/README.adoc[function-runner] application (_we will use the `1.0.0.BUILD-SNAPSHOT` built by the Spring CI system_) + - ``` dataflow:>app register --name function-runner --type processor --uri http://repo.spring.io/libs-snapshot/org/springframework/cloud/stream/app/function-app-rabbit/1.0.0.BUILD-SNAPSHOT/function-app-rabbit-1.0.0.BUILD-SNAPSHOT.jar --metadata-uri http://repo.spring.io/libs-snapshot/org/springframework/cloud/stream/app/function-app-rabbit/1.0.0.BUILD-SNAPSHOT/function-app-rabbit-1.0.0.BUILD-SNAPSHOT-metadata.jar ``` - + . Create and deploy the following stream + @@ -72,29 +36,33 @@ dataflow:>app register --name function-runner --type processor --uri http://repo dataflow:>stream create foo --definition "http --server.port=9001 | function-runner --function.className=com.example.functions.CharCounter --function.location=file:////spring-cloud-function-samples/function-sample/target/spring-cloud-function-sample-1.0.0.BUILD-SNAPSHOT.jar | log" --deploy ``` ++ NOTE: Replace the `` with the correct path. + NOTE: The source core of `CharCounter` function is in Spring cloud Function's https://github.com/spring-cloud/spring-cloud-function/blob/master/spring-cloud-function-samples/function-sample/src/main/java/com/example/functions/CharCounter.java[samples repo]. -+ - + . Verify the stream is successfully deployed. + +``` +dataflow:>stream list +``` ++ +//// +TODO: Doesn't render correctly in PDF [source,bash,options="nowrap"] ---- -dataflow:>stream list ╔══════╤══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════╗ ║Stream│ Stream Definition │ Status ║ ║ Name │ │ ║ ╠══════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╣ ║foo │http --server.port=9001 | function-runner --function.className=com.example.functions.CharCounter │All apps ║ -║ │--function.location=file:////spring-cloud-function-samples/function-sample/target/spring-cloud-function-sample-1.0.0.BUILD- │have been ║ +║ │--function.location=file:////spring-cloud-function-samples/function-sample/target/spring-cloud-function-sample-1.0.0.BUILD- │have been ║ ║ │| log │successfully║ ║ │ │deployed ║ ╚══════╧══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════╝ ---- - +//// + . Notice that `foo-http`, `foo-function-runner`, and `foo-log` link:https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as Spring Boot applications and the log locations will be printed in the Local-server console. + @@ -116,7 +84,7 @@ dataflow:>stream list ---- + -. Post sample data pointing to the `http` endpoint: `http://localhost:9001` [`9001` is the `server.port` we specified for the `http` source in this case] +. Post sample data to the `http` endpoint: `http://localhost:9001` (`9001` is the `port` we specified for the `http` source in this case) + ``` dataflow:>http post --target http://localhost:9001 --data "hello world" @@ -146,7 +114,7 @@ $ tail -f /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gs/T/spring-cloud-dataflow .... ---- -== Summary +==== Summary In this sample, you have learned: diff --git a/src/main/asciidoc/geode-setup.adoc b/src/main/asciidoc/geode-setup.adoc new file mode 100644 index 0000000..8968808 --- /dev/null +++ b/src/main/asciidoc/geode-setup.adoc @@ -0,0 +1,13 @@ + +If you do not have access an existing Geode installation, install http://geode.apache.org[Apache Geode] or +http://geode.apache.org/[Pivotal Gemfire] and start the `gfsh` CLI in a separate terminal. +``` + _________________________ __ + / _____/ ______/ ______/ /____/ / + / / __/ /___ /_____ / _____ / + / /__/ / ____/ _____/ / / / / +/______/_/ /______/_/ /_/ 1.2.1 + +Monitor and Manage Apache Geode +gfsh> +``` diff --git a/src/main/asciidoc/images/scdf-dashboard.png b/src/main/asciidoc/images/scdf-dashboard.png new file mode 100644 index 0000000..7f534cc Binary files /dev/null and b/src/main/asciidoc/images/scdf-dashboard.png differ diff --git a/analytics/twitter-analytics/images/twitter_analytics.png b/src/main/asciidoc/images/twitter_analytics.png similarity index 100% rename from analytics/twitter-analytics/images/twitter_analytics.png rename to src/main/asciidoc/images/twitter_analytics.png diff --git a/src/main/asciidoc/index-docinfo.xml b/src/main/asciidoc/index-docinfo.xml new file mode 100644 index 0000000..8c1b99d --- /dev/null +++ b/src/main/asciidoc/index-docinfo.xml @@ -0,0 +1,27 @@ +Spring Cloud Data Flow Samples +{project-version} + + 2013-2017 + Pivotal Software, Inc. + + + + Copies of this document may be made for your own use and for distribution to + others, provided that you do not charge any fee for such copies and further + provided that each copy contains this Copyright Notice, whether distributed in + print or electronically. + + + + + + + + + Pivotal Software, Inc. + + + + + + diff --git a/src/main/asciidoc/index.adoc b/src/main/asciidoc/index.adoc new file mode 100644 index 0000000..d602974 --- /dev/null +++ b/src/main/asciidoc/index.adoc @@ -0,0 +1,35 @@ += Spring Cloud Data Flow Samples +Sabby Anandan; David Turanski; Glenn Renfro; Eric Bottard; Mark Pollack; +:doctype: book +:toc: +:toclevels: 4 +:source-highlighter: prettify +:numbered: +:icons: font +:hide-uri-scheme: +:docinfo: shared + +:spring-cloud-stream-docs: http://docs.spring.io/spring-cloud-stream/docs/{scst-core-version}/reference/htmlsingle/index.html +:github-code: https://github.com/spring-cloud/spring-cloud-dataflow-samples + + +ifdef::backend-html5[] + +Version {project-version} + +(C) 2012-2017 Pivotal Software, Inc. + +_Copies of this document may be made for your own use and for distribution to +others, provided that you do not charge any fee for such copies and further +provided that each copy contains this Copyright Notice, whether distributed in +print or electronically._ + +endif::backend-html5[] + +// ====================================================================================== + +[[overview]] + +include::overview.adoc[] + +// ====================================================================================== diff --git a/src/main/asciidoc/local-server.adoc b/src/main/asciidoc/local-server.adoc new file mode 100644 index 0000000..759b20a --- /dev/null +++ b/src/main/asciidoc/local-server.adoc @@ -0,0 +1,9 @@ + +The Local Data Flow Server is Spring Boot application available for http://cloud.spring.io/spring-cloud-dataflow/#platform-implementations[download] or you can https://github.com/spring-cloud/spring-cloud-dataflow[build] it yourself. +If you build it yourself, the executable jar will be in `spring-cloud-dataflow-server-local/target` + +To run the Local Data Flow server Open a new terminal session: +``` +$cd +$java -jar spring-cloud-dataflow-server-local-.jar +``` diff --git a/src/main/asciidoc/maven-access.adoc b/src/main/asciidoc/maven-access.adoc new file mode 100644 index 0000000..d8214a5 --- /dev/null +++ b/src/main/asciidoc/maven-access.adoc @@ -0,0 +1,7 @@ +NOTE: These samples assume that the Data Flow Server can access a remote Maven repository, `https://repo.spring.io/libs-release` by default. If your Data Flow server is running behind a firewall, or you are using a maven proxy preventing +access to public repositories, you will need to install the sample apps in your internal Maven repository and https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-maven-configuration[configure] +the server accordingly. The sample applications are typically registered using Data Flow's bulk import facility. For example, the Shell command `dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven` (The actual URI is release and binder specific so refer to the sample instructions for the actual URL). +The bulk import URI references a plain text file containing entries for all of the publicly available Spring Cloud Stream and Task applications published to `https://repo.spring.io`. For example, +`source.http=maven://org.springframework.cloud.stream.app:http-source-rabbit:1.2.0.RELEASE` registers the `http` source app at the corresponding Maven address, relative to the remote repository(ies) configured for the +Data Flow server. The format is `maven://::` You will need to https://repo.spring.io/libs-release/org/springframework/cloud/stream/app/spring-cloud-stream-app-descriptor/Bacon.RELEASE/spring-cloud-stream-app-descriptor-Bacon.RELEASE.rabbit-apps-maven-repo-url.properties[download] the required apps or https://github.com/spring-cloud-stream-app-starters[build] them and then install them in your Maven repository, using whatever group, artifact, and version you choose. If you do +this, register individual apps using `dataflow:>app register...` using the `maven://` resource URI format corresponding to your installed app. diff --git a/src/main/asciidoc/overview.adoc b/src/main/asciidoc/overview.adoc new file mode 100644 index 0000000..1eb7288 --- /dev/null +++ b/src/main/asciidoc/overview.adoc @@ -0,0 +1,23 @@ +[[spring-cloud-data-flow-samples-overview]] +== Overview +This guide contains samples and demonstrations of how to build data pipelines with https://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow]. + +== Streaming +include::streaming/cassandra/http-to-cassandra/main.adoc[] +include::streaming/jdbc/http-mysql/main.adoc[] +include::streaming/gemfire/http-gemfire/main.adoc[] +include::streaming/gemfire/gemfire-cq-log/main.adoc[] +include::streaming/gemfire/gemfire-log/main.adoc[] +include::streaming/custom-apps/celsius-converter-processor/main.adoc[] + +== Task / Batch +include::tasks/simple-batch-job/main.adoc[] + +== Analytics +include::analytics/twitter-analytics/main.adoc[] + +== Data Science +include::datascience/species-prediction/main.adoc[] + +== Functions +include::functions/main.adoc[] diff --git a/src/main/asciidoc/shell.adoc b/src/main/asciidoc/shell.adoc new file mode 100644 index 0000000..6d0724f --- /dev/null +++ b/src/main/asciidoc/shell.adoc @@ -0,0 +1,30 @@ + +The Spring Cloud Data Flow Shell is available for https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started-deploying-spring-cloud-dataflow[download] or you can https://github.com/spring-cloud/spring-cloud-dataflow[build] it yourself. + +NOTE: the Spring Cloud Data Flow Shell and Local server implementation are in the same repository and are both built by running `./mvnw install` from the project root directory. If you have already run the build, use the jar in `spring-cloud-dataflow-shell/target` + +To run the Shell open a new terminal session: +``` +$ cd +$ java -jar spring-cloud-dataflow-shell-.jar + ____ ____ _ __ + / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | + \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | + ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | + |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| + ____ |_| _ __|___/ __________ + | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ + | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ + | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / + |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ + + +Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". +dataflow:> +``` +NOTE: The Spring Cloud Data Flow Shell is a Spring Boot application that connects to the Data Flow Server’s REST API and supports a DSL that simplifies the process of defining a stream or task and managing its lifecycle. Most of these samples +use the shell. If you prefer, you can use the Data Flow UI http://localhost:9393/dashboard, (or wherever it the server is hosted) to perform equivalent operations. + +ifdef::backend-pdf[] +image::scdf-dashboard.png[SCDF Dashboard, scaledwidth="50%"] +endif::[] diff --git a/src/main/asciidoc/streaming/cassandra/http-to-cassandra/local.adoc b/src/main/asciidoc/streaming/cassandra/http-to-cassandra/local.adoc new file mode 100644 index 0000000..2012b00 --- /dev/null +++ b/src/main/asciidoc/streaming/cassandra/http-to-cassandra/local.adoc @@ -0,0 +1,78 @@ +[[http-cassandra-local]] +==== Using the Local Server + +===== Additional Prerequisites + +* A running local Data Flow Server +include::{docs_dir}/local-server.adoc[] + +* Running instance of link:http://kafka.apache.org/downloads.html[Kafka] +* Running instance of link:http://cassandra.apache.org/[Apache Cassandra] + +* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using. +* Create a keyspace and a `book` table in Cassandra using: + +``` +CREATE KEYSPACE clouddata WITH REPLICATION = { 'class' : 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1' } AND DURABLE_WRITES = true; +USE clouddata; +CREATE TABLE book ( + id uuid PRIMARY KEY, + isbn text, + author text, + title text +); +``` + +===== Building and Running the Demo + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Kafka binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven +``` ++ +. Create the stream ++ +``` +dataflow:>stream create cassandrastream --definition "http --server.port=8888 --spring.cloud.stream.bindings.output.contentType='application/json' | cassandra --ingestQuery='insert into book (id, isbn, title, author) values (uuid(), ?, ?, ?)' --keyspace=clouddata" --deploy + +Created and deployed new stream 'cassandrastream' +``` +NOTE: If Cassandra isn't running on default port on `localhost` or if you need username and password to connect, use one of the following options to specify the necessary connection parameters: `--username='' --password='' --port= --contact-points=` + ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `cassandrastream-http` and `cassandrastream-cassandra` link:https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the `server` as a collocated process. ++ + +[source,console,options=nowrap] +---- +2015-12-15 15:52:31.576 INFO 18337 --- [nio-9393-exec-1] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:cassandra-sink:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/cassandrastream.cassandra +2015-12-15 15:52:31.583 INFO 18337 --- [nio-9393-exec-1] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:http-source:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/cassandrastream.http +---- ++ +. Post sample data pointing to the `http` endpoint: `http://localhost:8888` (`8888` is the `server.port` we specified for the `http` source in this case) ++ +``` +dataflow:>http post --contentType 'application/json' --data '{"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"}' --target http://localhost:8888 +> POST (application/json;charset=UTF-8) http://localhost:8888 {"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"} +> 202 ACCEPTED +``` ++ +. Connect to the Cassandra instance and query the table `clouddata.book` to list the persisted records ++ +``` +select * from clouddata.book; +``` + ++ +. You're done! diff --git a/src/main/asciidoc/streaming/cassandra/http-to-cassandra/main.adoc b/src/main/asciidoc/streaming/cassandra/http-to-cassandra/main.adoc new file mode 100644 index 0000000..b446885 --- /dev/null +++ b/src/main/asciidoc/streaming/cassandra/http-to-cassandra/main.adoc @@ -0,0 +1,25 @@ +[[spring-cloud-data-flow-samples-http-cassandra-overview]] +:sectnums: +:docs_dir: ../../.. +=== HTTP to Cassandra Demo + +In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an _HTTP_ endpoint and write the payload to a _Cassandra_ database. + +We will take you through the steps to configure and Spring Cloud Data Flow server in either a https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started/[local] or https://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current/reference/htmlsingle/#getting-started[Cloud Foundry] environment. + +==== Prerequisites +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] + +include::local.adoc[] + +include::pcf.adoc[] + +==== Summary + +In this sample, you have learned: + +* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers +* How to use Spring Cloud Data Flow's `shell` +* How to create streaming data pipeline to connect and write to `Cassandra` +* How to scale applications on `Pivotal Cloud Foundry` diff --git a/src/main/asciidoc/streaming/cassandra/http-to-cassandra/pcf.adoc b/src/main/asciidoc/streaming/cassandra/http-to-cassandra/pcf.adoc new file mode 100644 index 0000000..05e44af --- /dev/null +++ b/src/main/asciidoc/streaming/cassandra/http-to-cassandra/pcf.adoc @@ -0,0 +1,99 @@ +[[http-cassandra-cf]] +==== Using the Cloud Foundry Server + +===== Additional Prerequisites + +* Cloud Foundry instance +* A `rabbit` service instance + +* A Running instance of `cassandra` in Cloud Foundry or from another Cloud provider +* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using. +* Create a `book` table in your Cassandra keyspace using: ++ +``` +CREATE TABLE book ( + id uuid PRIMARY KEY, + isbn text, + author text, + title text +); +``` +* The Spring Cloud Data Flow Cloud Foundry Server +include::{docs_dir}/cloudfoundry-server.adoc[] + +===== Building and Running the Demo + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` ++ +. Create the stream ++ + +``` +dataflow:>stream create cassandrastream --definition "http --spring.cloud.stream.bindings.output.contentType='application/json' | cassandra --ingestQuery='insert into book (id, isbn, title, author) values (uuid(), ?, ?, ?)' --username='' --password='' --port= --contact-points= --keyspace=''" --deploy + +Created and deployed new stream 'cassandrastream' +``` ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry ++ + +``` +$ cf apps +Getting apps in org [your-org] / space [your-space] as user... +OK + +name requested state instances memory disk urls +cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.io +cassandrastream-http started 1/1 1G 1G cassandrastream-http.app.io +dataflow-server started 1/1 1G 1G dataflow-server.app.io +``` ++ +. Lookup the `url` for `cassandrastream-http` application from the list above. Post sample data pointing to the `http` endpoint: `` ++ +``` +http post --contentType 'application/json' --data '{"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"}' --target http:// +> POST (application/json;charset=UTF-8) http://cassandrastream-http.app.io {"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"} +> 202 ACCEPTED +``` ++ +. Connect to the Cassandra instance and query the table `book` to list the data inserted ++ +``` +select * from book; +``` + ++ +. Now, let's try to take advantage of Pivotal Cloud Foundry's platform capability. Let's scale the `cassandrastream-http` application from 1 to 3 instances ++ +``` +$ cf scale cassandrastream-http -i 3 +Scaling app cassandrastream-http in org user-dataflow / space development as user... +OK +``` ++ +. Verify App instances (3/3) running successfully ++ +``` +$ cf apps +Getting apps in org user-dataflow / space development as user... +OK + +name requested state instances memory disk urls +cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.io +cassandrastream-http started 3/3 1G 1G cassandrastream-http.app.io +dataflow-server started 1/1 1G 1G dataflow-server.app.io +``` ++ +. You're done! diff --git a/src/main/asciidoc/streaming/custom-apps/celsius-converter-processor/main.adoc b/src/main/asciidoc/streaming/custom-apps/celsius-converter-processor/main.adoc new file mode 100644 index 0000000..681b788 --- /dev/null +++ b/src/main/asciidoc/streaming/custom-apps/celsius-converter-processor/main.adoc @@ -0,0 +1,161 @@ +[[spring-cloud-data-flow-samples-custom-application-overview]] +:sectnums: +:docs_dir: ../../.. + +=== Custom Spring Cloud Stream Processor + +==== Prerequisites +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* A running local Data Flow Server +include::{docs_dir}/local-server.adoc[] +* A Java IDE +* https://maven.apache.org/[Maven] Installed +* A running instance of https://www.rabbitmq.com/[Rabbit MQ] + +==== Creating the Custom Stream App +We will create a custom https://cloud.spring.io/spring-cloud-stream/[Spring Cloud Stream] application and run it on Spring Cloud Data Flow. +We'll go through the steps to make a simple processor that converts temperature from Fahrenheit to Celsius. +We will be running the demo locally, but all the steps will work in a Cloud Foundry environment as well. + +. Create a new spring cloud stream project ++ +* Create a http://start.spring.io/[Spring initializer] project + +* Set the group to `demo.celsius.converter` and the artifact name as `celsius-converter-processor` + +* Choose a message transport binding as a dependency for the custom app +There are options for choosing `Rabbit MQ` or `Kafka` as the message transport. +For this demo, we will use `rabbit`. Type _rabbit_ in the search bar under _Search for dependencies_ and select `Stream Rabbit`. + +* Hit the generate project button and open the new project in an IDE of your choice ++ +. Develop the app ++ +We can now create our custom app. Our Spring Cloud Stream application is a Spring Boot application that runs as an executable jar. The application will include two Java classes: + +* `CelsiusConverterProcessorAplication.java` - the main Spring Boot application class, generated by Spring initializr + +* `CelsiusConverterProcessorConfiguration.java` - the Spring Cloud Stream code that we will write ++ +We are creating a transformer that takes a Fahrenheit input and converts it to Celsius. +Following the same naming convention as the application file, create a new Java class in the same package called `CelsiusConverterProcessorConfiguration.java`. ++ +.CelsiusConverterProcessorConfiguration.java +[source,java] +``` +@EnableBinding(Processor.class) +public class CelsiusConverterProcessorConfiguration { + + @Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT) + public int convertToCelsius(String payload) { + int fahrenheitTemperature = Integer.parseInt(payload); + return (farenheitTemperature-32)*5/9; + } +} +``` ++ +Here we introduced two important Spring annotations. +First we annotated the class with `@EnableBinding(Processor.class)`. +Second we created a method and annotated it with `@Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)`. +By adding these two annotations we have configured this stream app as a `Processor` (as opposed to a `Source` or a `Sink`). +This means that the application receives input from an upstream application via the `Processor.INPUT` channel and sends its output to a downstream application via the `Processor.OUTPUT` channel. ++ +The `convertToCelsius` method takes a `String` as input for Fahrenheit and then returns the converted Celsius as an integer. +This method is very simple, but that is also the beauty of this programming style. +We can add as much logic as we want to this method to enrich this processor. +As long as we annotate it properly and return valid output, it works as a Spring Cloud Stream Processor. Also note that it is straightforward to unit test this code. ++ +. Build the Spring Boot application with Maven ++ +``` +$cd +$./mvnw clean package +``` ++ +. Run the Application standalone ++ +``` +java -jar target/celsius-converter-processor-0.0.1-SNAPSHOT.jar +``` ++ +If all goes well, we should have a running standalone Spring Boot Application. +Once we verify that the app is started and running without any errors, we can stop it. + +==== Deploying the App to Spring Cloud Data Flow + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` ++ +. Register the custom processor ++ +``` +app register --type processor --name convertToCelsius --uri --force +``` ++ +. Create the stream ++ +We will create a stream that uses the out of the box `http` source and `log` sink and our custom transformer. ++ +``` +dataflow:>stream create --name convertToCelsiusStream --definition "http --port=9090 | convertToCelsius | log" --deploy + +Created and deployed new stream 'convertToCelsiusStream' +``` ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Verify that the apps have successfully deployed ++ +``` +dataflow:>runtime apps +``` ++ +.Note the file location of the application logs ++ +[source,console,options=nowrap] +---- +2016-09-27 10:03:11.988 INFO 95234 --- [nio-9393-exec-9] o.s.c.d.spi.local.LocalAppDeployer : deploying app convertToCelsiusStream.log instance 0 + Logs will be in /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-3236898888473815319/convertToCelsiusStream-1474984991968/convertToCelsiusStream.log +2016-09-27 10:03:12.397 INFO 95234 --- [nio-9393-exec-9] o.s.c.d.spi.local.LocalAppDeployer : deploying app convertToCelsiusStream.convertToCelsius instance 0 + Logs will be in /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-3236898888473815319/convertToCelsiusStream-1474984992392/convertToCelsiusStream.convertToCelsius +2016-09-27 10:03:14.445 INFO 95234 --- [nio-9393-exec-9] o.s.c.d.spi.local.LocalAppDeployer : deploying app convertToCelsiusStream.http instance 0 + Logs will be in /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-3236898888473815319/convertToCelsiusStream-1474984994440/convertToCelsiusStream.http +---- ++ +. Post sample data to the `http` endpoint: `http://localhost:9090` (`9090` is the `port` we specified for the `http` source in this case) ++ + +``` +dataflow:>http post --target http://localhost:9090 --data 76 +> POST (text/plain;Charset=UTF-8) http://localhost:9090 76 +> 202 ACCEPTED +``` ++ +. Open the log file for the `convertToCelsiusStream.log` app to see the output of our stream ++ +[source,console,options=nowrap] +---- +tail -f /var/folders/2q/krqwcbhj2d58csmthyq_n1nw0000gp/T/spring-cloud-dataflow-7563139704229890655/convertToCelsiusStream-1474990317406/convertToCelsiusStream.log/stdout_0.log +---- ++ +You should see the temperature you posted converted to Celsius! +``` +2016-09-27 10:05:34.933 INFO 95616 --- [CelsiusStream-1] log.sink : 24 +``` + +==== Summary +In this sample, you have learned: + +* How to write a custom `Processor` stream application +* How to use Spring Cloud Data Flow's `Local` server +* How to use Spring Cloud Data Flow's `shell` application diff --git a/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/local.adoc b/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/local.adoc new file mode 100644 index 0000000..c169989 --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/local.adoc @@ -0,0 +1,91 @@ +[[gemfire-cq-log-local]] +==== Using the Local Server +===== Additional Prerequisites +* A Running Data Flow Server +include::{docs_dir}/local-server.adoc[] +* A running instance of https://www.rabbitmq.com[Rabbit MQ] + +===== Building and Running the Demo + +. Use gfsh to start a locator and server ++ +``` +gfsh>start locator --name=locator1 +gfsh>start server --name=server1 + +``` +. Create a region called `Orders` ++ +``` +gfsh>create region --name Orders --type=REPLICATE +``` ++ +*Use the Shell to create the sample stream* ++ +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` +. Create the stream ++ +This example creates an gemfire-cq source to which will publish events matching a query criteria on a region. In this case we will monitor the `Orders` region. For simplicity, we will avoid creating a data structure for the order. +Each cache entry contains an integer value representing the quantity of the ordered item. This stream will fire a message whenever the value>999. By default, the source emits only the value. Here we will override that using the +`cq-event-expression` property. This accepts a SpEL expression bound to a https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/query/CqEvent.html[CQEvent]. To reference the entire CQEvent instace, we use `#this`. +In order to display the contents in the log, we will invoke `toString()` on the instance. ++ +``` +dataflow:>stream create --name orders --definition " gemfire-cq --query='SELECT * from /Orders o where o > 999' --cq-event-expression=#this.toString() | log" --deploy +Created and deployed new stream 'events' +``` +NOTE: If the Geode locator isn't running on default port on `localhost`, add the options `--connect-type=locator --host-addresses=:`. If there are multiple +locators, you can provide a comma separated list of locator addresses. This is not necessary for the sample but is typical for production environments to enable fail-over. + +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` + +. Monitor stdout for the log sink. When you deploy the stream, you will see log messages in the Data Flow server console like this ++ +``` +2017-10-30 09:39:36.283 INFO 8167 --- [nio-9393-exec-5] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId orders.log instance 0. + Logs will be in /var/folders/hd/5yqz2v2d3sxd3n879f4sg4gr0000gn/T/spring-cloud-dataflow-5375107584795488581/orders-1509370775940/orders.log +``` ++ +Copy the location of the `log` sink logs. This is a directory that ends in `orders.log`. The log files will be in `stdout_0.log` under this directory. You can monitor the output of the log sink using `tail`, or something similar: ++ +``` +$tail -f /var/folders/hd/5yqz2v2d3sxd3n879f4sg4gr0000gn/T/spring-cloud-dataflow-5375107584795488581/orders-1509370775940/orders.log/stdout_0.log +``` ++ +. Using `gfsh`, create and update some cache entries ++ +``` +gfsh>put --region Orders --value-class java.lang.Integer --key 01234 --value 1000 +gfsh>put --region Orders --value-class java.lang.Integer --key 11234 --value 1005 +gfsh>put --region Orders --value-class java.lang.Integer --key 21234 --value 100 +gfsh>put --region Orders --value-class java.lang.Integer --key 31234 --value 999 +gfsh>put --region Orders --value-class java.lang.Integer --key 21234 --value 1000 + +``` ++ +. Observe the log output +You should see messages like: ++ +``` +2017-10-30 09:53:02.231 INFO 8563 --- [ire-cq.orders-1] log-sink : CqEvent [CqName=GfCq1; base operation=CREATE; cq operation=CREATE; key=01234; value=1000] +2017-10-30 09:53:19.732 INFO 8563 --- [ire-cq.orders-1] log-sink : CqEvent [CqName=GfCq1; base operation=CREATE; cq operation=CREATE; key=11234; value=1005] +2017-10-30 09:53:53.242 INFO 8563 --- [ire-cq.orders-1] log-sink : CqEvent [CqName=GfCq1; base operation=UPDATE; cq operation=CREATE; key=21234; value=1000] +``` ++ +. Another interesting demonstration combines `gemfire-cq` with the link::../http-gemfire/README.adoc[http-gemfire] example. +``` +dataflow:> stream create --name stocks --definition "http --port=9090 | gemfire-json-server --regionName=Stocks --keyExpression=payload.getField('symbol')" --deploy +dataflow:> stream create --name stock_watch --definition "gemfire-cq --query='Select * from /Stocks where symbol=''VMW''' | log" --deploy +``` + +. You're done! diff --git a/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/main.adoc b/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/main.adoc new file mode 100644 index 0000000..0591be0 --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/main.adoc @@ -0,0 +1,31 @@ +[[spring-cloud-data-flow-samples-gemfire-cq-log-overview]] +:sectnums: +:docs_dir: ../../.. + +=== Gemfire CQ to Log Demo + +In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from a `gemfire-cq` (Continuous Query) endpoint and write to a log using the `log` sink. +The `gemfire-cq` source creates a Continuous Query to monitor events for a region that match the query's result set and publish a message whenever such an event is emitted. In this example, we simulate monitoring orders to trigger a process whenever +the quantity ordered is above a defined limit. + +We will take you through the steps to configure and run Spring Cloud Data Flow server in either a https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started/[local] or https://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current/reference/htmlsingle/#getting-started[Cloud Foundry] environment. + +NOTE: For legacy reasons the `gemfire` Spring Cloud Stream Apps are named after `Pivotal GemFire`. The code base for the commercial product has since been open sourced as `Apache Geode`. These samples should work with compatible versions of Pivotal GemFire or Apache Geode. Herein we will refer to the installed IMDG simply as `Geode`. + +==== Prerequisites +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* A Geode installation with a locator and cache server running +include::{docs_dir}/geode-setup.adoc[] + +include::local.adoc[] + +include::pcf.adoc[] + +==== Summary + +In this sample, you have learned: + +* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers +* How to use Spring Cloud Data Flow's `shell` +* How to create streaming data pipeline to connect and publish CQ events from `gemfire` diff --git a/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/pcf.adoc b/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/pcf.adoc new file mode 100644 index 0000000..a895586 --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/gemfire-cq-log/pcf.adoc @@ -0,0 +1,114 @@ +[[gemfire-cq-log-cf]] +==== Using the Cloud Foundry Server +===== Additional Prerequisites + +* A Cloud Foundry instance + +* Running instance of a `rabbit` service in Cloud Foundry + +* Running instance of the https://docs.pivotal.io/p-cloud-cache/1-0/developer.html[Pivotal Cloud Cache for PCF] (PCC) service `cloudcache` in Cloud Foundry. + +* The Spring Cloud Data Flow Cloud Foundry Server + +include::{docs_dir}/cloudfoundry-server.adoc[] + +===== Building and Running the Demo + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` ++ +. Get the PCC connection information ++ +``` +$ cf service-key cloudcache my-service-key +Getting key my-service-key for service instance cloudcache as ... + +{ + "locators": [ + "10.0.16.9[55221]", + "10.0.16.11[55221]", + "10.0.16.10[55221]" + ], + "urls": { + "gfsh": "http://...", + "pulse": "http://.../pulse" + }, + "users": [ + { + "password": , + "username": "cluster_operator" + }, + { + "password": , + "username": "developer" + } + ] +} +``` ++ +. Using `gfsh`, connect to the PCC instance as `cluster_operator` using the service key values and create the Test region. ++ +``` +gfsh>connect --use-http --url= --user=cluster_operator --password= +gfsh>create region --name Orders --type=REPLICATE +``` ++ +. Create the stream using the Data Flow Shell ++ +This example creates an gemfire-cq source to which will publish events matching a query criteria on a region. In this case we will monitor the `Orders` region. For simplicity, we will avoid creating a data structure for the order. +Each cache entry contains an integer value representing the quantity of the ordered item. This stream will fire a message whenever the value>999. By default, the source emits only the value. Here we will override that using the +`cq-event-expression` property. This accepts a SpEL expression bound to a https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/query/CqEvent.html[CQEvent]. To reference the entire CQEvent instance, we use `#this`. +In order to display the contents in the log, we will invoke `toString()` on the instance. ++ +``` +dataflow:>stream create --name orders --definition " gemfire-cq --username=developer --password= --connect-type=locator --host-addresses=10.0.16.9:55221 --query='SELECT * from /Orders o where o > 999' --cq-event-expression=#this.toString() | log" --deploy +Created and deployed new stream 'events' +``` ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Monitor stdout for the log sink ++ +``` +cf logs +``` ++ +. Using `gfsh`, create and update some cache entries ++ +``` +gfsh>connect --use-http --url= --user=cluster_operator --password= +gfsh>put --region Orders --value-class java.lang.Integer --key 01234 --value 1000 +gfsh>put --region Orders --value-class java.lang.Integer --key 11234 --value 1005 +gfsh>put --region Orders --value-class java.lang.Integer --key 21234 --value 100 +gfsh>put --region Orders --value-class java.lang.Integer --key 31234 --value 999 +gfsh>put --region Orders --value-class java.lang.Integer --key 21234 --value 1000 + +``` ++ +. Observe the log output +You should see messages like: ++ +[source,console,options=nowrap] +---- +2017-10-30 09:53:02.231 INFO 8563 --- [ire-cq.orders-1] log-sink : CqEvent [CqName=GfCq1; base operation=CREATE; cq operation=CREATE; key=01234; value=1000] +2017-10-30 09:53:19.732 INFO 8563 --- [ire-cq.orders-1] log-sink : CqEvent [CqName=GfCq1; base operation=CREATE; cq operation=CREATE; key=11234; value=1005] +2017-10-30 09:53:53.242 INFO 8563 --- [ire-cq.orders-1] log-sink : CqEvent [CqName=GfCq1; base operation=UPDATE; cq operation=CREATE; key=21234; value=1000] +---- ++ +. Another interesting demonstration combines `gemfire-cq` with the link:../http-gemfire/README.adoc[http-gemfire] example. ++ +``` +dataflow:> stream create --name stocks --definition "http --port=9090 | gemfire-json-server --regionName=Stocks --keyExpression=payload.getField('symbol')" --deploy +dataflow:> stream create --name stock_watch --definition "gemfire-cq --query='Select * from /Stocks where symbol=''VMW''' | log" --deploy +``` ++ +. You're done! diff --git a/src/main/asciidoc/streaming/gemfire/gemfire-log/local.adoc b/src/main/asciidoc/streaming/gemfire/gemfire-log/local.adoc new file mode 100644 index 0000000..48722ac --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/gemfire-log/local.adoc @@ -0,0 +1,106 @@ +[[gemfire-log-local]] +==== Using the Local Server +===== Additional Prerequisites + +* A Running Data Flow Server +include::{docs_dir}/local-server.adoc[] +* A running instance of https://www.rabbitmq.com[Rabbit MQ] + +===== Building and Running the Demo + +. Use gfsh to start a locator and server ++ +``` +gfsh>start locator --name=locator1 +gfsh>start server --name=server1 + +``` +. Create a region called `Test` ++ +``` +gfsh>create region --name Test --type=REPLICATE +``` ++ +*Use the Shell to create the sample stream* ++ +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` +. Create the stream ++ +This example creates an gemfire source to which will publish events on a region ++ +``` +dataflow:>stream create --name events --definition " gemfire --regionName=Test | log" --deploy +Created and deployed new stream 'events' +``` +NOTE: If the Geode locator isn't running on default port on `localhost`, add the options `--connect-type=locator --host-addresses=:`. If there are multiple +locators, you can provide a comma separated list of locator addresses. This is not necessary for the sample but is typical for production environments to enable fail-over. + +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` + +. Monitor stdout for the log sink. When you deploy the stream, you will see log messages in the Data Flow server console like this ++ + +[source,console,options=nowrap] +---- +2017-10-28 17:28:23.275 INFO 15603 --- [nio-9393-exec-2] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId events.log instance 0. + Logs will be in /var/folders/hd/5yqz2v2d3sxd3n879f4sg4gr0000gn/T/spring-cloud-dataflow-4093992067314402881/events-1509226103269/events.log +2017-10-28 17:28:23.277 INFO 15603 --- [nio-9393-exec-2] o.s.c.d.s.c.StreamDeploymentController : Downloading resource URI [maven://org.springframework.cloud.stream.app:gemfire-source-rabbit:1.2.0.RELEASE] +2017-10-28 17:28:23.311 INFO 15603 --- [nio-9393-exec-2] o.s.c.d.s.c.StreamDeploymentController : Deploying application named [gemfire] as part of stream named [events] with resource URI [maven://org.springframework.cloud.stream.app:gemfire-source-rabbit:1.2.0.RELEASE] +2017-10-28 17:28:23.318 INFO 15603 --- [nio-9393-exec-2] o.s.c.d.spi.local.LocalAppDeployer : Deploying app with deploymentId events.gemfire instance 0. + Logs will be in /var/folders/hd/5yqz2v2d3sxd3n879f4sg4gr0000gn/T/spring-cloud-dataflow-4093992067314402881/events-1509226103311/events.gemfire +---- + ++ +Copy the location of the `log` sink logs. This is a directory that ends in `events.log`. The log files will be in `stdout_0.log` under this directory. You can monitor the output of the log sink using `tail`, or something similar: ++ + +[source,console,options=nowrap] +---- +$tail -f /var/folders/hd/5yqz2v2d3sxd3n879f4sg4gr0000gn/T/spring-cloud-dataflow-4093992067314402881/events-1509226103269/events.log/stdout_0.log +---- + ++ +. Using `gfsh`, create and update some cache entries ++ +``` +gfsh>put --region /Test --key 1 --value "value 1" +gfsh>put --region /Test --key 2 --value "value 2" +gfsh>put --region /Test --key 3 --value "value 3" +gfsh>put --region /Test --key 1 --value "new value 1" +``` ++ +. Observe the log output +You should see messages like: ++ +[source,console,options=nowrap] +---- +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : value 1" +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : value 2" +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : value 3" +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : new value 1" +---- + ++ +By default, the message payload contains the updated value. Depending on your application, you may need additional information. The data comes from https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/EntryEvent.html[EntryEvent]. You +can access any fields using the source's `cache-event-expression` property. This takes a SpEL expression bound to the EntryEvent. Try something like `--cache-event-expression='{key:'\+key+',new_value:'\+newValue+'}'` (HINT: You will need to destroy the stream and recreate it to +add this property, an exercise left to the reader). Now you should see log messages like: ++ + +[source,console,options=nowrap] +---- +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log-sink : {key:1,new_value:value 1} +2017-10-28 17:41:24.466 INFO 18986 --- [emfire.events-1] log-sink : {key:2,new_value:value 2} +---- + ++ +. You're done! diff --git a/src/main/asciidoc/streaming/gemfire/gemfire-log/main.adoc b/src/main/asciidoc/streaming/gemfire/gemfire-log/main.adoc new file mode 100644 index 0000000..530c14b --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/gemfire-log/main.adoc @@ -0,0 +1,30 @@ +[[spring-cloud-data-flow-samples-gemfire-log-overview]] +:sectnums: +:docs_dir: ../../.. + +=== Gemfire to Log Demo + +In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from a `gemfire` endpoint and write to a log using the `log` sink. +The `gemfire` source creates a https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheListener.html[CacheListener] to monitor events for a region and publish a message whenever an entry is changed. + +We will take you through the steps to configure and run Spring Cloud Data Flow server in either a https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started/[local] or https://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current/reference/htmlsingle/#getting-started[Cloud Foundry] environment. + +NOTE: For legacy reasons the `gemfire` Spring Cloud Stream Apps are named after `Pivotal GemFire`. The code base for the commercial product has since been open sourced as `Apache Geode`. These samples should work with compatible versions of Pivotal GemFire or Apache Geode. Herein we will refer to the installed IMDG simply as `Geode`. + +==== Prerequisites +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* A Geode installation with a locator and cache server running +include::{docs_dir}/geode-setup.adoc[] + +include::local.adoc[] + +include::pcf.adoc[] + +==== Summary + +In this sample, you have learned: + +* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers +* How to use Spring Cloud Data Flow's `shell` +* How to create streaming data pipeline to connect and publish events from `gemfire` diff --git a/src/main/asciidoc/streaming/gemfire/gemfire-log/pcf.adoc b/src/main/asciidoc/streaming/gemfire/gemfire-log/pcf.adoc new file mode 100644 index 0000000..288bf0d --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/gemfire-log/pcf.adoc @@ -0,0 +1,110 @@ +[[gemfire-log-cf]] +==== Using the Cloud Foundry Server +===== Additional Prerequisites +* A Cloud Foundry instance + +* Running instance of a `rabbit` service in Cloud Foundry + +* Running instance of the https://docs.pivotal.io/p-cloud-cache/1-0/developer.html[Pivotal Cloud Cache for PCF] (PCC) service `cloudcache` in Cloud Foundry. + +* The Spring Cloud Data Flow Cloud Foundry Server + +include::{docs_dir}/cloudfoundry-server.adoc[] + +===== Building and Running the Demo + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` ++ +. Get the PCC connection information ++ +``` +$ cf service-key cloudcache my-service-key +Getting key my-service-key for service instance cloudcache as ... + +{ + "locators": [ + "10.0.16.9[55221]", + "10.0.16.11[55221]", + "10.0.16.10[55221]" + ], + "urls": { + "gfsh": "http://...", + "pulse": "http://.../pulse" + }, + "users": [ + { + "password": , + "username": "cluster_operator" + }, + { + "password": , + "username": "developer" + } + ] +} +``` ++ +. Using `gfsh`, connect to the PCC instance as `cluster_operator` using the service key values and create the Test region. ++ +``` +gfsh>connect --use-http --url= --user=cluster_operator --password= +gfsh>create region --name Test --type=REPLICATE +``` ++ +. Create the stream, connecting to the PCC instance as developer. This example creates an gemfire source to which will publish events on a region ++ +``` +dataflow stream create --name events --definition " gemfire --username=developer --password= --connect-type=locator --host-addresses=10.0.16.9:55221 --regionName=Test | log" --deploy +``` + +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Monitor stdout for the log sink ++ +``` +cf logs +``` ++ +. Using `gfsh`, create and update some cache entries ++ +``` +gfsh>connect --use-http --url= --user=cluster_operator --password= +gfsh>put --region /Test --key 1 --value "value 1" +gfsh>put --region /Test --key 2 --value "value 2" +gfsh>put --region /Test --key 3 --value "value 3" +gfsh>put --region /Test --key 1 --value "new value 1" +``` ++ +. Observe the log output ++ +You should see messages like: ++ +[source,console,options=nowrap] +---- +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : value 1" +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : value 2" +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : value 3" +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log sink : new value 1" +---- ++ +By default, the message payload contains the updated value. Depending on your application, you may need additional information. The data comes from https://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/EntryEvent.html[EntryEvent]. You +can access any fields using the source's `cache-event-expression` property. This takes a SpEL expression bound to the EntryEvent. Try something like `--cache-event-expression='{key:'\+key+',new_value:'\+newValue+'}'` (HINT: You will need to destroy the stream and recreate it to +add this property, an exercise left to the reader). Now you should see log messages like: ++ +[source,console,options=nowrap] +---- +2017-10-28 17:28:52.893 INFO 18986 --- [emfire.events-1] log-sink : {key:1,new_value:value 1} +2017-10-28 17:41:24.466 INFO 18986 --- [emfire.events-1] log-sink : {key:2,new_value:value 2} +---- ++ +. You're done! diff --git a/src/main/asciidoc/streaming/gemfire/http-gemfire/local.adoc b/src/main/asciidoc/streaming/gemfire/http-gemfire/local.adoc new file mode 100644 index 0000000..cd3ae4b --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/http-gemfire/local.adoc @@ -0,0 +1,78 @@ +[[gemfire-http-local]] +==== Using the Local Server +===== Additional Prerequisites + +* A running local Data Flow Server +include::{docs_dir}/local-server.adoc[] +* A running instance of https://www.rabbitmq.com[Rabbit MQ] + +===== Building and Running the Demo + +. Use gfsh to start a locator and server ++ +``` +gfsh>start locator --name=locator1 +gfsh>start server --name=server1 + +``` +. Create a region called `Stocks` ++ +``` +gfsh>create region --name Stocks --type=REPLICATE +``` ++ +*Use the Shell to create the sample stream* ++ +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` +. Create the stream ++ +This example creates an http endpoint to which we will post stock prices as a JSON document containing `symbol` and `price` fields. +The property `--json=true` to enable Geode's JSON support and configures the sink to convert JSON String payloads to https://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html[PdxInstance], the recommended way +to store JSON documents in Geode. The `keyExpression` property is a SpEL expression used to extract the `symbol` value the PdxInstance to use as an entry key. ++ +NOTE: PDX serialization is very efficient and supports OQL queries without requiring a custom domain class. +Use of custom domain types requires these classes to be in the class path of both the stream apps and the cache server. +For this reason, the use of custom payload types is generally discouraged. ++ +``` +dataflow:>stream create --name stocks --definition "http --port=9090 | gemfire --json=true --regionName=Stocks --keyExpression=payload.getField('symbol')" --deploy +Created and deployed new stream 'stocks' +``` +NOTE: If the Geode locator isn't running on default port on `localhost`, add the options `--connect-type=locator --host-addresses=:`. If there are multiple +locators, you can provide a comma separated list of locator addresses. This is not necessary for the sample but is typical for production environments to enable fail-over. + +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` + +. Post sample data pointing to the `http` endpoint: `http://localhost:9090` (`9090` is the `port` we specified for the `http` source) ++ +``` +dataflow:>http post --target http://localhost:9090 --contentType application/json --data '{"symbol":"VMW","price":117.06}' +> POST (application/json) http://localhost:9090 {"symbol":"VMW","price":117.06} +> 202 ACCEPTED +``` ++ +. Using `gfsh`, connect to the locator if not already connected, and verify the cache entry was created. ++ +``` +gfsh>get --key='VMW' --region=/Stocks +Result : true +Key Class : java.lang.String +Key : VMW +Value Class : org.apache.geode.pdx.internal.PdxInstanceImpl + +symbol | price +------ | ------ +VMW | 117.06 +``` ++ +. You're done! diff --git a/src/main/asciidoc/streaming/gemfire/http-gemfire/main.adoc b/src/main/asciidoc/streaming/gemfire/http-gemfire/main.adoc new file mode 100644 index 0000000..e18dc09 --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/http-gemfire/main.adoc @@ -0,0 +1,29 @@ +[[spring-cloud-data-flow-samples-gemfire-http-overview]] +:sectnums: +:docs_dir: ../../.. + +=== HTTP to Gemfire Demo + +In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to Gemfire using the `gemfire` sink. + +We will take you through the steps to configure and run Spring Cloud Data Flow server in either a https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started/[local] or https://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current/reference/htmlsingle/#getting-started[Cloud Foundry] environment. + +NOTE: For legacy reasons the `gemfire` Spring Cloud Stream Apps are named after `Pivotal GemFire`. The code base for the commercial product has since been open sourced as `Apache Geode`. These samples should work with compatible versions of Pivotal GemFire or Apache Geode. Herein we will refer to the installed IMDG simply as `Geode`. + +==== Prerequisites +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* A Geode installation with a locator and cache server running +include::{docs_dir}/geode-setup.adoc[] + +include::local.adoc[] + +include::pcf.adoc[] + +==== Summary + +In this sample, you have learned: + +* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers +* How to use Spring Cloud Data Flow's `shell` +* How to create streaming data pipeline to connect and write to `gemfire` diff --git a/src/main/asciidoc/streaming/gemfire/http-gemfire/pcf.adoc b/src/main/asciidoc/streaming/gemfire/http-gemfire/pcf.adoc new file mode 100644 index 0000000..0799337 --- /dev/null +++ b/src/main/asciidoc/streaming/gemfire/http-gemfire/pcf.adoc @@ -0,0 +1,106 @@ +[[gemfire-http-cf]] +===== Using the Cloud Foundry Server +====== Additional Prerequisites + +* A Cloud Foundry instance + +* Running instance of a `rabbit` service in Cloud Foundry + +* Running instance of the https://docs.pivotal.io/p-cloud-cache/1-0/developer.html[Pivotal Cloud Cache for PCF] (PCC) service `cloudcache` in Cloud Foundry. + +* Cloud Data Flow Cloud Foundry Server +include::{docs_dir}/cloudfoundry-server.adoc[] + +====== Building and Running the Demo + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` ++ +. Get the PCC connection information ++ +``` +$ cf service-key cloudcache my-service-key +Getting key my-service-key for service instance cloudcache as ... + +{ + "locators": [ + "10.0.16.9[55221]", + "10.0.16.11[55221]", + "10.0.16.10[55221]" + ], + "urls": { + "gfsh": "http://...", + "pulse": "http://.../pulse" + }, + "users": [ + { + "password": , + "username": "cluster_operator" + }, + { + "password": , + "username": "developer" + } + ] +} +``` ++ +. Using `gfsh`, connect to the PCC instance as `cluster_operator` using the service key values and create the Stocks region. ++ +``` +gfsh>connect --use-http --url= --user=cluster_operator --password= +gfsh>create region --name Stocks --type=REPLICATE +``` ++ +. Create the stream, connecting to the PCC instance as developer ++ +This example creates an http endpoint to which we will post stock prices as a JSON document containing `symbol` and `price` fields. +The property `--json=true` to enable Geode's JSON support and configures the sink to convert JSON String payloads to https://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html[PdxInstance], the recommended way +to store JSON documents in Geode. The `keyExpression` property is a SpEL expression used to extract the `symbol` value the PdxInstance to use as an entry key. ++ +NOTE: PDX serialization is very efficient and supports OQL queries without requiring a custom domain class. +Use of custom domain types requires these classes to be in the class path of both the stream apps and the cache server. +For this reason, the use of custom payload types is generally discouraged. ++ +``` +dataflow:>stream create --name stocks --definition "http --security.basic.enabled=false | gemfire --username=developer --password= --connect-type=locator --host-addresses=10.0.16.9:55221 --regionName=Stocks --keyExpression=payload.getField('symbol')" --deploy +``` + +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` + +. Post sample data pointing to the `http` endpoint ++ + +Get the url of the http source using `cf apps` ++ +``` +dataflow:>http post --target http:// --contentType application/json --data '{"symbol":"VMW","price":117.06}' +> POST (application/json) http://... {"symbol":"VMW","price":117.06} +> 202 ACCEPTED +``` ++ +. Using `gfsh`, connect to the PCC instance as `cluster_operator` using the service key values. ++ +``` +gfsh>connect --use-http --url= --user=cluster_operator --password= +gfsh>get --key='VMW' --region=/Stocks +Result : true +Key Class : java.lang.String +Key : VMW +Value Class : org.apache.geode.pdx.internal.PdxInstanceImpl + +symbol | price +------ | ------ +VMW | 117.06 +``` ++ +. You're done! diff --git a/src/main/asciidoc/streaming/jdbc/http-mysql/local.adoc b/src/main/asciidoc/streaming/jdbc/http-mysql/local.adoc new file mode 100644 index 0000000..b0f1dea --- /dev/null +++ b/src/main/asciidoc/streaming/jdbc/http-mysql/local.adoc @@ -0,0 +1,74 @@ +==== Using the Local Server + +===== Additional Prerequisites + +* A running local Data Flow Server +include::{docs_dir}/local-server.adoc[] + +* Running instance of link:http://kafka.apache.org/downloads.html[Kafka] +* Running instance of link:http://www.mysql.com/[MySQL] +* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] or link:https://www.dbvis.com/[DbVisualizer] +* Create the `test` database with a `names` table (in MySQL) using: ++ +``` +CREATE DATABASE test; +USE test; +CREATE TABLE names +( + name varchar(255) +); +``` + +===== Building and Running the Demo + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Kafka binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven +``` ++ +. Create the stream ++ +``` +dataflow:>stream create --name mysqlstream --definition "http --server.port=8787 | jdbc --tableName=names --columns=name --spring.datasource.driver-class-name=org.mariadb.jdbc.Driver --spring.datasource.url='jdbc:mysql://localhost:3306/test'" --deploy + +Created and deployed new stream 'mysqlstream' +``` +NOTE: If MySQL isn't running on default port on `localhost` or if you need username and password to connect, use one of the following options to specify the necessary connection parameters: `--spring.datasource.url='jdbc:mysql://:/' --spring.datasource.username= --spring.datasource.password=` + ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the Local `server` as collocated processes. ++ + +[source,console,options=nowrap] +---- +2016-05-03 09:29:55.918 INFO 65162 --- [nio-9393-exec-3] o.s.c.d.spi.local.LocalAppDeployer : deploying app mysqlstream.jdbc instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-dataflow-6850863945840320040/mysqlstream1-1462292995903/mysqlstream.jdbc +2016-05-03 09:29:55.939 INFO 65162 --- [nio-9393-exec-3] o.s.c.d.spi.local.LocalAppDeployer : deploying app mysqlstream.http instance 0 + Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-dataflow-6850863945840320040/mysqlstream-1462292995934/mysqlstream.http +---- + +. Post sample data pointing to the `http` endpoint: `http://localhost:8787` [`8787` is the `server.port` we specified for the `http` source in this case] + ++ +``` +dataflow:>http post --contentType 'application/json' --target http://localhost:8787 --data "{\"name\": \"Foo\"}" +> POST (application/json;charset=UTF-8) http://localhost:8787 {"name": "Spring Boot"} +> 202 ACCEPTED +``` ++ +. Connect to the MySQL instance and query the table `test.names` to list the new rows: ++ +``` +select * from test.names; +``` ++ +. You're done! diff --git a/src/main/asciidoc/streaming/jdbc/http-mysql/main.adoc b/src/main/asciidoc/streaming/jdbc/http-mysql/main.adoc new file mode 100644 index 0000000..bd8ce3b --- /dev/null +++ b/src/main/asciidoc/streaming/jdbc/http-mysql/main.adoc @@ -0,0 +1,25 @@ +:sectnums: +:docs_dir: ../../.. + +=== HTTP to MySQL Demo + +In this demonstration, you will learn how to build a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to MySQL database using `jdbc` sink. + +We will take you through the steps to configure and Spring Cloud Data Flow server in either a https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#getting-started/[local] or https://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current/reference/htmlsingle/#getting-started[Cloud Foundry] environment. + +==== Prerequisites +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] + +include::local.adoc[] + +include::pcf.adoc[] + +==== Summary + +In this sample, you have learned: + +* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers +* How to use Spring Cloud Data Flow's `shell` +* How to create streaming data pipeline to connect and write to `MySQL` +* How to scale applications on `Pivotal Cloud Foundry` diff --git a/src/main/asciidoc/streaming/jdbc/http-mysql/pcf.adoc b/src/main/asciidoc/streaming/jdbc/http-mysql/pcf.adoc new file mode 100644 index 0000000..bdec14d --- /dev/null +++ b/src/main/asciidoc/streaming/jdbc/http-mysql/pcf.adoc @@ -0,0 +1,104 @@ + +==== Using the Cloud Foundry Server + +===== Additional Prerequisites + +* Cloud Foundry instance + +* Running instance of `rabbit` in Cloud Foundry +* Running instance of `mysql` in Cloud Foundry +* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] or link:https://www.dbvis.com/[DbVisualizer] +* Create the `names` table (in MySQL) using: ++ +``` +CREATE TABLE names +( + name varchar(255) +); +``` ++ +* The Spring Cloud Data Flow Cloud Foundry Server +include::{docs_dir}/cloudfoundry-server.adoc[] + +===== Building and Running the Demo + +. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] the out-of-the-box applications for the Rabbit binder ++ +include::{docs_dir}/maven-access.adoc[] ++ +``` +dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven +``` ++ +. Create the stream ++ + +``` +dataflow:>stream create --name mysqlstream --definition "http | jdbc --tableName=names --columns=name" +Created new stream 'mysqlstream' + +dataflow:>stream deploy --name mysqlstream --properties "app.jdbc.spring.cloud.deployer.cloudfoundry.services=mysql" +Deployed stream 'mysqlstream' + +``` ++ + +NOTE: By supplying the `app.jdbc.spring.cloud.deployer.cloudfoundry.services=mysql` property, we are deploying the stream with `jdbc-sink` to automatically bind to `mysql` service and only this application in the stream gets the service binding. This also eliminates the requirement to supply `datasource` credentials in stream definition. ++ +. Verify the stream is successfully deployed ++ +``` +dataflow:>stream list +``` ++ +. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry ++ + +``` +$ cf apps +Getting apps in org user-dataflow / space development as user... +OK + +name requested state instances memory disk urls +mysqlstream-http started 1/1 1G 1G mysqlstream-http.app.io +mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.io +dataflow-server started 1/1 1G 1G dataflow-server.app.io +``` ++ +. Lookup the `url` for `mysqlstream-http` application from the list above. Post sample data pointing to the `http` endpoint: `` ++ +``` +http post --contentType 'application/json' --target http://mysqlstream-http.app.io --data "{\"name\": \"Bar"}" +> POST (application/json;charset=UTF-8) http://mysqlstream-http.app.io {"name": "Bar"} +> 202 ACCEPTED +``` ++ +. Connect to the MySQL instance and query the table `names` to list the new rows: ++ +``` +select * from names; +``` + ++ +. Now, let's take advantage of Pivotal Cloud Foundry's platform capability. Let's scale the `mysqlstream-http` application from 1 to 3 instances ++ +``` +$ cf scale mysqlstream-http -i 3 +Scaling app mysqlstream-http in org user-dataflow / space development as user... +OK +``` ++ +. Verify App instances (3/3) running successfully ++ +``` +$ cf apps +Getting apps in org user-dataflow / space development as user... +OK + +name requested state instances memory disk urls +mysqlstream-http started 3/3 1G 1G mysqlstream-http.app.io +mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.io +dataflow-server started 1/1 1G 1G dataflow-server.app.io +``` ++ +. You're done! diff --git a/tasks/simple-batch-job/batch-job-1.3.0.BUILD-SNAPSHOT.jar b/src/main/asciidoc/tasks/simple-batch-job/batch-job-1.3.0.BUILD-SNAPSHOT.jar similarity index 100% rename from tasks/simple-batch-job/batch-job-1.3.0.BUILD-SNAPSHOT.jar rename to src/main/asciidoc/tasks/simple-batch-job/batch-job-1.3.0.BUILD-SNAPSHOT.jar diff --git a/tasks/simple-batch-job/README.adoc b/src/main/asciidoc/tasks/simple-batch-job/main.adoc similarity index 72% rename from tasks/simple-batch-job/README.adoc rename to src/main/asciidoc/tasks/simple-batch-job/main.adoc index 662243b..d912aa5 100644 --- a/tasks/simple-batch-job/README.adoc +++ b/src/main/asciidoc/tasks/simple-batch-job/main.adoc @@ -1,39 +1,23 @@ :sectnums: -= Batch Job on Cloud Foundry +:docs_dir: ../.. +=== Batch Job on Cloud Foundry In this demonstration, you will learn how to orchestrate short-lived data processing application (_eg: Spring Batch Jobs_) using http://cloud.spring.io/spring-cloud-task/[Spring Cloud Task] and http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] on Cloud Foundry. -== Using Cloud Foundry Server +==== Prerequisites -=== Prerequisites - -In order to get started, make sure that you have the following components: - -* Local https://pivotal.io/pcf-dev[PCFDev] instance +* Local https://pivotal.io/pcf-dev[PCFDev] instance * Local install of https://github.com/cloudfoundry/cli[cf CLI] command line tool -* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] -* Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry Server] * Running instance of mysql in PCFDev +* A Running Data Flow Shell +include::{docs_dir}/shell.adoc[] +* The Spring Cloud Data Flow Cloud Foundry Server running in PCFDev +include::{docs_dir}/cloudfoundry-server.adoc[] + +==== Building and Running the Demo NOTE: PCF 1.7.12 or greater is required to run Tasks on Spring Cloud Data Flow. As of this writing, PCFDev and PWS supports builds upon this version. -=== Running the Sample in Cloud Foundry - -. Verify that CF instance is reachable -+ - -``` -→ cf api -api endpoint: https://api.local.pcfdev.io -api version: 2.75.0 - -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -No apps found -``` -+ . Task support needs to be enabled on pcf-dev. Being logged as `admin`, issue the following command: + ``` @@ -44,32 +28,14 @@ OK Feature task_creation Enabled. ``` -+ -. Follow the http://docs.spring.io/spring-cloud-dataflow-server-cloudfoundry/docs/current-SNAPSHOT/reference/htmlsingle/#getting-started[getting-started] instructions to deploy Spring Cloud Data Flow's Cloud Foundry Server - + NOTE: For this sample, all you need is the `mysql` service and in PCFDev, the `mysql` service comes with a different plan. From CF CLI, create the service by: `cf create-service p-mysql 512mb mysql` and bind this service to `dataflow-server` by: `cf bind-service dataflow-server mysql`. + NOTE: All the apps deployed to PCFDev start with low memory by default. It is recommended to change it to at least 768MB for `dataflow-server`. Ditto for every app spawned *by* Spring Cloud Data Flow. Change the memory by: `cf set-env dataflow-server SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_MEMORY 512`. Likewise, we would have to skip SSL validation by: `cf set-env dataflow-server SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SKIP_SSL_VALIDATION true`. -. Once you complete the instructions, you'll be able to list the newly deployed `dataflow-server` -+ - -``` -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -name requested state instances memory disk urls -dataflow-server started 1/1 512M 512M dataflow-server.local.pcfdev.io -``` -+ - -NOTE: The `ORG`, `SPACE` and `USER` could vary depending on your Cloud Foundry environment in use. - -. Tasks in Spring Cloud Data Flow require an RDBMS to host "task repository" (see http://docs.spring.io/spring-cloud-dataflow/docs/current-SNAPSHOT/reference/htmlsingle/#spring-cloud-dataflow-task-repository[here] for more details), so let's instruct the Spring Cloud Data Flow server to bind the `mysql` service to each deployed task: +. Tasks in Spring Cloud Data Flow require an RDBMS to host "task repository" (see http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-task-repository[here] for more details), so let's instruct the Spring Cloud Data Flow server to bind the `mysql` service to each deployed task: + @@ -106,43 +72,10 @@ No staging env variables have been set . Notice that `dataflow-server` application is started and ready for interaction via `http://dataflow-server.local.pcfdev.io` endpoint -. Connect to Spring Cloud Data Flow's `shell`. + - -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -server-unknown:> -``` -+ -. Connect `shell` with the `server` running at `http://dataflow-server.local.pcfdev.io` -+ - -``` -server-unknown:>dataflow config server http://dataflow-server.local.pcfdev.io -Successfully targeted http://dataflow-server.local.pcfdev.io -``` -+ - -. Let's build and register the batch-job https://github.com/spring-cloud/spring-cloud-task/tree/master/spring-cloud-task-samples/batch-job[example] from Spring Cloud Task samples. For convenience, the final https://github.com/spring-cloud/spring-cloud-dataflow-samples/raw/master/tasks/simple-batch-job/batch-job-1.0.0.BUILD-SNAPSHOT.jar[uber-jar artifact] is provided with this sample +. Build and register the batch-job https://github.com/spring-cloud/spring-cloud-task/tree/master/spring-cloud-task-samples/batch-job[example] from Spring Cloud Task samples. For convenience, the final https://github.com/spring-cloud/spring-cloud-dataflow-samples/raw/master/src/main/asciidoc/tasks/simple-batch-job/batch-job-1.0.0.BUILD-SNAPSHOT.jar[uber-jar artifact] is provided with this sample. + - ``` dataflow:>app register --type task --name simple_batch_job --uri https://github.com/spring-cloud/spring-cloud-dataflow-samples/raw/master/tasks/simple-batch-job/batch-job-1.3.0.BUILD-SNAPSHOT.jar ``` @@ -154,7 +87,7 @@ dataflow:>app register --type task --name simple_batch_job --uri https://github. ``` dataflow:>task create foo --definition "simple_batch_job" ``` -NOTE: Unlike Streams, the Task definitions don't require explicit deployment. They can be launched on-demand, scheduled, or triggered by streams. +NOTE: Unlike Streams, the Task definitions don't require explicit deployment. They can be launched on-demand, scheduled, or triggered by streams. + @@ -162,7 +95,7 @@ NOTE: Unlike Streams, the Task definitions don't require explicit deployment. Th + ``` -cf apps +$ cf apps Getting apps in org pcfdev-org / space pcfdev-space as user... OK @@ -176,7 +109,7 @@ dataflow-server started 1/1 768M 512M dataflow-server. + ``` -dataflow:>task launch foo +dataflow:>task launch foo ``` + @@ -184,8 +117,9 @@ dataflow:>task launch foo + -``` -→ cf logs foo +[source,console,options=nowrap] +---- +$ cf logs foo Retrieving logs for app foo in org pcfdev-org / space pcfdev-space as user... 2016-08-14T18:48:54.22-0700 [APP/TASK/foo/0]OUT Creating container @@ -214,12 +148,13 @@ Retrieving logs for app foo in org pcfdev-org / space pcfdev-space as user... 2016-08-14T18:49:07.71-0700 [APP/TASK/foo/0]OUT Exit status 0 2016-08-14T18:49:07.78-0700 [APP/TASK/foo/0]OUT Destroying container 2016-08-14T18:49:08.47-0700 [APP/TASK/foo/0]OUT Successfully destroyed container - -``` -NOTE: Verify `job1` and `job2` operations embeddded in `simple-batch-job` application are launched independently and they returned with the status `COMPLETED`. +---- ++ +NOTE: Verify `job1` and `job2` operations embeddded in `simple-batch-job` application are launched independently and they returned with the status `COMPLETED`. + ++ NOTE: Unlike LRPs in Cloud Foundry, tasks are short-lived, so the logs aren't always available. They are generated only when the Task application runs; at the end of Task operation, the container that ran the Task application is destroyed to free-up resources. + @@ -228,7 +163,7 @@ NOTE: Unlike LRPs in Cloud Foundry, tasks are short-lived, so the logs aren't al + ``` -→ cf apps +$ cf apps Getting apps in org pcfdev-org / space pcfdev-space as user... OK @@ -242,20 +177,22 @@ foo stopped 0/1 1G 1G + -``` +[source,console,options=nowrap] +---- dataflow:>task execution list ╔══════════════════════════╤══╤════════════════════════════╤════════════════════════════╤═════════╗ ║ Task Name │ID│ Start Time │ End Time │Exit Code║ ╠══════════════════════════╪══╪════════════════════════════╪════════════════════════════╪═════════╣ ║foo │1 │Sun Aug 14 18:49:05 PDT 2016│Sun Aug 14 18:49:07 PDT 2016│0 ║ ╚══════════════════════════╧══╧════════════════════════════╧════════════════════════════╧═════════╝ -``` +---- . Verify Job execution details + -``` +[source,console,options=nowrap] +---- dataflow:>job execution list ╔═══╤═══════╤═════════╤════════════════════════════╤═════════════════════╤══════════════════╗ ║ID │Task ID│Job Name │ Start Time │Step Execution Count │Definition Status ║ @@ -263,14 +200,14 @@ dataflow:>job execution list ║2 │1 │job2 │Sun Aug 14 18:49:07 PDT 2016│1 │Destroyed ║ ║1 │1 │job1 │Sun Aug 14 18:49:06 PDT 2016│1 │Destroyed ║ ╚═══╧═══════╧═════════╧════════════════════════════╧═════════════════════╧══════════════════╝ -``` +---- + -== Summary +==== Summary In this sample, you have learned: * How to register and orchestrate Spring Batch jobs in Spring Cloud Data Flow * How to use the `cf` CLI in the context of Task applications orchestrated by Spring Cloud Data Flow -* How to verify task executions and task repository +* How to verify task executions and task repository diff --git a/src/main/docbook/css/highlight.css b/src/main/docbook/css/highlight.css new file mode 100644 index 0000000..ffefef7 --- /dev/null +++ b/src/main/docbook/css/highlight.css @@ -0,0 +1,35 @@ +/* + code highlight CSS resemblign the Eclipse IDE default color schema + @author Costin Leau +*/ + +.hl-keyword { + color: #7F0055; + font-weight: bold; +} + +.hl-comment { + color: #3F5F5F; + font-style: italic; +} + +.hl-multiline-comment { + color: #3F5FBF; + font-style: italic; +} + +.hl-tag { + color: #3F7F7F; +} + +.hl-attribute { + color: #7F007F; +} + +.hl-value { + color: #2A00FF; +} + +.hl-string { + color: #2A00FF; +} \ No newline at end of file diff --git a/src/main/docbook/css/manual-multipage.css b/src/main/docbook/css/manual-multipage.css new file mode 100644 index 0000000..0c48453 --- /dev/null +++ b/src/main/docbook/css/manual-multipage.css @@ -0,0 +1,9 @@ +@IMPORT url("manual.css"); + +body.firstpage { + background: url("../images/background.png") no-repeat center top; +} + +div.part h1 { + border-top: none; +} diff --git a/src/main/docbook/css/manual-singlepage.css b/src/main/docbook/css/manual-singlepage.css new file mode 100644 index 0000000..4a7fd14 --- /dev/null +++ b/src/main/docbook/css/manual-singlepage.css @@ -0,0 +1,6 @@ +@IMPORT url("manual.css"); + +body { + background: url("../images/background.png") no-repeat center top; +} + diff --git a/src/main/docbook/css/manual.css b/src/main/docbook/css/manual.css new file mode 100644 index 0000000..2500583 --- /dev/null +++ b/src/main/docbook/css/manual.css @@ -0,0 +1,348 @@ +@IMPORT url("highlight.css"); + +html { + padding: 0pt; + margin: 0pt; +} + +body { + color: #333333; + margin: 15px 30px; + font-family: Helvetica, Arial, Freesans, Clean, Sans-serif; + line-height: 1.6; + -webkit-font-smoothing: antialiased; +} + +code { + font-size: 16px; + font-family: Consolas, "Liberation Mono", Courier, monospace; +} + +:not(a)>code { + color: #6D180B; +} + +:not(pre)>code { + background-color: #F2F2F2; + border: 1px solid #CCCCCC; + border-radius: 4px; + padding: 1px 3px 0; + text-shadow: none; + white-space: nowrap; +} + +body>*:first-child { + margin-top: 0 !important; +} + +div { + margin: 0pt; +} + +hr { + border: 1px solid #CCCCCC; + background: #CCCCCC; +} + +h1,h2,h3,h4,h5,h6 { + color: #000000; + cursor: text; + font-weight: bold; + margin: 30px 0 10px; + padding: 0; +} + +h1,h2,h3 { + margin: 40px 0 10px; +} + +h1 { + margin: 70px 0 30px; + padding-top: 20px; +} + +div.part h1 { + border-top: 1px dotted #CCCCCC; +} + +h1,h1 code { + font-size: 32px; +} + +h2,h2 code { + font-size: 24px; +} + +h3,h3 code { + font-size: 20px; +} + +h4,h1 code,h5,h5 code,h6,h6 code { + font-size: 18px; +} + +div.book,div.chapter,div.appendix,div.part,div.preface { + min-width: 300px; + max-width: 1200px; + margin: 0 auto; +} + +p.releaseinfo { + font-weight: bold; + margin-bottom: 40px; + margin-top: 40px; +} + +div.authorgroup { + line-height: 1; +} + +p.copyright { + line-height: 1; + margin-bottom: -5px; +} + +.legalnotice p { + font-style: italic; + font-size: 14px; + line-height: 1; +} + +div.titlepage+p,div.titlepage+p { + margin-top: 0; +} + +pre { + line-height: 1.0; + color: black; +} + +a { + color: #4183C4; + text-decoration: none; +} + +p { + margin: 15px 0; + text-align: left; +} + +ul,ol { + padding-left: 30px; +} + +li p { + margin: 0; +} + +div.table { + margin: 1em; + padding: 0.5em; + text-align: center; +} + +div.table table,div.informaltable table { + display: table; + width: 100%; +} + +div.table td { + padding-left: 7px; + padding-right: 7px; +} + +.sidebar { + line-height: 1.4; + padding: 0 20px; + background-color: #F8F8F8; + border: 1px solid #CCCCCC; + border-radius: 3px 3px 3px 3px; +} + +.sidebar p.title { + color: #6D180B; +} + +pre.programlisting,pre.screen { + font-size: 15px; + padding: 6px 10px; + background-color: #F8F8F8; + border: 1px solid #CCCCCC; + border-radius: 3px 3px 3px 3px; + clear: both; + overflow: auto; + line-height: 1.4; + font-family: Consolas, "Liberation Mono", Courier, monospace; +} + +table { + border-collapse: collapse; + border-spacing: 0; + border: 1px solid #DDDDDD !important; + border-radius: 4px !important; + border-collapse: separate !important; + line-height: 1.6; +} + +table thead { + background: #F5F5F5; +} + +table tr { + border: none; + border-bottom: none; +} + +table th { + font-weight: bold; +} + +table th,table td { + border: none !important; + padding: 6px 13px; +} + +table tr:nth-child(2n) { + background-color: #F8F8F8; +} + +td p { + margin: 0 0 15px 0; +} + +div.table-contents td p { + margin: 0; +} + +div.important *,div.note *,div.tip *,div.warning *,div.navheader *,div.navfooter *,div.calloutlist * + { + border: none !important; + background: none !important; + margin: 0; +} + +div.important p,div.note p,div.tip p,div.warning p { + color: #6F6F6F; + line-height: 1.6; +} + +div.important code,div.note code,div.tip code,div.warning code { + background-color: #F2F2F2 !important; + border: 1px solid #CCCCCC !important; + border-radius: 4px !important; + padding: 1px 3px 0 !important; + text-shadow: none !important; + white-space: nowrap !important; +} + +.note th,.tip th,.warning th { + display: none; +} + +.note tr:first-child td,.tip tr:first-child td,.warning tr:first-child td + { + border-right: 1px solid #CCCCCC !important; + padding-top: 10px; +} + +div.calloutlist p,div.calloutlist td { + padding: 0; + margin: 0; +} + +div.calloutlist>table>tbody>tr>td:first-child { + padding-left: 10px; + width: 30px !important; +} + +div.important,div.note,div.tip,div.warning { + margin-left: 0px !important; + margin-right: 20px !important; + margin-top: 20px; + margin-bottom: 20px; + padding-top: 10px; + padding-bottom: 10px; +} + +div.toc { + line-height: 1.2; +} + +dl,dt { + margin-top: 1px; + margin-bottom: 0; +} + +div.toc>dl>dt { + font-size: 32px; + font-weight: bold; + margin: 30px 0 10px 0; + display: block; +} + +div.toc>dl>dd>dl>dt { + font-size: 24px; + font-weight: bold; + margin: 20px 0 10px 0; + display: block; +} + +div.toc>dl>dd>dl>dd>dl>dt { + font-weight: bold; + font-size: 20px; + margin: 10px 0 0 0; +} + +tbody.footnotes * { + border: none !important; +} + +div.footnote p { + margin: 0; + line-height: 1; +} + +div.footnote p sup { + margin-right: 6px; + vertical-align: middle; +} + +div.navheader { + border-bottom: 1px solid #CCCCCC; +} + +div.navfooter { + border-top: 1px solid #CCCCCC; +} + +.title { + margin-left: -1em; + padding-left: 1em; +} + +.title>a { + position: absolute; + visibility: hidden; + display: block; + font-size: 0.85em; + margin-top: 0.05em; + margin-left: -1em; + vertical-align: text-top; + color: black; +} + +.title>a:before { + content: "\00A7"; +} + +.title:hover>a,.title>a:hover,.title:hover>a:hover { + visibility: visible; +} + +.title:focus>a,.title>a:focus,.title:focus>a:focus { + outline: 0; +} + +.mediaobject img { + max-width: 100%; +} diff --git a/src/main/docbook/images/background.png b/src/main/docbook/images/background.png new file mode 100644 index 0000000..d4195e5 Binary files /dev/null and b/src/main/docbook/images/background.png differ diff --git a/src/main/docbook/images/caution.png b/src/main/docbook/images/caution.png new file mode 100644 index 0000000..8a5e4fc Binary files /dev/null and b/src/main/docbook/images/caution.png differ diff --git a/src/main/docbook/images/cover.png b/src/main/docbook/images/cover.png new file mode 100644 index 0000000..b0db312 Binary files /dev/null and b/src/main/docbook/images/cover.png differ diff --git a/src/main/docbook/images/important.png b/src/main/docbook/images/important.png new file mode 100644 index 0000000..ec54df6 Binary files /dev/null and b/src/main/docbook/images/important.png differ diff --git a/src/main/docbook/images/logo.png b/src/main/docbook/images/logo.png new file mode 100644 index 0000000..bb1f4f9 Binary files /dev/null and b/src/main/docbook/images/logo.png differ diff --git a/src/main/docbook/images/logo.svg b/src/main/docbook/images/logo.svg new file mode 100644 index 0000000..e4624f7 --- /dev/null +++ b/src/main/docbook/images/logo.svg @@ -0,0 +1,16 @@ + + + + + + + + + + + + + + + + diff --git a/src/main/docbook/images/note.png b/src/main/docbook/images/note.png new file mode 100644 index 0000000..88d997b Binary files /dev/null and b/src/main/docbook/images/note.png differ diff --git a/src/main/docbook/images/tip.png b/src/main/docbook/images/tip.png new file mode 100644 index 0000000..6530abb Binary files /dev/null and b/src/main/docbook/images/tip.png differ diff --git a/src/main/docbook/images/warning.png b/src/main/docbook/images/warning.png new file mode 100644 index 0000000..0d5b524 Binary files /dev/null and b/src/main/docbook/images/warning.png differ diff --git a/src/main/docbook/xsl/common.xsl b/src/main/docbook/xsl/common.xsl new file mode 100644 index 0000000..157bf9d --- /dev/null +++ b/src/main/docbook/xsl/common.xsl @@ -0,0 +1,45 @@ + + + + + + + + 1 + 0 + 1 + + + + images/ + .png + + + book toc,title + 3 + + + + + diff --git a/src/main/docbook/xsl/epub.xsl b/src/main/docbook/xsl/epub.xsl new file mode 100644 index 0000000..031406c --- /dev/null +++ b/src/main/docbook/xsl/epub.xsl @@ -0,0 +1,31 @@ + + + + + + + + + + diff --git a/src/main/docbook/xsl/html-multipage.xsl b/src/main/docbook/xsl/html-multipage.xsl new file mode 100644 index 0000000..be9cc52 --- /dev/null +++ b/src/main/docbook/xsl/html-multipage.xsl @@ -0,0 +1,73 @@ + + + + + + + + + + css/manual-multipage.css + + '5' + '1' + + + + + + + + + + + + + + + + + + + + firstpage + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/main/docbook/xsl/html-singlepage.xsl b/src/main/docbook/xsl/html-singlepage.xsl new file mode 100644 index 0000000..6bd4ac8 --- /dev/null +++ b/src/main/docbook/xsl/html-singlepage.xsl @@ -0,0 +1,30 @@ + + + + + + + + + + css/manual-singlepage.css + + diff --git a/src/main/docbook/xsl/html.xsl b/src/main/docbook/xsl/html.xsl new file mode 100644 index 0000000..fd96f9a --- /dev/null +++ b/src/main/docbook/xsl/html.xsl @@ -0,0 +1,141 @@ + + + + + + + + + + + 1 + + + 1 + + + + 120 + images/callouts/ + .png + + + text/css + + text-align: left + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + , + + + + + + + +
+

Authors

+ +
+ + + + + + + + + + + + + + + + + + # + + + + + + + diff --git a/src/main/docbook/xsl/pdf.xsl b/src/main/docbook/xsl/pdf.xsl new file mode 100644 index 0000000..3074094 --- /dev/null +++ b/src/main/docbook/xsl/pdf.xsl @@ -0,0 +1,591 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + auto + + + + + underline + #204060 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + , + + + + + + + + + + + + + Copyright © + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + -5em + -5em + 8pt + + + + + + + + + + + + + + + please define title in your docbook file! + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 8pt + + + + + + + + + + + + + + + + + + + + + + + + + please define title in your docbook file! + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + 0 + 0 + + + + false + + + Helvetica + 10 + 8 + Helvetica + + + 1.4 + + + + left + bold + + + pt + + + + + + + + + + + + + + + 0.6em + 0.6em + 0.6em + + + pt + + 0.1em + 0.1em + 0.1em + + + + 0.4em + 0.4em + 0.4em + + + pt + + 0.1em + 0.1em + 0.1em + + + + 0.4em + 0.4em + 0.4em + + + pt + + 0.1em + 0.1em + 0.1em + + + + 0.3em + 0.3em + 0.3em + + + pt + + 0.1em + 0.1em + 0.1em + + + + + + + + 4pt + 4pt + 4pt + 4pt + + + + 0.1pt + 0.1pt + + + + + + + + + + + + + + + + 7pt + wrap + 1 + + + + 1em + 1em + 1em + 0.1em + 0.1em + 0.1em + + #444444 + solid + 0.1pt + 0.5em + 0.5em + 0.5em + 0.5em + 0.5em + 0.5em + + + + 1 + + #F0F0F0 + + + + 0.1em + 0.1em + 0.1em + 0.1em + 0.1em + 0.1em + + + + 0.5em + 0.5em + 0.5em + 0.1em + 0.1em + 0.1em + + + + #444444 + solid + 0.1pt + #F0F0F0 + + + + + + + normal + italic + + + pt + + false + 0.1em + 0.1em + 0.1em + + + + + + 0 + 1 + + + 90 + + + + + + figure after + example after + equation before + table before + procedure before + + + + 1 + 0pt + + + + + + + + + + + + + + + + + + + + 18pt + + + + 0.1em + 2em + .75pt + solid + #5c5c4f + 0.5em + 1.5em + 1.5em + 1.5em + 1.5em + 1.5em + 1.5em + + + + 10pt + bold + false + always + 0 + + + + 0em + 0em + 0em + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/main/docbook/xsl/xslthl-config.xml b/src/main/docbook/xsl/xslthl-config.xml new file mode 100644 index 0000000..e4d677f --- /dev/null +++ b/src/main/docbook/xsl/xslthl-config.xml @@ -0,0 +1,23 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/main/docbook/xsl/xslthl/asciidoc-hl.xml b/src/main/docbook/xsl/xslthl/asciidoc-hl.xml new file mode 100644 index 0000000..5478b1d --- /dev/null +++ b/src/main/docbook/xsl/xslthl/asciidoc-hl.xml @@ -0,0 +1,41 @@ + + + + + //// + //// + + + // + + + + ^(={1,6} .+)$ + + MULTILINE + + + ^(\.[^\.\s].+)$ + + MULTILINE + + + ^(:!?\w.*?:) + + MULTILINE + + + ^(-|\*{1,5}|\d*\.{1,5})(?= .+$) + + MULTILINE + + + ^(\[.+\])$ + + MULTILINE + + diff --git a/src/main/docbook/xsl/xslthl/bourne-hl.xml b/src/main/docbook/xsl/xslthl/bourne-hl.xml new file mode 100644 index 0000000..e2cd98d --- /dev/null +++ b/src/main/docbook/xsl/xslthl/bourne-hl.xml @@ -0,0 +1,95 @@ + + + + # + + << + ' + " + - + + + + + " + \ + + + ' + \ + + + + 0x + + + + . + + + + + + if + then + else + elif + fi + case + esac + for + while + until + do + done + + exec + shift + exit + times + break + export + trap + continue + readonly + wait + eval + return + + cd + echo + hash + pwd + read + set + test + type + ulimit + umask + unset + + diff --git a/src/main/docbook/xsl/xslthl/c-hl.xml b/src/main/docbook/xsl/xslthl/c-hl.xml new file mode 100644 index 0000000..176cc37 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/c-hl.xml @@ -0,0 +1,117 @@ + + + + + /** + */ + + + + + + + + /* + */ + + // + + + # + \ + + + + + " + \ + + + ' + \ + + + 0x + ul + lu + u + l + + + + . + + e + ul + lu + u + f + l + + + + auto + _Bool + break + case + char + _Complex + const + continue + default + do + double + else + enum + extern + float + for + goto + if + _Imaginary + inline + int + long + register + restrict + return + short + signed + sizeof + static + struct + switch + typedef + union + unsigned + void + volatile + while + + diff --git a/src/main/docbook/xsl/xslthl/cpp-hl.xml b/src/main/docbook/xsl/xslthl/cpp-hl.xml new file mode 100644 index 0000000..ef83c4f --- /dev/null +++ b/src/main/docbook/xsl/xslthl/cpp-hl.xml @@ -0,0 +1,151 @@ + + + + + /** + */ + + + + + + + + /* + */ + + // + + + # + \ + + + + + " + \ + + + ' + \ + + + 0x + ul + lu + u + l + + + + . + + e + ul + lu + u + f + l + + + + + auto + _Bool + break + case + char + _Complex + const + continue + default + do + double + else + enum + extern + float + for + goto + if + _Imaginary + inline + int + long + register + restrict + return + short + signed + sizeof + static + struct + switch + typedef + union + unsigned + void + volatile + while + + asm + dynamic_cast + namespace + reinterpret_cast + try + bool + explicit + new + static_cast + typeid + catch + false + operator + template + typename + class + friend + private + this + using + const_cast + inline + public + throw + virtual + delete + mutable + protected + true + wchar_t + + diff --git a/src/main/docbook/xsl/xslthl/csharp-hl.xml b/src/main/docbook/xsl/xslthl/csharp-hl.xml new file mode 100644 index 0000000..d57e631 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/csharp-hl.xml @@ -0,0 +1,194 @@ + + + + + /** + */ + + + + /// + + + + /* + */ + + // + + + [ + ] + ( + ) + + + + # + \ + + + + + + @" + " + \ + + + + " + \ + + + ' + \ + + + 0x + ul + lu + u + l + + + + . + + e + ul + lu + u + f + d + m + l + + + + abstract + as + base + bool + break + byte + case + catch + char + checked + class + const + continue + decimal + default + delegate + do + double + else + enum + event + explicit + extern + false + finally + fixed + float + for + foreach + goto + if + implicit + in + int + interface + internal + is + lock + long + namespace + new + null + object + operator + out + override + params + private + protected + public + readonly + ref + return + sbyte + sealed + short + sizeof + stackalloc + static + string + struct + switch + this + throw + true + try + typeof + uint + ulong + unchecked + unsafe + ushort + using + virtual + void + volatile + while + + + + add + alias + from + get + global + group + into + join + orderby + partial + remove + select + set + value + where + yield + + diff --git a/src/main/docbook/xsl/xslthl/css-hl.xml b/src/main/docbook/xsl/xslthl/css-hl.xml new file mode 100644 index 0000000..164c48c --- /dev/null +++ b/src/main/docbook/xsl/xslthl/css-hl.xml @@ -0,0 +1,176 @@ + + + + + /* + */ + + + " + \ + + + + ' + \ + + + + . + + + + @charset + @import + @media + @page + + + + - + azimuth + background-attachment + background-color + background-image + background-position + background-repeat + background + border-collapse + border-color + border-spacing + border-style + border-top + border-right + border-bottom + border-left + border-top-color + border-right-color + border-bottom-color + border-left-color + border-top-style + border-right-style + border-bottom-style + border-left-style + border-top-width + border-right-width + border-bottom-width + border-left-width + border-width + border + bottom + caption-side + clear + clip + color + content + counter-increment + counter-reset + cue-after + cue-before + cue + cursor + direction + display + elevation + empty-cells + float + font-family + font-size + font-style + font-variant + font-weight + font + height + left + letter-spacing + line-height + list-style-image + list-style-position + list-style-type + list-style + margin-right + margin-left + margin-top + margin-bottom + margin + max-height + max-width + min-height + min-width + orphans + outline-color + outline-style + outline-width + outline + overflow + padding-top + padding-right + padding-bottom + padding-left + padding + page-break-after + page-break-before + page-break-inside + pause-after + pause-before + pause + pitch-range + pitch + play-during + position + quotes + richness + right + speak-header + speak-numeral + speak-punctuation + speak + speech-rate + stress + table-layout + text-align + text-decoration + text-indent + text-transform + top + unicode-bidi + vertical-align + visibility + voice-family + volume + white-space + widows + width + word-spacing + z-index + + diff --git a/src/main/docbook/xsl/xslthl/html-hl.xml b/src/main/docbook/xsl/xslthl/html-hl.xml new file mode 100644 index 0000000..5b6761b --- /dev/null +++ b/src/main/docbook/xsl/xslthl/html-hl.xml @@ -0,0 +1,122 @@ + + + + + + + a + abbr + address + area + article + aside + audio + b + base + bdi + blockquote + body + br + button + caption + canvas + cite + code + command + col + colgroup + dd + del + dialog + div + dl + dt + em + embed + fieldset + figcaption + figure + font + form + footer + h1 + h2 + h3 + h4 + h5 + h6 + head + header + hr + html + i + iframe + img + input + ins + kbd + label + legend + li + link + map + mark + menu + menu + meta + nav + noscript + object + ol + optgroup + option + p + param + pre + q + samp + script + section + select + small + source + span + strong + style + sub + summary + sup + table + tbody + td + textarea + tfoot + th + thead + time + title + tr + track + u + ul + var + video + wbr + xmp + + + + + xsl: + + + diff --git a/src/main/docbook/xsl/xslthl/ini-hl.xml b/src/main/docbook/xsl/xslthl/ini-hl.xml new file mode 100644 index 0000000..34c1036 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/ini-hl.xml @@ -0,0 +1,45 @@ + + + + ; + + + ^(\[.+\]\s*)$ + + MULTILINE + + + + ^(.+)(?==) + + MULTILINE + + diff --git a/src/main/docbook/xsl/xslthl/java-hl.xml b/src/main/docbook/xsl/xslthl/java-hl.xml new file mode 100644 index 0000000..f7bb164 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/java-hl.xml @@ -0,0 +1,117 @@ + + + + + /** + */ + + + + /* + */ + + // + + " + \ + + + ' + \ + + + @ + ( + ) + + + 0x + + + + . + e + f + d + l + + + + abstract + boolean + break + byte + case + catch + char + class + const + continue + default + do + double + else + extends + final + finally + float + for + goto + if + implements + import + instanceof + int + interface + long + native + new + package + private + protected + public + return + short + static + strictfp + super + switch + synchronized + this + throw + throws + transient + try + void + volatile + while + + diff --git a/src/main/docbook/xsl/xslthl/javascript-hl.xml b/src/main/docbook/xsl/xslthl/javascript-hl.xml new file mode 100644 index 0000000..99b8a71 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/javascript-hl.xml @@ -0,0 +1,147 @@ + + + + + /* + */ + + // + + " + \ + + + ' + \ + + + 0x + + + + . + e + + + + break + case + catch + continue + default + delete + do + else + finally + for + function + if + in + instanceof + new + return + switch + this + throw + try + typeof + var + void + while + with + + abstract + boolean + byte + char + class + const + debugger + double + enum + export + extends + final + float + goto + implements + import + int + interface + long + native + package + private + protected + public + short + static + super + synchronized + throws + transient + volatile + + + prototype + + Array + Boolean + Date + Error + EvalError + Function + Math + Number + Object + RangeError + ReferenceError + RegExp + String + SyntaxError + TypeError + URIError + + decodeURI + decodeURIComponent + encodeURI + encodeURIComponent + eval + isFinite + isNaN + parseFloat + parseInt + + Infinity + NaN + undefined + + diff --git a/src/main/docbook/xsl/xslthl/json-hl.xml b/src/main/docbook/xsl/xslthl/json-hl.xml new file mode 100644 index 0000000..59b9c48 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/json-hl.xml @@ -0,0 +1,37 @@ + + + # + + " + \ + + + ' + \ + + + @ + ( + ) + + + . + e + f + d + l + + + + true + false + + + { + } + , + [ + ] + + + diff --git a/src/main/docbook/xsl/xslthl/perl-hl.xml b/src/main/docbook/xsl/xslthl/perl-hl.xml new file mode 100644 index 0000000..73d71cc --- /dev/null +++ b/src/main/docbook/xsl/xslthl/perl-hl.xml @@ -0,0 +1,120 @@ + + + + # + + << + ' + " + + + + " + \ + + + ' + \ + + + + 0x + + + + . + + + + + if + unless + while + until + foreach + else + elsif + for + when + default + given + + caller + continue + die + do + dump + eval + exit + goto + last + next + redo + return + sub + wantarray + + caller + import + local + my + package + use + + do + import + no + package + require + use + + bless + dbmclose + dbmopen + package + ref + tie + tied + untie + use + + and + or + not + eq + ne + lt + gt + le + ge + cmp + + diff --git a/src/main/docbook/xsl/xslthl/php-hl.xml b/src/main/docbook/xsl/xslthl/php-hl.xml new file mode 100644 index 0000000..1da25b8 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/php-hl.xml @@ -0,0 +1,154 @@ + + + + + /** + */ + + + + + + + + /* + */ + + // + # + + " + \ + + + + ' + \ + + + + <<< + + + 0x + + + + . + e + + + + and + or + xor + __FILE__ + exception + __LINE__ + array + as + break + case + class + const + continue + declare + default + die + do + echo + else + elseif + empty + enddeclare + endfor + endforeach + endif + endswitch + endwhile + eval + exit + extends + for + foreach + function + global + if + include + include_once + isset + list + new + print + require + require_once + return + static + switch + unset + use + var + while + __FUNCTION__ + __CLASS__ + __METHOD__ + final + php_user_filter + interface + implements + extends + public + private + protected + abstract + clone + try + catch + throw + cfunction + old_function + true + false + + namespace + __NAMESPACE__ + goto + __DIR__ + + + + + ?> + <?php + <?= + + + diff --git a/src/main/docbook/xsl/xslthl/properties-hl.xml b/src/main/docbook/xsl/xslthl/properties-hl.xml new file mode 100644 index 0000000..775f2f1 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/properties-hl.xml @@ -0,0 +1,38 @@ + + + + # + + ^(.+?)(?==|:) + + MULTILINE + + diff --git a/src/main/docbook/xsl/xslthl/python-hl.xml b/src/main/docbook/xsl/xslthl/python-hl.xml new file mode 100644 index 0000000..a467443 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/python-hl.xml @@ -0,0 +1,100 @@ + + + + + + @ + ( + ) + + # + + """ + + + + ''' + + + + " + \ + + + ' + \ + + + 0x + l + + + + . + + e + l + + + + and + del + from + not + while + as + elif + global + or + with + assert + else + if + pass + yield + break + except + import + print + class + exec + in + raise + continue + finally + is + return + def + for + lambda + try + + diff --git a/src/main/docbook/xsl/xslthl/ruby-hl.xml b/src/main/docbook/xsl/xslthl/ruby-hl.xml new file mode 100644 index 0000000..d105640 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/ruby-hl.xml @@ -0,0 +1,109 @@ + + + + # + + << + + + + " + \ + + + %Q{ + } + \ + + + %/ + / + \ + + + ' + \ + + + %q{ + } + \ + + + 0x + + + + . + e + + + + alias + and + BEGIN + begin + break + case + class + def + defined + do + else + elsif + END + end + ensure + false + for + if + in + module + next + nil + not + or + redo + rescue + retry + return + self + super + then + true + undef + unless + until + when + while + yield + + diff --git a/src/main/docbook/xsl/xslthl/sql2003-hl.xml b/src/main/docbook/xsl/xslthl/sql2003-hl.xml new file mode 100644 index 0000000..ac1d5d0 --- /dev/null +++ b/src/main/docbook/xsl/xslthl/sql2003-hl.xml @@ -0,0 +1,565 @@ + + + + -- + + /* + */ + + + ' + + + + U' + ' + + + + B' + ' + + + + N' + ' + + + + X' + ' + + + + . + + e + + + + + + A + ABS + ABSOLUTE + ACTION + ADA + ADMIN + AFTER + ALWAYS + ASC + ASSERTION + ASSIGNMENT + ATTRIBUTE + ATTRIBUTES + AVG + BEFORE + BERNOULLI + BREADTH + C + CARDINALITY + CASCADE + CATALOG_NAME + CATALOG + CEIL + CEILING + CHAIN + CHAR_LENGTH + CHARACTER_LENGTH + CHARACTER_SET_CATALOG + CHARACTER_SET_NAME + CHARACTER_SET_SCHEMA + CHARACTERISTICS + CHARACTERS + CHECKED + CLASS_ORIGIN + COALESCE + COBOL + CODE_UNITS + COLLATION_CATALOG + COLLATION_NAME + COLLATION_SCHEMA + COLLATION + COLLECT + COLUMN_NAME + COMMAND_FUNCTION_CODE + COMMAND_FUNCTION + COMMITTED + CONDITION_NUMBER + CONDITION + CONNECTION_NAME + CONSTRAINT_CATALOG + CONSTRAINT_NAME + CONSTRAINT_SCHEMA + CONSTRAINTS + CONSTRUCTORS + CONTAINS + CONVERT + CORR + COUNT + COVAR_POP + COVAR_SAMP + CUME_DIST + CURRENT_COLLATION + CURSOR_NAME + DATA + DATETIME_INTERVAL_CODE + DATETIME_INTERVAL_PRECISION + DEFAULTS + DEFERRABLE + DEFERRED + DEFINED + DEFINER + DEGREE + DENSE_RANK + DEPTH + DERIVED + DESC + DESCRIPTOR + DIAGNOSTICS + DISPATCH + DOMAIN + DYNAMIC_FUNCTION_CODE + DYNAMIC_FUNCTION + EQUALS + EVERY + EXCEPTION + EXCLUDE + EXCLUDING + EXP + EXTRACT + FINAL + FIRST + FLOOR + FOLLOWING + FORTRAN + FOUND + FUSION + G + GENERAL + GO + GOTO + GRANTED + HIERARCHY + IMPLEMENTATION + INCLUDING + INCREMENT + INITIALLY + INSTANCE + INSTANTIABLE + INTERSECTION + INVOKER + ISOLATION + K + KEY_MEMBER + KEY_TYPE + KEY + LAST + LENGTH + LEVEL + LN + LOCATOR + LOWER + M + MAP + MATCHED + MAX + MAXVALUE + MESSAGE_LENGTH + MESSAGE_OCTET_LENGTH + MESSAGE_TEXT + MIN + MINVALUE + MOD + MORE + MUMPS + NAME + NAMES + NESTING + NEXT + NORMALIZE + NORMALIZED + NULLABLE + NULLIF + NULLS + NUMBER + OBJECT + OCTET_LENGTH + OCTETS + OPTION + OPTIONS + ORDERING + ORDINALITY + OTHERS + OVERLAY + OVERRIDING + PAD + PARAMETER_MODE + PARAMETER_NAME + PARAMETER_ORDINAL_POSITION + PARAMETER_SPECIFIC_CATALOG + PARAMETER_SPECIFIC_NAME + PARAMETER_SPECIFIC_SCHEMA + PARTIAL + PASCAL + PATH + PERCENT_RANK + PERCENTILE_CONT + PERCENTILE_DISC + PLACING + PLI + POSITION + POWER + PRECEDING + PRESERVE + PRIOR + PRIVILEGES + PUBLIC + RANK + READ + RELATIVE + REPEATABLE + RESTART + RETURNED_CARDINALITY + RETURNED_LENGTH + RETURNED_OCTET_LENGTH + RETURNED_SQLSTATE + ROLE + ROUTINE_CATALOG + ROUTINE_NAME + ROUTINE_SCHEMA + ROUTINE + ROW_COUNT + ROW_NUMBER + SCALE + SCHEMA_NAME + SCHEMA + SCOPE_CATALOG + SCOPE_NAME + SCOPE_SCHEMA + SECTION + SECURITY + SELF + SEQUENCE + SERIALIZABLE + SERVER_NAME + SESSION + SETS + SIMPLE + SIZE + SOURCE + SPACE + SPECIFIC_NAME + SQRT + STATE + STATEMENT + STDDEV_POP + STDDEV_SAMP + STRUCTURE + STYLE + SUBCLASS_ORIGIN + SUBSTRING + SUM + TABLE_NAME + TABLESAMPLE + TEMPORARY + TIES + TOP_LEVEL_COUNT + TRANSACTION_ACTIVE + TRANSACTION + TRANSACTIONS_COMMITTED + TRANSACTIONS_ROLLED_BACK + TRANSFORM + TRANSFORMS + TRANSLATE + TRIGGER_CATALOG + TRIGGER_NAME + TRIGGER_SCHEMA + TRIM + TYPE + UNBOUNDED + UNCOMMITTED + UNDER + UNNAMED + USAGE + USER_DEFINED_TYPE_CATALOG + USER_DEFINED_TYPE_CODE + USER_DEFINED_TYPE_NAME + USER_DEFINED_TYPE_SCHEMA + VIEW + WORK + WRITE + ZONE + + ADD + ALL + ALLOCATE + ALTER + AND + ANY + ARE + ARRAY + AS + ASENSITIVE + ASYMMETRIC + AT + ATOMIC + AUTHORIZATION + BEGIN + BETWEEN + BIGINT + BINARY + BLOB + BOOLEAN + BOTH + BY + CALL + CALLED + CASCADED + CASE + CAST + CHAR + CHARACTER + CHECK + CLOB + CLOSE + COLLATE + COLUMN + COMMIT + CONNECT + CONSTRAINT + CONTINUE + CORRESPONDING + CREATE + CROSS + CUBE + CURRENT_DATE + CURRENT_DEFAULT_TRANSFORM_GROUP + CURRENT_PATH + CURRENT_ROLE + CURRENT_TIME + CURRENT_TIMESTAMP + CURRENT_TRANSFORM_GROUP_FOR_TYPE + CURRENT_USER + CURRENT + CURSOR + CYCLE + DATE + DAY + DEALLOCATE + DEC + DECIMAL + DECLARE + DEFAULT + DELETE + DEREF + DESCRIBE + DETERMINISTIC + DISCONNECT + DISTINCT + DOUBLE + DROP + DYNAMIC + EACH + ELEMENT + ELSE + END + END-EXEC + ESCAPE + EXCEPT + EXEC + EXECUTE + EXISTS + EXTERNAL + FALSE + FETCH + FILTER + FLOAT + FOR + FOREIGN + FREE + FROM + FULL + FUNCTION + GET + GLOBAL + GRANT + GROUP + GROUPING + HAVING + HOLD + HOUR + IDENTITY + IMMEDIATE + IN + INDICATOR + INNER + INOUT + INPUT + INSENSITIVE + INSERT + INT + INTEGER + INTERSECT + INTERVAL + INTO + IS + ISOLATION + JOIN + LANGUAGE + LARGE + LATERAL + LEADING + LEFT + LIKE + LOCAL + LOCALTIME + LOCALTIMESTAMP + MATCH + MEMBER + MERGE + METHOD + MINUTE + MODIFIES + MODULE + MONTH + MULTISET + NATIONAL + NATURAL + NCHAR + NCLOB + NEW + NO + NONE + NOT + NULL + NUMERIC + OF + OLD + ON + ONLY + OPEN + OR + ORDER + OUT + OUTER + OUTPUT + OVER + OVERLAPS + PARAMETER + PARTITION + PRECISION + PREPARE + PRIMARY + PROCEDURE + RANGE + READS + REAL + RECURSIVE + REF + REFERENCES + REFERENCING + REGR_AVGX + REGR_AVGY + REGR_COUNT + REGR_INTERCEPT + REGR_R2 + REGR_SLOPE + REGR_SXX + REGR_SXY + REGR_SYY + RELEASE + RESULT + RETURN + RETURNS + REVOKE + RIGHT + ROLLBACK + ROLLUP + ROW + ROWS + SAVEPOINT + SCROLL + SEARCH + SECOND + SELECT + SENSITIVE + SESSION_USER + SET + SIMILAR + SMALLINT + SOME + SPECIFIC + SPECIFICTYPE + SQL + SQLEXCEPTION + SQLSTATE + SQLWARNING + START + STATIC + SUBMULTISET + SYMMETRIC + SYSTEM_USER + SYSTEM + TABLE + THEN + TIME + TIMESTAMP + TIMEZONE_HOUR + TIMEZONE_MINUTE + TO + TRAILING + TRANSLATION + TREAT + TRIGGER + TRUE + UESCAPE + UNION + UNIQUE + UNKNOWN + UNNEST + UPDATE + UPPER + USER + USING + VALUE + VALUES + VAR_POP + VAR_SAMP + VARCHAR + VARYING + WHEN + WHENEVER + WHERE + WIDTH_BUCKET + WINDOW + WITH + WITHIN + WITHOUT + YEAR + + diff --git a/src/main/docbook/xsl/xslthl/yaml-hl.xml b/src/main/docbook/xsl/xslthl/yaml-hl.xml new file mode 100644 index 0000000..a28008e --- /dev/null +++ b/src/main/docbook/xsl/xslthl/yaml-hl.xml @@ -0,0 +1,47 @@ + + + # + + " + \ + + + ' + \ + + + @ + ( + ) + + + . + e + f + d + l + + + + true + false + + + { + } + , + [ + ] + + + + ^(---)$ + + MULTILINE + + + ^(.+?)(?==|:) + + MULTILINE + + diff --git a/streaming/http-to-cassandra/README.adoc b/streaming/http-to-cassandra/README.adoc deleted file mode 100644 index 58ecca1..0000000 --- a/streaming/http-to-cassandra/README.adoc +++ /dev/null @@ -1,295 +0,0 @@ -:sectnums: -= HTTP to Cassandra Demo - -In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an _HTTP_ endpoint and write the payload to a _Cassandra_ database. - -We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `server` Spring Boot application. We will deploy the `server` using https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-server-local[Local] as well as https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry] SPIs (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _dev-sandbox_ and _cloud-native_ platform capabilities respectively. - -== Using Local Server - -=== Prerequisites - -Make sure that you have the following components: - -* Local build of link:https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] -* Running instance of link:http://kafka.apache.org/downloads.html[Kafka] -* Running instance of link:http://cassandra.apache.org/[Apache Cassandra] -* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using. -* Create a keyspace and a `book` table in Cassandra using: -+ -``` -CREATE KEYSPACE clouddata WITH REPLICATION = { 'class' : 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1' } AND DURABLE_WRITES = true; -USE clouddata; -CREATE TABLE book ( - id uuid PRIMARY KEY, - isbn text, - author text, - title text -); -``` - -=== Running the Sample Locally - -. Launch the locally built `server` application -+ -``` -$ cd -$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-.jar - -``` -+ - -. Connect to Spring Cloud Data Flow's `shell` -+ -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -dataflow:>version - -``` - -+ -. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] Kafka binder variant of out-of-the-box applications -+ - -``` -dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven -``` - -+ -. Create the stream -+ -``` -dataflow:>stream create cassandrastream --definition "http --server.port=8888 --spring.cloud.stream.bindings.output.contentType='application/json' | cassandra --ingestQuery='insert into book (id, isbn, title, author) values (uuid(), ?, ?, ?)' --keyspace=clouddata" --deploy - -Created and deployed new stream 'cassandrastream' -``` -NOTE: If Cassandra isn't running on default port on `localhost` or if you need username and password to connect, use one of the following options to specify the necessary connection parameters: `--username='' --password='' --port= --contact-points=` - -+ -. Verify the stream is successfully deployed -+ -``` -dataflow:>stream list -``` -+ -. Notice that `cassandrastream-http` and `cassandrastream-cassandra` link:https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the `server` as a collocated process. -+ - -``` -2015-12-15 15:52:31.576 INFO 18337 --- [nio-9393-exec-1] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:cassandra-sink:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 - Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/cassandrastream.cassandra -2015-12-15 15:52:31.583 INFO 18337 --- [nio-9393-exec-1] o.s.c.d.a.s.l.OutOfProcessModuleDeployer : deploying module org.springframework.cloud.stream.module:http-source:jar:exec:1.0.0.BUILD-SNAPSHOT instance 0 - Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-data-flow-284240942697761420/cassandrastream.http -``` -+ -. Post sample data pointing to the `http` endpoint: `http://localhost:8888` [`8888` is the `server.port` we specified for the `http` source in this case] -+ -``` -dataflow:>http post --contentType 'application/json' --data '{"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"}' --target http://localhost:8888 -> POST (application/json;charset=UTF-8) http://localhost:8888 {"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"} -> 202 ACCEPTED -``` -+ -. Connect to the Cassandra instance and query the table `clouddata.book` to list the persisted records -+ -``` -select * from clouddata.book; -``` - -+ -. That's it; you're done! - - -== Using Cloud Foundry Server - -=== Prerequisites - -In order to get started, make sure that you have the following components: - -* Cloud Foundry instance -* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] -* Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry Server] -* Running instance of `rabbit` in Cloud Foundry -* Running instance of `cassandra` in Cloud Foundry or from another Cloud provider -* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] to connect to the Cassandra instance. You might have to provide `host`, `port`, `username` and `password` depending on the Cassandra configuration you are using. -* Create a `book` table in your Cassandra keyspace using: -+ -``` -CREATE TABLE book ( - id uuid PRIMARY KEY, - isbn text, - author text, - title text -); -``` - - -=== Running the Sample in Cloud Foundry - -. Verify that CF instance is reachable -+ - -``` -$ cf api -API endpoint: https://api.system.io (API version: 2.43.0) - -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -No apps found -``` -+ -. Follow the instructions to deploy Spring Cloud Data Flow's `server` from https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry/blob/master/README.adoc[Cloud Foundry Server] repo - -+ -. Once you complete step#3 from https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry/blob/master/README.adoc[Cloud Foundry Server] instructions, you'll be able to list the newly deployed `dataflow-server` application in Cloud Foundry -+ - -``` -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -name requested state instances memory disk urls -dataflow-server started 1/1 1G 1G dataflow-server.app.io -``` - -+ -. Notice that `dataflow-server` application is started and ready for interaction via `http://dataflow-server.app.io` endpoint - -. Connect to Spring Cloud Data Flow's `shell`. -+ - -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -server-unknown:> -``` -+ -. Connect the `shell` with `server` running at `http://dataflow-server.app.io` -+ - -``` -server-unknown:>dataflow config server http://dataflow-server.app.io -Successfully targeted http://dataflow-server.app.io -dataflow:>version - -``` - -+ -. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] RabbitMQ binder variant of out-of-the-box applications -+ - -``` -dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven -``` - -+ -. Create the stream -+ - -``` -dataflow:>stream create cassandrastream --definition "http --spring.cloud.stream.bindings.output.contentType='application/json' | cassandra --ingestQuery='insert into book (id, isbn, title, author) values (uuid(), ?, ?, ?)' --username='' --password='' --port= --contact-points= --keyspace=''" --deploy - -Created and deployed new stream 'cassandrastream' -``` -+ -. Verify the stream is successfully deployed -+ -``` -dataflow:>stream list -``` -+ -. Notice that `cassandrastream-http` and `cassandrastream-cassandra` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry -+ - -``` -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -name requested state instances memory disk urls -cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.io -cassandrastream-http started 1/1 1G 1G cassandrastream-http.app.io -dataflow-server started 1/1 1G 1G dataflow-server.app.io -``` -+ -. Lookup the `url` for `cassandrastream-http` application from the list above. Post sample data pointing to the `http` endpoint: `` -+ -``` -http post --contentType 'application/json' --data '{"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"}' --target http:// -> POST (application/json;charset=UTF-8) http://cassandrastream-http.app.io {"isbn": "1599869772", "title": "The Art of War", "author": "Sun Tzu"} -> 202 ACCEPTED -``` -+ -. Connect to the Cassandra instance and query the table `book` to list the data inserted -+ -``` -select * from book; -``` - -+ -. Now, let's try to take advantage of Pivotal Cloud Foundry's platform capability. Let's scale the `cassandrastream-http` application from 1 to 3 instances -+ -``` -$ cf scale cassandrastream-http -i 3 -Scaling app cassandrastream-http in org user-dataflow / space development as user... -OK -``` -+ -. Verify App instances (3/3) running successfully -+ -``` -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -name requested state instances memory disk urls -cassandrastream-cassandra started 1/1 1G 1G cassandrastream-cassandra.app.io -cassandrastream-http started 3/3 1G 1G cassandrastream-http.app.io -dataflow-server started 1/1 1G 1G dataflow-server.app.io -``` -+ -. That's it; you're done! - -:!sectnums: -== Summary - -In this sample, you have learned: - -* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers -* How to use Spring Cloud Data Flow's `shell` -* How to create streaming data pipeline to connect and write to `Cassandra` -* How to scale data microservice applications on `Pivotal Cloud Foundry` diff --git a/streaming/http-to-mysql/README.adoc b/streaming/http-to-mysql/README.adoc deleted file mode 100644 index 904b22e..0000000 --- a/streaming/http-to-mysql/README.adoc +++ /dev/null @@ -1,297 +0,0 @@ -:sectnums: -= HTTP to MySQL Demo - -In this demonstration, you will learn how to orchestrate a data pipeline using http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] to consume data from an `http` endpoint and write to MySQL database using `jdbc` sink. - -We will begin by discussing the steps to prep, configure and operationalize Spring Cloud Data Flow's `server` Spring Boot application. We will deploy the `server` using https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-server-local[Local] as well as https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry] SPIs (Service Provider Interface) to demonstrate how Spring Cloud Data Flow takes advantage of _dev-sandbox_ and _cloud-native_ platform capabilities, respectively. - -== Using Local Server - -=== Prerequisites - -Make sure that you have the following components: - -* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] -* Running instance of link:http://kafka.apache.org/downloads.html[Kafka] -* Running instance of link:http://www.mysql.com/[MySQL] -* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] or link:https://www.dbvis.com/[DbVisualizer] -* Create the `test` database with a `names` table (in MySQL) using: -+ -``` -CREATE DATABASE test; -USE test; -CREATE TABLE names -( - name varchar(255) -); -``` - -=== Running the Sample Locally - -. Launch the locally built `server` -+ - -``` -$ cd -$ java -jar spring-cloud-dataflow-server-local/target/spring-cloud-dataflow-server-local-.jar - -``` -+ - -. Connect to Spring Cloud Data Flow's `shell` -+ - -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -dataflow:>version - -``` - -+ -. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] Kafka binder variant of out-of-the-box applications -+ - -``` -dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-maven -``` - -+ -. Create the stream -+ -``` -dataflow:>stream create --name mysqlstream --definition "http --server.port=8787 | jdbc --tableName=names --columns=name --spring.datasource.driver-class-name=org.mariadb.jdbc.Driver --spring.datasource.url='jdbc:mysql://localhost:3306/test'" --deploy - -Created and deployed new stream 'mysqlstream' -``` -NOTE: If MySQL isn't running on default port on `localhost` or if you need username and password to connect, use one of the following options to specify the necessary connection parameters: `--spring.datasource.url='jdbc:mysql://:/' --spring.datasource.username= --spring.datasource.password=` - -+ -. Verify the stream is successfully deployed -+ -``` -dataflow:>stream list -``` -+ -. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters//[Spring Cloud Stream] applications are running as Spring Boot applications within the Local `server` as collocated processes. -+ - -``` -2016-05-03 09:29:55.918 INFO 65162 --- [nio-9393-exec-3] o.s.c.d.spi.local.LocalAppDeployer : deploying app mysqlstream.jdbc instance 0 - Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-dataflow-6850863945840320040/mysqlstream1-1462292995903/mysqlstream.jdbc -2016-05-03 09:29:55.939 INFO 65162 --- [nio-9393-exec-3] o.s.c.d.spi.local.LocalAppDeployer : deploying app mysqlstream.http instance 0 - Logs will be in /var/folders/c3/ctx7_rns6x30tq7rb76wzqwr0000gp/T/spring-cloud-dataflow-6850863945840320040/mysqlstream-1462292995934/mysqlstream.http -``` - -. Post sample data pointing to the `http` endpoint: `http://localhost:8787` [`8787` is the `server.port` we specified for the `http` source in this case] - -+ -``` -dataflow:>http post --contentType 'application/json' --target http://localhost:8787 --data "{\"name\": \"Foo\"}" -> POST (application/json;charset=UTF-8) http://localhost:8787 {"name": "Spring Boot"} -> 202 ACCEPTED -``` -+ -. Connect to the MySQL instance and query the table `test.names` to list the new rows: -+ -``` -select * from test.names; -``` -+ -. That's it; you're done! - -== Using Cloud Foundry Server - -=== Prerequisites - -In order to get started, make sure that you have the following components: - -* Cloud Foundry instance -* Local build of https://github.com/spring-cloud/spring-cloud-dataflow[Spring Cloud Data Flow] -* Local build of Spring Cloud Data Flow's https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry Server] -* Running instance of `rabbit` in Cloud Foundry -* Running instance of `mysql` in Cloud Foundry -* A database utility tool such as link:http://dbeaver.jkiss.org/[DBeaver] or link:https://www.dbvis.com/[DbVisualizer] -* Create the `names` table (in MySQL) using: -+ -``` -CREATE TABLE names -( - name varchar(255) -); -``` - -=== Running the Sample in Cloud Foundry - -. Verify that CF instance is reachable -+ - -``` -$ cf api -API endpoint: https://api.system.io (API version: 2.43.0) - -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -No apps found -``` -+ -. Follow the instructions to deploy Spring Cloud Data Flow's `server` from https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry/blob/master/README.adoc[Cloud Foundry Server] repo - -+ -. Once you complete step#3 from https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry/blob/master/README.adoc[Cloud Foundry Server] instructions, you'll be able to list the newly deployed `dataflow-server` application in Cloud Foundry -+ - -``` -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -name requested state instances memory disk urls -dataflow-server started 1/1 1G 1G dataflow-server.app.io -``` - -+ -. Notice that `dataflow-server` application is started and ready for interaction via `http://dataflow-server.app.io` endpoint - -. Connect to Spring Cloud Data Flow's `shell` -+ - -``` -$ cd -$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-.jar - - ____ ____ _ __ - / ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| | - \___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` | - ___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| | - |____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_| - ____ |_| _ __|___/ __________ - | _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \ - | | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \ - | |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / / - |____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/ - - - -Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help". -server-unknown:> -``` -+ -. Connect the `shell` with `server` running at `http://dataflow-server.app.io` -+ - -``` -server-unknown:>dataflow config server http://dataflow-server.app.io -Successfully targeted http://dataflow-server.app.io -dataflow:>version - -``` - -+ -. https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/streams.adoc#register-a-stream-app[Register] RabbitMQ binder variant of out-of-the-box applications -+ - -``` -dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-rabbit-maven -``` - -+ -. Create the stream -+ - -``` -dataflow:>stream create --name mysqlstream --definition "http | jdbc --tableName=names --columns=name" -Created new stream 'mysqlstream' - -dataflow:>stream deploy --name mysqlstream --properties "app.jdbc.spring.cloud.deployer.cloudfoundry.services=mysql" -Deployed stream 'mysqlstream' - -``` -+ - -NOTE: By supplying `mysql` property through `app.jdbc.spring.cloud.deployer.cloudfoundry.services` token, we are deploying the stream with `jdbc-sink` to automatically bind to `mysql` service and only this application in the stream gets the service binding. This also eliminates the requirement to supply `datasource` credentials in stream definition. -+ -. Verify the stream is successfully deployed -+ -``` -dataflow:>stream list -``` -+ -. Notice that `mysqlstream-http` and `mysqlstream-jdbc` https://github.com/spring-cloud-stream-app-starters/[Spring Cloud Stream] applications are running as _cloud-native_ (microservice) applications in Cloud Foundry -+ - -``` -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -name requested state instances memory disk urls -mysqlstream-http started 1/1 1G 1G mysqlstream-http.app.io -mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.io -dataflow-server started 1/1 1G 1G dataflow-server.app.io -``` -+ -. Lookup the `url` for `mysqlstream-http` application from the list above. Post sample data pointing to the `http` endpoint: `` -+ -``` -http post --contentType 'application/json' --target http://mysqlstream-http.app.io --data "{\"name\": \"Bar"}" -> POST (application/json;charset=UTF-8) http://mysqlstream-http.app.io {"name": "Bar"} -> 202 ACCEPTED -``` -+ -. Connect to the MySQL instance and query the table `names` to list the new rows: -+ -``` -select * from names; -``` - -+ -. Now, let's take advantage of Pivotal Cloud Foundry's platform capability. Let's scale the `mysqlstream-http` application from 1 to 3 instances -+ -``` -$ cf scale mysqlstream-http -i 3 -Scaling app mysqlstream-http in org user-dataflow / space development as user... -OK -``` -+ -. Verify App instances (3/3) running successfully -+ -``` -$ cf apps -Getting apps in org user-dataflow / space development as user... -OK - -name requested state instances memory disk urls -mysqlstream-http started 3/3 1G 1G mysqlstream-http.app.io -mysqlstream-jdbc started 1/1 1G 1G mysqlstream-jdbc.app.io -dataflow-server started 1/1 1G 1G dataflow-server.app.io -``` -+ -. That's it; you're done! - -:!sectnums: -== Summary - -In this sample, you have learned: - -* How to use Spring Cloud Data Flow's `Local` and `Cloud Foundry` servers -* How to use Spring Cloud Data Flow's `shell` -* How to create streaming data pipeline to connect and write to `MySQL` -* How to scale data microservice applications on `Pivotal Cloud Foundry`