233 lines
11 KiB
Plaintext
233 lines
11 KiB
Plaintext
== Stream applications
|
|
|
|
This repository provides a collection of components that help with various data integration use cases.
|
|
It provides standalone Java functions that can be reused in end-user applications.
|
|
|
|
In addition to providing these functions, this repository is also used to generate out of the box Spring Cloud Stream applications those using the functions as a baseline.
|
|
These applications can be run standalone as Spring Cloud Stream applications or as part of a data flow such as the one orchestrated using Spring Cloud Data Flow.
|
|
|
|
The repository is divided into two sections - `Functions` and `Applications`. The former is used for hosting the various functions and the latter is for generating the apps.
|
|
|
|
The following are the four major components of this repository.
|
|
|
|
* https://github.com/spring-cloud/stream-applications/tree/master/functions[Standalone Java Functions] as java.util.function.Function/Consumer/Supplier
|
|
* https://github.com/spring-cloud/stream-applications/tree/master/applications/stream-applications-core[Core components] for the out of the box applications
|
|
* https://github.com/spring-cloud/stream-applications/tree/master/applications[Out of the box Spring Cloud Stream applications] as Source/Sink/Processor
|
|
* https://github.com/spring-cloud/stream-applications/tree/master/applications/stream-applications-build[Aggregator] for the out of the box applications.
|
|
|
|
=== Available Functions
|
|
|
|
|
|
|===
|
|
|Supplier |Consumer |Function
|
|
|
|
|link:functions/supplier/file-supplier/README.adoc[File]
|
|
|link:functions/consumer/cassandra-consumer/README.adoc[Cassandra]
|
|
|link:functions/function/filter-function/README.adoc[Filter]
|
|
|link:functions/supplier/ftp-supplier/README.adoc[FTP]
|
|
|link:functions/consumer/counter-consumer/README.adoc[Counter]
|
|
|link:functions/function/header-enricher-function/README.adoc[Header-Enricher]
|
|
|link:functions/supplier/geode-supplier/README.adoc[Geode]
|
|
|link:functions/consumer/file-consumer/README.adoc[File]
|
|
|link:functions/function/http-request-function/README.adoc[HTTP Request]
|
|
|link:functions/supplier/http-supplier/README.adoc[HTTP]
|
|
|link:functions/consumer/ftp-consumer/README.adoc[FTP]
|
|
|
|
|
|link:functions/supplier/jdbc-supplier/README.adoc[JDBC]
|
|
|link:functions/consumer/geode-consumer/README.adoc[Geode]
|
|
|link:functions/function/spel-function/README.adoc[SpEL]
|
|
|link:functions/supplier/jms-supplier/README.adoc[JMS]
|
|
|link:functions/consumer/jdbc-consumer/README.adoc[JDBC]
|
|
|link:functions/function/splitter-function/README.adoc[Splitter]
|
|
|link:functions/supplier/mongodb-supplier/README.adoc[MongoDB]
|
|
|link:functions/consumer/log-consumer/README.adoc[Log]
|
|
|link:functions/function/task-launch-request-function/README.adoc[Task Launch Request]
|
|
|link:functions/supplier/mqtt-supplier/README.adoc[MQTT]
|
|
|link:functions/consumer/mongodb-consumer/README.adoc[MongoDB]
|
|
|link:functions/function/tasklauncher-function/README.adoc[Task Launcher]
|
|
|link:functions/supplier/rabbit-supplier/README.adoc[RabbitMQ]
|
|
|link:functions/consumer/mqtt-consumer/README.adoc[MQTT]
|
|
|link:functions/function/twitter-function/README.adoc[Twitter]
|
|
|link:functions/supplier/s3-supplier/README.adoc[AWS S3]
|
|
|link:functions/consumer/rabbit-consumer/README.adoc[RabbitMQ]
|
|
|
|
|
|link:functions/supplier/tcp-supplier/README.adoc[TCP]
|
|
|link:functions/consumer/redis-consumer/README.adoc[Redis]
|
|
|
|
|
|link:functions/supplier/time-supplier/README.adoc[Time]
|
|
|link:functions/consumer/s3-consumer/README.adoc[AWS S3]
|
|
|
|
|
|link:functions/supplier/twitter-supplier/README.adoc[Twitter]
|
|
|link:functions/consumer/sftp-consumer/README.adoc[SFTP]
|
|
|
|
|
|link:functions/supplier/websocket-supplier/README.adoc[Websocket]
|
|
|link:functions/consumer/tcp-consumer/README.adoc[TCP]
|
|
|
|
|
|
|
|
|link:functions/consumer/twitter-consumer/README.adoc[Twitter]
|
|
|
|
|
|
|
|
|link:functions/consumer/websocket-consumer/README.adoc[Websocker]
|
|
|
|
|
|===
|
|
|
|
=== Available Spring Cloud Stream Applications
|
|
|
|
|===
|
|
|Source |Sink |Processor
|
|
|
|
|link:applications/source/file-source/README.adoc[File]
|
|
|link:applications/sink/cassandra-sink/README.adoc[Cassandra]
|
|
|link:applications/processor/bridge-processor/README.adoc[Bridge]
|
|
|link:applications/source/ftp-source/README.adoc[FTP]
|
|
|link:applications/sink/counter-sink/README.adoc[Counter]
|
|
|link:applications/processor/filter-processor/README.adoc[Filter]
|
|
|link:applications/source/geode-source/README.adoc[Geode]
|
|
|link:applications/sink/file-sink/README.adoc[Fiile]
|
|
|link:applications/processor/groovy-processor/README.adoc[Groovy]
|
|
|link:applications/source/http-source/README.adoc[HTTP]
|
|
|link:applications/sink/ftp-sink/README.adoc[FTP]
|
|
|link:applications/processor/header-enricher-processor/README.adoc[Header-Enricher]
|
|
|link:applications/source/jdbc-source/README.adoc[JDBC]
|
|
|link:applications/sink/geode-sink/README.adoc[Geode]
|
|
|link:applications/processor/http-request-processor/README.adoc[HTTP Request]
|
|
|link:applications/source/jms-source/README.adoc[JMS]
|
|
|link:applications/sink/jdbc-sink/README.adoc[JDBC]
|
|
|link:applications/processor/script-processor/README.adoc[Script]
|
|
|link:applications/source/load-generator-source/README.adoc[Load-Generator]
|
|
|link:applications/sink/log-sink/README.adoc[Log]
|
|
|link:applications/processor/splitter-processor/README.adoc[Splitter]
|
|
|link:applications/source/mongodb-source/README.adoc[MongoDB]
|
|
|link:applications/sink/mongodb-sink/README.adoc[MongoDB]
|
|
|link:applications/processor/transform-processor/README.adoc[Transform]
|
|
|link:applications/source/mqtt-source/README.adoc[MQTT]
|
|
|link:applications/sink/mqtt-sink/README.adoc[MQTT]
|
|
|link:applications/processor/twitter-trend-processor/README.adoc[Twitter Trend]
|
|
|link:applications/source/rabbit-source/README.adoc[RabbitMQ]
|
|
|link:applications/sink/rabbit-sink/README.adoc[RabbitMQ]
|
|
|
|
|
|link:applications/source/s3-source/README.adoc[AWS S3]
|
|
|link:applications/sink/redis-sink/README.adoc[Redis]
|
|
|
|
|
|link:applications/source/tcp-source/README.adoc[TCP]
|
|
|link:applications/sink/router-sink/README.adoc[Router]
|
|
|
|
|
|link:applications/source/time-source/README.adoc[Time]
|
|
|link:applications/sink/sftp-sink/README.adoc[SFTP]
|
|
|
|
|
|link:applications/source/twitter-message-source/README.adoc[Twitter Message]
|
|
|link:applications/sink/tasklauncher-sink/README.adoc[Task Launcher]
|
|
|
|
|
|link:applications/source/twitter-search-source/README.adoc[Twitter Search]
|
|
|link:applications/sink/tcp-sink/README.adoc[TCP]
|
|
|
|
|
|link:applications/source/twitter-stream-source/README.adoc[Twitter Stream]
|
|
|link:applications/sink/throughput-sink/README.adoc[Throughput]
|
|
|
|
|
|link:applications/source/websocket-source/README.adoc[Websocket]
|
|
|link:applications/sink/twitter-message-sink/README.adoc[Twitter Message]
|
|
|
|
|
|
|
|
|link:applications/sink/twitter-update-sink/README.adoc[Twitter Update]
|
|
|
|
|
|===
|
|
|
|
=== Composite Function Support
|
|
|
|
By default, the `source` applications are auto-configured with link:functions/function[functions] which may optionally be included in a composite function definition.
|
|
This feature enables the sources to do things like
|
|
- execute SpEL transformations
|
|
- enrich message headers
|
|
- filter events
|
|
- produce task launch requests
|
|
|
|
or any combination of the above, without requiring a separate processor.
|
|
|
|
For example, the time source executed, as shown below, will perform a series of transformations to publish a task launch request every second to the rabbit exchange `time-test`.
|
|
|
|
```
|
|
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar --spring.cloud.stream.bindings.output.destination=time-test --spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" --spel.function.expression="payload.length()" --header.enricher.headers=task-id=payload*2 --task.launch.request.task-name-expression="'task-'+headers['task-id']"
|
|
```
|
|
|
|
The transformed message looks like this:
|
|
|
|
```
|
|
headers:
|
|
task-id: 34
|
|
content_type: application/json
|
|
Payload
|
|
49 bytes
|
|
Encoding: string
|
|
{"args":[],"deploymentProps":{},"name":"task-34"}
|
|
```
|
|
|
|
Let's upack this function definition:
|
|
|
|
`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction`
|
|
|
|
This creates a composed Supplier beginning with the default `timeSupplier` which is the foundation for `time-source`.
|
|
This produces a String like `06/18/20 16:01:38`. We transform this using the SpEL expression `payload.length()`.
|
|
The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, in accordance with common Spring Integration conventions.
|
|
|
|
The output of `spelFunction` is the length of the date-time String, `17`.
|
|
From here we apply the header enricher to add a Message header, `task-id` with the value of `payload*2` or ,`34`.
|
|
We use the `task-id` header to generate the task name for the task launch request using the SpEL expression "'task-'+headers['task-id']", or `task-34`.
|
|
|
|
This somewhat contrived example is intended to show the power of function composition.
|
|
Even so, if you have `task-34` as a task definition in Data Flow, you could build a simple pipeline `time | tasklauncher` to launch the task every second.
|
|
|
|
Prior to `3.0` release of Stream Applications, this composition required extensive customization.
|
|
|
|
Please see the individual function documentation for an explanation of its configuration properties.
|
|
|
|
NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`) derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`.
|
|
In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured property
|
|
`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`.
|
|
|
|
=== Build
|
|
|
|
You can build everything from the root of the repository.
|
|
|
|
`./mvnw clean install`
|
|
|
|
But, this may not be what you are interested in doing since you are probably interested in a single application or a few of them.
|
|
In order to build the functions and applications that you are interested in, you need to build them selectively as shown below.
|
|
|
|
==== Building functions
|
|
|
|
`./mvnw clean install -f functions`
|
|
|
|
You can also build a single function or group of functions.
|
|
For e.g if you are only interested in jdbc-supplier and log-consumer, do the following.
|
|
|
|
`./mvnw clean install -pl :jdbc-suppler,:log-consumer`
|
|
|
|
==== Building core for Stream Applications
|
|
|
|
`./mvnw clean install -f applications/stream-applications-core`
|
|
|
|
=== Building the applications
|
|
|
|
Let's assume that you want to build JDBC Source application based on Kafka Binder in Spring Cloud Stream and Log Sink application based on Rabbit binder.
|
|
Here is what you need to do.
|
|
Assuming that you built both functions and stream-applications-core as above.
|
|
|
|
```
|
|
./mvnw clean package -pl :jdbc-source
|
|
cd applications/source/jdbc-source/apps/jdbc-source-kafka
|
|
./mvnw clean package
|
|
```
|
|
|
|
This will generate the Kafka binder based uber jar in the target folder.
|
|
|
|
Similarly for the log sink, do the following.
|
|
|
|
```
|
|
./mvnw clean package -pl :log-sink
|
|
cd applications/sink/log-sink/apps/log-sink-rabbit
|
|
./mvnw clean package
|
|
```
|
|
|
|
=== Code of Conduct
|
|
|
|
Please see our https://github.com/spring-projects/.github/blob/master/CODE_OF_CONDUCT.md[Code of Conduct] |