== Stream applications
This repository provides a collection of components that help with various data integration use cases.
It provides standalone Java functions that can be reused in end-user applications.
In addition to providing these functions, this repository is also used to generate out of the box Spring Cloud Stream applications those using the functions as a baseline.
These applications can be run standalone as Spring Cloud Stream applications or as part of a data flow such as the one orchestrated using Spring Cloud Data Flow.
The repository is divided into two sections - `Functions` and `Applications`. The former is used for hosting the various functions and the latter is for generating the apps.
The following are the four major components of this repository.
* https://github.com/spring-cloud/stream-applications/tree/master/functions[Standalone Java Functions] as java.util.function.Function/Consumer/Supplier
* https://github.com/spring-cloud/stream-applications/tree/master/applications/stream-applications-core[Core components] for the out of the box applications
* https://github.com/spring-cloud/stream-applications/tree/master/applications[Out of the box Spring Cloud Stream applications] as Source/Sink/Processor
* https://github.com/spring-cloud/stream-applications/tree/master/applications/stream-applications-build[Aggregator] for the out of the box applications.
=== Available Functions
|===
|Supplier |Consumer |Function
|link:functions/supplier/file-supplier/README.adoc[File]
|link:functions/consumer/cassandra-consumer/README.adoc[Cassandra]
|link:functions/function/filter-function/README.adoc[Filter]
|link:functions/supplier/ftp-supplier/README.adoc[FTP]
|link:functions/consumer/counter-consumer/README.adoc[Counter]
|link:functions/function/header-enricher-function/README.adoc[Header-Enricher]
|link:functions/supplier/geode-supplier/README.adoc[Geode]
|link:functions/consumer/file-consumer/README.adoc[File]
|link:functions/function/http-request-function/README.adoc[HTTP Request]
|link:functions/supplier/http-supplier/README.adoc[HTTP]
|link:functions/consumer/ftp-consumer/README.adoc[FTP]
|
|link:functions/supplier/jdbc-supplier/README.adoc[JDBC]
|link:functions/consumer/geode-consumer/README.adoc[Geode]
|link:functions/function/spel-function/README.adoc[SpEL]
|link:functions/supplier/jms-supplier/README.adoc[JMS]
|link:functions/consumer/jdbc-consumer/README.adoc[JDBC]
|link:functions/function/splitter-function/README.adoc[Splitter]
|link:functions/supplier/mongodb-supplier/README.adoc[MongoDB]
|link:functions/consumer/log-consumer/README.adoc[Log]
|link:functions/function/task-launch-request-function/README.adoc[Task Launch Request]
|link:functions/supplier/mqtt-supplier/README.adoc[MQTT]
|link:functions/consumer/mongodb-consumer/README.adoc[MongoDB]
|link:functions/function/tasklauncher-function/README.adoc[Task Launcher]
|link:functions/supplier/rabbit-supplier/README.adoc[RabbitMQ]
|link:functions/consumer/mqtt-consumer/README.adoc[MQTT]
|link:functions/function/twitter-function/README.adoc[Twitter]
|link:functions/supplier/s3-supplier/README.adoc[AWS S3]
|link:functions/consumer/rabbit-consumer/README.adoc[RabbitMQ]
|
|link:functions/supplier/tcp-supplier/README.adoc[TCP]
|link:functions/consumer/redis-consumer/README.adoc[Redis]
|
|link:functions/supplier/time-supplier/README.adoc[Time]
|link:functions/consumer/s3-consumer/README.adoc[AWS S3]
|
|link:functions/supplier/twitter-supplier/README.adoc[Twitter]
|link:functions/consumer/sftp-consumer/README.adoc[SFTP]
|
|link:functions/supplier/websocket-supplier/README.adoc[Websocket]
|link:functions/consumer/tcp-consumer/README.adoc[TCP]
|
|
|link:functions/consumer/twitter-consumer/README.adoc[Twitter]
|
|
|link:functions/consumer/websocket-consumer/README.adoc[Websocker]
|
|===
=== Available Spring Cloud Stream Applications
|===
|Source |Sink |Processor
|link:applications/source/file-source/README.adoc[File]
|link:applications/sink/cassandra-sink/README.adoc[Cassandra]
|link:applications/processor/bridge-processor/README.adoc[Bridge]
|link:applications/source/ftp-source/README.adoc[FTP]
|link:applications/sink/counter-sink/README.adoc[Counter]
|link:applications/processor/filter-processor/README.adoc[Filter]
|link:applications/source/geode-source/README.adoc[Geode]
|link:applications/sink/file-sink/README.adoc[Fiile]
|link:applications/processor/groovy-processor/README.adoc[Groovy]
|link:applications/source/http-source/README.adoc[HTTP]
|link:applications/sink/ftp-sink/README.adoc[FTP]
|link:applications/processor/header-enricher-processor/README.adoc[Header-Enricher]
|link:applications/source/jdbc-source/README.adoc[JDBC]
|link:applications/sink/geode-sink/README.adoc[Geode]
|link:applications/processor/http-request-processor/README.adoc[HTTP Request]
|link:applications/source/jms-source/README.adoc[JMS]
|link:applications/sink/jdbc-sink/README.adoc[JDBC]
|link:applications/processor/script-processor/README.adoc[Script]
|link:applications/source/load-generator-source/README.adoc[Load-Generator]
|link:applications/sink/log-sink/README.adoc[Log]
|link:applications/processor/splitter-processor/README.adoc[Splitter]
|link:applications/source/mongodb-source/README.adoc[MongoDB]
|link:applications/sink/mongodb-sink/README.adoc[MongoDB]
|link:applications/processor/transform-processor/README.adoc[Transform]
|link:applications/source/mqtt-source/README.adoc[MQTT]
|link:applications/sink/mqtt-sink/README.adoc[MQTT]
|link:applications/processor/twitter-trend-processor/README.adoc[Twitter Trend]
|link:applications/source/rabbit-source/README.adoc[RabbitMQ]
|link:applications/sink/rabbit-sink/README.adoc[RabbitMQ]
|
|link:applications/source/s3-source/README.adoc[AWS S3]
|link:applications/sink/redis-sink/README.adoc[Redis]
|
|link:applications/source/tcp-source/README.adoc[TCP]
|link:applications/sink/router-sink/README.adoc[Router]
|
|link:applications/source/time-source/README.adoc[Time]
|link:applications/sink/sftp-sink/README.adoc[SFTP]
|
|link:applications/source/twitter-message-source/README.adoc[Twitter Message]
|link:applications/sink/tasklauncher-sink/README.adoc[Task Launcher]
|
|link:applications/source/twitter-search-source/README.adoc[Twitter Search]
|link:applications/sink/tcp-sink/README.adoc[TCP]
|
|link:applications/source/twitter-stream-source/README.adoc[Twitter Stream]
|link:applications/sink/throughput-sink/README.adoc[Throughput]
|
|link:applications/source/websocket-source/README.adoc[Websocket]
|link:applications/sink/twitter-message-sink/README.adoc[Twitter Message]
|
|
|link:applications/sink/twitter-update-sink/README.adoc[Twitter Update]
|
|===
=== Composite Function Support
By default, the `source` applications are auto-configured with link:functions/function[functions] which may optionally be included in a composite function definition.
This feature enables the sources to do things like
- execute SpEL transformations
- enrich message headers
- filter events
- produce task launch requests
or any combination of the above, without requiring a separate processor.
For example, the time source executed, as shown below, will perform a series of transformations to publish a task launch request every second to the rabbit exchange `time-test`.
```
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar --spring.cloud.stream.bindings.output.destination=time-test --spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" --spel.function.expression="payload.length()" --header.enricher.headers=task-id=payload*2 --task.launch.request.task-name-expression="'task-'+headers['task-id']"
```
The transformed message looks like this:
```
headers:
task-id: 34
content_type: application/json
Payload
49 bytes
Encoding: string
{"args":[],"deploymentProps":{},"name":"task-34"}
```
Let's upack this function definition:
`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction`
This creates a composed Supplier beginning with the default `timeSupplier` which is the foundation for `time-source`.
This produces a String like `06/18/20 16:01:38`. We transform this using the SpEL expression `payload.length()`.
The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, in accordance with common Spring Integration conventions.
The output of `spelFunction` is the length of the date-time String, `17`.
From here we apply the header enricher to add a Message header, `task-id` with the value of `payload*2` or ,`34`.
We use the `task-id` header to generate the task name for the task launch request using the SpEL expression "'task-'+headers['task-id']", or `task-34`.
This somewhat contrived example is intended to show the power of function composition.
Even so, if you have `task-34` as a task definition in Data Flow, you could build a simple pipeline `time | tasklauncher` to launch the task every second.
Prior to `3.0` release of Stream Applications, this composition required extensive customization.
Please see the individual function documentation for an explanation of its configuration properties.
NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`) derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`.
In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured property
`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`.
=== Build
You can build everything from the root of the repository.
`./mvnw clean install`
But, this may not be what you are interested in doing since you are probably interested in a single application or a few of them.
In order to build the functions and applications that you are interested in, you need to build them selectively as shown below.
==== Building functions
`./mvnw clean install -f functions`
You can also build a single function or group of functions.
For e.g if you are only interested in jdbc-supplier and log-consumer, do the following.
`./mvnw clean install -pl :jdbc-suppler,:log-consumer`
==== Building core for Stream Applications
`./mvnw clean install -f applications/stream-applications-core`
=== Building the applications
Let's assume that you want to build JDBC Source application based on Kafka Binder in Spring Cloud Stream and Log Sink application based on Rabbit binder.
Here is what you need to do.
Assuming that you built both functions and stream-applications-core as above.
```
./mvnw clean package -pl :jdbc-source
cd applications/source/jdbc-source/apps/jdbc-source-kafka
./mvnw clean package
```
This will generate the Kafka binder based uber jar in the target folder.
Similarly for the log sink, do the following.
```
./mvnw clean package -pl :log-sink
cd applications/sink/log-sink/apps/log-sink-rabbit
./mvnw clean package
```
=== Code of Conduct
Please see our https://github.com/spring-projects/.github/blob/master/CODE_OF_CONDUCT.md[Code of Conduct]