SpringCloud/stream-applications
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

README changes

Browse Source 
* Adding steps for PR contributions in the README
This commit is contained in:
Soby Chacko
2020-07-10 20:21:35 -04:00
committed by David Turanski
parent b7d41bf04c
commit 4a5bb50659
3 changed files with 185 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
README.adoc
View File

@@ -145,84 +145,6 @@ The following are the four major components of this repository.
|link:applications/sink/wavefront-sink/README.adoc[Wavefront]
|===
=== Composable Functions
Spring Cloud Stream includes integration with Spring Cloud Function's function-based programming model that lets the
business logic of an application be modeled as a `java.util.Supplier`, a `java.util.Function`, and a `java.util.Consumer`,
representing the roles of a `Source`, a `Processor`, and a `Sink`, respectively.
Building on this foundation, we can extend existing `Source` and `Sink` applications by importing the configuration of an
existing `Source` or `Sink` and adding code that defines a `java.util.Function` — this delivers a lot of powerful composition
possibilities.
Take for instance, the `Source` applications are auto-configured with link:functions/function[functions], which may optionally
be included in a composite function definition.
With this, the same Source application can potentially do one or all of the following without having to build it out as a
standalone processor.
- execute SpEL transformations
- enrich message headers
- filter events
- produce task launch requests on upstream events
For example, the `time` source when it is running, as shown below, will perform a series of internal transformations to
finally publish a task launch request every second to the rabbit exchange with the name `time-test`.
```
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar \
--spring.cloud.stream.bindings.output.destination=time-test \
--spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" \
--spel.function.expression="payload.length()" \
--header.enricher.headers=task-id=payload*2 \
--task.launch.request.task-name-expression="'task-'+headers['task-id']"
```
Now, the transformed message would look like:
```
headers:
task-id: 34
content_type: application/json
Payload
49 bytes
Encoding: string
{"args":[],"deploymentProps":{},"name":"task-34"}
```
Let us unpack what is happening behind the scenes. We will start with the following function definition.
`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction`
- Here, the function definition creates a composite `Supplier` beginning with the default `timeSupplier` Java function included
in this repository, which is the foundation for `time-source`.
- The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, by following
standard Spring Integration conventions.
- The output of `spelFunction` is the length of the date-time String, `17`.
- From here, we apply the header-enricher Java function to add a Message header, `task-id` with the value of `payload*2`.
That would be `34`.
- We use the `task-id` in the header to generate the "task name", and to programmatically derive the task launch request
using the SpEL expression "'task-'+headers['task-id']", or `task-34`.
This somewhat contrived example, but the goal here was to highlight the power of function composition.
If you have had a task definition in Spring Cloud Data Flow with the name `task-34`, you could build a `time | tasklauncher`
streaming data pipeline to launch that task every second.
Before the `3.0` release of Stream Applications, this composition required extensive customization. And a lot more manual
configuration changes, extensions, and custom build of the applications.
NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`)
derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`.
In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured
property
`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`.
=== Build
You can build everything from the root of the repository.
@@ -248,7 +170,7 @@ If you want to re-run the common core build, you can build it with the following
`./mvnw clean install -f applications/stream-applications-core`
=== Building Stream applications
=== Building Stream Applications
Let's assume that you want to build a `jdbc-source` application based on Kafka Binder in Spring Cloud Stream and Log Sink
application based on Rabbit binder.
@@ -272,6 +194,13 @@ cd applications/sink/log-sink/apps/log-sink-rabbit
./mvnw clean package
```
=== Additional Resources
Here is a list of resources where you can find out more about using and developing functions and stream applications:
* link:docs/FunctionComposition.adoc[Function Composition]
* link:docs/Contributing.adoc[Contributing a New Function or Application to this Repository]
=== Code of Conduct
Please see our https://github.com/spring-projects/.github/blob/master/CODE_OF_CONDUCT.md[Code of Conduct]

100
docs/Contributing.adoc Normal file
View File

@@ -0,0 +1,100 @@
= Contributing a new function component to this repository
If you do not see a function or application that you are interested in, we encourage you to contribute to this repository.
In most cases, you start by writing the function.
Once the function is ready, you set up a Maven project for the corresponding stream application, provide the pom, and configuration properties metadata, and some basic tests.
If you have followed the prescribed structure and naming conventions, the Maven build will generate the application.
Here are some things you need to know before contributing a new function.
== Creating a function
What kind of function do you want to write? If you are not sure, consult the table below.
We expect the majority of contributions to be either suppliers or consumers.
The reason for this is that it is fairly straightforward to create a reusable component to integrate with an external system and
there are continual opportunities to do so as new technology becomes available.
On the other hand, data transformation typically requires custom logic.
We have provided a few generic functions as processors for which simple logic can be configured using the Spring Expression Language (SpEL).
If you want write a custom application or function, this repository contains many useful examples, but the following sections are intended for those interested in contributing.
=== Project structure
Each function is a Maven project under the link:../functions[functions] directory.
They are organized by type:
.Function Types
|===
|Type | Purpose | Package Name
|link:../functions/supplier/[Supplier]
|produce information (poll an external source, receive data on demand, etc.)
|org.springframework.cloud.fn.supplier.<function_name>
|link:../functions/consumer/[Consumer]
|consume data and send it to an external system
|org.springframework.cloud.fn.consumer.<function_name>
|link:../functions/function/[Function]
|consume data, process it, and produce a result
|org.springframework.cloud.fn.<function_name>
|===
There is also link:../functions/common/[common] for code that is shared across multiple functions.
It is easiest to start with the maven configuration for an existing function.
If your function module includes `ConfigurationProperties`, make sure to add the `spring-boot-configuration-processor` dependency.
If your modules require Spring, make sure to use `spring-functions-parent` as the parent, otherwise you can use `java-functions-parent`.
=== Naming conventions
Following are required conventions when writing a new function.
Aside from enforcing a consistent structure, the Maven plugin used to generate the application may rely on some of these conventions.
Any function module that uses Spring is expected to contain at a minimum:
* The function configuration class - `<FunctionName><FunctionType>Configuration`. For example, if writing a Supplier for Google Spanner,
the configuration class should be `SpannerSupplierConfiguration`
* The name of the function bean - `<functionName><FunctionType`: `spannerSupplier`
* `ConfigurationProperties` class is named as - `<FunctionName><FunctionType>Properties`: `SpannerSupplierProperties`.
=== Documentation
Please follow the conventions used for the existing documentation.
=== Testing
All contributions must include unit tests. There is no single strategy for unit testing these functions.
Based on your use case and what type of function variant you write, you can employ varying levels of testing capabilities.
Take a look at the existing tests and pick the one that works for you.
== Generating Spring Cloud Stream applications
We will gladly consider a pull request for just the function, but if you want to generate Spring Cloud Stream applications from the function you wrote, please read on.
Now that you have a function, you can generate Spring Cloud Stream applications.
All the application generation modules use `stream-applications-core` as the parent.
For the most part, you can simply generate the applications based on a cookie cutter pattern.
Take a look at the many examples we provide for
link:../applications/source[sources], link:../applications/sink[sinks] or link:../applications/processor[processors].
If you have custom code to add at the application level, then it becomes a bit more involved, but still relatively straightforward.
link:../applications/processor/image-recognition-processor[Here] is an example showing how you can add extra customizations at the application level.
Pay close attention to the code that is provided there and the Maven configuration.
If you don't see the need for anything at the function layer and want to write a straight up Spring Cloud Stream application, then that pattern is supported as well.
Here is an link:../applications/processor/bridge-processor[example] that does exactly that.
=== Documentation
Please follow the conventions used for the existing documentation. If you are using Spring ConfigurationProperties, the properties section of the
README is generated by a Maven plugin. Also take a look at the contents of `src/main/resources/META-INF`.
=== Testing a Spring Cloud Stream application
It is important that you add some basic tests using the Spring Cloud Stream test binder as part of your application.
The parent will provide the test binder as a transitive dependency.
Source, Sink and Processor applications are tested slightly differently.
Take a look at the existing tests for more information.
When you are ready, please send a pull request so we can review it and merge it.

77
docs/FunctionComposition.adoc Normal file
View File

@@ -0,0 +1,77 @@
= Function Composition
Spring Cloud Stream includes integration with Spring Cloud Function's function-based programming model that lets the
business logic of an application be modeled as a `java.util.Supplier`, a `java.util.Function`, and a `java.util.Consumer`,
representing the roles of a `Source`, a `Processor`, and a `Sink`, respectively.
Building on this foundation, we can extend existing `Source` and `Sink` applications by importing the configuration of an
existing `Source` or `Sink` and adding code that defines a `java.util.Function` — this delivers a lot of powerful composition
possibilities.
Take for instance, the `Source` applications are auto-configured with link:../functions/function[functions], which may optionally
be included in a composite function definition.
With this, the same Source application can potentially do one or all of the following without having to build it out as a
standalone processor.
- execute SpEL transformations
- enrich message headers
- filter events
- produce task launch requests on upstream events
For example, the `time` source when it is running, as shown below, will perform a series of internal transformations to
finally publish a task launch request every second to the rabbit exchange with the name `time-test`.
```
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar \
--spring.cloud.stream.bindings.output.destination=time-test \
--spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" \
--spel.function.expression="payload.length()" \
--header.enricher.headers=task-id=payload*2 \
--task.launch.request.task-name-expression="'task-'+headers['task-id']"
```
Now, the transformed message would look like:
```
headers:
task-id: 34
content_type: application/json
Payload
49 bytes
Encoding: string
{"args":[],"deploymentProps":{},"name":"task-34"}
```
Let us unpack what is happening behind the scenes. We will start with the following function definition.
`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction`
- Here, the function definition creates a composite `Supplier` beginning with the default `timeSupplier` Java function included
in this repository, which is the foundation for `time-source`.
- The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, by following
standard Spring Integration conventions.
- The output of `spelFunction` is the length of the date-time String, `17`.
- From here, we apply the header-enricher Java function to add a Message header, `task-id` with the value of `payload*2`.
That would be `34`.
- We use the `task-id` in the header to generate the "task name", and to programmatically derive the task launch request
using the SpEL expression "'task-'+headers['task-id']", or `task-34`.
This somewhat contrived example, but the goal here was to highlight the power of function composition.
If you have had a task definition in Spring Cloud Data Flow with the name `task-34`, you could build a `time | tasklauncher`
streaming data pipeline to launch that task every second.
Before the `3.0` release of Stream Applications, this composition required extensive customization. And a lot more manual
configuration changes, extensions, and custom build of the applications.
NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`)
derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`.
In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured
property
`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`.