README changes
* Adding steps for PR contributions in the README
This commit is contained in:
committed by
David Turanski
parent
b7d41bf04c
commit
4a5bb50659
87
README.adoc
87
README.adoc
@@ -145,84 +145,6 @@ The following are the four major components of this repository.
|
|||||||
|link:applications/sink/wavefront-sink/README.adoc[Wavefront]
|
|link:applications/sink/wavefront-sink/README.adoc[Wavefront]
|
||||||
|===
|
|===
|
||||||
|
|
||||||
=== Composable Functions
|
|
||||||
|
|
||||||
Spring Cloud Stream includes integration with Spring Cloud Function's function-based programming model that lets the
|
|
||||||
business logic of an application be modeled as a `java.util.Supplier`, a `java.util.Function`, and a `java.util.Consumer`,
|
|
||||||
representing the roles of a `Source`, a `Processor`, and a `Sink`, respectively.
|
|
||||||
|
|
||||||
Building on this foundation, we can extend existing `Source` and `Sink` applications by importing the configuration of an
|
|
||||||
existing `Source` or `Sink` and adding code that defines a `java.util.Function` — this delivers a lot of powerful composition
|
|
||||||
possibilities.
|
|
||||||
|
|
||||||
Take for instance, the `Source` applications are auto-configured with link:functions/function[functions], which may optionally
|
|
||||||
be included in a composite function definition.
|
|
||||||
|
|
||||||
With this, the same Source application can potentially do one or all of the following without having to build it out as a
|
|
||||||
standalone processor.
|
|
||||||
|
|
||||||
- execute SpEL transformations
|
|
||||||
- enrich message headers
|
|
||||||
- filter events
|
|
||||||
- produce task launch requests on upstream events
|
|
||||||
|
|
||||||
For example, the `time` source when it is running, as shown below, will perform a series of internal transformations to
|
|
||||||
finally publish a task launch request every second to the rabbit exchange with the name `time-test`.
|
|
||||||
|
|
||||||
```
|
|
||||||
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar \
|
|
||||||
--spring.cloud.stream.bindings.output.destination=time-test \
|
|
||||||
--spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" \
|
|
||||||
--spel.function.expression="payload.length()" \
|
|
||||||
--header.enricher.headers=task-id=payload*2 \
|
|
||||||
--task.launch.request.task-name-expression="'task-'+headers['task-id']"
|
|
||||||
```
|
|
||||||
|
|
||||||
Now, the transformed message would look like:
|
|
||||||
|
|
||||||
```
|
|
||||||
headers:
|
|
||||||
task-id: 34
|
|
||||||
content_type: application/json
|
|
||||||
Payload
|
|
||||||
49 bytes
|
|
||||||
Encoding: string
|
|
||||||
{"args":[],"deploymentProps":{},"name":"task-34"}
|
|
||||||
```
|
|
||||||
|
|
||||||
Let us unpack what is happening behind the scenes. We will start with the following function definition.
|
|
||||||
|
|
||||||
`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction`
|
|
||||||
|
|
||||||
- Here, the function definition creates a composite `Supplier` beginning with the default `timeSupplier` Java function included
|
|
||||||
in this repository, which is the foundation for `time-source`.
|
|
||||||
|
|
||||||
- The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, by following
|
|
||||||
standard Spring Integration conventions.
|
|
||||||
|
|
||||||
- The output of `spelFunction` is the length of the date-time String, `17`.
|
|
||||||
|
|
||||||
- From here, we apply the header-enricher Java function to add a Message header, `task-id` with the value of `payload*2`.
|
|
||||||
That would be `34`.
|
|
||||||
|
|
||||||
- We use the `task-id` in the header to generate the "task name", and to programmatically derive the task launch request
|
|
||||||
using the SpEL expression "'task-'+headers['task-id']", or `task-34`.
|
|
||||||
|
|
||||||
This somewhat contrived example, but the goal here was to highlight the power of function composition.
|
|
||||||
|
|
||||||
If you have had a task definition in Spring Cloud Data Flow with the name `task-34`, you could build a `time | tasklauncher`
|
|
||||||
streaming data pipeline to launch that task every second.
|
|
||||||
|
|
||||||
Before the `3.0` release of Stream Applications, this composition required extensive customization. And a lot more manual
|
|
||||||
configuration changes, extensions, and custom build of the applications.
|
|
||||||
|
|
||||||
NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`)
|
|
||||||
derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`.
|
|
||||||
|
|
||||||
In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured
|
|
||||||
property
|
|
||||||
`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`.
|
|
||||||
|
|
||||||
=== Build
|
=== Build
|
||||||
|
|
||||||
You can build everything from the root of the repository.
|
You can build everything from the root of the repository.
|
||||||
@@ -248,7 +170,7 @@ If you want to re-run the common core build, you can build it with the following
|
|||||||
|
|
||||||
`./mvnw clean install -f applications/stream-applications-core`
|
`./mvnw clean install -f applications/stream-applications-core`
|
||||||
|
|
||||||
=== Building Stream applications
|
=== Building Stream Applications
|
||||||
|
|
||||||
Let's assume that you want to build a `jdbc-source` application based on Kafka Binder in Spring Cloud Stream and Log Sink
|
Let's assume that you want to build a `jdbc-source` application based on Kafka Binder in Spring Cloud Stream and Log Sink
|
||||||
application based on Rabbit binder.
|
application based on Rabbit binder.
|
||||||
@@ -272,6 +194,13 @@ cd applications/sink/log-sink/apps/log-sink-rabbit
|
|||||||
./mvnw clean package
|
./mvnw clean package
|
||||||
```
|
```
|
||||||
|
|
||||||
|
=== Additional Resources
|
||||||
|
|
||||||
|
Here is a list of resources where you can find out more about using and developing functions and stream applications:
|
||||||
|
|
||||||
|
* link:docs/FunctionComposition.adoc[Function Composition]
|
||||||
|
* link:docs/Contributing.adoc[Contributing a New Function or Application to this Repository]
|
||||||
|
|
||||||
=== Code of Conduct
|
=== Code of Conduct
|
||||||
|
|
||||||
Please see our https://github.com/spring-projects/.github/blob/master/CODE_OF_CONDUCT.md[Code of Conduct]
|
Please see our https://github.com/spring-projects/.github/blob/master/CODE_OF_CONDUCT.md[Code of Conduct]
|
||||||
|
|||||||
100
docs/Contributing.adoc
Normal file
100
docs/Contributing.adoc
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
= Contributing a new function component to this repository
|
||||||
|
|
||||||
|
If you do not see a function or application that you are interested in, we encourage you to contribute to this repository.
|
||||||
|
In most cases, you start by writing the function.
|
||||||
|
Once the function is ready, you set up a Maven project for the corresponding stream application, provide the pom, and configuration properties metadata, and some basic tests.
|
||||||
|
If you have followed the prescribed structure and naming conventions, the Maven build will generate the application.
|
||||||
|
|
||||||
|
Here are some things you need to know before contributing a new function.
|
||||||
|
|
||||||
|
== Creating a function
|
||||||
|
|
||||||
|
What kind of function do you want to write? If you are not sure, consult the table below.
|
||||||
|
We expect the majority of contributions to be either suppliers or consumers.
|
||||||
|
The reason for this is that it is fairly straightforward to create a reusable component to integrate with an external system and
|
||||||
|
there are continual opportunities to do so as new technology becomes available.
|
||||||
|
On the other hand, data transformation typically requires custom logic.
|
||||||
|
We have provided a few generic functions as processors for which simple logic can be configured using the Spring Expression Language (SpEL).
|
||||||
|
If you want write a custom application or function, this repository contains many useful examples, but the following sections are intended for those interested in contributing.
|
||||||
|
|
||||||
|
=== Project structure
|
||||||
|
|
||||||
|
Each function is a Maven project under the link:../functions[functions] directory.
|
||||||
|
They are organized by type:
|
||||||
|
|
||||||
|
.Function Types
|
||||||
|
|===
|
||||||
|
|Type | Purpose | Package Name
|
||||||
|
|link:../functions/supplier/[Supplier]
|
||||||
|
|produce information (poll an external source, receive data on demand, etc.)
|
||||||
|
|org.springframework.cloud.fn.supplier.<function_name>
|
||||||
|
|
||||||
|
|link:../functions/consumer/[Consumer]
|
||||||
|
|consume data and send it to an external system
|
||||||
|
|org.springframework.cloud.fn.consumer.<function_name>
|
||||||
|
|
||||||
|
|link:../functions/function/[Function]
|
||||||
|
|consume data, process it, and produce a result
|
||||||
|
|org.springframework.cloud.fn.<function_name>
|
||||||
|
|===
|
||||||
|
|
||||||
|
There is also link:../functions/common/[common] for code that is shared across multiple functions.
|
||||||
|
|
||||||
|
It is easiest to start with the maven configuration for an existing function.
|
||||||
|
If your function module includes `ConfigurationProperties`, make sure to add the `spring-boot-configuration-processor` dependency.
|
||||||
|
|
||||||
|
If your modules require Spring, make sure to use `spring-functions-parent` as the parent, otherwise you can use `java-functions-parent`.
|
||||||
|
|
||||||
|
=== Naming conventions
|
||||||
|
|
||||||
|
Following are required conventions when writing a new function.
|
||||||
|
Aside from enforcing a consistent structure, the Maven plugin used to generate the application may rely on some of these conventions.
|
||||||
|
|
||||||
|
Any function module that uses Spring is expected to contain at a minimum:
|
||||||
|
|
||||||
|
* The function configuration class - `<FunctionName><FunctionType>Configuration`. For example, if writing a Supplier for Google Spanner,
|
||||||
|
the configuration class should be `SpannerSupplierConfiguration`
|
||||||
|
* The name of the function bean - `<functionName><FunctionType`: `spannerSupplier`
|
||||||
|
* `ConfigurationProperties` class is named as - `<FunctionName><FunctionType>Properties`: `SpannerSupplierProperties`.
|
||||||
|
|
||||||
|
=== Documentation
|
||||||
|
|
||||||
|
Please follow the conventions used for the existing documentation.
|
||||||
|
|
||||||
|
=== Testing
|
||||||
|
|
||||||
|
All contributions must include unit tests. There is no single strategy for unit testing these functions.
|
||||||
|
Based on your use case and what type of function variant you write, you can employ varying levels of testing capabilities.
|
||||||
|
Take a look at the existing tests and pick the one that works for you.
|
||||||
|
|
||||||
|
== Generating Spring Cloud Stream applications
|
||||||
|
|
||||||
|
We will gladly consider a pull request for just the function, but if you want to generate Spring Cloud Stream applications from the function you wrote, please read on.
|
||||||
|
|
||||||
|
Now that you have a function, you can generate Spring Cloud Stream applications.
|
||||||
|
All the application generation modules use `stream-applications-core` as the parent.
|
||||||
|
|
||||||
|
For the most part, you can simply generate the applications based on a cookie cutter pattern.
|
||||||
|
Take a look at the many examples we provide for
|
||||||
|
link:../applications/source[sources], link:../applications/sink[sinks] or link:../applications/processor[processors].
|
||||||
|
|
||||||
|
If you have custom code to add at the application level, then it becomes a bit more involved, but still relatively straightforward.
|
||||||
|
link:../applications/processor/image-recognition-processor[Here] is an example showing how you can add extra customizations at the application level.
|
||||||
|
Pay close attention to the code that is provided there and the Maven configuration.
|
||||||
|
|
||||||
|
If you don't see the need for anything at the function layer and want to write a straight up Spring Cloud Stream application, then that pattern is supported as well.
|
||||||
|
Here is an link:../applications/processor/bridge-processor[example] that does exactly that.
|
||||||
|
|
||||||
|
=== Documentation
|
||||||
|
|
||||||
|
Please follow the conventions used for the existing documentation. If you are using Spring ConfigurationProperties, the properties section of the
|
||||||
|
README is generated by a Maven plugin. Also take a look at the contents of `src/main/resources/META-INF`.
|
||||||
|
|
||||||
|
=== Testing a Spring Cloud Stream application
|
||||||
|
|
||||||
|
It is important that you add some basic tests using the Spring Cloud Stream test binder as part of your application.
|
||||||
|
The parent will provide the test binder as a transitive dependency.
|
||||||
|
Source, Sink and Processor applications are tested slightly differently.
|
||||||
|
Take a look at the existing tests for more information.
|
||||||
|
|
||||||
|
When you are ready, please send a pull request so we can review it and merge it.
|
||||||
77
docs/FunctionComposition.adoc
Normal file
77
docs/FunctionComposition.adoc
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
= Function Composition
|
||||||
|
|
||||||
|
Spring Cloud Stream includes integration with Spring Cloud Function's function-based programming model that lets the
|
||||||
|
business logic of an application be modeled as a `java.util.Supplier`, a `java.util.Function`, and a `java.util.Consumer`,
|
||||||
|
representing the roles of a `Source`, a `Processor`, and a `Sink`, respectively.
|
||||||
|
|
||||||
|
Building on this foundation, we can extend existing `Source` and `Sink` applications by importing the configuration of an
|
||||||
|
existing `Source` or `Sink` and adding code that defines a `java.util.Function` — this delivers a lot of powerful composition
|
||||||
|
possibilities.
|
||||||
|
|
||||||
|
Take for instance, the `Source` applications are auto-configured with link:../functions/function[functions], which may optionally
|
||||||
|
be included in a composite function definition.
|
||||||
|
|
||||||
|
With this, the same Source application can potentially do one or all of the following without having to build it out as a
|
||||||
|
standalone processor.
|
||||||
|
|
||||||
|
- execute SpEL transformations
|
||||||
|
- enrich message headers
|
||||||
|
- filter events
|
||||||
|
- produce task launch requests on upstream events
|
||||||
|
|
||||||
|
For example, the `time` source when it is running, as shown below, will perform a series of internal transformations to
|
||||||
|
finally publish a task launch request every second to the rabbit exchange with the name `time-test`.
|
||||||
|
|
||||||
|
```
|
||||||
|
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar \
|
||||||
|
--spring.cloud.stream.bindings.output.destination=time-test \
|
||||||
|
--spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" \
|
||||||
|
--spel.function.expression="payload.length()" \
|
||||||
|
--header.enricher.headers=task-id=payload*2 \
|
||||||
|
--task.launch.request.task-name-expression="'task-'+headers['task-id']"
|
||||||
|
```
|
||||||
|
|
||||||
|
Now, the transformed message would look like:
|
||||||
|
|
||||||
|
```
|
||||||
|
headers:
|
||||||
|
task-id: 34
|
||||||
|
content_type: application/json
|
||||||
|
Payload
|
||||||
|
49 bytes
|
||||||
|
Encoding: string
|
||||||
|
{"args":[],"deploymentProps":{},"name":"task-34"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Let us unpack what is happening behind the scenes. We will start with the following function definition.
|
||||||
|
|
||||||
|
`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction`
|
||||||
|
|
||||||
|
- Here, the function definition creates a composite `Supplier` beginning with the default `timeSupplier` Java function included
|
||||||
|
in this repository, which is the foundation for `time-source`.
|
||||||
|
|
||||||
|
- The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, by following
|
||||||
|
standard Spring Integration conventions.
|
||||||
|
|
||||||
|
- The output of `spelFunction` is the length of the date-time String, `17`.
|
||||||
|
|
||||||
|
- From here, we apply the header-enricher Java function to add a Message header, `task-id` with the value of `payload*2`.
|
||||||
|
That would be `34`.
|
||||||
|
|
||||||
|
- We use the `task-id` in the header to generate the "task name", and to programmatically derive the task launch request
|
||||||
|
using the SpEL expression "'task-'+headers['task-id']", or `task-34`.
|
||||||
|
|
||||||
|
This somewhat contrived example, but the goal here was to highlight the power of function composition.
|
||||||
|
|
||||||
|
If you have had a task definition in Spring Cloud Data Flow with the name `task-34`, you could build a `time | tasklauncher`
|
||||||
|
streaming data pipeline to launch that task every second.
|
||||||
|
|
||||||
|
Before the `3.0` release of Stream Applications, this composition required extensive customization. And a lot more manual
|
||||||
|
configuration changes, extensions, and custom build of the applications.
|
||||||
|
|
||||||
|
NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`)
|
||||||
|
derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`.
|
||||||
|
|
||||||
|
In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured
|
||||||
|
property
|
||||||
|
`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`.
|
||||||
Reference in New Issue
Block a user