diff --git a/README.adoc b/README.adoc index 8ce2ccaf..0de8a947 100644 --- a/README.adoc +++ b/README.adoc @@ -145,84 +145,6 @@ The following are the four major components of this repository. |link:applications/sink/wavefront-sink/README.adoc[Wavefront] |=== -=== Composable Functions - -Spring Cloud Stream includes integration with Spring Cloud Function's function-based programming model that lets the -business logic of an application be modeled as a `java.util.Supplier`, a `java.util.Function`, and a `java.util.Consumer`, -representing the roles of a `Source`, a `Processor`, and a `Sink`, respectively. - -Building on this foundation, we can extend existing `Source` and `Sink` applications by importing the configuration of an -existing `Source` or `Sink` and adding code that defines a `java.util.Function` — this delivers a lot of powerful composition -possibilities. - -Take for instance, the `Source` applications are auto-configured with link:functions/function[functions], which may optionally -be included in a composite function definition. - -With this, the same Source application can potentially do one or all of the following without having to build it out as a -standalone processor. - -- execute SpEL transformations -- enrich message headers -- filter events -- produce task launch requests on upstream events - -For example, the `time` source when it is running, as shown below, will perform a series of internal transformations to -finally publish a task launch request every second to the rabbit exchange with the name `time-test`. - -``` -java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar \ - --spring.cloud.stream.bindings.output.destination=time-test \ - --spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" \ - --spel.function.expression="payload.length()" \ - --header.enricher.headers=task-id=payload*2 \ - --task.launch.request.task-name-expression="'task-'+headers['task-id']" -``` - -Now, the transformed message would look like: - -``` -headers: -task-id: 34 -content_type: application/json -Payload -49 bytes -Encoding: string -{"args":[],"deploymentProps":{},"name":"task-34"} -``` - -Let us unpack what is happening behind the scenes. We will start with the following function definition. - -`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction` - -- Here, the function definition creates a composite `Supplier` beginning with the default `timeSupplier` Java function included -in this repository, which is the foundation for `time-source`. - -- The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, by following -standard Spring Integration conventions. - -- The output of `spelFunction` is the length of the date-time String, `17`. - -- From here, we apply the header-enricher Java function to add a Message header, `task-id` with the value of `payload*2`. -That would be `34`. - -- We use the `task-id` in the header to generate the "task name", and to programmatically derive the task launch request -using the SpEL expression "'task-'+headers['task-id']", or `task-34`. - -This somewhat contrived example, but the goal here was to highlight the power of function composition. - -If you have had a task definition in Spring Cloud Data Flow with the name `task-34`, you could build a `time | tasklauncher` -streaming data pipeline to launch that task every second. - -Before the `3.0` release of Stream Applications, this composition required extensive customization. And a lot more manual -configuration changes, extensions, and custom build of the applications. - -NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`) -derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`. - -In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured -property -`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`. - === Build You can build everything from the root of the repository. @@ -248,7 +170,7 @@ If you want to re-run the common core build, you can build it with the following `./mvnw clean install -f applications/stream-applications-core` -=== Building Stream applications +=== Building Stream Applications Let's assume that you want to build a `jdbc-source` application based on Kafka Binder in Spring Cloud Stream and Log Sink application based on Rabbit binder. @@ -272,6 +194,13 @@ cd applications/sink/log-sink/apps/log-sink-rabbit ./mvnw clean package ``` +=== Additional Resources + +Here is a list of resources where you can find out more about using and developing functions and stream applications: + +* link:docs/FunctionComposition.adoc[Function Composition] +* link:docs/Contributing.adoc[Contributing a New Function or Application to this Repository] + === Code of Conduct Please see our https://github.com/spring-projects/.github/blob/master/CODE_OF_CONDUCT.md[Code of Conduct] diff --git a/docs/Contributing.adoc b/docs/Contributing.adoc new file mode 100644 index 00000000..bc197dfc --- /dev/null +++ b/docs/Contributing.adoc @@ -0,0 +1,100 @@ += Contributing a new function component to this repository + +If you do not see a function or application that you are interested in, we encourage you to contribute to this repository. +In most cases, you start by writing the function. +Once the function is ready, you set up a Maven project for the corresponding stream application, provide the pom, and configuration properties metadata, and some basic tests. +If you have followed the prescribed structure and naming conventions, the Maven build will generate the application. + +Here are some things you need to know before contributing a new function. + +== Creating a function + +What kind of function do you want to write? If you are not sure, consult the table below. +We expect the majority of contributions to be either suppliers or consumers. +The reason for this is that it is fairly straightforward to create a reusable component to integrate with an external system and +there are continual opportunities to do so as new technology becomes available. +On the other hand, data transformation typically requires custom logic. +We have provided a few generic functions as processors for which simple logic can be configured using the Spring Expression Language (SpEL). +If you want write a custom application or function, this repository contains many useful examples, but the following sections are intended for those interested in contributing. + +=== Project structure + +Each function is a Maven project under the link:../functions[functions] directory. +They are organized by type: + +.Function Types +|=== +|Type | Purpose | Package Name +|link:../functions/supplier/[Supplier] +|produce information (poll an external source, receive data on demand, etc.) +|org.springframework.cloud.fn.supplier. + +|link:../functions/consumer/[Consumer] +|consume data and send it to an external system +|org.springframework.cloud.fn.consumer. + +|link:../functions/function/[Function] +|consume data, process it, and produce a result +|org.springframework.cloud.fn. +|=== + +There is also link:../functions/common/[common] for code that is shared across multiple functions. + +It is easiest to start with the maven configuration for an existing function. +If your function module includes `ConfigurationProperties`, make sure to add the `spring-boot-configuration-processor` dependency. + +If your modules require Spring, make sure to use `spring-functions-parent` as the parent, otherwise you can use `java-functions-parent`. + +=== Naming conventions + +Following are required conventions when writing a new function. +Aside from enforcing a consistent structure, the Maven plugin used to generate the application may rely on some of these conventions. + +Any function module that uses Spring is expected to contain at a minimum: + +* The function configuration class - `Configuration`. For example, if writing a Supplier for Google Spanner, +the configuration class should be `SpannerSupplierConfiguration` +* The name of the function bean - `Properties`: `SpannerSupplierProperties`. + +=== Documentation + +Please follow the conventions used for the existing documentation. + +=== Testing + +All contributions must include unit tests. There is no single strategy for unit testing these functions. +Based on your use case and what type of function variant you write, you can employ varying levels of testing capabilities. +Take a look at the existing tests and pick the one that works for you. + +== Generating Spring Cloud Stream applications + +We will gladly consider a pull request for just the function, but if you want to generate Spring Cloud Stream applications from the function you wrote, please read on. + +Now that you have a function, you can generate Spring Cloud Stream applications. +All the application generation modules use `stream-applications-core` as the parent. + +For the most part, you can simply generate the applications based on a cookie cutter pattern. +Take a look at the many examples we provide for +link:../applications/source[sources], link:../applications/sink[sinks] or link:../applications/processor[processors]. + +If you have custom code to add at the application level, then it becomes a bit more involved, but still relatively straightforward. +link:../applications/processor/image-recognition-processor[Here] is an example showing how you can add extra customizations at the application level. +Pay close attention to the code that is provided there and the Maven configuration. + +If you don't see the need for anything at the function layer and want to write a straight up Spring Cloud Stream application, then that pattern is supported as well. +Here is an link:../applications/processor/bridge-processor[example] that does exactly that. + +=== Documentation + +Please follow the conventions used for the existing documentation. If you are using Spring ConfigurationProperties, the properties section of the +README is generated by a Maven plugin. Also take a look at the contents of `src/main/resources/META-INF`. + +=== Testing a Spring Cloud Stream application + +It is important that you add some basic tests using the Spring Cloud Stream test binder as part of your application. +The parent will provide the test binder as a transitive dependency. +Source, Sink and Processor applications are tested slightly differently. +Take a look at the existing tests for more information. + +When you are ready, please send a pull request so we can review it and merge it. diff --git a/docs/FunctionComposition.adoc b/docs/FunctionComposition.adoc new file mode 100644 index 00000000..a328deb6 --- /dev/null +++ b/docs/FunctionComposition.adoc @@ -0,0 +1,77 @@ += Function Composition + +Spring Cloud Stream includes integration with Spring Cloud Function's function-based programming model that lets the +business logic of an application be modeled as a `java.util.Supplier`, a `java.util.Function`, and a `java.util.Consumer`, +representing the roles of a `Source`, a `Processor`, and a `Sink`, respectively. + +Building on this foundation, we can extend existing `Source` and `Sink` applications by importing the configuration of an +existing `Source` or `Sink` and adding code that defines a `java.util.Function` — this delivers a lot of powerful composition +possibilities. + +Take for instance, the `Source` applications are auto-configured with link:../functions/function[functions], which may optionally +be included in a composite function definition. + +With this, the same Source application can potentially do one or all of the following without having to build it out as a +standalone processor. + +- execute SpEL transformations +- enrich message headers +- filter events +- produce task launch requests on upstream events + +For example, the `time` source when it is running, as shown below, will perform a series of internal transformations to +finally publish a task launch request every second to the rabbit exchange with the name `time-test`. + +``` +java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar \ + --spring.cloud.stream.bindings.output.destination=time-test \ + --spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" \ + --spel.function.expression="payload.length()" \ + --header.enricher.headers=task-id=payload*2 \ + --task.launch.request.task-name-expression="'task-'+headers['task-id']" +``` + +Now, the transformed message would look like: + +``` +headers: +task-id: 34 +content_type: application/json +Payload +49 bytes +Encoding: string +{"args":[],"deploymentProps":{},"name":"task-34"} +``` + +Let us unpack what is happening behind the scenes. We will start with the following function definition. + +`timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction` + +- Here, the function definition creates a composite `Supplier` beginning with the default `timeSupplier` Java function included +in this repository, which is the foundation for `time-source`. + +- The `spelFunction` applies to a `Message` from which we can extract and transform the `payload` or `headers`, by following +standard Spring Integration conventions. + +- The output of `spelFunction` is the length of the date-time String, `17`. + +- From here, we apply the header-enricher Java function to add a Message header, `task-id` with the value of `payload*2`. +That would be `34`. + +- We use the `task-id` in the header to generate the "task name", and to programmatically derive the task launch request +using the SpEL expression "'task-'+headers['task-id']", or `task-34`. + +This somewhat contrived example, but the goal here was to highlight the power of function composition. + +If you have had a task definition in Spring Cloud Data Flow with the name `task-34`, you could build a `time | tasklauncher` +streaming data pipeline to launch that task every second. + +Before the `3.0` release of Stream Applications, this composition required extensive customization. And a lot more manual +configuration changes, extensions, and custom build of the applications. + +NOTE: Support for composite functions includes auto-configuration for conventional binding name mappings (`input` and `output`) +derived from the function definition and the presence of `spring.cloud.stream.bindings.output...`. + +In this example, `--spring.cloud.stream.bindings.output.destination=time-test` is enabled behind the scenes by the auto-configured +property +`--spring.cloud.stream.function.bindings.timeSupplierspelFunctionheaderEnricherFunctiontaskLaunchRequestFunction-out-0=output`.