SpringCloud/stream-applications
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #261 from corneil/main

Browse Source 
Update baseimage version
This commit is contained in:
Corneil du Plessis
2022-05-18 09:19:55 +02:00
committed by GitHub
parent bc571c98de ad3d7cd8ac
commit db6403f4e3
10 changed files with 156 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
README.adoc
View File

@@ -203,21 +203,30 @@ Let's assume that you want to build a `jdbc-source` application based on Kafka B
Here is what you need to do.
Assuming that you already built both functions and stream-applications-core as above,
```
[source,shell]
----
./mvnw clean package -pl :jdbc-source
cd applications/source/jdbc-source/apps/jdbc-source-kafka
./mvnw clean package
```
----
This will generate the Kafka binder based uber jar in the target folder.
Similarly, for the `log-sink`, do the following.
```
[source,shell]
----
./mvnw clean package -pl :log-sink
cd applications/sink/log-sink/apps/log-sink-rabbit
./mvnw clean package
```
----
Rebuild all container images.
[source,shell]
----
./local/build-images.sh
----
=== Additional Resources

8
applications/sink/router-sink/README.adoc
View File

@@ -47,14 +47,14 @@ https://docs.spring.io/spring-integration/reference/html/xml.html#xpath-spel-fun
For example for `text` content type one should use:
[source]
[source,java]
----
new String(payload).contains('a')
new String(payload).contains('a');
----
and for `json` content type SpEL expressions like this:
[source]
[source,text]
----
#jsonPath(payload, '$.person.name')
----
@@ -66,7 +66,7 @@ Instead of SpEL expressions, Groovy scripts can also be used. Let's create a Gro
[source,groovy]
----
println("Groovy processing payload '" + payload + "'");
println("Groovy processing payload '" + payload + "'")
if (payload.contains('a')) {
return "foo"
}

22
applications/source/cdc-debezium-source/README.adoc
View File

@@ -158,9 +158,9 @@ mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIE
----
====
Use following properties to connect the CDC Source to the MySQL DB:
Use following properties to connect the CDC Source to MySQL DB:
[source]
[source,properties]
----
cdc.connector=mysql # <1>
@@ -176,6 +176,7 @@ cdc.config.database.port=3306 # <3>
cdc.schema=true # <4>
cdc.flattening.enabled=true # <5>
----
<1> Configures the CDC Source to use https://debezium.io/docs/connectors/mysql/[MySqlConnector]. (equivalent to setting `cdc.config.connector.class=io.debezium.connector.mysql.MySqlConnector`).
<2> Metadata used to identify and dispatch the incoming events.
<3> Connection to the MySQL server running on `localhost:3306` as `debezium` user.
@@ -198,9 +199,9 @@ You can connect to this server like this:
psql -U postgres -h localhost -p 5432
----
Use following properties to connect the CDC Source to the PostgreSQL:
Use following properties to connect the CDC Source to PostgreSQL:
[source]
[source,properties]
----
cdc.connector=postgres # <1>
cdc.offset.storage=memory #<2>
@@ -251,9 +252,9 @@ In the `mongodb` terminal output, search for a log entry like `host: "3f95a8a651
Add `127.0.0.1 3f95a8a6516e` entry to your `/etc/hosts`
Use following properties to connect the CDC Source to the MongoDB:
Use following properties to connect the CDC Source to MongoDB:
[source]
[source,properties]
----
cdc.connector=mongodb # <1>
cdc.offset.storage=memory #<2>
@@ -294,9 +295,9 @@ wget https://raw.githubusercontent.com/debezium/debezium-examples/master/tutoria
cat ./inventory.sql | docker exec -i sqlserver bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
----
Use following properties to connect the CDC Source to the SQLServer:
Use following properties to connect the CDC Source to SQLServer:
[source]
[source,properties]
----
cdc.connector=sqlserver # <1>
cdc.offset.storage=memory #<2>
@@ -335,7 +336,8 @@ cat ./inventory.sql | docker exec -i dbz_oracle sqlplus debezium/dbz@//localhost
== Run standalone
```
[source,shell]
----
java -jar cdc-debezium-source.jar --cdc.connector=mysql --cdc.name=my-sql-connector --cdc.config.database.server.id=85744 --cdc.config.database.server.name=my-app-connector --cdc.config.database.user=debezium --cdc.config.database.password=dbz --cdc.config.database.hostname=localhost --cdc.config.database.port=3306 --cdc.schema=true --cdc.flattening.enabled=true
```
----

5
applications/source/ftp-source/README.adoc
View File

@@ -139,8 +139,9 @@ $$root$$:: $$Root node - store entries are children of this node.$$ *($$String$$
== Examples
```
[source,shell]
----
java -jar ftp_source.jar --ftp.supplier.remote-dir=foo --file.consumer.mode=lines --ftp.factory.host=ftpserver \
--ftp.factory.username=user --ftp.factory.password=pw --ftp.local-dir=/foo
```
----
//end::ref-doc[]

8
applications/stream-applications-core/pom.xml
View File

@@ -19,13 +19,13 @@
<revision>3.2.1-SNAPSHOT</revision>
<stream-apps-core.version>${revision}</stream-apps-core.version>
<java-functions.version>1.2.0</java-functions.version>
<apps.base-image>springcloud/baseimage:1.0.0</apps.base-image>
<apps.base-image>springcloud/baseimage:1.0.1</apps.base-image>
<mockserver.version>5.10</mockserver.version>
<spring-cloud-stream-dependencies.version>3.2.2</spring-cloud-stream-dependencies.version>
<spring-cloud-stream.version>3.2.2</spring-cloud-stream.version>
<spring-cloud-dataflow-apps-generator-plugin.version>1.0.6</spring-cloud-dataflow-apps-generator-plugin.version>
<spring-cloud-dataflow-apps-docs-plugin.version>1.0.6</spring-cloud-dataflow-apps-docs-plugin.version>
<spring-cloud-dataflow-apps-metadata-plugin.version>1.0.6</spring-cloud-dataflow-apps-metadata-plugin.version>
<spring-cloud-dataflow-apps-generator-plugin.version>1.0.7-SNAPSHOT</spring-cloud-dataflow-apps-generator-plugin.version>
<spring-cloud-dataflow-apps-docs-plugin.version>1.0.7-SNAPSHOT</spring-cloud-dataflow-apps-docs-plugin.version>
<spring-cloud-dataflow-apps-metadata-plugin.version>1.0.7-SNAPSHOT</spring-cloud-dataflow-apps-metadata-plugin.version>
<java-cfenv-boot.version>2.1.2.RELEASE</java-cfenv-boot.version>
<prometheus-rsocket.version>1.3.0</prometheus-rsocket.version>
<!-- Boot 2.4.x needs wavefront-spring-boot version 2.1.0-RC1+ -->

10
docs/FunctionComposition.adoc
View File

@@ -22,18 +22,20 @@ standalone processor.
For example, the `time` source when it is running, as shown below, will perform a series of internal transformations to
finally publish a task launch request every second to the rabbit exchange with the name `time-test`.
```
[source,shell]
----
java -jar target/time-source-rabbit-3.0.0-SNAPSHOT.jar \
--spring.cloud.stream.bindings.output.destination=time-test \
--spring.cloud.stream.function.definition="timeSupplier|spelFunction|headerEnricherFunction|taskLaunchRequestFunction" \
--spel.function.expression="payload.length()" \
--header.enricher.headers=task-id=payload*2 \
--task.launch.request.task-name-expression="'task-'+headers['task-id']"
```
----
Now, the transformed message would look like:
```
[source,text]
----
headers:
task-id: 34
content_type: application/json
@@ -41,7 +43,7 @@ Payload
49 bytes
Encoding: string
{"args":[],"deploymentProps":{},"name":"task-34"}
```
----
Let us unpack what is happening behind the scenes. We will start with the following function definition.

2
functions/common/cdc-debezium-boot-starter/README.adoc
View File

@@ -79,7 +79,7 @@ mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIE
Configure `cdc-debezium-boot-starter` for consuming cdc events from the MySQL:
[source]
[source,properties]
----
cdc.name=my-sql-connector # <1>

10
functions/consumer/analytics-consumer/README.adoc
View File

@@ -46,7 +46,7 @@ The https://docs.spring.io/spring-integration/reference/html/message.html#messag
The `SpEL` expressions use the `headers` and `payload` keywords to access message’s headers and payload values. For example a counter metrics can have a value amount computed from the size of the input message payload add a `my_tag` tag, extracted from the `kind` header value:
[source]
[source,properties]
----
analytics.amount-expression=payload.lenght()
analytics.tag.expression.my_tag=headers['kind']
@@ -74,13 +74,13 @@ To enable support for another monitoring system you have to add the spring-boot-
Follow the https://docs.spring.io/spring-boot/docs/2.3.1.RELEASE/reference/html/production-ready-features.html#production-ready-metrics-export[configuration instructions] for the selected monitoring system. All monitoring configuration properties start with a prefix: `management.metrics.export`.
## Configuration Options
== Configuration Options
All `analytics-consumer` configuration properties use the `analytics` prefix. For more information on the various options available, please see link:src/main/java/org/springframework/cloud/fn/consumer/analytics/AnalyticsConsumerProperties.java[AnalyticsConsumerProperties].
All monitoring configuration properties start with a prefix `management.metrics.export`. For configuring a particular monitoring system follow the provided https://docs.spring.io/spring-boot/docs/2.3.1.RELEASE/reference/html/production-ready-features.html#production-ready-metrics-export[configuration instructions].
#### Sample Configuration
==== Sample Configuration
Following examples show how to configure `counter` and `gauge` metrics over a series of stock-exchange messages like this:
@@ -166,11 +166,11 @@ To enable one or more https://micrometer.io/docs[supported monitoring systems] y
|===
## Tests
== Tests
See this link:src/test/java/org/springframework/cloud/fn/consumer/analytics[test suite] for the various ways, this consumer is used.
## Other usage
== Other usage
* See the https://github.com/spring-cloud/stream-applications/blob/master/applications/sink/analytics-sink/README.adoc[Analytics Sink README] where this consumer is used to create a Spring Cloud Stream application where it makes a Counter sink.

60
local/build-images.sh Executable file
View File

@@ -0,0 +1,60 @@
#!/usr/bin/env bash
java --version
if [ $? -eq 0 ]
then
java_version=$(java --version)
echo "$java_version"
grep -e "1.8" "$java_version"
if [ $? -ne 0 ]
then
echo "Java 1.8 required not $java_version"
exit 2
fi
fi
version="3.2.1-SNAPSHOT"
rootdir="$(pwd)"
processors="aggregator bridge filter groovy header-enricher http-request image-recognition object-detection script semantic-segmentation splitter transform twitter-trend"
echo "Build processors:$processor"
pushd applications/processor
for app in $processors
do
pushd "${app}-processor"
echo "$Building $(pwd)"
rm -rf apps
$rootdir/mvnw clean package -Pintegration
pushd apps
$rootdir/mvnw package jib:dockerBuild -DskipTests -Djib.to.tags=${version}
popd
popd
done
popd
sinks="analytics cassandra elasticsearch file ftp geode jdbc log mongodb mqtt pgcopy rabbit redis router rsocket s3 sftp tcp throughput twitter-message twitter-update wavefront websocket zeromq"
echo "Build sinks:$sinks"
pushd applications/sink
for app in $sinks
do
pushd "${app}-sink"
echo "$Building $(pwd)"
rm -rf apps
$rootdir/mvnw clean package -Pintegration
pushd apps
$rootdir/mvnw package jib:dockerBuild -DskipTests -Djib.to.tags=${version}
popd
popd
done
popd
sources="cdc-debezium file ftp geode http jdbc jms load-generator mail mongodb mqtt rabbit s3 sftp syslog tcp time twitter-message twitter-search twitter-stream websocket zeromq"
echo "Build sources:sources"
pushd applications/source
for app in $sources
do
pushd "${app}-source"
echo "$Building $(pwd)"
rm -rf apps
$rootdir/mvnw clean package -Pintegration
pushd apps
$rootdir/mvnw package jib:dockerBuild -DskipTests -Djib.to.tags=${version}
popd
popd
done
popd

80
stream-applications-release-train/stream-applications-docs/src/main/asciidoc/overview.adoc
View File

@@ -97,21 +97,23 @@ Let's assume that you want to build JDBC Source application based on Kafka Binde
Here is what you need to do.
Assuming you built both functions and stream-applications-core as above.
```
[source,shell]
----
./mvnw clean package -pl :jdbc-source
cd applications/source/jdbc-source/apps/jdbc-source-kafka
./mvnw clean package
```
----
This will generate the Kafka binder based uber jar in the target folder.
Similarly for the log sink, do the following.
```
[source,shell]
----
./mvnw clean package -pl :log-sink
cd applications/sink/log-sink/apps/log-sink-rabbit
./mvnw clean package
```
----
==== Building a Docker image
The apps use the https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin[Jib Maven Plugin] to build and publish the Docker image.
@@ -119,25 +121,28 @@ If you have made some changes to an app, you may want to build the image and tes
NOTE: If you plan to use the image with minikube, run the following command before building the image:
```
[source,shell]
----
eval $(minikube docker-env)
```
----
To build the image in your local registry:
```
[source,shell]
----
./mvn clean package jib:dockerBuild
```
----
To publish the image to a remote registry:
```
jib:build \
[source,shell]
----
./mvnw jib:build \
-Djib.to.image=myregistry/myimage:latest \
-Djib.to.auth.username=$USERNAME \
-Djib.to.auth.password=$PASSWORD
```
----
== Patching Pre-built Applications
@@ -149,15 +154,18 @@ To add `mysql` driver to `jdbc-sink` application:
1. Clone the GitHub repository at https://github.com/spring-cloud/stream-applications
2. Find the module that you want to patch and add the additional dependencies, `jdbc-sink` in this case. For example, you can add the following mysql dependency to the application generator plugin's configuration in the pom.xml:
```
[source,xml]
----
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.37</version>
</dependency>
```
----
This is how the complete plugin configuration should look like.
```
[source,xml]
----
<plugin>
<groupId>org.springframework.cloud.stream.app.plugin</groupId>
<artifactId>spring-cloud-stream-app-maven-plugin</artifactId>
@@ -182,23 +190,25 @@ This is how the complete plugin configuration should look like.
</dependencies>
</configuration>
</plugin>
```
----
Once the above changes are done, you can generate the binder based apps as below from the root of the repository.
```
[source,shell]
----
./mvnw clean install -pl :jdbc-sink
```
----
This generates the binder based applications in the `apps` folder under `jdbc-sink` folder.
In order to build the app with the binder flavor that you are interested in, you need to do the following step.
```
[source,shell]
----
cd applications/sink/jdbc-sink
cd apps/jdbc-sink-kafka (or Rabbit if you are interested in that)
cd apps/jdbc-sink-kafka # (or Rabbit if you are interested in that)
./mvnw clean package
cd target
```
----
There you will find the binder based uber jar with your changes.
@@ -210,10 +220,11 @@ If we have to update the binder dependencies from a new release of Spring Cloud
Here are the steps (again, we are using `jdbc-sink-kafka` as an example).
```
[source,shell]
----
./mvnw clean install -pl :jdbc-sink
cd applications/sink/jdbc-sink/apps/jdbc-sink-kafka
```
----
Open the generated application's `pom.xml` and update the dependencies.
If there is a new version of Spring Cloud Stream update available that contains the enhancements we are looking for, then it is easier to update the BOM itself.
@@ -221,7 +232,8 @@ Find where the bom is declared in `pom.xml` and update the version.
For example, if we have to update Spring Cloud Stream to `Horsham.SR10`, this version must be specified in the BOM declaration as below:
```
[source,xml]
----
<dependencyManagement>
<dependencies>
<dependency>
@@ -233,7 +245,7 @@ For example, if we have to update Spring Cloud Stream to `Horsham.SR10`, this ve
</dependency>
</dependencies>
</dependencyManagement>
```
----
We can also update any individual dependencies directly, but it is preferred to use the above `dependencyManagement` approach if there is a BOM available.
This is because, when using a BOM, maven will properly use and align any transitive dependencies.
@@ -248,10 +260,11 @@ First, add the resources to the classpath by placing them under `src/main/resour
Then rebuild the application.
```
[source,shell]
----
./mvnw clean package
cd target
```
----
Here you can find the modified application jar file.
@@ -268,7 +281,8 @@ cd applications/stream-applications-core
We need to edit the pom.xml in this module.
Find the following configuration where it defines the Kafka and RabbitMQ binders for the maven plugin.
```
[source,xml]
----
<kafka>
<maven>
<dependencies>
@@ -289,12 +303,13 @@ Find the following configuration where it defines the Kafka and RabbitMQ binders
</dependencies>
</maven>
</rabbit>
```
----
Add the binder for which you want to generate new apps for.
For example, if we want to generate applications for the https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis[Kinesis binder], then modify as below.
```
[source,xml]
----
<binders>
<kafka>
<maven>
@@ -328,7 +343,7 @@ For example, if we want to generate applications for the https://github.com/spri
</maven>
</kinesis>
</binders>
```
----
Note that, we need to use the Kinesis binder version here explicitly, while both Kafka and RabbitMQ do not need them.
This is because, those versions come from a dependency management while the Kinesis binder is not available through such mechanisms.
@@ -343,9 +358,10 @@ If we go to the applications folder and look at the generated applications, we s
For instance, if we follow the configuration above for adding the Kinesis binder, then we should see the Kinesis binder based app in the generated apps.
Let's take `time-source` as an example.
```
[source,shell]
----
cd applications/source/time-souce/apps
```
----
Here, we should see three different binder based apps projects - `time-source-kafka`, `time-source-rabbit` and `time-source-kineses`.
Similarly, this should happen for all the out of the box application projects.