SpringCloud/stream-applications
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Files
6cf66ec779faaed28a759dde409b034c92cb7356
stream-applications/README.adoc
Christian Tzolov 6cf66ec779 Refactor CDC Debezium Source 
- Add AVRO format support. Defaults to JSON
 - Remove the Metadata repo configuration.
 - Remove the Spring property sort-cuts. Only the native Debezium properties are used.
 - change the native properties prefix from `cdc.config to cdc.debezium`.
 - Removes the need to fork and modify the Debezium engine code.
 - Add binding name strategy.

Remove old CDC projects and replace with the new implementation
Remove duplicate pom properties
remove non existing pom module
Rename `cdc-debezium-supplier/source` to `debezium-supplier/source`
streamline naming and tests
streamline supplier tests
test streamline. Github workflow
Fix Debezium engine generics mess
Break `DebeziumConfiguration` into engine auto-config and consumer configuration
add initial reactive consumer impl.
Splits Debezium into a pure supplier and a streaming source

  - Remove any SCS related dependencies from the supplier.
  - Supplier uses Sinks.Many.

complete the supplier vs source separation
Improve Debezium Egine auto-conf

 - Provide configuration hooks for OffsetCommitPolicy, ConnectorCallback,
   CompletionCallback and Clock with default implementations.
 - Remove obsolete code.

minor improvements
Rename cdc property prefix to `debezium` and `cdc.debezium` to `debezium.inner`
remove obsolete `StreamBridge` tests
address reviews. improve testing
Supplier tests. Rename `inner` to `properties`.
Streamline the README. Minor improvements
streamline supplier tests
final fixes
2023-05-02 16:07:28 -04:00

224 lines
9.7 KiB
Plaintext
Raw Blame History

== Stream Applications
In this repository, you will find a collection of components that can meet various data integration use cases and requirements.
The repository's primary focus is to provide a set of standalone Java functions that can be useful in the end-user applications as-is.
Besides, this repository builds on the Java functions to generate standalone Spring Cloud Stream applications that can run against Spring Cloud Stream's RabbitMQ or Apache Kafka binder implementations.
It is also possible to extend the generator to bundle the Java functions with the other supported binder implementations.
These applications can run standalone or as part of a data flow, such as the one orchestrated using Spring Cloud Data Flow.
=== Project Structure
The repository includes two major sections - `Functions` and `Applications`.
The former hosts the various Java functions, and the latter is for generating the standalone Spring Cloud Stream applications and hosting their related components.
The following are the various components of this repository.
* https://github.com/spring-cloud/stream-applications/tree/master/functions[Standalone Java Functions]
* https://github.com/spring-cloud/stream-applications/tree/master/applications/stream-applications-core[Common Core Components for Applications]
* https://github.com/spring-cloud/stream-applications/tree/master/applications[Spring Cloud Stream Applications]
* https://github.com/spring-cloud/stream-applications/tree/master/stream-applications-build[Build Parent]
* https://github.com/spring-cloud/stream-applications/tree/master/stream-applications-release-train[Release Train]
=== Reusable Functions
|===
| `java.util.Supplier` | `java.util.Function` | `java.util.Consumer`
|link:functions/supplier/debezium-supplier/README.adoc[Debezium supplier]
|link:functions/function/aggregator-function/README.adoc[Aggregator]
|link:functions/consumer/analytics-consumer/README.adoc[Analytics]
|link:functions/supplier/file-supplier/README.adoc[File]
|link:functions/function/filter-function/README.adoc[Filter]
|link:functions/consumer/cassandra-consumer/README.adoc[Cassandra]
|link:functions/supplier/ftp-supplier/README.adoc[FTP]
|link:functions/function/header-enricher-function/README.adoc[Header-Enricher]
|link:functions/consumer/elasticsearch-consumer/README.adoc[Elasticsearch]
|
|link:functions/function/header-filter-function/README.adoc[Header-Filter]
|link:functions/consumer/file-consumer/README.adoc[File]
|link:functions/supplier/http-supplier/README.adoc[HTTP]
|link:functions/function/http-request-function/README.adoc[HTTP Request]
|link:functions/consumer/ftp-consumer/README.adoc[FTP]
|link:functions/supplier/jdbc-supplier/README.adoc[JDBC]
|link:functions/function/image-recognition-function/README.adoc[Image Recognition(Tensorflow)]
|
|link:functions/supplier/jms-supplier/README.adoc[JMS]
|link:functions/function/object-detection-function/README.adoc[Object Detection(Tensorflow)]
|link:functions/consumer/jdbc-consumer/README.adoc[JDBC]
|link:functions/supplier/mail-supplier/README.adoc[Mail]
|link:functions/function/semantic-segmentation-function/README.adoc[Semantic Segmentation(Tensorflow)]
|link:functions/consumer/log-consumer/README.adoc[Log]
|link:functions/supplier/mongodb-supplier/README.adoc[MongoDB]
|link:functions/function/spel-function/README.adoc[SpEL]
|link:functions/consumer/mongodb-consumer/README.adoc[MongoDB]
|link:functions/supplier/mqtt-supplier/README.adoc[MQTT]
|link:functions/function/splitter-function/README.adoc[Splitter]
|link:functions/consumer/mqtt-consumer/README.adoc[MQTT]
|link:functions/supplier/rabbit-supplier/README.adoc[RabbitMQ]
|link:functions/function/task-launch-request-function/README.adoc[Task Launch Request]
|link:functions/consumer/rabbit-consumer/README.adoc[RabbitMQ]
|link:functions/supplier/s3-supplier/README.adoc[AWS S3]
|link:functions/function/tasklauncher-function/README.adoc[Task Launcher]
|link:functions/consumer/redis-consumer/README.adoc[Redis]
|link:functions/supplier/sftp-supplier/README.adoc[SFTP]
|
|link:functions/consumer/rsocket-consumer/README.adoc[RSocket]
|link:functions/supplier/syslog-supplier/README.adoc[Syslog]
|
|link:functions/consumer/s3-consumer/README.adoc[AWS S3]
|link:functions/supplier/tcp-supplier/README.adoc[TCP]
|
|link:functions/consumer/sftp-consumer/README.adoc[SFTP]
|link:functions/supplier/time-supplier/README.adoc[Time]
|
|link:functions/consumer/tcp-consumer/README.adoc[TCP]
|link:functions/supplier/twitter-supplier/README.adoc[Twitter]
|link:functions/function/twitter-function/README.adoc[Twitter]
|link:functions/consumer/twitter-consumer/README.adoc[Twitter]
|link:functions/supplier/websocket-supplier/README.adoc[Websocket]
|
|link:functions/consumer/websocket-consumer/README.adoc[Websocket]
|
|
|link:functions/consumer/wavefront-consumer/README.adoc[Wavefront]
|link:functions/supplier/xmpp-supplier/README.adoc[XMPP]
|
|link:functions/consumer/xmpp-consumer/README.adoc[XMPP]
|===
=== Reusable Spring Cloud Stream Applications
|===
| Source | Processor | Sink
|link:applications/source/debezium-source/README.adoc[Debezium supplier]
|link:applications/processor/aggregator-processor/README.adoc[Aggregator]
|link:applications/sink/analytics-sink/README.adoc[Analytics]
|link:applications/source/file-source/README.adoc[File]
|link:applications/processor/bridge-processor/README.adoc[Bridge]
|link:applications/sink/cassandra-sink/README.adoc[Cassandra]
|link:applications/source/ftp-source/README.adoc[FTP]
|link:applications/processor/filter-processor/README.adoc[Filter]
|link:applications/sink/elasticsearch-sink/README.adoc[Elasticsearch]
|link:applications/processor/groovy-processor/README.adoc[Groovy]
|link:applications/sink/file-sink/README.adoc[Fiile]
|link:applications/source/http-source/README.adoc[HTTP]
|link:applications/processor/header-enricher-processor/README.adoc[Header-Enricher]
|link:applications/sink/ftp-sink/README.adoc[FTP]
|link:applications/source/jdbc-source/README.adoc[JDBC]
|link:applications/processor/http-request-processor/README.adoc[HTTP Request]
|link:applications/source/jms-source/README.adoc[JMS]
|link:applications/processor/image-recognition-processor/README.adoc[Image Recognition(Tensorflow)]
|link:applications/sink/jdbc-sink/README.adoc[JDBC]
|link:applications/source/load-generator-source/README.adoc[Load-Generator]
|link:applications/processor/object-detection-processor/README.adoc[Object Detection(Tensorflow)]
|link:applications/sink/log-sink/README.adoc[Log]
|link:applications/source/mail-source/README.adoc[Mail]
|
|link:applications/sink/mongodb-sink/README.adoc[MongoDB]
|link:applications/source/mongodb-source/README.adoc[MongoDB]
|link:applications/processor/semantic-segmentation-processor/README.adoc[Semantic Segmentation(Tensorflow)]
|link:applications/sink/mqtt-sink/README.adoc[MQTT]
|link:applications/source/mqtt-source/README.adoc[MQTT]
|link:applications/processor/script-processor/README.adoc[Script]
|link:applications/sink/pgcopy-sink/README.adoc[Pgcopy]
|link:applications/source/rabbit-source/README.adoc[RabbitMQ]
|link:applications/processor/splitter-processor/README.adoc[Splitter]
|link:applications/sink/rabbit-sink/README.adoc[RabbitMQ]
|link:applications/source/s3-source/README.adoc[AWS S3]
|link:applications/processor/transform-processor/README.adoc[Transform]
|link:applications/sink/redis-sink/README.adoc[Redis]
|link:applications/source/sftp-source/README.adoc[SFTP]
|link:applications/processor/twitter-trend-processor/README.adoc[Twitter Trend]
|link:applications/sink/router-sink/README.adoc[Router]
|link:applications/source/syslog-source/README.adoc[Syslog]
|
|link:applications/sink/rsocket-sink/README.adoc[RSocket]
|link:applications/source/tcp-source/README.adoc[TCP]
|
|link:applications/sink/sftp-sink/README.adoc[SFTP]
|link:applications/source/time-source/README.adoc[Time]
|
|link:applications/sink/tasklauncher-sink/README.adoc[Task Launcher]
|link:applications/source/twitter-message-source/README.adoc[Twitter Message]
|
|link:applications/sink/tcp-sink/README.adoc[TCP]
|link:applications/source/twitter-search-source/README.adoc[Twitter Search]
|
|link:applications/sink/throughput-sink/README.adoc[Throughput]
|link:applications/source/twitter-stream-source/README.adoc[Twitter Stream]
|
|link:applications/sink/twitter-message-sink/README.adoc[Twitter Message]
|link:applications/source/websocket-source/README.adoc[Websocket]
|
|link:applications/sink/twitter-update-sink/README.adoc[Twitter Update]
|link:applications/source/xmpp-source/README.adoc[XMPP]
|
|link:applications/sink/wavefront-sink/README.adoc[Wavefront]
|
|
|link:applications/sink/websocket-sink/README.adoc[Websocket]
|
|
|link:applications/sink/xmpp-sink/README.adoc[XMPP]
|===
=== Build
This build requires GraalVM JDK 17 distribution in order to build the script function and processor.
If you are building locally and don't want to publish to a repository set: `export LOCAL=true`
==== Core
This will build the core functionality and all functions.
[source,shell]
....
./build-core.sh
....
You can then build the desired apps.
==== Application
[source,shell]
....
./build-app.sh . applications/sink/log-sink
....
NOTE: In order to disable metrics by default there needs to be application properties configured like in `default-application.properties`. The `build-app.sh` script will copy default-application.properties into src/main/resources if no application.properties,yml,yaml or json is present.
=== Additional Resources
Here is a list of resources where you can find out more about using and developing functions and stream applications:
* link:docs/FunctionComposition.adoc[Function Composition]
* link:docs/Contributing.adoc[Contributing a New Function or Application to this Repository]
=== Code of Conduct
Please see our https://github.com/spring-projects/.github/blob/master/CODE_OF_CONDUCT.md[Code of Conduct]
Reference in New Issue View Git Blame Copy Permalink