* GH-265: Add Kafka Consumer and Sink
Fixes https://github.com/spring-cloud/stream-applications/issues/265
* Make `kafkaPublisher` based on the Spring for Apache Kafka auto-configuration
and Spring Integration channel adapter, essentially `KafkaProducerMessageHandler`
* Make this `KafkaPublisherConfiguration` as an auto-configuration by itself.
* Expose those simple properties required by the `KafkaProducerMessageHandlerSpec`
* Add `kafka-sink` module based on the `kafkaPublisher`
* Add `kafka-sink` into apps metadata properties
Fixes https://github.com/spring-cloud/stream-applications/issues/275
Many functions make use of `ComponentCustomizer` ot let end-user
to provide any custom configuration for target component used by the function.
First of all for use-case which are not covered by standard configuration
properties and secondly for those options which cannot be represented
as a configuration option.
* Add a sentence to READMEs of those functions which make use of `ComponentCustomizer`
with an expected generic argument type
* Add README into `rabbit-supplier` where it is fully missed at the moment
- Move the `DebeziumEngineBuilderAutoConfiguration` and META-INF configurations to new function project: debezium-autoconfigure.
- Introduce the `EmbeddedEngineExecutorService` helper class for help DebezuimEngine implementations.
- Move the relevant tests from debezium-supplier to the debeizum-autoconfigure.
- Add detailed README documentation.
- Adjust the debeizum-supplier and debezium-source configurations.
- Clean the related tests.
Resolves#462
---------
Co-authored-by: Artem Bilan <abilan@vmware.com>
Add `JsonBytesToMap` function to allow conversion between byte[] and JSON Map.
Fixes https://github.com/spring-cloud/stream-applications/issues/441
* Introduce a `JsonBytesToMap` as a part of a `payload-converter-function` module
which is auto-discovered by Spring Cloud Function scanning algorithm - the `functions` package.
* Add a `payload-converter-function` as dependency into an `aggregator-function`
* Compose `jsonBytesToMap|aggregatorFunction` for the `aggregator-processor`
* Verify a `JsonBytesToMap` function in action with an `AggregatorProcessorTests`
* Mentioned such a payload conversion in the `aggregator-processor` README
* Factor out the Debezium Engine Builder auto-configuration
- Rename DebeziumEngineAutoConfiguration into DebeziumEngineBuilderAutoConfiguration.
- Make the DebeziumEngineBuilderAutoConfiguration auto-create a DebeizumEngine.Builder instance.
- Move the DebeizumEngine bean creation to the DebeziumReactiveConsumerConfiguration.
- Make DebeziumEngineBuilderAutoConfiguration conditional on Debeizum connecgtor on the class path and debezium.properties.connector.class property set.
- Minor improvment on the serialization format class resolution.
- Add support for debezium header serialization format, required for the change event headers to work.
Resolves#458
- Add AVRO format support. Defaults to JSON
- Remove the Metadata repo configuration.
- Remove the Spring property sort-cuts. Only the native Debezium properties are used.
- change the native properties prefix from `cdc.config to cdc.debezium`.
- Removes the need to fork and modify the Debezium engine code.
- Add binding name strategy.
Remove old CDC projects and replace with the new implementation
Remove duplicate pom properties
remove non existing pom module
Rename `cdc-debezium-supplier/source` to `debezium-supplier/source`
streamline naming and tests
streamline supplier tests
test streamline. Github workflow
Fix Debezium engine generics mess
Break `DebeziumConfiguration` into engine auto-config and consumer configuration
add initial reactive consumer impl.
Splits Debezium into a pure supplier and a streaming source
- Remove any SCS related dependencies from the supplier.
- Supplier uses Sinks.Many.
complete the supplier vs source separation
Improve Debezium Egine auto-conf
- Provide configuration hooks for OffsetCommitPolicy, ConnectorCallback,
CompletionCallback and Clock with default implementations.
- Remove obsolete code.
minor improvements
Rename cdc property prefix to `debezium` and `cdc.debezium` to `debezium.inner`
remove obsolete `StreamBridge` tests
address reviews. improve testing
Supplier tests. Rename `inner` to `properties`.
Streamline the README. Minor improvements
streamline supplier tests
final fixes
Fixes https://github.com/spring-cloud/stream-applications/issues/431
* Use `WatchService` for local file `MessageSource` in `AwsS3SupplierConfiguration`
for `S3InboundFileSynchronizingMessageSource`.
This way we are able to scan sub-dirs in the local file store and skip those
sub-dirs as entities.
The `S3InboundFileSynchronizingMessageSource` is able to properly create a local sub-dir
according to the complex key of S3 object, for example if we have a remove entry as `myDir/myFile.txt`,
so in the `/local` dir it will be stored as `/local/myDir/myFile.txt`.
The mentioned `WatchService` can react to entries of entire file tree.
The script file may have several DDL expression to execute.
They probably have to be called one-by-one as they are declared.
For example our `init-db.cql` has DDL to delete table and the next to create.
* Fix `CassandraAppClusterConfiguration.keyspaceInitializer()`
to use a `Flux.concatMap()` instead of `flatMap()` to ensure sequential
items processing in the order they appear upstream
