Fixes https://github.com/spring-cloud/stream-applications/issues/500
When `listeners` are provided for `DefaultKafkaConsumerFactory`,
the target `KafkaConsumer` instance is proxied.
The `java.lang.reflect.Proxy` is `Serializable`,
but the value it is wrapping is not.
When the `MessageHeaders` is serialized (e.g. into persistent `MessageStore`),
it checks for `Serializable` type only on top-level object of the header.
Therefore, the `Proxy` is passing condition, but eventually we fail
with `NotSerializableException`, since the proxied object is not like that
* Remove `kafka_consumer` from a message before it reaches an aggregator
with its logic to serialize message into the store
This is a workaround until Spring for Apache Kafka is released
with the fix: https://github.com/spring-projects/spring-kafka/pull/2822
* Upgrade to Spring Integration AWS `3.0.1`
* Upgrade to Spring Cloud AWS `3.0.1`
* Use `spring-cloud-aws-dependencies` BOM to manage Spring Cloud AWS deps, as well as AWS SDK
* Remove `AmazonS3Properties` in favor of similar properties in the Spring Cloud AWS auto-configuration
* Remove `CompatibleStorageAmazonS3Configuration` in favor of its code migration to the single `AmazonS3Configuration`
* Fix `supplier`, `consumer`, `source` and `sink` for S3 to use API and programing model from the mentioned upgrades
* Add extra `dataflow-configuration-metadata` entries for the mentioned properties from the Spring Cloud AWS auto-configuration
* Regenerate README for `s3-source` and `s3-sink`
* Fix `metadata-store-common` to use the latest Spring Integration AWS for DynamoDB store impl
* * Fix `AmazonS3ConfigurationTests` for missed `AwsAutoConfiguration`
in an `ApplicationContextRunner` setup
* * Remove unused import in the `MetadataStoreAutoConfigurationTests`
* * Remove unused import in the `AbstractAwsS3ConsumerMockTests`
* GH-265: Add Kafka Consumer and Sink
Fixes https://github.com/spring-cloud/stream-applications/issues/265
* Make `kafkaPublisher` based on the Spring for Apache Kafka auto-configuration
and Spring Integration channel adapter, essentially `KafkaProducerMessageHandler`
* Make this `KafkaPublisherConfiguration` as an auto-configuration by itself.
* Expose those simple properties required by the `KafkaProducerMessageHandlerSpec`
* Add `kafka-sink` module based on the `kafkaPublisher`
* Add `kafka-sink` into apps metadata properties
Fixes https://github.com/spring-cloud/stream-applications/issues/275
Many functions make use of `ComponentCustomizer` ot let end-user
to provide any custom configuration for target component used by the function.
First of all for use-case which are not covered by standard configuration
properties and secondly for those options which cannot be represented
as a configuration option.
* Add a sentence to READMEs of those functions which make use of `ComponentCustomizer`
with an expected generic argument type
* Add README into `rabbit-supplier` where it is fully missed at the moment
- Move the `DebeziumEngineBuilderAutoConfiguration` and META-INF configurations to new function project: debezium-autoconfigure.
- Introduce the `EmbeddedEngineExecutorService` helper class for help DebezuimEngine implementations.
- Move the relevant tests from debezium-supplier to the debeizum-autoconfigure.
- Add detailed README documentation.
- Adjust the debeizum-supplier and debezium-source configurations.
- Clean the related tests.
Resolves#462
---------
Co-authored-by: Artem Bilan <abilan@vmware.com>
Add `JsonBytesToMap` function to allow conversion between byte[] and JSON Map.
Fixes https://github.com/spring-cloud/stream-applications/issues/441
* Introduce a `JsonBytesToMap` as a part of a `payload-converter-function` module
which is auto-discovered by Spring Cloud Function scanning algorithm - the `functions` package.
* Add a `payload-converter-function` as dependency into an `aggregator-function`
* Compose `jsonBytesToMap|aggregatorFunction` for the `aggregator-processor`
* Verify a `JsonBytesToMap` function in action with an `AggregatorProcessorTests`
* Mentioned such a payload conversion in the `aggregator-processor` README
* Factor out the Debezium Engine Builder auto-configuration
- Rename DebeziumEngineAutoConfiguration into DebeziumEngineBuilderAutoConfiguration.
- Make the DebeziumEngineBuilderAutoConfiguration auto-create a DebeizumEngine.Builder instance.
- Move the DebeizumEngine bean creation to the DebeziumReactiveConsumerConfiguration.
- Make DebeziumEngineBuilderAutoConfiguration conditional on Debeizum connecgtor on the class path and debezium.properties.connector.class property set.
- Minor improvment on the serialization format class resolution.
- Add support for debezium header serialization format, required for the change event headers to work.
Resolves#458
- Add AVRO format support. Defaults to JSON
- Remove the Metadata repo configuration.
- Remove the Spring property sort-cuts. Only the native Debezium properties are used.
- change the native properties prefix from `cdc.config to cdc.debezium`.
- Removes the need to fork and modify the Debezium engine code.
- Add binding name strategy.
Remove old CDC projects and replace with the new implementation
Remove duplicate pom properties
remove non existing pom module
Rename `cdc-debezium-supplier/source` to `debezium-supplier/source`
streamline naming and tests
streamline supplier tests
test streamline. Github workflow
Fix Debezium engine generics mess
Break `DebeziumConfiguration` into engine auto-config and consumer configuration
add initial reactive consumer impl.
Splits Debezium into a pure supplier and a streaming source
- Remove any SCS related dependencies from the supplier.
- Supplier uses Sinks.Many.
complete the supplier vs source separation
Improve Debezium Egine auto-conf
- Provide configuration hooks for OffsetCommitPolicy, ConnectorCallback,
CompletionCallback and Clock with default implementations.
- Remove obsolete code.
minor improvements
Rename cdc property prefix to `debezium` and `cdc.debezium` to `debezium.inner`
remove obsolete `StreamBridge` tests
address reviews. improve testing
Supplier tests. Rename `inner` to `properties`.
Streamline the README. Minor improvements
streamline supplier tests
final fixes
Fixes https://github.com/spring-cloud/stream-applications/issues/431
* Use `WatchService` for local file `MessageSource` in `AwsS3SupplierConfiguration`
for `S3InboundFileSynchronizingMessageSource`.
This way we are able to scan sub-dirs in the local file store and skip those
sub-dirs as entities.
The `S3InboundFileSynchronizingMessageSource` is able to properly create a local sub-dir
according to the complex key of S3 object, for example if we have a remove entry as `myDir/myFile.txt`,
so in the `/local` dir it will be stored as `/local/myDir/myFile.txt`.
The mentioned `WatchService` can react to entries of entire file tree.
