Commit Graph

347 Commits

Author SHA1 Message Date
Christian Tzolov
6cf66ec779 Refactor CDC Debezium Source
- Add AVRO format support. Defaults to JSON
 - Remove the Metadata repo configuration.
 - Remove the Spring property sort-cuts. Only the native Debezium properties are used.
 - change the native properties prefix from `cdc.config to cdc.debezium`.
 - Removes the need to fork and modify the Debezium engine code.
 - Add binding name strategy.

Remove old CDC projects and replace with the new implementation
Remove duplicate pom properties
remove non existing pom module
Rename `cdc-debezium-supplier/source` to `debezium-supplier/source`
streamline naming and tests
streamline supplier tests
test streamline. Github workflow
Fix Debezium engine generics mess
Break `DebeziumConfiguration` into engine auto-config and consumer configuration
add initial reactive consumer impl.
Splits Debezium into a pure supplier and a streaming source

  - Remove any SCS related dependencies from the supplier.
  - Supplier uses Sinks.Many.

complete the supplier vs source separation
Improve Debezium Egine auto-conf

 - Provide configuration hooks for OffsetCommitPolicy, ConnectorCallback,
   CompletionCallback and Clock with default implementations.
 - Remove obsolete code.

minor improvements
Rename cdc property prefix to `debezium` and `cdc.debezium` to `debezium.inner`
remove obsolete `StreamBridge` tests
address reviews. improve testing
Supplier tests. Rename `inner` to `properties`.
Streamline the README. Minor improvements
streamline supplier tests
final fixes
2023-05-02 16:07:28 -04:00
Corneil du Plessis
85f1629fee Revert deploy-plugin to 2.8.2 for stream-applications-descriptor 2023-04-21 11:07:43 +02:00
Chris Bono
d61bb6db2b Fix ElasticsearchSink index name header (#446)
- Use `Consumer<Message<?>>` on IntegrationFlow
  rather than `Consumer`

Fixes #440
2023-04-11 15:50:59 -05:00
Corneil du Plessis
b0b3f14992 Fix header filter test. Improve aggregator-filter test configuration. 2023-04-06 12:21:08 +02:00
Corneil du Plessis
2b66ea3bff Added header-filter function and processor. 2023-04-05 18:02:33 +02:00
Chris Bono
92dc5b1047 Clarify sftp-supplier docs
* Highlight the `ref` mode invalid when streaming
* Use properties rather than shell options in docs

See https://github.com/spring-cloud/spring-cloud-dataflow/issues/5225
2023-02-23 09:10:17 -05:00
Corneil du Plessis
8c07e87309 Fix checkstyle errors. 2023-02-23 11:09:27 +02:00
abilan
f6939fcc25 GH-431: Support sub-dirs in S3 Source
Fixes https://github.com/spring-cloud/stream-applications/issues/431

* Use `WatchService` for local file `MessageSource` in `AwsS3SupplierConfiguration`
for `S3InboundFileSynchronizingMessageSource`.
This way we are able to scan sub-dirs in the local file store and skip those
sub-dirs as entities.

The `S3InboundFileSynchronizingMessageSource` is able to properly create a local sub-dir
according to the complex key of S3 object, for example if we have a remove entry as `myDir/myFile.txt`,
so in the `/local` dir it will be stored as `/local/myDir/myFile.txt`.
The mentioned `WatchService` can react to entries of entire file tree.
2023-02-23 10:37:46 +02:00
Corneil du Plessis
8e4c20dc77 GH-430: Change Filter function to non-reactive
Fixes https://github.com/spring-cloud/stream-applications/issues/430

Fix greenmail exclusion.
Added -X for github.debug
2023-02-14 11:23:31 -05:00
Corneil du Plessis
d29546c9bd Added logging to aggregator. 2023-02-14 16:59:41 +02:00
Corneil du Plessis
49736da55e Remove assertion for list 2023-02-09 12:29:59 +02:00
Corneil du Plessis
3bf8656c8f Remove assertion for list 2023-02-09 11:39:33 +02:00
Corneil du Plessis
3a0479a3ad Increase timeout for DefaultAggregatorTests 2023-02-07 21:40:28 +02:00
Corneil du Plessis
e9d8fb0c31 Added xmpp to the documentation.
Fixing remaining versions.

Updates #429
2023-02-02 18:51:01 +02:00
Corneil du Plessis
5a7e570bc1 Improve WebSocket stability. 2023-01-17 12:58:43 +02:00
Corneil du Plessis
722fb9182a Added <protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact> to protoc-jar-maven-plugin in object-detection-function. 2023-01-17 11:25:55 +02:00
Corneil du Plessis
cf13062d8d Fix checkstyle issue 2023-01-13 16:16:00 +02:00
Corneil du Plessis
731c48a839 Added GreenMail.
Update all tests for mail-supplier and mail-sink.
2023-01-12 16:39:53 +02:00
Corneil du Plessis
9f9b6a0c19 Fix checkstyle errors. 2023-01-12 11:05:43 +02:00
Corneil du Plessis
1abad6d76c Fix object-detection dependencies and cleanup samples. 2023-01-12 10:50:28 +02:00
Chris Bono
14390fe040 GH-327: Use TestSocketUtils from spring-test (#420)
Fixes https://github.com/spring-cloud/stream-applications/issues/327

- Delete local TSU temporary implementations
- Replace local TSU w/ spring-test TSU
2023-01-11 11:33:31 -05:00
Corneil du Plessis
570a8ec777 Upodated aggregator RedisMessageStoreAggregatorTests to use RedisTestContainerSupport. 2023-01-11 16:20:14 +02:00
Corneil du Plessis
fc8dc13e07 Fix } else 2023-01-09 12:59:33 +02:00
Corneil du Plessis
189341b8f4 Added TODOs update versions. 2023-01-06 16:49:14 +02:00
Corneil du Plessis
8eee517a49 Update test-binder dependency.
Added elasticsearch-x-content.
2022-12-01 15:17:44 +02:00
Corneil du Plessis
0c78c98439 [SB3] Update startup timeouts and retries for containers. 2022-11-25 15:36:10 +02:00
Corneil du Plessis
ceabfb483b [SB3] Update Elasticsearch Client. Update Boot and Spring versions. Improve Rabbit listener (#416) 2022-11-25 11:51:53 +02:00
Corneil du Plessis
ad5445e25e [GitHub Actions] Tune thread count for Maven and surefire plugin forks. Increase runners with 10% more than requested. 2022-11-17 21:53:41 +02:00
Corneil du Plessis
f55fa3d6ae [SB3] Resolve race conditions. 2022-11-17 16:39:50 +02:00
Corneil du Plessis
7d79c96f8b [SB3] Change from doHandshake to execute. 2022-11-17 16:16:04 +02:00
Corneil du Plessis
e8218f8df9 [SB3] Fix tests for changes in SB3. 2022-11-17 16:11:36 +02:00
Corneil du Plessis
0550284333 [SB3] Fix tests for changes in SB3. 2022-11-17 14:41:25 +02:00
Corneil du Plessis
38f24fd2ed [SB3] Fix tests for changes in SB3. 2022-11-17 14:16:18 +02:00
Corneil du Plessis
09381b491a [SB3] Fix tests for changes in SB3. 2022-11-17 13:11:50 +02:00
abilan
d7e2975af3 Fix Cassandra init script execution
The script file may have several DDL expression to execute.
They probably have to be called one-by-one as they are declared.
For example our `init-db.cql` has DDL to delete table and the next to create.

* Fix `CassandraAppClusterConfiguration.keyspaceInitializer()`
to use a `Flux.concatMap()` instead of `flatMap()` to ensure sequential
items processing in the order they appear upstream
2022-11-15 11:50:06 -05:00
Chris Bono
1e5887f48c Remove all remnants of Geode (#414) 2022-11-11 16:20:33 -06:00
Daniel Frey
0a2c5554b9 Update XMPP supplier test for consistency (#412) 2022-11-10 11:55:47 -06:00
Daniel Frey
2cbf762422 Update xmpp consumer for consistency (#404) 2022-11-10 11:47:20 -06:00
Daniel Frey
1fe892d617 add xmpp-sink application 2022-11-09 21:50:17 -06:00
Corneil du Plessis
23bc53a71a [SB 3] Add mail version. 2022-11-08 16:06:36 +02:00
Daniel Frey
985ab8b035 Xmpp supplier (#403)
* WIP: initial xmpp supplier support

* implement xmpp supplier and test

* remove test logging config

* update for consistency

* add documentation references for xmpp consumer and supplier, fix formatting for functions where geode was commented out
2022-11-07 17:02:01 +02:00
Corneil du Plessis
3917ce241b [SB 3] Fix checkstyle in http-request-function. (#406)
* [SB 3] Fix checkstyle in http-request-function.
2022-11-07 16:57:11 +02:00
Corneil du Plessis
dd2a70dda3 Merge remote-tracking branch 'origin/main'
# Conflicts:
#	.github/workflows/ci-pr.yml
2022-11-07 15:56:35 +02:00
Corneil du Plessis
00249f911e [GitHub Actions] Improve dependencies for main. Provide for building impacted dependencies for a PR. 2022-11-07 15:56:07 +02:00
Corneil du Plessis
392b5bfbbe [Issues] Change http-request to non-reactive (#402)
Fixes #206
2022-11-04 17:40:01 +02:00
Chris Bono
a9ed883124 Polish "Move xmpp testcontainer to function-test-support"
- Removed custom container
- Removed extra whitespace
- Moved static props to SBT.properties
2022-10-28 13:38:32 -05:00
Daniel Frey
9a6f05def7 Move xmpp testcontainer to function-test-support 2022-10-28 13:22:03 -05:00
Chris Bono
256b5ccaa3 Fix checkstyle from previous commit 2022-10-25 23:44:46 -05:00
Chris Bono
42cb06306d Fix pgcopy-sink tests
- Update build steps in README.adoc
- Remove junit4 usages across project
- Replace postgres ClassRule w/ Extension
2022-10-25 23:15:47 -05:00
Chris Bono
cb0a9276cc Update CDC Debezium to 1.9.6
- The update fixes integration tests
2022-10-25 23:15:47 -05:00