417 Commits

Author SHA1 Message Date
Soby Chacko
40df881d05 Cleaning up - pgcopy sink apps 2020-09-28 18:18:31 -04:00
Soby Chacko
3ce5f50cf9 RSocket consumer/sink
* This RSocket consumer is using the fire-and-forget strategy of execution.
* Generating the corresponding sink.
* Addressing the original PR review comments from
  https://github.com/spring-cloud/stream-applications/pull/116

* Addressing PR review comments

* Fix README

* Addressing PR review
2020-09-28 16:28:09 -04:00
David Turanski
8143c63190 Fix S3 consumer dependencies 2020-09-28 08:22:41 -04:00
David Turanski
4048221f1e Fix race condition and other errors in S3Supplier 2020-09-25 18:20:16 -04:00
David Turanski
6c69d06bc9 Add spring-integration-bom to app dependencyManagement 2020-09-25 14:59:40 -04:00
Artem Bilan
696ebca135 Defer subscription to the S3 until downstream one
It turns out that endpoint for S3 Source is starting to
consume the `Flux` earlier than an actual subscription
happens for the whole flow downstream

* Introduce a `MonoProcessor<Subscription> downstreamSubscription`
to fulfill from the downstream subscription and handle from the
`delaySubscription()` on the `Flux` for source data
2020-09-25 13:28:18 -04:00
David Turanski
3bb3c0b6a1 Upgate Spring Integration dependencies 2020-09-24 13:09:48 -04:00
David Turanski
3e3b770742 Enhance FunctionBindingEnvironmentPostProcessor to handle quotes in function definition 2020-09-24 11:30:15 -04:00
David Turanski
c2b778fd3b Fix TimeSourceTests 2020-09-22 15:06:34 -04:00
Christian Tzolov
d42d6fdc3a Set apps baseImage global to springcloud/baseimage:1.0.0 2020-09-22 11:43:21 +02:00
David Turanski
02ade7ae25 Add list-only to README 2020-09-21 13:47:06 -04:00
Soby Chacko
a7a37950e9 Migrating pgcopy-sink
* pgcopy-sink is migrated from Spring Cloud Stream App Starters repository.
 * This is not converted as a function, rather preserved as a sink app.
 * Old location: https://github.com/spring-cloud-stream-app-starters/jdbc/tree/master/spring-cloud-starter-stream-sink-pgcopy
2020-09-18 20:05:03 -04:00
David Turanski
ddcc10363e List only enhancements
* Add metadata.store.type Property and idempotent SftpSupplier for list-only

* Implement list-only for S3 source and Optimize metadastore access

* Fixed build and READMEs

* Change to ConditionalOnProperty

* Change to ReactiveMessageProducer

* Update cdc-debezium-source/README.adoc

* Make all MetadataStoreProperties visible
2020-09-17 15:36:36 -04:00
Soby Chacko
0378f7d3a0 Migrating Mail Source
* Mail supplier for receiving email from a URL (IMAP/POP3)
 * Generating mail source apps
 * Migrating tests

* Addressing PR review comments

* Addressing PR review comments
2020-09-17 15:18:59 -04:00
David Turanski
f7efc39164 Remove SplitterFunction autoconfiguration for sources 2020-09-14 14:00:52 -04:00
David Turanski
d5d437ec04 Add composite function support to S3 source 2020-09-14 12:56:25 -04:00
David Turanski
fd5cadc60d Add use-path-style-access to AmazonS3Properties 2020-09-09 18:26:20 -04:00
David Turanski
7dacd25a60 Fix s3 READMEs 2020-09-08 09:31:47 -04:00
David Turanski
53369a2af2 Geode test container will not start locator by default 2020-09-07 13:21:27 -04:00
David Turanski
d920b8cb27 Replace deprecated code 2020-09-04 09:54:52 -04:00
Christian Tzolov
f67cdb1237 Fix application tables 2020-09-02 20:56:53 +02:00
Christian Tzolov
87ab441633 set global container org name 2020-09-02 17:20:03 +02:00
Soby Chacko
ac965dd756 Update metadata plugin version
The generated apps use a new version of the metadata plugin --
spring-cloud-dataflow-apps-metadata-plugin 1.0.0-SNAPSHOT.
2020-08-31 16:27:21 -04:00
Soby Chacko
25f324c605 Update docs plugin maven coordinates 2020-08-31 15:20:18 -04:00
Christian Tzolov
62fed3b21b Re-enable the stream-applications-micrometer-common dependency as it adds CF metrics tags when the Cloud profile is set 2020-08-31 09:51:36 +02:00
Christian Tzolov
b6456d2b7d Configure a default base container image
Resolves #112
2020-08-31 09:45:50 +02:00
Christian Tzolov
b9f6d9e9a9 Fix another filter function checkstyle issue 2020-08-31 08:47:15 +02:00
Christian Tzolov
d899d0856b Fix a [ImportOrder] checkstyle issue 2020-08-31 08:05:36 +02:00
David Turanski
dc4f55d417 Fix http source properties 2020-08-30 15:11:14 -04:00
Christian Tzolov
6d7b0051c3 Replace Micrometer Env postprocessor and common tags class by core, global property configuraitons 2020-08-29 20:46:23 +02:00
Artem Bilan
a91aa2cd3f GH-110: Fix filter function for emitting null (#111)
* GH-110: Fix filter function for emitting `null`

Fixes https://github.com/spring-cloud/stream-applications/issues/110

Turns out when functions are composed in the Spring Cloud Stream environment,
they are called via reactive wrappers which don't allow to emit `null` from the
`map()` operator.

* Make `filterFunction` fully reactive to reply on the `Flux.filter()` operator
* Add `config-common` for auto-conversion string configuration options into `Expression` instances
* Add `spring-boot-starter-json` dependency since it is required by the `SpelExpressionConverterConfiguration`
* Add `reactor-test` dependency to test the final solution
* Remove redundant dependencies from the `filter-processor`

* * Add `proxyBeanMethods = false` into `FilterFunctionConfiguration`
* Fix default expression to `true` instead of `payload`, which does not fit to filter logic
* Fix JavaDoc for `expression` property
* Remove redundant `application.properties` from the `filter-function`
2020-08-28 15:55:08 -04:00
Soby Chacko
d962e6e233 Fixing missing filter processor properties in README 2020-08-25 14:07:06 -04:00
Soby Chacko
6a3aec8616 Syslog source - fix maven plugin coordinates for app generator 2020-08-20 17:10:50 -04:00
Artem Bilan
b27897227d Add Aggregator function & processor
* Have all the required dependencies in class path which give us a `MessageGroupStore` implementation
* Disable all their auto-configurations in the `ExcludeStoresAutoConfigurationEnvironmentPostProcessor`
* Enable only those which are configured by the `aggregator.messageStoreType` property
* Implement a fully reactive function around an aggregator making an interaction via a pair of `FluxMessageChannel`
* Fix checkstyle issues
* Change the app generator maven plugin to new coordinates
2020-08-20 15:54:51 -04:00
Christian Tzolov
e1a1c0d149 align the syslog-source's app generator plugin config with latest changes 2020-08-20 21:26:22 +02:00
Christian Tzolov
704c8ba569 fix checkstyle issue 2020-08-20 20:51:34 +02:00
Christian Tzolov
87434fdbff align POMs with the new maven app generator plugin 2020-08-20 20:27:10 +02:00
David Turanski
cc11b59c29 Fix no factory bean warnings 2020-08-19 12:37:15 -04:00
Soby Chacko
8cbddb49e6 Fix typo in debezium source 2020-08-18 15:41:50 -04:00
Soby Chacko
71ec62b58e Syslog supplier/source
- Provide TCP/UDP based syslog supplier and then generate apps based on that
2020-08-18 15:37:50 -04:00
David Turanski
19353bf7d9 GH-103: Fix file source
Fixes https://github.com/spring-cloud/stream-applications/issues/103

Excludes `splitter-function` from dependencies. 
There may be some other sources that have the same issue. 
Also, includes changes to handle function composition with , or `|` delimiter
2020-08-17 11:38:48 -04:00
Christian Tzolov
02a14b31e6 Fix cdc tests 2020-08-16 22:07:28 +02:00
Christian Tzolov
6f3dddd582 Update prometheus-rsocket.version to 1.0.0 to match Boot 2.3.x + RSocket 1.0.x. Bump the metadata-maven-plugin to 2.0.2.BS to resolve Rsocket classnotfound related issue 2020-08-16 21:16:05 +02:00
Soby Chacko
9e3c1caaf2 Fixing test 2020-08-13 16:05:32 -04:00
Christian Tzolov
8ab33a195d CDC Debezium Supplier, Source Boot Starter
- Add supplier, source and boot auto-config.
 - Add work around for KafkaNull handling.
 - Update CDC to 1.2.1.Final. Replace the deprecated properties.

remove comments

Add missing metadata configuration. Improve README docs.

CDC: add all metadata for doc auto-generation

Remove custom KafkaNullConverter.

  - The fixed KafkaNullConverter implementation comes with kafka binder 3.0.8-BUILD-SNAPSHOT or newer.

Add repositories to maven config in app generation project.

Remove the unnecessary function version in maven plugin configuration for app generation.
2020-08-13 15:41:25 -04:00
David Turanski
4acf224790 Wait for data 2020-08-13 15:01:31 -04:00
Soby Chacko
52bb359609 Increase timeout in mongodb consumer test 2020-08-13 13:36:59 -04:00
David Turanski
d13a1fceaf Add max-buffer-size and fix app deps 2020-08-13 13:27:15 -04:00
Soby Chacko
f863db2af1 Adding repos/pluginRepos to apps generation
This might be a temporory solution to address some issues on the CI envrionment.
We will most likely remove the changes from this commit later on once the CI
issues are addressed
2020-08-13 12:00:42 -04:00
David Turanski
eee707da00 Refactor to use SpEL in function properties
* Clean up per review
2020-08-12 15:39:56 -04:00