70 Commits

Author SHA1 Message Date
Soby Chacko
255fc69b0c Fix checkstyle issues 2021-05-27 10:25:12 -04:00
Soby Chacko
2712dab49b Refactor the usage of output channel in suppliers
Redesigning suppliers that use reactive output channel name as
the literal "output" explicitly, as this  cuases some conflicts
with the output channel name usage in Spring Cloud Stream.
The fix is done either by simply renaming the channel name
or by avoid using it directly, but delegate its use through the SI Java DSL.

Resolves https://github.com/spring-cloud/stream-applications/issues/160
2021-05-26 15:53:01 -04:00
Andrea Montemaggio
f2e3ce271e GH-167: rename remote files in sftp-supplier
Fixes https://github.com/spring-cloud/stream-applications/issues/167

* Support for renaming remote files after processing via 'sftp.supplier.rename-remote-files-to' property.
2021-05-04 15:00:22 -04:00
Artem Bilan
04beb9ecaf GH-156: Fix NPE in the RemoteFileDeletingAdvice
Fixes https://github.com/spring-cloud/stream-applications/issues/156

The `MessageSource.receive()` may produce `null`:
The `MessageSourceMutator` impl must honor such an input

* Add `MonoProcessor<Boolean> subscriptionBarrier` to delay
subscription to the source `Flux` until subscription happens
to the supplier's flux.
This way we don't have unexpected interaction with the source when
there are regular endpoints in the flow in between
2021-04-13 09:26:30 -04:00
Dan Smith
9faa2a6a38 Use Spring Test Data Geode instead of containerized geode server
Use spring test data geode in the geode integration tests
test to fork a separate geode process, instead of running
a geode server in a docker container.

STDG is faster than launching Geode through gfsh inside of a container.
- It doesn't run in docker
- It doesn't start management components necessary for gfsh
- It doesn't start extra gfsh shell processes

Removing the GeodeContainer test utility
2021-04-13 08:39:16 -04:00
Soby Chacko
2a0edea722 Fixing TCP Source issues
Exclude `IpHeaders.LOCAL_ADDRESS` from TCP source output as it
causes issues with message conversion downstream.

Update SCSt/Spring-Cloud to the latest snapshots.

* Refactor `TcpSupplierConfiguration` for better readability and
current state of requirements for Spring libraries configuration
2021-03-30 09:00:21 -04:00
Thomas Recloux
126c53e8ad GH-151: Remove output bean from the Rabbit Supplier
Fixes https://github.com/spring-cloud/stream-applications/issues/151

The `RabbitSupplierConfiguration` comes with the `output` bean definition which
clashes with the `output` binding for Spring Cloud Stream.
The last one doesn't register the binding bean and leave the rest of the logic
to deal with existing bean when we resolve it by the `output` name.

* Rework the logic of the `RabbitSupplierConfiguration` to avoid an extra bean
in between and the binding in Spring Cloud Stream do it job
2021-03-10 13:09:12 -05:00
Artem Bilan
4eb34fb245 GH-154: Change FORM CT to JSON for HTTP supplier
Fixes https://github.com/spring-cloud/stream-applications/issues/154

The Binder conversion mechanism is not aware about a `MediaType.APPLICATION_FORM_URLENCODED`
and does not know how to convert the `MultiValueMap` with such a content-type

* Change the Http-Supplier to replace a `MediaType.APPLICATION_FORM_URLENCODED`
content-type header to the `MediaType.APPLICATION_JSON` for the proper
JSON serialization for the `MultiValueMap` on the binder level before sending to
the output destination
2021-03-09 11:49:08 -05:00
David Turanski
f1d9f36dee Tag integration tests and enable integration profile 2021-02-22 14:09:22 -05:00
Artem Bilan
e2c11b4773 Upgrade Testcontainers tests to the latest images
* Un`@Disable` updated tests
2021-02-11 16:21:06 -05:00
Soby Chacko
63c22a85d7 tests debugging continues 2021-02-10 14:26:24 -05:00
Artem Bilan
1e6a2e1f44 GH-143: Add an update option to MongoDB Supplier (#147)
* GH-143: Add an update option to MongoDB Supplier

Fixes https://github.com/spring-cloud/stream-applications/issues/143

* * Fix Checkstyle violations
* Update `README.adoc` for `mongodb-source`
2021-02-05 15:36:03 -05:00
Soby Chacko
fec91314b9 More checkstyle fixes 2021-02-05 11:36:33 -05:00
Soby Chacko
172e345c2d Fixing checkstyle issues 2021-02-05 11:16:45 -05:00
Artem Bilan
17162dd775 GH-146: SFTP supplier: Use proper FileListFilter
Fixes https://github.com/spring-cloud/stream-applications/issues/146

The `SftpRegexPatternFileListFilter` is used for the `filenamePattern` option, too.
It seems like a copy/paste artefact

* Change `filenamePattern`-based configuration for the `SftpSimplePatternFileListFilter`
* Fix `SftpSupplierApplicationTests` for multi-OS compatibility
2021-02-03 10:20:10 -05:00
Andrea Montemaggio
65f3b82d0b Introduce basic support for user-configurable sorting of the remote file list. 2021-01-14 10:57:43 -05:00
Soby Chacko
c07aeeae13 remove unnecessary mvnw scripts from rabbit-supplier 2020-12-15 14:09:26 -05:00
Christian Tzolov
a9692efc75 Fix CDC issues
- Fix cdc_key header string wrapping issue that causes problems with the spring.cloud.stream.kafka.default.producer.messageKeyExpression.
 - Rename Flattering property to the correct Flattening.
 - Remove wrong dependencies.
2020-12-04 19:50:45 +01:00
David Turanski
426e058ad1 Increase timeouts on SftpSupplierApplicationTests 2020-11-10 14:11:57 -05:00
David Turanski
236ef55863 Upgrade Testcontainers, Debezium, and fix exceptions in MailSupplierTests 2020-11-06 12:32:33 -05:00
David Turanski
5e0f24a8ee Refactored debezium tests to use TestContainers 2020-10-28 16:32:28 -04:00
Soby Chacko
760683cc2a Restructuing with maven revision feature
Preserving the maven revision feature from the previous commit,
but perform a bit more restructuring the hierarchy.
2020-10-13 15:17:52 -04:00
Daniel Frey
5e9382c23a Refactor hardcoded versions in functions submodule branches
This is so they can be updated in a minimal amount of places and leverage the maven hierarchy to propagate them where appropriate.
While working with SNAPSHOT releases of Spring Boot, Spring Integration, and Spring Cloud, it required touching every pom.xml file
in order to reflect the use of the SNAPSHOT. After this change, it will only be required to change the property in the top
functions pom.xml and the versions will reflect to all submodules.

Maven introduced the ability use a key property, revision to propagate this information to the <parent> declarations in each pom.

This was enabled in Maven 3.5+
https://maven.apache.org/maven-ci-friendly.html
2020-10-13 15:15:59 -04:00
David Turanski
6645b390e8 Avoid timeout when attemting to connect to AWS 2020-10-11 11:30:04 -04:00
Soby Chacko
f257f48f78 Next version updates
Functions: 1.0.0-SNAPSHOT
  Core Apps: 3.0.0-SNAPSHOT
  Apps: 3.0.0-SNAPSHOT
  Aggregate Next:2020.0.0-SNAPSHOT
2020-09-29 18:28:20 -04:00
Soby Chacko
73a3e55299 Functions: Release - 1.0.0-M3 2020-09-29 14:59:33 -04:00
David Turanski
4048221f1e Fix race condition and other errors in S3Supplier 2020-09-25 18:20:16 -04:00
Artem Bilan
696ebca135 Defer subscription to the S3 until downstream one
It turns out that endpoint for S3 Source is starting to
consume the `Flux` earlier than an actual subscription
happens for the whole flow downstream

* Introduce a `MonoProcessor<Subscription> downstreamSubscription`
to fulfill from the downstream subscription and handle from the
`delaySubscription()` on the `Flux` for source data
2020-09-25 13:28:18 -04:00
David Turanski
02ade7ae25 Add list-only to README 2020-09-21 13:47:06 -04:00
David Turanski
ddcc10363e List only enhancements
* Add metadata.store.type Property and idempotent SftpSupplier for list-only

* Implement list-only for S3 source and Optimize metadastore access

* Fixed build and READMEs

* Change to ConditionalOnProperty

* Change to ReactiveMessageProducer

* Update cdc-debezium-source/README.adoc

* Make all MetadataStoreProperties visible
2020-09-17 15:36:36 -04:00
Soby Chacko
0378f7d3a0 Migrating Mail Source
* Mail supplier for receiving email from a URL (IMAP/POP3)
 * Generating mail source apps
 * Migrating tests

* Addressing PR review comments

* Addressing PR review comments
2020-09-17 15:18:59 -04:00
David Turanski
53369a2af2 Geode test container will not start locator by default 2020-09-07 13:21:27 -04:00
Christian Tzolov
704c8ba569 fix checkstyle issue 2020-08-20 20:51:34 +02:00
David Turanski
cc11b59c29 Fix no factory bean warnings 2020-08-19 12:37:15 -04:00
Soby Chacko
71ec62b58e Syslog supplier/source
- Provide TCP/UDP based syslog supplier and then generate apps based on that
2020-08-18 15:37:50 -04:00
Christian Tzolov
8ab33a195d CDC Debezium Supplier, Source Boot Starter
- Add supplier, source and boot auto-config.
 - Add work around for KafkaNull handling.
 - Update CDC to 1.2.1.Final. Replace the deprecated properties.

remove comments

Add missing metadata configuration. Improve README docs.

CDC: add all metadata for doc auto-generation

Remove custom KafkaNullConverter.

  - The fixed KafkaNullConverter implementation comes with kafka binder 3.0.8-BUILD-SNAPSHOT or newer.

Add repositories to maven config in app generation project.

Remove the unnecessary function version in maven plugin configuration for app generation.
2020-08-13 15:41:25 -04:00
David Turanski
7d4e5a9ea0 Add OSS JDBC drivers to consumer and supplier 2020-08-07 17:45:02 -04:00
Christian Tzolov
08de5c3355 Fix javadoc checkstyle issue. Update generated READMEs 2020-08-06 12:51:47 +02:00
Christian Tzolov
2ffbe371b6 Remove double quotes in the properties comments 2020-08-05 15:18:25 +02:00
David Turanski
3e776e611a Fix checkstyle 2020-07-21 09:48:57 -04:00
David Turanski
a841941736 Add try-finally 2020-07-21 09:48:01 -04:00
Soby Chacko
d59e4f5d86 Next version updates
Functions: 1.0.0-SNAPSHOT
  Core Apps: 3.0.0-SNAPSHOT
  Apps: 3.0.0-SNAPSHOT
  Aggregate Next:2020.0.0-SNAPSHOT
2020-07-13 16:02:05 -04:00
Soby Chacko
7066283aa2 Functions: Release - 1.0.0-M2 2020-07-13 14:43:36 -04:00
Timo Salm
c21533e42c Support for s3 compatible storage
* Add support for s3 compatible storage to s3 consumer and supplier.

* Add enhancements to s3 compatible storage implementation based on code review

* Move s3 compatible storage endpoint configuration bean to AmazonS3Configuration to remove code duplication

* Disable auto configuration of AmazonS3 Bean to enable custom configurations

* Add documentation for endpoint url configuration property

* Use localhost instead of a real URL to avoid an unexpected connection to the internet
2020-07-07 14:27:31 -04:00
Soby Chacko
49468e6148 Renaming internal test boot applications in functions 2020-07-02 18:31:58 -04:00
David Turanski
2b358838c7 Fix import order 2020-07-01 19:00:52 -04:00
David Turanski
8cc6f546b6 Implement SFTP Source
Address review comments

* Code style clean up - generics warnings
* Fix `SftpSupplierApplicationTests` for compatibility with Windows
2020-07-01 15:35:27 -04:00
Soby Chacko
5c59ff6823 Next version updates
Functions: 1.0.0-SNAPSHOT
  Core Apps: 3.0.0-SNAPSHOT
  Apps: 3.0.0-SNAPSHOT
  Aggregate Build: 2020.0.0-SNAPSHOT
2020-06-26 13:49:35 -04:00
Soby Chacko
327641fb05 Functions: Release - 1.0.0-M1 2020-06-25 17:42:02 -04:00
Christian Tzolov
9792768d55 Address twitter review suggestions 2020-06-24 09:58:26 +02:00