Commit Graph

399 Commits

Author SHA1 Message Date
Soby Chacko
255fc69b0c Fix checkstyle issues 2021-05-27 10:25:12 -04:00
Soby Chacko
2712dab49b Refactor the usage of output channel in suppliers
Redesigning suppliers that use reactive output channel name as
the literal "output" explicitly, as this  cuases some conflicts
with the output channel name usage in Spring Cloud Stream.
The fix is done either by simply renaming the channel name
or by avoid using it directly, but delegate its use through the SI Java DSL.

Resolves https://github.com/spring-cloud/stream-applications/issues/160
2021-05-26 15:53:01 -04:00
Soby Chacko
c90a5e44f8 Back to 1.0.3-SNAPSHOT 2021-05-21 14:50:51 -04:00
Soby Chacko
c37c403b08 stream-applications-build 1.0.3 Release 2021-05-21 14:42:27 -04:00
Soby Chacko
4ebd18ee4e Update versions
boot 2.3.11
Spring Kafka 2.5.13
SCSt Horsham.SR12/3.0.12.RELEASE
SCFn 3.0.14.RELEASE
Spring Cloud Hoxton.SR11
2021-05-20 18:27:05 -04:00
David Turanski
fbeb89b5b7 Fix restore Geode test support used by integration tests 2021-05-20 09:39:08 -04:00
Soby Chacko
31b6fa1d89 Fix issues with message conversion in mongodb sink
Fore more info see: https://stackoverflow.com/questions/67131500/mongodb-sink-issue-while-writing-a-json-data-from-http-source

Resolves https://github.com/spring-cloud/stream-applications/issues/165

This commit also addresses a doc update for sftp-source
2021-05-19 17:10:01 -04:00
David Turanski
5cbd171f1b Restore Geode test support used by integration tests 2021-05-19 10:55:32 -04:00
Andrea Montemaggio
f2e3ce271e GH-167: rename remote files in sftp-supplier
Fixes https://github.com/spring-cloud/stream-applications/issues/167

* Support for renaming remote files after processing via 'sftp.supplier.rename-remote-files-to' property.
2021-05-04 15:00:22 -04:00
Andrea Montemaggio
63dd5bc043 Added support for sync and async bulk-indexing.
New properties for controlling the batch size and the timeout for a message group have been added, consistently with the jdbc-consumer implementation.
Added support for per message index override via INDEX_NAME header.
Overriding behavior of message headers mentioned in properties javadoc.
Improved debug log messages.

Copyright and authors update.

README
2021-05-03 17:49:31 -04:00
Soby Chacko
62575ffc75 Pushing harbor bulk import files to Artifactory 2021-04-20 10:35:43 -04:00
Soby Chacko
619554a79f Adding harbor bulk import files for the apps 2021-04-20 09:39:08 -04:00
Artem Bilan
04beb9ecaf GH-156: Fix NPE in the RemoteFileDeletingAdvice
Fixes https://github.com/spring-cloud/stream-applications/issues/156

The `MessageSource.receive()` may produce `null`:
The `MessageSourceMutator` impl must honor such an input

* Add `MonoProcessor<Boolean> subscriptionBarrier` to delay
subscription to the source `Flux` until subscription happens
to the supplier's flux.
This way we don't have unexpected interaction with the source when
there are regular endpoints in the flow in between
2021-04-13 09:26:30 -04:00
Dan Smith
9faa2a6a38 Use Spring Test Data Geode instead of containerized geode server
Use spring test data geode in the geode integration tests
test to fork a separate geode process, instead of running
a geode server in a docker container.

STDG is faster than launching Geode through gfsh inside of a container.
- It doesn't run in docker
- It doesn't start management components necessary for gfsh
- It doesn't start extra gfsh shell processes

Removing the GeodeContainer test utility
2021-04-13 08:39:16 -04:00
Soby Chacko
2fed22a13f Update spring-cloud-function to 3.0.14.BUILD-SNAPSHOT.
Enable filter processor test.

Resolves https://github.com/spring-cloud/stream-applications/issues/162
2021-04-07 17:05:50 -04:00
Artem Bilan
bbf04eff2e Limit aggregator tests for 10 seconds to wait 2021-04-07 12:02:02 -04:00
Soby Chacko
b2540806ab Revert "Fix filter processor issues"
This reverts commit d8c75d5677.
2021-04-06 12:49:00 -04:00
Soby Chacko
d8c75d5677 Fix filter processor issues
Update fiilter processor to add an extra processing at the end of the chain
to convert the receeived text data into byte[].

See the previous commit message for more details.
2021-04-05 18:30:22 -04:00
Soby Chacko
b6ddad51c3 Fixing router sink issues
Router sink receives the input as String through byteArrayTextToString|routerSinkConsumer.
Spring Cloud Stream used to convert the string back to byte[], but the commit below removed that logic.
5d9de8ad57

Therefore we are adding it back in the router sink app by programmatically converting the String back to
byte[] before sending it out to the bound router channel.
2021-04-05 18:05:14 -04:00
Soby Chacko
55b306e160 Revert "Fix router sink tests"
This reverts commit d43893c1ef.
2021-04-05 15:43:52 -04:00
Soby Chacko
d43893c1ef Fix router sink tests
Disable filter integ tests temporarily
2021-04-01 16:33:24 -04:00
Soby Chacko
3db9b078d1 Spring Cloud Stream dependencies takes precedence
When given spring-cloud-stream-dependencies and spring-cloud-dependencies,
the former gets the precedence. This way, apps can upddate to a stanadlone
stream release indepndent of any upstream spring cloud release.
2021-03-31 21:15:40 -04:00
Soby Chacko
2a0edea722 Fixing TCP Source issues
Exclude `IpHeaders.LOCAL_ADDRESS` from TCP source output as it
causes issues with message conversion downstream.

Update SCSt/Spring-Cloud to the latest snapshots.

* Refactor `TcpSupplierConfiguration` for better readability and
current state of requirements for Spring libraries configuration
2021-03-30 09:00:21 -04:00
Thomas Recloux
126c53e8ad GH-151: Remove output bean from the Rabbit Supplier
Fixes https://github.com/spring-cloud/stream-applications/issues/151

The `RabbitSupplierConfiguration` comes with the `output` bean definition which
clashes with the `output` binding for Spring Cloud Stream.
The last one doesn't register the binding bean and leave the rest of the logic
to deal with existing bean when we resolve it by the `output` name.

* Rework the logic of the `RabbitSupplierConfiguration` to avoid an extra bean
in between and the binding in Spring Cloud Stream do it job
2021-03-10 13:09:12 -05:00
Artem Bilan
4eb34fb245 GH-154: Change FORM CT to JSON for HTTP supplier
Fixes https://github.com/spring-cloud/stream-applications/issues/154

The Binder conversion mechanism is not aware about a `MediaType.APPLICATION_FORM_URLENCODED`
and does not know how to convert the `MultiValueMap` with such a content-type

* Change the Http-Supplier to replace a `MediaType.APPLICATION_FORM_URLENCODED`
content-type header to the `MediaType.APPLICATION_JSON` for the proper
JSON serialization for the `MultiValueMap` on the binder level before sending to
the output destination
2021-03-09 11:49:08 -05:00
Soby Chacko
4c044d2d14 Downgrade config client starter dep to 3.2.0.RELEASE 2021-03-08 13:05:42 -05:00
Soby Chacko
0d2221d4c9 Update spring-cloud-services-starter-config-client to 3.2.1.RELEASE 2021-03-08 12:27:49 -05:00
Christian Tzolov
d304b3ed93 add spring-cloud-services-starter-config-client 2021-03-05 19:53:54 +01:00
Christian Tzolov
2c58bb36ad add spring-cloud-starter-config dependency 2021-03-05 16:51:54 +01:00
Nicolas Labrot
c899e71a64 GH-152: RabbitMQ consumer: Add headersMappedLast
Fixes https://github.com/spring-cloud/stream-applications/issues/152

* Add the possibilities to configure the `headersMappedLast` for the RabbitMQ consumer

* Changes following the code review
2021-02-26 17:00:36 -05:00
Soby Chacko
9c49c57d44 Releast train next version updates: 2020.0.2-SNAPSHOT 2021-02-26 15:41:05 -05:00
Soby Chacko
559264c197 Next apps version update: 3.0.2-SNAPSHOT 2021-02-26 15:37:46 -05:00
Soby Chacko
fa58243b31 Next version updates (non-apps) 2021-02-26 15:25:26 -05:00
Soby Chacko
21086cd248 stream-applications-release-train release: 2020.0.1 2021-02-26 14:47:25 -05:00
Soby Chacko
3c89b81088 stream-applicaitons: 3.0.1 Release 2021-02-26 10:40:11 -05:00
Soby Chacko
a603a092f7 stream-applications-core: 3.0.3 Release 2021-02-25 17:14:01 -05:00
Soby Chacko
eceda16e0a Functions 1.0.2 Release 2021-02-25 15:34:54 -05:00
Soby Chacko
f0c11dc52c stream-applications-build 1.0.2 Release 2021-02-25 15:26:17 -05:00
David Turanski
f1d9f36dee Tag integration tests and enable integration profile 2021-02-22 14:09:22 -05:00
Artem Bilan
386dad6562 Use compatible image version with Debezium
* Fix MongoDb driver problem.
Probably we need to think about upgrading Debezium support to the latest
version, so its MongoDb connector is based on the latest Mongo Driver
which already is pulled by Spring Boot
2021-02-19 12:31:34 -05:00
Soby Chacko
6d2ade2e7b Update parent in apps 2021-02-18 17:12:57 -05:00
Soby Chacko
721c506891 Update dependencies
Spring Boot 2.3.9
Spring Kafka 2.5.11
Spring Cloud Stream - Horsham.SR11
Spring Cloud - Hoxon.SR10
Spring Cloud Function - 3.0.13
2021-02-18 16:09:25 -05:00
Soby Chacko
dc0e31e132 Update next versions 2021-02-18 15:49:23 -05:00
Soby Chacko
eec5e00a0a stream-applications-core: 3.0.2 Release 2021-02-18 15:07:59 -05:00
Soby Chacko
b986495332 Functions 1.0.1 Release 2021-02-18 14:31:44 -05:00
Soby Chacko
c9fb4e5bdf stream-applications-build 1.0.1 Release 2021-02-18 14:23:46 -05:00
Artem Bilan
e2c11b4773 Upgrade Testcontainers tests to the latest images
* Un`@Disable` updated tests
2021-02-11 16:21:06 -05:00
Artem Bilan
d3c38406e8 Upgrade to the latest Testcontainers
* Use `debezium/example-mysql:latest` image in the `CdcBootStarterIntegrationTest`
to avoid failing when some transitive dependency does not exist any more
2021-02-11 11:08:12 -05:00
Soby Chacko
63c22a85d7 tests debugging continues 2021-02-10 14:26:24 -05:00
Artem Bilan
1e6a2e1f44 GH-143: Add an update option to MongoDB Supplier (#147)
* GH-143: Add an update option to MongoDB Supplier

Fixes https://github.com/spring-cloud/stream-applications/issues/143

* * Fix Checkstyle violations
* Update `README.adoc` for `mongodb-source`
2021-02-05 15:36:03 -05:00