GH-107: Make Splitter Function as Flux-based

Fixes: https://github.com/spring-cloud/spring-functions-catalog/issues/107

When we have a composition like this:

```
spring.cloud.function.definition = fileSupplier|splitterFunction
```

Then final "function" signature is like this `Supplier<Flux<Message<List<Message<?>>>>>`.
And that is exactly what we don't expected from the splitter in the end of the composition.
While Spring Cloud Stream supports de-batching, it works for a `List` output only if function is bound by itself.
In case of composition we got just a `Supplier`.

* Rework `SplitterFunctionConfiguration` for `splitterFunction` from `Function<Message<?>, List<Message<?>>>`
to `Function<Flux<Message<?>>, Flux<Message<?>>>` signature to support every possible simple and composed bindings
in Spring Cloud Stream
* Rework `SplitterFunctionApplicationTests` for new expected `Function<Flux<Message<?>>, Flux<Message<?>>>` signature
* Rework `zip-split-rabbit-binder` sample to not use a `flattenFunction` workaround
and fully rely on whatever is new for the `splitterFunction`
* Fix `ZipSplitRabbitBinderApplicationTests` moving the `@RabbitListener` into a `@TestConfiguration`.
Apparently in a new Spring Boot version the test class is registered as a bean much later than normal application context startup.
Therefore, even if the `@RabbitListener` parsed and registered properly, the `RabbitAdmin` bean
has been already started to see our extra bean definition for the `@QueueBinding`

Changing signature for the splitterFunction to reactive types would make it working even with a Supplier composition.

Fix JDBC & MongoDB suppliers to deal with a new version of Splitter function

Fix Checkstyle violations

Use `IntegrationReactiveUtils.messageSourceToFlux()` API

The `IntegrationReactiveUtils.messageSourceToFlux()` provides convenient API to represent a `MessageSource`
as a `Flux` to poll this source.
The API has an error handling logic and delay when no data emitted by the source

* Remove `org.springframework.cloud` dependencies from the project
since we don't use `@PollableBean` anymore, which comes from the `spring-cloud-function-context`
* Simplify `JdbcSupplierConfiguration` and `MongodbSupplierConfiguration` code more: more injections to the respective bean method.
* Use `(__) ->` lambda syntax for unused argument
* Remove unused `ThreadLocalFluxSinkMessageChannel` internal class
* Update Copyrights of the classes in this change

Upgrade to Gradle `8.12`
This commit is contained in:
Artem Bilan
2025-01-16 17:15:41 -05:00
committed by GitHub
parent 9ce6a1b326
commit 3ebce8858f
18 changed files with 69 additions and 133 deletions

View File

@@ -18,8 +18,7 @@ The second one is for `UnZipTransformer`, which we use for a custom function to
The `splitterFunction` is used in a `FileSplitter` mode to read lines from unzipped entries and emit each of them as an individual message.
Essentially, we are splitting twice: zip entries, and content of each file.
The composition is like this: `fileSupplier|unzipFunction|splitterFunction|flattenFunction`.
(The `flattenFunction` will be explained latter).
The composition is like this: `fileSupplier|unzipFunction|splitterFunction.
The result of this composition is a `Supplier<Flux<Mesage<?>>>` and we bind it into a RabbitMQ `unzipped_data_exchange` using Spring Cloud Stream.
For `fileSupplier` we provide these configuration properties:
@@ -49,8 +48,6 @@ Which is a trigger for that function to use a `FileSplitter` for zip entries to
The custom `ZipSplitRabbitBinderApplication.unzipFunction()` (might be a candidate for the future Functions Catalog version) uses `Flux` API to unzip polled files via `UnZipTransformer` and then `flatMapIterable()` for zip entries.
Then those entries are fed into a `splitterFunction` for `FileSplitter` mode.
The mention `ZipSplitRabbitBinderApplication.flattenFunction()` is needed for now here since `splitterFucntion` produces a `List<Message>` which cannot be https://docs.spring.io/spring-cloud-stream/reference/spring-cloud-stream/producing-and-consuming-messages.html#batch-producers[de-batched] by Spring Cloud Stream since our final product of the composition is, essentially, `Supplier<Flux<Message<?>>>`.
To run the application from main `ZipSplitRabbitBinderApplication` class (`./gradlew bootRun`), the RabbitMQ broker must be supplied on the target environment.
The test environment for this sample uses `org.springframework.boot:spring-boot-testcontainers` and `org.testcontainers:rabbitmq` to run RabbitMQ in Docker container and wire it properly into Spring Boot auto-configuration.

View File

@@ -1,7 +1,6 @@
package com.example;
import java.io.File;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
@@ -34,10 +33,4 @@ public class ZipSplitRabbitBinderApplication {
.flatMapIterable(Map::values);
}
// TODO until 'splitterFunction' is fixed this way: https://github.com/spring-cloud/spring-functions-catalog/issues/107
@Bean
Function<Flux<Message<List<Message<?>>>>, Flux<Message<?>>> flattenFunction() {
return messageFlux -> messageFlux.map(Message::getPayload).flatMapIterable(Function.identity());
}
}

View File

@@ -4,11 +4,11 @@ spring:
cloud:
function:
definition: fileSupplier|unzipFunction|splitterFunction|flattenFunction
definition: fileSupplier|unzipFunction|splitterFunction
stream:
bindings:
fileSupplier|unzipFunction|splitterFunction|flattenFunction-out-0:
fileSupplier|unzipFunction|splitterFunction-out-0:
destination: unzipped_data_exchange
file:

View File

@@ -19,6 +19,7 @@ import org.springframework.amqp.rabbit.annotation.Queue;
import org.springframework.amqp.rabbit.annotation.QueueBinding;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.springframework.test.annotation.DirtiesContext;
@@ -47,11 +48,16 @@ class ZipSplitRabbitBinderApplicationTests {
}
}
@RabbitListener(bindings = @QueueBinding(value = @Queue,
exchange = @Exchange(value = "unzipped_data_exchange", type = ExchangeTypes.TOPIC), key = "#"))
void receiveDataFromSplittedZips(String payload) {
LOG.info("A line from zip entry: " + payload);
DATA_SINK.offer(payload);
@TestConfiguration
static class RabbitListenerTestConfiguration {
@RabbitListener(bindings = @QueueBinding(value = @Queue,
exchange = @Exchange(value = "unzipped_data_exchange", type = ExchangeTypes.TOPIC), key = "#"))
void receiveDataFromSplittedZips(String payload) {
LOG.info("A line from zip entry: " + payload);
DATA_SINK.offer(payload);
}
}
}