GH-107: Make Splitter Function as Flux-based
Fixes: https://github.com/spring-cloud/spring-functions-catalog/issues/107 When we have a composition like this: ``` spring.cloud.function.definition = fileSupplier|splitterFunction ``` Then final "function" signature is like this `Supplier<Flux<Message<List<Message<?>>>>>`. And that is exactly what we don't expected from the splitter in the end of the composition. While Spring Cloud Stream supports de-batching, it works for a `List` output only if function is bound by itself. In case of composition we got just a `Supplier`. * Rework `SplitterFunctionConfiguration` for `splitterFunction` from `Function<Message<?>, List<Message<?>>>` to `Function<Flux<Message<?>>, Flux<Message<?>>>` signature to support every possible simple and composed bindings in Spring Cloud Stream * Rework `SplitterFunctionApplicationTests` for new expected `Function<Flux<Message<?>>, Flux<Message<?>>>` signature * Rework `zip-split-rabbit-binder` sample to not use a `flattenFunction` workaround and fully rely on whatever is new for the `splitterFunction` * Fix `ZipSplitRabbitBinderApplicationTests` moving the `@RabbitListener` into a `@TestConfiguration`. Apparently in a new Spring Boot version the test class is registered as a bean much later than normal application context startup. Therefore, even if the `@RabbitListener` parsed and registered properly, the `RabbitAdmin` bean has been already started to see our extra bean definition for the `@QueueBinding` Changing signature for the splitterFunction to reactive types would make it working even with a Supplier composition. Fix JDBC & MongoDB suppliers to deal with a new version of Splitter function Fix Checkstyle violations Use `IntegrationReactiveUtils.messageSourceToFlux()` API The `IntegrationReactiveUtils.messageSourceToFlux()` provides convenient API to represent a `MessageSource` as a `Flux` to poll this source. The API has an error handling logic and delay when no data emitted by the source * Remove `org.springframework.cloud` dependencies from the project since we don't use `@PollableBean` anymore, which comes from the `spring-cloud-function-context` * Simplify `JdbcSupplierConfiguration` and `MongodbSupplierConfiguration` code more: more injections to the respective bean method. * Use `(__) ->` lambda syntax for unused argument * Remove unused `ThreadLocalFluxSinkMessageChannel` internal class * Update Copyrights of the classes in this change Upgrade to Gradle `8.12`
This commit is contained in:
@@ -18,8 +18,7 @@ The second one is for `UnZipTransformer`, which we use for a custom function to
|
||||
The `splitterFunction` is used in a `FileSplitter` mode to read lines from unzipped entries and emit each of them as an individual message.
|
||||
Essentially, we are splitting twice: zip entries, and content of each file.
|
||||
|
||||
The composition is like this: `fileSupplier|unzipFunction|splitterFunction|flattenFunction`.
|
||||
(The `flattenFunction` will be explained latter).
|
||||
The composition is like this: `fileSupplier|unzipFunction|splitterFunction.
|
||||
The result of this composition is a `Supplier<Flux<Mesage<?>>>` and we bind it into a RabbitMQ `unzipped_data_exchange` using Spring Cloud Stream.
|
||||
|
||||
For `fileSupplier` we provide these configuration properties:
|
||||
@@ -49,8 +48,6 @@ Which is a trigger for that function to use a `FileSplitter` for zip entries to
|
||||
The custom `ZipSplitRabbitBinderApplication.unzipFunction()` (might be a candidate for the future Functions Catalog version) uses `Flux` API to unzip polled files via `UnZipTransformer` and then `flatMapIterable()` for zip entries.
|
||||
Then those entries are fed into a `splitterFunction` for `FileSplitter` mode.
|
||||
|
||||
The mention `ZipSplitRabbitBinderApplication.flattenFunction()` is needed for now here since `splitterFucntion` produces a `List<Message>` which cannot be https://docs.spring.io/spring-cloud-stream/reference/spring-cloud-stream/producing-and-consuming-messages.html#batch-producers[de-batched] by Spring Cloud Stream since our final product of the composition is, essentially, `Supplier<Flux<Message<?>>>`.
|
||||
|
||||
To run the application from main `ZipSplitRabbitBinderApplication` class (`./gradlew bootRun`), the RabbitMQ broker must be supplied on the target environment.
|
||||
|
||||
The test environment for this sample uses `org.springframework.boot:spring-boot-testcontainers` and `org.testcontainers:rabbitmq` to run RabbitMQ in Docker container and wire it properly into Spring Boot auto-configuration.
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package com.example;
|
||||
|
||||
import java.io.File;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Function;
|
||||
|
||||
@@ -34,10 +33,4 @@ public class ZipSplitRabbitBinderApplication {
|
||||
.flatMapIterable(Map::values);
|
||||
}
|
||||
|
||||
// TODO until 'splitterFunction' is fixed this way: https://github.com/spring-cloud/spring-functions-catalog/issues/107
|
||||
@Bean
|
||||
Function<Flux<Message<List<Message<?>>>>, Flux<Message<?>>> flattenFunction() {
|
||||
return messageFlux -> messageFlux.map(Message::getPayload).flatMapIterable(Function.identity());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -4,11 +4,11 @@ spring:
|
||||
|
||||
cloud:
|
||||
function:
|
||||
definition: fileSupplier|unzipFunction|splitterFunction|flattenFunction
|
||||
definition: fileSupplier|unzipFunction|splitterFunction
|
||||
|
||||
stream:
|
||||
bindings:
|
||||
fileSupplier|unzipFunction|splitterFunction|flattenFunction-out-0:
|
||||
fileSupplier|unzipFunction|splitterFunction-out-0:
|
||||
destination: unzipped_data_exchange
|
||||
|
||||
file:
|
||||
|
||||
@@ -19,6 +19,7 @@ import org.springframework.amqp.rabbit.annotation.Queue;
|
||||
import org.springframework.amqp.rabbit.annotation.QueueBinding;
|
||||
import org.springframework.amqp.rabbit.annotation.RabbitListener;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.TestConfiguration;
|
||||
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
|
||||
import org.springframework.test.annotation.DirtiesContext;
|
||||
|
||||
@@ -47,11 +48,16 @@ class ZipSplitRabbitBinderApplicationTests {
|
||||
}
|
||||
}
|
||||
|
||||
@RabbitListener(bindings = @QueueBinding(value = @Queue,
|
||||
exchange = @Exchange(value = "unzipped_data_exchange", type = ExchangeTypes.TOPIC), key = "#"))
|
||||
void receiveDataFromSplittedZips(String payload) {
|
||||
LOG.info("A line from zip entry: " + payload);
|
||||
DATA_SINK.offer(payload);
|
||||
@TestConfiguration
|
||||
static class RabbitListenerTestConfiguration {
|
||||
|
||||
@RabbitListener(bindings = @QueueBinding(value = @Queue,
|
||||
exchange = @Exchange(value = "unzipped_data_exchange", type = ExchangeTypes.TOPIC), key = "#"))
|
||||
void receiveDataFromSplittedZips(String payload) {
|
||||
LOG.info("A line from zip entry: " + payload);
|
||||
DATA_SINK.offer(payload);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user