GH-8632: Add DSL for Debezium module
Fixes https://github.com/spring-projects/spring-integration/issues/8632 * Debezium DSL initial support * additional dsl debezium factory * debezium dsl improvements and tests * impove debezium docs and streamline dsl testing * docs clarifications * fix doc cross-reference * updgrade debezium to 2.2.1.Final. Clean docs * fix multiflow config tests * improve batch tests * Code and doc formatting * Make `name` Debezium property as random according to its docs: ``` Unique name for the connector. Attempting to register again with the same name fails. This property is required by all Kafka Connect connectors. ``` * Code style clean up
This commit is contained in:
Christian Tzolov
@@ -1,12 +1,10 @@
|
||||
[[debezium]]
|
||||
== Debezium Support
|
||||
|
||||
Spring Integration provides channel adapter for handling Change Events using Debezium.
|
||||
https://debezium.io/documentation/reference/development/engine.html[Debezium Engine], Change Data Capture (CDC) inbound channel adapter.
|
||||
The `DebeziumMessageProducer` allows capturing database change events, converting them into messages and streaming later to the outbound channels.
|
||||
|
||||
https://debezium.io/documentation/reference/development/engine.html[Debezium Engine] based Change Data Capture (CDC) channel adapter.
|
||||
The Debezium adapter allows capturing database change events, converting them into messages and streaming those to the outbound channels.
|
||||
|
||||
You need to include this dependency into your project:
|
||||
You need to include the spring integration Debezium dependency to your project:
|
||||
|
||||
====
|
||||
[source, xml, subs="normal", role="primary"]
|
||||
@@ -25,10 +23,48 @@ compile "org.springframework.integration:spring-integration-debezium:{project-ve
|
||||
----
|
||||
====
|
||||
|
||||
You also need to include a https://debezium.io/documentation/reference/connectors/index.html[debezium connector] dependency for your input Database.
|
||||
For example to use Debezium with PostgreSQL you will need the postgres debezium connector:
|
||||
|
||||
====
|
||||
[source, xml, subs="normal", role="primary"]
|
||||
.Maven
|
||||
----
|
||||
<dependency>
|
||||
<groupId>io.debezium</groupId>
|
||||
<artifactId>debezium-connector-postgres</artifactId>
|
||||
<version>${debezium-version}</version>
|
||||
</dependency>
|
||||
|
||||
----
|
||||
[source, groovy, subs="normal", role="secondary"]
|
||||
.Gradle
|
||||
----
|
||||
compile "io.debezium:debezium-connector-postgres:{debezium-version}"
|
||||
----
|
||||
====
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
Replace the `debezium-version` with the version compatible with the `spring-integration-debezium` version being used.
|
||||
====
|
||||
|
||||
[[debezium-inbound]]
|
||||
=== Inbound Debezium Channel Adapter
|
||||
|
||||
The Debezium adapter expects a pre-configured `DebeziumEngine.Builder<ChangeEvent<byte[], byte[]>>` bean instance.
|
||||
The Debezium adapter expects a pre-configured `DebeziumEngine.Builder<ChangeEvent<byte[], byte[]>>` instance.
|
||||
|
||||
[TIP]
|
||||
====
|
||||
The https://github.com/spring-cloud/stream-applications/tree/main/functions/supplier/debezium-supplier[debezium-supplier] provides an out of the box `DebeziumEngine.Builder` Spring Boot auto-configuration with a handy https://github.com/spring-cloud/stream-applications/blob/main/functions/supplier/debezium-supplier/src/main/java/org/springframework/cloud/fn/supplier/debezium/DebeziumProperties.java[DebeziumProperties] configuration abstraction.
|
||||
====
|
||||
|
||||
[TIP]
|
||||
====
|
||||
The <<debezium-java-dsl,Debezium Java DSL>> can create a `DebeziumMessageProducer` instance from a provided `DebeziumEngine.Builder`, as well as from a plain Debezium configuration (e.g. `java.util.Properties`).
|
||||
Later can be handy for some common use-cases with opinionated configuration and serialization formats.
|
||||
====
|
||||
|
||||
Additionally, the `DebeziumMessageProducer` can be tuned with the following configuration properties:
|
||||
|
||||
- `contentType` - allows handling for `JSON` (default), `AVRO` and `PROTOBUF` message contents.
|
||||
@@ -40,7 +76,8 @@ Such a payload is not serializable and would require a custom serialization/dese
|
||||
On a database row delete, Debezium can send a tombstone change event that has the same key as the deleted row and a value of `Optional.empty`.
|
||||
Defaults to `false`.
|
||||
- `headerMapper` - custom `HeaderMapper` implementation that allows for selecting and converting the `ChangeEvent` headers into `Message` headers.
|
||||
The default `DefaultDebeziumHeaderMapper` implementation (no headers are mapped) provides a setter for `setHeaderNamesToMap`.
|
||||
The default `DefaultDebeziumHeaderMapper` implementation provides a setter for `setHeaderNamesToMap`.
|
||||
By default, all headers are mapped.
|
||||
- `threadFactory` - Set custom `ThreadFactory` for the Debezium executor service.
|
||||
Debezium Engine is designed to be submitted to an `Executor` or `ExecutorService` for execution by single thread.
|
||||
|
||||
@@ -58,7 +95,7 @@ public class DebeziumJavaApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
new SpringApplicationBuilder(DebeziumJavaApplication.class)
|
||||
.web(false)
|
||||
.web(WebApplicationType.NONE)
|
||||
.run(args);
|
||||
}
|
||||
|
||||
@@ -79,12 +116,31 @@ public class DebeziumJavaApplication {
|
||||
}
|
||||
|
||||
@ServiceActivator(inputChannel = "debeziumInputChannel")
|
||||
public void handler(String changeEventData) {
|
||||
System.out.println(changeEventData);
|
||||
public void handler(Message<?> message) {
|
||||
|
||||
Object destination = message.getHeaders().get(DebeziumHeaders.DESTINATION); # <1>
|
||||
|
||||
String key = new String((byte[]) message.getHeaders().get(DebeziumHeaders.KEY)); # <2>
|
||||
|
||||
String payload = new String((byte[]) message.getPayload()); # <3>
|
||||
|
||||
System.out.println("KEY: " + key + ", DESTINATION: " + destination + ", PAYLOAD: " + payload);
|
||||
}
|
||||
|
||||
}
|
||||
----
|
||||
<1> A name of the logical destination for which the event is intended.
|
||||
Usually the destination is composed of the `topic.prefix` configuration option, the database name and the table name. For example: `my-topic.inventory.orders`.
|
||||
<2> Contains the schema for the changed table's key and the changed row's actual key.
|
||||
Both the key schema and its corresponding key payload contain a field for each column in the changed table's `PRIMARY KEY` (or unique constraint) at the time the connector created the event.
|
||||
<3> Like the key, the payload has a schema section and a payload value section.
|
||||
The schema section contains the schema that describes the Envelope structure of the payload value section, including its nested fields.
|
||||
Change events for operations that create, update or delete data all have a value payload with an envelope structure.
|
||||
====
|
||||
|
||||
[TIP]
|
||||
====
|
||||
The `key.converter.schemas.enable=false` and/or `value.converter.schemas.enable=false` permit disabling the in-message schema content for key or payload respectively.
|
||||
====
|
||||
|
||||
Similarly, we can configure the `DebeziumMessageProducer` to process the incoming change events in batches:
|
||||
@@ -110,8 +166,36 @@ public void handler(List<ChangeEvent<Object, Object>> payload) {
|
||||
----
|
||||
====
|
||||
|
||||
[[debezium-java-dsl]]
|
||||
=== Debezium Java DSL Support
|
||||
|
||||
==== Configuring with the Java DSL
|
||||
The `spring-integration-debezium` provides a convenient Java DSL fluent API via the `Debezium` factory and the `DebeziumMessageProducerSpec` implementations.
|
||||
|
||||
The Inbound Channel Adapter for Debezium Java DSL is:
|
||||
====
|
||||
[source, java]
|
||||
----
|
||||
DebeziumEngine.Builder<ChangeEvent<byte[], byte[]>> debeziumEngineBuilder = ...
|
||||
IntegrationFlow.from(
|
||||
Debezium.inboundChannelAdapter(debeziumEngineBuilder)
|
||||
.headerNames("special*")
|
||||
.contentType("application/json")
|
||||
.enableBatch(false))
|
||||
.handle(m -> System.out.println(new String((byte[]) m.getPayload())))
|
||||
----
|
||||
====
|
||||
|
||||
Or create an `DebeziumMessageProducerSpec` instance from native debezium configuration properties and default to `JSON` serialization formats.
|
||||
|
||||
====
|
||||
[source, java]
|
||||
----
|
||||
Properties debeziumConfig = ...
|
||||
IntegrationFlow
|
||||
.from(Debezium.inboundChannelAdapter(debeziumConfig))
|
||||
.handle(m -> System.out.println(new String((byte[]) m.getPayload())))
|
||||
----
|
||||
====
|
||||
|
||||
The following Spring Boot application provides an example of configuring the inbound adapter with the Java DSL:
|
||||
|
||||
@@ -131,7 +215,12 @@ public class DebeziumJavaApplication {
|
||||
public IntegrationFlow debeziumInbound(
|
||||
DebeziumEngine.Builder<ChangeEvent<byte[], byte[]>> debeziumEngineBuilder) {
|
||||
|
||||
return IntegrationFlow.from(new DebeziumMessageProducer(debeziumEngineBuilder))
|
||||
return IntegrationFlow
|
||||
.from(Debezium
|
||||
.inboundChannelAdapter(debeziumEngineBuilder)
|
||||
.headerNames("special*")
|
||||
.contentType("application/json")
|
||||
.enableBatch(false))
|
||||
.handle(m -> System.out.println(new String((byte[]) m.getPayload())))
|
||||
.get();
|
||||
}
|
||||
|
||||
@@ -21,7 +21,7 @@ In general the project has been moved to the latest dependency versions.
|
||||
==== Debezium Inbound Channel Adapter
|
||||
|
||||
The Debezium Engine based Change Data Capture (CDC) channel adapter, that allows capturing database change events, converting them into Messages and streaming those to the outbound channels.
|
||||
See <<./debezium.adoc#debezium-inbound, Debezium Inbound Channel Adapter>> for more information.
|
||||
See <<./debezium.adoc#debezium, Debezium Support>> for more information.
|
||||
|
||||
[[x6.2-general]]
|
||||
=== General Changes
|
||||
|
||||
Reference in New Issue
Block a user