SpringCloud/spring-functions-catalog
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

CDC Debezium Supplier, Source Boot Starter

Browse Source 
- Add supplier, source and boot auto-config.
 - Add work around for KafkaNull handling.
 - Update CDC to 1.2.1.Final. Replace the deprecated properties.

remove comments

Add missing metadata configuration. Improve README docs.

CDC: add all metadata for doc auto-generation

Remove custom KafkaNullConverter.

  - The fixed KafkaNullConverter implementation comes with kafka binder 3.0.8-BUILD-SNAPSHOT or newer.

Add repositories to maven config in app generation project.

Remove the unnecessary function version in maven plugin configuration for app generation.
This commit is contained in:
Christian Tzolov
2020-07-09 12:45:58 +02:00
committed by Soby Chacko
parent fa2c0ec583
commit ceee2fb724
22 changed files with 2796 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
common/cdc-debezium-boot-starter/README.adoc Normal file
View File

@@ -0,0 +1,124 @@
= Debezium Spring Boot Starter
Spring Boot Starter for easy integration of https://debezium.io[Debezium] in Spring Boot applications.
To use it you just add the `cdc-debezium-boot-starter` dependency to your application POM and implement your own `Consumer<SourceRecord>` handler to process the incoming database change events. Follow the link below for further instructions.
https://en.wikipedia.org/wiki/Change_data_capture[Change Data Capture] (CDC) Spring Boot Starter that allows capturing change events from various databases including `MySQL`, `PostgreSQL`, `MongoDB`, `Oracle` and `SQL Server`.
== Quick Start
https://start.spring.io/[Start new Spring Boot project] and add the following starter dependency to activate the CDC:
[source, xml]
----
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>cdc-debezium-boot-starter</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
----
NOTE: for MySQL you need to adjust the mysql client version to `mysql:mysql-connector-java:8.0.13`.
The CDC starter setups an embedded https://debezium.io/documentation/reference/0.10/connectors/index.html[debezium connector] to capture the database changes.
Then implement a `Consumer<SourceRecord>` bean handler to receive and process the CDC notifications:
[source, java]
----
@Bean
public Consumer<SourceRecord> mySourceRecordConsumer( # <1>
Function<SourceRecord, byte[]> keySerializer, # <2>
Function<SourceRecord, byte[]> valueSerializer) { # <3>
return sourceRecord -> {
System.out.println(sourceRecord.topic() # <4>
+ " : " + new String(keySerializer.apply(sourceRecord)) # <5>
+ " : " + new String(valueSerializer.apply(sourceRecord))); # <6>
};
}
----
<1> CDC event listener that is called on every data change.
<2> Optional keySerializer converter that allows to serialize the keys of the SourceRecord into a compact binary format.
<3> Optional valueSerializer converter (by default JsonConverter) that allows to serialize the SourceRecord into a compact binary format.
<4> The `topic()` provides a logical identifier of the changed data (e.g. cdc-name.DB-name.table-name)
<5> Convert kay to text message using the keySerializer
<6> Convert payload to JSON message using the valueSerializer.
NOTE: you may have to add `@AutoConfigureBefore(CdcAutoConfiguration.class)` to your Boot class to ensure that your record consumer overrides the default handlers.
Above handler will produce output messages like this:
[source, bash]
----
my-app-connector.inventory.addresses : {"id":10,"customer_id":1001,"street":"3183 Moore Avenue","city":"Euless","state":"Texas","zip":"76036","type":"SHIPPING"}
my-app-connector.inventory.customers : {"id":1004,"first_name":"Anne","last_name":"Kretchmar","email":"annek@noanswer.org"}
my-app-connector.inventory.orders : {"order_number":10001,"order_date":16816,"purchaser":1001,"quantity":1,"product_id":102}
my-app-connector.inventory.products : {"id":109,"name":"spare tire","description":"24 inch spare tire","weight":22.200000762939453}
my-app-connector.inventory.products_on_hand : {"product_id":101,"quantity":3}
----
All Debezium connector properties are supported. Just prefix those properties with `cdc.config` prefix.
Following snipped shows how to configure the https://debezium.io/docs/connectors/mysql/[MySqlConnector] to connect to a local MySQL database run via the `debezium/example-mysql` docker image:
[source, bash]
----
docker run -it --rm --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=debezium -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw debezium/example-mysql:1.0
----
[TIP]
====
(optional) Use `mysql` client to connected to the database and to create a `debezium` user with required credentials:
[source, bash]
----
docker run -it --rm --name mysqlterm --link mysql --rm mysql:5.7 sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium' IDENTIFIED BY 'dbz';
----
====
Configure `cdc-debezium-boot-starter` for consuming cdc events from the MySQL:
[source]
----
cdc.name=my-sql-connector # <1>
cdc.connector=mysql # <2>
cdc.config.database.server.id=85744 # <3>
cdc.config.database.server.name=my-app-connector # <3>
cdc.config.database.user=debezium # <3>
cdc.config.database.password=dbz # <3>
cdc.config.database.hostname=localhost # <3>
cdc.config.database.port=3306 # <3>
cdc.schema=false # <4>
cdc.flattering.enabled=true # <5>
----
<1> Metadata used to identify and dispatch the events received by this cdc consumer instance.
<2> Configures the CDC Source to use https://debezium.io/docs/connectors/mysql/[MySqlConnector]. (equivalent to setting `cdc.config.connector.class=io.debezium.connector.mysql.MySqlConnector`).
<3> Connector specific configurations. MySQL server logical name, connect location and access credentials.
<4> Do not serialize the record's schema in the output messages.
<5> Enables the https://debezium.io/docs/configuration/event-flattening/[CDC Event Flattering] feature.
The full list of properties:
==== Options
//tag::configuration-properties[]
$$cdc.config$$:: $$Spring pass-trough wrapper for debezium configuration properties. All properties with a 'cdc.config.' prefix are native Debezium properties. The prefix is removed, converting them into Debezium io.debezium.config.Configuration.$$ *($$Map<String, String>$$, default: `$$<none>$$`)*
$$cdc.connector$$:: $$Shortcut for the cdc.config.connector.class property. Either of those can be used as long as they do not contradict with each other.$$ *($$ConnectorType$$, default: `$$<none>$$`, possible values: `mysql`,`postgres`,`mongodb`,`oracle`,`sqlserver`)*
$$cdc.flattering.add-fields$$:: $$Comma separated list of metadata fields to add to the flattened message. The fields will be prefixed with "__" or "__[<]struct]__", depending on the specification of the struct.$$ *($$String$$, default: `$$<none>$$`)*
$$cdc.flattering.add-headers$$:: $$Comma separated list specify a list of metadata fields to add to the header of the flattened message. The fields will be prefixed with "__" or "__[struct]__".$$ *($$String$$, default: `$$<none>$$`)*
$$cdc.flattering.delete-handling-mode$$:: $$Options for handling deleted records: (1) none - pass the records through, (2) drop - remove the records and (3) rewrite - add a '__deleted' field to the records.$$ *($$DeleteHandlingMode$$, default: `$$<none>$$`, possible values: `drop`,`rewrite`,`none`)*
$$cdc.flattering.drop-tombstones$$:: $$By default Debezium generates tombstone records to enable Kafka compaction on deleted records. The dropTombstones can suppress the tombstone records.$$ *($$Boolean$$, default: `$$true$$`)*
$$cdc.flattering.enabled$$:: $$Enable flattering the source record events (https://debezium.io/docs/configuration/event-flattening).$$ *($$Boolean$$, default: `$$true$$`)*
$$cdc.name$$:: $$Unique name for this sourceConnector instance.$$ *($$String$$, default: `$$<none>$$`)*
$$cdc.offset.commit-timeout$$:: $$Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to offset storage before cancelling the process and restoring the offset data to be committed in a future attempt.$$ *($$Duration$$, default: `$$5000ms$$`)*
$$cdc.offset.flush-interval$$:: $$Interval at which to try committing offsets. The default is 1 minute.$$ *($$Duration$$, default: `$$60000ms$$`)*
$$cdc.offset.policy$$:: $$Offset storage commit policy.$$ *($$OffsetPolicy$$, default: `$$<none>$$`)*
$$cdc.offset.storage$$:: $$Kafka connector tracks the number processed records and regularly stores the count (as "offsets") in a preconfigured metadata storage. On restart the connector resumes the reading from the last recorded source offset.$$ *($$OffsetStorageType$$, default: `$$<none>$$`, possible values: `memory`,`file`,`kafka`,`metadata`)*
$$cdc.schema$$:: $$Include the schema's as part of the outbound message.$$ *($$Boolean$$, default: `$$false$$`)*
//end::configuration-properties[]

175
common/cdc-debezium-boot-starter/pom.xml Normal file
View File

@@ -0,0 +1,175 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>cdc-debezium-boot-starter</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>cdc-debezium-boot-starter</name>
<description>Change Data Capture (CDC) Debezium Boot Starter</description>
<parent>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>spring-functions-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../spring-functions-parent</relativePath>
</parent>
<properties>
<version.debezium>1.2.1.Final</version.debezium>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>cdc-debezium-common</artifactId>
<version>${project.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- -->
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-mysql</artifactId>
<exclusions>
<exclusion>
<artifactId>mysql-connector-java</artifactId>
<groupId>mysql</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>io.debezium</groupId>-->
<!-- <artifactId>debezium-connector-mongodb</artifactId>-->
<!-- <exclusions>-->
<!-- <exclusion>-->
<!-- <artifactId>slf4j-log4j12</artifactId>-->
<!-- <groupId>org.slf4j</groupId>-->
<!-- </exclusion>-->
<!-- </exclusions>-->
<!-- <version>${version.debezium}</version>-->
<!-- </dependency>-->
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-postgres</artifactId>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-oracle</artifactId>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-sqlserver</artifactId>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.13</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-jdbc</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>0.33.0</version>
<configuration>
<images>
<image>
<alias>mysql</alias>
<name>debezium/example-mysql:1.0</name>
<run>
<env>
<MYSQL_ROOT_PASSWORD>debezium</MYSQL_ROOT_PASSWORD>
<MYSQL_USER>mysqluser</MYSQL_USER>
<MYSQL_PASSWORD>mysqlpw</MYSQL_PASSWORD>
</env>
<ports>
<port>3306:3306</port>
</ports>
<wait>
<log>port: 3306</log>
<time>30000</time>
</wait>
</run>
</image>
</images>
</configuration>
<executions>
<execution>
<id>start</id>
<!--<phase>pre-integration-test</phase>-->
<phase>generate-test-resources</phase>
<goals>
<goal>build</goal>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
<execution>
<id>stop-pre</id>
<phase>clean</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

71
common/cdc-debezium-boot-starter/src/main/java/org/springframework/cloud/fn/common/cdc/CdcAutoConfiguration.java Normal file
View File

@@ -0,0 +1,71 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc;
import java.util.function.Consumer;
import java.util.function.Function;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.kafka.connect.source.SourceRecord;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
/**
* @author Christian Tzolov
*/
@Configuration
@Import(CdcCommonConfiguration.class)
public class CdcAutoConfiguration {
private static final Log logger = LogFactory.getLog(CdcAutoConfiguration.class);
@Bean
@ConditionalOnMissingBean
public Consumer<SourceRecord> defaultSourceRecordConsumer() {
return sourceRecord -> logger.info("[CDC Event]: " + ((sourceRecord == null) ? "null" : sourceRecord.toString()));
}
@Bean
public EmbeddedEngineExecutorService embeddedEngine(EmbeddedEngine.Builder embeddedEngineBuilder,
Consumer<SourceRecord> sourceRecordConsumer, Function<SourceRecord, SourceRecord> recordFlattering) {
EmbeddedEngine embeddedEngine = embeddedEngineBuilder
.notifying(sourceRecord -> sourceRecordConsumer.accept(recordFlattering.apply(sourceRecord)))
.build();
return new EmbeddedEngineExecutorService(embeddedEngine) {
@PostConstruct
@Override
public void start() {
super.start();
}
@PreDestroy
@Override
public void close() {
super.close();
}
};
}
}

1
common/cdc-debezium-boot-starter/src/main/resources/META-INF/spring.factories Normal file
View File

@@ -0,0 +1 @@
org.springframework.boot.autoconfigure.EnableAutoConfiguration=org.springframework.cloud.fn.common.cdc.CdcAutoConfiguration

80
common/cdc-debezium-boot-starter/src/test/java/org/springframework/cloud/fn/common/cdc/CdcBootStarterIntegrationTest.java Normal file
View File

@@ -0,0 +1,80 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc;
import org.junit.Test;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.test.jdbc.JdbcTestUtils;
import static org.assertj.core.api.Assertions.assertThat;
/**
* @author Christian Tzolov
*/
public class CdcBootStarterIntegrationTest {
private final JdbcTemplate jdbcTemplate = jdbcTemplate(
"com.mysql.cj.jdbc.Driver",
"jdbc:mysql://localhost:3306/inventory",
"root",
"debezium");
private final ApplicationContextRunner contextRunner = new ApplicationContextRunner()
.withUserConfiguration(TestCdcApplication.class)
.withPropertyValues(
"cdc.name=my-sql-connector",
"cdc.schema=false",
"cdc.flattering.enabled=true",
"cdc.stream.header.offset=true",
"cdc.connector=mysql",
"cdc.config.database.user=debezium",
"cdc.config.database.password=dbz",
"cdc.config.database.hostname=localhost",
"cdc.config.database.port=3306",
"cdc.config.database.server.id=85744",
"cdc.config.database.server.name=my-app-connector",
"cdc.config.database.history=io.debezium.relational.history.MemoryDatabaseHistory");
@Test
public void consumerTest() {
contextRunner
.withPropertyValues(
"cdc.flattering.deleteHandlingMode=drop",
"cdc.flattering.dropTombstones=true")
.run(context -> {
TestCdcApplication.TestSourceRecordConsumer testConsumer =
context.getBean(TestCdcApplication.TestSourceRecordConsumer.class);
jdbcTemplate.update("insert into `customers`(`first_name`,`last_name`,`email`) VALUES('Test666', 'Test666', 'Test666@spring.org')");
JdbcTestUtils.deleteFromTableWhere(jdbcTemplate, "customers", "first_name = ?", "Test666");
Thread.sleep(10000);
assertThat(testConsumer.recordList).hasSize(53);
});
}
public static JdbcTemplate jdbcTemplate(String jdbcDriver, String jdbcUrl, String user, String password) {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(jdbcDriver);
dataSource.setUrl(jdbcUrl);
dataSource.setUsername(user);
dataSource.setPassword(password);
return new JdbcTemplate(dataSource);
}
}

79
common/cdc-debezium-boot-starter/src/test/java/org/springframework/cloud/fn/common/cdc/TestCdcApplication.java Normal file
View File

@@ -0,0 +1,79 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Consumer;
import java.util.function.Function;
import org.apache.kafka.connect.source.SourceRecord;
import org.springframework.boot.SpringBootConfiguration;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.metadata.SimpleMetadataStore;
/**
* @author Christian Tzolov
*/
@SpringBootConfiguration
@EnableAutoConfiguration
public class TestCdcApplication {
@Bean
public Consumer<SourceRecord> mySourceRecordConsumer(Function<SourceRecord, byte[]> valueSerializer,
Function<SourceRecord, byte[]> keySerializer) {
return new TestSourceRecordConsumer(valueSerializer, keySerializer);
}
@Bean
public SimpleMetadataStore simpleMetadataStore() {
return new SimpleMetadataStore();
}
public static class TestSourceRecordConsumer implements Consumer<SourceRecord> {
private final Function<SourceRecord, byte[]> valueSerializer;
private final Function<SourceRecord, byte[]> keySerializer;
public Map<Object, Object> keyValue = new HashMap<>();
public List<SourceRecord> recordList = new ArrayList<>();
public TestSourceRecordConsumer(Function<SourceRecord, byte[]> valueSerializer,
Function<SourceRecord, byte[]> keySerializer) {
this.valueSerializer = valueSerializer;
this.keySerializer = keySerializer;
}
@Override
public void accept(SourceRecord sourceRecord) {
if (sourceRecord != null) { // ignore null records
recordList.add(sourceRecord);
Object payload = valueSerializer.apply(sourceRecord);
Object key = keySerializer.apply(sourceRecord);
keyValue.put(key, payload);
System.out.println("[CDC Event]: " + sourceRecord.toString());
}
}
}
}

117
common/cdc-debezium-common/pom.xml Normal file
View File

@@ -0,0 +1,117 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>cdc-debezium-common</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>cdc-debezium-common</name>
<description>Change Data Capture (CDC) Debezium Common</description>
<parent>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>spring-functions-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../spring-functions-parent</relativePath>
</parent>
<properties>
<version.debezium>1.2.1.Final</version.debezium>
<!-- Note: postgresql version MUST match the version used by the Debezium postgres connector -->
<postgresql.version>42.2.5</postgresql.version>
<spring-jdbc.version>5.2.1.RELEASE</spring-jdbc.version>
<!-- Debezium requires a particular mysql-connector version -->
<mysql-connector-java>5.1.46</mysql-connector-java>
</properties>
<dependencies>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-embedded</artifactId>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-mysql</artifactId>
<exclusions>
<exclusion>
<artifactId>mysql-connector-java</artifactId>
<groupId>mysql</groupId>
</exclusion>
</exclusions>
<optional>true</optional>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<optional>true</optional>
<version>${mysql-connector-java}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-postgres</artifactId>
<optional>true</optional>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-mongodb</artifactId>
<optional>true</optional>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-oracle</artifactId>
<optional>true</optional>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-sqlserver</artifactId>
<optional>true</optional>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>metadata-store-common</artifactId>
<version>${project.version}</version>
</dependency>
<!-- -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-json</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-ip</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>config-common</artifactId>
<version>${spring-cloud-fn.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

131
common/cdc-debezium-common/src/main/java/org/springframework/cloud/fn/common/cdc/CdcCommonConfiguration.java Normal file
View File

@@ -0,0 +1,131 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc;
import java.util.Collections;
import java.util.Map;
import java.util.function.Function;
import io.debezium.transforms.ExtractNewRecordState;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.kafka.connect.json.JsonConverter;
import org.apache.kafka.connect.source.SourceRecord;
import org.apache.kafka.connect.storage.Converter;
import org.apache.kafka.connect.storage.OffsetBackingStore;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.util.StringUtils;
/**
* @author Christian Tzolov
*/
@Configuration
@EnableConfigurationProperties(CdcCommonProperties.class)
@Import(CdcOffsetBackingStoreConfiguration.class)
public class CdcCommonConfiguration {
private static final Log logger = LogFactory.getLog(CdcCommonConfiguration.class);
@Bean
public io.debezium.config.Configuration configuration(CdcCommonProperties properties) {
Map<String, String> configMap = properties.getConfig();
return io.debezium.config.Configuration.from(configMap);
}
@Bean
public Function<SourceRecord, SourceRecord> recordFlattering(CdcCommonProperties properties,
ExtractNewRecordState extractNewRecordState) {
return sourceRecord -> properties.getFlattering().isEnabled() ?
(SourceRecord) extractNewRecordState.apply(sourceRecord) : sourceRecord;
}
@Bean
public ExtractNewRecordState extractNewRecordState(CdcCommonProperties properties) {
ExtractNewRecordState extractNewRecordState = new ExtractNewRecordState();
Map<String, Object> config = extractNewRecordState.config().defaultValues();
config.put("drop.tombstones", properties.getFlattering().isDropTombstones());
config.put("delete.handling.mode", properties.getFlattering().getDeleteHandlingMode().name());
if (!StringUtils.isEmpty(properties.getFlattering().getAddHeaders())) {
config.put("add.headers", properties.getFlattering().getAddHeaders());
}
if (!StringUtils.isEmpty(properties.getFlattering().getAddFields())) {
config.put("add.fields", properties.getFlattering().getAddFields());
}
extractNewRecordState.configure(config);
return extractNewRecordState;
}
@Bean
@ConditionalOnMissingBean
public JsonConverter jsonConverter(CdcCommonProperties properties) {
JsonConverter jsonConverter = new JsonConverter();
jsonConverter.configure(Collections.singletonMap("schemas.enable", properties.isSchema()), false);
return jsonConverter;
}
@Bean
public Function<SourceRecord, byte[]> valueSerializer(Converter valueConverter) {
return sourceRecord -> valueConverter.fromConnectData(
sourceRecord.topic(), sourceRecord.valueSchema(), sourceRecord.value());
}
@Bean
public Function<SourceRecord, byte[]> keySerializer(Converter valueConverter) {
return sourceRecord -> valueConverter.fromConnectData(
sourceRecord.topic(), sourceRecord.keySchema(), sourceRecord.key());
}
@Bean
public EmbeddedEngine.Builder embeddedEngineBuilder(CdcCommonProperties properties,
OffsetBackingStore offsetBackingStore) {
if (!properties.getConfig().containsKey("connector.class")) {
properties.getConfig().put("connector.class", properties.getConnector().connectorClass);
}
if (!properties.getConfig().containsKey("name")) {
properties.getConfig().put("name", properties.getName());
}
if (!properties.getConfig().containsKey("offset.flush.interval.ms")) {
properties.getConfig().put("offset.flush.interval.ms", properties.getOffset().getFlushInterval().toMillis() + "");
}
if (!properties.getConfig().containsKey("offset.flush.timeout.ms")) {
properties.getConfig().put("offset.flush.timeout.ms", properties.getOffset().getCommitTimeout().toMillis() + "");
}
if (!properties.getConfig().containsKey("offset.commit.policy")) {
properties.getConfig().put("offset.commit.policy", properties.getOffset().getPolicy().policyClass);
}
if (!properties.getConfig().containsKey("offset.storage")) {
properties.getConfig().put("offset.storage", properties.getOffset().getStorage().offsetStorageClass);
}
return EmbeddedEngine.create()
.using(io.debezium.config.Configuration.from(properties.getConfig()))
.offsetBackingStore(offsetBackingStore);
}
}

312
common/cdc-debezium-common/src/main/java/org/springframework/cloud/fn/common/cdc/CdcCommonProperties.java Normal file
View File

@@ -0,0 +1,312 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
import javax.validation.constraints.AssertTrue;
import javax.validation.constraints.NotEmpty;
import javax.validation.constraints.NotNull;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.validation.annotation.Validated;
/**
*
* @author Christian Tzolov
*/
@ConfigurationProperties("cdc")
@Validated
public class CdcCommonProperties {
/**
* Unique name for this sourceConnector instance.
*/
@NotEmpty
private String name;
/**
* Shortcut for the cdc.config.connector.class property. Either of those can be used as long as they do not
* contradict with each other.
*/
@NotNull
private ConnectorType connector = null;
private final Offset offset = new Offset();
/**
* Include the schema's as part of the outbound message.
*/
private boolean schema = false;
/**
* Event Flattering (https://debezium.io/docs/configuration/event-flattening).
*/
private final Flattering flattering = new Flattering();
/**
* Spring pass-trough wrapper for debezium configuration properties.
* All properties with a 'cdc.config.' prefix are native Debezium properties.
* The prefix is removed, converting them into Debezium io.debezium.config.Configuration.
*/
private Map<String, String> config = defaultConfig();
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public Offset getOffset() {
return offset;
}
public Flattering getFlattering() {
return flattering;
}
public Map<String, String> getConfig() {
return config;
}
public boolean isSchema() {
return schema;
}
public void setSchema(boolean schema) {
this.schema = schema;
}
private Map<String, String> defaultConfig() {
Map<String, String> defaultConfig = new HashMap<>();
defaultConfig.put("database.history", "io.debezium.relational.history.MemoryDatabaseHistory");
//defaultConfig.put("offset.flush.interval.ms", "60000");
return defaultConfig;
}
public ConnectorType getConnector() {
return connector;
}
public void setConnector(ConnectorType connector) {
this.connector = connector;
}
@AssertTrue
public boolean connectorIsSet() {
return this.getConnector() != null || this.getConfig().containsKey("connector.class");
}
public static class Offset {
/**
* Interval at which to try committing offsets. The default is 1 minute.
*/
private Duration flushInterval = Duration.ofMillis(60000);
/**
* Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to
* offset storage before cancelling the process and restoring the offset data to be committed in a future attempt.
*/
private Duration commitTimeout = Duration.ofMillis(5000);
/**
* Offset storage commit policy.
*/
private OffsetPolicy policy = OffsetPolicy.periodic;
/**
* Kafka connector tracks the number processed records and regularly stores the count (as "offsets") in a
* preconfigured metadata storage. On restart the connector resumes the reading from the last recorded source offset.
*/
private OffsetStorageType storage = OffsetStorageType.metadata;
public enum OffsetPolicy {
/** periodic. */
periodic("io.debezium.embedded.spi.OffsetCommitPolicy$PeriodicCommitOffsetPolicy"),
/** always. */
always("OffsetCommitPolicy.AlwaysCommitOffsetPolicy.class.getName()");
/** policy. */
public final String policyClass;
OffsetPolicy(String policyClassName) {
this.policyClass = policyClassName;
}
}
public Duration getFlushInterval() {
return flushInterval;
}
public void setFlushInterval(Duration flushInterval) {
this.flushInterval = flushInterval;
}
public Duration getCommitTimeout() {
return commitTimeout;
}
public void setCommitTimeout(Duration commitTimeout) {
this.commitTimeout = commitTimeout;
}
public OffsetPolicy getPolicy() {
return policy;
}
public void setPolicy(OffsetPolicy policy) {
this.policy = policy;
}
public OffsetStorageType getStorage() {
return storage;
}
public void setStorage(OffsetStorageType storage) {
this.storage = storage;
}
}
public enum OffsetStorageType {
/** memory offset storage type. */
memory("org.apache.kafka.connect.storage.MemoryOffsetBackingStore"),
/** File offset storage type. */
file("org.apache.kafka.connect.storage.FileOffsetBackingStore"),
/** Kafka offset storage type. */
kafka("org.apache.kafka.connect.storage.KafkaOffsetBackingStore"),
/** Metadata offset storage type. */
metadata("org.springframework.cloud.stream.app.cdc.common.core.store.MetadataStoreOffsetBackingStore");
/** Class name of the Offset Storage. */
public final String offsetStorageClass;
OffsetStorageType(String type) {
this.offsetStorageClass = type;
}
}
public enum ConnectorType {
/** MySql connector type. */
mysql("io.debezium.connector.mysql.MySqlConnector"),
/** Postgres connector type. */
postgres("io.debezium.connector.postgresql.PostgresConnector"),
/** Mongodb connector type. */
mongodb("io.debezium.connector.mongodb.MongoDbConnector"),
/** Oracle connector type. */
oracle("io.debezium.connector.oracle.OracleConnector"),
/** SqlServer connector type. */
sqlserver("io.debezium.connector.sqlserver.SqlServerConnector");
/** Connector class name. */
public final String connectorClass;
ConnectorType(String type) {
this.connectorClass = type;
}
}
public enum DeleteHandlingMode {
/** records are removed. */
drop,
/** add a __deleted column with true/false values based on record operation. */
rewrite,
/** pass delete events. */
none
}
/**
* https://debezium.io/documentation/reference/0.10/configuration/event-flattening.html .
* https://debezium.io/documentation/reference/0.10/configuration/event-flattening.html#configuration_options
*/
public static class Flattering {
/**
* Enable flattering the source record events (https://debezium.io/docs/configuration/event-flattening).
*/
private boolean enabled = true;
/**
* By default Debezium generates tombstone records to enable Kafka compaction on deleted records.
* The dropTombstones can suppress the tombstone records.
*/
private boolean dropTombstones = true;
/**
* Options for handling deleted records: (1) none - pass the records through, (2) drop - remove the records and
* (3) rewrite - add a '__deleted' field to the records.
*/
private DeleteHandlingMode deleteHandlingMode = DeleteHandlingMode.drop;
/**
* Comma separated list of metadata fields to add to the flattened message.
* The fields will be prefixed with "__" or "__[<]struct]__", depending on the specification of the struct.
*/
private String addFields = null;
/**
* Comma separated list specify a list of metadata fields to add to the header of the flattened message.
* The fields will be prefixed with "__" or "__[struct]__".
*/
private String addHeaders = null;
public boolean isEnabled() {
return enabled;
}
public void setEnabled(boolean enabled) {
this.enabled = enabled;
}
public boolean isDropTombstones() {
return dropTombstones;
}
public void setDropTombstones(boolean dropTombstones) {
this.dropTombstones = dropTombstones;
}
public DeleteHandlingMode getDeleteHandlingMode() {
return deleteHandlingMode;
}
public void setDeleteHandlingMode(DeleteHandlingMode deleteHandlingMode) {
this.deleteHandlingMode = deleteHandlingMode;
}
public String getAddFields() {
return addFields;
}
public void setAddFields(String addFields) {
this.addFields = addFields;
}
public String getAddHeaders() {
return addHeaders;
}
public void setAddHeaders(String addHeaders) {
this.addHeaders = addHeaders;
}
}
}

68
common/cdc-debezium-common/src/main/java/org/springframework/cloud/fn/common/cdc/CdcOffsetBackingStoreConfiguration.java Normal file
View File

@@ -0,0 +1,68 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc;
import org.apache.kafka.connect.storage.FileOffsetBackingStore;
import org.apache.kafka.connect.storage.KafkaOffsetBackingStore;
import org.apache.kafka.connect.storage.MemoryOffsetBackingStore;
import org.apache.kafka.connect.storage.OffsetBackingStore;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.cloud.fn.common.cdc.store.MetadataStoreOffsetBackingStore;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.integration.metadata.MetadataStore;
/**
* @author Christian Tzolov
*/
@Configuration
public class CdcOffsetBackingStoreConfiguration {
@Bean
@ConditionalOnMissingBean
@ConditionalOnExpression("'${cdc.config.offset.storage}'.equalsIgnoreCase('org.springframework.cloud.stream.app.cdc.common.core.store.MetadataStoreOffsetBackingStore') " +
"or '${cdc.offset.storage:metadata}'.equals('metadata')")
public OffsetBackingStore metadataStoreOffsetBackingStore(MetadataStore metadataStore) {
return new MetadataStoreOffsetBackingStore(metadataStore);
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnExpression("'${cdc.config.offset.storage}'.equalsIgnoreCase('org.apache.kafka.connect.storage.FileOffsetBackingStore') " +
"or '${cdc.offset.storage:metadata}'.equalsIgnoreCase('file')")
public OffsetBackingStore fileOffsetBackingStore() {
return new FileOffsetBackingStore();
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnExpression("'${cdc.config.offset.storage}'.equalsIgnoreCase('org.apache.kafka.connect.storage.KafkaOffsetBackingStore') " +
"or '${cdc.offset.storage:metadata}'.equalsIgnoreCase('kafka')")
public OffsetBackingStore kafkaOffsetBackingStore() {
return new KafkaOffsetBackingStore();
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnExpression("'${cdc.config.offset.storage}'.equalsIgnoreCase('org.apache.kafka.connect.storage.MemoryOffsetBackingStore') " +
"or '${cdc.offset.storage:metadata}'.equalsIgnoreCase('memory')")
public OffsetBackingStore memoryOffsetBackingStore() {
return new MemoryOffsetBackingStore();
}
}

1074
common/cdc-debezium-common/src/main/java/org/springframework/cloud/fn/common/cdc/EmbeddedEngine.java Normal file
View File

File diff suppressed because it is too large Load Diff

52
common/cdc-debezium-common/src/main/java/org/springframework/cloud/fn/common/cdc/EmbeddedEngineExecutorService.java Normal file
View File

@@ -0,0 +1,52 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc;
import java.io.Closeable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
* @author Christian Tzolov
*/
public class EmbeddedEngineExecutorService implements Closeable {
private static final Log logger = LogFactory.getLog(EmbeddedEngineExecutorService.class);
private final EmbeddedEngine engine;
private final ExecutorService executor;
public EmbeddedEngineExecutorService(EmbeddedEngine engine) {
this.engine = engine;
this.executor = Executors.newSingleThreadExecutor();
}
public void start() {
logger.info("Start Embedded Engine");
this.executor.execute(this.engine);
}
@Override
public void close() {
logger.info("Stop Embedded Engine");
this.engine.stop();
this.executor.shutdown();
}
}

90
common/cdc-debezium-common/src/main/java/org/springframework/cloud/fn/common/cdc/store/MetadataStoreOffsetBackingStore.java Normal file
View File

@@ -0,0 +1,90 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.common.cdc.store;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.kafka.connect.runtime.WorkerConfig;
import org.apache.kafka.connect.runtime.standalone.StandaloneConfig;
import org.apache.kafka.connect.storage.MemoryOffsetBackingStore;
import org.springframework.integration.metadata.MetadataStore;
/**
* @author Christian Tzolov
*/
public class MetadataStoreOffsetBackingStore extends MemoryOffsetBackingStore {
private MetadataStore metadataStore;
private String offsetStoreName;
public MetadataStoreOffsetBackingStore(MetadataStore metadataStore) {
this.metadataStore = metadataStore;
}
@Override
public void configure(WorkerConfig config) {
super.configure(config);
this.offsetStoreName = config.getString(StandaloneConfig.OFFSET_STORAGE_FILE_FILENAME_CONFIG);
}
@Override
public synchronized void start() {
super.start();
//log.info("Starting FileOffsetBackingStore with file {}", file);
load();
}
@Override
public synchronized void stop() {
super.stop();
// Nothing to do since this doesn't maintain any outstanding connections/data
// log.info("Stopped FileOffsetBackingStore");
}
private void load() {
if (this.metadataStore.get("keys") != null) {
String[] keys = this.metadataStore.get("keys").split(",");
for (String keySting : keys) {
String valueString = this.metadataStore.get(keySting);
ByteBuffer key = ByteBuffer.wrap(keySting.getBytes());
ByteBuffer value = ByteBuffer.wrap(valueString.getBytes());
this.data.put(key, value);
}
}
}
@Override
protected void save() {
List<String> keys = new ArrayList<>();
for (Map.Entry<ByteBuffer, ByteBuffer> mapEntry : this.data.entrySet()) {
byte[] key = (mapEntry.getKey() != null) ? mapEntry.getKey().array() : null;
byte[] value = (mapEntry.getValue() != null) ? mapEntry.getValue().array() : null;
if (key != null && value != null) {
keys.add(new String(key, StandardCharsets.UTF_8));
this.metadataStore.put(
new String(key, StandardCharsets.UTF_8),
new String(value, StandardCharsets.UTF_8));
}
}
this.metadataStore.put("keys", String.join(",", keys));
}
}

2
common/cdc-debezium-common/src/main/resources/application.properties Normal file
View File

@@ -0,0 +1,2 @@
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration
spring.cloud.stream.kafka.default.producer.messageKeyExpression=headers['cdc_key']

7
function-dependencies/pom.xml
View File

@@ -15,6 +15,11 @@
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>cdc-debezium-supplier</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>file-supplier</artifactId>
@@ -244,4 +249,4 @@
</dependencies>
</dependencyManagement>
</project>
</project>

3
pom.xml
View File

@@ -51,6 +51,8 @@
<module>common/tcp-common</module>
<module>common/twitter-common</module>
<module>common/tensorflow-common</module>
<module>common/cdc-debezium-common</module>
<module>common/cdc-debezium-boot-starter</module>
<module>consumer/cassandra-consumer</module>
<module>consumer/analytics-consumer</module>
@@ -98,6 +100,7 @@
<module>supplier/websocket-supplier</module>
<module>supplier/s3-supplier</module>
<module>supplier/twitter-supplier</module>
<module>supplier/cdc-debezium-supplier</module>
<module>spring-functions-parent</module>
<module>function-dependencies</module>

2
spring-functions-parent/pom.xml
View File

@@ -38,4 +38,4 @@
</dependencies>
</dependencyManagement>
</project>
</project>

45
supplier/cdc-debezium-supplier/README.adoc Normal file
View File

@@ -0,0 +1,45 @@
# CDC Debezium Supplier
This module provides a CDC Debezium supplier that can be reused and composed in other applications.
The `Supplier` uses the `Debezium` library.
`cdcSupplier` is implemented as a `java.util.function.Supplier`.
This supplier gives you a reactive stream from CDC sources. The supplier has a signature of `Supplier<Flux<Message<?>>>`.
Users have to subscribe to this `Flux` and then receive the data.
https://en.wikipedia.org/wiki/Change_data_capture[Change Data Capture] (CDC) `source` that captures and streams change events from various databases.
Currently, it supports `MySQL`, `PostgreSQL`, `MongoDB`, `Oracle` and `SQL Server` databases.
Build upon https://debezium.io/docs/embedded/[Debezium Embedded Connector], the `CDC Source` allows capturing and streaming database changes over different message binders such Apache Kafka, RabbitMQ and all Spring Cloud Stream supporter brokers.
## Beans for injection
You can import the `CdcSupplierConfiguration` in the application and then inject the following bean.
`cdcSupplier`
You need to inject this as `Supplier<Flux<Message<?>>>`.
You can use `cdcSupplier` as a qualifier when injecting.
Once injected, you can use the `get` method of the `Supplier` to invoke it and then subscribe to the returned `Flux`.
## Configuration Options
All configuration properties are prefixed with `cdc`.
Use the `cdc.config.` prefix to inject native, Debezium properties. Convenient shortcuts for the most frequently used
Debezium properties are available.
Uses `BackingOffsetStore` configuration, based on the [MetadataStore](https://github.com/spring-cloud/stream-applications/tree/master/functions/common/metadata-store-common) service.
Later provides various microservices friendly ways for storing the offset metadata.
For more information on the various options available, please see:
* link:../../common/cdc-debezium-common/src/main/java/org/springframework/cloud/fn/common/cdc/CdcCommonProperties.java[CdcCommonProperties].
* link:src/main/java/org/springframework/cloud/fn/supplier/cdc-debezium-supplier/CdcSupplierProperties.java[CdcSupplierProperties].
## Examples
See this link:../../../applications/source/cdc-debezium-source/src/test/java/org/springframework/cloud/stream/app/source/cdc[test suite] for the various ways, this supplier is used.
## Other usage
See this link:../../../applications/source/cdc-debezium-source/README.adoc[README] where this supplier is used to create a Spring Cloud Stream CDC Source.
Also check link:../../common/cdc-debezium-boot-starter/README.adoc[README] to see how the core CDC can be used a Spring Boot Starter.

124
supplier/cdc-debezium-supplier/pom.xml Normal file
View File

@@ -0,0 +1,124 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>cdc-debezium-supplier</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>cdc-debezium-supplier</name>
<description>CDC Debezium Suppliers</description>
<parent>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>spring-functions-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
<relativePath>../../spring-functions-parent</relativePath>
</parent>
<properties>
<version.debezium>1.2.1.Final</version.debezium>
<mysql.version>8.0.13</mysql.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.fn</groupId>
<artifactId>cdc-debezium-common</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql.version}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-mysql</artifactId>
<exclusions>
<exclusion>
<artifactId>mysql-connector-java</artifactId>
<groupId>mysql</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-mongodb</artifactId>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-postgres</artifactId>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-oracle</artifactId>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-sqlserver</artifactId>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
<version>${version.debezium}</version>
</dependency>
<!-- -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-integration</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

171
supplier/cdc-debezium-supplier/src/main/java/org/springframework/cloud/fn/supplier/cdc/CdcSupplierConfiguration.java Normal file
View File

@@ -0,0 +1,171 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.supplier.cdc;
import java.lang.reflect.Field;
import java.util.Iterator;
import java.util.function.Consumer;
import java.util.function.Function;
import java.util.function.Supplier;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.kafka.connect.header.Header;
import org.apache.kafka.connect.source.SourceRecord;
import reactor.core.publisher.EmitterProcessor;
import reactor.core.publisher.Flux;
import reactor.core.publisher.FluxSink;
import org.springframework.beans.factory.BeanClassLoaderAware;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.cloud.fn.common.cdc.CdcCommonConfiguration;
import org.springframework.cloud.fn.common.cdc.EmbeddedEngine;
import org.springframework.cloud.fn.common.cdc.EmbeddedEngineExecutorService;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Import;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHeaders;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.util.ClassUtils;
import org.springframework.util.MimeTypeUtils;
/**
* CDC source that uses the Debezium Connectors to monitor and record all of the row-level changes in the databases.
* https://debezium.io/docs/connectors
*
* @author Christian Tzolov
*/
@EnableConfigurationProperties(CdcSupplierProperties.class)
@Import(CdcCommonConfiguration.class)
public class CdcSupplierConfiguration implements BeanClassLoaderAware {
private static final Log logger = LogFactory.getLog(CdcSupplierConfiguration.class);
/**
* ORG_SPRINGFRAMEWORK_KAFKA_SUPPORT_KAFKA_NULL.
*/
public static final String ORG_SPRINGFRAMEWORK_KAFKA_SUPPORT_KAFKA_NULL = "org.springframework.kafka.support.KafkaNull";
private Object kafkaNull = null;
@Override
public void setBeanClassLoader(ClassLoader classLoader) {
try {
Class<?> clazz = ClassUtils.forName(ORG_SPRINGFRAMEWORK_KAFKA_SUPPORT_KAFKA_NULL, classLoader);
Field field = clazz.getDeclaredField("INSTANCE");
this.kafkaNull = field.get(null);
}
catch (ClassNotFoundException | NoSuchFieldException | IllegalAccessException e) {
}
}
@Bean
public Supplier<Flux<Message<?>>> cdcSupplier(EmbeddedEngineExecutorService embeddedEngineExecutorService) {
return () -> emitterProcessor
.doOnSubscribe(subscription -> {
embeddedEngineExecutorService.start();
})
.doAfterTerminate(() -> {
logger.info("Proactive shutdown");
embeddedEngineExecutorService.close();
})
.doOnError(throwable -> logger.error(throwable.getMessage(), throwable));
}
private EmitterProcessor<Message<?>> emitterProcessor = EmitterProcessor.create(1, false);
@Bean
public EmbeddedEngineExecutorService embeddedEngineExecutorService(
EmbeddedEngine.Builder embeddedEngineBuilder,
Function<SourceRecord, byte[]> valueSerializer, Function<SourceRecord, byte[]> keySerializer,
Function<SourceRecord, SourceRecord> recordFlattering,
ObjectMapper mapper, CdcSupplierProperties cdcStreamingEngineProperties) {
FluxSink<Message<?>> sink = emitterProcessor.sink();
Consumer<SourceRecord> messageConsumer = sourceRecord -> {
// When cdc.flattering.deleteHandlingMode=none and cdc.flattering.dropTombstones=false
// then on deletion event an additional sourceRecord is sent with value Null.
// Here we filter out such condition.
if (sourceRecord == null) {
logger.debug("Ignore disabled tombstone events");
return;
}
Object cdcJsonPayload = valueSerializer.apply(sourceRecord);
// When the tombstone event is enabled, Debezium serializes the payload to null (e.g. empty payload)
// while the metadata information is carried through the headers (cdc_key).
// Note: Event for none flattered responses, when the cdc.config.tombstones.on.delete=true (default),
// tombstones are generate by Debezium and handled by the code below.
if (cdcJsonPayload == null) {
cdcJsonPayload = this.kafkaNull;
}
// If payload is still null ignore the message.
if (cdcJsonPayload == null) {
logger.info("dropped null payload message");
return;
}
byte[] key = keySerializer.apply(sourceRecord);
if (key == null) {
logger.warn("Null serialised key for sourceRecord: " + sourceRecord);
key = new byte[0];
}
MessageBuilder<?> messageBuilder = MessageBuilder
.withPayload(cdcJsonPayload)
.setHeader("cdc_key", new String(key))
.setHeader("cdc_topic", sourceRecord.topic())
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON_VALUE);
if (cdcStreamingEngineProperties.getHeader().isConvertConnectHeaders()) {
// Convert the Connect Headers into Message Headers.
if (sourceRecord.headers() != null && !sourceRecord.headers().isEmpty()) {
Iterator<Header> itr = sourceRecord.headers().iterator();
while (itr.hasNext()) {
Header header = itr.next();
messageBuilder.setHeader(header.key(), header.value());
}
}
}
if (cdcStreamingEngineProperties.getHeader().isOffset()) {
try {
messageBuilder.setHeader("cdc_offset",
mapper.writeValueAsString(sourceRecord.sourceOffset()));
}
catch (JsonProcessingException e) {
logger.warn("Failed to record cdc_offset header", e);
}
}
sink.next(messageBuilder.build());
};
EmbeddedEngine engine = embeddedEngineBuilder
.notifying(record -> messageConsumer.accept(recordFlattering.apply(record)))
.build();
return new EmbeddedEngineExecutorService(engine);
}
}

68
supplier/cdc-debezium-supplier/src/main/java/org/springframework/cloud/fn/supplier/cdc/CdcSupplierProperties.java Normal file
View File

@@ -0,0 +1,68 @@
/*
* Copyright 2020-2020 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.cloud.fn.supplier.cdc;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.validation.annotation.Validated;
/**
*
* @author Christian Tzolov
*/
@ConfigurationProperties("cdc.stream")
@Validated
public class CdcSupplierProperties {
/**
* Control what information to be serialized in the outbound message headers.
*/
private Header header = new Header();
public Header getHeader() {
return header;
}
public static class Header {
/**
* Serializes the source record's offset metadata into the outbound message header under cdc.offset.
*/
private boolean offset = false;
/**
* When true the {@link org.apache.kafka.connect.header.Header} are converted into message headers with the
* {@link org.apache.kafka.connect.header.Header#key()} as name and {@link org.apache.kafka.connect.header.Header#value()}.
*/
private boolean convertConnectHeaders = true;
public boolean isOffset() {
return offset;
}
public void setOffset(boolean offset) {
this.offset = offset;
}
public boolean isConvertConnectHeaders() {
return convertConnectHeaders;
}
public void setConvertConnectHeaders(boolean convertConnectHeaders) {
this.convertConnectHeaders = convertConnectHeaders;
}
}
}

2
supplier/cdc-debezium-supplier/src/main/resources/application.properties Normal file
View File

@@ -0,0 +1,2 @@
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration
spring.cloud.stream.kafka.default.producer.messageKeyExpression=headers['cdc_key']