175 lines
7.7 KiB
Plaintext
175 lines
7.7 KiB
Plaintext
== Spring Cloud Stream and Schema Evolution in Action with Kafka Binder.
|
|
|
|
:project-version: 4.0.0-SNAPSHOT
|
|
|
|
These are a set of Spring Boot applications to demonstrate Schema Evolution using Spring Cloud Stream with Kafka binder.
|
|
Producer V1 (`producer1`), Producer V2 (`producer2`), and Consumer (`consumer`) are included in this project.
|
|
|
|
=== Requirement
|
|
As a developer, I'd like to design my consumer to be resilient to differing payload schemas.
|
|
|
|
=== Assumptions
|
|
For this demonstration, we will simply assume there are two producers producing events with different payload schemas.
|
|
A consumer that consumes both the payload versions will be designed to adapt to evolving schemas.
|
|
|
|
Both producers and consumers interact with the SCSt schema registry to register and evolve the schema.
|
|
|
|
[[build-apps]]
|
|
=== Building
|
|
|
|
====
|
|
NOTE: It is expected that this repo has been checked out locally and All commands are executed from this sample's directory `spring-cloud-stream-schema-registry-integration` unless otherwise noted.
|
|
====
|
|
|
|
==== Build the schema registry
|
|
The schema registry must be built by executing the following command:
|
|
[source,bash]
|
|
----
|
|
pushd ../../schema-registry
|
|
../mvnw clean install
|
|
popd
|
|
----
|
|
|
|
==== Build the apps
|
|
To build the applications simply execute the following command:
|
|
[source,bash]
|
|
----
|
|
./mvnw clean install
|
|
----
|
|
|
|
NOTE: The apps can be built and run from w/in an IDE (such as IntelliJ) but you will need to invoke the Maven `package` goal and then `refresh` the project as the Avro Maven plugin needs to execute so that it generates the required model classes - otherwise you will see compile failures around missing `Sensor` class.
|
|
|
|
[[run-apps]]
|
|
=== Running
|
|
|
|
====
|
|
NOTE: It is expected that this repo has been checked out locally and All commands are executed from this sample's directory `spring-cloud-stream-schema-registry-integration` unless otherwise noted.
|
|
====
|
|
|
|
==== Pre-requisites
|
|
****
|
|
* The components have all been built by following the <<build-apps>> steps.
|
|
* Apache Kafka broker available at `localhost:9092`
|
|
|
|
TIP: The included link:../../../tools/kafka/docker-compose/README.adoc#_all_the_things[Kafka tools] can be used to easily start a broker locally on the required coordinates
|
|
|
|
* By default, the schema registry is backed by an `H2` database.
|
|
** To instead use `Postgres` it must be available at `localhost:5432`
|
|
** To instead use `MySQL` it must be available at `localhost:3306`.
|
|
|
|
TIP: Docker compose files are provided for both link:./postgres.yml[Postgres] and link:./mysql.yml[MySQL]. You can simply run `docker-compose -f <postgres.yml|mysql.yml> <up|down>` to start/stop the database server
|
|
****
|
|
|
|
==== Steps
|
|
Make sure the above pre-requisites are satisfied and follow the steps below.
|
|
|
|
===== Start Schema Registry
|
|
Start the Schema Registry server (_adjust commands accordingly if you are not on a Unix like platform_)
|
|
[source,bash,subs="attributes"]
|
|
----
|
|
java -jar ../../schema-registry/spring-cloud-stream-schema-registry-server/target/spring-cloud-stream-schema-registry-server-4.0.0-SNAPSHOT.jar
|
|
----
|
|
|
|
By default the schema registry starts with a local `H2` database.
|
|
|
|
.To use `Postgres` database instead of `H2`...
|
|
[%collapsible]
|
|
====
|
|
additional properties must be specified when starting the server:
|
|
[source,bash,subs="attributes"]
|
|
----
|
|
java -jar ../../schema-registry/spring-cloud-stream-schema-registry-server/target/spring-cloud-stream-schema-registry-server-{project-version}.jar \
|
|
--spring.datasource.url=jdbc:postgresql://localhost:5432/registry \
|
|
--spring.datasource.username=root \
|
|
--spring.datasource.password=rootpw \
|
|
--spring.datasource.driver-class-name=org.postgresql.Driver \
|
|
--spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect \
|
|
--spring.jpa.hibernate.ddl-auto-create=true \
|
|
--spring.jpa.hibernate.ddl-auto=update \
|
|
--spring.jpa.generate-ddl=true
|
|
----
|
|
====
|
|
|
|
|
|
.To use `MySQL` database instead of `H2`...
|
|
[%collapsible]
|
|
====
|
|
additional properties must be specified when starting the server:
|
|
[source,bash,subs="attributes"]
|
|
----
|
|
java -jar ../../schema-registry/spring-cloud-stream-schema-registry-server/target/spring-cloud-stream-schema-registry-server-{project-version}.jar \
|
|
--spring.datasource.url=jdbc:mariadb://localhost:3306/registry \
|
|
--spring.datasource.username=root \
|
|
--spring.datasource.password=rootpw \
|
|
--spring.datasource.driver-class-name=org.mariadb.jdbc.Driver \
|
|
--spring.jpa.database-platform=org.hibernate.dialect.MariaDB53Dialect \
|
|
--spring.jpa.hibernate.ddl-auto-create=true \
|
|
--spring.jpa.hibernate.ddl-auto=update \
|
|
--spring.jpa.generate-ddl=true
|
|
----
|
|
====
|
|
|
|
===== Start consumer
|
|
Start `consumer` on another terminal session (or run it from an IDE)
|
|
[source,bash,subs="attributes"]
|
|
----
|
|
java -jar schema-registry-consumer-kafka/target/schema-registry-consumer-kafka-{project-version}.jar
|
|
----
|
|
|
|
===== Start V1 producer
|
|
Start `producer1` on another terminal session (or run it from an IDE)
|
|
[source,bash,subs="attributes"]
|
|
----
|
|
java -jar schema-registry-producer1-kafka/target/schema-registry-producer1-kafka-{project-version}.jar
|
|
----
|
|
|
|
===== Start V2 producer
|
|
Start `producer2` on another terminal session (or run it from an IDE)
|
|
[source,bash,subs="attributes"]
|
|
----
|
|
java -jar schema-registry-producer2-kafka/target/schema-registry-producer2-kafka-{project-version}.jar
|
|
----
|
|
|
|
=== Sample Data
|
|
Both the producers in the demonstration are _also_ REST controllers. We will hit the `/randomMessage` endpoint on each producer
|
|
to POST sample data.
|
|
|
|
_Example:_
|
|
[source,bash]
|
|
----
|
|
curl -X POST http://localhost:9009/randomMessage
|
|
curl -X POST http://localhost:9010/randomMessage
|
|
curl -X POST http://localhost:9009/randomMessage
|
|
curl -X POST http://localhost:9009/randomMessage
|
|
curl -X POST http://localhost:9010/randomMessage
|
|
----
|
|
|
|
=== Output
|
|
The consumer should log the results.
|
|
|
|
[source,bash,options=nowrap,subs=attributes]
|
|
----
|
|
{"id": "d5657e55-c2cd-48f0-a22e-d28d1ef10873-v1", "internalTemperature": 19.534815, "externalTemperature": 0.0, "acceleration": 5.286502, "velocity": 25.349945}
|
|
{"id": "6a6de265-997c-4bf9-8eae-97accccb78e9-v2", "internalTemperature": 39.443855, "externalTemperature": 40.365253, "acceleration": 1.8879288, "velocity": 2.5296867}
|
|
{"id": "f09defad-828f-43ae-93a4-e777754cf57a-v1", "internalTemperature": 15.895501, "externalTemperature": 0.0, "acceleration": 1.9341749, "velocity": 52.868507}
|
|
{"id": "b39b8c73-eec3-4abd-b8d2-cc405eb39bd7-v1", "internalTemperature": 44.90698, "externalTemperature": 0.0, "acceleration": 1.5393275, "velocity": 87.0358}
|
|
{"id": "19d5c20e-ec18-4b35-a82a-c8322d7fea27-v2", "internalTemperature": 19.203693, "externalTemperature": 47.290142, "acceleration": 1.125809, "velocity": 11.153614}
|
|
----
|
|
|
|
NOTE: Refer to the payload suffix in the `id` field. Each of them are appended with `-v1` or `-v2` indicating they are from
|
|
`producer1` and `producer2` respectively.
|
|
|
|
=== What just happened?
|
|
The schema evolved on the `temperature` field. That field is now split into `internalTemperature` and `externalTemperature`,
|
|
as two separate fields. The `producer1` produces payload only with `temperature` and on the other hand, `producer2` produces
|
|
payload with `internalTemperature` and `externalTemperature` fields in it.
|
|
|
|
The `consumer` is coded against a base schema that include the split fields.
|
|
|
|
The `consumer` app can happily deserialize the payload with `internalTemperature` and `externalTemperature` fields. However, when
|
|
a `producer1` payload arrives (which includes `temperature` field), the schema evolution and compatibility check are automatically
|
|
applied.
|
|
|
|
Because each payload also includes the payload version in the header, Spring Cloud Stream with the help of Schema Registry server and Avro, the schema evolution occurs behind the scenes.
|
|
The automatic mapping of `temperature` to `internalTemperature` field is applied, since that's the field where the `aliases` is defined.
|