spring-cloud-stream/samples/spring-cloud-stream-schema-registry-integration

== Spring Cloud Stream and Schema Evolution in Action with Kafka Binder.

:project-version: 4.0.0-SNAPSHOT

These are a set of Spring Boot applications to demonstrate Schema Evolution using Spring Cloud Stream with Kafka binder.
Producer V1 (`producer1`), Producer V2 (`producer2`), and Consumer (`consumer`) are included in this project.

=== Requirement
As a developer, I'd like to design my consumer to be resilient to differing payload schemas.

=== Assumptions
For this demonstration, we will simply assume there are two producers producing events with different payload schemas.
A consumer that consumes both the payload versions will be designed to adapt to evolving schemas.

Both producers and consumers interact with the SCSt schema registry to register and evolve the schema.

[[build-apps]]
=== Building

====
NOTE: It is expected that this repo has been checked out locally and All commands are executed from this sample's directory `spring-cloud-stream-schema-registry-integration` unless otherwise noted.
====

==== Build the schema registry
The schema registry must be built by executing the following command:
[source,bash]
----
pushd ../../schema-registry
../mvnw clean install
popd
----

==== Build the apps
To build the applications simply execute the following command:
[source,bash]
----
./mvnw clean install
----

NOTE: The apps can be built and run from w/in an IDE (such as IntelliJ) but you will need to invoke the Maven `package` goal and then `refresh` the project as the Avro Maven plugin needs to execute so that it generates the required model classes - otherwise you will see compile failures around missing `Sensor` class.

[[run-apps]]
=== Running

====
NOTE: It is expected that this repo has been checked out locally and All commands are executed from this sample's directory `spring-cloud-stream-schema-registry-integration` unless otherwise noted.
====

==== Pre-requisites
****
* The components have all been built by following the <<build-apps>> steps.
* Apache Kafka broker available at `localhost:9092`

TIP: The included link:../../../tools/kafka/docker-compose/README.adoc#_all_the_things[Kafka tools] can be used to easily start a broker locally on the required coordinates

* By default, the schema registry is backed by an `H2` database.
** To instead use `Postgres` it must be available at `localhost:5432`
** To instead use `MySQL` it must be available at `localhost:3306`.

TIP: Docker compose files are provided for both link:./postgres.yml[Postgres] and link:./mysql.yml[MySQL]. You can simply run `docker-compose -f <postgres.yml|mysql.yml> <up|down>` to start/stop the database server
****

==== Steps
Make sure the above pre-requisites are satisfied and follow the steps below.

===== Start Schema Registry
Start the Schema Registry server (_adjust commands accordingly if you are not on a Unix like platform_)
[source,bash,subs="attributes"]
----
java -jar ../../schema-registry/spring-cloud-stream-schema-registry-server/target/spring-cloud-stream-schema-registry-server-4.0.0-SNAPSHOT.jar
----

By default the schema registry starts with a local `H2` database.

.To use `Postgres` database instead of `H2`...
[%collapsible]
====
additional properties must be specified when starting the server:
[source,bash,subs="attributes"]
----
java -jar ../../schema-registry/spring-cloud-stream-schema-registry-server/target/spring-cloud-stream-schema-registry-server-{project-version}.jar \
  --spring.datasource.url=jdbc:postgresql://localhost:5432/registry \
  --spring.datasource.username=root \
  --spring.datasource.password=rootpw \
  --spring.datasource.driver-class-name=org.postgresql.Driver \
  --spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect \
  --spring.jpa.hibernate.ddl-auto-create=true \
  --spring.jpa.hibernate.ddl-auto=update \
  --spring.jpa.generate-ddl=true
----
====


.To use `MySQL` database instead of `H2`...
[%collapsible]
====
additional properties must be specified when starting the server:
[source,bash,subs="attributes"]
----
java -jar ../../schema-registry/spring-cloud-stream-schema-registry-server/target/spring-cloud-stream-schema-registry-server-{project-version}.jar \
  --spring.datasource.url=jdbc:mariadb://localhost:3306/registry \
  --spring.datasource.username=root \
  --spring.datasource.password=rootpw \
  --spring.datasource.driver-class-name=org.mariadb.jdbc.Driver \
  --spring.jpa.database-platform=org.hibernate.dialect.MariaDB53Dialect \
  --spring.jpa.hibernate.ddl-auto-create=true \
  --spring.jpa.hibernate.ddl-auto=update \
  --spring.jpa.generate-ddl=true
----
====

===== Start consumer
Start `consumer` on another terminal session (or run it from an IDE)
[source,bash,subs="attributes"]
----
java -jar schema-registry-consumer-kafka/target/schema-registry-consumer-kafka-{project-version}.jar
----

===== Start V1 producer
Start `producer1` on another terminal session (or run it from an IDE)
[source,bash,subs="attributes"]
----
java -jar schema-registry-producer1-kafka/target/schema-registry-producer1-kafka-{project-version}.jar
----

===== Start V2 producer
Start `producer2` on another terminal session (or run it from an IDE)
[source,bash,subs="attributes"]
----
java -jar schema-registry-producer2-kafka/target/schema-registry-producer2-kafka-{project-version}.jar
----

=== Sample Data
Both the producers in the demonstration are _also_ REST controllers. We will hit the `/randomMessage` endpoint on each producer
to POST sample data.

_Example:_
[source,bash]
----
curl -X POST http://localhost:9009/randomMessage
curl -X POST http://localhost:9010/randomMessage
curl -X POST http://localhost:9009/randomMessage
curl -X POST http://localhost:9009/randomMessage
curl -X POST http://localhost:9010/randomMessage
----

=== Output
The consumer should log the results.

[source,bash,options=nowrap,subs=attributes]
----
{"id": "d5657e55-c2cd-48f0-a22e-d28d1ef10873-v1", "internalTemperature": 19.534815, "externalTemperature": 0.0, "acceleration": 5.286502, "velocity": 25.349945}
{"id": "6a6de265-997c-4bf9-8eae-97accccb78e9-v2", "internalTemperature": 39.443855, "externalTemperature": 40.365253, "acceleration": 1.8879288, "velocity": 2.5296867}
{"id": "f09defad-828f-43ae-93a4-e777754cf57a-v1", "internalTemperature": 15.895501, "externalTemperature": 0.0, "acceleration": 1.9341749, "velocity": 52.868507}
{"id": "b39b8c73-eec3-4abd-b8d2-cc405eb39bd7-v1", "internalTemperature": 44.90698, "externalTemperature": 0.0, "acceleration": 1.5393275, "velocity": 87.0358}
{"id": "19d5c20e-ec18-4b35-a82a-c8322d7fea27-v2", "internalTemperature": 19.203693, "externalTemperature": 47.290142, "acceleration": 1.125809, "velocity": 11.153614}
----

NOTE: Refer to the payload suffix in the `id` field. Each of them are appended with `-v1` or `-v2` indicating they are from
`producer1` and `producer2` respectively.

=== What just happened?
The schema evolved on the `temperature` field. That field is now split into `internalTemperature` and `externalTemperature`,
as two separate fields. The `producer1` produces payload only with `temperature` and on the other hand, `producer2` produces
payload with `internalTemperature` and `externalTemperature` fields in it.

The `consumer` is coded against a base schema that include the split fields.

The `consumer` app can happily deserialize the payload with `internalTemperature` and `externalTemperature` fields. However, when
a `producer1` payload arrives (which includes `temperature` field), the schema evolution and compatibility check are automatically
applied.

Because each payload also includes the payload version in the header, Spring Cloud Stream with the help of Schema Registry server and Avro, the schema evolution occurs behind the scenes.
The automatic mapping of `temperature` to `internalTemperature` field is applied, since that's the field where the `aliases` is defined.