diff --git a/schema-registry-samples/kafka-streams-schema-evolution/.mvn b/schema-registry-samples/kafka-streams-schema-evolution/.mvn new file mode 120000 index 0000000..d21aa17 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/.mvn @@ -0,0 +1 @@ +../../.mvn \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/README.adoc b/schema-registry-samples/kafka-streams-schema-evolution/README.adoc new file mode 100644 index 0000000..467dc26 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/README.adoc @@ -0,0 +1,88 @@ +== Spring Cloud Stream, Kafka Streams and Schema Evolution in Action with Confluent Schema Registry Server! + +This repo includes three Spring Boot applications to demonstrate Schema Evolution using Spring Cloud Stream Kafka and Kafka Streams binders support. +Producer V1 (`kafka-streams-confluent-producer1`), Producer V2 (`kafka-streams-confluent-producer2`), and Consumer (`kafka-streams-confluent-consumer`) are included in this project. + +Both producers are standard spring cloud stream sources that use the kafka binder. +They both skip the framework provided serialization on the outbound and relies on Kafka's native serialization mechanism. +The consumer app uses the kafka streams binder and also skips the binder provided message deserialization. Instead, it delegates those responsibilities to native Kafka Streams. + +The samples do *not* use the schema registry support provided by Spring Cloud Stream, but rather they use the Confluent Schema Registry. +Both producers and the consumer use the same exact serializer and deserializer respectively. +Producers use the `SpecificAvroSerializer` and the consumer uses the `SpecificAvroSerde`. Please see the implementations for more details. + +=== Running the application + +Make sure you are in the directory `kafka-streams-schema-evolution` + +Build the project: `./mvnw clean package` + +Ensure that you have Kafka running locally. + +- Start the Confluent Schema Registry server in a terminal window session +[source,bash] +---- +/bin/schema-registry-start ./etc/schema-registry/schema-registry.properties +---- + +In order to run this sample, you need to set compatibility to `NONE` on Confluent schema registry server. + +`curl -X PUT http://127.0.0.1:8081/config -d '{"compatibility": "NONE"}' -H "Content-Type:application/json"` + +- Start `kafka-streams-confluent-consumer` on another terminal session +[source,bash] +---- +java -jar kafka-streams-confluent-consumer/target/kafka-streams-confluent-consumer-0.0.1-SNAPSHOT.jar +---- +- Start `kafka-streams-confluent-producer1` on another terminal session +[source,bash] +---- +java -jar kafka-streams-confluent-producer1/target/kafka-streams-confluent-producer1-0.0.1-SNAPSHOT.jar +---- +- Start `kafka-streams-confluent-producer2` on another terminal session +[source,bash] +---- +java -jar kafka-streams-confluent-producer1/target/kafka-streams-confluent-producer1-0.0.1-SNAPSHOT.jar +---- + +=== Sample Data +Both the producers in the demonstration are _also_ REST controllers. We will hit the `/messages` endpoint on each producer +to POST sample data. + +_Example:_ +[source,bash] +---- +curl -X POST http://localhost:9009/messages +curl -X POST http://localhost:9010/messages +curl -X POST http://localhost:9009/messages +curl -X POST http://localhost:9009/messages +curl -X POST http://localhost:9010/messages +---- + +=== Output +The consumer application has a printer that runs every 30 seconds to print the current counts. + +[source,bash,options=nowrap,subs=attributes] +---- +2018-07-31 16:57:37.041 INFO 29393 --- [ask-scheduler-3] ication$$EnhancerBySpringCGLIB$$27af0d3a : Count for v1 is=10 +2018-07-31 16:57:37.041 INFO 29393 --- [ask-scheduler-3] ication$$EnhancerBySpringCGLIB$$27af0d3a : Count for v2 is=12 +2018-07-31 16:58:07.037 INFO 29393 --- [ask-scheduler-3] ication$$EnhancerBySpringCGLIB$$27af0d3a : Count for v1 is=10 +2018-07-31 16:58:07.037 INFO 29393 --- [ask-scheduler-3] ication$$EnhancerBySpringCGLIB$$27af0d3a : Count for v2 is=12 +---- + +NOTE: Refer to the payload suffix in the `id` field. Each of them are appended with `-v1` or `-v2` indicating they are from +`producer1` and `producer2` respectively. + +=== What just happened? +The schema evolved on the `temperature` field. That field is now split into `internalTemperature` and `externalTemperature`, +as two separate fields. The `producer1` produces payload only with `temperature` and on the other hand, `producer2` produces +payload with `internalTemperature` and `externalTemperature` fields in it. + +The `consumer` is coded against a base schema that include the split fields. + +The `consumer` app can happily deserialize the payload with `internalTemperature` and `externalTemperature` fields. However, when +a `producer1` payload arrives (which includes `temperature` field), the schema evolution and compatibility check are automatically +applied. + +Because each payload also includes the payload version, the applications co-ordinate with the Confluent Schema Registry server and thus schema evolution occurs behind the scenes. The automatic mapping of `temperature` to +`internalTemperature` field is applied, since that's the field where the `aliases` is defined. \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.gitignore b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.gitignore new file mode 100644 index 0000000..82eca33 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.gitignore @@ -0,0 +1,25 @@ +/target/ +!.mvn/wrapper/maven-wrapper.jar + +### STS ### +.apt_generated +.classpath +.factorypath +.project +.settings +.springBeans +.sts4-cache + +### IntelliJ IDEA ### +.idea +*.iws +*.iml +*.ipr + +### NetBeans ### +/nbproject/private/ +/build/ +/nbbuild/ +/dist/ +/nbdist/ +/.nb-gradle/ \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/jvm.config b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/jvm.config new file mode 100644 index 0000000..0e7dabe --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/jvm.config @@ -0,0 +1 @@ +-Xmx1024m -XX:CICompilerCount=1 -XX:TieredStopAtLevel=1 -Djava.security.egd=file:/dev/./urandom \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/maven.config b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/maven.config new file mode 100644 index 0000000..3b8cf46 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/maven.config @@ -0,0 +1 @@ +-DaltSnapshotDeploymentRepository=repo.spring.io::default::https://repo.spring.io/libs-snapshot-local -P spring diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/wrapper/maven-wrapper.jar b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/wrapper/maven-wrapper.jar new file mode 100644 index 0000000..5fd4d50 Binary files /dev/null and b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/wrapper/maven-wrapper.jar differ diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/wrapper/maven-wrapper.properties b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/wrapper/maven-wrapper.properties new file mode 100644 index 0000000..eb91947 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/.mvn/wrapper/maven-wrapper.properties @@ -0,0 +1 @@ +distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.3.3/apache-maven-3.3.3-bin.zip \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/mvnw b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/mvnw new file mode 100755 index 0000000..6efc7bd --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/mvnw @@ -0,0 +1,226 @@ +#!/bin/sh +# ---------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# ---------------------------------------------------------------------------- + +# ---------------------------------------------------------------------------- +# Maven2 Start Up Batch script +# +# Required ENV vars: +# ------------------ +# JAVA_HOME - location of a JDK home dir +# +# Optional ENV vars +# ----------------- +# M2_HOME - location of maven2's installed home dir +# MAVEN_OPTS - parameters passed to the Java VM when running Maven +# e.g. to debug Maven itself, use +# set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +# MAVEN_SKIP_RC - flag to disable loading of mavenrc files +# ---------------------------------------------------------------------------- + +if [ -z "$MAVEN_SKIP_RC" ] ; then + + if [ -f /etc/mavenrc ] ; then + . /etc/mavenrc + fi + + if [ -f "$HOME/.mavenrc" ] ; then + . "$HOME/.mavenrc" + fi + +fi + +# OS specific support. $var _must_ be set to either true or false. +cygwin=false; +darwin=false; +mingw=false +case "`uname`" in + CYGWIN*) cygwin=true ;; + MINGW*) mingw=true;; + Darwin*) darwin=true + # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home + # See https://developer.apple.com/library/mac/qa/qa1170/_index.html + if [ -z "$JAVA_HOME" ]; then + if [ -x "/usr/libexec/java_home" ]; then + export JAVA_HOME="`/usr/libexec/java_home`" + else + export JAVA_HOME="/Library/Java/Home" + fi + fi + ;; +esac + +if [ -z "$JAVA_HOME" ] ; then + if [ -r /etc/gentoo-release ] ; then + JAVA_HOME=`java-config --jre-home` + fi +fi + +if [ -z "$M2_HOME" ] ; then + ## resolve links - $0 may be a link to maven's home + PRG="$0" + + # need this for relative symlinks + while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG="`dirname "$PRG"`/$link" + fi + done + + saveddir=`pwd` + + M2_HOME=`dirname "$PRG"`/.. + + # make it fully qualified + M2_HOME=`cd "$M2_HOME" && pwd` + + cd "$saveddir" + # echo Using m2 at $M2_HOME +fi + +# For Cygwin, ensure paths are in UNIX format before anything is touched +if $cygwin ; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --unix "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --unix "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --unix "$CLASSPATH"` +fi + +# For Migwn, ensure paths are in UNIX format before anything is touched +if $mingw ; then + [ -n "$M2_HOME" ] && + M2_HOME="`(cd "$M2_HOME"; pwd)`" + [ -n "$JAVA_HOME" ] && + JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`" + # TODO classpath? +fi + +if [ -z "$JAVA_HOME" ]; then + javaExecutable="`which javac`" + if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then + # readlink(1) is not available as standard on Solaris 10. + readLink=`which readlink` + if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then + if $darwin ; then + javaHome="`dirname \"$javaExecutable\"`" + javaExecutable="`cd \"$javaHome\" && pwd -P`/javac" + else + javaExecutable="`readlink -f \"$javaExecutable\"`" + fi + javaHome="`dirname \"$javaExecutable\"`" + javaHome=`expr "$javaHome" : '\(.*\)/bin'` + JAVA_HOME="$javaHome" + export JAVA_HOME + fi + fi +fi + +if [ -z "$JAVACMD" ] ; then + if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + else + JAVACMD="`which java`" + fi +fi + +if [ ! -x "$JAVACMD" ] ; then + echo "Error: JAVA_HOME is not defined correctly." >&2 + echo " We cannot execute $JAVACMD" >&2 + exit 1 +fi + +if [ -z "$JAVA_HOME" ] ; then + echo "Warning: JAVA_HOME environment variable is not set." +fi + +CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher + +# traverses directory structure from process work directory to filesystem root +# first directory with .mvn subdirectory is considered project base directory +find_maven_basedir() { + + if [ -z "$1" ] + then + echo "Path not specified to find_maven_basedir" + return 1 + fi + + basedir="$1" + wdir="$1" + while [ "$wdir" != '/' ] ; do + if [ -d "$wdir"/.mvn ] ; then + basedir=$wdir + break + fi + # workaround for JBEAP-8937 (on Solaris 10/Sparc) + if [ -d "${wdir}" ]; then + wdir=`cd "$wdir/.."; pwd` + fi + # end of workaround + done + echo "${basedir}" +} + +# concatenates all lines of a file +concat_lines() { + if [ -f "$1" ]; then + echo "$(tr -s '\n' ' ' < "$1")" + fi +} + +BASE_DIR=`find_maven_basedir "$(pwd)"` +if [ -z "$BASE_DIR" ]; then + exit 1; +fi + +export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"} +echo $MAVEN_PROJECTBASEDIR +MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS" + +# For Cygwin, switch paths to Windows format before running java +if $cygwin; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --path --windows "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --windows "$CLASSPATH"` + [ -n "$MAVEN_PROJECTBASEDIR" ] && + MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"` +fi + +WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +"$JAVACMD" \ + $MAVEN_OPTS \ + -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \ + "-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \ + ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@" + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/mvnw.cmd b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/mvnw.cmd new file mode 100644 index 0000000..b0dc0e7 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/mvnw.cmd @@ -0,0 +1,145 @@ +@REM ---------------------------------------------------------------------------- +@REM Licensed to the Apache Software Foundation (ASF) under one +@REM or more contributor license agreements. See the NOTICE file +@REM distributed with this work for additional information +@REM regarding copyright ownership. The ASF licenses this file +@REM to you under the Apache License, Version 2.0 (the +@REM "License"); you may not use this file except in compliance +@REM with the License. You may obtain a copy of the License at +@REM +@REM http://www.apache.org/licenses/LICENSE-2.0 +@REM +@REM Unless required by applicable law or agreed to in writing, +@REM software distributed under the License is distributed on an +@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +@REM KIND, either express or implied. See the License for the +@REM specific language governing permissions and limitations +@REM under the License. +@REM ---------------------------------------------------------------------------- + +@REM ---------------------------------------------------------------------------- +@REM Maven2 Start Up Batch script +@REM +@REM Required ENV vars: +@REM JAVA_HOME - location of a JDK home dir +@REM +@REM Optional ENV vars +@REM M2_HOME - location of maven2's installed home dir +@REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands +@REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a key stroke before ending +@REM MAVEN_OPTS - parameters passed to the Java VM when running Maven +@REM e.g. to debug Maven itself, use +@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +@REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files +@REM ---------------------------------------------------------------------------- + +@REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on' +@echo off +@REM enable echoing my setting MAVEN_BATCH_ECHO to 'on' +@if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO% + +@REM set %HOME% to equivalent of $HOME +if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%") + +@REM Execute a user defined script before this one +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre +@REM check for pre script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_pre.bat" call "%HOME%\mavenrc_pre.bat" +if exist "%HOME%\mavenrc_pre.cmd" call "%HOME%\mavenrc_pre.cmd" +:skipRcPre + +@setlocal + +set ERROR_CODE=0 + +@REM To isolate internal variables from possible post scripts, we use another setlocal +@setlocal + +@REM ==== START VALIDATION ==== +if not "%JAVA_HOME%" == "" goto OkJHome + +echo. +echo Error: JAVA_HOME not found in your environment. >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +:OkJHome +if exist "%JAVA_HOME%\bin\java.exe" goto init + +echo. +echo Error: JAVA_HOME is set to an invalid directory. >&2 +echo JAVA_HOME = "%JAVA_HOME%" >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +@REM ==== END VALIDATION ==== + +:init + +set MAVEN_CMD_LINE_ARGS=%* + +@REM Find the project base dir, i.e. the directory that contains the folder ".mvn". +@REM Fallback to current working directory if not found. + +set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR% +IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir + +set EXEC_DIR=%CD% +set WDIR=%EXEC_DIR% +:findBaseDir +IF EXIST "%WDIR%"\.mvn goto baseDirFound +cd .. +IF "%WDIR%"=="%CD%" goto baseDirNotFound +set WDIR=%CD% +goto findBaseDir + +:baseDirFound +set MAVEN_PROJECTBASEDIR=%WDIR% +cd "%EXEC_DIR%" +goto endDetectBaseDir + +:baseDirNotFound +set MAVEN_PROJECTBASEDIR=%EXEC_DIR% +cd "%EXEC_DIR%" + +:endDetectBaseDir + +IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig + +@setlocal EnableExtensions EnableDelayedExpansion +for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a +@endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS% + +:endReadAdditionalConfig + +SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe" + +set WRAPPER_JAR="".\.mvn\wrapper\maven-wrapper.jar"" +set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +%MAVEN_JAVA_EXE% %JVM_CONFIG_MAVEN_PROPS% %MAVEN_OPTS% %MAVEN_DEBUG_OPTS% -classpath %WRAPPER_JAR% "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" %WRAPPER_LAUNCHER% %MAVEN_CMD_LINE_ARGS% +if ERRORLEVEL 1 goto error +goto end + +:error +set ERROR_CODE=1 + +:end +@endlocal & set ERROR_CODE=%ERROR_CODE% + +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPost +@REM check for post script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_post.bat" call "%HOME%\mavenrc_post.bat" +if exist "%HOME%\mavenrc_post.cmd" call "%HOME%\mavenrc_post.cmd" +:skipRcPost + +@REM pause the script if MAVEN_BATCH_PAUSE is set to 'on' +if "%MAVEN_BATCH_PAUSE%" == "on" pause + +if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERROR_CODE% + +exit /B %ERROR_CODE% diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/pom.xml b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/pom.xml new file mode 100644 index 0000000..693a2be --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/pom.xml @@ -0,0 +1,95 @@ + + + 4.0.0 + + kafka-streams-confluent-consumer + 0.0.1-SNAPSHOT + jar + kafka-streams-confluent-consumer + Kafka Streams Confluent Consumer + + + io.spring.cloud.stream.sample + spring-cloud-stream-samples-parent + 0.0.1-SNAPSHOT + ../../.. + + + + 4.0.0 + 1.8.2 + + + + + org.springframework.cloud + spring-cloud-stream-schema + + + org.apache.avro + avro + ${avro.version} + + + org.springframework.cloud + spring-cloud-stream-binder-kafka-streams + + + + io.confluent + kafka-streams-avro-serde + ${confluent.version} + + + org.slf4j + slf4j-log4j12 + + + + + io.confluent + kafka-avro-serializer + ${confluent.version} + + + io.confluent + kafka-schema-registry-client + ${confluent.version} + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + org.apache.avro + avro-maven-plugin + ${avro.version} + + + generate-sources + + schema + protocol + idl-protocol + + + src/main/resources/avro + + + + + + + + + + confluent + http://packages.confluent.io/maven/ + + + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/java/sample/consumer/CountVersionApplication.java b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/java/sample/consumer/CountVersionApplication.java new file mode 100644 index 0000000..75bcf91 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/java/sample/consumer/CountVersionApplication.java @@ -0,0 +1,84 @@ +package sample.consumer; + +import com.example.Sensor; +import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig; +import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.kafka.common.serialization.Serdes; +import org.apache.kafka.common.utils.Bytes; +import org.apache.kafka.streams.KeyValue; +import org.apache.kafka.streams.kstream.KStream; +import org.apache.kafka.streams.kstream.Materialized; +import org.apache.kafka.streams.state.KeyValueStore; +import org.apache.kafka.streams.state.QueryableStoreTypes; +import org.apache.kafka.streams.state.ReadOnlyKeyValueStore; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.cloud.stream.annotation.EnableBinding; +import org.springframework.cloud.stream.annotation.StreamListener; +import org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService; +import org.springframework.cloud.stream.binder.kafka.streams.annotations.KafkaStreamsProcessor; +import org.springframework.messaging.handler.annotation.SendTo; +import org.springframework.scheduling.annotation.EnableScheduling; +import org.springframework.scheduling.annotation.Scheduled; + +import java.util.Collections; +import java.util.Map; + +@SpringBootApplication +@EnableBinding(KafkaStreamsProcessor.class) +@EnableScheduling +public class CountVersionApplication { + + private static final String STORE_NAME = "sensor-store"; + + private final Log logger = LogFactory.getLog(getClass()); + + ReadOnlyKeyValueStore keyValueStore; + + @Autowired + private InteractiveQueryService queryService; + + public static void main(String[] args) { + SpringApplication.run(CountVersionApplication.class, args); + } + + @StreamListener("input") + @SendTo("output") + public KStream process(KStream input) { + + final Map serdeConfig = Collections.singletonMap( + AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081"); + + final SpecificAvroSerde sensorSerde = new SpecificAvroSerde<>(); + sensorSerde.configure(serdeConfig, false); + + return input + .map((k, value) -> { + String newKey = "v1"; + if (value.getId().toString().endsWith("v2")) { + newKey = "v2"; + } + return new KeyValue<>(newKey, value); + }) + .groupByKey() + .count(Materialized.>as(STORE_NAME) + .withKeySerde(Serdes.String()) + .withValueSerde(Serdes.Long())) + .toStream(); + + } + + @Scheduled(fixedRate = 30000, initialDelay = 5000) + public void printVersionCounts() { + if (keyValueStore == null) { + keyValueStore = queryService.getQueryableStore(STORE_NAME, QueryableStoreTypes.keyValueStore()); + } + + logger.info("Count for v1 is=" + keyValueStore.get("v1")); + logger.info("Count for v2 is=" + keyValueStore.get("v2")); + } + +} \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/resources/application.yml b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/resources/application.yml new file mode 100644 index 0000000..a3441da --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/resources/application.yml @@ -0,0 +1,20 @@ +server.port: 9998 +spring.cloud.stream.bindings.output: + destination: sensor-versions +spring.cloud.stream.bindings.input: + destination: sensors + consumer: + useNativeDecoding: true +spring.cloud.stream.kafka.streams.binder: + brokers: localhost + configuration: + schema.registry.url: http://localhost:8081 + commit.interval.ms: 1000 + default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde + default.value.serde: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde +spring.cloud.stream.kafka.streams.bindings.input: + consumer: + # The following 2 props are not needed as they are same as the global ones. + # Adding it here in order to illustrate the usage of it if they are indeed different from the global Serde's + keySerde: org.apache.kafka.common.serialization.Serdes$StringSerde + valueSerde: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/resources/avro/sensor.avsc b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/resources/avro/sensor.avsc new file mode 100644 index 0000000..95afdfc --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-consumer/src/main/resources/avro/sensor.avsc @@ -0,0 +1,12 @@ +{ + "namespace" : "com.example", + "type" : "record", + "name" : "Sensor", + "fields" : [ + {"name":"id","type":"string"}, + {"name":"internalTemperature", "type":"float", "default":0.0, "aliases":["temperature"]}, + {"name":"externalTemperature", "type":"float", "default":0.0}, + {"name":"acceleration", "type":"float","default":0.0}, + {"name":"velocity","type":"float","default":0.0} + ] +} \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.gitignore b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.gitignore new file mode 100644 index 0000000..82eca33 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.gitignore @@ -0,0 +1,25 @@ +/target/ +!.mvn/wrapper/maven-wrapper.jar + +### STS ### +.apt_generated +.classpath +.factorypath +.project +.settings +.springBeans +.sts4-cache + +### IntelliJ IDEA ### +.idea +*.iws +*.iml +*.ipr + +### NetBeans ### +/nbproject/private/ +/build/ +/nbbuild/ +/dist/ +/nbdist/ +/.nb-gradle/ \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/jvm.config b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/jvm.config new file mode 100644 index 0000000..0e7dabe --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/jvm.config @@ -0,0 +1 @@ +-Xmx1024m -XX:CICompilerCount=1 -XX:TieredStopAtLevel=1 -Djava.security.egd=file:/dev/./urandom \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/maven.config b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/maven.config new file mode 100644 index 0000000..3b8cf46 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/maven.config @@ -0,0 +1 @@ +-DaltSnapshotDeploymentRepository=repo.spring.io::default::https://repo.spring.io/libs-snapshot-local -P spring diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/wrapper/maven-wrapper.jar b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/wrapper/maven-wrapper.jar new file mode 100644 index 0000000..5fd4d50 Binary files /dev/null and b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/wrapper/maven-wrapper.jar differ diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/wrapper/maven-wrapper.properties b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/wrapper/maven-wrapper.properties new file mode 100644 index 0000000..6637ced --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/.mvn/wrapper/maven-wrapper.properties @@ -0,0 +1 @@ +distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.3.9/apache-maven-3.3.9-bin.zip \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/mvnw b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/mvnw new file mode 100755 index 0000000..6efc7bd --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/mvnw @@ -0,0 +1,226 @@ +#!/bin/sh +# ---------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# ---------------------------------------------------------------------------- + +# ---------------------------------------------------------------------------- +# Maven2 Start Up Batch script +# +# Required ENV vars: +# ------------------ +# JAVA_HOME - location of a JDK home dir +# +# Optional ENV vars +# ----------------- +# M2_HOME - location of maven2's installed home dir +# MAVEN_OPTS - parameters passed to the Java VM when running Maven +# e.g. to debug Maven itself, use +# set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +# MAVEN_SKIP_RC - flag to disable loading of mavenrc files +# ---------------------------------------------------------------------------- + +if [ -z "$MAVEN_SKIP_RC" ] ; then + + if [ -f /etc/mavenrc ] ; then + . /etc/mavenrc + fi + + if [ -f "$HOME/.mavenrc" ] ; then + . "$HOME/.mavenrc" + fi + +fi + +# OS specific support. $var _must_ be set to either true or false. +cygwin=false; +darwin=false; +mingw=false +case "`uname`" in + CYGWIN*) cygwin=true ;; + MINGW*) mingw=true;; + Darwin*) darwin=true + # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home + # See https://developer.apple.com/library/mac/qa/qa1170/_index.html + if [ -z "$JAVA_HOME" ]; then + if [ -x "/usr/libexec/java_home" ]; then + export JAVA_HOME="`/usr/libexec/java_home`" + else + export JAVA_HOME="/Library/Java/Home" + fi + fi + ;; +esac + +if [ -z "$JAVA_HOME" ] ; then + if [ -r /etc/gentoo-release ] ; then + JAVA_HOME=`java-config --jre-home` + fi +fi + +if [ -z "$M2_HOME" ] ; then + ## resolve links - $0 may be a link to maven's home + PRG="$0" + + # need this for relative symlinks + while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG="`dirname "$PRG"`/$link" + fi + done + + saveddir=`pwd` + + M2_HOME=`dirname "$PRG"`/.. + + # make it fully qualified + M2_HOME=`cd "$M2_HOME" && pwd` + + cd "$saveddir" + # echo Using m2 at $M2_HOME +fi + +# For Cygwin, ensure paths are in UNIX format before anything is touched +if $cygwin ; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --unix "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --unix "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --unix "$CLASSPATH"` +fi + +# For Migwn, ensure paths are in UNIX format before anything is touched +if $mingw ; then + [ -n "$M2_HOME" ] && + M2_HOME="`(cd "$M2_HOME"; pwd)`" + [ -n "$JAVA_HOME" ] && + JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`" + # TODO classpath? +fi + +if [ -z "$JAVA_HOME" ]; then + javaExecutable="`which javac`" + if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then + # readlink(1) is not available as standard on Solaris 10. + readLink=`which readlink` + if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then + if $darwin ; then + javaHome="`dirname \"$javaExecutable\"`" + javaExecutable="`cd \"$javaHome\" && pwd -P`/javac" + else + javaExecutable="`readlink -f \"$javaExecutable\"`" + fi + javaHome="`dirname \"$javaExecutable\"`" + javaHome=`expr "$javaHome" : '\(.*\)/bin'` + JAVA_HOME="$javaHome" + export JAVA_HOME + fi + fi +fi + +if [ -z "$JAVACMD" ] ; then + if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + else + JAVACMD="`which java`" + fi +fi + +if [ ! -x "$JAVACMD" ] ; then + echo "Error: JAVA_HOME is not defined correctly." >&2 + echo " We cannot execute $JAVACMD" >&2 + exit 1 +fi + +if [ -z "$JAVA_HOME" ] ; then + echo "Warning: JAVA_HOME environment variable is not set." +fi + +CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher + +# traverses directory structure from process work directory to filesystem root +# first directory with .mvn subdirectory is considered project base directory +find_maven_basedir() { + + if [ -z "$1" ] + then + echo "Path not specified to find_maven_basedir" + return 1 + fi + + basedir="$1" + wdir="$1" + while [ "$wdir" != '/' ] ; do + if [ -d "$wdir"/.mvn ] ; then + basedir=$wdir + break + fi + # workaround for JBEAP-8937 (on Solaris 10/Sparc) + if [ -d "${wdir}" ]; then + wdir=`cd "$wdir/.."; pwd` + fi + # end of workaround + done + echo "${basedir}" +} + +# concatenates all lines of a file +concat_lines() { + if [ -f "$1" ]; then + echo "$(tr -s '\n' ' ' < "$1")" + fi +} + +BASE_DIR=`find_maven_basedir "$(pwd)"` +if [ -z "$BASE_DIR" ]; then + exit 1; +fi + +export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"} +echo $MAVEN_PROJECTBASEDIR +MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS" + +# For Cygwin, switch paths to Windows format before running java +if $cygwin; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --path --windows "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --windows "$CLASSPATH"` + [ -n "$MAVEN_PROJECTBASEDIR" ] && + MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"` +fi + +WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +"$JAVACMD" \ + $MAVEN_OPTS \ + -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \ + "-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \ + ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@" + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/mvnw.cmd b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/mvnw.cmd new file mode 100644 index 0000000..b0dc0e7 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/mvnw.cmd @@ -0,0 +1,145 @@ +@REM ---------------------------------------------------------------------------- +@REM Licensed to the Apache Software Foundation (ASF) under one +@REM or more contributor license agreements. See the NOTICE file +@REM distributed with this work for additional information +@REM regarding copyright ownership. The ASF licenses this file +@REM to you under the Apache License, Version 2.0 (the +@REM "License"); you may not use this file except in compliance +@REM with the License. You may obtain a copy of the License at +@REM +@REM http://www.apache.org/licenses/LICENSE-2.0 +@REM +@REM Unless required by applicable law or agreed to in writing, +@REM software distributed under the License is distributed on an +@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +@REM KIND, either express or implied. See the License for the +@REM specific language governing permissions and limitations +@REM under the License. +@REM ---------------------------------------------------------------------------- + +@REM ---------------------------------------------------------------------------- +@REM Maven2 Start Up Batch script +@REM +@REM Required ENV vars: +@REM JAVA_HOME - location of a JDK home dir +@REM +@REM Optional ENV vars +@REM M2_HOME - location of maven2's installed home dir +@REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands +@REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a key stroke before ending +@REM MAVEN_OPTS - parameters passed to the Java VM when running Maven +@REM e.g. to debug Maven itself, use +@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +@REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files +@REM ---------------------------------------------------------------------------- + +@REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on' +@echo off +@REM enable echoing my setting MAVEN_BATCH_ECHO to 'on' +@if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO% + +@REM set %HOME% to equivalent of $HOME +if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%") + +@REM Execute a user defined script before this one +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre +@REM check for pre script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_pre.bat" call "%HOME%\mavenrc_pre.bat" +if exist "%HOME%\mavenrc_pre.cmd" call "%HOME%\mavenrc_pre.cmd" +:skipRcPre + +@setlocal + +set ERROR_CODE=0 + +@REM To isolate internal variables from possible post scripts, we use another setlocal +@setlocal + +@REM ==== START VALIDATION ==== +if not "%JAVA_HOME%" == "" goto OkJHome + +echo. +echo Error: JAVA_HOME not found in your environment. >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +:OkJHome +if exist "%JAVA_HOME%\bin\java.exe" goto init + +echo. +echo Error: JAVA_HOME is set to an invalid directory. >&2 +echo JAVA_HOME = "%JAVA_HOME%" >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +@REM ==== END VALIDATION ==== + +:init + +set MAVEN_CMD_LINE_ARGS=%* + +@REM Find the project base dir, i.e. the directory that contains the folder ".mvn". +@REM Fallback to current working directory if not found. + +set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR% +IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir + +set EXEC_DIR=%CD% +set WDIR=%EXEC_DIR% +:findBaseDir +IF EXIST "%WDIR%"\.mvn goto baseDirFound +cd .. +IF "%WDIR%"=="%CD%" goto baseDirNotFound +set WDIR=%CD% +goto findBaseDir + +:baseDirFound +set MAVEN_PROJECTBASEDIR=%WDIR% +cd "%EXEC_DIR%" +goto endDetectBaseDir + +:baseDirNotFound +set MAVEN_PROJECTBASEDIR=%EXEC_DIR% +cd "%EXEC_DIR%" + +:endDetectBaseDir + +IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig + +@setlocal EnableExtensions EnableDelayedExpansion +for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a +@endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS% + +:endReadAdditionalConfig + +SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe" + +set WRAPPER_JAR="".\.mvn\wrapper\maven-wrapper.jar"" +set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +%MAVEN_JAVA_EXE% %JVM_CONFIG_MAVEN_PROPS% %MAVEN_OPTS% %MAVEN_DEBUG_OPTS% -classpath %WRAPPER_JAR% "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" %WRAPPER_LAUNCHER% %MAVEN_CMD_LINE_ARGS% +if ERRORLEVEL 1 goto error +goto end + +:error +set ERROR_CODE=1 + +:end +@endlocal & set ERROR_CODE=%ERROR_CODE% + +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPost +@REM check for post script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_post.bat" call "%HOME%\mavenrc_post.bat" +if exist "%HOME%\mavenrc_post.cmd" call "%HOME%\mavenrc_post.cmd" +:skipRcPost + +@REM pause the script if MAVEN_BATCH_PAUSE is set to 'on' +if "%MAVEN_BATCH_PAUSE%" == "on" pause + +if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERROR_CODE% + +exit /B %ERROR_CODE% diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/pom.xml b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/pom.xml new file mode 100644 index 0000000..272b690 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/pom.xml @@ -0,0 +1,105 @@ + + + 4.0.0 + + kafka-streams-confluent-producer1 + 0.0.1-SNAPSHOT + jar + kafka-streams-confluent-producer1 + Kafka Streams Confluent Producer1 + + + io.spring.cloud.stream.sample + spring-cloud-stream-samples-parent + 0.0.1-SNAPSHOT + ../../.. + + + + 4.0.0 + 1.8.2 + + + + + org.springframework.cloud + spring-cloud-stream-binder-kafka + + + org.springframework.boot + spring-boot-starter-actuator + + + org.springframework.boot + spring-boot-starter + + + org.springframework.boot + spring-boot-starter-web + + + io.confluent + kafka-avro-serializer + ${confluent.version} + + + org.slf4j + slf4j-api + + + org.slf4j + slf4j-log4j12 + + + + + io.confluent + kafka-schema-registry-client + ${confluent.version} + + + io.confluent + kafka-streams-avro-serde + ${confluent.version} + + + org.slf4j + slf4j-log4j12 + + + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + org.apache.avro + avro-maven-plugin + ${avro.version} + + + generate-sources + + schema + protocol + idl-protocol + + + src/main/resources/avro + + + + + + + + + confluent + http://packages.confluent.io/maven/ + + + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/java/sample/producer1/FooSerde.java b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/java/sample/producer1/FooSerde.java new file mode 100644 index 0000000..e4e06e3 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/java/sample/producer1/FooSerde.java @@ -0,0 +1,21 @@ +package sample.producer1; + +import com.example.Sensor; +import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig; +import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerializer; + +import java.util.Collections; +import java.util.Map; + +/** + * @author Soby Chacko + */ +public class FooSerde extends SpecificAvroSerializer { + + @Override + public void configure(Map serializerConfig, boolean isSerializerForRecordKeys) { + final Map serdeConfig = Collections.singletonMap( + AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081"); + super.configure(serdeConfig, isSerializerForRecordKeys); + } +} diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/java/sample/producer1/Producer1Application.java b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/java/sample/producer1/Producer1Application.java new file mode 100644 index 0000000..2f2acfe --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/java/sample/producer1/Producer1Application.java @@ -0,0 +1,48 @@ +package sample.producer1; + +import com.example.Sensor; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.cloud.stream.annotation.EnableBinding; +import org.springframework.cloud.stream.messaging.Source; +import org.springframework.messaging.support.MessageBuilder; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestMethod; +import org.springframework.web.bind.annotation.RestController; + +import java.util.Random; +import java.util.UUID; + +@SpringBootApplication +@EnableBinding(Source.class) +@RestController +public class Producer1Application { + + @Autowired + private Source source; + + private Random random = new Random(); + + public static void main(String[] args) { + SpringApplication.run(Producer1Application.class, args); + } + + @RequestMapping(value = "/messages", method = RequestMethod.POST) + public String sendMessage() { + source.output().send(MessageBuilder.withPayload(randomSensor()).build()); + return "ok, have fun with v1 payload!"; + } + + private Sensor randomSensor() { + Sensor sensor = new Sensor(); + sensor.setId(UUID.randomUUID().toString() + "-v1"); + sensor.setAcceleration(random.nextFloat() * 10); + sensor.setVelocity(random.nextFloat() * 100); + sensor.setTemperature(random.nextFloat() * 50); + return sensor; + } +} + + + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/resources/application.yml b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/resources/application.yml new file mode 100644 index 0000000..768fa2c --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/resources/application.yml @@ -0,0 +1,13 @@ +spring: + cloud: + stream: + bindings: + output: + producer: + useNativeEncoding: true + destination: sensors +server.port: 9009 +spring.cloud.stream.kafka.binder.configuration: + schema.registry.url: http://localhost:8081 + key.serializer: org.apache.kafka.common.serialization.ByteArraySerializer + value.serializer: sample.producer1.FooSerde \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/resources/avro/sensor.avsc b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/resources/avro/sensor.avsc new file mode 100644 index 0000000..c0e060d --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer1/src/main/resources/avro/sensor.avsc @@ -0,0 +1,11 @@ +{ + "namespace" : "com.example", + "type" : "record", + "name" : "Sensor", + "fields" : [ + {"name":"id","type":"string"}, + {"name":"temperature", "type":"float", "default":0.0}, + {"name":"acceleration", "type":"float","default":0.0}, + {"name":"velocity","type":"float","default":0.0} + ] +} diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.gitignore b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.gitignore new file mode 100644 index 0000000..82eca33 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.gitignore @@ -0,0 +1,25 @@ +/target/ +!.mvn/wrapper/maven-wrapper.jar + +### STS ### +.apt_generated +.classpath +.factorypath +.project +.settings +.springBeans +.sts4-cache + +### IntelliJ IDEA ### +.idea +*.iws +*.iml +*.ipr + +### NetBeans ### +/nbproject/private/ +/build/ +/nbbuild/ +/dist/ +/nbdist/ +/.nb-gradle/ \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/jvm.config b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/jvm.config new file mode 100644 index 0000000..0e7dabe --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/jvm.config @@ -0,0 +1 @@ +-Xmx1024m -XX:CICompilerCount=1 -XX:TieredStopAtLevel=1 -Djava.security.egd=file:/dev/./urandom \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/maven.config b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/maven.config new file mode 100644 index 0000000..3b8cf46 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/maven.config @@ -0,0 +1 @@ +-DaltSnapshotDeploymentRepository=repo.spring.io::default::https://repo.spring.io/libs-snapshot-local -P spring diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/wrapper/maven-wrapper.jar b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/wrapper/maven-wrapper.jar new file mode 100644 index 0000000..5fd4d50 Binary files /dev/null and b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/wrapper/maven-wrapper.jar differ diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/wrapper/maven-wrapper.properties b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/wrapper/maven-wrapper.properties new file mode 100644 index 0000000..6637ced --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/.mvn/wrapper/maven-wrapper.properties @@ -0,0 +1 @@ +distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.3.9/apache-maven-3.3.9-bin.zip \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/mvnw b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/mvnw new file mode 100755 index 0000000..6efc7bd --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/mvnw @@ -0,0 +1,226 @@ +#!/bin/sh +# ---------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# ---------------------------------------------------------------------------- + +# ---------------------------------------------------------------------------- +# Maven2 Start Up Batch script +# +# Required ENV vars: +# ------------------ +# JAVA_HOME - location of a JDK home dir +# +# Optional ENV vars +# ----------------- +# M2_HOME - location of maven2's installed home dir +# MAVEN_OPTS - parameters passed to the Java VM when running Maven +# e.g. to debug Maven itself, use +# set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +# MAVEN_SKIP_RC - flag to disable loading of mavenrc files +# ---------------------------------------------------------------------------- + +if [ -z "$MAVEN_SKIP_RC" ] ; then + + if [ -f /etc/mavenrc ] ; then + . /etc/mavenrc + fi + + if [ -f "$HOME/.mavenrc" ] ; then + . "$HOME/.mavenrc" + fi + +fi + +# OS specific support. $var _must_ be set to either true or false. +cygwin=false; +darwin=false; +mingw=false +case "`uname`" in + CYGWIN*) cygwin=true ;; + MINGW*) mingw=true;; + Darwin*) darwin=true + # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home + # See https://developer.apple.com/library/mac/qa/qa1170/_index.html + if [ -z "$JAVA_HOME" ]; then + if [ -x "/usr/libexec/java_home" ]; then + export JAVA_HOME="`/usr/libexec/java_home`" + else + export JAVA_HOME="/Library/Java/Home" + fi + fi + ;; +esac + +if [ -z "$JAVA_HOME" ] ; then + if [ -r /etc/gentoo-release ] ; then + JAVA_HOME=`java-config --jre-home` + fi +fi + +if [ -z "$M2_HOME" ] ; then + ## resolve links - $0 may be a link to maven's home + PRG="$0" + + # need this for relative symlinks + while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG="`dirname "$PRG"`/$link" + fi + done + + saveddir=`pwd` + + M2_HOME=`dirname "$PRG"`/.. + + # make it fully qualified + M2_HOME=`cd "$M2_HOME" && pwd` + + cd "$saveddir" + # echo Using m2 at $M2_HOME +fi + +# For Cygwin, ensure paths are in UNIX format before anything is touched +if $cygwin ; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --unix "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --unix "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --unix "$CLASSPATH"` +fi + +# For Migwn, ensure paths are in UNIX format before anything is touched +if $mingw ; then + [ -n "$M2_HOME" ] && + M2_HOME="`(cd "$M2_HOME"; pwd)`" + [ -n "$JAVA_HOME" ] && + JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`" + # TODO classpath? +fi + +if [ -z "$JAVA_HOME" ]; then + javaExecutable="`which javac`" + if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then + # readlink(1) is not available as standard on Solaris 10. + readLink=`which readlink` + if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then + if $darwin ; then + javaHome="`dirname \"$javaExecutable\"`" + javaExecutable="`cd \"$javaHome\" && pwd -P`/javac" + else + javaExecutable="`readlink -f \"$javaExecutable\"`" + fi + javaHome="`dirname \"$javaExecutable\"`" + javaHome=`expr "$javaHome" : '\(.*\)/bin'` + JAVA_HOME="$javaHome" + export JAVA_HOME + fi + fi +fi + +if [ -z "$JAVACMD" ] ; then + if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + else + JAVACMD="`which java`" + fi +fi + +if [ ! -x "$JAVACMD" ] ; then + echo "Error: JAVA_HOME is not defined correctly." >&2 + echo " We cannot execute $JAVACMD" >&2 + exit 1 +fi + +if [ -z "$JAVA_HOME" ] ; then + echo "Warning: JAVA_HOME environment variable is not set." +fi + +CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher + +# traverses directory structure from process work directory to filesystem root +# first directory with .mvn subdirectory is considered project base directory +find_maven_basedir() { + + if [ -z "$1" ] + then + echo "Path not specified to find_maven_basedir" + return 1 + fi + + basedir="$1" + wdir="$1" + while [ "$wdir" != '/' ] ; do + if [ -d "$wdir"/.mvn ] ; then + basedir=$wdir + break + fi + # workaround for JBEAP-8937 (on Solaris 10/Sparc) + if [ -d "${wdir}" ]; then + wdir=`cd "$wdir/.."; pwd` + fi + # end of workaround + done + echo "${basedir}" +} + +# concatenates all lines of a file +concat_lines() { + if [ -f "$1" ]; then + echo "$(tr -s '\n' ' ' < "$1")" + fi +} + +BASE_DIR=`find_maven_basedir "$(pwd)"` +if [ -z "$BASE_DIR" ]; then + exit 1; +fi + +export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"} +echo $MAVEN_PROJECTBASEDIR +MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS" + +# For Cygwin, switch paths to Windows format before running java +if $cygwin; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --path --windows "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --windows "$CLASSPATH"` + [ -n "$MAVEN_PROJECTBASEDIR" ] && + MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"` +fi + +WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +"$JAVACMD" \ + $MAVEN_OPTS \ + -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \ + "-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \ + ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@" + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/mvnw.cmd b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/mvnw.cmd new file mode 100644 index 0000000..b0dc0e7 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/mvnw.cmd @@ -0,0 +1,145 @@ +@REM ---------------------------------------------------------------------------- +@REM Licensed to the Apache Software Foundation (ASF) under one +@REM or more contributor license agreements. See the NOTICE file +@REM distributed with this work for additional information +@REM regarding copyright ownership. The ASF licenses this file +@REM to you under the Apache License, Version 2.0 (the +@REM "License"); you may not use this file except in compliance +@REM with the License. You may obtain a copy of the License at +@REM +@REM http://www.apache.org/licenses/LICENSE-2.0 +@REM +@REM Unless required by applicable law or agreed to in writing, +@REM software distributed under the License is distributed on an +@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +@REM KIND, either express or implied. See the License for the +@REM specific language governing permissions and limitations +@REM under the License. +@REM ---------------------------------------------------------------------------- + +@REM ---------------------------------------------------------------------------- +@REM Maven2 Start Up Batch script +@REM +@REM Required ENV vars: +@REM JAVA_HOME - location of a JDK home dir +@REM +@REM Optional ENV vars +@REM M2_HOME - location of maven2's installed home dir +@REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands +@REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a key stroke before ending +@REM MAVEN_OPTS - parameters passed to the Java VM when running Maven +@REM e.g. to debug Maven itself, use +@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +@REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files +@REM ---------------------------------------------------------------------------- + +@REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on' +@echo off +@REM enable echoing my setting MAVEN_BATCH_ECHO to 'on' +@if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO% + +@REM set %HOME% to equivalent of $HOME +if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%") + +@REM Execute a user defined script before this one +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre +@REM check for pre script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_pre.bat" call "%HOME%\mavenrc_pre.bat" +if exist "%HOME%\mavenrc_pre.cmd" call "%HOME%\mavenrc_pre.cmd" +:skipRcPre + +@setlocal + +set ERROR_CODE=0 + +@REM To isolate internal variables from possible post scripts, we use another setlocal +@setlocal + +@REM ==== START VALIDATION ==== +if not "%JAVA_HOME%" == "" goto OkJHome + +echo. +echo Error: JAVA_HOME not found in your environment. >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +:OkJHome +if exist "%JAVA_HOME%\bin\java.exe" goto init + +echo. +echo Error: JAVA_HOME is set to an invalid directory. >&2 +echo JAVA_HOME = "%JAVA_HOME%" >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +@REM ==== END VALIDATION ==== + +:init + +set MAVEN_CMD_LINE_ARGS=%* + +@REM Find the project base dir, i.e. the directory that contains the folder ".mvn". +@REM Fallback to current working directory if not found. + +set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR% +IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir + +set EXEC_DIR=%CD% +set WDIR=%EXEC_DIR% +:findBaseDir +IF EXIST "%WDIR%"\.mvn goto baseDirFound +cd .. +IF "%WDIR%"=="%CD%" goto baseDirNotFound +set WDIR=%CD% +goto findBaseDir + +:baseDirFound +set MAVEN_PROJECTBASEDIR=%WDIR% +cd "%EXEC_DIR%" +goto endDetectBaseDir + +:baseDirNotFound +set MAVEN_PROJECTBASEDIR=%EXEC_DIR% +cd "%EXEC_DIR%" + +:endDetectBaseDir + +IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig + +@setlocal EnableExtensions EnableDelayedExpansion +for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a +@endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS% + +:endReadAdditionalConfig + +SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe" + +set WRAPPER_JAR="".\.mvn\wrapper\maven-wrapper.jar"" +set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +%MAVEN_JAVA_EXE% %JVM_CONFIG_MAVEN_PROPS% %MAVEN_OPTS% %MAVEN_DEBUG_OPTS% -classpath %WRAPPER_JAR% "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" %WRAPPER_LAUNCHER% %MAVEN_CMD_LINE_ARGS% +if ERRORLEVEL 1 goto error +goto end + +:error +set ERROR_CODE=1 + +:end +@endlocal & set ERROR_CODE=%ERROR_CODE% + +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPost +@REM check for post script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_post.bat" call "%HOME%\mavenrc_post.bat" +if exist "%HOME%\mavenrc_post.cmd" call "%HOME%\mavenrc_post.cmd" +:skipRcPost + +@REM pause the script if MAVEN_BATCH_PAUSE is set to 'on' +if "%MAVEN_BATCH_PAUSE%" == "on" pause + +if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERROR_CODE% + +exit /B %ERROR_CODE% diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/pom.xml b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/pom.xml new file mode 100644 index 0000000..cc9dc57 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/pom.xml @@ -0,0 +1,105 @@ + + + 4.0.0 + + kafka-streams-confluent-producer2 + 0.0.1-SNAPSHOT + jar + kafka-streams-confluent-producer2 + Kafka Streams Confluent Producer2 + + + io.spring.cloud.stream.sample + spring-cloud-stream-samples-parent + 0.0.1-SNAPSHOT + ../../.. + + + + 4.0.0 + 1.8.2 + + + + + org.springframework.cloud + spring-cloud-stream-binder-kafka + + + org.springframework.boot + spring-boot-starter-actuator + + + org.springframework.boot + spring-boot-starter + + + org.springframework.boot + spring-boot-starter-web + + + io.confluent + kafka-avro-serializer + ${confluent.version} + + + org.slf4j + slf4j-api + + + org.slf4j + slf4j-log4j12 + + + + + io.confluent + kafka-schema-registry-client + ${confluent.version} + + + io.confluent + kafka-streams-avro-serde + ${confluent.version} + + + org.slf4j + slf4j-log4j12 + + + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + org.apache.avro + avro-maven-plugin + ${avro.version} + + + generate-sources + + schema + protocol + idl-protocol + + + src/main/resources/avro + + + + + + + + + confluent + http://packages.confluent.io/maven/ + + + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/java/sample/producer2/FooSerde.java b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/java/sample/producer2/FooSerde.java new file mode 100644 index 0000000..3e52fb3 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/java/sample/producer2/FooSerde.java @@ -0,0 +1,21 @@ +package sample.producer2; + +import com.example.Sensor; +import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig; +import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerializer; + +import java.util.Collections; +import java.util.Map; + +/** + * @author Soby Chacko + */ +public class FooSerde extends SpecificAvroSerializer { + + @Override + public void configure(Map serializerConfig, boolean isSerializerForRecordKeys) { + final Map serdeConfig = Collections.singletonMap( + AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081"); + super.configure(serdeConfig, isSerializerForRecordKeys); + } +} diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/java/sample/producer2/Producer2Application.java b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/java/sample/producer2/Producer2Application.java new file mode 100644 index 0000000..cedc8e9 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/java/sample/producer2/Producer2Application.java @@ -0,0 +1,48 @@ +package sample.producer2; + +import com.example.Sensor; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.cloud.stream.annotation.EnableBinding; +import org.springframework.cloud.stream.messaging.Source; +import org.springframework.messaging.support.MessageBuilder; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestMethod; +import org.springframework.web.bind.annotation.RestController; + +import java.util.Random; +import java.util.UUID; + +@SpringBootApplication +@EnableBinding(Source.class) +@RestController +public class Producer2Application { + + @Autowired + private Source source; + + private Random random = new Random(); + + public static void main(String[] args) { + SpringApplication.run(Producer2Application.class, args); + } + + @RequestMapping(value = "/messages", method = RequestMethod.POST) + public String sendMessage() { + source.output().send(MessageBuilder.withPayload(randomSensor()).build()); + return "ok, have fun with v2 payload!"; + } + + private Sensor randomSensor() { + Sensor sensor = new Sensor(); + sensor.setId(UUID.randomUUID().toString() + "-v2"); + sensor.setAcceleration(random.nextFloat() * 10); + sensor.setVelocity(random.nextFloat() * 100); + sensor.setInternalTemperature(random.nextFloat() * 50); + sensor.setAccelerometer(null); + sensor.setMagneticField(null); + return sensor; + } +} + diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/resources/application.yml b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/resources/application.yml new file mode 100644 index 0000000..631502a --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/resources/application.yml @@ -0,0 +1,13 @@ +spring: + cloud: + stream: + bindings: + output: + producer: + useNativeEncoding: true + destination: sensors +server.port: 9010 +spring.cloud.stream.kafka.binder.configuration: + schema.registry.url: http://localhost:8081 + key.serializer: org.apache.kafka.common.serialization.ByteArraySerializer + value.serializer: sample.producer2.FooSerde \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/resources/avro/sensor.avsc b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/resources/avro/sensor.avsc new file mode 100644 index 0000000..8d2e605 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/kafka-streams-confluent-producer2/src/main/resources/avro/sensor.avsc @@ -0,0 +1,25 @@ +{ + "namespace" : "com.example", + "type" : "record", + "name" : "Sensor", + "fields" : [ + {"name":"id","type":"string"}, + {"name":"internalTemperature", "type":"float", "default":0.0, "aliases":["temperature"]}, + {"name":"externalTemperature", "type":"float", "default":0.0}, + {"name":"acceleration", "type":"float","default":0.0}, + {"name":"velocity","type":"float","default":0.0}, + {"name":"accelerometer","type":[ + "null",{ + "type":"array", + "items":"float" + } + ]}, + {"name":"magneticField","type":[ + "null",{ + "type":"array", + "items":"float" + } + ]} + ] + +} \ No newline at end of file diff --git a/schema-registry-samples/kafka-streams-schema-evolution/mvnw b/schema-registry-samples/kafka-streams-schema-evolution/mvnw new file mode 100755 index 0000000..5bf251c --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/mvnw @@ -0,0 +1,225 @@ +#!/bin/sh +# ---------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# ---------------------------------------------------------------------------- + +# ---------------------------------------------------------------------------- +# Maven2 Start Up Batch script +# +# Required ENV vars: +# ------------------ +# JAVA_HOME - location of a JDK home dir +# +# Optional ENV vars +# ----------------- +# M2_HOME - location of maven2's installed home dir +# MAVEN_OPTS - parameters passed to the Java VM when running Maven +# e.g. to debug Maven itself, use +# set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +# MAVEN_SKIP_RC - flag to disable loading of mavenrc files +# ---------------------------------------------------------------------------- + +if [ -z "$MAVEN_SKIP_RC" ] ; then + + if [ -f /etc/mavenrc ] ; then + . /etc/mavenrc + fi + + if [ -f "$HOME/.mavenrc" ] ; then + . "$HOME/.mavenrc" + fi + +fi + +# OS specific support. $var _must_ be set to either true or false. +cygwin=false; +darwin=false; +mingw=false +case "`uname`" in + CYGWIN*) cygwin=true ;; + MINGW*) mingw=true;; + Darwin*) darwin=true + # Use /usr/libexec/java_home if available, otherwise fall back to /Library/Java/Home + # See https://developer.apple.com/library/mac/qa/qa1170/_index.html + if [ -z "$JAVA_HOME" ]; then + if [ -x "/usr/libexec/java_home" ]; then + export JAVA_HOME="`/usr/libexec/java_home`" + else + export JAVA_HOME="/Library/Java/Home" + fi + fi + ;; +esac + +if [ -z "$JAVA_HOME" ] ; then + if [ -r /etc/gentoo-release ] ; then + JAVA_HOME=`java-config --jre-home` + fi +fi + +if [ -z "$M2_HOME" ] ; then + ## resolve links - $0 may be a link to maven's home + PRG="$0" + + # need this for relative symlinks + while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG="`dirname "$PRG"`/$link" + fi + done + + saveddir=`pwd` + + M2_HOME=`dirname "$PRG"`/.. + + # make it fully qualified + M2_HOME=`cd "$M2_HOME" && pwd` + + cd "$saveddir" + # echo Using m2 at $M2_HOME +fi + +# For Cygwin, ensure paths are in UNIX format before anything is touched +if $cygwin ; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --unix "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --unix "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --unix "$CLASSPATH"` +fi + +# For Migwn, ensure paths are in UNIX format before anything is touched +if $mingw ; then + [ -n "$M2_HOME" ] && + M2_HOME="`(cd "$M2_HOME"; pwd)`" + [ -n "$JAVA_HOME" ] && + JAVA_HOME="`(cd "$JAVA_HOME"; pwd)`" + # TODO classpath? +fi + +if [ -z "$JAVA_HOME" ]; then + javaExecutable="`which javac`" + if [ -n "$javaExecutable" ] && ! [ "`expr \"$javaExecutable\" : '\([^ ]*\)'`" = "no" ]; then + # readlink(1) is not available as standard on Solaris 10. + readLink=`which readlink` + if [ ! `expr "$readLink" : '\([^ ]*\)'` = "no" ]; then + if $darwin ; then + javaHome="`dirname \"$javaExecutable\"`" + javaExecutable="`cd \"$javaHome\" && pwd -P`/javac" + else + javaExecutable="`readlink -f \"$javaExecutable\"`" + fi + javaHome="`dirname \"$javaExecutable\"`" + javaHome=`expr "$javaHome" : '\(.*\)/bin'` + JAVA_HOME="$javaHome" + export JAVA_HOME + fi + fi +fi + +if [ -z "$JAVACMD" ] ; then + if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + else + JAVACMD="`which java`" + fi +fi + +if [ ! -x "$JAVACMD" ] ; then + echo "Error: JAVA_HOME is not defined correctly." >&2 + echo " We cannot execute $JAVACMD" >&2 + exit 1 +fi + +if [ -z "$JAVA_HOME" ] ; then + echo "Warning: JAVA_HOME environment variable is not set." +fi + +CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher + +# traverses directory structure from process work directory to filesystem root +# first directory with .mvn subdirectory is considered project base directory +find_maven_basedir() { + + if [ -z "$1" ] + then + echo "Path not specified to find_maven_basedir" + return 1 + fi + + basedir="$1" + wdir="$1" + while [ "$wdir" != '/' ] ; do + if [ -d "$wdir"/.mvn ] ; then + basedir=$wdir + break + fi + # workaround for JBEAP-8937 (on Solaris 10/Sparc) + if [ -d "${wdir}" ]; then + wdir=`cd "$wdir/.."; pwd` + fi + # end of workaround + done + echo "${basedir}" +} + +# concatenates all lines of a file +concat_lines() { + if [ -f "$1" ]; then + echo "$(tr -s '\n' ' ' < "$1")" + fi +} + +BASE_DIR=`find_maven_basedir "$(pwd)"` +if [ -z "$BASE_DIR" ]; then + exit 1; +fi + +export MAVEN_PROJECTBASEDIR=${MAVEN_BASEDIR:-"$BASE_DIR"} +echo $MAVEN_PROJECTBASEDIR +MAVEN_OPTS="$(concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config") $MAVEN_OPTS" + +# For Cygwin, switch paths to Windows format before running java +if $cygwin; then + [ -n "$M2_HOME" ] && + M2_HOME=`cygpath --path --windows "$M2_HOME"` + [ -n "$JAVA_HOME" ] && + JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"` + [ -n "$CLASSPATH" ] && + CLASSPATH=`cygpath --path --windows "$CLASSPATH"` + [ -n "$MAVEN_PROJECTBASEDIR" ] && + MAVEN_PROJECTBASEDIR=`cygpath --path --windows "$MAVEN_PROJECTBASEDIR"` +fi + +WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +exec "$JAVACMD" \ + $MAVEN_OPTS \ + -classpath "$MAVEN_PROJECTBASEDIR/.mvn/wrapper/maven-wrapper.jar" \ + "-Dmaven.home=${M2_HOME}" "-Dmaven.multiModuleProjectDirectory=${MAVEN_PROJECTBASEDIR}" \ + ${WRAPPER_LAUNCHER} $MAVEN_CONFIG "$@" diff --git a/schema-registry-samples/kafka-streams-schema-evolution/mvnw.cmd b/schema-registry-samples/kafka-streams-schema-evolution/mvnw.cmd new file mode 100644 index 0000000..019bd74 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/mvnw.cmd @@ -0,0 +1,143 @@ +@REM ---------------------------------------------------------------------------- +@REM Licensed to the Apache Software Foundation (ASF) under one +@REM or more contributor license agreements. See the NOTICE file +@REM distributed with this work for additional information +@REM regarding copyright ownership. The ASF licenses this file +@REM to you under the Apache License, Version 2.0 (the +@REM "License"); you may not use this file except in compliance +@REM with the License. You may obtain a copy of the License at +@REM +@REM http://www.apache.org/licenses/LICENSE-2.0 +@REM +@REM Unless required by applicable law or agreed to in writing, +@REM software distributed under the License is distributed on an +@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +@REM KIND, either express or implied. See the License for the +@REM specific language governing permissions and limitations +@REM under the License. +@REM ---------------------------------------------------------------------------- + +@REM ---------------------------------------------------------------------------- +@REM Maven2 Start Up Batch script +@REM +@REM Required ENV vars: +@REM JAVA_HOME - location of a JDK home dir +@REM +@REM Optional ENV vars +@REM M2_HOME - location of maven2's installed home dir +@REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands +@REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a key stroke before ending +@REM MAVEN_OPTS - parameters passed to the Java VM when running Maven +@REM e.g. to debug Maven itself, use +@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 +@REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files +@REM ---------------------------------------------------------------------------- + +@REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on' +@echo off +@REM enable echoing my setting MAVEN_BATCH_ECHO to 'on' +@if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO% + +@REM set %HOME% to equivalent of $HOME +if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%") + +@REM Execute a user defined script before this one +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre +@REM check for pre script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_pre.bat" call "%HOME%\mavenrc_pre.bat" +if exist "%HOME%\mavenrc_pre.cmd" call "%HOME%\mavenrc_pre.cmd" +:skipRcPre + +@setlocal + +set ERROR_CODE=0 + +@REM To isolate internal variables from possible post scripts, we use another setlocal +@setlocal + +@REM ==== START VALIDATION ==== +if not "%JAVA_HOME%" == "" goto OkJHome + +echo. +echo Error: JAVA_HOME not found in your environment. >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +:OkJHome +if exist "%JAVA_HOME%\bin\java.exe" goto init + +echo. +echo Error: JAVA_HOME is set to an invalid directory. >&2 +echo JAVA_HOME = "%JAVA_HOME%" >&2 +echo Please set the JAVA_HOME variable in your environment to match the >&2 +echo location of your Java installation. >&2 +echo. +goto error + +@REM ==== END VALIDATION ==== + +:init + +@REM Find the project base dir, i.e. the directory that contains the folder ".mvn". +@REM Fallback to current working directory if not found. + +set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR% +IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir + +set EXEC_DIR=%CD% +set WDIR=%EXEC_DIR% +:findBaseDir +IF EXIST "%WDIR%"\.mvn goto baseDirFound +cd .. +IF "%WDIR%"=="%CD%" goto baseDirNotFound +set WDIR=%CD% +goto findBaseDir + +:baseDirFound +set MAVEN_PROJECTBASEDIR=%WDIR% +cd "%EXEC_DIR%" +goto endDetectBaseDir + +:baseDirNotFound +set MAVEN_PROJECTBASEDIR=%EXEC_DIR% +cd "%EXEC_DIR%" + +:endDetectBaseDir + +IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig + +@setlocal EnableExtensions EnableDelayedExpansion +for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a +@endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS% + +:endReadAdditionalConfig + +SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe" + +set WRAPPER_JAR="%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.jar" +set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain + +%MAVEN_JAVA_EXE% %JVM_CONFIG_MAVEN_PROPS% %MAVEN_OPTS% %MAVEN_DEBUG_OPTS% -classpath %WRAPPER_JAR% "-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" %WRAPPER_LAUNCHER% %MAVEN_CONFIG% %* +if ERRORLEVEL 1 goto error +goto end + +:error +set ERROR_CODE=1 + +:end +@endlocal & set ERROR_CODE=%ERROR_CODE% + +if not "%MAVEN_SKIP_RC%" == "" goto skipRcPost +@REM check for post script, once with legacy .bat ending and once with .cmd ending +if exist "%HOME%\mavenrc_post.bat" call "%HOME%\mavenrc_post.bat" +if exist "%HOME%\mavenrc_post.cmd" call "%HOME%\mavenrc_post.cmd" +:skipRcPost + +@REM pause the script if MAVEN_BATCH_PAUSE is set to 'on' +if "%MAVEN_BATCH_PAUSE%" == "on" pause + +if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERROR_CODE% + +exit /B %ERROR_CODE% diff --git a/schema-registry-samples/kafka-streams-schema-evolution/pom.xml b/schema-registry-samples/kafka-streams-schema-evolution/pom.xml new file mode 100644 index 0000000..6ef6f87 --- /dev/null +++ b/schema-registry-samples/kafka-streams-schema-evolution/pom.xml @@ -0,0 +1,16 @@ + + + 4.0.0 + io.spring.cloud.stream.sample + kafka-streams-schema-evolution + 0.0.1-SNAPSHOT + pom + kafka-streams-schema-evolution + Kafka Streams Schema Evolution Sample + + + kafka-streams-confluent-producer1 + kafka-streams-confluent-producer2 + kafka-streams-confluent-consumer + + diff --git a/schema-registry-samples/pom.xml b/schema-registry-samples/pom.xml index d847ce6..8232f0b 100644 --- a/schema-registry-samples/pom.xml +++ b/schema-registry-samples/pom.xml @@ -11,6 +11,7 @@ schema-registry-vanilla schema-registry-confluent + kafka-streams-schema-evolution