SpringCloud/spring-cloud-stream
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Sync docs from master to gh-pages

Browse Source
This commit is contained in:
Oleg Zhurakousky
2018-10-26 11:29:08 +02:00
parent 65f0b470a3
commit 5b1cdcf644
16 changed files with 71 additions and 272 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
multi/multi_schema-evolution.html
View File

@@ -1,6 +1,6 @@
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>10.&nbsp;Schema Evolution Support</title><link rel="stylesheet" type="text/css" href="css/manual-multipage.css"><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"><link rel="home" href="multi_spring-cloud-stream.html" title="Spring Cloud Stream Reference Guide"><link rel="up" href="multi__reference_guide.html" title="Part&nbsp;II.&nbsp;Reference Guide"><link rel="prev" href="multi_content-type-management.html" title="9.&nbsp;Content Type Negotiation"><link rel="next" href="multi__inter_application_communication.html" title="11.&nbsp;Inter-Application Communication"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">10.&nbsp;Schema Evolution Support</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="multi_content-type-management.html">Prev</a>&nbsp;</td><th width="60%" align="center">Part&nbsp;II.&nbsp;Reference Guide</th><td width="20%" align="right">&nbsp;<a accesskey="n" href="multi__inter_application_communication.html">Next</a></td></tr></table><hr></div><div class="chapter"><div class="titlepage"><div><div><h2 class="title"><a name="schema-evolution" href="#schema-evolution"></a>10.&nbsp;Schema Evolution Support</h2></div></div></div><p>Spring Cloud Stream provides support for schema evolution so that the data can be evolved over time and still work with older or newer producers and consumers and vice versa.
<title>10.&nbsp;Schema Evolution Support</title><link rel="stylesheet" type="text/css" href="css/manual-multipage.css"><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"><link rel="home" href="multi_spring-cloud-stream.html" title="Spring Cloud Stream Reference Guide"><link rel="up" href="multi__preface.html" title="Part&nbsp;I.&nbsp;Preface"><link rel="prev" href="multi_content-type-management.html" title="9.&nbsp;Content Type Negotiation"><link rel="next" href="multi__inter_application_communication.html" title="11.&nbsp;Inter-Application Communication"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">10.&nbsp;Schema Evolution Support</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="multi_content-type-management.html">Prev</a>&nbsp;</td><th width="60%" align="center">Part&nbsp;I.&nbsp;Preface</th><td width="20%" align="right">&nbsp;<a accesskey="n" href="multi__inter_application_communication.html">Next</a></td></tr></table><hr></div><div class="chapter"><div class="titlepage"><div><div><h2 class="title"><a name="schema-evolution" href="#schema-evolution"></a>10.&nbsp;Schema Evolution Support</h2></div></div></div><p>Spring Cloud Stream provides support for schema evolution so that the data can be evolved over time and still work with older or newer producers and consumers and vice versa.
Most serialization models, especially the ones that aim for portability across different platforms and languages, rely on a schema that describes how the data is serialized in the binary payload.
In order to serialize the data and then to interpret it, both the sending and receiving sides must have access to a schema that describes the binary format.
In certain cases, the schema can be inferred from the payload type on serialization or from the target type on deserialization.
@@ -23,7 +23,7 @@ A schema is referenceable as a tuple consisting of:</p><div class="itemizedlist"
Because of this, it uses a <code class="literal">DefaultSchemaRegistryClient</code> that does not cache responses.
If you intend to change the default behavior, you can use the client directly on your code and override it to the desired outcome.
To do so, you have to add the property <code class="literal">spring.cloud.stream.schemaRegistryClient.cached=true</code> to your application properties.</p></td></tr></table></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_schema_registry_client_properties" href="#_schema_registry_client_properties"></a>10.1.1&nbsp;Schema Registry Client Properties</h3></div></div></div><p>The Schema Registry Client supports the following properties:</p><div class="variablelist"><dl class="variablelist"><dt><span class="term"><code class="literal">spring.cloud.stream.schemaRegistryClient.endpoint</code></span></dt><dd>The location of the schema-server.
When setting this, use a full URL, including protocol (<code class="literal">http</code> or <code class="literal">https</code>) , port, and context path.</dd><dt><span class="term">Default</span></dt><dd><code class="literal"><a class="link" href="http://localhost:8990/" target="_top">localhost:8990/</a></code></dd><dt><span class="term"><code class="literal">spring.cloud.stream.schemaRegistryClient.cached</code></span></dt><dd>Whether the client should cache schema server responses.
When setting this, use a full URL, including protocol (<code class="literal">http</code> or <code class="literal">https</code>) , port, and context path.</dd><dt><span class="term">Default</span></dt><dd><code class="literal"><a class="link" href="http://localhost:8990/" target="_top">http://localhost:8990/</a></code></dd><dt><span class="term"><code class="literal">spring.cloud.stream.schemaRegistryClient.cached</code></span></dt><dd>Whether the client should cache schema server responses.
Normally set to <code class="literal">false</code>, as the caching happens in the message converter.
Clients using the schema registry client should set this to <code class="literal">true</code>.</dd><dt><span class="term">Default</span></dt><dd><code class="literal">false</code></dd></dl></div></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_avro_schema_registry_client_message_converters" href="#_avro_schema_registry_client_message_converters"></a>10.2&nbsp;Avro Schema Registry Client Message Converters</h2></div></div></div><p>For applications that have a SchemaRegistryClient bean registered with the application context, Spring Cloud Stream auto configures an Apache Avro message converter for schema management.
This eases schema evolution, as applications that receive messages can get easy access to a writer schema that can be reconciled with their own reader schema.</p><p>For outbound messages, if the content type of the channel is set to <code class="literal">application/*+avro</code>, the <code class="literal">MessageConverter</code> is activated, as shown in the following example:</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-attribute">spring.cloud.stream.bindings.output.contentType</span>=application/*+avro</pre><p>During the outbound conversion, the message converter tries to infer the schema of each outbound messages (based on its type) and register it to a subject (based on the payload type) by using the <code class="literal">SchemaRegistryClient</code>.
@@ -79,10 +79,10 @@ If you want to use the Confluent schema registry, you need to create a bean of t
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">return</span> client;
}</pre><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>The ConfluentSchemaRegistryClient is tested against Confluent platform version 4.0.0.</p></td></tr></table></div></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_schema_registration_and_resolution" href="#_schema_registration_and_resolution"></a>10.6&nbsp;Schema Registration and Resolution</h2></div></div></div><p>To better understand how Spring Cloud Stream registers and resolves new schemas and its use of Avro schema comparison features, we provide two separate subsections:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><span class="quote">&#8220;<span class="quote"><a class="xref" href="multi_schema-evolution.html#spring-cloud-stream-overview-schema-registration-process" title="10.6.1&nbsp;Schema Registration Process (Serialization)">Section&nbsp;10.6.1, &#8220;Schema Registration Process (Serialization)&#8221;</a></span>&#8221;</span></li><li class="listitem"><span class="quote">&#8220;<span class="quote"><a class="xref" href="multi_schema-evolution.html#spring-cloud-stream-overview-schema-resolution-process" title="10.6.2&nbsp;Schema Resolution Process (Deserialization)">Section&nbsp;10.6.2, &#8220;Schema Resolution Process (Deserialization)&#8221;</a></span>&#8221;</span></li></ul></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="spring-cloud-stream-overview-schema-registration-process" href="#spring-cloud-stream-overview-schema-registration-process"></a>10.6.1&nbsp;Schema Registration Process (Serialization)</h3></div></div></div><p>The first part of the registration process is extracting a schema from the payload that is being sent over a channel.
Avro types such as <code class="literal">SpecificRecord</code> or <code class="literal">GenericRecord</code> already contain a schema, which can be retrieved immediately from the instance.
In the case of POJOs, a schema is inferred if the <code class="literal">spring.cloud.stream.schema.avro.dynamicSchemaGenerationEnabled</code> property is set to <code class="literal">true</code> (the default).</p><div class="figure"><a name="d0e4098" href="#d0e4098"></a><p class="title"><b>Figure&nbsp;10.1.&nbsp;Schema Writer Resolution Process</b></p><div class="figure-contents"><div class="mediaobject" align="center"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-stream/master/docs/src/main/asciidoc/images/schema_resolution.png" align="middle" alt="schema resolution"></div></div></div><br class="figure-break"><p>Ones a schema is obtained, the converter loads its metadata (version) from the remote server.
In the case of POJOs, a schema is inferred if the <code class="literal">spring.cloud.stream.schema.avro.dynamicSchemaGenerationEnabled</code> property is set to <code class="literal">true</code> (the default).</p><div class="figure"><a name="d0e4089" href="#d0e4089"></a><p class="title"><b>Figure&nbsp;10.1.&nbsp;Schema Writer Resolution Process</b></p><div class="figure-contents"><div class="mediaobject" align="center"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-stream/master/docs/src/main/asciidoc/images/schema_resolution.png" align="middle" alt="schema resolution"></div></div></div><br class="figure-break"><p>Ones a schema is obtained, the converter loads its metadata (version) from the remote server.
First, it queries a local cache. If no result is found, it submits the data to the server, which replies with versioning information.
The converter always caches the results to avoid the overhead of querying the Schema Server for every new message that needs to be serialized.</p><div class="figure"><a name="d0e4109" href="#d0e4109"></a><p class="title"><b>Figure&nbsp;10.2.&nbsp;Schema Registration Process</b></p><div class="figure-contents"><div class="mediaobject" align="center"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-stream/master/docs/src/main/asciidoc/images/registration.png" align="middle" alt="registration"></div></div></div><br class="figure-break"><p>With the schema version information, the converter sets the <code class="literal">contentType</code> header of the message to carry the version information&#8201;&#8212;&#8201;for example: <code class="literal">application/vnd.user.v1+avro</code>.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="spring-cloud-stream-overview-schema-resolution-process" href="#spring-cloud-stream-overview-schema-resolution-process"></a>10.6.2&nbsp;Schema Resolution Process (Deserialization)</h3></div></div></div><p>When reading messages that contain version information (that is, a <code class="literal">contentType</code> header with a scheme like the one described under <span class="quote">&#8220;<span class="quote"><a class="xref" href="multi_schema-evolution.html#spring-cloud-stream-overview-schema-registration-process" title="10.6.1&nbsp;Schema Registration Process (Serialization)">Section&nbsp;10.6.1, &#8220;Schema Registration Process (Serialization)&#8221;</a></span>&#8221;</span>), the converter queries the Schema server to fetch the writer schema of the message.
Once it has found the correct schema of the incoming message, it retrieves the reader schema and, by using Avro&#8217;s schema resolution support, reads it into the reader definition (setting defaults and any missing properties).</p><div class="figure"><a name="d0e4137" href="#d0e4137"></a><p class="title"><b>Figure&nbsp;10.3.&nbsp;Schema Reading Resolution Process</b></p><div class="figure-contents"><div class="mediaobject" align="center"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-stream/master/docs/src/main/asciidoc/images/schema_reading.png" align="middle" alt="schema reading"></div></div></div><br class="figure-break"><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>You should understand the difference between a writer schema (the application that wrote the message) and a reader schema (the receiving application).
The converter always caches the results to avoid the overhead of querying the Schema Server for every new message that needs to be serialized.</p><div class="figure"><a name="d0e4100" href="#d0e4100"></a><p class="title"><b>Figure&nbsp;10.2.&nbsp;Schema Registration Process</b></p><div class="figure-contents"><div class="mediaobject" align="center"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-stream/master/docs/src/main/asciidoc/images/registration.png" align="middle" alt="registration"></div></div></div><br class="figure-break"><p>With the schema version information, the converter sets the <code class="literal">contentType</code> header of the message to carry the version information&#8201;&#8212;&#8201;for example: <code class="literal">application/vnd.user.v1+avro</code>.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="spring-cloud-stream-overview-schema-resolution-process" href="#spring-cloud-stream-overview-schema-resolution-process"></a>10.6.2&nbsp;Schema Resolution Process (Deserialization)</h3></div></div></div><p>When reading messages that contain version information (that is, a <code class="literal">contentType</code> header with a scheme like the one described under <span class="quote">&#8220;<span class="quote"><a class="xref" href="multi_schema-evolution.html#spring-cloud-stream-overview-schema-registration-process" title="10.6.1&nbsp;Schema Registration Process (Serialization)">Section&nbsp;10.6.1, &#8220;Schema Registration Process (Serialization)&#8221;</a></span>&#8221;</span>), the converter queries the Schema server to fetch the writer schema of the message.
Once it has found the correct schema of the incoming message, it retrieves the reader schema and, by using Avro&#8217;s schema resolution support, reads it into the reader definition (setting defaults and any missing properties).</p><div class="figure"><a name="d0e4128" href="#d0e4128"></a><p class="title"><b>Figure&nbsp;10.3.&nbsp;Schema Reading Resolution Process</b></p><div class="figure-contents"><div class="mediaobject" align="center"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-stream/master/docs/src/main/asciidoc/images/schema_reading.png" align="middle" alt="schema reading"></div></div></div><br class="figure-break"><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>You should understand the difference between a writer schema (the application that wrote the message) and a reader schema (the receiving application).
We suggest taking a moment to read <a class="link" href="https://avro.apache.org/docs/1.7.6/spec.html" target="_top">the Avro terminology</a> and understand the process.
Spring Cloud Stream always fetches the writer schema to determine how to read a message.
If you want to get Avro&#8217;s schema evolution support working, you need to make sure that a <code class="literal">readerSchema</code> was properly set for your application.</p></td></tr></table></div></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="multi_content-type-management.html">Prev</a>&nbsp;</td><td width="20%" align="center"><a accesskey="u" href="multi__reference_guide.html">Up</a></td><td width="40%" align="right">&nbsp;<a accesskey="n" href="multi__inter_application_communication.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">9.&nbsp;Content Type Negotiation&nbsp;</td><td width="20%" align="center"><a accesskey="h" href="multi_spring-cloud-stream.html">Home</a></td><td width="40%" align="right" valign="top">&nbsp;11.&nbsp;Inter-Application Communication</td></tr></table></div></body></html>
If you want to get Avro&#8217;s schema evolution support working, you need to make sure that a <code class="literal">readerSchema</code> was properly set for your application.</p></td></tr></table></div></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="multi_content-type-management.html">Prev</a>&nbsp;</td><td width="20%" align="center"><a accesskey="u" href="multi__preface.html">Up</a></td><td width="40%" align="right">&nbsp;<a accesskey="n" href="multi__inter_application_communication.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">9.&nbsp;Content Type Negotiation&nbsp;</td><td width="20%" align="center"><a accesskey="h" href="multi_spring-cloud-stream.html">Home</a></td><td width="40%" align="right" valign="top">&nbsp;11.&nbsp;Inter-Application Communication</td></tr></table></div></body></html>