SpringCloud/spring-cloud-static
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Files
08fbff534e5299308709b92e12f24a0fba25fe95
spring-cloud-static/spring-cloud-sleuth/2.1.1.RELEASE/spring-cloud-sleuth.xml
buildmaster 08fbff534e Sync docs from v2.1.1.RELEASE to gh-pages
2019-03-05 20:06:47 +00:00

2160 lines
126 KiB
XML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<?xml version="1.0" encoding="UTF-8"?>
<?asciidoc-toc maxdepth="8"?>
<?asciidoc-numbered?>
<book xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en">
<info>
<title>Spring Cloud Sleuth</title>
<date>2019-03-05</date>
<author>
<personname>
<firstname>Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer, Jay Bryant</firstname>
</personname>
</author>
<authorinitials>A</authorinitials>
</info>
<preface>
<title></title>
<simpara><emphasis role="strong">2.1.1.RELEASE</emphasis></simpara>
</preface>
<chapter xml:id="_introduction">
<title>Introduction</title>
<simpara>Spring Cloud Sleuth implements a distributed tracing solution for <link xl:href="http://cloud.spring.io">Spring Cloud</link>.</simpara>
<section xml:id="_terminology">
<title>Terminology</title>
<simpara>Spring Cloud Sleuth borrows <link xl:href="http://research.google.com/pubs/pub36356.html">Dapper&#8217;s</link> terminology.</simpara>
<simpara><emphasis role="strong">Span</emphasis>: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an RPC.
Spans are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of.
Spans also have other data, such as descriptions, timestamped events, key-value annotations (tags), the ID of the span that caused them, and process IDs (normally IP addresses).</simpara>
<simpara>Spans can be started and stopped, and they keep track of their timing information.
Once you create a span, you must stop it at some point in the future.</simpara>
<tip>
<simpara>The initial span that starts a trace is called a <literal>root span</literal>. The value of the ID
of that span is equal to the trace ID.</simpara>
</tip>
<simpara><emphasis role="strong">Trace:</emphasis> A set of spans forming a tree-like structure.
For example, if you run a distributed big-data store, a trace might be formed by a <literal>PUT</literal> request.</simpara>
<simpara><emphasis role="strong">Annotation:</emphasis> Used to record the existence of an event in time. With
<link xl:href="https://github.com/openzipkin/brave">Brave</link> instrumentation, we no longer need to set special events
for <link xl:href="https://zipkin.io/">Zipkin</link> to understand who the client and server are, where
the request started, and where it ended. For learning purposes,
however, we mark these events to highlight what kind
of an action took place.</simpara>
<itemizedlist>
<listitem>
<simpara><emphasis role="strong">cs</emphasis>: Client Sent. The client has made a request. This annotation indicates the start of the span.</simpara>
</listitem>
<listitem>
<simpara><emphasis role="strong">sr</emphasis>: Server Received: The server side got the request and started processing it.
Subtracting the <literal>cs</literal> timestamp from this timestamp reveals the network latency.</simpara>
</listitem>
<listitem>
<simpara><emphasis role="strong">ss</emphasis>: Server Sent. Annotated upon completion of request processing (when the response got sent back to the client).
Subtracting the <literal>sr</literal> timestamp from this timestamp reveals the time needed by the server side to process the request.</simpara>
</listitem>
<listitem>
<simpara><emphasis role="strong">cr</emphasis>: Client Received. Signifies the end of the span.
The client has successfully received the response from the server side.
Subtracting the <literal>cs</literal> timestamp from this timestamp reveals the whole time needed by the client to receive the response from the server.</simpara>
</listitem>
</itemizedlist>
<simpara>The following image shows how <emphasis role="strong">Span</emphasis> and <emphasis role="strong">Trace</emphasis> look in a system, together with the Zipkin annotations:</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/trace-id.png"/>
</imageobject>
<textobject><phrase>Trace Info propagation</phrase></textobject>
</mediaobject>
</informalfigure>
<simpara>Each color of a note signifies a span (there are seven spans - from <emphasis role="strong">A</emphasis> to <emphasis role="strong">G</emphasis>).
Consider the following note:</simpara>
<screen>Trace Id = X
Span Id = D
Client Sent</screen>
<simpara>This note indicates that the current span has <emphasis role="strong">Trace Id</emphasis> set to <emphasis role="strong">X</emphasis> and <emphasis role="strong">Span Id</emphasis> set to <emphasis role="strong">D</emphasis>.
Also, the <literal>Client Sent</literal> event took place.</simpara>
<simpara>The following image shows how parent-child relationships of spans look:</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/parents.png"/>
</imageobject>
<textobject><phrase>Parent child relationship</phrase></textobject>
</mediaobject>
</informalfigure>
</section>
<section xml:id="_purpose">
<title>Purpose</title>
<simpara>The following sections refer to the example shown in the preceding image.</simpara>
<section xml:id="_distributed_tracing_with_zipkin">
<title>Distributed Tracing with Zipkin</title>
<simpara>This example has seven spans.
If you go to traces in Zipkin, you can see this number in the second trace, as shown in the following image:</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/zipkin-traces.png"/>
</imageobject>
<textobject><phrase>Traces</phrase></textobject>
</mediaobject>
</informalfigure>
<simpara>However, if you pick a particular trace, you can see four spans, as shown in the following image:</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/zipkin-ui.png"/>
</imageobject>
<textobject><phrase>Traces Info propagation</phrase></textobject>
</mediaobject>
</informalfigure>
<note>
<simpara>When you pick a particular trace, you see merged spans.
That means that, if there were two spans sent to Zipkin with Server Received and Server Sent or Client Received and Client Sent annotations, they are presented as a single span.</simpara>
</note>
<simpara>Why is there a difference between the seven and four spans in this case?</simpara>
<itemizedlist>
<listitem>
<simpara>One span comes from the <literal>http:/start</literal> span. It has the Server Received (<literal>sr</literal>) and Server Sent (<literal>ss</literal>) annotations.</simpara>
</listitem>
<listitem>
<simpara>Two spans come from the RPC call from <literal>service1</literal> to <literal>service2</literal> to the <literal>http:/foo</literal> endpoint.
The Client Sent (<literal>cs</literal>) and Client Received (<literal>cr</literal>) events took place on the <literal>service1</literal> side.
Server Received (<literal>sr</literal>) and Server Sent (<literal>ss</literal>) events took place on the <literal>service2</literal> side.
These two spans form one logical span related to an RPC call.</simpara>
</listitem>
<listitem>
<simpara>Two spans come from the RPC call from <literal>service2</literal> to <literal>service3</literal> to the <literal>http:/bar</literal> endpoint.
The Client Sent (<literal>cs</literal>) and Client Received (<literal>cr</literal>) events took place on the <literal>service2</literal> side.
The Server Received (<literal>sr</literal>) and Server Sent (<literal>ss</literal>) events took place on the <literal>service3</literal> side.
These two spans form one logical span related to an RPC call.</simpara>
</listitem>
<listitem>
<simpara>Two spans come from the RPC call from <literal>service2</literal> to <literal>service4</literal> to the <literal>http:/baz</literal> endpoint.
The Client Sent (<literal>cs</literal>) and Client Received (<literal>cr</literal>) events took place on the <literal>service2</literal> side.
Server Received (<literal>sr</literal>) and Server Sent (<literal>ss</literal>) events took place on the <literal>service4</literal> side.
These two spans form one logical span related to an RPC call.</simpara>
</listitem>
</itemizedlist>
<simpara>So, if we count the physical spans, we have one from <literal>http:/start</literal>, two from <literal>service1</literal> calling <literal>service2</literal>, two from <literal>service2</literal>
calling <literal>service3</literal>, and two from <literal>service2</literal> calling <literal>service4</literal>. In sum, we have a total of seven spans.</simpara>
<simpara>Logically, we see the information of four total Spans because we have one span related to the incoming request
to <literal>service1</literal> and three spans related to RPC calls.</simpara>
</section>
<section xml:id="_visualizing_errors">
<title>Visualizing errors</title>
<simpara>Zipkin lets you visualize errors in your trace.
When an exception was thrown and was not caught, we set proper tags on the span, which Zipkin can then properly colorize.
You could see in the list of traces one trace that is red. That appears because an exception was thrown.</simpara>
<simpara>If you click that trace, you see a similar picture, as follows:</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/zipkin-error-traces.png"/>
</imageobject>
<textobject><phrase>Error Traces</phrase></textobject>
</mediaobject>
</informalfigure>
<simpara>If you then click on one of the spans, you see the following</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/zipkin-error-trace-screenshot.png"/>
</imageobject>
<textobject><phrase>Error Traces Info propagation</phrase></textobject>
</mediaobject>
</informalfigure>
<simpara>The span shows the reason for the error and the whole stack trace related to it.</simpara>
</section>
<section xml:id="_distributed_tracing_with_brave">
<title>Distributed Tracing with Brave</title>
<simpara>Starting with version <literal>2.0.0</literal>, Spring Cloud Sleuth uses <link xl:href="https://github.com/openzipkin/brave">Brave</link> as the tracing library.
Consequently, Sleuth no longer takes care of storing the context but delegates that work to Brave.</simpara>
<simpara>Due to the fact that Sleuth had different naming and tagging conventions than Brave, we decided to follow Brave&#8217;s conventions from now on.
However, if you want to use the legacy Sleuth approaches, you can set the <literal>spring.sleuth.http.legacy.enabled</literal> property to <literal>true</literal>.</simpara>
</section>
<section xml:id="_live_examples">
<title>Live examples</title>
<figure>
<title>Click the Pivotal Web Services icon to see it live!</title>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/pws.png" contentwidth="150" contentdepth="74"/>
</imageobject>
<textobject><phrase>Zipkin deployed on Pivotal Web Services</phrase></textobject>
</mediaobject>
</figure>
<simpara><link xl:href="http://docssleuth-zipkin-server.cfapps.io/">Click here to see it live!</link></simpara>
<simpara>The dependency graph in Zipkin should resemble the following image:</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/dependencies.png"/>
</imageobject>
<textobject><phrase>Dependencies</phrase></textobject>
</mediaobject>
</informalfigure>
<figure>
<title>Click the Pivotal Web Services icon to see it live!</title>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/pws.png" contentwidth="150" contentdepth="74"/>
</imageobject>
<textobject><phrase>Zipkin deployed on Pivotal Web Services</phrase></textobject>
</mediaobject>
</figure>
<simpara><link xl:href="http://docssleuth-zipkin-server.cfapps.io/dependency">Click here to see it live!</link></simpara>
</section>
<section xml:id="_log_correlation">
<title>Log correlation</title>
<simpara>When using grep to read the logs of those four applications by scanning for a trace ID equal to (for example) <literal>2485ec27856c56f4</literal>, you get output resembling the following:</simpara>
<screen>service1.log:2016-02-26 11:15:47.561 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Hello from service1. Calling service2
service2.log:2016-02-26 11:15:47.710 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Hello from service2. Calling service3 and then service4
service3.log:2016-02-26 11:15:47.895 INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application : Hello from service3
service2.log:2016-02-26 11:15:47.924 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service3 [Hello from service3]
service4.log:2016-02-26 11:15:48.134 INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application : Hello from service4
service2.log:2016-02-26 11:15:48.156 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service4 [Hello from service4]
service1.log:2016-02-26 11:15:48.182 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]</screen>
<simpara>If you use a log aggregating tool (such as <link xl:href="https://www.elastic.co/products/kibana">Kibana</link>, <link xl:href="http://www.splunk.com/">Splunk</link>, and others), you can order the events that took place.
An example from Kibana would resemble the following image:</simpara>
<informalfigure>
<mediaobject>
<imageobject>
<imagedata fileref="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/kibana.png"/>
</imageobject>
<textobject><phrase>Log correlation with Kibana</phrase></textobject>
</mediaobject>
</informalfigure>
<simpara>If you want to use <link xl:href="https://www.elastic.co/guide/en/logstash/current/index.html">Logstash</link>, the following listing shows the Grok pattern for Logstash:</simpara>
<screen>filter {
# pattern matching logback pattern
grok {
match =&gt; { "message" =&gt; "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
}
}</screen>
<note>
<simpara>If you want to use Grok together with the logs from Cloud Foundry, you have to use the following pattern:</simpara>
</note>
<screen>filter {
# pattern matching logback pattern
grok {
match =&gt; { "message" =&gt; "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
}
}</screen>
<section xml:id="_json_logback_with_logstash">
<title>JSON Logback with Logstash</title>
<simpara>Often, you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick.
To do so, you have to do the following (for readability, we pass the dependencies in the <literal>groupId:artifactId:version</literal> notation).</simpara>
<simpara><emphasis role="strong">Dependencies Setup</emphasis></simpara>
<orderedlist numeration="arabic">
<listitem>
<simpara>Ensure that Logback is on the classpath (<literal>ch.qos.logback:logback-core</literal>).</simpara>
</listitem>
<listitem>
<simpara>Add Logstash Logback encode. For example, to use version <literal>4.6</literal>, add <literal>net.logstash.logback:logstash-logback-encoder:4.6</literal>.</simpara>
</listitem>
</orderedlist>
<simpara><emphasis role="strong">Logback Setup</emphasis></simpara>
<simpara>Consider the following example of a Logback configuration file (named <link xl:href="https://github.com/spring-cloud-samples/sleuth-documentation-apps/blob/master/service1/src/main/resources/logback-spring.xml">logback-spring.xml</link>).</simpara>
<programlisting language="xml" linenumbering="unnumbered">&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;configuration&gt;
&lt;include resource="org/springframework/boot/logging/logback/defaults.xml"/&gt;
​
&lt;springProperty scope="context" name="springAppName" source="spring.application.name"/&gt;
&lt;!-- Example for logging into the build folder of your project --&gt;
&lt;property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${springAppName}"/&gt;​
&lt;!-- You can override this to have a custom pattern --&gt;
&lt;property name="CONSOLE_LOG_PATTERN"
value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/&gt;
&lt;!-- Appender to log to console --&gt;
&lt;appender name="console" class="ch.qos.logback.core.ConsoleAppender"&gt;
&lt;filter class="ch.qos.logback.classic.filter.ThresholdFilter"&gt;
&lt;!-- Minimum logging level to be presented in the console logs--&gt;
&lt;level&gt;DEBUG&lt;/level&gt;
&lt;/filter&gt;
&lt;encoder&gt;
&lt;pattern&gt;${CONSOLE_LOG_PATTERN}&lt;/pattern&gt;
&lt;charset&gt;utf8&lt;/charset&gt;
&lt;/encoder&gt;
&lt;/appender&gt;
&lt;!-- Appender to log to file --&gt;​
&lt;appender name="flatfile" class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;
&lt;file&gt;${LOG_FILE}&lt;/file&gt;
&lt;rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;
&lt;fileNamePattern&gt;${LOG_FILE}.%d{yyyy-MM-dd}.gz&lt;/fileNamePattern&gt;
&lt;maxHistory&gt;7&lt;/maxHistory&gt;
&lt;/rollingPolicy&gt;
&lt;encoder&gt;
&lt;pattern&gt;${CONSOLE_LOG_PATTERN}&lt;/pattern&gt;
&lt;charset&gt;utf8&lt;/charset&gt;
&lt;/encoder&gt;
&lt;/appender&gt;
​
&lt;!-- Appender to log to file in a JSON format --&gt;
&lt;appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;
&lt;file&gt;${LOG_FILE}.json&lt;/file&gt;
&lt;rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;
&lt;fileNamePattern&gt;${LOG_FILE}.json.%d{yyyy-MM-dd}.gz&lt;/fileNamePattern&gt;
&lt;maxHistory&gt;7&lt;/maxHistory&gt;
&lt;/rollingPolicy&gt;
&lt;encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"&gt;
&lt;providers&gt;
&lt;timestamp&gt;
&lt;timeZone&gt;UTC&lt;/timeZone&gt;
&lt;/timestamp&gt;
&lt;pattern&gt;
&lt;pattern&gt;
{
"severity": "%level",
"service": "${springAppName:-}",
"trace": "%X{X-B3-TraceId:-}",
"span": "%X{X-B3-SpanId:-}",
"parent": "%X{X-B3-ParentSpanId:-}",
"exportable": "%X{X-Span-Export:-}",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"rest": "%message"
}
&lt;/pattern&gt;
&lt;/pattern&gt;
&lt;/providers&gt;
&lt;/encoder&gt;
&lt;/appender&gt;
​
&lt;root level="INFO"&gt;
&lt;appender-ref ref="console"/&gt;
&lt;!-- uncomment this to have also JSON logs --&gt;
&lt;!--&lt;appender-ref ref="logstash"/&gt;--&gt;
&lt;!--&lt;appender-ref ref="flatfile"/&gt;--&gt;
&lt;/root&gt;
&lt;/configuration&gt;</programlisting>
<simpara>That Logback configuration file:</simpara>
<itemizedlist>
<listitem>
<simpara>Logs information from the application in a JSON format to a <literal>build/${spring.application.name}.json</literal> file.</simpara>
</listitem>
<listitem>
<simpara>Has commented out two additional appenders: console and standard log file.</simpara>
</listitem>
<listitem>
<simpara>Has the same logging pattern as the one presented in the previous section.</simpara>
</listitem>
</itemizedlist>
<note>
<simpara>If you use a custom <literal>logback-spring.xml</literal>, you must pass the <literal>spring.application.name</literal> in the <literal>bootstrap</literal> rather than the <literal>application</literal> property file.
Otherwise, your custom logback file does not properly read the property.</simpara>
</note>
</section>
</section>
<section xml:id="_propagating_span_context">
<title>Propagating Span Context</title>
<simpara>The span context is the state that must get propagated to any child spans across process boundaries.
Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context.
Baggage is an optional part.</simpara>
<simpara>Baggage is a set of key:value pairs stored in the span context.
Baggage travels together with the trace and is attached to every span.
Spring Cloud Sleuth understands that a header is baggage-related if the HTTP header is prefixed with <literal>baggage-</literal> and, for messaging, it starts with <literal>baggage_</literal>.</simpara>
<important>
<simpara>There is currently no limitation of the count or size of baggage items.
However, keep in mind that too many can decrease system throughput or increase RPC latency.
In extreme cases, too much baggage can crash the application, due to exceeding transport-level message or header capacity.</simpara>
</important>
<simpara>The following example shows setting baggage on a span:</simpara>
<programlisting language="java" linenumbering="unnumbered">Span initialSpan = this.tracer.nextSpan().name("span").start();
ExtraFieldPropagation.set(initialSpan.context(), "foo", "bar");
ExtraFieldPropagation.set(initialSpan.context(), "UPPER_CASE", "someValue");</programlisting>
<section xml:id="_baggage_versus_span_tags">
<title>Baggage versus Span Tags</title>
<simpara>Baggage travels with the trace (every child span contains the baggage of its parent).
Zipkin has no knowledge of baggage and does not receive that information.</simpara>
<important>
<simpara>Starting from Sleuth 2.0.0 you have to pass the baggage key names explicitly
in your project configuration. Read more about that setup <link linkend="prefixed-fields">here</link></simpara>
</important>
<simpara>Tags are attached to a specific span. In other words, they are presented only for that particular span.
However, you can search by tag to find the trace, assuming a span having the searched tag value exists.</simpara>
<simpara>If you want to be able to lookup a span based on baggage, you should add a corresponding entry as a tag in the root span.</simpara>
<important>
<simpara>The span must be in scope.</simpara>
</important>
<simpara>The following listing shows integration tests that use baggage:</simpara>
<formalpara>
<title>The setup</title>
<para>
<programlisting language="yml" linenumbering="unnumbered">spring.sleuth:
baggage-keys:
- baz
- bizarrecase
propagation-keys:
- foo
- upper_case</programlisting>
</para>
</formalpara>
<formalpara>
<title>The code</title>
<para>
<programlisting language="java" linenumbering="unnumbered">initialSpan.tag("foo",
ExtraFieldPropagation.get(initialSpan.context(), "foo"));
initialSpan.tag("UPPER_CASE",
ExtraFieldPropagation.get(initialSpan.context(), "UPPER_CASE"));</programlisting>
</para>
</formalpara>
</section>
</section>
</section>
<section xml:id="sleuth-adding-project">
<title>Adding Sleuth to the Project</title>
<simpara>This section addresses how to add Sleuth to your project with either Maven or Gradle.</simpara>
<important>
<simpara>To ensure that your application name is properly displayed in Zipkin, set the <literal>spring.application.name</literal> property in <literal>bootstrap.yml</literal>.</simpara>
</important>
<section xml:id="_only_sleuth_log_correlation">
<title>Only Sleuth (log correlation)</title>
<simpara>If you want to use only Spring Cloud Sleuth without the Zipkin integration, add the <literal>spring-cloud-starter-sleuth</literal> module to your project.</simpara>
<simpara>The following example shows how to add Sleuth with Maven:</simpara>
<formalpara role="primary">
<title>Maven</title>
<para>
<programlisting language="xml" linenumbering="unnumbered">&lt;dependencyManagement&gt; <co xml:id="CO1-1"/>
&lt;dependencies&gt;
&lt;dependency&gt;
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-dependencies&lt;/artifactId&gt;
&lt;version&gt;${release.train.version}&lt;/version&gt;
&lt;type&gt;pom&lt;/type&gt;
&lt;scope&gt;import&lt;/scope&gt;
&lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;/dependencyManagement&gt;
&lt;dependency&gt; <co xml:id="CO1-2"/>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-starter-sleuth&lt;/artifactId&gt;
&lt;/dependency&gt;</programlisting>
</para>
</formalpara>
<calloutlist>
<callout arearefs="CO1-1">
<para>We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.</para>
</callout>
<callout arearefs="CO1-2">
<para>Add the dependency to <literal>spring-cloud-starter-sleuth</literal>.</para>
</callout>
</calloutlist>
<simpara>The following example shows how to add Sleuth with Gradle:</simpara>
<formalpara role="secondary">
<title>Gradle</title>
<para>
<programlisting language="groovy" linenumbering="unnumbered">dependencyManagement { <co xml:id="CO2-1"/>
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
}
}
dependencies { <co xml:id="CO2-2"/>
compile "org.springframework.cloud:spring-cloud-starter-sleuth"
}</programlisting>
</para>
</formalpara>
<calloutlist>
<callout arearefs="CO2-1">
<para>We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.</para>
</callout>
<callout arearefs="CO2-2">
<para>Add the dependency to <literal>spring-cloud-starter-sleuth</literal>.</para>
</callout>
</calloutlist>
</section>
<section xml:id="_sleuth_with_zipkin_via_http">
<title>Sleuth with Zipkin via HTTP</title>
<simpara>If you want both Sleuth and Zipkin, add the <literal>spring-cloud-starter-zipkin</literal> dependency.</simpara>
<simpara>The following example shows how to do so for Maven:</simpara>
<formalpara role="primary">
<title>Maven</title>
<para>
<programlisting language="xml" linenumbering="unnumbered">&lt;dependencyManagement&gt; <co xml:id="CO3-1"/>
&lt;dependencies&gt;
&lt;dependency&gt;
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-dependencies&lt;/artifactId&gt;
&lt;version&gt;${release.train.version}&lt;/version&gt;
&lt;type&gt;pom&lt;/type&gt;
&lt;scope&gt;import&lt;/scope&gt;
&lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;/dependencyManagement&gt;
&lt;dependency&gt; <co xml:id="CO3-2"/>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-starter-zipkin&lt;/artifactId&gt;
&lt;/dependency&gt;</programlisting>
</para>
</formalpara>
<calloutlist>
<callout arearefs="CO3-1">
<para>We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.</para>
</callout>
<callout arearefs="CO3-2">
<para>Add the dependency to <literal>spring-cloud-starter-zipkin</literal>.</para>
</callout>
</calloutlist>
<simpara>The following example shows how to do so for Gradle:</simpara>
<formalpara role="secondary">
<title>Gradle</title>
<para>
<programlisting language="groovy" linenumbering="unnumbered">dependencyManagement { <co xml:id="CO4-1"/>
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
}
}
dependencies { <co xml:id="CO4-2"/>
compile "org.springframework.cloud:spring-cloud-starter-zipkin"
}</programlisting>
</para>
</formalpara>
<calloutlist>
<callout arearefs="CO4-1">
<para>We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.</para>
</callout>
<callout arearefs="CO4-2">
<para>Add the dependency to <literal>spring-cloud-starter-zipkin</literal>.</para>
</callout>
</calloutlist>
</section>
<section xml:id="_sleuth_with_zipkin_over_rabbitmq_or_kafka">
<title>Sleuth with Zipkin over RabbitMQ or Kafka</title>
<simpara>If you want to use RabbitMQ or Kafka instead of HTTP, add the <literal>spring-rabbit</literal> or <literal>spring-kafka</literal> dependency.
The default destination name is <literal>zipkin</literal>.</simpara>
<simpara>If using Kafka, you must set the property <literal>spring.zipkin.sender.type</literal> property accordingly:</simpara>
<programlisting language="yaml" linenumbering="unnumbered">spring.zipkin.sender.type: kafka</programlisting>
<caution>
<simpara><literal>spring-cloud-sleuth-stream</literal> is deprecated and incompatible with these destinations.</simpara>
</caution>
<simpara>If you want Sleuth over RabbitMQ, add the <literal>spring-cloud-starter-zipkin</literal> and <literal>spring-rabbit</literal>
dependencies.</simpara>
<simpara>The following example shows how to do so for Gradle:</simpara>
<formalpara role="primary">
<title>Maven</title>
<para>
<programlisting language="xml" linenumbering="unnumbered">&lt;dependencyManagement&gt; <co xml:id="CO5-1"/>
&lt;dependencies&gt;
&lt;dependency&gt;
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-dependencies&lt;/artifactId&gt;
&lt;version&gt;${release.train.version}&lt;/version&gt;
&lt;type&gt;pom&lt;/type&gt;
&lt;scope&gt;import&lt;/scope&gt;
&lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;/dependencyManagement&gt;
&lt;dependency&gt; <co xml:id="CO5-2"/>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-starter-zipkin&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;dependency&gt; <co xml:id="CO5-3"/>
&lt;groupId&gt;org.springframework.amqp&lt;/groupId&gt;
&lt;artifactId&gt;spring-rabbit&lt;/artifactId&gt;
&lt;/dependency&gt;</programlisting>
</para>
</formalpara>
<calloutlist>
<callout arearefs="CO5-1">
<para>We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.</para>
</callout>
<callout arearefs="CO5-2">
<para>Add the dependency to <literal>spring-cloud-starter-zipkin</literal>. That way, all nested dependencies get downloaded.</para>
</callout>
<callout arearefs="CO5-3">
<para>To automatically configure RabbitMQ, add the <literal>spring-rabbit</literal> dependency.</para>
</callout>
</calloutlist>
<formalpara role="secondary">
<title>Gradle</title>
<para>
<programlisting language="groovy" linenumbering="unnumbered">dependencyManagement { <co xml:id="CO6-1"/>
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
}
}
dependencies {
compile "org.springframework.cloud:spring-cloud-starter-zipkin" <co xml:id="CO6-2"/>
compile "org.springframework.amqp:spring-rabbit" <co xml:id="CO6-3"/>
}</programlisting>
</para>
</formalpara>
<calloutlist>
<callout arearefs="CO6-1">
<para>We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.</para>
</callout>
<callout arearefs="CO6-2">
<para>Add the dependency to <literal>spring-cloud-starter-zipkin</literal>. That way, all nested dependencies get downloaded.</para>
</callout>
<callout arearefs="CO6-3">
<para>To automatically configure RabbitMQ, add the <literal>spring-rabbit</literal> dependency.</para>
</callout>
</calloutlist>
</section>
</section>
<section xml:id="_overriding_the_auto_configuration_of_zipkin">
<title>Overriding the auto-configuration of Zipkin</title>
<simpara>Spring Cloud Sleuth supports sending traces to multiple tracing systems as of version 2.1.0.
In order to get this to work, every tracing system needs to have a <literal>Reporter&lt;Span&gt;</literal> and <literal>Sender</literal>.
If you want to override the provided beans you need to give them a specific name.
To do this you can use respectively <literal>ZipkinAutoConfiguration.REPORTER_BEAN_NAME</literal> and <literal>ZipkinAutoConfiguration.SENDER_BEAN_NAME</literal>.</simpara>
<programlisting language="java" linenumbering="unnumbered">@Configuration
protected static class MyConfig {
@Bean(ZipkinAutoConfiguration.REPORTER_BEAN_NAME)
Reporter&lt;zipkin2.Span&gt; myReporter() {
return AsyncReporter.create(mySender());
}
@Bean(ZipkinAutoConfiguration.SENDER_BEAN_NAME)
MySender mySender() {
return new MySender();
}
static class MySender extends Sender {
private boolean spanSent = false;
boolean isSpanSent() {
return this.spanSent;
}
@Override
public Encoding encoding() {
return Encoding.JSON;
}
@Override
public int messageMaxBytes() {
return Integer.MAX_VALUE;
}
@Override
public int messageSizeInBytes(List&lt;byte[]&gt; encodedSpans) {
return encoding().listSizeInBytes(encodedSpans);
}
@Override
public Call&lt;Void&gt; sendSpans(List&lt;byte[]&gt; encodedSpans) {
this.spanSent = true;
return Call.create(null);
}
}
}</programlisting>
</section>
</chapter>
<chapter xml:id="_additional_resources">
<title>Additional Resources</title>
<simpara>You can watch a video of <link xl:href="https://twitter.com/reshmi9k">Reshmi Krishna</link> and <link xl:href="https://twitter.com/mgrzejszczak">Marcin Grzejszczak</link> talking about Spring Cloud
Sleuth and Zipkin <link xl:href="https://content.pivotal.io/springone-platform-2017/distributed-tracing-latency-analysis-for-your-microservices-grzejszczak-krishna">by clicking here</link>.</simpara>
<simpara>You can check different setups of Sleuth and Brave <link xl:href="https://github.com/openzipkin/sleuth-webmvc-example">in the openzipkin/sleuth-webmvc-example repository</link>.</simpara>
</chapter>
<chapter xml:id="_features">
<title>Features</title>
<itemizedlist>
<listitem>
<simpara>Adds trace and span IDs to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator, as shown in the following example logs:</simpara>
<screen>2016-02-02 15:30:57.902 INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
2016-02-02 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
2016-02-02 15:31:01.936 INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...</screen>
<simpara>Notice the <literal>[appname,traceId,spanId,exportable]</literal> entries from the MDC:</simpara>
<itemizedlist>
<listitem>
<simpara><emphasis role="strong"><literal>spanId</literal></emphasis>: The ID of a specific operation that took place.</simpara>
</listitem>
<listitem>
<simpara><emphasis role="strong"><literal>appname</literal></emphasis>: The name of the application that logged the span.</simpara>
</listitem>
<listitem>
<simpara><emphasis role="strong"><literal>traceId</literal></emphasis>: The ID of the latency graph that contains the span.</simpara>
</listitem>
<listitem>
<simpara><emphasis role="strong"><literal>exportable</literal></emphasis>: Whether the log should be exported to Zipkin.
When would you like the span not to be exportable?
When you want to wrap some operation in a Span and have it written to the logs only.</simpara>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<simpara>Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, and key-value annotations.
Spring Cloud Sleuth is loosely based on HTrace but is compatible with Zipkin (Dapper).</simpara>
</listitem>
<listitem>
<simpara>Sleuth records timing information to aid in latency analysis.
By using sleuth, you can pinpoint causes of latency in your applications.</simpara>
</listitem>
<listitem>
<simpara>Sleuth is written to not log too much and to not cause your production application to crash.
To that end, Sleuth:</simpara>
<itemizedlist>
<listitem>
<simpara>Propagates structural data about your call graph in-band and the rest out-of-band.</simpara>
</listitem>
<listitem>
<simpara>Includes opinionated instrumentation of layers such as HTTP.</simpara>
</listitem>
<listitem>
<simpara>Includes a sampling policy to manage volume.</simpara>
</listitem>
<listitem>
<simpara>Can report to a Zipkin system for query and visualization.</simpara>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<simpara>Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, rest template, scheduled actions, message channels, Zuul filters, and Feign client).</simpara>
</listitem>
<listitem>
<simpara>Sleuth includes default logic to join a trace across HTTP or messaging boundaries.
For example, HTTP propagation works over Zipkin-compatible request headers.</simpara>
</listitem>
<listitem>
<simpara>Sleuth can propagate context (also known as baggage) between processes.
Consequently, if you set a baggage element on a Span, it is sent downstream to other processes over either HTTP or messaging.</simpara>
</listitem>
<listitem>
<simpara>Provides a way to create or continue spans and add tags and logs through annotations.</simpara>
</listitem>
<listitem>
<simpara>If <literal>spring-cloud-sleuth-zipkin</literal> is on the classpath, the app generates and collects Zipkin-compatible traces.
By default, it sends them over HTTP to a Zipkin server on localhost (port 9411).
You can configure the location of the service by setting <literal>spring.zipkin.baseUrl</literal>.</simpara>
<itemizedlist>
<listitem>
<simpara>If you depend on <literal>spring-rabbit</literal>, your app sends traces to a RabbitMQ broker instead of HTTP.</simpara>
</listitem>
<listitem>
<simpara>If you depend on <literal>spring-kafka</literal>, and set <literal>spring.zipkin.sender.type: kafka</literal>, your app sends traces to a Kafka broker instead of HTTP.</simpara>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
<caution>
<simpara><literal>spring-cloud-sleuth-stream</literal> is deprecated and should no longer be used.</simpara>
</caution>
<itemizedlist>
<listitem>
<simpara>Spring Cloud Sleuth is <link xl:href="http://opentracing.io/">OpenTracing</link> compatible.</simpara>
</listitem>
</itemizedlist>
<important>
<simpara>If you use Zipkin, configure the probability of spans exported by setting <literal>spring.sleuth.sampler.probability</literal>
(default: 0.1, which is 10 percent). Otherwise, you might think that Sleuth is not working be cause it omits some spans.</simpara>
</important>
<note>
<simpara>The SLF4J MDC is always set and logback users immediately see the trace and span IDs in logs per the example
shown earlier.
Other logging systems have to configure their own formatter to get the same result.
The default is as follows:
<literal>logging.pattern.level</literal> set to <literal>%5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]</literal>
(this is a Spring Boot feature for logback users).
If you do not use SLF4J, this pattern is NOT automatically applied.</simpara>
</note>
<section xml:id="_introduction_to_brave">
<title>Introduction to Brave</title>
<important>
<simpara>Starting with version <literal>2.0.0</literal>, Spring Cloud Sleuth uses
<link xl:href="https://github.com/openzipkin/brave">Brave</link> as the tracing library.
For your convenience, we embed part of the Brave&#8217;s docs here.</simpara>
</important>
<important>
<simpara>In the vast majority of cases you need to just use the <literal>Tracer</literal>
or <literal>SpanCustomizer</literal> beans from Brave that Sleuth provides. The documentation below contains
a high overview of what Brave is and how it works.</simpara>
</important>
<simpara>Brave is a library used to capture and report latency information about distributed operations to Zipkin.
Most users do not use Brave directly. They use libraries or frameworks rather than employ Brave on their behalf.</simpara>
<simpara>This module includes a tracer that creates and joins spans that model the latency of potentially distributed work.
It also includes libraries to propagate the trace context over network boundaries (for example, with HTTP headers).</simpara>
<section xml:id="_tracing">
<title>Tracing</title>
<simpara>Most importantly, you need a <literal>brave.Tracer</literal>, configured to <link xl:href="https://github.com/openzipkin/zipkin-reporter-java">report to Zipkin</link>.</simpara>
<simpara>The following example setup sends trace data (spans) to Zipkin over HTTP (as opposed to Kafka):</simpara>
<programlisting language="java" linenumbering="unnumbered">class MyClass {
private final Tracer tracer;
// Tracer will be autowired
MyClass(Tracer tracer) {
this.tracer = tracer;
}
void doSth() {
Span span = tracer.newTrace().name("encode").start();
// ...
}
}</programlisting>
<important>
<simpara>If your span contains a name longer than 50 chars, then that name is truncated to 50 chars.
Your names have to be explicit and concrete.
Big names lead to latency issues and sometimes even thrown exceptions.</simpara>
</important>
<simpara>The tracer creates and joins spans that model the latency of potentially distributed work.
It can employ sampling to reduce overhead during the process, to reduce the amount of data sent to Zipkin, or both.</simpara>
<simpara>Spans returned by a tracer report data to Zipkin when finished or do nothing if unsampled.
After starting a span, you can annotate events of interest or add tags containing details or lookup keys.</simpara>
<simpara>Spans have a context that includes trace identifiers that place the span at the correct spot in the tree representing the distributed operation.</simpara>
</section>
<section xml:id="_local_tracing">
<title>Local Tracing</title>
<simpara>When tracing code that never leaves your process, run it inside a scoped span.</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracer tracer;
// Start a new trace or a span within an existing trace representing an operation
ScopedSpan span = tracer.startScopedSpan("encode");
try {
// The span is in "scope" meaning downstream code such as loggers can see trace IDs
return encoder.encode();
} catch (RuntimeException | Error e) {
span.error(e); // Unless you handle exceptions, you might not know the operation failed!
throw e;
} finally {
span.finish(); // always finish the span
}</programlisting>
<simpara>When you need more features, or finer control, use the <literal>Span</literal> type:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracer tracer;
// Start a new trace or a span within an existing trace representing an operation
Span span = tracer.nextSpan().name("encode").start();
// Put the span in "scope" so that downstream code such as loggers can see trace IDs
try (SpanInScope ws = tracer.withSpanInScope(span)) {
return encoder.encode();
} catch (RuntimeException | Error e) {
span.error(e); // Unless you handle exceptions, you might not know the operation failed!
throw e;
} finally {
span.finish(); // note the scope is independent of the span. Always finish a span.
}</programlisting>
<simpara>Both of the above examples report the exact same span on finish!</simpara>
<simpara>In the above example, the span will be either a new root span or the
next child in an existing trace.</simpara>
</section>
<section xml:id="_customizing_spans">
<title>Customizing Spans</title>
<simpara>Once you have a span, you can add tags to it.
The tags can be used as lookup keys or details.
For example, you might add a tag with your runtime version, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">span.tag("clnt/finagle.version", "6.36.0");</programlisting>
<simpara>When exposing the ability to customize spans to third parties, prefer <literal>brave.SpanCustomizer</literal> as opposed to <literal>brave.Span</literal>.
The former is simpler to understand and test and does not tempt users with span lifecycle hooks.</simpara>
<programlisting language="java" linenumbering="unnumbered">interface MyTraceCallback {
void request(Request request, SpanCustomizer customizer);
}</programlisting>
<simpara>Since <literal>brave.Span</literal> implements <literal>brave.SpanCustomizer</literal>, you can pass it to users, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">for (MyTraceCallback callback : userCallbacks) {
callback.request(request, span);
}</programlisting>
</section>
<section xml:id="_implicitly_looking_up_the_current_span">
<title>Implicitly Looking up the Current Span</title>
<simpara>Sometimes, you do not know if a trace is in progress or not, and you do not want users to do null checks.
<literal>brave.CurrentSpanCustomizer</literal> handles this problem by adding data to any span that&#8217;s in progress or drops, as shown in the following example:</simpara>
<simpara>Ex.</simpara>
<programlisting language="java" linenumbering="unnumbered">// The user code can then inject this without a chance of it being null.
@Autowired SpanCustomizer span;
void userCode() {
span.annotate("tx.started");
...
}</programlisting>
</section>
<section xml:id="_rpc_tracing">
<title>RPC tracing</title>
<tip>
<simpara>Check for <link xl:href="https://github.com/openzipkin/brave/tree/master/instrumentation">instrumentation written here</link> and <link xl:href="http://zipkin.io/pages/existing_instrumentations.html">Zipkin&#8217;s list</link> before rolling your own RPC instrumentation.</simpara>
</tip>
<simpara>RPC tracing is often done automatically by interceptors. Behind the scenes, they add tags and events that relate to their role in an RPC operation.</simpara>
<simpara>The following example shows how to add a client span:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracing tracing;
@Autowired Tracer tracer;
// before you send a request, add metadata that describes the operation
span = tracer.nextSpan().name(service + "/" + method).kind(CLIENT);
span.tag("myrpc.version", "1.0.0");
span.remoteServiceName("backend");
span.remoteIpAndPort("172.3.4.1", 8108);
// Add the trace context to the request, so it can be propagated in-band
tracing.propagation().injector(Request::addHeader)
.inject(span.context(), request);
// when the request is scheduled, start the span
span.start();
// if there is an error, tag the span
span.tag("error", error.getCode());
// or if there is an exception
span.error(exception);
// when the response is complete, finish the span
span.finish();</programlisting>
<section xml:id="_one_way_tracing">
<title>One-Way tracing</title>
<simpara>Sometimes, you need to model an asynchronous operation where there is a
request but no response. In normal RPC tracing, you use <literal>span.finish()</literal>
to indicate that the response was received. In one-way tracing, you use
<literal>span.flush()</literal> instead, as you do not expect a response.</simpara>
<simpara>The following example shows how a client might model a one-way operation:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracing tracing;
@Autowired Tracer tracer;
// start a new span representing a client request
oneWaySend = tracer.nextSpan().name(service + "/" + method).kind(CLIENT);
// Add the trace context to the request, so it can be propagated in-band
tracing.propagation().injector(Request::addHeader)
.inject(oneWaySend.context(), request);
// fire off the request asynchronously, totally dropping any response
request.execute();
// start the client side and flush instead of finish
oneWaySend.start().flush();</programlisting>
<simpara>The following example shows how a server might handle a one-way operation:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracing tracing;
@Autowired Tracer tracer;
// pull the context out of the incoming request
extractor = tracing.propagation().extractor(Request::getHeader);
// convert that context to a span which you can name and add tags to
oneWayReceive = nextSpan(tracer, extractor.extract(request))
.name("process-request")
.kind(SERVER)
... add tags etc.
// start the server side and flush instead of finish
oneWayReceive.start().flush();
// you should not modify this span anymore as it is complete. However,
// you can create children to represent follow-up work.
next = tracer.newSpan(oneWayReceive.context()).name("step2").start();</programlisting>
</section>
</section>
</section>
</chapter>
<chapter xml:id="_sampling">
<title>Sampling</title>
<simpara>Sampling may be employed to reduce the data collected and reported out of process.
When a span is not sampled, it adds no overhead (a noop).</simpara>
<simpara>Sampling is an up-front decision, meaning that the decision to report data is made at the first operation in a trace and that decision is propagated downstream.</simpara>
<simpara>By default, a global sampler applies a single rate to all traced operations.
<literal>Tracer.Builder.sampler</literal> controls this setting, and it defaults to tracing every request.</simpara>
<section xml:id="_declarative_sampling">
<title>Declarative sampling</title>
<simpara>Some applications need to sample based on the type or annotations of a java method.</simpara>
<simpara>Most users use a framework interceptor to automate this sort of policy.
The following example shows how that might work internally:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracer tracer;
// derives a sample rate from an annotation on a java method
DeclarativeSampler&lt;Traced&gt; sampler = DeclarativeSampler.create(Traced::sampleRate);
@Around("@annotation(traced)")
public Object traceThing(ProceedingJoinPoint pjp, Traced traced) throws Throwable {
// When there is no trace in progress, this decides using an annotation
Sampler decideUsingAnnotation = declarativeSampler.toSampler(traced);
Tracer tracer = tracer.withSampler(decideUsingAnnotation);
// This code looks the same as if there was no declarative override
ScopedSpan span = tracer.startScopedSpan(spanName(pjp));
try {
return pjp.proceed();
} catch (RuntimeException | Error e) {
span.error(e);
throw e;
} finally {
span.finish();
}
}</programlisting>
</section>
<section xml:id="_custom_sampling">
<title>Custom sampling</title>
<simpara>Depending on what the operation is, you may want to apply different policies.
For example, you might not want to trace requests to static resources such as images, or you might want to trace all requests to a new api.</simpara>
<simpara>Most users use a framework interceptor to automate this sort of policy.
The following example shows how that might work internally:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracer tracer;
@Autowired Sampler fallback;
Span nextSpan(final Request input) {
Sampler requestBased = Sampler() {
@Override public boolean isSampled(long traceId) {
if (input.url().startsWith("/experimental")) {
return true;
} else if (input.url().startsWith("/static")) {
return false;
}
return fallback.isSampled(traceId);
}
};
return tracer.withSampler(requestBased).nextSpan();
}</programlisting>
</section>
<section xml:id="_sampling_in_spring_cloud_sleuth">
<title>Sampling in Spring Cloud Sleuth</title>
<simpara>By default Spring Cloud Sleuth sets all spans to non-exportable.
That means that traces appear in logs but not in any remote store.
For testing the default is often enough, and it probably is all you need if you use only the logs (for example, with an ELK aggregator).
If you export span data to Zipkin, there is also an <literal>Sampler.ALWAYS_SAMPLE</literal> setting that exports everything and a <literal>ProbabilityBasedSampler</literal> setting that samples a fixed fraction of spans.</simpara>
<note>
<simpara>The <literal>ProbabilityBasedSampler</literal> is the default if you use <literal>spring-cloud-sleuth-zipkin</literal>.
You can configure the exports by setting <literal>spring.sleuth.sampler.probability</literal>.
The passed value needs to be a double from <literal>0.0</literal> to <literal>1.0</literal>.</simpara>
</note>
<simpara>A sampler can be installed by creating a bean definition, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Bean
public Sampler defaultSampler() {
return Sampler.ALWAYS_SAMPLE;
}</programlisting>
<tip>
<simpara>You can set the HTTP header <literal>X-B3-Flags</literal> to <literal>1</literal>, or, when doing messaging, you can set the <literal>spanFlags</literal> header to <literal>1</literal>.
Doing so forces the current span to be exportable regardless of the sampling decision.</simpara>
</tip>
<simpara>In order to use the rate-limited sampler set the <literal>spring.sleuth.sampler.rate</literal> property to choose an amount of traces to accept on a per-second interval. The minimum number is 0 and the max is 2,147,483,647 (max int).</simpara>
</section>
</chapter>
<chapter xml:id="_propagation">
<title>Propagation</title>
<simpara>Propagation is needed to ensure activities originating from the same root are collected together in the same trace.
The most common propagation approach is to copy a trace context from a client by sending an RPC request to a server receiving it.</simpara>
<simpara>For example, when a downstream HTTP call is made, its trace context is encoded as request headers and sent along with it, as shown in the following image:</simpara>
<screen> Client Span Server Span
┌──────────────────┐ ┌──────────────────┐
│ │ │ │
│ TraceContext │ Http Request Headers │ TraceContext │
│ ┌──────────────┐ │ ┌───────────────────┐ │ ┌──────────────┐ │
│ │ TraceId │ │ │ X─B3─TraceId │ │ │ TraceId │ │
│ │ │ │ │ │ │ │ │ │
│ │ ParentSpanId │ │ Extract │ X─B3─ParentSpanId │ Inject │ │ ParentSpanId │ │
│ │ ├─┼─────────&gt;│ ├────────┼&gt;│ │ │
│ │ SpanId │ │ │ X─B3─SpanId │ │ │ SpanId │ │
│ │ │ │ │ │ │ │ │ │
│ │ Sampled │ │ │ X─B3─Sampled │ │ │ Sampled │ │
│ └──────────────┘ │ └───────────────────┘ │ └──────────────┘ │
│ │ │ │
└──────────────────┘ └──────────────────┘</screen>
<simpara>The names above are from <link xl:href="https://github.com/openzipkin/b3-propagation">B3 Propagation</link>, which is built-in to Brave and has implementations in many languages and frameworks.</simpara>
<simpara>Most users use a framework interceptor to automate propagation.
The next two examples show how that might work for a client and a server.</simpara>
<simpara>The following example shows how client-side propagation might work:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracing tracing;
// configure a function that injects a trace context into a request
injector = tracing.propagation().injector(Request.Builder::addHeader);
// before a request is sent, add the current span's context to it
injector.inject(span.context(), request);</programlisting>
<simpara>The following example shows how server-side propagation might work:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracing tracing;
@Autowired Tracer tracer;
// configure a function that extracts the trace context from a request
extractor = tracing.propagation().extractor(Request::getHeader);
// when a server receives a request, it joins or starts a new trace
span = tracer.nextSpan(extractor.extract(request));</programlisting>
<section xml:id="_propagating_extra_fields">
<title>Propagating extra fields</title>
<simpara>Sometimes you need to propagate extra fields, such as a request ID or an alternate trace context.
For example, if you are in a Cloud Foundry environment, you might want to pass the request ID, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">// when you initialize the builder, define the extra field you want to propagate
Tracing.newBuilder().propagationFactory(
ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-vcap-request-id")
);
// later, you can tag that request ID or use it in log correlation
requestId = ExtraFieldPropagation.get("x-vcap-request-id");</programlisting>
<simpara>You may also need to propagate a trace context that you are not using.
For example, you may be in an Amazon Web Services environment but not be reporting data to X-Ray.
To ensure X-Ray can co-exist correctly, pass-through its tracing header, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">tracingBuilder.propagationFactory(
ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-amzn-trace-id")
);</programlisting>
<tip>
<simpara>In Spring Cloud Sleuth all elements of the tracing builder <literal>Tracing.newBuilder()</literal>
are defined as beans. So if you want to pass a custom <literal>PropagationFactory</literal>, it&#8217;s enough
for you to create a bean of that type and we will set it in the <literal>Tracing</literal> bean.</simpara>
</tip>
<section xml:id="prefixed-fields">
<title>Prefixed fields</title>
<simpara>If they follow a common pattern, you can also prefix fields.
The following example shows how to propagate <literal>x-vcap-request-id</literal> the field as-is but send the <literal>country-code</literal> and <literal>user-id</literal> fields on the wire as <literal>x-baggage-country-code</literal> and <literal>x-baggage-user-id</literal>, respectively:</simpara>
<programlisting language="java" linenumbering="unnumbered">Tracing.newBuilder().propagationFactory(
ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY)
.addField("x-vcap-request-id")
.addPrefixedFields("x-baggage-", Arrays.asList("country-code", "user-id"))
.build()
);</programlisting>
<simpara>Later, you can call the following code to affect the country code of the current trace context:</simpara>
<programlisting language="java" linenumbering="unnumbered">ExtraFieldPropagation.set("x-country-code", "FO");
String countryCode = ExtraFieldPropagation.get("x-country-code");</programlisting>
<simpara>Alternatively, if you have a reference to a trace context, you can use it explicitly, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">ExtraFieldPropagation.set(span.context(), "x-country-code", "FO");
String countryCode = ExtraFieldPropagation.get(span.context(), "x-country-code");</programlisting>
<important>
<simpara>A difference from previous versions of Sleuth is that, with Brave, you must pass the list of baggage keys.
There are two properties to achieve this.
With the <literal>spring.sleuth.baggage-keys</literal>, you set keys that get prefixed with <literal>baggage-</literal> for HTTP calls and <literal>baggage_</literal> for messaging.
You can also use the <literal>spring.sleuth.propagation-keys</literal> property to pass a list of prefixed keys that are whitelisted without any prefix.
Notice that there&#8217;s no <literal>x-</literal> in front of the header keys.</simpara>
</important>
<simpara>In order to automatically set the baggage values to Slf4j&#8217;s MDC, you have to set
the <literal>spring.sleuth.log.slf4j.whitelisted-mdc-keys</literal> property with a list of whitelisted
baggage and propagation keys. E.g. <literal>spring.sleuth.log.slf4j.whitelisted-mdc-keys=foo</literal> will set the value of the <literal>foo</literal> baggage into MDC.</simpara>
<important>
<simpara>Remember that adding entries to MDC can drastically decrease the performance of your application!</simpara>
</important>
<simpara>If you want to add the baggage entries as tags, to make it possible to search for spans via the baggage entries, you can set the value of
<literal>spring.sleuth.propagation.tag.whitelisted-keys</literal> with a list of whitelisted baggage keys. To disable the feature you have to pass the <literal>spring.sleuth.propagation.tag.enabled=false</literal> property.</simpara>
</section>
<section xml:id="_extracting_a_propagated_context">
<title>Extracting a Propagated Context</title>
<simpara>The <literal>TraceContext.Extractor&lt;C&gt;</literal> reads trace identifiers and sampling status from an incoming request or message.
The carrier is usually a request object or headers.</simpara>
<simpara>This utility is used in standard instrumentation (such as <literal>HttpServerHandler</literal>) but can also be used for custom RPC or messaging code.</simpara>
<simpara><literal>TraceContextOrSamplingFlags</literal> is usually used only with <literal>Tracer.nextSpan(extracted)</literal>, unless you are
sharing span IDs between a client and a server.</simpara>
</section>
<section xml:id="_sharing_span_ids_between_client_and_server">
<title>Sharing span IDs between Client and Server</title>
<simpara>A normal instrumentation pattern is to create a span representing the server side of an RPC.
<literal>Extractor.extract</literal> might return a complete trace context when applied to an incoming client request.
<literal>Tracer.joinSpan</literal> attempts to continue this trace, using the same span ID if supported or creating a child span
if not. When the span ID is shared, the reported data includes a flag saying so.</simpara>
<simpara>The following image shows an example of B3 propagation:</simpara>
<screen> ┌───────────────────┐ ┌───────────────────┐
Incoming Headers │ TraceContext │ │ TraceContext │
┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
│ X─B3-TraceId │─────────┼─┼&gt; TraceId │ │──────┼─┼&gt; TraceId │ │
│ │ │ │ │ │ │ │ │ │
│ X─B3-ParentSpanId │─────────┼─┼&gt; ParentSpanId │ │──────┼─┼&gt; ParentSpanId │ │
│ │ │ │ │ │ │ │ │ │
│ X─B3-SpanId │─────────┼─┼&gt; SpanId │ │──────┼─┼&gt; SpanId │ │
└───────────────────┘ │ │ │ │ │ │ │ │
│ │ │ │ │ │ Shared: true │ │
│ └───────────────┘ │ │ └───────────────┘ │
└───────────────────┘ └───────────────────┘</screen>
<simpara>Some propagation systems forward only the parent span ID, detected when <literal>Propagation.Factory.supportsJoin() == false</literal>.
In this case, a new span ID is always provisioned, and the incoming context determines the parent ID.</simpara>
<simpara>The following image shows an example of AWS propagation:</simpara>
<screen> ┌───────────────────┐ ┌───────────────────┐
x-amzn-trace-id │ TraceContext │ │ TraceContext │
┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
│ Root │─────────┼─┼&gt; TraceId │ │──────┼─┼&gt; TraceId │ │
│ │ │ │ │ │ │ │ │ │
│ Parent │─────────┼─┼&gt; SpanId │ │──────┼─┼&gt; ParentSpanId │ │
└───────────────────┘ │ └───────────────┘ │ │ │ │ │
└───────────────────┘ │ │ SpanId: New │ │
│ └───────────────┘ │
└───────────────────┘</screen>
<simpara>Note: Some span reporters do not support sharing span IDs.
For example, if you set <literal>Tracing.Builder.spanReporter(amazonXrayOrGoogleStackdrive)</literal>, you should disable join by setting <literal>Tracing.Builder.supportsJoin(false)</literal>.
Doing so forces a new child span on <literal>Tracer.joinSpan()</literal>.</simpara>
</section>
<section xml:id="_implementing_propagation">
<title>Implementing Propagation</title>
<simpara><literal>TraceContext.Extractor&lt;C&gt;</literal> is implemented by a <literal>Propagation.Factory</literal> plugin.
Internally, this code creates the union type, <literal>TraceContextOrSamplingFlags</literal>, with one of the following:
* <literal>TraceContext</literal> if trace and span IDs were present.
* <literal>TraceIdContext</literal> if a trace ID was present but span IDs were not present.
* <literal>SamplingFlags</literal> if no identifiers were present.</simpara>
<simpara>Some <literal>Propagation</literal> implementations carry extra data from the point of extraction (for example, reading incoming headers) to injection (for example, writing outgoing headers).
For example, it might carry a request ID.
When implementations have extra data, they handle it as follows:
* If a <literal>TraceContext</literal> were extracted, add the extra data as <literal>TraceContext.extra()</literal>.
* Otherwise, add it as <literal>TraceContextOrSamplingFlags.extra()</literal>, which <literal>Tracer.nextSpan</literal> handles.</simpara>
</section>
</section>
</chapter>
<chapter xml:id="_current_tracing_component">
<title>Current Tracing Component</title>
<simpara>Brave supports a "<literal>current tracing component</literal>" concept, which should only be used when you have no other way to get a reference.
This was made for JDBC connections, as they often initialize prior to the tracing component.</simpara>
<simpara>The most recent tracing component instantiated is available through <literal>Tracing.current()</literal>.
You can also use <literal>Tracing.currentTracer()</literal> to get only the tracer.
If you use either of these methods, do not cache the result.
Instead, look them up each time you need them.</simpara>
</chapter>
<chapter xml:id="_current_span">
<title>Current Span</title>
<simpara>Brave supports a "<literal>current span</literal>" concept which represents the in-flight operation.
You can use <literal>Tracer.currentSpan()</literal> to add custom tags to a span and <literal>Tracer.nextSpan()</literal> to create a child of whatever is in-flight.</simpara>
<important>
<simpara>In Sleuth, you can autowire the <literal>Tracer</literal> bean to retrieve the current span via
<literal>tracer.currentSpan()</literal> method. To retrieve the current context just call
<literal>tracer.currentSpan().context()</literal>. To get the current trace id as String
you can use the <literal>traceIdString()</literal> method like this: <literal>tracer.currentSpan().context().traceIdString()</literal>.</simpara>
</important>
<section xml:id="_setting_a_span_in_scope_manually">
<title>Setting a span in scope manually</title>
<simpara>When writing new instrumentation, it is important to place a span you created in scope as the current span.
Not only does doing so let users access it with <literal>Tracer.currentSpan()</literal>, but it also allows customizations such as SLF4J MDC to see the current trace IDs.</simpara>
<simpara><literal>Tracer.withSpanInScope(Span)</literal> facilitates this and is most conveniently employed by using the try-with-resources idiom.
Whenever external code might be invoked (such as proceeding an interceptor or otherwise), place the span in scope, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracer tracer;
try (SpanInScope ws = tracer.withSpanInScope(span)) {
return inboundRequest.invoke();
} finally { // note the scope is independent of the span
span.finish();
}</programlisting>
<simpara>In edge cases, you may need to clear the current span temporarily (for example, launching a task that should not be associated with the current request). To do tso, pass null to <literal>withSpanInScope</literal>, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired Tracer tracer;
try (SpanInScope cleared = tracer.withSpanInScope(null)) {
startBackgroundThread();
}</programlisting>
</section>
</chapter>
<chapter xml:id="_instrumentation">
<title>Instrumentation</title>
<simpara>Spring Cloud Sleuth automatically instruments all your Spring applications, so you should not have to do anything to activate it.
The instrumentation is added by using a variety of technologies according to the stack that is available. For example, for a servlet web application, we use a <literal>Filter</literal>, and, for Spring Integration, we use <literal>ChannelInterceptors</literal>.</simpara>
<simpara>You can customize the keys used in span tags.
To limit the volume of span data, an HTTP request is, by default, tagged only with a handful of metadata, such as the status code, the host, and the URL.
You can add request headers by configuring <literal>spring.sleuth.keys.http.headers</literal> (a list of header names).</simpara>
<note>
<simpara>Tags are collected and exported only if there is a <literal>Sampler</literal> that allows it. By default, there is no such <literal>Sampler</literal>, to ensure that there is no danger of accidentally collecting too much data without configuring something).</simpara>
</note>
</chapter>
<chapter xml:id="_span_lifecycle">
<title>Span lifecycle</title>
<simpara>You can do the following operations on the Span by means of <literal>brave.Tracer</literal>:</simpara>
<itemizedlist>
<listitem>
<simpara><link linkend="creating-and-finishing-spans">start</link>: When you start a span, its name is assigned and the start timestamp is recorded.</simpara>
</listitem>
<listitem>
<simpara><link linkend="creating-and-finishing-spans">close</link>: The span gets finished (the end time of the span is recorded) and, if the span is sampled, it is eligible for collection (for example, to Zipkin).</simpara>
</listitem>
<listitem>
<simpara><link linkend="continuing-spans">continue</link>: A new instance of span is created.
It is a copy of the one that it continues.</simpara>
</listitem>
<listitem>
<simpara><link linkend="continuing-spans">detach</link>: The span does not get stopped or closed.
It only gets removed from the current thread.</simpara>
</listitem>
<listitem>
<simpara><link linkend="creating-spans-with-explicit-parent">create with explicit parent</link>: You can create a new span and set an explicit parent for it.</simpara>
</listitem>
</itemizedlist>
<tip>
<simpara>Spring Cloud Sleuth creates an instance of <literal>Tracer</literal> for you. In order to use it, you can autowire it.</simpara>
</tip>
<section xml:id="creating-and-finishing-spans">
<title>Creating and finishing spans</title>
<simpara>You can manually create spans by using the <literal>Tracer</literal>, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">// Start a span. If there was a span present in this thread it will become
// the `newSpan`'s parent.
Span newSpan = this.tracer.nextSpan().name("calculateTax");
try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(newSpan.start())) {
// ...
// You can tag a span
newSpan.tag("taxValue", taxValue);
// ...
// You can log an event on a span
newSpan.annotate("taxCalculated");
}
finally {
// Once done remember to finish the span. This will allow collecting
// the span to send it to Zipkin
newSpan.finish();
}</programlisting>
<simpara>In the preceding example, we could see how to create a new instance of the span.
If there is already a span in this thread, it becomes the parent of the new span.</simpara>
<important>
<simpara>Always clean after you create a span. Also, always finish any span that you want to send to Zipkin.</simpara>
</important>
<important>
<simpara>If your span contains a name greater than 50 chars, that name is truncated to 50 chars.
Your names have to be explicit and concrete. Big names lead to latency issues and sometimes even exceptions.</simpara>
</important>
</section>
<section xml:id="continuing-spans">
<title>Continuing Spans</title>
<simpara>Sometimes, you do not want to create a new span but you want to continue one. An example of such a
situation might be as follows:</simpara>
<itemizedlist>
<listitem>
<simpara><emphasis role="strong">AOP</emphasis>: If there was already a span created before an aspect was reached, you might not want to create a new span.</simpara>
</listitem>
<listitem>
<simpara><emphasis role="strong">Hystrix</emphasis>: Executing a Hystrix command is most likely a logical part of the current processing.
It is in fact merely a technical implementation detail that you would not necessarily want to reflect in tracing as a separate being.</simpara>
</listitem>
</itemizedlist>
<simpara>To continue a span, you can use <literal>brave.Tracer</literal>, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">// let's assume that we're in a thread Y and we've received
// the `initialSpan` from thread X
Span continuedSpan = this.tracer.toSpan(newSpan.context());
try {
// ...
// You can tag a span
continuedSpan.tag("taxValue", taxValue);
// ...
// You can log an event on a span
continuedSpan.annotate("taxCalculated");
}
finally {
// Once done remember to flush the span. That means that
// it will get reported but the span itself is not yet finished
continuedSpan.flush();
}</programlisting>
</section>
<section xml:id="creating-spans-with-explicit-parent">
<title>Creating a Span with an explicit Parent</title>
<simpara>You might want to start a new span and provide an explicit parent of that span.
Assume that the parent of a span is in one thread and you want to start a new span in another thread.
In Brave, whenever you call <literal>nextSpan()</literal>, it creates a span in reference to the span that is currently in scope.
You can put the span in scope and then call <literal>nextSpan()</literal>, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">// let's assume that we're in a thread Y and we've received
// the `initialSpan` from thread X. `initialSpan` will be the parent
// of the `newSpan`
Span newSpan = null;
try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) {
newSpan = this.tracer.nextSpan().name("calculateCommission");
// ...
// You can tag a span
newSpan.tag("commissionValue", commissionValue);
// ...
// You can log an event on a span
newSpan.annotate("commissionCalculated");
}
finally {
// Once done remember to finish the span. This will allow collecting
// the span to send it to Zipkin. The tags and events set on the
// newSpan will not be present on the parent
if (newSpan != null) {
newSpan.finish();
}
}</programlisting>
<important>
<simpara>After creating such a span, you must finish it. Otherwise it is not reported (for example, to Zipkin).</simpara>
</important>
</section>
</chapter>
<chapter xml:id="_naming_spans">
<title>Naming spans</title>
<simpara>Picking a span name is not a trivial task. A span name should depict an operation name.
The name should be low cardinality, so it should not include identifiers.</simpara>
<simpara>Since there is a lot of instrumentation going on, some span names are artificial:</simpara>
<itemizedlist>
<listitem>
<simpara><literal>controller-method-name</literal> when received by a Controller with a method name of <literal>controllerMethodName</literal></simpara>
</listitem>
<listitem>
<simpara><literal>async</literal> for asynchronous operations done with wrapped <literal>Callable</literal> and <literal>Runnable</literal> interfaces.</simpara>
</listitem>
<listitem>
<simpara>Methods annotated with <literal>@Scheduled</literal> return the simple name of the class.</simpara>
</listitem>
</itemizedlist>
<simpara>Fortunately, for asynchronous processing, you can provide explicit naming.</simpara>
<section xml:id="_spanname_annotation">
<title><literal>@SpanName</literal> Annotation</title>
<simpara>You can name the span explicitly by using the <literal>@SpanName</literal> annotation, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered"> @SpanName("calculateTax")
class TaxCountingRunnable implements Runnable {
@Override
public void run() {
// perform logic
}
}
}</programlisting>
<simpara>In this case, when processed in the following manner, the span is named <literal>calculateTax</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">Runnable runnable = new TraceRunnable(this.tracing, spanNamer,
new TaxCountingRunnable());
Future&lt;?&gt; future = executorService.submit(runnable);
// ... some additional logic ...
future.get();</programlisting>
</section>
<section xml:id="_tostring_method">
<title><literal>toString()</literal> method</title>
<simpara>It is pretty rare to create separate classes for <literal>Runnable</literal> or <literal>Callable</literal>.
Typically, one creates an anonymous instance of those classes.
You cannot annotate such classes.
To overcome that limitation, if there is no <literal>@SpanName</literal> annotation present, we check whether the class has a custom implementation of the <literal>toString()</literal> method.</simpara>
<simpara>Running such code leads to creating a span named <literal>calculateTax</literal>, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">Runnable runnable = new TraceRunnable(this.tracing, spanNamer, new Runnable() {
@Override
public void run() {
// perform logic
}
@Override
public String toString() {
return "calculateTax";
}
});
Future&lt;?&gt; future = executorService.submit(runnable);
// ... some additional logic ...
future.get();</programlisting>
</section>
</chapter>
<chapter xml:id="_managing_spans_with_annotations">
<title>Managing Spans with Annotations</title>
<simpara>You can manage spans with a variety of annotations.</simpara>
<section xml:id="_rationale">
<title>Rationale</title>
<simpara>There are a number of good reasons to manage spans with annotations, including:</simpara>
<itemizedlist>
<listitem>
<simpara>API-agnostic means to collaborate with a span. Use of annotations lets users add to a span with no library dependency on a span api.
Doing so lets Sleuth change its core API to create less impact to user code.</simpara>
</listitem>
<listitem>
<simpara>Reduced surface area for basic span operations. Without this feature, you must use the span api, which has lifecycle commands that could be used incorrectly.
By only exposing scope, tag, and log functionality, you can collaborate without accidentally breaking span lifecycle.</simpara>
</listitem>
<listitem>
<simpara>Collaboration with runtime generated code. With libraries such as Spring Data and Feign, the implementations of interfaces are generated at runtime.
Consequently, span wrapping of objects was tedious.
Now you can provide annotations over interfaces and the arguments of those interfaces.</simpara>
</listitem>
</itemizedlist>
</section>
<section xml:id="_creating_new_spans">
<title>Creating New Spans</title>
<simpara>If you do not want to create local spans manually, you can use the <literal>@NewSpan</literal> annotation.
Also, we provide the <literal>@SpanTag</literal> annotation to add tags in an automated fashion.</simpara>
<simpara>Now we can consider some examples of usage.</simpara>
<programlisting language="java" linenumbering="unnumbered">@NewSpan
void testMethod();</programlisting>
<simpara>Annotating the method without any parameter leads to creating a new span whose name equals the annotated method name.</simpara>
<programlisting language="java" linenumbering="unnumbered">@NewSpan("customNameOnTestMethod4")
void testMethod4();</programlisting>
<simpara>If you provide the value in the annotation (either directly or by setting the <literal>name</literal> parameter), the created span has the provided value as the name.</simpara>
<programlisting language="java" linenumbering="unnumbered">// method declaration
@NewSpan(name = "customNameOnTestMethod5")
void testMethod5(@SpanTag("testTag") String param);
// and method execution
this.testBean.testMethod5("test");</programlisting>
<simpara>You can combine both the name and a tag. Let&#8217;s focus on the latter.
In this case, the value of the annotated method&#8217;s parameter runtime value becomes the value of the tag.
In our sample, the tag key is <literal>testTag</literal>, and the tag value is <literal>test</literal>.</simpara>
<programlisting language="java" linenumbering="unnumbered">@NewSpan(name = "customNameOnTestMethod3")
@Override
public void testMethod3() {
}</programlisting>
<simpara>You can place the <literal>@NewSpan</literal> annotation on both the class and an interface.
If you override the interface&#8217;s method and provide a different value for the <literal>@NewSpan</literal> annotation, the most
concrete one wins (in this case <literal>customNameOnTestMethod3</literal> is set).</simpara>
</section>
<section xml:id="_continuing_spans">
<title>Continuing Spans</title>
<simpara>If you want to add tags and annotations to an existing span, you can use the <literal>@ContinueSpan</literal> annotation, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">// method declaration
@ContinueSpan(log = "testMethod11")
void testMethod11(@SpanTag("testTag11") String param);
// method execution
this.testBean.testMethod11("test");
this.testBean.testMethod13();</programlisting>
<simpara>(Note that, in contrast with the <literal>@NewSpan</literal> annotation ,you can also add logs with the <literal>log</literal> parameter.)</simpara>
<simpara>That way, the span gets continued and:</simpara>
<itemizedlist>
<listitem>
<simpara>Log entries named <literal>testMethod11.before</literal> and <literal>testMethod11.after</literal> are created.</simpara>
</listitem>
<listitem>
<simpara>If an exception is thrown, a log entry named <literal>testMethod11.afterFailure</literal> is also created.</simpara>
</listitem>
<listitem>
<simpara>A tag with a key of <literal>testTag11</literal> and a value of <literal>test</literal> is created.</simpara>
</listitem>
</itemizedlist>
</section>
<section xml:id="_advanced_tag_setting">
<title>Advanced Tag Setting</title>
<simpara>There are 3 different ways to add tags to a span. All of them are controlled by the <literal>SpanTag</literal> annotation.
The precedence is as follows:</simpara>
<orderedlist numeration="arabic">
<listitem>
<simpara>Try with a bean of <literal>TagValueResolver</literal> type and a provided name.</simpara>
</listitem>
<listitem>
<simpara>If the bean name has not been provided, try to evaluate an expression.
We search for a <literal>TagValueExpressionResolver</literal> bean.
The default implementation uses SPEL expression resolution.
<emphasis role="strong">IMPORTANT</emphasis> You can only reference properties from the SPEL expression. Method execution is not allowed due to security constraints.</simpara>
</listitem>
<listitem>
<simpara>If we do not find any expression to evaluate, return the <literal>toString()</literal> value of the parameter.</simpara>
</listitem>
</orderedlist>
<section xml:id="_custom_extractor">
<title>Custom extractor</title>
<simpara>The value of the tag for the following method is computed by an implementation of <literal>TagValueResolver</literal> interface.
Its class name has to be passed as the value of the <literal>resolver</literal> attribute.</simpara>
<simpara>Consider the following annotated method:</simpara>
<programlisting language="java" linenumbering="unnumbered">@NewSpan
public void getAnnotationForTagValueResolver(
@SpanTag(key = "test", resolver = TagValueResolver.class) String test) {
}</programlisting>
<simpara>Now further consider the following <literal>TagValueResolver</literal> bean implementation:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Bean(name = "myCustomTagValueResolver")
public TagValueResolver tagValueResolver() {
return parameter -&gt; "Value from myCustomTagValueResolver";
}</programlisting>
<simpara>The two preceding examples lead to setting a tag value equal to <literal>Value from myCustomTagValueResolver</literal>.</simpara>
</section>
<section xml:id="_resolving_expressions_for_a_value">
<title>Resolving Expressions for a Value</title>
<simpara>Consider the following annotated method:</simpara>
<programlisting language="java" linenumbering="unnumbered">@NewSpan
public void getAnnotationForTagValueExpression(
@SpanTag(key = "test", expression = "'hello' + ' characters'") String test) {
}</programlisting>
<simpara>No custom implementation of a <literal>TagValueExpressionResolver</literal> leads to evaluation of the SPEL expression, and a tag with a value of <literal>4 characters</literal> is set on the span.
If you want to use some other expression resolution mechanism, you can create your own implementation of the bean.</simpara>
</section>
<section xml:id="_using_the_tostring_method">
<title>Using the <literal>toString()</literal> method</title>
<simpara>Consider the following annotated method:</simpara>
<programlisting language="java" linenumbering="unnumbered">@NewSpan
public void getAnnotationForArgumentToString(@SpanTag("test") Long param) {
}</programlisting>
<simpara>Running the preceding method with a value of <literal>15</literal> leads to setting a tag with a String value of <literal>"15"</literal>.</simpara>
</section>
</section>
</chapter>
<chapter xml:id="_customizations">
<title>Customizations</title>
<section xml:id="_http">
<title>HTTP</title>
<simpara>If a customization of client / server parsing of the HTTP related spans is required,
just register a bean of type <literal>brave.http.HttpClientParser</literal> or
<literal>brave.http.HttpServerParser</literal>. If client /server sampling is required, just
register a bean of type <literal>brave.http.HttpSampler</literal> and name the bean
<literal>sleuthClientSampler</literal> for client sampler and <literal>sleuthServerSampler</literal> for server sampler.
For your convenience the <literal>@ClientSampler</literal> and <literal>@ServerSampler</literal>
annotations can be used to inject the proper beans or to
reference the bean names via their static String <literal>NAME</literal> fields.</simpara>
<simpara>Check out Brave&#8217;s code to see an example of how to make a path-based sampler
<link xl:href="https://github.com/openzipkin/brave/tree/master/instrumentation/http#sampling-policy">https://github.com/openzipkin/brave/tree/master/instrumentation/http#sampling-policy</link></simpara>
<simpara>If you want to completely rewrite the <literal>HttpTracing</literal> bean you can use the <literal>SkipPatternProvider</literal>
interface to retrieve the URL <literal>Pattern</literal> for spans that should be not sampled. Below you can see
an example of usage of <literal>SkipPatternProvider</literal> inside a server side, <literal>HttpSampler</literal>.</simpara>
<programlisting language="java" linenumbering="unnumbered">@Configuration
class Config {
@Bean(name = ServerSampler.NAME)
HttpSampler myHttpSampler(SkipPatternProvider provider) {
Pattern pattern = provider.skipPattern();
return new HttpSampler() {
@Override
public &lt;Req&gt; Boolean trySample(HttpAdapter&lt;Req, ?&gt; adapter, Req request) {
String url = adapter.path(request);
boolean shouldSkip = pattern.matcher(url).matches();
if (shouldSkip) {
return false;
}
return null;
}
};
}
}</programlisting>
</section>
<section xml:id="_tracingfilter">
<title><literal>TracingFilter</literal></title>
<simpara>You can also modify the behavior of the <literal>TracingFilter</literal>, which is the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response.
You can customize the tags or modify the response headers by registering your own instance of the <literal>TracingFilter</literal> bean.</simpara>
<simpara>In the following example, we register the <literal>TracingFilter</literal> bean, add the <literal>ZIPKIN-TRACE-ID</literal> response header containing the current Span&#8217;s trace id, and add a tag with key <literal>custom</literal> and a value <literal>tag</literal> to the span.</simpara>
<programlisting language="java" linenumbering="unnumbered">@Component
@Order(TraceWebServletAutoConfiguration.TRACING_FILTER_ORDER + 1)
class MyFilter extends GenericFilterBean {
private final Tracer tracer;
MyFilter(Tracer tracer) {
this.tracer = tracer;
}
@Override
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException, ServletException {
Span currentSpan = this.tracer.currentSpan();
if (currentSpan == null) {
chain.doFilter(request, response);
return;
}
// for readability we're returning trace id in a hex form
((HttpServletResponse) response).addHeader("ZIPKIN-TRACE-ID",
currentSpan.context().traceIdString());
// we can also add some custom tags
currentSpan.tag("custom", "tag");
chain.doFilter(request, response);
}
}</programlisting>
</section>
<section xml:id="_custom_service_name">
<title>Custom service name</title>
<simpara>By default, Sleuth assumes that, when you send a span to Zipkin, you want the span&#8217;s service name to be equal to the value of the <literal>spring.application.name</literal> property.
That is not always the case, though.
There are situations in which you want to explicitly provide a different service name for all spans coming from your application.
To achieve that, you can pass the following property to your application to override that value (the example is for a service named <literal>myService</literal>):</simpara>
<programlisting language="yaml" linenumbering="unnumbered">spring.zipkin.service.name: myService</programlisting>
</section>
<section xml:id="_customization_of_reported_spans">
<title>Customization of Reported Spans</title>
<simpara>Before reporting spans (for example, to Zipkin) you may want to modify that span in some way.
You can do so by using the <literal>FinishedSpanHandler</literal> interface.</simpara>
<simpara>In Sleuth, we generate spans with a fixed name.
Some users want to modify the name depending on values of tags.
You can implement the <literal>FinishedSpanHandler</literal> interface to alter that name.</simpara>
<simpara>The following example shows how to register two beans that implement <literal>FinishedSpanHandler</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Bean
FinishedSpanHandler handlerOne() {
return new FinishedSpanHandler() {
@Override
public boolean handle(TraceContext traceContext, MutableSpan span) {
span.name("foo");
return true; // keep this span
}
};
}
@Bean
FinishedSpanHandler handlerTwo() {
return new FinishedSpanHandler() {
@Override
public boolean handle(TraceContext traceContext, MutableSpan span) {
span.name(span.name() + " bar");
return true; // keep this span
}
};
}</programlisting>
<simpara>The preceding example results in changing the name of the reported span to <literal>foo bar</literal>, just before it gets reported (for example, to Zipkin).</simpara>
</section>
<section xml:id="_host_locator">
<title>Host Locator</title>
<important>
<simpara>This section is about defining <emphasis role="strong">host</emphasis> from service discovery.
It is <emphasis role="strong">NOT</emphasis> about finding Zipkin through service discovery.</simpara>
</important>
<simpara>To define the host that corresponds to a particular span, we need to resolve the host name and port.
The default approach is to take these values from server properties.
If those are not set, we try to retrieve the host name from the network interfaces.</simpara>
<simpara>If you have the discovery client enabled and prefer to retrieve the host address from the registered instance in a service registry, you have to set the <literal>spring.zipkin.locator.discovery.enabled</literal> property (it is applicable for both HTTP-based and Stream-based span reporting), as follows:</simpara>
<programlisting language="yaml" linenumbering="unnumbered">spring.zipkin.locator.discovery.enabled: true</programlisting>
</section>
</chapter>
<chapter xml:id="_sending_spans_to_zipkin">
<title>Sending Spans to Zipkin</title>
<simpara>By default, if you add <literal>spring-cloud-starter-zipkin</literal> as a dependency to your project, when the span is closed, it is sent to Zipkin over HTTP.
The communication is asynchronous.
You can configure the URL by setting the <literal>spring.zipkin.baseUrl</literal> property, as follows:</simpara>
<programlisting language="yaml" linenumbering="unnumbered">spring.zipkin.baseUrl: http://192.168.99.100:9411/</programlisting>
<simpara>If you want to find Zipkin through service discovery, you can pass the Zipkin&#8217;s service ID inside the URL, as shown in the following example for <literal>zipkinserver</literal> service ID:</simpara>
<programlisting language="yaml" linenumbering="unnumbered">spring.zipkin.baseUrl: http://zipkinserver/</programlisting>
<simpara>To disable this feature just set <literal>spring.zipkin.discoveryClientEnabled</literal> to `false.</simpara>
<simpara>When the Discovery Client feature is enabled, Sleuth uses
<literal>LoadBalancerClient</literal> to find the URL of the Zipkin Server. It means
that you can set up the load balancing configuration e.g. via Ribbon.</simpara>
<programlisting language="yaml" linenumbering="unnumbered">zipkinserver:
ribbon:
ListOfServers: host1,host2</programlisting>
<simpara>If you have web, rabbit, or kafka together on the classpath, you might need to pick the means by which you would like to send spans to zipkin.
To do so, set <literal>web</literal>, <literal>rabbit</literal>, or <literal>kafka</literal> to the <literal>spring.zipkin.sender.type</literal> property.
The following example shows setting the sender type for <literal>web</literal>:</simpara>
<programlisting language="yaml" linenumbering="unnumbered">spring.zipkin.sender.type: web</programlisting>
<simpara>To customize the <literal>RestTemplate</literal> that sends spans to Zipkin via HTTP, you can register
the <literal>ZipkinRestTemplateCustomizer</literal> bean.</simpara>
<programlisting language="java" linenumbering="unnumbered">@Configuration
class MyConfig {
@Bean ZipkinRestTemplateCustomizer myCustomizer() {
return new ZipkinRestTemplateCustomizer() {
@Override
void customize(RestTemplate restTemplate) {
// customize the RestTemplate
}
};
}
}</programlisting>
<simpara>If, however, you would like to control the full process of creating the <literal>RestTemplate</literal>
object, you will have to create a bean of <literal>zipkin2.reporter.Sender</literal> type.</simpara>
<programlisting language="java" linenumbering="unnumbered"> @Bean Sender myRestTemplateSender(ZipkinProperties zipkin,
ZipkinRestTemplateCustomizer zipkinRestTemplateCustomizer) {
RestTemplate restTemplate = mySuperCustomRestTemplate();
zipkinRestTemplateCustomizer.customize(restTemplate);
return myCustomSender(zipkin, restTemplate);
}</programlisting>
</chapter>
<chapter xml:id="_zipkin_stream_span_consumer">
<title>Zipkin Stream Span Consumer</title>
<important>
<simpara>We recommend using Zipkin&#8217;s native support for message-based span sending.
Starting from the Edgware release, the Zipkin Stream server is deprecated.
In the Finchley release, it got removed.</simpara>
</important>
<simpara>If for some reason you need to create the deprecated Stream Zipkin server, see the <link xl:href="http://cloud.spring.io/spring-cloud-static/Dalston.SR4/multi/multi__span_data_as_messages.html#_zipkin_consumer">Dalston Documentation</link>.</simpara>
</chapter>
<chapter xml:id="_integrations">
<title>Integrations</title>
<section xml:id="_opentracing">
<title>OpenTracing</title>
<simpara>Spring Cloud Sleuth is compatible with <link xl:href="http://opentracing.io/">OpenTracing</link>.
If you have OpenTracing on the classpath, we automatically register the OpenTracing <literal>Tracer</literal> bean.
If you wish to disable this, set <literal>spring.sleuth.opentracing.enabled</literal> to <literal>false</literal></simpara>
</section>
<section xml:id="_runnable_and_callable">
<title>Runnable and Callable</title>
<simpara>If you wrap your logic in <literal>Runnable</literal> or <literal>Callable</literal>, you can wrap those classes in their Sleuth representative, as shown in the following example for <literal>Runnable</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">Runnable runnable = new Runnable() {
@Override
public void run() {
// do some work
}
@Override
public String toString() {
return "spanNameFromToStringMethod";
}
};
// Manual `TraceRunnable` creation with explicit "calculateTax" Span name
Runnable traceRunnable = new TraceRunnable(this.tracing, spanNamer, runnable,
"calculateTax");
// Wrapping `Runnable` with `Tracing`. That way the current span will be available
// in the thread of `Runnable`
Runnable traceRunnableFromTracer = this.tracing.currentTraceContext()
.wrap(runnable);</programlisting>
<simpara>The following example shows how to do so for <literal>Callable</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">Callable&lt;String&gt; callable = new Callable&lt;String&gt;() {
@Override
public String call() throws Exception {
return someLogic();
}
@Override
public String toString() {
return "spanNameFromToStringMethod";
}
};
// Manual `TraceCallable` creation with explicit "calculateTax" Span name
Callable&lt;String&gt; traceCallable = new TraceCallable&lt;&gt;(this.tracing, spanNamer,
callable, "calculateTax");
// Wrapping `Callable` with `Tracing`. That way the current span will be available
// in the thread of `Callable`
Callable&lt;String&gt; traceCallableFromTracer = this.tracing.currentTraceContext()
.wrap(callable);</programlisting>
<simpara>That way, you ensure that a new span is created and closed for each execution.</simpara>
</section>
<section xml:id="_hystrix">
<title>Hystrix</title>
<section xml:id="_custom_concurrency_strategy">
<title>Custom Concurrency Strategy</title>
<simpara>We register a custom <link xl:href="https://github.com/Netflix/Hystrix/wiki/Plugins#concurrencystrategy"><literal>HystrixConcurrencyStrategy</literal></link> called <literal>TraceCallable</literal> that wraps all <literal>Callable</literal> instances in their Sleuth representative.
The strategy either starts or continues a span, depending on whether tracing was already going on before the Hystrix command was called.
To disable the custom Hystrix Concurrency Strategy, set the <literal>spring.sleuth.hystrix.strategy.enabled</literal> to <literal>false</literal>.</simpara>
</section>
<section xml:id="_manual_command_setting">
<title>Manual Command setting</title>
<simpara>Assume that you have the following <literal>HystrixCommand</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">HystrixCommand&lt;String&gt; hystrixCommand = new HystrixCommand&lt;String&gt;(setter) {
@Override
protected String run() throws Exception {
return someLogic();
}
};</programlisting>
<simpara>To pass the tracing information, you have to wrap the same logic in the Sleuth version of the <literal>HystrixCommand</literal>, which is called
<literal>TraceCommand</literal>, as shown in the following example:</simpara>
<programlisting language="java" linenumbering="unnumbered">TraceCommand&lt;String&gt; traceCommand = new TraceCommand&lt;String&gt;(tracer, setter) {
@Override
public String doRun() throws Exception {
return someLogic();
}
};</programlisting>
</section>
</section>
<section xml:id="_rxjava">
<title>RxJava</title>
<simpara>We registering a custom <link xl:href="https://github.com/ReactiveX/RxJava/wiki/Plugins#rxjavaschedulershook"><literal>RxJavaSchedulersHook</literal></link> that wraps all <literal>Action0</literal> instances in their Sleuth representative, which is called <literal>TraceAction</literal>.
The hook either starts or continues a span, depending on whether tracing was already going on before the Action was scheduled.
To disable the custom <literal>RxJavaSchedulersHook</literal>, set the <literal>spring.sleuth.rxjava.schedulers.hook.enabled</literal> to <literal>false</literal>.</simpara>
<simpara>You can define a list of regular expressions for thread names for which you do not want spans to be created.
To do so, provide a comma-separated list of regular expressions in the <literal>spring.sleuth.rxjava.schedulers.ignoredthreads</literal> property.</simpara>
<important>
<simpara>The suggest approach to reactive programming and Sleuth is to use
the Reactor support.</simpara>
</important>
</section>
<section xml:id="_http_integration">
<title>HTTP integration</title>
<simpara>Features from this section can be disabled by setting the <literal>spring.sleuth.web.enabled</literal> property with value equal to <literal>false</literal>.</simpara>
<section xml:id="_http_filter">
<title>HTTP Filter</title>
<simpara>Through the <literal>TracingFilter</literal>, all sampled incoming requests result in creation of a Span.
That Span&#8217;s name is <literal>http:</literal> + the path to which the request was sent.
For example, if the request was sent to <literal>/this/that</literal> then the name will be <literal>http:/this/that</literal>.
You can configure which URIs you would like to skip by setting the <literal>spring.sleuth.web.skipPattern</literal> property.
If you have <literal>ManagementServerProperties</literal> on classpath, its value of <literal>contextPath</literal> gets appended to the provided skip pattern.
If you want to reuse the Sleuth&#8217;s default skip patterns and just append your own, pass those patterns by using the <literal>spring.sleuth.web.additionalSkipPattern</literal>.</simpara>
<simpara>By default, all the spring boot actuator endpoints are automatically added to the skip pattern.
If you want to disable this behaviour set <literal>spring.sleuth.web.ignore-auto-configured-skip-patterns</literal>
to <literal>true</literal>.</simpara>
<simpara>To change the order of tracing filter registration, please set the
<literal>spring.sleuth.web.filter-order</literal> property.</simpara>
<simpara>To disable the filter that logs uncaught exceptions you can disable the
<literal>spring.sleuth.web.exception-throwing-filter-enabled</literal> property.</simpara>
</section>
<section xml:id="_handlerinterceptor">
<title>HandlerInterceptor</title>
<simpara>Since we want the span names to be precise, we use a <literal>TraceHandlerInterceptor</literal> that either wraps an existing <literal>HandlerInterceptor</literal> or is added directly to the list of existing <literal>HandlerInterceptors</literal>.
The <literal>TraceHandlerInterceptor</literal> adds a special request attribute to the given <literal>HttpServletRequest</literal>.
If the the <literal>TracingFilter</literal> does not see this attribute, it creates a "<literal>fallback</literal>" span, which is an additional span created on the server side so that the trace is presented properly in the UI.
If that happens, there is probably missing instrumentation.
In that case, please file an issue in Spring Cloud Sleuth.</simpara>
</section>
<section xml:id="_async_servlet_support">
<title>Async Servlet support</title>
<simpara>If your controller returns a <literal>Callable</literal> or a <literal>WebAsyncTask</literal>, Spring Cloud Sleuth continues the existing span instead of creating a new one.</simpara>
</section>
<section xml:id="_webflux_support">
<title>WebFlux support</title>
<simpara>Through <literal>TraceWebFilter</literal>, all sampled incoming requests result in creation of a Span.
That Span&#8217;s name is <literal>http:</literal> + the path to which the request was sent.
For example, if the request was sent to <literal>/this/that</literal>, the name is <literal>http:/this/that</literal>.
You can configure which URIs you would like to skip by using the <literal>spring.sleuth.web.skipPattern</literal> property.
If you have <literal>ManagementServerProperties</literal> on the classpath, its value of <literal>contextPath</literal> gets appended to the provided skip pattern.
If you want to reuse Sleuth&#8217;s default skip patterns and append your own, pass those patterns by using the <literal>spring.sleuth.web.additionalSkipPattern</literal>.</simpara>
<simpara>To change the order of tracing filter registration, please set the
<literal>spring.sleuth.web.filter-order</literal> property.</simpara>
</section>
<section xml:id="_dubbo_rpc_support">
<title>Dubbo RPC support</title>
<simpara>Via the integration with Brave, Spring Cloud Sleuth supports <link xl:href="http://dubbo.io/">Dubbo</link>.
It&#8217;s enough to add the <literal>brave-instrumentation-dubbo-rpc</literal> dependency:</simpara>
<programlisting language="xml" linenumbering="unnumbered">&lt;dependency&gt;
&lt;groupId&gt;io.zipkin.brave&lt;/groupId&gt;
&lt;artifactId&gt;brave-instrumentation-dubbo-rpc&lt;/artifactId&gt;
&lt;/dependency&gt;</programlisting>
<simpara>You need to also set a <literal>dubbo.properties</literal> file with the following contents:</simpara>
<programlisting language="properties" linenumbering="unnumbered">dubbo.provider.filter=tracing
dubbo.consumer.filter=tracing</programlisting>
<simpara>You can read more about Brave - Dubbo integration <link xl:href="https://github.com/openzipkin/brave/tree/master/instrumentation/dubbo-rpc">here</link>.
An example of Spring Cloud Sleuth and Dubbo can be found <link xl:href="https://github.com/openzipkin/sleuth-webmvc-example/compare/add-dubbo-tracing">here</link>.</simpara>
</section>
</section>
<section xml:id="_http_client_integration">
<title>HTTP Client Integration</title>
<section xml:id="_synchronous_rest_template">
<title>Synchronous Rest Template</title>
<simpara>We inject a <literal>RestTemplate</literal> interceptor to ensure that all the tracing information is passed to the requests.
Each time a call is made, a new Span is created.
It gets closed upon receiving the response.
To block the synchronous <literal>RestTemplate</literal> features, set <literal>spring.sleuth.web.client.enabled</literal> to <literal>false</literal>.</simpara>
<important>
<simpara>You have to register <literal>RestTemplate</literal> as a bean so that the interceptors get injected.
If you create a <literal>RestTemplate</literal> instance with a <literal>new</literal> keyword, the instrumentation does NOT work.</simpara>
</important>
</section>
<section xml:id="_asynchronous_rest_template">
<title>Asynchronous Rest Template</title>
<important>
<simpara>Starting with Sleuth <literal>2.0.0</literal>, we no longer register a bean of <literal>AsyncRestTemplate</literal> type.
It is up to you to create such a bean.
Then we instrument it.</simpara>
</important>
<simpara>To block the <literal>AsyncRestTemplate</literal> features, set <literal>spring.sleuth.web.async.client.enabled</literal> to <literal>false</literal>.
To disable creation of the default <literal>TraceAsyncClientHttpRequestFactoryWrapper</literal>, set <literal>spring.sleuth.web.async.client.factory.enabled</literal>
to <literal>false</literal>.
If you do not want to create <literal>AsyncRestClient</literal> at all, set <literal>spring.sleuth.web.async.client.template.enabled</literal> to <literal>false</literal>.</simpara>
<section xml:id="_multiple_asynchronous_rest_templates">
<title>Multiple Asynchronous Rest Templates</title>
<simpara>Sometimes you need to use multiple implementations of the Asynchronous Rest Template.
In the following snippet, you can see an example of how to set up such a custom <literal>AsyncRestTemplate</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Configuration
@EnableAutoConfiguration
static class Config {
@Bean(name = "customAsyncRestTemplate")
public AsyncRestTemplate traceAsyncRestTemplate() {
return new AsyncRestTemplate(asyncClientFactory(),
clientHttpRequestFactory());
}
private ClientHttpRequestFactory clientHttpRequestFactory() {
ClientHttpRequestFactory clientHttpRequestFactory = new CustomClientHttpRequestFactory();
// CUSTOMIZE HERE
return clientHttpRequestFactory;
}
private AsyncClientHttpRequestFactory asyncClientFactory() {
AsyncClientHttpRequestFactory factory = new CustomAsyncClientHttpRequestFactory();
// CUSTOMIZE HERE
return factory;
}
}</programlisting>
</section>
</section>
<section xml:id="_webclient">
<title><literal>WebClient</literal></title>
<simpara>We inject a <literal>ExchangeFilterFunction</literal> implementation that creates a span and, through on-success and on-error callbacks, takes care of closing client-side spans.</simpara>
<simpara>To block this feature, set <literal>spring.sleuth.web.client.enabled</literal> to <literal>false</literal>.</simpara>
<important>
<simpara>You have to register <literal>WebClient</literal> as a bean so that the tracing instrumentation gets applied.
If you create a <literal>WebClient</literal> instance with a <literal>new</literal> keyword, the instrumentation does NOT work.</simpara>
</important>
</section>
<section xml:id="_traverson">
<title>Traverson</title>
<simpara>If you use the <link xl:href="http://docs.spring.io/spring-hateoas/docs/current/reference/html/#client.traverson">Traverson</link> library, you can inject a <literal>RestTemplate</literal> as a bean into your Traverson object.
Since <literal>RestTemplate</literal> is already intercepted, you get full support for tracing in your client. The following pseudo code
shows how to do that:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Autowired RestTemplate restTemplate;
Traverson traverson = new Traverson(URI.create("http://some/address"),
MediaType.APPLICATION_JSON, MediaType.APPLICATION_JSON_UTF8).setRestOperations(restTemplate);
// use Traverson</programlisting>
</section>
<section xml:id="_apache_httpclientbuilder_and_httpasyncclientbuilder">
<title>Apache <literal>HttpClientBuilder</literal> and <literal>HttpAsyncClientBuilder</literal></title>
<simpara>We instrument the <literal>HttpClientBuilder</literal> and <literal>HttpAsyncClientBuilder</literal> so that
tracing context gets injected to the sent requests.</simpara>
<simpara>To block these features, set <literal>spring.sleuth.web.client.enabled</literal> to <literal>false</literal>.</simpara>
</section>
<section xml:id="_netty_httpclient">
<title>Netty <literal>HttpClient</literal></title>
<simpara>We instrument the Netty&#8217;s <literal>HttpClient</literal>.</simpara>
<simpara>To block this feature, set <literal>spring.sleuth.web.client.enabled</literal> to <literal>false</literal>.</simpara>
<important>
<simpara>You have to register <literal>HttpClient</literal> as a bean so that the instrumentation happens.
If you create a <literal>HttpClient</literal> instance with a <literal>new</literal> keyword, the instrumentation does NOT work.</simpara>
</important>
</section>
<section xml:id="_userinforesttemplatecustomizer">
<title><literal>UserInfoRestTemplateCustomizer</literal></title>
<simpara>We instrument the Spring Security&#8217;s <literal>UserInfoRestTemplateCustomizer</literal>.</simpara>
<simpara>To block this feature, set <literal>spring.sleuth.web.client.enabled</literal> to <literal>false</literal>.</simpara>
</section>
</section>
<section xml:id="_feign">
<title>Feign</title>
<simpara>By default, Spring Cloud Sleuth provides integration with Feign through <literal>TraceFeignClientAutoConfiguration</literal>.
You can disable it entirely by setting <literal>spring.sleuth.feign.enabled</literal> to <literal>false</literal>.
If you do so, no Feign-related instrumentation take place.</simpara>
<simpara>Part of Feign instrumentation is done through a <literal>FeignBeanPostProcessor</literal>.
You can disable it by setting <literal>spring.sleuth.feign.processor.enabled</literal> to <literal>false</literal>.
If you set it to <literal>false</literal>, Spring Cloud Sleuth does not instrument any of your custom Feign components.
However, all the default instrumentation is still there.</simpara>
</section>
<section xml:id="_grpc">
<title>gRPC</title>
<simpara>Spring Cloud Sleuth provides instrumentation for <link xl:href="https://grpc.io/">gRPC</link> through <literal>TraceGrpcAutoConfiguration</literal>. You can disable it entirely by setting <literal>spring.sleuth.grpc.enabled</literal> to <literal>false</literal>.</simpara>
<section xml:id="_variant_1">
<title>Variant 1</title>
<section xml:id="_dependencies">
<title>Dependencies</title>
<important>
<simpara>The gRPC integration relies on two external libraries to instrument clients and servers and both of those libraries must be on the class path to enable the instrumentation.</simpara>
</important>
<simpara>Maven:</simpara>
<screen> &lt;dependency&gt;
&lt;groupId&gt;io.github.lognet&lt;/groupId&gt;
&lt;artifactId&gt;grpc-spring-boot-starter&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;dependency&gt;
&lt;groupId&gt;io.zipkin.brave&lt;/groupId&gt;
&lt;artifactId&gt;brave-instrumentation-grpc&lt;/artifactId&gt;
&lt;/dependency&gt;</screen>
<simpara>Gradle:</simpara>
<screen> compile("io.github.lognet:grpc-spring-boot-starter")
compile("io.zipkin.brave:brave-instrumentation-grpc")</screen>
</section>
<section xml:id="_server_instrumentation">
<title>Server Instrumentation</title>
<simpara>Spring Cloud Sleuth leverages grpc-spring-boot-starter to register Brave&#8217;s gRPC server interceptor with all services annotated with <literal>@GRpcService</literal>.</simpara>
</section>
<section xml:id="_client_instrumentation">
<title>Client Instrumentation</title>
<simpara>gRPC clients leverage a <literal>ManagedChannelBuilder</literal> to construct a <literal>ManagedChannel</literal> used to communicate to the gRPC server. The native <literal>ManagedChannelBuilder</literal> provides static methods as entry points for construction of <literal>ManagedChannel</literal> instances, however, this mechanism is outside the influence of the Spring application context.</simpara>
<important>
<simpara>Spring Cloud Sleuth provides a <literal>SpringAwareManagedChannelBuilder</literal> that can be customized through the Spring application context and injected by gRPC clients. <emphasis role="strong">This builder must be used when creating <literal>ManagedChannel</literal> instances.</emphasis></simpara>
</important>
<simpara>Sleuth creates a <literal>TracingManagedChannelBuilderCustomizer</literal> which inject Brave&#8217;s client interceptor into the <literal>SpringAwareManagedChannelBuilder</literal>.</simpara>
</section>
</section>
<section xml:id="_variant_2">
<title>Variant 2</title>
<simpara><link xl:href="https://github.com/yidongnan/grpc-spring-boot-starter">Grpc Spring Boot Starter</link> automatically detects the presence of Spring Cloud Sleuth and brave&#8217;s instrumentation for gRPC and registers the necessary client and/or server tooling.</simpara>
</section>
</section>
<section xml:id="_asynchronous_communication">
<title>Asynchronous Communication</title>
<section xml:id="_async_annotated_methods">
<title><literal>@Async</literal> Annotated methods</title>
<simpara>In Spring Cloud Sleuth, we instrument async-related components so that the tracing information is passed between threads.
You can disable this behavior by setting the value of <literal>spring.sleuth.async.enabled</literal> to <literal>false</literal>.</simpara>
<simpara>If you annotate your method with <literal>@Async</literal>, we automatically create a new Span with the following characteristics:</simpara>
<itemizedlist>
<listitem>
<simpara>If the method is annotated with <literal>@SpanName</literal>, the value of the annotation is the Span&#8217;s name.</simpara>
</listitem>
<listitem>
<simpara>If the method is not annotated with <literal>@SpanName</literal>, the Span name is the annotated method name.</simpara>
</listitem>
<listitem>
<simpara>The span is tagged with the method&#8217;s class name and method name.</simpara>
</listitem>
</itemizedlist>
</section>
<section xml:id="_scheduled_annotated_methods">
<title><literal>@Scheduled</literal> Annotated Methods</title>
<simpara>In Spring Cloud Sleuth, we instrument scheduled method execution so that the tracing information is passed between threads.
You can disable this behavior by setting the value of <literal>spring.sleuth.scheduled.enabled</literal> to <literal>false</literal>.</simpara>
<simpara>If you annotate your method with <literal>@Scheduled</literal>, we automatically create a new span with the following characteristics:</simpara>
<itemizedlist>
<listitem>
<simpara>The span name is the annotated method name.</simpara>
</listitem>
<listitem>
<simpara>The span is tagged with the method&#8217;s class name and method name.</simpara>
</listitem>
</itemizedlist>
<simpara>If you want to skip span creation for some <literal>@Scheduled</literal> annotated classes, you can set the <literal>spring.sleuth.scheduled.skipPattern</literal> with a regular expression that matches the fully qualified name of the <literal>@Scheduled</literal> annotated class.
If you use <literal>spring-cloud-sleuth-stream</literal> and <literal>spring-cloud-netflix-hystrix-stream</literal> together, a span is created for each Hystrix metrics and sent to Zipkin.
This behavior may be annoying. That&#8217;s why, by default, <literal>spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask</literal>.</simpara>
</section>
<section xml:id="_executor_executorservice_and_scheduledexecutorservice">
<title>Executor, ExecutorService, and ScheduledExecutorService</title>
<simpara>We provide <literal>LazyTraceExecutor</literal>, <literal>TraceableExecutorService</literal>, and <literal>TraceableScheduledExecutorService</literal>. Those implementations create spans each time a new task is submitted, invoked, or scheduled.</simpara>
<simpara>The following example shows how to pass tracing information with <literal>TraceableExecutorService</literal> when working with <literal>CompletableFuture</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">CompletableFuture&lt;Long&gt; completableFuture = CompletableFuture.supplyAsync(() -&gt; {
// perform some logic
return 1_000_000L;
}, new TraceableExecutorService(beanFactory, executorService,
// 'calculateTax' explicitly names the span - this param is optional
"calculateTax"));</programlisting>
<important>
<simpara>Sleuth does not work with <literal>parallelStream()</literal> out of the box.
If you want to have the tracing information propagated through the stream, you have to use the approach with <literal>supplyAsync(...)</literal>, as shown earlier.</simpara>
</important>
<simpara>If there are beans that implement the <literal>Executor</literal> interface that you would like
to exclude from span creation, you can use the <literal>spring.sleuth.async.ignored-beans</literal>
property where you can provide a list of bean names.</simpara>
<section xml:id="_customization_of_executors">
<title>Customization of Executors</title>
<simpara>Sometimes, you need to set up a custom instance of the <literal>AsyncExecutor</literal>.
The following example shows how to set up such a custom <literal>Executor</literal>:</simpara>
<programlisting language="java" linenumbering="unnumbered">@Configuration
@EnableAutoConfiguration
@EnableAsync
// add the infrastructure role to ensure that the bean gets auto-proxied
@Role(BeanDefinition.ROLE_INFRASTRUCTURE)
static class CustomExecutorConfig extends AsyncConfigurerSupport {
@Autowired
BeanFactory beanFactory;
@Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
// CUSTOMIZE HERE
executor.setCorePoolSize(7);
executor.setMaxPoolSize(42);
executor.setQueueCapacity(11);
executor.setThreadNamePrefix("MyExecutor-");
// DON'T FORGET TO INITIALIZE
executor.initialize();
return new LazyTraceExecutor(this.beanFactory, executor);
}
}</programlisting>
<tip>
<simpara>To ensure that your configuration gets post processed, remember
to add the <literal>@Role(BeanDefinition.ROLE_INFRASTRUCTURE)</literal> on your
<literal>@Configuration</literal> class</simpara>
</tip>
</section>
</section>
</section>
<section xml:id="_messaging">
<title>Messaging</title>
<simpara>Features from this section can be disabled by setting the <literal>spring.sleuth.messaging.enabled</literal> property with value equal to <literal>false</literal>.</simpara>
<section xml:id="_spring_integration_and_spring_cloud_stream">
<title>Spring Integration and Spring Cloud Stream</title>
<simpara>Spring Cloud Sleuth integrates with <link xl:href="http://projects.spring.io/spring-integration/">Spring Integration</link>.
It creates spans for publish and subscribe events.
To disable Spring Integration instrumentation, set <literal>spring.sleuth.integration.enabled</literal> to <literal>false</literal>.</simpara>
<simpara>You can provide the <literal>spring.sleuth.integration.patterns</literal> pattern to explicitly provide the names of channels that you want to include for tracing.
By default, all channels but <literal>hystrixStreamOutput</literal> channel are included.</simpara>
<important>
<simpara>When using the <literal>Executor</literal> to build a Spring Integration <literal>IntegrationFlow</literal>, you must use the untraced version of the <literal>Executor</literal>.
Decorating the Spring Integration Executor Channel with <literal>TraceableExecutorService</literal> causes the spans to be improperly closed.</simpara>
</important>
<simpara>If you want to customize the way tracing context is read from and written to message headers,
it&#8217;s enough for you to register beans of types:</simpara>
<itemizedlist>
<listitem>
<simpara><literal>Propagation.Setter&lt;MessageHeaderAccessor, String&gt;</literal> - for writing headers to the message</simpara>
</listitem>
<listitem>
<simpara><literal>Propagation.Getter&lt;MessageHeaderAccessor, String&gt;</literal> - for reading headers from the message</simpara>
</listitem>
</itemizedlist>
</section>
<section xml:id="_spring_rabbitmq">
<title>Spring RabbitMq</title>
<simpara>We instrument the <literal>RabbitTemplate</literal> so that tracing headers get injected
into the message.</simpara>
<simpara>To block this feature, set <literal>spring.sleuth.messaging.rabbit.enabled</literal> to <literal>false</literal>.</simpara>
</section>
<section xml:id="_spring_kafka">
<title>Spring Kafka</title>
<simpara>We instrument the Spring Kafka&#8217;s <literal>ProducerFactory</literal> and <literal>ConsumerFactory</literal>
so that tracing headers get injected into the created Spring Kafka&#8217;s
<literal>Producer</literal> and <literal>Consumer</literal>.</simpara>
<simpara>To block this feature, set <literal>spring.sleuth.messaging.kafka.enabled</literal> to <literal>false</literal>.</simpara>
</section>
<section xml:id="_spring_jms">
<title>Spring JMS</title>
<simpara>We instrument the <literal>JmsTemplate</literal> so that tracing headers get injected
into the message. We also support <literal>@JmsListener</literal> annotated methods on the consumer side.</simpara>
<simpara>To block this feature, set <literal>spring.sleuth.messaging.jms.enabled</literal> to <literal>false</literal>.</simpara>
<important>
<simpara>We don&#8217;t support baggage propagation for JMS</simpara>
</important>
</section>
</section>
<section xml:id="_zuul">
<title>Zuul</title>
<simpara>We instrument the Zuul Ribbon integration by enriching the Ribbon requests with tracing information.
To disable Zuul support, set the <literal>spring.sleuth.zuul.enabled</literal> property to <literal>false</literal>.</simpara>
</section>
</chapter>
<chapter xml:id="_running_examples">
<title>Running examples</title>
<simpara>You can see the running examples deployed in the <link xl:href="https://run.pivotal.io/">Pivotal Web Services</link>.
Check them out at the following links:</simpara>
<itemizedlist>
<listitem>
<simpara><link xl:href="http://docssleuth-zipkin-server.cfapps.io/">Zipkin for apps presented in the samples to the top</link>. First make
a request to <link xl:href="http://docssleuth-service1.cfapps.io/start">Service 1</link> and then check out the trace in Zipkin.</simpara>
</listitem>
<listitem>
<simpara><link xl:href="http://docsbrewing-zipkin-server.cfapps.io/">Zipkin for Brewery on PWS</link>, its <link xl:href="https://github.com/spring-cloud-samples/brewery">Github Code</link>.
Ensure that you&#8217;ve picked the lookback period of 7 days. If there are no traces, go to <link xl:href="https://docsbrewing-presenting.cfapps.io/">Presenting application</link>
and order some beers. Then check Zipkin for traces.</simpara>
</listitem>
</itemizedlist>
</chapter>
</book>
Reference in New Issue View Git Blame Copy Permalink