1.0.5.BUILD-SNAPSHOT
+Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.
+Terminology
+Spring Cloud Sleuth borrows Dapper’s terminology.
+Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an +RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span +is a part of. Spans also have other data, such as descriptions, timestamped events, key-value +annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address).
+Spans are started and stopped, and they keep track of their timing information. Once you create a +span, you must stop it at some point in the future.
+Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed +big-data store, a trace might be formed by a put request.
+Annotation: is used to record existence of an event in time. Some of the core annotations used to define +the start and stop of a request are:
+-
+
-
+
cs - Client Sent - The client has made a request. This annotation depicts the start of the span.
+
+ -
+
sr - Server Received - The server side got the request and will start processing it. +If one subtracts the cs timestamp from this timestamp one will receive the network latency.
+
+ -
+
ss - Server Sent - Annotated upon completion of request processing (when the response +got sent back to the client). If one subtracts the sr timestamp from this timestamp one +will receive the time needed by the server side to process the request.
+
+ -
+
cr - Client Received - Signifies the end of the span. The client has successfully received the +response from the server side. If one subtracts the cs timestamp from this timestamp one +will receive the whole time needed by the client to receive the response from the server.
+
+
Visualization of what Span and Trace will look in a system together with the Zipkin annotations:
+
+Each color of a note signifies a span (7 spans - from A to G). If you have such information in the note:
+Trace Id = X
+Span Id = D
+Client Sent
+That means that the current span has Trace-Id set to X, Span-Id set to D. It also has emitted + Client Sent event.
+This is how the visualization of the parent / child relationship of spans would look like:
+
+Purpose
+In the following sections the example from the image above will be taken into consideration.
+Distributed tracing with Zipkin
+Altogether there are 10 spans . If you go to traces in Zipkin you will see this number:
+
+However if you pick a particular trace then you will see 7 spans:
+
+|
+ Note
+ |
++When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to +Zipkin with Server Received and Server Sent / Client Received and Client Sent +annotations then they will presented as a single span. + | +
In the image depicting the visualization of what Span and Trace is you can see 20 +colorful labels. How does it happen that in Zipkin 10 spans are received?
+-
+
-
+
2 span A labels signify span started and closed. Upon closing a single span is sent to Zipkin.
+
+ -
+
4 span B labels are in fact are single span with 4 annotations. However this span is composed of +two separate instances. One sent from service 1 and one from service 2. So in fact two span instances will be sent +to Zipkin and merged there.
+
+ -
+
2 span C labels signify span started and closed. Upon closing a single span is sent to Zipkin.
+
+ -
+
4 span B labels are in fact are single span with 4 annotations. However this span is composed of +two separate instances. One sent from service 2 and one from service 3. So in fact two span instances will be sent +to Zipkin and merged there.
+
+ -
+
2 span E labels signify span started and closed. Upon closing a single span is sent to Zipkin.
+
+ -
+
4 span B labels are in fact are single span with 4 annotations. However this span is composed of +two separate instances. One sent from service 2 and one from service 4. So in fact two span instances will be sent +to Zipkin and merged there.
+
+ -
+
2 span G labels signify span started and closed. Upon closing a single span is sent to Zipkin.
+
+
So 1 span from A, 2 spans from B, 1 span from C, 2 spans from D, 1 span from E, 2 spans from F and 1 from G. +Altogether 10 spans.
+The dependency graph in Zipkin would look like this:
+
+Log correlation
+When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following:
service1.log:2016-02-26 11:15:47.561 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Hello from service1. Calling service2
+service2.log:2016-02-26 11:15:47.710 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Hello from service2. Calling service3 and then service4
+service3.log:2016-02-26 11:15:47.895 INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application : Hello from service3
+service2.log:2016-02-26 11:15:47.924 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service3 [Hello from service3]
+service4.log:2016-02-26 11:15:48.134 INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application : Hello from service4
+service2.log:2016-02-26 11:15:48.156 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service4 [Hello from service4]
+service1.log:2016-02-26 11:15:48.182 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]
+If you’re using a log aggregating tool like Kibana, +Splunk etc. you can order the events that took place. An example of +Kibana would look like this:
+
+If you want to use Logstash here is the Grok pattern for Logstash:
+filter {
+ # pattern matching logback pattern
+ grok {
+ match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
+ }
+}
+|
+ Note
+ |
++If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern: + | +
filter {
+ # pattern matching logback pattern
+ grok {
+ match => { "message" => "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
+ }
+}
+JSON Logback with Logstash
+Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability
+we’re passing the dependencies in the groupId:artifactId:version notation.
Dependencies setup
+-
+
-
+
Ensure that Logback is on the classpath (
+ch.qos.logback:logback-core)
+ -
+
Add Logstash Logback encode - example for version
+4.6:net.logstash.logback:logstash-logback-encoder:4.6
+
Logback setup
+Below you can find an example of a Logback configuration (file named https://github.com/spring-cloud-samples/sleuth-documentation-apps/blob/master/service1/src/main/resources/logback-spring.xml[logback-spring.xml]) that:
-
+
-
+
logs information from the application in a JSON format to a
+build/${spring.application.name}.jsonfile
+ -
+
has commented out two additional appenders - console and standard log file
+
+ -
+
has the same logging pattern as the one presented in the previous section
+
+
<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+ <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
+
+ <springProperty scope="context" name="springAppName" source="spring.application.name"/>
+ <!-- Example for logging into the build folder of your project -->
+ <property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${springAppName}"/>
+
+ <property name="CONSOLE_LOG_PATTERN"
+ value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr([${springAppName:-},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]){yellow} %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>
+
+ <!-- Appender to log to console -->
+ <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
+ <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
+ <!-- Minimum logging level to be presented in the console logs-->
+ <level>INFO</level>
+ </filter>
+ <encoder>
+ <pattern>${CONSOLE_LOG_PATTERN}</pattern>
+ <charset>utf8</charset>
+ </encoder>
+ </appender>
+
+ <!-- Appender to log to file -->
+ <appender name="flatfile" class="ch.qos.logback.core.rolling.RollingFileAppender">
+ <file>${LOG_FILE}</file>
+ <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+ <fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.gz</fileNamePattern>
+ <maxHistory>7</maxHistory>
+ </rollingPolicy>
+ <encoder>
+ <pattern>${CONSOLE_LOG_PATTERN}</pattern>
+ <charset>utf8</charset>
+ </encoder>
+ </appender>
+
+ <!-- Appender to log to file in a JSON format -->
+ <appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender">
+ <file>${LOG_FILE}.json</file>
+ <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+ <fileNamePattern>${LOG_FILE}.json.%d{yyyy-MM-dd}.gz</fileNamePattern>
+ <maxHistory>7</maxHistory>
+ </rollingPolicy>
+ <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
+ <providers>
+ <timestamp>
+ <timeZone>UTC</timeZone>
+ </timestamp>
+ <pattern>
+ <pattern>
+ {
+ "severity": "%level",
+ "service": "${springAppName:-}",
+ "trace": "%X{X-B3-TraceId:-}",
+ "span": "%X{X-B3-SpanId:-}",
+ "exportable": "%X{X-Span-Export:-}",
+ "pid": "${PID:-}",
+ "thread": "%thread",
+ "class": "%logger{40}",
+ "rest": "%message"
+ }
+ </pattern>
+ </pattern>
+ </providers>
+ </encoder>
+ </appender>
+
+ <root level="INFO">
+ <!--<appender-ref ref="console"/>-->
+ <appender-ref ref="logstash"/>
+ <!--<appender-ref ref="flatfile"/>-->
+ </root>
+</configuration>
+Adding to the project
+Only Sleuth (log correlation)
+If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add
+the spring-cloud-starter-sleuth module to your project.
<dependencyManagement> (1)
+ <dependencies>
+ <dependency>
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-dependencies</artifactId>
+ <version>Brixton.RELEASE</version>
+ <type>pom</type>
+ <scope>import</scope>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+
+ <dependency> (2)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-starter-sleuth</artifactId>
+ </dependency>
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-starter-sleuth
+
dependencyManagement { (1)
+ imports {
+ mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
+ }
+}
+
+dependencies { (2)
+ compile "org.springframework.cloud:spring-cloud-starter-sleuth"
+}
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-starter-sleuth
+
Sleuth with Zipkin via HTTP
+If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency.
<dependencyManagement> (1)
+ <dependencies>
+ <dependency>
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-dependencies</artifactId>
+ <version>Brixton.RELEASE</version>
+ <type>pom</type>
+ <scope>import</scope>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+
+ <dependency> (2)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-starter-zipkin</artifactId>
+ </dependency>
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-starter-zipkin
+
dependencyManagement { (1)
+ imports {
+ mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
+ }
+}
+
+dependencies { (2)
+ compile "org.springframework.cloud:spring-cloud-starter-zipkin"
+}
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-starter-zipkin
+
Sleuth with Zipkin via Spring Cloud Stream
+If you want both Sleuth and Zipkin just add the spring-cloud-sleuth-stream dependency.
<dependencyManagement> (1)
+ <dependencies>
+ <dependency>
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-dependencies</artifactId>
+ <version>Brixton.RELEASE</version>
+ <type>pom</type>
+ <scope>import</scope>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+
+ <dependency> (2)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-sleuth-stream</artifactId>
+ </dependency>
+ <dependency> (3)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-starter-sleuth</artifactId>
+ </dependency>
+ <!-- EXAMPLE FOR RABBIT BINDING -->
+ <dependency> (4)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-stream-binder-rabbit</artifactId>
+ </dependency>
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-sleuth-stream
+ -
+
Add the dependency to
+spring-cloud-starter-sleuth- that way all dependant dependencies will be downloaded
+ -
+
Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
+
+
dependencyManagement { (1)
+ imports {
+ mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
+ }
+}
+
+dependencies {
+ compile "org.springframework.cloud:spring-cloud-sleuth-stream" (2)
+ compile "org.springframework.cloud:spring-cloud-starter-sleuth" (3)
+ // Example for Rabbit binding
+ compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" (4)
+}
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-sleuth-stream
+ -
+
Add the dependency to
+spring-cloud-starter-sleuth- that way all dependant dependencies will be downloaded
+ -
+
Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
+
+
Spring Cloud Sleuth Stream Zipkin Collector
+If you want to start a Spring Cloud Sleuth Stream Zipkin collector just add the spring-cloud-sleuth-zipkin-stream
+dependency
<dependencyManagement> (1)
+ <dependencies>
+ <dependency>
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-dependencies</artifactId>
+ <version>Brixton.RELEASE</version>
+ <type>pom</type>
+ <scope>import</scope>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+
+ <dependency> (2)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-sleuth-zipkin-stream</artifactId>
+ </dependency>
+ <dependency> (3)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-starter-sleuth</artifactId>
+ </dependency>
+ <!-- EXAMPLE FOR RABBIT BINDING -->
+ <dependency> (4)
+ <groupId>org.springframework.cloud</groupId>
+ <artifactId>spring-cloud-stream-binder-rabbit</artifactId>
+ </dependency>
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-sleuth-zipkin-stream
+ -
+
Add the dependency to
+spring-cloud-starter-sleuth- that way all dependant dependencies will be downloaded
+ -
+
Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
+
+
dependencyManagement { (1)
+ imports {
+ mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
+ }
+}
+
+dependencies {
+ compile "org.springframework.cloud:spring-cloud-sleuth-zipkin-stream" (2)
+ compile "org.springframework.cloud:spring-cloud-starter-sleuth" (3)
+ // Example for Rabbit binding
+ compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" (4)
+}
+-
+
-
+
In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM
+
+ -
+
Add the dependency to
+spring-cloud-sleuth-zipkin-stream
+ -
+
Add the dependency to
+spring-cloud-starter-sleuth- that way all dependant dependencies will be downloaded
+ -
+
Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
+
+
and then just annotate your main class with @EnableZipkinStreamServer annotation:
package example;
+
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.cloud.sleuth.zipkin.stream.EnableZipkinStreamServer;
+
+@SpringBootApplication
+@EnableZipkinStreamServer
+public class ZipkinStreamServerApplication {
+
+ public static void main(String[] args) throws Exception {
+ SpringApplication.run(ZipkinStreamServerApplication.class, args);
+ }
+
+}
+Features
+-
+
-
+
Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Example logs:
+++++2016-02-02 15:30:57.902 INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ... +2016-02-02 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ... +2016-02-02 15:31:01.936 INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...
+++notice the
+[appname,traceId,spanId,exportable]entries from the MDC:++-
+
-
+
spanId - the id of a specific operation that took place
+
+ -
+
appname - the name of the application that logged the span
+
+ -
+
traceId - the id of the latency graph that contains the span
+
+ -
+
exportable - whether the log should be exported to Zipkin or not. When would you like the span not to be +exportable? In the case in which you want to wrap some operation in a Span and have it written to the logs +only.
+
+
+ -
+
-
+
Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, +key-value annotations. Loosely based on HTrace, but Zipkin (Dapper) compatible.
+
+ -
+
Sleuth records timing information to aid in latency analysis. Using sleuth, you can pinpoint causes of +latency in your applications. Sleuth is written to not log too much, and to not cause your production application to crash.
+++-
+
-
+
propagates structural data about your call-graph in-band, and the rest out-of-band.
+
+ -
+
includes opinionated instrumentation of layers such as HTTP
+
+ -
+
includes sampling policy to manage volume
+
+ -
+
can report to a Zipkin system for query and visualization
+
+
+ -
+
-
+
Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, +rest template, scheduled actions, message channels, zuul filters, feign client).
+
+ -
+
Sleuth includes default logic to join a trace across http or messaging boundaries. For example, http propagation +works via Zipkin-compatible request headers. This propagation logic is defined and customized via +
+SpanInjectorandSpanExtractorimplementations.
+ -
+
Provides simple metrics of accepted / dropped spans.
+
+ -
+
If
+spring-cloud-sleuth-zipkinthen the app will generate and collect Zipkin-compatible traces. +By default it sends them via HTTP to a Zipkin server on localhost (port 9411). +Configure the location of the service usingspring.zipkin.baseUrl.
+ -
+
If
+spring-cloud-sleuth-streamthen the app will generate and collect traces via Spring Cloud Stream. +Your app automatically becomes a producer of tracer messages that are sent over your broker of choice +(e.g. RabbitMQ, Apache Kafka, Redis).
+
|
+ Important
+ |
+
+If using Zipkin or Stream, configure the percentage of spans exported using spring.sleuth.sampler.percentage
+(default 0.1, i.e. 10%). Otherwise you might think that Sleuth is not working cause it’s omitting some spans.
+ |
+
|
+ Note
+ |
+
+the SLF4J MDC is always set and logback users will immediately see the trace and span ids in logs per the example
+ above. Other logging systems have to configure their own formatter to get the same result. The default is
+ logging.pattern.level set to %clr(%5p) %clr([${spring.application.name:},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]){yellow}
+ (this is a Spring Boot feature for logback users).
+ This means that if you’re not using SLF4J this pattern WILL NOT be automatically applied.
+ |
+
Sampling
+In distributed tracing the data volumes can be very high so sampling
+can be important (you usually don’t need to export all spans to get a
+good picture of what is happening). Spring Cloud Sleuth has a
+Sampler strategy that you can implement to take control of the
+sampling algorithm. Samplers do not stop span (correlation) ids from
+being generated, but they do prevent the tags and events being
+attached and exported. By default you get a strategy that continues to
+trace if a span is already active, but new ones are always marked as
+non-exportable. If all your apps run with this sampler you will see
+traces in logs, but not in any remote store. For testing the default
+is often enough, and it probably is all you need if you are only using
+the logs (e.g. with an ELK aggregator). If you are exporting span data
+to Zipkin or Spring Cloud Stream, there is also an AlwaysSampler
+that exports everything and a PercentageBasedSampler that samples a
+fixed fraction of spans.
|
+ Note
+ |
+
+the PercentageBasedSampler is the default if you are using
+spring-cloud-sleuth-zipkin or spring-cloud-sleuth-stream. You can
+configure the exports using spring.sleuth.sampler.percentage.
+ |
+
A sampler can be installed just by creating a bean definition, e.g:
+@Bean
+public Sampler defaultSampler() {
+ return new AlwaysSampler();
+}
+Instrumentation
+Spring Cloud Sleuth instruments all your Spring application
+automatically, so you shouldn’t have to do anything to activate
+it. The instrumentation is added using a variety of technologies
+according to the stack that is available, e.g. for a servlet web
+application we use a Filter, and for Spring Integration we use
+ChannelInterceptors.
You can customize the keys used in span tags. To limit the volume of
+span data, by default an HTTP request will be tagged only with a
+handful of metadata like the status code, host and URL. You can add
+request headers by configuring spring.sleuth.keys.http.headers (a
+list of header names).
|
+ Note
+ |
+
+Remember that tags are only collected and exported if there is a
+Sampler that allows it (by default there is not, so there is no
+danger of accidentally collecting too much data without configuring
+something).
+ |
+
|
+ Note
+ |
++Currently the instrumentation in Spring Cloud Sleuth is eager - it means that +we’re actively trying to pass the tracing context between threads. Also timing events +are captured even when sleuth isn’t exporting data to a tracing system. +This approach may change in the future towards being lazy on this matter. + | +
Span lifecycle
+You can do the following operations on the Span by means of Tracer interface:
+-
+
-
+
start - when you start a span its name is assigned and start timestamp is recorded.
+
+ -
+
close - the span gets finished (the end time of the span is recorded) and if +the span is exportable then it will be eligible for collection to Zipkin. +The span is also removed from the current thread.
+
+ -
+
continue - a new instance of span will be created whereas it will be a copy of the +one that it continues.
+
+ -
+
detach - the span doesn’t get stopped or closed. It only gets removed from the current thread.
+
+ -
+
create with explicit parent - you can create a new span and set an explicit parent to it
+
+
Creating and closing spans
+You can manually create spans by using the Tracer interface.
+// Start a span. If there was a span present in this thread it will become
+// the `newSpan`'s parent.
+Span newSpan = this.tracer.createSpan("calculateTax");
+try {
+ // ...
+ // You can tag a span
+ this.tracer.addTag("taxValue", taxValue);
+ // ...
+ // You can log an event on a span
+ newSpan.logEvent("taxCalculated");
+} finally {
+ // Once done remember to close the span. This will allow collecting
+ // the span to send it to Zipkin
+ this.tracer.close(newSpan);
+}
+In this example we could see how to create a new instance of span. Assuming that there already +was a span present in this thread then it would become the parent of that span.
+|
+ Important
+ |
++Always clean after you create a span! Don’t forget to close a span if you want to send it to Zipkin. + | +
Continuing spans
+Sometimes you don’t want to create a new span but you want to continue one. Example of such a +situation might be (of course it all depends on the use-case):
+-
+
-
+
AOP - If there was already a span created before an aspect was reached then you might not want to create a new span.
+
+ -
+
Hystrix - executing a Hystrix command is most likely a logical part of the current processing. It’s in fact +only a technical implementation detail that you wouldn’t necessarily want to reflect in tracing as a separate being.
+
+
The continued instance of span is equal to the one that it continues:
+Span continuedSpan = this.tracer.continueSpan(spanToContinue);
+assertThat(continuedSpan).isEqualTo(spanToContinue);
+To continue a span you can use the Tracer interface.
+// let's assume that we're in a thread Y and we've received
+// the `initialSpan` from thread X
+Span continuedSpan = this.tracer.continueSpan(initialSpan);
+try {
+ // ...
+ // You can tag a span
+ this.tracer.addTag("taxValue", taxValue);
+ // ...
+ // You can log an event on a span
+ continuedSpan.logEvent("taxCalculated");
+} finally {
+ // Once done remember to detach the span. That way you'll
+ // safely remove it from the current thread without closing it
+ this.tracer.detach(continuedSpan);
+}
+|
+ Important
+ |
++Always clean after you create a span! Don’t forget to detach a span if some work was done started in one + thread (e.g. thread X) and it’s waiting for other threads (e.g. Y, Z) to finish. + Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected + the span in thread X should be closed. + | +
Creating spans with an explicit parent
+There is a possibility that you want to start a new span and provide an explicit parent of that span.
+Let’s assume that the parent of a span is in one thread and you want to start a new span in another thread. The
+startSpan method of the Tracer interface is the method you are looking for.
// let's assume that we're in a thread Y and we've received
+// the `initialSpan` from thread X. `initialSpan` will be the parent
+// of the `newSpan`
+Span newSpan = this.tracer.createSpan("calculateCommission", initialSpan);
+try {
+ // ...
+ // You can tag a span
+ this.tracer.addTag("commissionValue", commissionValue);
+ // ...
+ // You can log an event on a span
+ newSpan.logEvent("commissionCalculated");
+} finally {
+ // Once done remember to close the span. This will allow collecting
+ // the span to send it to Zipkin. The tags and events set on the
+ // newSpan will not be present on the parent
+ this.tracer.close(newSpan);
+}
+|
+ Important
+ |
++After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs + related to the fact that you have a span present in the current thread other than the one you’re trying to close. + What’s worse your spans won’t get closed properly thus will not get collected to Zipkin. + | +
Naming spans
+Picking a span name is not a trivial task. Span name should depict an operation name. The name should +be low cardinality (e.g. not include identifiers).
+Since there is a lot of instrumentation going on some of the span names will be +artificial like:
+-
+
-
+
+controller-method-namewhen received by a Controller with a method nameconrollerMethodName
+ -
+
+asyncfor asynchronous operations done via wrappedCallableandRunnable.
+ -
+
+@Scheduledannotated methods will return the simple name of the class.
+
Fortunately, for the asynchronous processing you can provide explicit naming.
+@SpanName annotation
+You can do name the span explicitly via the @SpanName annotation.
@SpanName("calculateTax")
+class TaxCountingRunnable implements Runnable {
+
+ @Override public void run() {
+ // perform logic
+ }
+}
+In this case, when processed in the following manner:
+Runnable runnable = new TraceRunnable(tracer, spanNamer, new TaxCountingRunnable());
+Future<?> future = executorService.submit(runnable);
+// ... some additional logic ...
+future.get();
+The span will be named calculateTax.
toString() method
+It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous
+instance of those classes. You can’t annotate such classes thus to override that, if there is no @SpanName annotation present,
+we’re checking if the class has a custom implementation of the toString() method.
So executing such code:
+Runnable runnable = new TraceRunnable(tracer, spanNamer, new Runnable() {
+ @Override public void run() {
+ // perform logic
+ }
+
+ @Override public String toString() {
+ return "calculateTax";
+ }
+});
+Future<?> future = executorService.submit(runnable);
+// ... some additional logic ...
+future.get();
+will lead in creating a span named calculateTax.
Customizations
+Thanks to the SpanInjector and SpanExtractor you can customize the way spans
+are created and propagated.
There are currently two built-in ways to pass tracing information between processes:
+-
+
-
+
via Spring Integration
+
+ -
+
via HTTP
+
+
Span ids are extracted from Zipkin-compatible (B3) headers (either Message
+or HTTP headers), to start or join an existing trace. Trace information is
+injected into any outbound requests so the next hop can extract them.
Spring Integration
+For Spring Integration these are the beans responsible for creation of a Span from a Message
+ and filling in the MessageBuilder with tracing information.
@Bean
+public SpanExtractor<Message> messagingSpanExtractor() {
+ ...
+}
+
+@Bean
+public SpanInjector<MessageBuilder> messagingSpanInjector() {
+ ...
+}
+You can override them by providing your own implementation and by adding a @Primary annotation
+to your bean definition.
HTTP
+For HTTP these are the beans responsible for creation of a Span from a HttpServletRequest
+ and filling in the HttpServletResponse with tracing information.
@Bean
+public SpanExtractor<HttpServletRequest> httpServletRequestSpanExtractor() {
+ ...
+}
+
+@Bean
+public SpanInjector<HttpServletResponse> httpServletResponseSpanInjector() {
+ ...
+}
+You can override them by providing your own implementation and by adding a @Primary annotation
+to your bean definition.
Example
+Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names +you have
+-
+
-
+
for trace id -
+correlationId
+ -
+
for span id -
+mySpanId
+
This is a an example of a SpanExtractor
static class CustomHttpServletRequestSpanExtractor
+ implements SpanExtractor<HttpServletRequest> {
+
+ @Override
+ public Span joinTrace(HttpServletRequest carrier) {
+ long traceId = Span.hexToId(carrier.getHeader("correlationId"));
+ long spanId = Span.hexToId(carrier.getHeader("mySpanId"));
+ // extract all necessary headers
+ Span.SpanBuilder builder = Span.builder().traceId(traceId).spanId(spanId);
+ // build rest of the Span
+ return builder.build();
+ }
+}
+The following SpanInjector could be created
static class CustomHttpServletResponseSpanInjector
+ implements SpanInjector<HttpServletResponse> {
+
+ @Override
+ public void inject(Span span, HttpServletResponse carrier) {
+ carrier.addHeader("correlationId", Span.idToHex(span.getTraceId()));
+ carrier.addHeader("mySpanId", Span.idToHex(span.getSpanId()));
+ // inject the rest of Span values to the header
+ }
+}
+And you could register them like this:
+@Bean
+@Primary
+SpanExtractor<HttpServletRequest> customHttpServletRequestSpanExtractor() {
+ return new CustomHttpServletRequestSpanExtractor();
+}
+
+@Bean
+@Primary
+SpanInjector<HttpServletResponse> customHttpServletResponseSpanInjector() {
+ return new CustomHttpServletResponseSpanInjector();
+}
+Span Data as Messages
+You can accumulate and send span data over
+Spring Cloud Stream by
+including the spring-cloud-sleuth-stream jar as a dependency, and
+adding a Channel Binder implementation
+(e.g. spring-cloud-starter-stream-rabbit for RabbitMQ or
+spring-cloud-starter-stream-kafka for Kafka). This will
+automatically turn your app into a producer of messages with payload
+type Spans.
Zipkin Consumer
+There is a special convenience annotation for setting up a message consumer
+for the Span data and pushing it into a Zipkin SpanStore. This application
@SpringBootApplication
+@EnableZipkinStreamServer
+public class Consumer {
+ public static void main(String[] args) {
+ SpringApplication.run(Consumer.class, args);
+ }
+}
+will listen for the Span data on whatever transport you provide via a
+Spring Cloud Stream Binder (e.g. include
+spring-cloud-starter-stream-rabbit for RabbitMQ, and similar
+starters exist for Redis and Kafka). If you add the following UI dependency
<groupId>io.zipkin.java</groupId>
+<artifactId>zipkin-autoconfigure-ui</artifactId>
+Then you’ll have your app a +Zipkin server, which hosts +the UI and api on port 9411.
+The default SpanStore is in-memory (good for demos and getting
+started quickly). For a more robust solution you can add MySQL and
+spring-boot-starter-jdbc to your classpath and enable the JDBC
+SpanStore via configuration, e.g.:
spring:
+ rabbitmq:
+ host: ${RABBIT_HOST:localhost}
+ datasource:
+ schema: classpath:/mysql.sql
+ url: jdbc:mysql://${MYSQL_HOST:localhost}/test
+ username: root
+ password: root
+# Switch this on to create the schema on startup:
+ initialize: true
+ continueOnError: true
+ sleuth:
+ enabled: false
+zipkin:
+ storage:
+ type: mysql
+|
+ Note
+ |
+
+The @EnableZipkinStreamServer is also annotated with
+@EnableZipkinServer so the process will also expose the standard
+Zipkin server endpoints for collecting spans over HTTP, and for
+querying in the Zipkin Web UI.
+ |
+
Custom Consumer
+A custom consumer can also easily be implemented using
+spring-cloud-sleuth-stream and binding to the SleuthSink. Example:
@EnableBinding(SleuthSink.class)
+@SpringBootApplication(exclude = SleuthStreamAutoConfiguration.class)
+@MessageEndpoint
+public class Consumer {
+
+ @ServiceActivator(inputChannel = SleuthSink.INPUT)
+ public void sink(Spans input) throws Exception {
+ // ... process spans
+ }
+}
+|
+ Note
+ |
+
+the sample consumer application above explicitly excludes
+SleuthStreamAutoConfiguration so it doesn’t send messages to itself,
+but this is optional (you might actually want to trace requests into
+the consumer app).
+ |
+
Metrics
+Currently Spring Cloud Sleuth registers very simple metrics related to spans. +It’s using the Spring Boot’s metrics support +to calculate the number of accepted and dropped spans. Each time a span gets +sent to Zipkin the number of accepted spans will increase. If there’s an error then +the number of dropped spans will get increased.
+Integrations
+Runnable and Callable
+If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative.
Example for Runnable:
Runnable runnable = new Runnable() {
+ @Override
+ public void run() {
+ // do some work
+ }
+
+ @Override
+ public String toString() {
+ return "spanNameFromToStringMethod";
+ }
+};
+// Manual `TraceRunnable` creation with explicit "calculateTax" Span name
+Runnable traceRunnable = new TraceRunnable(tracer, spanNamer, runnable, "calculateTax");
+// Wrapping `Runnable` with `Tracer`. The Span name will be taken either from the
+// `@SpanName` annotation or from `toString` method
+Runnable traceRunnableFromTracer = tracer.wrap(runnable);
+Example for Callable:
Callable<String> callable = new Callable<String>() {
+ @Override
+ public String call() throws Exception {
+ return someLogic();
+ }
+
+ @Override
+ public String toString() {
+ return "spanNameFromToStringMethod";
+ }
+};
+// Manual `TraceCallable` creation with explicit "calculateTax" Span name
+Callable<String> traceCallable = new TraceCallable<>(tracer, spanNamer, callable, "calculateTax");
+// Wrapping `Callable` with `Tracer`. The Span name will be taken either from the
+// `@SpanName` annotation or from `toString` method
+Callable<String> traceCallableFromTracer = tracer.wrap(callable);
+That way you will ensure that a new Span is created and closed for each execution.
+Hystrix
+Custom Concurrency Strategy
+We’re registering a custom HystrixConcurrencyStrategy
+that wraps all Callable instances into their Sleuth representative -
+the TraceCallable. The strategy either starts or continues a span depending on the fact whether tracing was already going
+on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false.
Manual Command setting
+Assuming that you have the following HystrixCommand:
HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
+ @Override
+ protected String run() throws Exception {
+ return someLogic();
+ }
+};
+In order to pass the tracing information you have to wrap the same logic in the Sleuth version of the HystrixCommand which is the
+TraceCommand:
TraceCommand<String> traceCommand = new TraceCommand<String>(tracer, traceKeys, setter) {
+ @Override
+ public String doRun() throws Exception {
+ return someLogic();
+ }
+};
+RxJava
+We’re registering a custom RxJavaSchedulersHook
+that wraps all Action0 instances into their Sleuth representative -
+the TraceAction. The hook either starts or continues a span depending on the fact whether tracing was already going
+on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the spring.sleuth.rxjava.schedulers.hook.enabled to false.
You can define a list of regular expressions for thread names, for which you don’t want a Span to be created. Just provide a comma separated list
+of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.
HTTP integration
+Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false.
HTTP Filter
+Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which
+ the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would
+ like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then
+ its value of contextPath gets appended to the provided skip pattern.
HandlerInterceptor
+Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an
+ existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. The
+ TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. If the
+ the TraceFilter doesn’t see this attribute set it will create a "fallback" span which is an additional
+ span created on the server side so that the trace is presented properly in the UI. Seeing that most likely
+ signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.
Async Servlet support
+If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one.
HTTP client integration
+Synchronous Rest Template
+We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a
+call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous RestTemplate features
+just set spring.sleuth.web.client.enabled to false.
|
+ Important
+ |
+
+You have to register RestTemplate as a bean so that the interceptors will get injected.
+If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work.
+ |
+
Asynchronous Rest Template
+Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the ClientHttpRequestFactory
+and the AsyncClientHttpRequestFactory by registering your beans. Remember to use tracing compatible implementations (e.g. don’t forget to
+wrap ThreadPoolTaskScheduler in a TraceAsyncListenableTaskExecutor). Example of custom request factories:
Unresolved directive in spring-cloud-sleuth.adoc - include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/web/client/TraceWebAsyncClientAutoConfigurationTest.java[tags=async_template_factories,indent=0]
+To block the AsyncRestTemplate features set spring.sleuth.web.async.client.enabled to false.
+To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper set spring.sleuth.web.async.client.factory.enabled
+to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false.
Feign
+By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely
+by setting spring.sleuth.feign.enabled to false. If you do so then no Feign related instrumentation will take place.
Part of Feign instrumentation is done via a FeignBeanPostProcessor. You can disable it by providing the spring.sleuth.feign.processor.enabled equal to false.
+If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation
+however will be still there.
Asynchronous communication
+@Async annotated methods
+In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads.
+You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false.
If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics:
-
+
-
+
the Span name will be the annotated method name
+
+ -
+
the Span will be tagged with that method’s class name and the method name too
+
+
@Scheduled annotated methods
+In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour
+by setting the value of spring.sleuth.scheduled.enabled to false.
If you annotate your method with @Scheduled then we’ll automatically create a new Span with the following characteristics:
-
+
-
+
the Span name will be the annotated method name
+
+ -
+
the Span will be tagged with that method’s class name and the method name too
+
+
If you want to skip Span creation for some @Scheduled annotated classes you can set the
+spring.sleuth.scheduled.skipPattern with a regular expression that will match the fully qualified name of the
+@Scheduled annotated class.
Executor, ExecutorService and ScheduledExecutorService
+We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations
+are creating Spans each time a new task is submitted, invoked or scheduled.
Here you can see an example of how to pass tracing information with TraceableExecutorService when working with CompletableFuture:
CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> {
+ // perform some logic
+ return 1_000_000L;
+}, new TraceableExecutorService(executorService,
+ // 'calculateTax' explicitly names the span - this param is optional
+ tracer, traceKeys, spanNamer, "calculateTax"));
+Messaging
+Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and
+subscribe events. To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false.
Spring Cloud Sleuth up till version 1.0.4 is sending invalid tracing headers when using messaging. Those headers are actually
+the same as the ones sent in HTTP (they contain a -) in its name. For the sake of
+backwards compatibility in 1.0.4 we’ve started sending both valid and invalid headers. Please upgrade to 1.0.4 because
+in Spring Cloud Sleuth 1.1 we will remove the support for the deprecated headers.
Since 1.0.4 you can provide the spring.sleuth.integration.patterns pattern to explicitly
+provide the names of channels that you want to include for tracing. By default all channels
+are included.
Zuul
+We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data).
+To disable Zuul support set the spring.sleuth.zuul.enabled property to false.
Running examples
+You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links:
+