:github-tag: master
:github-repo: spring-cloud/spring-cloud-sleuth
:github-raw: https://raw.githubusercontent.com/{github-repo}/{github-tag}
:github-code: https://github.com/{github-repo}/tree/{github-tag}
:toc: left
:toclevels: 8
:nofooter:
:sectlinks: true

Spring Cloud Sleuth
====================
Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer

*{spring-cloud-version}*

:doctype: book

include::intro.adoc[]

include::features.adoc[]

== Sampling

In distributed tracing the data volumes can be very high so sampling
can be important (you usually don't need to export all spans to get a
good picture of what is happening). Spring Cloud Sleuth has a
`Sampler` strategy that you can implement to take control of the
sampling algorithm. Samplers do not stop span (correlation) ids from
being generated, but they do prevent the tags and events being
attached and exported. By default you get a strategy that continues to
trace if a span is already active, but new ones are always marked as
non-exportable. If all your apps run with this sampler you will see
traces in logs, but not in any remote store. For testing the default
is often enough, and it probably is all you need if you are only using
the logs (e.g. with an ELK aggregator). If you are exporting span data
to Zipkin or Spring Cloud Stream, there is also an `AlwaysSampler`
that exports everything and a `PercentageBasedSampler` that samples a
fixed fraction of spans.

NOTE: the `PercentageBasedSampler` is the default if you are using
`spring-cloud-sleuth-zipkin` or `spring-cloud-sleuth-stream`. You can
configure the exports using `spring.sleuth.sampler.percentage`. The passed
value needs to be a double from `0.0` to `1.0` so it's not a percentage.
For backwards compatibility reasons we're not changing the property name.

A sampler can be installed just by creating a bean definition, e.g:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=always_sampler,indent=0]
----

TIP: You can set the HTTP header `X-B3-Flags` to `1` or when doing messaging you can
set `spanFlags` header to `1`. Then the current span will be forced to be exportable
regardless of the sampling decision.

== Instrumentation

Spring Cloud Sleuth instruments all your Spring application
automatically, so you shouldn't have to do anything to activate
it. The instrumentation is added using a variety of technologies
according to the stack that is available, e.g. for a servlet web
application we use a `Filter`, and for Spring Integration we use
`ChannelInterceptors`.

You can customize the keys used in span tags. To limit the volume of
span data, by default an HTTP request will be tagged only with a
handful of metadata like the status code, host and URL. You can add
request headers by configuring `spring.sleuth.keys.http.headers` (a
list of header names).

NOTE: Remember that tags are only collected and exported if there is a
`Sampler` that allows it (by default there is not, so there is no
danger of accidentally collecting too much data without configuring
something).

NOTE: Currently the instrumentation in Spring Cloud Sleuth is eager - it means that
we're actively trying to pass the tracing context between threads. Also timing events
are captured even when sleuth isn't exporting data to a tracing system.
This approach may change in the future towards being lazy on this matter.

== Span lifecycle

You can do the following operations on the Span by means of *org.springframework.cloud.sleuth.Tracer* interface:

- <<creating-and-closing-spans, start>> - when you start a span its name is assigned and start timestamp is recorded.
- <<creating-and-closing-spans, close>> - the span gets finished (the end time of the span is recorded) and if
the span is *exportable* then it will be eligible for collection to Zipkin.
The span is also removed from the current thread.
- <<continuing-spans, continue>> - a new instance of span will be created whereas it will be a copy of the
one that it continues.
- <<continuing-spans, detach>> - the span doesn't get stopped or closed. It only gets removed from the current thread.
- <<creating-spans-with-explicit-parent, create with explicit parent>> - you can create a new span and set an explicit parent to it

TIP: Spring creates the instance of `Tracer` for you. In order to use it all you need is to just autowire it.

=== Creating and closing spans [[creating-and-closing-spans]]

You can manually create spans by using the *Tracer* interface.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=manual_span_creation,indent=0]
----

In this example we could see how to create a new instance of span. Assuming that there already
was a span present in this thread then it would become the parent of that span.

IMPORTANT: Always clean after you create a span! Don't forget to close a span if you want to send it to Zipkin.

IMPORTANT: If your span contains a name greater than 50 chars, then that name will
be truncated to 50 chars. Your names have to be explicit and concrete. Big names lead to
latency issues and sometimes even thrown exceptions.

=== Continuing spans [[continuing-spans]]

Sometimes you don't want to create a new span but you want to continue one. Example of such a
situation might be (of course it all depends on the use-case):

    - *AOP* - If there was already a span created before an aspect was reached then you might not want to create a new span.
    - *Hystrix* - executing a Hystrix command is most likely a logical part of the current processing. It's in fact
    only a technical implementation detail that you wouldn't necessarily want to reflect in tracing as a separate being.

The continued instance of span is equal to the one that it continues:
[source,java]
----
Span continuedSpan = this.tracer.continueSpan(spanToContinue);
assertThat(continuedSpan).isEqualTo(spanToContinue);
----

To continue a span you can use the *Tracer* interface.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=manual_span_continuation,indent=0]
----

IMPORTANT: Always clean after you create a span! Don't forget to detach a span if some work was done started in one
 thread (e.g. thread X) and it's waiting for other threads (e.g. Y, Z) to finish.
 Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected
 the span in thread X should be closed.

=== Creating spans with an explicit parent [[creating-spans-with-explicit-parent]]

There is a possibility that you want to start a new span and provide an explicit parent of that span.
Let's assume that the parent of a span is in one thread and you want to start a new span in another thread. The
`startSpan` method of the `Tracer` interface is the method you are looking for.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=manual_span_joining,indent=0]
----

IMPORTANT: After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs
 related to the fact that you have a span present in the current thread other than the one you're trying to close.
 What's worse your spans won't get closed properly thus will not get collected to Zipkin.

== Naming spans

Picking a span name is not a trivial task. Span name should depict an operation name. The name should
be low cardinality (e.g. not include identifiers).

Since there is a lot of instrumentation going on some of the span names will be
artificial like:

- `controller-method-name` when received by a Controller with a method name `conrollerMethodName`
- `async` for asynchronous operations done via wrapped `Callable` and `Runnable`.
- `@Scheduled` annotated methods will return the simple name of the class.

Fortunately, for the asynchronous processing you can provide explicit naming.

=== @SpanName annotation

You can name the span explicitly via the `@SpanName` annotation.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=span_name_annotation,indent=0]
----

In this case, when processed in the following manner:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=span_name_annotated_runnable_execution,indent=0]
----

The span will be named `calculateTax`.

=== toString() method

It's pretty rare to create separate classes for `Runnable` or `Callable`. Typically one creates an anonymous
instance of those classes. You can't annotate such classes thus to override that, if there is no `@SpanName` annotation present,
we're checking if the class has a custom implementation of the `toString()` method.

So executing such code:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=span_name_to_string_runnable_execution,indent=0]
----

will lead in creating a span named `calculateTax`.

== Managing spans with annotations

=== Rationale

The main arguments for this features are

* api-agnostic means to collaborate with a span
    - use of annotations allows users to add to a span with no library dependency on a span api.
    This allows Sleuth to change its core api less impact to user code.
* reduced surface area for basic span operations.
    - without this feature one has to use the span api, which has lifecycle commands that
    could be used incorrectly. By only exposing scope, tag and log functionality, users can
    collaborate without accidentally breaking span lifecycle.
* collaboration with runtime generated code
    - with libraries such as Spring Data / Feign the implementations of interfaces are generated
    at runtime thus span wrapping of objects was tedious. Now you can provide annotations
     over interfaces and arguments of those interfaces

=== Creating new spans

If you really don't want to take care of creating local spans manually you can profit from the
`@NewSpan` annotation. Also we give you the `@SpanTag` annotation to add tags in an automated
fashion.

Let's look at some examples of usage.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SleuthSpanCreatorAspectTests.java[tags=annotated_method,indent=0]
----

Annotating the method without any parameter will lead to a creation of a new span whose name
will be equal to annotated method name.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SleuthSpanCreatorAspectTests.java[tags=custom_name_on_annotated_method,indent=0]
----

If you provide the value in the annotation (either directly or via the `name` parameter) then
the created span will have the name as the provided value.

[source,java]
----
// method declaration
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SleuthSpanCreatorAspectTests.java[tags=custom_name_and_tag_on_annotated_method,indent=0]

// and method execution
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SleuthSpanCreatorAspectTests.java[tags=execution,indent=0]
----

You can combine both the name and a tag. Let's focus on the latter. In this case whatever the value of
the annotated method's parameter runtime value will be - that will be the value of the tag. In our sample
the tag key will be `testTag` and the tag value will be `test`.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SleuthSpanCreatorAspectTests.java[tags=name_on_implementation,indent=0]
----

You can place the `@NewSpan` annotation on both the class and an interface. If you override the
interface's method and provide a different value of the `@NewSpan` annotation then the most
concrete one wins (in this case `customNameOnTestMethod3` will be set).

=== Continuing spans

If you want to just add tags and annotations to an existing span it's enough
to use the `@ContinueSpan` annotation as presented below. Note that in contrast
with the `@NewSpan` annotation you can also add logs via the `log` parameter:

[source,java]
----
// method declaration
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SleuthSpanCreatorAspectTests.java[tags=continue_span,indent=0]

// method execution
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SleuthSpanCreatorAspectTests.java[tags=continue_span_execution,indent=0]
----

That way the span will get continued and:

 - logs with name `testMethod11.before` and `testMethod11.after` will be created
 - if an exception will be thrown a log `testMethod11.afterFailure` will also be created
 - tag with key `testTag11` and value `test` will be created

=== More advanced tag setting

There are 3 different ways to add tags to a span. All of them are controlled by the `SpanTag` annotation.
Precedence is:

- try with the bean of `TagValueResolver` type and provided name
- if one hasn't provided the bean name, try to evaluate an expression. We're searching for a `TagValueExpressionResolver` bean.
The default implementation uses SPEL expression resolution.
- if one hasn't provided any expression to evaluate just return a `toString()` value of the parameter

==== Custom extractor

The value of the tag for following method will be computed by an implementation of `TagValueResolver` interface.
Its class name has to be passed as the value of the `resolver` attribute.

Having such an annotated method:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SpanTagAnnotationHandlerTests.java[tags=resolver_bean,indent=0]
----

and such a `TagValueResolver` bean implementation

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SpanTagAnnotationHandlerTests.java[tags=custom_resolver,indent=0]
----

Will lead to setting of a tag value equal to `Value from myCustomTagValueResolver`.

==== Resolving expressions for value

Having such an annotated method:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SpanTagAnnotationHandlerTests.java[tags=spel,indent=0]
----

and no custom implementation of a `TagValueExpressionResolver` will lead to evaluation of the SPEL expression and a tag with value `4 characters` will be set on the span.
If you want to use some other expression resolution mechanism you can create your own implementation
of the bean.

==== Using toString method

Having such an annotated method:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/annotation/SpanTagAnnotationHandlerTests.java[tags=toString,indent=0]
----

if executed with a value of `15` will lead to setting of a tag with a String value of `"15"`.

== Customizations

Thanks to the `SpanInjector` and `SpanExtractor` you can customize the way spans
are created and propagated.

There are currently two built-in ways to pass tracing information between processes:

 * via Spring Integration
 * via HTTP

Span ids are extracted from Zipkin-compatible (B3) headers (either `Message`
or HTTP headers), to start or join an existing trace. Trace information is
injected into any outbound requests so the next hop can extract them.

The key change in comparison to the previous versions of Sleuth is that Sleuth is implementing
the Open Tracing's `TextMap` notion. In Sleuth it's called `SpanTextMap`. Basically the idea
is that any means of communication (e.g. message, http request, etc.) can be abstracted via
a `SpanTextMap`. This abstraction defines how one can insert data into the carrier and
how to retrieve it from there. Thanks to this if you want to instrument a new HTTP library
that uses a `FooRequest` as a mean of sending HTTP requests then you have to create an
implementation of  a `SpanTextMap` that delegates calls to `FooRequest` in terms of retrieval
and insertion of HTTP headers.

=== Spring Integration

For Spring Integration there are 2 interfaces responsible for creation of a Span from a `Message`.
These are:

- `MessagingSpanTextMapExtractor`
- `MessagingSpanTextMapInjector`

You can override them by providing your own implementation.

=== HTTP

For HTTP there are 2 interfaces responsible for creation of a Span from a `Message`.
These are:

- `HttpSpanExtractor`
- `HttpSpanInjector`

You can override them by providing your own implementation.

=== Example

Let's assume that instead of the standard Zipkin compatible tracing HTTP header names
you have

* for trace id - `correlationId`
* for span id - `mySpanId`

This is a an example of a `SpanExtractor`

[source,java]
----
include::../../../..//spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/web/TraceFilterCustomExtractorTests.java[tags=extractor,indent=0]
----

And you could register it like this:

[source,java]
----
include::../../../..//spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/web/TraceFilterCustomExtractorTests.java[tags=configuration,indent=0]
----

Spring Cloud Sleuth does not add trace/span related headers to the Http Response for security reasons. If you need the headers then a custom `SpanInjector`
that injects the headers into the Http Response and a Servlet filter which makes use of this can be added the following way:

[source,java]
----
include::../../../..//spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/web/TraceCustomFilterResponseInjectorTests.java[tags=injector,indent=0]
----

And you could register them like this:
[source,java]
----
include::../../../..//spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/web/TraceCustomFilterResponseInjectorTests.java[tags=configuration,indent=0]
----

=== TraceFilter

You can also modify the behaviour of the `TraceFilter` - the component that is responsible
for processing the input HTTP request and adding tags basing on the HTTP response. You can customize
the tags, or modify the response headers by registering your own instance of the `TraceFilter` bean.

In the following example we will register the `TraceFilter` bean and we will add the
`ZIPKIN-TRACE-ID` response header containing the current Span's trace id. Also we will
add to the Span a tag with key `custom` and a value `tag`.

[source,java]
----
include::../../../..//spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/web/TraceFilterIntegrationTests.java[tags=response_headers,indent=0]
----

=== Custom SA tag in Zipkin

Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented.
What you can do is to create a span with the `peer.service` tag that will contain a value of the service that you want to call.
Below you can see an example of a call to Redis that is wrapped in such a span.

[source,java]
----
include::../../../..//spring-cloud-sleuth-zipkin/src/test/java/org/springframework/cloud/sleuth/zipkin/HttpZipkinSpanReporterTest.java[tags=service_name,indent=0]
----

IMPORTANT: Remember not to add both `peer.service` tag and the `SA` tag! You have to add only `peer.service`.

=== Custom service name

By default Sleuth assumes that when you send a span to Zipkin, you want the span's service name
 to be equal to `spring.application.name` value. That's not always the case though. There
 are situations in which you want to explicitly provide a different service name for all spans coming
 from your application. To achieve that it's enough to just pass the following property
 to your application to override that value (example for `foo` service name):

[source,yaml]
----
spring.zipkin.service.name: foo
----

=== Customization of reported spans

Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way.
 You can achieve that by using the `SpanAdjuster` interface.

Example of usage:

In Sleuth we're generating spans with a fixed name. Some users want to modify the name depending on values
of tags. Implementation of the `SpanAdjuster` interface can be used to alter that name. Example:

[source,yaml]
----
@Bean
SpanAdjuster customSpanAdjuster() {
    return span -> span.toBuilder().name(scrub(span.getName())).build();
}
----

This will lead in changing the name of the reported span just before it gets sent to Zipkin.

IMPORTANT: Your `SpanReporter` should inject the `SpanAdjuster` and
 allow span manipulation before the actual reporting is done.

=== Host locator

In order to define the host that is corresponding to a particular span we need to resolve the host name
and port. The default approach is to take it from server properties. If those for some reason are not set
then we're trying to retrieve the host name from the network interfaces.

If you have the discovery client enabled and prefer to retrieve the host address from the registered
instance in a service registry then you have to set the property (it's applicable for both HTTP and
Stream based span reporting).

[source,yaml]
----
spring.zipkin.locator.discovery.enabled: true
----

== Sending spans to Zipkin

By default if you add `spring-cloud-starter-zipkin` as a dependency to your project,
when the span is closed, it will be sent to Zipkin over HTTP. The communication
is asynchronous. You can configure the URL by setting the `spring.zipkin.baseUrl`
property as follows:

[source,yaml]
----
spring.zipkin.baseUrl: http://192.168.99.100:9411/
----

If you want to find Zipkin via service discovery it's enough to pass the
Zipkin's service id inside the URL (example for `zipkinserver` service id)

[source,yaml]
----
spring.zipkin.baseUrl: http://zipkinserver/
----

== Span Data as Messages

You can accumulate and send span data over
http://cloud.spring.io/spring-cloud-stream[Spring Cloud Stream] by
including the `spring-cloud-sleuth-stream` jar as a dependency, and
adding a Channel Binder implementation
(e.g. `spring-cloud-starter-stream-rabbit` for RabbitMQ or
`spring-cloud-starter-stream-kafka` for Kafka). This will
automatically turn your app into a producer of messages with payload
type `Spans`.

=== Zipkin Consumer

There is a special convenience annotation for setting up a message consumer
for the Span data and pushing it into a Zipkin `SpanStore`. This application

[source,java]
----
include::../../../../spring-cloud-sleuth-zipkin-stream/src/test/java/org/springframework/cloud/sleuth/zipkin/stream/documentation/Consumer.java[tags=zipkin_consumer,indent=0]
----

will listen for the Span data on whatever transport you provide via a
Spring Cloud Stream `Binder` (e.g. include
`spring-cloud-starter-stream-rabbit` for RabbitMQ, and similar
starters exist for Redis and Kafka). If you add the following UI dependency

[source,xml]
----
<groupId>io.zipkin.java</groupId>
<artifactId>zipkin-autoconfigure-ui</artifactId>
----

Then you'll have your app a
https://github.com/openzipkin/zipkin[Zipkin server], which hosts
the UI and api on port 9411.


The default `SpanStore` is in-memory (good for demos and getting
started quickly). For a more robust solution you can add MySQL and
`spring-boot-starter-jdbc` to your classpath and enable the JDBC
`SpanStore` via configuration, e.g.:

[source,yaml]
----
spring:
  rabbitmq:
    host: ${RABBIT_HOST:localhost}
  datasource:
    schema: classpath:/mysql.sql
    url: jdbc:mysql://${MYSQL_HOST:localhost}/test
    username: root
    password: root
# Switch this on to create the schema on startup:
    initialize: true
    continueOnError: true
  sleuth:
    enabled: false
zipkin:
  storage:
    type: mysql
----

NOTE: The `@EnableZipkinStreamServer` is also annotated with
`@EnableZipkinServer` so the process will also expose the standard
Zipkin server endpoints for collecting spans over HTTP, and for
querying in the Zipkin Web UI.

=== Custom Consumer

A custom consumer can also easily be implemented using
`spring-cloud-sleuth-stream` and binding to the `SleuthSink`. Example:

[source,java]
----
@EnableBinding(SleuthSink.class)
@SpringBootApplication(exclude = SleuthStreamAutoConfiguration.class)
@MessageEndpoint
public class Consumer {

    @ServiceActivator(inputChannel = SleuthSink.INPUT)
    public void sink(Spans input) throws Exception {
        // ... process spans
    }
}
----

NOTE: the sample consumer application above explicitly excludes
`SleuthStreamAutoConfiguration` so it doesn't send messages to itself,
but this is optional (you might actually want to trace requests into
the consumer app).

In order to customize the polling mechanism you can create a bean of `PollerMetadata` type
with name equal to `StreamSpanReporter.POLLER`. Here you can find an example of such a configuration.

[source,java]
----
include::../../../../spring-cloud-sleuth-stream/src/test/java/org/springframework/cloud/sleuth/stream/SleuthStreamAutoConfigurationTest.java[tags=custom_poller,indent=0]
----

== Metrics

Currently Spring Cloud Sleuth registers very simple metrics related to spans.
It's using the http://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html#production-ready-recording-metrics[Spring Boot's metrics support]
to calculate the number of accepted and dropped spans. Each time a span gets
sent to Zipkin the number of accepted spans will increase. If there's an error then
the number of dropped spans will get increased.

== Integrations

=== Runnable and Callable

If you're wrapping your logic in `Runnable` or `Callable` it's enough to wrap those classes in their Sleuth representative.

Example for `Runnable`:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=trace_runnable,indent=0]
----

Example for `Callable`:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/documentation/SpringCloudSleuthDocTests.java[tags=trace_callable,indent=0]
----

That way you will ensure that a new Span is created and closed for each execution.

=== Hystrix

==== Custom Concurrency Strategy

We're registering a custom https://github.com/Netflix/Hystrix/wiki/Plugins#concurrencystrategy[`HystrixConcurrencyStrategy`]
that wraps all `Callable` instances into their Sleuth representative -
the `TraceCallable`. The strategy either starts or continues a span depending on the fact whether tracing was already going
on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the `spring.sleuth.hystrix.strategy.enabled` to `false`.

==== Manual Command setting

Assuming that you have the following `HystrixCommand`:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/hystrix/TraceCommandTests.java[tags=hystrix_command,indent=0]
----

In order to pass the tracing information you have to wrap the same logic in the Sleuth version of the `HystrixCommand` which is the
`TraceCommand`:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/hystrix/TraceCommandTests.java[tags=trace_hystrix_command,indent=0]
----

=== RxJava

We're registering a custom https://github.com/ReactiveX/RxJava/wiki/Plugins#rxjavaschedulershook[`RxJavaSchedulersHook`]
that wraps all `Action0` instances into their Sleuth representative -
the `TraceAction`. The hook either starts or continues a span depending on the fact whether tracing was already going
on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the `spring.sleuth.rxjava.schedulers.hook.enabled` to `false`.

You can define a list of regular expressions for thread names, for which you don't want a Span to be created. Just provide a comma separated list
of regular expressions in the `spring.sleuth.rxjava.schedulers.ignoredthreads` property.

=== HTTP integration

Features from this section can be disabled by providing the `spring.sleuth.web.enabled` property with value equal to `false`.

==== HTTP Filter

Via the `TraceFilter` all sampled incoming requests result in creation of a Span. That Span's name is `http:` + the path to which
 the request was sent. E.g. if the request was sent to `/foo/bar` then the name will be `http:/foo/bar`. You can configure which URIs you would
 like to skip via the `spring.sleuth.web.skipPattern` property. If you have `ManagementServerProperties` on classpath then
 its value of `contextPath` gets appended to the provided skip pattern.

==== HandlerInterceptor

Since we want the span names to be precise we're using a `TraceHandlerInterceptor` that either wraps an
 existing `HandlerInterceptor` or is added directly to the list of existing `HandlerInterceptors`. The
 `TraceHandlerInterceptor` adds a special request attribute to the given `HttpServletRequest`. If the
 the `TraceFilter` doesn't see this attribute set it will create a "fallback" span which is an additional
 span created on the server side so that the trace is presented properly in the UI. Seeing that most likely
 signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.

==== Async Servlet support

If your controller returns a `Callable` or a `WebAsyncTask` Spring Cloud Sleuth will continue the existing span instead of creating a new one.

==== WebFlux support

Via the `TraceWebFilter` all sampled incoming requests result in creation of a Span. That Span's name is `http:` + the path to which
 the request was sent. E.g. if the request was sent to `/foo/bar` then the name will be `http:/foo/bar`. You can configure which URIs you would
 like to skip via the `spring.sleuth.web.skipPattern` property. If you have `ManagementServerProperties` on classpath then
 its value of `contextPath` gets appended to the provided skip pattern.


=== HTTP client integration

==== Synchronous Rest Template

We're injecting a `RestTemplate` interceptor that ensures that all the tracing information is passed to the requests. Each time a
call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous `RestTemplate` features
just set `spring.sleuth.web.client.enabled` to `false`.

IMPORTANT: You have to register `RestTemplate` as a bean so that the interceptors will get injected.
If you create a `RestTemplate` instance with a `new` keyword then the instrumentation WILL NOT work.

==== Asynchronous Rest Template

IMPORTANT: A traced version of an `AsyncRestTemplate` bean is registered for you out of the box. If you
have your own bean you have to wrap it in a `TraceAsyncRestTemplate` representation. The best solution
is to only customize the `ClientHttpRequestFactory` and / or `AsyncClientHttpRequestFactory`.
*If you have your own `AsyncRestTemplate` and you don't wrap it your calls WILL NOT GET TRACED*.

Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the `ClientHttpRequestFactory`
and the `AsyncClientHttpRequestFactory` by registering your beans. Remember to use tracing compatible implementations (e.g. don't forget to
wrap `ThreadPoolTaskScheduler` in a `TraceAsyncListenableTaskExecutor`). Example of custom request factories:

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/web/client/TraceWebAsyncClientAutoConfigurationTests.java[tags=async_template_factories,indent=0]
----

To block the `AsyncRestTemplate` features set `spring.sleuth.web.async.client.enabled` to `false`.
To disable creation of the default `TraceAsyncClientHttpRequestFactoryWrapper` set `spring.sleuth.web.async.client.factory.enabled`
to `false`. If you don't want to create `AsyncRestClient` at all set `spring.sleuth.web.async.client.template.enabled` to `false`.

===== Multiple Asynchronous Rest Templates

Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you
can see an example of how to set up such a custom `AsyncRestTemplate`.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/async/MultipleAsyncRestTemplateTests.java[tags=custom_async_rest_template,indent=0]
----

==== WebClient

We inject a `ExchangeFilterFunction` implementation that creates a span and via on success and on
error callbacks takes care of closing client side spans.

IMPORTANT: You have to register `WebClient` as a bean so that the tracing instrumention gets applied.
If you create a `WebClient` instance with a `new` keyword then the instrumentation WILL NOT work.

==== Traverson

If you're using the http://docs.spring.io/spring-hateoas/docs/current/reference/html/#client.traverson[Traverson] library
it's enough for you to inject a `RestTemplate` as a bean into your Traverson object. Since `RestTemplate`
is already intercepted, you will get full support of tracing in your client. Below you can find a pseudo code
of how to do that:

[source,java]
----
@Autowired RestTemplate restTemplate;

Traverson traverson = new Traverson(URI.create("http://some/address"),
    MediaType.APPLICATION_JSON, MediaType.APPLICATION_JSON_UTF8).setRestOperations(restTemplate);
// use Traverson
----

=== Feign

By default Spring Cloud Sleuth provides integration with feign via the `TraceFeignClientAutoConfiguration`. You can disable it entirely
by setting `spring.sleuth.feign.enabled` to false. If you do so then no Feign related instrumentation will take place.

Part of Feign instrumentation is done via a `FeignBeanPostProcessor`. You can disable it by providing the `spring.sleuth.feign.processor.enabled` equal to `false`.
If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation
however will be still there.

=== Asynchronous communication

==== @Async annotated methods

In Spring Cloud Sleuth we're instrumenting async related components so that the tracing information is passed between threads.
You can disable this behaviour by setting the value of `spring.sleuth.async.enabled` to `false`.

If you annotate your method with `@Async` then we'll automatically create a new Span with the following characteristics:

    - if the method is annotated with `@SpanName` then the value of the annotation will be the Span's name
    - if the method is *not* annotated with `@SpanName` the Span name will be the annotated method name
    - the Span will be tagged with that method's class name and the method name too

==== @Scheduled annotated methods

In Spring Cloud Sleuth we're instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour
by setting the value of `spring.sleuth.scheduled.enabled` to `false`.

If you annotate your method with `@Scheduled` then we'll automatically create a new Span with the following characteristics:

    - the Span name will be the annotated method name
    - the Span will be tagged with that method's class name and the method name too

If you want to skip Span creation for some `@Scheduled` annotated classes you can set the
`spring.sleuth.scheduled.skipPattern` with a regular expression that will match the fully qualified name of the
`@Scheduled` annotated class.

TIP: If you are using `spring-cloud-sleuth-stream` and `spring-cloud-netflix-hystrix-stream` together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting `spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask`

==== Executor, ExecutorService and ScheduledExecutorService

We're providing `LazyTraceExecutor`, `TraceableExecutorService` and `TraceableScheduledExecutorService`. Those implementations
are creating Spans each time a new task is submitted, invoked or scheduled.

Here you can see an example of how to pass tracing information with `TraceableExecutorService` when working with `CompletableFuture`:

[source,java]
----

include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/async/TraceableExecutorServiceTests.java[tags=completablefuture,indent=0]
----

===== Customization of Executors

Sometimes you need to set up a custom instance of the `AsyncExecutor`. In the following snippet you
can see an example of how to set up such a custom `Executor`.

[source,java]
----
include::../../../../spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/async/MultipleAsyncRestTemplateTests.java[tags=custom_executor,indent=0]
----

=== Messaging

Spring Cloud Sleuth integrates with http://projects.spring.io/spring-integration/[Spring Integration]. It creates spans for publish and
subscribe events. To disable Spring Integration instrumentation, set `spring.sleuth.integration.enabled` to false.

You can provide the `spring.sleuth.integration.patterns` pattern to explicitly
provide the names of channels that you want to include for tracing. By default all channels
are included.

IMPORTANT: When using the `Executor` to build a Spring Integration `IntegrationFlow` remember to use the *untraced* version of the `Executor`.
Decorating Spring Integration Executor Channel with `TraceableExecutorService` will cause the spans to be improperly closed.

=== Zuul

We're registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data).
To disable Zuul support set the `spring.sleuth.zuul.enabled` property to `false`.

== Running examples

You can find the running examples deployed in the https://run.pivotal.io/[Pivotal Web Services]. Check them out in the following links:

- http://docssleuth-zipkin-server.cfapps.io/[Zipkin for apps presented in the samples to the top]
- http://docsbrewing-zipkin-server.cfapps.io/[Zipkin for Brewery on PWS], its https://github.com/spring-cloud-samples/brewery[Github Code]
