diff --git a/checkstyle-cachefile b/checkstyle-cachefile index c203c9d25..846e315b1 100644 --- a/checkstyle-cachefile +++ b/checkstyle-cachefile @@ -1,2 +1,2 @@ -#Tue Aug 29 10:25:58 CEST 2017 +#Tue Aug 29 10:48:25 CEST 2017 configuration*?=AAF64E6F43093085564EA8A1303772CC40B65829 diff --git a/multi/_additional_resources.html b/multi/_additional_resources.html index 5a0ead44d..47ee6b3d1 100644 --- a/multi/_additional_resources.html +++ b/multi/_additional_resources.html @@ -1,3 +1,3 @@ - 1. Additional resources
1. Additional resources
Prev   Next

1. Additional resources

Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin

click here to see the video

Prev   Next
 Home 2. Features
\ No newline at end of file + 2. Additional resources
2. Additional resources
Prev   Next

2. Additional resources

Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin

click here to see the video

Prev   Next
1. Introduction Home 3. Features
\ No newline at end of file diff --git a/multi/_customizations.html b/multi/_customizations.html index bd820bf2d..4ec2a18fb 100644 --- a/multi/_customizations.html +++ b/multi/_customizations.html @@ -1,6 +1,6 @@ - 8. Customizations
8. Customizations
Prev   Next

8. Customizations

Thanks to the SpanInjector and SpanExtractor you can customize the way spans + 9. Customizations

9. Customizations
Prev   Next

9. Customizations

Thanks to the SpanInjector and SpanExtractor you can customize the way spans are created and propagated.

There are currently two built-in ways to pass tracing information between processes:

  • via Spring Integration
  • via HTTP

Span ids are extracted from Zipkin-compatible (B3) headers (either Message or HTTP headers), to start or join an existing trace. Trace information is injected into any outbound requests so the next hop can extract them.

The key change in comparison to the previous versions of Sleuth is that Sleuth is implementing @@ -10,9 +10,9 @@ a SpanTextMap. This abstraction defines how one can how to retrieve it from there. Thanks to this if you want to instrument a new HTTP library that uses a FooRequest as a mean of sending HTTP requests then you have to create an implementation of a SpanTextMap that delegates calls to FooRequest in terms of retrieval -and insertion of HTTP headers.

8.1 Spring Integration

For Spring Integration there are 2 interfaces responsible for creation of a Span from a Message. -These are:

  • MessagingSpanTextMapExtractor
  • MessagingSpanTextMapInjector

You can override them by providing your own implementation.

8.2 HTTP

For HTTP there are 2 interfaces responsible for creation of a Span from a Message. -These are:

  • HttpSpanExtractor
  • HttpSpanInjector

You can override them by providing your own implementation.

8.3 Example

Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names +and insertion of HTTP headers.

9.1 Spring Integration

For Spring Integration there are 2 interfaces responsible for creation of a Span from a Message. +These are:

  • MessagingSpanTextMapExtractor
  • MessagingSpanTextMapInjector

You can override them by providing your own implementation.

9.2 HTTP

For HTTP there are 2 interfaces responsible for creation of a Span from a Message. +These are:

  • HttpSpanExtractor
  • HttpSpanInjector

You can override them by providing your own implementation.

9.3 Example

Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names you have

  • for trace id - correlationId
  • for span id - mySpanId

This is a an example of a SpanExtractor

static class CustomHttpSpanExtractor implements HttpSpanExtractor {
 
 	@Override public Span joinTrace(SpanTextMap carrier) {
@@ -99,7 +99,7 @@ that injects the headers into the Http Response and a Servlet filter which makes
 @Bean
 HttpResponseInjectingTraceFilter responseInjectingTraceFilter(Tracer tracer) {
 	return new HttpResponseInjectingTraceFilter(tracer, customHttpServletResponseSpanInjector());
-}

8.4 TraceFilter

You can also modify the behaviour of the TraceFilter - the component that is responsible +}

9.4 TraceFilter

You can also modify the behaviour of the TraceFilter - the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response. You can customize the tags, or modify the response headers by registering your own instance of the TraceFilter bean.

In the following example we will register the TraceFilter bean and we will add the ZIPKIN-TRACE-ID response header containing the current Span’s trace id. Also we will @@ -117,7 +117,7 @@ TraceFilter myTraceFilter(BeanFactory beanFactory, "custom", "tag"); } }; -}

8.5 Custom SA tag in Zipkin

Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented. +}

9.5 Custom SA tag in Zipkin

Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented. What you can do is to create a span with the peer.service tag that will contain a value of the service that you want to call. Below you can see an example of a call to Redis that is wrapped in such a span.

org.springframework.cloud.sleuth.Span newSpan = tracer.createSpan("redis");
 try {
@@ -132,18 +132,18 @@ Below you can see an example of a call to Redis that is wrapped in such a span.<
 	newSpan.tag("peer.port", "1234");
 	newSpan.logEvent(org.springframework.cloud.sleuth.Span.CLIENT_RECV);
 	tracer.close(newSpan);
-}
[Important]Important

Remember not to add both peer.service tag and the SA tag! You have to add only peer.service.

8.6 Custom service name

By default Sleuth assumes that when you send a span to Zipkin, you want the span’s service name +}

[Important]Important

Remember not to add both peer.service tag and the SA tag! You have to add only peer.service.

9.6 Custom service name

By default Sleuth assumes that when you send a span to Zipkin, you want the span’s service name to be equal to spring.application.name value. That’s not always the case though. There are situations in which you want to explicitly provide a different service name for all spans coming from your application. To achieve that it’s enough to just pass the following property - to your application to override that value (example for foo service name):

spring.zipkin.service.name: foo

8.7 Customization of reported spans

Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way. + to your application to override that value (example for foo service name):

spring.zipkin.service.name: foo

9.7 Customization of reported spans

Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way. You can achieve that by using the SpanAdjuster interface.

Example of usage:

In Sleuth we’re generating spans with a fixed name. Some users want to modify the name depending on values of tags. Implementation of the SpanAdjuster interface can be used to alter that name. Example:

@Bean
 SpanAdjuster customSpanAdjuster() {
     return span -> span.toBuilder().name(scrub(span.getName())).build();
 }

This will lead in changing the name of the reported span just before it gets sent to Zipkin.

[Important]Important

Your SpanReporter should inject the SpanAdjuster and - allow span manipulation before the actual reporting is done.

8.8 Host locator

In order to define the host that is corresponding to a particular span we need to resolve the host name + allow span manipulation before the actual reporting is done.

9.8 Host locator

In order to define the host that is corresponding to a particular span we need to resolve the host name and port. The default approach is to take it from server properties. If those for some reason are not set then we’re trying to retrieve the host name from the network interfaces.

If you have the discovery client enabled and prefer to retrieve the host address from the registered instance in a service registry then you have to set the property (it’s applicable for both HTTP and -Stream based span reporting).

spring.zipkin.locator.discovery.enabled: true
Prev   Next
7. Managing spans with annotations Home 9. Sending spans to Zipkin
\ No newline at end of file +Stream based span reporting).

spring.zipkin.locator.discovery.enabled: true
Prev   Next
8. Managing spans with annotations Home 10. Sending spans to Zipkin
\ No newline at end of file diff --git a/multi/_features.html b/multi/_features.html index ae8e7958a..525a7015e 100644 --- a/multi/_features.html +++ b/multi/_features.html @@ -1,6 +1,6 @@ - 2. Features
2. Features
Prev   Next

2. Features

Prev   Next
2. Additional resources Home 4. Sampling
\ No newline at end of file diff --git a/multi/_instrumentation.html b/multi/_instrumentation.html index 305c20e42..38b699079 100644 --- a/multi/_instrumentation.html +++ b/multi/_instrumentation.html @@ -1,6 +1,6 @@ - 4. Instrumentation
4. Instrumentation
Prev   Next

4. Instrumentation

Spring Cloud Sleuth instruments all your Spring application + 5. Instrumentation

5. Instrumentation
Prev   Next

5. Instrumentation

Spring Cloud Sleuth instruments all your Spring application automatically, so you shouldn’t have to do anything to activate it. The instrumentation is added using a variety of technologies according to the stack that is available, e.g. for a servlet web @@ -15,4 +15,4 @@ danger of accidentally collecting too much data without configuring something).

[Note]Note

Currently the instrumentation in Spring Cloud Sleuth is eager - it means that we’re actively trying to pass the tracing context between threads. Also timing events are captured even when sleuth isn’t exporting data to a tracing system. -This approach may change in the future towards being lazy on this matter.

Prev   Next
3. Sampling Home 5. Span lifecycle
\ No newline at end of file +This approach may change in the future towards being lazy on this matter.

Prev   Next
4. Sampling Home 6. Span lifecycle
\ No newline at end of file diff --git a/multi/_integrations.html b/multi/_integrations.html index 97d3c9d5f..fa555c12e 100644 --- a/multi/_integrations.html +++ b/multi/_integrations.html @@ -1,6 +1,6 @@ - 12. Integrations
12. Integrations
Prev   Next

12. Integrations

12.1 Runnable and Callable

If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative.

Example for Runnable:

Runnable runnable = new Runnable() {
+   13. Integrations
13. Integrations
Prev   Next
13. Integrations
13.1 Runnable and Callable
If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative.
Example for Runnable:
Runnable runnable = new Runnable() {
 	@Override
 	public void run() {
 		// do some work
@@ -30,10 +30,10 @@ Runnable traceRunnableFromTracer = tracer.wrap(runnable);
Example for new TraceCallable<>(tracer, spanNamer, callable, "calculateTax");
 // Wrapping `Callable` with `Tracer`. The Span name will be taken either from the
 // `@SpanName` annotation or from `toString` method
-Callable<String> traceCallableFromTracer = tracer.wrap(callable);

That way you will ensure that a new Span is created and closed for each execution.

12.2 Hystrix

Custom Concurrency Strategy

We’re registering a custom HystrixConcurrencyStrategy +Callable<String> traceCallableFromTracer = tracer.wrap(callable);

That way you will ensure that a new Span is created and closed for each execution.

13.2 Hystrix

13.2.1 Custom Concurrency Strategy

We’re registering a custom HystrixConcurrencyStrategy that wraps all Callable instances into their Sleuth representative - the TraceCallable. The strategy either starts or continues a span depending on the fact whether tracing was already going -on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false.

Manual Command setting

Assuming that you have the following HystrixCommand:

HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
+on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false.

13.2.2 Manual Command setting

Assuming that you have the following HystrixCommand:

HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
 	@Override
 	protected String run() throws Exception {
 		return someLogic();
@@ -44,22 +44,22 @@ on before the Hystrix command was called. To disable the custom Hystrix Concurre
 	public String doRun() throws Exception {
 		return someLogic();
 	}
-};

12.3 RxJava

We’re registering a custom RxJavaSchedulersHook +};

13.3 RxJava

We’re registering a custom RxJavaSchedulersHook that wraps all Action0 instances into their Sleuth representative - the TraceAction. The hook either starts or continues a span depending on the fact whether tracing was already going on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the spring.sleuth.rxjava.schedulers.hook.enabled to false.

You can define a list of regular expressions for thread names, for which you don’t want a Span to be created. Just provide a comma separated list -of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

12.4 HTTP integration

Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false.

HTTP Filter

Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which +of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

13.4 HTTP integration

Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false.

13.4.1 HTTP Filter

Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern.

HandlerInterceptor

Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an + its value of contextPath gets appended to the provided skip pattern.

13.4.2 HandlerInterceptor

Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. The TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. If the the TraceFilter doesn’t see this attribute set it will create a "fallback" span which is an additional span created on the server side so that the trace is presented properly in the UI. Seeing that most likely - signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.

Async Servlet support

If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one.

12.5 HTTP client integration

Synchronous Rest Template

We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a + signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.

13.4.3 Async Servlet support

If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one.

13.5 HTTP client integration

13.5.1 Synchronous Rest Template

We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous RestTemplate features just set spring.sleuth.web.client.enabled to false.

[Important]Important

You have to register RestTemplate as a bean so that the interceptors will get injected. -If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work.

Asynchronous Rest Template

[Important]Important

A traced version of an AsyncRestTemplate bean is registered for you out of the box. If you +If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work.

13.5.2 Asynchronous Rest Template

[Important]Important

A traced version of an AsyncRestTemplate bean is registered for you out of the box. If you have your own bean you have to wrap it in a TraceAsyncRestTemplate representation. The best solution is to only customize the ClientHttpRequestFactory and / or AsyncClientHttpRequestFactory. If you have your own AsyncRestTemplate and you don’t wrap it your calls WILL NOT GET TRACED.

Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the ClientHttpRequestFactory @@ -79,7 +79,7 @@ wrap ThreadPoolTaskScheduler in a AsyncRestTemplate features set spring.sleuth.web.async.client.enabled to false. To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper set spring.sleuth.web.async.client.factory.enabled -to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false.

Multiple Asynchronous Rest Templates

Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you +to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false.

Multiple Asynchronous Rest Templates

Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you can see an example of how to set up such a custom AsyncRestTemplate.

@Configuration
 @EnableAutoConfiguration
 static class Config {
@@ -113,27 +113,27 @@ can see an example of how to set up such a custom AsyncRes
 		//CUSTOMIZE HERE
 		return factory;
 	}
-}

Traverson

If you’re using the Traverson library +}

13.5.3 Traverson

If you’re using the Traverson library it’s enough for you to inject a RestTemplate as a bean into your Traverson object. Since RestTemplate is already intercepted, you will get full support of tracing in your client. Below you can find a pseudo code of how to do that:

@Autowired RestTemplate restTemplate;
 
 Traverson traverson = new Traverson(URI.create("http://some/address"),
     MediaType.APPLICATION_JSON, MediaType.APPLICATION_JSON_UTF8).setRestOperations(restTemplate);
-// use Traverson

12.6 Feign

By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely +// use Traverson

13.6 Feign

By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely by setting spring.sleuth.feign.enabled to false. If you do so then no Feign related instrumentation will take place.

Part of Feign instrumentation is done via a FeignBeanPostProcessor. You can disable it by providing the spring.sleuth.feign.processor.enabled equal to false. If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation -however will be still there.

12.7 Asynchronous communication

@Async annotated methods

In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads. -You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false.

If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics:

  • if the method is annotated with @SpanName then the value of the annotation will be the Span’s name
  • if the method is not annotated with @SpanName the Span name will be the annotated method name
  • the Span will be tagged with that method’s class name and the method name too

@Scheduled annotated methods

In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour +however will be still there.

13.7 Asynchronous communication

13.7.1 @Async annotated methods

In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads. +You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false.

If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics:

  • if the method is annotated with @SpanName then the value of the annotation will be the Span’s name
  • if the method is not annotated with @SpanName the Span name will be the annotated method name
  • the Span will be tagged with that method’s class name and the method name too

13.7.2 @Scheduled annotated methods

In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour by setting the value of spring.sleuth.scheduled.enabled to false.

If you annotate your method with @Scheduled then we’ll automatically create a new Span with the following characteristics:

  • the Span name will be the annotated method name
  • the Span will be tagged with that method’s class name and the method name too

If you want to skip Span creation for some @Scheduled annotated classes you can set the spring.sleuth.scheduled.skipPattern with a regular expression that will match the fully qualified name of the -@Scheduled annotated class.

[Tip]Tip

If you are using spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask

Executor, ExecutorService and ScheduledExecutorService

We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations +@Scheduled annotated class.

[Tip]Tip

If you are using spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask

13.7.3 Executor, ExecutorService and ScheduledExecutorService

We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations are creating Spans each time a new task is submitted, invoked or scheduled.

Here you can see an example of how to pass tracing information with TraceableExecutorService when working with CompletableFuture:

CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> {
 	// perform some logic
 	return 1_000_000L;
 }, new TraceableExecutorService(executorService,
 		// 'calculateTax' explicitly names the span - this param is optional
-		tracer, traceKeys, spanNamer, "calculateTax"));
Customization of Executors

Sometimes you need to set up a custom instance of the AsyncExecutor. In the following snippet you + tracer, traceKeys, spanNamer, "calculateTax"));

Customization of Executors

Sometimes you need to set up a custom instance of the AsyncExecutor. In the following snippet you can see an example of how to set up such a custom Executor.

@Configuration
 @EnableAutoConfiguration
 @EnableAsync
@@ -152,9 +152,9 @@ can see an example of how to set up such a custom Executor
 		executor.initialize();
 		return new LazyTraceExecutor(this.beanFactory, executor);
 	}
-}

12.8 Messaging

Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and +}

13.8 Messaging

Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and subscribe events. To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false.

You can provide the spring.sleuth.integration.patterns pattern to explicitly provide the names of channels that you want to include for tracing. By default all channels are included.

[Important]Important

When using the Executor to build a Spring Integration IntegrationFlow remember to use the untraced version of the Executor. -Decorating Spring Integration Executor Channel with TraceableExecutorService will cause the spans to be improperly closed.

12.9 Zuul

We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data). -To disable Zuul support set the spring.sleuth.zuul.enabled property to false.

Prev   Next
11. Metrics Home 13. Running examples
\ No newline at end of file +Decorating Spring Integration Executor Channel with TraceableExecutorService will cause the spans to be improperly closed.

13.9 Zuul

We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data). +To disable Zuul support set the spring.sleuth.zuul.enabled property to false.

Prev   Next
12. Metrics Home 14. Running examples
\ No newline at end of file diff --git a/multi/_introduction.html b/multi/_introduction.html new file mode 100644 index 000000000..db262d7a0 --- /dev/null +++ b/multi/_introduction.html @@ -0,0 +1,297 @@ + + + 1. Introduction
1. Introduction
Prev   Next

1. Introduction

Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.

1.1 Terminology

Spring Cloud Sleuth borrows Dapper’s terminology.

Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an +RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span +is a part of. Spans also have other data, such as descriptions, timestamped events, key-value +annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address).

Spans are started and stopped, and they keep track of their timing information. Once you create a +span, you must stop it at some point in the future.

[Tip]Tip

The initial span that starts a trace is called a root span. The value of span id +of that span is equal to trace id.

Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed +big-data store, a trace might be formed by a put request.

Annotation: is used to record existence of an event in time. Some of the core annotations used to define +the start and stop of a request are:

  • cs - Client Sent - The client has made a request. This annotation depicts the start of the span.
  • sr - Server Received - The server side got the request and will start processing it. +If one subtracts the cs timestamp from this timestamp one will receive the network latency.
  • ss - Server Sent - Annotated upon completion of request processing (when the response +got sent back to the client). If one subtracts the sr timestamp from this timestamp one +will receive the time needed by the server side to process the request.
  • cr - Client Received - Signifies the end of the span. The client has successfully received the +response from the server side. If one subtracts the cs timestamp from this timestamp one +will receive the whole time needed by the client to receive the response from the server.

Visualization of what Span and Trace will look in a system together with the Zipkin annotations:

Trace Info propagation

Each color of a note signifies a span (7 spans - from A to G). If you have such information in the note:

Trace Id = X
+Span Id = D
+Client Sent

That means that the current span has Trace-Id set to X, Span-Id set to D. It also has emitted + Client Sent event.

This is how the visualization of the parent / child relationship of spans would look like:

Parent child relationship

1.2 Purpose

In the following sections the example from the image above will be taken into consideration.

1.2.1 Distributed tracing with Zipkin

Altogether there are 7 spans . If you go to traces in Zipkin you will see this number in the second trace:

Traces

However if you pick a particular trace then you will see 4 spans:

Traces Info propagation
[Note]Note

When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to +Zipkin with Server Received and Server Sent / Client Received and Client Sent +annotations then they will presented as a single span.

Why is there a difference between the 7 and 4 spans in this case?

  • 2 spans come from http:/start span. It has the Server Received (SR) and Server Sent (SS) annotations.
  • 2 spans come from the RPC call from service1 to service2 to the http:/foo endpoint. It has the Client Sent (CS) +and Client Received (CR) annotations on service1 side. It also has Server Received (SR) and Server Sent (SS) annotations +on the service2 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
  • 2 spans come from the RPC call from service2 to service3 to the http:/bar endpoint. It has the Client Sent (CS) +and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations +on the service3 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
  • 2 spans come from the RPC call from service2 to service4 to the http:/baz endpoint. It has the Client Sent (CS) +and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations +on the service4 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.

So if we count the physical spans we have 1 from http:/start, 2 from service1 calling service2, 2 form service2 +calling service3 and 2 from service2 calling service4. Altogether 7 spans.

Logically we see the information of Total Spans: 4 because we have 1 span related to the incoming request +to service1 and 3 spans related to RPC calls.

1.2.2 Visualizing errors

Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn’t caught then we’re +setting proper tags on the span which Zipkin can properly colorize. You could see in the list of traces one + trace that was in red color. That’s because there was an exception thrown.

If you click that trace then you’ll see a similar picture

Error Traces

Then if you click on one of the spans you’ll see the following

Error Traces Info propagation

As you can see you can easily see the reason for an error and the whole stacktrace related to it.

1.2.3 Live examples

Figure 1.1. Click Pivotal Web Services icon to see it live!

Zipkin deployed on Pivotal Web Services

The dependency graph in Zipkin would look like this:

Dependencies

Figure 1.2. Click Pivotal Web Services icon to see it live!

Zipkin deployed on Pivotal Web Services

1.2.4 Log correlation

When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following:

service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
+service2.log:2016-02-26 11:15:47.710  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Hello from service2. Calling service3 and then service4
+service3.log:2016-02-26 11:15:47.895  INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application   : Hello from service3
+service2.log:2016-02-26 11:15:47.924  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service3 [Hello from service3]
+service4.log:2016-02-26 11:15:48.134  INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application   : Hello from service4
+service2.log:2016-02-26 11:15:48.156  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service4 [Hello from service4]
+service1.log:2016-02-26 11:15:48.182  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]

If you’re using a log aggregating tool like Kibana, +Splunk etc. you can order the events that took place. An example of +Kibana would look like this:

Log correlation with Kibana

If you want to use Logstash here is the Grok pattern for Logstash:

filter {
+       # pattern matching logback pattern
+       grok {
+              match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
+       }
+}
[Note]Note

If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern:

filter {
+       # pattern matching logback pattern
+       grok {
+              match => { "message" => "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
+       }
+}

JSON Logback with Logstash

Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability +we’re passing the dependencies in the groupId:artifactId:version notation.

Dependencies setup

  • Ensure that Logback is on the classpath (ch.qos.logback:logback-core)
  • Add Logstash Logback encode - example for version 4.6 : net.logstash.logback:logstash-logback-encoder:4.6

Logback setup

Below you can find an example of a Logback configuration (file named logback-spring.xml) that:

  • logs information from the application in a JSON format to a build/${spring.application.name}.json file
  • has commented out two additional appenders - console and standard log file
  • has the same logging pattern as the one presented in the previous section
<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+	<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
+	​
+	<springProperty scope="context" name="springAppName" source="spring.application.name"/>
+	<!-- Example for logging into the build folder of your project -->
+	<property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${springAppName}"/>​
+
+	<!-- You can override this to have a custom pattern -->
+	<property name="CONSOLE_LOG_PATTERN"
+			  value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>
+
+	<!-- Appender to log to console -->
+	<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
+		<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
+			<!-- Minimum logging level to be presented in the console logs-->
+			<level>DEBUG</level>
+		</filter>
+		<encoder>
+			<pattern>${CONSOLE_LOG_PATTERN}</pattern>
+			<charset>utf8</charset>
+		</encoder>
+	</appender>
+
+	<!-- Appender to log to file -->​
+	<appender name="flatfile" class="ch.qos.logback.core.rolling.RollingFileAppender">
+		<file>${LOG_FILE}</file>
+		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+			<fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.gz</fileNamePattern>
+			<maxHistory>7</maxHistory>
+		</rollingPolicy>
+		<encoder>
+			<pattern>${CONSOLE_LOG_PATTERN}</pattern>
+			<charset>utf8</charset>
+		</encoder>
+	</appender>
+	​
+	<!-- Appender to log to file in a JSON format -->
+	<appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender">
+		<file>${LOG_FILE}.json</file>
+		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
+			<fileNamePattern>${LOG_FILE}.json.%d{yyyy-MM-dd}.gz</fileNamePattern>
+			<maxHistory>7</maxHistory>
+		</rollingPolicy>
+		<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
+			<providers>
+				<timestamp>
+					<timeZone>UTC</timeZone>
+				</timestamp>
+				<pattern>
+					<pattern>
+						{
+						"severity": "%level",
+						"service": "${springAppName:-}",
+						"trace": "%X{X-B3-TraceId:-}",
+						"span": "%X{X-B3-SpanId:-}",
+						"parent": "%X{X-B3-ParentSpanId:-}",
+						"exportable": "%X{X-Span-Export:-}",
+						"pid": "${PID:-}",
+						"thread": "%thread",
+						"class": "%logger{40}",
+						"rest": "%message"
+						}
+					</pattern>
+				</pattern>
+			</providers>
+		</encoder>
+	</appender>
+	​
+	<root level="INFO">
+		<appender-ref ref="console"/>
+		<!-- uncomment this to have also JSON logs -->
+		<!--<appender-ref ref="logstash"/>-->
+		<!--<appender-ref ref="flatfile"/>-->
+	</root>
+</configuration>
[Note]Note

If you’re using a custom logback-spring.xml then you have to pass the spring.application.name in +bootstrap instead of application property file. Otherwise your custom logback file won’t read the property properly.

1.2.5 Propagating Span Context

The span context is the state that must get propagated to any child Spans across process boundaries. +Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context. +Baggage is an optional part.

Baggage is a set of key:value pairs stored in the span context. Baggage travels together with the trace +and is attached to every span. Spring Cloud Sleuth will understand that a header is baggage related if the HTTP + header is prefixed with baggage- and for messaging it starts with baggage_.

[Important]Important

There’s currently no limitation of the count or size of baggage items. However, keep in mind that +too many can decrease system throughput or increase RPC latency. In extreme cases, it could crash the app due +to exceeding transport-level message or header capacity.

Example of setting baggage on a span:

Span initialSpan = this.tracer.createSpan("span");
+initialSpan.setBaggageItem("foo", "bar");
+initialSpan.setBaggageItem("UPPER_CASE", "someValue");

Baggage vs. Span Tags

Baggage travels with the trace (i.e. every child span contains the baggage of its parent). Zipkin has no knowledge of +baggage and will not even receive that information.

Tags are attached to a specific span - they are presented for that particular span only. However you +can search by tag to find the trace, where there exists a span having the searched tag value.

If you want to be able to lookup a span based on baggage, you should add corresponding entry as a tag in the root span.

@Autowired Tracer tracer;
+
+Span span = tracer.getCurrentSpan();
+String baggageKey = "key";
+String baggageValue = "foo";
+span.setBaggageItem(baggageKey, baggageValue);
+tracer.addTag(baggageKey, baggageValue);

1.3 Adding to the project

1.3.1 Only Sleuth (log correlation)

If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add +the spring-cloud-starter-sleuth module to your project.

Maven.  +

<dependencyManagement> 1
+         <dependencies>
+             <dependency>
+                 <groupId>org.springframework.cloud</groupId>
+                 <artifactId>spring-cloud-dependencies</artifactId>
+                 <version>${release.train.version}</version>
+                 <type>pom</type>
+                 <scope>import</scope>
+             </dependency>
+         </dependencies>
+   </dependencyManagement>
+
+   <dependency> 2
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-starter-sleuth</artifactId>
+   </dependency>

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-starter-sleuth

Gradle.  +

dependencyManagement { 1
+    imports {
+        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
+    }
+}
+
+dependencies { 2
+    compile "org.springframework.cloud:spring-cloud-starter-sleuth"
+}

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-starter-sleuth

1.3.2 Sleuth with Zipkin via HTTP

If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency.

Maven.  +

<dependencyManagement> 1
+         <dependencies>
+             <dependency>
+                 <groupId>org.springframework.cloud</groupId>
+                 <artifactId>spring-cloud-dependencies</artifactId>
+                 <version>${release.train.version}</version>
+                 <type>pom</type>
+                 <scope>import</scope>
+             </dependency>
+         </dependencies>
+   </dependencyManagement>
+
+   <dependency> 2
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-starter-zipkin</artifactId>
+   </dependency>

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin

Gradle.  +

dependencyManagement { 1
+    imports {
+        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
+    }
+}
+
+dependencies { 2
+    compile "org.springframework.cloud:spring-cloud-starter-zipkin"
+}

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin

1.3.3 Sleuth with Zipkin via Spring Cloud Stream

If you want both Sleuth and Zipkin just add the spring-cloud-sleuth-stream dependency.

Maven.  +

<dependencyManagement> 1
+         <dependencies>
+             <dependency>
+                 <groupId>org.springframework.cloud</groupId>
+                 <artifactId>spring-cloud-dependencies</artifactId>
+                 <version>${release.train.version}</version>
+                 <type>pom</type>
+                 <scope>import</scope>
+             </dependency>
+         </dependencies>
+   </dependencyManagement>
+
+   <dependency> 2
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-sleuth-stream</artifactId>
+   </dependency>
+   <dependency> 3
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-starter-sleuth</artifactId>
+   </dependency>
+   <!-- EXAMPLE FOR RABBIT BINDING -->
+   <dependency> 4
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-stream-binder-rabbit</artifactId>
+   </dependency>

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-sleuth-stream

3

Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

4

Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

Gradle.  +

dependencyManagement { 1
+    imports {
+        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
+    }
+}
+
+dependencies {
+    compile "org.springframework.cloud:spring-cloud-sleuth-stream" 2
+    compile "org.springframework.cloud:spring-cloud-starter-sleuth" 3
+    // Example for Rabbit binding
+    compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" 4
+}

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-sleuth-stream

3

Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

4

Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

1.3.4 Spring Cloud Sleuth Stream Zipkin Collector

If you want to start a Spring Cloud Sleuth Stream Zipkin collector just add the spring-cloud-sleuth-zipkin-stream +dependency

Maven.  +

<dependencyManagement> 1
+         <dependencies>
+             <dependency>
+                 <groupId>org.springframework.cloud</groupId>
+                 <artifactId>spring-cloud-dependencies</artifactId>
+                 <version>${release.train.version}</version>
+                 <type>pom</type>
+                 <scope>import</scope>
+             </dependency>
+         </dependencies>
+   </dependencyManagement>
+
+   <dependency> 2
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-sleuth-zipkin-stream</artifactId>
+   </dependency>
+   <dependency> 3
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-starter-sleuth</artifactId>
+   </dependency>
+   <!-- EXAMPLE FOR RABBIT BINDING -->
+   <dependency> 4
+       <groupId>org.springframework.cloud</groupId>
+       <artifactId>spring-cloud-stream-binder-rabbit</artifactId>
+   </dependency>

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-sleuth-zipkin-stream

3

Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

4

Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

Gradle.  +

dependencyManagement { 1
+    imports {
+        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
+    }
+}
+
+dependencies {
+    compile "org.springframework.cloud:spring-cloud-sleuth-zipkin-stream" 2
+    compile "org.springframework.cloud:spring-cloud-starter-sleuth" 3
+    // Example for Rabbit binding
+    compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" 4
+}

+

1

In order not to pick versions by yourself it’s much better if you add the dependency management via +the Spring BOM

2

Add the dependency to spring-cloud-sleuth-zipkin-stream

3

Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

4

Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

and then just annotate your main class with @EnableZipkinStreamServer annotation:

package example;
+
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.cloud.sleuth.zipkin.stream.EnableZipkinStreamServer;
+
+@SpringBootApplication
+@EnableZipkinStreamServer
+public class ZipkinStreamServerApplication {
+
+	public static void main(String[] args) throws Exception {
+		SpringApplication.run(ZipkinStreamServerApplication.class, args);
+	}
+
+}
Prev   Next
 Home 2. Additional resources
\ No newline at end of file diff --git a/multi/_managing_spans_with_annotations.html b/multi/_managing_spans_with_annotations.html index 83594c458..9ab04a4e4 100644 --- a/multi/_managing_spans_with_annotations.html +++ b/multi/_managing_spans_with_annotations.html @@ -1,11 +1,11 @@ - 7. Managing spans with annotations
7. Managing spans with annotations
Prev   Next

7. Managing spans with annotations

7.1 Rationale

The main arguments for this features are

  • api-agnostic means to collaborate with a span

    • use of annotations allows users to add to a span with no library dependency on a span api. + 8. Managing spans with annotations
      8. Managing spans with annotations
      Prev   Next

      8. Managing spans with annotations

      8.1 Rationale

      The main arguments for this features are

      • api-agnostic means to collaborate with a span

        • use of annotations allows users to add to a span with no library dependency on a span api. This allows Sleuth to change its core api less impact to user code.

      • reduced surface area for basic span operations.

        • without this feature one has to use the span api, which has lifecycle commands that could be used incorrectly. By only exposing scope, tag and log functionality, users can collaborate without accidentally breaking span lifecycle.

      • collaboration with runtime generated code

        • with libraries such as Spring Data / Feign the implementations of interfaces are generated at runtime thus span wrapping of objects was tedious. Now you can provide annotations - over interfaces and arguments of those interfaces

      7.2 Creating new spans

      If you really don’t want to take care of creating local spans manually you can profit from the + over interfaces and arguments of those interfaces

8.2 Creating new spans

If you really don’t want to take care of creating local spans manually you can profit from the @NewSpan annotation. Also we give you the @SpanTag annotation to add tags in an automated fashion.

Let’s look at some examples of usage.

@NewSpan
 void testMethod();

Annotating the method without any parameter will lead to a creation of a new span whose name @@ -23,25 +23,25 @@ the tag key will be testTag and the tag value will public void testMethod3() { }

You can place the @NewSpan annotation on both the class and an interface. If you override the interface’s method and provide a different value of the @NewSpan annotation then the most -concrete one wins (in this case customNameOnTestMethod3 will be set).

7.3 Continuing spans

If you want to just add tags and annotations to an existing span it’s enough +concrete one wins (in this case customNameOnTestMethod3 will be set).

8.3 Continuing spans

If you want to just add tags and annotations to an existing span it’s enough to use the @ContinueSpan annotation as presented below. Note that in contrast with the @NewSpan annotation you can also add logs via the log parameter:

// method declaration
 @ContinueSpan(log = "testMethod11")
 void testMethod11(@SpanTag("testTag11") String param);
 
 // method execution
-this.testBean.testMethod11("test");

That way the span will get continued and:

  • logs with name testMethod11.before and testMethod11.after will be created
  • if an exception will be thrown a log testMethod11.afterFailure will also be created
  • tag with key testTag11 and value test will be created

7.4 More advanced tag setting

There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation. +this.testBean.testMethod11("test");

That way the span will get continued and:

  • logs with name testMethod11.before and testMethod11.after will be created
  • if an exception will be thrown a log testMethod11.afterFailure will also be created
  • tag with key testTag11 and value test will be created

8.4 More advanced tag setting

There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation. Precedence is:

  • try with the bean of TagValueResolver type and provided name
  • if one hasn’t provided the bean name, try to evaluate an expression. We’re searching for a TagValueExpressionResolver bean. -The default implementation uses SPEL expression resolution.
  • if one hasn’t provided any expression to evaluate just return a toString() value of the parameter

Custom extractor

The value of the tag for following method will be computed by an implementation of TagValueResolver interface. +The default implementation uses SPEL expression resolution.

  • if one hasn’t provided any expression to evaluate just return a toString() value of the parameter

    • 8.4.1 Custom extractor

    The value of the tag for following method will be computed by an implementation of TagValueResolver interface. Its class name has to be passed as the value of the resolver attribute.

    Having such an annotated method:

    @NewSpan
 public void getAnnotationForTagValueResolver(@SpanTag(key = "test", resolver = TagValueResolver.class) String test) {
 }

    and such a TagValueResolver bean implementation

    @Bean(name = "myCustomTagValueResolver")
 public TagValueResolver tagValueResolver() {
 	return parameter -> "Value from myCustomTagValueResolver";
-}

    Will lead to setting of a tag value equal to Value from myCustomTagValueResolver.

    Resolving expressions for value

    Having such an annotated method:

    @NewSpan
+}

    Will lead to setting of a tag value equal to Value from myCustomTagValueResolver.

    8.4.2 Resolving expressions for value

    Having such an annotated method:

    @NewSpan
 public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "length() + ' characters'") String test) {
 }

    and no custom implementation of a TagValueExpressionResolver will lead to evaluation of the SPEL expression and a tag with value 4 characters will be set on the span. If you want to use some other expression resolution mechanism you can create your own implementation -of the bean.

    Using toString method

    Having such an annotated method:

    @NewSpan
+of the bean.

    8.4.3 Using toString method

    Having such an annotated method:

    @NewSpan
 public void getAnnotationForArgumentToString(@SpanTag("test") Long param) {
-}

    if executed with a value of 15 will lead to setting of a tag with a String value of "15".

    Prev   Next
    6. Naming spans Home 8. Customizations
    \ No newline at end of file +}

    if executed with a value of 15 will lead to setting of a tag with a String value of "15".

    Prev   Next
    7. Naming spans Home 9. Customizations
    \ No newline at end of file diff --git a/multi/_metrics.html b/multi/_metrics.html index dcd47ef03..58125728b 100644 --- a/multi/_metrics.html +++ b/multi/_metrics.html @@ -1,7 +1,7 @@ - 11. Metrics
    11. Metrics
    Prev   Next

    11. Metrics

    Currently Spring Cloud Sleuth registers very simple metrics related to spans. + 12. Metrics

    12. Metrics
    Prev   Next

    12. Metrics

    Currently Spring Cloud Sleuth registers very simple metrics related to spans. It’s using the Spring Boot’s metrics support to calculate the number of accepted and dropped spans. Each time a span gets sent to Zipkin the number of accepted spans will increase. If there’s an error then -the number of dropped spans will get increased.

    Prev   Next
    10. Span Data as Messages Home 12. Integrations
    \ No newline at end of file +the number of dropped spans will get increased.

    Prev   Next
    11. Span Data as Messages Home 13. Integrations
    \ No newline at end of file diff --git a/multi/_naming_spans.html b/multi/_naming_spans.html index f63a4e785..4a7e689ea 100644 --- a/multi/_naming_spans.html +++ b/multi/_naming_spans.html @@ -1,8 +1,8 @@ - 6. Naming spans
    6. Naming spans
    Prev   Next

    6. Naming spans

    Picking a span name is not a trivial task. Span name should depict an operation name. The name should + 7. Naming spans

    7. Naming spans
    Prev   Next

    7. Naming spans

    Picking a span name is not a trivial task. Span name should depict an operation name. The name should be low cardinality (e.g. not include identifiers).

    Since there is a lot of instrumentation going on some of the span names will be -artificial like:

    • controller-method-name when received by a Controller with a method name conrollerMethodName
    • async for asynchronous operations done via wrapped Callable and Runnable.
    • @Scheduled annotated methods will return the simple name of the class.

    Fortunately, for the asynchronous processing you can provide explicit naming.

    6.1 @SpanName annotation

    You can name the span explicitly via the @SpanName annotation.

    @SpanName("calculateTax")
+artificial like:
    • controller-method-name when received by a Controller with a method name conrollerMethodName
    • async for asynchronous operations done via wrapped Callable and Runnable.
    • @Scheduled annotated methods will return the simple name of the class.
    Fortunately, for the asynchronous processing you can provide explicit naming.
    7.1 @SpanName annotation
    You can name the span explicitly via the @SpanName annotation.
    @SpanName("calculateTax")
 class TaxCountingRunnable implements Runnable {
 
 	@Override public void run() {
@@ -11,7 +11,7 @@ artificial like:
      Runnable runnable = new TraceRunnable(tracer, spanNamer, new TaxCountingRunnable());
 Future<?> future = executorService.submit(runnable);
 // ... some additional logic ...
-future.get();
    The span will be named calculateTax.
    6.2 toString() method
    It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous
+future.get();

    The span will be named calculateTax.

    7.2 toString() method

    It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous instance of those classes. You can’t annotate such classes thus to override that, if there is no @SpanName annotation present, we’re checking if the class has a custom implementation of the toString() method.

    So executing such code:

    Runnable runnable = new TraceRunnable(tracer, spanNamer, new Runnable() {
 	@Override public void run() {
@@ -24,4 +24,4 @@ we’re checking if the class has a custom implementation of the // ... some additional logic ...
-future.get();

    will lead in creating a span named calculateTax.

    Prev   Next
    5. Span lifecycle Home 7. Managing spans with annotations
    \ No newline at end of file +future.get();

    will lead in creating a span named calculateTax.

    Prev   Next
    6. Span lifecycle Home 8. Managing spans with annotations
    \ No newline at end of file diff --git a/multi/_running_examples.html b/multi/_running_examples.html index 37deea183..10e0992e9 100644 --- a/multi/_running_examples.html +++ b/multi/_running_examples.html @@ -1,3 +1,3 @@ - 13. Running examples
    13. Running examples
    Prev   

    13. Running examples

    You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links:

    Prev   
    12. Integrations Home 
    \ No newline at end of file + 14. Running examples
    14. Running examples
    Prev   

    14. Running examples

    You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links:

    Prev   
    13. Integrations Home 
    \ No newline at end of file diff --git a/multi/_sampling.html b/multi/_sampling.html index d679f6204..d8d9418ef 100644 --- a/multi/_sampling.html +++ b/multi/_sampling.html @@ -1,6 +1,6 @@ - 3. Sampling
    3. Sampling
    Prev   Next

    3. Sampling

    In distributed tracing the data volumes can be very high so sampling + 4. Sampling

    4. Sampling
    Prev   Next

    4. Sampling

    In distributed tracing the data volumes can be very high so sampling can be important (you usually don’t need to export all spans to get a good picture of what is happening). Spring Cloud Sleuth has a Sampler strategy that you can implement to take control of the @@ -23,4 +23,4 @@ For backwards compatibility reasons we’re not changing the property name.< return new AlwaysSampler(); }

    [Tip]Tip

    You can set the HTTP header X-B3-Flags to 1 or when doing messaging you can set spanFlags header to 1. Then the current span will be forced to be exportable -regardless of the sampling decision.

    Prev   Next
    2. Features Home 4. Instrumentation
    \ No newline at end of file +regardless of the sampling decision.

    Prev   Next
    3. Features Home 5. Instrumentation
    \ No newline at end of file diff --git a/multi/_sending_spans_to_zipkin.html b/multi/_sending_spans_to_zipkin.html index 5b2524b87..e781acca2 100644 --- a/multi/_sending_spans_to_zipkin.html +++ b/multi/_sending_spans_to_zipkin.html @@ -1,7 +1,7 @@ - 9. Sending spans to Zipkin
    9. Sending spans to Zipkin
    Prev   Next

    9. Sending spans to Zipkin

    By default if you add spring-cloud-starter-zipkin as a dependency to your project, + 10. Sending spans to Zipkin

    10. Sending spans to Zipkin
    Prev   Next

    10. Sending spans to Zipkin

    By default if you add spring-cloud-starter-zipkin as a dependency to your project, when the span is closed, it will be sent to Zipkin over HTTP. The communication is asynchronous. You can configure the URL by setting the spring.zipkin.baseUrl property as follows:

    spring.zipkin.baseUrl: http://192.168.99.100:9411/

    If you want to find Zipkin via service discovery it’s enough to pass the -Zipkin’s service id inside the URL (example for zipkinserver service id)

    spring.zipkin.baseUrl: http://zipkinserver/
    Prev   Next
    8. Customizations Home 10. Span Data as Messages
    \ No newline at end of file +Zipkin’s service id inside the URL (example for zipkinserver service id)

    spring.zipkin.baseUrl: http://zipkinserver/
    Prev   Next
    9. Customizations Home 11. Span Data as Messages
    \ No newline at end of file diff --git a/multi/_span_data_as_messages.html b/multi/_span_data_as_messages.html index d0fb73810..6dc020349 100644 --- a/multi/_span_data_as_messages.html +++ b/multi/_span_data_as_messages.html @@ -1,13 +1,13 @@ - 10. Span Data as Messages
    10. Span Data as Messages
    Prev   Next

    10. Span Data as Messages

    You can accumulate and send span data over + 11. Span Data as Messages

    11. Span Data as Messages
    Prev   Next

    11. Span Data as Messages

    You can accumulate and send span data over Spring Cloud Stream by including the spring-cloud-sleuth-stream jar as a dependency, and adding a Channel Binder implementation (e.g. spring-cloud-starter-stream-rabbit for RabbitMQ or spring-cloud-starter-stream-kafka for Kafka). This will automatically turn your app into a producer of messages with payload -type Spans.

    10.1 Zipkin Consumer

    There is a special convenience annotation for setting up a message consumer +type Spans.

    11.1 Zipkin Consumer

    There is a special convenience annotation for setting up a message consumer for the Span data and pushing it into a Zipkin SpanStore. This application

    @SpringBootApplication
 @EnableZipkinStreamServer
 public class Consumer {
@@ -41,7 +41,7 @@ started quickly). For a more robust solution you can add MySQL and
     type: mysql
    [Note]Note

    The @EnableZipkinStreamServer is also annotated with @EnableZipkinServer so the process will also expose the standard Zipkin server endpoints for collecting spans over HTTP, and for -querying in the Zipkin Web UI.

    10.2 Custom Consumer

    A custom consumer can also easily be implemented using +querying in the Zipkin Web UI.

    11.2 Custom Consumer

    A custom consumer can also easily be implemented using spring-cloud-sleuth-stream and binding to the SleuthSink. Example:

    @EnableBinding(SleuthSink.class)
 @SpringBootApplication(exclude = SleuthStreamAutoConfiguration.class)
 @MessageEndpoint
@@ -65,4 +65,4 @@ with name equal to StreamSpanReporter.POLLER. Here
 		poller.setTrigger(new PeriodicTrigger(5000L));
 		return poller;
 	}
-}
    Prev   Next
    9. Sending spans to Zipkin Home 11. Metrics
    \ No newline at end of file +}
    Prev   Next
    10. Sending spans to Zipkin Home 12. Metrics
    \ No newline at end of file diff --git a/multi/_span_lifecycle.html b/multi/_span_lifecycle.html index e5afd238f..dcfa078c3 100644 --- a/multi/_span_lifecycle.html +++ b/multi/_span_lifecycle.html @@ -1,9 +1,9 @@ - 5. Span lifecycle
    5. Span lifecycle
    Prev   Next

    5. Span lifecycle

    You can do the following operations on the Span by means of org.springframework.cloud.sleuth.Tracer interface:

    • start - when you start a span its name is assigned and start timestamp is recorded.
    • close - the span gets finished (the end time of the span is recorded) and if + 6. Span lifecycle
      6. Span lifecycle
      Prev   Next

      6. Span lifecycle

      You can do the following operations on the Span by means of org.springframework.cloud.sleuth.Tracer interface:

      • start - when you start a span its name is assigned and start timestamp is recorded.
      • close - the span gets finished (the end time of the span is recorded) and if the span is exportable then it will be eligible for collection to Zipkin. -The span is also removed from the current thread.
      • continue - a new instance of span will be created whereas it will be a copy of the -one that it continues.
      • detach - the span doesn’t get stopped or closed. It only gets removed from the current thread.
      • create with explicit parent - you can create a new span and set an explicit parent to it
      [Tip]Tip

      Spring creates the instance of Tracer for you. In order to use it all you need is to just autowire it.

      5.1 Creating and closing spans

      You can manually create spans by using the Tracer interface.

      // Start a span. If there was a span present in this thread it will become
+The span is also removed from the current thread.
    • continue - a new instance of span will be created whereas it will be a copy of the +one that it continues.
    • detach - the span doesn’t get stopped or closed. It only gets removed from the current thread.
    • create with explicit parent - you can create a new span and set an explicit parent to it
    [Tip]Tip

    Spring creates the instance of Tracer for you. In order to use it all you need is to just autowire it.

    6.1 Creating and closing spans

    You can manually create spans by using the Tracer interface.

    // Start a span. If there was a span present in this thread it will become
 // the `newSpan`'s parent.
 Span newSpan = this.tracer.createSpan("calculateTax");
 try {
@@ -20,7 +20,7 @@ Span newSpan = 
 }

    In this example we could see how to create a new instance of span. Assuming that there already was a span present in this thread then it would become the parent of that span.

    [Important]Important

    Always clean after you create a span! Don’t forget to close a span if you want to send it to Zipkin.

    [Important]Important

    If your span contains a name greater than 50 chars, then that name will be truncated to 50 chars. Your names have to be explicit and concrete. Big names lead to -latency issues and sometimes even thrown exceptions.

    5.2 Continuing spans

    Sometimes you don’t want to create a new span but you want to continue one. Example of such a +latency issues and sometimes even thrown exceptions.

    6.2 Continuing spans

    Sometimes you don’t want to create a new span but you want to continue one. Example of such a situation might be (of course it all depends on the use-case):

    • AOP - If there was already a span created before an aspect was reached then you might not want to create a new span.
    • Hystrix - executing a Hystrix command is most likely a logical part of the current processing. It’s in fact only a technical implementation detail that you wouldn’t necessarily want to reflect in tracing as a separate being.

    The continued instance of span is equal to the one that it continues:

    Span continuedSpan = this.tracer.continueSpan(spanToContinue);
 assertThat(continuedSpan).isEqualTo(spanToContinue);

    To continue a span you can use the Tracer interface.

    // let's assume that we're in a thread Y and we've received
@@ -40,7 +40,7 @@ Span continuedSpan = 
    [Important]Important
    Always clean after you create a span! Don’t forget to detach a span if some work was done started in one
  thread (e.g. thread X) and it’s waiting for other threads (e.g. Y, Z) to finish.
  Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected
- the span in thread X should be closed.

    5.3 Creating spans with an explicit parent

    There is a possibility that you want to start a new span and provide an explicit parent of that span. + the span in thread X should be closed.

    6.3 Creating spans with an explicit parent

    There is a possibility that you want to start a new span and provide an explicit parent of that span. Let’s assume that the parent of a span is in one thread and you want to start a new span in another thread. The startSpan method of the Tracer interface is the method you are looking for.

    // let's assume that we're in a thread Y and we've received
 // the `initialSpan` from thread X. `initialSpan` will be the parent
@@ -60,4 +60,4 @@ Span newSpan = 
 	this.tracer.close(newSpan);
 }
    [Important]Important

    After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs related to the fact that you have a span present in the current thread other than the one you’re trying to close. - What’s worse your spans won’t get closed properly thus will not get collected to Zipkin.

    Prev   Next
    4. Instrumentation Home 6. Naming spans
    \ No newline at end of file + What’s worse your spans won’t get closed properly thus will not get collected to Zipkin.

    Prev   Next
    5. Instrumentation Home 7. Naming spans
    \ No newline at end of file diff --git a/multi/pr01.html b/multi/pr01.html index f0f927940..3291e3fe5 100644 --- a/multi/pr01.html +++ b/multi/pr01.html @@ -1,3 +1,3 @@ -
    Prev   Next

    1.3.0.BUILD-SNAPSHOT

    Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.

    Prev   Next
    Spring Cloud Sleuth Home 1. Additional resources
    \ No newline at end of file +
    Prev   Next

    1.3.0.BUILD-SNAPSHOT

    Prev   Next
    Spring Cloud Sleuth Home 1. Introduction
    \ No newline at end of file diff --git a/multi/spring-cloud-sleuth.html b/multi/spring-cloud-sleuth.html index 1de1a9a8b..6da5af4f9 100644 --- a/multi/spring-cloud-sleuth.html +++ b/multi/spring-cloud-sleuth.html @@ -1,297 +1,3 @@ - Spring Cloud Sleuth
    Spring Cloud Sleuth
       Next

    Spring Cloud Sleuth

    Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer

    Table of Contents

    1. Additional resources
    2. Features
    3. Sampling
    4. Instrumentation
    5. Span lifecycle
    5.1. Creating and closing spans
    5.2. Continuing spans
    5.3. Creating spans with an explicit parent
    6. Naming spans
    6.1. @SpanName annotation
    6.2. toString() method
    7. Managing spans with annotations
    7.1. Rationale
    7.2. Creating new spans
    7.3. Continuing spans
    7.4. More advanced tag setting
    Custom extractor
    Resolving expressions for value
    Using toString method
    8. Customizations
    8.1. Spring Integration
    8.2. HTTP
    8.3. Example
    8.4. TraceFilter
    8.5. Custom SA tag in Zipkin
    8.6. Custom service name
    8.7. Customization of reported spans
    8.8. Host locator
    9. Sending spans to Zipkin
    10. Span Data as Messages
    10.1. Zipkin Consumer
    10.2. Custom Consumer
    11. Metrics
    12. Integrations
    12.1. Runnable and Callable
    12.2. Hystrix
    Custom Concurrency Strategy
    Manual Command setting
    12.3. RxJava
    12.4. HTTP integration
    HTTP Filter
    HandlerInterceptor
    Async Servlet support
    12.5. HTTP client integration
    Synchronous Rest Template
    Asynchronous Rest Template
    Multiple Asynchronous Rest Templates
    Traverson
    12.6. Feign
    12.7. Asynchronous communication
    @Async annotated methods
    @Scheduled annotated methods
    Executor, ExecutorService and ScheduledExecutorService
    Customization of Executors
    12.8. Messaging
    12.9. Zuul
    13. Running examples

    1 Terminology

    Spring Cloud Sleuth borrows Dapper’s terminology.

    Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an -RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span -is a part of. Spans also have other data, such as descriptions, timestamped events, key-value -annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address).

    Spans are started and stopped, and they keep track of their timing information. Once you create a -span, you must stop it at some point in the future.

    [Tip]Tip

    The initial span that starts a trace is called a root span. The value of span id -of that span is equal to trace id.

    Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed -big-data store, a trace might be formed by a put request.

    Annotation: is used to record existence of an event in time. Some of the core annotations used to define -the start and stop of a request are:

    • cs - Client Sent - The client has made a request. This annotation depicts the start of the span.
    • sr - Server Received - The server side got the request and will start processing it. -If one subtracts the cs timestamp from this timestamp one will receive the network latency.
    • ss - Server Sent - Annotated upon completion of request processing (when the response -got sent back to the client). If one subtracts the sr timestamp from this timestamp one -will receive the time needed by the server side to process the request.
    • cr - Client Received - Signifies the end of the span. The client has successfully received the -response from the server side. If one subtracts the cs timestamp from this timestamp one -will receive the whole time needed by the client to receive the response from the server.

    Visualization of what Span and Trace will look in a system together with the Zipkin annotations:

    Trace Info propagation

    Each color of a note signifies a span (7 spans - from A to G). If you have such information in the note:

    Trace Id = X
-Span Id = D
-Client Sent

    That means that the current span has Trace-Id set to X, Span-Id set to D. It also has emitted - Client Sent event.

    This is how the visualization of the parent / child relationship of spans would look like:

    Parent child relationship

    2 Purpose

    In the following sections the example from the image above will be taken into consideration.

    2.1 Distributed tracing with Zipkin

    Altogether there are 7 spans . If you go to traces in Zipkin you will see this number in the second trace:

    Traces

    However if you pick a particular trace then you will see 4 spans:

    Traces Info propagation
    [Note]Note

    When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to -Zipkin with Server Received and Server Sent / Client Received and Client Sent -annotations then they will presented as a single span.

    Why is there a difference between the 7 and 4 spans in this case?

    • 2 spans come from http:/start span. It has the Server Received (SR) and Server Sent (SS) annotations.
    • 2 spans come from the RPC call from service1 to service2 to the http:/foo endpoint. It has the Client Sent (CS) -and Client Received (CR) annotations on service1 side. It also has Server Received (SR) and Server Sent (SS) annotations -on the service2 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
    • 2 spans come from the RPC call from service2 to service3 to the http:/bar endpoint. It has the Client Sent (CS) -and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations -on the service3 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
    • 2 spans come from the RPC call from service2 to service4 to the http:/baz endpoint. It has the Client Sent (CS) -and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations -on the service4 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.

    So if we count the physical spans we have 1 from http:/start, 2 from service1 calling service2, 2 form service2 -calling service3 and 2 from service2 calling service4. Altogether 7 spans.

    Logically we see the information of Total Spans: 4 because we have 1 span related to the incoming request -to service1 and 3 spans related to RPC calls.

    2.2 Visualizing errors

    Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn’t caught then we’re -setting proper tags on the span which Zipkin can properly colorize. You could see in the list of traces one - trace that was in red color. That’s because there was an exception thrown.

    If you click that trace then you’ll see a similar picture

    Error Traces

    Then if you click on one of the spans you’ll see the following

    Error Traces Info propagation

    As you can see you can easily see the reason for an error and the whole stacktrace related to it.

    2.3 Live examples

    Figure 1. Click Pivotal Web Services icon to see it live!

    Zipkin deployed on Pivotal Web Services

    The dependency graph in Zipkin would look like this:

    Dependencies

    Figure 2. Click Pivotal Web Services icon to see it live!

    Zipkin deployed on Pivotal Web Services

    2.4 Log correlation

    When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following:

    service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
-service2.log:2016-02-26 11:15:47.710  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Hello from service2. Calling service3 and then service4
-service3.log:2016-02-26 11:15:47.895  INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application   : Hello from service3
-service2.log:2016-02-26 11:15:47.924  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service3 [Hello from service3]
-service4.log:2016-02-26 11:15:48.134  INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application   : Hello from service4
-service2.log:2016-02-26 11:15:48.156  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service4 [Hello from service4]
-service1.log:2016-02-26 11:15:48.182  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]

    If you’re using a log aggregating tool like Kibana, -Splunk etc. you can order the events that took place. An example of -Kibana would look like this:

    Log correlation with Kibana

    If you want to use Logstash here is the Grok pattern for Logstash:

    filter {
-       # pattern matching logback pattern
-       grok {
-              match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
-       }
-}
    [Note]Note

    If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern:

    filter {
-       # pattern matching logback pattern
-       grok {
-              match => { "message" => "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
-       }
-}

    JSON Logback with Logstash

    Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability -we’re passing the dependencies in the groupId:artifactId:version notation.

    Dependencies setup

    • Ensure that Logback is on the classpath (ch.qos.logback:logback-core)
    • Add Logstash Logback encode - example for version 4.6 : net.logstash.logback:logstash-logback-encoder:4.6

    Logback setup

    Below you can find an example of a Logback configuration (file named logback-spring.xml) that:

    • logs information from the application in a JSON format to a build/${spring.application.name}.json file
    • has commented out two additional appenders - console and standard log file
    • has the same logging pattern as the one presented in the previous section
    <?xml version="1.0" encoding="UTF-8"?>
-<configuration>
-	<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
-	​
-	<springProperty scope="context" name="springAppName" source="spring.application.name"/>
-	<!-- Example for logging into the build folder of your project -->
-	<property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${springAppName}"/>​
-
-	<!-- You can override this to have a custom pattern -->
-	<property name="CONSOLE_LOG_PATTERN"
-			  value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>
-
-	<!-- Appender to log to console -->
-	<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
-		<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
-			<!-- Minimum logging level to be presented in the console logs-->
-			<level>DEBUG</level>
-		</filter>
-		<encoder>
-			<pattern>${CONSOLE_LOG_PATTERN}</pattern>
-			<charset>utf8</charset>
-		</encoder>
-	</appender>
-
-	<!-- Appender to log to file -->​
-	<appender name="flatfile" class="ch.qos.logback.core.rolling.RollingFileAppender">
-		<file>${LOG_FILE}</file>
-		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
-			<fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.gz</fileNamePattern>
-			<maxHistory>7</maxHistory>
-		</rollingPolicy>
-		<encoder>
-			<pattern>${CONSOLE_LOG_PATTERN}</pattern>
-			<charset>utf8</charset>
-		</encoder>
-	</appender>
-	​
-	<!-- Appender to log to file in a JSON format -->
-	<appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender">
-		<file>${LOG_FILE}.json</file>
-		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
-			<fileNamePattern>${LOG_FILE}.json.%d{yyyy-MM-dd}.gz</fileNamePattern>
-			<maxHistory>7</maxHistory>
-		</rollingPolicy>
-		<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
-			<providers>
-				<timestamp>
-					<timeZone>UTC</timeZone>
-				</timestamp>
-				<pattern>
-					<pattern>
-						{
-						"severity": "%level",
-						"service": "${springAppName:-}",
-						"trace": "%X{X-B3-TraceId:-}",
-						"span": "%X{X-B3-SpanId:-}",
-						"parent": "%X{X-B3-ParentSpanId:-}",
-						"exportable": "%X{X-Span-Export:-}",
-						"pid": "${PID:-}",
-						"thread": "%thread",
-						"class": "%logger{40}",
-						"rest": "%message"
-						}
-					</pattern>
-				</pattern>
-			</providers>
-		</encoder>
-	</appender>
-	​
-	<root level="INFO">
-		<appender-ref ref="console"/>
-		<!-- uncomment this to have also JSON logs -->
-		<!--<appender-ref ref="logstash"/>-->
-		<!--<appender-ref ref="flatfile"/>-->
-	</root>
-</configuration>
    [Note]Note

    If you’re using a custom logback-spring.xml then you have to pass the spring.application.name in -bootstrap instead of application property file. Otherwise your custom logback file won’t read the property properly.

    2.5 Propagating Span Context

    The span context is the state that must get propagated to any child Spans across process boundaries. -Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context. -Baggage is an optional part.

    Baggage is a set of key:value pairs stored in the span context. Baggage travels together with the trace -and is attached to every span. Spring Cloud Sleuth will understand that a header is baggage related if the HTTP - header is prefixed with baggage- and for messaging it starts with baggage_.

    [Important]Important

    There’s currently no limitation of the count or size of baggage items. However, keep in mind that -too many can decrease system throughput or increase RPC latency. In extreme cases, it could crash the app due -to exceeding transport-level message or header capacity.

    Example of setting baggage on a span:

    Span initialSpan = this.tracer.createSpan("span");
-initialSpan.setBaggageItem("foo", "bar");
-initialSpan.setBaggageItem("UPPER_CASE", "someValue");

    Baggage vs. Span Tags

    Baggage travels with the trace (i.e. every child span contains the baggage of its parent). Zipkin has no knowledge of -baggage and will not even receive that information.

    Tags are attached to a specific span - they are presented for that particular span only. However you -can search by tag to find the trace, where there exists a span having the searched tag value.

    If you want to be able to lookup a span based on baggage, you should add corresponding entry as a tag in the root span.

    @Autowired Tracer tracer;
-
-Span span = tracer.getCurrentSpan();
-String baggageKey = "key";
-String baggageValue = "foo";
-span.setBaggageItem(baggageKey, baggageValue);
-tracer.addTag(baggageKey, baggageValue);

    3 Adding to the project

    3.1 Only Sleuth (log correlation)

    If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add -the spring-cloud-starter-sleuth module to your project.

    Maven.  -

    <dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
-
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-sleuth</artifactId>
-   </dependency>

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-starter-sleuth

    Gradle.  -

    dependencyManagement { 1
-    imports {
-        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
-    }
-}
-
-dependencies { 2
-    compile "org.springframework.cloud:spring-cloud-starter-sleuth"
-}

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-starter-sleuth

    3.2 Sleuth with Zipkin via HTTP

    If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency.

    Maven.  -

    <dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
-
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-zipkin</artifactId>
-   </dependency>

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-starter-zipkin

    Gradle.  -

    dependencyManagement { 1
-    imports {
-        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
-    }
-}
-
-dependencies { 2
-    compile "org.springframework.cloud:spring-cloud-starter-zipkin"
-}

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-starter-zipkin

    3.3 Sleuth with Zipkin via Spring Cloud Stream

    If you want both Sleuth and Zipkin just add the spring-cloud-sleuth-stream dependency.

    Maven.  -

    <dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
-
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-sleuth-stream</artifactId>
-   </dependency>
-   <dependency> 3
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-sleuth</artifactId>
-   </dependency>
-   <!-- EXAMPLE FOR RABBIT BINDING -->
-   <dependency> 4
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-stream-binder-rabbit</artifactId>
-   </dependency>

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-sleuth-stream

    3

    Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

    4

    Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

    Gradle.  -

    dependencyManagement { 1
-    imports {
-        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
-    }
-}
-
-dependencies {
-    compile "org.springframework.cloud:spring-cloud-sleuth-stream" 2
-    compile "org.springframework.cloud:spring-cloud-starter-sleuth" 3
-    // Example for Rabbit binding
-    compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" 4
-}

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-sleuth-stream

    3

    Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

    4

    Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

    3.4 Spring Cloud Sleuth Stream Zipkin Collector

    If you want to start a Spring Cloud Sleuth Stream Zipkin collector just add the spring-cloud-sleuth-zipkin-stream -dependency

    Maven.  -

    <dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
-
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-sleuth-zipkin-stream</artifactId>
-   </dependency>
-   <dependency> 3
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-sleuth</artifactId>
-   </dependency>
-   <!-- EXAMPLE FOR RABBIT BINDING -->
-   <dependency> 4
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-stream-binder-rabbit</artifactId>
-   </dependency>

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-sleuth-zipkin-stream

    3

    Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

    4

    Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

    Gradle.  -

    dependencyManagement { 1
-    imports {
-        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
-    }
-}
-
-dependencies {
-    compile "org.springframework.cloud:spring-cloud-sleuth-zipkin-stream" 2
-    compile "org.springframework.cloud:spring-cloud-starter-sleuth" 3
-    // Example for Rabbit binding
-    compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" 4
-}

    -

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-sleuth-zipkin-stream

    3

    Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

    4

    Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

    and then just annotate your main class with @EnableZipkinStreamServer annotation:

    package example;
-
-import org.springframework.boot.SpringApplication;
-import org.springframework.boot.autoconfigure.SpringBootApplication;
-import org.springframework.cloud.sleuth.zipkin.stream.EnableZipkinStreamServer;
-
-@SpringBootApplication
-@EnableZipkinStreamServer
-public class ZipkinStreamServerApplication {
-
-	public static void main(String[] args) throws Exception {
-		SpringApplication.run(ZipkinStreamServerApplication.class, args);
-	}
-
-}
       Next
       
    \ No newline at end of file + Spring Cloud Sleuth
    Spring Cloud Sleuth
       Next

    Spring Cloud Sleuth

    Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer

    Table of Contents

    1. Introduction
    1.1. Terminology
    1.2. Purpose
    1.2.1. Distributed tracing with Zipkin
    1.2.2. Visualizing errors
    1.2.3. Live examples
    1.2.4. Log correlation
    JSON Logback with Logstash
    1.2.5. Propagating Span Context
    Baggage vs. Span Tags
    1.3. Adding to the project
    1.3.1. Only Sleuth (log correlation)
    1.3.2. Sleuth with Zipkin via HTTP
    1.3.3. Sleuth with Zipkin via Spring Cloud Stream
    1.3.4. Spring Cloud Sleuth Stream Zipkin Collector
    2. Additional resources
    3. Features
    4. Sampling
    5. Instrumentation
    6. Span lifecycle
    6.1. Creating and closing spans
    6.2. Continuing spans
    6.3. Creating spans with an explicit parent
    7. Naming spans
    7.1. @SpanName annotation
    7.2. toString() method
    8. Managing spans with annotations
    8.1. Rationale
    8.2. Creating new spans
    8.3. Continuing spans
    8.4. More advanced tag setting
    8.4.1. Custom extractor
    8.4.2. Resolving expressions for value
    8.4.3. Using toString method
    9. Customizations
    9.1. Spring Integration
    9.2. HTTP
    9.3. Example
    9.4. TraceFilter
    9.5. Custom SA tag in Zipkin
    9.6. Custom service name
    9.7. Customization of reported spans
    9.8. Host locator
    10. Sending spans to Zipkin
    11. Span Data as Messages
    11.1. Zipkin Consumer
    11.2. Custom Consumer
    12. Metrics
    13. Integrations
    13.1. Runnable and Callable
    13.2. Hystrix
    13.2.1. Custom Concurrency Strategy
    13.2.2. Manual Command setting
    13.3. RxJava
    13.4. HTTP integration
    13.4.1. HTTP Filter
    13.4.2. HandlerInterceptor
    13.4.3. Async Servlet support
    13.5. HTTP client integration
    13.5.1. Synchronous Rest Template
    13.5.2. Asynchronous Rest Template
    Multiple Asynchronous Rest Templates
    13.5.3. Traverson
    13.6. Feign
    13.7. Asynchronous communication
    13.7.1. @Async annotated methods
    13.7.2. @Scheduled annotated methods
    13.7.3. Executor, ExecutorService and ScheduledExecutorService
    Customization of Executors
    13.8. Messaging
    13.9. Zuul
    14. Running examples
       Next
       
    \ No newline at end of file diff --git a/single/spring-cloud-sleuth.html b/single/spring-cloud-sleuth.html index 7f94fcb7d..58d55951c 100644 --- a/single/spring-cloud-sleuth.html +++ b/single/spring-cloud-sleuth.html @@ -1,6 +1,6 @@ - Spring Cloud Sleuth

    Spring Cloud Sleuth

    Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer

    Table of Contents

    1. Additional resources
    2. Features
    3. Sampling
    4. Instrumentation
    5. Span lifecycle
    5.1. Creating and closing spans
    5.2. Continuing spans
    5.3. Creating spans with an explicit parent
    6. Naming spans
    6.1. @SpanName annotation
    6.2. toString() method
    7. Managing spans with annotations
    7.1. Rationale
    7.2. Creating new spans
    7.3. Continuing spans
    7.4. More advanced tag setting
    Custom extractor
    Resolving expressions for value
    Using toString method
    8. Customizations
    8.1. Spring Integration
    8.2. HTTP
    8.3. Example
    8.4. TraceFilter
    8.5. Custom SA tag in Zipkin
    8.6. Custom service name
    8.7. Customization of reported spans
    8.8. Host locator
    9. Sending spans to Zipkin
    10. Span Data as Messages
    10.1. Zipkin Consumer
    10.2. Custom Consumer
    11. Metrics
    12. Integrations
    12.1. Runnable and Callable
    12.2. Hystrix
    Custom Concurrency Strategy
    Manual Command setting
    12.3. RxJava
    12.4. HTTP integration
    HTTP Filter
    HandlerInterceptor
    Async Servlet support
    12.5. HTTP client integration
    Synchronous Rest Template
    Asynchronous Rest Template
    Multiple Asynchronous Rest Templates
    Traverson
    12.6. Feign
    12.7. Asynchronous communication
    @Async annotated methods
    @Scheduled annotated methods
    Executor, ExecutorService and ScheduledExecutorService
    Customization of Executors
    12.8. Messaging
    12.9. Zuul
    13. Running examples

    1.3.0.BUILD-SNAPSHOT

    Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.

    1 Terminology

    Spring Cloud Sleuth borrows Dapper’s terminology.

    Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an + Spring Cloud Sleuth

    Spring Cloud Sleuth

    Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer

    Table of Contents

    1. Introduction
    1.1. Terminology
    1.2. Purpose
    1.2.1. Distributed tracing with Zipkin
    1.2.2. Visualizing errors
    1.2.3. Live examples
    1.2.4. Log correlation
    JSON Logback with Logstash
    1.2.5. Propagating Span Context
    Baggage vs. Span Tags
    1.3. Adding to the project
    1.3.1. Only Sleuth (log correlation)
    1.3.2. Sleuth with Zipkin via HTTP
    1.3.3. Sleuth with Zipkin via Spring Cloud Stream
    1.3.4. Spring Cloud Sleuth Stream Zipkin Collector
    2. Additional resources
    3. Features
    4. Sampling
    5. Instrumentation
    6. Span lifecycle
    6.1. Creating and closing spans
    6.2. Continuing spans
    6.3. Creating spans with an explicit parent
    7. Naming spans
    7.1. @SpanName annotation
    7.2. toString() method
    8. Managing spans with annotations
    8.1. Rationale
    8.2. Creating new spans
    8.3. Continuing spans
    8.4. More advanced tag setting
    8.4.1. Custom extractor
    8.4.2. Resolving expressions for value
    8.4.3. Using toString method
    9. Customizations
    9.1. Spring Integration
    9.2. HTTP
    9.3. Example
    9.4. TraceFilter
    9.5. Custom SA tag in Zipkin
    9.6. Custom service name
    9.7. Customization of reported spans
    9.8. Host locator
    10. Sending spans to Zipkin
    11. Span Data as Messages
    11.1. Zipkin Consumer
    11.2. Custom Consumer
    12. Metrics
    13. Integrations
    13.1. Runnable and Callable
    13.2. Hystrix
    13.2.1. Custom Concurrency Strategy
    13.2.2. Manual Command setting
    13.3. RxJava
    13.4. HTTP integration
    13.4.1. HTTP Filter
    13.4.2. HandlerInterceptor
    13.4.3. Async Servlet support
    13.5. HTTP client integration
    13.5.1. Synchronous Rest Template
    13.5.2. Asynchronous Rest Template
    Multiple Asynchronous Rest Templates
    13.5.3. Traverson
    13.6. Feign
    13.7. Asynchronous communication
    13.7.1. @Async annotated methods
    13.7.2. @Scheduled annotated methods
    13.7.3. Executor, ExecutorService and ScheduledExecutorService
    Customization of Executors
    13.8. Messaging
    13.9. Zuul
    14. Running examples

    1.3.0.BUILD-SNAPSHOT

    1. Introduction

    Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.

    1.1 Terminology

    Spring Cloud Sleuth borrows Dapper’s terminology.

    Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of. Spans also have other data, such as descriptions, timestamped events, key-value annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address).

    Spans are started and stopped, and they keep track of their timing information. Once you create a @@ -15,7 +15,7 @@ response from the server side. If one subtracts the cs timestamp from this times will receive the whole time needed by the client to receive the response from the server.

    Visualization of what Span and Trace will look in a system together with the Zipkin annotations:

    Trace Info propagation

    Each color of a note signifies a span (7 spans - from A to G). If you have such information in the note:

    Trace Id = X
 Span Id = D
 Client Sent

    That means that the current span has Trace-Id set to X, Span-Id set to D. It also has emitted - Client Sent event.

    This is how the visualization of the parent / child relationship of spans would look like:

    Parent child relationship

    2 Purpose

    In the following sections the example from the image above will be taken into consideration.

    2.1 Distributed tracing with Zipkin

    Altogether there are 7 spans . If you go to traces in Zipkin you will see this number in the second trace:

    Traces

    However if you pick a particular trace then you will see 4 spans:

    Traces Info propagation
    [Note]Note

    When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to + Client Sent event.

    This is how the visualization of the parent / child relationship of spans would look like:

    Parent child relationship

    1.2 Purpose

    In the following sections the example from the image above will be taken into consideration.

    1.2.1 Distributed tracing with Zipkin

    Altogether there are 7 spans . If you go to traces in Zipkin you will see this number in the second trace:

    Traces

    However if you pick a particular trace then you will see 4 spans:

    Traces Info propagation
    [Note]Note

    When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to Zipkin with Server Received and Server Sent / Client Received and Client Sent annotations then they will presented as a single span.

    Why is there a difference between the 7 and 4 spans in this case?

    • 2 spans come from http:/start span. It has the Server Received (SR) and Server Sent (SS) annotations.
    • 2 spans come from the RPC call from service1 to service2 to the http:/foo endpoint. It has the Client Sent (CS) and Client Received (CR) annotations on service1 side. It also has Server Received (SR) and Server Sent (SS) annotations @@ -25,9 +25,9 @@ on the service3 side. Physically there are 2 spans and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations on the service4 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.

    So if we count the physical spans we have 1 from http:/start, 2 from service1 calling service2, 2 form service2 calling service3 and 2 from service2 calling service4. Altogether 7 spans.

    Logically we see the information of Total Spans: 4 because we have 1 span related to the incoming request -to service1 and 3 spans related to RPC calls.

    2.2 Visualizing errors

    Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn’t caught then we’re +to service1 and 3 spans related to RPC calls.

    1.2.2 Visualizing errors

    Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn’t caught then we’re setting proper tags on the span which Zipkin can properly colorize. You could see in the list of traces one - trace that was in red color. That’s because there was an exception thrown.

    If you click that trace then you’ll see a similar picture

    Error Traces

    Then if you click on one of the spans you’ll see the following

    Error Traces Info propagation

    As you can see you can easily see the reason for an error and the whole stacktrace related to it.

    2.3 Live examples

    Figure 1. Click Pivotal Web Services icon to see it live!

    Zipkin deployed on Pivotal Web Services

    The dependency graph in Zipkin would look like this:

    Dependencies

    Figure 2. Click Pivotal Web Services icon to see it live!

    Zipkin deployed on Pivotal Web Services

    2.4 Log correlation

    When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following:

    service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
+  trace that was in red color. That’s because there was an exception thrown.
    If you click that trace then you’ll see a similar picture
    Error Traces
    Then if you click on one of the spans you’ll see the following
    Error Traces Info propagation
    As you can see you can easily see the reason for an error and the whole stacktrace related to it.

    1.2.3 Live examples

    Figure 1.1. Click Pivotal Web Services icon to see it live!

    Zipkin deployed on Pivotal Web Services

    The dependency graph in Zipkin would look like this:

    Dependencies

    Figure 1.2. Click Pivotal Web Services icon to see it live!

    Zipkin deployed on Pivotal Web Services

    1.2.4 Log correlation

    When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following:

    service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
 service2.log:2016-02-26 11:15:47.710  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Hello from service2. Calling service3 and then service4
 service3.log:2016-02-26 11:15:47.895  INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application   : Hello from service3
 service2.log:2016-02-26 11:15:47.924  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service3 [Hello from service3]
@@ -122,7 +122,7 @@ we’re passing the dependencies in the groupId:artifa
 		<!--<appender-ref ref="flatfile"/>-->
 	</root>
 </configuration>
    [Note]Note

    If you’re using a custom logback-spring.xml then you have to pass the spring.application.name in -bootstrap instead of application property file. Otherwise your custom logback file won’t read the property properly.

    2.5 Propagating Span Context

    The span context is the state that must get propagated to any child Spans across process boundaries. +bootstrap instead of application property file. Otherwise your custom logback file won’t read the property properly.

    1.2.5 Propagating Span Context

    The span context is the state that must get propagated to any child Spans across process boundaries. Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context. Baggage is an optional part.

    Baggage is a set of key:value pairs stored in the span context. Baggage travels together with the trace and is attached to every span. Spring Cloud Sleuth will understand that a header is baggage related if the HTTP @@ -138,7 +138,7 @@ Span span = tracer.getCurrentSpan(); String baggageKey = "key"; String baggageValue = "foo"; span.setBaggageItem(baggageKey, baggageValue); -tracer.addTag(baggageKey, baggageValue);

    3 Adding to the project

    3.1 Only Sleuth (log correlation)

    If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add +tracer.addTag(baggageKey, baggageValue);

    1.3 Adding to the project

    1.3.1 Only Sleuth (log correlation)

    If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add the spring-cloud-starter-sleuth module to your project.

    Maven.  

    <dependencyManagement> 1
          <dependencies>
@@ -168,7 +168,7 @@ dependencies { "org.springframework.cloud:spring-cloud-starter-sleuth"
 }

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-starter-sleuth

    3.2 Sleuth with Zipkin via HTTP

    If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency.

    Maven.  +the Spring BOM

    2

    Add the dependency to spring-cloud-starter-sleuth

    1.3.2 Sleuth with Zipkin via HTTP

    If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency.

    Maven.  

    <dependencyManagement> 1
          <dependencies>
              <dependency>
@@ -197,7 +197,7 @@ dependencies { "org.springframework.cloud:spring-cloud-starter-zipkin"
 }

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-starter-zipkin

    3.3 Sleuth with Zipkin via Spring Cloud Stream

    If you want both Sleuth and Zipkin just add the spring-cloud-sleuth-stream dependency.

    Maven.  +the Spring BOM

    2

    Add the dependency to spring-cloud-starter-zipkin

    1.3.3 Sleuth with Zipkin via Spring Cloud Stream

    If you want both Sleuth and Zipkin just add the spring-cloud-sleuth-stream dependency.

    Maven.  

    <dependencyManagement> 1
          <dependencies>
              <dependency>
@@ -238,7 +238,7 @@ dependencies {
     compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" 4
 }

    1

    In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

    2

    Add the dependency to spring-cloud-sleuth-stream

    3

    Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

    4

    Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

    3.4 Spring Cloud Sleuth Stream Zipkin Collector

    If you want to start a Spring Cloud Sleuth Stream Zipkin collector just add the spring-cloud-sleuth-zipkin-stream +the Spring BOM

    2

    Add the dependency to spring-cloud-sleuth-stream

    3

    Add the dependency to spring-cloud-starter-sleuth - that way all dependant dependencies will be downloaded

    4

    Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to

    1.3.4 Spring Cloud Sleuth Stream Zipkin Collector

    If you want to start a Spring Cloud Sleuth Stream Zipkin collector just add the spring-cloud-sleuth-zipkin-stream dependency

    Maven.  

    <dependencyManagement> 1
          <dependencies>
@@ -294,7 +294,7 @@ the Spring BOM
    class, args);
 	}
 
-}

    1. Additional resources

    Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin

    click here to see the video

    2. Features

    • Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Example logs:

      2016-02-02 15:30:57.902  INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
+}

    2. Additional resources

    Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin

    click here to see the video

    3. Features

    • Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Example logs:

      2016-02-02 15:30:57.902  INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
 2016-02-02 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
 2016-02-02 15:31:01.936  INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...

      notice the [appname,traceId,spanId,exportable] entries from the MDC:

      • spanId - the id of a specific operation that took place
      • appname - the name of the application that logged the span
      • traceId - the id of the latency graph that contains the span
      • exportable - whether the log should be exported to Zipkin or not. When would you like the span not to be exportable? In the case in which you want to wrap some operation in a Span and have it written to the logs @@ -313,7 +313,7 @@ Your app automatically becomes a producer of tracer messages that are sent over above. Other logging systems have to configure their own formatter to get the same result. The default is logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}] (this is a Spring Boot feature for logback users). - This means that if you’re not using SLF4J this pattern WILL NOT be automatically applied.

    3. Sampling

    In distributed tracing the data volumes can be very high so sampling + This means that if you’re not using SLF4J this pattern WILL NOT be automatically applied.

    4. Sampling

    In distributed tracing the data volumes can be very high so sampling can be important (you usually don’t need to export all spans to get a good picture of what is happening). Spring Cloud Sleuth has a Sampler strategy that you can implement to take control of the @@ -336,7 +336,7 @@ For backwards compatibility reasons we’re not changing the property name.< return new AlwaysSampler(); }

    [Tip]Tip

    You can set the HTTP header X-B3-Flags to 1 or when doing messaging you can set spanFlags header to 1. Then the current span will be forced to be exportable -regardless of the sampling decision.

    4. Instrumentation

    Spring Cloud Sleuth instruments all your Spring application +regardless of the sampling decision.

    5. Instrumentation

    Spring Cloud Sleuth instruments all your Spring application automatically, so you shouldn’t have to do anything to activate it. The instrumentation is added using a variety of technologies according to the stack that is available, e.g. for a servlet web @@ -351,10 +351,10 @@ danger of accidentally collecting too much data without configuring something).

    [Note]Note

    Currently the instrumentation in Spring Cloud Sleuth is eager - it means that we’re actively trying to pass the tracing context between threads. Also timing events are captured even when sleuth isn’t exporting data to a tracing system. -This approach may change in the future towards being lazy on this matter.

    5. Span lifecycle

    You can do the following operations on the Span by means of org.springframework.cloud.sleuth.Tracer interface:

    • start - when you start a span its name is assigned and start timestamp is recorded.
    • close - the span gets finished (the end time of the span is recorded) and if +This approach may change in the future towards being lazy on this matter.

    6. Span lifecycle

    You can do the following operations on the Span by means of org.springframework.cloud.sleuth.Tracer interface:

    • start - when you start a span its name is assigned and start timestamp is recorded.
    • close - the span gets finished (the end time of the span is recorded) and if the span is exportable then it will be eligible for collection to Zipkin. -The span is also removed from the current thread.
    • continue - a new instance of span will be created whereas it will be a copy of the -one that it continues.
    • detach - the span doesn’t get stopped or closed. It only gets removed from the current thread.
    • create with explicit parent - you can create a new span and set an explicit parent to it
    [Tip]Tip

    Spring creates the instance of Tracer for you. In order to use it all you need is to just autowire it.

    5.1 Creating and closing spans

    You can manually create spans by using the Tracer interface.

    // Start a span. If there was a span present in this thread it will become
+The span is also removed from the current thread.
  • continue - a new instance of span will be created whereas it will be a copy of the
+one that it continues.
  • detach - the span doesn’t get stopped or closed. It only gets removed from the current thread.
  • create with explicit parent - you can create a new span and set an explicit parent to it
    • [Tip]Tip

    Spring creates the instance of Tracer for you. In order to use it all you need is to just autowire it.

    6.1 Creating and closing spans

    You can manually create spans by using the Tracer interface.

    // Start a span. If there was a span present in this thread it will become
 // the `newSpan`'s parent.
 Span newSpan = this.tracer.createSpan("calculateTax");
 try {
@@ -371,7 +371,7 @@ Span newSpan = 
 }

    In this example we could see how to create a new instance of span. Assuming that there already was a span present in this thread then it would become the parent of that span.

    [Important]Important

    Always clean after you create a span! Don’t forget to close a span if you want to send it to Zipkin.

    [Important]Important

    If your span contains a name greater than 50 chars, then that name will be truncated to 50 chars. Your names have to be explicit and concrete. Big names lead to -latency issues and sometimes even thrown exceptions.

    5.2 Continuing spans

    Sometimes you don’t want to create a new span but you want to continue one. Example of such a +latency issues and sometimes even thrown exceptions.

    6.2 Continuing spans

    Sometimes you don’t want to create a new span but you want to continue one. Example of such a situation might be (of course it all depends on the use-case):

    • AOP - If there was already a span created before an aspect was reached then you might not want to create a new span.
    • Hystrix - executing a Hystrix command is most likely a logical part of the current processing. It’s in fact only a technical implementation detail that you wouldn’t necessarily want to reflect in tracing as a separate being.

    The continued instance of span is equal to the one that it continues:

    Span continuedSpan = this.tracer.continueSpan(spanToContinue);
 assertThat(continuedSpan).isEqualTo(spanToContinue);

    To continue a span you can use the Tracer interface.

    // let's assume that we're in a thread Y and we've received
@@ -391,7 +391,7 @@ Span continuedSpan = 
    [Important]Important
    Always clean after you create a span! Don’t forget to detach a span if some work was done started in one
  thread (e.g. thread X) and it’s waiting for other threads (e.g. Y, Z) to finish.
  Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected
- the span in thread X should be closed.

    5.3 Creating spans with an explicit parent

    There is a possibility that you want to start a new span and provide an explicit parent of that span. + the span in thread X should be closed.

    6.3 Creating spans with an explicit parent

    There is a possibility that you want to start a new span and provide an explicit parent of that span. Let’s assume that the parent of a span is in one thread and you want to start a new span in another thread. The startSpan method of the Tracer interface is the method you are looking for.

    // let's assume that we're in a thread Y and we've received
 // the `initialSpan` from thread X. `initialSpan` will be the parent
@@ -411,9 +411,9 @@ Span newSpan = 
 	this.tracer.close(newSpan);
 }
    [Important]Important

    After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs related to the fact that you have a span present in the current thread other than the one you’re trying to close. - What’s worse your spans won’t get closed properly thus will not get collected to Zipkin.

    6. Naming spans

    Picking a span name is not a trivial task. Span name should depict an operation name. The name should + What’s worse your spans won’t get closed properly thus will not get collected to Zipkin.

    7. Naming spans

    Picking a span name is not a trivial task. Span name should depict an operation name. The name should be low cardinality (e.g. not include identifiers).

    Since there is a lot of instrumentation going on some of the span names will be -artificial like:

    • controller-method-name when received by a Controller with a method name conrollerMethodName
    • async for asynchronous operations done via wrapped Callable and Runnable.
    • @Scheduled annotated methods will return the simple name of the class.

    Fortunately, for the asynchronous processing you can provide explicit naming.

    6.1 @SpanName annotation

    You can name the span explicitly via the @SpanName annotation.

    @SpanName("calculateTax")
+artificial like:
    • controller-method-name when received by a Controller with a method name conrollerMethodName
    • async for asynchronous operations done via wrapped Callable and Runnable.
    • @Scheduled annotated methods will return the simple name of the class.
    Fortunately, for the asynchronous processing you can provide explicit naming.
    7.1 @SpanName annotation
    You can name the span explicitly via the @SpanName annotation.
    @SpanName("calculateTax")
 class TaxCountingRunnable implements Runnable {
 
 	@Override public void run() {
@@ -422,7 +422,7 @@ artificial like:
      Runnable runnable = new TraceRunnable(tracer, spanNamer, new TaxCountingRunnable());
 Future<?> future = executorService.submit(runnable);
 // ... some additional logic ...
-future.get();
    The span will be named calculateTax.
    6.2 toString() method
    It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous
+future.get();

    The span will be named calculateTax.

    7.2 toString() method

    It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous instance of those classes. You can’t annotate such classes thus to override that, if there is no @SpanName annotation present, we’re checking if the class has a custom implementation of the toString() method.

    So executing such code:

    Runnable runnable = new TraceRunnable(tracer, spanNamer, new Runnable() {
 	@Override public void run() {
@@ -435,12 +435,12 @@ we’re checking if the class has a custom implementation of the // ... some additional logic ...
-future.get();

    will lead in creating a span named calculateTax.

    7. Managing spans with annotations

    7.1 Rationale

    The main arguments for this features are

    • api-agnostic means to collaborate with a span

      • use of annotations allows users to add to a span with no library dependency on a span api. +future.get();

        will lead in creating a span named calculateTax.

    8. Managing spans with annotations

    8.1 Rationale

    The main arguments for this features are

    • api-agnostic means to collaborate with a span

      • use of annotations allows users to add to a span with no library dependency on a span api. This allows Sleuth to change its core api less impact to user code.

    • reduced surface area for basic span operations.

      • without this feature one has to use the span api, which has lifecycle commands that could be used incorrectly. By only exposing scope, tag and log functionality, users can collaborate without accidentally breaking span lifecycle.

    • collaboration with runtime generated code

      • with libraries such as Spring Data / Feign the implementations of interfaces are generated at runtime thus span wrapping of objects was tedious. Now you can provide annotations - over interfaces and arguments of those interfaces

    7.2 Creating new spans

    If you really don’t want to take care of creating local spans manually you can profit from the + over interfaces and arguments of those interfaces

    8.2 Creating new spans

    If you really don’t want to take care of creating local spans manually you can profit from the @NewSpan annotation. Also we give you the @SpanTag annotation to add tags in an automated fashion.

    Let’s look at some examples of usage.

    @NewSpan
 void testMethod();

    Annotating the method without any parameter will lead to a creation of a new span whose name @@ -458,28 +458,28 @@ the tag key will be testTag and the tag value will public void testMethod3() { }

    You can place the @NewSpan annotation on both the class and an interface. If you override the interface’s method and provide a different value of the @NewSpan annotation then the most -concrete one wins (in this case customNameOnTestMethod3 will be set).

    7.3 Continuing spans

    If you want to just add tags and annotations to an existing span it’s enough +concrete one wins (in this case customNameOnTestMethod3 will be set).

    8.3 Continuing spans

    If you want to just add tags and annotations to an existing span it’s enough to use the @ContinueSpan annotation as presented below. Note that in contrast with the @NewSpan annotation you can also add logs via the log parameter:

    // method declaration
 @ContinueSpan(log = "testMethod11")
 void testMethod11(@SpanTag("testTag11") String param);
 
 // method execution
-this.testBean.testMethod11("test");

    That way the span will get continued and:

    • logs with name testMethod11.before and testMethod11.after will be created
    • if an exception will be thrown a log testMethod11.afterFailure will also be created
    • tag with key testTag11 and value test will be created

    7.4 More advanced tag setting

    There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation. +this.testBean.testMethod11("test");

    That way the span will get continued and:

    • logs with name testMethod11.before and testMethod11.after will be created
    • if an exception will be thrown a log testMethod11.afterFailure will also be created
    • tag with key testTag11 and value test will be created

    8.4 More advanced tag setting

    There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation. Precedence is:

    • try with the bean of TagValueResolver type and provided name
    • if one hasn’t provided the bean name, try to evaluate an expression. We’re searching for a TagValueExpressionResolver bean. -The default implementation uses SPEL expression resolution.
    • if one hasn’t provided any expression to evaluate just return a toString() value of the parameter

    Custom extractor

    The value of the tag for following method will be computed by an implementation of TagValueResolver interface. +The default implementation uses SPEL expression resolution.

  • if one hasn’t provided any expression to evaluate just return a toString() value of the parameter

    • 8.4.1 Custom extractor

    The value of the tag for following method will be computed by an implementation of TagValueResolver interface. Its class name has to be passed as the value of the resolver attribute.

    Having such an annotated method:

    @NewSpan
 public void getAnnotationForTagValueResolver(@SpanTag(key = "test", resolver = TagValueResolver.class) String test) {
 }

    and such a TagValueResolver bean implementation

    @Bean(name = "myCustomTagValueResolver")
 public TagValueResolver tagValueResolver() {
 	return parameter -> "Value from myCustomTagValueResolver";
-}

    Will lead to setting of a tag value equal to Value from myCustomTagValueResolver.

    Resolving expressions for value

    Having such an annotated method:

    @NewSpan
+}

    Will lead to setting of a tag value equal to Value from myCustomTagValueResolver.

    8.4.2 Resolving expressions for value

    Having such an annotated method:

    @NewSpan
 public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "length() + ' characters'") String test) {
 }

    and no custom implementation of a TagValueExpressionResolver will lead to evaluation of the SPEL expression and a tag with value 4 characters will be set on the span. If you want to use some other expression resolution mechanism you can create your own implementation -of the bean.

    Using toString method

    Having such an annotated method:

    @NewSpan
+of the bean.

    8.4.3 Using toString method

    Having such an annotated method:

    @NewSpan
 public void getAnnotationForArgumentToString(@SpanTag("test") Long param) {
-}

    if executed with a value of 15 will lead to setting of a tag with a String value of "15".

    8. Customizations

    Thanks to the SpanInjector and SpanExtractor you can customize the way spans +}

    if executed with a value of 15 will lead to setting of a tag with a String value of "15".

    9. Customizations

    Thanks to the SpanInjector and SpanExtractor you can customize the way spans are created and propagated.

    There are currently two built-in ways to pass tracing information between processes:

    • via Spring Integration
    • via HTTP

    Span ids are extracted from Zipkin-compatible (B3) headers (either Message or HTTP headers), to start or join an existing trace. Trace information is injected into any outbound requests so the next hop can extract them.

    The key change in comparison to the previous versions of Sleuth is that Sleuth is implementing @@ -489,9 +489,9 @@ a SpanTextMap. This abstraction defines how one can how to retrieve it from there. Thanks to this if you want to instrument a new HTTP library that uses a FooRequest as a mean of sending HTTP requests then you have to create an implementation of a SpanTextMap that delegates calls to FooRequest in terms of retrieval -and insertion of HTTP headers.

    8.1 Spring Integration

    For Spring Integration there are 2 interfaces responsible for creation of a Span from a Message. -These are:

    • MessagingSpanTextMapExtractor
    • MessagingSpanTextMapInjector

    You can override them by providing your own implementation.

    8.2 HTTP

    For HTTP there are 2 interfaces responsible for creation of a Span from a Message. -These are:

    • HttpSpanExtractor
    • HttpSpanInjector

    You can override them by providing your own implementation.

    8.3 Example

    Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names +and insertion of HTTP headers.

    9.1 Spring Integration

    For Spring Integration there are 2 interfaces responsible for creation of a Span from a Message. +These are:

    • MessagingSpanTextMapExtractor
    • MessagingSpanTextMapInjector

    You can override them by providing your own implementation.

    9.2 HTTP

    For HTTP there are 2 interfaces responsible for creation of a Span from a Message. +These are:

    • HttpSpanExtractor
    • HttpSpanInjector

    You can override them by providing your own implementation.

    9.3 Example

    Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names you have

    • for trace id - correlationId
    • for span id - mySpanId

    This is a an example of a SpanExtractor

    static class CustomHttpSpanExtractor implements HttpSpanExtractor {
 
 	@Override public Span joinTrace(SpanTextMap carrier) {
@@ -578,7 +578,7 @@ that injects the headers into the Http Response and a Servlet filter which makes
 @Bean
 HttpResponseInjectingTraceFilter responseInjectingTraceFilter(Tracer tracer) {
 	return new HttpResponseInjectingTraceFilter(tracer, customHttpServletResponseSpanInjector());
-}

    8.4 TraceFilter

    You can also modify the behaviour of the TraceFilter - the component that is responsible +}

    9.4 TraceFilter

    You can also modify the behaviour of the TraceFilter - the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response. You can customize the tags, or modify the response headers by registering your own instance of the TraceFilter bean.

    In the following example we will register the TraceFilter bean and we will add the ZIPKIN-TRACE-ID response header containing the current Span’s trace id. Also we will @@ -596,7 +596,7 @@ TraceFilter myTraceFilter(BeanFactory beanFactory, "custom", "tag"); } }; -}

    8.5 Custom SA tag in Zipkin

    Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented. +}

    9.5 Custom SA tag in Zipkin

    Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented. What you can do is to create a span with the peer.service tag that will contain a value of the service that you want to call. Below you can see an example of a call to Redis that is wrapped in such a span.

    org.springframework.cloud.sleuth.Span newSpan = tracer.createSpan("redis");
 try {
@@ -611,32 +611,32 @@ Below you can see an example of a call to Redis that is wrapped in such a span.<
 	newSpan.tag("peer.port", "1234");
 	newSpan.logEvent(org.springframework.cloud.sleuth.Span.CLIENT_RECV);
 	tracer.close(newSpan);
-}
    [Important]Important

    Remember not to add both peer.service tag and the SA tag! You have to add only peer.service.

    8.6 Custom service name

    By default Sleuth assumes that when you send a span to Zipkin, you want the span’s service name +}

    [Important]Important

    Remember not to add both peer.service tag and the SA tag! You have to add only peer.service.

    9.6 Custom service name

    By default Sleuth assumes that when you send a span to Zipkin, you want the span’s service name to be equal to spring.application.name value. That’s not always the case though. There are situations in which you want to explicitly provide a different service name for all spans coming from your application. To achieve that it’s enough to just pass the following property - to your application to override that value (example for foo service name):

    spring.zipkin.service.name: foo

    8.7 Customization of reported spans

    Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way. + to your application to override that value (example for foo service name):

    spring.zipkin.service.name: foo

    9.7 Customization of reported spans

    Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way. You can achieve that by using the SpanAdjuster interface.

    Example of usage:

    In Sleuth we’re generating spans with a fixed name. Some users want to modify the name depending on values of tags. Implementation of the SpanAdjuster interface can be used to alter that name. Example:

    @Bean
 SpanAdjuster customSpanAdjuster() {
     return span -> span.toBuilder().name(scrub(span.getName())).build();
 }

    This will lead in changing the name of the reported span just before it gets sent to Zipkin.

    [Important]Important

    Your SpanReporter should inject the SpanAdjuster and - allow span manipulation before the actual reporting is done.

    8.8 Host locator

    In order to define the host that is corresponding to a particular span we need to resolve the host name + allow span manipulation before the actual reporting is done.

    9.8 Host locator

    In order to define the host that is corresponding to a particular span we need to resolve the host name and port. The default approach is to take it from server properties. If those for some reason are not set then we’re trying to retrieve the host name from the network interfaces.

    If you have the discovery client enabled and prefer to retrieve the host address from the registered instance in a service registry then you have to set the property (it’s applicable for both HTTP and -Stream based span reporting).

    spring.zipkin.locator.discovery.enabled: true

    9. Sending spans to Zipkin

    By default if you add spring-cloud-starter-zipkin as a dependency to your project, +Stream based span reporting).

    spring.zipkin.locator.discovery.enabled: true

    10. Sending spans to Zipkin

    By default if you add spring-cloud-starter-zipkin as a dependency to your project, when the span is closed, it will be sent to Zipkin over HTTP. The communication is asynchronous. You can configure the URL by setting the spring.zipkin.baseUrl property as follows:

    spring.zipkin.baseUrl: http://192.168.99.100:9411/

    If you want to find Zipkin via service discovery it’s enough to pass the -Zipkin’s service id inside the URL (example for zipkinserver service id)

    spring.zipkin.baseUrl: http://zipkinserver/

    10. Span Data as Messages

    You can accumulate and send span data over +Zipkin’s service id inside the URL (example for zipkinserver service id)

    spring.zipkin.baseUrl: http://zipkinserver/

    11. Span Data as Messages

    You can accumulate and send span data over Spring Cloud Stream by including the spring-cloud-sleuth-stream jar as a dependency, and adding a Channel Binder implementation (e.g. spring-cloud-starter-stream-rabbit for RabbitMQ or spring-cloud-starter-stream-kafka for Kafka). This will automatically turn your app into a producer of messages with payload -type Spans.

    10.1 Zipkin Consumer

    There is a special convenience annotation for setting up a message consumer +type Spans.

    11.1 Zipkin Consumer

    There is a special convenience annotation for setting up a message consumer for the Span data and pushing it into a Zipkin SpanStore. This application

    @SpringBootApplication
 @EnableZipkinStreamServer
 public class Consumer {
@@ -670,7 +670,7 @@ started quickly). For a more robust solution you can add MySQL and
     type: mysql
    [Note]Note

    The @EnableZipkinStreamServer is also annotated with @EnableZipkinServer so the process will also expose the standard Zipkin server endpoints for collecting spans over HTTP, and for -querying in the Zipkin Web UI.

    10.2 Custom Consumer

    A custom consumer can also easily be implemented using +querying in the Zipkin Web UI.

    11.2 Custom Consumer

    A custom consumer can also easily be implemented using spring-cloud-sleuth-stream and binding to the SleuthSink. Example:

    @EnableBinding(SleuthSink.class)
 @SpringBootApplication(exclude = SleuthStreamAutoConfiguration.class)
 @MessageEndpoint
@@ -694,11 +694,11 @@ with name equal to StreamSpanReporter.POLLER. Here
 		poller.setTrigger(new PeriodicTrigger(5000L));
 		return poller;
 	}
-}

    11. Metrics

    Currently Spring Cloud Sleuth registers very simple metrics related to spans. +}

    12. Metrics

    Currently Spring Cloud Sleuth registers very simple metrics related to spans. It’s using the Spring Boot’s metrics support to calculate the number of accepted and dropped spans. Each time a span gets sent to Zipkin the number of accepted spans will increase. If there’s an error then -the number of dropped spans will get increased.

    12. Integrations

    12.1 Runnable and Callable

    If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative.

    Example for Runnable:

    Runnable runnable = new Runnable() {
+the number of dropped spans will get increased.

    13. Integrations

    13.1 Runnable and Callable

    If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative.

    Example for Runnable:

    Runnable runnable = new Runnable() {
 	@Override
 	public void run() {
 		// do some work
@@ -728,10 +728,10 @@ Runnable traceRunnableFromTracer = tracer.wrap(runnable);

    Example for new TraceCallable<>(tracer, spanNamer, callable, "calculateTax"); // Wrapping `Callable` with `Tracer`. The Span name will be taken either from the // `@SpanName` annotation or from `toString` method -Callable<String> traceCallableFromTracer = tracer.wrap(callable);

    That way you will ensure that a new Span is created and closed for each execution.

    12.2 Hystrix

    Custom Concurrency Strategy

    We’re registering a custom HystrixConcurrencyStrategy +Callable<String> traceCallableFromTracer = tracer.wrap(callable);

    That way you will ensure that a new Span is created and closed for each execution.

    13.2 Hystrix

    13.2.1 Custom Concurrency Strategy

    We’re registering a custom HystrixConcurrencyStrategy that wraps all Callable instances into their Sleuth representative - the TraceCallable. The strategy either starts or continues a span depending on the fact whether tracing was already going -on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false.

    Manual Command setting

    Assuming that you have the following HystrixCommand:

    HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
+on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false.

    13.2.2 Manual Command setting

    Assuming that you have the following HystrixCommand:

    HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
 	@Override
 	protected String run() throws Exception {
 		return someLogic();
@@ -742,22 +742,22 @@ on before the Hystrix command was called. To disable the custom Hystrix Concurre
 	public String doRun() throws Exception {
 		return someLogic();
 	}
-};

    12.3 RxJava

    We’re registering a custom RxJavaSchedulersHook +};

    13.3 RxJava

    We’re registering a custom RxJavaSchedulersHook that wraps all Action0 instances into their Sleuth representative - the TraceAction. The hook either starts or continues a span depending on the fact whether tracing was already going on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the spring.sleuth.rxjava.schedulers.hook.enabled to false.

    You can define a list of regular expressions for thread names, for which you don’t want a Span to be created. Just provide a comma separated list -of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

    12.4 HTTP integration

    Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false.

    HTTP Filter

    Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which +of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

    13.4 HTTP integration

    Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false.

    13.4.1 HTTP Filter

    Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern.

    HandlerInterceptor

    Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an + its value of contextPath gets appended to the provided skip pattern.

    13.4.2 HandlerInterceptor

    Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. The TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. If the the TraceFilter doesn’t see this attribute set it will create a "fallback" span which is an additional span created on the server side so that the trace is presented properly in the UI. Seeing that most likely - signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.

    Async Servlet support

    If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one.

    12.5 HTTP client integration

    Synchronous Rest Template

    We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a + signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.

    13.4.3 Async Servlet support

    If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one.

    13.5 HTTP client integration

    13.5.1 Synchronous Rest Template

    We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous RestTemplate features just set spring.sleuth.web.client.enabled to false.

    [Important]Important

    You have to register RestTemplate as a bean so that the interceptors will get injected. -If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work.

    Asynchronous Rest Template

    [Important]Important

    A traced version of an AsyncRestTemplate bean is registered for you out of the box. If you +If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work.

    13.5.2 Asynchronous Rest Template

    [Important]Important

    A traced version of an AsyncRestTemplate bean is registered for you out of the box. If you have your own bean you have to wrap it in a TraceAsyncRestTemplate representation. The best solution is to only customize the ClientHttpRequestFactory and / or AsyncClientHttpRequestFactory. If you have your own AsyncRestTemplate and you don’t wrap it your calls WILL NOT GET TRACED.

    Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the ClientHttpRequestFactory @@ -777,7 +777,7 @@ wrap ThreadPoolTaskScheduler in a AsyncRestTemplate features set spring.sleuth.web.async.client.enabled to false. To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper set spring.sleuth.web.async.client.factory.enabled -to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false.

    Multiple Asynchronous Rest Templates

    Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you +to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false.

    Multiple Asynchronous Rest Templates

    Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you can see an example of how to set up such a custom AsyncRestTemplate.

    @Configuration
 @EnableAutoConfiguration
 static class Config {
@@ -811,27 +811,27 @@ can see an example of how to set up such a custom AsyncRes
 		//CUSTOMIZE HERE
 		return factory;
 	}
-}

    Traverson

    If you’re using the Traverson library +}

    13.5.3 Traverson

    If you’re using the Traverson library it’s enough for you to inject a RestTemplate as a bean into your Traverson object. Since RestTemplate is already intercepted, you will get full support of tracing in your client. Below you can find a pseudo code of how to do that:

    @Autowired RestTemplate restTemplate;
 
 Traverson traverson = new Traverson(URI.create("http://some/address"),
     MediaType.APPLICATION_JSON, MediaType.APPLICATION_JSON_UTF8).setRestOperations(restTemplate);
-// use Traverson

    12.6 Feign

    By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely +// use Traverson

    13.6 Feign

    By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely by setting spring.sleuth.feign.enabled to false. If you do so then no Feign related instrumentation will take place.

    Part of Feign instrumentation is done via a FeignBeanPostProcessor. You can disable it by providing the spring.sleuth.feign.processor.enabled equal to false. If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation -however will be still there.

    12.7 Asynchronous communication

    @Async annotated methods

    In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads. -You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false.

    If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics:

    • if the method is annotated with @SpanName then the value of the annotation will be the Span’s name
    • if the method is not annotated with @SpanName the Span name will be the annotated method name
    • the Span will be tagged with that method’s class name and the method name too

    @Scheduled annotated methods

    In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour +however will be still there.

    13.7 Asynchronous communication

    13.7.1 @Async annotated methods

    In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads. +You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false.

    If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics:

    • if the method is annotated with @SpanName then the value of the annotation will be the Span’s name
    • if the method is not annotated with @SpanName the Span name will be the annotated method name
    • the Span will be tagged with that method’s class name and the method name too

    13.7.2 @Scheduled annotated methods

    In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour by setting the value of spring.sleuth.scheduled.enabled to false.

    If you annotate your method with @Scheduled then we’ll automatically create a new Span with the following characteristics:

    • the Span name will be the annotated method name
    • the Span will be tagged with that method’s class name and the method name too

    If you want to skip Span creation for some @Scheduled annotated classes you can set the spring.sleuth.scheduled.skipPattern with a regular expression that will match the fully qualified name of the -@Scheduled annotated class.

    [Tip]Tip

    If you are using spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask

    Executor, ExecutorService and ScheduledExecutorService

    We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations +@Scheduled annotated class.

    [Tip]Tip

    If you are using spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask

    13.7.3 Executor, ExecutorService and ScheduledExecutorService

    We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations are creating Spans each time a new task is submitted, invoked or scheduled.

    Here you can see an example of how to pass tracing information with TraceableExecutorService when working with CompletableFuture:

    CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> {
 	// perform some logic
 	return 1_000_000L;
 }, new TraceableExecutorService(executorService,
 		// 'calculateTax' explicitly names the span - this param is optional
-		tracer, traceKeys, spanNamer, "calculateTax"));
    Customization of Executors

    Sometimes you need to set up a custom instance of the AsyncExecutor. In the following snippet you + tracer, traceKeys, spanNamer, "calculateTax"));

    Customization of Executors

    Sometimes you need to set up a custom instance of the AsyncExecutor. In the following snippet you can see an example of how to set up such a custom Executor.

    @Configuration
 @EnableAutoConfiguration
 @EnableAsync
@@ -850,9 +850,9 @@ can see an example of how to set up such a custom Executor
 		executor.initialize();
 		return new LazyTraceExecutor(this.beanFactory, executor);
 	}
-}

    12.8 Messaging

    Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and +}

    13.8 Messaging

    Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and subscribe events. To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false.

    You can provide the spring.sleuth.integration.patterns pattern to explicitly provide the names of channels that you want to include for tracing. By default all channels are included.

    [Important]Important

    When using the Executor to build a Spring Integration IntegrationFlow remember to use the untraced version of the Executor. -Decorating Spring Integration Executor Channel with TraceableExecutorService will cause the spans to be improperly closed.

    12.9 Zuul

    We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data). -To disable Zuul support set the spring.sleuth.zuul.enabled property to false.

    13. Running examples

    You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links:

    \ No newline at end of file +Decorating Spring Integration Executor Channel with TraceableExecutorService will cause the spans to be improperly closed.

    13.9 Zuul

    We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data). +To disable Zuul support set the spring.sleuth.zuul.enabled property to false.

    14. Running examples

    You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links:

    \ No newline at end of file diff --git a/spring-cloud-sleuth.xml b/spring-cloud-sleuth.xml index a1e290955..133f4199c 100644 --- a/spring-cloud-sleuth.xml +++ b/spring-cloud-sleuth.xml @@ -15,8 +15,10 @@ 1.3.0.BUILD-SNAPSHOT -Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud. + +Introduction +Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.
    Terminology Spring Cloud Sleuth borrows Dapper’s terminology. @@ -663,6 +665,7 @@ public class ZipkinStreamServerApplication { }
    +     Additional resources Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin