@SpanName AnnotationtoString() methodX-B3-Tr
+
+
+
+4. Introduction to Brave
+
+
+Brave is a distributed tracing instrumentation library. Brave typically
+intercepts production requests to gather timing data, correlate and propagate
+trace contexts. While typically trace data is sent to Zipkin server,
+third-party plugins are available to send to alternate services such as Amazon
+X-Ray.
+
+
+Spring Cloud Sleuth is a layer over Brave.
+It configures everything you need to get started with tracing. Sleuth
+configures where trace data (spans) are reported to, how many traces to keep
+(sampling), if remote fields (baggage) and which libraries are traced.
+Sleuth also adds annotation based tracing features and some instrumentation not
+available otherwise, such as Reactor.
+
-3.1. Introduction to Brave
-
-
-
-
-
-
-
-Starting with version 2.0.0, Spring Cloud Sleuth uses
-Brave as the tracing library.
-For your convenience, we embed part of the Brave’s docs here.
-
-
-
-
-
-
-
-
-
-
-
-In the vast majority of cases you need to just use the Tracer
-or SpanCustomizer beans from Brave that Sleuth provides. The documentation below contains
-a high overview of what Brave is and how it works.
-
-
-
+4.1. Brave Basics
+
+Most instrumentation work is done for you by default. Sleuth provides beans to
+allow you to change what’s traced, and it even provides annotations to avoid
+using tracing libraries! All of this is explained later in this document.
-Brave is a library used to capture and report latency information about distributed operations to Zipkin.
-Most users do not use Brave directly. They use libraries or frameworks rather than employ Brave on their behalf.
+That said, you might want to know more about how things work underneath. Here
+are some pointers.
-This module includes a tracer that creates and joins spans that model the latency of potentially distributed work.
-It also includes libraries to propagate the trace context over network boundaries (for example, with HTTP headers).
-
-
-3.1.1. Tracing
-
-Most importantly, you need a brave.Tracer, configured to report to Zipkin.
+Here are the most core types you might use:
+* SpanCustomizer - to change the span currently in progress
+* Tracer - to get a start new spans ad-hoc
-The following example setup sends trace data (spans) to Zipkin over HTTP (as opposed to Kafka):
-
-
-
-class MyClass {
-
- private final Tracer tracer;
-
- // Tracer will be autowired
- MyClass(Tracer tracer) {
- this.tracer = tracer;
- }
-
- void doSth() {
- Span span = tracer.newTrace().name("encode").start();
- // ...
- }
-}
-
-
-
-
-
-
-
-
-
-If your span contains a name longer than 50 chars, then that name is truncated to 50 chars.
-Your names have to be explicit and concrete.
-Big names lead to latency issues and sometimes even thrown exceptions.
-
-
-
-
-
-The tracer creates and joins spans that model the latency of potentially distributed work.
-It can employ sampling to reduce overhead during the process, to reduce the amount of data sent to Zipkin, or both.
-
-
-Spans returned by a tracer report data to Zipkin when finished or do nothing if unsampled.
-After starting a span, you can annotate events of interest or add tags containing details or lookup keys.
-
-
-Spans have a context that includes trace identifiers that place the span at the correct spot in the tree representing the distributed operation.
-
-
-
-3.1.2. Local Tracing
-
-When tracing code that never leaves your process, run it inside a scoped span.
-
-
-
-@Autowired Tracer tracer;
-
-// Start a new trace or a span within an existing trace representing an operation
-ScopedSpan span = tracer.startScopedSpan("encode");
-try {
- // The span is in "scope" meaning downstream code such as loggers can see trace IDs
- return encoder.encode();
-} catch (RuntimeException | Error e) {
- span.error(e); // Unless you handle exceptions, you might not know the operation failed!
- throw e;
-} finally {
- span.finish(); // always finish the span
-}
-
-
-
-When you need more features, or finer control, use the Span type:
-
-
-
-@Autowired Tracer tracer;
-
-// Start a new trace or a span within an existing trace representing an operation
-Span span = tracer.nextSpan().name("encode").start();
-// Put the span in "scope" so that downstream code such as loggers can see trace IDs
-try (SpanInScope ws = tracer.withSpanInScope(span)) {
- return encoder.encode();
-} catch (RuntimeException | Error e) {
- span.error(e); // Unless you handle exceptions, you might not know the operation failed!
- throw e;
-} finally {
- span.finish(); // note the scope is independent of the span. Always finish a span.
-}
-
-
-
-Both of the above examples report the exact same span on finish!
-
-
-In the above example, the span will be either a new root span or the
-next child in an existing trace.
-
-
-
-3.1.3. Customizing Spans
-
-Once you have a span, you can add tags to it.
-The tags can be used as lookup keys or details.
-For example, you might add a tag with your runtime version, as shown in the following example:
-
-
-
-span.tag("clnt/finagle.version", "6.36.0");
-
-
-
-When exposing the ability to customize spans to third parties, prefer brave.SpanCustomizer as opposed to brave.Span.
-The former is simpler to understand and test and does not tempt users with span lifecycle hooks.
-
-
-
-interface MyTraceCallback {
- void request(Request request, SpanCustomizer customizer);
-}
-
-
-
-Since brave.Span implements brave.SpanCustomizer, you can pass it to users, as shown in the following example:
-
-
-
-for (MyTraceCallback callback : userCallbacks) {
- callback.request(request, span);
-}
-
-
-
-
-3.1.4. Implicitly Looking up the Current Span
-
-Sometimes, you do not know if a trace is in progress or not, and you do not want users to do null checks.
-brave.CurrentSpanCustomizer handles this problem by adding data to any span that’s in progress or drops, as shown in the following example:
-
-
-Ex.
-
-
-
-// The user code can then inject this without a chance of it being null.
-@Autowired SpanCustomizer span;
-
-void userCode() {
- span.annotate("tx.started");
- ...
-}
-
-
-
-
-3.1.5. RPC tracing
-
-
-
-
-
-
-
-Check for instrumentation written here and Zipkin’s list before rolling your own RPC instrumentation.
-
-
-
-
-
-RPC tracing is often done automatically by interceptors. Behind the scenes, they add tags and events that relate to their role in an RPC operation.
-
-
-The following example shows how to add a client span:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// before you send a request, add metadata that describes the operation
-span = tracer.nextSpan().name(service + "/" + method).kind(CLIENT);
-span.tag("myrpc.version", "1.0.0");
-span.remoteServiceName("backend");
-span.remoteIpAndPort("172.3.4.1", 8108);
-
-// Add the trace context to the request, so it can be propagated in-band
-tracing.propagation().injector(Request::addHeader)
- .inject(span.context(), request);
-
-// when the request is scheduled, start the span
-span.start();
-
-// if there is an error, tag the span
-span.tag("error", error.getCode());
-// or if there is an exception
-span.error(exception);
-
-// when the response is complete, finish the span
-span.finish();
-
-
-
-One-Way tracing
-
-Sometimes, you need to model an asynchronous operation where there is a
-request but no response. In normal RPC tracing, you use span.finish()
-to indicate that the response was received. In one-way tracing, you use
-span.flush() instead, as you do not expect a response.
-
-
-The following example shows how a client might model a one-way operation:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// start a new span representing a client request
-oneWaySend = tracer.nextSpan().name(service + "/" + method).kind(CLIENT);
-
-// Add the trace context to the request, so it can be propagated in-band
-tracing.propagation().injector(Request::addHeader)
- .inject(oneWaySend.context(), request);
-
-// fire off the request asynchronously, totally dropping any response
-request.execute();
-
-// start the client side and flush instead of finish
-oneWaySend.start().flush();
-
-
-
-The following example shows how a server might handle a one-way operation:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// pull the context out of the incoming request
-extractor = tracing.propagation().extractor(Request::getHeader);
-
-// convert that context to a span which you can name and add tags to
-oneWayReceive = nextSpan(tracer, extractor.extract(request))
- .name("process-request")
- .kind(SERVER)
- ... add tags etc.
-
-// start the server side and flush instead of finish
-oneWayReceive.start().flush();
-
-// you should not modify this span anymore as it is complete. However,
-// you can create children to represent follow-up work.
-next = tracer.newSpan(oneWayReceive.context()).name("step2").start();
-
-
-
+Here are the most relevant links from the OpenZipkin Brave project:
+* [Brave’s core library](github.com/openzipkin/brave/tree/master/brave)
+* [Baggage (propagated fields)](github.com/openzipkin/brave/tree/master/brave#baggage)
+* [HTTP tracing](github.com/openzipkin/brave/tree/master/instrumentation/http)
-4. Sampling
+5. Sampling
-Sampling may be employed to reduce the data collected and reported out of process.
-When a span is not sampled, it adds no overhead (a noop).
-
-
-Sampling is an up-front decision, meaning that the decision to report data is made at the first operation in a trace and that decision is propagated downstream.
-
-
-By default, a global sampler applies a single rate to all traced operations.
-Tracer.Builder.sampler controls this setting, and it defaults to tracing every request.
-
-
-4.1. Declarative sampling
-
-Some applications need to sample based on the type or annotations of a java method.
-
-
-Most users use a framework interceptor to automate this sort of policy.
-The following example shows how that might work internally:
-
-
-
-@Autowired Tracer tracer;
-
-// derives a sample rate from an annotation on a java method
-DeclarativeSampler<Traced> sampler = DeclarativeSampler.create(Traced::sampleRate);
-
-@Around("@annotation(traced)")
-public Object traceThing(ProceedingJoinPoint pjp, Traced traced) throws Throwable {
- // When there is no trace in progress, this decides using an annotation
- Sampler decideUsingAnnotation = declarativeSampler.toSampler(traced);
- Tracer tracer = tracer.withSampler(decideUsingAnnotation);
-
- // This code looks the same as if there was no declarative override
- ScopedSpan span = tracer.startScopedSpan(spanName(pjp));
- try {
- return pjp.proceed();
- } catch (RuntimeException | Error e) {
- span.error(e);
- throw e;
- } finally {
- span.finish();
- }
-}
-
-
-
-
-4.2. Custom sampling
-
-Depending on what the operation is, you may want to apply different policies.
-For example, you might not want to trace requests to static resources such as images, or you might want to trace all requests to a new api.
-
-
-Most users use a framework interceptor to automate this sort of policy.
-The following example shows how that might work internally:
-
-
-
-@Autowired Tracer tracer;
-@Autowired Sampler fallback;
-
-Span nextSpan(final Request input) {
- Sampler requestBased = Sampler() {
- @Override public boolean isSampled(long traceId) {
- if (input.url().startsWith("/experimental")) {
- return true;
- } else if (input.url().startsWith("/static")) {
- return false;
- }
- return fallback.isSampled(traceId);
- }
- };
- return tracer.withSampler(requestBased).nextSpan();
-}
-
-
-
-
-4.3. Sampling in Spring Cloud Sleuth
-
By default Spring Cloud Sleuth sets all spans to non-exportable.
That means that traces appear in logs but not in any remote store.
For testing the default is often enough, and it probably is all you need if you use only the logs (for example, with an ELK aggregator).
@@ -1669,7 +1316,7 @@ public Sampler defaultSampler() {
-You can set the HTTP header X-B3-Flags to 1, or, when doing messaging, you can set the spanFlags header to 1.
+You can set the HTTP header b3 to 1, or, when doing messaging, you can set the spanFlags header to 1.
Doing so forces the current span to be exportable regardless of the sampling decision.
@@ -1680,165 +1327,14 @@ Doing so forces the current span to be exportable regardless of the sampling dec
-
-5. Propagation
+6. Baggage
-Propagation is needed to ensure activities originating from the same root are collected together in the same trace.
-The most common propagation approach is to copy a trace context from a client by sending an RPC request to a server receiving it.
-
-
-For example, when a downstream HTTP call is made, its trace context is encoded as request headers and sent along with it, as shown in the following image:
-
-
-
- Client Span Server Span
-┌──────────────────┐ ┌──────────────────┐
-│ │ │ │
-│ TraceContext │ Http Request Headers │ TraceContext │
-│ ┌──────────────┐ │ ┌───────────────────┐ │ ┌──────────────┐ │
-│ │ TraceId │ │ │ X─B3─TraceId │ │ │ TraceId │ │
-│ │ │ │ │ │ │ │ │ │
-│ │ ParentSpanId │ │ Extract │ X─B3─ParentSpanId │ Inject │ │ ParentSpanId │ │
-│ │ ├─┼─────────>│ ├────────┼>│ │ │
-│ │ SpanId │ │ │ X─B3─SpanId │ │ │ SpanId │ │
-│ │ │ │ │ │ │ │ │ │
-│ │ Sampled │ │ │ X─B3─Sampled │ │ │ Sampled │ │
-│ └──────────────┘ │ └───────────────────┘ │ └──────────────┘ │
-│ │ │ │
-└──────────────────┘ └──────────────────┘
-
-
-
-The names above are from B3 Propagation, which is built-in to Brave and has implementations in many languages and frameworks.
-
-
-Most users use a framework interceptor to automate propagation.
-The next two examples show how that might work for a client and a server.
-
-
-The following example shows how client-side propagation might work:
-
-
-
-@Autowired Tracing tracing;
-
-// configure a function that injects a trace context into a request
-injector = tracing.propagation().injector(Request.Builder::addHeader);
-
-// before a request is sent, add the current span's context to it
-injector.inject(span.context(), request);
-
-
-
-The following example shows how server-side propagation might work:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// configure a function that extracts the trace context from a request
-extractor = tracing.propagation().extractor(Request::getHeader);
-
-// when a server receives a request, it joins or starts a new trace
-span = tracer.nextSpan(extractor.extract(request));
-
-
-
-5.1. Propagating extra fields
-
-Sometimes you need to propagate extra fields, such as a request ID or an alternate trace context.
-For example, if you are in a Cloud Foundry environment, you might want to pass the request ID, as shown in the following example:
-
-
-
-// when you initialize the builder, define the extra field you want to propagate
-Tracing.newBuilder().propagationFactory(
- ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-vcap-request-id")
-);
-
-// later, you can tag that request ID or use it in log correlation
-requestId = ExtraFieldPropagation.get("x-vcap-request-id");
-
-
-
-You may also need to propagate a trace context that you are not using.
-For example, you may be in an Amazon Web Services environment but not be reporting data to X-Ray.
-To ensure X-Ray can co-exist correctly, pass-through its tracing header, as shown in the following example:
-
-
-
-tracingBuilder.propagationFactory(
- ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-amzn-trace-id")
-);
-
-
-
-
-
-
-
-
-
-In Spring Cloud Sleuth all elements of the tracing builder Tracing.newBuilder()
-are defined as beans. So if you want to pass a custom PropagationFactory, it’s enough
-for you to create a bean of that type and we will set it in the Tracing bean.
-
-
-
-
-
-5.1.1. Prefixed fields
-
-If they follow a common pattern, you can also prefix fields.
-The following example shows how to propagate x-vcap-request-id the field as-is but send the country-code and user-id fields on the wire as x-baggage-country-code and x-baggage-user-id, respectively:
-
-
-
-Tracing.newBuilder().propagationFactory(
- ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY)
- .addField("x-vcap-request-id")
- .addPrefixedFields("x-baggage-", Arrays.asList("country-code", "user-id"))
- .build()
-);
-
-
-
-Later, you can call the following code to affect the country code of the current trace context:
-
-
-
-ExtraFieldPropagation.set("x-country-code", "FO");
-String countryCode = ExtraFieldPropagation.get("x-country-code");
-
-
-
-Alternatively, if you have a reference to a trace context, you can use it explicitly, as shown in the following example:
-
-
-
-ExtraFieldPropagation.set(span.context(), "x-country-code", "FO");
-String countryCode = ExtraFieldPropagation.get(span.context(), "x-country-code");
-
-
-
-
-
-
-
-
-
-A difference from previous versions of Sleuth is that, with Brave, you must pass the list of baggage keys.
-There are the following properties to achieve this.
-With the spring.sleuth.baggage-keys, you set keys that get prefixed with baggage- for HTTP calls and baggage_ for messaging.
+With the spring.sleuth.baggage-keys, you set keys that get prefixed with baggage- for HTTP calls and baggage_ for messaging.
You can also use the spring.sleuth.propagation-keys property to pass a list of prefixed keys that are propagated to remote services without any prefix.
You can also use the spring.sleuth.local-keys property to pass a list keys that will be propagated locally but will not be propagated over the wire.
-Notice that there’s no x- in front of the header keys.
-
-
-
+Notice that there’s no x- in front of the header keys.
In order to automatically set the baggage values to Slf4j’s MDC, you have to set
@@ -1862,169 +1358,9 @@ Remember that adding entries to MDC can drastically decrease the performance of
spring.sleuth.propagation.tag.whitelisted-keys with a list of whitelisted baggage keys. To disable the feature you have to pass the spring.sleuth.propagation.tag.enabled=false property.
-
-5.1.2. Extracting a Propagated Context
-
-The TraceContext.Extractor<C> reads trace identifiers and sampling status from an incoming request or message.
-The carrier is usually a request object or headers.
-
-
-This utility is used in standard instrumentation (such as HttpServerHandler) but can also be used for custom RPC or messaging code.
-
-
-TraceContextOrSamplingFlags is usually used only with Tracer.nextSpan(extracted), unless you are
-sharing span IDs between a client and a server.
-
-
-
-5.1.3. Sharing span IDs between Client and Server
-
-A normal instrumentation pattern is to create a span representing the server side of an RPC.
-Extractor.extract might return a complete trace context when applied to an incoming client request.
-Tracer.joinSpan attempts to continue this trace, using the same span ID if supported or creating a child span
-if not. When the span ID is shared, the reported data includes a flag saying so.
-
-
-The following image shows an example of B3 propagation:
-
-
-
- ┌───────────────────┐ ┌───────────────────┐
- Incoming Headers │ TraceContext │ │ TraceContext │
-┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
-│ X─B3-TraceId │─────────┼─┼> TraceId │ │──────┼─┼> TraceId │ │
-│ │ │ │ │ │ │ │ │ │
-│ X─B3-ParentSpanId │─────────┼─┼> ParentSpanId │ │──────┼─┼> ParentSpanId │ │
-│ │ │ │ │ │ │ │ │ │
-│ X─B3-SpanId │─────────┼─┼> SpanId │ │──────┼─┼> SpanId │ │
-└───────────────────┘ │ │ │ │ │ │ │ │
- │ │ │ │ │ │ Shared: true │ │
- │ └───────────────┘ │ │ └───────────────┘ │
- └───────────────────┘ └───────────────────┘
-
-
-
-Some propagation systems forward only the parent span ID, detected when Propagation.Factory.supportsJoin() == false.
-In this case, a new span ID is always provisioned, and the incoming context determines the parent ID.
-
-
-The following image shows an example of AWS propagation:
-
-
-
- ┌───────────────────┐ ┌───────────────────┐
- x-amzn-trace-id │ TraceContext │ │ TraceContext │
-┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
-│ Root │─────────┼─┼> TraceId │ │──────┼─┼> TraceId │ │
-│ │ │ │ │ │ │ │ │ │
-│ Parent │─────────┼─┼> SpanId │ │──────┼─┼> ParentSpanId │ │
-└───────────────────┘ │ └───────────────┘ │ │ │ │ │
- └───────────────────┘ │ │ SpanId: New │ │
- │ └───────────────┘ │
- └───────────────────┘
-
-
-
-Note: Some span reporters do not support sharing span IDs.
-For example, if you set Tracing.Builder.spanReporter(amazonXrayOrGoogleStackdrive), you should disable join by setting Tracing.Builder.supportsJoin(false).
-Doing so forces a new child span on Tracer.joinSpan().
-
-
-
-5.1.4. Implementing Propagation
-
-TraceContext.Extractor<C> is implemented by a Propagation.Factory plugin.
-Internally, this code creates the union type, TraceContextOrSamplingFlags, with one of the following:
-* TraceContext if trace and span IDs were present.
-* TraceIdContext if a trace ID was present but span IDs were not present.
-* SamplingFlags if no identifiers were present.
-
-
-Some Propagation implementations carry extra data from the point of extraction (for example, reading incoming headers) to injection (for example, writing outgoing headers).
-For example, it might carry a request ID.
-When implementations have extra data, they handle it as follows:
-* If a TraceContext were extracted, add the extra data as TraceContext.extra().
-* Otherwise, add it as TraceContextOrSamplingFlags.extra(), which Tracer.nextSpan handles.
-
-
-
-
-6. Current Tracing Component
-
-
-Brave supports a "current tracing component" concept, which should only be used when you have no other way to get a reference.
-This was made for JDBC connections, as they often initialize prior to the tracing component.
-
-
-The most recent tracing component instantiated is available through Tracing.current().
-You can also use Tracing.currentTracer() to get only the tracer.
-If you use either of these methods, do not cache the result.
-Instead, look them up each time you need them.
-
-
-
-
-7. Current Span
-
-
-Brave supports a "current span" concept which represents the in-flight operation.
-You can use Tracer.currentSpan() to add custom tags to a span and Tracer.nextSpan() to create a child of whatever is in-flight.
-
-
-
-
-
-
-
-
-In Sleuth, you can autowire the Tracer bean to retrieve the current span via
-tracer.currentSpan() method. To retrieve the current context just call
-tracer.currentSpan().context(). To get the current trace id as String
-you can use the traceIdString() method like this: tracer.currentSpan().context().traceIdString().
-
-
-
-
-
-7.1. Setting a span in scope manually
-
-When writing new instrumentation, it is important to place a span you created in scope as the current span.
-Not only does doing so let users access it with Tracer.currentSpan(), but it also allows customizations such as SLF4J MDC to see the current trace IDs.
-
-
-Tracer.withSpanInScope(Span) facilitates this and is most conveniently employed by using the try-with-resources idiom.
-Whenever external code might be invoked (such as proceeding an interceptor or otherwise), place the span in scope, as shown in the following example:
-
-
-
-@Autowired Tracer tracer;
-
-try (SpanInScope ws = tracer.withSpanInScope(span)) {
- return inboundRequest.invoke();
-} finally { // note the scope is independent of the span
- span.finish();
-}
-
-
-
-In edge cases, you may need to clear the current span temporarily (for example, launching a task that should not be associated with the current request). To do tso, pass null to withSpanInScope, as shown in the following example:
-
-
-
-@Autowired Tracer tracer;
-
-try (SpanInScope cleared = tracer.withSpanInScope(null)) {
- startBackgroundThread();
-}
-
-
-
-
-
-
-8. Instrumentation
+7. Instrumentation
Spring Cloud Sleuth automatically instruments all your Spring applications, so you should not have to do anything to activate it.
@@ -2050,7 +1386,7 @@ Tags are collected and exported only if there is a Sampler that all
-9. Span lifecycle
+8. Span lifecycle
You can do the following operations on the Span by means of brave.Tracer:
@@ -2089,7 +1425,7 @@ Spring Cloud Sleuth creates an instance of Tracer for you. In order
-9.1. Creating and finishing spans
+8.1. Creating and finishing spans
You can manually create spans by using the Tracer, as shown in the following example:
@@ -2144,7 +1480,7 @@ Your names have to be explicit and concrete. Big names lead to latency issues an
-9.2. Continuing Spans
+8.2. Continuing Spans
Sometimes, you do not want to create a new span but you want to continue one. An example of such a
situation might be as follows:
@@ -2181,7 +1517,7 @@ finally {
-9.3. Creating a Span with an explicit Parent
+8.3. Creating a Span with an explicit Parent
You might want to start a new span and provide an explicit parent of that span.
Assume that the parent of a span is in one thread and you want to start a new span in another thread.
@@ -2229,7 +1565,7 @@ After creating such a span, you must finish it. Otherwise it is not reported (fo
-10. Naming spans
+9. Naming spans
Picking a span name is not a trivial task. A span name should depict an operation name.
@@ -2255,7 +1591,7 @@ The name should be low cardinality, so it should not include identifiers.
Fortunately, for asynchronous processing, you can provide explicit naming.
-10.1. @SpanName Annotation
+9.1. @SpanName Annotation
You can name the span explicitly by using the @SpanName annotation, as shown in the following example:
@@ -2286,7 +1622,7 @@ future.get();
-10.2. toString() method
+9.2. toString() method
It is pretty rare to create separate classes for Runnable or Callable.
Typically, one creates an anonymous instance of those classes.
@@ -2318,13 +1654,13 @@ future.get();
-11. Managing Spans with Annotations
+10. Managing Spans with Annotations
You can manage spans with a variety of annotations.
-11.1. Rationale
+10.1. Rationale
There are a number of good reasons to manage spans with annotations, including:
@@ -2347,7 +1683,7 @@ Now you can provide annotations over interfaces and the arguments of those inter
-11.2. Creating New Spans
+10.2. Creating New Spans
If you do not want to create local spans manually, you can use the @NewSpan annotation.
Also, we provide the @SpanTag annotation to add tags in an automated fashion.
@@ -2403,7 +1739,7 @@ concrete one wins (in this case customNameOnTestMethod3 is set).
-11.3. Continuing Spans
+10.3. Continuing Spans
If you want to add tags and annotations to an existing span, you can use the @ContinueSpan annotation, as shown in the following example:
@@ -2439,7 +1775,7 @@ this.testBean.testMethod13();
-11.4. Advanced Tag Setting
+10.4. Advanced Tag Setting
There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation.
The precedence is as follows:
@@ -2461,7 +1797,7 @@ The default implementation uses SPEL expression resolution.
-11.4.1. Custom extractor
+10.4.1. Custom extractor
The value of the tag for the following method is computed by an implementation of TagValueResolver interface.
Its class name has to be passed as the value of the resolver attribute.
@@ -2493,7 +1829,7 @@ public TagValueResolver tagValueResolver() {
-11.4.2. Resolving Expressions for a Value
+10.4.2. Resolving Expressions for a Value
Consider the following annotated method:
@@ -2511,7 +1847,7 @@ If you want to use some other expression resolution mechanism, you can create yo
-11.4.3. Using the toString() method
+10.4.3. Using the toString() method
Consider the following annotated method:
@@ -2530,34 +1866,43 @@ public void getAnnotationForArgumentToString(@SpanTag("test") Long param) {
-12. Customizations
+11. Customizations
-
-12.1. Customizers
-With Brave 5.7 you have various options of providing customizers for your project. Brave ships with
+The Tracer object is fully managed by sleuth, so you rarely need to affect it. That said,
+Sleuth supports a number of Customizer types, that allow you to configure
+anything not already done by Sleuth with auto-configuration or properties.
+
+
+If you define one of the following as a Bean, Sleuth will invoke it to
+customize behaviour:
-
-
TracingCustomizer - allows configuration plugins to collaborate on building an instance of Tracing.
+RpcTracingCustomizer - for RPC tagging and sampling policy
-
-
CurrentTraceContextCustomizer - allows configuration plugins to collaborate on building an instance of CurrentTraceContext.
+HttpTracingCustomizer - for HTTP tagging and sampling policy
-
-
ExtraFieldCustomizer - allows configuration plugins to collaborate on building an instance of ExtraFieldPropagation.Factory.
+MessagingTracingCustomizer - for messaging tagging and sampling policy
+
+-
+
CurrentTraceContextCustomizer - to integrate decorators such as correlation.
+
+-
+
BaggagePropagationCustomizer - for propagating baggage fields in process and over headers
+
+-
+
CorrelationScopeDecoratorCustomizer - for scope decorations such as MDC (logging) field correlation
-
-Sleuth will search for beans of those types and automatically apply customizations.
-
-
-12.2. HTTP
+11.1. HTTP
-12.2.1. Data Policy
+11.1.1. Data Policy
The default span data policy for HTTP requests is described in Brave:
github.com/openzipkin/brave/tree/master/instrumentation/http#span-data-policy
@@ -2599,7 +1944,7 @@ class Config {
-12.2.2. Sampling
+11.1.2. Sampling
If client /server sampling is required, just register a bean of type
brave.sampler.SamplerFunction<HttpRequest> and name the bean
@@ -2642,7 +1987,7 @@ class Config {
-12.3. TracingFilter
+11.2. TracingFilter
You can also modify the behavior of the TracingFilter, which is the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response.
You can customize the tags or modify the response headers by registering your own instance of the TracingFilter bean.
@@ -2683,7 +2028,7 @@ class MyFilter extends GenericFilterBean {
-12.4. Messaging
+11.3. Messaging
Sleuth automatically configures the MessagingTracing bean which serves as a
foundation for Messaging instrumentation such as Kafka or JMS.
@@ -2716,7 +2061,7 @@ class Config {
-12.5. RPC
+11.4. RPC
Sleuth automatically configures the RpcTracing bean which serves as a
foundation for RPC instrumentation such as gRPC or Dubbo.
@@ -2757,7 +2102,7 @@ class Config {
-12.6. Custom service name
+11.5. Custom service name
By default, Sleuth assumes that, when you send a span to Zipkin, you want the span’s service name to be equal to the value of the spring.application.name property.
That is not always the case, though.
@@ -2771,7 +2116,7 @@ To achieve that, you can pass the following property to your application to over
-12.7. Customization of Reported Spans
+11.6. Customization of Reported Spans
Before reporting spans (for example, to Zipkin) you may want to modify that span in some way.
You can do so by using the FinishedSpanHandler interface.
@@ -2814,7 +2159,7 @@ FinishedSpanHandler handlerTwo() {
-12.8. Host Locator
+11.7. Host Locator
@@ -2845,7 +2190,7 @@ If those are not set, we try to retrieve the host name from the network interfac
-13. Sending Spans to Zipkin
+12. Sending Spans to Zipkin
By default, if you add spring-cloud-starter-zipkin as a dependency to your project, when the span is closed, it is sent to Zipkin over HTTP.
@@ -2926,7 +2271,7 @@ object, you will have to create a bean of zipkin2.reporter.Sender t
-14. Zipkin Stream Span Consumer
+13. Zipkin Stream Span Consumer
@@ -2948,10 +2293,10 @@ In the Finchley release, it got removed.
-15. Integrations
+14. Integrations
-15.1. OpenTracing
+14.1. OpenTracing
Spring Cloud Sleuth is compatible with OpenTracing.
If you have OpenTracing on the classpath, we automatically register the OpenTracing Tracer bean.
@@ -2959,7 +2304,7 @@ If you wish to disable this, set spring.sleuth.opentracing.enabled
-15.2. Runnable and Callable
+14.2. Runnable and Callable
If you wrap your logic in Runnable or Callable, you can wrap those classes in their Sleuth representative, as shown in the following example for Runnable:
@@ -3015,13 +2360,13 @@ Callable<String> traceCallableFromTracer = this.tracing.currentTraceContex
-15.3. Spring Cloud CircuitBreaker
+14.3. Spring Cloud CircuitBreaker
If you have Spring Cloud CircuitBreaker on the classpath, we will wrap the passed command Supplier and the fallback Function in its trace representations. In order to disable this instrumentation set spring.sleuth.circuitbreaker.enabled to false.
-15.4. RxJava
+14.4. RxJava
We registering a custom RxJavaSchedulersHook that wraps all Action0 instances in their Sleuth representative, which is called TraceAction.
The hook either starts or continues a span, depending on whether tracing was already going on before the Action was scheduled.
@@ -3046,12 +2391,12 @@ the Reactor support.
-15.5. HTTP integration
+14.5. HTTP integration
Features from this section can be disabled by setting the spring.sleuth.web.enabled property with value equal to false.
-15.5.1. HTTP Filter
+14.5.1. HTTP Filter
Through the TracingFilter, all sampled incoming requests result in creation of a Span.
That Span’s name is http: + the path to which the request was sent.
@@ -3075,7 +2420,7 @@ to true.
-15.5.2. HandlerInterceptor
+14.5.2. HandlerInterceptor
Since we want the span names to be precise, we use a TraceHandlerInterceptor that either wraps an existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors.
The TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest.
@@ -3085,13 +2430,13 @@ In that case, please file an issue in Spring Cloud Sleuth.
-15.5.3. Async Servlet support
+14.5.3. Async Servlet support
If your controller returns a Callable or a WebAsyncTask, Spring Cloud Sleuth continues the existing span instead of creating a new one.
-15.5.4. WebFlux support
+14.5.4. WebFlux support
Through TraceWebFilter, all sampled incoming requests result in creation of a Span.
That Span’s name is http: + the path to which the request was sent.
@@ -3106,7 +2451,7 @@ If you want to reuse Sleuth’s default skip patterns and append your own, p
-15.5.5. Dubbo RPC support
+14.5.5. Dubbo RPC support
Via the integration with Brave, Spring Cloud Sleuth supports Dubbo.
It’s enough to add the brave-instrumentation-dubbo dependency:
@@ -3135,9 +2480,9 @@ An example of Spring Cloud Sleuth and Dubbo can be found
-15.6. HTTP Client Integration
+14.6. HTTP Client Integration
-15.6.1. Synchronous Rest Template
+14.6.1. Synchronous Rest Template
We inject a RestTemplate interceptor to ensure that all the tracing information is passed to the requests.
Each time a call is made, a new Span is created.
@@ -3159,7 +2504,7 @@ If you create a RestTemplate instance with a new keywo
-15.6.2. Asynchronous Rest Template
+14.6.2. Asynchronous Rest Template
@@ -3216,7 +2561,7 @@ static class Config {
-15.6.3. WebClient
+14.6.3. WebClient
We inject a ExchangeFilterFunction implementation that creates a span and, through on-success and on-error callbacks, takes care of closing client-side spans.
@@ -3238,7 +2583,7 @@ If you create a WebClient instance with a new keyword,
-15.6.4. Traverson
+14.6.4. Traverson
If you use the Traverson library, you can inject a RestTemplate as a bean into your Traverson object.
Since RestTemplate is already intercepted, you get full support for tracing in your client. The following pseudo code
@@ -3255,7 +2600,7 @@ Traverson traverson = new Traverson(URI.create("https://some/address"),
-15.6.5. Apache HttpClientBuilder and HttpAsyncClientBuilder
+14.6.5. Apache HttpClientBuilder and HttpAsyncClientBuilder
We instrument the HttpClientBuilder and HttpAsyncClientBuilder so that
tracing context gets injected to the sent requests.
@@ -3265,7 +2610,7 @@ tracing context gets injected to the sent requests.
-15.6.6. Netty HttpClient
+14.6.6. Netty HttpClient
We instrument the Netty’s HttpClient.
@@ -3287,7 +2632,7 @@ If you create a HttpClient instance with a new keyword
-15.6.7. UserInfoRestTemplateCustomizer
+14.6.7. UserInfoRestTemplateCustomizer
We instrument the Spring Security’s UserInfoRestTemplateCustomizer.
@@ -3297,7 +2642,7 @@ If you create a HttpClient instance with a new keyword
-15.7. Feign
+14.7. Feign
By default, Spring Cloud Sleuth provides integration with Feign through TraceFeignClientAutoConfiguration.
You can disable it entirely by setting spring.sleuth.feign.enabled to false.
@@ -3311,12 +2656,12 @@ However, all the default instrumentation is still there.
-15.8. gRPC
+14.8. gRPC
Spring Cloud Sleuth provides instrumentation for gRPC through TraceGrpcAutoConfiguration. You can disable it entirely by setting spring.sleuth.grpc.enabled to false.
-15.8.1. Variant 1
+14.8.1. Variant 1
Dependencies
@@ -3385,16 +2730,16 @@ Spring Cloud Sleuth provides a SpringAwareManagedChannelBuilder tha
-15.8.2. Variant 2
+14.8.2. Variant 2
Grpc Spring Boot Starter automatically detects the presence of Spring Cloud Sleuth and brave’s instrumentation for gRPC and registers the necessary client and/or server tooling.
-15.9. Asynchronous Communication
+14.9. Asynchronous Communication
-15.9.1. @Async Annotated methods
+14.9.1. @Async Annotated methods
In Spring Cloud Sleuth, we instrument async-related components so that the tracing information is passed between threads.
You can disable this behavior by setting the value of spring.sleuth.async.enabled to false.
@@ -3417,7 +2762,7 @@ You can disable this behavior by setting the value of spring.sleuth.async.
-15.9.2. @Scheduled Annotated Methods
+14.9.2. @Scheduled Annotated Methods
In Spring Cloud Sleuth, we instrument scheduled method execution so that the tracing information is passed between threads.
You can disable this behavior by setting the value of spring.sleuth.scheduled.enabled to false.
@@ -3440,7 +2785,7 @@ You can disable this behavior by setting the value of spring.sleuth.schedu
-15.9.3. Executor, ExecutorService, and ScheduledExecutorService
+14.9.3. Executor, ExecutorService, and ScheduledExecutorService
We provide LazyTraceExecutor, TraceableExecutorService, and TraceableScheduledExecutorService. Those implementations create spans each time a new task is submitted, invoked, or scheduled.
@@ -3527,12 +2872,12 @@ to add the @Role(BeanDefinition.ROLE_INFRASTRUCTURE) on your
-15.10. Messaging
+14.10. Messaging
Features from this section can be disabled by setting the spring.sleuth.messaging.enabled property with value equal to false.
-15.10.1. Spring Integration and Spring Cloud Stream
+14.10.1. Spring Integration and Spring Cloud Stream
Spring Cloud Sleuth integrates with Spring Integration.
It creates spans for publish and subscribe events.
@@ -3571,7 +2916,7 @@ it’s enough for you to register beans of types:
-15.10.2. Spring RabbitMq
+14.10.2. Spring RabbitMq
We instrument the RabbitTemplate so that tracing headers get injected
into the message.
@@ -3581,7 +2926,7 @@ into the message.
-15.10.3. Spring Kafka
+14.10.3. Spring Kafka
We instrument the Spring Kafka’s ProducerFactory and ConsumerFactory
so that tracing headers get injected into the created Spring Kafka’s
@@ -3592,7 +2937,7 @@ so that tracing headers get injected into the created Spring Kafka’s
-15.10.4. Spring Kafka Streams
+14.10.4. Spring Kafka Streams
We instrument the KafkaStreams KafkaClientSupplier so that tracing headers
get injected into the Producer and Consumer`s. A `KafkaStreamsTracing bean
@@ -3604,7 +2949,7 @@ allows for further instrumentation through additional TransformerSupplier<
-15.10.5. Spring JMS
+14.10.5. Spring JMS
We instrument the JmsTemplate so that tracing headers get injected
into the message. We also support @JmsListener annotated methods on the consumer side.
@@ -3626,7 +2971,7 @@ We don’t support baggage propagation for JMS
-15.10.6. Spring Cloud AWS Messaging SQS
+14.10.6. Spring Cloud AWS Messaging SQS
We instrument @SqsListener which is provided by org.springframework.cloud:spring-cloud-aws-messaging
so that tracing headers get extracted from the message and a trace gets put into the context.
@@ -3637,14 +2982,14 @@ so that tracing headers get extracted from the message and a trace gets put into
-15.11. Redis
+14.11. Redis
We set tracing property to Lettcue ClientResources instance to enable Brave tracing built in Lettuce .
To disable Redis support, set the spring.sleuth.redis.enabled property to false.
-15.12. Quartz
+14.12. Quartz
We instrument quartz jobs by adding Job/Trigger listeners to the Quartz Scheduler.
@@ -3653,7 +2998,7 @@ To disable Redis support, set the spring.sleuth.redis.enabled prope
-15.13. Project Reactor
+14.13. Project Reactor
For projects depending on Project Reactor such as Spring Cloud Gateway, we suggest turning the spring.sleuth.reactor.decorate-on-each option to false. That way an increased performance gain should be observed in comparison to the standard instrumentation mechanism. What this option does is it will wrap decorate onLast operator instead of onEach which will result in creation of far fewer objects. The downside of this is that when Project Reactor will change threads, the trace propagation will continue without issues, however anything relying on the ThreadLocal such as e.g. MDC entries can be buggy.
@@ -3661,7 +3006,7 @@ To disable Redis support, set the spring.sleuth.redis.enabled prope
-16. Configuration properties
+15. Configuration properties
To see the list of all Sleuth related configuration properties please check the Appendix page.
@@ -3669,7 +3014,7 @@ To disable Redis support, set the spring.sleuth.redis.enabled prope
-17. Running examples
+16. Running examples
You can see the running examples deployed in the Pivotal Web Services.
diff --git a/reference/html/spring-cloud-sleuth.html b/reference/html/spring-cloud-sleuth.html
index f87d57094..570339542 100644
--- a/reference/html/spring-cloud-sleuth.html
+++ b/reference/html/spring-cloud-sleuth.html
@@ -128,84 +128,68 @@ $(globalSwitch);
2. Additional Resources
-3. Features
+ 3. Features
+4. Introduction to Brave
-4. Sampling
+ 5. Sampling
+6. Baggage
+7. Instrumentation
+8. Span lifecycle
-5. Propagation
+ 9. Naming spans
-6. Current Tracing Component
-7. Current Span
+ 10. Managing Spans with Annotations
-8. Instrumentation
-9. Span lifecycle
+ 11. Customizations
-10. Naming spans
+ 12. Sending Spans to Zipkin
+13. Zipkin Stream Span Consumer
+14. Integrations
-- 10.1.
@SpanName Annotation
-- 10.2.
toString() method
+- 14.1. OpenTracing
+- 14.2. Runnable and Callable
+- 14.3. Spring Cloud CircuitBreaker
+- 14.4. RxJava
+- 14.5. HTTP integration
+- 14.6. HTTP Client Integration
+- 14.7. Feign
+- 14.8. gRPC
+- 14.9. Asynchronous Communication
+- 14.10. Messaging
+- 14.11. Redis
+- 14.12. Quartz
+- 14.13. Project Reactor
-11. Managing Spans with Annotations
-
-
-12. Customizations
-
-
-13. Sending Spans to Zipkin
-14. Zipkin Stream Span Consumer
-15. Integrations
-
-
-16. Configuration properties
-17. Running examples
+15. Configuration properties
+16. Running examples
@@ -1247,392 +1231,55 @@ We’ve converted the MDC entries from B3 to non B3 keys (e.g. X-B3-Tr
+
+
+
+4. Introduction to Brave
+
+
+Brave is a distributed tracing instrumentation library. Brave typically
+intercepts production requests to gather timing data, correlate and propagate
+trace contexts. While typically trace data is sent to Zipkin server,
+third-party plugins are available to send to alternate services such as Amazon
+X-Ray.
+
+
+Spring Cloud Sleuth is a layer over Brave.
+It configures everything you need to get started with tracing. Sleuth
+configures where trace data (spans) are reported to, how many traces to keep
+(sampling), if remote fields (baggage) and which libraries are traced.
+Sleuth also adds annotation based tracing features and some instrumentation not
+available otherwise, such as Reactor.
+
-3.1. Introduction to Brave
-
-
-
-
-
-
-
-Starting with version 2.0.0, Spring Cloud Sleuth uses
-Brave as the tracing library.
-For your convenience, we embed part of the Brave’s docs here.
-
-
-
-
-
-
-
-
-
-
-
-In the vast majority of cases you need to just use the Tracer
-or SpanCustomizer beans from Brave that Sleuth provides. The documentation below contains
-a high overview of what Brave is and how it works.
-
-
-
+4.1. Brave Basics
+
+Most instrumentation work is done for you by default. Sleuth provides beans to
+allow you to change what’s traced, and it even provides annotations to avoid
+using tracing libraries! All of this is explained later in this document.
-Brave is a library used to capture and report latency information about distributed operations to Zipkin.
-Most users do not use Brave directly. They use libraries or frameworks rather than employ Brave on their behalf.
+That said, you might want to know more about how things work underneath. Here
+are some pointers.
-This module includes a tracer that creates and joins spans that model the latency of potentially distributed work.
-It also includes libraries to propagate the trace context over network boundaries (for example, with HTTP headers).
-
-
-3.1.1. Tracing
-
-Most importantly, you need a brave.Tracer, configured to report to Zipkin.
+Here are the most core types you might use:
+* SpanCustomizer - to change the span currently in progress
+* Tracer - to get a start new spans ad-hoc
-The following example setup sends trace data (spans) to Zipkin over HTTP (as opposed to Kafka):
-
-
-
-class MyClass {
-
- private final Tracer tracer;
-
- // Tracer will be autowired
- MyClass(Tracer tracer) {
- this.tracer = tracer;
- }
-
- void doSth() {
- Span span = tracer.newTrace().name("encode").start();
- // ...
- }
-}
-
-
-
-
-
-
-
-
-
-If your span contains a name longer than 50 chars, then that name is truncated to 50 chars.
-Your names have to be explicit and concrete.
-Big names lead to latency issues and sometimes even thrown exceptions.
-
-
-
-
-
-The tracer creates and joins spans that model the latency of potentially distributed work.
-It can employ sampling to reduce overhead during the process, to reduce the amount of data sent to Zipkin, or both.
-
-
-Spans returned by a tracer report data to Zipkin when finished or do nothing if unsampled.
-After starting a span, you can annotate events of interest or add tags containing details or lookup keys.
-
-
-Spans have a context that includes trace identifiers that place the span at the correct spot in the tree representing the distributed operation.
-
-
-
-3.1.2. Local Tracing
-
-When tracing code that never leaves your process, run it inside a scoped span.
-
-
-
-@Autowired Tracer tracer;
-
-// Start a new trace or a span within an existing trace representing an operation
-ScopedSpan span = tracer.startScopedSpan("encode");
-try {
- // The span is in "scope" meaning downstream code such as loggers can see trace IDs
- return encoder.encode();
-} catch (RuntimeException | Error e) {
- span.error(e); // Unless you handle exceptions, you might not know the operation failed!
- throw e;
-} finally {
- span.finish(); // always finish the span
-}
-
-
-
-When you need more features, or finer control, use the Span type:
-
-
-
-@Autowired Tracer tracer;
-
-// Start a new trace or a span within an existing trace representing an operation
-Span span = tracer.nextSpan().name("encode").start();
-// Put the span in "scope" so that downstream code such as loggers can see trace IDs
-try (SpanInScope ws = tracer.withSpanInScope(span)) {
- return encoder.encode();
-} catch (RuntimeException | Error e) {
- span.error(e); // Unless you handle exceptions, you might not know the operation failed!
- throw e;
-} finally {
- span.finish(); // note the scope is independent of the span. Always finish a span.
-}
-
-
-
-Both of the above examples report the exact same span on finish!
-
-
-In the above example, the span will be either a new root span or the
-next child in an existing trace.
-
-
-
-3.1.3. Customizing Spans
-
-Once you have a span, you can add tags to it.
-The tags can be used as lookup keys or details.
-For example, you might add a tag with your runtime version, as shown in the following example:
-
-
-
-span.tag("clnt/finagle.version", "6.36.0");
-
-
-
-When exposing the ability to customize spans to third parties, prefer brave.SpanCustomizer as opposed to brave.Span.
-The former is simpler to understand and test and does not tempt users with span lifecycle hooks.
-
-
-
-interface MyTraceCallback {
- void request(Request request, SpanCustomizer customizer);
-}
-
-
-
-Since brave.Span implements brave.SpanCustomizer, you can pass it to users, as shown in the following example:
-
-
-
-for (MyTraceCallback callback : userCallbacks) {
- callback.request(request, span);
-}
-
-
-
-
-3.1.4. Implicitly Looking up the Current Span
-
-Sometimes, you do not know if a trace is in progress or not, and you do not want users to do null checks.
-brave.CurrentSpanCustomizer handles this problem by adding data to any span that’s in progress or drops, as shown in the following example:
-
-
-Ex.
-
-
-
-// The user code can then inject this without a chance of it being null.
-@Autowired SpanCustomizer span;
-
-void userCode() {
- span.annotate("tx.started");
- ...
-}
-
-
-
-
-3.1.5. RPC tracing
-
-
-
-
-
-
-
-Check for instrumentation written here and Zipkin’s list before rolling your own RPC instrumentation.
-
-
-
-
-
-RPC tracing is often done automatically by interceptors. Behind the scenes, they add tags and events that relate to their role in an RPC operation.
-
-
-The following example shows how to add a client span:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// before you send a request, add metadata that describes the operation
-span = tracer.nextSpan().name(service + "/" + method).kind(CLIENT);
-span.tag("myrpc.version", "1.0.0");
-span.remoteServiceName("backend");
-span.remoteIpAndPort("172.3.4.1", 8108);
-
-// Add the trace context to the request, so it can be propagated in-band
-tracing.propagation().injector(Request::addHeader)
- .inject(span.context(), request);
-
-// when the request is scheduled, start the span
-span.start();
-
-// if there is an error, tag the span
-span.tag("error", error.getCode());
-// or if there is an exception
-span.error(exception);
-
-// when the response is complete, finish the span
-span.finish();
-
-
-
-One-Way tracing
-
-Sometimes, you need to model an asynchronous operation where there is a
-request but no response. In normal RPC tracing, you use span.finish()
-to indicate that the response was received. In one-way tracing, you use
-span.flush() instead, as you do not expect a response.
-
-
-The following example shows how a client might model a one-way operation:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// start a new span representing a client request
-oneWaySend = tracer.nextSpan().name(service + "/" + method).kind(CLIENT);
-
-// Add the trace context to the request, so it can be propagated in-band
-tracing.propagation().injector(Request::addHeader)
- .inject(oneWaySend.context(), request);
-
-// fire off the request asynchronously, totally dropping any response
-request.execute();
-
-// start the client side and flush instead of finish
-oneWaySend.start().flush();
-
-
-
-The following example shows how a server might handle a one-way operation:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// pull the context out of the incoming request
-extractor = tracing.propagation().extractor(Request::getHeader);
-
-// convert that context to a span which you can name and add tags to
-oneWayReceive = nextSpan(tracer, extractor.extract(request))
- .name("process-request")
- .kind(SERVER)
- ... add tags etc.
-
-// start the server side and flush instead of finish
-oneWayReceive.start().flush();
-
-// you should not modify this span anymore as it is complete. However,
-// you can create children to represent follow-up work.
-next = tracer.newSpan(oneWayReceive.context()).name("step2").start();
-
-
-
+Here are the most relevant links from the OpenZipkin Brave project:
+* [Brave’s core library](github.com/openzipkin/brave/tree/master/brave)
+* [Baggage (propagated fields)](github.com/openzipkin/brave/tree/master/brave#baggage)
+* [HTTP tracing](github.com/openzipkin/brave/tree/master/instrumentation/http)
-4. Sampling
+5. Sampling
-Sampling may be employed to reduce the data collected and reported out of process.
-When a span is not sampled, it adds no overhead (a noop).
-
-
-Sampling is an up-front decision, meaning that the decision to report data is made at the first operation in a trace and that decision is propagated downstream.
-
-
-By default, a global sampler applies a single rate to all traced operations.
-Tracer.Builder.sampler controls this setting, and it defaults to tracing every request.
-
-
-4.1. Declarative sampling
-
-Some applications need to sample based on the type or annotations of a java method.
-
-
-Most users use a framework interceptor to automate this sort of policy.
-The following example shows how that might work internally:
-
-
-
-@Autowired Tracer tracer;
-
-// derives a sample rate from an annotation on a java method
-DeclarativeSampler<Traced> sampler = DeclarativeSampler.create(Traced::sampleRate);
-
-@Around("@annotation(traced)")
-public Object traceThing(ProceedingJoinPoint pjp, Traced traced) throws Throwable {
- // When there is no trace in progress, this decides using an annotation
- Sampler decideUsingAnnotation = declarativeSampler.toSampler(traced);
- Tracer tracer = tracer.withSampler(decideUsingAnnotation);
-
- // This code looks the same as if there was no declarative override
- ScopedSpan span = tracer.startScopedSpan(spanName(pjp));
- try {
- return pjp.proceed();
- } catch (RuntimeException | Error e) {
- span.error(e);
- throw e;
- } finally {
- span.finish();
- }
-}
-
-
-
-
-4.2. Custom sampling
-
-Depending on what the operation is, you may want to apply different policies.
-For example, you might not want to trace requests to static resources such as images, or you might want to trace all requests to a new api.
-
-
-Most users use a framework interceptor to automate this sort of policy.
-The following example shows how that might work internally:
-
-
-
-@Autowired Tracer tracer;
-@Autowired Sampler fallback;
-
-Span nextSpan(final Request input) {
- Sampler requestBased = Sampler() {
- @Override public boolean isSampled(long traceId) {
- if (input.url().startsWith("/experimental")) {
- return true;
- } else if (input.url().startsWith("/static")) {
- return false;
- }
- return fallback.isSampled(traceId);
- }
- };
- return tracer.withSampler(requestBased).nextSpan();
-}
-
-
-
-
-4.3. Sampling in Spring Cloud Sleuth
-
By default Spring Cloud Sleuth sets all spans to non-exportable.
That means that traces appear in logs but not in any remote store.
For testing the default is often enough, and it probably is all you need if you use only the logs (for example, with an ELK aggregator).
@@ -1669,7 +1316,7 @@ public Sampler defaultSampler() {
-You can set the HTTP header X-B3-Flags to 1, or, when doing messaging, you can set the spanFlags header to 1.
+You can set the HTTP header b3 to 1, or, when doing messaging, you can set the spanFlags header to 1.
Doing so forces the current span to be exportable regardless of the sampling decision.
@@ -1680,165 +1327,14 @@ Doing so forces the current span to be exportable regardless of the sampling dec
-
-5. Propagation
+6. Baggage
-Propagation is needed to ensure activities originating from the same root are collected together in the same trace.
-The most common propagation approach is to copy a trace context from a client by sending an RPC request to a server receiving it.
-
-
-For example, when a downstream HTTP call is made, its trace context is encoded as request headers and sent along with it, as shown in the following image:
-
-
-
- Client Span Server Span
-┌──────────────────┐ ┌──────────────────┐
-│ │ │ │
-│ TraceContext │ Http Request Headers │ TraceContext │
-│ ┌──────────────┐ │ ┌───────────────────┐ │ ┌──────────────┐ │
-│ │ TraceId │ │ │ X─B3─TraceId │ │ │ TraceId │ │
-│ │ │ │ │ │ │ │ │ │
-│ │ ParentSpanId │ │ Extract │ X─B3─ParentSpanId │ Inject │ │ ParentSpanId │ │
-│ │ ├─┼─────────>│ ├────────┼>│ │ │
-│ │ SpanId │ │ │ X─B3─SpanId │ │ │ SpanId │ │
-│ │ │ │ │ │ │ │ │ │
-│ │ Sampled │ │ │ X─B3─Sampled │ │ │ Sampled │ │
-│ └──────────────┘ │ └───────────────────┘ │ └──────────────┘ │
-│ │ │ │
-└──────────────────┘ └──────────────────┘
-
-
-
-The names above are from B3 Propagation, which is built-in to Brave and has implementations in many languages and frameworks.
-
-
-Most users use a framework interceptor to automate propagation.
-The next two examples show how that might work for a client and a server.
-
-
-The following example shows how client-side propagation might work:
-
-
-
-@Autowired Tracing tracing;
-
-// configure a function that injects a trace context into a request
-injector = tracing.propagation().injector(Request.Builder::addHeader);
-
-// before a request is sent, add the current span's context to it
-injector.inject(span.context(), request);
-
-
-
-The following example shows how server-side propagation might work:
-
-
-
-@Autowired Tracing tracing;
-@Autowired Tracer tracer;
-
-// configure a function that extracts the trace context from a request
-extractor = tracing.propagation().extractor(Request::getHeader);
-
-// when a server receives a request, it joins or starts a new trace
-span = tracer.nextSpan(extractor.extract(request));
-
-
-
-5.1. Propagating extra fields
-
-Sometimes you need to propagate extra fields, such as a request ID or an alternate trace context.
-For example, if you are in a Cloud Foundry environment, you might want to pass the request ID, as shown in the following example:
-
-
-
-// when you initialize the builder, define the extra field you want to propagate
-Tracing.newBuilder().propagationFactory(
- ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-vcap-request-id")
-);
-
-// later, you can tag that request ID or use it in log correlation
-requestId = ExtraFieldPropagation.get("x-vcap-request-id");
-
-
-
-You may also need to propagate a trace context that you are not using.
-For example, you may be in an Amazon Web Services environment but not be reporting data to X-Ray.
-To ensure X-Ray can co-exist correctly, pass-through its tracing header, as shown in the following example:
-
-
-
-tracingBuilder.propagationFactory(
- ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-amzn-trace-id")
-);
-
-
-
-
-
-
-
-
-
-In Spring Cloud Sleuth all elements of the tracing builder Tracing.newBuilder()
-are defined as beans. So if you want to pass a custom PropagationFactory, it’s enough
-for you to create a bean of that type and we will set it in the Tracing bean.
-
-
-
-
-
-5.1.1. Prefixed fields
-
-If they follow a common pattern, you can also prefix fields.
-The following example shows how to propagate x-vcap-request-id the field as-is but send the country-code and user-id fields on the wire as x-baggage-country-code and x-baggage-user-id, respectively:
-
-
-
-Tracing.newBuilder().propagationFactory(
- ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY)
- .addField("x-vcap-request-id")
- .addPrefixedFields("x-baggage-", Arrays.asList("country-code", "user-id"))
- .build()
-);
-
-
-
-Later, you can call the following code to affect the country code of the current trace context:
-
-
-
-ExtraFieldPropagation.set("x-country-code", "FO");
-String countryCode = ExtraFieldPropagation.get("x-country-code");
-
-
-
-Alternatively, if you have a reference to a trace context, you can use it explicitly, as shown in the following example:
-
-
-
-ExtraFieldPropagation.set(span.context(), "x-country-code", "FO");
-String countryCode = ExtraFieldPropagation.get(span.context(), "x-country-code");
-
-
-
-
-
-
-
-
-
-A difference from previous versions of Sleuth is that, with Brave, you must pass the list of baggage keys.
-There are the following properties to achieve this.
-With the spring.sleuth.baggage-keys, you set keys that get prefixed with baggage- for HTTP calls and baggage_ for messaging.
+With the spring.sleuth.baggage-keys, you set keys that get prefixed with baggage- for HTTP calls and baggage_ for messaging.
You can also use the spring.sleuth.propagation-keys property to pass a list of prefixed keys that are propagated to remote services without any prefix.
You can also use the spring.sleuth.local-keys property to pass a list keys that will be propagated locally but will not be propagated over the wire.
-Notice that there’s no x- in front of the header keys.
-
-
-
+Notice that there’s no x- in front of the header keys.
In order to automatically set the baggage values to Slf4j’s MDC, you have to set
@@ -1862,169 +1358,9 @@ Remember that adding entries to MDC can drastically decrease the performance of
spring.sleuth.propagation.tag.whitelisted-keys with a list of whitelisted baggage keys. To disable the feature you have to pass the spring.sleuth.propagation.tag.enabled=false property.
-
-5.1.2. Extracting a Propagated Context
-
-The TraceContext.Extractor<C> reads trace identifiers and sampling status from an incoming request or message.
-The carrier is usually a request object or headers.
-
-
-This utility is used in standard instrumentation (such as HttpServerHandler) but can also be used for custom RPC or messaging code.
-
-
-TraceContextOrSamplingFlags is usually used only with Tracer.nextSpan(extracted), unless you are
-sharing span IDs between a client and a server.
-
-
-
-5.1.3. Sharing span IDs between Client and Server
-
-A normal instrumentation pattern is to create a span representing the server side of an RPC.
-Extractor.extract might return a complete trace context when applied to an incoming client request.
-Tracer.joinSpan attempts to continue this trace, using the same span ID if supported or creating a child span
-if not. When the span ID is shared, the reported data includes a flag saying so.
-
-
-The following image shows an example of B3 propagation:
-
-
-
- ┌───────────────────┐ ┌───────────────────┐
- Incoming Headers │ TraceContext │ │ TraceContext │
-┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
-│ X─B3-TraceId │─────────┼─┼> TraceId │ │──────┼─┼> TraceId │ │
-│ │ │ │ │ │ │ │ │ │
-│ X─B3-ParentSpanId │─────────┼─┼> ParentSpanId │ │──────┼─┼> ParentSpanId │ │
-│ │ │ │ │ │ │ │ │ │
-│ X─B3-SpanId │─────────┼─┼> SpanId │ │──────┼─┼> SpanId │ │
-└───────────────────┘ │ │ │ │ │ │ │ │
- │ │ │ │ │ │ Shared: true │ │
- │ └───────────────┘ │ │ └───────────────┘ │
- └───────────────────┘ └───────────────────┘
-
-
-
-Some propagation systems forward only the parent span ID, detected when Propagation.Factory.supportsJoin() == false.
-In this case, a new span ID is always provisioned, and the incoming context determines the parent ID.
-
-
-The following image shows an example of AWS propagation:
-
-
-
- ┌───────────────────┐ ┌───────────────────┐
- x-amzn-trace-id │ TraceContext │ │ TraceContext │
-┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
-│ Root │─────────┼─┼> TraceId │ │──────┼─┼> TraceId │ │
-│ │ │ │ │ │ │ │ │ │
-│ Parent │─────────┼─┼> SpanId │ │──────┼─┼> ParentSpanId │ │
-└───────────────────┘ │ └───────────────┘ │ │ │ │ │
- └───────────────────┘ │ │ SpanId: New │ │
- │ └───────────────┘ │
- └───────────────────┘
-
-
-
-Note: Some span reporters do not support sharing span IDs.
-For example, if you set Tracing.Builder.spanReporter(amazonXrayOrGoogleStackdrive), you should disable join by setting Tracing.Builder.supportsJoin(false).
-Doing so forces a new child span on Tracer.joinSpan().
-
-
-
-5.1.4. Implementing Propagation
-
-TraceContext.Extractor<C> is implemented by a Propagation.Factory plugin.
-Internally, this code creates the union type, TraceContextOrSamplingFlags, with one of the following:
-* TraceContext if trace and span IDs were present.
-* TraceIdContext if a trace ID was present but span IDs were not present.
-* SamplingFlags if no identifiers were present.
-
-
-Some Propagation implementations carry extra data from the point of extraction (for example, reading incoming headers) to injection (for example, writing outgoing headers).
-For example, it might carry a request ID.
-When implementations have extra data, they handle it as follows:
-* If a TraceContext were extracted, add the extra data as TraceContext.extra().
-* Otherwise, add it as TraceContextOrSamplingFlags.extra(), which Tracer.nextSpan handles.
-
-
-
-
-6. Current Tracing Component
-
-
-Brave supports a "current tracing component" concept, which should only be used when you have no other way to get a reference.
-This was made for JDBC connections, as they often initialize prior to the tracing component.
-
-
-The most recent tracing component instantiated is available through Tracing.current().
-You can also use Tracing.currentTracer() to get only the tracer.
-If you use either of these methods, do not cache the result.
-Instead, look them up each time you need them.
-
-
-
-
-7. Current Span
-
-
-Brave supports a "current span" concept which represents the in-flight operation.
-You can use Tracer.currentSpan() to add custom tags to a span and Tracer.nextSpan() to create a child of whatever is in-flight.
-
-
-
-
-
-
-
-
-In Sleuth, you can autowire the Tracer bean to retrieve the current span via
-tracer.currentSpan() method. To retrieve the current context just call
-tracer.currentSpan().context(). To get the current trace id as String
-you can use the traceIdString() method like this: tracer.currentSpan().context().traceIdString().
-
-
-
-
-
-7.1. Setting a span in scope manually
-
-When writing new instrumentation, it is important to place a span you created in scope as the current span.
-Not only does doing so let users access it with Tracer.currentSpan(), but it also allows customizations such as SLF4J MDC to see the current trace IDs.
-
-
-Tracer.withSpanInScope(Span) facilitates this and is most conveniently employed by using the try-with-resources idiom.
-Whenever external code might be invoked (such as proceeding an interceptor or otherwise), place the span in scope, as shown in the following example:
-
-
-
-@Autowired Tracer tracer;
-
-try (SpanInScope ws = tracer.withSpanInScope(span)) {
- return inboundRequest.invoke();
-} finally { // note the scope is independent of the span
- span.finish();
-}
-
-
-
-In edge cases, you may need to clear the current span temporarily (for example, launching a task that should not be associated with the current request). To do tso, pass null to withSpanInScope, as shown in the following example:
-
-
-
-@Autowired Tracer tracer;
-
-try (SpanInScope cleared = tracer.withSpanInScope(null)) {
- startBackgroundThread();
-}
-
-
-
-
-
-
-8. Instrumentation
+7. Instrumentation
Spring Cloud Sleuth automatically instruments all your Spring applications, so you should not have to do anything to activate it.
@@ -2050,7 +1386,7 @@ Tags are collected and exported only if there is a Sampler that all
-9. Span lifecycle
+8. Span lifecycle
You can do the following operations on the Span by means of brave.Tracer:
@@ -2089,7 +1425,7 @@ Spring Cloud Sleuth creates an instance of Tracer for you. In order
-9.1. Creating and finishing spans
+8.1. Creating and finishing spans
You can manually create spans by using the Tracer, as shown in the following example:
@@ -2144,7 +1480,7 @@ Your names have to be explicit and concrete. Big names lead to latency issues an
-9.2. Continuing Spans
+8.2. Continuing Spans
Sometimes, you do not want to create a new span but you want to continue one. An example of such a
situation might be as follows:
@@ -2181,7 +1517,7 @@ finally {
-9.3. Creating a Span with an explicit Parent
+8.3. Creating a Span with an explicit Parent
You might want to start a new span and provide an explicit parent of that span.
Assume that the parent of a span is in one thread and you want to start a new span in another thread.
@@ -2229,7 +1565,7 @@ After creating such a span, you must finish it. Otherwise it is not reported (fo
-10. Naming spans
+9. Naming spans
Picking a span name is not a trivial task. A span name should depict an operation name.
@@ -2255,7 +1591,7 @@ The name should be low cardinality, so it should not include identifiers.
Fortunately, for asynchronous processing, you can provide explicit naming.
-10.1. @SpanName Annotation
+9.1. @SpanName Annotation
You can name the span explicitly by using the @SpanName annotation, as shown in the following example:
@@ -2286,7 +1622,7 @@ future.get();
-10.2. toString() method
+9.2. toString() method
It is pretty rare to create separate classes for Runnable or Callable.
Typically, one creates an anonymous instance of those classes.
@@ -2318,13 +1654,13 @@ future.get();
-11. Managing Spans with Annotations
+10. Managing Spans with Annotations
You can manage spans with a variety of annotations.
-11.1. Rationale
+10.1. Rationale
There are a number of good reasons to manage spans with annotations, including:
@@ -2347,7 +1683,7 @@ Now you can provide annotations over interfaces and the arguments of those inter
-11.2. Creating New Spans
+10.2. Creating New Spans
If you do not want to create local spans manually, you can use the @NewSpan annotation.
Also, we provide the @SpanTag annotation to add tags in an automated fashion.
@@ -2403,7 +1739,7 @@ concrete one wins (in this case customNameOnTestMethod3 is set).
-11.3. Continuing Spans
+10.3. Continuing Spans
If you want to add tags and annotations to an existing span, you can use the @ContinueSpan annotation, as shown in the following example:
@@ -2439,7 +1775,7 @@ this.testBean.testMethod13();
-11.4. Advanced Tag Setting
+10.4. Advanced Tag Setting
There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation.
The precedence is as follows:
@@ -2461,7 +1797,7 @@ The default implementation uses SPEL expression resolution.
-11.4.1. Custom extractor
+10.4.1. Custom extractor
The value of the tag for the following method is computed by an implementation of TagValueResolver interface.
Its class name has to be passed as the value of the resolver attribute.
@@ -2493,7 +1829,7 @@ public TagValueResolver tagValueResolver() {
-11.4.2. Resolving Expressions for a Value
+10.4.2. Resolving Expressions for a Value
Consider the following annotated method:
@@ -2511,7 +1847,7 @@ If you want to use some other expression resolution mechanism, you can create yo
-11.4.3. Using the toString() method
+10.4.3. Using the toString() method
Consider the following annotated method:
@@ -2530,34 +1866,43 @@ public void getAnnotationForArgumentToString(@SpanTag("test") Long param) {
-12. Customizations
+11. Customizations
-
-12.1. Customizers
-With Brave 5.7 you have various options of providing customizers for your project. Brave ships with
+The Tracer object is fully managed by sleuth, so you rarely need to affect it. That said,
+Sleuth supports a number of Customizer types, that allow you to configure
+anything not already done by Sleuth with auto-configuration or properties.
+
+
+If you define one of the following as a Bean, Sleuth will invoke it to
+customize behaviour:
-
-
TracingCustomizer - allows configuration plugins to collaborate on building an instance of Tracing.
+RpcTracingCustomizer - for RPC tagging and sampling policy
-
-
CurrentTraceContextCustomizer - allows configuration plugins to collaborate on building an instance of CurrentTraceContext.
+HttpTracingCustomizer - for HTTP tagging and sampling policy
-
-
ExtraFieldCustomizer - allows configuration plugins to collaborate on building an instance of ExtraFieldPropagation.Factory.
+MessagingTracingCustomizer - for messaging tagging and sampling policy
+
+-
+
CurrentTraceContextCustomizer - to integrate decorators such as correlation.
+
+-
+
BaggagePropagationCustomizer - for propagating baggage fields in process and over headers
+
+-
+
CorrelationScopeDecoratorCustomizer - for scope decorations such as MDC (logging) field correlation
-
-Sleuth will search for beans of those types and automatically apply customizations.
-
-
-12.2. HTTP
+11.1. HTTP
-12.2.1. Data Policy
+11.1.1. Data Policy
The default span data policy for HTTP requests is described in Brave:
github.com/openzipkin/brave/tree/master/instrumentation/http#span-data-policy
@@ -2599,7 +1944,7 @@ class Config {
-12.2.2. Sampling
+11.1.2. Sampling
If client /server sampling is required, just register a bean of type
brave.sampler.SamplerFunction<HttpRequest> and name the bean
@@ -2642,7 +1987,7 @@ class Config {
-12.3. TracingFilter
+11.2. TracingFilter
You can also modify the behavior of the TracingFilter, which is the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response.
You can customize the tags or modify the response headers by registering your own instance of the TracingFilter bean.
@@ -2683,7 +2028,7 @@ class MyFilter extends GenericFilterBean {
-12.4. Messaging
+11.3. Messaging
Sleuth automatically configures the MessagingTracing bean which serves as a
foundation for Messaging instrumentation such as Kafka or JMS.
@@ -2716,7 +2061,7 @@ class Config {
-12.5. RPC
+11.4. RPC
Sleuth automatically configures the RpcTracing bean which serves as a
foundation for RPC instrumentation such as gRPC or Dubbo.
@@ -2757,7 +2102,7 @@ class Config {
-12.6. Custom service name
+11.5. Custom service name
By default, Sleuth assumes that, when you send a span to Zipkin, you want the span’s service name to be equal to the value of the spring.application.name property.
That is not always the case, though.
@@ -2771,7 +2116,7 @@ To achieve that, you can pass the following property to your application to over
-12.7. Customization of Reported Spans
+11.6. Customization of Reported Spans
Before reporting spans (for example, to Zipkin) you may want to modify that span in some way.
You can do so by using the FinishedSpanHandler interface.
@@ -2814,7 +2159,7 @@ FinishedSpanHandler handlerTwo() {
-12.8. Host Locator
+11.7. Host Locator