af7593eaf5
we had a couple of questions related to missing attributes in the logback file or the logs as such. It's related to the fact that once you provide a custom logback file you have to place the propery of spring.application.name in the bootstrap file fixes #376
386 lines
18 KiB
Plaintext
386 lines
18 KiB
Plaintext
Spring Cloud Sleuth implements a distributed tracing solution for http://cloud.spring.io[Spring Cloud].
|
|
|
|
=== Terminology
|
|
|
|
Spring Cloud Sleuth borrows http://research.google.com/pubs/pub36356.html[Dapper's] terminology.
|
|
|
|
*Span:* The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an
|
|
RPC. Span's are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span
|
|
is a part of. Spans also have other data, such as descriptions, timestamped events, key-value
|
|
annotations (tags), the ID of the span that caused them, and process ID's (normally IP address).
|
|
|
|
Spans are started and stopped, and they keep track of their timing information. Once you create a
|
|
span, you must stop it at some point in the future.
|
|
|
|
*Trace:* A set of spans forming a tree-like structure. For example, if you are running a distributed
|
|
big-data store, a trace might be formed by a put request.
|
|
|
|
*Annotation:* is used to record existence of an event in time. Some of the core annotations used to define
|
|
the start and stop of a request are:
|
|
|
|
- *cs* - Client Sent - The client has made a request. This annotation depicts the start of the span.
|
|
- *sr* - Server Received - The server side got the request and will start processing it.
|
|
If one subtracts the cs timestamp from this timestamp one will receive the network latency.
|
|
- *ss* - Server Sent - Annotated upon completion of request processing (when the response
|
|
got sent back to the client). If one subtracts the sr timestamp from this timestamp one
|
|
will receive the time needed by the server side to process the request.
|
|
- *cr* - Client Received - Signifies the end of the span. The client has successfully received the
|
|
response from the server side. If one subtracts the cs timestamp from this timestamp one
|
|
will receive the whole time needed by the client to receive the response from the server.
|
|
|
|
Visualization of what *Span* and *Trace* will look in a system together with the Zipkin annotations:
|
|
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/trace-id.png[Trace Info propagation]
|
|
|
|
Each color of a note signifies a span (7 spans - from *A* to *G*). If you have such information in the note:
|
|
|
|
[source]
|
|
Trace Id = X
|
|
Span Id = D
|
|
Client Sent
|
|
|
|
That means that the current span has *Trace-Id* set to *X*, *Span-Id* set to *D*. It also has emitted
|
|
*Client Sent* event.
|
|
|
|
This is how the visualization of the parent / child relationship of spans would look like:
|
|
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/parents.png[Parent child relationship]
|
|
|
|
=== Purpose
|
|
|
|
In the following sections the example from the image above will be taken into consideration.
|
|
|
|
==== Distributed tracing with Zipkin
|
|
|
|
Altogether there are *10 spans* . If you go to traces in Zipkin you will see this number:
|
|
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/zipkin-traces.png[Traces]
|
|
|
|
However if you pick a particular trace then you will see *7 spans*:
|
|
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/zipkin-ui.png[Traces Info propagation]
|
|
|
|
NOTE: When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to
|
|
Zipkin with Server Received and Server Sent / Client Received and Client Sent
|
|
annotations then they will presented as a single span.
|
|
|
|
In the image depicting the visualization of what *Span* and *Trace* is you can see 20
|
|
colorful labels. How does it happen that in Zipkin 10 spans are received?
|
|
|
|
- 2 span *A* labels signify span started and closed. Upon closing a single span is sent to Zipkin.
|
|
- 4 span *B* labels are in fact are single span with 4 annotations. However this span is composed of
|
|
two separate instances. One sent from service 1 and one from service 2. So in fact two span instances will be sent
|
|
to Zipkin and merged there.
|
|
- 2 span *C* labels signify span started and closed. Upon closing a single span is sent to Zipkin.
|
|
- 4 span *B* labels are in fact are single span with 4 annotations. However this span is composed of
|
|
two separate instances. One sent from service 2 and one from service 3. So in fact two span instances will be sent
|
|
to Zipkin and merged there.
|
|
- 2 span *E* labels signify span started and closed. Upon closing a single span is sent to Zipkin.
|
|
- 4 span *B* labels are in fact are single span with 4 annotations. However this span is composed of
|
|
two separate instances. One sent from service 2 and one from service 4. So in fact two span instances will be sent
|
|
to Zipkin and merged there.
|
|
- 2 span *G* labels signify span started and closed. Upon closing a single span is sent to Zipkin.
|
|
|
|
So 1 span from *A*, 2 spans from *B*, 1 span from *C*, 2 spans from *D*, 1 span from *E*, 2 spans from *F* and 1 from *G*.
|
|
Altogether *10* spans.
|
|
|
|
.Click Pivotal Web Services icon to see it live!
|
|
[caption="Click Pivotal Web Services icon to see it live!"]
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/pws.png["Zipkin deployed on Pivotal Web Services", link="http://docssleuth-zipkin-server.cfapps.io/", width=150, height=74]
|
|
|
|
The dependency graph in Zipkin would look like this:
|
|
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/dependencies.png[Dependencies]
|
|
|
|
.Click Pivotal Web Services icon to see it live!
|
|
[caption="Click Pivotal Web Services icon to see it live!"]
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/pws.png["Zipkin deployed on Pivotal Web Services", link="http://docssleuth-zipkin-server.cfapps.io/dependency", width=150, height=74]
|
|
|
|
|
|
==== Log correlation
|
|
|
|
When grepping the logs of those four applications by trace id equal to e.g. `2485ec27856c56f4` one would get the following:
|
|
|
|
[source]
|
|
service1.log:2016-02-26 11:15:47.561 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Hello from service1. Calling service2
|
|
service2.log:2016-02-26 11:15:47.710 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Hello from service2. Calling service3 and then service4
|
|
service3.log:2016-02-26 11:15:47.895 INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application : Hello from service3
|
|
service2.log:2016-02-26 11:15:47.924 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service3 [Hello from service3]
|
|
service4.log:2016-02-26 11:15:48.134 INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application : Hello from service4
|
|
service2.log:2016-02-26 11:15:48.156 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service4 [Hello from service4]
|
|
service1.log:2016-02-26 11:15:48.182 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]
|
|
|
|
If you're using a log aggregating tool like https://www.elastic.co/products/kibana[Kibana],
|
|
http://www.splunk.com/[Splunk] etc. you can order the events that took place. An example of
|
|
Kibana would look like this:
|
|
|
|
image::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/docs/src/main/asciidoc/images/kibana.png[Log correlation with Kibana]
|
|
|
|
If you want to use https://www.elastic.co/guide/en/logstash/current/index.html[Logstash] here is the Grok pattern for Logstash:
|
|
|
|
[source]
|
|
filter {
|
|
# pattern matching logback pattern
|
|
grok {
|
|
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
|
|
}
|
|
}
|
|
|
|
NOTE: If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern:
|
|
[source]
|
|
filter {
|
|
# pattern matching logback pattern
|
|
grok {
|
|
match => { "message" => "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
|
|
}
|
|
}
|
|
|
|
===== JSON Logback with Logstash
|
|
|
|
Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability
|
|
we're passing the dependencies in the `groupId:artifactId:version` notation.
|
|
|
|
*Dependencies setup*
|
|
|
|
- Ensure that Logback is on the classpath (`ch.qos.logback:logback-core`)
|
|
- Add Logstash Logback encode - example for version `4.6` : `net.logstash.logback:logstash-logback-encoder:4.6`
|
|
|
|
*Logback setup*
|
|
|
|
Below you can find an example of a Logback configuration (file named `https://github.com/spring-cloud-samples/sleuth-documentation-apps/blob/master/service1/src/main/resources/logback-spring.xml[logback-spring.xml]`) that:
|
|
|
|
- logs information from the application in a JSON format to a `build/${spring.application.name}.json` file
|
|
- has commented out two additional appenders - console and standard log file
|
|
- has the same logging pattern as the one presented in the previous section
|
|
|
|
[source,xml]
|
|
-----
|
|
include::https://raw.githubusercontent.com/spring-cloud-samples/sleuth-documentation-apps/master/service1/src/main/resources/logback-spring.xml[]
|
|
-----
|
|
|
|
NOTE: If you're using a custom `logback-spring.xml` then you have to pass the `spring.application.name` in
|
|
`bootstrap` instead of `application` property file. Otherwise your custom logback file won't read the property properly.
|
|
|
|
=== Adding to the project
|
|
|
|
==== Only Sleuth (log correlation)
|
|
|
|
If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add
|
|
the `spring-cloud-starter-sleuth` module to your project.
|
|
|
|
[source,xml,indent=0,subs="verbatim,attributes",role="primary"]
|
|
.Maven
|
|
----
|
|
<dependencyManagement> <1>
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-dependencies</artifactId>
|
|
<version>Brixton.RELEASE</version>
|
|
<type>pom</type>
|
|
<scope>import</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
</dependencyManagement>
|
|
|
|
<dependency> <2>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-starter-sleuth</artifactId>
|
|
</dependency>
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-starter-sleuth`
|
|
|
|
[source,groovy,indent=0,subs="verbatim,attributes",role="secondary"]
|
|
.Gradle
|
|
----
|
|
dependencyManagement { <1>
|
|
imports {
|
|
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
|
|
}
|
|
}
|
|
|
|
dependencies { <2>
|
|
compile "org.springframework.cloud:spring-cloud-starter-sleuth"
|
|
}
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-starter-sleuth`
|
|
|
|
==== Sleuth with Zipkin via HTTP
|
|
|
|
If you want both Sleuth and Zipkin just add the `spring-cloud-starter-zipkin` dependency.
|
|
|
|
[source,xml,indent=0,subs="verbatim,attributes",role="primary"]
|
|
.Maven
|
|
----
|
|
<dependencyManagement> <1>
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-dependencies</artifactId>
|
|
<version>Brixton.RELEASE</version>
|
|
<type>pom</type>
|
|
<scope>import</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
</dependencyManagement>
|
|
|
|
<dependency> <2>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-starter-zipkin</artifactId>
|
|
</dependency>
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-starter-zipkin`
|
|
|
|
[source,groovy,indent=0,subs="verbatim,attributes",role="secondary"]
|
|
.Gradle
|
|
----
|
|
dependencyManagement { <1>
|
|
imports {
|
|
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
|
|
}
|
|
}
|
|
|
|
dependencies { <2>
|
|
compile "org.springframework.cloud:spring-cloud-starter-zipkin"
|
|
}
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-starter-zipkin`
|
|
|
|
==== Sleuth with Zipkin via Spring Cloud Stream
|
|
|
|
If you want both Sleuth and Zipkin just add the `spring-cloud-sleuth-stream` dependency.
|
|
|
|
[source,xml,indent=0,subs="verbatim,attributes",role="primary"]
|
|
.Maven
|
|
----
|
|
<dependencyManagement> <1>
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-dependencies</artifactId>
|
|
<version>Brixton.RELEASE</version>
|
|
<type>pom</type>
|
|
<scope>import</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
</dependencyManagement>
|
|
|
|
<dependency> <2>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-sleuth-stream</artifactId>
|
|
</dependency>
|
|
<dependency> <3>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-starter-sleuth</artifactId>
|
|
</dependency>
|
|
<!-- EXAMPLE FOR RABBIT BINDING -->
|
|
<dependency> <4>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
|
|
</dependency>
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-sleuth-stream`
|
|
<3> Add the dependency to `spring-cloud-starter-sleuth` - that way all dependant dependencies will be downloaded
|
|
<4> Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
|
|
|
|
[source,groovy,indent=0,subs="verbatim,attributes",role="secondary"]
|
|
.Gradle
|
|
----
|
|
dependencyManagement { <1>
|
|
imports {
|
|
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
|
|
}
|
|
}
|
|
|
|
dependencies {
|
|
compile "org.springframework.cloud:spring-cloud-sleuth-stream" <2>
|
|
compile "org.springframework.cloud:spring-cloud-starter-sleuth" <3>
|
|
// Example for Rabbit binding
|
|
compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" <4>
|
|
}
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-sleuth-stream`
|
|
<3> Add the dependency to `spring-cloud-starter-sleuth` - that way all dependant dependencies will be downloaded
|
|
<4> Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
|
|
|
|
==== Spring Cloud Sleuth Stream Zipkin Collector
|
|
|
|
If you want to start a Spring Cloud Sleuth Stream Zipkin collector just add the `spring-cloud-sleuth-zipkin-stream`
|
|
dependency
|
|
|
|
[source,xml,indent=0,subs="verbatim,attributes",role="primary"]
|
|
.Maven
|
|
----
|
|
<dependencyManagement> <1>
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-dependencies</artifactId>
|
|
<version>Brixton.RELEASE</version>
|
|
<type>pom</type>
|
|
<scope>import</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
</dependencyManagement>
|
|
|
|
<dependency> <2>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-sleuth-zipkin-stream</artifactId>
|
|
</dependency>
|
|
<dependency> <3>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-starter-sleuth</artifactId>
|
|
</dependency>
|
|
<!-- EXAMPLE FOR RABBIT BINDING -->
|
|
<dependency> <4>
|
|
<groupId>org.springframework.cloud</groupId>
|
|
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
|
|
</dependency>
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-sleuth-zipkin-stream`
|
|
<3> Add the dependency to `spring-cloud-starter-sleuth` - that way all dependant dependencies will be downloaded
|
|
<4> Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
|
|
|
|
[source,groovy,indent=0,subs="verbatim,attributes",role="secondary"]
|
|
.Gradle
|
|
----
|
|
dependencyManagement { <1>
|
|
imports {
|
|
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
|
|
}
|
|
}
|
|
|
|
dependencies {
|
|
compile "org.springframework.cloud:spring-cloud-sleuth-zipkin-stream" <2>
|
|
compile "org.springframework.cloud:spring-cloud-starter-sleuth" <3>
|
|
// Example for Rabbit binding
|
|
compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" <4>
|
|
}
|
|
----
|
|
<1> In order not to pick versions by yourself it's much better if you add the dependency management via
|
|
the Spring BOM
|
|
<2> Add the dependency to `spring-cloud-sleuth-zipkin-stream`
|
|
<3> Add the dependency to `spring-cloud-starter-sleuth` - that way all dependant dependencies will be downloaded
|
|
<4> Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to
|
|
|
|
and then just annotate your main class with `@EnableZipkinStreamServer` annotation:
|
|
|
|
[source,java]
|
|
----
|
|
include::https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/master/spring-cloud-sleuth-samples/spring-cloud-sleuth-sample-zipkin-stream/src/main/java/example/ZipkinStreamServerApplication.java[]
|
|
----
|