SpringCloud/spring-cloud-sleuth
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Files
45a9216c851bb8604bfd2b54dc2ec593eaee402f
spring-cloud-sleuth/1.0.x/index.html
Spring Operator 07a561f92d URL Cleanup 
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener).

# HTTP URLs that Could Not Be Fixed
These URLs were unable to be fixed. Please review them to see if they can be manually resolved.

* [ ] http://dubbo.io/ (200) with 7 occurrences could not be migrated:
   ([https](https://dubbo.io/) result SSLHandshakeException).
* [ ] http://erik.eae.net/archives/2007/07/27/18.54.15/ (200) with 1 occurrences could not be migrated:
   ([https](https://erik.eae.net/archives/2007/07/27/18.54.15/) result SSLHandshakeException).
* [ ] http://javascript.nwbox.com/IEContentLoaded/ (200) with 1 occurrences could not be migrated:
   ([https](https://javascript.nwbox.com/IEContentLoaded/) result SSLHandshakeException).
* [ ] http://blindsignals.com/index.php/2009/07/jquery-delay/ (301) with 1 occurrences could not be migrated:
   ([https](https://blindsignals.com/index.php/2009/07/jquery-delay/) result SSLHandshakeException).

# Fixed URLs

## Fixed But Review Recommended
These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended.

* [ ] http://fontawesome.io (301) with 4 occurrences migrated to:
  https://fontawesome.com?from=io ([https](https://fontawesome.io) result AnnotatedConnectException).
* [ ] http://jsperf.com/getall-vs-sizzle/2 (301) with 1 occurrences migrated to:
  https://jsperf.com/getall-vs-sizzle/2 ([https](https://jsperf.com/getall-vs-sizzle/2) result ReadTimeoutException).
* [ ] http://zenorocha.mit-license.org (301) with 1 occurrences migrated to:
  https://zenorocha.mit-license.org/ ([https](https://zenorocha.mit-license.org) result NullPointerException).
* [ ] http://192.168.99.100:9411/ (ConnectTimeoutException) with 1 occurrences migrated to:
  https://192.168.99.100:9411/ ([https](https://192.168.99.100:9411/) result ConnectTimeoutException).
* [ ] http://wiki.ecmascript.org/doku.php?id=harmony:egal (ConnectTimeoutException) with 1 occurrences migrated to:
  https://wiki.ecmascript.org/doku.php?id=harmony:egal ([https](https://wiki.ecmascript.org/doku.php?id=harmony:egal) result ConnectTimeoutException).
* [ ] http://www.w3.org/TR/html4/loose.dtd (ReadTimeoutException) with 1 occurrences migrated to:
  https://www.w3.org/TR/html4/loose.dtd ([https](https://www.w3.org/TR/html4/loose.dtd) result ReadTimeoutException).
* [ ] http://cloud.spring.io/spring-cloud-static/Dalston.SR4/multi/multi__span_data_as_messages.html (404) with 9 occurrences migrated to:
  https://cloud.spring.io/spring-cloud-static/Dalston.SR4/multi/multi__span_data_as_messages.html ([https](https://cloud.spring.io/spring-cloud-static/Dalston.SR4/multi/multi__span_data_as_messages.html) result 404).
* [ ] http://docsbrewing-zipkin-web.cfapps.io/ (404) with 4 occurrences migrated to:
  https://docsbrewing-zipkin-web.cfapps.io/ ([https](https://docsbrewing-zipkin-web.cfapps.io/) result 404).
* [ ] http://docssleuth-zipkin-server.cfapps.io/dependency (404) with 10 occurrences migrated to:
  https://docssleuth-zipkin-server.cfapps.io/dependency ([https](https://docssleuth-zipkin-server.cfapps.io/dependency) result 404).
* [ ] http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ (301) with 1 occurrences migrated to:
  https://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ ([https](https://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/) result 404).
* [ ] http://github.com/sprinframework (301) with 4 occurrences migrated to:
  https://github.com/sprinframework ([https](https://github.com/sprinframework) result 404).
* [ ] http://github.com/spring_hadoop (301) with 4 occurrences migrated to:
  https://github.com/spring_hadoop ([https](https://github.com/spring_hadoop) result 404).
* [ ] http://json.org/json2.js (404) with 1 occurrences migrated to:
  https://json.org/json2.js ([https](https://json.org/json2.js) result 404).
* [ ] http://pivotal.github.com/jasmine (404) with 1 occurrences migrated to:
  https://pivotal.github.com/jasmine ([https](https://pivotal.github.com/jasmine) result 404).
* [ ] http://zipkin.io/pages/existing_instrumentations.html (301) with 7 occurrences migrated to:
  https://zipkin.io/pages/existing_instrumentations.html ([https](https://zipkin.io/pages/existing_instrumentations.html) result 404).

## Fixed Success
These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended.

* [ ] http://asciidoctor.org with 2 occurrences migrated to:
  https://asciidoctor.org ([https](https://asciidoctor.org) result 200).
* [ ] http://backbonejs.org with 1 occurrences migrated to:
  https://backbonejs.org ([https](https://backbonejs.org) result 200).
* [ ] http://bugs.jquery.com/ticket/12282 with 1 occurrences migrated to:
  https://bugs.jquery.com/ticket/12282 ([https](https://bugs.jquery.com/ticket/12282) result 200).
* [ ] http://bugs.jquery.com/ticket/12359 with 1 occurrences migrated to:
  https://bugs.jquery.com/ticket/12359 ([https](https://bugs.jquery.com/ticket/12359) result 200).
* [ ] http://bugs.jquery.com/ticket/13378 with 1 occurrences migrated to:
  https://bugs.jquery.com/ticket/13378 ([https](https://bugs.jquery.com/ticket/13378) result 200).
* [ ] http://cdnjs.cloudflare.com/ajax/libs/zepto/1.2.0/zepto.min.js with 6 occurrences migrated to:
  https://cdnjs.cloudflare.com/ajax/libs/zepto/1.2.0/zepto.min.js ([https](https://cdnjs.cloudflare.com/ajax/libs/zepto/1.2.0/zepto.min.js) result 200).
* [ ] http://cloud.spring.io with 15 occurrences migrated to:
  https://cloud.spring.io ([https](https://cloud.spring.io) result 200).
* [ ] http://creativecommons.org/licenses/by/3.0/ with 2 occurrences migrated to:
  https://creativecommons.org/licenses/by/3.0/ ([https](https://creativecommons.org/licenses/by/3.0/) result 200).
* [ ] http://diveintohtml5.info/history.html with 1 occurrences migrated to:
  https://diveintohtml5.info/history.html ([https](https://diveintohtml5.info/history.html) result 200).
* [ ] http://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html with 10 occurrences migrated to:
  https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html ([https](https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html) result 200).
* [ ] http://docs.spring.io/spring-data/jpa/docs/1.3.4.RELEASE/api/ with 1 occurrences migrated to:
  https://docs.spring.io/spring-data/jpa/docs/1.3.4.RELEASE/api/ ([https](https://docs.spring.io/spring-data/jpa/docs/1.3.4.RELEASE/api/) result 200).
* [ ] http://docs.spring.io/spring-data/jpa/docs/1.3.4.RELEASE/reference/html/ with 1 occurrences migrated to:
  https://docs.spring.io/spring-data/jpa/docs/1.3.4.RELEASE/reference/html/ ([https](https://docs.spring.io/spring-data/jpa/docs/1.3.4.RELEASE/reference/html/) result 200).
* [ ] http://docs.spring.io/spring-data/jpa/docs/1.4.0.RC1/api/ with 1 occurrences migrated to:
  https://docs.spring.io/spring-data/jpa/docs/1.4.0.RC1/api/ ([https](https://docs.spring.io/spring-data/jpa/docs/1.4.0.RC1/api/) result 200).
* [ ] http://docs.spring.io/spring-data/jpa/docs/1.4.0.RC1/reference/html/ with 1 occurrences migrated to:
  https://docs.spring.io/spring-data/jpa/docs/1.4.0.RC1/reference/html/ ([https](https://docs.spring.io/spring-data/jpa/docs/1.4.0.RC1/reference/html/) result 200).
* [ ] http://docs.spring.io/spring-hateoas/docs/current/reference/html/ with 13 occurrences migrated to:
  https://docs.spring.io/spring-hateoas/docs/current/reference/html/ ([https](https://docs.spring.io/spring-hateoas/docs/current/reference/html/) result 200).
* [ ] http://docssleuth-service1.cfapps.io/start with 7 occurrences migrated to:
  https://docssleuth-service1.cfapps.io/start ([https](https://docssleuth-service1.cfapps.io/start) result 200).
* [ ] http://github.com/spring-cloud/spring-cloud-sleuth with 1 occurrences migrated to:
  https://github.com/spring-cloud/spring-cloud-sleuth ([https](https://github.com/spring-cloud/spring-cloud-sleuth) result 200).
* [ ] http://github.com/spring-projects/gh-pages with 1 occurrences migrated to:
  https://github.com/spring-projects/gh-pages ([https](https://github.com/spring-projects/gh-pages) result 200).
* [ ] http://grigio.org with 1 occurrences migrated to:
  https://grigio.org ([https](https://grigio.org) result 200).
* [ ] http://imgs.mi9.com/uploads/landscape/2101/beautiful-leaf-wallpapers_1280x960_28083.jpg with 1 occurrences migrated to:
  https://imgs.mi9.com/uploads/landscape/2101/beautiful-leaf-wallpapers_1280x960_28083.jpg ([https](https://imgs.mi9.com/uploads/landscape/2101/beautiful-leaf-wallpapers_1280x960_28083.jpg) result 200).
* [ ] http://jekyllrb.com with 1 occurrences migrated to:
  https://jekyllrb.com ([https](https://jekyllrb.com) result 200).
* [ ] http://jquery.com/ with 1 occurrences migrated to:
  https://jquery.com/ ([https](https://jquery.com/) result 200).
* [ ] http://nn.mit-license.org/ with 1 occurrences migrated to:
  https://nn.mit-license.org/ ([https](https://nn.mit-license.org/) result 200).
* [ ] http://opensource.org/licenses/mit-license.html with 2 occurrences migrated to:
  https://opensource.org/licenses/mit-license.html ([https](https://opensource.org/licenses/mit-license.html) result 200).
* [ ] http://opentracing.io/ with 14 occurrences migrated to:
  https://opentracing.io/ ([https](https://opentracing.io/) result 200).
* [ ] http://pages.github.com/ with 1 occurrences migrated to:
  https://pages.github.com/ ([https](https://pages.github.com/) result 200).
* [ ] http://projects.spring.io/spring-integration/ with 15 occurrences migrated to:
  https://projects.spring.io/spring-integration/ ([https](https://projects.spring.io/spring-integration/) result 200).
* [ ] http://rubygems.org with 1 occurrences migrated to:
  https://rubygems.org ([https](https://rubygems.org) result 200).
* [ ] http://rubygems.org/ with 1 occurrences migrated to:
  https://rubygems.org/ ([https](https://rubygems.org/) result 200).
* [ ] http://silviomoreto.github.io/bootstrap-select/ with 2 occurrences migrated to:
  https://silviomoreto.github.io/bootstrap-select/ ([https](https://silviomoreto.github.io/bootstrap-select/) result 200).
* [ ] http://sizzlejs.com/ with 2 occurrences migrated to:
  https://sizzlejs.com/ ([https](https://sizzlejs.com/) result 200).
* [ ] http://spring.io/guides/gs/gradle/ with 1 occurrences migrated to:
  https://spring.io/guides/gs/gradle/ ([https](https://spring.io/guides/gs/gradle/) result 200).
* [ ] http://spring.io/guides/gs/maven/ with 1 occurrences migrated to:
  https://spring.io/guides/gs/maven/ ([https](https://spring.io/guides/gs/maven/) result 200).
* [ ] http://spring.io/projects/spring-cloud-sleuth with 4 occurrences migrated to:
  https://spring.io/projects/spring-cloud-sleuth ([https](https://spring.io/projects/spring-cloud-sleuth) result 200).
* [ ] http://stackoverflow.com/questions/tagged/spring-cloud with 1 occurrences migrated to:
  https://stackoverflow.com/questions/tagged/spring-cloud ([https](https://stackoverflow.com/questions/tagged/spring-cloud) result 200).
* [ ] http://stackoverflow.com/questions/tagged/spring-data-jpa with 1 occurrences migrated to:
  https://stackoverflow.com/questions/tagged/spring-data-jpa ([https](https://stackoverflow.com/questions/tagged/spring-data-jpa) result 200).
* [ ] http://twitter.com/byscuits with 2 occurrences migrated to:
  https://twitter.com/byscuits ([https](https://twitter.com/byscuits) result 200).
* [ ] http://twitter.com/fontawesome with 2 occurrences migrated to:
  https://twitter.com/fontawesome ([https](https://twitter.com/fontawesome) result 200).
* [ ] http://underscorejs.org with 1 occurrences migrated to:
  https://underscorejs.org ([https](https://underscorejs.org) result 200).
* [ ] http://kyruus.com (301) with 2 occurrences migrated to:
  https://www.kyruus.com/ ([https](https://kyruus.com) result 200).
* [ ] http://www.splunk.com/ with 15 occurrences migrated to:
  https://www.splunk.com/ ([https](https://www.splunk.com/) result 200).
* [ ] http://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html with 1 occurrences migrated to:
  https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html ([https](https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html) result 200).
* [ ] http://www.w3.org/TR/2011/REC-css3-selectors-20110929/ with 2 occurrences migrated to:
  https://www.w3.org/TR/2011/REC-css3-selectors-20110929/ ([https](https://www.w3.org/TR/2011/REC-css3-selectors-20110929/) result 200).
* [ ] http://www.w3.org/TR/CSS21/syndata.html with 2 occurrences migrated to:
  https://www.w3.org/TR/CSS21/syndata.html ([https](https://www.w3.org/TR/CSS21/syndata.html) result 200).
* [ ] http://www.w3.org/TR/selectors/ with 4 occurrences migrated to:
  https://www.w3.org/TR/selectors/ ([https](https://www.w3.org/TR/selectors/) result 200).
* [ ] http://cloud.spring.io/spring-cloud-stream with 10 occurrences migrated to:
  https://cloud.spring.io/spring-cloud-stream ([https](https://cloud.spring.io/spring-cloud-stream) result 301).
* [ ] http://dev.w3.org/csswg/cssom/ with 1 occurrences migrated to:
  https://dev.w3.org/csswg/cssom/ ([https](https://dev.w3.org/csswg/cssom/) result 301).
* [ ] http://docs.python.org/library/functions.html with 1 occurrences migrated to:
  https://docs.python.org/library/functions.html ([https](https://docs.python.org/library/functions.html) result 301).
* [ ] http://forum.spring.io/forum/spring-projects/data with 1 occurrences migrated to:
  https://forum.spring.io/forum/spring-projects/data ([https](https://forum.spring.io/forum/spring-projects/data) result 301).
* [ ] http://github.com/SpringSource/spring-data-jpa with 1 occurrences migrated to:
  https://github.com/SpringSource/spring-data-jpa ([https](https://github.com/SpringSource/spring-data-jpa) result 301).
* [ ] http://github.com/pivotal/jasmine-ajax with 1 occurrences migrated to:
  https://github.com/pivotal/jasmine-ajax ([https](https://github.com/pivotal/jasmine-ajax) result 301).
* [ ] http://helpful.knobs-dials.com/index.php/Component_returned_failure_code:_0x80040111_ with 1 occurrences migrated to:
  https://helpful.knobs-dials.com/index.php/Component_returned_failure_code:_0x80040111_ ([https](https://helpful.knobs-dials.com/index.php/Component_returned_failure_code:_0x80040111_) result 301).
* [ ] http://jira.springsource.org/browse/DATAJPA with 1 occurrences migrated to:
  https://jira.springsource.org/browse/DATAJPA ([https](https://jira.springsource.org/browse/DATAJPA) result 301).
* [ ] http://jquery.org/license with 2 occurrences migrated to:
  https://jquery.org/license ([https](https://jquery.org/license) result 301).
* [ ] http://projects.spring.io/spring-data-jpa with 1 occurrences migrated to:
  https://projects.spring.io/spring-data-jpa ([https](https://projects.spring.io/spring-data-jpa) result 301).
* [ ] http://research.google.com/pubs/pub36356.html with 15 occurrences migrated to:
  https://research.google.com/pubs/pub36356.html ([https](https://research.google.com/pubs/pub36356.html) result 301).
* [ ] http://spring-projects.github.io/ with 1 occurrences migrated to:
  https://spring-projects.github.io/ ([https](https://spring-projects.github.io/) result 301).
* [ ] http://twitter.github.com/bootstrap/javascript.html with 13 occurrences migrated to:
  https://twitter.github.com/bootstrap/javascript.html ([https](https://twitter.github.com/bootstrap/javascript.html) result 301).
* [ ] http://www.modernizr.com/ with 1 occurrences migrated to:
  https://www.modernizr.com/ ([https](https://www.modernizr.com/) result 301).
* [ ] http://www.spring.io with 8 occurrences migrated to:
  https://www.spring.io ([https](https://www.spring.io) result 301).
* [ ] http://www.w3.org/TR/css3-selectors/ with 2 occurrences migrated to:
  https://www.w3.org/TR/css3-selectors/ ([https](https://www.w3.org/TR/css3-selectors/) result 301).
* [ ] http://www.w3.org/TR/css3-syntax/ with 1 occurrences migrated to:
  https://www.w3.org/TR/css3-syntax/ ([https](https://www.w3.org/TR/css3-syntax/) result 301).
* [ ] http://docsbrewing-zipkin-server.cfapps.io/ with 11 occurrences migrated to:
  https://docsbrewing-zipkin-server.cfapps.io/ ([https](https://docsbrewing-zipkin-server.cfapps.io/) result 302).
* [ ] http://docssleuth-zipkin-server.cfapps.io/ with 25 occurrences migrated to:
  https://docssleuth-zipkin-server.cfapps.io/ ([https](https://docssleuth-zipkin-server.cfapps.io/) result 302).
* [ ] http://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx with 2 occurrences migrated to:
  https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx ([https](https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx) result 302).
* [ ] http://repo.spring.io/milestone with 4 occurrences migrated to:
  https://repo.spring.io/milestone ([https](https://repo.spring.io/milestone) result 302).
* [ ] http://scripts.sil.org/OFL with 2 occurrences migrated to:
  https://scripts.sil.org/OFL ([https](https://scripts.sil.org/OFL) result 302).
* [ ] http://weblogs.java.net/blog/driscoll/archive/2009/09/08/eval-javascript-global-context with 1 occurrences migrated to:
  https://weblogs.java.net/blog/driscoll/archive/2009/09/08/eval-javascript-global-context ([https](https://weblogs.java.net/blog/driscoll/archive/2009/09/08/eval-javascript-global-context) result 302).

# Ignored
These URLs were intentionally ignored.

* http://docbook.org/ns/docbook with 6728 occurrences
* http://localhost:4000/spring-xyz/ with 1 occurrences
* http://some/address with 13 occurrences
* http://www with 1 occurrences
* http://www.w3.org/1999/xhtml with 2 occurrences
* http://www.w3.org/1999/xlink with 1 occurrences
* http://www.w3.org/2000/svg with 1 occurrences
* http://zipkinserver/ with 11 occurrences
2019-03-26 03:51:33 -05:00

2510 lines
111 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=edge"><![endif]-->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 1.5.3">
<meta name="author" content="Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer">
<title>Spring Cloud Sleuth</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
<style>
/* Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */
/* Remove comment around @import statement below when using as a custom stylesheet */
/*@import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700";*/
article,aside,details,figcaption,figure,footer,header,hgroup,main,nav,section,summary{display:block}
audio,canvas,video{display:inline-block}
audio:not([controls]){display:none;height:0}
[hidden],template{display:none}
script{display:none!important}
html{font-family:sans-serif;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}
body{margin:0}
a{background:transparent}
a:focus{outline:thin dotted}
a:active,a:hover{outline:0}
h1{font-size:2em;margin:.67em 0}
abbr[title]{border-bottom:1px dotted}
b,strong{font-weight:bold}
dfn{font-style:italic}
hr{-moz-box-sizing:content-box;box-sizing:content-box;height:0}
mark{background:#ff0;color:#000}
code,kbd,pre,samp{font-family:monospace;font-size:1em}
pre{white-space:pre-wrap}
q{quotes:"\201C" "\201D" "\2018" "\2019"}
small{font-size:80%}
sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}
sup{top:-.5em}
sub{bottom:-.25em}
img{border:0}
svg:not(:root){overflow:hidden}
figure{margin:0}
fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}
legend{border:0;padding:0}
button,input,select,textarea{font-family:inherit;font-size:100%;margin:0}
button,input{line-height:normal}
button,select{text-transform:none}
button,html input[type="button"],input[type="reset"],input[type="submit"]{-webkit-appearance:button;cursor:pointer}
button[disabled],html input[disabled]{cursor:default}
input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0}
input[type="search"]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}
input[type="search"]::-webkit-search-cancel-button,input[type="search"]::-webkit-search-decoration{-webkit-appearance:none}
button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}
textarea{overflow:auto;vertical-align:top}
table{border-collapse:collapse;border-spacing:0}
*,*:before,*:after{-moz-box-sizing:border-box;-webkit-box-sizing:border-box;box-sizing:border-box}
html,body{font-size:100%}
body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;font-weight:400;font-style:normal;line-height:1;position:relative;cursor:auto}
a:hover{cursor:pointer}
img,object,embed{max-width:100%;height:auto}
object,embed{height:100%}
img{-ms-interpolation-mode:bicubic}
.left{float:left!important}
.right{float:right!important}
.text-left{text-align:left!important}
.text-right{text-align:right!important}
.text-center{text-align:center!important}
.text-justify{text-align:justify!important}
.hide{display:none}
body{-webkit-font-smoothing:antialiased}
img,object,svg{display:inline-block;vertical-align:middle}
textarea{height:auto;min-height:50px}
select{width:100%}
.center{margin-left:auto;margin-right:auto}
.spread{width:100%}
p.lead,.paragraph.lead>p,#preamble>.sectionbody>.paragraph:first-of-type p{font-size:1.21875em;line-height:1.6}
.subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em}
div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0;direction:ltr}
a{color:#2156a5;text-decoration:underline;line-height:inherit}
a:hover,a:focus{color:#1d4b8f}
a img{border:none}
p{font-family:inherit;font-weight:400;font-size:1em;line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility}
p aside{font-size:.875em;line-height:1.35;font-style:italic}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em}
h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0}
h1{font-size:2.125em}
h2{font-size:1.6875em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em}
h4,h5{font-size:1.125em}
h6{font-size:1em}
hr{border:solid #ddddd8;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em;height:0}
em,i{font-style:italic;line-height:inherit}
strong,b{font-weight:bold;line-height:inherit}
small{font-size:60%;line-height:inherit}
code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)}
ul,ol,dl{font-size:1em;line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit}
ul,ol,ul.no-bullet,ol.no-bullet{margin-left:1.5em}
ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0;font-size:1em}
ul.square li ul,ul.circle li ul,ul.disc li ul{list-style:inherit}
ul.square{list-style-type:square}
ul.circle{list-style-type:circle}
ul.disc{list-style-type:disc}
ul.no-bullet{list-style:none}
ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0}
dl dt{margin-bottom:.3125em;font-weight:bold}
dl dd{margin-bottom:1.25em}
abbr,acronym{text-transform:uppercase;font-size:90%;color:rgba(0,0,0,.8);border-bottom:1px dotted #ddd;cursor:help}
abbr{text-transform:none}
blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd}
blockquote cite{display:block;font-size:.9375em;color:rgba(0,0,0,.6)}
blockquote cite:before{content:"\2014 \0020"}
blockquote cite a,blockquote cite a:visited{color:rgba(0,0,0,.6)}
blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)}
@media only screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2}
h1{font-size:2.75em}
h2{font-size:2.3125em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em}
h4{font-size:1.4375em}}
table{background:#fff;margin-bottom:1.25em;border:solid 1px #dedede}
table thead,table tfoot{background:#f7f8f7;font-weight:bold}
table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left}
table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)}
table tr.even,table tr.alt,table tr:nth-of-type(even){background:#f8f8f7}
table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{display:table-cell;line-height:1.6}
body{tab-size:4}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em}
h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400}
.clearfix:before,.clearfix:after,.float-group:before,.float-group:after{content:" ";display:table}
.clearfix:after,.float-group:after{clear:both}
*:not(pre)>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background-color:#f7f7f8;-webkit-border-radius:4px;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed}
pre,pre>code{line-height:1.45;color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;text-rendering:optimizeSpeed}
.keyseq{color:rgba(51,51,51,.8)}
kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background-color:#f7f7f7;border:1px solid #ccc;-webkit-border-radius:3px;border-radius:3px;-webkit-box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em white inset;box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em #fff inset;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap}
.keyseq kbd:first-child{margin-left:0}
.keyseq kbd:last-child{margin-right:0}
.menuseq,.menu{color:rgba(0,0,0,.8)}
b.button:before,b.button:after{position:relative;top:-1px;font-weight:400}
b.button:before{content:"[";padding:0 3px 0 2px}
b.button:after{content:"]";padding:0 2px 0 3px}
p a>code:hover{color:rgba(0,0,0,.9)}
#header,#content,#footnotes,#footer{width:100%;margin-left:auto;margin-right:auto;margin-top:0;margin-bottom:0;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em}
#header:before,#header:after,#content:before,#content:after,#footnotes:before,#footnotes:after,#footer:before,#footer:after{content:" ";display:table}
#header:after,#content:after,#footnotes:after,#footer:after{clear:both}
#content{margin-top:1.25em}
#content:before{content:none}
#header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0}
#header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #ddddd8}
#header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #ddddd8;padding-bottom:8px}
#header .details{border-bottom:1px solid #ddddd8;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:-ms-flexbox;display:-webkit-flex;display:flex;-ms-flex-flow:row wrap;-webkit-flex-flow:row wrap;flex-flow:row wrap}
#header .details span:first-child{margin-left:-.125em}
#header .details span.email a{color:rgba(0,0,0,.85)}
#header .details br{display:none}
#header .details br+span:before{content:"\00a0\2013\00a0"}
#header .details br+span.author:before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)}
#header .details br+span#revremark:before{content:"\00a0|\00a0"}
#header #revnumber{text-transform:capitalize}
#header #revnumber:after{content:"\00a0"}
#content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #ddddd8;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem}
#toc{border-bottom:1px solid #efefed;padding-bottom:.5em}
#toc>ul{margin-left:.125em}
#toc ul.sectlevel0>li>a{font-style:italic}
#toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0}
#toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none}
#toc li{line-height:1.3334;margin-top:.3334em}
#toc a{text-decoration:none}
#toc a:active{text-decoration:underline}
#toctitle{color:#7a2518;font-size:1.2em}
@media only screen and (min-width:768px){#toctitle{font-size:1.375em}
body.toc2{padding-left:15em;padding-right:0}
#toc.toc2{margin-top:0!important;background-color:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #efefed;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto}
#toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em}
#toc.toc2>ul{font-size:.9em;margin-bottom:0}
#toc.toc2 ul ul{margin-left:0;padding-left:1em}
#toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em}
body.toc2.toc-right{padding-left:0;padding-right:15em}
body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #efefed;left:auto;right:0}}
@media only screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0}
#toc.toc2{width:20em}
#toc.toc2 #toctitle{font-size:1.375em}
#toc.toc2>ul{font-size:.95em}
#toc.toc2 ul ul{padding-left:1.25em}
body.toc2.toc-right{padding-left:0;padding-right:20em}}
#content #toc{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px}
#content #toc>:first-child{margin-top:0}
#content #toc>:last-child{margin-bottom:0}
#footer{max-width:100%;background-color:rgba(0,0,0,.8);padding:1.25em}
#footer-text{color:rgba(255,255,255,.8);line-height:1.44}
.sect1{padding-bottom:.625em}
@media only screen and (min-width:768px){.sect1{padding-bottom:1.25em}}
.sect1+.sect1{border-top:1px solid #efefed}
#content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400}
#content h1>a.anchor:before,h2>a.anchor:before,h3>a.anchor:before,#toctitle>a.anchor:before,.sidebarblock>.content>.title>a.anchor:before,h4>a.anchor:before,h5>a.anchor:before,h6>a.anchor:before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em}
#content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible}
#content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none}
#content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221}
.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em}
.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic}
table.tableblock>caption.title{white-space:nowrap;overflow:visible;max-width:0}
.paragraph.lead>p,#preamble>.sectionbody>.paragraph:first-of-type p{color:rgba(0,0,0,.85)}
table.tableblock #preamble>.sectionbody>.paragraph:first-of-type p{font-size:inherit}
.admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%}
.admonitionblock>table td.icon{text-align:center;width:80px}
.admonitionblock>table td.icon img{max-width:none}
.admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase}
.admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #ddddd8;color:rgba(0,0,0,.6)}
.admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0}
.exampleblock>.content{border-style:solid;border-width:1px;border-color:#e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;-webkit-border-radius:4px;border-radius:4px}
.exampleblock>.content>:first-child{margin-top:0}
.exampleblock>.content>:last-child{margin-bottom:0}
.sidebarblock{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px}
.sidebarblock>:first-child{margin-top:0}
.sidebarblock>:last-child{margin-bottom:0}
.sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center}
.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0}
.literalblock pre,.listingblock pre:not(.highlight),.listingblock pre[class="highlight"],.listingblock pre[class^="highlight "],.listingblock pre.CodeRay,.listingblock pre.prettyprint{background:#f7f7f8}
.sidebarblock .literalblock pre,.sidebarblock .listingblock pre:not(.highlight),.sidebarblock .listingblock pre[class="highlight"],.sidebarblock .listingblock pre[class^="highlight "],.sidebarblock .listingblock pre.CodeRay,.sidebarblock .listingblock pre.prettyprint{background:#f2f1f1}
.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{-webkit-border-radius:4px;border-radius:4px;word-wrap:break-word;padding:1em;font-size:.8125em}
.literalblock pre.nowrap,.literalblock pre[class].nowrap,.listingblock pre.nowrap,.listingblock pre[class].nowrap{overflow-x:auto;white-space:pre;word-wrap:normal}
@media only screen and (min-width:768px){.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{font-size:.90625em}}
@media only screen and (min-width:1280px){.literalblock pre,.literalblock pre[class],.listingblock pre,.listingblock pre[class]{font-size:1em}}
.literalblock.output pre{color:#f7f7f8;background-color:rgba(0,0,0,.9)}
.listingblock pre.highlightjs{padding:0}
.listingblock pre.highlightjs>code{padding:1em;-webkit-border-radius:4px;border-radius:4px}
.listingblock pre.prettyprint{border-width:0}
.listingblock>.content{position:relative}
.listingblock code[data-lang]:before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:#999}
.listingblock:hover code[data-lang]:before{display:block}
.listingblock.terminal pre .command:before{content:attr(data-prompt);padding-right:.5em;color:#999}
.listingblock.terminal pre .command:not([data-prompt]):before{content:"$"}
table.pyhltable{border-collapse:separate;border:0;margin-bottom:0;background:none}
table.pyhltable td{vertical-align:top;padding-top:0;padding-bottom:0;line-height:1.45}
table.pyhltable td.code{padding-left:.75em;padding-right:0}
pre.pygments .lineno,table.pyhltable td:not(.code){color:#999;padding-left:0;padding-right:.5em;border-right:1px solid #ddddd8}
pre.pygments .lineno{display:inline-block;margin-right:.25em}
table.pyhltable .linenodiv{background:none!important;padding-right:0!important}
.quoteblock{margin:0 1em 1.25em 1.5em;display:table}
.quoteblock>.title{margin-left:-1.5em;margin-bottom:.75em}
.quoteblock blockquote,.quoteblock blockquote p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify}
.quoteblock blockquote{margin:0;padding:0;border:0}
.quoteblock blockquote:before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)}
.quoteblock blockquote>.paragraph:last-child p{margin-bottom:0}
.quoteblock .attribution{margin-top:.5em;margin-right:.5ex;text-align:right}
.quoteblock .quoteblock{margin-left:0;margin-right:0;padding:.5em 0;border-left:3px solid rgba(0,0,0,.6)}
.quoteblock .quoteblock blockquote{padding:0 0 0 .75em}
.quoteblock .quoteblock blockquote:before{display:none}
.verseblock{margin:0 1em 1.25em 1em}
.verseblock pre{font-family:"Open Sans","DejaVu Sans",sans;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility}
.verseblock pre strong{font-weight:400}
.verseblock .attribution{margin-top:1.25rem;margin-left:.5ex}
.quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic}
.quoteblock .attribution br,.verseblock .attribution br{display:none}
.quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)}
.quoteblock.abstract{margin:0 0 1.25em 0;display:block}
.quoteblock.abstract blockquote,.quoteblock.abstract blockquote p{text-align:left;word-spacing:0}
.quoteblock.abstract blockquote:before,.quoteblock.abstract blockquote p:first-of-type:before{display:none}
table.tableblock{max-width:100%;border-collapse:separate}
table.tableblock td>.paragraph:last-child p>p:last-child,table.tableblock th>p:last-child,table.tableblock td>p:last-child{margin-bottom:0}
table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede}
table.grid-all th.tableblock,table.grid-all td.tableblock{border-width:0 1px 1px 0}
table.grid-all tfoot>tr>th.tableblock,table.grid-all tfoot>tr>td.tableblock{border-width:1px 1px 0 0}
table.grid-cols th.tableblock,table.grid-cols td.tableblock{border-width:0 1px 0 0}
table.grid-all *>tr>.tableblock:last-child,table.grid-cols *>tr>.tableblock:last-child{border-right-width:0}
table.grid-rows th.tableblock,table.grid-rows td.tableblock{border-width:0 0 1px 0}
table.grid-all tbody>tr:last-child>th.tableblock,table.grid-all tbody>tr:last-child>td.tableblock,table.grid-all thead:last-child>tr>th.tableblock,table.grid-rows tbody>tr:last-child>th.tableblock,table.grid-rows tbody>tr:last-child>td.tableblock,table.grid-rows thead:last-child>tr>th.tableblock{border-bottom-width:0}
table.grid-rows tfoot>tr>th.tableblock,table.grid-rows tfoot>tr>td.tableblock{border-width:1px 0 0 0}
table.frame-all{border-width:1px}
table.frame-sides{border-width:0 1px}
table.frame-topbot{border-width:1px 0}
th.halign-left,td.halign-left{text-align:left}
th.halign-right,td.halign-right{text-align:right}
th.halign-center,td.halign-center{text-align:center}
th.valign-top,td.valign-top{vertical-align:top}
th.valign-bottom,td.valign-bottom{vertical-align:bottom}
th.valign-middle,td.valign-middle{vertical-align:middle}
table thead th,table tfoot th{font-weight:bold}
tbody tr th{display:table-cell;line-height:1.6;background:#f7f8f7}
tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold}
p.tableblock>code:only-child{background:none;padding:0}
p.tableblock{font-size:1em}
td>div.verse{white-space:pre}
ol{margin-left:1.75em}
ul li ol{margin-left:1.5em}
dl dd{margin-left:1.125em}
dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0}
ol>li p,ul>li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em}
ul.unstyled,ol.unnumbered,ul.checklist,ul.none{list-style-type:none}
ul.unstyled,ol.unnumbered,ul.checklist{margin-left:.625em}
ul.checklist li>p:first-child>.fa-square-o:first-child,ul.checklist li>p:first-child>.fa-check-square-o:first-child{width:1em;font-size:.85em}
ul.checklist li>p:first-child>input[type="checkbox"]:first-child{width:1em;position:relative;top:1px}
ul.inline{margin:0 auto .625em auto;margin-left:-1.375em;margin-right:0;padding:0;list-style:none;overflow:hidden}
ul.inline>li{list-style:none;float:left;margin-left:1.375em;display:block}
ul.inline>li>*{display:block}
.unstyled dl dt{font-weight:400;font-style:normal}
ol.arabic{list-style-type:decimal}
ol.decimal{list-style-type:decimal-leading-zero}
ol.loweralpha{list-style-type:lower-alpha}
ol.upperalpha{list-style-type:upper-alpha}
ol.lowerroman{list-style-type:lower-roman}
ol.upperroman{list-style-type:upper-roman}
ol.lowergreek{list-style-type:lower-greek}
.hdlist>table,.colist>table{border:0;background:none}
.hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none}
td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em}
td.hdlist1{font-weight:bold;padding-bottom:1.25em}
.literalblock+.colist,.listingblock+.colist{margin-top:-.5em}
.colist>table tr>td:first-of-type{padding:0 .75em;line-height:1}
.colist>table tr>td:last-of-type{padding:.25em 0}
.thumb,.th{line-height:0;display:inline-block;border:solid 4px #fff;-webkit-box-shadow:0 0 0 1px #ddd;box-shadow:0 0 0 1px #ddd}
.imageblock.left,.imageblock[style*="float: left"]{margin:.25em .625em 1.25em 0}
.imageblock.right,.imageblock[style*="float: right"]{margin:.25em 0 1.25em .625em}
.imageblock>.title{margin-bottom:0}
.imageblock.thumb,.imageblock.th{border-width:6px}
.imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em}
.image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0}
.image.left{margin-right:.625em}
.image.right{margin-left:.625em}
a.image{text-decoration:none;display:inline-block}
a.image object{pointer-events:none}
sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super}
sup.footnote a,sup.footnoteref a{text-decoration:none}
sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline}
#footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em}
#footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em 0;border-width:1px 0 0 0}
#footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;text-indent:-1.05em;margin-bottom:.2em}
#footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none}
#footnotes .footnote:last-of-type{margin-bottom:0}
#content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0}
.gist .file-data>table{border:0;background:#fff;width:100%;margin-bottom:0}
.gist .file-data>table td.line-data{width:99%}
div.unbreakable{page-break-inside:avoid}
.big{font-size:larger}
.small{font-size:smaller}
.underline{text-decoration:underline}
.overline{text-decoration:overline}
.line-through{text-decoration:line-through}
.aqua{color:#00bfbf}
.aqua-background{background-color:#00fafa}
.black{color:#000}
.black-background{background-color:#000}
.blue{color:#0000bf}
.blue-background{background-color:#0000fa}
.fuchsia{color:#bf00bf}
.fuchsia-background{background-color:#fa00fa}
.gray{color:#606060}
.gray-background{background-color:#7d7d7d}
.green{color:#006000}
.green-background{background-color:#007d00}
.lime{color:#00bf00}
.lime-background{background-color:#00fa00}
.maroon{color:#600000}
.maroon-background{background-color:#7d0000}
.navy{color:#000060}
.navy-background{background-color:#00007d}
.olive{color:#606000}
.olive-background{background-color:#7d7d00}
.purple{color:#600060}
.purple-background{background-color:#7d007d}
.red{color:#bf0000}
.red-background{background-color:#fa0000}
.silver{color:#909090}
.silver-background{background-color:#bcbcbc}
.teal{color:#006060}
.teal-background{background-color:#007d7d}
.white{color:#bfbfbf}
.white-background{background-color:#fafafa}
.yellow{color:#bfbf00}
.yellow-background{background-color:#fafa00}
span.icon>.fa{cursor:default}
.admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default}
.admonitionblock td.icon .icon-note:before{content:"\f05a";color:#19407c}
.admonitionblock td.icon .icon-tip:before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111}
.admonitionblock td.icon .icon-warning:before{content:"\f071";color:#bf6900}
.admonitionblock td.icon .icon-caution:before{content:"\f06d";color:#bf3400}
.admonitionblock td.icon .icon-important:before{content:"\f06a";color:#bf0000}
.conum[data-value]{display:inline-block;color:#fff!important;background-color:rgba(0,0,0,.8);-webkit-border-radius:100px;border-radius:100px;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold}
.conum[data-value] *{color:#fff!important}
.conum[data-value]+b{display:none}
.conum[data-value]:after{content:attr(data-value)}
pre .conum[data-value]{position:relative;top:-.125em}
b.conum *{color:inherit!important}
.conum:not([data-value]):empty{display:none}
dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility}
h1,h2,p,td.content,span.alt{letter-spacing:-.01em}
p strong,td.content strong,div.footnote strong{letter-spacing:-.005em}
p,blockquote,dt,td.content,span.alt{font-size:1.0625rem}
p{margin-bottom:1.25rem}
.sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em}
.exampleblock>.content{background-color:#fffef7;border-color:#e0e0dc;-webkit-box-shadow:0 1px 4px #e0e0dc;box-shadow:0 1px 4px #e0e0dc}
.print-only{display:none!important}
@media print{@page{margin:1.25cm .75cm}
*{-webkit-box-shadow:none!important;box-shadow:none!important;text-shadow:none!important}
a{color:inherit!important;text-decoration:underline!important}
a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important}
a[href^="http:"]:not(.bare):after,a[href^="https:"]:not(.bare):after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em}
abbr[title]:after{content:" (" attr(title) ")"}
pre,blockquote,tr,img,object,svg{page-break-inside:avoid}
thead{display:table-header-group}
svg{max-width:100%}
p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3}
h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid}
#toc,.sidebarblock,.exampleblock>.content{background:none!important}
#toc{border-bottom:1px solid #ddddd8!important;padding-bottom:0!important}
.sect1{padding-bottom:0!important}
.sect1+.sect1{border:0!important}
#header>h1:first-child{margin-top:1.25rem}
body.book #header{text-align:center}
body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em 0}
body.book #header .details{border:0!important;display:block;padding:0!important}
body.book #header .details span:first-child{margin-left:0!important}
body.book #header .details br{display:block}
body.book #header .details br+span:before{content:none!important}
body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important}
body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always}
.listingblock code[data-lang]:before{display:block}
#footer{background:none!important;padding:0 .9375em}
#footer-text{color:rgba(0,0,0,.6)!important;font-size:.9em}
.hide-on-print{display:none!important}
.print-only{display:block!important}
.hide-for-print{display:none!important}
.show-for-print{display:inherit!important}}
</style>
</head>
<body class="article toc2 toc-left">
<div id="header">
<h1>Spring Cloud Sleuth</h1>
<div class="details">
<span id="author" class="author">Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer</span><br>
</div>
<div id="toc" class="toc2">
<div id="toctitle">Table of Contents</div>
<ul class="sectlevel2">
<li><a href="#_terminology">Terminology</a></li>
<li><a href="#_purpose">Purpose</a>
<ul class="sectlevel3">
<li><a href="#_distributed_tracing_with_zipkin">Distributed tracing with Zipkin</a></li>
<li><a href="#_visualizing_errors">Visualizing errors</a></li>
<li><a href="#_live_examples">Live examples</a></li>
<li><a href="#_log_correlation">Log correlation</a>
<ul class="sectlevel4">
<li><a href="#_json_logback_with_logstash">JSON Logback with Logstash</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#_adding_to_the_project">Adding to the project</a>
<ul class="sectlevel3">
<li><a href="#_only_sleuth_log_correlation">Only Sleuth (log correlation)</a></li>
<li><a href="#_sleuth_with_zipkin_via_http">Sleuth with Zipkin via HTTP</a></li>
<li><a href="#_sleuth_with_zipkin_via_spring_cloud_stream">Sleuth with Zipkin via Spring Cloud Stream</a></li>
<li><a href="#_spring_cloud_sleuth_stream_zipkin_collector">Spring Cloud Sleuth Stream Zipkin Collector</a></li>
</ul>
</li>
<li><a href="#_additional_resources">Additional resources</a></li>
<li><a href="#_features">Features</a></li>
<li><a href="#_sampling">Sampling</a></li>
<li><a href="#_instrumentation">Instrumentation</a></li>
<li><a href="#_span_lifecycle">Span lifecycle</a>
<ul class="sectlevel2">
<li><a href="#creating-and-closing-spans">Creating and closing spans</a></li>
<li><a href="#continuing-spans">Continuing spans</a></li>
<li><a href="#creating-spans-with-explicit-parent">Creating spans with an explicit parent</a></li>
</ul>
</li>
<li><a href="#_naming_spans">Naming spans</a>
<ul class="sectlevel2">
<li><a href="#__spanname_annotation">@SpanName annotation</a></li>
<li><a href="#_tostring_method">toString() method</a></li>
</ul>
</li>
<li><a href="#_customizations">Customizations</a>
<ul class="sectlevel2">
<li><a href="#_spring_integration">Spring Integration</a></li>
<li><a href="#_http">HTTP</a></li>
<li><a href="#_example">Example</a></li>
<li><a href="#_custom_sa_tag_in_zipkin">Custom SA tag in Zipkin</a></li>
<li><a href="#_custom_service_name">Custom service name</a></li>
<li><a href="#_host_locator">Host locator</a></li>
</ul>
</li>
<li><a href="#_span_data_as_messages">Span Data as Messages</a>
<ul class="sectlevel2">
<li><a href="#_zipkin_consumer">Zipkin Consumer</a></li>
<li><a href="#_custom_consumer">Custom Consumer</a></li>
</ul>
</li>
<li><a href="#_metrics">Metrics</a></li>
<li><a href="#_integrations">Integrations</a>
<ul class="sectlevel2">
<li><a href="#_runnable_and_callable">Runnable and Callable</a></li>
<li><a href="#_hystrix">Hystrix</a>
<ul class="sectlevel3">
<li><a href="#_custom_concurrency_strategy">Custom Concurrency Strategy</a></li>
<li><a href="#_manual_command_setting">Manual Command setting</a></li>
</ul>
</li>
<li><a href="#_rxjava">RxJava</a></li>
<li><a href="#_http_integration">HTTP integration</a>
<ul class="sectlevel3">
<li><a href="#_http_filter">HTTP Filter</a></li>
<li><a href="#_handlerinterceptor">HandlerInterceptor</a></li>
<li><a href="#_async_servlet_support">Async Servlet support</a></li>
</ul>
</li>
<li><a href="#_http_client_integration">HTTP client integration</a>
<ul class="sectlevel3">
<li><a href="#_synchronous_rest_template">Synchronous Rest Template</a></li>
<li><a href="#_asynchronous_rest_template">Asynchronous Rest Template</a></li>
</ul>
</li>
<li><a href="#_feign">Feign</a></li>
<li><a href="#_asynchronous_communication">Asynchronous communication</a>
<ul class="sectlevel3">
<li><a href="#__async_annotated_methods">@Async annotated methods</a></li>
<li><a href="#__scheduled_annotated_methods">@Scheduled annotated methods</a></li>
<li><a href="#_executor_executorservice_and_scheduledexecutorservice">Executor, ExecutorService and ScheduledExecutorService</a></li>
</ul>
</li>
<li><a href="#_messaging">Messaging</a></li>
<li><a href="#_zuul">Zuul</a></li>
</ul>
</li>
<li><a href="#_running_examples">Running examples</a></li>
</ul>
</div>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p><strong>1.0.13.BUILD-SNAPSHOT</strong></p>
</div>
<div class="paragraph">
<p>Spring Cloud Sleuth implements a distributed tracing solution for <a href="https://cloud.spring.io">Spring Cloud</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_terminology">Terminology</h3>
<div class="paragraph">
<p>Spring Cloud Sleuth borrows <a href="https://research.google.com/pubs/pub36356.html">Dapper&#8217;s</a> terminology.</p>
</div>
<div class="paragraph">
<p><strong>Span:</strong> The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an
RPC. Span&#8217;s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span
is a part of. Spans also have other data, such as descriptions, timestamped events, key-value
annotations (tags), the ID of the span that caused them, and process ID&#8217;s (normally IP address).</p>
</div>
<div class="paragraph">
<p>Spans are started and stopped, and they keep track of their timing information. Once you create a
span, you must stop it at some point in the future.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<div class="title">Tip</div>
</td>
<td class="content">
The initial span that starts a trace is called a <code>root span</code>. The value of span id
of that span is equal to trace id.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p><strong>Trace:</strong> A set of spans forming a tree-like structure. For example, if you are running a distributed
big-data store, a trace might be formed by a put request.</p>
</div>
<div class="paragraph">
<p><strong>Annotation:</strong> is used to record existence of an event in time. Some of the core annotations used to define
the start and stop of a request are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>cs</strong> - Client Sent - The client has made a request. This annotation depicts the start of the span.</p>
</li>
<li>
<p><strong>sr</strong> - Server Received - The server side got the request and will start processing it.
If one subtracts the cs timestamp from this timestamp one will receive the network latency.</p>
</li>
<li>
<p><strong>ss</strong> - Server Sent - Annotated upon completion of request processing (when the response
got sent back to the client). If one subtracts the sr timestamp from this timestamp one
will receive the time needed by the server side to process the request.</p>
</li>
<li>
<p><strong>cr</strong> - Client Received - Signifies the end of the span. The client has successfully received the
response from the server side. If one subtracts the cs timestamp from this timestamp one
will receive the whole time needed by the client to receive the response from the server.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Visualization of what <strong>Span</strong> and <strong>Trace</strong> will look in a system together with the Zipkin annotations:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/trace-id.png" alt="Trace Info propagation">
</div>
</div>
<div class="paragraph">
<p>Each color of a note signifies a span (7 spans - from <strong>A</strong> to <strong>G</strong>). If you have such information in the note:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code>Trace Id = X
Span Id = D
Client Sent</code></pre>
</div>
</div>
<div class="paragraph">
<p>That means that the current span has <strong>Trace-Id</strong> set to <strong>X</strong>, <strong>Span-Id</strong> set to <strong>D</strong>. It also has emitted
<strong>Client Sent</strong> event.</p>
</div>
<div class="paragraph">
<p>This is how the visualization of the parent / child relationship of spans would look like:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/parents.png" alt="Parent child relationship">
</div>
</div>
</div>
<div class="sect2">
<h3 id="_purpose">Purpose</h3>
<div class="paragraph">
<p>In the following sections the example from the image above will be taken into consideration.</p>
</div>
<div class="sect3">
<h4 id="_distributed_tracing_with_zipkin">Distributed tracing with Zipkin</h4>
<div class="paragraph">
<p>Altogether there are <strong>7 spans</strong> . If you go to traces in Zipkin you will see this number in the second trace:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/zipkin-traces.png" alt="Traces">
</div>
</div>
<div class="paragraph">
<p>However if you pick a particular trace then you will see <strong>4 spans</strong>:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/zipkin-ui.png" alt="Traces Info propagation">
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to
Zipkin with Server Received and Server Sent / Client Received and Client Sent
annotations then they will presented as a single span.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Why is there a difference between the 7 and 4 spans in this case?</p>
</div>
<div class="ulist">
<ul>
<li>
<p>2 spans come from <code>http:/start</code> span. It has the Server Received (SR) and Server Sent (SS) annotations.</p>
</li>
<li>
<p>2 spans come from the RPC call from <code>service1</code> to <code>service2</code> to the <code>http:/foo</code> endpoint. It has the Client Sent (CS)
and Client Received (CR) annotations on <code>service1</code> side. It also has Server Received (SR) and Server Sent (SS) annotations
on the <code>service2</code> side. Physically there are 2 spans but they form 1 logical span related to an RPC call.</p>
</li>
<li>
<p>2 spans come from the RPC call from <code>service2</code> to <code>service3</code> to the <code>http:/bar</code> endpoint. It has the Client Sent (CS)
and Client Received (CR) annotations on <code>service2</code> side. It also has Server Received (SR) and Server Sent (SS) annotations
on the <code>service3</code> side. Physically there are 2 spans but they form 1 logical span related to an RPC call.</p>
</li>
<li>
<p>2 spans come from the RPC call from <code>service2</code> to <code>service4</code> to the <code>http:/baz</code> endpoint. It has the Client Sent (CS)
and Client Received (CR) annotations on <code>service2</code> side. It also has Server Received (SR) and Server Sent (SS) annotations
on the <code>service4</code> side. Physically there are 2 spans but they form 1 logical span related to an RPC call.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>So if we count the physical spans we have <strong>1</strong> from <code>http:/start</code>, <strong>2</strong> from <code>service1</code> calling <code>service2</code>, <strong>2</strong> form <code>service2</code>
calling <code>service3</code> and <strong>2</strong> from <code>service2</code> calling <code>service4</code>. Altogether <strong>7</strong> spans.</p>
</div>
<div class="paragraph">
<p>Logically we see the information of <strong>Total Spans: 4</strong> because we have <strong>1</strong> span related to the incoming request
to <code>service1</code> and <strong>3</strong> spans related to RPC calls.</p>
</div>
</div>
<div class="sect3">
<h4 id="_visualizing_errors">Visualizing errors</h4>
<div class="paragraph">
<p>Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn&#8217;t caught then we&#8217;re
setting proper tags on the span which Zipkin can properly colorize. You could see in the list of traces one
trace that was in red color. That&#8217;s because there was an exception thrown.</p>
</div>
<div class="paragraph">
<p>If you click that trace then you&#8217;ll see a similar picture</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/zipkin-error-traces.png" alt="Error Traces">
</div>
</div>
<div class="paragraph">
<p>Then if you click on one of the spans you&#8217;ll see the following</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/zipkin-error-trace-screenshot.png" alt="Error Traces Info propagation">
</div>
</div>
<div class="paragraph">
<p>As you can see you can easily see the reason for an error and the whole stacktrace related to it.</p>
</div>
</div>
<div class="sect3">
<h4 id="_live_examples">Live examples</h4>
<div class="imageblock">
<div class="content">
<a class="image" href="https://docssleuth-zipkin-server.cfapps.io/"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/pws.png" alt="Zipkin deployed on Pivotal Web Services" width="150" height="74"></a>
</div>
<div class="title">Click Pivotal Web Services icon to see it live!Click Pivotal Web Services icon to see it live!</div>
</div>
<div class="paragraph">
<p>The dependency graph in Zipkin would look like this:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/dependencies.png" alt="Dependencies">
</div>
</div>
<div class="imageblock">
<div class="content">
<a class="image" href="https://docssleuth-zipkin-server.cfapps.io/dependency"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/pws.png" alt="Zipkin deployed on Pivotal Web Services" width="150" height="74"></a>
</div>
<div class="title">Click Pivotal Web Services icon to see it live!Click Pivotal Web Services icon to see it live!</div>
</div>
</div>
<div class="sect3">
<h4 id="_log_correlation">Log correlation</h4>
<div class="paragraph">
<p>When grepping the logs of those four applications by trace id equal to e.g. <code>2485ec27856c56f4</code> one would get the following:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code>service1.log:2016-02-26 11:15:47.561 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Hello from service1. Calling service2
service2.log:2016-02-26 11:15:47.710 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Hello from service2. Calling service3 and then service4
service3.log:2016-02-26 11:15:47.895 INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application : Hello from service3
service2.log:2016-02-26 11:15:47.924 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service3 [Hello from service3]
service4.log:2016-02-26 11:15:48.134 INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application : Hello from service4
service2.log:2016-02-26 11:15:48.156 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service4 [Hello from service4]
service1.log:2016-02-26 11:15:48.182 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]</code></pre>
</div>
</div>
<div class="paragraph">
<p>If you&#8217;re using a log aggregating tool like <a href="https://www.elastic.co/products/kibana">Kibana</a>,
<a href="https://www.splunk.com/">Splunk</a> etc. you can order the events that took place. An example of
Kibana would look like this:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-sleuth/1.0.x/docs/src/main/asciidoc/images/kibana.png" alt="Log correlation with Kibana">
</div>
</div>
<div class="paragraph">
<p>If you want to use <a href="https://www.elastic.co/guide/en/logstash/current/index.html">Logstash</a> here is the Grok pattern for Logstash:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code>filter {
# pattern matching logback pattern
grok {
match =&gt; { "message" =&gt; "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
}
}</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern:
</td>
</tr>
</table>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code>filter {
# pattern matching logback pattern
grok {
match =&gt; { "message" =&gt; "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
}
}</code></pre>
</div>
</div>
<div class="sect4">
<h5 id="_json_logback_with_logstash">JSON Logback with Logstash</h5>
<div class="paragraph">
<p>Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability
we&#8217;re passing the dependencies in the <code>groupId:artifactId:version</code> notation.</p>
</div>
<div class="paragraph">
<p><strong>Dependencies setup</strong></p>
</div>
<div class="ulist">
<ul>
<li>
<p>Ensure that Logback is on the classpath (<code>ch.qos.logback:logback-core</code>)</p>
</li>
<li>
<p>Add Logstash Logback encode - example for version <code>4.6</code> : <code>net.logstash.logback:logstash-logback-encoder:4.6</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><strong>Logback setup</strong></p>
</div>
<div class="paragraph">
<p>Below you can find an example of a Logback configuration (file named <a href="https://github.com/spring-cloud-samples/sleuth-documentation-apps/blob/master/service1/src/main/resources/logback-spring.xml">logback-spring.xml</a>) that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>logs information from the application in a JSON format to a <code>build/${spring.application.name}.json</code> file</p>
</li>
<li>
<p>has commented out two additional appenders - console and standard log file</p>
</li>
<li>
<p>has the same logging pattern as the one presented in the previous section</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-xml" data-lang="xml">&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;configuration&gt;
&lt;include resource="org/springframework/boot/logging/logback/defaults.xml"/&gt;
​
&lt;springProperty scope="context" name="springAppName" source="spring.application.name"/&gt;
&lt;!-- Example for logging into the build folder of your project --&gt;
&lt;property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${springAppName}"/&gt;​
&lt;!-- You can override this to have a custom pattern --&gt;
&lt;property name="CONSOLE_LOG_PATTERN"
value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/&gt;
&lt;!-- Appender to log to console --&gt;
&lt;appender name="console" class="ch.qos.logback.core.ConsoleAppender"&gt;
&lt;filter class="ch.qos.logback.classic.filter.ThresholdFilter"&gt;
&lt;!-- Minimum logging level to be presented in the console logs--&gt;
&lt;level&gt;DEBUG&lt;/level&gt;
&lt;/filter&gt;
&lt;encoder&gt;
&lt;pattern&gt;${CONSOLE_LOG_PATTERN}&lt;/pattern&gt;
&lt;charset&gt;utf8&lt;/charset&gt;
&lt;/encoder&gt;
&lt;/appender&gt;
&lt;!-- Appender to log to file --&gt;​
&lt;appender name="flatfile" class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;
&lt;file&gt;${LOG_FILE}&lt;/file&gt;
&lt;rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;
&lt;fileNamePattern&gt;${LOG_FILE}.%d{yyyy-MM-dd}.gz&lt;/fileNamePattern&gt;
&lt;maxHistory&gt;7&lt;/maxHistory&gt;
&lt;/rollingPolicy&gt;
&lt;encoder&gt;
&lt;pattern&gt;${CONSOLE_LOG_PATTERN}&lt;/pattern&gt;
&lt;charset&gt;utf8&lt;/charset&gt;
&lt;/encoder&gt;
&lt;/appender&gt;
​
&lt;!-- Appender to log to file in a JSON format --&gt;
&lt;appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;
&lt;file&gt;${LOG_FILE}.json&lt;/file&gt;
&lt;rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;
&lt;fileNamePattern&gt;${LOG_FILE}.json.%d{yyyy-MM-dd}.gz&lt;/fileNamePattern&gt;
&lt;maxHistory&gt;7&lt;/maxHistory&gt;
&lt;/rollingPolicy&gt;
&lt;encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"&gt;
&lt;providers&gt;
&lt;timestamp&gt;
&lt;timeZone&gt;UTC&lt;/timeZone&gt;
&lt;/timestamp&gt;
&lt;pattern&gt;
&lt;pattern&gt;
{
"severity": "%level",
"service": "${springAppName:-}",
"trace": "%X{X-B3-TraceId:-}",
"span": "%X{X-B3-SpanId:-}",
"parent": "%X{X-B3-ParentSpanId:-}",
"exportable": "%X{X-Span-Export:-}",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"rest": "%message"
}
&lt;/pattern&gt;
&lt;/pattern&gt;
&lt;/providers&gt;
&lt;/encoder&gt;
&lt;/appender&gt;
​
&lt;root level="INFO"&gt;
&lt;appender-ref ref="console"/&gt;
&lt;!-- uncomment this to have also JSON logs --&gt;
&lt;!--&lt;appender-ref ref="logstash"/&gt;--&gt;
&lt;!--&lt;appender-ref ref="flatfile"/&gt;--&gt;
&lt;/root&gt;
&lt;/configuration&gt;</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
If you&#8217;re using a custom <code>logback-spring.xml</code> then you have to pass the <code>spring.application.name</code> in
<code>bootstrap</code> instead of <code>application</code> property file. Otherwise your custom logback file won&#8217;t read the property properly.
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_adding_to_the_project">Adding to the project</h3>
<div class="sect3">
<h4 id="_only_sleuth_log_correlation">Only Sleuth (log correlation)</h4>
<div class="paragraph">
<p>If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add
the <code>spring-cloud-starter-sleuth</code> module to your project.</p>
</div>
<div class="listingblock primary">
<div class="title">Maven</div>
<div class="content">
<pre class="highlight"><code class="language-xml" data-lang="xml">&lt;dependencyManagement&gt; <b class="conum">(1)</b>
&lt;dependencies&gt;
&lt;dependency&gt;
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-dependencies&lt;/artifactId&gt;
&lt;version&gt;Brixton.RELEASE&lt;/version&gt;
&lt;type&gt;pom&lt;/type&gt;
&lt;scope&gt;import&lt;/scope&gt;
&lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;/dependencyManagement&gt;
&lt;dependency&gt; <b class="conum">(2)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-starter-sleuth&lt;/artifactId&gt;
&lt;/dependency&gt;</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-sleuth</code></p>
</li>
</ol>
</div>
<div class="listingblock secondary">
<div class="title">Gradle</div>
<div class="content">
<pre class="highlight"><code class="language-groovy" data-lang="groovy">dependencyManagement { <b class="conum">(1)</b>
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
}
}
dependencies { <b class="conum">(2)</b>
compile "org.springframework.cloud:spring-cloud-starter-sleuth"
}</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-sleuth</code></p>
</li>
</ol>
</div>
</div>
<div class="sect3">
<h4 id="_sleuth_with_zipkin_via_http">Sleuth with Zipkin via HTTP</h4>
<div class="paragraph">
<p>If you want both Sleuth and Zipkin just add the <code>spring-cloud-starter-zipkin</code> dependency.</p>
</div>
<div class="listingblock primary">
<div class="title">Maven</div>
<div class="content">
<pre class="highlight"><code class="language-xml" data-lang="xml">&lt;dependencyManagement&gt; <b class="conum">(1)</b>
&lt;dependencies&gt;
&lt;dependency&gt;
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-dependencies&lt;/artifactId&gt;
&lt;version&gt;Brixton.RELEASE&lt;/version&gt;
&lt;type&gt;pom&lt;/type&gt;
&lt;scope&gt;import&lt;/scope&gt;
&lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;/dependencyManagement&gt;
&lt;dependency&gt; <b class="conum">(2)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-starter-zipkin&lt;/artifactId&gt;
&lt;/dependency&gt;</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-zipkin</code></p>
</li>
</ol>
</div>
<div class="listingblock secondary">
<div class="title">Gradle</div>
<div class="content">
<pre class="highlight"><code class="language-groovy" data-lang="groovy">dependencyManagement { <b class="conum">(1)</b>
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
}
}
dependencies { <b class="conum">(2)</b>
compile "org.springframework.cloud:spring-cloud-starter-zipkin"
}</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-zipkin</code></p>
</li>
</ol>
</div>
</div>
<div class="sect3">
<h4 id="_sleuth_with_zipkin_via_spring_cloud_stream">Sleuth with Zipkin via Spring Cloud Stream</h4>
<div class="paragraph">
<p>If you want both Sleuth and Zipkin just add the <code>spring-cloud-sleuth-stream</code> dependency.</p>
</div>
<div class="listingblock primary">
<div class="title">Maven</div>
<div class="content">
<pre class="highlight"><code class="language-xml" data-lang="xml">&lt;dependencyManagement&gt; <b class="conum">(1)</b>
&lt;dependencies&gt;
&lt;dependency&gt;
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-dependencies&lt;/artifactId&gt;
&lt;version&gt;Brixton.RELEASE&lt;/version&gt;
&lt;type&gt;pom&lt;/type&gt;
&lt;scope&gt;import&lt;/scope&gt;
&lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;/dependencyManagement&gt;
&lt;dependency&gt; <b class="conum">(2)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-sleuth-stream&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;dependency&gt; <b class="conum">(3)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-starter-sleuth&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;!-- EXAMPLE FOR RABBIT BINDING --&gt;
&lt;dependency&gt; <b class="conum">(4)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-stream-binder-rabbit&lt;/artifactId&gt;
&lt;/dependency&gt;</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-sleuth-stream</code></p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-sleuth</code> - that way all dependant dependencies will be downloaded</p>
</li>
<li>
<p>Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to</p>
</li>
</ol>
</div>
<div class="listingblock secondary">
<div class="title">Gradle</div>
<div class="content">
<pre class="highlight"><code class="language-groovy" data-lang="groovy">dependencyManagement { <b class="conum">(1)</b>
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
}
}
dependencies {
compile "org.springframework.cloud:spring-cloud-sleuth-stream" <b class="conum">(2)</b>
compile "org.springframework.cloud:spring-cloud-starter-sleuth" <b class="conum">(3)</b>
// Example for Rabbit binding
compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" <b class="conum">(4)</b>
}</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-sleuth-stream</code></p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-sleuth</code> - that way all dependant dependencies will be downloaded</p>
</li>
<li>
<p>Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to</p>
</li>
</ol>
</div>
</div>
<div class="sect3">
<h4 id="_spring_cloud_sleuth_stream_zipkin_collector">Spring Cloud Sleuth Stream Zipkin Collector</h4>
<div class="paragraph">
<p>If you want to start a Spring Cloud Sleuth Stream Zipkin collector just add the <code>spring-cloud-sleuth-zipkin-stream</code>
dependency</p>
</div>
<div class="listingblock primary">
<div class="title">Maven</div>
<div class="content">
<pre class="highlight"><code class="language-xml" data-lang="xml">&lt;dependencyManagement&gt; <b class="conum">(1)</b>
&lt;dependencies&gt;
&lt;dependency&gt;
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-dependencies&lt;/artifactId&gt;
&lt;version&gt;Brixton.RELEASE&lt;/version&gt;
&lt;type&gt;pom&lt;/type&gt;
&lt;scope&gt;import&lt;/scope&gt;
&lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;/dependencyManagement&gt;
&lt;dependency&gt; <b class="conum">(2)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-sleuth-zipkin-stream&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;dependency&gt; <b class="conum">(3)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-starter-sleuth&lt;/artifactId&gt;
&lt;/dependency&gt;
&lt;!-- EXAMPLE FOR RABBIT BINDING --&gt;
&lt;dependency&gt; <b class="conum">(4)</b>
&lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt;
&lt;artifactId&gt;spring-cloud-stream-binder-rabbit&lt;/artifactId&gt;
&lt;/dependency&gt;</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-sleuth-zipkin-stream</code></p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-sleuth</code> - that way all dependant dependencies will be downloaded</p>
</li>
<li>
<p>Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to</p>
</li>
</ol>
</div>
<div class="listingblock secondary">
<div class="title">Gradle</div>
<div class="content">
<pre class="highlight"><code class="language-groovy" data-lang="groovy">dependencyManagement { <b class="conum">(1)</b>
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.RELEASE"
}
}
dependencies {
compile "org.springframework.cloud:spring-cloud-sleuth-zipkin-stream" <b class="conum">(2)</b>
compile "org.springframework.cloud:spring-cloud-starter-sleuth" <b class="conum">(3)</b>
// Example for Rabbit binding
compile "org.springframework.cloud:spring-cloud-stream-binder-rabbit" <b class="conum">(4)</b>
}</code></pre>
</div>
</div>
<div class="colist arabic">
<ol>
<li>
<p>In order not to pick versions by yourself it&#8217;s much better if you add the dependency management via
the Spring BOM</p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-sleuth-zipkin-stream</code></p>
</li>
<li>
<p>Add the dependency to <code>spring-cloud-starter-sleuth</code> - that way all dependant dependencies will be downloaded</p>
</li>
<li>
<p>Add a binder (e.g. Rabbit binder) to tell Spring Cloud Stream what it should bind to</p>
</li>
</ol>
</div>
<div class="paragraph">
<p>and then just annotate your main class with <code>@EnableZipkinStreamServer</code> annotation:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">package example;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.sleuth.zipkin.stream.EnableZipkinStreamServer;
@SpringBootApplication
@EnableZipkinStreamServer
public class ZipkinStreamServerApplication {
public static void main(String[] args) throws Exception {
SpringApplication.run(ZipkinStreamServerApplication.class, args);
}
}</code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_additional_resources">Additional resources</h2>
<div class="sectionbody">
<div class="paragraph">
<p><strong>Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin</strong></p>
</div>
<div class="videoblock">
<div class="content">
<iframe src="https://www.youtube.com/embed/eQV71Mw1u1c?rel=0" frameborder="0" allowfullscreen></iframe>
</div>
</div>
<div class="paragraph">
<p><a href="https://www.youtube.com/watch?v=eQV71Mw1u1c">click here to see the video</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_features">Features</h2>
<div class="sectionbody">
<div class="ulist">
<ul>
<li>
<p>Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Example logs:</p>
<div class="listingblock">
<div class="content">
<pre>2016-02-02 15:30:57.902 INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
2016-02-02 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
2016-02-02 15:31:01.936 INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...</pre>
</div>
</div>
<div class="paragraph">
<p>notice the <code>[appname,traceId,spanId,exportable]</code> entries from the MDC:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>spanId</strong> - the id of a specific operation that took place</p>
</li>
<li>
<p><strong>appname</strong> - the name of the application that logged the span</p>
</li>
<li>
<p><strong>traceId</strong> - the id of the latency graph that contains the span</p>
</li>
<li>
<p><strong>exportable</strong> - whether the log should be exported to Zipkin or not. When would you like the span not to be
exportable? In the case in which you want to wrap some operation in a Span and have it written to the logs
only.</p>
</li>
</ul>
</div>
</li>
<li>
<p>Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations,
key-value annotations. Loosely based on HTrace, but Zipkin (Dapper) compatible.</p>
</li>
<li>
<p>Sleuth records timing information to aid in latency analysis. Using sleuth, you can pinpoint causes of
latency in your applications. Sleuth is written to not log too much, and to not cause your production application to crash.</p>
<div class="ulist">
<ul>
<li>
<p>propagates structural data about your call-graph in-band, and the rest out-of-band.</p>
</li>
<li>
<p>includes opinionated instrumentation of layers such as HTTP</p>
</li>
<li>
<p>includes sampling policy to manage volume</p>
</li>
<li>
<p>can report to a Zipkin system for query and visualization</p>
</li>
</ul>
</div>
</li>
<li>
<p>Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints,
rest template, scheduled actions, message channels, zuul filters, feign client).</p>
</li>
<li>
<p>Sleuth includes default logic to join a trace across http or messaging boundaries. For example, http propagation
works via Zipkin-compatible request headers. This propagation logic is defined and customized via
<code>SpanInjector</code> and <code>SpanExtractor</code> implementations.</p>
</li>
<li>
<p>Provides simple metrics of accepted / dropped spans.</p>
</li>
<li>
<p>If <code>spring-cloud-sleuth-zipkin</code> then the app will generate and collect Zipkin-compatible traces.
By default it sends them via HTTP to a Zipkin server on localhost (port 9411).
Configure the location of the service using <code>spring.zipkin.baseUrl</code>.</p>
</li>
<li>
<p>If <code>spring-cloud-sleuth-stream</code> then the app will generate and collect traces via <a href="https://github.com/spring-cloud/spring-cloud-stream">Spring Cloud Stream</a>.
Your app automatically becomes a producer of tracer messages that are sent over your broker of choice
(e.g. RabbitMQ, Apache Kafka, Redis).</p>
</li>
</ul>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
If using Zipkin or Stream, configure the percentage of spans exported using <code>spring.sleuth.sampler.percentage</code>
(default 0.1, i.e. 10%). <strong>Otherwise you might think that Sleuth is not working cause it&#8217;s omitting some spans.</strong>
</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
the SLF4J MDC is always set and logback users will immediately see the trace and span ids in logs per the example
above. Other logging systems have to configure their own formatter to get the same result. The default is
<code>logging.pattern.level</code> set to <code>%clr(%5p) %clr([${spring.application.name:},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]){yellow}</code>
(this is a Spring Boot feature for logback users).
<strong>This means that if you&#8217;re not using SLF4J this pattern WILL NOT be automatically applied</strong>.
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_sampling">Sampling</h2>
<div class="sectionbody">
<div class="paragraph">
<p>In distributed tracing the data volumes can be very high so sampling
can be important (you usually don&#8217;t need to export all spans to get a
good picture of what is happening). Spring Cloud Sleuth has a
<code>Sampler</code> strategy that you can implement to take control of the
sampling algorithm. Samplers do not stop span (correlation) ids from
being generated, but they do prevent the tags and events being
attached and exported. By default you get a strategy that continues to
trace if a span is already active, but new ones are always marked as
non-exportable. If all your apps run with this sampler you will see
traces in logs, but not in any remote store. For testing the default
is often enough, and it probably is all you need if you are only using
the logs (e.g. with an ELK aggregator). If you are exporting span data
to Zipkin or Spring Cloud Stream, there is also an <code>AlwaysSampler</code>
that exports everything and a <code>PercentageBasedSampler</code> that samples a
fixed fraction of spans.</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
the <code>PercentageBasedSampler</code> is the default if you are using
<code>spring-cloud-sleuth-zipkin</code> or <code>spring-cloud-sleuth-stream</code>. You can
configure the exports using <code>spring.sleuth.sampler.percentage</code>. The passed
value needs to be a double from <code>0.0</code> to <code>1.0</code> so it&#8217;s not a percentage.
For backwards compatibility reasons we&#8217;re not changing the property name.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>A sampler can be installed just by creating a bean definition, e.g:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@Bean
public Sampler defaultSampler() {
return new AlwaysSampler();
}</code></pre>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_instrumentation">Instrumentation</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Spring Cloud Sleuth instruments all your Spring application
automatically, so you shouldn&#8217;t have to do anything to activate
it. The instrumentation is added using a variety of technologies
according to the stack that is available, e.g. for a servlet web
application we use a <code>Filter</code>, and for Spring Integration we use
<code>ChannelInterceptors</code>.</p>
</div>
<div class="paragraph">
<p>You can customize the keys used in span tags. To limit the volume of
span data, by default an HTTP request will be tagged only with a
handful of metadata like the status code, host and URL. You can add
request headers by configuring <code>spring.sleuth.keys.http.headers</code> (a
list of header names).</p>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
Remember that tags are only collected and exported if there is a
<code>Sampler</code> that allows it (by default there is not, so there is no
danger of accidentally collecting too much data without configuring
something).
</td>
</tr>
</table>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
Currently the instrumentation in Spring Cloud Sleuth is eager - it means that
we&#8217;re actively trying to pass the tracing context between threads. Also timing events
are captured even when sleuth isn&#8217;t exporting data to a tracing system.
This approach may change in the future towards being lazy on this matter.
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_span_lifecycle">Span lifecycle</h2>
<div class="sectionbody">
<div class="paragraph">
<p>You can do the following operations on the Span by means of <strong>org.springframework.cloud.sleuth.Tracer</strong> interface:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#creating-and-closing-spans">start</a> - when you start a span its name is assigned and start timestamp is recorded.</p>
</li>
<li>
<p><a href="#creating-and-closing-spans">close</a> - the span gets finished (the end time of the span is recorded) and if
the span is <strong>exportable</strong> then it will be eligible for collection to Zipkin.
The span is also removed from the current thread.</p>
</li>
<li>
<p><a href="#continuing-spans">continue</a> - a new instance of span will be created whereas it will be a copy of the
one that it continues.</p>
</li>
<li>
<p><a href="#continuing-spans">detach</a> - the span doesn&#8217;t get stopped or closed. It only gets removed from the current thread.</p>
</li>
<li>
<p><a href="#creating-spans-with-explicit-parent">create with explicit parent</a> - you can create a new span and set an explicit parent to it</p>
</li>
</ul>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<div class="title">Tip</div>
</td>
<td class="content">
Spring creates the instance of <code>Tracer</code> for you. In order to use it all you need is to just autowire it.
</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="creating-and-closing-spans">Creating and closing spans</h3>
<div class="paragraph">
<p>You can manually create spans by using the <strong>Tracer</strong> interface.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">// Start a span. If there was a span present in this thread it will become
// the `newSpan`'s parent.
Span newSpan = this.tracer.createSpan("calculateTax");
try {
// ...
// You can tag a span
this.tracer.addTag("taxValue", taxValue);
// ...
// You can log an event on a span
newSpan.logEvent("taxCalculated");
} finally {
// Once done remember to close the span. This will allow collecting
// the span to send it to Zipkin
this.tracer.close(newSpan);
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>In this example we could see how to create a new instance of span. Assuming that there already
was a span present in this thread then it would become the parent of that span.</p>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
Always clean after you create a span! Don&#8217;t forget to close a span if you want to send it to Zipkin.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="continuing-spans">Continuing spans</h3>
<div class="paragraph">
<p>Sometimes you don&#8217;t want to create a new span but you want to continue one. Example of such a
situation might be (of course it all depends on the use-case):</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>AOP</strong> - If there was already a span created before an aspect was reached then you might not want to create a new span.</p>
</li>
<li>
<p><strong>Hystrix</strong> - executing a Hystrix command is most likely a logical part of the current processing. It&#8217;s in fact
only a technical implementation detail that you wouldn&#8217;t necessarily want to reflect in tracing as a separate being.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The continued instance of span is equal to the one that it continues:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Span continuedSpan = this.tracer.continueSpan(spanToContinue);
assertThat(continuedSpan).isEqualTo(spanToContinue);</code></pre>
</div>
</div>
<div class="paragraph">
<p>To continue a span you can use the <strong>Tracer</strong> interface.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">// let's assume that we're in a thread Y and we've received
// the `initialSpan` from thread X
Span continuedSpan = this.tracer.continueSpan(initialSpan);
try {
// ...
// You can tag a span
this.tracer.addTag("taxValue", taxValue);
// ...
// You can log an event on a span
continuedSpan.logEvent("taxCalculated");
} finally {
// Once done remember to detach the span. That way you'll
// safely remove it from the current thread without closing it
this.tracer.detach(continuedSpan);
}</code></pre>
</div>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
Always clean after you create a span! Don&#8217;t forget to detach a span if some work was done started in one
thread (e.g. thread X) and it&#8217;s waiting for other threads (e.g. Y, Z) to finish.
Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected
the span in thread X should be closed.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="creating-spans-with-explicit-parent">Creating spans with an explicit parent</h3>
<div class="paragraph">
<p>There is a possibility that you want to start a new span and provide an explicit parent of that span.
Let&#8217;s assume that the parent of a span is in one thread and you want to start a new span in another thread. The
<code>startSpan</code> method of the <code>Tracer</code> interface is the method you are looking for.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">// let's assume that we're in a thread Y and we've received
// the `initialSpan` from thread X. `initialSpan` will be the parent
// of the `newSpan`
Span newSpan = this.tracer.createSpan("calculateCommission", initialSpan);
try {
// ...
// You can tag a span
this.tracer.addTag("commissionValue", commissionValue);
// ...
// You can log an event on a span
newSpan.logEvent("commissionCalculated");
} finally {
// Once done remember to close the span. This will allow collecting
// the span to send it to Zipkin. The tags and events set on the
// newSpan will not be present on the parent
this.tracer.close(newSpan);
}</code></pre>
</div>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs
related to the fact that you have a span present in the current thread other than the one you&#8217;re trying to close.
What&#8217;s worse your spans won&#8217;t get closed properly thus will not get collected to Zipkin.
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_naming_spans">Naming spans</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Picking a span name is not a trivial task. Span name should depict an operation name. The name should
be low cardinality (e.g. not include identifiers).</p>
</div>
<div class="paragraph">
<p>Since there is a lot of instrumentation going on some of the span names will be
artificial like:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>controller-method-name</code> when received by a Controller with a method name <code>conrollerMethodName</code></p>
</li>
<li>
<p><code>async</code> for asynchronous operations done via wrapped <code>Callable</code> and <code>Runnable</code>.</p>
</li>
<li>
<p><code>@Scheduled</code> annotated methods will return the simple name of the class.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Fortunately, for the asynchronous processing you can provide explicit naming.</p>
</div>
<div class="sect2">
<h3 id="__spanname_annotation">@SpanName annotation</h3>
<div class="paragraph">
<p>You can do name the span explicitly via the <code>@SpanName</code> annotation.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@SpanName("calculateTax")
class TaxCountingRunnable implements Runnable {
@Override public void run() {
// perform logic
}
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>In this case, when processed in the following manner:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Runnable runnable = new TraceRunnable(tracer, spanNamer, new TaxCountingRunnable());
Future&lt;?&gt; future = executorService.submit(runnable);
// ... some additional logic ...
future.get();</code></pre>
</div>
</div>
<div class="paragraph">
<p>The span will be named <code>calculateTax</code>.</p>
</div>
</div>
<div class="sect2">
<h3 id="_tostring_method">toString() method</h3>
<div class="paragraph">
<p>It&#8217;s pretty rare to create separate classes for <code>Runnable</code> or <code>Callable</code>. Typically one creates an anonymous
instance of those classes. You can&#8217;t annotate such classes thus to override that, if there is no <code>@SpanName</code> annotation present,
we&#8217;re checking if the class has a custom implementation of the <code>toString()</code> method.</p>
</div>
<div class="paragraph">
<p>So executing such code:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Runnable runnable = new TraceRunnable(tracer, spanNamer, new Runnable() {
@Override public void run() {
// perform logic
}
@Override public String toString() {
return "calculateTax";
}
});
Future&lt;?&gt; future = executorService.submit(runnable);
// ... some additional logic ...
future.get();</code></pre>
</div>
</div>
<div class="paragraph">
<p>will lead in creating a span named <code>calculateTax</code>.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_customizations">Customizations</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Thanks to the <code>SpanInjector</code> and <code>SpanExtractor</code> you can customize the way spans
are created and propagated.</p>
</div>
<div class="paragraph">
<p>There are currently two built-in ways to pass tracing information between processes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>via Spring Integration</p>
</li>
<li>
<p>via HTTP</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Span ids are extracted from Zipkin-compatible (B3) headers (either <code>Message</code>
or HTTP headers), to start or join an existing trace. Trace information is
injected into any outbound requests so the next hop can extract them.</p>
</div>
<div class="sect2">
<h3 id="_spring_integration">Spring Integration</h3>
<div class="paragraph">
<p>For Spring Integration these are the beans responsible for creation of a Span from a <code>Message</code>
and filling in the <code>MessageBuilder</code> with tracing information.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@Bean
public SpanExtractor&lt;Message&gt; messagingSpanExtractor() {
...
}
@Bean
public SpanInjector&lt;MessageBuilder&gt; messagingSpanInjector() {
...
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>You can override them by providing your own implementation and by adding a <code>@Primary</code> annotation
to your bean definition.</p>
</div>
</div>
<div class="sect2">
<h3 id="_http">HTTP</h3>
<div class="paragraph">
<p>For HTTP these are the beans responsible for creation of a Span from a <code>HttpServletRequest</code>.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@Bean
public SpanExtractor&lt;HttpServletRequest&gt; httpServletRequestSpanExtractor() {
...
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>You can override them by providing your own implementation and by adding a <code>@Primary</code> annotation
to your bean definition.</p>
</div>
</div>
<div class="sect2">
<h3 id="_example">Example</h3>
<div class="paragraph">
<p>Let&#8217;s assume that instead of the standard Zipkin compatible tracing HTTP header names
you have</p>
</div>
<div class="ulist">
<ul>
<li>
<p>for trace id - <code>correlationId</code></p>
</li>
<li>
<p>for span id - <code>mySpanId</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This is a an example of a <code>SpanExtractor</code></p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">static class CustomHttpServletRequestSpanExtractor
implements SpanExtractor&lt;HttpServletRequest&gt; {
@Override
public Span joinTrace(HttpServletRequest carrier) {
long traceId = Span.hexToId(carrier.getHeader("correlationId"));
long spanId = Span.hexToId(carrier.getHeader("mySpanId"));
// extract all necessary headers
Span.SpanBuilder builder = Span.builder().traceId(traceId).spanId(spanId);
// build rest of the Span
return builder.build();
}
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>And you could register it like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@Bean
@Primary
SpanExtractor&lt;HttpServletRequest&gt; customHttpServletRequestSpanExtractor() {
return new CustomHttpServletRequestSpanExtractor();
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>Spring Cloud Sleuth does not add trace/span related headers to the Http Response for security reasons. If you need the headers then a custom <code>SpanInjector</code>
that injects the headers into the Http Response and a Servlet filter which makes use of this can be added the following way:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">static class CustomHttpServletResponseSpanInjector
implements SpanInjector&lt;HttpServletResponse&gt; {
@Override
public void inject(Span span, HttpServletResponse carrier) {
carrier.addHeader(Span.TRACE_ID_NAME, span.traceIdString());
carrier.addHeader(Span.SPAN_ID_NAME, Span.idToHex(span.getSpanId()));
}
}
static class HttpResponseInjectingTraceFilter extends GenericFilterBean {
private final Tracer tracer;
private final SpanInjector&lt;HttpServletResponse&gt; spanInjector;
public HttpResponseInjectingTraceFilter(Tracer tracer, SpanInjector&lt;HttpServletResponse&gt; spanInjector) {
this.tracer = tracer;
this.spanInjector = spanInjector;
}
@Override
public void doFilter(ServletRequest request, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
HttpServletResponse response = (HttpServletResponse) servletResponse;
Span currentSpan = this.tracer.getCurrentSpan();
this.spanInjector.inject(currentSpan, response);
filterChain.doFilter(request, response);
}
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>And you could register them like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@Bean
SpanInjector&lt;HttpServletResponse&gt; customHttpServletResponseSpanInjector() {
return new CustomHttpServletResponseSpanInjector();
}
@Bean
HttpResponseInjectingTraceFilter responseInjectingTraceFilter(Tracer tracer) {
return new HttpResponseInjectingTraceFilter(tracer, customHttpServletResponseSpanInjector());
}</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_custom_sa_tag_in_zipkin">Custom SA tag in Zipkin</h3>
<div class="paragraph">
<p>Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented.
What you can do is to create a span with the <code>peer.service</code> tag that will contain a value of the service that you want to call.
Below you can see an example of a call to Redis that is wrapped in such a span.</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">org.springframework.cloud.sleuth.Span newSpan = tracer.createSpan("redis");
try {
newSpan.tag("redis.op", "get");
newSpan.tag("lc", "redis");
newSpan.logEvent(org.springframework.cloud.sleuth.Span.CLIENT_SEND);
// call redis service e.g
// return (SomeObj) redisTemplate.opsForHash().get("MYHASH", someObjKey);
} finally {
newSpan.tag("peer.service", "redisService");
newSpan.tag("peer.ipv4", "1.2.3.4");
newSpan.tag("peer.port", "1234");
newSpan.logEvent(org.springframework.cloud.sleuth.Span.CLIENT_RECV);
tracer.close(newSpan);
}</code></pre>
</div>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
Remember not to add both <code>peer.service</code> tag and the <code>SA</code> tag! You have to add only <code>peer.service</code>.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_custom_service_name">Custom service name</h3>
<div class="paragraph">
<p>By default Sleuth assumes that when you send a span to Zipkin, you want the span&#8217;s service name
to be equal to <code>spring.application.name</code> value. That&#8217;s not always the case though. There
are situations in which you want to explicitly provide a different service name for all spans coming
from your application. To achieve that it&#8217;s enough to just pass the following property
to your application to override that value (example for <code>foo</code> service name):</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-yaml" data-lang="yaml">spring.zipkin.service.name: foo</code></pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_host_locator">Host locator</h3>
<div class="paragraph">
<p>In order to define the host that is corresponding to a particular span we need to resolve the host name
and port. The default approach is to take it from server properties. If those for some reason are not set
then we&#8217;re trying to retrieve the host name from the network interfaces.</p>
</div>
<div class="paragraph">
<p>If you have the discovery client enabled and prefer to retrieve the host address from the registered
instance in a service registry then you have to set the property (it&#8217;s applicable for both HTTP and
Stream based span reporting).</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-yaml" data-lang="yaml">spring.zipkin.locator.discovery.enabled: true</code></pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_span_data_as_messages">Span Data as Messages</h2>
<div class="sectionbody">
<div class="paragraph">
<p>You can accumulate and send span data over
<a href="https://cloud.spring.io/spring-cloud-stream">Spring Cloud Stream</a> by
including the <code>spring-cloud-sleuth-stream</code> jar as a dependency, and
adding a Channel Binder implementation
(e.g. <code>spring-cloud-starter-stream-rabbit</code> for RabbitMQ or
<code>spring-cloud-starter-stream-kafka</code> for Kafka). This will
automatically turn your app into a producer of messages with payload
type <code>Spans</code>.</p>
</div>
<div class="sect2">
<h3 id="_zipkin_consumer">Zipkin Consumer</h3>
<div class="paragraph">
<p>There is a special convenience annotation for setting up a message consumer
for the Span data and pushing it into a Zipkin <code>SpanStore</code>. This application</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@SpringBootApplication
@EnableZipkinStreamServer
public class Consumer {
public static void main(String[] args) {
SpringApplication.run(Consumer.class, args);
}
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>will listen for the Span data on whatever transport you provide via a
Spring Cloud Stream <code>Binder</code> (e.g. include
<code>spring-cloud-starter-stream-rabbit</code> for RabbitMQ, and similar
starters exist for Redis and Kafka). If you add the following UI dependency</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-xml" data-lang="xml">&lt;groupId&gt;io.zipkin.java&lt;/groupId&gt;
&lt;artifactId&gt;zipkin-autoconfigure-ui&lt;/artifactId&gt;</code></pre>
</div>
</div>
<div class="paragraph">
<p>Then you&#8217;ll have your app a
<a href="https://github.com/openzipkin/zipkin">Zipkin server</a>, which hosts
the UI and api on port 9411.</p>
</div>
<div class="paragraph">
<p>The default <code>SpanStore</code> is in-memory (good for demos and getting
started quickly). For a more robust solution you can add MySQL and
<code>spring-boot-starter-jdbc</code> to your classpath and enable the JDBC
<code>SpanStore</code> via configuration, e.g.:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-yaml" data-lang="yaml">spring:
rabbitmq:
host: ${RABBIT_HOST:localhost}
datasource:
schema: classpath:/mysql.sql
url: jdbc:mysql://${MYSQL_HOST:localhost}/test
username: root
password: root
# Switch this on to create the schema on startup:
initialize: true
continueOnError: true
sleuth:
enabled: false
zipkin:
storage:
type: mysql</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
The <code>@EnableZipkinStreamServer</code> is also annotated with
<code>@EnableZipkinServer</code> so the process will also expose the standard
Zipkin server endpoints for collecting spans over HTTP, and for
querying in the Zipkin Web UI.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_custom_consumer">Custom Consumer</h3>
<div class="paragraph">
<p>A custom consumer can also easily be implemented using
<code>spring-cloud-sleuth-stream</code> and binding to the <code>SleuthSink</code>. Example:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@EnableBinding(SleuthSink.class)
@SpringBootApplication(exclude = SleuthStreamAutoConfiguration.class)
@MessageEndpoint
public class Consumer {
@ServiceActivator(inputChannel = SleuthSink.INPUT)
public void sink(Spans input) throws Exception {
// ... process spans
}
}</code></pre>
</div>
</div>
<div class="admonitionblock note">
<table>
<tr>
<td class="icon">
<div class="title">Note</div>
</td>
<td class="content">
the sample consumer application above explicitly excludes
<code>SleuthStreamAutoConfiguration</code> so it doesn&#8217;t send messages to itself,
but this is optional (you might actually want to trace requests into
the consumer app).
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_metrics">Metrics</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Currently Spring Cloud Sleuth registers very simple metrics related to spans.
It&#8217;s using the <a href="https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html#production-ready-recording-metrics">Spring Boot&#8217;s metrics support</a>
to calculate the number of accepted and dropped spans. Each time a span gets
sent to Zipkin the number of accepted spans will increase. If there&#8217;s an error then
the number of dropped spans will get increased.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_integrations">Integrations</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_runnable_and_callable">Runnable and Callable</h3>
<div class="paragraph">
<p>If you&#8217;re wrapping your logic in <code>Runnable</code> or <code>Callable</code> it&#8217;s enough to wrap those classes in their Sleuth representative.</p>
</div>
<div class="paragraph">
<p>Example for <code>Runnable</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Runnable runnable = new Runnable() {
@Override
public void run() {
// do some work
}
@Override
public String toString() {
return "spanNameFromToStringMethod";
}
};
// Manual `TraceRunnable` creation with explicit "calculateTax" Span name
Runnable traceRunnable = new TraceRunnable(tracer, spanNamer, runnable, "calculateTax");
// Wrapping `Runnable` with `Tracer`. The Span name will be taken either from the
// `@SpanName` annotation or from `toString` method
Runnable traceRunnableFromTracer = tracer.wrap(runnable);</code></pre>
</div>
</div>
<div class="paragraph">
<p>Example for <code>Callable</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">Callable&lt;String&gt; callable = new Callable&lt;String&gt;() {
@Override
public String call() throws Exception {
return someLogic();
}
@Override
public String toString() {
return "spanNameFromToStringMethod";
}
};
// Manual `TraceCallable` creation with explicit "calculateTax" Span name
Callable&lt;String&gt; traceCallable = new TraceCallable&lt;&gt;(tracer, spanNamer, callable, "calculateTax");
// Wrapping `Callable` with `Tracer`. The Span name will be taken either from the
// `@SpanName` annotation or from `toString` method
Callable&lt;String&gt; traceCallableFromTracer = tracer.wrap(callable);</code></pre>
</div>
</div>
<div class="paragraph">
<p>That way you will ensure that a new Span is created and closed for each execution.</p>
</div>
</div>
<div class="sect2">
<h3 id="_hystrix">Hystrix</h3>
<div class="sect3">
<h4 id="_custom_concurrency_strategy">Custom Concurrency Strategy</h4>
<div class="paragraph">
<p>We&#8217;re registering a custom <a href="https://github.com/Netflix/Hystrix/wiki/Plugins#concurrencystrategy"><code>HystrixConcurrencyStrategy</code></a>
that wraps all <code>Callable</code> instances into their Sleuth representative -
the <code>TraceCallable</code>. The strategy either starts or continues a span depending on the fact whether tracing was already going
on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the <code>spring.sleuth.hystrix.strategy.enabled</code> to <code>false</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="_manual_command_setting">Manual Command setting</h4>
<div class="paragraph">
<p>Assuming that you have the following <code>HystrixCommand</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">HystrixCommand&lt;String&gt; hystrixCommand = new HystrixCommand&lt;String&gt;(setter) {
@Override
protected String run() throws Exception {
return someLogic();
}
};</code></pre>
</div>
</div>
<div class="paragraph">
<p>In order to pass the tracing information you have to wrap the same logic in the Sleuth version of the <code>HystrixCommand</code> which is the
<code>TraceCommand</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">TraceCommand&lt;String&gt; traceCommand = new TraceCommand&lt;String&gt;(tracer, traceKeys, setter) {
@Override
public String doRun() throws Exception {
return someLogic();
}
};</code></pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_rxjava">RxJava</h3>
<div class="paragraph">
<p>We&#8217;re registering a custom <a href="https://github.com/ReactiveX/RxJava/wiki/Plugins#rxjavaschedulershook"><code>RxJavaSchedulersHook</code></a>
that wraps all <code>Action0</code> instances into their Sleuth representative -
the <code>TraceAction</code>. The hook either starts or continues a span depending on the fact whether tracing was already going
on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the <code>spring.sleuth.rxjava.schedulers.hook.enabled</code> to <code>false</code>.</p>
</div>
<div class="paragraph">
<p>You can define a list of regular expressions for thread names, for which you don&#8217;t want a Span to be created. Just provide a comma separated list
of regular expressions in the <code>spring.sleuth.rxjava.schedulers.ignoredthreads</code> property.</p>
</div>
</div>
<div class="sect2">
<h3 id="_http_integration">HTTP integration</h3>
<div class="paragraph">
<p>Features from this section can be disabled by providing the <code>spring.sleuth.web.enabled</code> property with value equal to <code>false</code>.</p>
</div>
<div class="sect3">
<h4 id="_http_filter">HTTP Filter</h4>
<div class="paragraph">
<p>Via the <code>TraceFilter</code> all sampled incoming requests result in creation of a Span. That Span&#8217;s name is <code>http:</code> + the path to which
the request was sent. E.g. if the request was sent to <code>/foo/bar</code> then the name will be <code>http:/foo/bar</code>. You can configure which URIs you would
like to skip via the <code>spring.sleuth.web.skipPattern</code> property. If you have <code>ManagementServerProperties</code> on classpath then
its value of <code>contextPath</code> gets appended to the provided skip pattern.</p>
</div>
</div>
<div class="sect3">
<h4 id="_handlerinterceptor">HandlerInterceptor</h4>
<div class="paragraph">
<p>Since we want the span names to be precise we&#8217;re using a <code>TraceHandlerInterceptor</code> that either wraps an
existing <code>HandlerInterceptor</code> or is added directly to the list of existing <code>HandlerInterceptors</code>. The
<code>TraceHandlerInterceptor</code> adds a special request attribute to the given <code>HttpServletRequest</code>. If the
the <code>TraceFilter</code> doesn&#8217;t see this attribute set it will create a "fallback" span which is an additional
span created on the server side so that the trace is presented properly in the UI. Seeing that most likely
signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.</p>
</div>
</div>
<div class="sect3">
<h4 id="_async_servlet_support">Async Servlet support</h4>
<div class="paragraph">
<p>If your controller returns a <code>Callable</code> or a <code>WebAsyncTask</code> Spring Cloud Sleuth will continue the existing span instead of creating a new one.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_http_client_integration">HTTP client integration</h3>
<div class="sect3">
<h4 id="_synchronous_rest_template">Synchronous Rest Template</h4>
<div class="paragraph">
<p>We&#8217;re injecting a <code>RestTemplate</code> interceptor that ensures that all the tracing information is passed to the requests. Each time a
call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous <code>RestTemplate</code> features
just set <code>spring.sleuth.web.client.enabled</code> to <code>false</code>.</p>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
You have to register <code>RestTemplate</code> as a bean so that the interceptors will get injected.
If you create a <code>RestTemplate</code> instance with a <code>new</code> keyword then the instrumentation WILL NOT work.
</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_asynchronous_rest_template">Asynchronous Rest Template</h4>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<div class="title">Important</div>
</td>
<td class="content">
A traced version of an <code>AsyncRestTemplate</code> bean is registered for you out of the box. If you
have your own bean you have to wrap it in a <code>TraceAsyncRestTemplate</code> representation. The best solution
is to only customize the <code>ClientHttpRequestFactory</code> and / or <code>AsyncClientHttpRequestFactory</code>.
<strong>If you have your own <code>AsyncRestTemplate</code> and you don&#8217;t wrap it your calls WILL NOT GET TRACED</strong>.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the <code>ClientHttpRequestFactory</code>
and the <code>AsyncClientHttpRequestFactory</code> by registering your beans. Remember to use tracing compatible implementations (e.g. don&#8217;t forget to
wrap <code>ThreadPoolTaskScheduler</code> in a <code>TraceAsyncListenableTaskExecutor</code>). Example of custom request factories:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">@EnableAutoConfiguration
@Configuration
public static class TestConfiguration {
@Bean
ClientHttpRequestFactory mySyncClientFactory() {
return new MySyncClientHttpRequestFactory();
}
@Bean
AsyncClientHttpRequestFactory myAsyncClientFactory() {
return new MyAsyncClientHttpRequestFactory();
}
}</code></pre>
</div>
</div>
<div class="paragraph">
<p>To block the <code>AsyncRestTemplate</code> features set <code>spring.sleuth.web.async.client.enabled</code> to <code>false</code>.
To disable creation of the default <code>TraceAsyncClientHttpRequestFactoryWrapper</code> set <code>spring.sleuth.web.async.client.factory.enabled</code>
to <code>false</code>. If you don&#8217;t want to create <code>AsyncRestClient</code> at all set <code>spring.sleuth.web.async.client.template.enabled</code> to <code>false</code>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_feign">Feign</h3>
<div class="paragraph">
<p>By default Spring Cloud Sleuth provides integration with feign via the <code>TraceFeignClientAutoConfiguration</code>. You can disable it entirely
by setting <code>spring.sleuth.feign.enabled</code> to false. If you do so then no Feign related instrumentation will take place.</p>
</div>
<div class="paragraph">
<p>Part of Feign instrumentation is done via a <code>FeignBeanPostProcessor</code>. You can disable it by providing the <code>spring.sleuth.feign.processor.enabled</code> equal to <code>false</code>.
If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation
however will be still there.</p>
</div>
</div>
<div class="sect2">
<h3 id="_asynchronous_communication">Asynchronous communication</h3>
<div class="sect3">
<h4 id="__async_annotated_methods">@Async annotated methods</h4>
<div class="paragraph">
<p>In Spring Cloud Sleuth we&#8217;re instrumenting async related components so that the tracing information is passed between threads.
You can disable this behaviour by setting the value of <code>spring.sleuth.async.enabled</code> to <code>false</code>.</p>
</div>
<div class="paragraph">
<p>If you annotate your method with <code>@Async</code> then we&#8217;ll automatically create a new Span with the following characteristics:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the Span name will be the annotated method name</p>
</li>
<li>
<p>the Span will be tagged with that method&#8217;s class name and the method name too</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="__scheduled_annotated_methods">@Scheduled annotated methods</h4>
<div class="paragraph">
<p>In Spring Cloud Sleuth we&#8217;re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour
by setting the value of <code>spring.sleuth.scheduled.enabled</code> to <code>false</code>.</p>
</div>
<div class="paragraph">
<p>If you annotate your method with <code>@Scheduled</code> then we&#8217;ll automatically create a new Span with the following characteristics:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the Span name will be the annotated method name</p>
</li>
<li>
<p>the Span will be tagged with that method&#8217;s class name and the method name too</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>If you want to skip Span creation for some <code>@Scheduled</code> annotated classes you can set the
<code>spring.sleuth.scheduled.skipPattern</code> with a regular expression that will match the fully qualified name of the
<code>@Scheduled</code> annotated class.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<div class="title">Tip</div>
</td>
<td class="content">
If you are using <code>spring-cloud-sleuth-stream</code> and <code>spring-cloud-netflix-hystrix-stream</code> together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting <code>spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask</code>
</td>
</tr>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_executor_executorservice_and_scheduledexecutorservice">Executor, ExecutorService and ScheduledExecutorService</h4>
<div class="paragraph">
<p>We&#8217;re providing <code>LazyTraceExecutor</code>, <code>TraceableExecutorService</code> and <code>TraceableScheduledExecutorService</code>. Those implementations
are creating Spans each time a new task is submitted, invoked or scheduled.</p>
</div>
<div class="paragraph">
<p>Here you can see an example of how to pass tracing information with <code>TraceableExecutorService</code> when working with <code>CompletableFuture</code>:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code class="language-java" data-lang="java">CompletableFuture&lt;Long&gt; completableFuture = CompletableFuture.supplyAsync(() -&gt; {
// perform some logic
return 1_000_000L;
}, new TraceableExecutorService(executorService,
// 'calculateTax' explicitly names the span - this param is optional
tracer, traceKeys, spanNamer, "calculateTax"));</code></pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_messaging">Messaging</h3>
<div class="paragraph">
<p>Spring Cloud Sleuth integrates with <a href="https://projects.spring.io/spring-integration/">Spring Integration</a>. It creates spans for publish and
subscribe events. To disable Spring Integration instrumentation, set <code>spring.sleuth.integration.enabled</code> to false.</p>
</div>
<div class="paragraph">
<p>Spring Cloud Sleuth up till version 1.0.4 is sending invalid tracing headers when using messaging. Those headers are actually
the same as the ones sent in HTTP (they contain a <code>-</code>) in its name. For the sake of
backwards compatibility in 1.0.4 we&#8217;ve started sending both valid and invalid headers. Please upgrade to 1.0.4 because
in Spring Cloud Sleuth 1.1 we will remove the support for the deprecated headers.</p>
</div>
<div class="paragraph">
<p>Since 1.0.4 you can provide the <code>spring.sleuth.integration.patterns</code> pattern to explicitly
provide the names of channels that you want to include for tracing. By default all channels
are included.</p>
</div>
</div>
<div class="sect2">
<h3 id="_zuul">Zuul</h3>
<div class="paragraph">
<p>We&#8217;re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data).
To disable Zuul support set the <code>spring.sleuth.zuul.enabled</code> property to <code>false</code>.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_running_examples">Running examples</h2>
<div class="sectionbody">
<div class="paragraph">
<p>You can find the running examples deployed in the <a href="https://run.pivotal.io/">Pivotal Web Services</a>. Check them out in the following links:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://docssleuth-zipkin-server.cfapps.io/">Zipkin for apps presented in the samples to the top</a></p>
</li>
<li>
<p><a href="https://docsbrewing-zipkin-web.cfapps.io/">Zipkin for Brewery on PWS</a>, its <a href="https://github.com/spring-cloud-samples/brewery">Github Code</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink