From 2301f52d4e15e2eb9d5f87ae858ed53b12af1c61 Mon Sep 17 00:00:00 2001 From: buildmaster Date: Wed, 25 Jul 2018 09:54:08 +0000 Subject: [PATCH] Sync docs from 2.0.x to gh-pages --- 2.0.x/images/callouts/1.png | Bin 0 -> 329 bytes 2.0.x/images/callouts/2.png | Bin 0 -> 353 bytes 2.0.x/images/callouts/3.png | Bin 0 -> 350 bytes 2.0.x/images/trace-id.png | Bin 53518 -> 86174 bytes 2.0.x/index.html | 2 +- 2.0.x/multi/images/callouts/1.png | Bin 0 -> 329 bytes 2.0.x/multi/images/callouts/2.png | Bin 0 -> 353 bytes 2.0.x/multi/images/callouts/3.png | Bin 0 -> 350 bytes 2.0.x/multi/multi__additional_resources.html | 3 +- 2.0.x/multi/multi__current_span.html | 19 + .../multi__current_tracing_component.html | 7 + 2.0.x/multi/multi__customizations.html | 183 +- 2.0.x/multi/multi__features.html | 140 +- 2.0.x/multi/multi__instrumentation.html | 20 +- 2.0.x/multi/multi__integrations.html | 179 +- 2.0.x/multi/multi__introduction.html | 233 +- ...ulti__managing_spans_with_annotations.html | 58 +- 2.0.x/multi/multi__naming_spans.html | 17 +- 2.0.x/multi/multi__propagation.html | 92 + 2.0.x/multi/multi__running_examples.html | 6 +- 2.0.x/multi/multi__sampling.html | 60 +- .../multi/multi__sending_spans_to_zipkin.html | 31 +- 2.0.x/multi/multi__span_lifecycle.html | 72 +- .../multi__zipkin_stream_span_consumer.html | 5 + 2.0.x/multi/multi_pr01.html | 2 +- 2.0.x/multi/multi_spring-cloud-sleuth.html | 2 +- 2.0.x/single/images/callouts/1.png | Bin 0 -> 329 bytes 2.0.x/single/images/callouts/2.png | Bin 0 -> 353 bytes 2.0.x/single/images/callouts/3.png | Bin 0 -> 350 bytes 2.0.x/single/spring-cloud-sleuth.html | 1129 ++++++---- 2.0.x/spring-cloud-sleuth.xml | 2003 ++++++++++------- 31 files changed, 2428 insertions(+), 1835 deletions(-) create mode 100644 2.0.x/images/callouts/1.png create mode 100644 2.0.x/images/callouts/2.png create mode 100644 2.0.x/images/callouts/3.png create mode 100644 2.0.x/multi/images/callouts/1.png create mode 100644 2.0.x/multi/images/callouts/2.png create mode 100644 2.0.x/multi/images/callouts/3.png create mode 100644 2.0.x/multi/multi__current_span.html create mode 100644 2.0.x/multi/multi__current_tracing_component.html create mode 100644 2.0.x/multi/multi__propagation.html create mode 100644 2.0.x/multi/multi__zipkin_stream_span_consumer.html create mode 100644 2.0.x/single/images/callouts/1.png create mode 100644 2.0.x/single/images/callouts/2.png create mode 100644 2.0.x/single/images/callouts/3.png diff --git a/2.0.x/images/callouts/1.png b/2.0.x/images/callouts/1.png new file mode 100644 index 0000000000000000000000000000000000000000..7d473430b7bec514f7de12f5769fe7c5859e8c5d GIT binary patch literal 329 zcmeAS@N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQC}X^4DKU-G|w_t}fLBA)Suv#nrW z!^h2QnY_`l!BOq-UXEX{m2up>JTQkX)2m zTvF+fTUlI^nXH#utd~++ke^qgmzgTe~DWM4ffP81J literal 0 HcmV?d00001 diff --git a/2.0.x/images/callouts/2.png b/2.0.x/images/callouts/2.png new file mode 100644 index 0000000000000000000000000000000000000000..5d09341b2f6d2ea2d1d5dad5d980f14b4b05dfd2 GIT binary patch literal 353 zcmeAS@N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQxaY7e*=hH)_rZeB4|imU1$R#1`!P>&$poQl;nzm}mD5ZFopaX|GsS%q*{P~< z;WtmO%lhToBL0i}yfkaOt?EN=nkLNGuU`ywhI5H)L`iUdT1k0gQ7VIjhO(w-Zen_> zZ(@38a<+nro{^q~f~BRtfrY+-p+a&|W^qZSLvCepNoKNMYO!8QX+eHoiC%Jk?!;Y+ zJAlS%fsM;d&r2*R1)67JkeZlkYGj#gX_9E3W@4U_nw*@Ln38B@k(iuhnUeN2eF0kK0(Y1u|9Rc(19XFPiEBhjaDG}zd16s2gM)^$re|(qda7?? zdS-IAf{C7yo`r&?rM`iMzJZ}aa#3b+Nu@(>WpPPnvR-PjUP@^}eqM=Qa(?c_U5Yz^ z#%Y0#%S_KpEGY$=XJL?(l#*ybuErX#^g`ttQfwnF(}Er5mI~x}>|LyGy#eyBjv|!t1%eU!NYr z_S|)@wPuVl=A1#Y(jthjabH6q5JWLiL3s!S#ti~_HvbY1T=^O_odJHq=}CwPLLQ&~ zrZwh7gDbCWM8DWWAaBr~{yl^IO2P&g;T^=Jgy2^Y@Lr-ZTEfY^gh1Xw!~{Pny3Fn` zxOzxVJ$9c=ts5om*9)Qw1fsm{XgZQmGdGnp`+%B#kzHh^CZRGx#ZOC?n_XEMlR=q{ zE3q$uLL&GXHAv84$$0Zh#DEHwit7D|D<7fj&syh0&b6dogGKv8UU**=ftSSpe?L0( zGk%c&_asK^beX<;t|FDOX;Am7oviF%uN|M$DR|WLe{U4bev6IGA(}H*6dUJaar#Z# z84hvBhhhmnTbJ^H;@?a9xRa$-uv%u)F(>1~N8cGzo+z`hM`t2_Q)_|N-=(DlP$&`P zi++wxryfzMMRhMyB@+FM_V31G<(0bC6zwnL{3#Uz{v_ag4mU2c@|tQ)iTuh#ICxgv zp>-!?h%L(e`oBd8ZjgME`M(8G;cz-C{Cmt3>jZDT8qv&VOU?NfqK*e&=d>*nfB8V@ zs>yGm`ZYS!JgR&RP4C*aU4T67Zv9(7r{^V#IXm6z+ErXS(V0;lZDemeb3P2+vzRye6I~ zXULzyvQJ<~lKB5t6?iv8&=7={b2ZkJgP~UB?}&#r(saRGI9K^|>n=3ATEz4B@ar^* z0I0>n4Gb?AvLGXaLI07t$>CHK5gW~V*sIhvoh?PZNcYF+k!xpyj&%L{vhqsW@<+*O zIV6I)qUNJb`HE?ig}~O5v&~9ke2xYMft1L!|4y;PXLr%`-2yk_J5uJb_yB2Q&JFVF zNu>AWol(D>LgJ+AmzOqRXKPL=sJWy4`i)i(HUcszIXLUe=?(B6QVEm_=bvso&(gZi z8TYG-r8XP1I->^)w>X%)bC}(1ZDAl^h#(}FySm*L=Bp|>VKn+h;Csq{CWtH`hYkr( zxenpKvi|M4XPmpMLGX3H0e`#(CN3(9aR(>Qz|eTN2E(_ijw~)QG2SPtte5xmyie%1 z+$vYy zv*~dvq_H$TocadNj%V&cNVzyUBsQCtkj$4i)ot)+jR_UvE~}yQaY^T8jm2z2oD}!c z)y;ja_Zw}mYtrD1;}xtmP8$OX%U{kVkq7RyY`>}!y+4!>&W;9uO(!LC)%;+Z=ZnuR zFf6SZVxe7gg9UftTCBKHICmcc<=ix9@u<0U*CM<}^{;7u_wNj3YRmMyFJUH%gFwU49%AdYa3YBzR@Se0qSHx!Hod^s_MTSnkcrrdV6w?(6@YgMwd9 z0MBg<_8!u*_@Mq|N79z=V}-T>sQok?mr~z9S1ZhA@&$j|;S2BXfDWpcc;*q=dRXoU z?7x}O(w$rE8x5Y6QjfG@j=Jf`JQ-0x%4B2v5M3ufCO>nzRjl{P`k(|)9OJx@wkYrq zYOk}o%Gv3%?!c`P-Gqi_zfw>1lNHIx>gX)LOO~!tR$DFWLw@;k+rjv6q?gOQ!>_Mj zzlNh;3cA`|^{B%fZ92brJBJ!i_vBM`DTpB!d(+iZynBBtBxR6^cz8VscWW0l1B$|6 zJnyyhOD(M}O@$JF)BeH4D{4lxA!YoQ#!Age==LLG(Jj@d8_E89oe}cfKRiol?KeI$ zZ)cm5#4~Ju4+JR>yfT{!Ch1gepP-;uM=~DKR7l}ull4c(pOo+KVVu~U47N7WXX0Z# zLU6PPsmd+8_KtLn&xSiViDVMQkK}xMr`GMkxn4b98P%Grzj70Jd6LC1CUsZ{@b$mz^Hy!uyueG_nxuJrC%7Zs zyQ@9$nbAMu`pW5>SK6{e2CmI8&9S_?Vmu5sYZD!$0?f2ce}ovif668nF%a{U*7n|T=!qiTPiKQm{a>bDs4zZ$|r;S>E+2s+zIQiwFx*k zxQPD2#NVuCNVwSpl9KTC*49d^E8FG^b)F~Tlr*=`lf40(9t_P4c%FtGhwoV2DXed0 z=3o1zH6lW#k>pNcO#Vi1WJ;)dfj1+*WQ4(ch6L_%w;R0e#-va2>2{u?cno`FX^${}>Zy$2eecM(jkOY(e`}@OwQv~yw_yCwwpu+M%oe1dVoPB8S^ud_ zh~?hR{fhybLg16lCEB79-_}AbxSPDGC(PMfMrIzq_0Lqe;)JPPbPSDPe1)l>Tp7zE zog|}Sqwz(FkO!%uVpFQC<>RM<2^|B20rir>5gDgb%7(iUQIJkvu5V_UpCoNG+(YWRQ5j3{g1)@Y~p(OZ&ebN?! z;7$vHJPs{vFH})HJCY?^ysJ|nDLq4Wu=5~g*m!Bh2cOS1 zsp?I0yR4k*@eclN1WsSvhWU}JRN4z(u*Z*I8iiq(D7~2N-hKTFfD*pZt%8T-Ea z$$Ck7V4z3><2*^j{O$P+$Ks9In;Zcro5ShU${jXQE`^tJo6@PgzKIf2T7+D%Ut`U; zP%*vm>4*L2#iieG-eS&)Uak+dfW1!!M}fn;Nis= zeXwgjzO4iPxqq*#U#2gNfZrDQ?Q^h&I_K5g-K3nVkytlht`n`=Sh0}#gC`b+F_QI+ z=Haa3(=jVnr%nBNw~d2Wa8dItZa6`3f5PUM^L|8h0sLKc2{}7EJaS3bIRO+?>=k1n z^wtt$=5{mUzP*?9dZsm1T_uC^H7}fWOMi`0kYiW3VEOG{ja=d{H1`XV18g;e$A~vM zKjQ2ZyqnuANP}aPn#Du}kdIco{S;pDqqT8sTSkSWKL*hx$o;08K*gytpET}_&`wuQ z97)q+_UGk)Dp^2!%~VesZS`YGFhOrE7Mw+PbfTMrvNYsu@Rqe%}j z3=tKLqII7Rw0;$`Tej25TI5rOYKiI${grBcGr43`O(iscOERsR#Z}2+@ zf`ai@{NNULj|cH}lGu4H1Jl@#FKl-cb1MTWF=QSDze6mn%p?z3li*t-;j(KVTanyIN#oMJ|G+{yU3= z8>IfrYH0`4r3ku%JKODPLMTtEA9%QfC8dH6DRo|Hed-ON^bF-fL5e^*OCIEQ$!O-FjKH+*Z7Q z<_}IoyVY*_m>ZV8-OrH$o@kjV%>;OH4b>tRbPDR=MC5RHv`SZ`hM}k3D`8Sb6I8NThH` zBlO9~lo}ynn;O@!ZcbFUWNUSU%?!E2_Ssa_CWFBw%wEdQw$zR|{}4G3OFfkkmTpZg zX_o6vwGp&3zO=Hsd$2vRGA0q>by9D>tF?}i979yFx|H>-1JOW|bBahoZ;n{~;#Hx( z?6PFs@C{Min{C*)qbzwq#2eJyxU;Sm5O&Pqdo9Me>4Sm(9eKq^Yvj93< zW;$3>{m+H_1EwdEpVh9}J=PtIpbpHm_js!_*QveMB+|mRI|k@b`Wo+LJAo@dwDOJj z-zwnu`Y$ZE{H(d@Zwygme*uX9HJp>T6QV=<9-S{^*l@V|l z(wE&fq+tMFd&R$0{ioIi0*jV~UUznxJXc6-xiejjCEV2ofO-f(%GZo=u+CRyu_&h>FY?kJVT+%GkG5`BtCR$1F|EtG%&8HI>XgTS6kl1q()gpvMb5 zOs3p>()nK(1aT-irwKoNA2W+Cra-f$dObBO{l^t^fb8_D78_OnNA6f9xv~@HRUtbc zRaA4k9s?sQT{S%~S@isX#)GFmvp!AX<4yH^O)}ONvNrj3y#CLhxVlE^agp#E5%uEJ zooGVUN{;z}N@*vI$Hqo~S&wPi-*N*PoXnd>W$M|g0~_94yXluv%ay(u+6kH{YzVV(|2cYk*xW{YO8RT8k_mRGzp=8(wMDOKFcZJTaC(FFrTJhAb+5O#SNF2!u{+px>2&i?P}c8kj2Z1e zs<+Af&h@s;Q2=P}dV`J4)~L=4eTt1J5pr{0QpaZU(S_$0`HIf(7|HXNecC4y=R+mZ zAHEumo6EgHUNSRbRr9*`pib5@kv5a(VpA&QaAx@adjSuf`pTk*?4eAS;k64z1l)v; zP1W62d#Gk_^a3^x-jC?m5aGwmo)DNWeM8g!$z*<;UCEbCi-S9BkE6VM zyl$uTUP~2(SX--TmV&~Xl&CLh2MU8@3@r@W%%2gRI{szhsyYa(O~?-28KVV=iMh?o3M0PGkw?U!kIB*Y?iO)>OhnKtMR&lTY2%Qj(xi zNi2EDL26%pMho4Q(j$Q;+>B3*y`s3vAhKbjz4obo_+>*)97Y>*c|6~7II4SP!waR# z8S-VYTE7GL`ktFR+Ef9voZAJB`*HrO_Y{`u$(oACW$WuUdcUWuW>uWMy+I9+T~qg^ zy3gu;*mr!^l2M~qZ*Hc@Ja`;&0wNBGNlrDQ9)L)}k~_7ZYNORg6G3?BXu3J|4ZtQ6 zlf&?wx)LU)v7Wx6jupaav7l4k7h#V5nOu(=2L6>etQrj&FHKIHG;DI zw9%ThntrAaH+}?!^LjaCG)#4FX4-aTR47fy;b=%qX*u<6qxjpD4MQ?tl-RM9!TvuE zhm^uyW?WaUUz+b*Gs_JJRn7@--J&erz178{s~bV4qF3UaD$(lr$k@?4;S=5{9J6~J zO!a{dXUR$Hr&{hSekZ?t!tilx&BP0PXPqsJ_LM4Omc2zt>ST&9QKLX+_{DUC2|In$ z!ka{@PWt;ROO*geZL9XKk>H2K@HP30KFzx${<6NHg{>a(;oqiLaKHna5-NY41<>z_ zycDt>yyudisO7>9^RL*7U;^^Tytr7$Tz^Ea`CAs#Y0RqGP@qehZjb$N4f*{8XgyEA zQli|3aqvpIg+_8eQHO>}noRYjT6z~!mp>|tlNb*XO)S+gRAdwzj|6#@S1@0YhEfPz zJ*f>jBa9IjM=2a%E<-z)ZCM@j*3l6B<-u{Uq-HJ&Yc`{=G;6E@pusvSK z`ynsSMnHx{DK^#X z35|K^#myxVucCK-un%bne=5b3@FqCJU1$!F2SKc7$t zMng9NWJ83c`RFqdlAUXhH+P6 zhEe@8%VavwPbxKL42$yC6%!~tR%8@*xBJ9q?k;;Wh(Ik}8=pbhw@f}y=LJgY*$M@) zMN|-1^K0Ee=J^QNC6d!n?u3zn!SAS)sNm>K>XEW2z&5*cw3t1$MKnW}&Cbc}&g>Z3 znCPZFCo*gmoS>y;9iB%GT(v#x6fmg={Vm3KnC=>AOTxF_Z>*PUOKj!G{k* z6>Ky6`-wW`W-*OLGs+^ARXPD#AUTMrsEHsSx_sSKWMG^f9iKQDC}v+;TD=9RpO{3g zn7BL-jEMKggWJKI%QZ&IU}C^7D|h&QI`Ia?sjBQxgcPdt5LD%Hj;OQ+AJEg(%LW*W zEN{%^_vr8E|5@JN51$=Rg5jQ0{^doP#9AQ>OyYJ9bJECxaK5}l|B$a(uG{UZaBYZ5 zg6s0rf@XjeMggh_l8~)Zasis11v|f$>PK(}0J2|&iPS1vV}paCH?7@ij|;Lg9=lX$Louzw_Cv4O>e#2&M0 zu=|MCefPGjuP+Y7fYB^n^@kp+mIk4tm7Nu!xxT7s_VRJ27}W30|H3iN`4kzT*XC$P z(!WAt4O>rAWwZw=v>W}k+pA4jagp$6dqHX|2j7^4uW343Y8jQ5GqR4BDr=LxKd~** zLD8|>KKHyrTj%qzBvvqti9_ zyf__pZ&j78U}kTQ$`#hrvp_KXlz#ch?#Jy|Z0W^l!Z6lrARp zI5(@&yL0E^CmcN3qk+&!sIr^24#;8SBm#Qc7lOtYi-Zm!|Iy~{#_P204>p6i&Mjc! zxM9CVA?}Z1B>=W|D_TD3YU?OOGMZgQ>w;R_M1d!oq_)m+Hy~>M_=z47n=Mt={gCKX zbZky@N-fB> zs_=e9B?N2acga>k$IDUH*emv_U=Un$P24>Q${9b7#%C!O#TUG3hK2cN6uQzBrnCBPkX37I(DP{Igrc$ zsK4|AzwWIae)2-^P@m51v{QD|kW3)O!6EXk-(d=*1C(zlM5f@W6v&ZpB945@qLMf~ zzw$xpNB-uJ`P{;uc2yEzvs#2yWz#C*Ye2AhEnaVK!hV|nCtxqADkQpF1~V(8prR@mfQ{s)Du{zK(nd^d?2Ce;(ixxP zzFHNz93CkH8(Zx|9djI3hl1#)6sAC+xXJM55l3l(&U1Kf5tVr}x6DS~d`yhDuUo=? zQoFd^*m^uF56~7qIY8wZk@9ZcJWoSVM9_<6@#`SHI|@nd*73)=;>TL<^@W3>k>+4& zK>ARU1QZQ$KW4~-vQJ;?T}`Ru7BUcg+OGHq zbHv)KK@ic^NsEk12+zPSUkxY;CMZx&)CIT!XbtlPuJRMT$qfzom9yVxEWK5mA)U*u zT@qSux+LB&tlu;3KDShJ*pi%n3sff#M>b!8ZB`XZa44jYD8O@j?DQLg+g`Jk+pT#O z#7}UeTRSl@{GnbnL6iI3cxIRWJhNr+feXO&*0mZa6ox%|)pVxt4R{~fuG)*{qc8OI zfjN;+yhD(T#byA$@YfZbdrdJ2JKY7JsBu}2`dT&jUIGbl|HFeE`se41nu1QsT<%_) zeWNoadRDqc$ZAeX8s{4euJw_F2P6K(Yy7%9`>(KZ3K$uyKFWz-<#cup1ql&Fw0>k1 z1X{&XU=4Mri56fi5{nP7*kz36t$$}un7m5jy^fk>bZ!$jf@(P69!tG}rulh)=E1Ay zx8dacQd{^F`eu{O=;xQ*T06R5a?C01#ra|POhK>vr+a)Oa_C4o47dSY%Puzl{P3Y@ zZ0R;G*}Iqmr0YCukFc_+SHo8XJ{JU;_Vh?UKe`#U#(xOT*p46mYr-O#SdJnj=hAfa z6Lqi2zFy6;>#oFGO4tZ@QOgU+7T=a54_y9CP4K4z__5NmqWOoTcC5^jK5bcT06gVz z@D~{X1)v&t%r`8nRRL3WH5|YDqjL5l?Xewlq{ib2BMom^5|Zndea$nA7K_bQm@(#P5f!+kz^PJC8dwMpWFOY))l0Qo`V=z@p3 z4nWU#&%CokfX(02^ z>zk#MGm;0@(VFS%S~@M=mXB5dtk!u+wj@ehzA)ca*kh3fICqFlB3I?kUJm``(OJrbtWJ3KlktkO21u5j8Ks5qakKGL#?3x~JBPe*iTizpl7jlWbqFwWm%` zN0hgu3)v&z#Lxivtam(?`26DIjI?kK-v4?5ek`ARYi2jQls~gLdkb;&aRvGct#es_xIbedUsb)4`0iEp9}txw?gq! ze{A+&OVvLz>+3XPR=r zY*z@yOHMT#NoDEQGbHMj7&GESk<8v`VVAboGH6hG13>`GQ%IcY&ENHfgWFb!3n#yy z#%M#*Hd_16kNs%aS|NC*eeguSa;vnMt?kk0F&m%IuHVqi zVU%b}y-=hpdM8dn#`|rLog`4~k0{8!RdrOc;&ii~Nh{JhYaZ)D;eLLowikWghcK=x zSfG;D*HX+1e`2W)4>;s?^%)hmBR8s8>~3cXe`t81{zClRHd`Sg;;idi0=ZJB*>+!g znC$CbF7JTO$k@1|W@|Iu@3F~#P_MAtV!@=~wgNFJ$Z96)E9eIHu*~5Hf>*$`JNqk$ z*A_jV@cJ+I!`$ZZ-Km3upN}*20EU;FbaoGTPy{Y`Tk7o}J40NqGNV|`=f9O6qdiH* zl^>RvP8POIW-9H2Dp(KS%t(XW1z=)#gOEbVfwO(yx7rzg&nxSYw0f1Zb+>t?9YHyg zWNDG>*hrF<;z;pqjgcJ|_^jAIEh{ZT_Xt0m)q9-Lqn{hI9n(pE#CufF{Q>^heg zT3tKs-XO(muCA^b67#V5HhZb(H6N|-e%flvd_2qgpg!p#)U8EIn4d&z-js3mcdFl# zU#UbrR*A?m!?Kwin-UIx(J45JIfayyI84EPSfRfLH|H`Xr;`~`Oro!)Z|FI~Ag$@c^K1gotNhW!G^gqqOL~U(UU!U<-iyD)$_$63 zVPmI{Y{GCU=At@s4KjcF^eJdm+|YUs%8UPo`nKLG7yI|-`lbVv-t*nmQ%-}mJ1^0$ z1=7X0Bf8auR`tiz2k+Z-E>af;r5xKAKlh z>0bOn(;waU%Bp>KyTAU-DIJRTIxgSjQA45O*W~*Tbbc#D=k2|X4y`l23*S^W5k3J> zkrdh0PlQN0QJ5UNun41mt=s+_T__zSvfJxsLEw{L`Q}YoaM3#>$1Bxo)Oi!%ekNx& zozJ0!6VKZ%uW=?}<*gN+akjJ1`{d&`c+PJ{$9kx~c(|+$xJZ#h2Bn;xTI)sKkdtFs zP4TmB2$SRXEj%o&W+Wz+yIOL%eSG_y#gK#7!rO=l2zIBeH^a~=zR0&vH1VUIeTB7m z?m&0ZZ+0|(wXh8AjFI&UP&Mq8lp&^PPkbA_nFcO6To!jD$2H%}>*_ zK0h>zTj-n6Fsaz$&~in*SRX`XQ(J8-!Ss1J));@ zhEJLgNmVn}Gb>CDtB3@pjE94^k1kt1^9z{{vNi{r>9<9^k7M>#)2mJu@If_~x!BPRZQ!F>N(YA{6mhhRkO-nJJ` zxx@S|n-ZoFV4HK(%LCw#Kc8kv@?mfQrJ^vo+o*| zQ{x*pUcyrsnf#blRBZAWgWBg;W+|cg@;0x_Y7@vHPrXVY~ckFa7OS z$o$l9WeI=&|F#xzG2T=nnWOZQ=s3IR>fZ)Y+LM^kiQ zdc+@^8z_rfEV4sX!p2I$O$`Gy7@*IFLa?s5Fv^j`LJzqHaqx#wxP zAazrrOL}*y)&T=_qGWg*=sI6@QS#F+u5GRi7L2LrSZ;MxJDxFCRCW!G#2)`eY4A`# z0j287L$;Ry6+g9X1nc#HBE7DWZV`Qj)5B(=_T^hhqkogMTb~quLEYJq3M}z?PUje? z*F`6!2skOj33riqh=}yP5#X`)t1u7lrue|AjJxu-CQ~{`{b+ZWYvWTySRElP;`PgU zaS38(?!ENekN-L=m^=fTJ3)1o!O@|0J5eTDn#F7at%y8z!=9Ycc^3s(s(%0Z$L8DV*Xu@?PgK*fGR96c7(2s--+Wa%TQr#C`mB*Z)w(bi|7nfSV2FD-lhXgXdi!^?R|!KX zo9R^IK&1hCCkp6qn7untJuvFA_QKv9Ca~Bm?G-ATx zVn8vaM`~;5kgcD%2vpMlNC7IIH-@@-fVbw-p)^i{X{Yq#z+k`b5wr+2f5-t~KRH;x zS8{bnzX4TQ@g^dF?usK9^I^ z+2_Ro?FNBNxFQqT5*$GD3@@jwj}Nn4_wVgtKEs7lkdju7uk04#w5u}QO_9uD(%qq4 znN^vB7gGoz12Hx(K7jA~i)N76Ng>qKD+sMw(EuQakd0VM&v*aY5l|oo#_yw|Z6$D5 zU-d~$MJA@_{e27oR9nx`RN~QCg-}<%6VMOqHl72x zUSN3G`rk5MQ1$xADC^?7jSlKNsHrIJL;UQ28^<7?oN`kn*i1S+K-+L}XI)v2`i^5_ zO-4~}HKxbJByYNOi}TcT092|!ghZVH2!WPUCcWdm{cC-Kt+og>Ot0sFLb^VC{a@Q> z?z96JZQ06Q?;i{aw^PS*`PGs2t;L&2LDTH=!xy3rkz$uc9_7{>QPCNn5=+L8l=BpO zF;1F6rzqx8om8+zJ#*4CLE+@?`GM~+0nSfaL||+5mxnQr%fqGMMooJ@K082rr?afpmmqG9_jh=9@!Z7Wt-sq0?&=b;`{vi339*( zaBxf;xJ?PqFr~=uM@{P-aX;$GH~8`p|F_w(?Hl@ufr#|3$zNi{9Yy70ngIq@p3ieY z9Wt0VzXDAqPAVOb@wK%WI5>Gi)nw$I`sSMVZVjc!Yd{_cJN)k9lJU|>$z(hYBRE5L zA`cgY{;R@*dPx3^aY?9RBw`{CEAu?B_yU5&A%&6+@_5Vb61PW#aO+;qLYk|qf(z?KZ7vj z<+nG4W5z~#u^fF*PuI-}o`v^nvgELF$$=^kf|C4V3S%f}Pr~|bm&^MO`nuRFgU?XfxosH&u@+5JuUI8J~Z|Z=R84+Qf zG#Mp>5@!T-kjt57b!d~*Q^0(#!pK$3>hk+04SyUK64Jje!s}Q?`9)d&6EohdS#MoN zT0b@F=_=LkV}I`m#$&OMi~TGLV5q1!SFT!wY}wN!yAaR-u6vkq^@P4jPHCdMJ;j}z zjDj|}ra3FT&g9`PabUjwrIQ9@7qysCM>Sbhw)-z|9w!#`=1~oguKz z7}ht`xyRJR8lhZv!zk7U2IuvaKT!TP6U|c?h*FF4zm!{ipVJAWW336pjD}adgg~kt z{RYD$DAfJc%a=9~d{F96V#N(B5YPTG+>`MV2F8c7NEuHhxI*YZXB7xMB_nIp*;JHJ z=>8#~<;PFpyxtRKhK3f+_E9D+LOQ1aCT+q@F3&*TO)g2f0QMaN0|Puq-q?n{C01hsn>h(dA>zkmPcl8}&qdGqGYz`($IOjeHSN}gzKZ7o=^?cOA0X{f; zu&<8^%q6H7C>PB>ih0SAYwx@!7VNw`mdCo5ESR{2VPIhypDg?`)-pI42KFBt8wW>> zG)Nwp)t|b$)!kk33ZucEo}T5=Yzg0hfas*8eK-#b3yXWBj~`na=I7@h3toPci5qaK zgN1`@DjM4%2Qu`|Y*oSP;UQfZNna8uXg9iFu}zm7hOkjn7ffC3O~pepQ^wEavLrel zSWHK=NdJzsOGqzxV3(<6zLr^-ceuYk85<>8ZN}#JYOr7H3fp@A>1$0{S=r?9u*^iJ z2y$#d0Q}|UC1r3!QOM)a^fb1yvGMLyNd)++ot>SmoScGh;lUXQXJ=>iZmS(ZXQT2N zJhs7xGW|u%dI|$xa?LmE7CC^R8&;lt`4P+78`Bt?4H{wq9m>VOyLjX)7l|)5d74fY6OfaWgV3lIoRI`pVS#L#EdBr;GXxO} zV{=egE~eyJd!6In=0u6MHkTX^qvP&)y3yb-2qsmmS@DzBhT^u?9Z?e)7PbI9YVm~w z0k=~YShcFUy2M%_#c!=fHw0XkK(mSb^s$|NC8~+=IW*@WadI{`Hrbr92VDz`26J#C zm>Ul_i_p>Z8~_re&o&1)^bAirsx0R)_4W0OfUj{l?N19w)O#g_3I&P_V%E?rL^!V8 zjbw^U4Gj-3Js)RfWu0(3T3lGZG_?m@#;gk0+uK_@m2WVLROIiD1M~Ce&&}%p{0YrB zkzfqGv!lFk_?4R4>5hEkWp>yG;~F;N0J&@M-}-Cr?J-Vba-WPCjP z{_Vvc8`vHq`IPprE+!jX`wN7$C&2h|nwpwU{Xg`Dffv~Cj(16!dfDY!S@aV^Bw*lY11Tq7JH&dEx(3+l>*MVJ)RdW z9w&ja@ns@kX=kDU3)t{G;6w4knK!n;dL9f7N={A2bKhdoY5X-bHjdvaWrwu1x3__giKQY-o`9EO|UA5V{mi;a!&d=rfB#kR$m!p2)lx}Q1TB6y_yrd5yqqPe2G)ic=p zq5Qz8ElfWCOmYc|;9X3=$cHE@DZ1RE?ViBz;p%5SW!TWH>3 z+RszWZ-ua$PqM@kqHTl$caEpm2D;D%C>2vRnp!tWD;trsRXaI&ZsJUO)F4PuQx`ux zu!kr=ezcNCQlNN5b{7$`B7e`Yw+EZV>G*@SKTglo&@U{R;Ar4Jx1)TmM0+ba2+hW} zpsT9+aq~vEJ$PRmdLV#wU!?--?E>ZiWreri5C~F{44F-Sx+3VB*DYhv7uEqpNEO zw(i%jUuF*v4~8rDaw~0Lo12@>`r~Np?(gq)0n?2C{{6cVkQt__&1-UsODqixVm4M+ z!E|E`H;5;G+M<<7jQaj7Yg}uezNfh835VfD)RfS7Bl&@;GkXR$#+Brt5>zMh82tdG zk*yo7?U4d@@ML6COvmHkZK%v9l*njvbKv1zzpZ^}Wkb_rcm0CD&`7X_gUe}e=+qhu zv*arFMCiMwm@4+dEjYug$Q zMVYyQf8{6Qf!YI_BtYeOuSS#0*?0CLQ6-QK836$wL~bo5bfKw6cV`#{g_}YWDnbzv z5x4X0^xK1)C5M(f5UK94=(QTu)r8x$61;x0vJ@6gn3Yvm4*}0S-Tcr;fQE(!5`(C; zv~qW6=LGB!U$~p0E9}WSRA(MZc1l0P)qS;KAeA?yV=hOnOIW*oFdU5kU&r_q(#PCI6lZ#%lqlTnveei(~pq+;?#_bTCygrGJMi_)C+=l)qS`wGCrQ-_)=( zcMBe}ytgGXoxsduITI{&wkf|fg+*^|E!@bI@~g^>+58s%3g5KgXfCoH9%*?)GgUu8 zAlR?b8*k*z8$>dfVIrD`6AHb|%mfZM^a3_$aCYp3!WhM2N)wPZ?ST3;{cJeQ4f(6@ zA449DNsXqI6W#(ub;?)o|TMp}~XJ zhF9TLjB-#!*(ZWda~zVBlhg1a@b&QUa0I{`dU|?E-)eg<++vybW8b1dfO)` z;5j}UU*D>L226M`Ir9al6Hl0p>+GAO%``xbcLd*^+$ymF2^W}sR6smWAka~$KHGA1MI}`+zecQupEgcLgZq@FqkMT{JH|Bd(B<(Uf`dflP-kkt0 zF7iU%iLj;5)jAR^wre9bDmhjfG`Bf;I0=^7Ws(TwDUadR%<)qoAG+jjIiEE0FOhoVZySNsEntn=}F!FNXZ36 z|2_?oy1ax7e%^BOpw3z8&if$JDAL$PAAF}6{kv*C3ybey+yq>>_IwZkg0rsWmSEzck1E%l5(4^XX~OwWbB8_>G{Zp1 z3_VA$E=rVXyU5kMD>_ID)`)hlp~jL%DR<{3@~A))VqCAKWl=%=UoSv>4(j>LpHr{o z#cl{l;Y^X%8A66xl&_y2zkK!TQ_fi9E1tMCtp5$>6S}ZK4=i9_x&OOQMF!y9s&m{J zwMVj5T#$=IhmN+sa4__umE4rH-NOI@J*&)5?XFhs)wg0wPyZW<(rw^?qJbfz;0Z8m zw^BC9bxualo})NC(IYsxF5@LA_j++s&57`9k+-jg5^CZQeTi`k$-J zCJ47}Q7@l(eTqPTv!>{A%6A;JIp@RMlD+BJNq~(E#;MnAZ5+m}xv}rDAVqV|l#Z@X zerkNjeIuYwj!QBGilqrP)>0 z)^=XHZ8%kiZ$=*3Rj4jS}{YO@9;19ugQ94)wkSouOTKk1*GJ965Ls zd7rODMht{1tAqrVS>CoW_|}G;hJENC7$jk?v5zL^!WfjauqvSaRVbw{I2k82Lbc(L z$r0mVFFkb>k+e5uD;*Q%b9HvIoo=6Ozi_QFwYBS(JnZPncbvEz#`lmUr=}4qHfF4| ztm9nxr3s;`iZyb+*2jviTJ?Aabs&Z1nZTPjJh{bP0IZ=jJ$~(T37o14(;Bk&q zR#vStWtQ&j)a%QAtJ=f>nV6V>Kopqm3ssxnstZOy4D-Rq|vCxr&yCN z?N4C#e%ADQE)4$ah`qk}^IZnxp7&X)-h(j0DR%xjG>2N5ju4q`DdtYdAZ@ zncp}R_mLQhX!>#XsF~V3rE?~F>5~WPRs#Lq;eq@-mE56A_VVmfk(#upCSviuYurO# zab3Y8#iG-JFy!ObQ1VW-)amU#ZE2+iJOmLD5hj;KE45UF*4D%INGu@%;- z!9YSKLt_^ocSDqZ@|r%sLvt1^6X~ij{voTP9x@Y6zP!vna($CS6l+E8nY_P%A82nJ z4uM=+KNilk4y8^mndaECt#D`W+&Ozp=5L5v#S2tg(G?pcb@K1@s-i6iWi-%K(W)Q? zFcX@1cm+mI`Q_#1-CsLrj;580ymW9m7rK`mz&e&_63L4$39L6ZPwTX|9b5w_?h$M*jL|nq&o${HclsaTQVI~E zk6k|WTESa9Ps2u&eTGZBHE{U)q@cbid3*B5*7Ku&AEm9S2E*4)!%w)BeOwFBnfCfA z#awP70(tr5o>+YD!PgrJK||<>pJxZhjd38aA@iS1CdS7HFD||awf2>yZ}2_{YG{xU z>$8$@|a0c11z=7q8j7O@f3YwZ^28Lu} z=SuMZPEPy}V`y$7A|fND&CPR%L_=j5&Tb(@T>jzm7}ZB$63}HT#L{i=@4pxu`zwHG z{`HF}CML#?_z)@dL}r4n3i)=qv zw`EsoMpak9Hkh)tBTN7BLBvjUbE@HJVzZ8D`kD^EDc;{;C9 zLXsaFuFu%&hxF8uiI7mE2gl^4Agi&P*y!Y1s+ydbgh?K;W%GGf64;Yb%(j-YX_Vyu zYw=tELmV*|PyWEUZolwbRsAtW%%)+KgG}(Yeyjh!6KzhC`RVqgtpr_cmz9;(;LV_V zA>e46mz-}QECeln4r=9RQqa?v@^K3al0di^udpN#bV_wV0=a^JGL=*yQkXiBn0pf*jS%_; zMr_+b0MG2-RHkEzo!jIAZny|O=N`6MXD`p*6~9Fb!@$XimE((LS&>=~?#*c)^Njd4 zi!sc+wAyT&BM)`Ggd%4NFKynIev{j8`f8tdV>T*2mi)bHH#$e187{j(h>s_5tN&v} zE7Ce@v_#p)#-^{vfXCI%ZDme;cVceugeFq%x}DvIJ%XpG*FE*ahqkdXZIg*e61Fn0 zJ&T|3??t8k`}Z$z|9f6ga4-WYcDeO1tK)YAMDza2aJl`I=pDUEtZ;l<)BNg@6n-;O zy{-4}6q!*&oFyCn+|I5KI!b(m0OJ!95&{@QxYuT6(AU@3zc)0}50QBJhw0OXS`?j^ zQ^5AY0lz1bj}zjqCZ^W0KJ=MpLi&EASN85t zIFs4pA89@rvoDQ;Wk-;#zo#+zIma@PZVOi~)&s+IyFj41s{T6m9Na zqui%U$9a)9;-^1Z?LbhdF)=?tK~B!d&5Z{&o05tB<@pI=REMIfYCzs(_mXqVlJnaS zMaL`?6vn=<3S;+0i}5w8)(fAl@i3H9z5K<*dl&f~ZznTI&1`D@W7#DM@A$Y_&a5}< z$Em9;UHM<4Qsa%I^z+pBJ#)_L2JK66V6VU)xD5?lvm#|nGn7rgE({?6Gj7&#vupjT z%a>MomTS37$HLS7%4!ORxk;PXWauyN$X8`^0bMrya5M>tTARpv|3za*M-du9w~3P1 zudzcXg}FF69cP=bDW{9;&ba`gNf;@IJ#1@dH@CX_+Geao`Zi&~`isbc<+k6y$$>Ws zi;sU+^o<)F#VdavoK(0$Km%473m?C;#ISC5aq$KwCMHn1pEr^pU4Mm;4aZ;lDB^S- zVNh9B)rVMowsAWoh)!KLoM2r+$ys0@=FXy(xqh-o%2ajXsUB5KKtPR#>{75a(&av)Q3xQv=K7v^s^BUTo3GoPWBbh`FJ z;MVk2GX*5U#>XBDr&1Kn6qKgjI#P!&cm(=8r!_Gq0m(L^+*TI^HJ1bE^~?8jM<@UM zsv(ha&p5e?;zVPzsvd)BPc)u=8b!yjO18f^(7Cj@2vL|xM1&If7VHez$#PCk<*j$U z*aQJU433Q0y3+!KQ~Z)4AzqPLY0dW3b}Hm1$?U&>VF?MO2KAmqAA}t`JEN(~-PW{3 zL_`VcJrffFJDl@| z_HHVI!)Gs@fAdDA%<*LTm}Q6b35`9Cg2VnX0e`m{`#k;? zqNv2MygT52Yvai`Co#$NaI&?g)|C)?C)OQOPjMc}kGn*t_>L`q1RovyUsVr|jVXA= zr;i!A^;WYCwp`I$OIV02%CI9FM8d+tbQ-+x0hc$6L>7=9zBjY6!2wz5e7lVxMbb|c z@qn4QrN1BF&CRXg%a@P}mxW|qP`I!g9DA1j{c9Z>`r0vhx>nl6$jOP_6+>&5fwSoR zol?Z{_R!GK$Y+CFVJ5P4$-RjqzkeS_y?JvdOHca0!4C6|meFMiv?TD4ySX z!j(pX5{=2s?9RpfV*{nfvh9bZ{B|+C?V-ZkvZlmTFP{EP6TS7!;$*hVOySpn`E5Yn zH}vb$Qv`Z3o37r!k*M=9IbZxyqplX{5X<4wOYgDBH#CqhdEVC97S!<=fmoB(x_!r+ z{ul#8XLm<@KVX3Dtyp8{;w?$fu^%^QHX>zCiUrp3r-ClZrLH6LVyIqSmXxqJjZ*#& zAx}EylAwLa+8NjLDelqQPMgPB`Eqz?OIyAl9~Hlo6r=4Lxg5RZ_0-C1OooQYz}q2l z+JQj)0wo<>WivLvJ=bQ7V+eb)2VWDkysW~)$c`uqCU$l#1fS>j^OX1R8DaCnUd6)2 z4av%4Ks1kzl4WIO=~TTW%X^ZB&66Aj8Atnl;*S>s%b->pPC%r68(kaAo; zv_q}sJPQkUZk~s`1Nn}ZN2G|rVnbC#G}?QcZBIljrp<@V8t)SU7)R5?E}Sy&AEr?L z1RE2pUPnx?OdWrjp;oDVW8Y)**Yn(Aj!lG&A}R;d?D=qo6YU>eT^*$GhYJHYO|o|R z(DjO7)YMtb;U$n$ECdjsaT*)bCmvk})QgvvFT@?2(cGXFeZGHuv%9~iwXap?aIdyj zw4l)#&(JWfQ(vMguw*Ks`k9{bQP27sK=?)TtH_wnw87zDcnHU$Pl#IY>win%J$Uwv z8iQ6vzg|35`Lpx({@T^2teR~qcSmL%L z_N`h*CMFY8)9WCe$vu2%dbBwe@%r`kK~MbxC>0^e-zLR|e1k13E4x13Kwamt)d_{R zy1qV+zP`Tg-x~F~@1nr>GPANW^7E@(rvcADyLSR|ZF>x@Sefk~0Syg}=Aj{CBO@c* zBGYGZS3Cd78u7)YrKNHPRc2ztT&7q;Y;U=~v{QaMI3@$Nsv*0`CG%(@5N6LvaB1&j zSDY`;tuFO;*M7W-nBDcnHhDRmT=e`ZhUu=SxNAj~x&Hp;$r|tC#fQmkJId92*cAO~ zZPnS1KX)@QZK$cg%rugKcW#`52*2@_B#_EZ9Nc(YqW4YK-t*4g_r^ty|JWs^)wbvO z_Z2R{EPu=!H;@r!+#e7K!AoHRkFCEC*&aV$pP)xWjF+28xVpNjd2P{s-qMzpeM};} zi-SNc<+5})c%zS%zL?wGnB0|b93l8Xk8K>e`|f)Es0S8LdbF&w@7;gJZ#w^kYz?-V z-2S{Z6Pn&=QWP}3$!lpg-D4GM?KER2t!_l_eLxo?Sgu}^Kuq9pb&uQV-_&I8c*T7} zoUwMbCFYRIZqOh3VejOwKzhf-r+IE%YFPJ8zPF;HqHWALp!lwYurLMeDAn`7IoEa{ zg~*shRTt-G_$2jzydMnmzN5%TszMD>c1Fe^89D|?GriCI&(U?7c3qTdBuNp24$Tj0 zaNQMxm0#cMO=cMM5PI>HLsFLxmrfOpR@7Ak>luRYrLS4?#R6ME2yc?_Yxzs-g5tuC zcbdlReN6c|ML2$FXoqgi^8>R|W>gP-`N%UcO50z{OV2ugN7|Z)K1oWOqp!qT(=KUl zR3w@%+YH-m@Md`aL*T~o&l}0T9f)e_!J(lr=u`C+$3rw{X0@q#Xk`FJd&pp9v4JCQx#Jzw~-5z-)bDHFkEX4xIh1`A110!01ReS^7s zGtGg1F#{uiX-uBXaMsWK_$v5qD^e>(2(l!G7p`W?Z`X_XJdWu@0smc%HyK|<5Oy=y zc#vYv=*s~Y=qpY6vzuv?vyLLqDf6tmU3^k~$a?!~1L^R$IFzX+?8;l$xNcdR*iNIH z&x;#09^>Sp+jmC#QSY1$gb)5wj6f4hX0U2AdGRT|9sbJiJz+{o`F@@||A;I}(Dx~# zQE&fRZT$(kcfBCtW{w{MF%w=I;zdF2c~;so#+7I4JqwPozV?oaofNv7oBSrdZ&Tf= z(uy}so?E@DN_k0tpbaSQ0 z^Wt?YHHEq&PD0?)(9bZsXW-K9((*4SM|0!gZPF$4xs0eL1fur9HOM-@1Zis*3pK^y zFTudDyriMxbcg#5v2HV;Opu-de|bpER07#v2ymeGYX3~(I0aS6>gT%dAGYfu`A5xf zQ4{KLKs!`p;J@JPwvUUjj^+G}!>DajBp$+|NHRG}Ix_?FFyNJvjC$<_-<<<#=>b@&Y$j6@= z^-KGWJ<=mn3TL;S7kqKIQ$i{i%_zmpbSrv`y&UO(Ncp2x6I}&FeVOdOg>^FXynf^E zja*?Dlj%R?kZkzZ+q{mwJ2dv`M2LY1C_64P<4i6rVte4VgLfXE{__x53CqhbF=3F; z?kcgdQ4*{+dNjLWaQYBd&!iEABFtWv)Re_xXWcUI=vdz>)$5L@7M-St;yJkHHYS=x ze)ISj9K`eS@V(5e!cbTjq3#BB@cyPhl0+5_8*X$BD4^bHWNfrKyNDK%sXrjOz7^FZ z8dg{RnC!RUBAT(t_n0ua)P)4A%=jOS2WFUj60}oldi%Vnh_G>3i(C*5OH;k9bN4a4 zv8DVe_#NTX>j>>qqmI#{ksk{Ius3IymJZs>JeGRMyt>iC+jdAIb%v(fmUm)mlebCsGR$rg;|AtjWhcmjjtGvF~Yd@2nNT$ zj{mBOZ}rx=nBEr*i5r8`1pDaV039L78J*w7_l6o#?Q^s#A%uh!$-u%QFR7ZT$ox8c zhzZKGEUBbYrS2jVN>#$}3vg;hw^3N$T zaNBM?l#{c~dH28H4k;-9JSnX1zu@qf^O%Hyvv=v0kqnD07DF^PHun3ldR~r2B3L|x zG9$!ocQa>8pRse*guGYEot7ze{{EMllSKy0dL8)>Ai%DZnJlcfph6AQcwjNSEQC+j zIGrqqmx>@Lg%)=_>!o@3nvX&Vckjres*csQwQILEn2EVRJ!O!)7z|v2e;J(mx`sdm zKXC}o?+duQs4leu17Za9@o-#q2IQyJDA-R7i&^l@*AR*3f2;~hv*O@pDvHd+06DT7 z8|iS{Xf{U*hm|?WWJV~ekVG@o<3<_V9WTB*i_qAJFaA%gV@n(hvwK z;DsVz^{&UJK~Y#31`dAsaHI*P9p_suj^!QC6}wyHn@LORg;JqCuUqr}8<2?et)8hm zk$_8aI&n_~BL9B|ss5A2&y=j<+}wK!CMi3JPT{+Et_}@-T}O` z1psS6B40BR@!-dS3;TEf80^ax^=_t+I94DO7sn7B;&fYR*j@w1T3Y1aZ2ttkd4r1x z1n>o=2?`}69#~kgCh;1VI2!>M>FMLM0HcqEho|Wo+q;wvbYhBBfMkda!{OFUEPT~= zqVkdT{mF0lVv@i+7-%>7tpzNH)6>)IXYkpMS7?}V85snCtQ-B#=7&|KLUVIj5Vu)b zG5GoUIjf!e-rp;$u68)QGK4yki;L@Lniz$g=~pMwFd57u}?_nc!RE&+km z&YiJG!IM`WcelZo#BM{pD_(dsw_I9VONB#zr@6*yq0FQW!|LVBd3e?1^SuEgIy!tG zAD@xYQRzG6OyUM!1!iyppZoo%g`{9qT=yX(BOCu_M-4hv5NP);fBxKpasVhFLhMlN zF@A9MSfXXhF-o5uZu~j&*q8_f=8uV)`37*WsM=1Expn;yB<(>4tQ*%gw6tVYRSAGO zQJSTUnw*)TXHtlfgEdTkM<;N6TnDi9;45kASaEJ+a*&23H@8SIU(_P+}lJZ5=wA_K6+K zlEuX1Otb6)uO0Y(99>-(4H~@3zzEwKj7jiUgyukdAQ0f%#BIDd z)DJ~*!Lj^iodqAS0NbJziy`hr10Lz@uCZ{Km7Z55*{UAppBiKN>{r}`;zmo3QJYjwcmK4<;c3be|VKAF>3$aodi1Eouv!5yaz};&c+rpS8{)>%{HrLFu3cUhZnt*3rRARR-gwP}y)8?yaD{55i1rY?xr4W9Q;(gR-~Wc}Xd8Qd?K|$28q3 zGthZTlkB9}MAX!i>*D6-=D@UK0Ixl}yo?D8MU)*l3K|(18PAT6j*_aXswObujMW!x zq)7OD3)SGfxb?sT+{|HMVsnD#Mk~xJB5rPOVE3|JmixhZ^U%-l0=Hbp_TSdl6@#D6 zsrRV7Om}bB%#5)rYy1Ho0fB1v(CvZX=xDre_EWD{S6`kVj2SDcsC2+WWq7MdDk`T= zmI5pG&&-LhBtUG2_#bfz2_dbmG9jU%EfW)Z3_7)Sbxfd>WccphBqkyG)f5mwNk#P; zpFljl9`iTb*Xru7_xEfP{sqv$%$)BJtHO3nx$nY?i-*VWy>BJTo_;+#XRNc`w4A<; zFCR>S0f(^QeAFp`%09wGOu-hc%85HNt>5XSrUcL2`L$&efz&&!%*-K;fnF-B zj0}spxLy`d3-_^~G&g1N)3t_zv|KQF)= zy2~}jL?sfVcJw@*JD0CbQv5*^{Ru%%M8!v)(QW`#3@DJTW*Q}+7~mj}ss@Ra&h0JX z?$fY(V}xtKqBw+zD!()L&_T4j(?jExW{LR>0T}k+M#CQ7*w|=U-XkI*35FLC5fc-x z!-W?E`>r%IF^BR%Umq&yzqh;T`>wL`E{aTMc<^s|S?XXHYpV7n`v))I+l5zq8hfSXGSFDCWsl6d{Nc}$G-R0J)DcVPm z9;JvlF=F7;V8K)ayD-<2$feZA@kOsn$=V-sh;oLc;nG((S637ta03Iw6dbGA5~eY? z1Th%jVnJ%cE)I&FcmA$ZVS$H=u#m;T2QAI`_BO0961FEHQBhGcYR{^o!A~0&930$w z7sv`7U0nt~KEk$e0+_nDHbANSNJ%pke!j#Ic2m!^=aCG5GEr ziXA2n#5D!AknIDQa{)m?b!}}-VPWAfMh)-nroMjtDx;_v(Q~qt;dQoA4ZpNdu-_OJ zg(KuJjhm{R$L9xcONe!YQ-2Slc`;v78Yqlls^8fgkjkm7Bn3|K7tXD9A6uMn7ezvmYtle4Tf5g|9vlBUS60Z^_IBh{y%@d|Iud$ z9h={IvAe6^!P@#JFrbNB-$5lf_Zj~yKyT-VMkVI`Hr6ZuHN*;V_30f1{2m$cT^D$? zHYn)*lJF4JgInUu%OP~`+dKCW^?UtdP*zr=jSz;ny#hhPY{_3_D<3AYrk6lze@CYp)^mii`1}7lRX-9QgJv5sW%zSa~W~HUpJHS^+wu;G^1L zPFel~u=8I>h`J&lpr5~kw?mGXGa&-I7Jj9ukRD<&Kt;gtNmK7$ee1{jE@93hG%txy z9cc8Z4%0^mabR!_5f6-wGnB4avh(x9^{ee)^)9(zbBtM?k1CA1yAMVGE?ry4*jO_5 z_R7+0xolES45)qw!LvQ7RKVESSQsX|em1jTH@*L9W<~k<$enR;?Ut=v2M@b3+r#SlRww8 z6JmirfMsK|#`e1-@8CLulZ(sg-_KhwEGz(_wYIce+uYp5cH@Ug1)3@VA`PA2Ic}BT zyZtT`fnVxyHQ$HgWW0;`QSDoyE23)&gHS8p{GcG*mhLV zDHHeOg;E2@803`tdV>|JNB$DK1>q99*$}kKs;bbFdo{ghy1Nw%D@wn96+o`6+@lq` z4gk1&U;s0@H#9mry444eH|LcFLhZY+Og9(#{`|rM84n?hBoP(WYxCYDlYQi`^^RP~ zT~Ckx5zvW04*=t4R}eZ53V}gj>wY0XBkGC;O(&?(^v5ecKK=uqHy{ygieyNZ+N(*8 z*0Bjdz{{AJF#MgGLJ6vIae-S+S9DOiqGt?fBUv~(IBo*?S$D70)b1rcRD8T<3cwWl zM8;VcpA}+gTB>LKqhOq*0WD?mbar%bjBI@S z5(Nv+OG1VYVLSN^53&_@O9=#CsA^CN2XG|f`AC5}=dy>Zs~kveh}#Sd*QkZ<<)B&s z_k|i9djcU7g%c2Qsi`-mq@;4qQDAU5Pi6h2Bs=?hOA8z^+Z|Y3%nkTWOW2-hW4z2HxOLZJARP}j=3%WJByADjV?H=^@phup4kR2%iJS&a_m1TCJw0oN zPBI90vC$ewZf1zSP zvh6h)SDDAww3fppl>Q$iZZ0h^FF;vyyp(J_?(h>L9?CM&-qqDMIZ4f>U7}GF=Rb7y zs+G>`ElA3VtMecd6VlT5VKlIf$^(#vb}y6J=2z?J5%u;cLjb%bOoq+DH|TvTqmYm7 z>FH@r^!Ru-m_V}mM?4$9`?^kOEhPoT(Ri)Of1zPu)m_EI!y|N0)!dxr zU*CstXt{X4F;T^hEdlEkCBg~W|HY!9pg=@(+ry|rh7WE%If1fZeTz8q2C5>2ZrWE%q>)B9JQIUsrgHJE{Gd<(k z)2D=_q#=@ir)9(MF|=eNyk8Dm?oU)Z#7<01kOxSD+XNc0ppb5Cb%oLxB~rnkpe1a; zm3pa~8bE-&i67*DPX?2cT$9ojK_8S^B~SK)IFyk1F9dYq98w4-4ZoVx1*FvZOAWY=71@DVdHARR3eNq}Zhkf1?U)lRv}iIr@f zsImdcs0zz(3-KKEAb@hH*PheONrG=^d)q&{6pvM74~dN>ki0xL1NhYf?j?MD zd`B0TXOlkt=-01>;nViqcbdO}sIjuK`Eac}zwHC!fr&|0Pc0ZT$LDK3fk%VdOJzmq zai-I(I~0Pj4%w$pPnf|N6#}84r`F2OE*hFio&&>F_U+q4=zE%j=nHVKsw9ezj&1`r zWTq#lr&LwuU0e_=6%`fLtnSAf<4AZLkzw5!{PpVV3!O01w}9C?-tFZ@g<(KL)uI~TLQhY#`SCNn}7eTsy%Xh z8#mVDikP}jgAWsN`cu{jmcNe?I#rV|ov152jA+jGcL=*&;%m4FXyRD6iw7{$0hfFG z@#CBAZAU9>>z1}QG$`mITPbW*GKPx^3LePIqr>%YX72RX{p(-era_RlL{G82 zMuXDX7+3J0zX_C>Lw9n#7qs2FY zg0VeJKY^SFgWJ~AgH6J!+zfUOHP|i?1c1VjU8*+bJ#7_$_|3w?vWwH}zi&Wj{W;+3 za=n5XqXFWrTrmM>{i>5fDsE)F-GQ^?#bl!I95u0TaaMb3|Fmh|!CPh4uDH+xpIr@UZ;DuX&7 z@QLX_x`d*VQY*OR6F0qe>fATXGN^z5`qcuOHMHhwpj;BIp*Z7>equoSWe*C!N=~LY z6U!Lrs@rPpozAnivWkFrP3%pcd3l1oLiG&;w=rJtB?!&B3I9m2Ls7inKUff87CMK8 z2b5qausmn$nNVPWw&%-7%var7=8Bmag@X> zJv}{WZ#1^P1p}mt8?3^@%4!N-X4u1b@7?=V?O+Va`txYN8D&^QneAA~)CLSjG7K8A z=$9W0tMd?~%f0sgX-PbWzTqJFHxvX;`v!KC7cXAShSU28!XrUh@+T^0ZLE}wn@L!> zTC>;@z^0}yg`t7L`;FfpU_fsmpd&DCW^iy2vw0M-Q2)+CSL*VP@2!jhg$EB1(0%j@ z@=?`le;8m4qdA)VoE$W0zC%n~pc=$W`q0o&DCLfi7h=nSpae{i(7@)`9mmiO$q@AJxDOwqfMJV} zV?;T1_6^YQPCvdvo-SS+86GBq@zwcedkbJH19=n*LIOF*<(rd-Pmz?bnqk3vhnk8C z10;CzUkobgpH}FPl1AiDI+WGa2!RH2+()WMbeO_cYaJLMFflPfU21Z5b8ha{{=Vz_ zL=_HP9>}{6)Kh>9as*h3-sBI(QBX2JeDVZet5~n&cSn1>oSeM;+{MvMeyt*w*B6vQ zf-Y7Vt9cZ3eP<|2QjV!UUkl3zh9CR_Prbiw{@Up8?pE{s-QU@X0mu#8I9*#q<~Hx&U#Yc(CSX1SG&RuxMFXv1eyzZ$pP%P}f3BXXnF}@1js_ z0MfauPXWwaxzCZkoSYo{h&ir&0#wx0IAUUAZ~%h|Fa)qLf$m;U zBPjM#mV5Rr{-V_mG8bggFkrrQb#Fn>K3V6lXKsy&0g-IW5Y}L&zM+?*gOLHry%pNA z`CXA}p>yT1LN#N^zyXv_xI6EpcPb4ecOpV&QDVLj?WEXp^{a#(3xIn_mrx56KomSx zkdu2+Z+jLat{YEYb&K&WlG<%L17t}cEo^xmH7E)tiHF|rP3!?1S;_`H7^0@qVTKME zFHWC*`L0a=Ty95ZDKl;d}tE+F~ z|Ae$K-QZMBe^8R7$qqkKZ0 zsNR4g2}WT(H1~f*>=3y2Cv%h7t_3dv-1u|2aC=9`-0xz24L($9yx5`i_;I|)Qw9`w z6UcMtW)^HxYrpX|khciKOz@>Vw-Gx#JM0`;_|7Quq#Ng5YHB1(EsEpE7361RoF`4h zX3(Ix$xuu|Ow70eO~b-%eBmEr0v=&8l)l%J%28lzyLtltBfuBjjb$)qcHSmlAxd#- zhM(Y8Ck`gO%*^fjcgV*^#>P}t6cq#O>cm7nHkCtVvAnp0jcy`PWWRRl)2+EHbf9;V zlgCJS`AQAGGd7jo_#buD)#OnD&r_fW#dFFqXPIE-?4F*UHUknsMV&5chx%XHCvfn> zPJQQ+4=s3u)EEe}d89!L36-L89e(qAV}&1g0i$_iy1^&5q2_0AudI^NYv1$-z)8#J zD*W%Y_gq$GpqAppCGmt-V)D=iSjuo{hWebu2N(XSbeZlbeD7TxWzMfx|K`wZCl@5+ zlas@T^HqaQDB(MlL`3D$N-_WQ(q8^14q%PZL`X{co!$i)4r?mbLW-CEF+sBv;%;Tn zvw24mZFxo_i8bX2(Hj!G=TkLKIV+({$?&RJC{}%sf{2Jn+_L|pGa}1!ARWkgyC0NU zU6anU9=o0g2L~Eha(0h$hlGsT4r&V@g7?x93~FH}lnAWohzP8_rwsosi@w%rp)ix{ zude8M;|rwl!=Ircj<#mnE1>JZDI$mMxB z$3X#>4pHD8CZ^yGx8#C+5bDSf&Cm-(Rd6?=_FXj+P(0ij6dgd?&p$oo2scS-lgV;q z{oj?wt0gDbPUJOiBM`%?E(>ofPY_IZAR#?+Kq>F$b`K2!OHH11-zXlZ7j!d_k;2;e ze$G8f6ZMj!84D*wJw`U2i36;jLOA$D?&(uvI?1}nBZ?XtBW?(J&^<(1SnlefsY^k6 z173s%!C~j+CoR2&UijsUt+z?4GDT7dgwT)z^Kr2bzMq6JaPg3YVXkwsUTqR;$cA5= z1^=t-CF{%lf`W3QSa^K9??ylpcSHZa+OubE*W$9XvgYLf?>cO66%!Zx@AjwxP)p8)KBu+vXHEYY5QA=QP3D(hD z5L!NdHz>JbHMvvP9nX)ucOUU7FYotLO19NmKq!#0pVOuDPWURrJJ`MHT7YenVR`%i zX4IJ|Qcn9;m=|WVfs4x?lLq5Uvi2t64VPcq*i?bqjFeW|ZH117A7Uo=kIP2C+m*Um zQB)R(QzWQs5dZfE^&e44jNtZ1B^ho%+*z524>5qK*zTlu0BMDS5vpPs4Ya`bkd0N1 z+l9dB59dnAV0p2?S~6aA6nRV;X6*ss<`!P{1EUZyoP8AYz()YlZc%wq;pGf{74Tb0 zaoVuu{-a6-IU|=DkL3S`{eE3m-2(1w8P@ndRb}M_&_{OPL`Pefz7Zc= zZkPx`CL7A_LY8G+&ZcJ}*pG-P0`DQJ8r*N*y!kdPB*YVFi)BIF3M5F73XdM$JU!Zq z1yy`FhIwFQs?Ou>yLUn0F~1&&+ib}zx_Ggi$tLwyR73cD1)Jxh*9{56qRLqhn^XLi(fT z-tO)PfbS^7u7$U8SWsF7)!v;IkGI^Xp`n?DL%V#=I0iG*Q8UCapz7&)90{dZBWyvX zROLf;3kz0ZE-oCX5FpIXu~5P@ zsgzcZx>m^Dg`pXa<2gCz)ioR7D{u@A{X`l!f_8E&r1sn{(vwh_@XhXdM4Fa=9jQ;_ z8FrM^^vL|uxxOPA{DMIHP_r5|oV+kMZsp~0N?ESht(uA@2yahWSy-qD>ssJW{_%ny zT-IQabGe29-(esah(Kt_%FoBNv$KQFAvJW*zR};mv-9#W!0cF1@?Za=NlILz%|8x+DXRxhw);WBiovJp7OHB;>$wk5E^;aD6Kt<&qBJ*=C@k3e_ z*m-LHbvL|gPX|ZmxNcM(PKd7GJy{mTdXrh0m0MNauZ@^IpRs-GaNc})T#@KyFwmS%AGqh5|g(I&k8O*vz=mx55l15Y_xtkB@H=b#a3 zdcyj;-AiffhyJZ4c##CX^F03=L`L!Z*OiZdq}*G2hTIbyd}1>{S7V`$$o^ls86Px@ zFan=<)tNRg7$LKOL6Kw5QA8g8Oa^6akEiapG0DPkn&*rguJ-{F?b*VUWBwh4h02MB_4l#M z23(SbC#a`MnBnR8mzKd%&+B0BOT&|l6K>Rh%)$ad&NV;nfRaGW?z8rGjI%u$h5>gH(6a1P}Y$H<9gt*`#ySoEnuNq%3r3sM1tXMg4u43m!je+-4W;3U6R{% zU-v8wB=h$#xOeF$TeXeRmS2`tw+yXI3mfHva7b5%YBWP-*vkNzyt6 zF+smtCB=^NGBJcyKE-@RRO`Od(kLXn@$+(%--@R3ehryej#f*{^eqqOrNL#7W(AV# zW!B4s@YO{mF_#{VvEImcUqNG*yr;3=YwgNrvjfclpG6*4i;~PG8 zl8(NUxrr@;=#tN*%=OXGmP=Z)%;u0=i`T$GJG6-=CDjxO{;E$o<$a2vu4CS#;IRs* zyq^lRw7yO2G!B6`oMj~I<#k`H)CdnD;;~87pCJ_sJz-t_M}5LGz*m?}oblG8C`5*ZJ~g^ zV~{2r`l&!Zqta(VrGB`9g6xlrE5?KLnzGTo!j9q(K$>(xs*HR8{xhE+B+d0L(2I4d z?WahR;m9Q+p!0Kp#5-QE7$|d`ld1XseQxnbJc?z8_`f|DPM-~SM%27$l^EOvzExG~ zf)skC6dyg3R#wJ^#Ga63`0QCQNd71d1$xM|uwwp87VtYQJR^)qx0NAuFqCuvCGcmG zDeCoWH4TlBkdPbT%$5!YWu!zPP~s(Yd!z_E&_Ti=BqF+oDERzY25=l`A%k#+%oqJ? zd?4f4IXPRw8^h;+?g=g(RM>0U=}{4v zy4zKHYo!8aLqO8n{I>-eef{t>*7zTeO{Ydho%idF4R0dWg?e5H{4J_)JKR-@&;#iN z^*9)KRP7kw-@PLaoxIog^QW4o=4f^%98-dc=SD~1R zSGqt2bcl8^M?A(xAY>|dkYkgy7v2Q^|JFp`mJ#H2rp;bXQ@ZP&Y!J88+_V|prtn~{ zF56j?AE*3EKQSRXPJ6*J{nf>r#aokQKugSRA_YJ@w!1tmEgqJU0A`mF^0|+(hF!8{tK5q-d^8_Zv#KIyB^c)$^M0k3crCkF zynj!14s@66J}T{(z6xr!>25rv5b`4mc5C@HIJ(5NQoVo(^hxw^EZxQ5RP3SuPRVQ8 z)f*r1gf$)&aYshSIExpIa&HKvl0H}v@y@NbhuNpn96i#jDu$E5kYio6Ye6~X{4b5aF{%$i^dh{Fx9#VC!MGhB?5Y3@d0UfWH#NO^t zI)PewSR>+khsN?5$@r4tsTewD%zArO0I!sL!xg!~e;?D1E8aI+DgMy?Q^_pglDp>f zb|Zut@o;6C_Qr$tS}=5K8hJ*g2M41~I$N5XhXQt!h1mynC?X=_fukdzr#Kxn!z744 zx`u#$+tAm zu(s%LSKX0+BzmLqtXylg(pK_RZ6qM9@iAD%2}uj^$i@~QJANHEj1%s1Z}4TOOFMu7 z{hIAlB*~u{VECe8=}Nj^Dzs@0FM-pbd;odx9#B!Zw4xd&6T3^j`jyd458pIHuts*H z^1czt={f$fvo7g7f3X;VKCqIbGriVK7}l{RX#L!gLFkV=TZjFwPrDq-QS=I{Gxy7) zX4zSik{`|R&Ws$u6F?fT3Z2+;-7e01iQdrO$AKjWzt&|LswJK2)8nKE9hXW3o4mDzQp}VUpO!$|(^y;Ecy3(nU>%^+Mw^x?$EKzdsr?Cy zoE=y#;N2ob>%2F39l!tW)GBw1uHK=IitX%bG;VI!>GqH5U6N*vpL4WjR(>OvRisG} zswL*dMiaZ{xnRFUi(I{Xc<=ehFJ-7JzsZ-vLZEz#b*zd{i__K6Bj)Hqi)Lte5q|c* z7Xt}U*5K-h9)Jx0;2#oz1T{0a~m#{?)J1jC1TpmoTCLUC|MzV+jP^(jAtzN4$I z9ZWWP3OVM&n~$CrkyRO}8sF*V`{!+@6UHwzW|p#6MBnyRpwocbtoQeJV${8VjJ?$c zEEL7AxgrTU&(h8>3nbS>%qhu@jwXvgxJZsI@7>Q>5Fa@EY+>?2;@&V@h~s7}$B^Jt zx|hau_Kh^6#9@zRc@GsrR?#1MmJv5p+B}lWxc?BerVk#z^2;$>J@V(6&MsAOe#}q_ zixYBt!1H@`^OclDs;F?Bo)GWO{^t`-ztgrnih11PPV>@>QJ?vo zG0{=CoF!|vZ-EliaEskBA-eWz5M;=)}{9Lt#J=rr{pJ6 z!egfEm`n)d-Asxs8zHG*exaKe9k-F8SMCu!rA<3((xqO{=wZOp`=pCf>wNhSt8A{; z7)pQMXy9!;4g%={h z3eO@hPee?V|1kLN&kJXM3LfLdZx7qz{I_=WDw0?>UA9WkZW=XV*c7cAJc$ranQ>Dy zZdishMLVlZn3lZil1hm#CmCR<=!&&^7h^jUB3*P@gVObiQKv1!d9-1|)1JX#8;66t z1g^k~i87!gkyd4!5KHfaW(*M0cDxLwsFgo&xo)mp4^6fo)2=u#pQx%X!{@;@^|ahn z9?nH2Ea3nd3IjcWI3jS-QIm!Py$%E(KQ>PsVEq1?@PS8~wpc%vd|5%=vka zjO(`_a*|cwqhiIS@9X=i?@Bw8e6vkKFrb6N7GF<=EUM?rbbY#uQ@KHn?^{irU`PCp~1;SvvyAid3!0NmNekWt*|V`)v6*&Z9WdCK-BK*>L||8AeXeCmPT1DUqCdZ1veihyB2@D1W}Y3?s8P z(d|qR3M+2wSuLHr`MOinhwu!4qrFs#DCDCSH9Ir4BUN!Z%&CT)0m`p>Mh}i$lGS0E1Zz zS807K2>20cX|%B6L886xxcT>f!=XDEUEqXcRLrFkvh9X8BCDZKk6JL@O&Qsqd~ELQ z41-t=<>z>n4F%}H0F3!OHvfi;@_-!&^xnsy&+FE~?+wMW!=@5vgLBATlHOR%7j%sO z`OPGz-&SLNVTC*v(0#9;lB$bDuZL^~Q&V1iLQKZ>9u6+9DgK^&4v?_y%D!c9iQJn- zs9Iojf9oEVFoi?r^Eco8!rg}sqK^`qb+dvZjczM8xF`NK;CBCZkgyP~TK}Mrsw$CB zz2V?T)>HQeai`6(Dfq>eg=*}@2U{z`^%Sgb+ z{BvlgHy)(Pl)p8yv{`whTQ%v(wRuD;yv63FIEzy4o5|B`%|}UWIZH7Xy?dQym-fUE zS`2J@R)skwT~xU|{}IOzSl-9Va#d#chKb_b%nbCp;+{hDIn{1$2*NcY@&jtnO^`03 ze-Ni3q6UNf(E^>Db?3WxVO~Pc!xIO%H&hrf|Jj_ZiefQ&H0>(ev#|q7gMdN%hH9y) z7Pt+_Wh+TrOg9nZ@v#D?4-Scd*188;GEMN_Y1J#AA%W6h+DnW9-lqI#xhCIK>iB~= zjn`-Jc-*;4g()sLoG3qYhzVK!lexDtEQ^0(gk^uej{)0Cw?%6Dd)O&C6jGvP=VTQ* ze_t@HgvdNDT5jK2AFYd%82210F(5@*E2Gt03;bZeG1|C_aTj9M?#bt$Pxzd^3u*=~ z9TVTYhR>Hn3?HrZm=WzGycy*ht!rg0QQ~oUtvh^}g^o=xlDeM=K!&!{cvhD%(z1%t z=UFZ$Al20pH@k=S=a|9tsbnBse_}ip8Y9SEAjUeczDcI(NcT8unDs?C zV|LxhiPpd$se8`*?|M94*@wlefLC^!E{7INGUv+CXT?-j@4mF}EP3;~sGU${rYb+~ zj*PMS$MIO@4Y!6~(%U?V7*^-*i}Ula761`uD-|RS*u}-ggaXegDG|rIg6}O1a%cLd!6bY4k$bJr z@t3=?=bATUmoT-KT|x^ojHGN5FqzYrL_M8z{ZF#lU{h&%rdsa~SiDHN6p=K?MB%95 z5X~?>FiyeRvs<6PhJNr=j3y>>+r;d6s=TZwK1x}y;!P7N)m3$E{_Te^TL*EG(quYQ zqx0+j8ijNJ;rg_sYXh36-j=Qj$~!Ts>oa^_nnPx0tK8d{Qyvc2A$S`c&t$}M%fFv> zpxQ%+3n<=y(bpHc-0#MYf=h`{wKka3|9#yQ1L{5a_p@I)<8rTUt(cvaJc?ttec|7~ zBsDuV5(xo6Y_1bLKze>+l`&d;&{ufxRb-62;_v~M_x72O_iG`?x#!Wo7NBg@TivU zmV4gh<|3!CO9glY^tm4za(M_vAkfWuJefaJpMF2tmam%%#VN!aqm?zyp2Ll{iZgSr z177(pU(vI}r(+^d&(J5{r_F7y`sqwIKjA#Ld4u0NOp)RSFQmS}T*o{;Z^!r{9~?CP z>4v!v_9)a|nvU+^1`s+Y$z*I>9*}K}$wRQyQLF#Lv?Mt@mQEqvD_oxQR~K>_-T8sV z;$*y?YTbv6p%gd^5we=A<`)0uK5V%Lo3<4>hg>DZi?Ub2@6R1RE4=G@<+*#bRzKG1 zH(KPuRdr;+^keJ9`tG@ko%H4EpcSjqV(X5Z>BJ>jJ3p^Z_obPwo6c_=2MKc%X8*Vo zg2`|D;Jo|#`4!&5$Pt5HefG`h!)N0woauSHY`W*^j5b68$g}$5STBKJ@`L!#P=S%l z{csxUa1Ff4$1HYxjcn zxs$U!K~mU_x~ol1mcPDyw7fZRIy4?i53mauLr!W?P7pcc->g$mw=|L4p8FJ~ynyXF zv-e??t)AK1R=a+_TSQdi-v-GANBM<;-jkXZlEsOr|2$4*P937dQhU8kZp5?zDf+w1 zKi{65equ^rlkU(lJ63!vQNjGP z%Hza&4{CTNOAeB$50}#z5gHL-)Pby=`$=Ekv#fe5TR*= zl|fNr{HQo5M_c;+!dUA&FIu|10k_GHj>g7c&wf=@l|SzA_>Nuk-AU}%YU;5V4EqTP z2&|v~r}?l5NVLdyAL)t8xUQURTSf~_;mCxn4m!o$*i$Fx8jcE;uJ|j@|MnWfKP@CM zgZXR6=-kBLp_#sITs`roZ_s zu_xOHYux?v+EwYs@^lTGpCWp2TDwcSTaB)2%&!3)s8|=5iG$Jih=0A!%MM9A!hxJp z`_M+9itlfP_O#2_9A1j3f#Z z)(hfLJ`Ev=nDNK8X3<5NwzAz6MFxp z1+7WlxVKNiZBOSqfyu}_U+JMvZck(&*0wMahUN1@O+vTrvvOlBSG!ng34=@qJp*g~ z;3(SbEh{U*{za+FSRb1-+1`;aa1`U&FR0|K0x8I2MgmN-Wgaw`-RNCiUXXRc@oi$O zyZ>7X5~C^G>sd>6Hpwi0tKj zPcVH)2el#mpFhII|rn2x$%6A z_cGf?S*?}(cF@j$FCvDaH1Q!R9ALs)$T}_Fm+_k5Y<56`ye z*9`VbIFuI>hQHqgOQRp`#+3QX8A3(t|G-y^4<>U!D?%I8V1FHA*IKr>b zcwg6-dqZPRNV|{hvcHl_nnAccG)H>K$CvlsYrVT-=PCq&itG^HerZZA$aOqWuu%GF zA&HP(c#UHy|rB9LYlY}EOCKT?rZgV`6mvq++gdLo1X_htWG7Q!l=#7rU z&!E_^uw8fpR(VZrdG@z6AJ@KdvaP;%y5zhdS zm!182(E)mO$^b^Ygbm>Toe4p*a7$C5?EGbDByVXQb+qkvYej{=?1z6#G3AyNUpuH- zF5P)DjFq2$ghF9<{e+L6A>)e;ssjqtxSyv^x9Ye)iQ?WJYb8*m*P1G}YmDUy44rFF z$(1cO-MJ|B$l)5y8ocr0^!R>PY|z^j#-y_tPmEIO{gLTB@`F|ONG0u8jeKs3zfb_o zeE$K-NeVNx2&*H2IZgTE#e8fm1|-Ybzjpu~g?aDZjr`!E9Yc81k!~;(lZV)~Lf5bc z;_afZs@VK-4?>iGLC3mg35pn2r4~5LAE_^G9cIa7SlaB++7Su{$y4G&BYOdZipZlU z6szy%St0;JRSB`NL!KLeX3ZBzXDz9&&-a6b(%QpA-+!H7Sc!LsUzGW&kTnWmZRkFr zz(~Be%E1EcC$0e|X%-q9udF7!;<)Di{i_EdK@Q8Ak5kp6;gtUXNe?o%Q1$1imR(1H z#hx386eNTbFeC!{1`Qy+!Ny4?ikOhF9Uxv6)(dw4&sc0a8Ui0j#UpnCzb&fT=Hj;B zE8J#6v2kz(KiJ=z)||SB>;gyq$&N}P>oMM2N)!LX0-)t;_$`zQsWWq^cXia^Ztk+X zkOk`WdRTH${0q+gCbx_mL=E^bxK0qxkN}m*Ya?YA#BQhiTEHPxQd0W<<3}rS`g)HRU z5BBzm!z}C}4qzTGoG*~7N%$=sJ}sH5wq9t9Q_PUNxj@acur*v!@rC#*Ak-kaS9e-E zRgsxhq_cgPXW-yF7`TnXp!9v#py_C#;O)1UmzTDHKZJqc?vSd*wGUcaZ?tfyWne1! z;C|wCXgE+?_LjO0z&j1;bAxCBKn{K?ssIG{Q$Yd4)YKIEcn3fv0d|Gl>?4}XOscK5 zH88nShdJ*1R5sI*ZHu6Sasb`}A2R^B1c2t))kkijk!cC8zp^#QXQ} zBVtq1)|R~8V%hijYK*){y63d=gq=9zW~G%?RFV|3I2^a1ajL(W)Cw1EVw-kN{!;C+ z?Yf320(b^gaygsIUJ7 zyfOx$bwUlfy}JuG2!4kf2Q7YJTLm5B$qyG9vT2k-z}|)Z*y~aY90b5>B#W>0rQC%T z@;bTq19;7;4O%PrN3yD}Dd;yC(zZe`H+7XgPLe!SoCI4rHLWg@^st)aI!7~Xwfg^dA_JhpdsKx*Wa zxDFF0=AJsmg@qw^sgaDhB(u12FU3LPwv*nU-XDBKs;%HUav2P=JfC+|)zl23{SCW# z5jc_B(3(Wpk}K;V`!JArFjmG;@Qk1+D@hwtSV5Fp@o zw9FXK)~w@&E*+hf29SaQaw+<~xUB4IT<1U4N8MH>r-)>OPGXn1EYVfycf17Zk{F_I z)SYI1xoLc*-^t%dos$Bd_X(H1zeVkdP*kc+hT%y5PsWLcWO^1BJuup<20wCr=5{Dn0QuTDIcT{^GvJyPrHsffPG z_a@=x5bA2y*~Q}5xbxeG>56HY`z?263-0cVWgm}UP+hK1RD=TK9UBO#u-Jhe19%)s zicZhZ*pMM9maQ0-BGov9Z-g1WE&916m&K z76Q_tit^Q~SL(UKqvb@S9Aok%BqTw4=lrl@f|XTOLH=Pn-~GaQC{I%gfT`e33Cx}M zMqBy@2Eq{Sfak!hq?y(aarzAmK;TXoegJ4_D>N$VFMwIVR4<3I23D@?(WCo0q}*{bmTh#FAG6F1=7Gu&rj5g(*s$4ZRru}75t6$^-aQkm=_ujWk*#LAge14 z=Ww37_251h{*wh9?knKUq+RbHD>fpP$4wvs47cDvo7Yg?sBn;4^W7myFoEDf6y~4& zpV4 zooDth$lx0hiyPRT%C)W>YePB7(C>xN4wT0qjC#I(|N4Md1=ceZR+GS&1g7HgtREe) z23dY=EiN_#7qOtQP#dJul~hz1A*OgPKs@@BE}e|X(tx_tOM3eE1j0-}Ffi&*Mbu_s zLBU-NWXeO=#(?UR1#C4k`>7k2;It^Cc?2o4AU_{s3_Z*mvAM#q;W7Qkbk#8xS>`~0 zEwY*u0=Gu2*GfwGFoAjj;3Wu&C~#r7Fl#-8&B4G$dvzeA&89@96Ihf{KzD@sW-`Q4 z@n~irhn0s!v4-b_QY;Gp+H{RUmxvH zq+I+{SX}%IAXMNu^ausE3KZ{+kc0OlV9V^wnh~H2lNM1AyZAZ*ESf$GvM{ zAgr73O3gw=K_Q}^0Tj`}!O19pGU)=(2M!=CEV$6gcanvu0_hlb?x;RI6xiR&b?!W{ zM}b(Q1wR1Y*@LkxL1E#&#Tbj#9C%ahaCPb2SQr?CCTRv*T2MYEXc*qRbt@5AFtB=I z@!rP5LTnbD$$Xpi^!0hSN4|otf-V?&I`?npNUDSYcd0puxclQZ)Xi}#KO_skLG}w1 z2--&=nazBH9l2R7EF?IBX#*9RCRU#iVlo!QdwT05P}XeLM;O4EF>Tl-6|h^t5CoZ|!aJ;cdUz^ongf*}5pu|s4^0A4Dj)8sNWc*PqO%~fP>hJum^nK+ zg*!fh>K;mgP^dSA-8Ucw+#_K80uO=Wn*9HJ!chkB^dRja?DbT5GS)!OzL^UnFR*9` ziHOjrXBL1p053-miZUPuMzZH>1cR~wF!L}<3$&Z=FhA}jEz5x8qh4DWg?JRB7nGpw zgl~{oLn6-xONtv8bUKzm3<0U80C)*TSK>e_q){nOgsjkJO;*y23S}c3BwJdmai|Sl zcmI+Doz@e!EmeIl$h*NL8J6*1C=J}E=Zf<4Km2R(LiSyLK0WYxMc%!8oi$hpm{UN# zhUu}HjWq%%`d%pm4A(*9V?PyqOz#8;{RtrxQ!3Q;VRQq9jke=Y@P@zq$BDH21yoEB za^Q1j;Q1}vassP)pHmoYn}4^rXEh#}3}&X>Xb!n?A%C}5#Gp8 z`k+uPiv#)xaD^gJ?t(iAn3zII|C8;>AMylOeqbH>w7l;tcl1QZKqL~einIU!>iRo1 zMdadg2JyC}sw&|O?NCc`d)$hL{k;be%0)<7@g-k$TSj3B^3U@z7`P{CFW=T%V+tQgLd9gFspNp2@HzZThR2Hx{65vd9_NMb6Id zn4tEEWL^WQjpB>rYisxS5A@oDN!x5b#yrMA`D@J5Ra!bRWX$1kKxvnglc8^p`C7xB zm*Mv9fgEX1N-r~X3zHJHcOpN7b7)@TQZ8?-Eyi)RS&vPqb-g6h(${V{IFk#e7O^2G zFD2$_8x#1pZDp&w_wSD@n{%<6xK^UL+~gBh(;oqnX@1oJk?;T)-s$h{&%aFg!38ze zk-gxK)o71+Q0zblUwQ8i=Hj;3-F=nHuvW3ip2fvJqx$QIfFSsYOU3>ePk25>Y+tkB zvu1Z(@|7=c3Oc))Ic-e@t=`p2zxZ_RVMxa_w}gRtxs6m)g<3Qk&sv9n}10|HZXvZzQo zZJ$x{#t+`aT<)Z!)T<_Ddie11$cQBgnU{)uR$CLc9^G>Ic9YDl9lA+Bpw86+opM;Z z>nkjPYq)o_G~&fXwO8osdM+0t!$XS3RcMC7sZif{PCQB8U8qRuwN%X_V<7buDlKI$ zEuyljf{RQ!O_Tpu*MwU_NIh$8(d|#fb1#B{iQ#8x8ab20x??)4`)t!rUijW4EGLL2 z%*F%na=+fc(r#_61`T~bVE@baQ#x!3bS zUY#VovUBIn2__eJrrea>{=frgk28lY$LD92$$w|%=CwRm>aHq}ewNv6F>FrPpvKWk zeV(cz$pc#IkG9{D{O4;VQk#OXKj@O0?d_e(ETNPLThigOo1Gag5+ARPdt)}%^Sc?{ zKvnf?nMLE46PyB>h5f*XSp^17BU|Wl!vHj&JxISpwO^4pito6@ESx5%Qnfqw{ zR6-r<%=mY&7mb>4Qz%tYA9EihQ&7yyXUd82YJGWy`oF7vuIS=Yv(y=PX)Ld+Ucq?x z^pHgT7-~jSnndJV=oFJ5b$t5g*{A+dj4+mu2orD^>S`b}u5D#w2o0Vr}$FfSOX3 zvb#k}v+X;t`||ECJFe3Y!TRY)a6XgsjQ_$-@?!or5}eNCxT2|Rr;9`2FJ=D@#1hx3 z{t2s~^5JpuogwR2#yRIHBFECZj6_Q@t`|$)SsdSbaM_NjPkR{=+TyOaQtjiPP@CPj zL3i%TyjOP~*dbnm{mk#j54pzvs_^>aS-1*D>vOecOqZCre@pH29InQ1r1<0SWA7q? z5o-9XZ*2B#GB4O(svB*evfw zl?|M4%?D$b87L3>a+O)2CiX7`M}(xaT|wnonyntF9(Q_=n91{vMOms*QE9mgi%Y@h z|Nmt5z)6Qa)ofZ)kd${lUW2Y3RtZf4g=~Jw zc~;Iy>#*ihNpnEL+M14;Y)s>^pDSVE_xY@ydBY-jBJb9&Y_)fn9j;mKZKs-jqjlw; zrvV>LpE}$4GUpo2S#Z#e?ypoQe;K&$DXOR2lMezjGC23PG8=Q6lQ^TS&K`0EhONQqyGz&CAJ+ahe{^sWcS_MkR8TvYfz zoUXg#*9-2|f%;#TrZtdb7<_z~3rkX>D>xur6x!H`C}c4(FeZU?>eOM;-fnq(cnWur zZ_pC-+acaMF6(smuZ-;cCGVbONtds}q#D}jb>Y?96kV`tO?A@+ieHI@nO;;q>C5EW z;A=Qxeb=&;%D%zPD=CyPE_xeF4H(#f2lA8{SdKCSXjZ$aG+ z>|!EsuA=TP$nZ~*hR^@4d3BU@zq=9?PB>AS3yXgZY4U0Kw9M@0s6ySu=sd+ct*}1X z?v44>adI-Q`rK7TDxs^jfng^Tr}i7AV&+?WbJ}loYR{M%Lm)lt$o#wx3k-?pDD<@1 zQ_3N0%F1%I=XE3Lu16}U^3yyU}Ga=;C@F|iOF=)2gL60;1T%!X@J_OW$dRwxKri1YpTU`*K**~g0ezw<1%woF6Xe6 zDhH+l8-ph%L-Vl5qB%IOE%VUshJ@on9z;?foFA1>+v#F!oWrf+IAIcsxwGf8+2x_q zCXa*&SyG4C#}!Ow>#xba@X(qFJbLxC42Q#R_g0!H!RMdM5CCr+9+C>j1f)4oZ5kFH z>$a-u?N;+fVDsE|;(rGr!+S{@mJQU9y)pU_FW?Eqx7HR{RE*$Jc?|%Sn1l>X(dF#h zSeHbA5Jxk-XKifk?Q~gTT0;tWnfODw>OOmw%tlEL5>QCxCt7HFF8D^k{q?C{P8mVX zDyDYz28}oEO?R%OTQsW+MtIvT8$U{vbPL^lojuWzhjVxNH`9%jAhT}OZ7xI-vn2tp z`TWqOdaUs2^jGS)db*8&lfN0BovK#%mT3B%^{wpQ(qN4H)#xZL(P?SXr)F^}#WfTf z>nP~7xiTqgk?uuYmh)5n_qgqbd{4lqG%2bMDphOY(1V^D2VSGA^0)s>U9%M!&TJOe z?5q(q!h!kp%wc~D3e?0U+@7C}d7LuG1U+H6Wp-3Lddl@uNC-+_1!w)(S~J3>(2#j; z-p>G){-9n_80$^mG59A&rDUyV&dveehrnzCUO|>@0wiSjkyDIhKK0k#m&A-Z+V$m* zE*@iTkd~vEnY>f>UZPme5A%%F$t7@zu@!0!&dMB>m|ZABX`sJ2Ytd_HPS{~Fv#xM| z^Ujs==*rb=Yl=hWfX6P^zeDX2DvTxFen_^vyW2M5bRkUk|8isZ0p5ET7qyLz6B^bC zxE)a`f2i9VQ&O#&X#@YNR~52xMhh5p%VZhLe=RZj2B(;%kPs!8<@->^K3rZxe!wLa z`|Nz*XpGfdJo2T4=%j_5D>ocs?ltS6md8dk7x;O|n=mmkko@kKb<$`W2t#u_V+3&7 z*W(kby{@Our>AXAt~aET<`wXf64FyzmVj3uy}&?fN1}&&)6jg+xZU3Iq&Nu*wu89G z$%Fc+J+mc&;!m0Glak^_k9(95m|aHmQHDQ|-()i7UmumNW-EU79L;Ol{b$yYuwmFk4<#wYG5=5nzJm zT?mMGLnj2f;>|-t3Q>8zhR{`Mffmx53$HhL@>)4Kv}ET0;Du=u>P<{bfK-Dz%+CHk z2-Ga(v8kIri%wo%@*;Y1}FzKjgPBFMSlPO8-@=1 zLuI9<%^)U+^g36JOQMlWh>muib}9Jfq6Z2R&iz|8ct9zJVZ)^xUyyIH+4#_m4IEFk zIpcn`ItT~t_O9jGz;Pzo%Bo{!&`YC_17bp*YlshpIe#J6a@_86!>&hD#$n0RPy;^GaB$`v6_Q$J!_pt(^ z&JF4*j3)x5igeefoWG^F{(+4e*O$8)nEI@TtXWfc?G)E_=O74BT>I{S#OS_U#8Jv; zg&r@FL0Km3M*6fpbMW9#O%Y*wf(2CUB#rpmlw%XT=s#zkuc!*l$`IbM@5OefB(HO4^fg4kAlxbdSGhR zj=?aN?SZ-H*GH^9@-PjGVZ{LWHT?ZSpb_-a#N#+|VWtB8Bm(a8M4~*9t5(U0kk3*8 zH9#yqRm4{c00?cDem2(DA>i!o*ouk_1EJlEexwcEc}3OJ zKo}AQK*B$v*F~*SjSu}DSP1tvn?Sk)b~s`~S)0!z3bPcLIwpd$J7_B1Cm{G+sMikC z?l?To`#n{7@bCZ5Hk}T#K&M&=SeLMjp_7QHvt?vZK)X*kYY-a;rx9`q0lQ`5u9NNY zwidMcMbCQ$p)->7&V`Llg_A2ZGNC)%WcbJ4Tu|eS7c+Cq?_l%8DnPm$M-AQe!6w5Y z_Iod7YtI6UW7yy*C$HVZ6%+Q>#K0gkDrV0-OW})<5bu7$!om{L`;Zrx2-k6bki%>F zvGW@i)gOH7r}V!QFD62J-2P(gJ=^<5$Y!W(((d1IRYW8YC(*YWwde25_U+@iZzn45 zszyM(Cw?e=`0x0iWu_doJw>1Q$K1lGogC(PNlxJn#afF>93>g66Xn z>781qXs0)it|dr0CXm)`{u}W5Q+?6({$gt~U}CGveUJ50t-?0`ngvTsMsGX(kqF98 zqj`nHt_^{kHRST@MF1`aS=m(yLLO)4EQK6xnBle+!@O(dB1CfnW)Y(`E^K^P=SA55 z%#Sf(6ohO_+sz4rk6-|>hQ<^pnO2quDrheyq8p*1mXs}9d;w!$)gsyEOnQ2{{Fhqm4Z7_KAAv9I#VXN{N5or z0ePh)&p+FUi1=@g1k3L1ylrZc?QQjr@tO-Zwy}Zg3i}}`sgT3EcTh4Z95tW;LSt2h z8{l8=w1kOLR#Zg1GO~ENjT*fU8tO>z!Jd7{Fj zy){-9xiVfx*pX*qCLyJ%xv(`Q6AdSthoRqJCngX>0ZFi%aJ9X|Gp(+H1Ir+)0+<&~ z&3#>+FvWhzW}&|qObQ1i{q?UHDEE{rLZJDV4^@tPdnDDHYG^}3hn!cVEKoBkX~gNg zyzfod_i(DHYC9?|X=&%P&h~C{5>ijNRw!y)TiY5ct@WV4xLoGSVMRgVuU3UDA*h9I zRq{W!wnrlW&Luh7$2W+4do9mLOyuYE?!pUQUCY^BJ?Q$5!pTDQP7I0Hmv4J(6*lX9 zZqZhRY}k<@{L^%G^RSAc*upP$W`Y0Qy4>WM*;bvzhUJP>`Jn0PT$vvD7E^uznG zNPIV&;CeP|@TB565wi^NDOsGJ@)I=s2O<$b-A{#?<1t8FZ*1h8PfCHf1qg78`lx|~ z1|VW6tdt?OdJ?MxP6&bLfR+nGdc*A=z!D(i2^cjQjTVxD*dbUq5l-j;6a)PTnK#t- zMSZ$Ym;KUiZPwV}Xn0AYz!G<6#V0uc1LDofcoG`yaOB_w2e8%*D#dLuH|3Nc<+grd z5E^o8d!Ko^li_!ND0iHsGTUw+S<2YbA45}%?o1MauqC$`wlsMy-2Otf%)J-u=f^ty zQt`Ubjz{{G{Z$SvRqk$>aXjtk4J4SUrju<44R2=r(D`rP^7C!zZAOQtqWM&XfH?nk z*(wLRTfQ-@6mKlwOxL0h@1_d<`K(^`?wu65NRIwS&rdlfHiM0y0>|u}vxYhXO?PmW z3LNOvV^~uvLLYu|cZVqe*~Spfqs=kF;ZzB$bZJk8fX4-UjkmeihR_nLg0TK5b?M^98^XB78aUdsD#n*-qyDG4Iu$&WTu=VTIc{GcmbAW zAp8{tF=b|UwgL2O!;O_e_PY9HG3ND~l?0WXRLGK|giqrn6<71l1T{^!XeRNw?5QEj z=t51jzc$?cAr2<2FxjimtGjH2F&=pGup-0i-TA+w0IBU5IQy(#oO^@EYOCf)Lc-Q@ z@n#cD;tRn1#FJQDv=UEUT^&a4{3$X8Fh2vqR&Z#$0e)2o_5#HB!KNvDLYQpQ5607S znc(esP)FNx#?|k`fUpmODR!16==MVC^a%Dr3t&WGtgi(_aNCtWFZh*JyD{6X zn&p|9XK$pu@qUPS-4V58e{HlU-&mm}f0#sf0ZCv^J}-XCda$F@NQ2+y+@;J&uJ)X? zEQ`vB9pwx!Kj+FT3`er5rGK*J+a@pXugOZmqb9#mkd*l$m;PE?Ye`1{h)X7_1>4i` z*GPnceojySa3r)t;2feC_piw~I?T2U2iy;@MAY!WaU&AcsQ2b81zrFb#rayils($4ZI{EA#RA=mVz{ zM)^B+q!VZExY+yqVRV}AF7ED^KcK@i_JrN?^%006XNeQd#iRl#O|&WzCYCT?^#pC4 zL{DP3ed&yi&rtIKcEZugi9%DfzpoFU&z)P-bq5U=hEsi~fnF0R5~XCrO#5dsDW$W9 z3QNiHu_FLA3Q9@_voejewCaC}gaW88_6<7}*8ueDFEvvoD~91X6OeanN<9H9*hl*= zEW^bLmJCq4K4bA7&-~nMN}?QY659b_3_|ZxF3?5;+=FYPOD~N20pbx>ju^Pr=!IWd z=Hx^%>fM1#`S)J@s<_}941uXOOo4KV!0r%~0IoEQKcO3Uf0n=ih7F#dJpp=d>9InF zJh!=J@0xI2OcS zGJtFpfb7tdZ7rd*v!Z`x)9)&+JYk%d&1v}&W+3;ScV9qMG8`{wFe)?eEpxKb-yHb_ zr)_yZH@eNiYyurN(+?6_Y3@*L|C+4+`Gc9403aDF^4-+fF!%?!kl2I=*i*m+1Qn=_!-ULMpv{6n zgPhwdVI4pmcTnRGD{%pK^%Mdm1j1ODBVU+)lP@mJ%!MtY0cQvj#n4LR+pZZ&67-YV z0;O>JbI}#2Y)CK=$(d^Muek@qS%*OG?><29Iy-X)NN{10z#Rkaq!HLC(2CNsQo@G? zOrPdpfuevxz`soFaUW)oZ%}V)Lm%;~dn&F)Q2xT>i3eDCJ-g-meD-od@KKImse$Y#w%>iJK-5`QZ% zHeVkjXLYD4%}CamlXLfcV`v8NF{^IRT{`@mAt&W74(n#7KZ?zt)=I?mHBZJ7b;k6k zYVf3*O}q@-qJTq|(ooET1z0c!_uj9a%H-{cK%iHeBqfWoruCh(566|g;DgZxVv zG@186Q~7b-#ZBKQx^Hq60C4se#_LXi7VBNH-CrHN3r-)BvF!E0>IUoPnk6>}SRP=) z2B;%?5UBUg%Ax{v031kvKwl$Y_iu={CYLp&n*9|W>sUB+J_9py@QpyMAXIW*a&yOm zYV=h3#aC1(&r%$gL_hGNxVJ(}q^b^}r5AvfcnP?n2c$w1vg+#t84sbnK+d|v#Kdq4 z1;hs;ru1_7d=jZTVZDIi6XM(8Z)XL^uUKBUXs^3?kIu#b+R}S=Sc7xo>gERJKs_ih zkL=bObQ3*){@ih^=E#37B`r-%9fu#sdjH(1OQXhD%&xwTwO6hpPmlE>v03)5WO^G&0z-fE{{S;t$q zodxe;VoH|pq3T10-r>8-ZIRYme?UMC4=<9R#{T=TaT(}J zK(Te}&YkD*eB&f<&SU8BLRco;Iw>eFre>}pt&d70L=u567HzrCOD9M zKih94&F}+4t=DQ2bVGES{k7oPSHnLAh+Wl?Ak>hx=euH%5dJ_D#(x}HNdf)eM@B{_ zN@yXcG6G^+zdPPc1`h-W+QAaXUtM@_Z8omu;xBmuc!=m``vJ11V${$jo2ImPw)OJQ(v{+{`IK(M>R>i8 zOPg+mef{-7$v9`t@jhB5kcpY!yc|I1+4)e|zC1%C6*BSdJvtV%*69Y;51M`F3pyY> zX@DM_qK4A~?x3e6?)&`k#6-yd{k3i`b8~RupR`kjLpj3thQ|~d5uvc`iGZyn7JDcI z!EXV?zH}iBs#V!z+`RdXAGH`v)^MG~p)&Q_>_;RP;Ktad&K#C1oaX}AHnUO$yhGCc zWc^O)doyIoLn$yfQ)m~5J!e<7>s=JK6!2OC2wT9a6P)#u0YL91R7#Ue`=|s91kM3F z<(uONnoX|`c|>mQ?4QzbMkA3!)oIam^3LzeWy4FD-;et|v^Ab!x0vskP>Wcf6l-kq z8?dx}k*FiQkB<7SD^AD5mKo(57;|uV-~138QJ>*@!kU_7Nr2M>wJmPafbIMs6ZGQ zmsC>SkB^GVfRZV6_t~Clc$kY3O}Qd~PO;1aX+Z8Iic5-nJ^yH6KoD8eBvPwM_h+Sh zYLB09(W$(>s<62|RXaJtvQ@Thco@s^$DeI_e#!fmJD*F0dYWW#SVS|MyZc02+iaxw zEpvU{?UmKRd%K5=1O^6+rlT_JwbyH}i0bS&mPg7NM9RKh)M+v`H2CwuZQzYqE!-)R zjMLtXVp^AcNB@DAc+bvU=1)KK_#Q+R}nobHnF&8kKxD z#t?fVBKEPd-QHcG(RfUaP)732>TqFU$@mhyr6S{bS`tM-L`%z@DV^NI>X|BPMjLt z7cKEx$jR>5x4%lcce$tDh`a=l?paHJ)iwqmcY}a73yx$kXQ>b$>No7p+eqpmiy(2!O2I8MxAGN zfM^kIK2xup;nG1*$kPnE`ivIVZ|IL2>q_uQX!~%3`Yu+QVtWKT463836qW2|P-3o* zJ==YbF1@P9I{fVq)|ku}L|gTa?_*3HeA4=e6k@y%M?j8nyQhv zbp_^vRCG1_A(?RSII0e4kED~8+55zU0dUIs7q>F}WT0{+`wjVf%-Qii0iLj+s%pHI zJu5F;bIZ5gp&?kULONQXN(F8{w~*@H?Yc{6$;1=43;5-qCs|YRY&nUUxRm!9b#I|0 z9IlJ#sy$g?e*AShzGcJR^m3V$g5FAzz?sLWEbSk=6W2C{FC!&0vu#{#obb|lRPxK9 z@QI0FtgL$jSbMKwxP0(iRV9a2I&at`-)>Do!+C|wcsSQ>csQQRArSfkqDFi-e3RZi z4E)0L-^<2|stJ(zS<$tglS5%)rf!(a6?9#;R<5{&@QnB8Uxb-M1^$xLvAelD?^+uZFN2#z`IlZAef$3=2@7 zR#L?Dcm1);uX%q@@}Q>-r-ZhgSvV8PFIrJZ%VFGR-!QGQi3g2(SCjJoe0Z?C{v4`>=m?iZ&=yL}j(apna$ zHTRKM!S8(?8zu0?OY0ZX=$@|L4K&o$=u?{*wo-ccA$UoXW41=QaO-U!dlJ1iG+h1y zdYvz>BW}h5dBgqXRyq`-Bt#4ufOXs1UJnX<$x3k>sy4&5f7gC*8Fcrh3t`ZzRecSu zaVrS@iAP37p4NIz-X1y~$jUpk;qx6gS5E0);V)L9`BNe8O+@FSoU7 z-PdR$wx+vgDkns`j7v9e?kR!x_4dG_k$?bMc{pS z-z=hI#1=58?LR$C(V1%&UmK5n92ls(Fpoc6P*lp=WpZ?}Iayuj$UqqQs>;gZ)8)azVv&V> zoSYVu%Z98^I2u4c9&*6bj~L&uZ1x+_BsFTO(a68ts0qQv#@>xfdgX>90H^vBnmhd7 z!eWp;__=?0`{CUUs65dsGk-Zr>fsl-2#uuat*OqEFdXD@1;+DrA3h{64-0$S+7;(| z-bRH4k0W5Eq&wTS88H{z+Axi7eo@qHa%jGHzSybrcdBOJaO%lNxUxQ^ygt}y&ra}o zxw%z$0{OPaplf>+W@u1B!Y50$SK4L3SiB@I_SX#HycKuhc@D$ME$9=kgq4~d9I#z(e2N{oY}LOp8&Fct;H z%X$i|bPlDW8{uWYNJ;5+=z8)jFU6vBt8TB?7kA>F*j!IVbuX41w1r}YJcUb)byp7{ z=g4%m`C2~?z=;LM1kuXPj!1b*`BP|Kj8xc=i%v~E<@aup#bf$KdxxZPvgBax3Awv<%q_=|;<|1?Z|yrQDr9K4D1K0!;kJ78poY6E&Bn4C$^oBnOBoq z2K$wiw$Db3O!=(%??`&5g`HY$D5dp#nUsvdMH#Qb$-$I*KIRn(k%9X9FPV%|^>WL4 z6=Y%ISU4_qRZo`Ekn89Wj1`c=Mnt)BxGLyGNLWl)>mBZe_W17IA@}WT!HT)}dkRP) zR2N!X8_LyiqI8-JvtHJ^52W*}zb<*o*^_2x@_cRE$TmN}r32aPERU}A?yo`e- zVm2`uK!TN+ltW|8!~`cpf6pRFm4*swfjhh$0bX7^ENC8dX}vfcPmu)$=SrIWSlN?Z z$DvHGQ+sc!^6t-diV)p7gaA8UBMhm8nO{1r6u$An?n{g??FX|v15L;^0O*?ySuS5* ziI&=K;qiZ-x`0t@FlBfq+O|1V8^a<6w|0(KpZAC(kImb4XJuu*+}cYyP4Yqf_wRM= z(p~=dQ(`KMpR}Hz?CE^GPk{~b9zM>o+WM1bp4z|TX%Cy^Z5-@i;}YkfShD{I3ul}k zrcxYnt2&TGdzDJEd!F+C367u$6B}EQwY_~H z_}2`G{a<|vy*{sT8&uZff`2j!k225&>c2;Bzc>#7^m=BNLYB+5THmDN;1#d8RzNid z-yM*BHaI&wBbLF@6EHX5%tX8UsPVtD9I%*$T%9>lmKYgZ<-*7*=Mk({q-aX?4Q9FB zOIA2^!5|+wa31HW&=7|v*aMV6*T5hYb_=+3!bv&eBi@Xjx%s8u>({?a0h4b9`SQX4 zbSN_HX_)%iD;Gm?cBkBq)GY+1Qz?9RpTKS6=H})iOdyyZ>Ri%=j@`ZX9X=8a=)kd# z{N>AL@w{ONkPcA)iuv?$L}(~eadFA^f5kN#AFgPtM8|%)X}Mr;GG(U6JCBBro;f!= zy9fpVnCReNlnQfc?!68s=qplKT6IJ3uEbXQ_@Gy5p-~5= zHoTOqAQ#)QE1Z2C!5tK^y8R^-@K~nrh=P*R7bG58X3$%|y_MG9UCY!7XdY3p41T!# z{e>ovSlgftC2}cU`Q!%wR&ej%pUECxbGX(8k9Hv?QlkMJ-6j2BF@RgCT>AS@(0Tq2 zY7@N;4YMu4S}+eP(%bquk-W6BA_BCslrO}up|Ku&W2aRvv)7x_PNmhyQ7KRWzQdXb zv{G;gkd?2IH+_#?F-su;7{~vIr?-I0YU{p-vA_;mP!K7L6a=Ic1*IjV8xbU>q(Mat z5K&Sj1u2o1PL&2pkq&93q#OS8x$pP;9OI6A$MuQxoU`}ZYt1$1Ts6gxgYOcx^0`Y& zO3eF;LeXJ-gYK>7-bD|sDnl0W{|rYMBFVuV|9H*T#l;1v{uu^&Aw}S*fVB7FsVq9X ziKeJbLRYz8fu%lz=ka;Q0`7PZB@vf2MkNWmkwM)m>3h0S#qAkQnJH$T0xo}>+$@Yc z!!u6xWt|}Myxz#SGugQG%t!G(5*7|{7TX}{I!%0-TJNsP8};P2lEaIt$!$^qG62o{ zKPC6m8mSFJ%kl^jqVF~c2g!uJ8tD9kh23CtN;lo%%lE=4c0K%O;*t!RqNUb|^LKIl zYu}j}MIME!V{Zc@9=KYVN-2Ic%Ak@XbNOTUZll1OZ6>NeP4^EjHb3ZtjNGH&Zl@QL z_lWBEt|;YMZyhwFa+Qc3<+#(CA50z(KC}-;OL~YaJc1(6^4Q_SmzI=ldPOQHh#;;} z6yWE-&EQ>C3DP3we#V~j4$9A8<0DnxzXIr=R@qLKQL1>! z47dCCThG+%?|xCTzbq-XFmz>2&mhK+%k2|~KxgRYaiPdweIJ25+tY`P-S150FdjA6 z|5JaAf2qoH+3KQ@qlx=P!|leo8-g;Cxf$72v!0>#o2$H}WbE^U7d?k}gM?R@Y`3AC z1}uB?PS+0|R3oC%3TyKtK~Rt0o_JN&fG(b|glEh-;TMNb4z4AxaFE4|GKcOu(`&a| zS)q0$BqZjX(b>7Ugm})y)_rPf8rI9d7%p7B%eeggq)q0$#%#`)(7G~(>&lN(y$HXF^8ryyk!6bHheKUKVvWbb^2SrE#uV!! zhCd~LHSfQ{&d%^GPaa3Z3LHWAA3p3F%Ts?$<*=byWKRgI)E=nYXn0M_!G=N5B6||) zXS351vtsuWE*Hz6;zneP4J<9)W! zBM*cAS`Obc*8iAscJ41TX_`( zw*TV-@L8X@_5DZQ@wFFPdSd&n{b=|sUu&eOC||fci>7E?+#GebMhew{R*9>V8x&%3 zO*fO8)Aes4y6^As5>e;4uKUxGXvp>QC znxpsdxX}%F&c>RWS8Qfg?(@~`I<%XjriG$)ORGMg;+dKgI}c5DXNIGeaa}0gR#P+O zysWC!n!-gFsWa8Hs=ZCC&#)xiJwF*gD=9Di$HYniyBD+Lan9QxpD7F0MQXJMK&ij8y z3ehMIb+^b5(If<`1I#t1uM+{N>ejz68)Ej+hKNIDXzK|$%`=(WPq@n~kdGWJ^QGnc zloNCUF|()0iL5NT*@RSTh%0BZE6OHkr)RuxWJ$lWpTsYsE>pnpm59?@21Cyp-toB! zKL7efp5UJ+eXL3qu)Zob0z&4ym+x*C&C?S5t>xVD(b>LISStK{>Wd6^Qh9FwxeIqw zoEoc;7#+Gut65MgO#Jw7HtoOB^d#mhiiybhUxAfy=y@?*Av%=E<4CUk{JC|vdVZZhGWXdi)uH8eG)H7w$X;sia6B;_Ph(6v|^nORvcgCG%L1UN2w zAW;@Qr2#K0oie-!VLlzhI~6^wC+rGtr>MAGVPuS*0YxWCG0yu}p5hF~KAiL3oqmy@ zKTr8eMb30f%C3u&=I3camuyTa~_J@w+J4KIr@Se8mi-N7(4-U$L~}F%FY2 zb6_xLWMl^)wQH+9e{hf{Lz0*~xtXFutJ_`Vv>?V{J2zOtQio`yd~WJz zgkT7iT!iO|wB#-eJ74;^+P_WbxkD(-tjLxY&qtw5DrPdfacKBd!(^PX*R3IKU(4=_Yl$h_NiT#i&v z8iZQGZrKszxLTw+xWH6I5b-&EFHSK|iX{!h=Gl~q(8Rp-^-J{Ix5u&I37|e?(Dx8? zz{1M^rK^P;qn8YGvO@j+2Tz9fg?Jf;D_4H= zn-1fcp6JYC&4F&s^w-M}UH4FUTytTN=2=EYCp1>Pd+&d1?{o5lRrZw}CQ9bBXTNu( zD5t1o*-a?wIR4oOFkN*@_(_~NaKFC({vAVc1=RXBdmS8B3~uq0`^m!~4IZ~dgvT#6 zX&s*s$2q(3?Nb7?At!a>Y-g~@b^zU95kg8rR?=&kIrU;-?yE_EM8OfwzJ*8b`|0F= zz1o8qT3yo+Q`K!aOb1D(<9?*fw_vIb&Lz=%!F$zj%Q3>p-tW&H^Ie!Y{prrlB%=pw z<8XbMy588<76htZsE|(HLx#5wt+Sx2+?J8~O}v$Z~QWbeEF!5pWB`)D%b zrd9m@ozvkqT&j|yLZvZliQIOfzy9_0MKBjGT&7Zkje!L6S5GLU8&3;4zP)16bvxyv z(eg~M&o@0HwUZd`si5Jv(D8(g$8`Itb$X!Hc6(H^YJ*Zn<{^Pj^>Xdk-}^ES21S_d za#|RO@Dq9-7x%o2PIlLAS;@1kf&r6rA&K4T`$k9Ot#0w9R-E)GbsiC#NUF)z4Y1d( z`tZU3CjB(iU(xR1#joVM``xaC!zup+;Fev3)x?Qo^$V64c`SRVAER^k9?;KSg1B}8 z$aY4>O|+%rn8tg3QC~?`*Etx6)NblIa(FvWE8XKypT&q}m7oiEV|0q06&Ivl!3rC@ zNF0`MM}nkyQ+epkC_=i-wKn%r2Ga-$b!9*KU@uO>&=DW0IZWo_2< zbb`_CL3=j+7^B~xHyY3AxnOSokO-zeluXEGrVJ()|*D{RH1nI`8p(V-+=x7@4#Z^Y$r zSl-JgjGjcC)Q7cRZti(JHFatI*v{_9Wgj^x3)3QKsW?s7HNJeQdw1L=wQ0d>ugo{S zRpUSd!3`;|-}$TU)|PQB0q-9kxNPgxO*SZ*aL$`#cxl|)&MuH_o}Qnd|4yugZ((YC zj5xA(VeA)9689$4?1=x0u-@!z1-SEA)FrW5{Kix~}u4zmLJIUk9uHf(~8D!l?f7QcpRa^GkcmQmoDj+ zgkw}Vm&1(K(Bq58q)|E3@=?wRKO8F`U%7DQN-&1(1+P7Iu(l2*l2Il10(B_eV`qEQ zr$m@Wb>Dpf!t<=?a1@F@>sp?dE!SC|-c*2{R7*k*4$3+_}gkLuK zA)m)=vMw%*HBqNUx_fH@7xDGW& zFdUmNu#Ci-_AE_hCT;|Jr=W=NNxyY5Dcddo_LBv5@UEtwUS6J)PW9>P8_<;%ac} zK$gTbvANnj^P{70idJg{d6t|*FSPx|+U22(w0O{(#6dJ&u*_$fHZZ$ZP$Y8rapaz= zcL5PKzj?nc{-c$S`^-B#sTP*K^hd>%+xYCV_0T$n$hv8c~RHh9rnNC2>J51 zWOAExwfc#{W|t{#KClYwa9qqSFIRSVck}9sXcep#g5k>L zfViD7g!K{BoC;;(D1XNYye;HxNCuB0N%a#ZJyZ@o-7H}1CFX?&PoBpjvA5NLDh z`cD0W80V}9g9=>6Nf_$GFJd1&8GH~-=P*+4U#q;RLW%}2CW_U-`@JL2nh}Jw`(V+E z&zNE0G)8s%fbtLD6|4Dhgy)RA83UP5VFKlyo6CoO4rGGO0B$HX5-!0ePGI-f<&~A$ z)N{3q7((HPrDF|8k1wzYP|G-ra~yqn7?OKpK=!ZjLQGl5LGB5T6!NATg8zJul zQ!oP^C+Fp9HpC?Ts=4hkArwbmNZpo=(-sMMTrAPf>R0iQPcU!v{L`bNF9?X)L)hML^;XZ2b zHIUi}JOEUOXB}NnC=ia~b`@E7qK+X{0$5TbZK}Ha=1Zm^QAq%I+)GT=y-6CQPJSE{ z#aM&q@J_+)YCzEhZpv_SS_zsyARdv?)9aoa#~|u397!}cwC>zF4t&Zh{$-+STBLOZ z0Ks4|EtCJRP7pnWvqYgm_e=l4eG ztp5@VDT{&>Qn^fWau0;J}?%;CD~eB*wKh<#&s5ilvLm^Yz4 z91g;N&JOiE_3^h|mwQB+BDi>^FKQ*)U5h`c_O2O)(`>(X+S-HJ4>DC}o;l9v7Zgyg zXF7Cd8o?rr)|2irq(K@fIUUok+iPRghlydN;|D$p5i)W*JRYOX)wxURnZqT17G;lj zt*(;0MOmAgUI6FX2r&x_dSH1Y_X%>{c|0&q3wG(O`wor-M$D4YIEz? zE>|sQI90TNU7Y#m=Mx`uny)`k*NR@$U5kgJ^w@nHhQHLtZ>D1{e^74bnOrcu|HZhU zqb^A+n^K`_p+KYEqQf7MXsgS=osPmyvq27VrO8khbFFpe*!Z+R{~~=)brnZC%SMhh z_UDS;k&ZqdGbJS}`!lMQyt**Wqd7)=b$$KA7pl{zWkK4g$wd7&G%-Oiq07k1x|n(K z6KH1`tucgnvT}M6l)i&(b)~rtz*~0hA?8FeiXx1l2!P)hkJ;}pzc8;7a3g}m!VeuX zUx_RaY=RU@HjTjZsvnQbKWr)tfU#EKo8x|xzzr#bKe%W(ICL01cP}WTKF+aif9kPZ z+InkxC?D0?oq_qa);vB--*3?gQrTBo(n{$(-Yp$UIN@-f=HjHy>orqhBF*2#tpzu% zF5FF$bZg8x)suhpamdD-Xz`61As0c%?ZqGFea<`5AW$T!<8E%xzWD5u58L29@$rA)!Lr&URggn;q4{5 zV$AgdTK3S^_A+}I&jV6W)&)SLJ!e056{!*Ot5MO{Jkb1ZEIvEOaNp+Ov*Qkb=QW7T zC}l=hKb|hv;g_xc>fpO4z9-2Pp=jzfp~?kXu6JK&bLEz5S~%-SW!u@P%Y3SrP3pBM z8)+o(SF(Nh`uDa?%UvO%Vt%K4RIgMy6#BE+3V%>q<50MW-onhHeovT?-?!+hVr#Qy zD@{qcoCn9wV8eG7O_?F<&*j=wSMR3r;}aANFxJFSy)He3L|c;;=X|1Pj^1c!!iXaZ zgvxrc6o;Mo>z>5?SFaN68iz9JJvQIrJhUBeVhDIZ2Af8(4KIH6#RB`RoMh9u(Np3k zh?~e#hrb$cOQymqyBNeU`zTlT@$GDlt*1S(WrX2aIiqs%qdnL&kH?2dI`ZcPuiDYy zE3!(_Zr^s&|F@)1R0PN!@wp+5w$$+7t$qXJ+1pt;B9*UN7d+B{V6>U>oe`LbidJp- z{Vh+JDCEP;qT9;lK@JH5$eH)cW^62|zo{sN<7bg;;*Pmym;H2O@UUR>K^SS(d8`5XzhICx##D|`ZIXsU+Q<{dJD%@|TA2!h+kb?(AMEL5Y@ewB@ zfBf+ED5Vz1#f!<+Pqb!6=n4@LsoSOHfwJ8{@n>sw9=x&cWQ7d;s@`aY?L2-qH&xKA}OTFYC=y`X9DO}hjv%bD(F#dYfT8vymfQr-q|1L>O zOLmXVIg+>S^gHO#VBS|?5?=`c#RG2Ap!nffAoWpSFNnf4$bu5#@H6XLvGqz7vg&A6&&5~=bTZ-66J6M7LCiF5ytKX*SXPkr5B)9i^gOvjD)h&6)z<{+F*dL<(Ot z&fJgk4&`ak+FRDhQ|H^+t0nW~Lm%)}*AxWpd^EPmHf$(4Z zGZ1??uMAG;u_Rpk&u5U}!vYK!kCL#Y@CHShc-7;~7yd4uSG4NmQ6K%Q5+VgV`05g^ z-oG);Ca>#A({xGl(zYUey<-$dYWbDF>@RCubwt=J>}*yFaC$WB}t(x$@0u-Szqr;cEqnRzO^O~PU7OoMa|sR za|7DwjmB|;XnDIDH~MBM1DkWq&9!CcBaibs>WW%6ct41?(>(6&&YfHFrTx@bBst5s zk@3)-Q*~(QLCe?k#r;#q%GU2*6nM0HyfMc_vQflMvee$kYr#{pWR6dGV=?Y}087P; zLz{oa%?J^%IEfoS1b+)YTII|LF?j8l+-^702|0k4;RQ9Z`3I% z0g~`=pllZsMdk4|=cT`M4T7_T+S5;TyBdDH(22=>Tnc8>eEv z(Wc`*y7eu`>Ucgc5;Y;p5~HLpRHa;8c)nC#p>u!gQ{$@RYP#6 z7PSl0UD8kB3g>8g-uvmZqFb#08_C%PQ3k>SuA2#39ay4bQsI-ZFj^=$PXRmzSw$?Z z@z@<~A5_bn_J5U!Y|-05u|h^cAr2RE6xyypu?|yn5)O}m=xmebvqQ^<1_r>ExJ62x zmrQXsqF(@YK|Ib!sL8x>{t)bhG2aes4DhF^IrXl6G^{`7S)mRKeRwEbMjtis5ElM9 zpQ{@sPoY@v92dOExwMns_kG&l$^YX5Nc^Z1)@5eg3a(i=U*lmrV)JD#L)&RX ztecitG(cq{g+c-OsIU_saE|$stenq%>nPnod4?hO zyExi+C1}g84RMA0dK>IS_R2(E4k>Zpv%0#qlGq;2;NmcI#%|>A+1&QA>0O&vi?^{s zGTcXw6_Fj+n|HOyEvR^Y^YH{3>pzX1?OzY=AP)%^-vT%OcNr2kHg6 zDcW#q?0X5noO-P$K`rE>@jQR2;BcUNmVo^r|K0H9`Oz|m`M%-WAj6rMINTE1sNAje z^Qv$9td_^t9`!Nllvb8Gyf8?;QtX=07z#NX1*dBA+GwFyxbK@7+`m7am-=uD+~ptDb{=Q`2<04*qGxbhwnjzZzA?>5kce{XKb2Ii z23|qwoC3X`q{-sa(mp)^JBR_0{E-GkkHWVE@H1i5C9-EDNV_mU_!Oi)27VlZf*Xu_hhmyV%QYIu#wdm7P)t-j!uWr-n?1(Wg~f^f;S_V}2=l(!>7Xwh5(dD%5tXwAfC%PM zH%&|uYvysB0Bt;Q=nx}@LUVAW8RRy@RRL)I+MgHn2T!j!^b0%6_e1Oa7!T@WnI6Tm&z)iY!~<{ddl;<{N$rP zmflPB&-<5WzGbP~njh>TaTH)7eoS-hAMspwpKzY=5Ib4iA%=CisZM{hITD zGmvv`;u1k87niWFEbtJ|Fm1cwzBWPy=!kGW2gr+XJ#gE|We?j|=u27E-oB#CdY1}B zzw-y2W{=U7(Yo&;q8?qL)ZN$U5ly5LKI=Kgy_TFN`&6gA`NuKx#TKjH!3BAdc*ER~ zwSNKfaYP6xVv$_KYwPPa>x&NrxMFUGhK02jyEqhJ*7WcKxHlEV#jc=&RuIR2)iaa@vHH^E#WqRgd1Z z_sFK(5HZo77sPjxY>eKV7O8Kf{cXrOKGB@OK{(&PJAG45(g^bWvtiQyhgs)uS4y?p z$wlYjXMgitxroieY#e!bBFqch?=>$+!bOFt+~mHbZAk;HGR^biSp zZ);3H?wD)dRetES;G97MyZNDRrhVwJk+FXeMKB&`P<#8JP5`zd*danbDPA zC?t7q?yKog#lrN$OgrJ+ur(gtmxsgQJw1=ludwrYrBRO@Ym_Pq9{%mREw?J@#BhCx zj}waJOfiz}Y1V{8-@45-UgEK*cr`wuuFn(I{@3pzA?_NA)v*g0}dMTHVDS6+wOD2dz8MXOOoJQ>&GZkk+M4()Rm zwLy#%>-MIFZ6o(c0sH|kCx$hz2ni%2|>OLEe3vTgfIukH9TST80jY`(NJ1gv6)axp( z2vuI4=?!9BNg!cP0pS-Ds*Qb}kjeNj(dM5byYcsDn5)nO?om@->x>NCb@6^2L+q*z z{3GAL^G?1LaCpL!~j-fN7YbYSQvXkB^7F9F!e(4+xj<_#NBe?r0L`b|{RB z={%{4BHr2J1S(AAwy5)4v8$2|CJkx@@%5j)>E%s9U*jI)L0>z9 zfj!8rtR7@5f1a|_?am`7yYce{e?}qBpUl(qL>RCUwX!4QTo7JPLJnxr)wM`qfS52R zgpjYs%Wsl5jQ!IGr&frGGR)uop+`ni|zsT^NNe&S@k=Oe!w)eDO zVe!~(b}N-YWf(v^KXce+@LNSyRh=Z^mz=9ZS5^{HEc|mpQ@(`k=k_oA2*uoeSIU=N zTCyouEe~%!WZGDny5F8-5$^Nm)5wBOQP09%(}Duhob6lT5y^w&!{t*P@d^#_pyA_t zewhASe74!Duq!=&xvZFa*UvEY`-YU;yi9pb75n;jJx17GHJxW7`pmC7@?$P4)1%RH zd8X$&UP$N$$%OKTX?N$?Y)=V1vgoco6meDhIx0%|O+Pz#xqg`ZrT_9oVA51qfmi!g z`++^vdfb){_Mum-chA0AeT&uMAM6~^zw#o$ZtE)6Z0v?UukZHkPDjxdj=U7jjMtU2 zQAU-+*D7V#U!T9;f1Jw2OUYxqCa5mN+TQl!uX*^r$qTp*?iJQuWGS0v18a`xJ@1HZ zAN9YZtz-2Sin03rX@aO>E}IXgtvm{_FS{z1yVh`R^1 zbi38yX1?v04Sj)0cQrXgzM;e@*+@6a>ae5mQN!?GZVK@~0Jm(o%auS9{Hb<~uqkfV z_)`ii3~uu|lT$TB>?teOzfs#6{3E*YqeP0vVfR@#7LO&~zSDw3<7zd-_LQc)Kcmck zF7CK_Tc!08DFct|Nc4NZ+16gvk#w28&5XRTKZ#7_kGV1vktH3S@O5%O29C{+HIo_2D@-ln<3cLpx`GB|rDZ%}6_}Qn5xe&SB~5pwHWBi6Fgwg;5zdZaH` zuv#=HeC64Yl#dJYGtyvs8J`q)Ki^&3M%VumCpnce`3WPXWqrN((<>z00#7wRp4FLc zuM4@nBD~@@BVc_0y;*}mI~TpfLcR){_V?W6=3 zLDuk6o+$dxIsexZj4(=4JUN(w){@okFOPjlhqr+7?0byE-N4Xr!Eg5j{MQI$4QFO{ z#y@!b_~bvDrXhk;!2BSE*luBlT2l(@>Ri!nt@|#X627(f`lmW`bwjIq9Ls;kK4 z9k-U+k-oeUIuXMT2L@m1pTHSRMkhc$8; zlK6QW`ZC*y|jZ>_Cl=T7!@e!M|q`_>^AD++Cz>Y6pP51>f>NYXC6 zLhzc(AA>HDg23>JaOl$qB@LluHzzpOJ_Bi}9_nrDs;l2p6&w$Gxj24c?wFCW&gK5# zGx9xgN3IouE>l!o9CxMMkb#RU!!B?BImw+KljtM}C&2zl{ddOde+6)$>x>x>(qts} z4IbMI^gdOvq=~K1fUXa)Sgi5|3~4fkM2ApUgMx+f1lFo&(u(H2#jSQo=w0xGCZ{I1s-~#KTx)T<76kpV{sKIv2zrhdwI{~vKOdg|mUuD3|k=3;|uKV@W zk?!jc79i&vbOh6k?ak8sd?xk%3(wwOlNZ{p7!VLgI+JS2R88ja>CU+;G`^L7AROHz zx;GV{I1eNaupUcUbnC?S?5?6JcWxSAt1uZBKlqfu#m;J>C(pYHBDp@-1zChB^?Vy< zOr9D6#r{HT@}q=uCR8#6TQ|Z#=t}$p)Nz82PEKE+0TcP+*R11;9SC#1fT*?#nYSQL z#8k~l;9tzT%gVvJE6AK00fzj%!Eg0DQGs!Lqkm`j!{sUUN04k|JNf}>5a2B0N_u@9 z8)D8xY7alf25W-BPGDD_@PYYqZ8WZ%Ux7EZ*H@yIpN;JZ{JN6J72>edPk4TdCR)CP z{W6B1#r}qtFSS_gV@GIk@NYZpJc_HXtpKu4naDh$W8pPtkDs4sAFPB2?66A7L=Tu>-5xXEY7qb7>Tn4JdjjV>)=f~ zVjYr0gn~7Ixryv&O?9L6N^{9qN zg63CA^3dPkDRofl08sG}PtyhbgkrK_a^7X>uQ+V`2wR!(OawXvU6wBaZPmxQrCBDYiXgwa)y($>C_oh9vXmH;yc@0BNdmP?RAd%yw64PkeG5F)X9)R@PUq| zLu&ZyLY*EbG(#qP$yov>5=hh^VO(n0&u==I`dIh#%9=d9-=HB8!{|AAk+n}wGt-Fi zwWqlY(CyI3wfy<=1S4Rj-3W$1Dk{9dXGBW^rpHU#A=;SQh8o{-V>-7JZjYERh)qih zY{iG8q#BwWhdM*)7Z34}`im@pFM`N<`QgAd#J1<)g&=eTeDNEL^61;ln;#v&E-C*9 z@nFP@1HS}_mm{2#*V&kvM+|V*ZsCj@92;W|I{Wz(1R{&h#k={*Po8vIyB^{p++mj1 zKAAf*JZ#qq#RkE-hm|#7HTQ1@E&5zo)uVLTg`uU(;L&mEl@VxuPtXUzPeB+%cXLnR z;jp5xg0OQ9VZ2-Q*OwL_8UD!2L~_tyn59P$hBJI_b;-$vI+_(tgWq` zuf^Rt*XN({)I~anuAH%cc>0Rc159#r<#ph~eJ7#nF6;#9(1z zAqA{tW{R_)!vW3*3=SGlr0avVvq3kS4b&_NHUqi=HDA-x)8FEJvt9ao_W=`><%CgL z7b3y+>L#>Vh~ZF+Pk7zaqlM5V^O)|v&dJpC@RfnTPlOKkg*L(n+^QMaWJ#G1F)$Hy z+zQ{27C7uFgi+%>?vf%yOf5)|=e5tcspHaIjq+2Qn(ny4DDrujOuRkVf^d?POlKv8 zLg;iVE{Dn}?E-V~K4#~7E5CFcju3VU2JQG_^zAssV%^TvG@gzsl zy#H%)lm(e+?NmK~_UzdQnfN9UK>k8WlDJ-Ya+jB52l}WTr5>g4{OJRBPE%5z`+IDaYLPU4Z7N5sAyqAfuDSR|t zkgPWgMLhsGO`4+J?`2p{@?HGzek){S?q?~4$h2c4BiViVwh14t`b(r>OrAtNcN^M$ z^WK8H32mkpkEem$wbBQf`MrGkp4)xXk(bjJASXc@jOMVW2yE*U7DtKCsA%lWf)eD7 zmvu1k{_mkG#DP4V1h1-g-H&%Rt-*fq0|b1s&~75;t)9Hl!RvjI!Y+%zNPk}|l8STw zv))vFs}i=L{h*U=^6G6vj0bjXuGXy?V%k)WfU_5BChxqgl}hvYNB`=-2XNtCb#--} zM6C7kXi>NIuLcl4Rm_iwE8oZWn)l=>garnE+HG@~_{I(#;g(`owP5hM8CP@f!{OuT zNoi??^Za8htF8UKU+VeMLkj9iZKgi$$^VN6#4e>LeJfM1bwQ+Rj-o5!=g*%(-Y0?@ zfEBAF|FDMEi|zO^JFU!nlHX%D!~gqaJOzH-LPKR`^3t8{LLNJ2)vJy2DJs&RSTC^o zN+{8c$4UQ(pG;-{w_7YA#4?V!RD9+_09X(i7|Qu$Nx-QPC#s-I{L=jPD+B{({JfVS zb7O*OFf#QA|DiUlc#Nm0Hx2}8K(@N*aIS2t1DS~JgUC25jSN<; z5X&q$r=nwh4m8JUki+j>7+Ddrc0h&)Q@e0M2f2Xo@Gsc#kNBoUb`{Z{I5FqpXM5Tg zK_^=(uFH&*S}WCxhH-NI?op(zp92E}lOIbX13;1=D0L#}JBlU3SzLym%}dH-ndS#D zO?9w&$7X?J!08ONiR+MikW^eY*qJ#Xs$Ufpya?-Yco?1{Cj>Dkzw;?7J`%Cb2tpJi zV`G!NSe?mjT{MTEhv|~D%CNK&dx>Rm;i@%EQ*T0_p%kZh(g>II<;$1nUN2sxz0*kI zu~?w|Ckl<@5{i?1j-HO=$QRIo;HIFUP!E-f|K>x*u`P+Z>Dh6Qy3(Dk*->mLBF^I` zM>`}_UKV~LHjrg$y&BuR>fyKU?6|V#rqXZ1v%EY%uYnHq=Mb8tQ-nUQv7w*-UG9Nmoyz69ji#RO9+ueA-i-n57G(8bL0n)#Ws%{8X1k~bB8jm zl9>k71jZ|0NJBTO4)O|E+3K4nQO+e6ihObl1?lQf)5jar_(S;gcM%ak!GHF3F4p;Y z=bSU_iA|#ZJ0^j-OKKqt9@9rbc$pv6>E4-H-eT15o2`IGcwi~1qBP|wzoD8xhzke?{yx0!WAlm zu@|kMAP&cSN}ZgYOV-m<#hnRLa`!95$sjjNthuMb#%tYA{dguwpabdegM*v9_@P=# z|M!YDGCd?x*t!&VKLfU!tw>l?9@n8LcOzthKMLq3nsHt;+zc|{4*jruwJ>jBV&Dvg z;uF$^`P<6LLDv|gTZo;pjpuNPxjnI)ONgnclZZ7@zY34a}}RVOd#Ov}x*D@#%=O%l^Jf z%1i+@pQdJt_o}s{SVfq< zMo5lw8h3CK$uY5Jjf^Ywo#q_KaQ#Lo_({g?-6p)6(8*i}jj6J;Q=q@SJ@*4>Cf{Y9 z-7B=Eq@@`J1VWLW3~p|^(q20dPSof7^5tr2tzF80dE*`?$+L8vIIf^_B+ET?>60uS z{|#)im0QdmlP8W}^LtF>jy&jxaU(Y`|7uB4(7TYTxOy2M3yp|3WHmJ_c9HQqI^#!( zgX#Ip2BOczyYC1~i;2-~Q0d@i%>xhioz4hS#*uQ%7h6SPw%zoVsVMcgFL=Z$U zkKhb+!#{Wbe=z{!fk`Bg_MoI5Oi?j&amzizY&+M7qE&+e8!^qior91Q30;emJlj*G zIiU7K`ogeCt)%=JzOACCf=s^WEXY``G^eyje0sM&$+arJO-){WB{Wm{A@SUIhuO;C zT_0Xr54|rirI=}GV7h;gGsVNB;EF}9hsu{Ts<8~xDWko+_LvOJG|-}xJ2NE{tnu?{ zbF8(!-KCYy9rcW-yHkR-eB{q>gW z3|hjOXZ3X86&HdMTAtds@*DQiH)JICkfG!|W2%FH-rag-8a`&|nk-Pv(O%p*rt;qt zxpAkpP#~K&C&W81>dg(y^|eCxJSAaGQy%}drbud6r$TQ?|t9PjWLhGfj2uEA{H%rKd|oE-C;Y=bfVB9L@+$_D+z}y zCOcog>}1%6wR#pCWeCbjk-g#EbT73k92odObG$kOYDz64~-iF zu8<5%K*TTqJUI*Q(3U`{5sHC9)B@}xQq4((IP)y;mh9xOUDvEQIUVXK!%v@jn3-GE z+Zig8{h^YMxmBb$B7D3pW?z^e9bvxd`|_JaPigqrSBmnzyc~6ApDeH&&vd}NH{!P3 zXhN1P zdL9X)!*+V|`#A5py=Gzl^bHQFe$n`wj!xA8VhX>?kaOaa^YcfVn+bQNxwjzWVbxJ@ zyQ?(6K3A~P@^Q!MiVmh}F&H$`xeDWuB$t8nGG1(AKeA%k=V<<`Gs|eXKj~K5wPd5e zEB#q|izYOAg@mV%?d(*gQ+DeKlbmz(%zJ*xu2`Gw7yq3F`_^)24wk9Xzy6c0UN>W= zPLh*yiR@QU?)0NTDGI4L@7W)HhKs{aQ)*Na*WKUjb&6@R$;nOVJ5p>mBPTc7?dkb+ z)nv=rfRyaP^w5oWaxYYUX&wI`pCo9ImRo$j9$_SQG7;x-^9uId%EJs_3(6bbCBgQ&vXC<$hu{ksY>%N^vf`?;bTjrN&m@MUW@jPHZl6 zUR|M)YRMlm*P1&C`KhJbEz%C=_nzIVC)^$$x^gAwu(K1f&eCQr+vPvQv*ub{R3x{I z=WV&`ru^P7Sov&xW85*cPVMt#o$vg0-F$QAaznZJ+;3J$@#gt;$`-Rv+r}m zG($fI#O)eDknyqj*eL2z$89NO9Ho*n-uk(%zu@<~2tm!Bl}EOW=2IQFN{&d8v9A5^ zT;ZtQILF1|cQad~!h)1FDCdRzP;g#;`#Y&ZKSlq>cfsKgyZ`rN=`M4TD*9XYZENGVs`RPgXQM&WxB|v%~iL4;=c9aTQ*rOYooE& zMH_3QHjP_9_-5xSJyZ?MW~&#QPuEV-FV5TfY_O+h*9qPKH5tWip{yyQzAY#fw<$>~ z)}wxzimI!+z+t70?u&EZ4i=%ycEo-(THMO_7=)3u&&54_r z`WrIxJZ$~w$_lD?TE@I2UFnzUT1QhuhzV!#g_r-8I@GuG?ShAHJ(`~x3u{iab+R&N zwf(26tOmIc?^#h7-&8jV#oX(c#o{@{RJF<*mQLgfCB*uKt#y2|t*VDZ>K{bnmr%pBH@>^RbXyUgqzA@$q9m&5;Xn{-!xOqZZDw z?}U6ugD1}ivr4BRJa@U$6*u3OuuT0Dbnd@>{<*%bH@RovbMP%;$maT8d5#LY+%j#q z*k9IkV8^Jh&@&^;;giuFjt^g_s_8fQ{^8~{Q+{?bz1Hb;D{jnHzh}n~FG`x3dTu7L zaM{#pwP*fnT0AphX)1|q?|S@)O!Pip@SuAnw3}`vu_OO4XM%_HnagZcX0#M?scrlJ zwtGjHq+BTa5|qVv;#V$3iN*HkTw4#8cmIq@-KriHPA-(s2>zrO9HbUaU^=q%VaK8( z*Y}u3)WNrWly7lZZeZBeyD%Tcq`pjl&cj)nxlK91I z+v#!<)Lj{|JO%eoZCMe?#~3XU7{9V zPi?e?zWG72O2$&JgTi@o*Gy~1K1)l42w56sG)S;^+BS5jzvhWfI1|fZ&}83+jb><7 z?o?Ai@M&iabK0f#gj}}R#qg2SF1ffrYKPmtW~HgjED3*h_MN(!_Aa6$XOSp1vSE~n zLwJIQs6F)_SyN4j^jy*>miu?okn0`k-Bn$uXnW=AyQ{utVY%8uPF`QWp4#-J(Vuk; z`|~wPK z>F{L7JGx!FKHW3EZ2oQc(fSWds=di=M$Se!(-7;vR@)`DO*GN^QWjs0>z1aHdg0GZ zYMzqvs^sCH$TZ&d@B0kyHwH@-9*UE+`h8P;*L%-&Bq}EMSN*dxsXJ0nb5XIseH;0v z5Y^)YISCy)7tBLQ4Wo_Nl6oFj#lmdO!t8a_m!`R_cPn^=1?JHkYYV^Fj8p&oEPkEy z`!}V!j^@T{lPKp~zu#$Nui|AT8|v?kMqh7vPiV`a?9uPU651y~v$@muzQZj+zeV7dt^CP zr@`RoPEO18+VZK@iDNX`UyL(EX9i9-(C=J)Yx1il?9DfeOLPl8wDOld25rozQ$l$> z0(m-)6lrO7*ltM|r@bJVjF;J^hb2<3oBtFiM!1}$K1fOaaVnSX+b-a6phw6CKDxY7&=4~BzFC`>;W#^_4nzH85KGI=7!jI@0x40 z5PbCdhj9bvs_u@#Y)!zIH{OGBmc1`KifL1H^3kw z*%o(52_(CGMu|Y$DWl<$IjNg_!JaC2AUUdMW?_+0`ZXjY5Z8@V*kbOzs>_l=$J2eM zL~NfgSrs!MVfhc>W1!2Kx#4L-+CgNWN*tmgcQ0;4&m#6L@7#sZ2l;j-!tC-*O$cyhFNRkIp;Zh|6=c*1WRf8Mw00wpy}bVxF3OCIxK103eT9p zZd#9_h=v8(xR6z*MYS>`%=|=lCN&NVYNaEpDViqRZ$nB8snazchPnly&yFya_=KMR zoBE)7Z7F(diU_{Oig!v#z*}f27PB?cQo$(qug&BSk;YHBY!M_|bczZ;qmm%7u&|He zv~oTQH{XbPO!M|a;VXB%ExTVj@b|--Z-w}R$nJ5_oJ(ozAHbRNggY)rE z#xK*SG;Kp~g&j}%FTf-=xo3EY^YbVDtKhVd$1p3uYU}>>JWt>51lz-Vji&DXKnCv8 zN?gC5xxU=oKrhB|NI00M=;rN>^~KP_W?D_2s-+F246U~B-y#0XC;q$pjy0cj;moa3 z{dd3fzblI&R?;4>O_`bxkkuwHh3r=TWp2eziS1rPN~6%l@& z*-l{XF_3zM{b2m>#u%Mj*eB(X!>5iKs@8?;r7g|8rQROR?%QG78D_JXB)&pl`BFWH zk;p@9wH(U6;NZJ=@A3Fvc+q%!swEG_Q|WWlkHagum+r&i%iM=-fgcD^Hqm4LLK0r& zlUj*dwU-L}fy7mJ?!IKH{k{HWEmRg{tbMeh$6lUlQ_~DOa&qE^23zMcRf#t?xGgN5 zk2mzN7hW(JrLFLFpCUJQRY8X^G#J#ZoofPizJfFn4kG@{=B3`>7%Tby=Qvf!2hJ84v!!ww)!T#Y(g;cujbFrlL{i>AW zLL~|xS&<*ady)5M@zB;x%WQhVHjFN4?JLd(s-n1w{F(bIyk8G*|nNk=TE(&w0P}tq9z}?fZvdx5sE`JL(QM_2}<4eMCyE4rJfMK@)?}<@^pO59goH zavn2s7o-_RMz(LnKG>?S_#PMRj}Qf|{mpyYqQvjt!Qr$|oVxG8KkY(8)63C9V52R? zF0{ZUpL_;T^ZP^sVQ<$=lDe^PAnqi&I!^7p1c+l<#D7VI8lu`6NSy1lDWG}tMlGFG z_Z~iBwi0g)=?uNG<-}PMM-8_H=dr7(LzP+i-Chr}tA#NtqvrJHsl1`LPgnGIR$ZH` z4;h9K3Lnl|10g)F`?kwxIN0;?Sgthp6gWV8l4CQXyjJOqtAr9s8A&upPlyLa=tfI_ zE2-&BgGH%1v;|qn_1WX?O5LkXI)ZTT+IEzDNWS0b^f&5sGe=sRAx`=ClKY!}FYSY6 z5YX0+l*hq$#c3-j#A^tP&gO}H*zmv=J|+}+-2qyS;0C>p>BqM%A7i{d$`NRln#?~cvDVwVM;Ka@ryu8SltDmPywyqu`dfEWao2#%hzpz@ z9AMhiM3LFP?NofeazcDSp_#nA5}1Ccq&?j4QXwW%1OOlokAgnd;D$MJ%>$DIZkRRy z@W#j7qHUgjosq2s{p#(@jQ6lHiD^Da;s}*1G2zceu_8l%9%$54kmXGwY#Z% zIwF?&iqx1ph)p^=jH`+Rs>T3=A5y6}!^$dEyk~wb!T%? zx_Bo#NSbLz_HK&nY$+6zJ2DvMg{B^dvv5`l7L{$!Z+#4@WIBGbw6@Zk_@=Ra>c~g! zn+)k!u&ouDm7ZYBA`oH;Jz8F8&DhvOyc5uiU(kJHre?SolN3u(DJR6DT<(-p&NDr09^sM7mrH17dNI|$iRViP5^7pkogu4sZg5|oAq=IQE zkLnDc)MQ=Lz~vyBj|i&xL){`EIIICu z1QDmb6=o0;m~oRsT?5zvY{y7VdP>#SY72SgKzLMMYhYSMw1Awf`0G-+DqMyD?HadW;M>g$Uzy44&cqw<5$;Yp)ttPg zdu>@jknB_(nMQW|eBAN;$gpH}GzhWIczh5$w1}~sANgs%`2;c(^CX&$E?aD-lp|du z7~v}){$_Q4J$E~n`!En}u%;|!ad5f}Jc3HZfQKM$RE&+ogaCWmh3jXdb8ih(iMj8(af zL4>co0bcrk-(?qz?u!y;udFZG-<-vF2=#Q?~k$^>%z|Qw#cW z{+3(sJ*sQp;(aKu1P$etB^3cA979`(a0Ynwm5-=}{CP4g=*Zb7#=b{pj}%Fc0IcNm#ey)jw%&gI=Dq8Usn2sK24~Ecqf*d z{fl-+>a!dHDe9*G|3=V*HG*_uAa)Egc_E~rS7FbTKcm;5~QGY8p*p8;Z8fzx9NwsV+l$?sFxDmj7H^*WC#27tV z>U_xj;k2{?9ojSHPsGV#xwH`}@;I!UB+aPRWJ4|h51;|5>!_X0Xo;fk*x1k${+N0bWtq1dQ08#%l1;!h@*ApnMEfe0y4o(KVTb%6WEO})VwoZvW# zGi3MBW&kM%`PNrJ*(*6GW!Fumb}QHGwk2Aj^-)KPJNYp|P)7(fI%z%mRL7~{S_fs9>u;V}Bir0(u z-8#`-^?G@tI6CDWfSmom8)uBoHZKa?wNDO(eHX6wEAx!SIM-g&Kr;4IpUIkK1&Ji*Iu zN3fWv2&ce@!Cres(%Fe$H7EMv5b?$c`<4!6%GdW+LuF-s5AL~qy3Gjs^*RgC zoc16T-=T$K^N*1Kpizk-@nFMO78f^(zP+K(=@n2vNqe`(RwwS+wt+qkx7)fU!`{i| zI4W1vwYBQlD@H#w4~3GM^}xav@%gR1y}{Zlez+84GHGk=XdLS6oA-MCyO@VMH1U;+ z--eWw4n(rET|Uau-7y&uh^JsE(dK9bBN0*N-J9h71FMS|E0sUaPbHDxH|*9b%KOH8 z!oOCDM-~_|im(%l*{<_$`lSLY1>xkh>@rog{b#9zast?MH`Z2knN4!Pz`Dy$`lx2E zJ(d#nqbx1$T!4ZI5@olP{tXRhUECCl>8o-Jszp3&Dh?whj@gm|K`2 zmt&Eq+)W;`oo}1JFBKUT5#&G%LEfLC2Q&!gQfa=DRA{H+yX1MzCUvXm{YlofJsVAEsK`~j3m^kvo5vrC1K1XdgmdL1GcNe~lMh_56v4dHZ?}uViA2*ht zme@3XY<-0TuCX!AAe5`IM&Ck8CDjq!egy!yW-QlWfmVnMXazP|v=FnN70q^g_=W#z zFb69B!_P-Z*jQu=z1VNp8Og&W$xwxqWVegcs_`7Ef{#H9TB!tpN3WW$6nY*?JK(=% z5To?t*`KYckD>I63hsgfIt9#j8@kLdW&1}P0|1lINyqDjSfL{>zB<7q40^a!z>aijL>u7BgTw^y#j`tJ50!1~2>oyHAMUEIG^ zd8%h!zVhWDjl~eZZn|%HGUu=*Yyr)&zLHO&6fboXEHYJC>?Xo`Y6@alH0@LC%2}&% zYFaHrIl8qc?5*10TOFy4#N@p+@L_(av08z)>mLtIHf^sVgA^3LNI%5JU*yFWpyH!U~B8vI+*4T z^sL{CR3u1AWf1qkWo7}rUH}X9KgbQ-H5MOG8i0Q*6A|st&|=5!zwU*YZCe)59AEwe zbG(R|?aN>y9d_Y#W-x&-)7KjZ7XX<6 zuU3WBj8;U0J88K^YU=mnED@3AvI0BYb^(qCtXc=;Rar#N|_Xb3oarv`N(c3wi-h;MlZS7+TD?Cj9 zK3R36J$08FED!v%AsOti#u#QNCOd(=wrF)vZK{hb$hTwCZ-Vz%CYA%3>@I=ss}BP*5)(9`Q+K9Qr^ zog#U8pJQx~oi7H#8-8%G^MFw(HmkmVLjoQNwl(_p&dri>D6>Ai+QWeWjFUMuYgy^d zBm){qaJ5%-C)wE?A$AwA8>88px1eR;!i_xNKSG4DUYtgJi)qf54&vw*0Xz~xPm-#Y zRBhxOa>V=O7XhIQhyXL>G@_OMwt84E&c<@3#c1ffk#p5{UF(y_&A8baJUU@Foi|Y6%aK zUhVh(lYf`rW%&(D}M~0V2P3~hCe;#~%(bdH+Sh6E=kV}I zi{)QF*R;FDs@6=Fh8AaK`G}hSVeJcH>e4%K;*eETvcS#v6uo3UU>$HOAfK=VL8H+qc z+3)}r)tvzp@cW+5&doQ}`uLuhFUb0d_$7J}-b%I)AOLPY+=noW<>CR_Wj2T|@ln!8 zmR*JDqK5^A-!K6m99>+~)Hlb|mKO`(j&OBnWMK8Tr#*)<-$M`9a9ROgsJQr+sIA#g zePvU>OJ9e^FE%5ULufZ!13J1^ZnmFBUGe9bI4d<>y`s%g9wZpnYkP81HHPK}9H*;w z|3`K`J@6r#4_4kQqHG3p^eDKgs3s8z+}y7jE(`?sOMAcmh8@LFX3o z4(A`!Y)6tI`Cc^ECXB0WI9oc8#W!r2}`Ee*3 zKofJrYSW0R*QwnyDVovW*XIIKU&}v@Oo;Ao4}2`~C|yrH9K5O&se^QC_hTucC><+V za+kNntnyaDA|>hVT`c={D`8fRzm4ZN=r=u`M{7AsQNC*hB^T+}ra~oLgpT^3w@p^= z!LH~5Rpqg9lGAQLPfx_j;$i_GqhJB!gDs)f^S41v#%_WD8*|^(!{q~sF)1l9jo`fl zY*9bTdVu$=u{jJisyz*Ns5(r*+z8v8Mifm|LhaT;$gI{c`9|7JJ85u~9`6O$GjDYT zshFdhT3kysSL58)qInThOVQ`u{6!0jTTZ%=jt7x5h4eY5wqe9*#1hS2d_xwZGiE&H4EOt>8n~ zEe$)iwu(I)Rd5qT|Fu__cW(jWed^I{Fb>-QY0WgXG%6uFtj}b2C@!Mp=9-M%pnHWB9}PF8 zwuRXmM^?E)`kxyWz7W6uBFUej$J?OS;G7SjX3wKqV=}>QU*z_^x(3w@vu(A0u^896 zTid>L&540C#p37wc#D=+dmz8x)f?6K6FcrWupfC#I~XcLTmy)?FsxMv5Vv-wPzE8(S1p%UrjFp*q}n$(d@Q^=gs6eH|^N=^8R2BobpgQ&z5C zUQdt*@t}`N3u=&QSjO+)@B4R`F&5Wm^}BZxbF$>C&WFzUe+hx)U<~3*WW|L6 z$&dvtc!*^bdh>K;Y=42b=7&Udc+GLxC%7(_x*=tC_9Dq*tR=0 z0orEZ1cgEe;pKp36j`l;tGgIqQJSae<-hUFV3431iC;92HJ5kK-BOw$nL6I=Rq1YO zk=SqQ$59ZU*_eE~YvBux(Vj!tRHTFA1|1y_&HiBi#J6kQ%6z0;(70>wz^MQzX~>Aan{b)$JpqZe2*?wq|cCpub7 zMa#PzKqNrw@*f?R>grzliN(gqB4wpr9JVXlC-sKXjpQHs_&vck_&0-H@u{Zkvxqa^ zJ8)2HWU>-1gL|)R&~c&V-b|t88P81t8?fITRV_~VZ5mc7ptBdo2 z($yAkaf#0(##?fjE#It}LZZBJH`A-FuIOJ}|6L=~$&v!yy)hdj_s`!)`GgU3I$bX9I`g9@N;G^P)gt8YVds=JsZv zL0p|iwiE|^r|p#{$erh+%15%Bul`#h8YP8*4iVxBv|`(zgeFxw;| z(b3t_1ZUqlAJXAm#-DBjc^eSefHrIx4c`=Sn0haXFT1lAqH1Pl8s`Ic z?0{ZD6li^3o8g1^Y$6%e6}6|0Wn&@Of27`wIo6MN%j+e>-L|Udb2>Giih8^Jl z@YdE=j-Lg^ODd@z!d0dd|E{fP;%W$JIp>`Ih>Z*}{>f0!Rfcpk8URm1CDb5&X$375>%pY!phf9@b8g#9+q@*H06GA~$QpLHGh8p1`R z+FrHCNiSL&BtH-fY&f@F_SNFCe!1jur;M^7F>7pHJr5#Hrb`gXzF>;4% zMKKp04YJ0Mt6joIQMaF{aMBd1iJ(P7W(uN29wIi&oSJn!jhF6Q_qUcbVCERvGMi!3Zca293R?QEB}JTzwE5Z=8}M zsNrJ>c{Mgx|9%$ql$(+ie$r|Rb#VZOKq2m+*|inV7Bhu0edx^Z$%I&?vR&}@IUwa^ zRRo2Sz?rQ{bbrJf`}Ncm{1WXk@Jbhv`Kq@yj0XlQd<`TOY^~%OjT*i)zs)L0lTOYy zfXROT*EsySC8W)qiBDv!;5PZ9gzNZZEw8O)?gQ0-owhXuXjCk%C}N|emyLo!Gf*uGls&P` z-(q%Xt%+*o?KQvqQ!b`oGe@v9tsA5aS~NvGQ2nd`!$mknc6f;XHRg^7s*!pU|I}`t z#PK<)Q%>X)!t=IK|9v*XSCKC@#X*@~ literal 53518 zcma%icRbbqA1@A0MpTrngY3+b?Qk@VtfWF_$tYxV9D5~+W2>wa3fW{HTO1=CBgfvH zW3OYc`_cFN{XOpe@BYK%;T-Sz8qeqR^?HXr(pJ4l%|cB=LUK`E?fzpD5)gufgmmlz zCGZ#ju%UF|CyDxfZwLqPfgBk5%-0hLEh0xyCE?y0@By3HKh^Siwruj$;aS%;D3 zci}r|?L1woSg9yM>=ms2So=|6^7L%a^Hk?){&TO@@mkyb=kROvF)#J z|Ns3u%6R(hS!DHk-O*Y!;(oI${fsC5fB#z&Q+)r1lb7=hs~Yq>eKYE-*Q0$e&i}@a zkK^U9q4{*&omR?YP`A%4i2O3Xj|ymo>h77hGU`lA0%D5$-Z!b@bGEL(K1pMi5E1OOE(&)d8tISO&uXc0mSJNAY9-o4NfVE%EzP zF=|m386LtYUYI{sFM%G;;%94*E>H74-kJJR=nKttzt-*3YJ#swI|}pinH(<-3Y+{S z`u_}CE531XiP9fhHY4B2P5;)VFwMk;(_oBfxAAKWnjB?=F6O!r{r{2Eac4>^#2<^=Hzg9?&NC|*ElA0P5qxA5u?PwgB z5~Fijr>`amb!Sk1-1N17;NsXa&m9D|F7%i;tFycR$ipmryOK0i7lkRtr@DR31!smR zJD4`V(YN}rH8|asagmWCkPI4-`oGJ?hLb>HzZL9k6MRPl#{M*`pb@1@30Wi`_X6Zl z5w|D6SiW&di7P~z1V@byn+hQal-!&Q0l!9pAcOwTHc*MAgyCi8G7UWlR7mydWHNrj z@AbAkUE9WtuGKn20y72Tf)1x;aHxAr23X)qUHgHGiBB#}2gU#2Wl#}INNbY{>qkki zimMX%qFS~C@McScc-_Rk%_&l|8jn9%)y8p~< z9UmW`asAgfNWb`aD0~9+IWGzoy4SVWFsve|;OBc{)SdgzUwCj?WFAX znV?!GoP#@t)GkO2GVpz(XF_3V=^*`g;r8jC9}8{u)C~`lWSy@`=PqKq7b1(cd|KqV z6aF|=vh=ljC_^H%h+g1#$^LH&Z=UPr^NHRLNQH@UV`q2#UAk#}J@?m>amXm`4*I(N z=Nn~01Pyu6R(E8kzy>2j+Gr}UE(WOTLtp&X+p0g};Q`qIGI-)TR z;-V>kf${H_`9#)eTnpTxf9=Uil!oxBR&VFDlU#1s8L|NmKj{01Sll z0lrkW&0Ij_wS+%(6~ziE5H4GJE=NeC@xe}ykyv#F#52D>X+CmWCJ`OaQxXYN)qqq# z;f8k}Vn#>^{`HTv&t!@XTJ_Um zh^sCMw#eXBoSBPPccEk?N)^s4ZZ#nRl8}Nid;Y;CKf5F5P7;ctr)Re(eZP2DdevZX zF$C(D66P-3Iz_)e^VyncTfz6}uRW91CX;oIVRq$w)`Io)dZg<%M{E2i+G~%>gwRWe z=rP&S*Velqpp7AkCabAE1||JvT*y?I4&O2r)*@81@5YerV@T1x@ADXoL@oDrVq=uW zr5-`~3VCei;brjwQx^3mv6y=CZ#(KMZ*NeDAp@8&U3qESG*_IcL^yWiH?sIM}{LR;tT>S)yBy{8B%<7&ppjHklIDNF{w=Dproe%({- zJvvE|NLxEmHoQI-J{m9);V~c&mv~V2fZes5;$Gmzqth%PUf@LZIzIMkL`REZK?R2? zz6C)7Bs~hR{0DRR(-s=2dzb}j;q*4#-^g|fAyRVjoonB9J*ws4BYgoAO;tUW)ogx! zR9WR9D5^^*0is=di4xi;95o8Wp}$$dkU0oa7*M4JKJ98`N5<_zT~J;%FXz~mx2xBy z@fquOj$dP~=_&{Q+*8o;pd)dZ?sq`r8(v)}%7{%Te;0^qUf%zFc4*x=x1jYL2;<8M zf|e>hS%QLA*n9A95`n3u?b;qk5{ggEZKUvoOg7|L-vT~xzg@{oR||v&u*n`aEs18MZ^x5U zf)J-Hjp76+NVh{%#`QYWsjceKHz}%*AkY9)(Zj)GS&gyaryVI)(uxK}?bdZLWad1N zQUwm)OLCKwS}Y|V{1(9QtJi@1dY&{SVakOkG`oGi)bRTgLiebmp3rtc>203jC} z{td5+zNM(=i~<*3uBLi&`MK-U9LYu=o`6Kr6HQJCkCo(O?G?cS4R7e9+c^va;Rtm(0a{zqw$z zU28YJD>z!U2^OTkfhQC1LML{)k$JvfZfG{Rxp;N0+AmXgyhkg6tdn6Qy6dEkI_hZ$FN*@I;FbA|L^PnhJLVgnjP9Qhjg5VH zo#vjD!a?;-Il^=(Zi$VSyL)Q+V?LS`pzPyW-za zYvf*CG8~(Q-XoRm3Zk@U($6mOXE*G%V^NVdm^OJ zsi;|!s)WQPZe^s!MPfDwC3GU=H)Gi;T@M#0)t#HJr50FF$Iwp@lB4~n$ZN^@yn3<~ z5)sZzQy^+Xe9JqCE}H)kwYTyG)Yz~s<_7s@=+-$J0O4%k`E<sMw`UpegjBY~<-1`xr0VwIg8%@8m~0YAKdmga6_>wIlMRII`Fl0hPM<}uF7TUN zMZ=$eAYUU*om?xY!m5rhYc>a#4o{a}LRB0MGX*L$8P9QkaC7PW^8E4FB=~Bb@3Oxz6UcDpTVfcfk7dep&kwd{!X*$7r-nX#FXD&sTdEZfrOuG)YF{!0IALgSG6? z#@4V37&n6YE=j}XNUxI2&4=<`gF{K|W z{!Ar%k&-8282n93iKC*ziC*TIVr}~Mn>*P(PSY00hq{iDd$=V>v7uZ2W@n+eTgg2= zeYb{cGw3YYtmWa)Kb!DF%+E|+?BgEHuLvb5AnH_gJc;K_^Vu#cRGOatb0hnu7MHC* z)6-}maf^nKQeoR;Rnwh2)S}IEF2gTY>Z%0p0M!^3Q$Q`FRU2r8gPb?zX?cH>PSw<-My@layB48t$ zD?apBNCNb8*_4s>tb}>Lm&-htA&MmIBr|??jw-Ty0r+PeI-KShAMzpD?zLMxc|K<~ zW7Z_JPU`{oBcN89cO2w0VjHg~^w(m`9?0S|AW>~jyUcT6x{`j|j#PMQ`;35hNuEW1 z``BYt90)Y#SMD=AbsXqE`z9{GuT>z_{};h^`gx zJ7%qfpSskmEr)FeS-1DK=Q2@3ry_3a4V9|N8*?L*843Xih+>k)E^BJeWYk!}2R5qr z%zQX$y{-g^PlZ=6xPi7dSNbTIU^MfwKS<^Qp|fgx4CiKB~w$xU;h>ckl&=|wlyOG)xUpUENF)A@Dm_H-Dx zU4Qqy7x6bYRpqdpPPZyFw`a1k%;-bJOM?u4s=trY^L8tUFa!-S^b^F^@y*_5TC9RQ?cD zFn>f;n*`W+X)4lD8-~O&nfc~}?ktfS1{=@(0qqO_jJ<8!ndr~w-p*s$*rK1heg%bRN!H%?E_Am%?N0st;n>rd0~7NsV7VK- zah?~UK4w%dJa(61Tex0bI<{#OiOOaMtEA+ice1B)5AWVty-tJs+2z(mojA%TR_lDna17-7?&i4xwr6`rO>~E<7M960WU;z*OD$WWrt@KaAPw>F}|+{9<0KU z<)qRK=JI+}W_gRfKHP+)?bRaaem5Ju@ZCaB8@PuqM6D2K33pPon2L zIw#OVx>yTL=j(9OON1Q~0;$_MMkIy2rly_nFEY(Kt`aIGpNFxT-no);)ZU@9<`w9r z#lAK24FvU%u}}*<#q)cYvlF7D*d0>Zb;kYGIw0?qLf)2lsjFV#MBkXPoc1huffTP5 zk2?U!Vysw1gaZ6m>;r@J_36t5b>@?e4Qwn~E!pzH4(oq#YNlF+9!^$Qz;eV3_JQ~N z7wPU9FHZ0~j5l=lXwNqQxVp%YdBfV?jT|u`5kC7U53t)$OttW+OpJnQ}h^ry@!H0oTG zgX!E9CrpPm=UNwTR$qnyDiI7)M9H{Sr<_9X@i`Cz`6W(>$mh5^=D4n}N6oLtKgQsY zz8*oYO@Z)Z-1zm5#qRBNVa8#&DSuU&US2WTi~wkBgX{}G78W^6BJ1u;>(K6iQe`in zswfdB46lXrDJ{CoiA@1qkq;YtqwMG)tvm>=M5eOir=bWag`M@Mi{N$29_Xf~c&+dA zBrkExtvIvL^?-A+3;?QaVVLjyy6u>4=B-73Xnp=mUv2l+8tA5Z;^_2{Ob}$swbj1| zJiA`so?kmFEAL450={PemUbobc86nJr0hr=HVBq9h-=mA@^O}EW*T$c-q^&N96!`G z*DKYY^&2gsff?J%&H8%kb#pAnE5UBmIcu2mdIMw!NKZCaPd#fs~vlD(jmDDLw~Pc@Ov;0I>)E+0)#&m zc5bM$(=tCg$OO?Ru}y&^=R&%a(+j+|2Ew=Jr>Y+Mt-ETb%LV>NJ$xef#_maz3UJa6 zizbjTPRGX`nBJ-JgirV@53Ub?7dWL9oZbz%?bUq?PHX6gi+81C1I3L<3b>YvC=8b$ z1G}vJZr|@9{(g^6hrI#AZAs||TxnG6^9I+6Dmy45ub(^LAExu6ujYuU`bhTdR5i%6 zG-0^*L`nVPG68ad=*(JRsPwJ-WIbsuZ5ERI#8l(*{o4otV$Yc-y)JFE4i~fp_ z`Hj6an!c%AzwNDAv@q}I`kGG76ZV0kk54n(Xep8l&yH2O>oziT{k}X~vrFV_)4lPZ zXt$WClyc9Bl2g>SIGHaAiq`@!XVd9SET*E}iL$@Yd~2YrJdbvNHhJJ1yYuNLq>4nH zMj6fXOsq9{(H}A$`|y*g0?c3kBX93tMFp-2rW>SCDVyRhbrKN;?-w?p@BZ8)Q#<=34|2*kx-Vb*LPcP{Gjgy| zLTW$wo%FjWu~Ma=y(T=4G`>z1d3&mtBxctenHu6p_ML>ol0Vt|%eiXJDR2*KR5|73 z;qYxO>?{NSV5E0!>^ST51gPdwL+Sw3Q@vNB4*YM106G6=2qod9>B?lI=dc|zKw}W{ zAf8E@Rb^B;fHVgV5^6W7O$|8c@ekxvq9#U8Z0crRTEE%>5;RC-^m7@iAE?;`cY`f#HTrhlB_gs((Fdg zSp>wf@?Hru9A0?c4;486FJNB*L{(>6&y@n%#e&bGWu%v^Jc)KV&5mZ)a#z02GNy@; z{_bYFWM4$6<}WMJpSo}UC-GxrPp@xP@10RChz)aMg;0=_?qu`9zQc4%=#{Urw;RQp zmlv{hI0IyJm;^xB0v1FNNUAeHHQlYl2HeB3qqO~b)>z&booOR(en;#^kL<#^v^UFi zr$B>O7taRchpGAW$$QuBF?aV1{8!^)AY832uQ{j>;$1d#F_-pn*LY+iAzO28MkT;B z)h%1lVB<*;2z@v#sr*zc-99z@{>;KlazMQ%pVM7Gpz7Id(z>1)Jc71H@A->5ujp#O zm50$5Wk4tR*G(#+Bo~KR9oLw~Hxu3lc>4y2_EE}itt6Kp@#7kc3wloHnz zu8k$CdB(awAbkfD+k0bkIuI^rscb2}8vTJ&*J$yc>3ZstQ5C=fsKk=&KQ=80!C1lu zno-h#z^C^j7Cr@(Rh~V^a{szrb6Er6a8ZVU z-1?o!q~O7Ut_v(2&)4rw{BS)81DsJ`%Ow7&UdKpQF#AV+p(>YH8p_&s%OA=~z5fMj zZ~2O_peIx}+jNI2XD%AI2&3;Y9@1-zO#RvOT%p^VD+9ifXVN4S&?JBbDUzx!(fozl zKprMXJjg%LP)Z5r`|;-ZQR6$!wIG<7^z(!9s+=<)E>5vYlr=;ORZtoS{yZqLZeBeb zP*sHK8C-GWS#skQ3HpaO0iKCs>MZIS+=2UL=jE{_E&5y{%&Ta)k8_mq6?$+TCS3V` z7IatI^4x`E+o=Hx@zo!i?ApT z@!He>Onz|s)t%nP?%>Bv@!3C4pKdw%uia~WdZ~>8mT=R{N;-#_+L{es)>h=9d5c8G3bgtK^*Q?EMlI zUMyst_?21<8kcB-v03>yO}XaeUu6T!BFc~q(1 zTk1K#%ijFCx#tP9(703YwSlBgD_*i3VT*`iZ{FOOgrT#?Tvq-a%$)7g8>E@GlK7rSA zMrXi*tD_ZQ39H}xzm_G0=Eh^M{z;slQB45r*bm>d%vlJ%c5^aHc>SF!AC>z~d(ZH) zy3h60U;6s`ez$!NiAXAW!*aXNcVieo%r!+_6EkJ|ou-r)Ri<;JVcp}{0Cr#XmbO9M z?TmfBS92Ssk8EdeqzO&NhhscMemU-ovyEI7*Y=bI?$G7 zzPvv>wemQIUSR`P1f0)Qg!IIP3BBg#q-)Km2SBku}!myuGP8B>n z{L*gRTDlrA89}0`NfC40@NGj=aPkIiWnJCr?4XvBduJ*Cc|Gk_iQUmWiLb~{Dorh| zE)0CB+Br^mYK9(Ue?|9Ndvhp#hJtVC~S-AG9%t*HBhqrA`8htO~1Xb-dr zb*=GZw1yK#+t6t=Q$4o3n4?+Bek8L_=&ELq(wX<0uF}?r00#lwwM=5>Xqc*(52ukJ zOtrf{StzEPV?I)18Qv*7NbOZ`*?ethAO_<1;SJN^9Nt+htxJ`|hp6<&?AzUNt-^hU zlB#$W#(zfo_K_WJ*B8iyGz0In2)0;mCnO|XQ%ZITK-j5wlfE%pU;p}YNV!OIU5v}c z@6oRyMlidaYnw=^O-QXW%hOlq+1#UkE{?XE{K{m-JC0~@kI0hm&L91qZ-On_c0)00 zJa);W>Ff&+UFaSO7bVuTra9)f_snCK(jFV%f6l5=h0=Y%2gn|7?Qvs@>xV^I=7nzh zT8RYj-Id|AB2apr@Hd&l@!b4;t^85Fwd@#*T7zcD#Fc{Lf`YZlcopRjM&#rNcAr&k z@>BCNVe8#qLV=xi$MI6n(FavEu&a2)ECD$Z=TuQ(eO9vx+bQOZOhNszj zQmZv5{b>~VWIx}cqe6NLR*xD+W67B>=^KRM+UKV$0aH~$XQK0u8ccP@RQWLNE;Zd% z3%@^ahjHrMEFaT_(AxD%+bsSjHQ->Yt}hO%os+D^ltt6sKhgb=G?9=+T6D6Xq4pPGs+%bc?~e!LjE<2l9Q+9d`OD1 zL+7{;cRU98M^N90Ffp4v!J9fO18x;fy=MJ|)9bPN8!RttUjuMlr$4NwTBA>=6Bh7m+fkXW6} z6@gAt;$Y1&vJt2fm-jYhO?K1V&-YtR3%WXa_}Z(oW6OKl{p=*?OTbmrYiX&2)9ib1d!Tvye2iz&mB+9xa(ZP;p6AK*Wt3nJg;=jZ?{r zJi~%Y&P(i5KbF_24^zB~y;47IzXEzF-e;aT9x#bk-OhEFS~*H<{0tJkn4q}Fy?40# zD@ePA1mPPvX+K)NNTYl6o5_N2?11{Aln6UEoO=amOPXje6E>~c_gM0KY}`_PG`P`K z%wqBiC@(g!LuxrrXA(u9#31b6-}ZSM>&3ASX`pmn9r0_3Ck>h2fm0&vQeR>wT+D9R zA|WL0Rh1BhFzZyGNE&t#3|5PicBm8wpRf`hX8u^^m9UNKGA+hH!Py(X4cQy)?~XqP zo5dN{E`86EKL?eF9T6j-;$f;{-mlMyBJ=v`*^XTf=W~@9UMn9YuRpvU_wa6T4|Jme z6fn3P&qrvvxH~izxNK01*at?Rz4g_VTZ;SiN4};p?$+MS)+D zz9!*9)=+L4q=oudxuV~W6zBDVE|d=6inVa4nq{@;bnflI1byC4qCn&qk_wtT9F*77 zP@2h(1h7Bym}TE-Y?bz1o&ZW#9)xTcGbIlJ_`9&V&@Y zJ$K$-!2bRkhyW0TPT^O2%UhX``vO2|L*7l>0Eqa7gAQasl!7BY@UA_3vkEglh zh>|{XK%o<2$2QMe6|!=84%gQ0Z`qpEChUMm2UHs-&YKcCNEN5Vsfh+Ucn;QrL_xxI z`?A*p4EzbW;gyC5qrugS;g4=gLU#uA20bQvrQa+Dm>Qdl#V*HXUJv;RK*aO#si54` zB%CFJ;M(taZ)8%*`{)6*#wpIw$DCGdqE-Up0)b9sMF?P2fW2S_bV@^;-FJCKal&6qm+JOp)r)!O3q%8JOZ(TP9gC^K#pj z4L)ZGXEZK2a^X#iDMWdlJ%m;Lo2cR>G)3uVwA$Fji1jx%8vw=(FX*o50-0U?9PI<@ z?eg6v&}hr;n^(z9E5H9KMDMuyCwR^ZZ1;0d8G;A~IGiqmLOua^>M3y<7KZB)%Mcm8 z#P=3J-rL``l6J$k6xF}wHHXrJWBGMXE3fWS?#y{Gp~r4j#Z(snc*fehkcI+icLCbu zVy|8z4ta;92_KXv2QK9Aj=as6)FBgtqG#aZyt3?Lg53Xy_^- z3S7#h6NJbp;Y6{q&KD8+RDD5$dfx@?^lq|hr751wJ}aECB!f1lu2t{=B=RATtIKwQ zgZ#c6{&+hmS2fc4MFNbusO5-+)8EQ-2hQVQnpWqd$7zkFXaH2*Ej+XMl^b93Rgudw zep3`{2i(i4uB-9#gyQnZi~1!d5{Y33dicVFZQQA4Kb zn*jwiWn;U#6@ZX7q3V^k9oqb`lZvP^W;{(!DCV}38TRyefG(f?Ug%gt>?RjZ9Yd$t zhS&I>bk`2qcVE%RS~T7Bl6POhCZ;`+dU<7H1nCYa-2cr^inV}3UPR?3f#!aE*Y=o@ zYM{-&@iErT34RFY`Pff#%Log)aoG}nsdR_n^j67uJJO5K{dJ02jz1&}V~=|XQTH+T zSNTHT$2s&isYd5{$h5Q;?T@5XZO?_a3e~srk!1s5U6iW)PgV2_?6Pnu1cm2lc8fxb zwW9XqM=Yf8`W&dwr=J|{87UoVB?DD`Y#a*<%g8tA>$h(sHKDL^$F&9kWtp!G6*qpm zWf3UrvOo@|W3!Q-HmdXXoVy>jJ!Cr)DVJpi7BQ*5b6xtau{eTAKou7kcOCSV1<05Q zHvH?y6tv^UC5ZeW0e@ORvb0`a+-Tw&m1HPnEzHrE{U&<#-QxNbb6IXy0MEo-_pZ62 z2zk?vn|2axKf?H(7mgQ7D4_lye$7%k1gm|)td@T`5WOEW=9Z~LGS_B;ajEfxrFxR7 z=aT%|^kpQFhMhMOk&Z(9A*)vipH-3DQ@x!}ONhon5C*o9hl;-GVv0FqPfSc!`HmR) zwU0H8Ve~dj{aHG!BrLMduN+s3F?~9EJCyVgwq%b9PrC2kTA#$Zo>~L+q;c3fZFQiU zPP&qi9v-NtEz3Jp+u5q@U&{x zd7RWm(f`ozp6cpVp??>44QN5~3AC@dfeq&;gaj-P^mjjuH&Vm9Rs5msTCQUSy3T%X z50QHNKVu|>;ARb(UnJ?9_~}y%qgkuNqK=IYF@ z41%-|ZZ26;Bhc=v&e(acHO=71eS;%qwl36&pI*OjVJdBRI?X)iui{NUvJxINnu^w- z{|THew8ReO7CHpbFLpt+X5|go!k>*bT&{lMPdry}|6S!$(%R|fP>;%TIx)c_4}dGm_i%~(Y)mz_*}ex6 zzUrI#NbXcTdHQsR>33 z#CV04RH;N#>}oF9f8+)tv<)g>(smFyu`7fYa$=;?={{rICR@YvDrZZ1wGYoTRNrTn zH9xO$#cx6b$4xSrOpP!I1u35w_FP5nlSMC@-x&l#k{)%XDNUZ_pX1!P@C~i=ZtZh{ zhpd+EovX@xy)#_`6?HZ#0VZ?Wc@WMN!r2vYtmxPYxsqW zB{rdD-c<$}OsA}d;P=)JQw;&dpxO&<%lVRs^ALK@H(!z?=D&OUr}FW{@X2kOWMJIL z?CbOkC9t5P3u#1;Uzv*bmj>HephnW6dlA*sQYs&>Pd~o>3XroIau_C#LtIqK%+#>* z2%-)3){op=rR?l%g3%6vvV{b1K4tYc$MBb_-pmQKc&jhM8*Vo5)Kc!#FVwz=>?r|%hFbrZ#cd`=_ zLJ8$`*qmt*Y|~Jr6Bsy2ICn-qY-iXYw6ypYY<*g5(jzJBw`p-!7oR|0F;Hq)UP|vU zJi>Sy6yoBSHE{(nofvY2ZqsP%NO?B9h+(WXL~e=1RJSDA?OKzpE8!8?`Msoj@Z-ll zW=41%XyZZ|ZDH-~kSm_(F*G-F;bIF-7(Y_WtjSdIh;`pfgKFze!9$j_bNL6k7{RR? z#4l}6XEothU#~F|(L#$@sXK_jTp>rotpn*135BqhX&Mpl4w_rp@4v)pxv$SDC>&Pp z+`e4YzJ>yv1+bm`wCO4#Oj#8`yq)$GqScP;N=lp`RgF0S9|ESpo9=NsTBsO;Hr%Ti zrIbII-z_!f4G{sdTKK{NlixQj=JvMX;XW9BG&*u`$*8=V5&TC>C#%LcQ;uZh5(KuC z26PFaz2TY88`7Sn$G)fL|d#qvY2OFYfsRJ2wtG%MDG(;YCRF>~GWS573&D^Z` zxP$s^8elb8Avsb{$WRDZp@+^yE)beGUL9;V-Nw#F1BFU)O7a;B@?Cg+4>P`sNIN$! zF(i)<P1ay(XF8Pgd<&pp6f> z&L>}@vW`SvAD>&k=bylYYdB9P5x+axX;QVAbJ0|~$}=`ds@@K0g@_=p1#+h==R4K) zn6Egr>I6Re@O0Z%y?66B&_q+Y;<47Uwv&^U694cWU7M3=Y@nHmXH^8xgro^!~ zt0QIZX-Yo;x$TeMq#pn-T9p7$ndtIIJ9R~FeNIr1n#}Wn9I9XiYo#Z3o6mIM|rXgoJ&W0ROl}AbkOw0urBFgXQvGI*cC9!`81vkI{{kMmlOL1XDW0 zsj2YfhkXGMuB5!)Z4XL7_F-@0Gr@w6utbeW}>P_=S!5@7ed>ItHc>{&wsZG zPz<*ppSQ1}mEj8@+rTf?%(9^hT57<0@AL|~W4!<+%BOAgibqcd&-wlU<|CM0aByeS z=c&DkDW}fAvwvVFPQYg)P<3?^gLA7kZ{MEk0Vnp)Khzr7o~)VO&u?E&DLl%g3d;>D^@ zm3~WmNL*yddH0S|KlE^VPh07PXg=2#-6&L8Gqa?bA}4uW(HlM|srgB~u|3rY{rwz< z0-)6c=K+8P@_V6l7u{D!;t^?wiv`ik>xA|1bgP@sXVWwjSR0n(*?uSTD*4xM^VjX{ z?9``8J4~#{f?D|j@!W~yNsh&N_A>y$ET9e0dp|KUI?}zs6S;7M!w6iI#j|@$+gl6^ zWS=htMpi-#^VUNrV<>#X91Fk#^S(^=#yX#)KoFRYTuV!cj4>2F79QnisehXu)bAU~9>U5p84K5eBW@(i8u2Q?-+ z)aA6R(*e@$ggFp}lub;=`FE8H^70g-yxCAXW}e!~K=y>`JR_fddT;sUP~DiTkP-8A z!K?un&8<2Thh&xRZcL&_u$SF4QLvbHUf^c4g}L`WV4`S08vWj#{vbSgugCNV1!kOS zr12W9H0A75Kn`#yz#Uutq*o(3la>ALQuYt?-2f8nr=~c#6blp)f#@t8=@LA*H<0PADraLHv1@nvrFvQ;8qoTysS(G8 z8#TkW3f95qhi(^=l5bl+7?i6eR6f?h*sMcmLFVkL;RljW%7D4{XC92@?{h`%z_=$@n!05_ueI4KzGKY4ylM$Zom zt=0jDOpa)Q_Hwbb`U>!dM5^com=?4g%1H50C4xNd-u9(nyKY&u$_&vsylaw^K~bx{ z3hjv((*I5YZD1gLq|Ql}ae7JpyX4|{X014RiD<`DflpKPIphz% z^kz#Jeh_A(-i%6pTzr$Ai|OE0dnO-!C9>J{yin0?WMXnc zC9nKu<}yd^;Zm+|!HtuL#_9h~=#tJr>iMOJinvwoH2VHBH`~g%WAjBiR}zTn=%ft_ z>>py8l$|;HkpguXJ2*h=D;tG*4t>mjKg9iT@#+SNGjydyT{cxZvaH7I^t+fYj!<}+ zc73FMvZJh&`6Bi;I_M3s^)7S9kvH|Ke4m2J`FeIB_*ZyUExDO2>r~IyyE1;d1h?e* z0jA~UWu>*8e-B@}cay8IxcHVq@yifCJF?oE@Js5aqA+AnO1&&~Prz3JCpI@V(=~p> zy_R>5-e(~h;W86SNGfrfGe4>`o?ndOuf|QUPh_X9v7G!`n%>YFc?IS2b9qt+jqoc; zzy6W%2ukIZU$r&l%`1Z-X7jP%V%)O?3WN#Pg{IK&_`{bvjeptqRos1kTF*RNT(B03 zAAr<`SwXtMN9$evuFZcC~^^lw7?V4coD@%KCd@W$T>7hj?pvS79G z-2R}tlJ|$&Yorbxdi1jH#owfqGV_U$8T9wMV*rw2DS3UKbvLG==wmr9-zcm58TpK2 z**k{poe$TFJ1Vt$6 z7Yb^c{bpKueV$Ah*2A=|qq3-y{(|lj(Tgev{Cf_(8WDb3Ef8okh+vP>EL;ll^9Ts&ie*aFM$ z9j@p*y0bhrM0f@A{8~RN81ZeqPdp$sb5o46^;PN4OeULvNqU=igXz{pSIIBzl1&SD zgTfowBQs%Or&s^oXlhQXOqz?s{ivGf}#0+hqNP5HL>tkG*&(ZEjCB`a$ zDB1%^o=;STZk%T^=ksxl&3;q-J5E}dwmxT1C^RC&gsD^AdB+fYj-2%PZ$5Wp+I3xj zXbZ!h2-}&hh^{Q2senaY7TBcL}1_Qe? zU{3(&S-Q_l>Q(IlAQZ)-tRw!{#zHIiRil81UO-fr(`^U7p8|BB(f>0pwRG?0IY8G7*~&bM!P8Ie;l+y=S0ld(zd@UHoFE+PPu#Hs z1B@S;C0fJqaR+Wg9~5uuu;FGBvqonss;Xq4;@hIR->NnjTVSfUH$xl&YN|a-XjkTA z9qK1Bs6#RRVLDI<^+?jhF`f7Z5Ws^>iyY@s|L_LJablw^_{gSOaSr!YtS@z^#c>j; zVfLGHcYuR8!WOF3+s-@Jiw8uAc&T(@V=-gQPlw4msU13q!gaf$qQ(=4ruADBA;>*E zw=O<2r7KAqy(zJDvhgXx)N57JgWZ=Ich|_+Xdha@H9pDorw%x_hZl(1xGbGCsn`NQ zM*w5kTs|gc8$y@P0nv3R!j<(Ol?mUSQZVs=$%Ik!^KD)=y1_^)VU_CdLRV@?=$+=7ES>|>I(4#@l< z5VAuM8>BzbG%_RalUok=tw)KaI(G;bK<0mX><(MYAH>Es27%FgO}Hy-8N`;$+$uli zvy@b@*H(?p*kl}0%i&Us8%UbH)zr1{&rPX^8Gtod^UGE3&JBAg&3+HN6<~*`#oFYz#exKb2&TQu%KoettcjU(bzsWO6Z5AmL1a1}2CVJO@G|Tyv zcl(nY1yPPA_|xyln6s9T+qZ^ky;vkpSg9|1v5<7K64MVUrhZw=_4nNJA!*p+=X};p z&#ft5O@=tl6cF}}wz@rk7W4a{au9%$KXcQ%$TOmfvQ7nF8!uOdX)88KW-PG&X8u(8 zkxlmY?5!{h{~RDOmTy+>Ed3GMxtAPZ3eVf@2?0lJCVNIJd5~_AJaR5M#N8k6`=-&~ zZJA9!bTBJ{Yjb**@HRCItEGMuc)t_3Zz<5<&P!FtY>(U_ZlL-Y&vZ6A7aOFW4?5 zs_wMh4Kih12{}stdn0&0xr67ixL)sO4t&u{e0qniwp_WSfoy~~{A(~qm z5MB0DNnQ*uIQDd`;|yrZS86>wvg_u>#%A6ILn-AcX&5Yz<&@l(2BbhV^{^1!;7|jB z{*^8ni@kXVB@sT}i-1t7ki?p`wMMYweXnXTxj?|!dxdp}ZbDDbuho4%oCXgO}Y?NcE55(~@?l^j04e*;IcqOfO7tim($d`>JPGn;_TKY0a20 ztOLXJcK2VRva4ucp!~Tf*}Zk{M?5Xh<2|m}D+4OC&IDS%$L(+49=bbTeZq0I(-K7gYms*Yjf z(Ku{6|B9>&+end&(P-_sAK2F-qoes16u$T%Djr;0xlwu_`jvrykV%wonjn@Fmnrl28--POy#ENs2AV z>%-oWhGI=jJ_BBwY6N{sWHP<~@>>GmdPqQIU0dOhZW0wE3-R`1_!1Hb*r}fhQ(8Ky zv(m}AIXR256%z;{F$lCtlvJS=yD`Q!dMm6r&XeBJ(u)B=wz|v%o6XS8g$G@J$Cu!v zl`R||ufwaDw^JkQKr+{QAd~Fn?vK7E0jR-NuT{+8`1yQc#iE^aJBn#yG0g1KzUnQ) zMrw%crQIoq?w2oPFGRMxPR%prX1mLnJ^78`QeG$YX5@rfZfUHZV4D5$rLmax3Ww&F!3wmcU{3KoUwJUa-FNMR&?zFx%DvuF$?Pqe~QTU+yVs(0F zXxa&PSLS5hpY?6!*KuUPh5et)fTR)wD>RV}qWbztitCm6Ek~M8N@3qBG6IRX?>PRl z4us*U->CUS#>d@#3K}Q&xC5@l|HIZ>hcy|tal<2|OF)nsA_yuV2Hju)Qi?@~NOzCX zQX;|-kQA7LAWAn#qafYg+H5tf&KnobETo|>5Q88R2J?1%@?G#rD=ljx?!Mn`6PrUcv)U!J z(eG}b7PrR_uAZ-R>_4?kHua%R$u&|LX3lQErC+ywf0^im)0o?x#-ZeQfv01-OAT#d9FoH8dsFz&4dt9k?5hoDTUmda&C^m!k zx|+!5M8oEVFX}WSN$KtjQklz}^cGSx9_eyMVnp{T;rlHcl(IxjmtDh+;w;-a?|Et@ z4-k-eLZPqvQ@VHAH^hpdZyt$^lU-gO7CdBY9A~t7c?~Q}bhxkK!_(072MBeDxHvSZs2r6tBXMSm0?JHUFOB+78ujM!?i{2zc z=HAWV$P2Z}=O1GSUa;;b*+b6+;Zy z+Qi+TlvA14o#mATk>@qTv~v;P!H>CJRu^!?|G&cF=d<{%fPup}zo=--zhP>s#>>qJ zWc@rnAl%E*emZ$DyLWZ}ipaxV4P@4#jzyvB3$!eQ5YSaFyjW=YUyyhn4-csznR~;aKDgci-m6-}v7O9sdE;2= z`T|l9UT$vR-&O*z>&+_(;#|qHi>o!bKGiBE{HW^nsFoOh98d@3hIhOPdS$koh{Ozn zM?|j=UOMgm{k0=r$yOU3llg{74yoHGdLv<4atUq*tGBQB#0a1KqTOLqHi#D_#Ms`P zYY!3PSms4!4@15ZDEJia{k;YSQyEfhO%TE)2ZaJ+%Wb+AXM?4qZ&KOcpv<7ELZoQ5 z^X~;)N?&n0+!|>LPn(@CRF(VaX3%=Ny>?cvD=JFVFM3d)tX*aa8SqCEd~*G}&;+OE zS>@0PSz&IKtZCsk($ghYF$al^dbv8q59S|8Xx`A_fBNJJR%IpBIG8ssiOx#?YVO}% z7k*k1N4{$19T|_OeoxT$O@B>OaT<)iqH3Fay0%*XZXWYN*T)bIC)Zk_4 zr0WK9Q^I|IUuFWaaPPol*#gj2j2j3++z;}Y7x{wet&3o~kHZ$3)CI1n^E@mT`X={~ zb@;7?K6k|1hABW05teNkNf39D1G&y&-9UFn#kWtN8sziq;C6{>H>UB`L{K`38FG`C z`n{4K_9Y|i-hYP3DZ*ySSaaT`+7=YMP>yn$YKdEZ;1k4`H)fj@+}ui&O&4r@xxz;m zw7}3JUl{&9MxT4nvyPwg(T4$^9SMmN0y?-k1*WH4LvaAXkR*P$w+ov?6_9oLWwUbGPQ9hoS&l?BM-2V(6tVM)#$J)k5M7Cw)#8DGZ&%i9(Xnk}AMa6-GQ;_Y| zF6Wxv#1A+<`h$_tQ@>c@ybHS(&jAJHYgN?&k;7`w9(%siaXhQhmx@TQd(_$SaVM7s zdJYyh^3I-A{qNk_YbjkQrM)@wL^fw6U@s*%x#%dMQ)mE{p}5j1HmzpY_Zsirqz_%QfEqAN;be#3&IxK7_tK2xhcdIz*B*zbkZfhjs zpQU;HI9fQb^_2l>O}lPQE+=NB@b0n3^#oD-3CsCOOsS}5CYUqw_XnVEtRK@_Me9|( z_wgAZ!)CZrYH|HUP?e1)D^{B&X+KP&%a(fxE^&6&eT+CFmYJIWb)ox(w9=}oD%#!k z$?qq8+X4!oM4hq8DIiif5Tp#s>KV?g!+Yl@$621V)yTe;S}pXwEujQr z!J0JNu@V(P^ZgEp!po?Ud;D)FMe+hl4i9cl2Y$hC0f4B|+AiLAi6d@jmg8Q0 zRLsUHkc8ni7Ls^jVD6H4MKz8%l!imgDAqX-)OAS#!PfuuuJYYhHEj?Zv#n0m=(TAf zq_>n2zOz;@v&r9sTDr<2F?{9uD%FndA?dYH!kxEkKdzB`iSgy{1`fG-HQ;$s-dbG{ zhyAU^6L_^zyc+8{HsxvG!!`N18iKz(9FVU@*x(69z7eq(TUGX&pF4yT$x3_cmD{^j zb`8jVv7L#0G!sn8eBi%(dj8vJ|FvX*pvtHVh;T)poq@ymWyHOF9DD1z8vI@AE`A(z z@b{A4Wb4NgJ}%D6rCR^t<(^z*G0aW|Bo}v|uSo+Nj)r(4w)5VHm1z~js>x}Td3Aqw zib!ON@s*5uqBJW|y|JH*^a$y`#^Bj(m0f-RE%DIgW~HT-6*-`GkD8M|-zvtXwYIca zTU*29SA*{3Ri{8O2~OPWW{9b|OpCzH@}W#a%)}lMpZvRF`N4if9n{wEKw0V$wmoM` zHF?^~BhqRxeC%T&TvK3re&+W0O3m7zhol*LLmaMA2S*3x#PHPj!vYd2X|YFA?!26* zcfB4cCw>Fl!jG@P-U}4tQm!mE;89;ZUN*>Zvd+GbBfk5eNQpgM3f1z^ZJLfE|gnsQJopByKEy3f=TyfEKM zo<3^M=AOZ>_-!YH!N9&I%JwpozA2u$8;cBd!- z3UyZC#;<(5a0>2tG~g4_;!}25=y2HHP*4LtH5{pwB*Wb|nOV{I3OvJ?|9f!I_LyQB-5Xz# zs=4=c{A%W{9m(x?Zxt>qZ1?f#?)t_NDe-+8;*r9mnl-6!Xp1g9a7pNBKX57|wn9Hz5ETFWYx-i~k!w@2@CAinx*Obi+>}Z6ep8z!!2YOSE zr{)|~i}-2cZp1LmX#bSRWa7!X%-qzx(|m0xL8p8-u)@IG$h41IQg{)le+=x_8lWY` zUv&cVz=7jz4TCEb6n$xk)w&X0?^Ieick-k-r|NOkcknLPb9rvA`tV|5{sIA>7cWxd zAkvf^9l=VpDex^TU}xi>ddiAORS;bSE>f9WS6D5nfo&qqy2Q@gb|py%BStjW;p3U? z3)5DlaLDJuyS^It7*NDa`yIh|ea}>tpo-jg7FF;B7?Qvj8tm*8HxqF4v$GNmaDFjO z4wF=vbwWpRo)CM|k{oRyw1wbpoG{+G3kxO3Rnfp9Y!Fg^C(Ubouli1QNwz0EWciP} zJ&ET4;X%2iH2%oA=R!|$H#25a;`esa;{fou|KpnU>@l6A{z>4aHb*{Kta4(6lWg^T z`t|Nr4jXJoc+0EP3Pea2rKMHyAX(H;DW& zB-}i5$Ol?VFWS8(y>^!w$!d-6u@ckl?!r;~X2nA$`giuMZWdBALgqa`@IHmHLjK<> zvpyT>JUQ-|v2cKPR;AQU`;Ew)ZX*2p3dA#jk{Tv?L^{WOeHb79dCbG*a zbc3)@z&8HyT+Y7DCRul>8|?9H-4Jif+-TE^#hTf5NS|hZJ?cB%u7uH<$>Z7pYG5iG zwNF^ne~7bt;r`1k-omE~xC2qty8n8%?m1cWtf45A$rINx0{P`UZo(sq7-PKBHbNK7 zNRgp@>AYm+rn8m45{y~HonSj3*kb&>A)cms;x;y}A`CYJQyJ@eI&TJ1*TNp%M86Jy zZ4sC64lUMms2**&(Cz65>Gs=-Nvq%xPtWV@6Va9#TeY`72f!#04J^6b05uZdOK)`Lu7H06 z2_P*Ta?#|yw9#PEfK2Uf4d*%L)YD)_vaKW(ZMQuQP3>M%ptND#>2X@sZhuPPuD}3& z!s=V*&EM+yd#fYsjKiU4y=zHIvADcl1?{D7I_T>Dv(FjAwjmz8ueO8=7&i%j|d)-z+uyQ zm4A|+%0R*kRHh+!qvH_(WeH%F*lJ9bjsDorRJrjLp@gV{QAi+y%?X|2s%5C)1BDsO zanoQk{qJQM<-1?R;p*j8MgJy=J^tHQ>xjwklM4qGZRQDddMZk8vTaVAzZ!rT5$@^X2;B@!eRFASIGyyY-){7 zP&QQ`%KERT3b#c~@-G(}#&lDj_JfOaQnV7YI>$;PvmWNT`pClZ zx7+4?0DUGu!q65dHVCUa@JuB}M*_2)?z^)Q&u+G5YNRaJhxa@L_E;6B>d=n6e!nue$CnDb4BKCf2L-XearG4leBlR{A&re}pTUMf-lB3q zIm)hbTSI1FSZubLpRD4RNtS^0`&|XvjVkyE`YA_hf5;cd0(N^I3%(bu7=px)%#UXD z2@=DARd)j9CS-06FrTH3S;qiIT`RfRch`7ypBlo{)2>B`67uQ^!s{-@-& z^sKNiy!U{L5@Z#9VXdxJ8G|N0rnkHArI^B~)ozWJ^{Axk={;s}oNCfjJ~lmVKm9-` ziOosmBXum6_Hh>=?-5iGbAwPLn*h-}jE_Qo0XS9BXFHJBShYxN!;afHa7XO02|!8j zaz5o=zj7+&HGMKHFD?JuYo`u-8v6KD6U3zBLqkpujmQ=mtek^yr0m&6IBZ>ZEnHrc zK~?Ji?lFgjU=t660IaT)t@c(fH7Pv zf3^I3h?*7cH2KD5-e(R@Kz|A+kZtR9(OE~5PV9#*d3K$KdWDG%o@n|*&FaNICHAhAUo%iuA!YUm7GV@ z5*8kS9y}GlY%Z`+Vv2egolyL~ZAwte#h1pmOuUD0W_g4E&!n_m+K(-_>(-*W@CWDgJiijAc}tJmcZU6Cngq(G&5x+fb5x^T$}87Y}1mh^uhUID8y z6<>cjizhOi1kG(KW0mry=`#%<)fK|1K{? zMBoMz@2l0G>f3jogtc;7e?+C`8yzI@lNJe-xad3y|89y@8ht-`$nhIwP0Ka-TNCy1 zJ3q7rIB-4;KpLD`SK|X_us-@KcrRV0_$&SwgyA(>?NVr&VsE$PZMZ*cs_-TI;eI^> z2?UQA>B7+~6bYQf`2>5lN2*41`YB7QB0kWaESo;rDZg^lK&<9vEw7n zz?OJ>H{;=v9>%9H2FGe#fF*!fJUzP5H?kCw(}HDaN9nEpY4z0kF*>OJOaNtiEdW|y z`q|Tot%nHhUyUEkBC$(A`IGf{C5hjIaP~m#ZJ~M&=P&Kv{mO;Iee#u<+V~#t^yM!kKYWT3 zBiD8kJ2!1Jn*KhJ1pAdk zbQ;`9j>p4!ZqNq()vD1C)CR!X!FIgNPE|^qmkZ-qORD<6A{L)nI^N=7*LtVpzCwF- zse;?~f~NHE6@%ZKcZPQ_eC0L^NJ~idb}!><4|3Y3`DxptHS?|P^C$t+4~GMZwH%p9 zF2TUHlCYLYQ_Jybr@`)NZmF&9+(dLoLI>I{v(*d2qZh%C6hktt{L5JtFTKY;CbcBmJHnpq zZ~DC#y(VRxWY@R-lBCA@gS=2?nJ3oV!_?YlT;QdB557`Oj>AdB3+IvJ=Dv1nt|9=| zFLh(yA`ipw9}?PSIG;^a0Ll9v8sQFkV=MTPR$rbKDyDdI^V{q`_xTzQqP<|mt;g|n znsPym`5dX$T=6uTr6#kV@EaFA%q00M!0zSv`T1L{UtYtX8E1I!(Qa`tCDh~5B{RMg z2dBVl@7K`TIM^mgY_f>>8hLe8Nh@Z}OnsoAJ}`x_gorANl7GU9xz8r(*&k%u+~Gj$ zOs?ctPNu|5)~DI^y^g1&D~aj_wj+-M8uQjiu7R`EZESW!m%vL)&(m}3^}?n7A3!4j za18uag~!t4JbF}J3rsU&D_;_M?(Yl`y1_Oexp;4X)gQ(yC_0;kMy@njODC%2N>^94 z0nbgEgTs#+<a?wAPq_2adtwO+X=%*ILWpLQx>+o-Vju_F4Myi}} z5KME+?EbvNDrF`kSIBez2pA!}=DHgqzWMRCmtRu|bpybsxqo>L=m2{0k6l2QU>49V zxhn3w@aB8}$OxLzjT*3I6)}XEkBF1$ z=yeS0e-hGmgl zhEHnojv}lH=`vnvWsy?r6~=Qv!y4K?@N*2L04`Dk;8<$8L8`D_QbJp-YZzrx&p;At zXVhP;094fhgTBye7b|znxW3Fb-WdQ!w_CDpSD>dIab^UUJZwjnIW$;qKq2bcc8U%= z8o#YB@MNm$ruQa$6kuo|2LV%xW_)(+vvKRCP9(iCmDdW4T6;`n4_OY6Wah zVZC(!oEF|VDUWrk=uT4A_#1XaUcMEk9`dx>#5`*vQ#F7XU6y^vdv8M&D$R)ll1O&f z2PxPOFAS~bd)t|C6{7O-bs!D;-?Q9KJIsgqP@*@;d51MRFK!XT_j~hePj?m6-6Ae& zG9kjQto+%CoV6pW^5>VfGRfB;I1uB+`74~>?K~{8NmbCZQ5ppltS9oGfYB~aYru$( z0O^>)6AM2g6d_sDaD3~S6sWeJ9XYQEYhFHcbIV?!8{GJX!w7x|6Pjz^O34)C59-G5N>CQjRn7blYi4|wgD zFI&gN6>pHI5hnpVcG@?$E0(Xi_HUuTjK4E$3?g42al9cV?NiT<2}O`+y_k_0zT>w1 zRIH&s22YOr0VfFqy#adj41Rzj9(V`<*?v(17`pN?{T5z*(sz6{;*hKV=k2UUn`Zlj zoRs1tSMT>0(KjcD#tLhNtmhdN#dR#0X}W)ow`@M9WBPD2Wf*p`QdT;u{^}CzZIuN3 zn`4jPl3!7a|2s@)?nD33x-%sAZ?l)bxYs`{5bvGTe|9MSL;dhS10YmzN*$PLzL|8< znQnvY`PJmqw2Vz()iDsYe7Kuhr^C47o!-^GS=Rcb3clK<$~ghVs|#LBd4=h9$u=1p zuySLtbI(kO&hFpL6YYT{ai`C!UX5Lqt7GkX6(=F8M-I`anaZONG;L!Uw`G$B2}}l9 zWW6iKJFYFW0=+}*Q}~6vz)2S$d7c+2z(Yh$%^&<8T_WO?TFd;=9C>0t_F3JXA@z<$ zB-o_?zN)n%&l5jD$~&bW8Ljp4`u0vg(Vx?Q@Nv}zqsNs(n>_^Z)uZ+mz`&E#rt~^c zJ|?u(sicL+7D@^ETVk-kZI*R(#3NKKUZRYULnjZU6BmdgwwSB|7K$Gg0FHu0Y`f(% zy?n5TfWEBn(Hp!|xOYxYwNuQ~{r|91Es zEdlAIKA-g8$-!gGrdDJKy#f0e|KwnOcee=W41Q~rk9(hc!zOGQMWpC|`{Tm~QkkK> zSv$%!goxLGX<&TbYiP@y8&zQD|NSMc2L}ekje*|xMKUlSW@Y>2>M^V?|JKcOcESw; zO|81>kU|UBr<8II;ffT!hC0skhwe&#e4sd=^zW0-Rwu*d?Urb|)LV_*I$|x&*I;A* z1>o4ubgxYS!WZ|J{pUm3V9%+}$7E0m{g@o}8RkH#!}(uz}(V=)*LLy}(-nB>D`~ z7tzsqm&VV^7Ra2G4EoD#;+Ed~46Zjce)%5AmpXZ6@iA5}&bm7G|2#z6ERH71Egx?P zwRT6x%7G>^0aGqNXh@V!FlU{gvNpB;Z5<_i9(IP3f*a`HA#=txAjdsd|9DZ~Y09$! z>3jSUrlBJ{v-HVn{-IO9_RZ9*kra$Vuvn9?h3T$*`(Zhc#b(@%Ot%I5h-@nZWEH{Tk2 zgN+w1e=6Y5AIP*#8P!xxN|;ANQzS_NFqlgUC0oBJDZH{$V>41GJ;-f99Odw}6~!ScUfFi{?Zcofp|4&~3-=q0+e{M`WG$EWGgug*3R+hHJQ z8RdoOdDwWXX*L$PzEe_Zg}lfIouP2sLO!hxhIuuBE+1f63)ogiB89OOH@Q9wJ%9GB zChOnufw;#mVR5K6Ca}F*qNRalO60rEB{Iv|nx?~@4YAtya86BQM)ao(9@8nWe*)kzG!v@>oJ$%G3L*W?_5z+AW0>ALtOB*XL+)c$3T|f~F1RZ5akM3tLB!t+x;)8Sv zU0}Op-5FFfL=~CSj)7jpM;k{3Zyp1G300jkXbrcZ;EkZz4fU>bLABy`y1lQbY#ffS zC)B;V%O+E8t!Fs1l<#*mZhTyZDdYEBU7FM>XQ+K(Sa-Z$M~Q2U@<8`+QJ;W2@LU32 zFqbBt6$DrYMb5-B)eg0!pg8QMZpm9fT5XsAT~*-%LO!X11C&t`Jr5gynCT|F3#8U` ziiqeK@b_3H|4evsa~Y3(yY0%cm3m?*IIGBm_UlVAzvW2#QL^;j+evj)W!{Hx z0>7;|oP;psXJe02Ije^2$9FJCx%}x+I(JmYXYRZ?>?W?960)Xy@Ijh^s=LiGMy_Rj6nF{4&<~4Ud}q2ED@^Poh5G zaBFAhLUQs%a)&xEZ|~2R%I)!;Z>v*2-+ti#Ku@=QPV>KMq!f2&%a(dzjB-ndlm19M zEXiZuThRV69mOyZk&2BvocbL;d;0iJUDycIy30@{^S$sFb`c@^w?Kb{rAwNDb-Zy} zEgEn76#h5>C&%#T>Sm>bca?wIa4&rgB4=Uj9zOvY2s{7i|P|_lsrw6Vmi$Z zhvPEl^cDL&WlFfafd6mLqhp$5v`23y$F%L`vR}d8(8Eu(Kiu0`ak{Ui&nB5ejLVw* zzoQHenqWe>Z^B}=7bk$QrM*{o8ynI@DJyxr=`rjrpW1J5o~MM*R{A#q^28e(g=(WR zpyr$2L*_+Y6f4g?qRIC~k(lpr&O%+&r4V%Do>u?fNER?~|7Mfv981 zc8Y@%9^{%j?SLE)*`>u8mH{&JL-NA~*w|auEv+qNJo(>GZu3j~P5j??1S=tZM&eh? zH4AwrRYO#f&)e*l4+=B-qh)RnhV#_hgrY8G4k#c{@*Jsm%2*OMleS@G8-z2K^O>6> zUCAEP%?4I?#rU%ZrCdm{^MEsF2y?m_oZaPp%EhgIfda-$Bypl z$z4uoyH6F4kHD3XkG#~KX1t{-Z=K8}s5|g7Yv5;5HSayU0O{>tNB=mYE&i$7#igx< z2=ec-Qi92{HT}G4K*oQmD^%8)7Q=CKyQUdC+b|A8rR4qIx42PpsR|Xs_PH?LkX7Uz zCq}+&rppid8u}@PJ;px5kQZ9xOg+hnzHnG?{!e)Ex$f}Us9SyBu^}2En4sylRrxzJ6D}d?> zI8gC7iTg_fs66RAbxw)f9}h*?r}n6!7$s8JXV|K1U6Tb-GAq(G$ zndy$bfhGqzv6hM2oB!y&n8*?~?pW>qx+ljOW7D-=Gm}KlSt9asv}E3h+mL4n4YvI{8E%WYN@a5s zTGQpcQ3e|vdYTy2uYp;-^wv?sdcH747Y1RQ5?LN*`TbZ7ar5Cg6Db@vV}3&yRYu^& zg*WpZyDM}FCAv2+GcFV_f}ANIUxTp6qL z^D8lKc>VnFrM?k)d8m&QP|M{;DrHtUDT3KtB!@$oS|yaf62ua?!h9ijhFl5U{h8dl zeh-kX5_r_JLB*DHg9|K>O3}2@0UOG%qssiDFLPPKq>ic_!T8fD(9*Jfir-=FW{1kw zzk8$>sNhebJb(V?DO$)WSlk?m6 zlG%U4?s3NuJ;6-Xq%&xUe`NFg1$d53guls?dv5BDp2T(`N(iywS=g#&{p2zdu+27bJZW) z{Wo)pyPW5e@J2dgJL7W>tz{dI>A?H6>mJY>1B&NNPJ}8OC#k2;X_9 z0*iypgUfaVpHE;vOPjZU?VO-LpUFrBdx|qI<^&LpDR&VZ6>=UDxy-_a`nJZoq4naR z3)iL2F?&k16FX8RMYP&$kpk|2TjxgZQw62m@_8@g(sh`|o~_q7&yuPhL*WL8%5I?i zCm4>bohO?-wMvtUeHB7^zAKD5V%J|_=Za2y$}^XbO8L8qD1aTlL5W4=JE=S?Qg43g z|662+V-+8U2!x5i_sND|y}a)=>&|PZ#&k@sx2IYwC&4sHt$FWUstVf_<=c z7l`84kHHv^Fz-(hy5XLHan&-KsB-(ha`))>tk>e6!a2t~J>e_iCzYwb2Uf#%8}-44 zVq+*$meVA3T1heu4$@0NL?V1BbvPdqkgX^N5|3e4^k|jX^7q+ZZt^zM7``9J39<7v z#X;9$+c3MH;4C4v<(Ht_Y>CCn+87~Rp7&9GZO(9d3W$T3XZH&NEd~u4bXWMYXHT4r z_L6mm)PpQ8IF?1}Mp81MgCHzZcMQj!VA}h#y?L~M;%fTkgbovx;xOlTd+VU%=F``9 zc83|AeY{a$kK;QfJG;K6D{Wo4?KoybS^DZYnN|zduc)afItNo-?MoG{2=R`6j%Ms> zA5N-vMS8RgmRA^U-Z>{T6Gonlp)T+faMx!?2P1>C1V2(G?JaUzV)^2O{BAAb@;gao zrdq=3LVRwil!g@m6GC&v*nAOBCuHb`cVf8!TMuPZ5_xCI{G5FELE3gOTXYPgZ%~*N z>6FY$mb-ch`RYE3T7fo-s_AZk=1TY(VhxHi5aQB7NUcruJbhI2J@$N#Palmc7U;`` zRt`39=!lyqyq=ctmSf5{Dr?nN-tXSrNm4oa8ZLAL@cD?D(9{HPyn;yM>dFZu{4Cxk z68)awBNC8KXK$~m?|<8Ix8>FZwx_@O7K#6ahcTOc@q(oHi@h#LB+T-Pqb%YIrRa$ev6OKHP?H~~(Pz|>bciy{-1@8Tn16!yaQ>DdCIt$7lkw%YI$TK1)(m_Ay8 zpV76ytEwCY!Ho2Zj#G(}7vh?(H*E(U70esO+$Y@Q`e=gA47SEf%nywearnQ}`wp12 z8C&OMrd?hdviPmB#aK(`)C1y^8DGgL2oYOOdwBg_+b<_~`NH)8(}jF?LWs z)j(QIJypEgjUF2`Q(J`2EEPMm1eF(2BX0}w-~8oNb`o88ZNs7kqHz=5QuLAFCy{;Ox=S_`qS z{?!LP{1$G)=s~9t=3(~6zxT;* zQOhMafJUecpCi%X-n&h;%F`Ql?K?GymGB_l5U8{?c8Ga1lR%FI?&Nb``3^>Gyot{G z6l3a$0_|!3ggPD|EqvZ@*whI{vfSYvWFDoV3Q6lU^22+bTu^%v9oa}K<4>-QB^YVb zfOz=seOE{h)*yTW`|Zz5Is9{l$wO_T@_6mfl@cvx7=13aw3B|iXLx{UUcYoz8K>OE zJwr3;64;5rPiETB!4^ro7kf%r-_?@p0!-Y-9M}`o#PA(l;ZxJ7o|5t2bJS4iCgtZf zfngQ>Z3Ok-wSH2+?Bl=@qbhB`qZvY;6Og2k8RpRiSn)oP z=LtFI@>xG2{+O7_)sy=ZAIuHS{Q@4ol0|9G)b%V;&7$%&4jeWs0`1pR#mk&>Vf!Np z8(!&tQkk(O-?^}Ck&xAPr`2w^;(Rkt5lehkNw)-8?C&V2uDWSRy8P+b-Ku;*+w+nh zt$C!ItUo95Mz8zs*9?WHKpa>LQLnaf${yH!xu>GnXfsy~iPQg~ej{6frG2FW#T0HAA{#J z1G5~ZX3l*aP0|=TB!Iii1T7Jo1zC=XO1tWpdtR|iNe$pTJZRMi#TElO2}A?bF2Ms& zXSKP-rJBc=03QA+YGXGcLWR#(~=}8t=LRFgR5A_M=e5ft9vU|VzobS8BT4qwX`=qZCPf61J z##C>9)O|YbQ)6jDdTj4sz0)shILtqx{?2~)FL=45pw7g-k1l14l6HbDZDuM2gS}5U zKsrJmXgEQ9hpN2ox!NW2^OWU)JU*sQEEpD;yGXuh@tenlux{*5JARcuOtjOpWI9&;`(SS_6EzB;BHWl{9si;SLvE2)E^{3DB>}# z3xPapRk^3|v}P+OJ-)s64fBmWA;Jgpth4buH4eE#eY8ilWT4T;vpqwLF=hg(Iw5+) zH9nIf8w$iCyNMC8>~ZaH)YdT39Vl2Jf3V&``E7suYIyE{pQXLKFsxr(AaUvKBZ>NM#Xkb?BJKR+%=&p; zX_fDpWIh14IvUDcHhJ2i7xJXA46n7dqIMcT$!i`cY)Ku!=r{fh42{X~ZjM_97`80c zoh)efcFCNIRC$e#mF&tFWceWpYi=jgWeYKN*8my9fY@6G99bw&A*qU>ww@W%+--!c zhUdp+o7@~rNeq!&4uWt>xwxC#CO?Npj~)G9;k}z_kh3oubCRcyM0tU_qL$?G^=!Iz zwGUxG>$Lh*knw&TC!2l`*veMlW~zO+ai+Dl&N`XifVzsW0*MBD0r?-`$M)6F9E9Wy zy!V#>UiyoiWRRP{s1=YLu3>R3GXy>`97G;TeUWHEV4@$NWv9x?1@#w69k8(byt(U3 z2xwt`5g^%nnALL1`k7_$>GKEV*vmfq?R6TmJ1QRdrbdHF+X!KzwQtL15uAAfE09 zxfjXV9B#}2OVc19=Nok!?g=HoLdAcI2HQWh)Af=x4f&BqtCj}MO%v_Bgh8~na0yNl z1sK{liP%_YRn;BmNO0ov+uBt8@UP`{KzhB0e;$A?pU<-NN>ak1X#xXz_LXh2{*oT` zYJ1rJ#uS0afPC@g(E*cWS?5>HNDGl(?Q|s=N~MK!I`fqPaAsjp&;RDU>vESH*tgj` z(>VFq?8+nKrPiiSn=+@W@6FZ3tt(_T8g3DKj1Xgi+^2Jy6QYAlI8wG_NIZWAeh4wy z|5HYKpn2(9Sfwe<$3?N|!jSIWobz*G8$FK;#-M^a;WE1`$bE8Mai%@Wn6g)wDI6(r z8ACWMHJg^>t!wWc5i0cw(XBFg82LL}nJ7Wc5O}cO*5zSQ^RO_yZ(Hw-=IHC2qR*_8 zqm81zpJ6*bg&Y3_mN<&c`BEFzT&W9c%xJf=+H$zto}Z0fCjQI(DCiu!)?ZXbdY{~D zYNqbTrYcXVLTlY_M3S~OsJCk;{F9Tyky(mk9JKIcRF~WCbL+G{DY%^4`eoLu{a*ge z6eBTXeEC=5&AUir0nY|i+t-!Af$jChqxRmvukyF}tVQl`sMveivzP0(#|o6OMdqsT zNb*b0mn%cLN&0{>;|2r`|LWA0=M!(V7VLMFSY_MgOr}o|mSTzwUNDgDll+xhm5w^jG{V{LxgUeRAe1Tte?Wg8L&DjaoMM8L-Uxqn>7AbEzZnd3->Q#x-v zsG+9_b7|q{5oj&OgDfKYC{D^G(v_sT69b@AwHT4q_$0XPwEpAw*FV*}1d2=hub%eK zN4of<%T=$D%Bk{g@4>om#o9n_=3bOUl(dVsNoz+Z_{w-m(x&63Z*D;r7A&gGWD@u)b<>^R2UTd6f z$_EYxKv4p%1vMb?o*il=T3qM|^z?K9)3}^V!#U6v(0!)6+L0-8R4c$m+WC2G%CLdl z*7;%HM$=HfZtHy=13B#Vko%+f_KrE+#+YySE^jK40PN%WJ>X^X?eEat;HEFW1^G={Vm(9ZuTh|V^ zIP2(Cq?ou$b!!AD-gQc@NhDa!07BPtEO{?opv=GFDuvd1E&(r|ZuTAv8TXLI4bU27LPe1)bn?Q^c)9Ng8Y4_HMs_fL) zx;FqhK=IuFB*glk$7@oQ`phcgXRf^~I;MT9Re}4lkDg+)Lg9Cex7VsKCR)a;L|Kj_APl*RX<@4X* z_KfZdr{8n=z4F}J6uY-MFQ;}-RjWz6s@~&iwAuhv!&mI!+L#IeHvjuOU)?acrFFwe zIXCO4Vpduzic>-43|$(EldBnqk(;^?weDQ0Alk}&A=f8f9mtp`dKI~XCz*)?Msz_d!Z@TZ4?!X*MbAn2fs5#yBi(aE>&t6 z-)^Z^($7~#0lM-nI&GfoQ0+;knv?0q_x###zSl~1gzAop*!rzm8JK|IYOR5@OBqne ze=kukHeme>ru{tF(u0lYrZYky4M9!N5X0iV`?Q|=b)K8RtT{at3Rj+zx|Vg&=Mz4SJ3IMaScZRUpbJtXX-`Tgj zceRBJ1*Xa0`;h(-uzIs@aQZHmyHmg5@`_gmzCiUvxhlt0+OriiDzsnw-gprR_m%^- zrncw8g{pdt18Y8mzCA&Uz(MT2fEUpY2vd|^Pc19a+(ST*0%N()G~!^IDcZk=#UDl1 zw0YP8vuNFLKDV!+i&5HZumZm+{Z+H!IpJrdGXXJdDu;JBP--vHn}5Zqq|A;M-|YI# z0bDvjEOR7}DVjA+#|p8sx}EM-_Ie%*xpZSDoq2>V-9MZy8@GA!U^}@s<(bqA#P`Td z0nZMZY4;hAUZjcX=TgmMZF~SCqeJ;%MCbDRi;KvL+tOB4+vj5-GY-ZmMbUI89Sr6#_nW0z1p=O8crzj zFM9a{*nDX$18MBAkCnYJ=7hPpYN*reRA$-ffzfdDSG|?bQWg_J2z+JwRW9PiV8rgq zqgy#5)&l7p+X)^{32gjSi z%xiJWU+kqY-8w(`mi?AKba97f(Tx1q$-HJ?cgyG_2Q{b6EkSgr_NJ?A?~IMUS5~U8 z`JJ9SLg`jR-%v@Vzi#PM9)ipi4+eL%@IQV0xbSi$C!M_OSt`!tD3OqU_4bCss9{diThn zWFQ@vs1Qc%Xo5F?t`Sp#o(qXBo9TQhtP#l-K!JrH{y5|dssalgQIEB;FDJYg<|@kd z@T~N4A$_o~pNH~aeovEhh_{i*(U$K{HS+DW>-{i%<)cTpG<@}RqxLk0;tNkn%r3nP zr&{Y2?f8Or;O~l_&LL@dPCF5k#QHl~W~m;KEJVv|#*M-TdwOPuszs)L@KNA&Lxthk zsmDe!6e>K#Ny1k2KB&phy5ENld%PCjD zv%e72IR@}44+>VCwzA!|kiprvO0OvAYk}R|>?&v=AEO~L*}!hI94U2@-t{DuZ!OZw zhP`9^aN)wbHZt4?2IOmjNja0)apHBd{S5V)n?SJD!mY>fhVNF;!@s1vrIzoYdc9Xr zUe=Q*A~9UqZ7uaZUnMy60>74<$4VOo?kejP;Bx}#z1w%E*m9)6q_YedPDrhkUFYXd z-~N9Zd-HH8!}srhY>~AnvXd=I%DxR_iR@9?4P}q9@4KQ>F@?xZwv>GrgDCr&eaX&D z)?o%A{I1dG^L@U@@q7Mw`lsVi=AQe$uj{<7>pb7*`?WS!j|Tgw*Q;x5ufJuFy};cY zQb{Y4#sd*-soU7`Vcg&LfFN$bES3hS<>!w5+n?q~NY*=iG{$-1RjdrY>a7Y=Geu+8 ziV&oCfZyo?eYr6Z0j&MuA9L-#Zp{yCpH!GEMHC+J2);SnQG>DbMfo{uX}95KI6C`By__RmI%`z1$X=^8#dy ztrFpp6amq2)L=sVGLV0-dXbqLSKKCZ)unP;VZq;RQMQn$%!m_?BmQ0Y{MsaK&AZx2 zmf(rX*4yShCd&T27eQ+lA-YY!F#;|mRe^^t_Wsw`W$PGs#R~4-Ns28eL1}BUKpteh zX2|6#5X8QS)QH?i?zrLQ1;1dQ@<9^VU6m;2<#5d;t&9dM0y;QJD_dSiZQ0C0@NY6XSy&-IdOEO&P_QVED&$Ivg>(ge?uI)&vV1Te$-5e+E5xAOJFMd$HrQ#zhav ztM#NhamCm9u9s|GUlZjVAMU<}haT^KJcr_XgA?v<)A&s`GSOWESp~bpYY&A8zGR98 zp5qsTB@(kz3L|;q>64gfJ`fj2ST0!I^0r)@gn(kFzi0+cgz zMZmvPbtpmMcf*^PKIKsOS)3A-yNxX66pcGwJmEPQSKl}L`N!yaruw{l6JwhHtX~5< zJ#Sgz2){Rx}wz0bAkfDs={y#2T=0^j2dkNfMBKO9Y!t^VA z9JMohj9Ma>1}aE}6QYM+eXjU)xMq3!Ro0I7^tXFc!8Xagz^r7EmCx4#)Stmq{|@Mk z3<5i2FkhddV#1&Vw1s8(Q;tlv4Y>xJ&Ty{7L1ak$PAZYxmEG+yAys!T>OVY740>GH zJIv1oGJ<|?fTG|yoiXpts@ua1aE_8-&>gN~AAlo;RXs5cq0M?s9C-Gl}9_tqx|H{8P|)v zba!Wku3t~^vf(d}0S!!XAdbD_#9w2-poV{2Bz8j(7W>XkvC?C?%MP)lV)^jl`=i1o zQi`!=Pm3Pzmc{Wt|8)sOd)m-!`|9|_L9Ne|?g8$xr56A~z>v$Nn_Mt^y+} z@ss!GG4sqMb3G72xESQ!;q3m*{PyBtHJ}dq%@S4$O~_ETI?_(#(fAnCRL}+d{M-HQ z3kfWkAJdBp(R1qa!~W6x=3TJPrYYwW7r#vp3noH7odAyT80!Dvni{mzIrvFG+kilF z@&fG`s-EaRoee4AF8nEQe{*wijL`gY#3t`IfXGIfxzzD}1t6MikS$cRPxCfpa9f&` z*7Fl~M*9W!9&`v^NRO*VaIQ!Tq`E1Kdn|Qep0m2V?1S!``}Uk1AN?W#q1r=20)zHq zYPY~K=`6Sslc7L!hRnEiFgw?eOl#2Vw*j(yPZ?n}rS-u3Jk7`zc<;DmXe-h2Oi;BC z!9kVt&z(MHjb7{p`hOT1~39L9mD9%j7v1K9=1 z$OwCVpbVOhr$_^Yzg@v`gn1q9ZRqs1*8zdTfb(fgM7{KQ3N+A%axWIi2_gu~sU{n~ z2fJ|KG()}6g4LkUsmv!OX|M4`SO^Xe0*#L;{W7)MFVn;TsP&Jbn@)K^)_=crKhgE5 zxcigi?xvx_VQ>wc&Po)aSbo!BW$g#Yx8{Lgn3Qve>DK7ZWG z;0ul@2Jo_Q6?GXa`CYsid#b()5&=u13-H_aQ}rC(bg$aE@l0S$N#q!l;~3~z~z*E|0)_I1&O8Y7^S(QKw zC#`4qkQ5G=`2g4tpEB*Oz`RNG>iSLl<^p&6zKhHPjM702(;CQV6cFwSF=d_;LbNpkU%D3N8zpKOAq?I=fj z74Fb7K~}`-`HS93GA@~!8FaEF3XA)6)ZUNq2x;&+VS!G5pR$jfBgwvxr>3dc^?4KA82VrpbRtkvI@$x(i;PJ&xp?FhR-X7}rw=QS8Ol&fCpKvH-q0`V(B6YnWkc zQg}q`m#os>yfl5?2c{TA^ELGH#g`=5FqPsla%e-C`4|6nX^xT%3lRj7Zz>WI22yUZ zU4hXYk3PUy$5I3_x0$X^<$Ytj>Ld_lK#^}F%)a(#oK`h9O3azWm3=}CC@&@j>db4? zBm&)Mekhw}k5Tyll06L30!udG8NQdmFnbZfB18{opd{D2V;tAEuz%*gty%BcWo7K8 zR(Zk_H>}&APuvbK-%vaY(7Na`e(`^(P>-drhg?4k`Tw}tikSj0qi*rL4SRr zfK3ti+ODYZA)q_v{M2a^1!ZP(eZ#oEE*AOIzqno-?)2eO&?vabw(ibWa-Qmr006{M)w3w>!)WlnqQ`OO6CT`}26ppT^ffVHfZzjP`?U6A23x8-9mx zMNPqZd!kj8qMDNfWu7|pDu*d{hvpx(hZyy=EcB=*^mRJ0d2#>;kb*b&H7tZKy#pVj zBG-sJa`=_4{&8a0$skx7kljbmHUkFjDu8hbU^$s%1$%o_uI=9}bj5T5&PXdhgV_wMr!()})-xyTq+zmaT<1R_Lr3%_TgM`fw zM(h-aALxEQ>0h$u1egYpeD$VX`W>Q07@cW$z!eQH9(?HKYYm`1EWN2R6&S9Pj~^GV zH^v*JNT_BS{hd#K7?=OG(d>Rr=C*oDSIv}%s|0~{0Q~fJ08}(Olo&-{3J$o(&pIFJ z=)9l{sw-8!>Ta!FTM!=^8``=r%LtG1r*2;u7TgrX*q`ixVqX#{wfQ= zix+h0I0H0+&fesAd(w6~@WBYDNhrMfi8p>ArakAz!_a;d*0?8-|f;Ue&K8>+@hGHZZ;5s$<#s*9OZNxxPe%HSz9^y z;`A%ey5(rD0PRlVpm>TQGI&!&96(b zn84G+3QPy`ly5b3@CPjz3!;Qk$dKm2RsvuZ2BrV&+a-m`H~8-b{8BOdct>dSbzZ;O zoYm)#d7~R*p})(A6&m^_Is6ev(e=A~m~C{^%vDTp-BCK`Yl=&Z%UA0lfDK%thFmOZ zt(K;{7qI)OU!Ao^@$-OA62J@QMCne%7)7am%!OjnM3S_W(|NhAb zHf?X$6!#P|ya5cN>~l$p?~%B;`1i(8AX!rdoo~KJJ!x`MU+15*J}!B*I$WTyqZ1w2 zmtEy#3VddrkZWsTF%Y-w`QQL{IR*O@ioYbPeM*3}DN8h)%+KI+fLCVnE}WwzzbAdM zh1cKt&nnN&^_`*9TRjM~ubV=HMZUUTQ`GClZ2m<~ivx6_>+&ukez|l3Y=P~G44dOZ zYUU&S1Fb1zHKC#80Ocq^I6eUU7aGvYoQM-AYXg=#E|TklNr*bq)>AyLd{%57Bf~#Q zfQ*XpDSmxcaBt6dPA0RXk$-L-q#?XZe<%O}t?$O>ulgEh2O_ui{x$tJ7RXOR$?BkKTMR>ozpQN^m>DZw{TpE4i<IHjYi@`2TZY=V2-#(} z%gUNDy-1{Wc!Q(=*+|65(FnTMrTi61QNY9L7BF_ae+ABYfwiWoh*MmTP)8J^AGrU` z%liqzJwfx)$5vB`(L#xk7$=F(j^@2p9pZI1ow0U+ZU{xcbJD}`m#RDD@M&+4ZJ0>& zqp{EsZVZu4L;}mb=Si{~lNB_-cq^`Tn)MvLM86wLIJgfGZ~P$gb^>G%IK9ad?}%4B znfJc#u!%zV#^(OXB&V(-$2%~H%x=`o&NzqeNr+fcxC7I7jxi?UD>LeIg+-a|$2e%H z-+mi#W~US!0no#Aw1mMx%m>T&7P-{USzVBPH~zP4=KHO&gyq; z8RT(9xB)#kbrtFG0@tk@C1FO$U0-u1E6NBD)bI&z%GzvNViN730km+PuSoI6i)z7g zvB{O*C^N0A;aatf$ik#gMH^*ypuq4i7;JMRceR|xibuwa-6cvMr?Eip2t%7G`V*R} zV^jsdA!!Af*|zs7y7@nhb3C^T{h=@O54MK zkWfQ%WGvZ<6J>0if1JVjni-kFPWj*l#m&(vi+N8``|@Z}DnC-k7gsUAVaURi_3pAt zW+23oC4qhSjzH+Qg&+DiI~unr{kyj=fR!Q#SShd{1Nk%U%&}{8RBqXM>`22Nwp&Nz zg8925R9GdvMLXT-LWH-JiwO{HwtD=Vd; zI8_$P*Yr{@AYk?cGd!g0xkZ$F$&Tr4IOXglV=77i_jmpD8^>7draAk?XNY}~{ey+V z--?2dOH39XQB!R^Gx7}{3YbXP74m>eC#`qUSlJDJ>-_^=0fsC%1Mm+}kw(n3i@hzsr&WE##{|k; zip{S#)`4rUQTn&NipA~e?JEi94hZ2#PlVtc9rW*sU3cYfE5!n8|s zYlV34zfjw=tvw;hGD+sIu)gh|Q|&nT@s7;V%`@5*0e49l5&|GRSF-Npv>g8be@?Lj zj4?so9{5v$LPf~oa5Z8Ij|5z4#CU1eP`j!UxCG%_gHQvYWp$uudLl>w^r?Ib4KkQ; z1a){9ZSB2htN?6=2l~ulv@N(l{nY`w6B*NrkY}=0H`edxTtXU_z>5gK&`Iz1z|fGE zVaf5Q&_EnMEfHLntYqv2-JgxngKjjtz&)={cm@@m!fDiD9 zb#G|YDQ{C4Y3ENlY!`T+!mpFO1atlSXPfh;VF#SO{UL7QypN}Gs zK7jU;50mzw0Hr@xY>`c17_R=zeI2L`ca4jBI{Ui=y(v-06VAS!U* zr~PQGZ+o37BoAyK)PeY|V9fkks7_^P)#TA%;i2G0Z0ecb9^CnzqQHdE(#@sx_xpEf z=v<@t$ABEXSXeSYarKYWc!U-)0IrWtb z8#5@ms_1^IFoOMa@7&Of>I?Ys45v9$K+Jqsy*_ym2;=~fy$WWMa1BroI2#*xvI>Us z>iUkF_t*!Azsf457gmMrGR|J)w?0G*ynl^_O^h4&%qIikES4nf;ND2AFV(fj8HC!{ zL|@fSlk%MNn8^0NG@zSc?hn_HcMY?you!eQo(t<~;;)|1!u78`<3Pfk%+1b)1{uzZ z6#%}|Kb~>{5D2SX*S`N&pu5x()wTaUBsek0gV5bL{l}P4swO!cZ{Mln7oo*ty>>UI zqMKe9#s=kA%l$GceoZAD=CyVj{6NNM&{{ljxHbFGfHgB{rx^#ZWMtJv&e~=L{lg|zfF_^$? z)^CM0UBtU=gee$~eV`Bm3i`(PL;XZtM9R$Hf?h+XCLk<%^i2soKtO>BHuK5v2a*&V}XW`|WdVt9z$QE*m0-_kp zH8gjYIwxmpX66Db?E5*6N#Zr(dh**5(QzW)4>(f*@DdqgoNeEEFANImyiS?q#>@i; zkAvZ0Uus7E>y23nQ_V_EadF?b)KHys0qd^ZsS2219^JwP5>Fc;rc^p-=pQ{p)6$x| znk8U^nIge`Slv;MF>+$YN+Bk5Lc|$uV>^gEA<1-sFuF&%VODHFiGXx!HUq6|j%2VQOIP)~*XL8tvPOh}r_Jx= z|5!AFD75Vj^VV<9%m-b>3b9z&xLrhIQTCB3Mc|`^ZgbmTyqD-5%6YY5mmhtoPUuf$ z7Yq68SHn5e47PA`%?rU!JPZS3Ry>OW?H-8AA5{@f`^p0qYbUI(8(jD7*C2uuFf$0x zRG}AfsMZYf$obr}f5E{<)WpQp!(g|j^V^Aia_U`M@j97dlVjoYf$g3l#nxT%f;z9F z?S(S0-&Em;b=yn4UGSGZXeaWH&?Gi349Bev5alKn;}-t{agW!=TWP+VX8edq@6GkE zt;Zmfu{x<7BCLS3$~$uAzWYyB5BxcnF(dWQl$rpxMC5aU!26Hzkb`9h@M^cP&=P>) z9!)wHPAnC;Jeu>*ei%F1&)>51u@6a43!H-BiZ7Hp4;Mfan>D$udZP zAEZ>y<`RG0@(ggSayxedQiZ;rNyPA2S+n61t&p06v={0ndRcgxmOR19nenao1H=FG zJH21_>7OaCIpO>7UB6spUbl1HrqQbsIf^DF@FqZz0A3(Bwh~g^Xsxh%y!2ba8%~x( zimC;e9})HL$~eQUQ6_};%gR=(EuhC8LH(=15$s50_E+noD5Hepd6lo*&9g76zkZxF zV64-|9T_AG(llcNjX=TZ-!E!TvJ5rV-$?bH4c@OOC?QW|H1F_) zq<#Pj4@(vSQ4x9;6C1_K8@I=&0F^tBq54ljHX1Oj8P;|FKCsq2bO6#T%Efp}Q$PuMMR3=zVHLR5ly>V;%fyfAQ_*nHIq(R` zqCOP${4mIJ0!zKV^>B0aY87E>jC-b|0O6#jKqD_S9!Nc&E>`o?>A|^2x}5~1wM!#J z-LYH)LN^u_+!PIq%@4?TwF>T40n#P_?11(Ee#v|A+P=Z*%qolkftKk#{4J#h2pOad z3BIi1=-lnac3;8V6C-_nchomH@DzfTghfY*?OV?@_%|8N#!~{PF7I;zs1YZEoNdEN1Q62?8eK*5I?zbCW@47qG`^9?FN~Q+R4+va@SdfU890Ie2-ZF_y_m9sE34c=FZSZ6w$$q z)k%@uKt}Zc&iE9sMbIN)XR-f{oapzB5Vv!!G9OGJd>}@Jn)94Kc}C4S!V$$l~~i1Vb*U;@<7`c5D1v$O&P!= zZp*OHczXpvgl2fCk)iSi{;zh^cy~=e&MT#J>xXN;ZrY(8J^X||#P>ry?>9S*x&W-{ zpYUAdxAfIVm#0w$i;X?W5?UH2IZriO3;lRiaDOMt=HVjthNvT;E?CanAnn#73<08hMdr0;Lii6jW4u1Mff|5Z+l` z`X8@vB!kBGtF(+yrR=crU1r`Inf33t$nL2#)9o^v2!6P%C*Xe}7e>lzVE;Qe5q|aa z3iPr@;+Z=Z_yiAk}2H&6D?)UJNtussXLm3EaB!yUOZ&!nY(uAko!Xi2&c6 zX{Wo~vqx=r&(+xZ1F}viz29B-G+R$lqTMp<$OD+6fB)aZhF4=Sxw~QYm+}j!@+j+A zn(TO-z1}JXfpIbT%XjuT#9Hy-4j}PLMR)w?5_Ks{6|}cafzuD(CKwg3LU3Df(9{yQ z;|&>Vk*>1S>&gehOHSb7h?*K#4CzGDutfjoayyWKqlBB$CtJ_xG-WQ88fiQpsSwy8iChoRGCpk$iycdf|^-IexU< z_1+aRT{{@L06qD$XprOtGlWSrD0AXLU!7!JuA1EMZucN>V#<=aVDVXE{=OSz>%xIiV%sC2^Yq(B2Q~JR?_6 zOb_M?}JHcKjYHtmQKiYp|HOIct+rRc#G zfS=M7&m~ zGyXo+5zqp>mfrK6Z3Vy$T;|1C?B~|?0yKnXPw=g5gs|gO`0SmU2O`nco2NQYrr3u@ zRq3{l6bV)xgJJ1>y5BsXzlcGHT|O$~kGZAW;0R-~Y7g86{NrOuqX*(z4UY*9OjjI$ zkBwQPuiy6Lv2%#!>p>M2N_tUoltS*Yo9d2y<*{Jyn-D#nX*jacz?=cS{m@O@1V z4r)`TLmO2>*+L~+Iq7H?U7&A9nS*rb$}{FU8?34yA6&x9<)Bw8Kg?}0%{Q-_q4B)e z#YUOEpIT{XIxek5Z_mJo@6Be70(VDD0++CtJX{pBsL(xkDD3T@rqc^}*UmVezv-n& zxfVi*<%OzB{a%_ia2+^_0Z<8Z_L}ad;=Wo)200I({uIjApsEgnsF=m5K6@S=1*Bt_ zb<7TJy5!oO_ACxC`$NhW$9b;mOf-%BhxnH%A35yti0awpvFTA&cZ9p}BL7V6y6Qec+ZHA@V;$I}2Y<(i?^_E^ntHLALxPYJLw_0lJL1}ayai(>xlNL<3vMP< zZ-ac;ggE|D1}mqq@o6lH63{cf0FFH-Lda;gYjc*EDI)7lDA*jXSkt`N=8--gRGf zP#A|sKuaKzu}7ct}i2A$q_$E{5)2~vH+TK4`LFqm2%B&iN+PB|Ut&F^|I zRW~NanU0yssE59rM9-k}@A%$e8?rI`^~_fmV|oysB>Z${!9rql{FSYgI0?$uvcz*M z#gDyO!f*J_ppICH_ZTUrGkdJ^T)OSV_rRa|f%|sw1Zh7iF^iMO$A@yvLqoA?u-5R=(;@b8;h=I^*)O4b z-JK#c0SNv&K5ep_=0NjMIDw zj!I9?AscIGBe}Q+oD?zcmIqS?pIfXbuPrvZZT(3x3s#10e2I#9ro5NE$2}W)VUHRw zU2WN)Uf`obQ1+kyL{*Z=NhD1^TT;tOd#1AhZ;6nFXQvK`+iDVCek7*Ho|Ps~zZ0e2 zH$%RvD_8O5ndl_5+mD0Z7D1p&VeGJG#f~!~Io#T-f*SfA2ajiy1O}-1M z5GpnOHn|~X?8W~1ERyvPOJYPdolq{RkD;M8oLevA0zm}|B4@wPpywQ?-cPOs`s85! z@Wsd<+2nY4x(VGhA+0aKp~`ZLFr>9G z@1+mUuho%mI?7UxPJeoV5v=@t*>G2$OVru#UUa zjS1SjJ3J(Cwf{A_hg*O|BbT}u<*12&ZyypnwAVSRaeLP-aJqU_^?C1Fqxe|VdP?2R z1UNPDdypCZJzr(rG+`!WKF;?jb)Bk~$h@*uQu+FX04*NiT^YlbG-Q#C$^KJsxoIhY_%xjXKqj(odEIxy%W~k2smLFc zoy2Sr?N7CcoF7Vc`*FE%>V;I|QLitgf0jzjK9A&AeO`3==pOR*mstEmZ;00~` ztc{{#mws0l`vw@Mb=V$Nu8l3f{xXuN!~D{JzX>D0I{Nj&?S0T!IrqBCb1z`EZCen@ zmK7IaRt~3=u2>R^@FaFTX98n_bx3C3l7BJFzg3YDZYc@StAn7$ZD~S~7{xU~$q-=m zMx%jm&T6O6Caubl@w2)#$J*L>A1PZ2RXYjEXaaP9Y9DF8Lwl=S^463SnKrOuJ>Q?) zlt=5DifXfN4_-4>i5D*+HY{wu#3z_N7euGq|DSyMSWA6 zesftbA9lf&i%Wgp6Na4;R%gr1B|%k)Tv@WZNYKIcD&K8={QHpuf2O+G@bM8g^dJyD zK?ts|E>eUwY`wqeM}haskOPlWVfg_Qzaw0XgN|TiSceY1O7=96?o|G=>_Lx~qOQ`I z;>nQf#}JZpxzKijbF@ByG&y_JgG&xi_h-~em2zH-e5M#38_Gxy(&=Xbe3{WX)@?!jS}j1@Q+1)Y_5K*+U7Qjf z1>Q1|1@lfutS8>&Q#m~~Xk=ppov$v1NsG6~5OD<(#ni-6!R3fUb||{AP*5$m=Exqv_YvMvHJAo*i_v2G%Z38DbPkqIxT0< z`f{lm^Fx>)Ib>S}H*a8%BZbkp1WR-sm{iz}b-oWHHM}E42x#t3!*&_!;>Q-AmoetnWhG(F-&~sit9W9a(II%#x>y){1?hN{m!{4MF zgbgX5gT|V@g+$s2*% z;(x?kQi>`_Hl3q6H1z$Vf!}1>9vg@sNOn4HyEyshPNievfaUP*MBA zuJ_E#Jor~ab{M(3SIbQHdN2HK@u3;O(%Y^wKR(3<$h3*9-RsM2pKU&A`-`IK3cJ?G zWJopGU)(8KpZGU;0=b^(9=7$21D%z?fq0SG9vD)FuT@n%xRH@{*46}GZTVGVl%rOL zw(T4M=&@n+Z_dYa2;4E6*>g0?Yo4Q_So^ulLz{Q)11r+#$j7WMw1>c`js_*uHTUL&t0(L)V4#>yqSF4^agpe`z}~_}?xUTWMTH%yTxa6` z1KPRWWsZntv7mv9mersdWrxN3B_=KP`KolUzuN^{i$)CR|GUfr_p{~C%EsB`@}Mn- zec-ti4F$=``f+1o`tp}^hsm$HBW)`?um~m5$OJR^39hX7(sNtl9CMYr zr|_?(U!@0Hk0kB!4pQ8q&O0`~Z2f|nE>$CiMP3|GN(2cF=V5)5=LU%@$U=BPtk#PA zevkHU^f&qx`oE^wx8a14l;TnH4Q)g166GSzO2P_zXiwrdFse)ki6hd0&|NoTcUWm; zi$_!89{WHqbS{L*YrQ*Is-NRg#$GTbQE#Jlm(1#CFyFdLR&~^ z&fPG7>qJBuMkOD*LRP7IUGgBGy&F>ah;%3;ZfzeN>9(0X*x81^guX3|Dg0tx1wapaQFC0ItCgGJv|hVK?h@2hm& zVPkZCxyx(kG@A)tZmRQ-IsEs4Biq?#sE&*k_WI}fc%={KO}!)SYZL?ZJ6|J&rkec) zTR28Xx>XS}c~_LBh(3vEH~u{TvO*%DnQaOpKgD$AJ5@(WGvn0X{bMwLY{{Q~B8Ab} zDn4ueb~pL4<-n_8enNQVCA3?TIu`wJ_j6>$@K+Q0hFxiH|DIuu&`CIDgHWg_`}a=* zB$~1i&G#^x$?7VvhW_2sU)6&*j|Qsw9?ef4_oSc5gwe;48;7Zpz$PJ*E#48!EMFbZ z8`h!|YM(dV4i&1Nn(d1%xF8u-gS=omvp7g!G483z91|QG<{v|TQr^4tYUgsX$+b*% zK~f?2X3sgt`ByJI1866wD;HCAs<)G6{U$0^@_KPcG}^X!QpN(bedN^tESYE72a}x- zUl8q1UkcPsNV~Z1v4m5n8ghh}Q^reLrT7njDpJ??)|43FxHGany3LU!=g+RpJO$B~ za|LON{MAKwU7K7MX%VZA+rtN~YaL~r(BBUd#q-!G^NzAou77~1!ZKw8mhS^95pnOXWyDFV#upQ$isa-Bd7Wr9}#(y zK&yw+**Twbg#BjHtTFZH?_(eB2@LArH`x9A@8}2?e~1v|74Zs;rP&U0cit=A5?VYt zGC(p3dN9y))FNzI>4vj0vVpKGVI7iBHi)7jSxT0E@XT$(HmiNGktkD%bHkIPue;Ik z)2}Xt_|NJhx`nxGFR2sHQXM77Eh?-DxdXv{BABe*DY#b^2GTgV*ciX_)>~%V2E9x& zfXy)}^MIR`d6Luxj&3*@$U}iKVpW&pZh>1LFmzsbol%p2+n`ogm<43)V5Ww{yPnzi z6N!xWr~NESq*_z;+rxe)GCpV0%yFlGdx6N?nS^oU3hbTr&yhv351wvS%P-}8Kw8B2 z&eSn(Ql^OJ)L#blKxkzeO&@a7&k)4`N}y+xt_&+05x;!v~}e zj(to1W>(^PC^z@GD4jF;{F{Tfk=V_$jFBmlhMw&95Q6=qG}511Mg*>;EE+ENOix+U z|0W2c1RZ&k=4xI2mXB49-|Db~hi&dJxqMJM{tEncy}b*ctvk?htO;tAt0aJ2$J8L$ zHGloiUl@vGsQAMH9d>Qzswo1ln7N~SX+d2Dz!DhVB==>daw3~Ne&J5#_nTpYw}1oP z*@M)Lw5RE4ks}X!_cn3^#&SX3m~U-4>~=HNL>Wj=gqZnI!xdtpb5=Me zZf{%>b@`cRt6PO5{tz;%C$z0(r`M-dt=B{&A6kj1Ixc3u)g1oVxGnp^8pgySZ}oqc zJHk85k#q>4H~qt^kQDKnIga03cF8+Joo<|@xF<#Z8ebI-a;w_xSoO&bgKO;A7OM7b z2=|N(0Fh-CMvjQtC~BJg$$TL#QmS^e(whyPgzyq6YFd{bi!qtct>TE{gTMOf*^(`r zda&=AfVLeIiGTME>tN??Tdm0>^v7pn$`)3fog5Kd=&FUaX09 z_XkD z-2*l>pNRtePS&po#cpz1e}n=|eX^T&gHIR-yfI<26#ah^DSSnTBU6iUY7Yz>2Tgxc zgu^!_N*+%xM+-^P=f_aUwH6G(<|?U7oxntVG(u zI+SXS*NdT_t@%YF-92scfWlVRfiFgJxXEPsKCN+nh~`}f*?}*da-(V4=A+;h|0iGy ziMhhPd&#YCTkEUDe=6MJE-}m7R|P&=BO`h~`>Og3}y|ymDl@7HV`jr z^bH1YgjT_r=99x1IiR6(%|8OGX=Cnn-%6q?#W0zmuUPdf3%6QpS0-&XK=O#}$Dc#V zvC>SyCXL68m(e@l5KZY11iOXkd+v{b0*g zO|QX?7v#)s)%I1uwQR^l^}kVp#)^$NBcT$Vx9Hip`puLDD?NhK7Ud2&+bs-~#J&oz zr<32YJn?wlSxBH`&2ygG4`T=VKKW=4|wf6jIKw3(l($@kI|n1&>uZKr_X#`Mz$(CT!uP z;C}Y+IDT7df6#w}nWHw>@a!+CXo9^bnv^q&v?fBj2y@I;Yl{pUVU TbH{lI_@i-0SG8E#>goRnq|Ye{ diff --git a/2.0.x/index.html b/2.0.x/index.html index 4767d56bf..bc8f5b535 100644 --- a/2.0.x/index.html +++ b/2.0.x/index.html @@ -90,7 +90,7 @@ $(addBlockSwitches);
-

2.0.0.BUILD-SNAPSHOT

+

2.0.1.BUILD-SNAPSHOT

diff --git a/2.0.x/multi/images/callouts/1.png b/2.0.x/multi/images/callouts/1.png new file mode 100644 index 0000000000000000000000000000000000000000..7d473430b7bec514f7de12f5769fe7c5859e8c5d GIT binary patch literal 329 zcmeAS@N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQC}X^4DKU-G|w_t}fLBA)Suv#nrW z!^h2QnY_`l!BOq-UXEX{m2up>JTQkX)2m zTvF+fTUlI^nXH#utd~++ke^qgmzgTe~DWM4ffP81J literal 0 HcmV?d00001 diff --git a/2.0.x/multi/images/callouts/2.png b/2.0.x/multi/images/callouts/2.png new file mode 100644 index 0000000000000000000000000000000000000000..5d09341b2f6d2ea2d1d5dad5d980f14b4b05dfd2 GIT binary patch literal 353 zcmeAS@N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQxaY7e*=hH)_rZeB4|imU1$R#1`!P>&$poQl;nzm}mD5ZFopaX|GsS%q*{P~< z;WtmO%lhToBL0i}yfkaOt?EN=nkLNGuU`ywhI5H)L`iUdT1k0gQ7VIjhO(w-Zen_> zZ(@38a<+nro{^q~f~BRtfrY+-p+a&|W^qZSLvCepNoKNMYO!8QX+eHoiC%Jk?!;Y+ zJAlS%fsM;d&r2*R1)67JkeZlkYGj#gX_9E3W@4U_nw*@Ln38B@k(iuhnUeN2eF0kK0(Y1u|9Rc(19XFPiEBhjaDG}zd16s2gM)^$re|(qda7?? zdS-IAf{C7yo`r&?rM`iMzJZ}aa#3b+Nu@(>WpPPnvR-PjUP@^}eqM=Qa(?c_U5Yz^ z#%Y0#%S_KpEGY$=XJL?(l#*ybuErX#^g`ttQfwn - 2. Additional resources
2. Additional resources
Prev   Next

2. Additional resources

Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin

click here to see the video

Prev   Next
1. Introduction Home 3. Features
\ No newline at end of file + 2. Additional Resources
2. Additional Resources
Prev   Next

2. Additional Resources

You can watch a video of Reshmi Krishna and Marcin Grzejszczak talking about Spring Cloud +Sleuth and Zipkin by clicking here.

You can check different setups of Sleuth and Brave in the openzipkin/sleuth-webmvc-example repository.

Prev   Next
1. Introduction Home 3. Features
\ No newline at end of file diff --git a/2.0.x/multi/multi__current_span.html b/2.0.x/multi/multi__current_span.html new file mode 100644 index 000000000..6df2346f9 --- /dev/null +++ b/2.0.x/multi/multi__current_span.html @@ -0,0 +1,19 @@ + + + 7. Current Span
7. Current Span
Prev   Next

7. Current Span

Brave supports a "current span" concept which represents the in-flight operation. +You can use Tracer.currentSpan() to add custom tags to a span and Tracer.nextSpan() to create a child of whatever is in-flight.

[Important]Important

In Sleuth, you can autowire the Tracer bean to retrieve the current span via +tracer.currentSpan() method. To retrieve the current context just call +tracer.currentSpan().context(). To get the current trace id as String +you can use the traceIdString() method like this: tracer.currentSpan().context().traceIdString().

7.1 Setting a span in scope manually

When writing new instrumentation, it is important to place a span you created in scope as the current span. +Not only does doing so let users access it with Tracer.currentSpan(), but it also allows customizations such as SLF4J MDC to see the current trace IDs.

Tracer.withSpanInScope(Span) facilitates this and is most conveniently employed by using the try-with-resources idiom. +Whenever external code might be invoked (such as proceeding an interceptor or otherwise), place the span in scope, as shown in the following example:

@Autowired Tracer tracer;
+
+try (SpanInScope ws = tracer.withSpanInScope(span)) {
+  return inboundRequest.invoke();
+} finally { // note the scope is independent of the span
+  span.finish();
+}

In edge cases, you may need to clear the current span temporarily (for example, launching a task that should not be associated with the current request). To do tso, pass null to withSpanInScope, as shown in the following example:

@Autowired Tracer tracer;
+
+try (SpanInScope cleared = tracer.withSpanInScope(null)) {
+  startBackgroundThread();
+}
Prev   Next
6. Current Tracing Component Home 8. Instrumentation
\ No newline at end of file diff --git a/2.0.x/multi/multi__current_tracing_component.html b/2.0.x/multi/multi__current_tracing_component.html new file mode 100644 index 000000000..75511b6ac --- /dev/null +++ b/2.0.x/multi/multi__current_tracing_component.html @@ -0,0 +1,7 @@ + + + 6. Current Tracing Component
6. Current Tracing Component
Prev   Next

6. Current Tracing Component

Brave supports a "current tracing component" concept, which should only be used when you have no other way to get a reference. +This was made for JDBC connections, as they often initialize prior to the tracing component.

The most recent tracing component instantiated is available through Tracing.current(). +You can also use Tracing.currentTracer() to get only the tracer. +If you use either of these methods, do not cache the result. +Instead, look them up each time you need them.

Prev   Next
5. Propagation Home 7. Current Span
\ No newline at end of file diff --git a/2.0.x/multi/multi__customizations.html b/2.0.x/multi/multi__customizations.html index 618062712..0b804d6c2 100644 --- a/2.0.x/multi/multi__customizations.html +++ b/2.0.x/multi/multi__customizations.html @@ -1,136 +1,71 @@ - 9. Customizations
9. Customizations
Prev   Next

9. Customizations

Thanks to the SpanInjector and SpanExtractor you can customize the way spans -are created and propagated.

There are currently two built-in ways to pass tracing information between processes:

  • via Spring Integration
  • via HTTP

Span ids are extracted from Zipkin-compatible (B3) headers (either Message -or HTTP headers), to start or join an existing trace. Trace information is -injected into any outbound requests so the next hop can extract them.

The key change in comparison to the previous versions of Sleuth is that Sleuth is implementing -the Open Tracing’s TextMap notion. In Sleuth it’s called SpanTextMap. Basically the idea -is that any means of communication (e.g. message, http request, etc.) can be abstracted via -a SpanTextMap. This abstraction defines how one can insert data into the carrier and -how to retrieve it from there. Thanks to this if you want to instrument a new HTTP library -that uses a FooRequest as a mean of sending HTTP requests then you have to create an -implementation of a SpanTextMap that delegates calls to FooRequest in terms of retrieval -and insertion of HTTP headers.

9.1 Spring Integration

For Spring Integration there are 2 interfaces responsible for creation of a Span from a Message. -These are:

  • MessagingSpanTextMapExtractor
  • MessagingSpanTextMapInjector

You can override them by providing your own implementation.

9.2 HTTP

For HTTP there are 2 interfaces responsible for creation of a Span from a Message. -These are:

  • HttpSpanExtractor
  • HttpSpanInjector

You can override them by providing your own implementation.

9.3 Example

Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names -you have

  • for trace id - correlationId
  • for span id - mySpanId

This is a an example of a SpanExtractor

static class CustomHttpSpanExtractor implements HttpSpanExtractor {
+   12. Customizations
12. Customizations
Prev   Next
12. Customizations
12.1 HTTP
If a customization of client / server parsing of the HTTP related spans is required,
+just register a bean of type brave.http.HttpClientParser or
+brave.http.HttpServerParser. If client /server sampling is required, just
+register a bean of type brave.http.HttpSampler and name the bean
+ sleuthClientSampler for client sampler and sleuthServerSampler for server sampler.
+ For your convenience the @ClientSampler and @ServerSampler
+ annotations can be used to inject the proper beans or to
+ reference the bean names via their static String NAME fields.
Check out Brave’s code to see an example of how to make a path-based sampler
+https://github.com/openzipkin/brave/tree/master/instrumentation/http#sampling-policy
If you want to completely rewrite the HttpTracing bean you can use the SkipPatternProvider
+interface to retrieve the URL Pattern for spans that should be not sampled. Below you can see
+an example of usage of SkipPatternProvider inside a server side, HttpSampler.
@Configuration
+class Config {
+  @Bean(name = ServerSampler.NAME)
+  HttpSampler myHttpSampler(SkipPatternProvider provider) {
+  	Pattern pattern = provider.skipPattern();
+  	return new HttpSampler() {
 
-	@Override public Span joinTrace(SpanTextMap carrier) {
-		Map<String, String> map = TextMapUtil.asMap(carrier);
-		long traceId = Span.hexToId(map.get("correlationid"));
-		long spanId = Span.hexToId(map.get("myspanid"));
-		// extract all necessary headers
-		Span.SpanBuilder builder = Span.builder().traceId(traceId).spanId(spanId);
-		// build rest of the Span
-		return builder.build();
-	}
-}
-
-static class CustomHttpSpanInjector implements HttpSpanInjector {
-
-	@Override
-	public void inject(Span span, SpanTextMap carrier) {
-		carrier.put("correlationId", span.traceIdString());
-		carrier.put("mySpanId", Span.idToHex(span.getSpanId()));
-	}
-}
And you could register it like this:
@Bean
-HttpSpanInjector customHttpSpanInjector() {
-	return new CustomHttpSpanInjector();
-}
-
-@Bean
-HttpSpanExtractor customHttpSpanExtractor() {
-	return new CustomHttpSpanExtractor();
-}
Spring Cloud Sleuth does not add trace/span related headers to the Http Response for security reasons. If you need the headers then a custom SpanInjector
-that injects the headers into the Http Response and a Servlet filter which makes use of this can be added the following way:
static class CustomHttpServletResponseSpanInjector extends ZipkinHttpSpanInjector {
-
-	@Override
-	public void inject(Span span, SpanTextMap carrier) {
-		super.inject(span, carrier);
-		carrier.put(Span.TRACE_ID_NAME, span.traceIdString());
-		carrier.put(Span.SPAN_ID_NAME, Span.idToHex(span.getSpanId()));
-	}
-}
-
-static class HttpResponseInjectingTraceFilter extends GenericFilterBean {
+  		@Override public <Req> Boolean trySample(HttpAdapter<Req, ?> adapter, Req request) {
+  			String url = adapter.path(request);
+  			boolean shouldSkip = pattern.matcher(url).matches();
+  			if (shouldSkip) {
+  				return false;
+  			}
+  			return null;
+  		}
+  	};
+  }
+}
12.2 TracingFilter
You can also modify the behavior of the TracingFilter, which is the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response.
+You can customize the tags or modify the response headers by registering your own instance of the TracingFilter bean.
In the following example, we register the TracingFilter bean, add the ZIPKIN-TRACE-ID response header containing the current Span’s trace id, and add a tag with key custom and a value tag to the span.
@Component
+@Order(TraceWebServletAutoConfiguration.TRACING_FILTER_ORDER + 1)
+class MyFilter extends GenericFilterBean {
 
 	private final Tracer tracer;
-	private final HttpSpanInjector spanInjector;
 
-	public HttpResponseInjectingTraceFilter(Tracer tracer, HttpSpanInjector spanInjector) {
+	MyFilter(Tracer tracer) {
 		this.tracer = tracer;
-		this.spanInjector = spanInjector;
 	}
 
-	@Override
-	public void doFilter(ServletRequest request, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
-		HttpServletResponse response = (HttpServletResponse) servletResponse;
-		Span currentSpan = this.tracer.getCurrentSpan();
-		this.spanInjector.inject(currentSpan, new HttpServletResponseTextMap(response));
-		filterChain.doFilter(request, response);
+	@Override public void doFilter(ServletRequest request, ServletResponse response,
+			FilterChain chain) throws IOException, ServletException {
+		Span currentSpan = this.tracer.currentSpan();
+		if (currentSpan == null) {
+			chain.doFilter(request, response);
+			return;
+		}
+		// for readability we're returning trace id in a hex form
+		((HttpServletResponse) response)
+				.addHeader("ZIPKIN-TRACE-ID",
+						currentSpan.context().traceIdString());
+		// we can also add some custom tags
+		currentSpan.tag("custom", "tag");
+		chain.doFilter(request, response);
 	}
-
-	 class HttpServletResponseTextMap implements SpanTextMap {
-
-		 private final HttpServletResponse delegate;
-
-		 HttpServletResponseTextMap(HttpServletResponse delegate) {
-			 this.delegate = delegate;
-		 }
-
-		 @Override
-		 public Iterator<Map.Entry<String, String>> iterator() {
-			 Map<String, String> map = new HashMap<>();
-			 for (String header : this.delegate.getHeaderNames()) {
-				map.put(header, this.delegate.getHeader(header));
-			 }
-			 return map.entrySet().iterator();
-		 }
-
-		 @Override
-		 public void put(String key, String value) {
-			this.delegate.addHeader(key, value);
-		 }
-	 }
-}
And you could register them like this:
@Bean HttpSpanInjector customHttpServletResponseSpanInjector() {
-	return new CustomHttpServletResponseSpanInjector();
+}
12.3 Custom service name
By default, Sleuth assumes that, when you send a span to Zipkin, you want the span’s service name to be equal to the value of the spring.application.name property.
+That is not always the case, though.
+There are situations in which you want to explicitly provide a different service name for all spans coming from your application.
+To achieve that, you can pass the following property to your application to override that value (the example is for a service named myService):
spring.zipkin.service.name: myService
12.4 Customization of Reported Spans
Before reporting spans (for example, to Zipkin) you may want to modify that span in some way.
+You can do so by using the SpanAdjuster interface.
In Sleuth, we generate spans with a fixed name.
+Some users want to modify the name depending on values of tags.
+You can implement the SpanAdjuster interface to alter that name.
The following example shows how to register two beans that implement SpanAdjuster:
@Bean SpanAdjuster adjusterOne() {
+	return span -> span.toBuilder().name("foo").build();
 }
 
-@Bean
-HttpResponseInjectingTraceFilter responseInjectingTraceFilter(Tracer tracer) {
-	return new HttpResponseInjectingTraceFilter(tracer, customHttpServletResponseSpanInjector());
-}
9.4 TraceFilter
You can also modify the behaviour of the TraceFilter - the component that is responsible
-for processing the input HTTP request and adding tags basing on the HTTP response. You can customize
-the tags, or modify the response headers by registering your own instance of the TraceFilter bean.
In the following example we will register the TraceFilter bean and we will add the
-ZIPKIN-TRACE-ID response header containing the current Span’s trace id. Also we will
-add to the Span a tag with key custom and a value tag.
@Bean
-TraceFilter myTraceFilter(BeanFactory beanFactory, final Tracer tracer) {
-	return new TraceFilter(beanFactory) {
-		@Override protected void addResponseTags(HttpServletResponse response,
-				Throwable e) {
-			// execute the default behaviour
-			super.addResponseTags(response, e);
-			// for readability we're returning trace id in a hex form
-			response.addHeader("ZIPKIN-TRACE-ID",
-					Span.idToHex(tracer.getCurrentSpan().getTraceId()));
-			// we can also add some custom tags
-			tracer.addTag("custom", "tag");
-		}
-	};
-}
9.5 Custom SA tag in Zipkin
Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented.
-What you can do is to create a span with the peer.service tag that will contain a value of the service that you want to call.
-Below you can see an example of a call to Redis that is wrapped in such a span.
Unresolved directive in spring-cloud-sleuth.adoc - include::../../../..//spring-cloud-sleuth-zipkin-legacy/src/test/java/org/springframework/cloud/sleuth/zipkin/HttpZipkinSpanReporterTest.java[tags=service_name,indent=0]
[Important]Important
Remember not to add both peer.service tag and the SA tag! You have to add only peer.service.
9.6 Custom service name
By default Sleuth assumes that when you send a span to Zipkin, you want the span’s service name
- to be equal to spring.application.name value. That’s not always the case though. There
- are situations in which you want to explicitly provide a different service name for all spans coming
- from your application. To achieve that it’s enough to just pass the following property
- to your application to override that value (example for foo service name):
spring.zipkin.service.name: foo
9.7 Customization of reported spans
Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way.
- You can achieve that by using the SpanAdjuster interface.
Example of usage:
In Sleuth we’re generating spans with a fixed name. Some users want to modify the name depending on values
-of tags. Implementation of the SpanAdjuster interface can be used to alter that name. Example:
@Bean
-SpanAdjuster customSpanAdjuster() {
-    return span -> span.toBuilder().name(scrub(span.getName())).build();
-}
This will lead in changing the name of the reported span just before it gets sent to Zipkin.
[Important]Important
Your SpanReporter should inject the SpanAdjuster and
- allow span manipulation before the actual reporting is done.
9.8 Host locator
In order to define the host that is corresponding to a particular span we need to resolve the host name
-and port. The default approach is to take it from server properties. If those for some reason are not set
-then we’re trying to retrieve the host name from the network interfaces.
If you have the discovery client enabled and prefer to retrieve the host address from the registered
-instance in a service registry then you have to set the property (it’s applicable for both HTTP and
-Stream based span reporting).
spring.zipkin.locator.discovery.enabled: true
Prev   Next
8. Managing spans with annotations Home 10. Sending spans to Zipkin

\ No newline at end of file
+@Bean SpanAdjuster adjusterTwo() {
+	return span -> span.toBuilder().name(span.name() + " bar").build();
+}

The preceding example results in changing the name of the reported span to foo bar, just before it gets reported (for example, to Zipkin).

12.5 Host Locator

[Important]Important

This section is about defining host from service discovery. +It is NOT about finding Zipkin through service discovery.

To define the host that corresponds to a particular span, we need to resolve the host name and port. +The default approach is to take these values from server properties. +If those are not set, we try to retrieve the host name from the network interfaces.

If you have the discovery client enabled and prefer to retrieve the host address from the registered instance in a service registry, you have to set the spring.zipkin.locator.discovery.enabled property (it is applicable for both HTTP-based and Stream-based span reporting), as follows:

spring.zipkin.locator.discovery.enabled: true
Prev   Next
11. Managing Spans with Annotations Home 13. Sending Spans to Zipkin
\ No newline at end of file diff --git a/2.0.x/multi/multi__features.html b/2.0.x/multi/multi__features.html index bb705d5d4..6c2740f7c 100644 --- a/2.0.x/multi/multi__features.html +++ b/2.0.x/multi/multi__features.html @@ -1,20 +1,126 @@ - 3. Features
3. Features
Prev   Next

3. Features

  • Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Example logs:

    2016-02-02 15:30:57.902  INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
+   3. Features
    3. Features
    Prev   Next
    3. Features
    • Adds trace and span IDs to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator, as shown in the following example logs:
      2016-02-02 15:30:57.902  INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
 2016-02-02 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
-2016-02-02 15:31:01.936  INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...
      notice the [appname,traceId,spanId,exportable] entries from the MDC:
      • spanId - the id of a specific operation that took place
      • appname - the name of the application that logged the span
      • traceId - the id of the latency graph that contains the span
      • exportable - whether the log should be exported to Zipkin or not. When would you like the span not to be
-exportable? In the case in which you want to wrap some operation in a Span and have it written to the logs
-only.
    • Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations,
-key-value annotations. Loosely based on HTrace, but Zipkin (Dapper) compatible.
    • Sleuth records timing information to aid in latency analysis. Using sleuth, you can pinpoint causes of
-latency in your applications. Sleuth is written to not log too much, and to not cause your production application to crash.
      • propagates structural data about your call-graph in-band, and the rest out-of-band.
      • includes opinionated instrumentation of layers such as HTTP
      • includes sampling policy to manage volume
      • can report to a Zipkin system for query and visualization
    • Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints,
-rest template, scheduled actions, message channels, zuul filters, feign client).
    • Sleuth includes default logic to join a trace across http or messaging boundaries. For example, http propagation
-works via Zipkin-compatible request headers. This propagation logic is defined and customized via
-SpanInjector and SpanExtractor implementations.
    • Sleuth gives you the possibility to propagate context (also known as baggage) between processes. That means that if you set on a Span
-a baggage element then it will be sent downstream either via HTTP or messaging to other processes.
    • Provides a way to create / continue spans and add tags and logs via annotations.
    • Provides simple metrics of accepted / dropped spans.
    • If spring-cloud-sleuth-zipkin then the app will generate and collect Zipkin-compatible traces.
-By default it sends them via HTTP to a Zipkin server on localhost (port 9411).
-Configure the location of the service using spring.zipkin.baseUrl.
      • If you depend on spring-rabbit or spring-kafka your app will send traces to a broker instead of http.
      • Note: spring-cloud-sleuth-stream is deprecated and should no longer be used.
    [Important]Important
    If using Zipkin, configure the percentage of spans exported using spring.sleuth.sampler.percentage
-(default 0.1, i.e. 10%). Otherwise you might think that Sleuth is not working cause it’s omitting some spans.
    [Note]Note
    the SLF4J MDC is always set and logback users will immediately see the trace and span ids in logs per the example
- above. Other logging systems have to configure their own formatter to get the same result. The default is
- logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]
- (this is a Spring Boot feature for logback users).
- This means that if you’re not using SLF4J this pattern WILL NOT be automatically applied.
    Prev   Next
    2. Additional resources Home 4. Sampling
    
\ No newline at end of file
+2016-02-02 15:31:01.936  INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...

    Notice the [appname,traceId,spanId,exportable] entries from the MDC:

    • spanId: The ID of a specific operation that took place.
    • appname: The name of the application that logged the span.
    • traceId: The ID of the latency graph that contains the span.
    • exportable: Whether the log should be exported to Zipkin. +When would you like the span not to be exportable? +When you want to wrap some operation in a Span and have it written to the logs only.
  • Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, and key-value annotations. +Spring Cloud Sleuth is loosely based on HTrace but is compatible with Zipkin (Dapper).
  • Sleuth records timing information to aid in latency analysis. +By using sleuth, you can pinpoint causes of latency in your applications.

  • Sleuth is written to not log too much and to not cause your production application to crash. +To that end, Sleuth:

    • Propagates structural data about your call graph in-band and the rest out-of-band.
    • Includes opinionated instrumentation of layers such as HTTP.
    • Includes a sampling policy to manage volume.
    • Can report to a Zipkin system for query and visualization.
  • Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, rest template, scheduled actions, message channels, Zuul filters, and Feign client).
  • Sleuth includes default logic to join a trace across HTTP or messaging boundaries. +For example, HTTP propagation works over Zipkin-compatible request headers.
  • Sleuth can propagate context (also known as baggage) between processes. +Consequently, if you set a baggage element on a Span, it is sent downstream to other processes over either HTTP or messaging.
  • Provides a way to create or continue spans and add tags and logs through annotations.

  • If spring-cloud-sleuth-zipkin is on the classpath, the app generates and collects Zipkin-compatible traces. +By default, it sends them over HTTP to a Zipkin server on localhost (port 9411). +You can configure the location of the service by setting spring.zipkin.baseUrl.

    • If you depend on spring-rabbit, your app sends traces to a RabbitMQ broker instead of HTTP.
    • If you depend on spring-kafka, and set spring.zipkin.sender.type: kafka, your app sends traces to a Kafka broker instead of HTTP.
[Caution]Caution

spring-cloud-sleuth-stream is deprecated and should no longer be used.

[Important]Important

If you use Zipkin, configure the probability of spans exported by setting spring.sleuth.sampler.probability +(default: 0.1, which is 10 percent). Otherwise, you might think that Sleuth is not working be cause it omits some spans.

[Note]Note

The SLF4J MDC is always set and logback users immediately see the trace and span IDs in logs per the example +shown earlier. +Other logging systems have to configure their own formatter to get the same result. +The default is as follows: +logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}] +(this is a Spring Boot feature for logback users). +If you do not use SLF4J, this pattern is NOT automatically applied.

3.1 Introduction to Brave

[Important]Important

Starting with version 2.0.0, Spring Cloud Sleuth uses +Brave as the tracing library. +For your convenience, we embed part of the Brave’s docs here.

[Important]Important

In the vast majority of cases you need to just use the Tracer +or SpanCustomizer beans from Brave that Sleuth provides. The documentation below contains +a high overview of what Brave is and how it works.

Brave is a library used to capture and report latency information about distributed operations to Zipkin. +Most users do not use Brave directly. They use libraries or frameworks rather than employ Brave on their behalf.

This module includes a tracer that creates and joins spans that model the latency of potentially distributed work. +It also includes libraries to propagate the trace context over network boundaries (for example, with HTTP headers).

3.1.1 Tracing

Most importantly, you need a brave.Tracer, configured to report to Zipkin.

The following example setup sends trace data (spans) to Zipkin over HTTP (as opposed to Kafka):

class MyClass {
+
+    private final Tracer tracer;
+
+    // Tracer will be autowired
+    MyClass(Tracer tracer) {
+        this.tracer = tracer;
+    }
+
+    void doSth() {
+        Span span = tracer.newTrace().name("encode").start();
+        // ...
+    }
+}
[Important]Important

If your span contains a name longer than 50 chars, then that name is truncated to 50 chars. +Your names have to be explicit and concrete. +Big names lead to latency issues and sometimes even thrown exceptions.

The tracer creates and joins spans that model the latency of potentially distributed work. +It can employ sampling to reduce overhead during the process, to reduce the amount of data sent to Zipkin, or both.

Spans returned by a tracer report data to Zipkin when finished or do nothing if unsampled. +After starting a span, you can annotate events of interest or add tags containing details or lookup keys.

Spans have a context that includes trace identifiers that place the span at the correct spot in the tree representing the distributed operation.

3.1.2 Local Tracing

When tracing local code, you can run it inside a span, as shown in the following example:

@Autowired Tracer tracer;
+
+Span span = tracer.newTrace().name("encode").start();
+try {
+  doSomethingExpensive();
+} finally {
+  span.finish();
+}

In the preceding example, the span is the root of the trace. +In many cases, the span is part of an existing trace. +When this is the case, call newChild instead of newTrace, as shown in the following example:

@Autowired Tracer tracer;
+
+Span span = tracer.newChild(root.context()).name("encode").start();
+try {
+  doSomethingExpensive();
+} finally {
+  span.finish();
+}

3.1.3 Customizing Spans

Once you have a span, you can add tags to it. +The tags can be used as lookup keys or details. +For example, you might add a tag with your runtime version, as shown in the following example:

span.tag("clnt/finagle.version", "6.36.0");

When exposing the ability to customize spans to third parties, prefer brave.SpanCustomizer as opposed to brave.Span. +The former is simpler to understand and test and does not tempt users with span lifecycle hooks.

interface MyTraceCallback {
+  void request(Request request, SpanCustomizer customizer);
+}

Since brave.Span implements brave.SpanCustomizer, you can pass it to users, as shown in the following example:

for (MyTraceCallback callback : userCallbacks) {
+  callback.request(request, span);
+}

3.1.4 Implicitly Looking up the Current Span

Sometimes, you do not know if a trace is in progress or not, and you do not want users to do null checks. +brave.CurrentSpanCustomizer handles this problem by adding data to any span that’s in progress or drops, as shown in the following example:

Ex.

// The user code can then inject this without a chance of it being null.
+@Autowired SpanCustomizer span;
+
+void userCode() {
+  span.annotate("tx.started");
+  ...
+}

3.1.5 RPC tracing

[Tip]Tip

Check for instrumentation written here and Zipkin’s list before rolling your own RPC instrumentation.

RPC tracing is often done automatically by interceptors. Behind the scenes, they add tags and events that relate to their role in an RPC operation.

The following example shows how to add a client span:

@Autowired Tracer tracer;
+
+// before you send a request, add metadata that describes the operation
+span = tracer.newTrace().name("get").type(CLIENT);
+span.tag("clnt/finagle.version", "6.36.0");
+span.tag(TraceKeys.HTTP_PATH, "/api");
+span.remoteEndpoint(Endpoint.builder()
+    .serviceName("backend")
+    .ipv4(127 << 24 | 1)
+    .port(8080).build());
+
+// when the request is scheduled, start the span
+span.start();
+
+// if you have callbacks for when data is on the wire, note those events
+span.annotate(Constants.WIRE_SEND);
+span.annotate(Constants.WIRE_RECV);
+
+// when the response is complete, finish the span
+span.finish();

One-Way tracing

Sometimes, you need to model an asynchronous operation where there is a +request but no response. In normal RPC tracing, you use span.finish() +to indicate that the response was received. In one-way tracing, you use +span.flush() instead, as you do not expect a response.

The following example shows how a client might model a one-way operation:

@Autowired Tracer tracer;
+
+// start a new span representing a client request
+oneWaySend = tracer.newSpan(parent).kind(Span.Kind.CLIENT);
+
+// Add the trace context to the request, so it can be propagated in-band
+tracing.propagation().injector(Request::addHeader)
+                     .inject(oneWaySend.context(), request);
+
+// fire off the request asynchronously, totally dropping any response
+request.execute();
+
+// start the client side and flush instead of finish
+oneWaySend.start().flush();

The following example shows how a server might handle a one-way operation:

@Autowired Tracing tracing;
+@Autowired Tracer tracer;
+
+// pull the context out of the incoming request
+extractor = tracing.propagation().extractor(Request::getHeader);
+
+// convert that context to a span which you can name and add tags to
+oneWayReceive = nextSpan(tracer, extractor.extract(request))
+    .name("process-request")
+    .kind(SERVER)
+    ... add tags etc.
+
+// start the server side and flush instead of finish
+oneWayReceive.start().flush();
+
+// you should not modify this span anymore as it is complete. However,
+// you can create children to represent follow-up work.
+next = tracer.newSpan(oneWayReceive.context()).name("step2").start();
Prev   Next
2. Additional Resources Home 4. Sampling
\ No newline at end of file diff --git a/2.0.x/multi/multi__instrumentation.html b/2.0.x/multi/multi__instrumentation.html index 77cf9b176..82c1d45de 100644 --- a/2.0.x/multi/multi__instrumentation.html +++ b/2.0.x/multi/multi__instrumentation.html @@ -1,18 +1,6 @@ - 5. Instrumentation
5. Instrumentation
Prev   Next

5. Instrumentation

Spring Cloud Sleuth instruments all your Spring application -automatically, so you shouldn’t have to do anything to activate -it. The instrumentation is added using a variety of technologies -according to the stack that is available, e.g. for a servlet web -application we use a Filter, and for Spring Integration we use -ChannelInterceptors.

You can customize the keys used in span tags. To limit the volume of -span data, by default an HTTP request will be tagged only with a -handful of metadata like the status code, host and URL. You can add -request headers by configuring spring.sleuth.keys.http.headers (a -list of header names).

[Note]Note

Remember that tags are only collected and exported if there is a -Sampler that allows it (by default there is not, so there is no -danger of accidentally collecting too much data without configuring -something).

[Note]Note

Currently the instrumentation in Spring Cloud Sleuth is eager - it means that -we’re actively trying to pass the tracing context between threads. Also timing events -are captured even when sleuth isn’t exporting data to a tracing system. -This approach may change in the future towards being lazy on this matter.

Prev   Next
4. Sampling Home 6. Span lifecycle
\ No newline at end of file + 8. Instrumentation
8. Instrumentation
Prev   Next

8. Instrumentation

Spring Cloud Sleuth automatically instruments all your Spring applications, so you should not have to do anything to activate it. +The instrumentation is added by using a variety of technologies according to the stack that is available. For example, for a servlet web application, we use a Filter, and, for Spring Integration, we use ChannelInterceptors.

You can customize the keys used in span tags. +To limit the volume of span data, an HTTP request is, by default, tagged only with a handful of metadata, such as the status code, the host, and the URL. +You can add request headers by configuring spring.sleuth.keys.http.headers (a list of header names).

[Note]Note

Tags are collected and exported only if there is a Sampler that allows it. By default, there is no such Sampler, to ensure that there is no danger of accidentally collecting too much data without configuring something).

Prev   Next
7. Current Span Home 9. Span lifecycle
\ No newline at end of file diff --git a/2.0.x/multi/multi__integrations.html b/2.0.x/multi/multi__integrations.html index 94def53ee..aa2def57b 100644 --- a/2.0.x/multi/multi__integrations.html +++ b/2.0.x/multi/multi__integrations.html @@ -1,6 +1,8 @@ - 13. Integrations
13. Integrations
Prev   Next

13. Integrations

13.1 Runnable and Callable

If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative.

Example for Runnable:

Runnable runnable = new Runnable() {
+   15. Integrations
15. Integrations
Prev   Next
15. Integrations
15.1 OpenTracing
Spring Cloud Sleuth is compatible with OpenTracing.
+If you have OpenTracing on the classpath, we automatically register the OpenTracing Tracer bean.
+If you wish to disable this, set spring.sleuth.opentracing.enabled to false
15.2 Runnable and Callable
If you wrap your logic in Runnable or Callable, you can wrap those classes in their Sleuth representative, as shown in the following example for Runnable:
Runnable runnable = new Runnable() {
 	@Override
 	public void run() {
 		// do some work
@@ -12,10 +14,11 @@
 	}
 };
 // Manual `TraceRunnable` creation with explicit "calculateTax" Span name
-Runnable traceRunnable = new TraceRunnable(tracer, spanNamer, runnable, "calculateTax");
-// Wrapping `Runnable` with `Tracer`. The Span name will be taken either from the
-// `@SpanName` annotation or from `toString` method
-Runnable traceRunnableFromTracer = tracer.wrap(runnable);
Example for Callable:
Callable<String> callable = new Callable<String>() {
+Runnable traceRunnable = new TraceRunnable(tracing, spanNamer, runnable,
+		"calculateTax");
+// Wrapping `Runnable` with `Tracing`. That way the current span will be available
+// in the thread of `Runnable`
+Runnable traceRunnableFromTracer = tracing.currentTraceContext().wrap(runnable);
The following example shows how to do so for Callable:
Callable<String> callable = new Callable<String>() {
 	@Override
 	public String call() throws Exception {
 		return someLogic();
@@ -27,82 +30,64 @@ Runnable traceRunnableFromTracer = tracer.wrap(runnable);
Example for // Manual `TraceCallable` creation with explicit "calculateTax" Span name
-Callable<String> traceCallable = new TraceCallable<>(tracer, spanNamer, callable, "calculateTax");
-// Wrapping `Callable` with `Tracer`. The Span name will be taken either from the
-// `@SpanName` annotation or from `toString` method
-Callable<String> traceCallableFromTracer = tracer.wrap(callable);

That way you will ensure that a new Span is created and closed for each execution.

13.2 Hystrix

13.2.1 Custom Concurrency Strategy

We’re registering a custom HystrixConcurrencyStrategy -that wraps all Callable instances into their Sleuth representative - -the TraceCallable. The strategy either starts or continues a span depending on the fact whether tracing was already going -on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false.

13.2.2 Manual Command setting

Assuming that you have the following HystrixCommand:

HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
+Callable<String> traceCallable = new TraceCallable<>(tracing, spanNamer, callable,
+		"calculateTax");
+// Wrapping `Callable` with `Tracing`. That way the current span will be available
+// in the thread of `Callable`
+Callable<String> traceCallableFromTracer = tracing.currentTraceContext().wrap(callable);

That way, you ensure that a new span is created and closed for each execution.

15.3 Hystrix

15.3.1 Custom Concurrency Strategy

We register a custom HystrixConcurrencyStrategy called TraceCallable that wraps all Callable instances in their Sleuth representative. +The strategy either starts or continues a span, depending on whether tracing was already going on before the Hystrix command was called. +To disable the custom Hystrix Concurrency Strategy, set the spring.sleuth.hystrix.strategy.enabled to false.

15.3.2 Manual Command setting

Assume that you have the following HystrixCommand:

HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
 	@Override
 	protected String run() throws Exception {
 		return someLogic();
 	}
-};

In order to pass the tracing information you have to wrap the same logic in the Sleuth version of the HystrixCommand which is the -TraceCommand:

TraceCommand<String> traceCommand = new TraceCommand<String>(tracer, traceKeys, setter) {
+};

To pass the tracing information, you have to wrap the same logic in the Sleuth version of the HystrixCommand, which is called +TraceCommand, as shown in the following example:

TraceCommand<String> traceCommand = new TraceCommand<String>(tracer, setter) {
 	@Override
 	public String doRun() throws Exception {
 		return someLogic();
 	}
-};

13.3 RxJava

We’re registering a custom RxJavaSchedulersHook -that wraps all Action0 instances into their Sleuth representative - -the TraceAction. The hook either starts or continues a span depending on the fact whether tracing was already going -on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the spring.sleuth.rxjava.schedulers.hook.enabled to false.

You can define a list of regular expressions for thread names, for which you don’t want a Span to be created. Just provide a comma separated list -of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

13.4 HTTP integration

Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false.

13.4.1 HTTP Filter

Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which - the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would - like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern.

13.4.2 HandlerInterceptor

Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an - existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. The - TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. If the - the TraceFilter doesn’t see this attribute set it will create a "fallback" span which is an additional - span created on the server side so that the trace is presented properly in the UI. Seeing that most likely - signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.

13.4.3 Async Servlet support

If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one.

13.4.4 WebFlux support

Via the TraceWebFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which - the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would - like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern.

13.5 HTTP client integration

13.5.1 Synchronous Rest Template

We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a -call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous RestTemplate features -just set spring.sleuth.web.client.enabled to false.

[Important]Important

You have to register RestTemplate as a bean so that the interceptors will get injected. -If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work.

13.5.2 Asynchronous Rest Template

[Important]Important

A traced version of an AsyncRestTemplate bean is registered for you out of the box. If you -have your own bean you have to wrap it in a TraceAsyncRestTemplate representation. The best solution -is to only customize the ClientHttpRequestFactory and / or AsyncClientHttpRequestFactory. -If you have your own AsyncRestTemplate and you don’t wrap it your calls WILL NOT GET TRACED.

Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the ClientHttpRequestFactory -and the AsyncClientHttpRequestFactory by registering your beans. Remember to use tracing compatible implementations (e.g. don’t forget to -wrap ThreadPoolTaskScheduler in a TraceAsyncListenableTaskExecutor). Example of custom request factories:

@EnableAutoConfiguration
-@Configuration
-public static class TestConfiguration {
-
-	@Bean
-	ClientHttpRequestFactory mySyncClientFactory() {
-		return new MySyncClientHttpRequestFactory();
-	}
-
-	@Bean
-	AsyncClientHttpRequestFactory myAsyncClientFactory() {
-		return new MyAsyncClientHttpRequestFactory();
-	}
-}

To block the AsyncRestTemplate features set spring.sleuth.web.async.client.enabled to false. -To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper set spring.sleuth.web.async.client.factory.enabled -to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false.

Multiple Asynchronous Rest Templates

Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you -can see an example of how to set up such a custom AsyncRestTemplate.

@Configuration
+};

15.4 RxJava

We registering a custom RxJavaSchedulersHook that wraps all Action0 instances in their Sleuth representative, which is called TraceAction. +The hook either starts or continues a span, depending on whether tracing was already going on before the Action was scheduled. +To disable the custom RxJavaSchedulersHook, set the spring.sleuth.rxjava.schedulers.hook.enabled to false.

You can define a list of regular expressions for thread names for which you do not want spans to be created. +To do so, provide a comma-separated list of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

[Important]Important

The suggest approach to reactive programming and Sleuth is to use +the Reactor support.

15.5 HTTP integration

Features from this section can be disabled by setting the spring.sleuth.web.enabled property with value equal to false.

15.5.1 HTTP Filter

Through the TracingFilter, all sampled incoming requests result in creation of a Span. +That Span’s name is http: + the path to which the request was sent. +For example, if the request was sent to /this/that then the name will be http:/this/that. +You can configure which URIs you would like to skip by setting the spring.sleuth.web.skipPattern property. +If you have ManagementServerProperties on classpath, its value of contextPath gets appended to the provided skip pattern. +If you want to reuse the Sleuth’s default skip patterns and just append your own, pass those patterns by using the spring.sleuth.web.additionalSkipPattern.

15.5.2 HandlerInterceptor

Since we want the span names to be precise, we use a TraceHandlerInterceptor that either wraps an existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. +The TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. +If the the TracingFilter does not see this attribute, it creates a "fallback" span, which is an additional span created on the server side so that the trace is presented properly in the UI. +If that happens, there is probably missing instrumentation. +In that case, please file an issue in Spring Cloud Sleuth.

15.5.3 Async Servlet support

If your controller returns a Callable or a WebAsyncTask, Spring Cloud Sleuth continues the existing span instead of creating a new one.

15.5.4 WebFlux support

Through TraceWebFilter, all sampled incoming requests result in creation of a Span. +That Span’s name is http: + the path to which the request was sent. +For example, if the request was sent to /this/that, the name is http:/this/that. +You can configure which URIs you would like to skip by using the spring.sleuth.web.skipPattern property. +If you have ManagementServerProperties on the classpath, its value of contextPath gets appended to the provided skip pattern. +If you want to reuse Sleuth’s default skip patterns and append your own, pass those patterns by using the spring.sleuth.web.additionalSkipPattern.

15.5.5 Dubbo RPC support

Via the integration with Brave, Spring Cloud Sleuth supports Dubbo. +It’s enough to add the brave-instrumentation-dubbo-rpc dependency:

<dependency>
+    <groupId>io.zipkin.brave</groupId>
+    <artifactId>brave-instrumentation-dubbo-rpc</artifactId>
+</dependency>

You need to also set a dubbo.properties file with the following contents:

dubbo.provider.filter=tracing
+dubbo.consumer.filter=tracing

You can read more about Brave - Dubbo integration here. +An example of Spring Cloud Sleuth and Dubbo can be found here.

15.6 HTTP Client Integration

15.6.1 Synchronous Rest Template

We inject a RestTemplate interceptor to ensure that all the tracing information is passed to the requests. +Each time a call is made, a new Span is created. +It gets closed upon receiving the response. +To block the synchronous RestTemplate features, set spring.sleuth.web.client.enabled to false.

[Important]Important

You have to register RestTemplate as a bean so that the interceptors get injected. +If you create a RestTemplate instance with a new keyword, the instrumentation does NOT work.

15.6.2 Asynchronous Rest Template

[Important]Important

Starting with Sleuth 2.0.0, we no longer register a bean of AsyncRestTemplate type. +It is up to you to create such a bean. +Then we instrument it.

To block the AsyncRestTemplate features, set spring.sleuth.web.async.client.enabled to false. +To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper, set spring.sleuth.web.async.client.factory.enabled +to false. +If you do not want to create AsyncRestClient at all, set spring.sleuth.web.async.client.template.enabled to false.

Multiple Asynchronous Rest Templates

Sometimes you need to use multiple implementations of the Asynchronous Rest Template. +In the following snippet, you can see an example of how to set up such a custom AsyncRestTemplate:

@Configuration
 @EnableAutoConfiguration
 static class Config {
-	@Autowired Tracer tracer;
-	@Autowired HttpTraceKeysInjector httpTraceKeysInjector;
-	@Autowired HttpSpanInjector spanInjector;
 
 	@Bean(name = "customAsyncRestTemplate")
-	public AsyncRestTemplate traceAsyncRestTemplate(@Qualifier("customHttpRequestFactoryWrapper")
-			TraceAsyncClientHttpRequestFactoryWrapper wrapper, ErrorParser errorParser) {
-		return new TraceAsyncRestTemplate(wrapper, this.tracer, errorParser);
-	}
-
-	@Bean(name = "customHttpRequestFactoryWrapper")
-	public TraceAsyncClientHttpRequestFactoryWrapper traceAsyncClientHttpRequestFactory() {
-		return new TraceAsyncClientHttpRequestFactoryWrapper(this.tracer,
-				this.spanInjector,
-				asyncClientFactory(),
-				clientHttpRequestFactory(),
-				this.httpTraceKeysInjector);
+	public AsyncRestTemplate traceAsyncRestTemplate() {
+		return new AsyncRestTemplate(asyncClientFactory(), clientHttpRequestFactory());
 	}
 
 	private ClientHttpRequestFactory clientHttpRequestFactory() {
@@ -116,32 +101,32 @@ can see an example of how to set up such a custom AsyncRes
 		//CUSTOMIZE HERE
 		return factory;
 	}
-}

13.5.3 WebClient

We inject a ExchangeFilterFunction implementation that creates a span and via on success and on -error callbacks takes care of closing client side spans.

[Important]Important

You have to register WebClient as a bean so that the tracing instrumention gets applied. -If you create a WebClient instance with a new keyword then the instrumentation WILL NOT work.

13.5.4 Traverson

If you’re using the Traverson library -it’s enough for you to inject a RestTemplate as a bean into your Traverson object. Since RestTemplate -is already intercepted, you will get full support of tracing in your client. Below you can find a pseudo code -of how to do that:

@Autowired RestTemplate restTemplate;
+}

15.6.3 WebClient

We inject a ExchangeFilterFunction implementation that creates a span and, through on-success and on-error callbacks, takes care of closing client-side spans.

To block this feature, set spring.sleuth.web.client.enabled to false.

[Important]Important

You have to register WebClient as a bean so that the tracing instrumentation gets applied. +If you create a WebClient instance with a new keyword, the instrumentation does NOT work.

15.6.4 Traverson

If you use the Traverson library, you can inject a RestTemplate as a bean into your Traverson object. +Since RestTemplate is already intercepted, you get full support for tracing in your client. The following pseudo code +shows how to do that:

@Autowired RestTemplate restTemplate;
 
 Traverson traverson = new Traverson(URI.create("http://some/address"),
     MediaType.APPLICATION_JSON, MediaType.APPLICATION_JSON_UTF8).setRestOperations(restTemplate);
-// use Traverson

13.6 Feign

By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely -by setting spring.sleuth.feign.enabled to false. If you do so then no Feign related instrumentation will take place.

Part of Feign instrumentation is done via a FeignBeanPostProcessor. You can disable it by providing the spring.sleuth.feign.processor.enabled equal to false. -If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation -however will be still there.

13.7 Asynchronous communication

13.7.1 @Async annotated methods

In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads. -You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false.

If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics:

  • if the method is annotated with @SpanName then the value of the annotation will be the Span’s name
  • if the method is not annotated with @SpanName the Span name will be the annotated method name
  • the Span will be tagged with that method’s class name and the method name too

13.7.2 @Scheduled annotated methods

In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour -by setting the value of spring.sleuth.scheduled.enabled to false.

If you annotate your method with @Scheduled then we’ll automatically create a new Span with the following characteristics:

  • the Span name will be the annotated method name
  • the Span will be tagged with that method’s class name and the method name too

If you want to skip Span creation for some @Scheduled annotated classes you can set the -spring.sleuth.scheduled.skipPattern with a regular expression that will match the fully qualified name of the -@Scheduled annotated class.

[Tip]Tip

If you are using spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask

13.7.3 Executor, ExecutorService and ScheduledExecutorService

We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations -are creating Spans each time a new task is submitted, invoked or scheduled.

Here you can see an example of how to pass tracing information with TraceableExecutorService when working with CompletableFuture:

CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> {
+// use Traverson

15.6.5 Apache HttpClientBuilder and HttpAsyncClientBuilder

We instrument the HttpClientBuilder and HttpAsyncClientBuilder so that +tracing context gets injected to the sent requests.

To block these features, set spring.sleuth.web.client.enabled to false.

15.6.6 Netty HttpClient

We instrument the Netty’s HttpClient.

To block this feature, set spring.sleuth.web.client.enabled to false.

[Important]Important

You have to register HttpClient as a bean so that the instrumentation happens. +If you create a HttpClient instance with a new keyword, the instrumentation does NOT work.

15.6.7 UserInfoRestTemplateCustomizer

We instrument the Spring Security’s UserInfoRestTemplateCustomizer.

To block this feature, set spring.sleuth.web.client.enabled to false.

15.7 Feign

By default, Spring Cloud Sleuth provides integration with Feign through TraceFeignClientAutoConfiguration. +You can disable it entirely by setting spring.sleuth.feign.enabled to false. +If you do so, no Feign-related instrumentation take place.

Part of Feign instrumentation is done through a FeignBeanPostProcessor. +You can disable it by setting spring.sleuth.feign.processor.enabled to false. +If you set it to false, Spring Cloud Sleuth does not instrument any of your custom Feign components. +However, all the default instrumentation is still there.

15.8 Asynchronous Communication

15.8.1 @Async Annotated methods

In Spring Cloud Sleuth, we instrument async-related components so that the tracing information is passed between threads. +You can disable this behavior by setting the value of spring.sleuth.async.enabled to false.

If you annotate your method with @Async, we automatically create a new Span with the following characteristics:

  • If the method is annotated with @SpanName, the value of the annotation is the Span’s name.
  • If the method is not annotated with @SpanName, the Span name is the annotated method name.
  • The span is tagged with the method’s class name and method name.

15.8.2 @Scheduled Annotated Methods

In Spring Cloud Sleuth, we instrument scheduled method execution so that the tracing information is passed between threads. +You can disable this behavior by setting the value of spring.sleuth.scheduled.enabled to false.

If you annotate your method with @Scheduled, we automatically create a new span with the following characteristics:

  • The span name is the annotated method name.
  • The span is tagged with the method’s class name and method name.

If you want to skip span creation for some @Scheduled annotated classes, you can set the spring.sleuth.scheduled.skipPattern with a regular expression that matches the fully qualified name of the @Scheduled annotated class. +If you use spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, a span is created for each Hystrix metrics and sent to Zipkin. +This behavior may be annoying. That’s why, by default, spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask.

15.8.3 Executor, ExecutorService, and ScheduledExecutorService

We provide LazyTraceExecutor, TraceableExecutorService, and TraceableScheduledExecutorService. Those implementations create spans each time a new task is submitted, invoked, or scheduled.

The following example shows how to pass tracing information with TraceableExecutorService when working with CompletableFuture:

CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> {
 	// perform some logic
 	return 1_000_000L;
-}, new TraceableExecutorService(executorService,
+}, new TraceableExecutorService(beanFactory, executorService,
 		// 'calculateTax' explicitly names the span - this param is optional
-		tracer, traceKeys, spanNamer, "calculateTax"));
[Important]Important

Sleuth doesn’t work with parallelStream() out of the box. If you want -to have the tracing information propagated through the stream you have to use the -approach with supplyAsync(...) as presented above.

Customization of Executors

Sometimes you need to set up a custom instance of the AsyncExecutor. In the following snippet you -can see an example of how to set up such a custom Executor.

@Configuration
+		"calculateTax"));
[Important]Important

Sleuth does not work with parallelStream() out of the box. +If you want to have the tracing information propagated through the stream, you have to use the approach with supplyAsync(...), as shown earlier.

Customization of Executors

Sometimes, you need to set up a custom instance of the AsyncExecutor. +The following example shows how to set up such a custom Executor:

@Configuration
 @EnableAutoConfiguration
 @EnableAsync
 static class CustomExecutorConfig extends AsyncConfigurerSupport {
@@ -159,9 +144,13 @@ can see an example of how to set up such a custom Executor
 		executor.initialize();
 		return new LazyTraceExecutor(this.beanFactory, executor);
 	}
-}

13.8 Messaging

Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and -subscribe events. To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false.

You can provide the spring.sleuth.integration.patterns pattern to explicitly -provide the names of channels that you want to include for tracing. By default all channels -are included.

[Important]Important

When using the Executor to build a Spring Integration IntegrationFlow remember to use the untraced version of the Executor. -Decorating Spring Integration Executor Channel with TraceableExecutorService will cause the spans to be improperly closed.

13.9 Zuul

We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data). -To disable Zuul support set the spring.sleuth.zuul.enabled property to false.

Prev   Next
12. Metrics Home 14. Running examples
\ No newline at end of file +}

15.9 Messaging

Features from this section can be disabled by setting the spring.sleuth.messaging.enabled property with value equal to false.

15.9.1 Spring Integration and Spring Cloud Stream

Spring Cloud Sleuth integrates with Spring Integration. +It creates spans for publish and subscribe events. +To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false.

You can provide the spring.sleuth.integration.patterns pattern to explicitly provide the names of channels that you want to include for tracing. +By default, all channels but hystrixStreamOutput channel are included.

[Important]Important

When using the Executor to build a Spring Integration IntegrationFlow, you must use the untraced version of the Executor. +Decorating the Spring Integration Executor Channel with TraceableExecutorService causes the spans to be improperly closed.

15.9.2 Spring RabbitMq

We instrument the RabbitTemplate so that tracing headers get injected +into the message.

To block this feature, set spring.sleuth.messaging.rabbit.enabled to false.

15.9.3 Spring Kafka

We instrument the Spring Kafka’s ProducerFactory and ConsumerFactory +so that tracing headers get injected into the created Spring Kafka’s +Producer and Consumer.

To block this feature, set spring.sleuth.messaging.kafka.enabled to false.

[Note]Note

We do not support context propagation via @KafkaListener annotation. +Check this issue for more information.

15.10 Zuul

We instrument the Zuul Ribbon integration by enriching the Ribbon requests with tracing information. +To disable Zuul support, set the spring.sleuth.zuul.enabled property to false.

Prev   Next
14. Zipkin Stream Span Consumer Home 16. Running examples
\ No newline at end of file diff --git a/2.0.x/multi/multi__introduction.html b/2.0.x/multi/multi__introduction.html index 0e1416cb8..081976882 100644 --- a/2.0.x/multi/multi__introduction.html +++ b/2.0.x/multi/multi__introduction.html @@ -1,52 +1,59 @@ - 1. Introduction
1. Introduction
Prev   Next

1. Introduction

Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.

1.1 Terminology

Spring Cloud Sleuth borrows Dapper’s terminology.

Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an -RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span -is a part of. Spans also have other data, such as descriptions, timestamped events, key-value -annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address).

Spans are started and stopped, and they keep track of their timing information. Once you create a -span, you must stop it at some point in the future.

[Tip]Tip

The initial span that starts a trace is called a root span. The value of span id -of that span is equal to trace id.

Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed -big-data store, a trace might be formed by a put request.

Annotation: is used to record existence of an event in time. Some of the core annotations used to define -the start and stop of a request are:

  • cs - Client Sent - The client has made a request. This annotation depicts the start of the span.
  • sr - Server Received - The server side got the request and will start processing it. -If one subtracts the cs timestamp from this timestamp one will receive the network latency.
  • ss - Server Sent - Annotated upon completion of request processing (when the response -got sent back to the client). If one subtracts the sr timestamp from this timestamp one -will receive the time needed by the server side to process the request.
  • cr - Client Received - Signifies the end of the span. The client has successfully received the -response from the server side. If one subtracts the cs timestamp from this timestamp one -will receive the whole time needed by the client to receive the response from the server.

Visualization of what Span and Trace will look in a system together with the Zipkin annotations:

Trace Info propagation

Each color of a note signifies a span (7 spans - from A to G). If you have such information in the note:

Trace Id = X
+   1. Introduction
1. Introduction
Prev   Next
1. Introduction
Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.
1.1 Terminology
Spring Cloud Sleuth borrows Dapper’s terminology.
Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an RPC.
+Spans are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of.
+Spans also have other data, such as descriptions, timestamped events, key-value annotations (tags), the ID of the span that caused them, and process IDs (normally IP addresses).
Spans can be started and stopped, and they keep track of their timing information.
+Once you create a span, you must stop it at some point in the future.
[Tip]Tip
The initial span that starts a trace is called a root span. The value of the ID
+of that span is equal to the trace ID.
Trace: A set of spans forming a tree-like structure.
+For example, if you run a distributed big-data store, a trace might be formed by a PUT request.
Annotation:  Used to record the existence of an event in time. With
+Brave instrumentation, we no longer need to set special events
+for Zipkin to understand who the client and server are, where
+the request started, and where it ended. For learning purposes,
+however, we mark these events to highlight what kind
+of an action took place.
  • cs: Client Sent. The client has made a request. This annotation indicates the start of the span.
  • sr: Server Received: The server side got the request and started processing it.
+Subtracting the cs timestamp from this timestamp reveals the network latency.
  • ss: Server Sent. Annotated upon completion of request processing (when the response got sent back to the client).
+Subtracting the sr timestamp from this timestamp reveals the time needed by the server side to process the request.
  • cr: Client Received. Signifies the end of the span.
+The client has successfully received the response from the server side.
+Subtracting the cs timestamp from this timestamp reveals the whole time needed by the client to receive the response from the server.
The following image shows how Span and Trace look in a system, together with the Zipkin annotations:
Trace Info propagation
Each color of a note signifies a span (there are seven spans - from A to G).
+Consider the following note:
Trace Id = X
 Span Id = D
-Client Sent
That means that the current span has Trace-Id set to X, Span-Id set to D. It also has emitted
- Client Sent event.
This is how the visualization of the parent / child relationship of spans would look like:
Parent child relationship
1.2 Purpose
In the following sections the example from the image above will be taken into consideration.
1.2.1 Distributed tracing with Zipkin
Altogether there are 7 spans . If you go to traces in Zipkin you will see this number in the second trace:
Traces
However if you pick a particular trace then you will see 4 spans:
Traces Info propagation
[Note]Note
When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to
-Zipkin with Server Received and Server Sent / Client Received and Client Sent
-annotations then they will presented as a single span.
Why is there a difference between the 7 and 4 spans in this case?
  • 2 spans come from http:/start span. It has the Server Received (SR) and Server Sent (SS) annotations.
  • 2 spans come from the RPC call from service1 to service2 to the http:/foo endpoint. It has the Client Sent (CS)
-and Client Received (CR) annotations on service1 side. It also has Server Received (SR) and Server Sent (SS) annotations
-on the service2 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
  • 2 spans come from the RPC call from service2 to service3 to the http:/bar endpoint. It has the Client Sent (CS)
-and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations
-on the service3 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
  • 2 spans come from the RPC call from service2 to service4 to the http:/baz endpoint. It has the Client Sent (CS)
-and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations
-on the service4 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
So if we count the physical spans we have 1 from http:/start, 2 from service1 calling service2, 2 form service2
-calling service3 and 2 from service2 calling service4. Altogether 7 spans.
Logically we see the information of Total Spans: 4 because we have 1 span related to the incoming request
-to service1 and 3 spans related to RPC calls.
1.2.2 Visualizing errors
Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn’t caught then we’re
-setting proper tags on the span which Zipkin can properly colorize. You could see in the list of traces one
-  trace that was in red color. That’s because there was an exception thrown.
If you click that trace then you’ll see a similar picture
Error Traces
Then if you click on one of the spans you’ll see the following
Error Traces Info propagation
As you can see you can easily see the reason for an error and the whole stacktrace related to it.
1.2.3 Live examples
Figure 1.1. Click Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

The dependency graph in Zipkin would look like this:
Dependencies
Figure 1.2. Click Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

1.2.4 Log correlation
When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following:
service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
+Client Sent
This note indicates that the current span has Trace Id set to X and Span Id set to D.
+Also, the Client Sent event took place.
The following image shows how parent-child relationships of spans look:
Parent child relationship
1.2 Purpose
The following sections refer to the example shown in the preceding image.
1.2.1 Distributed Tracing with Zipkin
This example has seven spans.
+If you go to traces in Zipkin, you can see this number in the second trace, as shown in the following image:
Traces
However, if you pick a particular trace, you can see four spans, as shown in the following image:
Traces Info propagation
[Note]Note
When you pick a particular trace, you see merged spans.
+That means that, if there were two spans sent to Zipkin with Server Received and Server Sent or Client Received and Client Sent annotations, they are presented as a single span.
Why is there a difference between the seven and four spans in this case?
  • Two spans come from the http:/start span. It has the Server Received (sr) and Server Sent (ss) annotations.
  • Two spans come from the RPC call from service1 to service2 to the http:/foo endpoint.
+The Client Sent (cs) and Client Received (cr) events took place on the service1 side.
+Server Received (sr) and Server Sent (ss) events took place on the service2 side.
+These two spans form one logical span related to an RPC call.
  • Two spans come from the RPC call from service2 to service3 to the http:/bar endpoint.
+The Client Sent (cs) and Client Received (cr) events took place on the service2 side.
+The Server Received (sr) and Server Sent (ss) events took place on the service3 side.
+These two spans form one logical span related to an RPC call.
  • Two spans come from the RPC call from service2 to service4 to the http:/baz endpoint.
+The Client Sent (cs) and Client Received (cr) events took place on the service2 side.
+Server Received (sr) and Server Sent (ss) events took place on the service4 side.
+These two spans form one logical span related to an RPC call.
So, if we count the physical spans, we have one from http:/start, two from service1 calling service2, two from service2
+calling service3, and two from service2 calling service4. In sum, we have a total of seven spans.
Logically, we see the information of four total Spans because we have one span related to the incoming request
+to service1 and three spans related to RPC calls.
1.2.2 Visualizing errors
Zipkin lets you visualize errors in your trace.
+When an exception was thrown and was not caught, we set proper tags on the span, which Zipkin can then properly colorize.
+You could see in the list of traces one trace that is red. That appears because an exception was thrown.
If you click that trace, you see a similar picture, as follows:
Error Traces
If you then click on one of the spans, you see the following
Error Traces Info propagation
The span shows the reason for the error and the whole stack trace related to it.
1.2.3 Distributed Tracing with Brave
Starting with version 2.0.0, Spring Cloud Sleuth uses Brave as the tracing library.
+Consequently, Sleuth no longer takes care of storing the context but delegates that work to Brave.
Due to the fact that Sleuth had different naming and tagging conventions than Brave, we decided to follow Brave’s conventions from now on.
+However, if you want to use the legacy Sleuth approaches, you can set the spring.sleuth.http.legacy.enabled property to true.
1.2.4 Live examples
Figure 1.1. Click the Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

Click here to see it live!
The dependency graph in Zipkin should resemble the following image:
Dependencies
Figure 1.2. Click the Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

Click here to see it live!
1.2.5 Log correlation
When using grep to read the logs of those four applications by scanning for a trace ID equal to (for example) 2485ec27856c56f4, you get output resembling the following:
service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
 service2.log:2016-02-26 11:15:47.710  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Hello from service2. Calling service3 and then service4
 service3.log:2016-02-26 11:15:47.895  INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application   : Hello from service3
 service2.log:2016-02-26 11:15:47.924  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service3 [Hello from service3]
 service4.log:2016-02-26 11:15:48.134  INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application   : Hello from service4
 service2.log:2016-02-26 11:15:48.156  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service4 [Hello from service4]
-service1.log:2016-02-26 11:15:48.182  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]
If you’re using a log aggregating tool like Kibana,
-Splunk etc. you can order the events that took place. An example of
-Kibana would look like this:
Log correlation with Kibana
If you want to use Logstash here is the Grok pattern for Logstash:
filter {
+service1.log:2016-02-26 11:15:48.182  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]
If you use a log aggregating tool (such as Kibana, Splunk, and others), you can order the events that took place.
+An example from Kibana would resemble the following image:
Log correlation with Kibana
If you want to use Logstash, the following listing shows the Grok pattern for Logstash:
filter {
        # pattern matching logback pattern
        grok {
               match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
        }
-}
[Note]Note
If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern:
filter {
+}
[Note]Note
If you want to use Grok together with the logs from Cloud Foundry, you have to use the following pattern:
filter {
        # pattern matching logback pattern
        grok {
               match => { "message" => "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
        }
-}
JSON Logback with Logstash
Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability
-we’re passing the dependencies in the groupId:artifactId:version notation.
Dependencies setup
  • Ensure that Logback is on the classpath (ch.qos.logback:logback-core)
  • Add Logstash Logback encode - example for version 4.6 : net.logstash.logback:logstash-logback-encoder:4.6
Logback setup
Below you can find an example of a Logback configuration (file named logback-spring.xml) that:
  • logs information from the application in a JSON format to a build/${spring.application.name}.json file
  • has commented out two additional appenders - console and standard log file
  • has the same logging pattern as the one presented in the previous section
<?xml version="1.0" encoding="UTF-8"?>
+}
JSON Logback with Logstash
Often, you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick.
+To do so, you have to do the following (for readability, we pass the dependencies in the groupId:artifactId:version notation).
Dependencies Setup
  1. Ensure that Logback is on the classpath (ch.qos.logback:logback-core).
  2. Add Logstash Logback encode. For example, to use version 4.6, add net.logstash.logback:logstash-logback-encoder:4.6.
Logback Setup
Consider the following example of a Logback configuration file (named logback-spring.xml).
<?xml version="1.0" encoding="UTF-8"?>
 <configuration>
 	<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
 	​
@@ -121,44 +128,51 @@ we’re passing the dependencies in the groupId:artifa
 		<!--<appender-ref ref="logstash"/>-->
 		<!--<appender-ref ref="flatfile"/>-->
 	</root>
-</configuration>
[Note]Note
If you’re using a custom logback-spring.xml then you have to pass the spring.application.name in
-bootstrap instead of application property file. Otherwise your custom logback file won’t read the property properly.
1.2.5 Propagating Span Context
The span context is the state that must get propagated to any child Spans across process boundaries.
+</configuration>

That Logback configuration file:

  • Logs information from the application in a JSON format to a build/${spring.application.name}.json file.
  • Has commented out two additional appenders: console and standard log file.
  • Has the same logging pattern as the one presented in the previous section.
[Note]Note

If you use a custom logback-spring.xml, you must pass the spring.application.name in the bootstrap rather than the application property file. +Otherwise, your custom logback file does not properly read the property.

1.2.6 Propagating Span Context

The span context is the state that must get propagated to any child spans across process boundaries. Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context. -Baggage is an optional part.

Baggage is a set of key:value pairs stored in the span context. Baggage travels together with the trace -and is attached to every span. Spring Cloud Sleuth will understand that a header is baggage related if the HTTP - header is prefixed with baggage- and for messaging it starts with baggage_.

[Important]Important

There’s currently no limitation of the count or size of baggage items. However, keep in mind that -too many can decrease system throughput or increase RPC latency. In extreme cases, it could crash the app due -to exceeding transport-level message or header capacity.

Example of setting baggage on a span:

Span initialSpan = this.tracer.createSpan("span");
-initialSpan.setBaggageItem("foo", "bar");
-initialSpan.setBaggageItem("UPPER_CASE", "someValue");

Baggage vs. Span Tags

Baggage travels with the trace (i.e. every child span contains the baggage of its parent). Zipkin has no knowledge of -baggage and will not even receive that information.

Tags are attached to a specific span - they are presented for that particular span only. However you -can search by tag to find the trace, where there exists a span having the searched tag value.

If you want to be able to lookup a span based on baggage, you should add corresponding entry as a tag in the root span.

@Autowired Tracer tracer;
-
-Span span = tracer.getCurrentSpan();
-String baggageKey = "key";
-String baggageValue = "foo";
-span.setBaggageItem(baggageKey, baggageValue);
-tracer.addTag(baggageKey, baggageValue);

1.3 Adding to the project

[Important]Important

To ensure that your application name is properly displayed in Zipkin - set the spring.application.name property in bootstrap.yml.

1.3.1 Only Sleuth (log correlation)

If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add -the spring-cloud-starter-sleuth module to your project.

Maven.  +Baggage is an optional part.

Baggage is a set of key:value pairs stored in the span context. +Baggage travels together with the trace and is attached to every span. +Spring Cloud Sleuth understands that a header is baggage-related if the HTTP header is prefixed with baggage- and, for messaging, it starts with baggage_.

[Important]Important

There is currently no limitation of the count or size of baggage items. +However, keep in mind that too many can decrease system throughput or increase RPC latency. +In extreme cases, too much baggage can crash the application, due to exceeding transport-level message or header capacity.

The following example shows setting baggage on a span:

Span initialSpan = this.tracer.nextSpan().name("span").start();
+try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) {
+	ExtraFieldPropagation.set("foo", "bar");
+	ExtraFieldPropagation.set("UPPER_CASE", "someValue");
+}

Baggage versus Span Tags

Baggage travels with the trace (every child span contains the baggage of its parent). +Zipkin has no knowledge of baggage and does not receive that information.

[Important]Important

Starting from Sleuth 2.0.0 you have to pass the baggage key names explicitly +in your project configuration. Read more about that setup here

Tags are attached to a specific span. In other words, they are presented only for that particular span. +However, you can search by tag to find the trace, assuming a span having the searched tag value exists.

If you want to be able to lookup a span based on baggage, you should add a corresponding entry as a tag in the root span.

[Important]Important

The span must be in scope.

The following listing shows integration tests that use baggage:

The setup.  +

spring.sleuth:
+  baggage-keys:
+    - baz
+    - bizarrecase
+  propagation-keys:
+    - foo
+    - upper_case

+

The code.  +

initialSpan.tag("foo",
+		ExtraFieldPropagation.get(initialSpan.context(), "foo"));
+initialSpan.tag("UPPER_CASE",
+		ExtraFieldPropagation.get(initialSpan.context(), "UPPER_CASE"));

+

1.3 Adding Sleuth to the Project

This section addresses how to add Sleuth to your project with either Maven or Gradle.

[Important]Important

To ensure that your application name is properly displayed in Zipkin, set the spring.application.name property in bootstrap.yml.

1.3.1 Only Sleuth (log correlation)

If you want to use only Spring Cloud Sleuth without the Zipkin integration, add the spring-cloud-starter-sleuth module to your project.

The following example shows how to add Sleuth with Maven:

Maven.  

<dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
+      <dependencies>
+          <dependency>
+              <groupId>org.springframework.cloud</groupId>
+              <artifactId>spring-cloud-dependencies</artifactId>
+              <version>${release.train.version}</version>
+              <type>pom</type>
+              <scope>import</scope>
+          </dependency>
+      </dependencies>
+</dependencyManagement>
 
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-sleuth</artifactId>
-   </dependency>

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-sleuth

Gradle.  +<dependency> 2 + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-sleuth</artifactId> +</dependency>

+

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-sleuth.

The following example shows how to add Sleuth with Gradle:

Gradle.  

dependencyManagement { 1
     imports {
         mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
@@ -168,26 +182,24 @@ the Spring BOM
2
     compile "org.springframework.cloud:spring-cloud-starter-sleuth"
 }

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-sleuth

1.3.2 Sleuth with Zipkin via HTTP

If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency.

Maven.  +

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-sleuth.

1.3.2 Sleuth with Zipkin via HTTP

If you want both Sleuth and Zipkin, add the spring-cloud-starter-zipkin dependency.

The following example shows how to do so for Maven:

Maven.  

<dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
+      <dependencies>
+          <dependency>
+              <groupId>org.springframework.cloud</groupId>
+              <artifactId>spring-cloud-dependencies</artifactId>
+              <version>${release.train.version}</version>
+              <type>pom</type>
+              <scope>import</scope>
+          </dependency>
+      </dependencies>
+</dependencyManagement>
 
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-zipkin</artifactId>
-   </dependency>

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin

Gradle.  +<dependency> 2 + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-zipkin</artifactId> +</dependency>

+

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-zipkin.

The following example shows how to do so for Gradle:

Gradle.  

dependencyManagement { 1
     imports {
         mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
@@ -197,32 +209,30 @@ the Spring BOM
2
     compile "org.springframework.cloud:spring-cloud-starter-zipkin"
 }

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin

1.3.3 Sleuth with Zipkin via RabbitMQ or Kafka

If you want to use RabbitMQ or Kafka instead of http, add the spring-rabbit or spring-kafka -dependencies. The default destination name is zipkin.

Note: spring-cloud-sleuth-stream is deprecated and incompatible with these destinations

If you want Sleuth over RabbitMQ add the spring-cloud-starter-zipkin and spring-rabbit -dependencies.

Maven.  +

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-zipkin.

1.3.3 Sleuth with Zipkin over RabbitMQ or Kafka

If you want to use RabbitMQ or Kafka instead of HTTP, add the spring-rabbit or spring-kafka dependency. +The default destination name is zipkin.

If using Kafka, you must set the property spring.zipkin.sender.type property accordingly:

spring.zipkin.sender.type: kafka
[Caution]Caution

spring-cloud-sleuth-stream is deprecated and incompatible with these destinations.

If you want Sleuth over RabbitMQ, add the spring-cloud-starter-zipkin and spring-rabbit +dependencies.

The following example shows how to do so for Gradle:

Maven.  

<dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
+      <dependencies>
+          <dependency>
+              <groupId>org.springframework.cloud</groupId>
+              <artifactId>spring-cloud-dependencies</artifactId>
+              <version>${release.train.version}</version>
+              <type>pom</type>
+              <scope>import</scope>
+          </dependency>
+      </dependencies>
+</dependencyManagement>
 
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-zipkin</artifactId>
-   </dependency>
-   <dependency> 3
-       <groupId>org.springframework.amqp</groupId>
-       <artifactId>spring-rabbit</artifactId>
-   </dependency>

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin - that way all dependent dependencies will be downloaded

3

To automatically configure rabbit, simply add the spring-rabbit dependency

Gradle.  +<dependency> 2 + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-zipkin</artifactId> +</dependency> +<dependency> 3 + <groupId>org.springframework.amqp</groupId> + <artifactId>spring-rabbit</artifactId> +</dependency>

+

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded.

3

To automatically configure RabbitMQ, add the spring-rabbit dependency.

Gradle.  

dependencyManagement { 1
     imports {
         mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
@@ -233,5 +243,4 @@ dependencies {
     compile "org.springframework.cloud:spring-cloud-starter-zipkin" 2
     compile "org.springframework.amqp:spring-rabbit" 3
 }

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin - that way all dependent dependencies will be downloaded

3

To automatically configure rabbit, simply add the spring-rabbit dependency

Prev   Next
 Home 2. Additional resources
\ No newline at end of file +

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded.

3

To automatically configure RabbitMQ, add the spring-rabbit dependency.

Prev   Next
 Home 2. Additional Resources
\ No newline at end of file diff --git a/2.0.x/multi/multi__managing_spans_with_annotations.html b/2.0.x/multi/multi__managing_spans_with_annotations.html index 4820f37f8..2b8601f0a 100644 --- a/2.0.x/multi/multi__managing_spans_with_annotations.html +++ b/2.0.x/multi/multi__managing_spans_with_annotations.html @@ -1,47 +1,43 @@ - 8. Managing spans with annotations
8. Managing spans with annotations
Prev   Next

8. Managing spans with annotations

8.1 Rationale

The main arguments for this features are

  • api-agnostic means to collaborate with a span

    • use of annotations allows users to add to a span with no library dependency on a span api. -This allows Sleuth to change its core api less impact to user code.

  • reduced surface area for basic span operations.

    • without this feature one has to use the span api, which has lifecycle commands that -could be used incorrectly. By only exposing scope, tag and log functionality, users can -collaborate without accidentally breaking span lifecycle.

  • collaboration with runtime generated code

    • with libraries such as Spring Data / Feign the implementations of interfaces are generated -at runtime thus span wrapping of objects was tedious. Now you can provide annotations - over interfaces and arguments of those interfaces

8.2 Creating new spans

If you really don’t want to take care of creating local spans manually you can profit from the -@NewSpan annotation. Also we give you the @SpanTag annotation to add tags in an automated -fashion.

Let’s look at some examples of usage.

@NewSpan
-void testMethod();

Annotating the method without any parameter will lead to a creation of a new span whose name -will be equal to annotated method name.

@NewSpan("customNameOnTestMethod4")
-void testMethod4();

If you provide the value in the annotation (either directly or via the name parameter) then -the created span will have the name as the provided value.

// method declaration
+   11. Managing Spans with Annotations
11. Managing Spans with Annotations
Prev   Next
11. Managing Spans with Annotations
You can manage spans with a variety of annotations.
11.1 Rationale
There are a number of good reasons to manage spans with annotations, including:
  • API-agnostic means to collaborate with a span. Use of annotations lets users add to a span with no library dependency on a span api.
+Doing so lets Sleuth change its core API to create less impact to user code.
  • Reduced surface area for basic span operations. Without this feature, you must use the span api, which has lifecycle commands that could be used incorrectly.
+By only exposing scope, tag, and log functionality, you can collaborate without accidentally breaking span lifecycle.
  • Collaboration with runtime generated code. With libraries such as Spring Data and Feign, the implementations of interfaces are generated at runtime.
+Consequently, span wrapping of objects was tedious.
+Now you can provide annotations over interfaces and the arguments of those interfaces.
11.2 Creating New Spans
If you do not want to create local spans manually, you can use the @NewSpan annotation.
+Also, we provide the @SpanTag annotation to add tags in an automated fashion.
Now we can consider some examples of usage.
@NewSpan
+void testMethod();
Annotating the method without any parameter leads to creating a new span whose name equals the annotated method name.
@NewSpan("customNameOnTestMethod4")
+void testMethod4();
If you provide the value in the annotation (either directly or by setting the name parameter), the created span has the provided value as the name.
// method declaration
 @NewSpan(name = "customNameOnTestMethod5")
 void testMethod5(@SpanTag("testTag") String param);
 
 // and method execution
-this.testBean.testMethod5("test");
You can combine both the name and a tag. Let’s focus on the latter. In this case whatever the value of
-the annotated method’s parameter runtime value will be - that will be the value of the tag. In our sample
-the tag key will be testTag and the tag value will be test.
@NewSpan(name = "customNameOnTestMethod3")
+this.testBean.testMethod5("test");
You can combine both the name and a tag. Let’s focus on the latter.
+In this case, the value of the annotated method’s parameter runtime value becomes the value of the tag.
+In our sample, the tag key is testTag, and the tag value is test.
@NewSpan(name = "customNameOnTestMethod3")
 @Override
 public void testMethod3() {
-}
You can place the @NewSpan annotation on both the class and an interface. If you override the
-interface’s method and provide a different value of the @NewSpan annotation then the most
-concrete one wins (in this case customNameOnTestMethod3 will be set).
8.3 Continuing spans
If you want to just add tags and annotations to an existing span it’s enough
-to use the @ContinueSpan annotation as presented below. Note that in contrast
-with the @NewSpan annotation you can also add logs via the log parameter:
// method declaration
+}
You can place the @NewSpan annotation on both the class and an interface.
+If you override the interface’s method and provide a different value for the @NewSpan annotation, the most
+concrete one wins (in this case customNameOnTestMethod3 is set).
11.3 Continuing Spans
If you want to add tags and annotations to an existing span, you can use the @ContinueSpan annotation, as shown in the following example:
// method declaration
 @ContinueSpan(log = "testMethod11")
 void testMethod11(@SpanTag("testTag11") String param);
 
 // method execution
-this.testBean.testMethod11("test");
That way the span will get continued and:
  • logs with name testMethod11.before and testMethod11.after will be created
  • if an exception will be thrown a log testMethod11.afterFailure will also be created
  • tag with key testTag11 and value test will be created
8.4 More advanced tag setting
There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation.
-Precedence is:
  • try with the bean of TagValueResolver type and provided name
  • if one hasn’t provided the bean name, try to evaluate an expression. We’re searching for a TagValueExpressionResolver bean.
-The default implementation uses SPEL expression resolution.
  • if one hasn’t provided any expression to evaluate just return a toString() value of the parameter
8.4.1 Custom extractor
The value of the tag for following method will be computed by an implementation of TagValueResolver interface.
-Its class name has to be passed as the value of the resolver attribute.
Having such an annotated method:
@NewSpan
+this.testBean.testMethod11("test");
+this.testBean.testMethod13();
(Note that, in contrast with the @NewSpan annotation ,you can also add logs with the log parameter.)
That way, the span gets continued and:
  • Log entries named testMethod11.before and testMethod11.after are created.
  • If an exception is thrown, a log entry named testMethod11.afterFailure is also created.
  • A tag with a key of testTag11 and a value of test is created.
11.4 Advanced Tag Setting
There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation.
+The precedence is as follows:
  1. Try with a bean of TagValueResolver type and a provided name.
  2. If the bean name has not been provided, try to evaluate an expression.
+We search for a TagValueExpressionResolver bean.
+The default implementation uses SPEL expression resolution.
+IMPORTANT You can only reference properties from the SPEL expression. Method execution is not allowed due to security constraints.
  3. If we do not find any expression to evaluate, return the toString() value of the parameter.
11.4.1 Custom extractor
The value of the tag for the following method is computed by an implementation of TagValueResolver interface.
+Its class name has to be passed as the value of the resolver attribute.
Consider the following annotated method:
@NewSpan
 public void getAnnotationForTagValueResolver(@SpanTag(key = "test", resolver = TagValueResolver.class) String test) {
-}
and such a TagValueResolver bean implementation
@Bean(name = "myCustomTagValueResolver")
+}
Now further consider the following TagValueResolver bean implementation:
@Bean(name = "myCustomTagValueResolver")
 public TagValueResolver tagValueResolver() {
 	return parameter -> "Value from myCustomTagValueResolver";
-}
Will lead to setting of a tag value equal to Value from myCustomTagValueResolver.
8.4.2 Resolving expressions for value
Having such an annotated method:
@NewSpan
-public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "length() + ' characters'") String test) {
-}
and no custom implementation of a TagValueExpressionResolver will lead to evaluation of the SPEL expression and a tag with value 4 characters will be set on the span.
-If you want to use some other expression resolution mechanism you can create your own implementation
-of the bean.
8.4.3 Using toString method
Having such an annotated method:
@NewSpan
+}
The two preceding examples lead to setting a tag value equal to Value from myCustomTagValueResolver.
11.4.2 Resolving Expressions for a Value
Consider the following annotated method:
@NewSpan
+public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "'hello' + ' characters'") String test) {
+}
No custom implementation of a TagValueExpressionResolver leads to evaluation of the SPEL expression, and a tag with a value of 4 characters is set on the span.
+If you want to use some other expression resolution mechanism, you can create your own implementation of the bean.
11.4.3 Using the toString() method
Consider the following annotated method:
@NewSpan
 public void getAnnotationForArgumentToString(@SpanTag("test") Long param) {
-}
if executed with a value of 15 will lead to setting of a tag with a String value of "15".
Prev   Next
7. Naming spans Home 9. Customizations

\ No newline at end of file
+}

Running the preceding method with a value of 15 leads to setting a tag with a String value of "15".

Prev   Next
10. Naming spans Home 12. Customizations
\ No newline at end of file diff --git a/2.0.x/multi/multi__naming_spans.html b/2.0.x/multi/multi__naming_spans.html index 3ce9a30d9..abfa42cf2 100644 --- a/2.0.x/multi/multi__naming_spans.html +++ b/2.0.x/multi/multi__naming_spans.html @@ -1,19 +1,20 @@ - 7. Naming spans
7. Naming spans
Prev   Next

7. Naming spans

Picking a span name is not a trivial task. Span name should depict an operation name. The name should -be low cardinality (e.g. not include identifiers).

Since there is a lot of instrumentation going on some of the span names will be -artificial like:

  • controller-method-name when received by a Controller with a method name conrollerMethodName
  • async for asynchronous operations done via wrapped Callable and Runnable.
  • @Scheduled annotated methods will return the simple name of the class.

Fortunately, for the asynchronous processing you can provide explicit naming.

7.1 @SpanName annotation

You can name the span explicitly via the @SpanName annotation.

@SpanName("calculateTax")
+   10. Naming spans
10. Naming spans
Prev   Next
10. Naming spans
Picking a span name is not a trivial task. A span name should depict an operation name.
+The name should be low cardinality, so it should not include identifiers.
Since there is a lot of instrumentation going on, some span names are artificial:
  • controller-method-name when received by a Controller with a method name of controllerMethodName
  • async for asynchronous operations done with wrapped Callable and Runnable interfaces.
  • Methods annotated with @Scheduled return the simple name of the class.
Fortunately, for asynchronous processing, you can provide explicit naming.
10.1 @SpanName Annotation
You can name the span explicitly by using the @SpanName annotation, as shown in the following example:
@SpanName("calculateTax")
 class TaxCountingRunnable implements Runnable {
 
 	@Override public void run() {
 		// perform logic
 	}
-}
In this case, when processed in the following manner:
Runnable runnable = new TraceRunnable(tracer, spanNamer, new TaxCountingRunnable());
+}
In this case, when processed in the following manner, the span is named calculateTax:
Runnable runnable = new TraceRunnable(tracing, spanNamer,
+		new TaxCountingRunnable());
 Future<?> future = executorService.submit(runnable);
 // ... some additional logic ...
-future.get();
The span will be named calculateTax.
7.2 toString() method
It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous
-instance of those classes. You can’t annotate such classes thus to override that, if there is no @SpanName annotation present,
-we’re checking if the class has a custom implementation of the toString() method.
So executing such code:
Runnable runnable = new TraceRunnable(tracer, spanNamer, new Runnable() {
+future.get();
10.2 toString() method
It is pretty rare to create separate classes for Runnable or Callable.
+Typically, one creates an anonymous instance of those classes.
+You cannot annotate such classes.
+To overcome that limitation, if there is no @SpanName annotation present, we check whether the class has a custom implementation of the toString() method.
Running such code leads to creating a span named calculateTax, as shown in the following example:
Runnable runnable = new TraceRunnable(tracing, spanNamer, new Runnable() {
 	@Override public void run() {
 		// perform logic
 	}
@@ -24,4 +25,4 @@ we’re checking if the class has a custom implementation of the // ... some additional logic ...
-future.get();
will lead in creating a span named calculateTax.
Prev   Next
6. Span lifecycle Home 8. Managing spans with annotations

\ No newline at end of file
+future.get();
Prev   Next
9. Span lifecycle Home 11. Managing Spans with Annotations
\ No newline at end of file diff --git a/2.0.x/multi/multi__propagation.html b/2.0.x/multi/multi__propagation.html new file mode 100644 index 000000000..e00d22cfc --- /dev/null +++ b/2.0.x/multi/multi__propagation.html @@ -0,0 +1,92 @@ + + + 5. Propagation
5. Propagation
Prev   Next

5. Propagation

Propagation is needed to ensure activities originating from the same root are collected together in the same trace. +The most common propagation approach is to copy a trace context from a client by sending an RPC request to a server receiving it.

For example, when a downstream HTTP call is made, its trace context is encoded as request headers and sent along with it, as shown in the following image:

   Client Span                                                Server Span
+┌──────────────────┐                                       ┌──────────────────┐
+│                  │                                       │                  │
+│   TraceContext   │           Http Request Headers        │   TraceContext   │
+│ ┌──────────────┐ │          ┌───────────────────┐        │ ┌──────────────┐ │
+│ │ TraceId      │ │          │ X─B3─TraceId      │        │ │ TraceId      │ │
+│ │              │ │          │                   │        │ │              │ │
+│ │ ParentSpanId │ │ Extract  │ X─B3─ParentSpanId │ Inject │ │ ParentSpanId │ │
+│ │              ├─┼─────────>│                   ├────────┼>│              │ │
+│ │ SpanId       │ │          │ X─B3─SpanId       │        │ │ SpanId       │ │
+│ │              │ │          │                   │        │ │              │ │
+│ │ Sampled      │ │          │ X─B3─Sampled      │        │ │ Sampled      │ │
+│ └──────────────┘ │          └───────────────────┘        │ └──────────────┘ │
+│                  │                                       │                  │
+└──────────────────┘                                       └──────────────────┘

The names above are from B3 Propagation, which is built-in to Brave and has implementations in many languages and frameworks.

Most users use a framework interceptor to automate propagation. +The next two examples show how that might work for a client and a server.

The following example shows how client-side propagation might work:

@Autowired Tracing tracing;
+
+// configure a function that injects a trace context into a request
+injector = tracing.propagation().injector(Request.Builder::addHeader);
+
+// before a request is sent, add the current span's context to it
+injector.inject(span.context(), request);

The following example shows how server-side propagation might work:

@Autowired Tracing tracing;
+@Autowired Tracer tracer;
+
+// configure a function that extracts the trace context from a request
+extractor = tracing.propagation().extractor(Request::getHeader);
+
+// when a server receives a request, it joins or starts a new trace
+span = tracer.nextSpan(extractor.extract(request));

5.1 Propagating extra fields

Sometimes you need to propagate extra fields, such as a request ID or an alternate trace context. +For example, if you are in a Cloud Foundry environment, you might want to pass the request ID, as shown in the following example:

// when you initialize the builder, define the extra field you want to propagate
+Tracing.newBuilder().propagationFactory(
+  ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-vcap-request-id")
+);
+
+// later, you can tag that request ID or use it in log correlation
+requestId = ExtraFieldPropagation.get("x-vcap-request-id");

You may also need to propagate a trace context that you are not using. +For example, you may be in an Amazon Web Services environment but not be reporting data to X-Ray. +To ensure X-Ray can co-exist correctly, pass-through its tracing header, as shown in the following example:

tracingBuilder.propagationFactory(
+  ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-amzn-trace-id")
+);
[Tip]Tip

In Spring Cloud Sleuth all elements of the tracing builder Tracing.newBuilder() +are defined as beans. So if you want to pass a custom PropagationFactory, it’s enough +for you to create a bean of that type and we will set it in the Tracing bean.

5.1.1 Prefixed fields

If they follow a common pattern, you can also prefix fields. +The following example shows how to propagate x-vcap-request-id the field as-is but send the country-code and user-id fields on the wire as x-baggage-country-code and x-baggage-user-id, respectively:

Tracing.newBuilder().propagationFactory(
+  ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY)
+                       .addField("x-vcap-request-id")
+                       .addPrefixedFields("baggage-", Arrays.asList("country-code", "user-id"))
+                       .build()
+);

Later, you can call the following code to affect the country code of the current trace context:

ExtraFieldPropagation.set("country-code", "FO");
+String countryCode = ExtraFieldPropagation.get("country-code");

Alternatively, if you have a reference to a trace context, you can use it explicitly, as shown in the following example:

ExtraFieldPropagation.set(span.context(), "country-code", "FO");
+String countryCode = ExtraFieldPropagation.get(span.context(), "country-code");
[Important]Important

A difference from previous versions of Sleuth is that, with Brave, you must pass the list of baggage keys. +There are two properties to achieve this. +With the spring.sleuth.baggage-keys, you set keys that get prefixed with baggage- for HTTP calls and baggage_ for messaging. +You can also use the spring.sleuth.propagation-keys property to pass a list of prefixed keys that are whitelisted without any prefix.

5.1.2 Extracting a Propagated Context

The TraceContext.Extractor<C> reads trace identifiers and sampling status from an incoming request or message. +The carrier is usually a request object or headers.

This utility is used in standard instrumentation (such as HttpServerHandler`) but can also be used for custom RPC or messaging code.

TraceContextOrSamplingFlags is usually used only with Tracer.nextSpan(extracted), unless you are +sharing span IDs between a client and a server.

5.1.3 Sharing span IDs between Client and Server

A normal instrumentation pattern is to create a span representing the server side of an RPC. +Extractor.extract might return a complete trace context when applied to an incoming client request. +Tracer.joinSpan attempts to continue this trace, using the same span ID if supported or creating a child span +if not. When the span ID is shared, the reported data includes a flag saying so.

The following image shows an example of B3 propagation:

                              ┌───────────────────┐      ┌───────────────────┐
+ Incoming Headers             │   TraceContext    │      │   TraceContext    │
+┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
+│ X─B3-TraceId      │─────────┼─┼> TraceId      │ │──────┼─┼> TraceId      │ │
+│                   │         │ │               │ │      │ │               │ │
+│ X─B3-ParentSpanId │─────────┼─┼> ParentSpanId │ │──────┼─┼> ParentSpanId │ │
+│                   │         │ │               │ │      │ │               │ │
+│ X─B3-SpanId       │─────────┼─┼> SpanId       │ │──────┼─┼> SpanId       │ │
+└───────────────────┘         │ │               │ │      │ │               │ │
+                              │ │               │ │      │ │  Shared: true │ │
+                              │ └───────────────┘ │      │ └───────────────┘ │
+                              └───────────────────┘      └───────────────────┘

Some propagation systems forward only the parent span ID, detected when Propagation.Factory.supportsJoin() == false. +In this case, a new span ID is always provisioned, and the incoming context determines the parent ID.

The following image shows an example of AWS propagation:

                              ┌───────────────────┐      ┌───────────────────┐
+ x-amzn-trace-id              │   TraceContext    │      │   TraceContext    │
+┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
+│ Root              │─────────┼─┼> TraceId      │ │──────┼─┼> TraceId      │ │
+│                   │         │ │               │ │      │ │               │ │
+│ Parent            │─────────┼─┼> SpanId       │ │──────┼─┼> ParentSpanId │ │
+└───────────────────┘         │ └───────────────┘ │      │ │               │ │
+                              └───────────────────┘      │ │  SpanId: New  │ │
+                                                         │ └───────────────┘ │
+                                                         └───────────────────┘

Note: Some span reporters do not support sharing span IDs. +For example, if you set Tracing.Builder.spanReporter(amazonXrayOrGoogleStackdrive), you should disable join by setting Tracing.Builder.supportsJoin(false). +Doing so forces a new child span on Tracer.joinSpan().

5.1.4 Implementing Propagation

TraceContext.Extractor<C> is implemented by a Propagation.Factory plugin. +Internally, this code creates the union type, TraceContextOrSamplingFlags, with one of the following: +* TraceContext if trace and span IDs were present. +* TraceIdContext if a trace ID was present but span IDs were not present. +* SamplingFlags if no identifiers were present.

Some Propagation implementations carry extra data from the point of extraction (for example, reading incoming headers) to injection (for example, writing outgoing headers). +For example, it might carry a request ID. +When implementations have extra data, they handle it as follows: +* If a TraceContext were extracted, add the extra data as TraceContext.extra(). +* Otherwise, add it as TraceContextOrSamplingFlags.extra(), which Tracer.nextSpan handles.

Prev   Next
4. Sampling Home 6. Current Tracing Component
\ No newline at end of file diff --git a/2.0.x/multi/multi__running_examples.html b/2.0.x/multi/multi__running_examples.html index ca327869e..6da03204d 100644 --- a/2.0.x/multi/multi__running_examples.html +++ b/2.0.x/multi/multi__running_examples.html @@ -1,3 +1,7 @@ - 14. Running examples
14. Running examples
Prev   

14. Running examples

You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links:

Prev   
13. Integrations Home 
\ No newline at end of file + 16. Running examples
16. Running examples
Prev   

16. Running examples

You can see the running examples deployed in the Pivotal Web Services. +Check them out at the following links:

Prev   
15. Integrations Home 
\ No newline at end of file diff --git a/2.0.x/multi/multi__sampling.html b/2.0.x/multi/multi__sampling.html index 3d4a50e92..115979dc0 100644 --- a/2.0.x/multi/multi__sampling.html +++ b/2.0.x/multi/multi__sampling.html @@ -1,26 +1,40 @@ - 4. Sampling
4. Sampling
Prev   Next

4. Sampling

In distributed tracing the data volumes can be very high so sampling -can be important (you usually don’t need to export all spans to get a -good picture of what is happening). Spring Cloud Sleuth has a -Sampler strategy that you can implement to take control of the -sampling algorithm. Samplers do not stop span (correlation) ids from -being generated, but they do prevent the tags and events being -attached and exported. By default you get a strategy that continues to -trace if a span is already active, but new ones are always marked as -non-exportable. If all your apps run with this sampler you will see -traces in logs, but not in any remote store. For testing the default -is often enough, and it probably is all you need if you are only using -the logs (e.g. with an ELK aggregator). If you are exporting span data -to Zipkin or Spring Cloud Stream, there is also an AlwaysSampler -that exports everything and a PercentageBasedSampler that samples a -fixed fraction of spans.

[Note]Note

the PercentageBasedSampler is the default if you are using -spring-cloud-sleuth-zipkin or spring-cloud-sleuth-stream. You can -configure the exports using spring.sleuth.sampler.percentage. The passed -value needs to be a double from 0.0 to 1.0 so it’s not a percentage. -For backwards compatibility reasons we’re not changing the property name.

A sampler can be installed just by creating a bean definition, e.g:

@Bean
+   4. Sampling
4. Sampling
Prev   Next
4. Sampling
Sampling may be employed to reduce the data collected and reported out of process.
+When a span is not sampled, it adds no overhead (a noop).
Sampling is an up-front decision, meaning that the decision to report data is made at the first operation in a trace and that decision is propagated downstream.
By default, a global sampler applies a single rate to all traced operations.
+Tracer.Builder.sampler controls this setting, and it defaults to tracing every request.
4.1 Declarative sampling
Some applications need to sample based on the type or annotations of a java method.
Most users use a framework interceptor to automate this sort of policy.
+The following example shows how that might work internally:
@Autowired Tracing tracing;
+
+// derives a sample rate from an annotation on a java method
+DeclarativeSampler<Traced> sampler = DeclarativeSampler.create(Traced::sampleRate);
+
+@Around("@annotation(traced)")
+public Object traceThing(ProceedingJoinPoint pjp, Traced traced) throws Throwable {
+  Span span = tracing.tracer().newTrace(sampler.sample(traced))...
+  try {
+    return pjp.proceed();
+  } finally {
+    span.finish();
+  }
+}
4.2 Custom sampling
Depending on what the operation is, you may want to apply different policies.
+For example, you might not want to trace requests to static resources such as images, or you might want to trace all requests to a new api.
Most users use a framework interceptor to automate this sort of policy.
+The following example shows how that might work internally:
@Autowired Tracer tracer;
+
+Span newTrace(Request input) {
+  SamplingFlags flags = SamplingFlags.NONE;
+  if (input.url().startsWith("/experimental")) {
+    flags = SamplingFlags.SAMPLED;
+  } else if (input.url().startsWith("/static")) {
+    flags = SamplingFlags.NOT_SAMPLED;
+  }
+  return tracer.newTrace(flags);
+}
4.3 Sampling in Spring Cloud Sleuth
By default Spring Cloud Sleuth sets all spans to non-exportable.
+That means that traces appear in logs but not in any remote store.
+For testing the default is often enough, and it probably is all you need if you use only the logs (for example, with an ELK aggregator).
+If you export span data to Zipkin, there is also an Sampler.ALWAYS_SAMPLE setting that exports everything and a ProbabilityBasedSampler setting that samples a fixed fraction of spans.
[Note]Note
The ProbabilityBasedSampler is the default if you use spring-cloud-sleuth-zipkin.
+You can configure the exports by setting spring.sleuth.sampler.probability.
+The passed value needs to be a double from 0.0 to 1.0.
A sampler can be installed by creating a bean definition, as shown in the following example:
@Bean
 public Sampler defaultSampler() {
-	return new AlwaysSampler();
-}
[Tip]Tip
You can set the HTTP header X-B3-Flags to 1 or when doing messaging you can
-set spanFlags header to 1. Then the current span will be forced to be exportable
-regardless of the sampling decision.
Prev   Next
3. Features Home 5. Instrumentation

\ No newline at end of file
+	return Sampler.ALWAYS_SAMPLE;
+}
[Tip]Tip

You can set the HTTP header X-B3-Flags to 1, or, when doing messaging, you can set the spanFlags header to 1. +Doing so forces the current span to be exportable regardless of the sampling decision.

Prev   Next
3. Features Home 5. Propagation
\ No newline at end of file diff --git a/2.0.x/multi/multi__sending_spans_to_zipkin.html b/2.0.x/multi/multi__sending_spans_to_zipkin.html index c63c7f4f7..d302fee54 100644 --- a/2.0.x/multi/multi__sending_spans_to_zipkin.html +++ b/2.0.x/multi/multi__sending_spans_to_zipkin.html @@ -1,7 +1,28 @@ - 10. Sending spans to Zipkin
10. Sending spans to Zipkin
Prev   Next

10. Sending spans to Zipkin

By default if you add spring-cloud-starter-zipkin as a dependency to your project, -when the span is closed, it will be sent to Zipkin over HTTP. The communication -is asynchronous. You can configure the URL by setting the spring.zipkin.baseUrl -property as follows:

spring.zipkin.baseUrl: http://192.168.99.100:9411/

If you want to find Zipkin via service discovery it’s enough to pass the -Zipkin’s service id inside the URL (example for zipkinserver service id)

spring.zipkin.baseUrl: http://zipkinserver/
Prev   Next
9. Customizations Home 11. Span Data as Messages
\ No newline at end of file + 13. Sending Spans to Zipkin
13. Sending Spans to Zipkin
Prev   Next

13. Sending Spans to Zipkin

By default, if you add spring-cloud-starter-zipkin as a dependency to your project, when the span is closed, it is sent to Zipkin over HTTP. +The communication is asynchronous. +You can configure the URL by setting the spring.zipkin.baseUrl property, as follows:

spring.zipkin.baseUrl: http://192.168.99.100:9411/

If you want to find Zipkin through service discovery, you can pass the Zipkin’s service ID inside the URL, as shown in the following example for zipkinserver service ID:

spring.zipkin.baseUrl: http://zipkinserver/

To disable this feature just set spring.zipkin.discoveryClientEnabled to `false.

When the Discovery Client feature is enabled, Sleuth uses +LoadBalancerClient to find the URL of the Zipkin Server. It means +that you can set up the load balancing configuration e.g. via Ribbon.

zipkinserver:
+  ribbon:
+    ListOfServers: host1,host2

If you have web, rabbit, or kafka together on the classpath, you might need to pick the means by which you would like to send spans to zipkin. +To do so, set web, rabbit, or kafka to the spring.zipkin.sender.type property. +The following example shows setting the sender type for web:

spring.zipkin.sender.type: web

To customize the RestTemplate that sends spans to Zipkin via HTTP, you can register +the ZipkinRestTemplateCustomizer bean.

@Configuration
+class MyConfig {
+	@Bean ZipkinRestTemplateCustomizer myCustomizer() {
+		return new ZipkinRestTemplateCustomizer() {
+			@Override
+			void customize(RestTemplate restTemplate) {
+				// customize the RestTemplate
+			}
+		};
+	}
+}

If, however, you would like to control the full process of creating the RestTemplate +object, you will have to create a bean of zipkin2.reporter.Sender type.

	@Bean Sender myRestTemplateSender(ZipkinProperties zipkin,
+			ZipkinRestTemplateCustomizer zipkinRestTemplateCustomizer) {
+		RestTemplate restTemplate = mySuperCustomRestTemplate();
+		zipkinRestTemplateCustomizer.customize(restTemplate);
+		return myCustomSender(zipkin, restTemplate);
+	}
Prev   Next
12. Customizations Home 14. Zipkin Stream Span Consumer
\ No newline at end of file diff --git a/2.0.x/multi/multi__span_lifecycle.html b/2.0.x/multi/multi__span_lifecycle.html index 5d06ae5d2..f68933a72 100644 --- a/2.0.x/multi/multi__span_lifecycle.html +++ b/2.0.x/multi/multi__span_lifecycle.html @@ -1,63 +1,59 @@ - 6. Span lifecycle
6. Span lifecycle
Prev   Next

6. Span lifecycle

You can do the following operations on the Span by means of org.springframework.cloud.sleuth.Tracer interface:

  • start - when you start a span its name is assigned and start timestamp is recorded.
  • close - the span gets finished (the end time of the span is recorded) and if -the span is exportable then it will be eligible for collection to Zipkin. -The span is also removed from the current thread.
  • continue - a new instance of span will be created whereas it will be a copy of the -one that it continues.
  • detach - the span doesn’t get stopped or closed. It only gets removed from the current thread.
  • create with explicit parent - you can create a new span and set an explicit parent to it
[Tip]Tip

Spring creates the instance of Tracer for you. In order to use it all you need is to just autowire it.

6.1 Creating and closing spans

You can manually create spans by using the Tracer interface.

// Start a span. If there was a span present in this thread it will become
+   9. Span lifecycle
9. Span lifecycle
Prev   Next
9. Span lifecycle
You can do the following operations on the Span by means of brave.Tracer:
  • start: When you start a span, its name is assigned and the start timestamp is recorded.
  • close: The span gets finished (the end time of the span is recorded) and, if the span is sampled, it is eligible for collection (for example, to Zipkin).
  • continue: A new instance of span is created.
+It is a copy of the one that it continues.
  • detach: The span does not get stopped or closed.
+It only gets removed from the current thread.
  • create with explicit parent: You can create a new span and set an explicit parent for it.
[Tip]Tip
Spring Cloud Sleuth creates an instance of Tracer for you. In order to use it, you can autowire it.
9.1 Creating and finishing spans
You can manually create spans by using the Tracer, as shown in the following example:
// Start a span. If there was a span present in this thread it will become
 // the `newSpan`'s parent.
-Span newSpan = this.tracer.createSpan("calculateTax");
-try {
+Span newSpan = this.tracer.nextSpan().name("calculateTax");
+try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(newSpan.start())) {
 	// ...
 	// You can tag a span
-	this.tracer.addTag("taxValue", taxValue);
+	newSpan.tag("taxValue", taxValue);
 	// ...
 	// You can log an event on a span
-	newSpan.logEvent("taxCalculated");
+	newSpan.annotate("taxCalculated");
 } finally {
-	// Once done remember to close the span. This will allow collecting
+	// Once done remember to finish the span. This will allow collecting
 	// the span to send it to Zipkin
-	this.tracer.close(newSpan);
-}
In this example we could see how to create a new instance of span. Assuming that there already
-was a span present in this thread then it would become the parent of that span.
[Important]Important
Always clean after you create a span! Don’t forget to close a span if you want to send it to Zipkin.
[Important]Important
If your span contains a name greater than 50 chars, then that name will
-be truncated to 50 chars. Your names have to be explicit and concrete. Big names lead to
-latency issues and sometimes even thrown exceptions.
6.2 Continuing spans
Sometimes you don’t want to create a new span but you want to continue one. Example of such a
-situation might be (of course it all depends on the use-case):
  • AOP - If there was already a span created before an aspect was reached then you might not want to create a new span.
  • Hystrix - executing a Hystrix command is most likely a logical part of the current processing. It’s in fact
-only a technical implementation detail that you wouldn’t necessarily want to reflect in tracing as a separate being.
The continued instance of span is equal to the one that it continues:
Span continuedSpan = this.tracer.continueSpan(spanToContinue);
-assertThat(continuedSpan).isEqualTo(spanToContinue);
To continue a span you can use the Tracer interface.
// let's assume that we're in a thread Y and we've received
+	newSpan.finish();
+}
In the preceding example, we could see how to create a new instance of the span.
+If there is already a span in this thread, it becomes the parent of the new span.
[Important]Important
Always clean after you create a span. Also, always finish any span that you want to send to Zipkin.
[Important]Important
If your span contains a name greater than 50 chars, that name is truncated to 50 chars.
+Your names have to be explicit and concrete. Big names lead to latency issues and sometimes even exceptions.
9.2 Continuing Spans
Sometimes, you do not want to create a new span but you want to continue one. An example of such a
+situation might be as follows:
  • AOP: If there was already a span created before an aspect was reached, you might not want to create a new span.
  • Hystrix: Executing a Hystrix command is most likely a logical part of the current processing.
+It is in fact merely a technical implementation detail that you would not necessarily want to reflect in tracing as a separate being.
To continue a span, you can use brave.Tracer, as shown in the following example:
// let's assume that we're in a thread Y and we've received
 // the `initialSpan` from thread X
-Span continuedSpan = this.tracer.continueSpan(initialSpan);
+Span continuedSpan = this.tracer.toSpan(newSpan.context());
 try {
 	// ...
 	// You can tag a span
-	this.tracer.addTag("taxValue", taxValue);
+	continuedSpan.tag("taxValue", taxValue);
 	// ...
 	// You can log an event on a span
-	continuedSpan.logEvent("taxCalculated");
+	continuedSpan.annotate("taxCalculated");
 } finally {
-	// Once done remember to detach the span. That way you'll
-	// safely remove it from the current thread without closing it
-	this.tracer.detach(continuedSpan);
-}
[Important]Important
Always clean after you create a span! Don’t forget to detach a span if some work was done started in one
- thread (e.g. thread X) and it’s waiting for other threads (e.g. Y, Z) to finish.
- Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected
- the span in thread X should be closed.
6.3 Creating spans with an explicit parent
There is a possibility that you want to start a new span and provide an explicit parent of that span.
-Let’s assume that the parent of a span is in one thread and you want to start a new span in another thread. The
-startSpan method of the Tracer interface is the method you are looking for.
// let's assume that we're in a thread Y and we've received
+	// Once done remember to flush the span. That means that
+	// it will get reported but the span itself is not yet finished
+	continuedSpan.flush();
+}
9.3 Creating a Span with an explicit Parent
You might want to start a new span and provide an explicit parent of that span.
+Assume that the parent of a span is in one thread and you want to start a new span in another thread.
+In Brave, whenever you call nextSpan(), it creates a span in reference to the span that is currently in scope.
+You can put the span in scope and then call nextSpan(), as shown in the following example:
// let's assume that we're in a thread Y and we've received
 // the `initialSpan` from thread X. `initialSpan` will be the parent
 // of the `newSpan`
-Span newSpan = this.tracer.createSpan("calculateCommission", initialSpan);
-try {
+Span newSpan = null;
+try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) {
+	newSpan = this.tracer.nextSpan().name("calculateCommission");
 	// ...
 	// You can tag a span
-	this.tracer.addTag("commissionValue", commissionValue);
+	newSpan.tag("commissionValue", commissionValue);
 	// ...
 	// You can log an event on a span
-	newSpan.logEvent("commissionCalculated");
+	newSpan.annotate("commissionCalculated");
 } finally {
-	// Once done remember to close the span. This will allow collecting
+	// Once done remember to finish the span. This will allow collecting
 	// the span to send it to Zipkin. The tags and events set on the
 	// newSpan will not be present on the parent
-	this.tracer.close(newSpan);
-}
[Important]Important
After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs
- related to the fact that you have a span present in the current thread other than the one you’re trying to close.
- What’s worse your spans won’t get closed properly thus will not get collected to Zipkin.
Prev   Next
5. Instrumentation Home 7. Naming spans

\ No newline at end of file
+	if (newSpan != null) {
+		newSpan.finish();
+	}
+}
[Important]Important

After creating such a span, you must finish it. Otherwise it is not reported (for example, to Zipkin).

Prev   Next
8. Instrumentation Home 10. Naming spans
\ No newline at end of file diff --git a/2.0.x/multi/multi__zipkin_stream_span_consumer.html b/2.0.x/multi/multi__zipkin_stream_span_consumer.html new file mode 100644 index 000000000..6570accf0 --- /dev/null +++ b/2.0.x/multi/multi__zipkin_stream_span_consumer.html @@ -0,0 +1,5 @@ + + + 14. Zipkin Stream Span Consumer
14. Zipkin Stream Span Consumer
Prev   Next

14. Zipkin Stream Span Consumer

[Important]Important

We recommend using Zipkin’s native support for message-based span sending. +Starting from the Edgware release, the Zipkin Stream server is deprecated. +In the Finchley release, it got removed.

If for some reason you need to create the deprecated Stream Zipkin server, see the Dalston Documentation.

Prev   Next
13. Sending Spans to Zipkin Home 15. Integrations
\ No newline at end of file diff --git a/2.0.x/multi/multi_pr01.html b/2.0.x/multi/multi_pr01.html index 36c974284..9d68e972d 100644 --- a/2.0.x/multi/multi_pr01.html +++ b/2.0.x/multi/multi_pr01.html @@ -1,3 +1,3 @@ -
Prev   Next

2.0.0.BUILD-SNAPSHOT

Prev   Next
Spring Cloud Sleuth Home 1. Introduction
\ No newline at end of file +
Prev   Next

2.0.1.BUILD-SNAPSHOT

Prev   Next
Spring Cloud Sleuth Home 1. Introduction
\ No newline at end of file diff --git a/2.0.x/multi/multi_spring-cloud-sleuth.html b/2.0.x/multi/multi_spring-cloud-sleuth.html index a735595ec..f52ccbb79 100644 --- a/2.0.x/multi/multi_spring-cloud-sleuth.html +++ b/2.0.x/multi/multi_spring-cloud-sleuth.html @@ -1,3 +1,3 @@ - Spring Cloud Sleuth
Spring Cloud Sleuth
   Next

Spring Cloud Sleuth

Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer

Table of Contents

1. Introduction
1.1. Terminology
1.2. Purpose
1.2.1. Distributed tracing with Zipkin
1.2.2. Visualizing errors
1.2.3. Live examples
1.2.4. Log correlation
JSON Logback with Logstash
1.2.5. Propagating Span Context
Baggage vs. Span Tags
1.3. Adding to the project
1.3.1. Only Sleuth (log correlation)
1.3.2. Sleuth with Zipkin via HTTP
1.3.3. Sleuth with Zipkin via RabbitMQ or Kafka
2. Additional resources
3. Features
4. Sampling
5. Instrumentation
6. Span lifecycle
6.1. Creating and closing spans
6.2. Continuing spans
6.3. Creating spans with an explicit parent
7. Naming spans
7.1. @SpanName annotation
7.2. toString() method
8. Managing spans with annotations
8.1. Rationale
8.2. Creating new spans
8.3. Continuing spans
8.4. More advanced tag setting
8.4.1. Custom extractor
8.4.2. Resolving expressions for value
8.4.3. Using toString method
9. Customizations
9.1. Spring Integration
9.2. HTTP
9.3. Example
9.4. TraceFilter
9.5. Custom SA tag in Zipkin
9.6. Custom service name
9.7. Customization of reported spans
9.8. Host locator
10. Sending spans to Zipkin
11. Span Data as Messages
11.1. Zipkin Consumer
11.2. Custom Consumer
12. Metrics
13. Integrations
13.1. Runnable and Callable
13.2. Hystrix
13.2.1. Custom Concurrency Strategy
13.2.2. Manual Command setting
13.3. RxJava
13.4. HTTP integration
13.4.1. HTTP Filter
13.4.2. HandlerInterceptor
13.4.3. Async Servlet support
13.4.4. WebFlux support
13.5. HTTP client integration
13.5.1. Synchronous Rest Template
13.5.2. Asynchronous Rest Template
Multiple Asynchronous Rest Templates
13.5.3. WebClient
13.5.4. Traverson
13.6. Feign
13.7. Asynchronous communication
13.7.1. @Async annotated methods
13.7.2. @Scheduled annotated methods
13.7.3. Executor, ExecutorService and ScheduledExecutorService
Customization of Executors
13.8. Messaging
13.9. Zuul
14. Running examples
   Next
   
\ No newline at end of file + Spring Cloud Sleuth
Spring Cloud Sleuth
   Next

Spring Cloud Sleuth

Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer, Jay Bryant

Table of Contents

1. Introduction
1.1. Terminology
1.2. Purpose
1.2.1. Distributed Tracing with Zipkin
1.2.2. Visualizing errors
1.2.3. Distributed Tracing with Brave
1.2.4. Live examples
1.2.5. Log correlation
JSON Logback with Logstash
1.2.6. Propagating Span Context
Baggage versus Span Tags
1.3. Adding Sleuth to the Project
1.3.1. Only Sleuth (log correlation)
1.3.2. Sleuth with Zipkin via HTTP
1.3.3. Sleuth with Zipkin over RabbitMQ or Kafka
2. Additional Resources
3. Features
3.1. Introduction to Brave
3.1.1. Tracing
3.1.2. Local Tracing
3.1.3. Customizing Spans
3.1.4. Implicitly Looking up the Current Span
3.1.5. RPC tracing
One-Way tracing
4. Sampling
4.1. Declarative sampling
4.2. Custom sampling
4.3. Sampling in Spring Cloud Sleuth
5. Propagation
5.1. Propagating extra fields
5.1.1. Prefixed fields
5.1.2. Extracting a Propagated Context
5.1.3. Sharing span IDs between Client and Server
5.1.4. Implementing Propagation
6. Current Tracing Component
7. Current Span
7.1. Setting a span in scope manually
8. Instrumentation
9. Span lifecycle
9.1. Creating and finishing spans
9.2. Continuing Spans
9.3. Creating a Span with an explicit Parent
10. Naming spans
10.1. @SpanName Annotation
10.2. toString() method
11. Managing Spans with Annotations
11.1. Rationale
11.2. Creating New Spans
11.3. Continuing Spans
11.4. Advanced Tag Setting
11.4.1. Custom extractor
11.4.2. Resolving Expressions for a Value
11.4.3. Using the toString() method
12. Customizations
12.1. HTTP
12.2. TracingFilter
12.3. Custom service name
12.4. Customization of Reported Spans
12.5. Host Locator
13. Sending Spans to Zipkin
14. Zipkin Stream Span Consumer
15. Integrations
15.1. OpenTracing
15.2. Runnable and Callable
15.3. Hystrix
15.3.1. Custom Concurrency Strategy
15.3.2. Manual Command setting
15.4. RxJava
15.5. HTTP integration
15.5.1. HTTP Filter
15.5.2. HandlerInterceptor
15.5.3. Async Servlet support
15.5.4. WebFlux support
15.5.5. Dubbo RPC support
15.6. HTTP Client Integration
15.6.1. Synchronous Rest Template
15.6.2. Asynchronous Rest Template
Multiple Asynchronous Rest Templates
15.6.3. WebClient
15.6.4. Traverson
15.6.5. Apache HttpClientBuilder and HttpAsyncClientBuilder
15.6.6. Netty HttpClient
15.6.7. UserInfoRestTemplateCustomizer
15.7. Feign
15.8. Asynchronous Communication
15.8.1. @Async Annotated methods
15.8.2. @Scheduled Annotated Methods
15.8.3. Executor, ExecutorService, and ScheduledExecutorService
Customization of Executors
15.9. Messaging
15.9.1. Spring Integration and Spring Cloud Stream
15.9.2. Spring RabbitMq
15.9.3. Spring Kafka
15.10. Zuul
16. Running examples
   Next
   
\ No newline at end of file diff --git a/2.0.x/single/images/callouts/1.png b/2.0.x/single/images/callouts/1.png new file mode 100644 index 0000000000000000000000000000000000000000..7d473430b7bec514f7de12f5769fe7c5859e8c5d GIT binary patch literal 329 zcmeAS@N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQC}X^4DKU-G|w_t}fLBA)Suv#nrW z!^h2QnY_`l!BOq-UXEX{m2up>JTQkX)2m zTvF+fTUlI^nXH#utd~++ke^qgmzgTe~DWM4ffP81J literal 0 HcmV?d00001 diff --git a/2.0.x/single/images/callouts/2.png b/2.0.x/single/images/callouts/2.png new file mode 100644 index 0000000000000000000000000000000000000000..5d09341b2f6d2ea2d1d5dad5d980f14b4b05dfd2 GIT binary patch literal 353 zcmeAS@N?(olHy`uVBq!ia0vp^JRr;gBp8b2n5}^nQxaY7e*=hH)_rZeB4|imU1$R#1`!P>&$poQl;nzm}mD5ZFopaX|GsS%q*{P~< z;WtmO%lhToBL0i}yfkaOt?EN=nkLNGuU`ywhI5H)L`iUdT1k0gQ7VIjhO(w-Zen_> zZ(@38a<+nro{^q~f~BRtfrY+-p+a&|W^qZSLvCepNoKNMYO!8QX+eHoiC%Jk?!;Y+ zJAlS%fsM;d&r2*R1)67JkeZlkYGj#gX_9E3W@4U_nw*@Ln38B@k(iuhnUeN2eF0kK0(Y1u|9Rc(19XFPiEBhjaDG}zd16s2gM)^$re|(qda7?? zdS-IAf{C7yo`r&?rM`iMzJZ}aa#3b+Nu@(>WpPPnvR-PjUP@^}eqM=Qa(?c_U5Yz^ z#%Y0#%S_KpEGY$=XJL?(l#*ybuErX#^g`ttQfwn - Spring Cloud Sleuth

Spring Cloud Sleuth

Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer

Table of Contents

1. Introduction
1.1. Terminology
1.2. Purpose
1.2.1. Distributed tracing with Zipkin
1.2.2. Visualizing errors
1.2.3. Live examples
1.2.4. Log correlation
JSON Logback with Logstash
1.2.5. Propagating Span Context
Baggage vs. Span Tags
1.3. Adding to the project
1.3.1. Only Sleuth (log correlation)
1.3.2. Sleuth with Zipkin via HTTP
1.3.3. Sleuth with Zipkin via RabbitMQ or Kafka
2. Additional resources
3. Features
4. Sampling
5. Instrumentation
6. Span lifecycle
6.1. Creating and closing spans
6.2. Continuing spans
6.3. Creating spans with an explicit parent
7. Naming spans
7.1. @SpanName annotation
7.2. toString() method
8. Managing spans with annotations
8.1. Rationale
8.2. Creating new spans
8.3. Continuing spans
8.4. More advanced tag setting
8.4.1. Custom extractor
8.4.2. Resolving expressions for value
8.4.3. Using toString method
9. Customizations
9.1. Spring Integration
9.2. HTTP
9.3. Example
9.4. TraceFilter
9.5. Custom SA tag in Zipkin
9.6. Custom service name
9.7. Customization of reported spans
9.8. Host locator
10. Sending spans to Zipkin
11. Span Data as Messages
11.1. Zipkin Consumer
11.2. Custom Consumer
12. Metrics
13. Integrations
13.1. Runnable and Callable
13.2. Hystrix
13.2.1. Custom Concurrency Strategy
13.2.2. Manual Command setting
13.3. RxJava
13.4. HTTP integration
13.4.1. HTTP Filter
13.4.2. HandlerInterceptor
13.4.3. Async Servlet support
13.4.4. WebFlux support
13.5. HTTP client integration
13.5.1. Synchronous Rest Template
13.5.2. Asynchronous Rest Template
Multiple Asynchronous Rest Templates
13.5.3. WebClient
13.5.4. Traverson
13.6. Feign
13.7. Asynchronous communication
13.7.1. @Async annotated methods
13.7.2. @Scheduled annotated methods
13.7.3. Executor, ExecutorService and ScheduledExecutorService
Customization of Executors
13.8. Messaging
13.9. Zuul
14. Running examples

2.0.0.BUILD-SNAPSHOT

1. Introduction

Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.

1.1 Terminology

Spring Cloud Sleuth borrows Dapper’s terminology.

Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an -RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span -is a part of. Spans also have other data, such as descriptions, timestamped events, key-value -annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address).

Spans are started and stopped, and they keep track of their timing information. Once you create a -span, you must stop it at some point in the future.

[Tip]Tip

The initial span that starts a trace is called a root span. The value of span id -of that span is equal to trace id.

Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed -big-data store, a trace might be formed by a put request.

Annotation: is used to record existence of an event in time. Some of the core annotations used to define -the start and stop of a request are:

  • cs - Client Sent - The client has made a request. This annotation depicts the start of the span.
  • sr - Server Received - The server side got the request and will start processing it. -If one subtracts the cs timestamp from this timestamp one will receive the network latency.
  • ss - Server Sent - Annotated upon completion of request processing (when the response -got sent back to the client). If one subtracts the sr timestamp from this timestamp one -will receive the time needed by the server side to process the request.
  • cr - Client Received - Signifies the end of the span. The client has successfully received the -response from the server side. If one subtracts the cs timestamp from this timestamp one -will receive the whole time needed by the client to receive the response from the server.

Visualization of what Span and Trace will look in a system together with the Zipkin annotations:

Trace Info propagation

Each color of a note signifies a span (7 spans - from A to G). If you have such information in the note:

Trace Id = X
+   Spring Cloud Sleuth
Spring Cloud Sleuth
Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer, Jay Bryant
Table of Contents
1. Introduction
1.1. Terminology
1.2. Purpose
1.2.1. Distributed Tracing with Zipkin
1.2.2. Visualizing errors
1.2.3. Distributed Tracing with Brave
1.2.4. Live examples
1.2.5. Log correlation
JSON Logback with Logstash
1.2.6. Propagating Span Context
Baggage versus Span Tags
1.3. Adding Sleuth to the Project
1.3.1. Only Sleuth (log correlation)
1.3.2. Sleuth with Zipkin via HTTP
1.3.3. Sleuth with Zipkin over RabbitMQ or Kafka
2. Additional Resources
3. Features
3.1. Introduction to Brave
3.1.1. Tracing
3.1.2. Local Tracing
3.1.3. Customizing Spans
3.1.4. Implicitly Looking up the Current Span
3.1.5. RPC tracing
One-Way tracing
4. Sampling
4.1. Declarative sampling
4.2. Custom sampling
4.3. Sampling in Spring Cloud Sleuth
5. Propagation
5.1. Propagating extra fields
5.1.1. Prefixed fields
5.1.2. Extracting a Propagated Context
5.1.3. Sharing span IDs between Client and Server
5.1.4. Implementing Propagation
6. Current Tracing Component
7. Current Span
7.1. Setting a span in scope manually
8. Instrumentation
9. Span lifecycle
9.1. Creating and finishing spans
9.2. Continuing Spans
9.3. Creating a Span with an explicit Parent
10. Naming spans
10.1. @SpanName Annotation
10.2. toString() method
11. Managing Spans with Annotations
11.1. Rationale
11.2. Creating New Spans
11.3. Continuing Spans
11.4. Advanced Tag Setting
11.4.1. Custom extractor
11.4.2. Resolving Expressions for a Value
11.4.3. Using the toString() method
12. Customizations
12.1. HTTP
12.2. TracingFilter
12.3. Custom service name
12.4. Customization of Reported Spans
12.5. Host Locator
13. Sending Spans to Zipkin
14. Zipkin Stream Span Consumer
15. Integrations
15.1. OpenTracing
15.2. Runnable and Callable
15.3. Hystrix
15.3.1. Custom Concurrency Strategy
15.3.2. Manual Command setting
15.4. RxJava
15.5. HTTP integration
15.5.1. HTTP Filter
15.5.2. HandlerInterceptor
15.5.3. Async Servlet support
15.5.4. WebFlux support
15.5.5. Dubbo RPC support
15.6. HTTP Client Integration
15.6.1. Synchronous Rest Template
15.6.2. Asynchronous Rest Template
Multiple Asynchronous Rest Templates
15.6.3. WebClient
15.6.4. Traverson
15.6.5. Apache HttpClientBuilder and HttpAsyncClientBuilder
15.6.6. Netty HttpClient
15.6.7. UserInfoRestTemplateCustomizer
15.7. Feign
15.8. Asynchronous Communication
15.8.1. @Async Annotated methods
15.8.2. @Scheduled Annotated Methods
15.8.3. Executor, ExecutorService, and ScheduledExecutorService
Customization of Executors
15.9. Messaging
15.9.1. Spring Integration and Spring Cloud Stream
15.9.2. Spring RabbitMq
15.9.3. Spring Kafka
15.10. Zuul
16. Running examples
2.0.1.BUILD-SNAPSHOT
1. Introduction
Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud.
1.1 Terminology
Spring Cloud Sleuth borrows Dapper’s terminology.
Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an RPC.
+Spans are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of.
+Spans also have other data, such as descriptions, timestamped events, key-value annotations (tags), the ID of the span that caused them, and process IDs (normally IP addresses).
Spans can be started and stopped, and they keep track of their timing information.
+Once you create a span, you must stop it at some point in the future.
[Tip]Tip
The initial span that starts a trace is called a root span. The value of the ID
+of that span is equal to the trace ID.
Trace: A set of spans forming a tree-like structure.
+For example, if you run a distributed big-data store, a trace might be formed by a PUT request.
Annotation:  Used to record the existence of an event in time. With
+Brave instrumentation, we no longer need to set special events
+for Zipkin to understand who the client and server are, where
+the request started, and where it ended. For learning purposes,
+however, we mark these events to highlight what kind
+of an action took place.
  • cs: Client Sent. The client has made a request. This annotation indicates the start of the span.
  • sr: Server Received: The server side got the request and started processing it.
+Subtracting the cs timestamp from this timestamp reveals the network latency.
  • ss: Server Sent. Annotated upon completion of request processing (when the response got sent back to the client).
+Subtracting the sr timestamp from this timestamp reveals the time needed by the server side to process the request.
  • cr: Client Received. Signifies the end of the span.
+The client has successfully received the response from the server side.
+Subtracting the cs timestamp from this timestamp reveals the whole time needed by the client to receive the response from the server.
The following image shows how Span and Trace look in a system, together with the Zipkin annotations:
Trace Info propagation
Each color of a note signifies a span (there are seven spans - from A to G).
+Consider the following note:
Trace Id = X
 Span Id = D
-Client Sent
That means that the current span has Trace-Id set to X, Span-Id set to D. It also has emitted
- Client Sent event.
This is how the visualization of the parent / child relationship of spans would look like:
Parent child relationship
1.2 Purpose
In the following sections the example from the image above will be taken into consideration.
1.2.1 Distributed tracing with Zipkin
Altogether there are 7 spans . If you go to traces in Zipkin you will see this number in the second trace:
Traces
However if you pick a particular trace then you will see 4 spans:
Traces Info propagation
[Note]Note
When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to
-Zipkin with Server Received and Server Sent / Client Received and Client Sent
-annotations then they will presented as a single span.
Why is there a difference between the 7 and 4 spans in this case?
  • 2 spans come from http:/start span. It has the Server Received (SR) and Server Sent (SS) annotations.
  • 2 spans come from the RPC call from service1 to service2 to the http:/foo endpoint. It has the Client Sent (CS)
-and Client Received (CR) annotations on service1 side. It also has Server Received (SR) and Server Sent (SS) annotations
-on the service2 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
  • 2 spans come from the RPC call from service2 to service3 to the http:/bar endpoint. It has the Client Sent (CS)
-and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations
-on the service3 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
  • 2 spans come from the RPC call from service2 to service4 to the http:/baz endpoint. It has the Client Sent (CS)
-and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations
-on the service4 side. Physically there are 2 spans but they form 1 logical span related to an RPC call.
So if we count the physical spans we have 1 from http:/start, 2 from service1 calling service2, 2 form service2
-calling service3 and 2 from service2 calling service4. Altogether 7 spans.
Logically we see the information of Total Spans: 4 because we have 1 span related to the incoming request
-to service1 and 3 spans related to RPC calls.
1.2.2 Visualizing errors
Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn’t caught then we’re
-setting proper tags on the span which Zipkin can properly colorize. You could see in the list of traces one
-  trace that was in red color. That’s because there was an exception thrown.
If you click that trace then you’ll see a similar picture
Error Traces
Then if you click on one of the spans you’ll see the following
Error Traces Info propagation
As you can see you can easily see the reason for an error and the whole stacktrace related to it.
1.2.3 Live examples
Figure 1.1. Click Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

The dependency graph in Zipkin would look like this:
Dependencies
Figure 1.2. Click Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

1.2.4 Log correlation
When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following:
service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
+Client Sent
This note indicates that the current span has Trace Id set to X and Span Id set to D.
+Also, the Client Sent event took place.
The following image shows how parent-child relationships of spans look:
Parent child relationship
1.2 Purpose
The following sections refer to the example shown in the preceding image.
1.2.1 Distributed Tracing with Zipkin
This example has seven spans.
+If you go to traces in Zipkin, you can see this number in the second trace, as shown in the following image:
Traces
However, if you pick a particular trace, you can see four spans, as shown in the following image:
Traces Info propagation
[Note]Note
When you pick a particular trace, you see merged spans.
+That means that, if there were two spans sent to Zipkin with Server Received and Server Sent or Client Received and Client Sent annotations, they are presented as a single span.
Why is there a difference between the seven and four spans in this case?
  • Two spans come from the http:/start span. It has the Server Received (sr) and Server Sent (ss) annotations.
  • Two spans come from the RPC call from service1 to service2 to the http:/foo endpoint.
+The Client Sent (cs) and Client Received (cr) events took place on the service1 side.
+Server Received (sr) and Server Sent (ss) events took place on the service2 side.
+These two spans form one logical span related to an RPC call.
  • Two spans come from the RPC call from service2 to service3 to the http:/bar endpoint.
+The Client Sent (cs) and Client Received (cr) events took place on the service2 side.
+The Server Received (sr) and Server Sent (ss) events took place on the service3 side.
+These two spans form one logical span related to an RPC call.
  • Two spans come from the RPC call from service2 to service4 to the http:/baz endpoint.
+The Client Sent (cs) and Client Received (cr) events took place on the service2 side.
+Server Received (sr) and Server Sent (ss) events took place on the service4 side.
+These two spans form one logical span related to an RPC call.
So, if we count the physical spans, we have one from http:/start, two from service1 calling service2, two from service2
+calling service3, and two from service2 calling service4. In sum, we have a total of seven spans.
Logically, we see the information of four total Spans because we have one span related to the incoming request
+to service1 and three spans related to RPC calls.
1.2.2 Visualizing errors
Zipkin lets you visualize errors in your trace.
+When an exception was thrown and was not caught, we set proper tags on the span, which Zipkin can then properly colorize.
+You could see in the list of traces one trace that is red. That appears because an exception was thrown.
If you click that trace, you see a similar picture, as follows:
Error Traces
If you then click on one of the spans, you see the following
Error Traces Info propagation
The span shows the reason for the error and the whole stack trace related to it.
1.2.3 Distributed Tracing with Brave
Starting with version 2.0.0, Spring Cloud Sleuth uses Brave as the tracing library.
+Consequently, Sleuth no longer takes care of storing the context but delegates that work to Brave.
Due to the fact that Sleuth had different naming and tagging conventions than Brave, we decided to follow Brave’s conventions from now on.
+However, if you want to use the legacy Sleuth approaches, you can set the spring.sleuth.http.legacy.enabled property to true.
1.2.4 Live examples
Figure 1.1. Click the Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

Click here to see it live!
The dependency graph in Zipkin should resemble the following image:
Dependencies
Figure 1.2. Click the Pivotal Web Services icon to see it live!
Zipkin deployed on Pivotal Web Services

Click here to see it live!
1.2.5 Log correlation
When using grep to read the logs of those four applications by scanning for a trace ID equal to (for example) 2485ec27856c56f4, you get output resembling the following:
service1.log:2016-02-26 11:15:47.561  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Hello from service1. Calling service2
 service2.log:2016-02-26 11:15:47.710  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Hello from service2. Calling service3 and then service4
 service3.log:2016-02-26 11:15:47.895  INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application   : Hello from service3
 service2.log:2016-02-26 11:15:47.924  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service3 [Hello from service3]
 service4.log:2016-02-26 11:15:48.134  INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application   : Hello from service4
 service2.log:2016-02-26 11:15:48.156  INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application   : Got response from service4 [Hello from service4]
-service1.log:2016-02-26 11:15:48.182  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]
If you’re using a log aggregating tool like Kibana,
-Splunk etc. you can order the events that took place. An example of
-Kibana would look like this:
Log correlation with Kibana
If you want to use Logstash here is the Grok pattern for Logstash:
filter {
+service1.log:2016-02-26 11:15:48.182  INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application   : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]
If you use a log aggregating tool (such as Kibana, Splunk, and others), you can order the events that took place.
+An example from Kibana would resemble the following image:
Log correlation with Kibana
If you want to use Logstash, the following listing shows the Grok pattern for Logstash:
filter {
        # pattern matching logback pattern
        grok {
               match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
        }
-}
[Note]Note
If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern:
filter {
+}
[Note]Note
If you want to use Grok together with the logs from Cloud Foundry, you have to use the following pattern:
filter {
        # pattern matching logback pattern
        grok {
               match => { "message" => "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
        }
-}
JSON Logback with Logstash
Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability
-we’re passing the dependencies in the groupId:artifactId:version notation.
Dependencies setup
  • Ensure that Logback is on the classpath (ch.qos.logback:logback-core)
  • Add Logstash Logback encode - example for version 4.6 : net.logstash.logback:logstash-logback-encoder:4.6
Logback setup
Below you can find an example of a Logback configuration (file named logback-spring.xml) that:
  • logs information from the application in a JSON format to a build/${spring.application.name}.json file
  • has commented out two additional appenders - console and standard log file
  • has the same logging pattern as the one presented in the previous section
<?xml version="1.0" encoding="UTF-8"?>
+}
JSON Logback with Logstash
Often, you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick.
+To do so, you have to do the following (for readability, we pass the dependencies in the groupId:artifactId:version notation).
Dependencies Setup
  1. Ensure that Logback is on the classpath (ch.qos.logback:logback-core).
  2. Add Logstash Logback encode. For example, to use version 4.6, add net.logstash.logback:logstash-logback-encoder:4.6.
Logback Setup
Consider the following example of a Logback configuration file (named logback-spring.xml).
<?xml version="1.0" encoding="UTF-8"?>
 <configuration>
 	<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
 	​
@@ -121,44 +128,51 @@ we’re passing the dependencies in the groupId:artifa
 		<!--<appender-ref ref="logstash"/>-->
 		<!--<appender-ref ref="flatfile"/>-->
 	</root>
-</configuration>
[Note]Note
If you’re using a custom logback-spring.xml then you have to pass the spring.application.name in
-bootstrap instead of application property file. Otherwise your custom logback file won’t read the property properly.
1.2.5 Propagating Span Context
The span context is the state that must get propagated to any child Spans across process boundaries.
+</configuration>

That Logback configuration file:

  • Logs information from the application in a JSON format to a build/${spring.application.name}.json file.
  • Has commented out two additional appenders: console and standard log file.
  • Has the same logging pattern as the one presented in the previous section.
[Note]Note

If you use a custom logback-spring.xml, you must pass the spring.application.name in the bootstrap rather than the application property file. +Otherwise, your custom logback file does not properly read the property.

1.2.6 Propagating Span Context

The span context is the state that must get propagated to any child spans across process boundaries. Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context. -Baggage is an optional part.

Baggage is a set of key:value pairs stored in the span context. Baggage travels together with the trace -and is attached to every span. Spring Cloud Sleuth will understand that a header is baggage related if the HTTP - header is prefixed with baggage- and for messaging it starts with baggage_.

[Important]Important

There’s currently no limitation of the count or size of baggage items. However, keep in mind that -too many can decrease system throughput or increase RPC latency. In extreme cases, it could crash the app due -to exceeding transport-level message or header capacity.

Example of setting baggage on a span:

Span initialSpan = this.tracer.createSpan("span");
-initialSpan.setBaggageItem("foo", "bar");
-initialSpan.setBaggageItem("UPPER_CASE", "someValue");

Baggage vs. Span Tags

Baggage travels with the trace (i.e. every child span contains the baggage of its parent). Zipkin has no knowledge of -baggage and will not even receive that information.

Tags are attached to a specific span - they are presented for that particular span only. However you -can search by tag to find the trace, where there exists a span having the searched tag value.

If you want to be able to lookup a span based on baggage, you should add corresponding entry as a tag in the root span.

@Autowired Tracer tracer;
-
-Span span = tracer.getCurrentSpan();
-String baggageKey = "key";
-String baggageValue = "foo";
-span.setBaggageItem(baggageKey, baggageValue);
-tracer.addTag(baggageKey, baggageValue);

1.3 Adding to the project

[Important]Important

To ensure that your application name is properly displayed in Zipkin - set the spring.application.name property in bootstrap.yml.

1.3.1 Only Sleuth (log correlation)

If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add -the spring-cloud-starter-sleuth module to your project.

Maven.  +Baggage is an optional part.

Baggage is a set of key:value pairs stored in the span context. +Baggage travels together with the trace and is attached to every span. +Spring Cloud Sleuth understands that a header is baggage-related if the HTTP header is prefixed with baggage- and, for messaging, it starts with baggage_.

[Important]Important

There is currently no limitation of the count or size of baggage items. +However, keep in mind that too many can decrease system throughput or increase RPC latency. +In extreme cases, too much baggage can crash the application, due to exceeding transport-level message or header capacity.

The following example shows setting baggage on a span:

Span initialSpan = this.tracer.nextSpan().name("span").start();
+try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) {
+	ExtraFieldPropagation.set("foo", "bar");
+	ExtraFieldPropagation.set("UPPER_CASE", "someValue");
+}

Baggage versus Span Tags

Baggage travels with the trace (every child span contains the baggage of its parent). +Zipkin has no knowledge of baggage and does not receive that information.

[Important]Important

Starting from Sleuth 2.0.0 you have to pass the baggage key names explicitly +in your project configuration. Read more about that setup here

Tags are attached to a specific span. In other words, they are presented only for that particular span. +However, you can search by tag to find the trace, assuming a span having the searched tag value exists.

If you want to be able to lookup a span based on baggage, you should add a corresponding entry as a tag in the root span.

[Important]Important

The span must be in scope.

The following listing shows integration tests that use baggage:

The setup.  +

spring.sleuth:
+  baggage-keys:
+    - baz
+    - bizarrecase
+  propagation-keys:
+    - foo
+    - upper_case

+

The code.  +

initialSpan.tag("foo",
+		ExtraFieldPropagation.get(initialSpan.context(), "foo"));
+initialSpan.tag("UPPER_CASE",
+		ExtraFieldPropagation.get(initialSpan.context(), "UPPER_CASE"));

+

1.3 Adding Sleuth to the Project

This section addresses how to add Sleuth to your project with either Maven or Gradle.

[Important]Important

To ensure that your application name is properly displayed in Zipkin, set the spring.application.name property in bootstrap.yml.

1.3.1 Only Sleuth (log correlation)

If you want to use only Spring Cloud Sleuth without the Zipkin integration, add the spring-cloud-starter-sleuth module to your project.

The following example shows how to add Sleuth with Maven:

Maven.  

<dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
+      <dependencies>
+          <dependency>
+              <groupId>org.springframework.cloud</groupId>
+              <artifactId>spring-cloud-dependencies</artifactId>
+              <version>${release.train.version}</version>
+              <type>pom</type>
+              <scope>import</scope>
+          </dependency>
+      </dependencies>
+</dependencyManagement>
 
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-sleuth</artifactId>
-   </dependency>

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-sleuth

Gradle.  +<dependency> 2 + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-sleuth</artifactId> +</dependency>

+

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-sleuth.

The following example shows how to add Sleuth with Gradle:

Gradle.  

dependencyManagement { 1
     imports {
         mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
@@ -168,26 +182,24 @@ the Spring BOM
2
     compile "org.springframework.cloud:spring-cloud-starter-sleuth"
 }

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-sleuth

1.3.2 Sleuth with Zipkin via HTTP

If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency.

Maven.  +

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-sleuth.

1.3.2 Sleuth with Zipkin via HTTP

If you want both Sleuth and Zipkin, add the spring-cloud-starter-zipkin dependency.

The following example shows how to do so for Maven:

Maven.  

<dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
+      <dependencies>
+          <dependency>
+              <groupId>org.springframework.cloud</groupId>
+              <artifactId>spring-cloud-dependencies</artifactId>
+              <version>${release.train.version}</version>
+              <type>pom</type>
+              <scope>import</scope>
+          </dependency>
+      </dependencies>
+</dependencyManagement>
 
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-zipkin</artifactId>
-   </dependency>

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin

Gradle.  +<dependency> 2 + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-zipkin</artifactId> +</dependency>

+

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-zipkin.

The following example shows how to do so for Gradle:

Gradle.  

dependencyManagement { 1
     imports {
         mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
@@ -197,32 +209,30 @@ the Spring BOM
2
     compile "org.springframework.cloud:spring-cloud-starter-zipkin"
 }

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin

1.3.3 Sleuth with Zipkin via RabbitMQ or Kafka

If you want to use RabbitMQ or Kafka instead of http, add the spring-rabbit or spring-kafka -dependencies. The default destination name is zipkin.

Note: spring-cloud-sleuth-stream is deprecated and incompatible with these destinations

If you want Sleuth over RabbitMQ add the spring-cloud-starter-zipkin and spring-rabbit -dependencies.

Maven.  +

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-zipkin.

1.3.3 Sleuth with Zipkin over RabbitMQ or Kafka

If you want to use RabbitMQ or Kafka instead of HTTP, add the spring-rabbit or spring-kafka dependency. +The default destination name is zipkin.

If using Kafka, you must set the property spring.zipkin.sender.type property accordingly:

spring.zipkin.sender.type: kafka
[Caution]Caution

spring-cloud-sleuth-stream is deprecated and incompatible with these destinations.

If you want Sleuth over RabbitMQ, add the spring-cloud-starter-zipkin and spring-rabbit +dependencies.

The following example shows how to do so for Gradle:

Maven.  

<dependencyManagement> 1
-         <dependencies>
-             <dependency>
-                 <groupId>org.springframework.cloud</groupId>
-                 <artifactId>spring-cloud-dependencies</artifactId>
-                 <version>${release.train.version}</version>
-                 <type>pom</type>
-                 <scope>import</scope>
-             </dependency>
-         </dependencies>
-   </dependencyManagement>
+      <dependencies>
+          <dependency>
+              <groupId>org.springframework.cloud</groupId>
+              <artifactId>spring-cloud-dependencies</artifactId>
+              <version>${release.train.version}</version>
+              <type>pom</type>
+              <scope>import</scope>
+          </dependency>
+      </dependencies>
+</dependencyManagement>
 
-   <dependency> 2
-       <groupId>org.springframework.cloud</groupId>
-       <artifactId>spring-cloud-starter-zipkin</artifactId>
-   </dependency>
-   <dependency> 3
-       <groupId>org.springframework.amqp</groupId>
-       <artifactId>spring-rabbit</artifactId>
-   </dependency>

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin - that way all dependent dependencies will be downloaded

3

To automatically configure rabbit, simply add the spring-rabbit dependency

Gradle.  +<dependency> 2 + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-zipkin</artifactId> +</dependency> +<dependency> 3 + <groupId>org.springframework.amqp</groupId> + <artifactId>spring-rabbit</artifactId> +</dependency>

+

1

We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.

2

Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded.

3

To automatically configure RabbitMQ, add the spring-rabbit dependency.

Gradle.  

dependencyManagement { 1
     imports {
         mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
@@ -233,136 +243,350 @@ dependencies {
     compile "org.springframework.cloud:spring-cloud-starter-zipkin" 2
     compile "org.springframework.amqp:spring-rabbit" 3
 }

-

1

In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM

2

Add the dependency to spring-cloud-starter-zipkin - that way all dependent dependencies will be downloaded

3

To automatically configure rabbit, simply add the spring-rabbit dependency

2. Additional resources

Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin

click here to see the video

3. Features

  • Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Example logs:

    2016-02-02 15:30:57.902  INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
+
    1 
    We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.
    2 
    Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded.
    3 
    To automatically configure RabbitMQ, add the spring-rabbit dependency.

2. Additional Resources

You can watch a video of Reshmi Krishna and Marcin Grzejszczak talking about Spring Cloud +Sleuth and Zipkin by clicking here.

You can check different setups of Sleuth and Brave in the openzipkin/sleuth-webmvc-example repository.

3. Features

  • Adds trace and span IDs to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator, as shown in the following example logs:

    2016-02-02 15:30:57.902  INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
 2016-02-02 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ...
-2016-02-02 15:31:01.936  INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...

    notice the [appname,traceId,spanId,exportable] entries from the MDC:

    • spanId - the id of a specific operation that took place
    • appname - the name of the application that logged the span
    • traceId - the id of the latency graph that contains the span
    • exportable - whether the log should be exported to Zipkin or not. When would you like the span not to be -exportable? In the case in which you want to wrap some operation in a Span and have it written to the logs -only.
  • Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, -key-value annotations. Loosely based on HTrace, but Zipkin (Dapper) compatible.

  • Sleuth records timing information to aid in latency analysis. Using sleuth, you can pinpoint causes of -latency in your applications. Sleuth is written to not log too much, and to not cause your production application to crash.

    • propagates structural data about your call-graph in-band, and the rest out-of-band.
    • includes opinionated instrumentation of layers such as HTTP
    • includes sampling policy to manage volume
    • can report to a Zipkin system for query and visualization
  • Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, -rest template, scheduled actions, message channels, zuul filters, feign client).
  • Sleuth includes default logic to join a trace across http or messaging boundaries. For example, http propagation -works via Zipkin-compatible request headers. This propagation logic is defined and customized via -SpanInjector and SpanExtractor implementations.
  • Sleuth gives you the possibility to propagate context (also known as baggage) between processes. That means that if you set on a Span -a baggage element then it will be sent downstream either via HTTP or messaging to other processes.
  • Provides a way to create / continue spans and add tags and logs via annotations.
  • Provides simple metrics of accepted / dropped spans.

  • If spring-cloud-sleuth-zipkin then the app will generate and collect Zipkin-compatible traces. -By default it sends them via HTTP to a Zipkin server on localhost (port 9411). -Configure the location of the service using spring.zipkin.baseUrl.

    • If you depend on spring-rabbit or spring-kafka your app will send traces to a broker instead of http.
    • Note: spring-cloud-sleuth-stream is deprecated and should no longer be used.
[Important]Important

If using Zipkin, configure the percentage of spans exported using spring.sleuth.sampler.percentage -(default 0.1, i.e. 10%). Otherwise you might think that Sleuth is not working cause it’s omitting some spans.

[Note]Note

the SLF4J MDC is always set and logback users will immediately see the trace and span ids in logs per the example - above. Other logging systems have to configure their own formatter to get the same result. The default is - logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}] - (this is a Spring Boot feature for logback users). - This means that if you’re not using SLF4J this pattern WILL NOT be automatically applied.

4. Sampling

In distributed tracing the data volumes can be very high so sampling -can be important (you usually don’t need to export all spans to get a -good picture of what is happening). Spring Cloud Sleuth has a -Sampler strategy that you can implement to take control of the -sampling algorithm. Samplers do not stop span (correlation) ids from -being generated, but they do prevent the tags and events being -attached and exported. By default you get a strategy that continues to -trace if a span is already active, but new ones are always marked as -non-exportable. If all your apps run with this sampler you will see -traces in logs, but not in any remote store. For testing the default -is often enough, and it probably is all you need if you are only using -the logs (e.g. with an ELK aggregator). If you are exporting span data -to Zipkin or Spring Cloud Stream, there is also an AlwaysSampler -that exports everything and a PercentageBasedSampler that samples a -fixed fraction of spans.

[Note]Note

the PercentageBasedSampler is the default if you are using -spring-cloud-sleuth-zipkin or spring-cloud-sleuth-stream. You can -configure the exports using spring.sleuth.sampler.percentage. The passed -value needs to be a double from 0.0 to 1.0 so it’s not a percentage. -For backwards compatibility reasons we’re not changing the property name.

A sampler can be installed just by creating a bean definition, e.g:

@Bean
+2016-02-02 15:31:01.936  INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...

Notice the [appname,traceId,spanId,exportable] entries from the MDC:

  • spanId: The ID of a specific operation that took place.
  • appname: The name of the application that logged the span.
  • traceId: The ID of the latency graph that contains the span.
  • exportable: Whether the log should be exported to Zipkin. +When would you like the span not to be exportable? +When you want to wrap some operation in a Span and have it written to the logs only.
  • Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, and key-value annotations. +Spring Cloud Sleuth is loosely based on HTrace but is compatible with Zipkin (Dapper).
  • Sleuth records timing information to aid in latency analysis. +By using sleuth, you can pinpoint causes of latency in your applications.

  • Sleuth is written to not log too much and to not cause your production application to crash. +To that end, Sleuth:

    • Propagates structural data about your call graph in-band and the rest out-of-band.
    • Includes opinionated instrumentation of layers such as HTTP.
    • Includes a sampling policy to manage volume.
    • Can report to a Zipkin system for query and visualization.
  • Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, rest template, scheduled actions, message channels, Zuul filters, and Feign client).
  • Sleuth includes default logic to join a trace across HTTP or messaging boundaries. +For example, HTTP propagation works over Zipkin-compatible request headers.
  • Sleuth can propagate context (also known as baggage) between processes. +Consequently, if you set a baggage element on a Span, it is sent downstream to other processes over either HTTP or messaging.
  • Provides a way to create or continue spans and add tags and logs through annotations.

  • If spring-cloud-sleuth-zipkin is on the classpath, the app generates and collects Zipkin-compatible traces. +By default, it sends them over HTTP to a Zipkin server on localhost (port 9411). +You can configure the location of the service by setting spring.zipkin.baseUrl.

    • If you depend on spring-rabbit, your app sends traces to a RabbitMQ broker instead of HTTP.
    • If you depend on spring-kafka, and set spring.zipkin.sender.type: kafka, your app sends traces to a Kafka broker instead of HTTP.
    • [Caution]Caution

    spring-cloud-sleuth-stream is deprecated and should no longer be used.

    [Important]Important

    If you use Zipkin, configure the probability of spans exported by setting spring.sleuth.sampler.probability +(default: 0.1, which is 10 percent). Otherwise, you might think that Sleuth is not working be cause it omits some spans.

    [Note]Note

    The SLF4J MDC is always set and logback users immediately see the trace and span IDs in logs per the example +shown earlier. +Other logging systems have to configure their own formatter to get the same result. +The default is as follows: +logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}] +(this is a Spring Boot feature for logback users). +If you do not use SLF4J, this pattern is NOT automatically applied.

    3.1 Introduction to Brave

    [Important]Important

    Starting with version 2.0.0, Spring Cloud Sleuth uses +Brave as the tracing library. +For your convenience, we embed part of the Brave’s docs here.

    [Important]Important

    In the vast majority of cases you need to just use the Tracer +or SpanCustomizer beans from Brave that Sleuth provides. The documentation below contains +a high overview of what Brave is and how it works.

    Brave is a library used to capture and report latency information about distributed operations to Zipkin. +Most users do not use Brave directly. They use libraries or frameworks rather than employ Brave on their behalf.

    This module includes a tracer that creates and joins spans that model the latency of potentially distributed work. +It also includes libraries to propagate the trace context over network boundaries (for example, with HTTP headers).

    3.1.1 Tracing

    Most importantly, you need a brave.Tracer, configured to report to Zipkin.

    The following example setup sends trace data (spans) to Zipkin over HTTP (as opposed to Kafka):

    class MyClass {
+
+    private final Tracer tracer;
+
+    // Tracer will be autowired
+    MyClass(Tracer tracer) {
+        this.tracer = tracer;
+    }
+
+    void doSth() {
+        Span span = tracer.newTrace().name("encode").start();
+        // ...
+    }
+}
    [Important]Important

    If your span contains a name longer than 50 chars, then that name is truncated to 50 chars. +Your names have to be explicit and concrete. +Big names lead to latency issues and sometimes even thrown exceptions.

    The tracer creates and joins spans that model the latency of potentially distributed work. +It can employ sampling to reduce overhead during the process, to reduce the amount of data sent to Zipkin, or both.

    Spans returned by a tracer report data to Zipkin when finished or do nothing if unsampled. +After starting a span, you can annotate events of interest or add tags containing details or lookup keys.

    Spans have a context that includes trace identifiers that place the span at the correct spot in the tree representing the distributed operation.

    3.1.2 Local Tracing

    When tracing local code, you can run it inside a span, as shown in the following example:

    @Autowired Tracer tracer;
+
+Span span = tracer.newTrace().name("encode").start();
+try {
+  doSomethingExpensive();
+} finally {
+  span.finish();
+}

    In the preceding example, the span is the root of the trace. +In many cases, the span is part of an existing trace. +When this is the case, call newChild instead of newTrace, as shown in the following example:

    @Autowired Tracer tracer;
+
+Span span = tracer.newChild(root.context()).name("encode").start();
+try {
+  doSomethingExpensive();
+} finally {
+  span.finish();
+}

    3.1.3 Customizing Spans

    Once you have a span, you can add tags to it. +The tags can be used as lookup keys or details. +For example, you might add a tag with your runtime version, as shown in the following example:

    span.tag("clnt/finagle.version", "6.36.0");

    When exposing the ability to customize spans to third parties, prefer brave.SpanCustomizer as opposed to brave.Span. +The former is simpler to understand and test and does not tempt users with span lifecycle hooks.

    interface MyTraceCallback {
+  void request(Request request, SpanCustomizer customizer);
+}

    Since brave.Span implements brave.SpanCustomizer, you can pass it to users, as shown in the following example:

    for (MyTraceCallback callback : userCallbacks) {
+  callback.request(request, span);
+}

    3.1.4 Implicitly Looking up the Current Span

    Sometimes, you do not know if a trace is in progress or not, and you do not want users to do null checks. +brave.CurrentSpanCustomizer handles this problem by adding data to any span that’s in progress or drops, as shown in the following example:

    Ex.

    // The user code can then inject this without a chance of it being null.
+@Autowired SpanCustomizer span;
+
+void userCode() {
+  span.annotate("tx.started");
+  ...
+}

    3.1.5 RPC tracing

    [Tip]Tip

    Check for instrumentation written here and Zipkin’s list before rolling your own RPC instrumentation.

    RPC tracing is often done automatically by interceptors. Behind the scenes, they add tags and events that relate to their role in an RPC operation.

    The following example shows how to add a client span:

    @Autowired Tracer tracer;
+
+// before you send a request, add metadata that describes the operation
+span = tracer.newTrace().name("get").type(CLIENT);
+span.tag("clnt/finagle.version", "6.36.0");
+span.tag(TraceKeys.HTTP_PATH, "/api");
+span.remoteEndpoint(Endpoint.builder()
+    .serviceName("backend")
+    .ipv4(127 << 24 | 1)
+    .port(8080).build());
+
+// when the request is scheduled, start the span
+span.start();
+
+// if you have callbacks for when data is on the wire, note those events
+span.annotate(Constants.WIRE_SEND);
+span.annotate(Constants.WIRE_RECV);
+
+// when the response is complete, finish the span
+span.finish();

    One-Way tracing

    Sometimes, you need to model an asynchronous operation where there is a +request but no response. In normal RPC tracing, you use span.finish() +to indicate that the response was received. In one-way tracing, you use +span.flush() instead, as you do not expect a response.

    The following example shows how a client might model a one-way operation:

    @Autowired Tracer tracer;
+
+// start a new span representing a client request
+oneWaySend = tracer.newSpan(parent).kind(Span.Kind.CLIENT);
+
+// Add the trace context to the request, so it can be propagated in-band
+tracing.propagation().injector(Request::addHeader)
+                     .inject(oneWaySend.context(), request);
+
+// fire off the request asynchronously, totally dropping any response
+request.execute();
+
+// start the client side and flush instead of finish
+oneWaySend.start().flush();

    The following example shows how a server might handle a one-way operation:

    @Autowired Tracing tracing;
+@Autowired Tracer tracer;
+
+// pull the context out of the incoming request
+extractor = tracing.propagation().extractor(Request::getHeader);
+
+// convert that context to a span which you can name and add tags to
+oneWayReceive = nextSpan(tracer, extractor.extract(request))
+    .name("process-request")
+    .kind(SERVER)
+    ... add tags etc.
+
+// start the server side and flush instead of finish
+oneWayReceive.start().flush();
+
+// you should not modify this span anymore as it is complete. However,
+// you can create children to represent follow-up work.
+next = tracer.newSpan(oneWayReceive.context()).name("step2").start();

    4. Sampling

    Sampling may be employed to reduce the data collected and reported out of process. +When a span is not sampled, it adds no overhead (a noop).

    Sampling is an up-front decision, meaning that the decision to report data is made at the first operation in a trace and that decision is propagated downstream.

    By default, a global sampler applies a single rate to all traced operations. +Tracer.Builder.sampler controls this setting, and it defaults to tracing every request.

    4.1 Declarative sampling

    Some applications need to sample based on the type or annotations of a java method.

    Most users use a framework interceptor to automate this sort of policy. +The following example shows how that might work internally:

    @Autowired Tracing tracing;
+
+// derives a sample rate from an annotation on a java method
+DeclarativeSampler<Traced> sampler = DeclarativeSampler.create(Traced::sampleRate);
+
+@Around("@annotation(traced)")
+public Object traceThing(ProceedingJoinPoint pjp, Traced traced) throws Throwable {
+  Span span = tracing.tracer().newTrace(sampler.sample(traced))...
+  try {
+    return pjp.proceed();
+  } finally {
+    span.finish();
+  }
+}

    4.2 Custom sampling

    Depending on what the operation is, you may want to apply different policies. +For example, you might not want to trace requests to static resources such as images, or you might want to trace all requests to a new api.

    Most users use a framework interceptor to automate this sort of policy. +The following example shows how that might work internally:

    @Autowired Tracer tracer;
+
+Span newTrace(Request input) {
+  SamplingFlags flags = SamplingFlags.NONE;
+  if (input.url().startsWith("/experimental")) {
+    flags = SamplingFlags.SAMPLED;
+  } else if (input.url().startsWith("/static")) {
+    flags = SamplingFlags.NOT_SAMPLED;
+  }
+  return tracer.newTrace(flags);
+}

    4.3 Sampling in Spring Cloud Sleuth

    By default Spring Cloud Sleuth sets all spans to non-exportable. +That means that traces appear in logs but not in any remote store. +For testing the default is often enough, and it probably is all you need if you use only the logs (for example, with an ELK aggregator). +If you export span data to Zipkin, there is also an Sampler.ALWAYS_SAMPLE setting that exports everything and a ProbabilityBasedSampler setting that samples a fixed fraction of spans.

    [Note]Note

    The ProbabilityBasedSampler is the default if you use spring-cloud-sleuth-zipkin. +You can configure the exports by setting spring.sleuth.sampler.probability. +The passed value needs to be a double from 0.0 to 1.0.

    A sampler can be installed by creating a bean definition, as shown in the following example:

    @Bean
 public Sampler defaultSampler() {
-	return new AlwaysSampler();
-}
    [Tip]Tip

    You can set the HTTP header X-B3-Flags to 1 or when doing messaging you can -set spanFlags header to 1. Then the current span will be forced to be exportable -regardless of the sampling decision.

    5. Instrumentation

    Spring Cloud Sleuth instruments all your Spring application -automatically, so you shouldn’t have to do anything to activate -it. The instrumentation is added using a variety of technologies -according to the stack that is available, e.g. for a servlet web -application we use a Filter, and for Spring Integration we use -ChannelInterceptors.

    You can customize the keys used in span tags. To limit the volume of -span data, by default an HTTP request will be tagged only with a -handful of metadata like the status code, host and URL. You can add -request headers by configuring spring.sleuth.keys.http.headers (a -list of header names).

    [Note]Note

    Remember that tags are only collected and exported if there is a -Sampler that allows it (by default there is not, so there is no -danger of accidentally collecting too much data without configuring -something).

    [Note]Note

    Currently the instrumentation in Spring Cloud Sleuth is eager - it means that -we’re actively trying to pass the tracing context between threads. Also timing events -are captured even when sleuth isn’t exporting data to a tracing system. -This approach may change in the future towards being lazy on this matter.

    6. Span lifecycle

    You can do the following operations on the Span by means of org.springframework.cloud.sleuth.Tracer interface:

    • start - when you start a span its name is assigned and start timestamp is recorded.
    • close - the span gets finished (the end time of the span is recorded) and if -the span is exportable then it will be eligible for collection to Zipkin. -The span is also removed from the current thread.
    • continue - a new instance of span will be created whereas it will be a copy of the -one that it continues.
    • detach - the span doesn’t get stopped or closed. It only gets removed from the current thread.
    • create with explicit parent - you can create a new span and set an explicit parent to it
    [Tip]Tip

    Spring creates the instance of Tracer for you. In order to use it all you need is to just autowire it.

    6.1 Creating and closing spans

    You can manually create spans by using the Tracer interface.

    // Start a span. If there was a span present in this thread it will become
+	return Sampler.ALWAYS_SAMPLE;
+}
    [Tip]Tip

    You can set the HTTP header X-B3-Flags to 1, or, when doing messaging, you can set the spanFlags header to 1. +Doing so forces the current span to be exportable regardless of the sampling decision.

    5. Propagation

    Propagation is needed to ensure activities originating from the same root are collected together in the same trace. +The most common propagation approach is to copy a trace context from a client by sending an RPC request to a server receiving it.

    For example, when a downstream HTTP call is made, its trace context is encoded as request headers and sent along with it, as shown in the following image:

       Client Span                                                Server Span
+┌──────────────────┐                                       ┌──────────────────┐
+│                  │                                       │                  │
+│   TraceContext   │           Http Request Headers        │   TraceContext   │
+│ ┌──────────────┐ │          ┌───────────────────┐        │ ┌──────────────┐ │
+│ │ TraceId      │ │          │ X─B3─TraceId      │        │ │ TraceId      │ │
+│ │              │ │          │                   │        │ │              │ │
+│ │ ParentSpanId │ │ Extract  │ X─B3─ParentSpanId │ Inject │ │ ParentSpanId │ │
+│ │              ├─┼─────────>│                   ├────────┼>│              │ │
+│ │ SpanId       │ │          │ X─B3─SpanId       │        │ │ SpanId       │ │
+│ │              │ │          │                   │        │ │              │ │
+│ │ Sampled      │ │          │ X─B3─Sampled      │        │ │ Sampled      │ │
+│ └──────────────┘ │          └───────────────────┘        │ └──────────────┘ │
+│                  │                                       │                  │
+└──────────────────┘                                       └──────────────────┘

    The names above are from B3 Propagation, which is built-in to Brave and has implementations in many languages and frameworks.

    Most users use a framework interceptor to automate propagation. +The next two examples show how that might work for a client and a server.

    The following example shows how client-side propagation might work:

    @Autowired Tracing tracing;
+
+// configure a function that injects a trace context into a request
+injector = tracing.propagation().injector(Request.Builder::addHeader);
+
+// before a request is sent, add the current span's context to it
+injector.inject(span.context(), request);

    The following example shows how server-side propagation might work:

    @Autowired Tracing tracing;
+@Autowired Tracer tracer;
+
+// configure a function that extracts the trace context from a request
+extractor = tracing.propagation().extractor(Request::getHeader);
+
+// when a server receives a request, it joins or starts a new trace
+span = tracer.nextSpan(extractor.extract(request));

    5.1 Propagating extra fields

    Sometimes you need to propagate extra fields, such as a request ID or an alternate trace context. +For example, if you are in a Cloud Foundry environment, you might want to pass the request ID, as shown in the following example:

    // when you initialize the builder, define the extra field you want to propagate
+Tracing.newBuilder().propagationFactory(
+  ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-vcap-request-id")
+);
+
+// later, you can tag that request ID or use it in log correlation
+requestId = ExtraFieldPropagation.get("x-vcap-request-id");

    You may also need to propagate a trace context that you are not using. +For example, you may be in an Amazon Web Services environment but not be reporting data to X-Ray. +To ensure X-Ray can co-exist correctly, pass-through its tracing header, as shown in the following example:

    tracingBuilder.propagationFactory(
+  ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-amzn-trace-id")
+);
    [Tip]Tip

    In Spring Cloud Sleuth all elements of the tracing builder Tracing.newBuilder() +are defined as beans. So if you want to pass a custom PropagationFactory, it’s enough +for you to create a bean of that type and we will set it in the Tracing bean.

    5.1.1 Prefixed fields

    If they follow a common pattern, you can also prefix fields. +The following example shows how to propagate x-vcap-request-id the field as-is but send the country-code and user-id fields on the wire as x-baggage-country-code and x-baggage-user-id, respectively:

    Tracing.newBuilder().propagationFactory(
+  ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY)
+                       .addField("x-vcap-request-id")
+                       .addPrefixedFields("baggage-", Arrays.asList("country-code", "user-id"))
+                       .build()
+);

    Later, you can call the following code to affect the country code of the current trace context:

    ExtraFieldPropagation.set("country-code", "FO");
+String countryCode = ExtraFieldPropagation.get("country-code");

    Alternatively, if you have a reference to a trace context, you can use it explicitly, as shown in the following example:

    ExtraFieldPropagation.set(span.context(), "country-code", "FO");
+String countryCode = ExtraFieldPropagation.get(span.context(), "country-code");
    [Important]Important

    A difference from previous versions of Sleuth is that, with Brave, you must pass the list of baggage keys. +There are two properties to achieve this. +With the spring.sleuth.baggage-keys, you set keys that get prefixed with baggage- for HTTP calls and baggage_ for messaging. +You can also use the spring.sleuth.propagation-keys property to pass a list of prefixed keys that are whitelisted without any prefix.

    5.1.2 Extracting a Propagated Context

    The TraceContext.Extractor<C> reads trace identifiers and sampling status from an incoming request or message. +The carrier is usually a request object or headers.

    This utility is used in standard instrumentation (such as HttpServerHandler`) but can also be used for custom RPC or messaging code.

    TraceContextOrSamplingFlags is usually used only with Tracer.nextSpan(extracted), unless you are +sharing span IDs between a client and a server.

    5.1.3 Sharing span IDs between Client and Server

    A normal instrumentation pattern is to create a span representing the server side of an RPC. +Extractor.extract might return a complete trace context when applied to an incoming client request. +Tracer.joinSpan attempts to continue this trace, using the same span ID if supported or creating a child span +if not. When the span ID is shared, the reported data includes a flag saying so.

    The following image shows an example of B3 propagation:

                                  ┌───────────────────┐      ┌───────────────────┐
+ Incoming Headers             │   TraceContext    │      │   TraceContext    │
+┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
+│ X─B3-TraceId      │─────────┼─┼> TraceId      │ │──────┼─┼> TraceId      │ │
+│                   │         │ │               │ │      │ │               │ │
+│ X─B3-ParentSpanId │─────────┼─┼> ParentSpanId │ │──────┼─┼> ParentSpanId │ │
+│                   │         │ │               │ │      │ │               │ │
+│ X─B3-SpanId       │─────────┼─┼> SpanId       │ │──────┼─┼> SpanId       │ │
+└───────────────────┘         │ │               │ │      │ │               │ │
+                              │ │               │ │      │ │  Shared: true │ │
+                              │ └───────────────┘ │      │ └───────────────┘ │
+                              └───────────────────┘      └───────────────────┘

    Some propagation systems forward only the parent span ID, detected when Propagation.Factory.supportsJoin() == false. +In this case, a new span ID is always provisioned, and the incoming context determines the parent ID.

    The following image shows an example of AWS propagation:

                                  ┌───────────────────┐      ┌───────────────────┐
+ x-amzn-trace-id              │   TraceContext    │      │   TraceContext    │
+┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │
+│ Root              │─────────┼─┼> TraceId      │ │──────┼─┼> TraceId      │ │
+│                   │         │ │               │ │      │ │               │ │
+│ Parent            │─────────┼─┼> SpanId       │ │──────┼─┼> ParentSpanId │ │
+└───────────────────┘         │ └───────────────┘ │      │ │               │ │
+                              └───────────────────┘      │ │  SpanId: New  │ │
+                                                         │ └───────────────┘ │
+                                                         └───────────────────┘

    Note: Some span reporters do not support sharing span IDs. +For example, if you set Tracing.Builder.spanReporter(amazonXrayOrGoogleStackdrive), you should disable join by setting Tracing.Builder.supportsJoin(false). +Doing so forces a new child span on Tracer.joinSpan().

    5.1.4 Implementing Propagation

    TraceContext.Extractor<C> is implemented by a Propagation.Factory plugin. +Internally, this code creates the union type, TraceContextOrSamplingFlags, with one of the following: +* TraceContext if trace and span IDs were present. +* TraceIdContext if a trace ID was present but span IDs were not present. +* SamplingFlags if no identifiers were present.

    Some Propagation implementations carry extra data from the point of extraction (for example, reading incoming headers) to injection (for example, writing outgoing headers). +For example, it might carry a request ID. +When implementations have extra data, they handle it as follows: +* If a TraceContext were extracted, add the extra data as TraceContext.extra(). +* Otherwise, add it as TraceContextOrSamplingFlags.extra(), which Tracer.nextSpan handles.

    6. Current Tracing Component

    Brave supports a "current tracing component" concept, which should only be used when you have no other way to get a reference. +This was made for JDBC connections, as they often initialize prior to the tracing component.

    The most recent tracing component instantiated is available through Tracing.current(). +You can also use Tracing.currentTracer() to get only the tracer. +If you use either of these methods, do not cache the result. +Instead, look them up each time you need them.

    7. Current Span

    Brave supports a "current span" concept which represents the in-flight operation. +You can use Tracer.currentSpan() to add custom tags to a span and Tracer.nextSpan() to create a child of whatever is in-flight.

    [Important]Important

    In Sleuth, you can autowire the Tracer bean to retrieve the current span via +tracer.currentSpan() method. To retrieve the current context just call +tracer.currentSpan().context(). To get the current trace id as String +you can use the traceIdString() method like this: tracer.currentSpan().context().traceIdString().

    7.1 Setting a span in scope manually

    When writing new instrumentation, it is important to place a span you created in scope as the current span. +Not only does doing so let users access it with Tracer.currentSpan(), but it also allows customizations such as SLF4J MDC to see the current trace IDs.

    Tracer.withSpanInScope(Span) facilitates this and is most conveniently employed by using the try-with-resources idiom. +Whenever external code might be invoked (such as proceeding an interceptor or otherwise), place the span in scope, as shown in the following example:

    @Autowired Tracer tracer;
+
+try (SpanInScope ws = tracer.withSpanInScope(span)) {
+  return inboundRequest.invoke();
+} finally { // note the scope is independent of the span
+  span.finish();
+}

    In edge cases, you may need to clear the current span temporarily (for example, launching a task that should not be associated with the current request). To do tso, pass null to withSpanInScope, as shown in the following example:

    @Autowired Tracer tracer;
+
+try (SpanInScope cleared = tracer.withSpanInScope(null)) {
+  startBackgroundThread();
+}

    8. Instrumentation

    Spring Cloud Sleuth automatically instruments all your Spring applications, so you should not have to do anything to activate it. +The instrumentation is added by using a variety of technologies according to the stack that is available. For example, for a servlet web application, we use a Filter, and, for Spring Integration, we use ChannelInterceptors.

    You can customize the keys used in span tags. +To limit the volume of span data, an HTTP request is, by default, tagged only with a handful of metadata, such as the status code, the host, and the URL. +You can add request headers by configuring spring.sleuth.keys.http.headers (a list of header names).

    [Note]Note

    Tags are collected and exported only if there is a Sampler that allows it. By default, there is no such Sampler, to ensure that there is no danger of accidentally collecting too much data without configuring something).

    9. Span lifecycle

    You can do the following operations on the Span by means of brave.Tracer:

    • start: When you start a span, its name is assigned and the start timestamp is recorded.
    • close: The span gets finished (the end time of the span is recorded) and, if the span is sampled, it is eligible for collection (for example, to Zipkin).
    • continue: A new instance of span is created. +It is a copy of the one that it continues.
    • detach: The span does not get stopped or closed. +It only gets removed from the current thread.
    • create with explicit parent: You can create a new span and set an explicit parent for it.
    [Tip]Tip

    Spring Cloud Sleuth creates an instance of Tracer for you. In order to use it, you can autowire it.

    9.1 Creating and finishing spans

    You can manually create spans by using the Tracer, as shown in the following example:

    // Start a span. If there was a span present in this thread it will become
 // the `newSpan`'s parent.
-Span newSpan = this.tracer.createSpan("calculateTax");
-try {
+Span newSpan = this.tracer.nextSpan().name("calculateTax");
+try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(newSpan.start())) {
 	// ...
 	// You can tag a span
-	this.tracer.addTag("taxValue", taxValue);
+	newSpan.tag("taxValue", taxValue);
 	// ...
 	// You can log an event on a span
-	newSpan.logEvent("taxCalculated");
+	newSpan.annotate("taxCalculated");
 } finally {
-	// Once done remember to close the span. This will allow collecting
+	// Once done remember to finish the span. This will allow collecting
 	// the span to send it to Zipkin
-	this.tracer.close(newSpan);
-}

    In this example we could see how to create a new instance of span. Assuming that there already -was a span present in this thread then it would become the parent of that span.

    [Important]Important

    Always clean after you create a span! Don’t forget to close a span if you want to send it to Zipkin.

    [Important]Important

    If your span contains a name greater than 50 chars, then that name will -be truncated to 50 chars. Your names have to be explicit and concrete. Big names lead to -latency issues and sometimes even thrown exceptions.

    6.2 Continuing spans

    Sometimes you don’t want to create a new span but you want to continue one. Example of such a -situation might be (of course it all depends on the use-case):

    • AOP - If there was already a span created before an aspect was reached then you might not want to create a new span.
    • Hystrix - executing a Hystrix command is most likely a logical part of the current processing. It’s in fact -only a technical implementation detail that you wouldn’t necessarily want to reflect in tracing as a separate being.

    The continued instance of span is equal to the one that it continues:

    Span continuedSpan = this.tracer.continueSpan(spanToContinue);
-assertThat(continuedSpan).isEqualTo(spanToContinue);

    To continue a span you can use the Tracer interface.

    // let's assume that we're in a thread Y and we've received
+	newSpan.finish();
+}

    In the preceding example, we could see how to create a new instance of the span. +If there is already a span in this thread, it becomes the parent of the new span.

    [Important]Important

    Always clean after you create a span. Also, always finish any span that you want to send to Zipkin.

    [Important]Important

    If your span contains a name greater than 50 chars, that name is truncated to 50 chars. +Your names have to be explicit and concrete. Big names lead to latency issues and sometimes even exceptions.

    9.2 Continuing Spans

    Sometimes, you do not want to create a new span but you want to continue one. An example of such a +situation might be as follows:

    • AOP: If there was already a span created before an aspect was reached, you might not want to create a new span.
    • Hystrix: Executing a Hystrix command is most likely a logical part of the current processing. +It is in fact merely a technical implementation detail that you would not necessarily want to reflect in tracing as a separate being.

    To continue a span, you can use brave.Tracer, as shown in the following example:

    // let's assume that we're in a thread Y and we've received
 // the `initialSpan` from thread X
-Span continuedSpan = this.tracer.continueSpan(initialSpan);
+Span continuedSpan = this.tracer.toSpan(newSpan.context());
 try {
 	// ...
 	// You can tag a span
-	this.tracer.addTag("taxValue", taxValue);
+	continuedSpan.tag("taxValue", taxValue);
 	// ...
 	// You can log an event on a span
-	continuedSpan.logEvent("taxCalculated");
+	continuedSpan.annotate("taxCalculated");
 } finally {
-	// Once done remember to detach the span. That way you'll
-	// safely remove it from the current thread without closing it
-	this.tracer.detach(continuedSpan);
-}
    [Important]Important

    Always clean after you create a span! Don’t forget to detach a span if some work was done started in one - thread (e.g. thread X) and it’s waiting for other threads (e.g. Y, Z) to finish. - Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected - the span in thread X should be closed.

    6.3 Creating spans with an explicit parent

    There is a possibility that you want to start a new span and provide an explicit parent of that span. -Let’s assume that the parent of a span is in one thread and you want to start a new span in another thread. The -startSpan method of the Tracer interface is the method you are looking for.

    // let's assume that we're in a thread Y and we've received
+	// Once done remember to flush the span. That means that
+	// it will get reported but the span itself is not yet finished
+	continuedSpan.flush();
+}

    9.3 Creating a Span with an explicit Parent

    You might want to start a new span and provide an explicit parent of that span. +Assume that the parent of a span is in one thread and you want to start a new span in another thread. +In Brave, whenever you call nextSpan(), it creates a span in reference to the span that is currently in scope. +You can put the span in scope and then call nextSpan(), as shown in the following example:

    // let's assume that we're in a thread Y and we've received
 // the `initialSpan` from thread X. `initialSpan` will be the parent
 // of the `newSpan`
-Span newSpan = this.tracer.createSpan("calculateCommission", initialSpan);
-try {
+Span newSpan = null;
+try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) {
+	newSpan = this.tracer.nextSpan().name("calculateCommission");
 	// ...
 	// You can tag a span
-	this.tracer.addTag("commissionValue", commissionValue);
+	newSpan.tag("commissionValue", commissionValue);
 	// ...
 	// You can log an event on a span
-	newSpan.logEvent("commissionCalculated");
+	newSpan.annotate("commissionCalculated");
 } finally {
-	// Once done remember to close the span. This will allow collecting
+	// Once done remember to finish the span. This will allow collecting
 	// the span to send it to Zipkin. The tags and events set on the
 	// newSpan will not be present on the parent
-	this.tracer.close(newSpan);
-}
    [Important]Important

    After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs - related to the fact that you have a span present in the current thread other than the one you’re trying to close. - What’s worse your spans won’t get closed properly thus will not get collected to Zipkin.

    7. Naming spans

    Picking a span name is not a trivial task. Span name should depict an operation name. The name should -be low cardinality (e.g. not include identifiers).

    Since there is a lot of instrumentation going on some of the span names will be -artificial like:

    • controller-method-name when received by a Controller with a method name conrollerMethodName
    • async for asynchronous operations done via wrapped Callable and Runnable.
    • @Scheduled annotated methods will return the simple name of the class.

    Fortunately, for the asynchronous processing you can provide explicit naming.

    7.1 @SpanName annotation

    You can name the span explicitly via the @SpanName annotation.

    @SpanName("calculateTax")
+	if (newSpan != null) {
+		newSpan.finish();
+	}
+}
    [Important]Important

    After creating such a span, you must finish it. Otherwise it is not reported (for example, to Zipkin).

    10. Naming spans

    Picking a span name is not a trivial task. A span name should depict an operation name. +The name should be low cardinality, so it should not include identifiers.

    Since there is a lot of instrumentation going on, some span names are artificial:

    • controller-method-name when received by a Controller with a method name of controllerMethodName
    • async for asynchronous operations done with wrapped Callable and Runnable interfaces.
    • Methods annotated with @Scheduled return the simple name of the class.

    Fortunately, for asynchronous processing, you can provide explicit naming.

    10.1 @SpanName Annotation

    You can name the span explicitly by using the @SpanName annotation, as shown in the following example:

    @SpanName("calculateTax")
 class TaxCountingRunnable implements Runnable {
 
 	@Override public void run() {
 		// perform logic
 	}
-}

    In this case, when processed in the following manner:

    Runnable runnable = new TraceRunnable(tracer, spanNamer, new TaxCountingRunnable());
+}

    In this case, when processed in the following manner, the span is named calculateTax:

    Runnable runnable = new TraceRunnable(tracing, spanNamer,
+		new TaxCountingRunnable());
 Future<?> future = executorService.submit(runnable);
 // ... some additional logic ...
-future.get();

    The span will be named calculateTax.

    7.2 toString() method

    It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous -instance of those classes. You can’t annotate such classes thus to override that, if there is no @SpanName annotation present, -we’re checking if the class has a custom implementation of the toString() method.

    So executing such code:

    Runnable runnable = new TraceRunnable(tracer, spanNamer, new Runnable() {
+future.get();

    10.2 toString() method

    It is pretty rare to create separate classes for Runnable or Callable. +Typically, one creates an anonymous instance of those classes. +You cannot annotate such classes. +To overcome that limitation, if there is no @SpanName annotation present, we check whether the class has a custom implementation of the toString() method.

    Running such code leads to creating a span named calculateTax, as shown in the following example:

    Runnable runnable = new TraceRunnable(tracing, spanNamer, new Runnable() {
 	@Override public void run() {
 		// perform logic
 	}
@@ -373,228 +597,144 @@ we’re checking if the class has a custom implementation of the // ... some additional logic ...
-future.get();

    will lead in creating a span named calculateTax.

    8. Managing spans with annotations

    8.1 Rationale

    The main arguments for this features are

    • api-agnostic means to collaborate with a span

      • use of annotations allows users to add to a span with no library dependency on a span api. -This allows Sleuth to change its core api less impact to user code.

    • reduced surface area for basic span operations.

      • without this feature one has to use the span api, which has lifecycle commands that -could be used incorrectly. By only exposing scope, tag and log functionality, users can -collaborate without accidentally breaking span lifecycle.

    • collaboration with runtime generated code

      • with libraries such as Spring Data / Feign the implementations of interfaces are generated -at runtime thus span wrapping of objects was tedious. Now you can provide annotations - over interfaces and arguments of those interfaces

    8.2 Creating new spans

    If you really don’t want to take care of creating local spans manually you can profit from the -@NewSpan annotation. Also we give you the @SpanTag annotation to add tags in an automated -fashion.

    Let’s look at some examples of usage.

    @NewSpan
-void testMethod();

    Annotating the method without any parameter will lead to a creation of a new span whose name -will be equal to annotated method name.

    @NewSpan("customNameOnTestMethod4")
-void testMethod4();

    If you provide the value in the annotation (either directly or via the name parameter) then -the created span will have the name as the provided value.

    // method declaration
+future.get();

    11. Managing Spans with Annotations

    You can manage spans with a variety of annotations.

    11.1 Rationale

    There are a number of good reasons to manage spans with annotations, including:

    • API-agnostic means to collaborate with a span. Use of annotations lets users add to a span with no library dependency on a span api. +Doing so lets Sleuth change its core API to create less impact to user code.
    • Reduced surface area for basic span operations. Without this feature, you must use the span api, which has lifecycle commands that could be used incorrectly. +By only exposing scope, tag, and log functionality, you can collaborate without accidentally breaking span lifecycle.
    • Collaboration with runtime generated code. With libraries such as Spring Data and Feign, the implementations of interfaces are generated at runtime. +Consequently, span wrapping of objects was tedious. +Now you can provide annotations over interfaces and the arguments of those interfaces.

    11.2 Creating New Spans

    If you do not want to create local spans manually, you can use the @NewSpan annotation. +Also, we provide the @SpanTag annotation to add tags in an automated fashion.

    Now we can consider some examples of usage.

    @NewSpan
+void testMethod();

    Annotating the method without any parameter leads to creating a new span whose name equals the annotated method name.

    @NewSpan("customNameOnTestMethod4")
+void testMethod4();

    If you provide the value in the annotation (either directly or by setting the name parameter), the created span has the provided value as the name.

    // method declaration
 @NewSpan(name = "customNameOnTestMethod5")
 void testMethod5(@SpanTag("testTag") String param);
 
 // and method execution
-this.testBean.testMethod5("test");

    You can combine both the name and a tag. Let’s focus on the latter. In this case whatever the value of -the annotated method’s parameter runtime value will be - that will be the value of the tag. In our sample -the tag key will be testTag and the tag value will be test.

    @NewSpan(name = "customNameOnTestMethod3")
+this.testBean.testMethod5("test");

    You can combine both the name and a tag. Let’s focus on the latter. +In this case, the value of the annotated method’s parameter runtime value becomes the value of the tag. +In our sample, the tag key is testTag, and the tag value is test.

    @NewSpan(name = "customNameOnTestMethod3")
 @Override
 public void testMethod3() {
-}

    You can place the @NewSpan annotation on both the class and an interface. If you override the -interface’s method and provide a different value of the @NewSpan annotation then the most -concrete one wins (in this case customNameOnTestMethod3 will be set).

    8.3 Continuing spans

    If you want to just add tags and annotations to an existing span it’s enough -to use the @ContinueSpan annotation as presented below. Note that in contrast -with the @NewSpan annotation you can also add logs via the log parameter:

    // method declaration
+}

    You can place the @NewSpan annotation on both the class and an interface. +If you override the interface’s method and provide a different value for the @NewSpan annotation, the most +concrete one wins (in this case customNameOnTestMethod3 is set).

    11.3 Continuing Spans

    If you want to add tags and annotations to an existing span, you can use the @ContinueSpan annotation, as shown in the following example:

    // method declaration
 @ContinueSpan(log = "testMethod11")
 void testMethod11(@SpanTag("testTag11") String param);
 
 // method execution
-this.testBean.testMethod11("test");

    That way the span will get continued and:

    • logs with name testMethod11.before and testMethod11.after will be created
    • if an exception will be thrown a log testMethod11.afterFailure will also be created
    • tag with key testTag11 and value test will be created

    8.4 More advanced tag setting

    There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation. -Precedence is:

    • try with the bean of TagValueResolver type and provided name
    • if one hasn’t provided the bean name, try to evaluate an expression. We’re searching for a TagValueExpressionResolver bean. -The default implementation uses SPEL expression resolution.
    • if one hasn’t provided any expression to evaluate just return a toString() value of the parameter

    8.4.1 Custom extractor

    The value of the tag for following method will be computed by an implementation of TagValueResolver interface. -Its class name has to be passed as the value of the resolver attribute.

    Having such an annotated method:

    @NewSpan
+this.testBean.testMethod11("test");
+this.testBean.testMethod13();

    (Note that, in contrast with the @NewSpan annotation ,you can also add logs with the log parameter.)

    That way, the span gets continued and:

    • Log entries named testMethod11.before and testMethod11.after are created.
    • If an exception is thrown, a log entry named testMethod11.afterFailure is also created.
    • A tag with a key of testTag11 and a value of test is created.

    11.4 Advanced Tag Setting

    There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation. +The precedence is as follows:

    1. Try with a bean of TagValueResolver type and a provided name.
    2. If the bean name has not been provided, try to evaluate an expression. +We search for a TagValueExpressionResolver bean. +The default implementation uses SPEL expression resolution. +IMPORTANT You can only reference properties from the SPEL expression. Method execution is not allowed due to security constraints.
    3. If we do not find any expression to evaluate, return the toString() value of the parameter.

    11.4.1 Custom extractor

    The value of the tag for the following method is computed by an implementation of TagValueResolver interface. +Its class name has to be passed as the value of the resolver attribute.

    Consider the following annotated method:

    @NewSpan
 public void getAnnotationForTagValueResolver(@SpanTag(key = "test", resolver = TagValueResolver.class) String test) {
-}

    and such a TagValueResolver bean implementation

    @Bean(name = "myCustomTagValueResolver")
+}

    Now further consider the following TagValueResolver bean implementation:

    @Bean(name = "myCustomTagValueResolver")
 public TagValueResolver tagValueResolver() {
 	return parameter -> "Value from myCustomTagValueResolver";
-}

    Will lead to setting of a tag value equal to Value from myCustomTagValueResolver.

    8.4.2 Resolving expressions for value

    Having such an annotated method:

    @NewSpan
-public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "length() + ' characters'") String test) {
-}

    and no custom implementation of a TagValueExpressionResolver will lead to evaluation of the SPEL expression and a tag with value 4 characters will be set on the span. -If you want to use some other expression resolution mechanism you can create your own implementation -of the bean.

    8.4.3 Using toString method

    Having such an annotated method:

    @NewSpan
+}

    The two preceding examples lead to setting a tag value equal to Value from myCustomTagValueResolver.

    11.4.2 Resolving Expressions for a Value

    Consider the following annotated method:

    @NewSpan
+public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "'hello' + ' characters'") String test) {
+}

    No custom implementation of a TagValueExpressionResolver leads to evaluation of the SPEL expression, and a tag with a value of 4 characters is set on the span. +If you want to use some other expression resolution mechanism, you can create your own implementation of the bean.

    11.4.3 Using the toString() method

    Consider the following annotated method:

    @NewSpan
 public void getAnnotationForArgumentToString(@SpanTag("test") Long param) {
-}

    if executed with a value of 15 will lead to setting of a tag with a String value of "15".

    9. Customizations

    Thanks to the SpanInjector and SpanExtractor you can customize the way spans -are created and propagated.

    There are currently two built-in ways to pass tracing information between processes:

    • via Spring Integration
    • via HTTP

    Span ids are extracted from Zipkin-compatible (B3) headers (either Message -or HTTP headers), to start or join an existing trace. Trace information is -injected into any outbound requests so the next hop can extract them.

    The key change in comparison to the previous versions of Sleuth is that Sleuth is implementing -the Open Tracing’s TextMap notion. In Sleuth it’s called SpanTextMap. Basically the idea -is that any means of communication (e.g. message, http request, etc.) can be abstracted via -a SpanTextMap. This abstraction defines how one can insert data into the carrier and -how to retrieve it from there. Thanks to this if you want to instrument a new HTTP library -that uses a FooRequest as a mean of sending HTTP requests then you have to create an -implementation of a SpanTextMap that delegates calls to FooRequest in terms of retrieval -and insertion of HTTP headers.

    9.1 Spring Integration

    For Spring Integration there are 2 interfaces responsible for creation of a Span from a Message. -These are:

    • MessagingSpanTextMapExtractor
    • MessagingSpanTextMapInjector

    You can override them by providing your own implementation.

    9.2 HTTP

    For HTTP there are 2 interfaces responsible for creation of a Span from a Message. -These are:

    • HttpSpanExtractor
    • HttpSpanInjector

    You can override them by providing your own implementation.

    9.3 Example

    Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names -you have

    • for trace id - correlationId
    • for span id - mySpanId

    This is a an example of a SpanExtractor

    static class CustomHttpSpanExtractor implements HttpSpanExtractor {
+}

    Running the preceding method with a value of 15 leads to setting a tag with a String value of "15".

    12. Customizations

    12.1 HTTP

    If a customization of client / server parsing of the HTTP related spans is required, +just register a bean of type brave.http.HttpClientParser or +brave.http.HttpServerParser. If client /server sampling is required, just +register a bean of type brave.http.HttpSampler and name the bean + sleuthClientSampler for client sampler and sleuthServerSampler for server sampler. + For your convenience the @ClientSampler and @ServerSampler + annotations can be used to inject the proper beans or to + reference the bean names via their static String NAME fields.

    Check out Brave’s code to see an example of how to make a path-based sampler +https://github.com/openzipkin/brave/tree/master/instrumentation/http#sampling-policy

    If you want to completely rewrite the HttpTracing bean you can use the SkipPatternProvider +interface to retrieve the URL Pattern for spans that should be not sampled. Below you can see +an example of usage of SkipPatternProvider inside a server side, HttpSampler.

    @Configuration
+class Config {
+  @Bean(name = ServerSampler.NAME)
+  HttpSampler myHttpSampler(SkipPatternProvider provider) {
+  	Pattern pattern = provider.skipPattern();
+  	return new HttpSampler() {
 
-	@Override public Span joinTrace(SpanTextMap carrier) {
-		Map<String, String> map = TextMapUtil.asMap(carrier);
-		long traceId = Span.hexToId(map.get("correlationid"));
-		long spanId = Span.hexToId(map.get("myspanid"));
-		// extract all necessary headers
-		Span.SpanBuilder builder = Span.builder().traceId(traceId).spanId(spanId);
-		// build rest of the Span
-		return builder.build();
-	}
-}
-
-static class CustomHttpSpanInjector implements HttpSpanInjector {
-
-	@Override
-	public void inject(Span span, SpanTextMap carrier) {
-		carrier.put("correlationId", span.traceIdString());
-		carrier.put("mySpanId", Span.idToHex(span.getSpanId()));
-	}
-}

    And you could register it like this:

    @Bean
-HttpSpanInjector customHttpSpanInjector() {
-	return new CustomHttpSpanInjector();
-}
-
-@Bean
-HttpSpanExtractor customHttpSpanExtractor() {
-	return new CustomHttpSpanExtractor();
-}

    Spring Cloud Sleuth does not add trace/span related headers to the Http Response for security reasons. If you need the headers then a custom SpanInjector -that injects the headers into the Http Response and a Servlet filter which makes use of this can be added the following way:

    static class CustomHttpServletResponseSpanInjector extends ZipkinHttpSpanInjector {
-
-	@Override
-	public void inject(Span span, SpanTextMap carrier) {
-		super.inject(span, carrier);
-		carrier.put(Span.TRACE_ID_NAME, span.traceIdString());
-		carrier.put(Span.SPAN_ID_NAME, Span.idToHex(span.getSpanId()));
-	}
-}
-
-static class HttpResponseInjectingTraceFilter extends GenericFilterBean {
+  		@Override public <Req> Boolean trySample(HttpAdapter<Req, ?> adapter, Req request) {
+  			String url = adapter.path(request);
+  			boolean shouldSkip = pattern.matcher(url).matches();
+  			if (shouldSkip) {
+  				return false;
+  			}
+  			return null;
+  		}
+  	};
+  }
+}

    12.2 TracingFilter

    You can also modify the behavior of the TracingFilter, which is the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response. +You can customize the tags or modify the response headers by registering your own instance of the TracingFilter bean.

    In the following example, we register the TracingFilter bean, add the ZIPKIN-TRACE-ID response header containing the current Span’s trace id, and add a tag with key custom and a value tag to the span.

    @Component
+@Order(TraceWebServletAutoConfiguration.TRACING_FILTER_ORDER + 1)
+class MyFilter extends GenericFilterBean {
 
 	private final Tracer tracer;
-	private final HttpSpanInjector spanInjector;
 
-	public HttpResponseInjectingTraceFilter(Tracer tracer, HttpSpanInjector spanInjector) {
+	MyFilter(Tracer tracer) {
 		this.tracer = tracer;
-		this.spanInjector = spanInjector;
 	}
 
-	@Override
-	public void doFilter(ServletRequest request, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
-		HttpServletResponse response = (HttpServletResponse) servletResponse;
-		Span currentSpan = this.tracer.getCurrentSpan();
-		this.spanInjector.inject(currentSpan, new HttpServletResponseTextMap(response));
-		filterChain.doFilter(request, response);
+	@Override public void doFilter(ServletRequest request, ServletResponse response,
+			FilterChain chain) throws IOException, ServletException {
+		Span currentSpan = this.tracer.currentSpan();
+		if (currentSpan == null) {
+			chain.doFilter(request, response);
+			return;
+		}
+		// for readability we're returning trace id in a hex form
+		((HttpServletResponse) response)
+				.addHeader("ZIPKIN-TRACE-ID",
+						currentSpan.context().traceIdString());
+		// we can also add some custom tags
+		currentSpan.tag("custom", "tag");
+		chain.doFilter(request, response);
 	}
-
-	 class HttpServletResponseTextMap implements SpanTextMap {
-
-		 private final HttpServletResponse delegate;
-
-		 HttpServletResponseTextMap(HttpServletResponse delegate) {
-			 this.delegate = delegate;
-		 }
-
-		 @Override
-		 public Iterator<Map.Entry<String, String>> iterator() {
-			 Map<String, String> map = new HashMap<>();
-			 for (String header : this.delegate.getHeaderNames()) {
-				map.put(header, this.delegate.getHeader(header));
-			 }
-			 return map.entrySet().iterator();
-		 }
-
-		 @Override
-		 public void put(String key, String value) {
-			this.delegate.addHeader(key, value);
-		 }
-	 }
-}

    And you could register them like this:

    @Bean HttpSpanInjector customHttpServletResponseSpanInjector() {
-	return new CustomHttpServletResponseSpanInjector();
+}

    12.3 Custom service name

    By default, Sleuth assumes that, when you send a span to Zipkin, you want the span’s service name to be equal to the value of the spring.application.name property. +That is not always the case, though. +There are situations in which you want to explicitly provide a different service name for all spans coming from your application. +To achieve that, you can pass the following property to your application to override that value (the example is for a service named myService):

    spring.zipkin.service.name: myService

    12.4 Customization of Reported Spans

    Before reporting spans (for example, to Zipkin) you may want to modify that span in some way. +You can do so by using the SpanAdjuster interface.

    In Sleuth, we generate spans with a fixed name. +Some users want to modify the name depending on values of tags. +You can implement the SpanAdjuster interface to alter that name.

    The following example shows how to register two beans that implement SpanAdjuster:

    @Bean SpanAdjuster adjusterOne() {
+	return span -> span.toBuilder().name("foo").build();
 }
 
-@Bean
-HttpResponseInjectingTraceFilter responseInjectingTraceFilter(Tracer tracer) {
-	return new HttpResponseInjectingTraceFilter(tracer, customHttpServletResponseSpanInjector());
-}

    9.4 TraceFilter

    You can also modify the behaviour of the TraceFilter - the component that is responsible -for processing the input HTTP request and adding tags basing on the HTTP response. You can customize -the tags, or modify the response headers by registering your own instance of the TraceFilter bean.

    In the following example we will register the TraceFilter bean and we will add the -ZIPKIN-TRACE-ID response header containing the current Span’s trace id. Also we will -add to the Span a tag with key custom and a value tag.

    @Bean
-TraceFilter myTraceFilter(BeanFactory beanFactory, final Tracer tracer) {
-	return new TraceFilter(beanFactory) {
-		@Override protected void addResponseTags(HttpServletResponse response,
-				Throwable e) {
-			// execute the default behaviour
-			super.addResponseTags(response, e);
-			// for readability we're returning trace id in a hex form
-			response.addHeader("ZIPKIN-TRACE-ID",
-					Span.idToHex(tracer.getCurrentSpan().getTraceId()));
-			// we can also add some custom tags
-			tracer.addTag("custom", "tag");
-		}
-	};
-}

    9.5 Custom SA tag in Zipkin

    Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented. -What you can do is to create a span with the peer.service tag that will contain a value of the service that you want to call. -Below you can see an example of a call to Redis that is wrapped in such a span.

    Unresolved directive in spring-cloud-sleuth.adoc - include::../../../..//spring-cloud-sleuth-zipkin-legacy/src/test/java/org/springframework/cloud/sleuth/zipkin/HttpZipkinSpanReporterTest.java[tags=service_name,indent=0]
    [Important]Important

    Remember not to add both peer.service tag and the SA tag! You have to add only peer.service.

    9.6 Custom service name

    By default Sleuth assumes that when you send a span to Zipkin, you want the span’s service name - to be equal to spring.application.name value. That’s not always the case though. There - are situations in which you want to explicitly provide a different service name for all spans coming - from your application. To achieve that it’s enough to just pass the following property - to your application to override that value (example for foo service name):

    spring.zipkin.service.name: foo

    9.7 Customization of reported spans

    Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way. - You can achieve that by using the SpanAdjuster interface.

    Example of usage:

    In Sleuth we’re generating spans with a fixed name. Some users want to modify the name depending on values -of tags. Implementation of the SpanAdjuster interface can be used to alter that name. Example:

    @Bean
-SpanAdjuster customSpanAdjuster() {
-    return span -> span.toBuilder().name(scrub(span.getName())).build();
-}

    This will lead in changing the name of the reported span just before it gets sent to Zipkin.

    [Important]Important

    Your SpanReporter should inject the SpanAdjuster and - allow span manipulation before the actual reporting is done.

    9.8 Host locator

    In order to define the host that is corresponding to a particular span we need to resolve the host name -and port. The default approach is to take it from server properties. If those for some reason are not set -then we’re trying to retrieve the host name from the network interfaces.

    If you have the discovery client enabled and prefer to retrieve the host address from the registered -instance in a service registry then you have to set the property (it’s applicable for both HTTP and -Stream based span reporting).

    spring.zipkin.locator.discovery.enabled: true

    10. Sending spans to Zipkin

    By default if you add spring-cloud-starter-zipkin as a dependency to your project, -when the span is closed, it will be sent to Zipkin over HTTP. The communication -is asynchronous. You can configure the URL by setting the spring.zipkin.baseUrl -property as follows:

    spring.zipkin.baseUrl: http://192.168.99.100:9411/

    If you want to find Zipkin via service discovery it’s enough to pass the -Zipkin’s service id inside the URL (example for zipkinserver service id)

    spring.zipkin.baseUrl: http://zipkinserver/

    11. Span Data as Messages

    [Important]Important

    The suggested approach is to use the Zipkin’s -native support for message based span sending. Starting from -Edgware Zipkin Stream server is deprecated and in Finchley -it got removed.

    You can accumulate and send span data over -Spring Cloud Stream by -including the spring-cloud-sleuth-stream jar as a dependency, and -adding a Channel Binder implementation -(e.g. spring-cloud-starter-stream-rabbit for RabbitMQ or -spring-cloud-starter-stream-kafka for Kafka). This will -automatically turn your app into a producer of messages with payload -type Spans.

    11.1 Zipkin Consumer

    Please refer to the Dalston Documentaion -on how to create a Stream Zipkin server. That approach has been -deprecated in Edgware and removed in Finchley release.

    11.2 Custom Consumer

    A custom consumer can also easily be implemented using -spring-cloud-sleuth-stream and binding to the SleuthSink. Example:

    @EnableBinding(SleuthSink.class)
-@SpringBootApplication(exclude = SleuthStreamAutoConfiguration.class)
-@MessageEndpoint
-public class Consumer {
-
-    @ServiceActivator(inputChannel = SleuthSink.INPUT)
-    public void sink(Spans input) throws Exception {
-        // ... process spans
-    }
-}
    [Note]Note

    the sample consumer application above explicitly excludes -SleuthStreamAutoConfiguration so it doesn’t send messages to itself, -but this is optional (you might actually want to trace requests into -the consumer app).

    In order to customize the polling mechanism you can create a bean of PollerMetadata type -with name equal to StreamSpanReporter.POLLER. Here you can find an example of such a configuration.

    @Configuration
-public static class CustomPollerConfiguration {
-
-	@Bean(name = StreamSpanReporter.POLLER)
-	PollerMetadata customPoller() {
-		PollerMetadata poller = new PollerMetadata();
-		poller.setMaxMessagesPerPoll(500);
-		poller.setTrigger(new PeriodicTrigger(5000L));
-		return poller;
+@Bean SpanAdjuster adjusterTwo() {
+	return span -> span.toBuilder().name(span.name() + " bar").build();
+}

    The preceding example results in changing the name of the reported span to foo bar, just before it gets reported (for example, to Zipkin).

    12.5 Host Locator

    [Important]Important

    This section is about defining host from service discovery. +It is NOT about finding Zipkin through service discovery.

    To define the host that corresponds to a particular span, we need to resolve the host name and port. +The default approach is to take these values from server properties. +If those are not set, we try to retrieve the host name from the network interfaces.

    If you have the discovery client enabled and prefer to retrieve the host address from the registered instance in a service registry, you have to set the spring.zipkin.locator.discovery.enabled property (it is applicable for both HTTP-based and Stream-based span reporting), as follows:

    spring.zipkin.locator.discovery.enabled: true

    13. Sending Spans to Zipkin

    By default, if you add spring-cloud-starter-zipkin as a dependency to your project, when the span is closed, it is sent to Zipkin over HTTP. +The communication is asynchronous. +You can configure the URL by setting the spring.zipkin.baseUrl property, as follows:

    spring.zipkin.baseUrl: http://192.168.99.100:9411/

    If you want to find Zipkin through service discovery, you can pass the Zipkin’s service ID inside the URL, as shown in the following example for zipkinserver service ID:

    spring.zipkin.baseUrl: http://zipkinserver/

    To disable this feature just set spring.zipkin.discoveryClientEnabled to `false.

    When the Discovery Client feature is enabled, Sleuth uses +LoadBalancerClient to find the URL of the Zipkin Server. It means +that you can set up the load balancing configuration e.g. via Ribbon.

    zipkinserver:
+  ribbon:
+    ListOfServers: host1,host2

    If you have web, rabbit, or kafka together on the classpath, you might need to pick the means by which you would like to send spans to zipkin. +To do so, set web, rabbit, or kafka to the spring.zipkin.sender.type property. +The following example shows setting the sender type for web:

    spring.zipkin.sender.type: web

    To customize the RestTemplate that sends spans to Zipkin via HTTP, you can register +the ZipkinRestTemplateCustomizer bean.

    @Configuration
+class MyConfig {
+	@Bean ZipkinRestTemplateCustomizer myCustomizer() {
+		return new ZipkinRestTemplateCustomizer() {
+			@Override
+			void customize(RestTemplate restTemplate) {
+				// customize the RestTemplate
+			}
+		};
 	}
-}

    12. Metrics

    Currently Spring Cloud Sleuth registers very simple metrics related to spans. -It’s using the Spring Boot’s metrics support -to calculate the number of accepted and dropped spans. Each time a span gets -sent to Zipkin the number of accepted spans will increase. If there’s an error then -the number of dropped spans will get increased.

    13. Integrations

    13.1 Runnable and Callable

    If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative.

    Example for Runnable:

    Runnable runnable = new Runnable() {
+}

    If, however, you would like to control the full process of creating the RestTemplate +object, you will have to create a bean of zipkin2.reporter.Sender type.

    	@Bean Sender myRestTemplateSender(ZipkinProperties zipkin,
+			ZipkinRestTemplateCustomizer zipkinRestTemplateCustomizer) {
+		RestTemplate restTemplate = mySuperCustomRestTemplate();
+		zipkinRestTemplateCustomizer.customize(restTemplate);
+		return myCustomSender(zipkin, restTemplate);
+	}

    14. Zipkin Stream Span Consumer

    [Important]Important

    We recommend using Zipkin’s native support for message-based span sending. +Starting from the Edgware release, the Zipkin Stream server is deprecated. +In the Finchley release, it got removed.

    If for some reason you need to create the deprecated Stream Zipkin server, see the Dalston Documentation.

    15. Integrations

    15.1 OpenTracing

    Spring Cloud Sleuth is compatible with OpenTracing. +If you have OpenTracing on the classpath, we automatically register the OpenTracing Tracer bean. +If you wish to disable this, set spring.sleuth.opentracing.enabled to false

    15.2 Runnable and Callable

    If you wrap your logic in Runnable or Callable, you can wrap those classes in their Sleuth representative, as shown in the following example for Runnable:

    Runnable runnable = new Runnable() {
 	@Override
 	public void run() {
 		// do some work
@@ -606,10 +746,11 @@ the number of dropped spans will get increased.
    // Manual `TraceRunnable` creation with explicit "calculateTax" Span name -Runnable traceRunnable = new TraceRunnable(tracer, spanNamer, runnable, "calculateTax"); -// Wrapping `Runnable` with `Tracer`. The Span name will be taken either from the -// `@SpanName` annotation or from `toString` method -Runnable traceRunnableFromTracer = tracer.wrap(runnable);

    Example for Callable:

    Callable<String> callable = new Callable<String>() {
+Runnable traceRunnable = new TraceRunnable(tracing, spanNamer, runnable,
+		"calculateTax");
+// Wrapping `Runnable` with `Tracing`. That way the current span will be available
+// in the thread of `Runnable`
+Runnable traceRunnableFromTracer = tracing.currentTraceContext().wrap(runnable);

    The following example shows how to do so for Callable:

    Callable<String> callable = new Callable<String>() {
 	@Override
 	public String call() throws Exception {
 		return someLogic();
@@ -621,82 +762,64 @@ Runnable traceRunnableFromTracer = tracer.wrap(runnable);

    Example for // Manual `TraceCallable` creation with explicit "calculateTax" Span name -Callable<String> traceCallable = new TraceCallable<>(tracer, spanNamer, callable, "calculateTax"); -// Wrapping `Callable` with `Tracer`. The Span name will be taken either from the -// `@SpanName` annotation or from `toString` method -Callable<String> traceCallableFromTracer = tracer.wrap(callable);

    That way you will ensure that a new Span is created and closed for each execution.

    13.2 Hystrix

    13.2.1 Custom Concurrency Strategy

    We’re registering a custom HystrixConcurrencyStrategy -that wraps all Callable instances into their Sleuth representative - -the TraceCallable. The strategy either starts or continues a span depending on the fact whether tracing was already going -on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false.

    13.2.2 Manual Command setting

    Assuming that you have the following HystrixCommand:

    HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
+Callable<String> traceCallable = new TraceCallable<>(tracing, spanNamer, callable,
+		"calculateTax");
+// Wrapping `Callable` with `Tracing`. That way the current span will be available
+// in the thread of `Callable`
+Callable<String> traceCallableFromTracer = tracing.currentTraceContext().wrap(callable);

    That way, you ensure that a new span is created and closed for each execution.

    15.3 Hystrix

    15.3.1 Custom Concurrency Strategy

    We register a custom HystrixConcurrencyStrategy called TraceCallable that wraps all Callable instances in their Sleuth representative. +The strategy either starts or continues a span, depending on whether tracing was already going on before the Hystrix command was called. +To disable the custom Hystrix Concurrency Strategy, set the spring.sleuth.hystrix.strategy.enabled to false.

    15.3.2 Manual Command setting

    Assume that you have the following HystrixCommand:

    HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) {
 	@Override
 	protected String run() throws Exception {
 		return someLogic();
 	}
-};

    In order to pass the tracing information you have to wrap the same logic in the Sleuth version of the HystrixCommand which is the -TraceCommand:

    TraceCommand<String> traceCommand = new TraceCommand<String>(tracer, traceKeys, setter) {
+};

    To pass the tracing information, you have to wrap the same logic in the Sleuth version of the HystrixCommand, which is called +TraceCommand, as shown in the following example:

    TraceCommand<String> traceCommand = new TraceCommand<String>(tracer, setter) {
 	@Override
 	public String doRun() throws Exception {
 		return someLogic();
 	}
-};

    13.3 RxJava

    We’re registering a custom RxJavaSchedulersHook -that wraps all Action0 instances into their Sleuth representative - -the TraceAction. The hook either starts or continues a span depending on the fact whether tracing was already going -on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the spring.sleuth.rxjava.schedulers.hook.enabled to false.

    You can define a list of regular expressions for thread names, for which you don’t want a Span to be created. Just provide a comma separated list -of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

    13.4 HTTP integration

    Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false.

    13.4.1 HTTP Filter

    Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which - the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would - like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern.

    13.4.2 HandlerInterceptor

    Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an - existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. The - TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. If the - the TraceFilter doesn’t see this attribute set it will create a "fallback" span which is an additional - span created on the server side so that the trace is presented properly in the UI. Seeing that most likely - signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth.

    13.4.3 Async Servlet support

    If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one.

    13.4.4 WebFlux support

    Via the TraceWebFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which - the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would - like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern.

    13.5 HTTP client integration

    13.5.1 Synchronous Rest Template

    We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a -call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous RestTemplate features -just set spring.sleuth.web.client.enabled to false.

    [Important]Important

    You have to register RestTemplate as a bean so that the interceptors will get injected. -If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work.

    13.5.2 Asynchronous Rest Template

    [Important]Important

    A traced version of an AsyncRestTemplate bean is registered for you out of the box. If you -have your own bean you have to wrap it in a TraceAsyncRestTemplate representation. The best solution -is to only customize the ClientHttpRequestFactory and / or AsyncClientHttpRequestFactory. -If you have your own AsyncRestTemplate and you don’t wrap it your calls WILL NOT GET TRACED.

    Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the ClientHttpRequestFactory -and the AsyncClientHttpRequestFactory by registering your beans. Remember to use tracing compatible implementations (e.g. don’t forget to -wrap ThreadPoolTaskScheduler in a TraceAsyncListenableTaskExecutor). Example of custom request factories:

    @EnableAutoConfiguration
-@Configuration
-public static class TestConfiguration {
-
-	@Bean
-	ClientHttpRequestFactory mySyncClientFactory() {
-		return new MySyncClientHttpRequestFactory();
-	}
-
-	@Bean
-	AsyncClientHttpRequestFactory myAsyncClientFactory() {
-		return new MyAsyncClientHttpRequestFactory();
-	}
-}

    To block the AsyncRestTemplate features set spring.sleuth.web.async.client.enabled to false. -To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper set spring.sleuth.web.async.client.factory.enabled -to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false.

    Multiple Asynchronous Rest Templates

    Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you -can see an example of how to set up such a custom AsyncRestTemplate.

    @Configuration
+};

    15.4 RxJava

    We registering a custom RxJavaSchedulersHook that wraps all Action0 instances in their Sleuth representative, which is called TraceAction. +The hook either starts or continues a span, depending on whether tracing was already going on before the Action was scheduled. +To disable the custom RxJavaSchedulersHook, set the spring.sleuth.rxjava.schedulers.hook.enabled to false.

    You can define a list of regular expressions for thread names for which you do not want spans to be created. +To do so, provide a comma-separated list of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property.

    [Important]Important

    The suggest approach to reactive programming and Sleuth is to use +the Reactor support.

    15.5 HTTP integration

    Features from this section can be disabled by setting the spring.sleuth.web.enabled property with value equal to false.

    15.5.1 HTTP Filter

    Through the TracingFilter, all sampled incoming requests result in creation of a Span. +That Span’s name is http: + the path to which the request was sent. +For example, if the request was sent to /this/that then the name will be http:/this/that. +You can configure which URIs you would like to skip by setting the spring.sleuth.web.skipPattern property. +If you have ManagementServerProperties on classpath, its value of contextPath gets appended to the provided skip pattern. +If you want to reuse the Sleuth’s default skip patterns and just append your own, pass those patterns by using the spring.sleuth.web.additionalSkipPattern.

    15.5.2 HandlerInterceptor

    Since we want the span names to be precise, we use a TraceHandlerInterceptor that either wraps an existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. +The TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. +If the the TracingFilter does not see this attribute, it creates a "fallback" span, which is an additional span created on the server side so that the trace is presented properly in the UI. +If that happens, there is probably missing instrumentation. +In that case, please file an issue in Spring Cloud Sleuth.

    15.5.3 Async Servlet support

    If your controller returns a Callable or a WebAsyncTask, Spring Cloud Sleuth continues the existing span instead of creating a new one.

    15.5.4 WebFlux support

    Through TraceWebFilter, all sampled incoming requests result in creation of a Span. +That Span’s name is http: + the path to which the request was sent. +For example, if the request was sent to /this/that, the name is http:/this/that. +You can configure which URIs you would like to skip by using the spring.sleuth.web.skipPattern property. +If you have ManagementServerProperties on the classpath, its value of contextPath gets appended to the provided skip pattern. +If you want to reuse Sleuth’s default skip patterns and append your own, pass those patterns by using the spring.sleuth.web.additionalSkipPattern.

    15.5.5 Dubbo RPC support

    Via the integration with Brave, Spring Cloud Sleuth supports Dubbo. +It’s enough to add the brave-instrumentation-dubbo-rpc dependency:

    <dependency>
+    <groupId>io.zipkin.brave</groupId>
+    <artifactId>brave-instrumentation-dubbo-rpc</artifactId>
+</dependency>

    You need to also set a dubbo.properties file with the following contents:

    dubbo.provider.filter=tracing
+dubbo.consumer.filter=tracing

    You can read more about Brave - Dubbo integration here. +An example of Spring Cloud Sleuth and Dubbo can be found here.

    15.6 HTTP Client Integration

    15.6.1 Synchronous Rest Template

    We inject a RestTemplate interceptor to ensure that all the tracing information is passed to the requests. +Each time a call is made, a new Span is created. +It gets closed upon receiving the response. +To block the synchronous RestTemplate features, set spring.sleuth.web.client.enabled to false.

    [Important]Important

    You have to register RestTemplate as a bean so that the interceptors get injected. +If you create a RestTemplate instance with a new keyword, the instrumentation does NOT work.

    15.6.2 Asynchronous Rest Template

    [Important]Important

    Starting with Sleuth 2.0.0, we no longer register a bean of AsyncRestTemplate type. +It is up to you to create such a bean. +Then we instrument it.

    To block the AsyncRestTemplate features, set spring.sleuth.web.async.client.enabled to false. +To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper, set spring.sleuth.web.async.client.factory.enabled +to false. +If you do not want to create AsyncRestClient at all, set spring.sleuth.web.async.client.template.enabled to false.

    Multiple Asynchronous Rest Templates

    Sometimes you need to use multiple implementations of the Asynchronous Rest Template. +In the following snippet, you can see an example of how to set up such a custom AsyncRestTemplate:

    @Configuration
 @EnableAutoConfiguration
 static class Config {
-	@Autowired Tracer tracer;
-	@Autowired HttpTraceKeysInjector httpTraceKeysInjector;
-	@Autowired HttpSpanInjector spanInjector;
 
 	@Bean(name = "customAsyncRestTemplate")
-	public AsyncRestTemplate traceAsyncRestTemplate(@Qualifier("customHttpRequestFactoryWrapper")
-			TraceAsyncClientHttpRequestFactoryWrapper wrapper, ErrorParser errorParser) {
-		return new TraceAsyncRestTemplate(wrapper, this.tracer, errorParser);
-	}
-
-	@Bean(name = "customHttpRequestFactoryWrapper")
-	public TraceAsyncClientHttpRequestFactoryWrapper traceAsyncClientHttpRequestFactory() {
-		return new TraceAsyncClientHttpRequestFactoryWrapper(this.tracer,
-				this.spanInjector,
-				asyncClientFactory(),
-				clientHttpRequestFactory(),
-				this.httpTraceKeysInjector);
+	public AsyncRestTemplate traceAsyncRestTemplate() {
+		return new AsyncRestTemplate(asyncClientFactory(), clientHttpRequestFactory());
 	}
 
 	private ClientHttpRequestFactory clientHttpRequestFactory() {
@@ -710,32 +833,32 @@ can see an example of how to set up such a custom AsyncRes
 		//CUSTOMIZE HERE
 		return factory;
 	}
-}

    13.5.3 WebClient

    We inject a ExchangeFilterFunction implementation that creates a span and via on success and on -error callbacks takes care of closing client side spans.

    [Important]Important

    You have to register WebClient as a bean so that the tracing instrumention gets applied. -If you create a WebClient instance with a new keyword then the instrumentation WILL NOT work.

    13.5.4 Traverson

    If you’re using the Traverson library -it’s enough for you to inject a RestTemplate as a bean into your Traverson object. Since RestTemplate -is already intercepted, you will get full support of tracing in your client. Below you can find a pseudo code -of how to do that:

    @Autowired RestTemplate restTemplate;
+}

    15.6.3 WebClient

    We inject a ExchangeFilterFunction implementation that creates a span and, through on-success and on-error callbacks, takes care of closing client-side spans.

    To block this feature, set spring.sleuth.web.client.enabled to false.

    [Important]Important

    You have to register WebClient as a bean so that the tracing instrumentation gets applied. +If you create a WebClient instance with a new keyword, the instrumentation does NOT work.

    15.6.4 Traverson

    If you use the Traverson library, you can inject a RestTemplate as a bean into your Traverson object. +Since RestTemplate is already intercepted, you get full support for tracing in your client. The following pseudo code +shows how to do that:

    @Autowired RestTemplate restTemplate;
 
 Traverson traverson = new Traverson(URI.create("http://some/address"),
     MediaType.APPLICATION_JSON, MediaType.APPLICATION_JSON_UTF8).setRestOperations(restTemplate);
-// use Traverson

    13.6 Feign

    By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely -by setting spring.sleuth.feign.enabled to false. If you do so then no Feign related instrumentation will take place.

    Part of Feign instrumentation is done via a FeignBeanPostProcessor. You can disable it by providing the spring.sleuth.feign.processor.enabled equal to false. -If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation -however will be still there.

    13.7 Asynchronous communication

    13.7.1 @Async annotated methods

    In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads. -You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false.

    If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics:

    • if the method is annotated with @SpanName then the value of the annotation will be the Span’s name
    • if the method is not annotated with @SpanName the Span name will be the annotated method name
    • the Span will be tagged with that method’s class name and the method name too

    13.7.2 @Scheduled annotated methods

    In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour -by setting the value of spring.sleuth.scheduled.enabled to false.

    If you annotate your method with @Scheduled then we’ll automatically create a new Span with the following characteristics:

    • the Span name will be the annotated method name
    • the Span will be tagged with that method’s class name and the method name too

    If you want to skip Span creation for some @Scheduled annotated classes you can set the -spring.sleuth.scheduled.skipPattern with a regular expression that will match the fully qualified name of the -@Scheduled annotated class.

    [Tip]Tip

    If you are using spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask

    13.7.3 Executor, ExecutorService and ScheduledExecutorService

    We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations -are creating Spans each time a new task is submitted, invoked or scheduled.

    Here you can see an example of how to pass tracing information with TraceableExecutorService when working with CompletableFuture:

    CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> {
+// use Traverson

    15.6.5 Apache HttpClientBuilder and HttpAsyncClientBuilder

    We instrument the HttpClientBuilder and HttpAsyncClientBuilder so that +tracing context gets injected to the sent requests.

    To block these features, set spring.sleuth.web.client.enabled to false.

    15.6.6 Netty HttpClient

    We instrument the Netty’s HttpClient.

    To block this feature, set spring.sleuth.web.client.enabled to false.

    [Important]Important

    You have to register HttpClient as a bean so that the instrumentation happens. +If you create a HttpClient instance with a new keyword, the instrumentation does NOT work.

    15.6.7 UserInfoRestTemplateCustomizer

    We instrument the Spring Security’s UserInfoRestTemplateCustomizer.

    To block this feature, set spring.sleuth.web.client.enabled to false.

    15.7 Feign

    By default, Spring Cloud Sleuth provides integration with Feign through TraceFeignClientAutoConfiguration. +You can disable it entirely by setting spring.sleuth.feign.enabled to false. +If you do so, no Feign-related instrumentation take place.

    Part of Feign instrumentation is done through a FeignBeanPostProcessor. +You can disable it by setting spring.sleuth.feign.processor.enabled to false. +If you set it to false, Spring Cloud Sleuth does not instrument any of your custom Feign components. +However, all the default instrumentation is still there.

    15.8 Asynchronous Communication

    15.8.1 @Async Annotated methods

    In Spring Cloud Sleuth, we instrument async-related components so that the tracing information is passed between threads. +You can disable this behavior by setting the value of spring.sleuth.async.enabled to false.

    If you annotate your method with @Async, we automatically create a new Span with the following characteristics:

    • If the method is annotated with @SpanName, the value of the annotation is the Span’s name.
    • If the method is not annotated with @SpanName, the Span name is the annotated method name.
    • The span is tagged with the method’s class name and method name.

    15.8.2 @Scheduled Annotated Methods

    In Spring Cloud Sleuth, we instrument scheduled method execution so that the tracing information is passed between threads. +You can disable this behavior by setting the value of spring.sleuth.scheduled.enabled to false.

    If you annotate your method with @Scheduled, we automatically create a new span with the following characteristics:

    • The span name is the annotated method name.
    • The span is tagged with the method’s class name and method name.

    If you want to skip span creation for some @Scheduled annotated classes, you can set the spring.sleuth.scheduled.skipPattern with a regular expression that matches the fully qualified name of the @Scheduled annotated class. +If you use spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, a span is created for each Hystrix metrics and sent to Zipkin. +This behavior may be annoying. That’s why, by default, spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask.

    15.8.3 Executor, ExecutorService, and ScheduledExecutorService

    We provide LazyTraceExecutor, TraceableExecutorService, and TraceableScheduledExecutorService. Those implementations create spans each time a new task is submitted, invoked, or scheduled.

    The following example shows how to pass tracing information with TraceableExecutorService when working with CompletableFuture:

    CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> {
 	// perform some logic
 	return 1_000_000L;
-}, new TraceableExecutorService(executorService,
+}, new TraceableExecutorService(beanFactory, executorService,
 		// 'calculateTax' explicitly names the span - this param is optional
-		tracer, traceKeys, spanNamer, "calculateTax"));
    [Important]Important

    Sleuth doesn’t work with parallelStream() out of the box. If you want -to have the tracing information propagated through the stream you have to use the -approach with supplyAsync(...) as presented above.

    Customization of Executors

    Sometimes you need to set up a custom instance of the AsyncExecutor. In the following snippet you -can see an example of how to set up such a custom Executor.

    @Configuration
+		"calculateTax"));
    [Important]Important

    Sleuth does not work with parallelStream() out of the box. +If you want to have the tracing information propagated through the stream, you have to use the approach with supplyAsync(...), as shown earlier.

    Customization of Executors

    Sometimes, you need to set up a custom instance of the AsyncExecutor. +The following example shows how to set up such a custom Executor:

    @Configuration
 @EnableAutoConfiguration
 @EnableAsync
 static class CustomExecutorConfig extends AsyncConfigurerSupport {
@@ -753,9 +876,17 @@ can see an example of how to set up such a custom Executor
 		executor.initialize();
 		return new LazyTraceExecutor(this.beanFactory, executor);
 	}
-}

    13.8 Messaging

    Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and -subscribe events. To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false.

    You can provide the spring.sleuth.integration.patterns pattern to explicitly -provide the names of channels that you want to include for tracing. By default all channels -are included.

    [Important]Important

    When using the Executor to build a Spring Integration IntegrationFlow remember to use the untraced version of the Executor. -Decorating Spring Integration Executor Channel with TraceableExecutorService will cause the spans to be improperly closed.

    13.9 Zuul

    We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data). -To disable Zuul support set the spring.sleuth.zuul.enabled property to false.

    14. Running examples

    You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links:

    \ No newline at end of file +}

    15.9 Messaging

    Features from this section can be disabled by setting the spring.sleuth.messaging.enabled property with value equal to false.

    15.9.1 Spring Integration and Spring Cloud Stream

    Spring Cloud Sleuth integrates with Spring Integration. +It creates spans for publish and subscribe events. +To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false.

    You can provide the spring.sleuth.integration.patterns pattern to explicitly provide the names of channels that you want to include for tracing. +By default, all channels but hystrixStreamOutput channel are included.

    [Important]Important

    When using the Executor to build a Spring Integration IntegrationFlow, you must use the untraced version of the Executor. +Decorating the Spring Integration Executor Channel with TraceableExecutorService causes the spans to be improperly closed.

    15.9.2 Spring RabbitMq

    We instrument the RabbitTemplate so that tracing headers get injected +into the message.

    To block this feature, set spring.sleuth.messaging.rabbit.enabled to false.

    15.9.3 Spring Kafka

    We instrument the Spring Kafka’s ProducerFactory and ConsumerFactory +so that tracing headers get injected into the created Spring Kafka’s +Producer and Consumer.

    To block this feature, set spring.sleuth.messaging.kafka.enabled to false.

    [Note]Note

    We do not support context propagation via @KafkaListener annotation. +Check this issue for more information.

    15.10 Zuul

    We instrument the Zuul Ribbon integration by enriching the Ribbon requests with tracing information. +To disable Zuul support, set the spring.sleuth.zuul.enabled property to false.

    16. Running examples

    You can see the running examples deployed in the Pivotal Web Services. +Check them out at the following links:

    \ No newline at end of file diff --git a/2.0.x/spring-cloud-sleuth.xml b/2.0.x/spring-cloud-sleuth.xml index f90d7ddc3..5e2bd9b1f 100644 --- a/2.0.x/spring-cloud-sleuth.xml +++ b/2.0.x/spring-cloud-sleuth.xml @@ -4,17 +4,17 @@ Spring Cloud Sleuth -2017-11-28 +2018-07-25 -Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer +Adrian Cole, Spencer Gibb, Marcin Grzejszczak, Dave Syer, Jay Bryant A -2.0.0.BUILD-SNAPSHOT +2.0.1.BUILD-SNAPSHOT Introduction @@ -22,40 +22,42 @@
    Terminology Spring Cloud Sleuth borrows Dapper’s terminology. -Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an -RPC. Span’s are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span -is a part of. Spans also have other data, such as descriptions, timestamped events, key-value -annotations (tags), the ID of the span that caused them, and process ID’s (normally IP address). -Spans are started and stopped, and they keep track of their timing information. Once you create a -span, you must stop it at some point in the future. +Span: The basic unit of work. For example, sending an RPC is a new span, as is sending a response to an RPC. +Spans are identified by a unique 64-bit ID for the span and another 64-bit ID for the trace the span is a part of. +Spans also have other data, such as descriptions, timestamped events, key-value annotations (tags), the ID of the span that caused them, and process IDs (normally IP addresses). +Spans can be started and stopped, and they keep track of their timing information. +Once you create a span, you must stop it at some point in the future. -The initial span that starts a trace is called a root span. The value of span id -of that span is equal to trace id. +The initial span that starts a trace is called a root span. The value of the ID +of that span is equal to the trace ID. -Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed -big-data store, a trace might be formed by a put request. -Annotation: is used to record existence of an event in time. Some of the core annotations used to define -the start and stop of a request are: +Trace: A set of spans forming a tree-like structure. +For example, if you run a distributed big-data store, a trace might be formed by a PUT request. +Annotation: Used to record the existence of an event in time. With +Brave instrumentation, we no longer need to set special events +for Zipkin to understand who the client and server are, where +the request started, and where it ended. For learning purposes, +however, we mark these events to highlight what kind +of an action took place. -cs - Client Sent - The client has made a request. This annotation depicts the start of the span. +cs: Client Sent. The client has made a request. This annotation indicates the start of the span. -sr - Server Received - The server side got the request and will start processing it. -If one subtracts the cs timestamp from this timestamp one will receive the network latency. +sr: Server Received: The server side got the request and started processing it. +Subtracting the cs timestamp from this timestamp reveals the network latency. -ss - Server Sent - Annotated upon completion of request processing (when the response -got sent back to the client). If one subtracts the sr timestamp from this timestamp one -will receive the time needed by the server side to process the request. +ss: Server Sent. Annotated upon completion of request processing (when the response got sent back to the client). +Subtracting the sr timestamp from this timestamp reveals the time needed by the server side to process the request. -cr - Client Received - Signifies the end of the span. The client has successfully received the -response from the server side. If one subtracts the cs timestamp from this timestamp one -will receive the whole time needed by the client to receive the response from the server. +cr: Client Received. Signifies the end of the span. +The client has successfully received the response from the server side. +Subtracting the cs timestamp from this timestamp reveals the whole time needed by the client to receive the response from the server. -Visualization of what Span and Trace will look in a system together with the Zipkin annotations: +The following image shows how Span and Trace look in a system, together with the Zipkin annotations: @@ -64,13 +66,14 @@ will receive the whole time needed by the client to receive the response from th Trace Info propagation -Each color of a note signifies a span (7 spans - from A to G). If you have such information in the note: +Each color of a note signifies a span (there are seven spans - from A to G). +Consider the following note: Trace Id = X Span Id = D Client Sent -That means that the current span has Trace-Id set to X, Span-Id set to D. It also has emitted - Client Sent event. -This is how the visualization of the parent / child relationship of spans would look like: +This note indicates that the current span has Trace Id set to X and Span Id set to D. +Also, the Client Sent event took place. +The following image shows how parent-child relationships of spans look: @@ -82,10 +85,11 @@ Client Sent
    Purpose -In the following sections the example from the image above will be taken into consideration. +The following sections refer to the example shown in the preceding image.
    -Distributed tracing with Zipkin -Altogether there are 7 spans . If you go to traces in Zipkin you will see this number in the second trace: +Distributed Tracing with Zipkin +This example has seven spans. +If you go to traces in Zipkin, you can see this number in the second trace, as shown in the following image: @@ -94,7 +98,7 @@ Client Sent Traces -However if you pick a particular trace then you will see 4 spans: +However, if you pick a particular trace, you can see four spans, as shown in the following image: @@ -104,42 +108,44 @@ Client Sent -When picking a particular trace you will see merged spans. That means that if there were 2 spans sent to -Zipkin with Server Received and Server Sent / Client Received and Client Sent -annotations then they will presented as a single span. +When you pick a particular trace, you see merged spans. +That means that, if there were two spans sent to Zipkin with Server Received and Server Sent or Client Received and Client Sent annotations, they are presented as a single span. -Why is there a difference between the 7 and 4 spans in this case? +Why is there a difference between the seven and four spans in this case? -2 spans come from http:/start span. It has the Server Received (SR) and Server Sent (SS) annotations. +Two spans come from the http:/start span. It has the Server Received (sr) and Server Sent (ss) annotations. -2 spans come from the RPC call from service1 to service2 to the http:/foo endpoint. It has the Client Sent (CS) -and Client Received (CR) annotations on service1 side. It also has Server Received (SR) and Server Sent (SS) annotations -on the service2 side. Physically there are 2 spans but they form 1 logical span related to an RPC call. +Two spans come from the RPC call from service1 to service2 to the http:/foo endpoint. +The Client Sent (cs) and Client Received (cr) events took place on the service1 side. +Server Received (sr) and Server Sent (ss) events took place on the service2 side. +These two spans form one logical span related to an RPC call. -2 spans come from the RPC call from service2 to service3 to the http:/bar endpoint. It has the Client Sent (CS) -and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations -on the service3 side. Physically there are 2 spans but they form 1 logical span related to an RPC call. +Two spans come from the RPC call from service2 to service3 to the http:/bar endpoint. +The Client Sent (cs) and Client Received (cr) events took place on the service2 side. +The Server Received (sr) and Server Sent (ss) events took place on the service3 side. +These two spans form one logical span related to an RPC call. -2 spans come from the RPC call from service2 to service4 to the http:/baz endpoint. It has the Client Sent (CS) -and Client Received (CR) annotations on service2 side. It also has Server Received (SR) and Server Sent (SS) annotations -on the service4 side. Physically there are 2 spans but they form 1 logical span related to an RPC call. +Two spans come from the RPC call from service2 to service4 to the http:/baz endpoint. +The Client Sent (cs) and Client Received (cr) events took place on the service2 side. +Server Received (sr) and Server Sent (ss) events took place on the service4 side. +These two spans form one logical span related to an RPC call. -So if we count the physical spans we have 1 from http:/start, 2 from service1 calling service2, 2 form service2 -calling service3 and 2 from service2 calling service4. Altogether 7 spans. -Logically we see the information of Total Spans: 4 because we have 1 span related to the incoming request -to service1 and 3 spans related to RPC calls. +So, if we count the physical spans, we have one from http:/start, two from service1 calling service2, two from service2 +calling service3, and two from service2 calling service4. In sum, we have a total of seven spans. +Logically, we see the information of four total Spans because we have one span related to the incoming request +to service1 and three spans related to RPC calls.
    Visualizing errors -Zipkin allows you to visualize errors in your trace. When an exception was thrown and wasn’t caught then we’re -setting proper tags on the span which Zipkin can properly colorize. You could see in the list of traces one - trace that was in red color. That’s because there was an exception thrown. -If you click that trace then you’ll see a similar picture +Zipkin lets you visualize errors in your trace. +When an exception was thrown and was not caught, we set proper tags on the span, which Zipkin can then properly colorize. +You could see in the list of traces one trace that is red. That appears because an exception was thrown. +If you click that trace, you see a similar picture, as follows: @@ -148,7 +154,7 @@ setting proper tags on the span which Zipkin can properly colorize. You could se Error Traces -Then if you click on one of the spans you’ll see the following +If you then click on one of the spans, you see the following @@ -157,12 +163,19 @@ setting proper tags on the span which Zipkin can properly colorize. You could se Error Traces Info propagation -As you can see you can easily see the reason for an error and the whole stacktrace related to it. +The span shows the reason for the error and the whole stack trace related to it. +
    +
    +Distributed Tracing with Brave +Starting with version 2.0.0, Spring Cloud Sleuth uses Brave as the tracing library. +Consequently, Sleuth no longer takes care of storing the context but delegates that work to Brave. +Due to the fact that Sleuth had different naming and tagging conventions than Brave, we decided to follow Brave’s conventions from now on. +However, if you want to use the legacy Sleuth approaches, you can set the spring.sleuth.http.legacy.enabled property to true.
    Live examples
    -Click Pivotal Web Services icon to see it live! +Click the Pivotal Web Services icon to see it live! @@ -170,7 +183,8 @@ setting proper tags on the span which Zipkin can properly colorize. You could se Zipkin deployed on Pivotal Web Services
    -The dependency graph in Zipkin would look like this: +Click here to see it live! +The dependency graph in Zipkin should resemble the following image: @@ -180,7 +194,7 @@ setting proper tags on the span which Zipkin can properly colorize. You could se
    -Click Pivotal Web Services icon to see it live! +Click the Pivotal Web Services icon to see it live! @@ -188,10 +202,11 @@ setting proper tags on the span which Zipkin can properly colorize. You could se Zipkin deployed on Pivotal Web Services
    +Click here to see it live!
    Log correlation -When grepping the logs of those four applications by trace id equal to e.g. 2485ec27856c56f4 one would get the following: +When using grep to read the logs of those four applications by scanning for a trace ID equal to (for example) 2485ec27856c56f4, you get output resembling the following: service1.log:2016-02-26 11:15:47.561 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Hello from service1. Calling service2 service2.log:2016-02-26 11:15:47.710 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Hello from service2. Calling service3 and then service4 service3.log:2016-02-26 11:15:47.895 INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application : Hello from service3 @@ -199,9 +214,8 @@ service2.log:2016-02-26 11:15:47.924 INFO [service2,2485ec27856c56f4,9aa10ee6fb service4.log:2016-02-26 11:15:48.134 INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application : Hello from service4 service2.log:2016-02-26 11:15:48.156 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service4 [Hello from service4] service1.log:2016-02-26 11:15:48.182 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]] -If you’re using a log aggregating tool like Kibana, -Splunk etc. you can order the events that took place. An example of -Kibana would look like this: +If you use a log aggregating tool (such as Kibana, Splunk, and others), you can order the events that took place. +An example from Kibana would resemble the following image: @@ -210,7 +224,7 @@ Kibana would look like this: Log correlation with Kibana -If you want to use Logstash here is the Grok pattern for Logstash: +If you want to use Logstash, the following listing shows the Grok pattern for Logstash: filter { # pattern matching logback pattern grok { @@ -218,7 +232,7 @@ Kibana would look like this: } } -If you want to use Grok together with the logs from Cloud Foundry you have to use this pattern: +If you want to use Grok together with the logs from Cloud Foundry, you have to use the following pattern: filter { # pattern matching logback pattern @@ -228,30 +242,19 @@ Kibana would look like this: }
    JSON Logback with Logstash -Often you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. To do that you have to do the following (for readability -we’re passing the dependencies in the groupId:artifactId:version notation. -Dependencies setup - +Often, you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick. +To do so, you have to do the following (for readability, we pass the dependencies in the groupId:artifactId:version notation). +Dependencies Setup + -Ensure that Logback is on the classpath (ch.qos.logback:logback-core) +Ensure that Logback is on the classpath (ch.qos.logback:logback-core). -Add Logstash Logback encode - example for version 4.6 : net.logstash.logback:logstash-logback-encoder:4.6 +Add Logstash Logback encode. For example, to use version 4.6, add net.logstash.logback:logstash-logback-encoder:4.6. - -Logback setup -Below you can find an example of a Logback configuration (file named logback-spring.xml) that: - - -logs information from the application in a JSON format to a build/${spring.application.name}.json file - - -has commented out two additional appenders - console and standard log file - - -has the same logging pattern as the one presented in the previous section - - + +Logback Setup +Consider the following example of a Logback configuration file (named logback-spring.xml). <?xml version="1.0" encoding="UTF-8"?> <configuration> <include resource="org/springframework/boot/logging/logback/defaults.xml"/> @@ -328,86 +331,122 @@ we’re passing the dependencies in the groupId:artifactId:version< <!--<appender-ref ref="flatfile"/>--> </root> </configuration> +That Logback configuration file: + + +Logs information from the application in a JSON format to a build/${spring.application.name}.json file. + + +Has commented out two additional appenders: console and standard log file. + + +Has the same logging pattern as the one presented in the previous section. + + -If you’re using a custom logback-spring.xml then you have to pass the spring.application.name in -bootstrap instead of application property file. Otherwise your custom logback file won’t read the property properly. +If you use a custom logback-spring.xml, you must pass the spring.application.name in the bootstrap rather than the application property file. +Otherwise, your custom logback file does not properly read the property.
    Propagating Span Context -The span context is the state that must get propagated to any child Spans across process boundaries. +The span context is the state that must get propagated to any child spans across process boundaries. Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context. Baggage is an optional part. -Baggage is a set of key:value pairs stored in the span context. Baggage travels together with the trace -and is attached to every span. Spring Cloud Sleuth will understand that a header is baggage related if the HTTP - header is prefixed with baggage- and for messaging it starts with baggage_. +Baggage is a set of key:value pairs stored in the span context. +Baggage travels together with the trace and is attached to every span. +Spring Cloud Sleuth understands that a header is baggage-related if the HTTP header is prefixed with baggage- and, for messaging, it starts with baggage_. -There’s currently no limitation of the count or size of baggage items. However, keep in mind that -too many can decrease system throughput or increase RPC latency. In extreme cases, it could crash the app due -to exceeding transport-level message or header capacity. +There is currently no limitation of the count or size of baggage items. +However, keep in mind that too many can decrease system throughput or increase RPC latency. +In extreme cases, too much baggage can crash the application, due to exceeding transport-level message or header capacity. -Example of setting baggage on a span: -Span initialSpan = this.tracer.createSpan("span"); -initialSpan.setBaggageItem("foo", "bar"); -initialSpan.setBaggageItem("UPPER_CASE", "someValue"); -
    -Baggage vs. Span Tags -Baggage travels with the trace (i.e. every child span contains the baggage of its parent). Zipkin has no knowledge of -baggage and will not even receive that information. -Tags are attached to a specific span - they are presented for that particular span only. However you -can search by tag to find the trace, where there exists a span having the searched tag value. -If you want to be able to lookup a span based on baggage, you should add corresponding entry as a tag in the root span. -@Autowired Tracer tracer; - -Span span = tracer.getCurrentSpan(); -String baggageKey = "key"; -String baggageValue = "foo"; -span.setBaggageItem(baggageKey, baggageValue); -tracer.addTag(baggageKey, baggageValue); -
    -
    -
    -
    -Adding to the project +The following example shows setting baggage on a span: +Span initialSpan = this.tracer.nextSpan().name("span").start(); +try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) { + ExtraFieldPropagation.set("foo", "bar"); + ExtraFieldPropagation.set("UPPER_CASE", "someValue"); +} +
    +Baggage versus Span Tags +Baggage travels with the trace (every child span contains the baggage of its parent). +Zipkin has no knowledge of baggage and does not receive that information. -To ensure that your application name is properly displayed in Zipkin - set the spring.application.name property in bootstrap.yml. +Starting from Sleuth 2.0.0 you have to pass the baggage key names explicitly +in your project configuration. Read more about that setup here + +Tags are attached to a specific span. In other words, they are presented only for that particular span. +However, you can search by tag to find the trace, assuming a span having the searched tag value exists. +If you want to be able to lookup a span based on baggage, you should add a corresponding entry as a tag in the root span. + +The span must be in scope. + +The following listing shows integration tests that use baggage: + +The setup + +spring.sleuth: + baggage-keys: + - baz + - bizarrecase + propagation-keys: + - foo + - upper_case + + + +The code + +initialSpan.tag("foo", + ExtraFieldPropagation.get(initialSpan.context(), "foo")); +initialSpan.tag("UPPER_CASE", + ExtraFieldPropagation.get(initialSpan.context(), "UPPER_CASE")); + + +
    +
    + +
    +Adding Sleuth to the Project +This section addresses how to add Sleuth to your project with either Maven or Gradle. + +To ensure that your application name is properly displayed in Zipkin, set the spring.application.name property in bootstrap.yml.
    Only Sleuth (log correlation) -If you want to profit only from Spring Cloud Sleuth without the Zipkin integration just add -the spring-cloud-starter-sleuth module to your project. +If you want to use only Spring Cloud Sleuth without the Zipkin integration, add the spring-cloud-starter-sleuth module to your project. +The following example shows how to add Sleuth with Maven: Maven <dependencyManagement> - <dependencies> - <dependency> - <groupId>org.springframework.cloud</groupId> - <artifactId>spring-cloud-dependencies</artifactId> - <version>${release.train.version}</version> - <type>pom</type> - <scope>import</scope> - </dependency> - </dependencies> - </dependencyManagement> + <dependencies> + <dependency> + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-dependencies</artifactId> + <version>${release.train.version}</version> + <type>pom</type> + <scope>import</scope> + </dependency> + </dependencies> +</dependencyManagement> - <dependency> - <groupId>org.springframework.cloud</groupId> - <artifactId>spring-cloud-starter-sleuth</artifactId> - </dependency> +<dependency> + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-sleuth</artifactId> +</dependency> -In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM +We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself. -Add the dependency to spring-cloud-starter-sleuth +Add the dependency to spring-cloud-starter-sleuth. +The following example shows how to add Sleuth with Gradle: Gradle @@ -424,47 +463,47 @@ dependencies { -In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM +We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself. -Add the dependency to spring-cloud-starter-sleuth +Add the dependency to spring-cloud-starter-sleuth.
    Sleuth with Zipkin via HTTP -If you want both Sleuth and Zipkin just add the spring-cloud-starter-zipkin dependency. +If you want both Sleuth and Zipkin, add the spring-cloud-starter-zipkin dependency. +The following example shows how to do so for Maven: Maven <dependencyManagement> - <dependencies> - <dependency> - <groupId>org.springframework.cloud</groupId> - <artifactId>spring-cloud-dependencies</artifactId> - <version>${release.train.version}</version> - <type>pom</type> - <scope>import</scope> - </dependency> - </dependencies> - </dependencyManagement> + <dependencies> + <dependency> + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-dependencies</artifactId> + <version>${release.train.version}</version> + <type>pom</type> + <scope>import</scope> + </dependency> + </dependencies> +</dependencyManagement> - <dependency> - <groupId>org.springframework.cloud</groupId> - <artifactId>spring-cloud-starter-zipkin</artifactId> - </dependency> +<dependency> + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-zipkin</artifactId> +</dependency> -In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM +We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself. -Add the dependency to spring-cloud-starter-zipkin +Add the dependency to spring-cloud-starter-zipkin. +The following example shows how to do so for Gradle: Gradle @@ -481,56 +520,59 @@ dependencies { -In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM +We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself. -Add the dependency to spring-cloud-starter-zipkin +Add the dependency to spring-cloud-starter-zipkin.
    -
    -Sleuth with Zipkin via RabbitMQ or Kafka -If you want to use RabbitMQ or Kafka instead of http, add the spring-rabbit or spring-kafka -dependencies. The default destination name is zipkin. -Note: spring-cloud-sleuth-stream is deprecated and incompatible with these destinations -If you want Sleuth over RabbitMQ add the spring-cloud-starter-zipkin and spring-rabbit +
    +Sleuth with Zipkin over RabbitMQ or Kafka +If you want to use RabbitMQ or Kafka instead of HTTP, add the spring-rabbit or spring-kafka dependency. +The default destination name is zipkin. +If using Kafka, you must set the property spring.zipkin.sender.type property accordingly: +spring.zipkin.sender.type: kafka + +spring-cloud-sleuth-stream is deprecated and incompatible with these destinations. + +If you want Sleuth over RabbitMQ, add the spring-cloud-starter-zipkin and spring-rabbit dependencies. +The following example shows how to do so for Gradle: Maven <dependencyManagement> - <dependencies> - <dependency> - <groupId>org.springframework.cloud</groupId> - <artifactId>spring-cloud-dependencies</artifactId> - <version>${release.train.version}</version> - <type>pom</type> - <scope>import</scope> - </dependency> - </dependencies> - </dependencyManagement> + <dependencies> + <dependency> + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-dependencies</artifactId> + <version>${release.train.version}</version> + <type>pom</type> + <scope>import</scope> + </dependency> + </dependencies> +</dependencyManagement> - <dependency> - <groupId>org.springframework.cloud</groupId> - <artifactId>spring-cloud-starter-zipkin</artifactId> - </dependency> - <dependency> - <groupId>org.springframework.amqp</groupId> - <artifactId>spring-rabbit</artifactId> - </dependency> +<dependency> + <groupId>org.springframework.cloud</groupId> + <artifactId>spring-cloud-starter-zipkin</artifactId> +</dependency> +<dependency> + <groupId>org.springframework.amqp</groupId> + <artifactId>spring-rabbit</artifactId> +</dependency> -In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM +We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself. -Add the dependency to spring-cloud-starter-zipkin - that way all dependent dependencies will be downloaded +Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded. -To automatically configure rabbit, simply add the spring-rabbit dependency +To automatically configure RabbitMQ, add the spring-rabbit dependency. @@ -550,323 +592,689 @@ dependencies { -In order not to pick versions by yourself it’s much better if you add the dependency management via -the Spring BOM +We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself. -Add the dependency to spring-cloud-starter-zipkin - that way all dependent dependencies will be downloaded +Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded. -To automatically configure rabbit, simply add the spring-rabbit dependency +To automatically configure RabbitMQ, add the spring-rabbit dependency.
    -Additional resources -Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin - -click here to see the video +Additional Resources +You can watch a video of Reshmi Krishna and Marcin Grzejszczak talking about Spring Cloud +Sleuth and Zipkin by clicking here. +You can check different setups of Sleuth and Brave in the openzipkin/sleuth-webmvc-example repository. Features -Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator. Example logs: +Adds trace and span IDs to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator, as shown in the following example logs: 2016-02-02 15:30:57.902 INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ... 2016-02-02 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ... 2016-02-02 15:31:01.936 INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ... -notice the [appname,traceId,spanId,exportable] entries from the MDC: +Notice the [appname,traceId,spanId,exportable] entries from the MDC: -spanId - the id of a specific operation that took place +spanId: The ID of a specific operation that took place. -appname - the name of the application that logged the span +appname: The name of the application that logged the span. -traceId - the id of the latency graph that contains the span +traceId: The ID of the latency graph that contains the span. -exportable - whether the log should be exported to Zipkin or not. When would you like the span not to be -exportable? In the case in which you want to wrap some operation in a Span and have it written to the logs -only. +exportable: Whether the log should be exported to Zipkin. +When would you like the span not to be exportable? +When you want to wrap some operation in a Span and have it written to the logs only. -Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, -key-value annotations. Loosely based on HTrace, but Zipkin (Dapper) compatible. +Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, and key-value annotations. +Spring Cloud Sleuth is loosely based on HTrace but is compatible with Zipkin (Dapper). -Sleuth records timing information to aid in latency analysis. Using sleuth, you can pinpoint causes of -latency in your applications. Sleuth is written to not log too much, and to not cause your production application to crash. +Sleuth records timing information to aid in latency analysis. +By using sleuth, you can pinpoint causes of latency in your applications. + + +Sleuth is written to not log too much and to not cause your production application to crash. +To that end, Sleuth: -propagates structural data about your call-graph in-band, and the rest out-of-band. +Propagates structural data about your call graph in-band and the rest out-of-band. -includes opinionated instrumentation of layers such as HTTP +Includes opinionated instrumentation of layers such as HTTP. -includes sampling policy to manage volume +Includes a sampling policy to manage volume. -can report to a Zipkin system for query and visualization +Can report to a Zipkin system for query and visualization. -Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, -rest template, scheduled actions, message channels, zuul filters, feign client). +Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, rest template, scheduled actions, message channels, Zuul filters, and Feign client). -Sleuth includes default logic to join a trace across http or messaging boundaries. For example, http propagation -works via Zipkin-compatible request headers. This propagation logic is defined and customized via -SpanInjector and SpanExtractor implementations. +Sleuth includes default logic to join a trace across HTTP or messaging boundaries. +For example, HTTP propagation works over Zipkin-compatible request headers. -Sleuth gives you the possibility to propagate context (also known as baggage) between processes. That means that if you set on a Span -a baggage element then it will be sent downstream either via HTTP or messaging to other processes. +Sleuth can propagate context (also known as baggage) between processes. +Consequently, if you set a baggage element on a Span, it is sent downstream to other processes over either HTTP or messaging. -Provides a way to create / continue spans and add tags and logs via annotations. +Provides a way to create or continue spans and add tags and logs through annotations. -Provides simple metrics of accepted / dropped spans. - - -If spring-cloud-sleuth-zipkin then the app will generate and collect Zipkin-compatible traces. -By default it sends them via HTTP to a Zipkin server on localhost (port 9411). -Configure the location of the service using spring.zipkin.baseUrl. +If spring-cloud-sleuth-zipkin is on the classpath, the app generates and collects Zipkin-compatible traces. +By default, it sends them over HTTP to a Zipkin server on localhost (port 9411). +You can configure the location of the service by setting spring.zipkin.baseUrl. -If you depend on spring-rabbit or spring-kafka your app will send traces to a broker instead of http. +If you depend on spring-rabbit, your app sends traces to a RabbitMQ broker instead of HTTP. -Note: spring-cloud-sleuth-stream is deprecated and should no longer be used. +If you depend on spring-kafka, and set spring.zipkin.sender.type: kafka, your app sends traces to a Kafka broker instead of HTTP. + +spring-cloud-sleuth-stream is deprecated and should no longer be used. + + + +Spring Cloud Sleuth is OpenTracing compatible. + + -If using Zipkin, configure the percentage of spans exported using spring.sleuth.sampler.percentage -(default 0.1, i.e. 10%). Otherwise you might think that Sleuth is not working cause it’s omitting some spans. +If you use Zipkin, configure the probability of spans exported by setting spring.sleuth.sampler.probability +(default: 0.1, which is 10 percent). Otherwise, you might think that Sleuth is not working be cause it omits some spans. -the SLF4J MDC is always set and logback users will immediately see the trace and span ids in logs per the example - above. Other logging systems have to configure their own formatter to get the same result. The default is - logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}] - (this is a Spring Boot feature for logback users). - This means that if you’re not using SLF4J this pattern WILL NOT be automatically applied. +The SLF4J MDC is always set and logback users immediately see the trace and span IDs in logs per the example +shown earlier. +Other logging systems have to configure their own formatter to get the same result. +The default is as follows: +logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}] +(this is a Spring Boot feature for logback users). +If you do not use SLF4J, this pattern is NOT automatically applied. +
    +Introduction to Brave + +Starting with version 2.0.0, Spring Cloud Sleuth uses +Brave as the tracing library. +For your convenience, we embed part of the Brave’s docs here. + + +In the vast majority of cases you need to just use the Tracer +or SpanCustomizer beans from Brave that Sleuth provides. The documentation below contains +a high overview of what Brave is and how it works. + +Brave is a library used to capture and report latency information about distributed operations to Zipkin. +Most users do not use Brave directly. They use libraries or frameworks rather than employ Brave on their behalf. +This module includes a tracer that creates and joins spans that model the latency of potentially distributed work. +It also includes libraries to propagate the trace context over network boundaries (for example, with HTTP headers). +
    +Tracing +Most importantly, you need a brave.Tracer, configured to report to Zipkin. +The following example setup sends trace data (spans) to Zipkin over HTTP (as opposed to Kafka): +class MyClass { + + private final Tracer tracer; + + // Tracer will be autowired + MyClass(Tracer tracer) { + this.tracer = tracer; + } + + void doSth() { + Span span = tracer.newTrace().name("encode").start(); + // ... + } +} + +If your span contains a name longer than 50 chars, then that name is truncated to 50 chars. +Your names have to be explicit and concrete. +Big names lead to latency issues and sometimes even thrown exceptions. + +The tracer creates and joins spans that model the latency of potentially distributed work. +It can employ sampling to reduce overhead during the process, to reduce the amount of data sent to Zipkin, or both. +Spans returned by a tracer report data to Zipkin when finished or do nothing if unsampled. +After starting a span, you can annotate events of interest or add tags containing details or lookup keys. +Spans have a context that includes trace identifiers that place the span at the correct spot in the tree representing the distributed operation. +
    +
    +Local Tracing +When tracing local code, you can run it inside a span, as shown in the following example: +@Autowired Tracer tracer; + +Span span = tracer.newTrace().name("encode").start(); +try { + doSomethingExpensive(); +} finally { + span.finish(); +} +In the preceding example, the span is the root of the trace. +In many cases, the span is part of an existing trace. +When this is the case, call newChild instead of newTrace, as shown in the following example: +@Autowired Tracer tracer; + +Span span = tracer.newChild(root.context()).name("encode").start(); +try { + doSomethingExpensive(); +} finally { + span.finish(); +} +
    +
    +Customizing Spans +Once you have a span, you can add tags to it. +The tags can be used as lookup keys or details. +For example, you might add a tag with your runtime version, as shown in the following example: +span.tag("clnt/finagle.version", "6.36.0"); +When exposing the ability to customize spans to third parties, prefer brave.SpanCustomizer as opposed to brave.Span. +The former is simpler to understand and test and does not tempt users with span lifecycle hooks. +interface MyTraceCallback { + void request(Request request, SpanCustomizer customizer); +} +Since brave.Span implements brave.SpanCustomizer, you can pass it to users, as shown in the following example: +for (MyTraceCallback callback : userCallbacks) { + callback.request(request, span); +} +
    +
    +Implicitly Looking up the Current Span +Sometimes, you do not know if a trace is in progress or not, and you do not want users to do null checks. +brave.CurrentSpanCustomizer handles this problem by adding data to any span that’s in progress or drops, as shown in the following example: +Ex. +// The user code can then inject this without a chance of it being null. +@Autowired SpanCustomizer span; + +void userCode() { + span.annotate("tx.started"); + ... +} +
    +
    +RPC tracing + +Check for instrumentation written here and Zipkin’s list before rolling your own RPC instrumentation. + +RPC tracing is often done automatically by interceptors. Behind the scenes, they add tags and events that relate to their role in an RPC operation. +The following example shows how to add a client span: +@Autowired Tracer tracer; + +// before you send a request, add metadata that describes the operation +span = tracer.newTrace().name("get").type(CLIENT); +span.tag("clnt/finagle.version", "6.36.0"); +span.tag(TraceKeys.HTTP_PATH, "/api"); +span.remoteEndpoint(Endpoint.builder() + .serviceName("backend") + .ipv4(127 << 24 | 1) + .port(8080).build()); + +// when the request is scheduled, start the span +span.start(); + +// if you have callbacks for when data is on the wire, note those events +span.annotate(Constants.WIRE_SEND); +span.annotate(Constants.WIRE_RECV); + +// when the response is complete, finish the span +span.finish(); +
    +One-Way tracing +Sometimes, you need to model an asynchronous operation where there is a +request but no response. In normal RPC tracing, you use span.finish() +to indicate that the response was received. In one-way tracing, you use +span.flush() instead, as you do not expect a response. +The following example shows how a client might model a one-way operation: +@Autowired Tracer tracer; + +// start a new span representing a client request +oneWaySend = tracer.newSpan(parent).kind(Span.Kind.CLIENT); + +// Add the trace context to the request, so it can be propagated in-band +tracing.propagation().injector(Request::addHeader) + .inject(oneWaySend.context(), request); + +// fire off the request asynchronously, totally dropping any response +request.execute(); + +// start the client side and flush instead of finish +oneWaySend.start().flush(); +The following example shows how a server might handle a one-way operation: +@Autowired Tracing tracing; +@Autowired Tracer tracer; + +// pull the context out of the incoming request +extractor = tracing.propagation().extractor(Request::getHeader); + +// convert that context to a span which you can name and add tags to +oneWayReceive = nextSpan(tracer, extractor.extract(request)) + .name("process-request") + .kind(SERVER) + ... add tags etc. + +// start the server side and flush instead of finish +oneWayReceive.start().flush(); + +// you should not modify this span anymore as it is complete. However, +// you can create children to represent follow-up work. +next = tracer.newSpan(oneWayReceive.context()).name("step2").start(); +
    +
    +
     Sampling -In distributed tracing the data volumes can be very high so sampling -can be important (you usually don’t need to export all spans to get a -good picture of what is happening). Spring Cloud Sleuth has a -Sampler strategy that you can implement to take control of the -sampling algorithm. Samplers do not stop span (correlation) ids from -being generated, but they do prevent the tags and events being -attached and exported. By default you get a strategy that continues to -trace if a span is already active, but new ones are always marked as -non-exportable. If all your apps run with this sampler you will see -traces in logs, but not in any remote store. For testing the default -is often enough, and it probably is all you need if you are only using -the logs (e.g. with an ELK aggregator). If you are exporting span data -to Zipkin or Spring Cloud Stream, there is also an AlwaysSampler -that exports everything and a PercentageBasedSampler that samples a -fixed fraction of spans. +Sampling may be employed to reduce the data collected and reported out of process. +When a span is not sampled, it adds no overhead (a noop). +Sampling is an up-front decision, meaning that the decision to report data is made at the first operation in a trace and that decision is propagated downstream. +By default, a global sampler applies a single rate to all traced operations. +Tracer.Builder.sampler controls this setting, and it defaults to tracing every request. +
    +Declarative sampling +Some applications need to sample based on the type or annotations of a java method. +Most users use a framework interceptor to automate this sort of policy. +The following example shows how that might work internally: +@Autowired Tracing tracing; + +// derives a sample rate from an annotation on a java method +DeclarativeSampler<Traced> sampler = DeclarativeSampler.create(Traced::sampleRate); + +@Around("@annotation(traced)") +public Object traceThing(ProceedingJoinPoint pjp, Traced traced) throws Throwable { + Span span = tracing.tracer().newTrace(sampler.sample(traced))... + try { + return pjp.proceed(); + } finally { + span.finish(); + } +} +
    +
    +Custom sampling +Depending on what the operation is, you may want to apply different policies. +For example, you might not want to trace requests to static resources such as images, or you might want to trace all requests to a new api. +Most users use a framework interceptor to automate this sort of policy. +The following example shows how that might work internally: +@Autowired Tracer tracer; + +Span newTrace(Request input) { + SamplingFlags flags = SamplingFlags.NONE; + if (input.url().startsWith("/experimental")) { + flags = SamplingFlags.SAMPLED; + } else if (input.url().startsWith("/static")) { + flags = SamplingFlags.NOT_SAMPLED; + } + return tracer.newTrace(flags); +} +
    +
    +Sampling in Spring Cloud Sleuth +By default Spring Cloud Sleuth sets all spans to non-exportable. +That means that traces appear in logs but not in any remote store. +For testing the default is often enough, and it probably is all you need if you use only the logs (for example, with an ELK aggregator). +If you export span data to Zipkin, there is also an Sampler.ALWAYS_SAMPLE setting that exports everything and a ProbabilityBasedSampler setting that samples a fixed fraction of spans. -the PercentageBasedSampler is the default if you are using -spring-cloud-sleuth-zipkin or spring-cloud-sleuth-stream. You can -configure the exports using spring.sleuth.sampler.percentage. The passed -value needs to be a double from 0.0 to 1.0 so it’s not a percentage. -For backwards compatibility reasons we’re not changing the property name. +The ProbabilityBasedSampler is the default if you use spring-cloud-sleuth-zipkin. +You can configure the exports by setting spring.sleuth.sampler.probability. +The passed value needs to be a double from 0.0 to 1.0. -A sampler can be installed just by creating a bean definition, e.g: +A sampler can be installed by creating a bean definition, as shown in the following example: @Bean public Sampler defaultSampler() { - return new AlwaysSampler(); + return Sampler.ALWAYS_SAMPLE; } -You can set the HTTP header X-B3-Flags to 1 or when doing messaging you can -set spanFlags header to 1. Then the current span will be forced to be exportable -regardless of the sampling decision. +You can set the HTTP header X-B3-Flags to 1, or, when doing messaging, you can set the spanFlags header to 1. +Doing so forces the current span to be exportable regardless of the sampling decision. +
    +     + +Propagation +Propagation is needed to ensure activities originating from the same root are collected together in the same trace. +The most common propagation approach is to copy a trace context from a client by sending an RPC request to a server receiving it. +For example, when a downstream HTTP call is made, its trace context is encoded as request headers and sent along with it, as shown in the following image: + Client Span Server Span +┌──────────────────┐ ┌──────────────────┐ +│ │ │ │ +│ TraceContext │ Http Request Headers │ TraceContext │ +│ ┌──────────────┐ │ ┌───────────────────┐ │ ┌──────────────┐ │ +│ │ TraceId │ │ │ X─B3─TraceId │ │ │ TraceId │ │ +│ │ │ │ │ │ │ │ │ │ +│ │ ParentSpanId │ │ Extract │ X─B3─ParentSpanId │ Inject │ │ ParentSpanId │ │ +│ │ ├─┼─────────>│ ├────────┼>│ │ │ +│ │ SpanId │ │ │ X─B3─SpanId │ │ │ SpanId │ │ +│ │ │ │ │ │ │ │ │ │ +│ │ Sampled │ │ │ X─B3─Sampled │ │ │ Sampled │ │ +│ └──────────────┘ │ └───────────────────┘ │ └──────────────┘ │ +│ │ │ │ +└──────────────────┘ └──────────────────┘ +The names above are from B3 Propagation, which is built-in to Brave and has implementations in many languages and frameworks. +Most users use a framework interceptor to automate propagation. +The next two examples show how that might work for a client and a server. +The following example shows how client-side propagation might work: +@Autowired Tracing tracing; + +// configure a function that injects a trace context into a request +injector = tracing.propagation().injector(Request.Builder::addHeader); + +// before a request is sent, add the current span's context to it +injector.inject(span.context(), request); +The following example shows how server-side propagation might work: +@Autowired Tracing tracing; +@Autowired Tracer tracer; + +// configure a function that extracts the trace context from a request +extractor = tracing.propagation().extractor(Request::getHeader); + +// when a server receives a request, it joins or starts a new trace +span = tracer.nextSpan(extractor.extract(request)); +
    +Propagating extra fields +Sometimes you need to propagate extra fields, such as a request ID or an alternate trace context. +For example, if you are in a Cloud Foundry environment, you might want to pass the request ID, as shown in the following example: +// when you initialize the builder, define the extra field you want to propagate +Tracing.newBuilder().propagationFactory( + ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-vcap-request-id") +); + +// later, you can tag that request ID or use it in log correlation +requestId = ExtraFieldPropagation.get("x-vcap-request-id"); +You may also need to propagate a trace context that you are not using. +For example, you may be in an Amazon Web Services environment but not be reporting data to X-Ray. +To ensure X-Ray can co-exist correctly, pass-through its tracing header, as shown in the following example: +tracingBuilder.propagationFactory( + ExtraFieldPropagation.newFactory(B3Propagation.FACTORY, "x-amzn-trace-id") +); + +In Spring Cloud Sleuth all elements of the tracing builder Tracing.newBuilder() +are defined as beans. So if you want to pass a custom PropagationFactory, it’s enough +for you to create a bean of that type and we will set it in the Tracing bean. + +
    +Prefixed fields +If they follow a common pattern, you can also prefix fields. +The following example shows how to propagate x-vcap-request-id the field as-is but send the country-code and user-id fields on the wire as x-baggage-country-code and x-baggage-user-id, respectively: +Tracing.newBuilder().propagationFactory( + ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY) + .addField("x-vcap-request-id") + .addPrefixedFields("baggage-", Arrays.asList("country-code", "user-id")) + .build() +); +Later, you can call the following code to affect the country code of the current trace context: +ExtraFieldPropagation.set("country-code", "FO"); +String countryCode = ExtraFieldPropagation.get("country-code"); +Alternatively, if you have a reference to a trace context, you can use it explicitly, as shown in the following example: +ExtraFieldPropagation.set(span.context(), "country-code", "FO"); +String countryCode = ExtraFieldPropagation.get(span.context(), "country-code"); + +A difference from previous versions of Sleuth is that, with Brave, you must pass the list of baggage keys. +There are two properties to achieve this. +With the spring.sleuth.baggage-keys, you set keys that get prefixed with baggage- for HTTP calls and baggage_ for messaging. +You can also use the spring.sleuth.propagation-keys property to pass a list of prefixed keys that are whitelisted without any prefix. + +
    +
    +Extracting a Propagated Context +The TraceContext.Extractor<C> reads trace identifiers and sampling status from an incoming request or message. +The carrier is usually a request object or headers. +This utility is used in standard instrumentation (such as HttpServerHandler`) but can also be used for custom RPC or messaging code. +TraceContextOrSamplingFlags is usually used only with Tracer.nextSpan(extracted), unless you are +sharing span IDs between a client and a server. +
    +
    +Sharing span IDs between Client and Server +A normal instrumentation pattern is to create a span representing the server side of an RPC. +Extractor.extract might return a complete trace context when applied to an incoming client request. +Tracer.joinSpan attempts to continue this trace, using the same span ID if supported or creating a child span +if not. When the span ID is shared, the reported data includes a flag saying so. +The following image shows an example of B3 propagation: + ┌───────────────────┐ ┌───────────────────┐ + Incoming Headers │ TraceContext │ │ TraceContext │ +┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │ +│ X─B3-TraceId │─────────┼─┼> TraceId │ │──────┼─┼> TraceId │ │ +│ │ │ │ │ │ │ │ │ │ +│ X─B3-ParentSpanId │─────────┼─┼> ParentSpanId │ │──────┼─┼> ParentSpanId │ │ +│ │ │ │ │ │ │ │ │ │ +│ X─B3-SpanId │─────────┼─┼> SpanId │ │──────┼─┼> SpanId │ │ +└───────────────────┘ │ │ │ │ │ │ │ │ + │ │ │ │ │ │ Shared: true │ │ + │ └───────────────┘ │ │ └───────────────┘ │ + └───────────────────┘ └───────────────────┘ +Some propagation systems forward only the parent span ID, detected when Propagation.Factory.supportsJoin() == false. +In this case, a new span ID is always provisioned, and the incoming context determines the parent ID. +The following image shows an example of AWS propagation: + ┌───────────────────┐ ┌───────────────────┐ + x-amzn-trace-id │ TraceContext │ │ TraceContext │ +┌───────────────────┐(extract)│ ┌───────────────┐ │(join)│ ┌───────────────┐ │ +│ Root │─────────┼─┼> TraceId │ │──────┼─┼> TraceId │ │ +│ │ │ │ │ │ │ │ │ │ +│ Parent │─────────┼─┼> SpanId │ │──────┼─┼> ParentSpanId │ │ +└───────────────────┘ │ └───────────────┘ │ │ │ │ │ + └───────────────────┘ │ │ SpanId: New │ │ + │ └───────────────┘ │ + └───────────────────┘ +Note: Some span reporters do not support sharing span IDs. +For example, if you set Tracing.Builder.spanReporter(amazonXrayOrGoogleStackdrive), you should disable join by setting Tracing.Builder.supportsJoin(false). +Doing so forces a new child span on Tracer.joinSpan(). +
    +
    +Implementing Propagation +TraceContext.Extractor<C> is implemented by a Propagation.Factory plugin. +Internally, this code creates the union type, TraceContextOrSamplingFlags, with one of the following: +* TraceContext if trace and span IDs were present. +* TraceIdContext if a trace ID was present but span IDs were not present. +* SamplingFlags if no identifiers were present. +Some Propagation implementations carry extra data from the point of extraction (for example, reading incoming headers) to injection (for example, writing outgoing headers). +For example, it might carry a request ID. +When implementations have extra data, they handle it as follows: +* If a TraceContext were extracted, add the extra data as TraceContext.extra(). +* Otherwise, add it as TraceContextOrSamplingFlags.extra(), which Tracer.nextSpan handles. +
    +
    +     + +Current Tracing Component +Brave supports a "current tracing component" concept, which should only be used when you have no other way to get a reference. +This was made for JDBC connections, as they often initialize prior to the tracing component. +The most recent tracing component instantiated is available through Tracing.current(). +You can also use Tracing.currentTracer() to get only the tracer. +If you use either of these methods, do not cache the result. +Instead, look them up each time you need them. + + +Current Span +Brave supports a "current span" concept which represents the in-flight operation. +You can use Tracer.currentSpan() to add custom tags to a span and Tracer.nextSpan() to create a child of whatever is in-flight. + +In Sleuth, you can autowire the Tracer bean to retrieve the current span via +tracer.currentSpan() method. To retrieve the current context just call +tracer.currentSpan().context(). To get the current trace id as String +you can use the traceIdString() method like this: tracer.currentSpan().context().traceIdString(). + +
    +Setting a span in scope manually +When writing new instrumentation, it is important to place a span you created in scope as the current span. +Not only does doing so let users access it with Tracer.currentSpan(), but it also allows customizations such as SLF4J MDC to see the current trace IDs. +Tracer.withSpanInScope(Span) facilitates this and is most conveniently employed by using the try-with-resources idiom. +Whenever external code might be invoked (such as proceeding an interceptor or otherwise), place the span in scope, as shown in the following example: +@Autowired Tracer tracer; + +try (SpanInScope ws = tracer.withSpanInScope(span)) { + return inboundRequest.invoke(); +} finally { // note the scope is independent of the span + span.finish(); +} +In edge cases, you may need to clear the current span temporarily (for example, launching a task that should not be associated with the current request). To do tso, pass null to withSpanInScope, as shown in the following example: +@Autowired Tracer tracer; + +try (SpanInScope cleared = tracer.withSpanInScope(null)) { + startBackgroundThread(); +} +
     Instrumentation -Spring Cloud Sleuth instruments all your Spring application -automatically, so you shouldn’t have to do anything to activate -it. The instrumentation is added using a variety of technologies -according to the stack that is available, e.g. for a servlet web -application we use a Filter, and for Spring Integration we use -ChannelInterceptors. -You can customize the keys used in span tags. To limit the volume of -span data, by default an HTTP request will be tagged only with a -handful of metadata like the status code, host and URL. You can add -request headers by configuring spring.sleuth.keys.http.headers (a -list of header names). +Spring Cloud Sleuth automatically instruments all your Spring applications, so you should not have to do anything to activate it. +The instrumentation is added by using a variety of technologies according to the stack that is available. For example, for a servlet web application, we use a Filter, and, for Spring Integration, we use ChannelInterceptors. +You can customize the keys used in span tags. +To limit the volume of span data, an HTTP request is, by default, tagged only with a handful of metadata, such as the status code, the host, and the URL. +You can add request headers by configuring spring.sleuth.keys.http.headers (a list of header names). -Remember that tags are only collected and exported if there is a -Sampler that allows it (by default there is not, so there is no -danger of accidentally collecting too much data without configuring -something). - - -Currently the instrumentation in Spring Cloud Sleuth is eager - it means that -we’re actively trying to pass the tracing context between threads. Also timing events -are captured even when sleuth isn’t exporting data to a tracing system. -This approach may change in the future towards being lazy on this matter. +Tags are collected and exported only if there is a Sampler that allows it. By default, there is no such Sampler, to ensure that there is no danger of accidentally collecting too much data without configuring something). Span lifecycle -You can do the following operations on the Span by means of org.springframework.cloud.sleuth.Tracer interface: +You can do the following operations on the Span by means of brave.Tracer: -start - when you start a span its name is assigned and start timestamp is recorded. +start: When you start a span, its name is assigned and the start timestamp is recorded. -close - the span gets finished (the end time of the span is recorded) and if -the span is exportable then it will be eligible for collection to Zipkin. -The span is also removed from the current thread. +close: The span gets finished (the end time of the span is recorded) and, if the span is sampled, it is eligible for collection (for example, to Zipkin). -continue - a new instance of span will be created whereas it will be a copy of the -one that it continues. +continue: A new instance of span is created. +It is a copy of the one that it continues. -detach - the span doesn’t get stopped or closed. It only gets removed from the current thread. +detach: The span does not get stopped or closed. +It only gets removed from the current thread. -create with explicit parent - you can create a new span and set an explicit parent to it +create with explicit parent: You can create a new span and set an explicit parent for it. -Spring creates the instance of Tracer for you. In order to use it all you need is to just autowire it. +Spring Cloud Sleuth creates an instance of Tracer for you. In order to use it, you can autowire it. -
    -Creating and closing spans -You can manually create spans by using the Tracer interface. +
    +Creating and finishing spans +You can manually create spans by using the Tracer, as shown in the following example: // Start a span. If there was a span present in this thread it will become // the `newSpan`'s parent. -Span newSpan = this.tracer.createSpan("calculateTax"); -try { +Span newSpan = this.tracer.nextSpan().name("calculateTax"); +try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(newSpan.start())) { // ... // You can tag a span - this.tracer.addTag("taxValue", taxValue); + newSpan.tag("taxValue", taxValue); // ... // You can log an event on a span - newSpan.logEvent("taxCalculated"); + newSpan.annotate("taxCalculated"); } finally { - // Once done remember to close the span. This will allow collecting + // Once done remember to finish the span. This will allow collecting // the span to send it to Zipkin - this.tracer.close(newSpan); + newSpan.finish(); } -In this example we could see how to create a new instance of span. Assuming that there already -was a span present in this thread then it would become the parent of that span. +In the preceding example, we could see how to create a new instance of the span. +If there is already a span in this thread, it becomes the parent of the new span. -Always clean after you create a span! Don’t forget to close a span if you want to send it to Zipkin. +Always clean after you create a span. Also, always finish any span that you want to send to Zipkin. -If your span contains a name greater than 50 chars, then that name will -be truncated to 50 chars. Your names have to be explicit and concrete. Big names lead to -latency issues and sometimes even thrown exceptions. +If your span contains a name greater than 50 chars, that name is truncated to 50 chars. +Your names have to be explicit and concrete. Big names lead to latency issues and sometimes even exceptions.
    -Continuing spans -Sometimes you don’t want to create a new span but you want to continue one. Example of such a -situation might be (of course it all depends on the use-case): +Continuing Spans +Sometimes, you do not want to create a new span but you want to continue one. An example of such a +situation might be as follows: -AOP - If there was already a span created before an aspect was reached then you might not want to create a new span. +AOP: If there was already a span created before an aspect was reached, you might not want to create a new span. -Hystrix - executing a Hystrix command is most likely a logical part of the current processing. It’s in fact -only a technical implementation detail that you wouldn’t necessarily want to reflect in tracing as a separate being. +Hystrix: Executing a Hystrix command is most likely a logical part of the current processing. +It is in fact merely a technical implementation detail that you would not necessarily want to reflect in tracing as a separate being. -The continued instance of span is equal to the one that it continues: -Span continuedSpan = this.tracer.continueSpan(spanToContinue); -assertThat(continuedSpan).isEqualTo(spanToContinue); -To continue a span you can use the Tracer interface. +To continue a span, you can use brave.Tracer, as shown in the following example: // let's assume that we're in a thread Y and we've received // the `initialSpan` from thread X -Span continuedSpan = this.tracer.continueSpan(initialSpan); +Span continuedSpan = this.tracer.toSpan(newSpan.context()); try { // ... // You can tag a span - this.tracer.addTag("taxValue", taxValue); + continuedSpan.tag("taxValue", taxValue); // ... // You can log an event on a span - continuedSpan.logEvent("taxCalculated"); + continuedSpan.annotate("taxCalculated"); } finally { - // Once done remember to detach the span. That way you'll - // safely remove it from the current thread without closing it - this.tracer.detach(continuedSpan); + // Once done remember to flush the span. That means that + // it will get reported but the span itself is not yet finished + continuedSpan.flush(); } - -Always clean after you create a span! Don’t forget to detach a span if some work was done started in one - thread (e.g. thread X) and it’s waiting for other threads (e.g. Y, Z) to finish. - Then the spans in the threads Y, Z should be detached at the end of their work. When the results are collected - the span in thread X should be closed. -
    -Creating spans with an explicit parent -There is a possibility that you want to start a new span and provide an explicit parent of that span. -Let’s assume that the parent of a span is in one thread and you want to start a new span in another thread. The -startSpan method of the Tracer interface is the method you are looking for. +Creating a Span with an explicit Parent +You might want to start a new span and provide an explicit parent of that span. +Assume that the parent of a span is in one thread and you want to start a new span in another thread. +In Brave, whenever you call nextSpan(), it creates a span in reference to the span that is currently in scope. +You can put the span in scope and then call nextSpan(), as shown in the following example: // let's assume that we're in a thread Y and we've received // the `initialSpan` from thread X. `initialSpan` will be the parent // of the `newSpan` -Span newSpan = this.tracer.createSpan("calculateCommission", initialSpan); -try { +Span newSpan = null; +try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) { + newSpan = this.tracer.nextSpan().name("calculateCommission"); // ... // You can tag a span - this.tracer.addTag("commissionValue", commissionValue); + newSpan.tag("commissionValue", commissionValue); // ... // You can log an event on a span - newSpan.logEvent("commissionCalculated"); + newSpan.annotate("commissionCalculated"); } finally { - // Once done remember to close the span. This will allow collecting + // Once done remember to finish the span. This will allow collecting // the span to send it to Zipkin. The tags and events set on the // newSpan will not be present on the parent - this.tracer.close(newSpan); + if (newSpan != null) { + newSpan.finish(); + } } -After having created such a span remember to close it. Otherwise you will see a lot of warnings in your logs - related to the fact that you have a span present in the current thread other than the one you’re trying to close. - What’s worse your spans won’t get closed properly thus will not get collected to Zipkin. +After creating such a span, you must finish it. Otherwise it is not reported (for example, to Zipkin).
    Naming spans -Picking a span name is not a trivial task. Span name should depict an operation name. The name should -be low cardinality (e.g. not include identifiers). -Since there is a lot of instrumentation going on some of the span names will be -artificial like: +Picking a span name is not a trivial task. A span name should depict an operation name. +The name should be low cardinality, so it should not include identifiers. +Since there is a lot of instrumentation going on, some span names are artificial: -controller-method-name when received by a Controller with a method name conrollerMethodName +controller-method-name when received by a Controller with a method name of controllerMethodName -async for asynchronous operations done via wrapped Callable and Runnable. +async for asynchronous operations done with wrapped Callable and Runnable interfaces. -@Scheduled annotated methods will return the simple name of the class. +Methods annotated with @Scheduled return the simple name of the class. -Fortunately, for the asynchronous processing you can provide explicit naming. -
    -@SpanName annotation -You can name the span explicitly via the @SpanName annotation. +Fortunately, for asynchronous processing, you can provide explicit naming. +
    +<literal>@SpanName</literal> Annotation +You can name the span explicitly by using the @SpanName annotation, as shown in the following example: @SpanName("calculateTax") class TaxCountingRunnable implements Runnable { @@ -874,20 +1282,21 @@ class TaxCountingRunnable implements Runnable { // perform logic } } -In this case, when processed in the following manner: -Runnable runnable = new TraceRunnable(tracer, spanNamer, new TaxCountingRunnable()); +In this case, when processed in the following manner, the span is named calculateTax: +Runnable runnable = new TraceRunnable(tracing, spanNamer, + new TaxCountingRunnable()); Future<?> future = executorService.submit(runnable); // ... some additional logic ... future.get(); -The span will be named calculateTax.
    -
    -toString() method -It’s pretty rare to create separate classes for Runnable or Callable. Typically one creates an anonymous -instance of those classes. You can’t annotate such classes thus to override that, if there is no @SpanName annotation present, -we’re checking if the class has a custom implementation of the toString() method. -So executing such code: -Runnable runnable = new TraceRunnable(tracer, spanNamer, new Runnable() { +
    +<literal>toString()</literal> method +It is pretty rare to create separate classes for Runnable or Callable. +Typically, one creates an anonymous instance of those classes. +You cannot annotate such classes. +To overcome that limitation, if there is no @SpanName annotation present, we check whether the class has a custom implementation of the toString() method. +Running such code leads to creating a span named calculateTax, as shown in the following example: +Runnable runnable = new TraceRunnable(tracing, spanNamer, new Runnable() { @Override public void run() { // perform logic } @@ -899,464 +1308,301 @@ we’re checking if the class has a custom implementation of the to Future<?> future = executorService.submit(runnable); // ... some additional logic ... future.get(); -will lead in creating a span named calculateTax.
    -Managing spans with annotations +Managing Spans with Annotations +You can manage spans with a variety of annotations.
    Rationale -The main arguments for this features are +There are a number of good reasons to manage spans with annotations, including: -api-agnostic means to collaborate with a span - - -use of annotations allows users to add to a span with no library dependency on a span api. -This allows Sleuth to change its core api less impact to user code. - - +API-agnostic means to collaborate with a span. Use of annotations lets users add to a span with no library dependency on a span api. +Doing so lets Sleuth change its core API to create less impact to user code. -reduced surface area for basic span operations. - - -without this feature one has to use the span api, which has lifecycle commands that -could be used incorrectly. By only exposing scope, tag and log functionality, users can -collaborate without accidentally breaking span lifecycle. - - +Reduced surface area for basic span operations. Without this feature, you must use the span api, which has lifecycle commands that could be used incorrectly. +By only exposing scope, tag, and log functionality, you can collaborate without accidentally breaking span lifecycle. -collaboration with runtime generated code - - -with libraries such as Spring Data / Feign the implementations of interfaces are generated -at runtime thus span wrapping of objects was tedious. Now you can provide annotations - over interfaces and arguments of those interfaces - - +Collaboration with runtime generated code. With libraries such as Spring Data and Feign, the implementations of interfaces are generated at runtime. +Consequently, span wrapping of objects was tedious. +Now you can provide annotations over interfaces and the arguments of those interfaces.
    -Creating new spans -If you really don’t want to take care of creating local spans manually you can profit from the -@NewSpan annotation. Also we give you the @SpanTag annotation to add tags in an automated -fashion. -Let’s look at some examples of usage. +Creating New Spans +If you do not want to create local spans manually, you can use the @NewSpan annotation. +Also, we provide the @SpanTag annotation to add tags in an automated fashion. +Now we can consider some examples of usage. @NewSpan void testMethod(); -Annotating the method without any parameter will lead to a creation of a new span whose name -will be equal to annotated method name. +Annotating the method without any parameter leads to creating a new span whose name equals the annotated method name. @NewSpan("customNameOnTestMethod4") void testMethod4(); -If you provide the value in the annotation (either directly or via the name parameter) then -the created span will have the name as the provided value. +If you provide the value in the annotation (either directly or by setting the name parameter), the created span has the provided value as the name. // method declaration @NewSpan(name = "customNameOnTestMethod5") void testMethod5(@SpanTag("testTag") String param); // and method execution this.testBean.testMethod5("test"); -You can combine both the name and a tag. Let’s focus on the latter. In this case whatever the value of -the annotated method’s parameter runtime value will be - that will be the value of the tag. In our sample -the tag key will be testTag and the tag value will be test. +You can combine both the name and a tag. Let’s focus on the latter. +In this case, the value of the annotated method’s parameter runtime value becomes the value of the tag. +In our sample, the tag key is testTag, and the tag value is test. @NewSpan(name = "customNameOnTestMethod3") @Override public void testMethod3() { } -You can place the @NewSpan annotation on both the class and an interface. If you override the -interface’s method and provide a different value of the @NewSpan annotation then the most -concrete one wins (in this case customNameOnTestMethod3 will be set). +You can place the @NewSpan annotation on both the class and an interface. +If you override the interface’s method and provide a different value for the @NewSpan annotation, the most +concrete one wins (in this case customNameOnTestMethod3 is set).
    -Continuing spans -If you want to just add tags and annotations to an existing span it’s enough -to use the @ContinueSpan annotation as presented below. Note that in contrast -with the @NewSpan annotation you can also add logs via the log parameter: +Continuing Spans +If you want to add tags and annotations to an existing span, you can use the @ContinueSpan annotation, as shown in the following example: // method declaration @ContinueSpan(log = "testMethod11") void testMethod11(@SpanTag("testTag11") String param); // method execution -this.testBean.testMethod11("test"); -That way the span will get continued and: +this.testBean.testMethod11("test"); +this.testBean.testMethod13(); +(Note that, in contrast with the @NewSpan annotation ,you can also add logs with the log parameter.) +That way, the span gets continued and: -logs with name testMethod11.before and testMethod11.after will be created +Log entries named testMethod11.before and testMethod11.after are created. -if an exception will be thrown a log testMethod11.afterFailure will also be created +If an exception is thrown, a log entry named testMethod11.afterFailure is also created. -tag with key testTag11 and value test will be created +A tag with a key of testTag11 and a value of test is created.
    -
    -More advanced tag setting +
    +Advanced Tag Setting There are 3 different ways to add tags to a span. All of them are controlled by the SpanTag annotation. -Precedence is: - +The precedence is as follows: + -try with the bean of TagValueResolver type and provided name +Try with a bean of TagValueResolver type and a provided name. -if one hasn’t provided the bean name, try to evaluate an expression. We’re searching for a TagValueExpressionResolver bean. -The default implementation uses SPEL expression resolution. +If the bean name has not been provided, try to evaluate an expression. +We search for a TagValueExpressionResolver bean. +The default implementation uses SPEL expression resolution. +IMPORTANT You can only reference properties from the SPEL expression. Method execution is not allowed due to security constraints. -if one hasn’t provided any expression to evaluate just return a toString() value of the parameter +If we do not find any expression to evaluate, return the toString() value of the parameter. - +
    Custom extractor -The value of the tag for following method will be computed by an implementation of TagValueResolver interface. +The value of the tag for the following method is computed by an implementation of TagValueResolver interface. Its class name has to be passed as the value of the resolver attribute. -Having such an annotated method: +Consider the following annotated method: @NewSpan public void getAnnotationForTagValueResolver(@SpanTag(key = "test", resolver = TagValueResolver.class) String test) { } -and such a TagValueResolver bean implementation +Now further consider the following TagValueResolver bean implementation: @Bean(name = "myCustomTagValueResolver") public TagValueResolver tagValueResolver() { return parameter -> "Value from myCustomTagValueResolver"; } -Will lead to setting of a tag value equal to Value from myCustomTagValueResolver. +The two preceding examples lead to setting a tag value equal to Value from myCustomTagValueResolver.
    -
    -Resolving expressions for value -Having such an annotated method: +
    +Resolving Expressions for a Value +Consider the following annotated method: @NewSpan -public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "length() + ' characters'") String test) { +public void getAnnotationForTagValueExpression(@SpanTag(key = "test", expression = "'hello' + ' characters'") String test) { } -and no custom implementation of a TagValueExpressionResolver will lead to evaluation of the SPEL expression and a tag with value 4 characters will be set on the span. -If you want to use some other expression resolution mechanism you can create your own implementation -of the bean. +No custom implementation of a TagValueExpressionResolver leads to evaluation of the SPEL expression, and a tag with a value of 4 characters is set on the span. +If you want to use some other expression resolution mechanism, you can create your own implementation of the bean.
    -
    -Using toString method -Having such an annotated method: +
    +Using the <literal>toString()</literal> method +Consider the following annotated method: @NewSpan public void getAnnotationForArgumentToString(@SpanTag("test") Long param) { } -if executed with a value of 15 will lead to setting of a tag with a String value of "15". +Running the preceding method with a value of 15 leads to setting a tag with a String value of "15".
    Customizations -Thanks to the SpanInjector and SpanExtractor you can customize the way spans -are created and propagated. -There are currently two built-in ways to pass tracing information between processes: - - -via Spring Integration - - -via HTTP - - -Span ids are extracted from Zipkin-compatible (B3) headers (either Message -or HTTP headers), to start or join an existing trace. Trace information is -injected into any outbound requests so the next hop can extract them. -The key change in comparison to the previous versions of Sleuth is that Sleuth is implementing -the Open Tracing’s TextMap notion. In Sleuth it’s called SpanTextMap. Basically the idea -is that any means of communication (e.g. message, http request, etc.) can be abstracted via -a SpanTextMap. This abstraction defines how one can insert data into the carrier and -how to retrieve it from there. Thanks to this if you want to instrument a new HTTP library -that uses a FooRequest as a mean of sending HTTP requests then you have to create an -implementation of a SpanTextMap that delegates calls to FooRequest in terms of retrieval -and insertion of HTTP headers. -
    -Spring Integration -For Spring Integration there are 2 interfaces responsible for creation of a Span from a Message. -These are: - - -MessagingSpanTextMapExtractor - - -MessagingSpanTextMapInjector - - -You can override them by providing your own implementation. -
    HTTP -For HTTP there are 2 interfaces responsible for creation of a Span from a Message. -These are: - - -HttpSpanExtractor - - -HttpSpanInjector - - -You can override them by providing your own implementation. +If a customization of client / server parsing of the HTTP related spans is required, +just register a bean of type brave.http.HttpClientParser or +brave.http.HttpServerParser. If client /server sampling is required, just +register a bean of type brave.http.HttpSampler and name the bean + sleuthClientSampler for client sampler and sleuthServerSampler for server sampler. + For your convenience the @ClientSampler and @ServerSampler + annotations can be used to inject the proper beans or to + reference the bean names via their static String NAME fields. +Check out Brave’s code to see an example of how to make a path-based sampler +https://github.com/openzipkin/brave/tree/master/instrumentation/http#sampling-policy +If you want to completely rewrite the HttpTracing bean you can use the SkipPatternProvider +interface to retrieve the URL Pattern for spans that should be not sampled. Below you can see +an example of usage of SkipPatternProvider inside a server side, HttpSampler. +@Configuration +class Config { + @Bean(name = ServerSampler.NAME) + HttpSampler myHttpSampler(SkipPatternProvider provider) { + Pattern pattern = provider.skipPattern(); + return new HttpSampler() { + + @Override public <Req> Boolean trySample(HttpAdapter<Req, ?> adapter, Req request) { + String url = adapter.path(request); + boolean shouldSkip = pattern.matcher(url).matches(); + if (shouldSkip) { + return false; + } + return null; + } + }; + } +}
    -
    -Example -Let’s assume that instead of the standard Zipkin compatible tracing HTTP header names -you have - - -for trace id - correlationId - - -for span id - mySpanId - - -This is a an example of a SpanExtractor -static class CustomHttpSpanExtractor implements HttpSpanExtractor { - - @Override public Span joinTrace(SpanTextMap carrier) { - Map<String, String> map = TextMapUtil.asMap(carrier); - long traceId = Span.hexToId(map.get("correlationid")); - long spanId = Span.hexToId(map.get("myspanid")); - // extract all necessary headers - Span.SpanBuilder builder = Span.builder().traceId(traceId).spanId(spanId); - // build rest of the Span - return builder.build(); - } -} - -static class CustomHttpSpanInjector implements HttpSpanInjector { - - @Override - public void inject(Span span, SpanTextMap carrier) { - carrier.put("correlationId", span.traceIdString()); - carrier.put("mySpanId", Span.idToHex(span.getSpanId())); - } -} -And you could register it like this: -@Bean -HttpSpanInjector customHttpSpanInjector() { - return new CustomHttpSpanInjector(); -} - -@Bean -HttpSpanExtractor customHttpSpanExtractor() { - return new CustomHttpSpanExtractor(); -} -Spring Cloud Sleuth does not add trace/span related headers to the Http Response for security reasons. If you need the headers then a custom SpanInjector -that injects the headers into the Http Response and a Servlet filter which makes use of this can be added the following way: -static class CustomHttpServletResponseSpanInjector extends ZipkinHttpSpanInjector { - - @Override - public void inject(Span span, SpanTextMap carrier) { - super.inject(span, carrier); - carrier.put(Span.TRACE_ID_NAME, span.traceIdString()); - carrier.put(Span.SPAN_ID_NAME, Span.idToHex(span.getSpanId())); - } -} - -static class HttpResponseInjectingTraceFilter extends GenericFilterBean { +
    +<literal>TracingFilter</literal> +You can also modify the behavior of the TracingFilter, which is the component that is responsible for processing the input HTTP request and adding tags basing on the HTTP response. +You can customize the tags or modify the response headers by registering your own instance of the TracingFilter bean. +In the following example, we register the TracingFilter bean, add the ZIPKIN-TRACE-ID response header containing the current Span’s trace id, and add a tag with key custom and a value tag to the span. +@Component +@Order(TraceWebServletAutoConfiguration.TRACING_FILTER_ORDER + 1) +class MyFilter extends GenericFilterBean { private final Tracer tracer; - private final HttpSpanInjector spanInjector; - public HttpResponseInjectingTraceFilter(Tracer tracer, HttpSpanInjector spanInjector) { + MyFilter(Tracer tracer) { this.tracer = tracer; - this.spanInjector = spanInjector; } - @Override - public void doFilter(ServletRequest request, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException { - HttpServletResponse response = (HttpServletResponse) servletResponse; - Span currentSpan = this.tracer.getCurrentSpan(); - this.spanInjector.inject(currentSpan, new HttpServletResponseTextMap(response)); - filterChain.doFilter(request, response); - } - - class HttpServletResponseTextMap implements SpanTextMap { - - private final HttpServletResponse delegate; - - HttpServletResponseTextMap(HttpServletResponse delegate) { - this.delegate = delegate; - } - - @Override - public Iterator<Map.Entry<String, String>> iterator() { - Map<String, String> map = new HashMap<>(); - for (String header : this.delegate.getHeaderNames()) { - map.put(header, this.delegate.getHeader(header)); - } - return map.entrySet().iterator(); - } - - @Override - public void put(String key, String value) { - this.delegate.addHeader(key, value); - } - } -} -And you could register them like this: -@Bean HttpSpanInjector customHttpServletResponseSpanInjector() { - return new CustomHttpServletResponseSpanInjector(); -} - -@Bean -HttpResponseInjectingTraceFilter responseInjectingTraceFilter(Tracer tracer) { - return new HttpResponseInjectingTraceFilter(tracer, customHttpServletResponseSpanInjector()); -} -
    -
    -TraceFilter -You can also modify the behaviour of the TraceFilter - the component that is responsible -for processing the input HTTP request and adding tags basing on the HTTP response. You can customize -the tags, or modify the response headers by registering your own instance of the TraceFilter bean. -In the following example we will register the TraceFilter bean and we will add the -ZIPKIN-TRACE-ID response header containing the current Span’s trace id. Also we will -add to the Span a tag with key custom and a value tag. -@Bean -TraceFilter myTraceFilter(BeanFactory beanFactory, final Tracer tracer) { - return new TraceFilter(beanFactory) { - @Override protected void addResponseTags(HttpServletResponse response, - Throwable e) { - // execute the default behaviour - super.addResponseTags(response, e); - // for readability we're returning trace id in a hex form - response.addHeader("ZIPKIN-TRACE-ID", - Span.idToHex(tracer.getCurrentSpan().getTraceId())); - // we can also add some custom tags - tracer.addTag("custom", "tag"); + @Override public void doFilter(ServletRequest request, ServletResponse response, + FilterChain chain) throws IOException, ServletException { + Span currentSpan = this.tracer.currentSpan(); + if (currentSpan == null) { + chain.doFilter(request, response); + return; } - }; + // for readability we're returning trace id in a hex form + ((HttpServletResponse) response) + .addHeader("ZIPKIN-TRACE-ID", + currentSpan.context().traceIdString()); + // we can also add some custom tags + currentSpan.tag("custom", "tag"); + chain.doFilter(request, response); + } }
    -
    -Custom SA tag in Zipkin -Sometimes you want to create a manual Span that will wrap a call to an external service which is not instrumented. -What you can do is to create a span with the peer.service tag that will contain a value of the service that you want to call. -Below you can see an example of a call to Redis that is wrapped in such a span. -Unresolved directive in spring-cloud-sleuth.adoc - include::../../../..//spring-cloud-sleuth-zipkin-legacy/src/test/java/org/springframework/cloud/sleuth/zipkin/HttpZipkinSpanReporterTest.java[tags=service_name,indent=0] - -Remember not to add both peer.service tag and the SA tag! You have to add only peer.service. - -
    Custom service name -By default Sleuth assumes that when you send a span to Zipkin, you want the span’s service name - to be equal to spring.application.name value. That’s not always the case though. There - are situations in which you want to explicitly provide a different service name for all spans coming - from your application. To achieve that it’s enough to just pass the following property - to your application to override that value (example for foo service name): -spring.zipkin.service.name: foo +By default, Sleuth assumes that, when you send a span to Zipkin, you want the span’s service name to be equal to the value of the spring.application.name property. +That is not always the case, though. +There are situations in which you want to explicitly provide a different service name for all spans coming from your application. +To achieve that, you can pass the following property to your application to override that value (the example is for a service named myService): +spring.zipkin.service.name: myService
    -Customization of reported spans -Before reporting spans to e.g. Zipkin you can be interested in modifying that span in some way. - You can achieve that by using the SpanAdjuster interface. -Example of usage: -In Sleuth we’re generating spans with a fixed name. Some users want to modify the name depending on values -of tags. Implementation of the SpanAdjuster interface can be used to alter that name. Example: -@Bean -SpanAdjuster customSpanAdjuster() { - return span -> span.toBuilder().name(scrub(span.getName())).build(); +Customization of Reported Spans +Before reporting spans (for example, to Zipkin) you may want to modify that span in some way. +You can do so by using the SpanAdjuster interface. +In Sleuth, we generate spans with a fixed name. +Some users want to modify the name depending on values of tags. +You can implement the SpanAdjuster interface to alter that name. +The following example shows how to register two beans that implement SpanAdjuster: +@Bean SpanAdjuster adjusterOne() { + return span -> span.toBuilder().name("foo").build(); +} + +@Bean SpanAdjuster adjusterTwo() { + return span -> span.toBuilder().name(span.name() + " bar").build(); } -This will lead in changing the name of the reported span just before it gets sent to Zipkin. - -Your SpanReporter should inject the SpanAdjuster and - allow span manipulation before the actual reporting is done. - +The preceding example results in changing the name of the reported span to foo bar, just before it gets reported (for example, to Zipkin).
    -Host locator -In order to define the host that is corresponding to a particular span we need to resolve the host name -and port. The default approach is to take it from server properties. If those for some reason are not set -then we’re trying to retrieve the host name from the network interfaces. -If you have the discovery client enabled and prefer to retrieve the host address from the registered -instance in a service registry then you have to set the property (it’s applicable for both HTTP and -Stream based span reporting). +Host Locator + +This section is about defining host from service discovery. +It is NOT about finding Zipkin through service discovery. + +To define the host that corresponds to a particular span, we need to resolve the host name and port. +The default approach is to take these values from server properties. +If those are not set, we try to retrieve the host name from the network interfaces. +If you have the discovery client enabled and prefer to retrieve the host address from the registered instance in a service registry, you have to set the spring.zipkin.locator.discovery.enabled property (it is applicable for both HTTP-based and Stream-based span reporting), as follows: spring.zipkin.locator.discovery.enabled: true
    -Sending spans to Zipkin -By default if you add spring-cloud-starter-zipkin as a dependency to your project, -when the span is closed, it will be sent to Zipkin over HTTP. The communication -is asynchronous. You can configure the URL by setting the spring.zipkin.baseUrl -property as follows: +Sending Spans to Zipkin +By default, if you add spring-cloud-starter-zipkin as a dependency to your project, when the span is closed, it is sent to Zipkin over HTTP. +The communication is asynchronous. +You can configure the URL by setting the spring.zipkin.baseUrl property, as follows: spring.zipkin.baseUrl: http://192.168.99.100:9411/ -If you want to find Zipkin via service discovery it’s enough to pass the -Zipkin’s service id inside the URL (example for zipkinserver service id) +If you want to find Zipkin through service discovery, you can pass the Zipkin’s service ID inside the URL, as shown in the following example for zipkinserver service ID: spring.zipkin.baseUrl: http://zipkinserver/ - - -Span Data as Messages - -The suggested approach is to use the Zipkin’s -native support for message based span sending. Starting from -Edgware Zipkin Stream server is deprecated and in Finchley -it got removed. - -You can accumulate and send span data over -Spring Cloud Stream by -including the spring-cloud-sleuth-stream jar as a dependency, and -adding a Channel Binder implementation -(e.g. spring-cloud-starter-stream-rabbit for RabbitMQ or -spring-cloud-starter-stream-kafka for Kafka). This will -automatically turn your app into a producer of messages with payload -type Spans. -
    -Zipkin Consumer -Please refer to the Dalston Documentaion -on how to create a Stream Zipkin server. That approach has been -deprecated in Edgware and removed in Finchley release. -
    -
    -Custom Consumer -A custom consumer can also easily be implemented using -spring-cloud-sleuth-stream and binding to the SleuthSink. Example: -@EnableBinding(SleuthSink.class) -@SpringBootApplication(exclude = SleuthStreamAutoConfiguration.class) -@MessageEndpoint -public class Consumer { - - @ServiceActivator(inputChannel = SleuthSink.INPUT) - public void sink(Spans input) throws Exception { - // ... process spans - } -} - -the sample consumer application above explicitly excludes -SleuthStreamAutoConfiguration so it doesn’t send messages to itself, -but this is optional (you might actually want to trace requests into -the consumer app). - -In order to customize the polling mechanism you can create a bean of PollerMetadata type -with name equal to StreamSpanReporter.POLLER. Here you can find an example of such a configuration. +To disable this feature just set spring.zipkin.discoveryClientEnabled to `false. +When the Discovery Client feature is enabled, Sleuth uses +LoadBalancerClient to find the URL of the Zipkin Server. It means +that you can set up the load balancing configuration e.g. via Ribbon. +zipkinserver: + ribbon: + ListOfServers: host1,host2 +If you have web, rabbit, or kafka together on the classpath, you might need to pick the means by which you would like to send spans to zipkin. +To do so, set web, rabbit, or kafka to the spring.zipkin.sender.type property. +The following example shows setting the sender type for web: +spring.zipkin.sender.type: web +To customize the RestTemplate that sends spans to Zipkin via HTTP, you can register +the ZipkinRestTemplateCustomizer bean. @Configuration -public static class CustomPollerConfiguration { - - @Bean(name = StreamSpanReporter.POLLER) - PollerMetadata customPoller() { - PollerMetadata poller = new PollerMetadata(); - poller.setMaxMessagesPerPoll(500); - poller.setTrigger(new PeriodicTrigger(5000L)); - return poller; +class MyConfig { + @Bean ZipkinRestTemplateCustomizer myCustomizer() { + return new ZipkinRestTemplateCustomizer() { + @Override + void customize(RestTemplate restTemplate) { + // customize the RestTemplate + } + }; } } -
    +If, however, you would like to control the full process of creating the RestTemplate +object, you will have to create a bean of zipkin2.reporter.Sender type. + @Bean Sender myRestTemplateSender(ZipkinProperties zipkin, + ZipkinRestTemplateCustomizer zipkinRestTemplateCustomizer) { + RestTemplate restTemplate = mySuperCustomRestTemplate(); + zipkinRestTemplateCustomizer.customize(restTemplate); + return myCustomSender(zipkin, restTemplate); + }     - -Metrics -Currently Spring Cloud Sleuth registers very simple metrics related to spans. -It’s using the Spring Boot’s metrics support -to calculate the number of accepted and dropped spans. Each time a span gets -sent to Zipkin the number of accepted spans will increase. If there’s an error then -the number of dropped spans will get increased. + +Zipkin Stream Span Consumer + +We recommend using Zipkin’s native support for message-based span sending. +Starting from the Edgware release, the Zipkin Stream server is deprecated. +In the Finchley release, it got removed. + +If for some reason you need to create the deprecated Stream Zipkin server, see the Dalston Documentation. Integrations +
    +OpenTracing +Spring Cloud Sleuth is compatible with OpenTracing. +If you have OpenTracing on the classpath, we automatically register the OpenTracing Tracer bean. +If you wish to disable this, set spring.sleuth.opentracing.enabled to false +
    Runnable and Callable -If you’re wrapping your logic in Runnable or Callable it’s enough to wrap those classes in their Sleuth representative. -Example for Runnable: +If you wrap your logic in Runnable or Callable, you can wrap those classes in their Sleuth representative, as shown in the following example for Runnable: Runnable runnable = new Runnable() { @Override public void run() { @@ -1369,11 +1615,12 @@ the number of dropped spans will get increased. } }; // Manual `TraceRunnable` creation with explicit "calculateTax" Span name -Runnable traceRunnable = new TraceRunnable(tracer, spanNamer, runnable, "calculateTax"); -// Wrapping `Runnable` with `Tracer`. The Span name will be taken either from the -// `@SpanName` annotation or from `toString` method -Runnable traceRunnableFromTracer = tracer.wrap(runnable); -Example for Callable: +Runnable traceRunnable = new TraceRunnable(tracing, spanNamer, runnable, + "calculateTax"); +// Wrapping `Runnable` with `Tracing`. That way the current span will be available +// in the thread of `Runnable` +Runnable traceRunnableFromTracer = tracing.currentTraceContext().wrap(runnable); +The following example shows how to do so for Callable: Callable<String> callable = new Callable<String>() { @Override public String call() throws Exception { @@ -1386,33 +1633,33 @@ Runnable traceRunnableFromTracer = tracer.wrap(runnable); } }; // Manual `TraceCallable` creation with explicit "calculateTax" Span name -Callable<String> traceCallable = new TraceCallable<>(tracer, spanNamer, callable, "calculateTax"); -// Wrapping `Callable` with `Tracer`. The Span name will be taken either from the -// `@SpanName` annotation or from `toString` method -Callable<String> traceCallableFromTracer = tracer.wrap(callable); -That way you will ensure that a new Span is created and closed for each execution. +Callable<String> traceCallable = new TraceCallable<>(tracing, spanNamer, callable, + "calculateTax"); +// Wrapping `Callable` with `Tracing`. That way the current span will be available +// in the thread of `Callable` +Callable<String> traceCallableFromTracer = tracing.currentTraceContext().wrap(callable); +That way, you ensure that a new span is created and closed for each execution.
    Hystrix
    Custom Concurrency Strategy -We’re registering a custom HystrixConcurrencyStrategy -that wraps all Callable instances into their Sleuth representative - -the TraceCallable. The strategy either starts or continues a span depending on the fact whether tracing was already going -on before the Hystrix command was called. To disable the custom Hystrix Concurrency Strategy set the spring.sleuth.hystrix.strategy.enabled to false. +We register a custom HystrixConcurrencyStrategy called TraceCallable that wraps all Callable instances in their Sleuth representative. +The strategy either starts or continues a span, depending on whether tracing was already going on before the Hystrix command was called. +To disable the custom Hystrix Concurrency Strategy, set the spring.sleuth.hystrix.strategy.enabled to false.
    Manual Command setting -Assuming that you have the following HystrixCommand: +Assume that you have the following HystrixCommand: HystrixCommand<String> hystrixCommand = new HystrixCommand<String>(setter) { @Override protected String run() throws Exception { return someLogic(); } }; -In order to pass the tracing information you have to wrap the same logic in the Sleuth version of the HystrixCommand which is the -TraceCommand: -TraceCommand<String> traceCommand = new TraceCommand<String>(tracer, traceKeys, setter) { +To pass the tracing information, you have to wrap the same logic in the Sleuth version of the HystrixCommand, which is called +TraceCommand, as shown in the following example: +TraceCommand<String> traceCommand = new TraceCommand<String>(tracer, setter) { @Override public String doRun() throws Exception { return someLogic(); @@ -1422,108 +1669,99 @@ on before the Hystrix command was called. To disable the custom Hystrix Concurre
    RxJava -We’re registering a custom RxJavaSchedulersHook -that wraps all Action0 instances into their Sleuth representative - -the TraceAction. The hook either starts or continues a span depending on the fact whether tracing was already going -on before the Action was scheduled. To disable the custom RxJavaSchedulersHook set the spring.sleuth.rxjava.schedulers.hook.enabled to false. -You can define a list of regular expressions for thread names, for which you don’t want a Span to be created. Just provide a comma separated list -of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property. +We registering a custom RxJavaSchedulersHook that wraps all Action0 instances in their Sleuth representative, which is called TraceAction. +The hook either starts or continues a span, depending on whether tracing was already going on before the Action was scheduled. +To disable the custom RxJavaSchedulersHook, set the spring.sleuth.rxjava.schedulers.hook.enabled to false. +You can define a list of regular expressions for thread names for which you do not want spans to be created. +To do so, provide a comma-separated list of regular expressions in the spring.sleuth.rxjava.schedulers.ignoredthreads property. + +The suggest approach to reactive programming and Sleuth is to use +the Reactor support. +
    HTTP integration -Features from this section can be disabled by providing the spring.sleuth.web.enabled property with value equal to false. +Features from this section can be disabled by setting the spring.sleuth.web.enabled property with value equal to false.
    HTTP Filter -Via the TraceFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which - the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would - like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern. +Through the TracingFilter, all sampled incoming requests result in creation of a Span. +That Span’s name is http: + the path to which the request was sent. +For example, if the request was sent to /this/that then the name will be http:/this/that. +You can configure which URIs you would like to skip by setting the spring.sleuth.web.skipPattern property. +If you have ManagementServerProperties on classpath, its value of contextPath gets appended to the provided skip pattern. +If you want to reuse the Sleuth’s default skip patterns and just append your own, pass those patterns by using the spring.sleuth.web.additionalSkipPattern.
    HandlerInterceptor -Since we want the span names to be precise we’re using a TraceHandlerInterceptor that either wraps an - existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. The - TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. If the - the TraceFilter doesn’t see this attribute set it will create a "fallback" span which is an additional - span created on the server side so that the trace is presented properly in the UI. Seeing that most likely - signifies that there is a missing instrumentation. In that case please file an issue in Spring Cloud Sleuth. +Since we want the span names to be precise, we use a TraceHandlerInterceptor that either wraps an existing HandlerInterceptor or is added directly to the list of existing HandlerInterceptors. +The TraceHandlerInterceptor adds a special request attribute to the given HttpServletRequest. +If the the TracingFilter does not see this attribute, it creates a "fallback" span, which is an additional span created on the server side so that the trace is presented properly in the UI. +If that happens, there is probably missing instrumentation. +In that case, please file an issue in Spring Cloud Sleuth.
    Async Servlet support -If your controller returns a Callable or a WebAsyncTask Spring Cloud Sleuth will continue the existing span instead of creating a new one. +If your controller returns a Callable or a WebAsyncTask, Spring Cloud Sleuth continues the existing span instead of creating a new one.
    WebFlux support -Via the TraceWebFilter all sampled incoming requests result in creation of a Span. That Span’s name is http: + the path to which - the request was sent. E.g. if the request was sent to /foo/bar then the name will be http:/foo/bar. You can configure which URIs you would - like to skip via the spring.sleuth.web.skipPattern property. If you have ManagementServerProperties on classpath then - its value of contextPath gets appended to the provided skip pattern. +Through TraceWebFilter, all sampled incoming requests result in creation of a Span. +That Span’s name is http: + the path to which the request was sent. +For example, if the request was sent to /this/that, the name is http:/this/that. +You can configure which URIs you would like to skip by using the spring.sleuth.web.skipPattern property. +If you have ManagementServerProperties on the classpath, its value of contextPath gets appended to the provided skip pattern. +If you want to reuse Sleuth’s default skip patterns and append your own, pass those patterns by using the spring.sleuth.web.additionalSkipPattern. +
    +
    +Dubbo RPC support +Via the integration with Brave, Spring Cloud Sleuth supports Dubbo. +It’s enough to add the brave-instrumentation-dubbo-rpc dependency: +<dependency> + <groupId>io.zipkin.brave</groupId> + <artifactId>brave-instrumentation-dubbo-rpc</artifactId> +</dependency> +You need to also set a dubbo.properties file with the following contents: +dubbo.provider.filter=tracing +dubbo.consumer.filter=tracing +You can read more about Brave - Dubbo integration here. +An example of Spring Cloud Sleuth and Dubbo can be found here.
    -HTTP client integration +HTTP Client Integration
    Synchronous Rest Template -We’re injecting a RestTemplate interceptor that ensures that all the tracing information is passed to the requests. Each time a -call is made a new Span is created. It gets closed upon receiving the response. In order to block the synchronous RestTemplate features -just set spring.sleuth.web.client.enabled to false. +We inject a RestTemplate interceptor to ensure that all the tracing information is passed to the requests. +Each time a call is made, a new Span is created. +It gets closed upon receiving the response. +To block the synchronous RestTemplate features, set spring.sleuth.web.client.enabled to false. -You have to register RestTemplate as a bean so that the interceptors will get injected. -If you create a RestTemplate instance with a new keyword then the instrumentation WILL NOT work. +You have to register RestTemplate as a bean so that the interceptors get injected. +If you create a RestTemplate instance with a new keyword, the instrumentation does NOT work.
    Asynchronous Rest Template -A traced version of an AsyncRestTemplate bean is registered for you out of the box. If you -have your own bean you have to wrap it in a TraceAsyncRestTemplate representation. The best solution -is to only customize the ClientHttpRequestFactory and / or AsyncClientHttpRequestFactory. -If you have your own AsyncRestTemplate and you don’t wrap it your calls WILL NOT GET TRACED. +Starting with Sleuth 2.0.0, we no longer register a bean of AsyncRestTemplate type. +It is up to you to create such a bean. +Then we instrument it. -Custom instrumentation is set to create and close Spans upon sending and receiving requests. You can customize the ClientHttpRequestFactory -and the AsyncClientHttpRequestFactory by registering your beans. Remember to use tracing compatible implementations (e.g. don’t forget to -wrap ThreadPoolTaskScheduler in a TraceAsyncListenableTaskExecutor). Example of custom request factories: -@EnableAutoConfiguration -@Configuration -public static class TestConfiguration { - - @Bean - ClientHttpRequestFactory mySyncClientFactory() { - return new MySyncClientHttpRequestFactory(); - } - - @Bean - AsyncClientHttpRequestFactory myAsyncClientFactory() { - return new MyAsyncClientHttpRequestFactory(); - } -} -To block the AsyncRestTemplate features set spring.sleuth.web.async.client.enabled to false. -To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper set spring.sleuth.web.async.client.factory.enabled -to false. If you don’t want to create AsyncRestClient at all set spring.sleuth.web.async.client.template.enabled to false. +To block the AsyncRestTemplate features, set spring.sleuth.web.async.client.enabled to false. +To disable creation of the default TraceAsyncClientHttpRequestFactoryWrapper, set spring.sleuth.web.async.client.factory.enabled +to false. +If you do not want to create AsyncRestClient at all, set spring.sleuth.web.async.client.template.enabled to false.
    Multiple Asynchronous Rest Templates -Sometimes you need to use multiple implementations of Asynchronous Rest Template. In the following snippet you -can see an example of how to set up such a custom AsyncRestTemplate. +Sometimes you need to use multiple implementations of the Asynchronous Rest Template. +In the following snippet, you can see an example of how to set up such a custom AsyncRestTemplate: @Configuration @EnableAutoConfiguration static class Config { - @Autowired Tracer tracer; - @Autowired HttpTraceKeysInjector httpTraceKeysInjector; - @Autowired HttpSpanInjector spanInjector; @Bean(name = "customAsyncRestTemplate") - public AsyncRestTemplate traceAsyncRestTemplate(@Qualifier("customHttpRequestFactoryWrapper") - TraceAsyncClientHttpRequestFactoryWrapper wrapper, ErrorParser errorParser) { - return new TraceAsyncRestTemplate(wrapper, this.tracer, errorParser); - } - - @Bean(name = "customHttpRequestFactoryWrapper") - public TraceAsyncClientHttpRequestFactoryWrapper traceAsyncClientHttpRequestFactory() { - return new TraceAsyncClientHttpRequestFactoryWrapper(this.tracer, - this.spanInjector, - asyncClientFactory(), - clientHttpRequestFactory(), - this.httpTraceKeysInjector); + public AsyncRestTemplate traceAsyncRestTemplate() { + return new AsyncRestTemplate(asyncClientFactory(), clientHttpRequestFactory()); } private ClientHttpRequestFactory clientHttpRequestFactory() { @@ -1540,95 +1778,111 @@ static class Config { }
    -
    -WebClient -We inject a ExchangeFilterFunction implementation that creates a span and via on success and on -error callbacks takes care of closing client side spans. +
    +<literal>WebClient</literal> +We inject a ExchangeFilterFunction implementation that creates a span and, through on-success and on-error callbacks, takes care of closing client-side spans. +To block this feature, set spring.sleuth.web.client.enabled to false. -You have to register WebClient as a bean so that the tracing instrumention gets applied. -If you create a WebClient instance with a new keyword then the instrumentation WILL NOT work. +You have to register WebClient as a bean so that the tracing instrumentation gets applied. +If you create a WebClient instance with a new keyword, the instrumentation does NOT work.
    Traverson -If you’re using the Traverson library -it’s enough for you to inject a RestTemplate as a bean into your Traverson object. Since RestTemplate -is already intercepted, you will get full support of tracing in your client. Below you can find a pseudo code -of how to do that: +If you use the Traverson library, you can inject a RestTemplate as a bean into your Traverson object. +Since RestTemplate is already intercepted, you get full support for tracing in your client. The following pseudo code +shows how to do that: @Autowired RestTemplate restTemplate; Traverson traverson = new Traverson(URI.create("http://some/address"), MediaType.APPLICATION_JSON, MediaType.APPLICATION_JSON_UTF8).setRestOperations(restTemplate); // use Traverson
    +
    +Apache <literal>HttpClientBuilder</literal> and <literal>HttpAsyncClientBuilder</literal> +We instrument the HttpClientBuilder and HttpAsyncClientBuilder so that +tracing context gets injected to the sent requests. +To block these features, set spring.sleuth.web.client.enabled to false. +
    +
    +Netty <literal>HttpClient</literal> +We instrument the Netty’s HttpClient. +To block this feature, set spring.sleuth.web.client.enabled to false. + +You have to register HttpClient as a bean so that the instrumentation happens. +If you create a HttpClient instance with a new keyword, the instrumentation does NOT work. + +
    +
    +<literal>UserInfoRestTemplateCustomizer</literal> +We instrument the Spring Security’s UserInfoRestTemplateCustomizer. +To block this feature, set spring.sleuth.web.client.enabled to false. +
    Feign -By default Spring Cloud Sleuth provides integration with feign via the TraceFeignClientAutoConfiguration. You can disable it entirely -by setting spring.sleuth.feign.enabled to false. If you do so then no Feign related instrumentation will take place. -Part of Feign instrumentation is done via a FeignBeanPostProcessor. You can disable it by providing the spring.sleuth.feign.processor.enabled equal to false. -If you set it like this then Spring Cloud Sleuth will not instrument any of your custom Feign components. All the default instrumentation -however will be still there. +By default, Spring Cloud Sleuth provides integration with Feign through TraceFeignClientAutoConfiguration. +You can disable it entirely by setting spring.sleuth.feign.enabled to false. +If you do so, no Feign-related instrumentation take place. +Part of Feign instrumentation is done through a FeignBeanPostProcessor. +You can disable it by setting spring.sleuth.feign.processor.enabled to false. +If you set it to false, Spring Cloud Sleuth does not instrument any of your custom Feign components. +However, all the default instrumentation is still there.
    -Asynchronous communication -
    -@Async annotated methods -In Spring Cloud Sleuth we’re instrumenting async related components so that the tracing information is passed between threads. -You can disable this behaviour by setting the value of spring.sleuth.async.enabled to false. -If you annotate your method with @Async then we’ll automatically create a new Span with the following characteristics: +Asynchronous Communication +
    +<literal>@Async</literal> Annotated methods +In Spring Cloud Sleuth, we instrument async-related components so that the tracing information is passed between threads. +You can disable this behavior by setting the value of spring.sleuth.async.enabled to false. +If you annotate your method with @Async, we automatically create a new Span with the following characteristics: -if the method is annotated with @SpanName then the value of the annotation will be the Span’s name +If the method is annotated with @SpanName, the value of the annotation is the Span’s name. -if the method is not annotated with @SpanName the Span name will be the annotated method name +If the method is not annotated with @SpanName, the Span name is the annotated method name. -the Span will be tagged with that method’s class name and the method name too +The span is tagged with the method’s class name and method name.
    -
    -@Scheduled annotated methods -In Spring Cloud Sleuth we’re instrumenting scheduled method execution so that the tracing information is passed between threads. You can disable this behaviour -by setting the value of spring.sleuth.scheduled.enabled to false. -If you annotate your method with @Scheduled then we’ll automatically create a new Span with the following characteristics: +
    +<literal>@Scheduled</literal> Annotated Methods +In Spring Cloud Sleuth, we instrument scheduled method execution so that the tracing information is passed between threads. +You can disable this behavior by setting the value of spring.sleuth.scheduled.enabled to false. +If you annotate your method with @Scheduled, we automatically create a new span with the following characteristics: -the Span name will be the annotated method name +The span name is the annotated method name. -the Span will be tagged with that method’s class name and the method name too +The span is tagged with the method’s class name and method name. -If you want to skip Span creation for some @Scheduled annotated classes you can set the -spring.sleuth.scheduled.skipPattern with a regular expression that will match the fully qualified name of the -@Scheduled annotated class. - -If you are using spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, Span will be created for each Hystrix metrics and sent to Zipkin. This may be annoying. You can prevent this by setting spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask - +If you want to skip span creation for some @Scheduled annotated classes, you can set the spring.sleuth.scheduled.skipPattern with a regular expression that matches the fully qualified name of the @Scheduled annotated class. +If you use spring-cloud-sleuth-stream and spring-cloud-netflix-hystrix-stream together, a span is created for each Hystrix metrics and sent to Zipkin. +This behavior may be annoying. That’s why, by default, spring.sleuth.scheduled.skipPattern=org.springframework.cloud.netflix.hystrix.stream.HystrixStreamTask.
    -Executor, ExecutorService and ScheduledExecutorService -We’re providing LazyTraceExecutor, TraceableExecutorService and TraceableScheduledExecutorService. Those implementations -are creating Spans each time a new task is submitted, invoked or scheduled. -Here you can see an example of how to pass tracing information with TraceableExecutorService when working with CompletableFuture: +Executor, ExecutorService, and ScheduledExecutorService +We provide LazyTraceExecutor, TraceableExecutorService, and TraceableScheduledExecutorService. Those implementations create spans each time a new task is submitted, invoked, or scheduled. +The following example shows how to pass tracing information with TraceableExecutorService when working with CompletableFuture: CompletableFuture<Long> completableFuture = CompletableFuture.supplyAsync(() -> { // perform some logic return 1_000_000L; -}, new TraceableExecutorService(executorService, +}, new TraceableExecutorService(beanFactory, executorService, // 'calculateTax' explicitly names the span - this param is optional - tracer, traceKeys, spanNamer, "calculateTax")); + "calculateTax")); -Sleuth doesn’t work with parallelStream() out of the box. If you want -to have the tracing information propagated through the stream you have to use the -approach with supplyAsync(...) as presented above. +Sleuth does not work with parallelStream() out of the box. +If you want to have the tracing information propagated through the stream, you have to use the approach with supplyAsync(...), as shown earlier.
    Customization of Executors -Sometimes you need to set up a custom instance of the AsyncExecutor. In the following snippet you -can see an example of how to set up such a custom Executor. +Sometimes, you need to set up a custom instance of the AsyncExecutor. +The following example shows how to set up such a custom Executor: @Configuration @EnableAutoConfiguration @EnableAsync @@ -1653,31 +1907,56 @@ static class CustomExecutorConfig extends AsyncConfigurerSupport {
    Messaging -Spring Cloud Sleuth integrates with Spring Integration. It creates spans for publish and -subscribe events. To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false. -You can provide the spring.sleuth.integration.patterns pattern to explicitly -provide the names of channels that you want to include for tracing. By default all channels -are included. +Features from this section can be disabled by setting the spring.sleuth.messaging.enabled property with value equal to false. +
    +Spring Integration and Spring Cloud Stream +Spring Cloud Sleuth integrates with Spring Integration. +It creates spans for publish and subscribe events. +To disable Spring Integration instrumentation, set spring.sleuth.integration.enabled to false. +You can provide the spring.sleuth.integration.patterns pattern to explicitly provide the names of channels that you want to include for tracing. +By default, all channels but hystrixStreamOutput channel are included. -When using the Executor to build a Spring Integration IntegrationFlow remember to use the untraced version of the Executor. -Decorating Spring Integration Executor Channel with TraceableExecutorService will cause the spans to be improperly closed. +When using the Executor to build a Spring Integration IntegrationFlow, you must use the untraced version of the Executor. +Decorating the Spring Integration Executor Channel with TraceableExecutorService causes the spans to be improperly closed.
    +
    +Spring RabbitMq +We instrument the RabbitTemplate so that tracing headers get injected +into the message. +To block this feature, set spring.sleuth.messaging.rabbit.enabled to false. +
    +
    +Spring Kafka +We instrument the Spring Kafka’s ProducerFactory and ConsumerFactory +so that tracing headers get injected into the created Spring Kafka’s +Producer and Consumer. +To block this feature, set spring.sleuth.messaging.kafka.enabled to false. + +We do not support context propagation via @KafkaListener annotation. +Check this issue for more information. + +
    +
    Zuul -We’re registering Zuul filters to propagate the tracing information (the request header is enriched with tracing data). -To disable Zuul support set the spring.sleuth.zuul.enabled property to false. +We instrument the Zuul Ribbon integration by enriching the Ribbon requests with tracing information. +To disable Zuul support, set the spring.sleuth.zuul.enabled property to false.
    Running examples -You can find the running examples deployed in the Pivotal Web Services. Check them out in the following links: +You can see the running examples deployed in the Pivotal Web Services. +Check them out at the following links: -Zipkin for apps presented in the samples to the top +Zipkin for apps presented in the samples to the top. First make +a request to Service 1 and then check out the trace in Zipkin. -Zipkin for Brewery on PWS, its Github Code +Zipkin for Brewery on PWS, its Github Code. +Ensure that you’ve picked the lookback period of 7 days. If there are no traces, go to Presenting application +and order some beers. Then check Zipkin for traces.