91 lines
24 KiB
HTML
91 lines
24 KiB
HTML
<html><head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
|
|
<title>20. Metrics: Spectator, Servo, and Atlas</title><link rel="stylesheet" type="text/css" href="css/manual-multipage.css"><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"><link rel="home" href="multi_spring-cloud.html" title="Spring Cloud"><link rel="up" href="multi__spring_cloud_netflix.html" title="Part III. Spring Cloud Netflix"><link rel="prev" href="multi__polyglot_support_with_sidecar.html" title="19. Polyglot support with Sidecar"><link rel="next" href="multi_netflix-metrics-atlas.html" title="21. Metrics Backend: Atlas"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">20. Metrics: Spectator, Servo, and Atlas</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="multi__polyglot_support_with_sidecar.html">Prev</a> </td><th width="60%" align="center">Part III. Spring Cloud Netflix</th><td width="20%" align="right"> <a accesskey="n" href="multi_netflix-metrics-atlas.html">Next</a></td></tr></table><hr></div><div class="chapter"><div class="titlepage"><div><div><h2 class="title"><a name="netflix-metrics" href="#netflix-metrics"></a>20. Metrics: Spectator, Servo, and Atlas</h2></div></div></div><p>When used together, Spectator (or Servo) and Atlas provide a near real-time operational insight platform.
|
|
Spectator and Servo are Netflix’s metrics collection libraries.
|
|
Atlas is a Netflix metrics backend that manages dimensional time-series data.</p><p>Servo served Netflix for several years and is still usable but is gradually being phased out in favor of Spectator, which is designed to work only with Java 8.
|
|
Spring Cloud Netflix provides support for both, but Java 8-based applications are encouraged to use Spectator.</p><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_dimensional_versus_hierarchical_metrics" href="#_dimensional_versus_hierarchical_metrics"></a>20.1 Dimensional Versus Hierarchical Metrics</h2></div></div></div><p>Spring Boot Actuator metrics are hierarchical, and the metrics are separated only by name.
|
|
These names often follow a naming convention that embeds key/value attribute pairs (dimensions) into the name (separated by periods).
|
|
Consider the following metrics for two endpoints, <code class="literal">root</code> and <code class="literal">star-star</code>:</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">{</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"counter.status.200.root"</span>: <span class="hl-number">20</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"counter.status.400.root"</span>: <span class="hl-number">3</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"counter.status.200.star-star"</span>: <span class="hl-number">5</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">}</span></pre><p>The first metric gives us a normalized count of successful requests against the root endpoint per unit of time.
|
|
But what if the system has 20 endpoints and you want to get a count of successful requests against all the endpoints?
|
|
Some hierarchical metrics backends would let you specify a wildcard, such as <code class="literal">counter.status.200.*</code>, that would read all 20 metrics and aggregate the results.
|
|
Alternatively, you could provide a <code class="literal">HandlerInterceptorAdapter</code> that intercepts and records a metric such as <code class="literal">counter.status.200.all</code> for all successful requests irrespective of the endpoint, but now you must write 20+1 different metrics.
|
|
Similarly, if you want to know the total number of successful requests for all endpoints in the service, you could specify a wildcard such as <code class="literal">counter.status.2*.*</code>.</p><p>Even in the presence of wildcarding support on a hierarchical metrics backend, naming consistency can be difficult.
|
|
Specifically, the position of these tags in the name string can slip with time, breaking queries.
|
|
For example, suppose we add an additional dimension to the earlier hierarchical metrics for an HTTP method.
|
|
Then <code class="literal">counter.status.200.root</code> becomes <code class="literal">counter.status.200.method.get.root</code> (or <code class="literal">post</code> and so on).
|
|
Suddenly, Our <code class="literal">counter.status.200.*</code> no longer has the same semantic meaning.
|
|
Furthermore, if the new dimension is not applied uniformly across the codebase, certain queries may become impossible.
|
|
This can quickly get out of hand.</p><p>Netflix metrics are tagged (in other words, they are dimensional).
|
|
Each metric has a name, but this single named metric can contain multiple statistics and 'tag' key/value pairs, which allows more querying flexibility.
|
|
In fact, the statistics themselves are recorded in a special tag.</p><p>When recorded with Netflix Servo or Spectator, a timer for the root endpoint described earlier contains four statistics for each status code, where the count statistic is identical to Spring Boot Actuator’s counter.
|
|
When we have encountered an HTTP 200 and 400 with the preceding examples, there are eight available data points, as shown in the following example:</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">{</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=200,stastic=count)"</span>: <span class="hl-number">20</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=200,stastic=max)"</span>: <span class="hl-number">0.7265630630000001</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=200,stastic=totalOfSquares)"</span>: <span class="hl-number">0.04759702862580789</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=200,stastic=totalTime)"</span>: <span class="hl-number">0.2093076914666667</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=400,stastic=count)"</span>: <span class="hl-number">1</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=400,stastic=max)"</span>: <span class="hl-number">0</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=400,stastic=totalOfSquares)"</span>: <span class="hl-number">0</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"root(status=400,stastic=totalTime)"</span>: <span class="hl-number">0</span><span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">,</span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">}</span></pre></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="_default_metrics_collection" href="#_default_metrics_collection"></a>20.2 Default Metrics Collection</h2></div></div></div><p>Without any additional dependencies or configuration, a Spring Cloud based service autoconfigures a Servo <code class="literal">MonitorRegistry</code> and begins collecting metrics on every Spring MVC request.
|
|
By default, a Servo timer with a name of <code class="literal">rest</code> is recorded for each MVC request, which is tagged with the following information:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">HTTP method (<code class="literal">GET</code>, <code class="literal">POST</code>, and so on).</li><li class="listitem">HTTP status (<code class="literal">200</code>, <code class="literal">400</code>, <code class="literal">500</code>, and so on).</li><li class="listitem">URI (or <code class="literal">root</code> if the URI is empty), sanitized for Atlas.</li><li class="listitem">The exception class name, if the request handler threw an exception.</li><li class="listitem">The caller, if a request header with a key matching <code class="literal">netflix.metrics.rest.callerHeader</code> is set on the request.
|
|
There is no default key for <code class="literal">netflix.metrics.rest.callerHeader</code>.
|
|
You must add it to your application properties if you wish to collect caller information.</li></ul></div><p>Set the <code class="literal">netflix.metrics.rest.metricName</code> property to change the name of the metric from <code class="literal">rest</code> to the name you provide.</p><p>If Spring AOP is enabled and <code class="literal">org.aspectj:aspectjweaver</code> is present on your runtime classpath, Spring Cloud also collects metrics on every client call made with <code class="literal">RestTemplate</code>.
|
|
A Servo timer with a name of <code class="literal">restclient</code> is recorded for each MVC request, which is tagged with the following information:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">HTTP method ('GET', 'POST', and so on).</li><li class="listitem">HTTP status (<code class="literal">200</code>, <code class="literal">400</code>, <code class="literal">500</code>, and so on) and possibly <code class="literal">CLIENT_ERROR</code> if the response returned null or <code class="literal">IO_ERROR</code> if an <code class="literal">IOException</code> occurred during the execution of the <code class="literal">RestTemplate</code> method.</li><li class="listitem">URI, sanitized for Atlas.</li><li class="listitem">Client name.</li></ul></div><div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Warning"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Warning]" src="images/warning.png"></td><th align="left">Warning</th></tr><tr><td align="left" valign="top"><p>Avoid using hard-coded URL parameters within <code class="literal">RestTemplate</code>.
|
|
When targeting dynamic endpoints, use URL variables.
|
|
Doing so avoids potential “GC Overhead Limit Reached” issues where <code class="literal">ServoMonitorCache</code> treats each URL as a unique key.
|
|
The following example shows both the recommended and the problematic ways to set URL parameters:</p></td></tr></table></div><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// recommended</span>
|
|
String orderid = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"1"</span>;
|
|
restTemplate.getForObject(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"http://testeurekabrixtonclient/orders/{orderid}"</span>, String.<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">class</span>, orderid)
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// avoid</span>
|
|
restTemplate.getForObject(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"http://testeurekabrixtonclient/orders/1"</span>, String.<span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">class</span>)</pre></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="netflix-metrics-spectator" href="#netflix-metrics-spectator"></a>20.3 Metrics Collection: Spectator</h2></div></div></div><p>To enable Spectator metrics, include a dependency on <code class="literal">spring-boot-starter-spectator</code>, as follows:</p><pre class="programlisting"> <span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><dependency></span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><groupId></span>org.springframework.cloud<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"></groupId></span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"><artifactId></span>spring-cloud-starter-netflix-spectator<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"></artifactId></span>
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-tag"></dependency></span></pre><p>In Spectator parlance, a meter is a named, typed, and tagged configuration, while a metric represents the value of a given meter at a point in time.
|
|
Spectator meters are created and controlled by a registry, which currently has several different implementations.
|
|
Spectator provides four meter types: counter, timer, gauge, and distribution summary.</p><p>Spring Cloud Spectator integration configures an injectable <code class="literal">com.netflix.spectator.api.Registry</code> instance for you.
|
|
Specifically, it configures a <code class="literal">ServoRegistry</code> instance in order to unify the collection of REST metrics and the exporting of metrics to the Atlas backend under a single Servo API.
|
|
Practically, this means that your code may use a mixture of Servo monitors and Spectator meters.
|
|
Spring Boot scoops up both Actuator <code class="literal">MetricReader</code> instances and ships them to the Atlas backend.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_spectator_counter" href="#_spectator_counter"></a>20.3.1 Spectator Counter</h3></div></div></div><p>A counter measures the rate at which some event is occurring, as shown in the following example:</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// create a counter with a name and a set of tags</span>
|
|
Counter counter = registry.counter(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"counterName"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagKey1"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagValue1"</span>, ...);
|
|
counter.increment(); <span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// increment when an event occurs</span>
|
|
counter.increment(<span class="hl-number">10</span>); <span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// increment by a discrete amount</span></pre><p>The counter records a single time-normalized statistic.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_spectator_timer" href="#_spectator_timer"></a>20.3.2 Spectator Timer</h3></div></div></div><p>A timer measures how long some event takes.
|
|
Spring Cloud automatically records timers for Spring MVC requests and, conditionally, <code class="literal">RestTemplate</code> requests, which can later be used to create dashboards for request related metrics like latency, as shown in the following example:</p><div class="figure"><a name="d0e6174" href="#d0e6174"></a><p class="title"><b>Figure 20.1. Request Latency</b></p><div class="figure-contents"><div class="mediaobject"><img src="https://raw.githubusercontent.com/spring-cloud/spring-cloud-netflix/master/docs/src/main/asciidoc/images/RequestLatency.png" alt="RequestLatency"></div></div></div><br class="figure-break"><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// create a timer with a name and a set of tags</span>
|
|
Timer timer = registry.timer(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"timerName"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagKey1"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagValue1"</span>, ...);
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// execute an operation and time it at the same time</span>
|
|
T result = timer.record(() -> fooReturnsT());
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// alternatively, if you must manually record the time</span>
|
|
Long start = System.nanoTime();
|
|
T result = fooReturnsT();
|
|
timer.record(System.nanoTime() - start, TimeUnit.NANOSECONDS);</pre><p>The timer simultaneously records four statistics: <code class="literal">count</code>, <code class="literal">max</code>, <code class="literal">totalOfSquares</code>, and <code class="literal">totalTime</code>.
|
|
The count statistic always matches the single normalized value provided by a counter as though you had called <code class="literal">increment()</code> once on the counter for each time you recorded a timing, so it is rarely necessary to count and time separately for a single operation.</p><p>For <a class="link" href="https://github.com/Netflix/spectator/wiki/Timer-Usage#longtasktimer" target="_top">long-running operations</a>, Spectator provides a special <code class="literal">LongTaskTimer</code>.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_spectator_gauge" href="#_spectator_gauge"></a>20.3.3 Spectator Gauge</h3></div></div></div><p>Gauges show some current value, such as the size of a queue or number of threads in a running state.
|
|
Since gauges are sampled, they provide no information about how these values fluctuate between samples.</p><p>The normal use of a gauge involves registering the gauge once on initialization with an ID, a reference to the object to be sampled, and a function to get or compute a numeric value based on the object.
|
|
The reference to the object is passed in separately, and the Spectator registry keeps a weak reference to the object.
|
|
If the object is garbage collected, Spectator automatically drops the registration.
|
|
See <a class="link" href="https://github.com/Netflix/spectator/wiki/Gauge-Usage#using-lambda" target="_top">the note</a> in Spectator’s documentation about potential memory leaks if this API is misused.
|
|
The following listing shows how to automatically and manually sample a gauge:</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// the registry automatically samples this gauge periodically</span>
|
|
registry.gauge(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"gaugeName"</span>, pool, Pool::numberOfRunningThreads);
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// manually sample a value in code at periodic intervals -- last resort!</span>
|
|
registry.gauge(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"gaugeName"</span>, Arrays.asList(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagKey1"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagValue1"</span>, ...), <span class="hl-number">1000</span>);</pre></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_spectator_distribution_summaries" href="#_spectator_distribution_summaries"></a>20.3.4 Spectator Distribution Summaries</h3></div></div></div><p>A distribution summary tracks the distribution of events.
|
|
It is similar to a timer but more general in that the size does not have to be a period of time.
|
|
For example, a distribution summary could be used to measure the payload sizes of requests hitting a server.
|
|
The following example defines a distribution summary:</p><pre class="programlisting"><span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// the registry automatically samples this gauge periodically</span>
|
|
DistributionSummary ds = registry.distributionSummary(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"dsName"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagKey1"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagValue1"</span>, ...);
|
|
ds.record(request.sizeInBytes());</pre></div></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="netflix-metrics-servo" href="#netflix-metrics-servo"></a>20.4 Metrics Collection: Servo</h2></div></div></div><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><table border="0" summary="Note"><tr><td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="images/note.png"></td><th align="left">Note</th></tr><tr><td align="left" valign="top"><p>If your code is compiled on Java 8, use Spectator instead of Servo, as Spectator is destined to replace Servo entirely.</p></td></tr></table></div><p>In Servo parlance, a monitor is a named, typed, and tagged configuration, and a metric represents the value of a given monitor at a point in time.
|
|
Servo monitors are logically equivalent to Spectator meters.
|
|
Servo monitors are created and controlled by a <code class="literal">MonitorRegistry</code>.
|
|
While it is still available, Servo has a <a class="link" href="https://github.com/Netflix/servo/wiki/Getting-Started" target="_top">wider array</a> of monitor options than Spectator has meters.</p><p>Spring Cloud integration configures an injectable <code class="literal">com.netflix.servo.MonitorRegistry</code> instance for you.
|
|
Once you have created the appropriate <code class="literal">Monitor</code> type in Servo, the process of recording data is similar to that of Spectator.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_creating_servo_monitors" href="#_creating_servo_monitors"></a>20.4.1 Creating Servo Monitors</h3></div></div></div><p>If you use the Servo <code class="literal">MonitorRegistry</code> instance provided by Spring Cloud (specifically, an instance of <code class="literal">DefaultMonitorRegistry</code>), Servo provides convenience classes for retrieving <a class="link" href="https://github.com/Netflix/spectator/wiki/Servo-Comparison#dynamiccounter" target="_top">counters</a> and <a class="link" href="https://github.com/Netflix/spectator/wiki/Servo-Comparison#dynamictimer" target="_top">timers</a>.
|
|
These convenience classes ensure that only one <code class="literal">Monitor</code> is registered for each unique combination of name and tags.</p><p>To manually create a Monitor type in Servo, especially for the more exotic monitor types for which convenience methods are not provided, instantiate the appropriate type by providing a <code class="literal">MonitorConfig</code> instance, as shown in the following example:</p><pre class="programlisting">MonitorConfig config = MonitorConfig.builder(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"timerName"</span>).withTag(<span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagKey1"</span>, <span xmlns:d="http://docbook.org/ns/docbook" class="hl-string">"tagValue1"</span>).build();
|
|
|
|
<span xmlns:d="http://docbook.org/ns/docbook" class="hl-comment">// somewhere we should cache this Monitor by MonitorConfig</span>
|
|
Timer timer = <span xmlns:d="http://docbook.org/ns/docbook" class="hl-keyword">new</span> BasicTimer(config);
|
|
monitorRegistry.register(timer);</pre></div></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="multi__polyglot_support_with_sidecar.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="multi__spring_cloud_netflix.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="multi_netflix-metrics-atlas.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">19. Polyglot support with Sidecar </td><td width="20%" align="center"><a accesskey="h" href="multi_spring-cloud.html">Home</a></td><td width="40%" align="right" valign="top"> 21. Metrics Backend: Atlas</td></tr></table></div></body></html> |