SpringCloud/spring-cloud-dataflow-samples
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

SCDF metrics with Prometheus, InfluxDB and Grafana

Browse Source 
Resolve #56
Resolve #57
Resolve #60

- Address the review questions
- Update SCDF common tags names (use dot notation)
- Move Prometheus service-discovery module to samples

address review comments
This commit is contained in:
Christian Tzolov
2018-04-26 15:36:23 +02:00
committed by Ilayaperumal Gopinathan
parent 596589a12f
commit 2960e249aa
23 changed files with 3576 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
src/main/asciidoc/images/grafana-influx-dashboard.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 189 KiB

BIN
src/main/asciidoc/images/grafana-influxdb-datasource.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 90 KiB

BIN
src/main/asciidoc/images/grafana-prometheus-dashboard.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 668 KiB

BIN
src/main/asciidoc/images/scdf-micrometer-influxdb-grafana-architecture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 107 KiB

BIN
src/main/asciidoc/images/scdf-micrometer-prometheus-grafana-architecture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 116 KiB

2
src/main/asciidoc/index.adoc
View File

@@ -1,5 +1,5 @@
= Spring Cloud Data Flow Samples
Sabby Anandan; David Turanski; Glenn Renfro; Eric Bottard; Mark Pollack; Chris Schaefer
Sabby Anandan; David Turanski; Glenn Renfro; Eric Bottard; Mark Pollack; Chris Schaefer; Christian Tzolov
:doctype: book
:toc:
:toclevels: 4

123
src/main/asciidoc/micrometer/influx/main.adoc Normal file
View File

@@ -0,0 +1,123 @@
[[spring-cloud-data-flow-samples-micrometer-influx-overview]]
:sectnums:
:docs_dir: ../..
=== SCDF metrics with InfluxDB and Grafana
In this demonstration, you will learn how http://micrometer.io[Micrometer] can help to monitor your http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] (SCDF) streams using https://docs.influxdata.com/influxdb/v1.5/[InfluxDB] and https://grafana.com/grafana[Grafana].
https://docs.influxdata.com/influxdb/v1.5/[InfluxDB] is a real-time storage for time-series data, such as SCDF metrics. It supports downsampling, automatically expiring and deleting unwanted data, as well as backup and restore. Analysis of data is done via a https://docs.influxdata.com/influxdb/v1.5/query_language/[SQL-like query] language.
https://grafana.com/grafana[Grafana] is open source metrics Dashboard platform. It supports multiple backend time-series databases including InluxDB.
The architecture (Fig.1) builds on the https://docs.spring.io/spring-boot/docs/2.0.1.RELEASE/reference/htmlsingle/#production-ready-metrics-getting-started[Spring Boot Micrometer] functionality. When a http://micrometer.io/docs/registry/influx[micrometer-registry-influx] dependency is found on the classpath the Spring Boot auto-configures the metrics export for `InfluxDB`.
The https://cloud.spring.io/spring-cloud-stream-app-starters/[Spring Cloud Stream] (SCSt) applications inherit the mircometer functionality, allowing them to compute and send various application metrics to the configured time-series database.
image::scdf-micrometer-influxdb-grafana-architecture.png[scaledwidth="100%", title="SCDF metrics analyzis with InfluxDB and Grafana"]
Out of the box, SCSt sends https://docs.spring.io/spring-boot/docs/2.0.1.RELEASE/reference/htmlsingle/#production-ready-metrics-meter[core metrics] such as `CPU`, `Memory`, `MVC` and `Health` to name some. Among those the https://docs.spring.io/spring-integration/docs/current/reference/html/system-management-chapter.html#micrometer-integration[Spring Integration metrics] allows computing the `Rate` and the `Latency` of the messages in the SCDF streams.
NOTE: Unlike Spring Cloud Data Flow Metrics Collector, metrics here are sent synchronously over HTTP not through a Binder channel topic.
All Spring Cloud Stream App Starers enrich the standard http://micrometer.io/docs/concepts#_supported_monitoring_systems[dimensional tags] with the following SCDF specific tags:
[width="100%",options="header"]
|====================
| tag name | SCDF property | default value
| stream.name | spring.cloud.dataflow.stream.name | unknown
| application.name | spring.cloud.dataflow.stream.app.label | unknown
| instance.index | instance.index | 0
| application.guid | spring.cloud.application.guid | unknown
| application.type | spring.cloud.dataflow.stream.app.type | unknown
|====================
NOTE: For custom app starters that don't extend from the https://github.com/spring-cloud-stream-app-starters/core[core] parent, you should add the `app-starters-common` : `org.springframework.cloud.stream.app` dependency to enable the SCDF tags.
Below we will present the steps to prep, configure the demo of Spring Cloud Data Flow's `Local` server integration with `InfluxDB`. For other deployment environment, such as `Cloud Foundry` or `Kubernetes`, additional configurations might be required.
==== Prerequisites
* A Running Data Flow Shell
include::{docs_dir}/shell.adoc[]
* A running local Data Flow Server
include::{docs_dir}/local-server.adoc[]
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
* Spring Cloud Stream 2.x based https://github.com/spring-cloud-stream-app-starters/time/blob/master/spring-cloud-starter-stream-source-time/README.adoc[Time] and https://github.com/spring-cloud-stream-app-starters/log/blob/master/spring-cloud-starter-stream-sink-log/README.adoc[Log] applications starters, pre-built with `io.micrometer:micrometer-registry-influx` dependency.
+
NOTE: Next versions of the https://start-scs.cfapps.io/[SCSt App Initializr] utility would add support for Micrometer dependencies to facilitate the injection of micrometer-registries with SCSt apps.
==== Building and Running the Demo
. Register `time` and `log` applications that are pre-built with `io.micrometer:micrometer-registry-influx`. The next version of https://start-scs.cfapps.io/[SCSt App Initializr] allows adding Micrometer registry dependencies as well.
+
```bash
app register --name time2 --type source --uri file://<path-to-your-time-app>/time-source-kafka-2.0.0.BUILD-SNAPSHOT.jar --metadata-uri file://<path-to-your-time-app>/time-source-kafka-2.0.0.BUILD-SNAPSHOT-metadata.jar
app register --name log2 --type sink --uri file://<path-to-your-log-app>/log-sink-kafka-2.0.0.BUILD-SNAPSHOT.jar --metadata-uri file://<path-to-your-log-app>/log-sink-kafka-2.0.0.BUILD-SNAPSHOT-metadata.jar
```
+
. Create InfluxDB and Grafana Docker containers
+
```bash
docker run -d --name grafana -p 3000:3000 grafana/grafana:5.1.0
docker run -d --name influxdb -p 8086:8086 influxdb:1.5.2-alpine
```
+
. Create and deploy the following stream
+
```bash
dataflow:>stream create --name t2 --definition "time2 | log2"
dataflow:>stream deploy --name t2 --properties "app.*.management.metrics.export.influx.db=myinfluxdb"
```
The `app.*.management.metrics.export.influx.db=myinfluxdb` instructs the `time2` and `log2` apps to use the `myinfluxdb` database (created automatically).
+
By default, the InfluxDB server runs on `http://localhost:8086`. You can add the `app.*.management.metrics.export.influx.uri={influxbb-server-url}` property to alter the default location.
+
You can connect to the InfluxDB and explore the measurements
+
[source,bash]
----
docker exec -it influxdb /bin/bash
root:/# influx
> show databases
> use myinfluxdb
> show measurements
> select * from spring_integration_send limit 10
----
+
. Configure Grafana
+
* Open Grafana UI (http://localhost:3000) and log-in (user: `admin`, password: `admin`).
* Create InfluxDB datasource called: `influx_auto_DataFlowMetricsCollector` that connects to our `myinfluxdb` influx database.
+
[.left]
image::grafana-influxdb-datasource.png[InfluxDB DataSource]
+
.DataSource Properties
[height="50%", width="60%",options=""]
|====================
| Name | influx_auto_DataFlowMetricsCollector
| Type | InfluxDB
| Host | http://localhost:8086
| Access | Browser
| Database | myinfluxdb
| User (DB) | admin
| Password (DB) | admin
|====================
+
NOTE: For previous `Grafana 4.x` set the `Access` property to `direct` instead.
+
* Import the link:micrometer/influx/scdf-influxdb-dashboard.json[scdf-influxdb-dashboard.json] dashboard
+
image::grafana-influx-dashboard.png[]
==== Summary
In this sample, you have learned:
* How to use Spring Cloud Data Flow's `Local` server
* How to use Spring Cloud Data Flow's `shell` application
* How to use `InfluxDB` and `Grafana` to monitor and visualize Spring Cloud Stream application metrics.

1688
src/main/asciidoc/micrometer/influx/scdf-influxdb-dashboard.json Normal file
View File

File diff suppressed because it is too large Load Diff

171
src/main/asciidoc/micrometer/prometheus/main.adoc Normal file
View File

@@ -0,0 +1,171 @@
[[spring-cloud-data-flow-samples-micrometer-prometheus-overview]]
:sectnums:
:docs_dir: ../..
=== SCDF metrics with Prometheus and Grafana
In this demonstration, you will learn how http://micrometer.io[Micrometer] can help to monitor your http://cloud.spring.io/spring-cloud-dataflow/[Spring Cloud Data Flow] Streams using http://prometheus.io[Prometheus] and https://grafana.com/grafana[Grafana].
Prometheus is time series database used for monitoring of highly dynamic service-oriented architectures. In a world of microservices, its support for multi-dimensional data collection and querying is a particular strength.
https://grafana.com/grafana[Grafana] is open source metrics Dashboard platform. It supports multiple backend time-series databases including Prometheus.
The architecture (Fig.1) builds on the https://docs.spring.io/spring-boot/docs/2.0.1.RELEASE/reference/htmlsingle/#production-ready-metrics-getting-started[Spring Boot Micrometer] functionality. When a http://micrometer.io/docs/registry/prometheus[micrometer-registry-prometheus] dependency is found on the classpath the Spring Boot auto-configures the metrics export for `Prometheus`.
The https://cloud.spring.io/spring-cloud-stream-app-starters/[Spring Cloud Stream] (SCSt) applications inherit the mircometer functionality, allowing them to compute and send various application metrics to the configured time-series database.
image::scdf-micrometer-prometheus-grafana-architecture.png[title="SCDF metrics analyzis with Prometheus and Grafana"]
Out of the box, SCSt sends https://docs.spring.io/spring-boot/docs/2.0.1.RELEASE/reference/htmlsingle/#production-ready-metrics-meter[core metrics] such as `CPU`, `Memory`, `MVC` and `Health` to name some. Among those the https://docs.spring.io/spring-integration/docs/current/reference/html/system-management-chapter.html#micrometer-integration[Spring Integration metrics] allows computing the `Rate` and the `Latency` of the messages in the SCDF streams.
NOTE: Unlike Spring Cloud Data Flow Metrics Collector, metrics here are sent synchronously over HTTP not through a Binder channel topic.
All Spring Cloud Stream App Starers enrich the standard http://micrometer.io/docs/concepts#_supported_monitoring_systems[dimensional tags] with the following SCDF specific tags:
[width="100%",options="header"]
|====================
| tag name | SCDF property | default value
| stream.name | spring.cloud.dataflow.stream.name | unknown
| application.name | spring.cloud.dataflow.stream.app.label | unknown
| instance.index | instance.index | 0
| application.guid | spring.cloud.application.guid | unknown
| application.gype | spring.cloud.dataflow.stream.app.type | unknown
|====================
NOTE: For custom app starters that don't extend from the https://github.com/spring-cloud-stream-app-starters/core[core] parent, you should add the `app-starters-common` : `org.springframework.cloud.stream.app` dependency to enable the SCDF tags.
Prometheus employs the pull-metrics model, called metrics scraping. Spring Boot provides an actuator endpoint available at `/actuator/prometheus` to present a Prometheus scrape with the appropriate format.
Furthermore Prometheus requires a mechanism to discover the target applications to be monitored (e.g. the URLs of the SCSt app instances). Targets may be statically configured via the `static_configs` parameter or dynamically discovered using one of the supported service-discovery mechanisms.
The https://github.com/tzolov/spring-cloud-dataflow-prometheus-service-discovery[SCDF Prometheus Service Discovery] is a standalone (Spring Boot) service, that uses the https://goo.gl/kE4eLV[runtime/apps] endpoint to retrieve the URLs of the running SCDF applications and generate `targets.json` file. The targets.json file is compliant with the https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E[<file_sd_config>] Prometheus discovery format.
Below we will present the steps to prepare, configure the demo of Spring Cloud Data Flow's `Local` server integration with `Prometheus`. For other deployment environment, such as `Cloud Foundry` or `Kubernetes`, additional configurations might be required.
==== Prerequisites
* A Running Data Flow Shell
include::{docs_dir}/shell.adoc[]
* A running local Data Flow Server
include::{docs_dir}/local-server.adoc[]
* Running instance of link:http://kafka.apache.org/downloads.html[Kafka]
* Spring Cloud Stream 2.x based https://github.com/spring-cloud-stream-app-starters/time/blob/master/spring-cloud-starter-stream-source-time/README.adoc[Time] and https://github.com/spring-cloud-stream-app-starters/log/blob/master/spring-cloud-starter-stream-sink-log/README.adoc[Log] applications starters, pre-built with `io.micrometer:micrometer-registry-prometheus` dependency.
+
NOTE: Next versions of the https://start-scs.cfapps.io/[SCSt App Initializr] utility would add support for Micrometer dependencies to facilitate the injection of micrometer-registries with SCSt apps.
==== Building and Running the Demo
. Register `time` and `log` applications that are pre-built with `io.micrometer:micrometer-registry-prometheus`. The next version of https://start-scs.cfapps.io/[SCSt App Initializr] allows adding Micrometer registry dependencies as well.
+
```bash
app register --name time2 --type source --uri file://<path-to-your-time-app>/time-source-kafka-2.0.0.BUILD-SNAPSHOT.jar --metadata-uri file://<path-to-your-time-app>/time-source-kafka-2.0.0.BUILD-SNAPSHOT-metadata.jar
app register --name log2 --type sink --uri file://<path-to-your-log-app>/log-sink-kafka-2.0.0.BUILD-SNAPSHOT.jar --metadata-uri file://<path-to-your-log-app>/log-sink-kafka-2.0.0.BUILD-SNAPSHOT-metadata.jar
```
+
. Create and deploy the following stream
+
```bash
dataflow:>stream create --name t2 --definition "time2 | log2"
dataflow:>stream deploy --name t2 --properties "app.*.management.endpoints.web.exposure.include=prometheus,app.*.spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration"
```
The deployment properties make sure that the prometheus actuator is enabled and the Spring Boot security is disabled
+
. Build and start the SCDF Prometheus Service Discovery application
+
Build the spring-cloud-dataflow-prometheus-service-discover project form: https://github.com/spring-cloud/spring-cloud-dataflow-samples/micrometer/spring-cloud-dataflow-prometheus-service-discovery
+
```bash
cd ./spring-cloud-dataflow-samples/micrometer/spring-cloud-dataflow-prometheus-service-discovery
./mvnw clean install
```
For convenience, the final https://github.com/spring-cloud/spring-cloud-dataflow-samples/raw/master/src/main/asciidoc/micrometer/prometheus/spring-cloud-dataflow-prometheus-service-discovery-0.0.1-SNAPSHOT.jar[spring-cloud-dataflow-prometheus-service-discovery-0.0.1-SNAPSHOT.jar] artifact is provided with this sample.
+
Start the service discovery application:
+
```bash
java -jar ./target/spring-cloud-dataflow-prometheus-service-discovery-0.0.1-SNAPSHOT.jar \
--metrics.prometheus.target.discovery.url=http://localhost:9393/runtime/apps \
--metrics.prometheus.target.file.path=/tmp/targets.json \
--metrics.prometheus.target.refresh.rate=10000 \
--metrics.prometheus.target.mode=local
```
+
It will connect to the SCDF runtime url, and generates /tmp/targets.json files every 10 sec.
+
. Create Prometheus configuration file (prometheus-local-file.yml)
+
```yaml
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'scdf'
metrics_path: '/actuator/prometheus'
file_sd_configs:
- files:
- targets.json
refresh_interval: 30s
```
+
Configure the file_sd_config discovery mechanism using the generated targets.json:
+
. Start Prometheus
+
```bash
docker run -d --name prometheus \
-p 9090:9090 \
-v <full-path-to>/prometheus-local-file.yml:/etc/prometheus/prometheus.yml \
-v /tmp/targets.json:/etc/prometheus/targets.json \
prom/prometheus:v2.2.1
```
+
Pass the prometheus.yml and map the /tmp/targets.json into /etc/prometheus/targets.json
+
Use the management UI: http://localhost:9090/graph to verify that SCDF apps metrics have been collected:
+
```
# Throughput
rate(spring_integration_send_seconds_count{type="channel"}[60s])
# Latency
rate(spring_integration_send_seconds_sum{type="channel"}[60s])/rate(spring_integration_send_seconds_count{type="channel"}[60s])
```
+
. Start Grafana Docker containers
+
```bash
docker run -d --name grafana -p 3000:3000 grafana/grafana:5.1.0
```
+
. Configure Grafana
+
* Open Grafana UI (http://localhost:3000) and log-in (user: `admin`, password: `admin`).
* Create Prometheus datasource called: `ScdfPrometheus`
+
.DataSource Properties
[width="60%",options=""]
|====================
| Name | ScdfPrometheus
| Type | Prometheus
| Host | http://localhost:9090
| Access | Browser
|====================
+
NOTE: For previous `Grafana 4.x` set the `Access` property to `direct` instead.
+
* Import the link:micrometer/prometheus/scdf-prometheus-grafana-dashboard.json[scdf-prometheus-grafana-dashboard.json] dashboard
+
image::grafana-prometheus-dashboard.png[]
==== Summary
In this sample, you have learned:
* How to use Spring Cloud Data Flow's `Local` server
* How to use Spring Cloud Data Flow's `shell` application
* How to use `Prometheus` and `Grafana` to monitor and visualize Spring Cloud Stream application metrics.

718
src/main/asciidoc/micrometer/prometheus/scdf-prometheus-grafana-dashboard.json Normal file
View File

@@ -0,0 +1,718 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"hideControls": false,
"id": 2,
"links": [
{
"icon": "external link",
"tags": [],
"type": "dashboards"
}
],
"refresh": "10s",
"rows": [
{
"collapse": false,
"height": 282,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "ScdfPrometheus",
"fill": 1,
"id": 1,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(spring_integration_send_seconds_sum{type=\"channel\",name!=\"errorChannel\",name!=\"nullChannel\",result=\"success\",stream_name=\"$stream_name\",application_name=~\"$application_name\",application_guid=~\"$application_guid\"}[60s])/rate(spring_integration_send_seconds_count{type=\"channel\",name!=\"errorChannel\",name!=\"nullChannel\",result=\"success\",stream_name=\"$stream_name\",application_name=~\"$application_name\"}[60s])",
"format": "time_series",
"instant": false,
"intervalFactor": 2,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Channel Latency",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "s",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "ScdfPrometheus",
"fill": 1,
"id": 2,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(spring_integration_send_seconds_count{type=\"channel\",name!=\"errorChannel\",name!=\"nullChannel\",result=\"success\",stream_name=~\"$stream_name\",application_name=~\"$application_name\",application_guid=~\"$application_guid\"}[60s])",
"format": "time_series",
"intervalFactor": 2,
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Channel Throughput",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
}
],
"repeat": null,
"repeatIteration": null,
"repeatRowId": null,
"showTitle": true,
"title": "Channels (success)",
"titleSize": "h4"
},
{
"collapse": false,
"height": 281,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "ScdfPrometheus",
"fill": 1,
"id": 4,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(spring_integration_send_seconds_sum{type=\"channel\",result=\"failure\",name!=\"errorChannel\",name!=\"nullChannel\",stream_name=\"$stream_name\",application_name=~\"$application_name\",application_guid=~\"$application_guid\"}[60s])/rate(spring_integration_send_seconds_count{type=\"channel\",result=\"failure\",name!=\"errorChannel\",name!=\"nullChannel\",stream_name=\"$stream_name\",application_name=~\"$application_name\"}[60s])",
"format": "time_series",
"instant": false,
"intervalFactor": 2,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Channel Latency",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "s",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "ScdfPrometheus",
"fill": 1,
"id": 5,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(spring_integration_send_seconds_count{type=\"channel\",result=\"failure\",name!=\"errorChannel\",name!=\"nullChannel\",stream_name=~\"$stream_name\",application_name=~\"$application_name\",application_guid=~\"$application_guid\"}[60s])",
"format": "time_series",
"intervalFactor": 2,
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Channel Throughput",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
}
],
"repeat": null,
"repeatIteration": null,
"repeatRowId": null,
"showTitle": true,
"title": "Channels (failure)",
"titleSize": "h4"
},
{
"collapse": true,
"height": 250,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "ScdfPrometheus",
"fill": 1,
"id": 7,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(spring_integration_send_seconds_sum{type=\"handler\",result=\"success\",stream_name=\"$stream_name\",application_name=~\"$application_name\",application_guid=~\"$application_guid\"}[60s])/rate(spring_integration_send_seconds_count{type=\"handler\",result=\"success\",stream_name=\"$stream_name\",application_name=~\"$application_name\"}[60s])",
"format": "time_series",
"instant": false,
"intervalFactor": 2,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Handler Latency",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "s",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "ScdfPrometheus",
"fill": 1,
"id": 8,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(spring_integration_send_seconds_count{type=\"handler\",result=\"success\",stream_name=~\"$stream_name\",application_name=~\"$application_name\",application_guid=~\"$application_guid\"}[60s])",
"format": "time_series",
"intervalFactor": 2,
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Handler Throughput",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
}
],
"repeat": null,
"repeatIteration": null,
"repeatRowId": null,
"showTitle": true,
"title": "Handlers",
"titleSize": "h4"
},
{
"collapse": true,
"height": 250,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "ScdfPrometheus",
"fill": 1,
"id": 10,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"span": 6,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(spring_integration_send_seconds_count{type=\"channel\",name=\"errorChannel\",stream_name=~\"$stream_name\",application_name=~\"$application_name\",application_guid=~\"$application_guid\"}[60s])",
"format": "time_series",
"intervalFactor": 2,
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Error Channel Throughput",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
]
}
],
"repeat": null,
"repeatIteration": null,
"repeatRowId": null,
"showTitle": true,
"title": "Error Channels",
"titleSize": "h4"
}
],
"schemaVersion": 14,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"allValue": null,
"current": {
"text": "t2",
"value": "t2"
},
"datasource": "ScdfPrometheus",
"hide": 0,
"includeAll": false,
"label": "Stream",
"multi": false,
"name": "stream_name",
"options": [],
"query": "label_values(stream_name)",
"refresh": 2,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "time2 + log2",
"value": [
"time2",
"log2"
]
},
"datasource": "ScdfPrometheus",
"hide": 0,
"includeAll": true,
"label": "Application",
"multi": true,
"name": "application_name",
"options": [],
"query": "label_values({stream_name=\"$stream_name\"},application_name)",
"refresh": 2,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"selected": true,
"text": "8082",
"value": "8082"
},
"datasource": "ScdfPrometheus",
"hide": 0,
"includeAll": false,
"label": "ApplicationGuid",
"multi": true,
"name": "application_guid",
"options": [],
"query": "label_values({stream_name=~\"$stream_name\",application_name=~\"$application_name\"},application_guid)",
"refresh": 2,
"regex": "",
"sort": 0,
"tagValuesQuery": "label_values({stream_name=~\"$stream_name\",application_name=~\"$tag\"},application_guid)",
"tags": [
"time2",
"log2"
],
"tagsQuery": "label_values({stream_name=~\"$stream_name\"},application_name)",
"type": "query",
"useTags": true
}
]
},
"time": {
"from": "now-15m",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "SCDF - Prometheus Dashboard",
"version": 19
}

BIN
src/main/asciidoc/micrometer/prometheus/spring-cloud-dataflow-prometheus-service-discovery-0.0.1-SNAPSHOT.jar Normal file
View File

Binary file not shown.

4
src/main/asciidoc/overview.adoc
View File

@@ -28,3 +28,7 @@ include::datascience/species-prediction/main.adoc[]
== Functions
include::functions/main.adoc[]
== Micrometer
include::micrometer/influx/main.adoc[]
include::micrometer/prometheus/main.adoc[]