-
Notifications
You must be signed in to change notification settings - Fork 7.9k
Envoy native telemetry
It is possible to get a subset of the full istio telemetry thru envoy only metrics. Note: by Istio 1.3/1.4 the proxy will emit full istio telemetry.
Envoy native telemetry has the following feature gaps compared to Istio standard metrics.
- Configurability
- mTLS indicator
- Peer service account
- Source information
Istio 1.2 supports stats annotations to specify metrics to be collected by Envoy.
Annotation | Description |
---|---|
sidecar.istio.io/statsInclusionPrefixes | Specifies the comma separated list of prefixes of the stats to be emitted by Envoy. |
sidecar.istio.io/statsInclusionRegexps | Specifies the comma separated list of regexes the stats should match to be emitted by Envoy. |
sidecar.istio.io/statsInclusionSuffixes | Specifies the comma separated list of suffixes of the stats to be emitted by Envoy. |
Envoy native metrics provide good client side metrics. The source is fully specified by the prometheus scraper and the destination information is encoded in the upstream cluster name.
The following stats set at ingress gateway gives a useful set of metrics. You can compute clientside traffic-by-responsecode and associated latency histograms. It also gives you downstream metrics by port. Since ingress gateway does not use the sidecar injector, you have to configure it as follows using environment variables.
containers:
- name: istio-proxy
env:
- name: ISTIO_METAJSON_STATS
value: |
{"sidecar.istio.io/statsInclusionSuffixes": "upstream_rq_1xx,upstream_rq_2xx,upstream_rq_3xx,upstream_rq_4xx,upstream_rq_5xx,upstream_rq_time,upstream_cx_tx_bytes_total,upstream_cx_rx_bytes_total,upstream_cx_total,downstream_rq_1xx,downstream_rq_2xx,downstream_rq_3xx,downstream_rq_4xx,downstream_rq_5xx,downstream_rq_time,downstream_cx_tx_bytes_total,downstream_cx_rx_bytes_total,downstream_cx_total"}
You can import a sample dashboard based on the above configuration, it should look like this
For a sidecar we collect the full set of upstream metrics. However, only remote downstream metrics are usually of interest. You can configure them as follows thru an annotation.
"sidecar.istio.io/statsInclusionSuffixes": "upstream_rq_1xx,upstream_rq_2xx,upstream_rq_3xx,upstream_rq_4xx,upstream_rq_5xx,upstream_rq_time,upstream_cx_tx_bytes_total,upstream_cx_rx_bytes_total,upstream_cx_total"
"sidecar.istio.io/statsInclusionPrefixes": "http.{pod_ip}_"
The {pod_ip}
is automatically substituted during bootstrap config generation.
1.1 does not automatically add required
stats and does not support suffixes.
Add the following annotation to a deployment where you want to collect metrics.
"sidecar.istio.io/statsInclusionPrefixes": "http.,tcp.,cluster.,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpc,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpc"
For gateways add the equivalent env
variables as shown in a previous section.
Visit istio.io to learn how to use Istio.
- Preparing for Development Mac
- Preparing for Development Linux
- Troubleshooting Development Environment
- Repository Map
- GitHub Workflow
- Github Gmail Filters
- Using the Code Base
- Developing with Minikube
- Remote Debugging
- Verify your Docker Environment
- Istio Test Framework
- Working with Prow
- Test Grid
- Code Coverage FAQ
- Writing Good Integration Tests
- Test Flakes
- Release Manager Expectations
- Preparing Istio Releases
- 1.5 Release Information
- 1.6 Release Information
- 1.7 Release Information
- 1.8 Release Information
- 1.9 Release Information
- 1.10 Release Information
- 1.11 Release Information
- 1.12 Release Information
- 1.13 Release Information
- 1.14 Release Information
- 1.15 Release Information
- 1.16 Release Information
- 1.17 Release Information
- 1.18 Release Information
- 1.19 Release Information
- 1.20 Release Information
- 1.21 Release Information
- 1.22 Release Information
- 1.23 Release Information
- 1.24 Release Information