Dataproc metrics

Cloud Monitoring provides visibility into the performance, uptime, and overall health of cloud-powered applications. Google Cloud Observability collects and ingests metrics, events, and metadata from Dataproc clusters, including per-cluster HDFS, YARN, job, and operation metrics, to generate insights via dashboards and charts (see Cloud Monitoring Dataproc metrics).

• See Cloud Monitoring Pricing to understand your costs.

• See Monitoring Quotas and limits for information on metric data retention.

Dataproc resource metric collection

Cloud Monitoring collects metrics related to the following Dataproc resources:

• Cloud Dataproc Cluster
• Cloud Dataproc Job
• Cloud Dataproc Batch
• Cloud Dataproc Session

Dataproc resource metrics are collected in the following format: dataproc.googleapis.com/RESOURCE/METRIC, and include the collection of several OSS metrics.

View Dataproc resource metrics

You can select and view the Dataproc resource metrics in the Metrics Explorer by typing "dataproc" in the Filter by resource or metric name box, then selecting a "Cloud Dataproc" resource.

Custom metric collection

When you create a Dataproc cluster, you can enable the collection of metrics from one or more custom metric sources. A standard set of metrics is collected from each enabled metric source unless you specify the metrics to collect from a metric source (user-specified metrics are called metric "overrides").

Custom OSS metrics are collected in the following format: custom.googleapis.com/OSS_COMPONENT/METRIC

Custom OSS metric examples:

custom.googleapis.com/spark/driver/DAGScheduler/job/allJobs
custom.googleapis.com/hiveserver2/memory/MaxNonHeapMemory

Enable custom metric collection

You can use the gcloud CLI or the Dataproc API to enable the collection of custom metrics from one or more metric sources.

gcloud dataproc clusters create cluster-name \
    --metric-sources=METRIC_SOURCE(s) \
    ... other flags

gcloud dataproc clusters create cluster-name \
    --metric-sources=METRIC_SOURCE(s) \
    --metric-overrides=LIST_OF_METRIC_OVERRIDES \
    ... other flags

gcloud dataproc clusters create cluster-name \
    --metric-sources=METRIC-SOURCE(s) \
    --metric-overrides-file=METRIC_OVERRIDES_FILENAME \
    ... other flags

View custom metrics

You can select and view the Dataproc resource metrics in the Metrics Explorer by selecting the VM Instanceresource, then selecting Custom metrics.

Custom metrics

You can enable Dataproc to collect the custom metrics listed in the following tables.

• The Enabled metrics column is marked with "y" if Dataproc collects the metric when you enable the associated metric source.

• Any of the metrics listed for a metric source, and all Spark metrics), can be enabled for collection if you override the collection of the standard set of enabled metrics for the metric source (see Enable custom metric collection).

• Dataproc uses the monitoring agent to collect metrics. Enabling any metric source enables the collection of agent metrics. These metrics are not billed to users; Dataproc uses them to diagnose metric collection issues.

Hadoop metrics

HDFS metrics

YARN metrics

Spark metrics

Spark driver metrics

Spark executor metrics

Flink metrics

Spark History Server metrics

Dataproc collects the following Spark history service JVM memory metrics:

HiveServer 2 metrics

Hive Metastore metrics

Hive Metastore metric measures

Dataproc monitoring agent metrics

Dataproc collects the following Dataproc monitoring agent metrics when you set --metric-sources=monitoring-agent-defaults. These metrics are published with the agent.googleapis.com prefix.

CPU
agent.googleapis.com/cpu/load_15m
agent.googleapis.com/cpu/load_1m
agent.googleapis.com/cpu/load_5m
agent.googleapis.com/cpu/usage_time*
agent.googleapis.com/cpu/utilization*

Disk
agent.googleapis.com/disk/bytes_used
agent.googleapis.com/disk/io_time
agent.googleapis.com/disk/merged_operations
agent.googleapis.com/disk/operation_count
agent.googleapis.com/disk/operation_time
agent.googleapis.com/disk/pending_operations
agent.googleapis.com/disk/percent_used
agent.googleapis.com/disk/read_bytes_count

Swap
agent.googleapis.com/swap/bytes_used
agent.googleapis.com/swap/io
agent.googleapis.com/swap/percent_used

Memory
agent.googleapis.com/memory/bytes_used
agent.googleapis.com/memory/percent_used

Processes - (follows slightly different quota policy for few attributes)
agent.googleapis.com/processes/count_by_state
agent.googleapis.com/processes/cpu_time
agent.googleapis.com/processes/disk/read_bytes_count
agent.googleapis.com/processes/disk/write_bytes_count
agent.googleapis.com/processes/fork_count
agent.googleapis.com/processes/rss_usage
agent.googleapis.com/processes/vm_usage

Interface
agent.googleapis.com/interface/errors
agent.googleapis.com/interface/packets
agent.googleapis.com/interface/traffic

Network
agent.googleapis.com/network/tcp_connections

Build a Monitoring dashboard

You can build a Monitoring dashboard that displays charts of selected Dataproc metrics.

1. Select + CREATE DASHBOARD from the Monitoring Dashboards Overview page. Provide a name for the dashboard, then click Add Chart in the upper-right menu to open the Add Chart window. Select "Cloud Dataproc Cluster" as the resource type. Select one or more metrics and metric and chart properties. Then Save the chart.

   

2. You can add additional charts to your dashboard. After you Save the dashboard, its title appears in the Monitoring Dashboards Overview page. Dashboard charts can be viewed, updated, and deleted from the dashboard display page.

   

Whats next

• See the Cloud Monitoring documentation
• Learn how to Create Dataproc metric alerts