Why Is GPU Utilization Zero or Low?

Background

RealTheory shows GPU requests and limits for every cluster, from Pod specs and Node capacity. GPU utilization comes from NVIDIA DCGM Exporter, which runs on each GPU Node. On the Metrics tab, GPU utilization appears as GPU Usage and GPU Percentage.

The RealTheory Collector, the agent that runs in your cluster, reads the DCGM Exporter metrics when GPU collection is on. GPU metrics for Pods, Services, workloads, and Namespaces also require DCGM Exporter to run in Kubernetes mode.

Because of this, GPU requests and limits might appear while GPU utilization shows as zero, is missing, or looks low.

GPU Utilization Is Zero on Every GPU Node

Cause: One of the following:

DCGM Exporter is not installed on the cluster.
GPU collection is off in the Collector. GPU collection is off by default.
THEORY_SCRAPER_DCGM_NAMESPACE or THEORY_SCRAPER_DCGM_LABEL_SELECTOR does not match the DCGM Exporter Pods.
The Collector runs from a newly generated Collector deployment YAML file that does not include the GPU environment variables, for example after a reinstall.
The cluster runs on GKE Autopilot, where the Collector does not collect GPU utilization.

Solution:

Confirm that one DCGM Exporter Pod runs on each GPU Node. See Step 1 in Enabling GPU Metric Collection.
Confirm that the Collector log shows Scraper->DCGMEnabled: true. See Step 4 in Enabling GPU Metric Collection.
Check that THEORY_SCRAPER_DCGM_NAMESPACE and THEORY_SCRAPER_DCGM_LABEL_SELECTOR match your DCGM Exporter Pods. See Step 3 in Enabling GPU Metric Collection.
If you reinstalled the Collector, add the GPU environment variables again. See Step 3 in Enabling GPU Metric Collection.

Note: GKE Autopilot has no fix for this cause. On Autopilot clusters, RealTheory shows GPU requests and limits only.

Pod GPU Utilization Is Zero While Node GPU Utilization Appears

Cause: DCGM Exporter does not label its metrics with Pods, so GPU metrics for Pods, Services, workloads, and Namespaces show as zero. Either DCGM Exporter is not running in Kubernetes mode, or --kubernetes-gpu-id-type does not match your device plugin.

Solution: Run the check that DCGM Exporter emits Pod labels, in Step 2 of Enabling GPU Metric Collection. If the check prints nothing, turn on Kubernetes mode and set --kubernetes-gpu-id-type as Step 2 of that topic describes.

GPU Utilization Is Missing on Some GPU Nodes

Cause: No DCGM Exporter Pod runs on those GPU Nodes.

Solution: Update the node selectors and tolerations on the DCGM Exporter DaemonSet so that the DaemonSet schedules a Pod on every GPU Node.

GPU Utilization Looks Low for Busy GPUs

Cause: DCGM Exporter reports the default utilization metric, an instantaneous sample that under-reports bursty workloads such as inference.

Solution: Add the profiling counters that Step 2 of Enabling GPU Metric Collection describes. When the averaged counters are available, the Collector uses them instead of the instantaneous sample.