--- title: "Enabling GPU Metric Collection" slug: "enabling-gpu-metric-collection" updated: 2026-10-29T19:28:28Z published: 2026-10-29T19:28:28Z canonical: "resources.realtheory.io/enabling-gpu-metric-collection" --- > ## Documentation Index > Fetch the complete documentation index at: https://resources.realtheory.io/llms.txt > Use this file to discover all available pages before exploring further. # Enabling GPU Metric Collection The RealTheory Collector, the agent that runs in your cluster, collects NVIDIA GPU metrics. RealTheory shows these metrics for Nodes, Pods, Services, workloads, Namespaces, and the whole cluster. To collect GPU metrics, you install NVIDIA DCGM Exporter on your GPU Nodes and turn on GPU collection in the Collector. ## GPU Data Available at Each Setup Level The GPU data that RealTheory shows depends on how far you take the setup. | Setup | GPU data in RealTheory | | --- | --- | | Collector only, without DCGM Exporter | GPU requests and limits, from Pod specs and Node capacity, plus GPU memory requests and limits for MIG. RealTheory estimates the physical GPU card count for cost reporting. | | DCGM Exporter with default settings | An exact physical GPU card count, plus **GPU Usage**, **GPU Percentage**, **GPU Memory**, and **GPU Memory Percentage** for Nodes and the cluster, in addition to requests and limits. | | DCGM Exporter in Kubernetes mode | The same GPU metrics for Pods, Services, workloads, and Namespaces, in addition to all Node-level data. | ## Prerequisites - Nodes with NVIDIA GPUs. - The NVIDIA device plugin, which advertises `nvidia.com/gpu` or MIG resources on each Node. - Access to Pods through the Kubernetes API server proxy. The Collector reaches each DCGM Exporter Pod through the API server, so you do not open any extra network paths. The Collector's standard permissions include this access. **Note:** On GKE Autopilot, the Collector does not collect DCGM Exporter metrics because Autopilot blocks Pod access through the API server proxy. RealTheory still shows GPU requests and limits for Autopilot clusters. ## Step 1: Install NVIDIA DCGM Exporter DCGM Exporter runs on every GPU Node and serves GPU metrics on port 9400 by default. The Collector finds DCGM Exporter automatically by the label on its Pods, and each install method applies its own default label. The NVIDIA GPU Operator is the recommended install method. 1. Do one of the following: - To install the GPU driver, device plugin, and DCGM Exporter together, install the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/). The GPU Operator installs into the `gpu-operator` Namespace and labels DCGM Exporter Pods `app=nvidia-dcgm-exporter`. - To install DCGM Exporter on its own, install the [dcgm-exporter Helm chart](https://github.com/NVIDIA/dcgm-exporter). The Helm chart labels DCGM Exporter Pods `app.kubernetes.io/name=dcgm-exporter`. - To control the configuration yourself, deploy DCGM Exporter as your own DaemonSet and label its Pods `app=dcgm-exporter`. 2. Confirm that one DCGM Exporter Pod runs on each GPU Node: ``` kubectl get pods -A -l app=nvidia-dcgm-exporter -o wide ``` **Note:** If you did not use the GPU Operator, replace `app=nvidia-dcgm-exporter` with the label for your install method: `app.kubernetes.io/name=dcgm-exporter` for the Helm chart, or `app=dcgm-exporter` for your own DaemonSet. ## Step 2: Configure DCGM Exporter Node-level metrics work with the default DCGM Exporter settings. Pod-level GPU metrics and accurate figures for bursty workloads each need one more setting. **Kubernetes mode for Pod-level GPU metrics:** DCGM Exporter must label each metric with the Pod that owns the GPU. The GPU Operator turns on Kubernetes mode by default. If you did not use the GPU Operator, configure Kubernetes mode: 1. Pass `--kubernetes` (or `-k`) to dcgm-exporter. 2. Mount `/var/lib/kubelet/pod-resources` into the DCGM Exporter container. 3. Set `--kubernetes-gpu-id-type` to match your device plugin: **Important:** If `--kubernetes-gpu-id-type` does not match your device plugin, DCGM Exporter starts normally but emits no Pod labels. - For the standard NVIDIA device plugin, use `uid`, the default. - For some MIG and vGPU setups, use `device-name`. 4. Check that DCGM Exporter emits Pod labels: **Note:** If the `curl` command prints nothing, GPU metrics for Pods, Services, workloads, and Namespaces show as zero in RealTheory. 1. Forward the metrics port of one DCGM Exporter Pod to your machine: ``` kubectl -n port-forward 9400:9400 ``` where `<namespace>` is the Namespace where DCGM Exporter runs, and `<dcgm-exporter-pod>` is the name of a DCGM Exporter Pod from the `kubectl get pods` output in Step 1. If DCGM Exporter serves metrics on a port other than 9400, replace `9400` in both commands. 2. In a second terminal, run: ``` curl -s localhost:9400/metrics | grep -m1 'pod=' ``` **Profiling counters for bursty workloads:** The default utilization metric is an instantaneous sample, which under-reports bursty workloads such as inference. To measure these workloads accurately, run DCGM Exporter with a counters file that includes `DCGM_FI_PROF_GR_ENGINE_ACTIVE` and `DCGM_FI_PROF_SM_ACTIVE`, for example `-f /etc/dcgm-exporter/counters.csv`. The Collector uses these averaged counters instead of the instantaneous sample whenever DCGM Exporter provides them. ## Step 3: Turn On GPU Collection in the RealTheory Collector GPU collection is off by default. You turn it on with environment variables on the `realtheorycollector` container, in the `realtheorycollector` Deployment in the `real-theory-system` Namespace. | Variable | What it controls | Default | | --- | --- | --- | | `THEORY_SCRAPER_DCGM_ENABLED` | Turns GPU collection on or off. Set to `true` to turn GPU collection on. | `false` | | `THEORY_SCRAPER_DCGM_NAMESPACE` | The Namespace where DCGM Exporter runs. Set to an empty string to search all Namespaces. | `gpu-operator` | | `THEORY_SCRAPER_DCGM_LABEL_SELECTOR` | Label selectors that identify DCGM Exporter Pods, separated by semicolons. The Collector includes Pods that match any selector. | `app=nvidia-dcgm-exporter;app=dcgm-exporter;app.kubernetes.io/name=dcgm-exporter` | | `THEORY_SCRAPER_DCGM_PORT` | The port where DCGM Exporter serves metrics. | `9400` | | `THEORY_SCRAPER_DCGM_CONCURRENCY` | The maximum number of DCGM Exporter Pods that the Collector scrapes at once, from 1 to 20. Raise this value for large GPU clusters. | `10` | 1. Choose the environment variables for your setup. Do one of the following: **Important:** In `THEORY_SCRAPER_DCGM_LABEL_SELECTOR`, add your label selector to the end of the default list, separated by a semicolon. If you replace the list, the Collector no longer finds DCGM Exporter Pods that carry the default labels. - If DCGM Exporter runs in the `gpu-operator` Namespace with its default labels and port, set only `THEORY_SCRAPER_DCGM_ENABLED`: ``` - name: THEORY_SCRAPER_DCGM_ENABLED value: "true" ``` - If DCGM Exporter runs in another Namespace or uses a custom label or port, also set the matching variables: ``` - name: THEORY_SCRAPER_DCGM_ENABLED value: "true" - name: THEORY_SCRAPER_DCGM_NAMESPACE value: "monitoring" - name: THEORY_SCRAPER_DCGM_LABEL_SELECTOR value: "app=nvidia-dcgm-exporter;app=dcgm-exporter;app.kubernetes.io/name=dcgm-exporter;team=gpu-metrics" - name: THEORY_SCRAPER_DCGM_PORT value: "9500" ``` 2. Do one of the following: **Note:** The Collector restarts with the new settings. If the Collector does not find DCGM Exporter right away, the Collector checks again every collection cycle, so a DCGM Exporter that you install later needs no Collector restart. **Important:** The Collector deployment YAML that the Console generates does not include the GPU environment variables. Before you apply a newly generated Collector deployment YAML file, for example to reinstall the Collector, add the GPU environment variables to that file. - To use your Collector deployment YAML file, open the file that you downloaded or copied when you installed the Collector. See [Installing RealTheory in a Cluster](https://resources.realtheory.io/docs/installing-realtheory-in-a-cluster). Add the variables to the end of the existing `env` list of the `realtheorycollector` container. Then run `kubectl apply -f <filename.yaml>`, where `<filename.yaml>` is the path to that file. For example: ``` spec: containers: - env: - name: THEORY_LOGGING_LEVEL value: Info # ...the other existing variables... - name: THEORY_SCRAPER_DCGM_ENABLED value: "true" image: realtheory/theory-k8s-collector ``` **Important:** The `realtheorycollector` container already has an `env` list. Add the GPU variables to that list and do not add a second `env` key. Keep the quotation marks around `"true"` and port numbers because Kubernetes requires environment variable values to be strings. - To set the variables on the running Collector Deployment without editing the YAML, run the following command. Add each other variable to the same command as a `NAME=value` pair. ``` kubectl -n real-theory-system set env deployment/realtheorycollector THEORY_SCRAPER_DCGM_ENABLED=true ``` **Important:** Put quotation marks around a label selector value because the shell treats each semicolon as the end of a command. For example: `THEORY_SCRAPER_DCGM_LABEL_SELECTOR="app=nvidia-dcgm-exporter;app=dcgm-exporter;app.kubernetes.io/name=dcgm-exporter;team=gpu-metrics"` ## Step 4: Verify GPU Metric Collection 1. Check that the Collector loaded the GPU settings: ``` kubectl -n real-theory-system logs deployment/realtheorycollector | grep DCGM ``` The output includes `Scraper->DCGMEnabled: true`. 2. After a few collection cycles, open the Information panel for a GPU Node in RealTheory. On the **Metrics** tab, check that **GPU Usage**, **GPU Percentage**, **GPU Memory**, and **GPU Memory Percentage** appear. **Note:** Pods, Services, workloads, and Namespaces show GPU metrics only when DCGM Exporter runs in Kubernetes mode. ## Known Limitations of GPU Metric Collection - **Shared GPUs (time-slicing or MPS):** For each Pod that shares a GPU, RealTheory shows the total utilization of that GPU. Per-Pod figures overstate usage and do not add up to the Node total. - **MIG:** RealTheory weights MIG utilization for a 7-slice card, such as A100 or H100. **See Also** [Why Is GPU Utilization Zero or Low?](/v1/docs/why-is-gpu-utilization-zero-or-low) [Installing RealTheory in a Cluster](/v1/docs/installing-realtheory-in-a-cluster)