Enabling GPU Metric Collection

The RealTheory Collector, the agent that runs in your cluster, collects NVIDIA GPU metrics. RealTheory shows these metrics for Nodes, Pods, Services, workloads, Namespaces, and the whole cluster. To collect GPU metrics, you install NVIDIA DCGM Exporter on your GPU Nodes and turn on GPU collection in the Collector.

GPU Data Available at Each Setup Level

The GPU data that RealTheory shows depends on how far you take the setup.

Setup	GPU data in RealTheory
Collector only, without DCGM Exporter	GPU requests and limits, from Pod specs and Node capacity, plus GPU memory requests and limits for MIG. RealTheory estimates the physical GPU card count for cost reporting.
DCGM Exporter with default settings	An exact physical GPU card count, plus GPU Usage, GPU Percentage, GPU Memory, and GPU Memory Percentage for Nodes and the cluster, in addition to requests and limits.
DCGM Exporter in Kubernetes mode	The same GPU metrics for Pods, Services, workloads, and Namespaces, in addition to all Node-level data.

Prerequisites

Nodes with NVIDIA GPUs.
The NVIDIA device plugin, which advertises nvidia.com/gpu or MIG resources on each Node.
Access to Pods through the Kubernetes API server proxy. The Collector reaches each DCGM Exporter Pod through the API server, so you do not open any extra network paths. The Collector's standard permissions include this access.

Note: On GKE Autopilot, the Collector does not collect DCGM Exporter metrics because Autopilot blocks Pod access through the API server proxy. RealTheory still shows GPU requests and limits for Autopilot clusters.

Step 1: Install NVIDIA DCGM Exporter

DCGM Exporter runs on every GPU Node and serves GPU metrics on port 9400 by default. The Collector finds DCGM Exporter automatically by the label on its Pods, and each install method applies its own default label. The NVIDIA GPU Operator is the recommended install method.

Do one of the following:
- To install the GPU driver, device plugin, and DCGM Exporter together, install the NVIDIA GPU Operator. The GPU Operator installs into the gpu-operator Namespace and labels DCGM Exporter Pods app=nvidia-dcgm-exporter.
- To install DCGM Exporter on its own, install the dcgm-exporter Helm chart. The Helm chart labels DCGM Exporter Pods app.kubernetes.io/name=dcgm-exporter.
- To control the configuration yourself, deploy DCGM Exporter as your own DaemonSet and label its Pods app=dcgm-exporter.
Confirm that one DCGM Exporter Pod runs on each GPU Node:
```
kubectl get pods -A -l app=nvidia-dcgm-exporter -o wide
```
Note: If you did not use the GPU Operator, replace app=nvidia-dcgm-exporter with the label for your install method: app.kubernetes.io/name=dcgm-exporter for the Helm chart, or app=dcgm-exporter for your own DaemonSet.

Step 2: Configure DCGM Exporter

Node-level metrics work with the default DCGM Exporter settings. Pod-level GPU metrics and accurate figures for bursty workloads each need one more setting.

Kubernetes mode for Pod-level GPU metrics: DCGM Exporter must label each metric with the Pod that owns the GPU. The GPU Operator turns on Kubernetes mode by default. If you did not use the GPU Operator, configure Kubernetes mode:

Pass --kubernetes (or -k) to dcgm-exporter.
Mount /var/lib/kubelet/pod-resources into the DCGM Exporter container.
Set --kubernetes-gpu-id-type to match your device plugin:
- For the standard NVIDIA device plugin, use uid, the default.
- For some MIG and vGPU setups, use device-name.
Important: If --kubernetes-gpu-id-type does not match your device plugin, DCGM Exporter starts normally but emits no Pod labels.
Check that DCGM Exporter emits Pod labels:
1. Forward the metrics port of one DCGM Exporter Pod to your machine:
```
kubectl -n <namespace> port-forward <dcgm-exporter-pod> 9400:9400
```
  where <namespace> is the Namespace where DCGM Exporter runs, and <dcgm-exporter-pod> is the name of a DCGM Exporter Pod from the kubectl get pods output in Step 1. If DCGM Exporter serves metrics on a port other than 9400, replace 9400 in both commands.
2. In a second terminal, run:
```
curl -s localhost:9400/metrics | grep -m1 'pod='
```
Note: If the curl command prints nothing, GPU metrics for Pods, Services, workloads, and Namespaces show as zero in RealTheory.

Profiling counters for bursty workloads: The default utilization metric is an instantaneous sample, which under-reports bursty workloads such as inference. To measure these workloads accurately, run DCGM Exporter with a counters file that includes DCGM_FI_PROF_GR_ENGINE_ACTIVE and DCGM_FI_PROF_SM_ACTIVE, for example -f /etc/dcgm-exporter/counters.csv. The Collector uses these averaged counters instead of the instantaneous sample whenever DCGM Exporter provides them.

Step 3: Turn On GPU Collection in the RealTheory Collector

GPU collection is off by default. You turn it on with environment variables on the realtheorycollector container, in the realtheorycollector Deployment in the real-theory-system Namespace.

Variable	What it controls	Default
`THEORY_SCRAPER_DCGM_ENABLED`	Turns GPU collection on or off. Set to `true` to turn GPU collection on.	`false`
`THEORY_SCRAPER_DCGM_NAMESPACE`	The Namespace where DCGM Exporter runs. Set to an empty string to search all Namespaces.	`gpu-operator`
`THEORY_SCRAPER_DCGM_LABEL_SELECTOR`	Label selectors that identify DCGM Exporter Pods, separated by semicolons. The Collector includes Pods that match any selector.	`app=nvidia-dcgm-exporter;app=dcgm-exporter;app.kubernetes.io/name=dcgm-exporter`
`THEORY_SCRAPER_DCGM_PORT`	The port where DCGM Exporter serves metrics.	`9400`
`THEORY_SCRAPER_DCGM_CONCURRENCY`	The maximum number of DCGM Exporter Pods that the Collector scrapes at once, from 1 to 20. Raise this value for large GPU clusters.	`10`

Choose the environment variables for your setup. Do one of the following:
- If DCGM Exporter runs in the gpu-operator Namespace with its default labels and port, set only THEORY_SCRAPER_DCGM_ENABLED:
```
- name: THEORY_SCRAPER_DCGM_ENABLED
  value: "true"
```
- If DCGM Exporter runs in another Namespace or uses a custom label or port, also set the matching variables:
```
- name: THEORY_SCRAPER_DCGM_ENABLED
  value: "true"
- name: THEORY_SCRAPER_DCGM_NAMESPACE
  value: "monitoring"
- name: THEORY_SCRAPER_DCGM_LABEL_SELECTOR
  value: "app=nvidia-dcgm-exporter;app=dcgm-exporter;app.kubernetes.io/name=dcgm-exporter;team=gpu-metrics"
- name: THEORY_SCRAPER_DCGM_PORT
  value: "9500"
```
Important: In THEORY_SCRAPER_DCGM_LABEL_SELECTOR, add your label selector to the end of the default list, separated by a semicolon. If you replace the list, the Collector no longer finds DCGM Exporter Pods that carry the default labels.
Do one of the following:
- To use your Collector deployment YAML file, open the file that you downloaded or copied when you installed the Collector. See Installing RealTheory in a Cluster. Add the variables to the end of the existing env list of the realtheorycollector container. Then run kubectl apply -f <filename.yaml>, where <filename.yaml> is the path to that file. For example:
```
    spec:
      containers:
      - env:
        - name: THEORY_LOGGING_LEVEL
          value: Info
        # ...the other existing variables...
        - name: THEORY_SCRAPER_DCGM_ENABLED
          value: "true"
        image: realtheory/theory-k8s-collector
```
  Important: The realtheorycollector container already has an env list. Add the GPU variables to that list and do not add a second env key. Keep the quotation marks around "true" and port numbers because Kubernetes requires environment variable values to be strings.
- To set the variables on the running Collector Deployment without editing the YAML, run the following command. Add each other variable to the same command as a NAME=value pair.
```
kubectl -n real-theory-system set env deployment/realtheorycollector THEORY_SCRAPER_DCGM_ENABLED=true
```
  Important: Put quotation marks around a label selector value because the shell treats each semicolon as the end of a command. For example: THEORY_SCRAPER_DCGM_LABEL_SELECTOR="app=nvidia-dcgm-exporter;app=dcgm-exporter;app.kubernetes.io/name=dcgm-exporter;team=gpu-metrics"
Note: The Collector restarts with the new settings. If the Collector does not find DCGM Exporter right away, the Collector checks again every collection cycle, so a DCGM Exporter that you install later needs no Collector restart.

Important: The Collector deployment YAML that the Console generates does not include the GPU environment variables. Before you apply a newly generated Collector deployment YAML file, for example to reinstall the Collector, add the GPU environment variables to that file.

Step 4: Verify GPU Metric Collection

Check that the Collector loaded the GPU settings:
```
kubectl -n real-theory-system logs deployment/realtheorycollector | grep DCGM
```
The output includes Scraper->DCGMEnabled: true.
After a few collection cycles, open the Information panel for a GPU Node in RealTheory. On the Metrics tab, check that GPU Usage, GPU Percentage, GPU Memory, and GPU Memory Percentage appear.

Note: Pods, Services, workloads, and Namespaces show GPU metrics only when DCGM Exporter runs in Kubernetes mode.

Known Limitations of GPU Metric Collection

Shared GPUs (time-slicing or MPS): For each Pod that shares a GPU, RealTheory shows the total utilization of that GPU. Per-Pod figures overstate usage and do not add up to the Node total.
MIG: RealTheory weights MIG utilization for a 7-slice card, such as A100 or H100.