feat(dcgm-exporter): expose per-pod GPU util config in ClusterPolicy

[zbennett10]

Summary

Adds spec.dcgmExporter.perPodGPUUtil to the ClusterPolicy CRD, enabling
per-pod GPU SM utilization metrics when CUDA time-slicing is active.

This is the GPU Operator half of a two-part contribution that closes
NVIDIA/dcgm-exporter#638
(issue: NVIDIA/dcgm-exporter#587).

The problem

With GPU time-slicing, dcgm_fi_dev_gpu_util reports only aggregate device
utilization — you cannot tell how much of the GPU proxy, embeddings, or
inference pods are each consuming.

The fix

dcgm-exporter PR #638 adds an opt-in collector that uses NVML per-process
utilization + kubelet pod-resources gRPC to emit dcgm_fi_dev_sm_util_per_pod
per (pod, namespace, container, gpu_uuid) tuple.

This PR wires that feature through ClusterPolicy so users can enable it
without hand-editing the dcgm-exporter DaemonSet:

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  dcgmExporter:
    perPodGPUUtil:
      enabled: true
      # podResourcesSocketPath defaults to /var/lib/kubelet/pod-resources/kubelet.sock

What GPU Operator does automatically when enabled: true

1. Sets DCGM_EXPORTER_ENABLE_PER_POD_GPU_UTIL=true env var in the dcgm-exporter DaemonSet
2. Mounts /var/lib/kubelet/pod-resources/ as a read-only hostPath volume
3. Sets hostPID: true on the DaemonSet so dcgm-exporter can resolve /proc/<pid>/cgroup

Compatibility

GPU Operator	dcgm-exporter	Feature available
< v24.x	any	No
≥ v24.x	< v3.4.0	Field accepted but no-op
≥ v24.x	≥ v3.4.0	Yes

Security considerations

Enabling perPodGPUUtil grants dcgm-exporter:

• Read access to /var/lib/kubelet/pod-resources/ (lists all GPU-using pods)
• Host PID namespace access (to read /proc/<pid>/cgroup)

These are the same permissions used by other node-level monitoring agents.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[zbennett10]

@rajathagasthya can anyone take a look at this? Thanks

[rajathagasthya]

@zbennett10 Has this been merged upstream in DCGM Exporter? I see NVIDIA/dcgm-exporter#638 is still open.

[zbennett10]

@zbennett10 Has this been merged upstream in DCGM Exporter? I see NVIDIA/dcgm-exporter#638 is still open.

Nope - still waiting for a review there. Are you able to help with that @rajathagasthya ? Thanks :) figured I'd try

[rajathagasthya]

Sorry, we don't maintain that repo! You'd have to first get it merged there and then we can add support for it once there's a new release of DCGM Exporter.

[zbennett10]

Sorry, we don't maintain that repo! You'd have to first get it merged there and then we can add support for it once there's a new release of DCGM Exporter.

Sure was just wondering if you could help contact them about it since you're also from NVIDIA. A lot of radio silence over there it seems. Thanks @rajathagasthya

[rajathagasthya]

@zbennett10 I've poked some people internally to take a look at this, so hopefully there's some action on it soon!