This code is in the process of migrating into the RHOSO EDPM Ansible repo. Currently it exists on a fork for testing pending preparations for downstream delivery, here: https://github.com/rhos-vaf/edpm-ansible/tree/rhos-vaf/roles/edpm_telemetry_accel_exporters
This repo should be ready to be archived.
This Ansible role installs and configures the NVIDIA DCGM Exporter for GPU monitoring and metrics collection.
- Ansible 2.14 or higher
- Target systems must have:
- Podman installed
- Network access to pull container images
| Variable | Default | Description |
|---|---|---|
dcgm_exporter_nvidia_repo_url |
"https://developer.download.nvidia.com/compute/cuda/repos/rhel9" |
Base URL for NVIDIA repo |
dcgm_exporter_image_tag |
"4.3.1-4.4.0-ubi9" |
Container image tag for the DCGM Exporter |
dcgm_exporter_libnvidia_container_toolkit_version_release |
"1.17.8-1" |
Version of the NVIDIA Container Tools RPM to install |
dcgm_exporter_nvidia_driver_module_version |
"open-dkms" |
NVIDIA Driver version to install if not already present |
None.
---
- hosts: gpu_servers
roles:
- nvidia-dcgm-exporterWith custom image tag:
---
- hosts: gpu_servers
vars:
dcgm_exporter_image_tag: "4.3.1-4.4.0-ubi9"
roles:
- nvidia-dcgm-exporter- Installs NVIDIA Driver: Installs nvidia-driver from the NVIDIA repository if not already installed
- Installs NVIDIA Management Library: Installs the libnvidia-ml RPM that matches the installed nvidia-driver version
- Installs NVIDIA Container Tools: Installs the libnvidia-container-tools RPM for container GPU support
- Updates CDI Configuration: Adds the required mount configuration to
/etc/cdi/nvidia.yamlfor nvidia-ml library access - Creates Systemd Service: Creates a systemd service file that manages the DCGM Exporter container
- Manages Service: Enables and starts the DCGM Exporter service
The role runs the DCGM Exporter in a container with the following configuration:
- Image:
nvcr.io/nvidia/k8s/dcgm-exporter:{{ dcgm_exporter_image_tag }} - Port: 9400 (Prometheus metrics endpoint)
- Devices: All NVIDIA GPUs (
nvidia.com/gpu=all) - Capabilities: SYS_ADMIN
- Environment: DCGM_EXPORTER_DEBUG=true
The role creates a systemd service that:
- Automatically starts on boot
- Restarts on failure
- Uses podman to manage the container lifecycle
Once deployed, metrics will be available at http://your-server:9400/metrics