feat: add support for gpu sharing metrics in k8s #432
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Fixes #307 |
|
Thank you for the contribution! We will test and review the PR in the coming weeks. Please be aware that we are on the verge of releasing a new 4.0 version of DCGM-Exporter so this PR will likely need to be updated after that happens. |
|
Thanks @glowkey - I actually may factor out the smaller bugfix in a separate PR:
That way, this change just focuses on the new |
|
Moved the bugfix fix over to #433 |
pintohutch
force-pushed
the
vgpu-fix
branch
3 times, most recently
from
1808a2b to
61ed255
Compare
We add support for capturing separate metrics when running dcgm-exporter in K8s clusters that have GPU sharing enabled, including time-sharing and MPS. This should now support GPU sharing on MIG devices as well. We ensure this is supported for both the NVIDIA and GKE device plugins, respectively at: * https://github.com/NVIDIA/k8s-device-plugin * https://github.com/GoogleCloudPlatform/container-engine-accelerators The change is guarded by a new configuration parameter, which can be passed in as a flag `--kubernetes-virtual-gpus` or as an environment variable `KUBERNETES_VIRTUAL_GPUS`. If set, the Kubernetes PodMapper tranform processor uses a different mechanism to build the device mapping, which creates a copy of the metric for every shared (i.e. virtual) GPU exposed by the device plugin. To disambiguate the generated timeseries, it adds a new label "vgpu" set to the detected shared GPU replica. This also fixes an issue where pod attributes are not guaranteed to be consistently associated with the same metric. If the podresources API does not consistently return the device-ids in the same order between calls, the device-to-pod association in the map can change between scrapes due to an overwrite that happens in the Process loop. Ultimately, we may wish to make this the default behavior. However, guarding it behind a flag: 1. Mitigates any risk of the change in case of bugs 2. Given the feature adds a new label, it is possible PromQL queries performing deaggregation using, e.g. `without` or `ignore` functions, may break existing dashboards and alerts. Allowing users to opt-in via a flag ensures backwards compatibility in these scenarios. Finally, we update the unit tests to ensure thorough coverage for the changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We add support for capturing separate metrics when running dcgm-exporter in K8s clusters that have GPU sharing enabled, including time-sharing and MPS. This should now support GPU sharing on MIG devices as well.
We ensure this is supported for both the NVIDIA and GKE device plugins, respectively at:
The bulk of the change is guarded by a new configuration parameter, which can be passed in as a flag
--kubernetes-virtual-gpusor as an environment variableKUBERNETES_VIRTUAL_GPUS. If set, the Kubernetes PodMapper tranform processor uses a different mechanism to build the device mapping, which creates a copy of the metric for every shared (i.e. virtual) GPU exposed by the device plugin. To disambiguate the generated timeseries, it adds a new label "vgpu" set to the detected shared GPU replica.This also fixes an issue where pod attributes are not guaranteed to be consistently associated with the same metric. If the podresources API does not consistently return the device-ids in the same order between calls, the device-to-pod association in the map can change between scrapes due to an overwrite that happens in the Process loop.
Ultimately, we may wish to make this the default behavior. However, guarding it behind a flag:
withoutorignorefunctions, may break existing dashboards and alerts. Allowing users to opt-in via a flag ensures backwards compatibility in these scenarios.Finally, we update the unit tests to ensure thorough coverage for the changes.