Memory usage increased 2.25x after upgrading from 3.3.6-3.4.2 to 3.3.9-3.6.1

[age9990]

After upgrading dcgm-exporter from 3.3.6-3.4.2 to 3.3.9-3.6.1, with no configuration change, memory usage rised from abount 200mb to 450mb, approximately 2.25x. Currently, I set memory request to 256mb, and memory limit to 512mb. It almost hit the memory limit after the upgrade.
Is it normal the memory usage increased so much?

[nvvfedorov]

@age9990 , Thank you for the reporting about the issue. Can you give us a hint about your configuration (what metrics enabled and what is a GPU)?

[age9990]

@nvvfedorov I use the default configuration shipped with gpu-operator v24.9.0,
DCGM_EXPORTER_COLLECTORS: /etc/dcgm-exporter/dcp-metrics-included.csv
GPU is A100 and H100.

[devlware]

I'm also seeing this behavior in memory usage with the latest version 3.3.9-3.6.{0,1}. Need at least 500M of memory otherwise the pod is killed.

[vanHavel]

Also seeing this behavior - memory usage is still fine in version 3.3.7-3.5.0.
Tested on GKE for L4 nodes.