High Virtual Memory (VIRT) Usage Despite Low Physical RAM (RES) in Triton Server

[qocharian]

Description
When running Triton Server, the virtual memory (VIRT) reported can be very large (e.g., 30GB), but the actual physical RAM usage (RES) remains low (e.g., around 3GB). This causes confusion about memory consumption and concern about potential memory leaks or inefficient usage. Is this expected behavior?

[whoisj]

virtual memory (VIRT) includes memory mapped files, depending on the size of your model files (including weights) this value can be surprisingly high. I would check the size of your models + their weights on disk and see if that accounts for the VIRT utilization you're seeing.

[qocharian]

whoisj, thanks for the reply! It looks like most of the VIRT comes from anonymous mmap regions without a file backing (no path in maps), with a smaller share from files and special regions like heap, [stack], [stack:tid], [vdso], [vvar], [vsyscall], [anon], as well as device mappings under /dev/nvidia*. From what I understand, this is expected, right?

virt_report.txt
virt_checker.py

[whoisj]

Nothing looks out of place to me. I'll pass this link around to a few folks here at NVIDIA. If any of them notice an oddity, I'll be sure to report back.

[whoisj]

@qocharian would you be able to provide a minimal reproducer?

Additionally, I'm wondering about mmaps for shared memory. Given that's the primary mechanism for sharing data between Python "threads", it's quite possible a large portion of the VIRT usage is related to these allocations.