Very high virtual memory consumption and slow performance

[jmather-sesi]

Hi there,

I'm currently evaluating tcmalloc as a potential replacement for our existing old memory allocator (jemalloc 3.6.0). One thing that I've noticed is that the virtual memory consumption of tcmalloc is much higher than other allocators such as mimalloc, and gperftools tcmalloc. I was wondering if there was something that I was doing wrong?

For testing, I built tcmalloc as a shared library following #27 (comment) and injected it into our application through LD_PRELOAD. (I know this isn't recommended due to potential ODR issues with Abseil - this is just for testing). I then ran one of our fluid simulation workloads. When building with Bazel, I made sure to specify -c opt. This seems to have built a release build of the allocator, but I'm new to Bazel, so perhaps I missed something crucial.

This is the virtual memory consumption over time for a number of allocators. tcmalloc_google is this repo, and tcmalloc is from gperftools:

As you can see, tcmalloc_google_stock's virtual memory consumption is easily double that of the other allocators, and goes past the bounds of the graph. The maximum that it hits is 145.44GB, almost 3 times higher than the next largest.

Looking at the tuning guide, it recommends that we make changes to THP settings. We unfortunately cannot control these settings on customer machines. I have therefore decided to ignore it for now.

For reference, this was run on an Ubuntu 22.04.4 machine with the default settings:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
$ cat /sys/kernel/mm/transparent_hugepage/defrag
always defer defer+madvise [madvise] never
$ cat /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
511
$ cat /proc/sys/vm/overcommit_memory
0

Looking at the existing issues, I came across this post: #260 (comment). I tried exporting GLIBC_TUNABLES=glibc.pthread.rseq=0 before running our workload, but it didn't seem to have much of an effect. (-1.5GB VSIZE) [rseq0 in the graph above]

I also tried creating a new thread in our application that calls ProcessBackgroundActions. This makes quite an improvement, but VSIZE is still excessive. (I configured the release rate to 10 MB/s) [rr10MBps in the graph above)

One additional thing that I've noticed is that tcmalloc seems to be slightly slower than other the allocator provided by gperftools. It takes tcmalloc around 4430.00s for this benchmark to finish, while tcmalloc from gperftools takes 4271.00s (159 seconds [3.6%] less) with considerably lower virtual memory consumption. These numbers were collected on a silent machine with 24 cores (16P 8E) with turbo boost disabled, the cores locked to their base frequencies and the CPU governor set to performance for consistent results.

Virtual memory consumption is a big issue for us as many of our customers configure their systems to have zero swap space and monitor the virtual memory size. If the virtual memory size gets close to the physical memory size, they'll kill the application as it's "swapping." We've tried to inform users that virtual memory consumption is not an indication of swapping, but alas, this practice is very common in our industry, so we must make concessions. This is the main reason why we've been stuck on jemalloc 3.6.0 for so long.

I have dumped the stats right at the end of the workload, and you can find it here: stats.txt

Any help that you could provide would be greatly appreciated. Thanks!

[ckennelly]

Thanks for the detailed report and including the malloc stats. We don't try to be frugal with virtual address space (we get memory in 1GB chunks), but this is definitely an outlier.

Can you say anything about your allocation pattern, and how modifiable it might be at least for prototyping? In particular the stats section here caught my eye:

MALLOC:           8192               Tcmalloc page size

HugePageAware: filler donations 59 (3338 pages from abandoned donations)

HugeRegionSet: 1 MiB+ allocations best-fit into 1024 MiB slabs
HugeRegionSet: 3 total regions
HugeRegion: 39840 KiB used, 15456 KiB free, 935168 KiB contiguous space, 970 MiB unbacked, 222422 MiB unbacked lifetime

HugePageAware: [     257 ,     512 ] page info:      10110 /      10073 a/f,       37 (  96.1 MiB) live,    2.196 allocs/s (   5.9 MiB/s)
HugePageAware: [     513 ,    1024 ] page info:       1498 /       1498 a/f,        0 (   0.0 MiB) live,    0.325 allocs/s (   1.7 MiB/s)
HugePageAware: [    1025 ,    2048 ] page info:         68 /         68 a/f,        0 (   0.0 MiB) live,    0.015 allocs/s (   0.1 MiB/s)
HugePageAware: [    2049 ,    4096 ] page info:        181 /        181 a/f,        0 (   0.0 MiB) live,    0.039 allocs/s (   0.8 MiB/s)
HugePageAware: [    4097 ,    8192 ] page info:          1 /          1 a/f,        0 (   0.0 MiB) live,    0.000 allocs/s (   0.0 MiB/s)
HugePageAware: [    8193 ,   16384 ] page info:        870 /        870 a/f,        0 (   0.0 MiB) live,    0.189 allocs/s (  15.1 MiB/s)
HugePageAware: [   16385 ,   32768 ] page info:       2617 /       2617 a/f,        0 (   0.0 MiB) live,    0.569 allocs/s (  94.4 MiB/s)
HugePageAware: [   32769 ,   65536 ] page info:        305 /        305 a/f,        0 (   0.0 MiB) live,    0.066 allocs/s (  24.8 MiB/s)
HugePageAware: [   65537 ,  131072 ] page info:        392 /        392 a/f,        0 (   0.0 MiB) live,    0.085 allocs/s (  52.7 MiB/s)
HugePageAware: [  131073 ,  262144 ] page info:        518 /        518 a/f,        0 (   0.0 MiB) live,    0.113 allocs/s ( 164.0 MiB/s)

The lines are in TCMalloc pages (so 8KB here). These allocations are (2MB,4MB], (4MB, 8MB], etc.

They should do one of:

• Be allocated normally with a tail donated to the filler. The filler doesn't look like it has many donations, so this isn't commonplace.
• The region should be able to handle almost all of them (<1GB), and it has a lot of contiguous space. It only has 3 1GB regions, so it's not growing boundlessly.
• Go to the huge cache (>=1GB).

Are these allocations, particularly the >=1GB ones the same size or do they vary? My suspicion is that either:

• They're the same size, but we're releasing small pieces of the contiguous memory to the OS, thereby making us think we don't have a contiguous region and we request more.

  I believe changing https://github.com/google/tcmalloc/blob/master/tcmalloc/internal/system_allocator.h#L195 from kDontNeed to kNever will simulate release failing. If that stabilizes VSS at a much lower level, this is the likely culprit.

• They're varying in size, and we end up not binpacking well, especially with any other singleton allocations (like a 2MB page handed off to the filler for binpacking)

[ckennelly]

re the overall topline performance number: Was that

• With or without GLIBC_TUNABLES=glibc.pthread.rseq=0? Not having that means we use per-thread caches, but our fast paths are heavily optimized for per-CPU caches being available.
• With or without the background thread (even with release rate = 0)? We make a number of resizing decisions (optimizing/rebalancing caches/etc.) based on usage patterns on a background thread, so omitting it can hurt performance.

[jmather-sesi]

Hi Chris, thanks for getting back to me so quickly.

Can you say anything about your allocation pattern, and how modifiable it might be at least for prototyping?

Unfortunately I'm not the original developer of this code, so I don't know for certain what the pattern is. I can provide a graph of the allocation sizes. One thing that I did notice is that there are a large number of temporary allocations. I'm currently looking into resolving these.

re the overall topline performance number: Was that

• With or without GLIBC_TUNABLES=glibc.pthread.rseq=0? Not having that means we use per-thread caches, but our fast paths are heavily optimized for per-CPU caches being available.
• With or without the background thread (even with release rate = 0)? We make a number of resizing decisions (optimizing/rebalancing caches/etc.) based on usage patterns on a background thread, so omitting it can hurt performance.

I made sure to test these cases individually. So in the case of _rseq0, that only had the rseq0 environment variable set in GLIBC_TUNABLES. For _rr10MBps, that only had the background thread running at 10 MB/s. I've run an additional test with everything enabled at once - rr10MBpsRseq0 has both the background thread active and the GLIBC_TUNABLES export mentioned above. Stats: tcmalloc_google_rr10MBpsRseq0.txt

I believe changing https://github.com/google/tcmalloc/blob/master/tcmalloc/internal/system_allocator.h#L195 from kDontNeed to kNever will simulate release failing. If that stabilizes VSS at a much lower level, this is the likely culprit.

I tested this as well. rr10MBpsRseq0MadviseNever uses the test above with your proposed change to kNever. The VSIZE dropped considerably. It also seemed to be notably faster on this graph. This may have been an error where the system wasn't properly tuned for benchmarking. I'll run it again overnight and see if it's reproducible. Stats: tcmalloc_google_rr10MBpsRseq0MadviseNever.txt