MGS is using a surprisingly large amount of memory

[jgallagher]

On dogfood from prstat:

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  1911 root       12G   10G cpu110   0    0   1:07:40 0.2% mgs/132

Chatted with @hawkw about this briefly; we think maybe a hiccup in Oximeter collection (possibly induced by an update rebooting the sled hosting it) caused MGS to fill its channel with samples, which does have an upper bound but maybe it's a little bigger than we expected.

Nothing urgent here, I don't think, but filing this so we can poke at it more and see if we can confirm what's going on (and decide whether we're okay with it).

[hawkw]

Yeah...I would certainly expect the memory actively used by samples to go back down when Oximeter comes back, but I'm not totally sure what prstat is actually showing us here and what the illumos allocator's behavior is (do we expect these numbers to decrease as allocations are freed, or is it giving us a high water mark?)

[JustinAzoff]

I profiled mgs on london and metrics do appear to be the cause

umem_mgs_core_1987-01-02-22:31:23.28511.txt

104939520 bytes for 854 allocations with data size 122880:
..
         alloc::raw_vec::finish_grow::he24bf95232e95278 +0x31
         alloc::raw_vec::RawVec<T,A>::grow_one::h834eba057047a5e0+0x80
         omicron_gateway::metrics::SpPoller::poll::{{closure}}::h8d135405271cf6b6+0x1f74
         omicron_gateway::metrics::SpPoller::run::_$u7b$$u7b$closure$u7d$$u7d$::hcdcbcadf13c1eeee +0x39e
         tokio::runtime::task::core::Core<T,S>::poll::ha5491eacdd0d20a9+0x3b
         tokio::runtime::task::harness::Harness<T,S>::poll::h0f4cc76b143bfe57+0x5d
         tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h2a66954dc1f39f13+0x131
         tokio::runtime::scheduler::multi_thread::worker::Context::run::h464176f8b45f2c3b+0xc0e
         tokio::runtime::context::scoped::Scoped<T>::set::h5b0972e640072193+0x2a
         tokio::runtime::context::runtime::enter_runtime::hc0a7491c2da47afe+0x19e

It's also still growing 30 minutes after being restarted:

root@oxz_switch0:~# ps auxw|egrep [R]SS;while true; do ps auxw|grep [m]gs; sleep 10;done
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
root     28511  1.9  1.7 17604920 17475760 ?        O 22:21:56  6:55 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  1.9  1.7 17657268 17522528 ?        O 22:21:56  7:19 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  1.9  1.7 17702736 17562956 ?        O 22:21:56  7:44 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  1.7  1.7 17717216 17574028 ?        O 22:21:56  8:03 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  1.6  1.7 17738224 17588024 ?        O 22:21:56  8:22 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  1.5  1.7 17759616 17604976 ?        S 22:21:56  8:41 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  1.4  1.7 17773296 17612960 ?        O 22:21:56  8:56 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  1.1  1.7 17773296 17612960 ?        O 22:21:56  9:03 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  0.9  1.7 17773296 17612960 ?        S 22:21:56  9:11 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest
root     28511  0.8  1.7 17773296 17614180 ?        S 22:21:56  9:19 /opt/oxide/mgs/bin/mgs run --id-and-address-from-smf /var/svc/manifest

[JustinAzoff]

note: london may not be the most representative case. I notice RSS has not completed and no zones are running other than oxz_switch, so there's no oximeter or even clickhouse running.