Staging shibd swap memory problem

[DevinSmithWork]

Summary

• /usr/sbin/shibd is eating up a ton of swap memory on staging, occasionally causing the system to reboot.
• Additional work is required to determine how this is occurring and implement fixes.

Problem Identification history

• Monday 2025-05-12: While troubleshooting a crash & reboot on staging over the weekend, Martin noticed that prior to the restart, the swap % climbed to 99-100%.
• Throughout the week, we continued to monitor Librato and started logging the output of top and ps to monitor PID swap usage over time.
• Tracking showed the following:
  • Friday 2025-05-16:
    • Around 8:15pm, /usr/sbin/shibd started running as process 3036
    • By 8:50pm, its subprocess 3440 had consumed 100% swap memory.
    • At midnight, the swap level fell to 86-87% (perhaps related to nightly restarts?)
    • The system did not reboot this time, and was running in this state on Monday.
• Monday 2025-05-19:
  • Collected the above data
  • Checked /proc/ memory usage, confirmed the 3036 subprocess 3440 was storing large amount of memory.
  • Ran eye restart shibd and swap memory was freed.