mgadm aborted with rust panic

[askfongjojo]

We found a mgadm core file on rack2.

> ::status
debugging core file of mgadm (64-bit) from oxz_switch
file: /zone/oxz_switch/root/opt/oxide/mgd/bin/mgadm
initial argv: mgadm bgp config router list
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=28311 uid=0 code=-1
> $C ! demangle
fffff5ffffdf0c00 libc.so.1`_lwp_kill+0xa()
fffff5ffffdf0c30 libc.so.1`raise+0x22(6)
fffff5ffffdf0c80 libc.so.1`abort+0x58()
fffff5ffffdf0c90 ~std::sys::pal::unix::abort_internal::h515b31030a784ec5+8()
fffff5ffffdf0ca0 ~std::process::abort::h263ae6dd5f7f06e2+8()
fffff5ffffdf0cb0 ~__rustc::__rust_abort+8()
fffff5ffffdf0cc0 ~__rustc::__rust_start_panic+8()
fffff5ffffdf0d20 __rustc::rust_panic+0xd()
fffff5ffffdf0de0 std::panicking::rust_panic_with_hook::h24c1a837a734501c+0x224()
fffff5ffffdf0e20 std::panicking::begin_panic_handler::{{closure}}::hebb980b0c684b90d+0x98()
fffff5ffffdf0e30 ~std::sys::backtrace::__rust_end_short_backtrace::h4b651dba38b6cba9+8()
fffff5ffffdf0e60 ~__rustc::rust_begin_unwind+0x1b()
fffff5ffffdf0e90 ~core::panicking::panic_fmt::h717aa56ae600c892+0x1e()
fffff5ffffdf0f20 std::io::stdio::_print::h7cc95148d8499c36+0xcb()
fffff5ffffdf2a10 mgadm::bgp::commands::{{closure}}::h5c4d2fa9e5258160+0x152e6()
fffff5ffffdf40a0 mgadm::run::_$u7b$$u7b$closure$u7d$$u7d$::hc8ecba0112647168 (.llvm.10680117334342463519)+0x3f0()
fffff5ffffdf4ff0 tokio::runtime::park::CachedParkThread::block_on::h2032e1c370f1d443+0xa7()
fffff5ffffdf6150 oxide_tokio_rt::run::hb3f3f8779801724c+0x220()
fffff5ffffdf7040 mgadm::main::h9bd44c10686f32de+0x1e()
fffff5ffffdf7050 std::sys::backtrace::__rust_begin_short_backtrace::h4d80df8f099b7c04+6()
fffff5ffffdf70b0 std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h3253e85a1fd849f5 (.llvm.7986615778376668083)+0x11()
fffff5ffffdf7150 std::rt::lang_start_internal::h4d69539b7902fe85+0x421()
fffff5ffffdf7170 main+0x2c()
fffff5ffffdf71a0 _start_crt+0x87()
fffff5ffffdf71b0 _start+0x18()

The core file can be found in /staff/dock/rack2/mupdate-20251201/corefile.

[askfongjojo]

It was suggested that the panic was a result of someone piping mgadm into something that closed stdin.

[rcgoodfellow]

If it was an mgadm before #362 was addressed it could be a failed API call. Since mgadm is not involved in the function of the product, I would not spend too many cycles on this.

[taspelund]

If the issue is piping mgadm into a different binary, then we could likely solve it by handling EPIPE similar to what we're doing in the oxide cli:
oxidecomputer/oxide.rs#746

Above and beyond that, they also added a clippy lint banning println and friends from non-test code:
oxidecomputer/oxide.rs#749

[taspelund]

I'm not super familiar with the mdb call stack, but it looks like the core was generated from within a std::io print method:

fffff5ffffdf0f20 std::io::stdio::_print::h7cc95148d8499c36+0xcb()
fffff5ffffdf2a10 mgadm::bgp::commands::{{closure}}::h5c4d2fa9e5258160+0x152e6()

So I imagine we just need to pull in something to handle print errors more gracefully.

We also use tabwriter in a lot of places rather than println, so that may require a different approach.