Skip to content

omdb should handle SIGPIPE cleanly instead of panicking #8352

@davepacheco

Description

@davepacheco

Easily reproduced (there's probably a better way, given this produced a head error):

$ ./target/debug/omdb --dns-server [::1]:34590 db sleds | head -n 0
head: Invalid "-n 0" option
usage: head [-q] [-v] [-n #] [-c #] [-#] [filename...]
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using database URL postgresql://root@[::1]:59338/omicron?sslmode=disable
WARN: found schema version 150.0.0, expected 148.0.0
It's possible the database is running a version that's different from what this
tool understands.  This may result in errors or incorrect output.
note: listing all commissioned sleds (use -F to filter, e.g. -F in-service)

thread 'main' panicked at library/std/src/io/stdio.rs:1165:9:
failed printing to stdout: Broken pipe (os error 32)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Jun 17 16:59:48.837 ERRO Pool dropped without invoking `terminate`. qorb background tasks
                 should be cancelled, but they may briefly still be initializing connections

We saw a core file on dogfood that looks like this:

dap@castle /staff/dock/rack2/mupdate-20250613 $ mdb /staff/core/dogfood-mupdate-20250613/14/core.oxz_switch.omdb.4133.1749654363
Loading modules: [ libumem.so.1 libnvpair.so.1 libc.so.1 ld.so.1 ]
> $G
C++ symbol demangling enabled
> $C 
fffff5ffffde9cd0 libc.so.1`_lwp_kill+0xa()
fffff5ffffde9d00 libc.so.1`raise+0x22(6)
fffff5ffffde9d50 libc.so.1`abort+0x58()
fffff5ffffde9d60 ~panic_abort::__rust_start_panic::abort::h349954c3ed15731c+8()
fffff5ffffde9d70 ~__rustc::__rust_start_panic+8()
fffff5ffffde9dd0 __rustc::rust_panic+0xd()
fffff5ffffde9e90 std::panicking::rust_panic_with_hook::hb99087f77e1043a2+0x22f()
fffff5ffffde9ed0 std::panicking::begin_panic_handler::{{closure}}::hdc6b5764a1eb26a4+0x98()
fffff5ffffde9ee0 ~std::sys::backtrace::__rust_end_short_backtrace::h2fdfec1ded489945+8()
fffff5ffffde9f10 ~__rustc::rust_begin_unwind+0x1b()
fffff5ffffde9f40 ~core::panicking::panic_fmt::h84330e1efd4e6522+0x1e()
fffff5ffffde9fd0 std::io::stdio::_print::h4cfd8b3a3c3cdb51+0xcc()
fffff5ffffdeae70 omdb::db::cmd_crucible_dataset_list::{{closure}}::he1e92ce8334d6568+0x45a()
fffff5ffffded0f0 omdb::main::_$u7b$$u7b$closure$u7d$$u7d$::h1ae782f3e11650e1 +0x7c39()
fffff5ffffdee4d0 tokio::runtime::park::CachedParkThread::block_on::h1b17cbb1ecc3507e+0xb3()
fffff5ffffdef8c0 tokio::runtime::context::runtime::enter_runtime::h6c358638de65487c+0x13d()
fffff5ffffdf2100 omdb::main::hcddea64eae948ab8+0xce()
fffff5ffffdf2110 std::sys::backtrace::__rust_begin_short_backtrace::h1826ce7cb0fec08e+6()
fffff5ffffdf2170 std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hd4a90b26aceba257 +0x11()
fffff5ffffdf2210 std::rt::lang_start_internal::hcda8dc089aca4d48+0x3af()
fffff5ffffdf2230 main+0x2c()
fffff5ffffdf2260 _start_crt+0x87()
fffff5ffffdf2270 _start+0x18()
> ::status
debugging core file of omdb (64-bit) from oxz_switch
initial argv: omdb db crucible-dataset list
threading model: native threads
status: process terminated by SIGABRT (Abort), pid=4133 uid=0 code=-1

By default on most systems:

  • When you pipe a command to another command, and that second command exits early, SIGPIPE is generated in the first process when it tries to write to the now-broken pipe.
  • The default disposition for SIGPIPE is to exit the process. This makes the basic foo | head flow work.

For whatever reason, Rust installs a handler for SIGPIPE that causes a Rust panic. For more information: https://github.com/kurtbuilds/sigpipe.

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueIssues that are good for learning the codebase

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions