Skip to content

[CORE-16773] lsan: don't report on leak:seastar::alien::message_queue#30972

Merged
WillemKauf merged 1 commit into
redpanda-data:devfrom
WillemKauf:lsan_fix
Jul 6, 2026
Merged

[CORE-16773] lsan: don't report on leak:seastar::alien::message_queue#30972
WillemKauf merged 1 commit into
redpanda-data:devfrom
WillemKauf:lsan_fix

Conversation

@WillemKauf

@WillemKauf WillemKauf commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Introduced by #30829 (commit 341302f) and observed in v26.1.x backport branch https://buildkite.com/redpanda/redpanda/builds/86538 - I haven't seen it reproduce on dev, but better safe than sorry.

==12==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0xaaaab535018c in posix_memalign /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:139:3
    #1 0xffffa01335ac in boost::alignment::aligned_alloc(unsigned long, unsigned long) external/rules_boost++non_module_dependencies+boost/libs/align/include/boost/align/detail/aligned_alloc_posix.hpp:26:9
    #2 0xffffa01335ac in boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul>::allocate(unsigned long, void const*) external/rules_boost++non_module_dependencies+boost/libs/align/include/boost/align/aligned_allocator.hpp:70:19
    #3 0xffffa01335ac in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul>>::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul>>(boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul> const&, unsigned long) external/rules_boost++non_module_dependencies+boost/libs/lockfree/include/boost/lockfree/detail/freelist.hpp:62:31
    #4 0xffffa01300ec in boost::lockfree::queue<seastar::alien::message_queue::work_item*>::queue(unsigned long) external/rules_boost++non_module_dependencies+boost/libs/lockfree/include/boost/lockfree/queue.hpp:234:9
    #5 0xffffa01300ec in seastar::alien::message_queue::lf_queue::lf_queue(seastar::reactor*) external/+non_module_dependencies+seastar/include/seastar/core/alien.hh:57:40
    #6 0xffffa01300ec in seastar::alien::message_queue::message_queue(seastar::reactor*) external/+non_module_dependencies+seastar/src/core/alien.cc:46:5
    #7 0xffffa0131c14 in seastar::alien::instance::create_qs(std::__1::vector<seastar::reactor*, std::__1::allocator<seastar::reactor*>> const&) external/+non_module_dependencies+seastar/src/core/alien.cc:138:26
    #8 0xffffa05582ac in seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&) external/+non_module_dependencies+seastar/src/core/reactor.cc:4765:18
    #9 0xffffa01670bc in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) external/+non_module_dependencies+seastar/src/core/app-template.cc:242:15
    #10 0xffffa01662a8 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) external/+non_module_dependencies+seastar/src/core/app-template.cc:160:12
    #11 0xffffa18cb0e8 in seastar::testing::test_runner::start_thread(int, char**)::$_0::operator()() external/+non_module_dependencies+seastar/src/testing/test_runner.cc:77:26
    #12 0xffffa18cb0e8 in decltype(std::declval<seastar::testing::test_runner::start_thread(int, char**)::$_0&>()()) std::__1::__invoke[abi:se200100]<seastar::testing::test_runner::start_thread(int, char**)::$_0&>(seastar::testing::test_runner::start_thread(int, char**)::$_0&) external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:179:25
    #13 0xffffa18cb0e8 in void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:se200100]<seastar::testing::test_runner::start_thread(int, char**)::$_0&>(seastar::testing::test_runner::start_thread(int, char**)::$_0&) external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:251:5
    #14 0xffffa18cb0e8 in void std::__1::__invoke_r[abi:se200100]<void, seastar::testing::test_runner::start_thread(int, char**)::$_0&>(seastar::testing::test_runner::start_thread(int, char**)::$_0&) external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:273:10
    #15 0xffffa18cb0e8 in std::__1::__function::__alloc_func<seastar::testing::test_runner::start_thread(int, char**)::$_0, std::__1::allocator<seastar::testing::test_runner::start_thread(int, char**)::$_0>, void ()>::operator()[abi:se200100]() external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:167:12
    #16 0xffffa18cb0e8 in std::__1::__function::__func<seastar::testing::test_runner::start_thread(int, char**)::$_0, std::__1::allocator<seastar::testing::test_runner::start_thread(int, char**)::$_0>, void ()>::operator()() external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:319:10
    #17 0xffffa042bad4 in std::__1::function<void ()>::operator()() const external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:995:10
    #18 0xffffa042bad4 in seastar::posix_thread::start_routine(void*) external/+non_module_dependencies+seastar/src/core/posix.cc:90:5
    #19 0xaaaab534d034 in asan_thread_start(void*) /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
    #20 0xffff95eebb48  (/lib/aarch64-linux-gnu/libc.so.6+0xebb48) (BuildId: d5ef86dde36cbd3289566cf5098226035d76f2e1)

Indirect leak of 8128 byte(s) in 127 object(s) allocated from:
    #0 0xaaaab535018c in posix_memalign /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:139:3
    #1 0xffffa01335ac in boost::alignment::aligned_alloc(unsigned long, unsigned long) external/rules_boost++non_module_dependencies+boost/libs/align/include/boost/align/detail/aligned_alloc_posix.hpp:26:9
    #2 0xffffa01335ac in boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul>::allocate(unsigned long, void const*) external/rules_boost++non_module_dependencies+boost/libs/align/include/boost/align/aligned_allocator.hpp:70:19
    #3 0xffffa01335ac in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul>>::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul>>(boost::alignment::aligned_allocator<boost::lockfree::queue<seastar::alien::message_queue::work_item*>::node, 64ul> const&, unsigned long) external/rules_boost++non_module_dependencies+boost/libs/lockfree/include/boost/lockfree/detail/freelist.hpp:62:31
    #4 0xffffa01300ec in boost::lockfree::queue<seastar::alien::message_queue::work_item*>::queue(unsigned long) external/rules_boost++non_module_dependencies+boost/libs/lockfree/include/boost/lockfree/queue.hpp:234:9
    #5 0xffffa01300ec in seastar::alien::message_queue::lf_queue::lf_queue(seastar::reactor*) external/+non_module_dependencies+seastar/include/seastar/core/alien.hh:57:40
    #6 0xffffa01300ec in seastar::alien::message_queue::message_queue(seastar::reactor*) external/+non_module_dependencies+seastar/src/core/alien.cc:46:5
    #7 0xffffa0131c14 in seastar::alien::instance::create_qs(std::__1::vector<seastar::reactor*, std::__1::allocator<seastar::reactor*>> const&) external/+non_module_dependencies+seastar/src/core/alien.cc:138:26
    #8 0xffffa05582ac in seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&) external/+non_module_dependencies+seastar/src/core/reactor.cc:4765:18
    #9 0xffffa01670bc in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) external/+non_module_dependencies+seastar/src/core/app-template.cc:242:15
    #10 0xffffa01662a8 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) external/+non_module_dependencies+seastar/src/core/app-template.cc:160:12
    #11 0xffffa18cb0e8 in seastar::testing::test_runner::start_thread(int, char**)::$_0::operator()() external/+non_module_dependencies+seastar/src/testing/test_runner.cc:77:26
    #12 0xffffa18cb0e8 in decltype(std::declval<seastar::testing::test_runner::start_thread(int, char**)::$_0&>()()) std::__1::__invoke[abi:se200100]<seastar::testing::test_runner::start_thread(int, char**)::$_0&>(seastar::testing::test_runner::start_thread(int, char**)::$_0&) external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:179:25
    #13 0xffffa18cb0e8 in void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:se200100]<seastar::testing::test_runner::start_thread(int, char**)::$_0&>(seastar::testing::test_runner::start_thread(int, char**)::$_0&) external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:251:5
    #14 0xffffa18cb0e8 in void std::__1::__invoke_r[abi:se200100]<void, seastar::testing::test_runner::start_thread(int, char**)::$_0&>(seastar::testing::test_runner::start_thread(int, char**)::$_0&) external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__type_traits/invoke.h:273:10
    #15 0xffffa18cb0e8 in std::__1::__function::__alloc_func<seastar::testing::test_runner::start_thread(int, char**)::$_0, std::__1::allocator<seastar::testing::test_runner::start_thread(int, char**)::$_0>, void ()>::operator()[abi:se200100]() external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:167:12
    #16 0xffffa18cb0e8 in std::__1::__function::__func<seastar::testing::test_runner::start_thread(int, char**)::$_0, std::__1::allocator<seastar::testing::test_runner::start_thread(int, char**)::$_0>, void ()>::operator()() external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:319:10
    #17 0xffffa042bad4 in std::__1::function<void ()>::operator()() const external/toolchains_llvm++llvm+current_llvm_toolchain/bin/../../toolchains_llvm++llvm+current_llvm_toolchain_llvm/bin/../include/c++/v1/__functional/function.h:995:10
    #18 0xffffa042bad4 in seastar::posix_thread::start_routine(void*) external/+non_module_dependencies+seastar/src/core/posix.cc:90:5
    #19 0xaaaab534d034 in asan_thread_start(void*) /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
    #20 0xffff95eebb48  (/lib/aarch64-linux-gnu/libc.so.6+0xebb48) (BuildId: d5ef86dde36cbd3289566cf5098226035d76f2e1)

SUMMARY: AddressSanitizer: 8192 byte(s) leaked in 128 allocation(s).
INFO: to ignore leaks on libFuzzer side use -detect_leaks=0.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v26.1.x
  • v25.3.x
  • v25.2.x

Release Notes

  • none

Copilot AI review requested due to automatic review settings June 30, 2026 21:11
@WillemKauf WillemKauf requested review from dotnwat and oleiman June 30, 2026 21:12
Copilot AI reviewed Jun 30, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

dotnwat
dotnwat reviewed Jun 30, 2026
View reviewed changes
Comment thread lsan_suppressions.txt
# When running seastar_fuzz, libFuzzer's during-fuzzing leak check probes
# while the reactor is still alive and cannot trace the owning pointer through
# the blocked reactor thread's stack, so it spuriously reports the allocation.
leak:seastar::alien::message_queue

@dotnwat dotnwat Jun 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cannot trace the owning pointer through the blocked reactor thread's stack

what does this mean exactly? there isn't anything particularly exotic going on with reactors, i'd expect lsan to be able to handle what the alien thread is doing.

@WillemKauf WillemKauf Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Err good point. I kinda took the initial AI interpretation at face value.

I think the real reason is this: alien::message_queue::_pending is a boost::lockfree::queue<work_item*>, and boost::lockfree::queue uses tagged pointers to reference memory & help achieve lock free semantics. LSan does the dumb thing of scanning these tagged pointers directly (which may not point to anything initialized) during run-time, and reports false positive memory leaks.

I'm trying to find existing reports of LSan/boost::lockfree::queue having this issue, but this feels like the proper conclusion to me.

@WillemKauf WillemKauf Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wdyt @dotnwat? this is actively flaking in dev

@WillemKauf WillemKauf requested a review from dotnwat July 1, 2026 03:57
@WillemKauf WillemKauf changed the title lsan: don't report on leak:seastar::alien::message_queue [CORE-16773] lsan: don't report on leak:seastar::alien::message_queue Jul 2, 2026
@dotnwat

dotnwat commented Jul 6, 2026

Copy link
Copy Markdown
Member

Did it just start happening recently? If so, then did something change? Did sanitizers get better or did we enable a new type of test or did we introduce a bug?

@WillemKauf

WillemKauf commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Did it just start happening recently? If so, then did something change? Did sanitizers get better or did we enable a new type of test or did we introduce a bug?

the test is new, so is the way the new library seastar_fuzz hooks into libFuzzer (see commit referenced in cover letter, 341302f)

@WillemKauf WillemKauf merged commit 813b106 into redpanda-data:dev Jul 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@dotnwat dotnwat dotnwat approved these changes
@oleiman oleiman Awaiting requested review from oleiman

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@WillemKauf @dotnwat