hang at end in loadCurrentStack

[aulwes]

Hi, I've built the latest repo but in gdb, it looks like Malt is hanging in loadCurrentStack. looking at backtrace in gdb, I see

#0 0x00001554da17c4cd in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00001554da175ac9 in pthread_mutex_lock () from /lib64/libpthread.so.0

so it looks like it's hanging on a pthread lock. is it possible to build without using pthread?

[aulwes]

I tried setting -DPORTABILITY_MUTEX=DUMMY -DPORTABILITY_SPINLOCK=DUMMY, but when I tried to compile, I get compile error

src/libinstrum/./portability/Mutex.hpp:36:11: fatal error: 'MutexDummy.hpp' file not found
36 | #include "MutexDummy.hpp"

[svalat]

Hi, thanks for reporting,
I first looked about your second message, true, I have a missing file. I made a fix so now it will build.

I will give a look in the hang.

Any chance to have a small reproducer on the conditions ?

[svalat]

In case the fix for the MutexDummy issue is in branch develop until next release.

[aulwes]

Let me see if I can create a reproducer. Thanks, I'll grab your fix from develop branch.

[aulwes]

I see LockDummy.hpp in develop branch but not MutexDummy.hpp

[svalat]

Oopos sorry, I forgot to push to the public repo. Done.

[aulwes]

Thank you!

[aulwes]

Just updated develop branch, but getting this compile error:

src/libinstrum/./portability/Spinlock.hpp:37:11: error: unknown type name 'SpinlockDummy'
37 | typedef SpinlockDummy Spinlock;
| ^

[aulwes]

So if I replace SpinlockDummy with LockDummy, then I can compile. Now, when I try to run, I'm getting an abort because of

MALT_ASSERT(this->prev->next == this);

in FreeChunk::removeFromList() in SimpleAllocator.cpp. This was called from SimpleAllocator::getInList(). Any ideas why this assertion is failing?

[svalat]

Hum, sorry, was not fresh enough when making the patch, I forgot is was shared and enabled Dummy only on Mutex for my test.

I fixed it in develop.

Question : is you app using threads ? Because without lock, it might corrupt MALT internally.

If no threads in the app, I would also deactivate two things on MALT to get a chance to pass :

1. Disable using threads for the addr2line calls : -o addr2line:threads=1
2. If you still get the previous crash, try to disable the internal allocator by add -DMALT_USE_STD_MALLOC=ON. Note it will change the memory layout of your program by mixing MALT blocks in the middle of the one from the app. And physical memory will also include the MALT memory in that case. But it still offer all the rest.
3. You can also maybe try to just apply 2. and keep the threads. The only part which is locking in loadCurrentStack is the MALT_MALLOC.

Note: I'm not 100% sure I'm not using some threads elsewhere in the last version so not 100% sure the LockDummy is usable in practice in last versions.

Question:

1. Is the original problem (deadlock) reproducable ?
2. Is it in MPI context ?
3. Any chance to get the MALT internal stack to see where I am in the tool at that deadlock ?
4. Is it python or C code ? (there are several loadCurrentStack function in MALT).

[aulwes]

Our app is not using threads. I tried 1. and 2., but still getting same error. Here is the backtrace:

#3 0x00001555550f543e in MALT::FreeChunk::removeFromList() ()
from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#4 0x00001555550f5008 in MALT::SimpleAllocator::getInList(unsigned long) ()
from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#5 0x00001555550f4eaa in MALT::SimpleAllocator::malloc(unsigned long) ()
from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#6 0x00001555550be6b1 in std::_Rb_tree_iterator<std::pair<void* const, MALT::SegmentInfo> > std::_Rb_tree<void*, std::pair<void* const, MALT::SegmentInfo>, std::_Select1st<std::pair<void* const, MALT::SegmentInfo> >, std::less<void*>, MALT::STLInternalAllocator<std::pair<void* const, MALT::SegmentInfo> > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<void* const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<void* const, MALT::SegmentInfo> >, std::piecewise_construct_t const&, std::tuple<void* const&>&&, std::tuple<>&&) () from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#7 0x00001555550bd8ae in MALT::SegmentTracker::add(void*, unsigned long, MALT::MMCallStackNode) ()
from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#8 0x00001555550d75bf in MALT::AllocStackProfiler::onAllocEvent(void*, unsigned long, MALT::Stack*, MALT::MMCallStackNode*, bool, MALT::AllocDomain) () from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#9 0x00001555550d733f in MALT::AllocStackProfiler::onMalloc(MALT::AllocTracerEvent&, void*, unsigned long, MALT::Stack*, MALT::AllocDomain) () from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#10 0x00001555550e1c66 in MALT::LocalAllocStackProfiler::onMalloc(void*, unsigned long, unsigned long long, MALT::MallocKind, MALT::Language, MALT::AllocDomain) () from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#11 0x00001555550e3f25 in MALT::malt_wrap_malloc(unsigned long, void* (* const&)(unsigned long), void*) ()
from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#12 0x00001555550fe296 in malloc () from malt/malt-1.6.2/hipcc-7.2.0/lib64/libmalt.so
#13 0x000015551994ed7c in operator new(unsigned long) () from /lib64/libstdc++.so.6
#14 0x000015553c51898e in amd::roc::Device::svmAlloc(amd::Context&, unsigned long, unsigned long, unsigned long, void*) const ()
from /opt/rocm-6.4.2/lib/libamdhip64.so.6
#15 0x000015553c507ee6 in amd::Context::svmAlloc(unsigned long, unsigned long, unsigned long, amd::Device const*) ()
from /opt/rocm-6.4.2/lib/libamdhip64.so.6
#16 0x000015553c50c067 in amd::SvmBuffer::malloc(amd::Context&, unsigned long, unsigned long, unsigned long, amd::Device const*) ()
from /opt/rocm-6.4.2/lib/libamdhip64.so.6
#17 0x000015553c402a9e in hip::MemoryPool::AllocateMemory(unsigned long, hip::Stream*, void*) ()
from /opt/rocm-6.4.2/lib/libamdhip64.so.6
#18 0x000015553c3f9eb4 in hip::hipMallocAsync(void**, unsigned long, ihipStream_t*) () from /opt/rocm-6.4.2/lib/libamdhip64.so.6
#19 0x000015554fb85be9 in Kokkos::HIPSpace::impl_allocate(int, ihipStream_t*, char const*, unsigned long, unsigned long, bool) const ()

[aulwes]

Looking at original hang and when I look at the call stack in gdb when attaching to the hung process, I see a calling stack of

getStack -> loadCurrentStack -> __backtrace -> _Unwind_Backtrace -> ... -> _Unwind_Find_FDE -> __pthread_mutex_lock

Is it possible to tell libunwind not to use pthreads?