emscripten_dlopen deadlock when re-opening already-loaded library

[sumleo]

Version of emscripten/emsdk

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 5.0.0 (a7c5deabd7c88ba1c38ebe988112256775f944c6)
clang version 23.0.0git (https:/github.com/llvm/llvm-project 358db292cc6a9a8a5448a296f643312289f328d7)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: ~/code/emsdk/upstream/bin

Bug description

emscripten_dlopen() deadlocks when opening a library that is already loaded. The find_existing() early-return path at system/lib/libc/dynlink.c:580-586 acquires do_write_lock() but never calls do_write_unlock():

  do_write_lock();                         // L580: lock acquired
  char buf[2*NAME_MAX+2];
  filename = find_dylib(buf, filename, sizeof buf);
  struct dso* p = find_existing(filename); // L583
  if (p) {
    onsuccess(user_data, p);               // L585: callback called
    return;                                // L586: MISSING do_write_unlock()
  }

Every other path in emscripten_dlopen properly releases the lock:

• L590: do_write_unlock() before onerror() callback
• L441: do_write_unlock() in dlopen_onsuccess() callback
• L449: do_write_unlock() in dlopen_onerror() callback

The leaked lock causes the next dlopen call to deadlock. Only affects -pthread builds where do_write_lock() is a real mutex (lines 97-99 define them as no-ops in non-_REENTRANT builds).

Reproducing

Save side.c:

int foo = 42;

Save test.c:

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <emscripten/emscripten.h>
#include <assert.h>

static int callback_count = 0;

void onsuccess_third(void *user_data, void *handle) {
  callback_count++;
  printf("Third dlopen succeeded (callback_count=%d)\n", callback_count);
  printf("PASS: All three dlopen calls succeeded without deadlock\n");
  exit(0);
}

void onerror_third(void *user_data) {
  printf("Third dlopen failed: %s\n", dlerror());
  exit(1);
}

void onsuccess_second(void *user_data, void *handle) {
  callback_count++;
  printf("Second dlopen succeeded (callback_count=%d)\n", callback_count);
  printf("Attempting third dlopen of same library...\n");
  emscripten_dlopen("libside.so", RTLD_NOW, NULL, onsuccess_third, onerror_third);
}

void onerror_second(void *user_data) {
  printf("Second dlopen failed: %s\n", dlerror());
  exit(1);
}

void onsuccess_first(void *user_data, void *handle) {
  callback_count++;
  printf("First dlopen succeeded (callback_count=%d)\n", callback_count);
  printf("Attempting second dlopen of same library...\n");
  emscripten_dlopen("libside.so", RTLD_NOW, NULL, onsuccess_second, onerror_second);
}

void onerror_first(void *user_data) {
  printf("First dlopen failed: %s\n", dlerror());
  exit(1);
}

int main() {
  printf("Test: dlopen same library three times (deadlock bug)\n");
  emscripten_dlopen("libside.so", RTLD_NOW, NULL, onsuccess_first, onerror_first);
  printf("Returning from main, waiting for callbacks...\n");
  return 99;
}

Three emscripten_dlopen calls are needed to trigger this:

1. 1st call loads the library fresh (async path, lock properly released in dlopen_onsuccess)
2. 2nd call finds the library via find_existing(), leaks the lock (bug)
3. 3rd call tries to acquire the leaked lock and deadlocks

Failing command line in full

emcc side.c -o libside.so -sSIDE_MODULE -pthread
emcc test.c libside.so -sMAIN_MODULE=2 -pthread -sPROXY_TO_PTHREAD -sEXIT_RUNTIME -o test.js
node test.js

Runtime output

Test: dlopen same library three times (deadlock bug)
Returning from main, waiting for callbacks...
First dlopen succeeded (callback_count=1)
Attempting second dlopen of same library...
Second dlopen succeeded (callback_count=2)
Attempting third dlopen of same library...
Aborted(Assertion failed: at || own != __pthread_self()->tid && "pthread mutex deadlock detected", at: /emsdk/emscripten/system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c,95,__pthread_mutex_timedlock)

Full link command and output with -v appended

Side module compile (emcc side.c -o libside.so -sSIDE_MODULE -pthread -v):

 ~code/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -fPIC -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=~code/emsdk/upstream/emscripten/cache/sysroot -D__EMSCRIPTEN_SHARED_MEMORY__=1 -DEMSCRIPTEN -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -pthread -v -c side.c -o /var/folders/wk/3v2syqm519z2kg7nqf4xtgy80000gn/T/emscripten_temp_vntpzynh/side.o
emcc: warning: dynamic linking + pthreads is experimental [-Wexperimental]
 ~code/emsdk/upstream/bin/wasm-ld -o libside.so --import-memory --shared-memory --strip-debug --export-dynamic --export=__wasm_call_ctors --export=_emscripten_tls_init --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=main --export-if-defined=__main_argc_argv --export-if-defined=fflush --experimental-pic --unresolved-symbols=import-dynamic -shared --stack-first --whole-archive /var/folders/wk/3v2syqm519z2kg7nqf4xtgy80000gn/T/emscripten_temp_vntpzynh/side.o -L~code/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic -L~code/emsdk/upstream/emscripten/src/lib ~code/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/crtbegin.o --no-whole-archive -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr

Main module link (emcc test.c libside.so -sMAIN_MODULE=2 -pthread -sPROXY_TO_PTHREAD -sEXIT_RUNTIME -o test.js -v):

 ~code/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -fPIC -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=~code/emsdk/upstream/emscripten/cache/sysroot -D__EMSCRIPTEN_SHARED_MEMORY__=1 -DEMSCRIPTEN -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -pthread -v -c test.c -o /var/folders/wk/3v2syqm519z2kg7nqf4xtgy80000gn/T/emscripten_temp_edrh4eqr/test_dynlink_unlock.o
emcc: warning: dynamic linking + pthreads is experimental [-Wexperimental]
 ~code/emsdk/upstream/bin/wasm-ld -o test.wasm /var/folders/wk/3v2syqm519z2kg7nqf4xtgy80000gn/T/tmpiiro9j26libemscripten_js_symbols.so --import-memory --shared-memory --strip-debug -Bdynamic --export=__stack_pointer --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=_emscripten_stack_alloc --export=_emscripten_thread_free_data --export=_emscripten_thread_crashed --export=_emscripten_dlsync_self --export=_emscripten_dlsync_self_async --export=_emscripten_proxy_dlsync --export=_emscripten_proxy_dlsync_async --export=__dl_seterr --export=__funcs_on_exit --export=__wasm_call_ctors --export=_emscripten_tls_init --export=setThrew --export=_emscripten_stack_restore --export=_emscripten_find_dylib --export=strerror --export=emscripten_get_sbrk_ptr --export=__heap_base --export=calloc --export=_emscripten_thread_init --export=emscripten_stack_set_limits --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=main --export-if-defined=__main_argc_argv --export-if-defined=fflush --export-if-defined=__memory_base --export-if-defined=__table_base --export-if-defined=emscripten_builtin_memalign --export-if-defined=pthread_self --experimental-pic --unresolved-symbols=import-dynamic --export-table --growable-table -z stack-size=65536 --no-growable-memory --initial-memory=16777216 --entry=_emscripten_proxy_main --stack-first --table-base=1 /var/folders/wk/3v2syqm519z2kg7nqf4xtgy80000gn/T/emscripten_temp_edrh4eqr/test_dynlink_unlock.o libside.so -L~code/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic -L~code/emsdk/upstream/emscripten/src/lib ~code/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/crtbegin.o ~code/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/crt1_proxy_main.o -lGL-mt-getprocaddr -lal -lhtml5 -lstubs-debug -lc-mt-debug -ldlmalloc-mt-debug -lcompiler_rt-mt -lc++-debug-mt-noexcept -lc++abi-debug-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr
 ~code/emsdk/node/22.16.0_64bit/bin/node ~code/emsdk/upstream/emscripten/tools/compiler.mjs -

Suggested fix

  if (p) {
+   do_write_unlock();
    onsuccess(user_data, p);
    return;
  }