Page-prealloc deadlock: a co-located kvcached engine hangs in `Init C++ PageAllocator` and never finishes startup

[jiarong0907]

Summary

With KVCACHED_PAGE_PREALLOC_ENABLED=true (the default), a kvcached vLLM engine can deadlock during KV-pool initialization: the process prints

Init C++ PageAllocator: ... enable_prealloc=1, ...

and then hangs forever in that C++ prealloc step — it never reaches uvicorn (Application startup complete), never binds /health, and the unit sits idle in futex_wait_queue (kernel futex wait) with no further output and no error.

We hit this in two situations, both removed by disabling prealloc:

1. Two co-located engines (init deadlock). Start a first kvcached engine (it comes up fine), then start a second kvcached engine on the same GPU while the first is running. The second engine deadlocks in Init C++ PageAllocator. The first is unaffected. Reproduced directly in our logs (details below).
2. A single engine (runtime stall). For the hybrid linear-attention coder Qwen/Qwen3.6-27B-FP8, a single kvcached engine with prealloc enabled stalls under sustained saturated load: the KV pool reports healthy/near-full but decode never makes progress and clients time out (600 s). Same fix.

Disabling prealloc (KVCACHED_PAGE_PREALLOC_ENABLED=false) fixes both — the engine maps KV pages on demand and starts/serves normally. Prealloc is the only changed variable.

Environment

Component	Value
GPU	NVIDIA RTX PRO 6000 Blackwell, 96 GB, sm_120
CUDA toolkit / driver	12.8 / 13.0
vLLM	0.19.0 (V1, VLLM_USE_V1=1)
torch	2.10.0+cu128
flashinfer	0.6.6 (VLLM_ATTENTION_BACKEND=FLASHINFER)
kvcached	0.1.5, built from source at upstream main incl. PR #367 / commit fea40de
kvcached mode	plugin autopatch (ENABLE_KVCACHED=true, KVCACHED_AUTOPATCH=1)
kvcached util	KVCACHED_GPU_UTILIZATION=0.95

Relevant C++ source: csrc/page_allocator.cpp (the Init C++ PageAllocator banner, alloc_page, prealloc_worker, map_pages), with the CUDA-VMM cuMemCreate/cuMemMap/cuMemRelease calls in csrc/page.cpp / csrc/ftensor.cpp.

Reproduction A — two co-located engines (init deadlock)

Two single-GPU kvcached vLLM servers launched sequentially, both with prealloc on:

• Engine 1 (coder): RedHatAI/Qwen2.5-Coder-32B-Instruct-FP8-Dynamic, port 8400, --gpu-memory-utilization 0.74, KVCACHED_CONTIGUOUS_LAYOUT=true, page 2 MB, KVCACHED_PAGE_PREALLOC_ENABLED=true.
• Engine 2 (co-tenant): Qwen/Qwen2-VL-7B-Instruct, port 8401, --gpu-memory-utilization 0.20, KVCACHED_CONTIGUOUS_LAYOUT=true, page 2 MB, KVCACHED_PAGE_PREALLOC_ENABLED=true.

Both engines use distinct IPC names (CODER vs VLMM) and distinct /tmp/kvcached-tp-* sockets, so this is not an IPC-name / shared-segment collision.

Observed

Engine 1 — initializes its PageAllocator with prealloc and starts normally:

Init C++ PageAllocator: num_layers=64, mem_size_per_layer=283MB, total_mem_size=36248MB,
  page_size=2MB, world_size=1, pp_rank=0, async_sched=1, contiguous_layout=1,
  enable_prealloc=1, num_kv_buffers=2, group_id=0, min_reserved_pages=5, max_reserved_pages=10
...
(APIServer) INFO:     Application startup complete.
(APIServer) INFO:     127.0.0.1 - "GET /health HTTP/1.1" 200 OK

Engine 2 — loads the model, captures CUDA graphs, prints the PageAllocator banner, then the log DEAD-ENDS. No Application startup complete, no /health, no error:

(EngineCore) INFO ... [core.py:283] init engine (profile, create kv cache, warmup model) took 10.58 seconds
Init C++ PageAllocator: num_layers=28, mem_size_per_layer=11MB, total_mem_size=667MB,
  page_size=2MB, world_size=1, pp_rank=0, async_sched=1, contiguous_layout=1,
  enable_prealloc=1, num_kv_buffers=2, group_id=0, min_reserved_pages=5, max_reserved_pages=5
<<< end of log — process hangs here indefinitely (futex_wait_queue) >>>

Init C++ PageAllocator … enable_prealloc=1 is the last line ever printed by the second engine. The health-wait times out and the server never serves.

Same engine 2 with enable_prealloc=0 → starts immediately:

Init C++ PageAllocator: num_layers=28, ... enable_prealloc=0, ...
(APIServer) INFO:     Application startup complete.
(APIServer) INFO:     127.0.0.1 - "GET /health HTTP/1.1" 200 OK

The deadlock reproduces with the dense coder as engine 1, so it is not specific to the hybrid model — it is triggered by bringing up a second prealloc-enabled engine while a first is live. (We also need prealloc=false on the co-located hybrid Qwen3.6 runs for the same reason.)

Reproduction B — single hybrid engine (runtime stall)

A single kvcached engine serving Qwen/Qwen3.6-27B-FP8 (hybrid linear attention: Gated-DeltaNet/Mamba + periodic full-attention; non-contiguous layout, KVCACHED_PAGE_SIZE_MB=4) with KVCACHED_PAGE_PREALLOC_ENABLED=true, under sustained saturated load:

• ~60 % of requests time out (600 s client timeout).
• The KV pool is reported healthy (≈199 k tokens) and near-full (KV usage ≈ 98.7 %) with ~21 requests "running", 0 preemptions — but decode never resumes (no forward progress).

Setting prealloc=false fixes it completely: the same single-engine workload then completes 112/112 and 76/76 turns at sub-second TTFT.

Signature / how to recognize it

• Last log line is Init C++ PageAllocator: ... enable_prealloc=1, ....
• The hung process (the vLLM EngineCore child that owns GPU memory) is parked in a futex wait (futex_wait_queue in /proc/<pid>/stack or wchan); CPU ~0 %.
• No exception, no CUDA error, no OOM — it is a lock wait, not a crash.
• Goes away entirely with KVCACHED_PAGE_PREALLOC_ENABLED=false.

Minimal repro + how to capture the confirming backtrace

Two single-GPU kvcached engines, both prealloc-enabled, started sequentially. The second hangs in Init C++ PageAllocator. (Any two models fit on the GPU work; sizes below are what we used.)

export ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 VLLM_USE_V1=1
export KVCACHED_GPU_UTILIZATION=0.95
export KVCACHED_PAGE_PREALLOC_ENABLED=true          # the trigger; set false to make it start

# Engine 1 — comes up fine (becomes the "already-running" engine)
KVCACHED_IPC_NAME=ENG1 \
vllm serve RedHatAI/Qwen2.5-Coder-32B-Instruct-FP8-Dynamic \
  --port 8400 --gpu-memory-utilization 0.74 --max-model-len 16384 \
  --no-enable-prefix-caching >eng1.log 2>&1 &
# wait until eng1 is healthy:
until curl -sf http://127.0.0.1:8400/health; do sleep 3; done

# Engine 2 — hangs at "Init C++ PageAllocator: ... enable_prealloc=1"
KVCACHED_IPC_NAME=ENG2 \
vllm serve Qwen/Qwen2-VL-7B-Instruct \
  --port 8401 --gpu-memory-utilization 0.20 --max-model-len 8192 \
  --no-enable-prefix-caching >eng2.log 2>&1 &

When eng2.log's last line is Init C++ PageAllocator: ... enable_prealloc=1, capture the stacks of the EngineCore child (the process that owns GPU memory — not the API-server parent). Find it and dump it:

# the EngineCore pid is printed in the log, e.g. "(EngineCore pid=NNNNN)"; or:
PID=$(pgrep -af 'VLLM::EngineCore|EngineCore' | awk '{print $1}' | tail -1)

# Python + native threads (shows the cond_.wait / map_pages frames):
py-spy dump --pid "$PID" --native            # pip install py-spy

# Kernel-side confirmation of the futex wait:
cat /proc/$PID/status | grep State           # expect: S (sleeping)
cat /proc/$PID/wchan; echo                    # expect: futex_wait_queue
for t in /proc/$PID/task/*; do echo "== $t =="; cat $t/stack 2>/dev/null; done

# (alternative) full C++ backtrace of every thread:
gdb -p "$PID" -batch -ex "thread apply all bt" 2>/dev/null

Expected confirmation: one thread parked in PageAllocator::alloc_page → std::condition_variable::wait, and the prealloc_worker thread inside PageAllocator::map_pages (→ broadcast_map_callback_ or FTensorAllocator::map_to_kv_tensors / CUDA-VMM cuMemMap). That pairing is the deadlock described below.

Why we believe the bug is in the prealloc path (not our harness)

• The only variable changed between hang and success is KVCACHED_PAGE_PREALLOC_ENABLED (enable_prealloc=1 → hang, =0 → starts). The model, ports, layout, page size, gmu, IPC name, and launch order are otherwise identical.
• The two engines do not share an IPC name or socket, ruling out a named-resource collision between processes.
• The reserved page counts are tiny (min_reserved_pages=5, max_reserved_pages=5..10, i.e. ~10–20 MB on a 96 GB GPU), so this is not a memory-exhaustion/OOM at prealloc time — it is a synchronization deadlock.

Root cause (from csrc/page_allocator.cpp @ main)

The hang is a structural deadlock between PageAllocator::alloc_page() and the background PageAllocator::prealloc_worker(). alloc_page() contains an unbounded condition-variable wait that can only be woken by the prealloc worker successfully mapping physical pages — and that mapping can fail to make progress exactly when a second engine is initializing on the same GPU (or when the device is near the utilization ceiling).

1. alloc_page() blocks forever when both page lists are momentarily empty:

std::shared_ptr<InternalPage> PageAllocator::alloc_page() {
  std::unique_lock<std::mutex> lock(lock_);
  page_id_t page_id = -1;
  while (page_id == -1) {
    if (!reserved_page_list_.empty()) { /* fast path */ ... return ...; }
    if (!free_page_list_.empty())     { /* slow path: pop + map_pages */ break; }
    if (num_free_pages_ <= 0)         { throw "No free pages left"; }
    if (!enable_page_prealloc_)       { throw "Inconsistent ... no free pages"; }
    cond_.wait(lock);          // <-- waits for the prealloc worker; NO timeout
  }
  ...
}

So the caller parks in cond_.wait(lock) whenever, at the same instant, reserved_page_list_ is empty and free_page_list_ is empty and num_free_pages_ > 0. It will only ever resume if prealloc_worker() adds to reserved_page_list_ (or returns the pages to free_page_list_) and calls cond_.notify_all().

2. The prealloc worker creates exactly that empty-list window, then does a blocking map outside the lock:

void PageAllocator::prealloc_worker() {
  while (prealloc_running_) {
    std::unique_lock<std::mutex> lock(lock_);
    while (!prealloc_needed_ && prealloc_running_) cond_.wait(lock);
    ...
    // pull pages OUT of free_page_list_ -> they are now in neither list ("in flight")
    for (...; i < to_reserve && !free_page_list_.empty(); ++i) {
      pages_to_reserve.push_back(free_page_list_.front());
      free_page_list_.pop_front();
    }
    lock.unlock();                 // <-- lock released with pages in flight
    map_pages(pages_to_reserve);   // <-- can BLOCK; only on success are pages
    lock.lock();                   //     re-inserted into reserved_page_list_
    reserved_page_list_.insert(... pages_to_reserve ...);
    cond_.notify_all();
  }
}

Between free_page_list_.pop_front() and the post-map_pages re-insert, the popped pages are in neither list. A concurrent alloc_page() in that window sees both lists empty and takes the unbounded cond_.wait. If map_pages() does not return promptly, both threads are stuck: the worker is inside the mapping call, and alloc_page() waits for a refill that never comes — the process sits in futex_wait_queue with no error and no timeout. This is the last-line-is-Init C++ PageAllocator hang we observe (the first KV allocation during engine warmup is the alloc_page() caller).

3. Why map_pages() stalls precisely for the second / loaded engine. map_pages() delegates the real work (and, for the worker-IPC / multi-rank path, a cross-process broadcast_map_callback_) down to the FTensor allocator's CUDA-VMM cuMem* mapping:

void PageAllocator::map_pages(const std::vector<page_id_t> &page_ids) {
  ...
  if ((world_size_ > 1 || should_use_worker_ipc()) && broadcast_map_callback_) {
    broadcast_map_callback_(world_size_, offsets);          // cross-process RPC; can block
  } else {
    auto allocator = FTensorAllocator::global_allocator(group_id_);
    bool success = allocator->map_to_kv_tensors(offsets);   // cuMemCreate/cuMemMap; can block/fail
    if (!success) throw std::runtime_error("Failed to map pages to KV tensors");
  }
}

When a first kvcached engine is already live (its own prealloc_worker + resize_watcher threads continuously calling mem_get_info/mem_create/mem_map), the second engine's map_pages() contends on the CUDA driver's device-wide VMM serialization and/or the worker-IPC broadcast at the worst possible moment — engine init — and stalls long enough (or indefinitely) to wedge alloc_page(). The same happens to a single hybrid Qwen3.6 engine under sustained saturation: near the KVCACHED_GPU_UTILIZATION=0.95 ceiling the reserve cannot complete, the reserved pool never refills, and decode wedges (KV ≈ 98.7 %, 0 progress).

Why prealloc=false fixes it: with prealloc off there is no worker and no in-flight window. reserved_page_list_ stays empty, so alloc_page() always finds a free_page_list_ entry and maps it synchronously on its own thread (the cond_.wait branch is never reached). Mapping may be slower under contention but it always completes or throws — it cannot deadlock against a second thread. This exactly matches the observed behavior.

Secondary bug: unsigned underflow in get_avail_physical_pages()

int64_t PageAllocator::get_avail_physical_pages() const {
  size_t avail_phy_mem_size = 0, total_phy_mem_size = 0;
  CHECK_GPU(gpu_vmm::mem_get_info(&avail_phy_mem_size, &total_phy_mem_size));
  size_t headroom = total_phy_mem_size * (1.0 - gpu_utilization_);
  avail_phy_mem_size = std::max(avail_phy_mem_size - headroom, (size_t)0); // underflow
  ...
}

avail_phy_mem_size and headroom are unsigned (size_t). When the device is fuller than the headroom (avail < headroom, i.e. < ~5 % free at util 0.95 ≈ < ~4.8 GiB on 96 GB), avail_phy_mem_size - headroom wraps to a huge value and std::max(..., 0) keeps it. The guard that is supposed to stop the worker from over-reserving under memory pressure is defeated, so the worker keeps attempting map_pages (which then blocks/fails) instead of backing off — feeding the deadlock above rather than avoiding it.

(Verified against ovg-project/kvcached main: csrc/page_allocator.cpp — alloc_page, prealloc_worker, map_pages, get_avail_physical_pages, trigger_preallocation; header csrc/inc/page_allocator.hpp.)

Workaround

Launch with page prealloc disabled (KV pages map on demand):

export KVCACHED_PAGE_PREALLOC_ENABLED=false

For co-located engines this is required on at least the second engine; we disable it on all co-located engines. For the single hybrid Qwen3.6 engine it is also required to avoid the runtime stall. With prealloc off, both the two-engine co-location and the single-engine hybrid run start and serve correctly.

Suggested fixes (csrc/page_allocator.cpp)

1. Don't let alloc_page() wait unboundedly. Replace the bare cond_.wait(lock) with a cond_.wait_for(...) and, on timeout (or whenever free_page_list_ is non-empty), fall through to the synchronous slow path (map_pages({page_id}) on the caller's thread) instead of depending on the worker. The caller should never be permanently blocked by the prealloc worker.
2. Close the in-flight-pages window. While prealloc_worker() is mapping, the popped pages are in neither list, so alloc_page() mistakes a transient reserve for "no pages." Either keep the pages discoverable to alloc_page() until mapping completes, or have alloc_page() treat "worker is mapping" distinctly from "truly out of pages."
3. Make a blocked/failed map_pages() recoverable and observable. A stalled broadcast_map_callback_ / map_to_kv_tensors() (CUDA-VMM contention from a second engine, or physical-memory exhaustion near the util ceiling) currently wedges startup silently. It should time out, log, and propagate — not hang in futex_wait_queue.
4. Fix the unsigned underflow in get_avail_physical_pages() so the over-reservation guard actually engages under memory pressure (e.g. compute in signed arithmetic or headroom = min(headroom, avail) before subtracting). Today, a nearly-full GPU makes the worker think it has near-unlimited headroom.
5. Guard concurrent cross-process VMM init. Bringing up a second prealloc-enabled engine on the same device must not be able to deadlock either engine; serialize or back off the reserve while another kvcached process is mid-VMM-init.

[cmssmc00]

I would like to work on the unsigned-underflow part of this issue as a scoped PR. I will keep it limited to get_avail_physical_pages() and avoid claiming to fix the full prealloc deadlock.