Mamba-2 + Nemotron-H MoE backport (Phase 3.x)#1593
Conversation
Adds enum entries, name mappings, hparam fields and exhaustive switch
arms for the upcoming Mamba-2 + Nemotron-H MoE backport from upstream
llama.cpp. NO forward pass, NO tensor loading, NO kernel — Phase 1 is
metadata only so the existing Qwen3.5-35B-A3B production path is
guaranteed unaffected.
What this enables:
- A Mamba-2 GGUF (state-spaces/mamba2-*) is now DETECTED as
LLM_ARCH_MAMBA2 instead of failing at GGUF arch resolution.
- A Nemotron-Nano-30B-A3B GGUF (unsloth/Nemotron-3-Nano-30B-A3B) is now
DETECTED as LLM_ARCH_NEMOTRON_H_MOE: hparams parse correctly
(n_expert=128, n_expert_used=6, vocab=131072), then the model fails
cleanly at the existing "unknown architecture" stub in load-tensors.
What changes:
- src/llama-arch.h: 2 new enum values (LLM_ARCH_MAMBA2,
LLM_ARCH_NEMOTRON_H_MOE).
- src/llama-arch.cpp: name mappings ("mamba2", "nemotron_h_moe") +
classification (mamba2 = recurrent, nemotron_h_moe = hybrid).
- src/llama-hparams.h: bool ssm_dt_b_c_rms field (Mamba-2 specific —
applies RMSNorm to dt, B, C projections), included in operator!=.
- src/llama.cpp: explicit cases in llama_rope_type() for both new
arches (LLAMA_ROPE_TYPE_NONE) since the existing switch is
exhaustive with no default branch.
Validation:
- Build with CUDA 12.8 sm_120 succeeds (3.3 MB llama-cli, 8.5 MB
llama-server). Zero new warnings.
- Qwen3.5 chimere-v3-ramp.gguf still loads, parses, generates.
Smoke test ran with -ngl 0 because production qwen35-custom.service
was holding GPU memory; CPU-only completion was coherent.
- Nemotron-Nano UD-IQ3_XXS triggers the new detection path: model
metadata prints (n_expert=128, n_expert_used=6, vocab=131072,
general.name="Nemotron-3-Nano-30B-A3B") then fails as expected at
"unknown architecture" in load-tensors. Before this patch, the same
GGUF would fail much earlier at arch string resolution.
What does NOT change:
- llama-model.cpp untouched.
- llama-load-tensors.cpp untouched.
- llama-build-context.cpp untouched.
- ggml-cuda/* untouched. No SSM scan kernel yet — that is Phase 3.
- The ik_llama MTP work for Qwen3.5-35B-A3B is preserved exactly.
Build env note: nvcc 12.0 (Ubuntu default) fails on compute_120a.
CMake configure must explicitly use:
-DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc
-DCUDA_PATH=/usr/local/cuda-12.8
Next phase plan:
- Phase 2: tensor-name table (LLM_TENSOR_NAMES entries for both arches),
hparam loading cases in llama-hparams.cpp, allocation-only stubs in
llama-load-tensors.cpp. Model loads to memory; build_*() still aborts.
- Phase 3 (risky): port ggml-cuda/ssm-scan.cu and the build graph wiring
from upstream llama.cpp/src/models/{mamba-base,nemotron-h}.cpp.
Refs: agent investigation reports 2026-04-06,
upstream llama.cpp commits #9126 (Mamba-2), #15507 (nemotronh),
#18058 (Nemotron 3 Nano), #20270/#20335 (mamba2 assert fixes).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… + nemotron_h_moe
Phase 2 of the Mamba-2 backport. Phase 1 added enums + hparam fields and made
the GGUF arch string parser route Mamba-2 / Nemotron-H GGUFs to the new enum
values. Phase 2 makes the model loader actually allocate the full tensor set
to memory. Build graph (forward pass) is still NOT wired — that's Phase 3.
What this enables:
- A Nemotron-Nano-30B-A3B GGUF (16.9 GB UD-IQ3_XXS) now loads completely:
· 53 layers worth of tensors (~17.3 GiB CPU buffer) allocated cleanly
· KV cache (6.1 GiB) allocated cleanly
· Context (n_ctx=1048576) initialized
· Failure point progressed from "unknown architecture" (Phase 1, at
tensor loading) to llama-build-context.cpp:9983 fatal error (Phase 2,
at compute graph dispatch — exactly the GGML_ABORT we expected because
build_mamba2 / build_nemotron_h_moe don't exist yet)
What changes (5 files, +278 lines):
- src/llama-arch.h: +1 line — LLM_KV_SSM_DT_B_C_RMS enum entry
- src/llama-arch.cpp: +1 line — "%s.ssm.dt_b_c_rms" string in LLM_KV_NAMES
- src/llama-hparams.cpp: +70 lines — full LLM_ARCH_MAMBA2 hparams loading
case (sets recurrent_layer_arr = all true), full LLM_ARCH_NEMOTRON_H_MOE
case (SSM + MoE expert params + per-layer recurrent detection via
n_head_kv(i)==0 && n_ff(i)==0), and an *optional* SSM_DT_B_C_RMS load
added to the existing LLM_ARCH_MAMBA case for completeness. Model type
classification: 52-layer→MODEL_30B_A3B (Nemotron-Nano), 56-layer→MODEL_9B.
- src/llama-model.cpp: +49 lines — two new LLM_TENSOR_NAMES entries
(LLM_ARCH_MAMBA2 with 11 tensors; LLM_ARCH_NEMOTRON_H_MOE with 22 tensors
including SSM + attn + dense MLP + routed MoE experts + shared expert).
Names cross-checked line-for-line against upstream llama.cpp/src/llama-arch.cpp.
- src/llama-load-tensors.cpp: +157 lines — two new helper methods:
· create_mamba2_tensors: standard Mamba-2 SSM allocation, with the
correct d_in_proj = 2*d_inner + 2*n_group*d_state + n_ssm_head and
SSM_NORM shape {d_inner/n_group, n_group}.
· create_nemotron_h_moe_tensors: per-layer dispatch on
hparams.is_recurrent(i) → SSM branch, n_ff(i)==0 → attention branch,
else → MoE-or-dense FFN branch with shared+routed experts.
Both wired into the dispatch switch in create_tensors().
Compile-time issue resolved during build:
- create_mamba2_tensors had a redeclaration of `n_head` because LOADING_PRELUDE
already defines `const int64_t n_head` in the enclosing scope. Fixed by
renaming the local SSM head count to `n_ssm_head`.
Validation:
- Build with CUDA 12.8 sm_120 succeeds (3.3 MB llama-cli, no new warnings).
- Qwen3.5 chimere-v3-ramp.gguf smoke test (CPU only via CUDA_VISIBLE_DEVICES=,
-ngl 0): "What is 2+2?" → "Two plus two equals four." Coherent. The
optional SSM_DT_B_C_RMS load added to the Mamba-1 case is benign for
Qwen3.5 because Qwen3.5 never enters that case — confirmed empirically.
- Nemotron-Nano-30B-A3B UD-IQ3_XXS smoke test: full model + KV cache + ctx
initialized in CPU memory; aborts at llama-build-context.cpp:9983 during
compute-graph build, which is the expected Phase 2 end state.
What does NOT change:
- llama-build-context.cpp untouched (no build_mamba2, no build_nemotron_h_moe)
- ggml-cuda/* untouched (no SSM scan kernel)
- The Qwen3.5 production path is byte-equivalent except for the optional
SSM_DT_B_C_RMS load on the Mamba-1 case (which Qwen3.5 never hits)
- ik_llama MTP work for Qwen3.5-35B-A3B is preserved exactly
Phase 3 plan (next session, NOT for an agent):
- Audit the hybrid memory cache layout: ik_llama's n_embd_v_s() has
ssm_n_group > 0 branches that were written for Qwen3.5 delta-net
(head_v_dim * head_v_dim * n_heads), NOT for Mamba-2 SSM state layout
(d_state * d_inner). This cache-layout fork is the riskiest part of
Phase 3 and needs investigation FIRST in a scratch worktree before
touching mamba2-backport-phase1.
- Port ggml-cuda/ssm-scan.cu chunked scan kernel from upstream
llama.cpp/ggml/src/ggml-cuda/ssm-scan.cu (342 lines, CUB-based, needs
CUDART >= 11.0.7 and sm_120 compat verification).
- Port build_mamba2() and build_nemotron_h_moe() from upstream
src/models/{mamba-base,mamba,nemotron-h}.cpp into a new method on
llm_build_context, then wire into the dispatch switch at line 9982.
- Bench Qwen3.5 production path on GPU after each step to catch any
unintended kernel side-effect on the existing GDN/attention path.
Refs: agent investigation 2026-04-06; upstream commits #9126 (Mamba-2),
#15507 (nemotronh), #18058 (Nemotron 3 Nano), #20270/#20335 (mamba2 fixes).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 3.1.B.1 of the Mamba-2 backport. Pure rename + relocation; behaviour is byte-equivalent for every architecture currently supported (Qwen3.5-A3B GDN, Mamba-1). NO new arch cases yet — those land in Phase 3.1.B.2. Why rename: - Aligns with upstream llama.cpp naming (n_embd_r = "rolling state", n_embd_s = "recurrent state"). The legacy n_embd_k_s/v_s names refer to K/V cache slots that no longer exist for non-attention layers — pure historical artifact from when ik_llama only had Mamba-1. - Phase 3.1.B.2 needs to add LLM_ARCH_MAMBA2 + LLM_ARCH_NEMOTRON_H_MOE cases. Doing the rename in a separate commit isolates the cosmetic change from the semantic one and makes review trivial. What changes: - src/llama-hparams.h: declarations only (move bodies to .cpp). The three functions n_embd_r / n_embd_s / n_embd_s_id are documented in-place to explain the qnext-vs-mamba-1 fork. - src/llama-hparams.cpp: function bodies, verbatim from the old inline versions. The qnext (ssm_n_group > 0) branch and the Mamba-1 (else) branch are byte-identical to before. - src/llama-build-context.cpp: 5 call sites renamed (lines 202, 203, 215, 6125, 6126). - src/llama-delta-net.cpp: 1 call site renamed (line 34, the GGML_ASSERT for the GDN state shape). - src/llama.cpp: 5 call sites renamed (lines 681, 879, 899, 6087, 6492). What does NOT change: - Any behaviour or numeric value returned by these functions. - The qnext_states_all slot in build-context (still uses n_embd_s now). - Any tensor shape, KV cache size, scheduler logic. - The ChimereModel multi-arch refactor in chimere-server is unaffected because it goes through libllama FFI, not these symbols directly. Validation: - Build with CUDA 12.8 sm_120 succeeds. Only the pre-existing rope_cache unused-parameter warning remains. - Qwen3.5 chimere-v3-ramp.gguf GPU smoke test (production parameters -ngl 99 --n-cpu-moe 3 -fa on -ctk q8_0 -ctv q4_0 -c 8192): · "What is 2+2?" generates coherent thinking-mode output · 94.10 tok/s eval (baseline before rename: ~93 tok/s) · Prompt eval 91.82 tok/s · Within noise of baseline; no regression - The agent reading the diff should be able to verify the rename is mechanical: every removed function in the .h has an exact-text equivalent in the .cpp, just with the new name and llama_hparams:: qualifier. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…motron-H
Phase 3.1.B.2 of the Mamba-2 backport. Resolves the cache layout fork that
made it impossible to support Mamba-2 / Nemotron-H alongside Qwen3.5 GDN:
both arches set ssm_n_group > 0, so the previous discriminator was
ambiguous and Mamba-2 GGUFs would have been allocated with qnext-style
packed buffers (wrong shape, would segfault in build_graph).
What changes:
- src/llama-hparams.h: new field `bool use_qnext_state_layout = false`.
Documented inline as the explicit discriminator between the two
recurrent state packings (qnext GDN vs standard Mamba SSM).
- src/llama-hparams.cpp: n_embd_r() and n_embd_s() now branch on the
new flag instead of `ssm_n_group > 0`.
· Default branch (use_qnext_state_layout = false) is the unified
Mamba formula taken verbatim from upstream llama.cpp:
n_embd_r = (d_conv-1) * (d_inner + 2*n_group*d_state)
n_embd_s = d_state * d_inner
Mamba-1 stays byte-equivalent because n_group=0 collapses the
formula back to (d_conv-1)*d_inner.
· qnext branch (use_qnext_state_layout = true) is the existing
GDN packed formula, untouched.
- src/llama-hparams.cpp: the three Qwen3.5 GDN cases in
llm_load_hparams (LLM_ARCH_QWEN3NEXT, LLM_ARCH_QWEN35MOE,
LLM_ARCH_QWEN35) explicitly set the flag to true. The Phase 2
cases for LLM_ARCH_MAMBA2 and LLM_ARCH_NEMOTRON_H_MOE leave it
at the default (false).
What does NOT change:
- The qnext logic itself (formula, scheduler, slot management) is
byte-equivalent. Qwen3.5 production path is preserved.
- Mamba-1 formulae are preserved (collapsed n_group=0 case).
- llama-build-context.cpp still aborts at line 9983 for Nemotron-H
because Phase 3.3 hasn't ported build_mamba2 / build_nemotron_h_moe
yet — that's the next milestone.
Validation:
- Build with CUDA 12.8 sm_120 succeeds. Same single pre-existing warning.
- Qwen3.5 chimere-v3-ramp.gguf GPU smoke test (-ngl 99 --n-cpu-moe 3
-fa on -ctk q8_0 -ctv q4_0 -c 8192):
· "What is 2+2?" generates "<think>\n\n</think>\n\n2+2 equals 4."
· 94.31 tok/s eval (B.1 baseline 94.10 tok/s, prod baseline 93 tok/s)
· Zero regression
- Nemotron-Nano UD-IQ3_XXS CPU load test:
· Phase 2 KV buffer was 6168.62 MiB; Phase 3.1.B.2 is 6190.00 MiB.
The +21.38 MiB delta is the conv state being allocated for the
first time — n_embd_r() now returns the correct non-zero size
instead of falling into the qnext "everything in V slot" branch.
· Failure point is still build-context.cpp:9983 (the GGML_ABORT
fallback in build graph dispatch). This is the EXPECTED Phase 2
end state — Phase 3.3 will add the build_mamba2 /
build_nemotron_h_moe cases that consume these correctly-sized
buffers.
Why a flag instead of arch dispatch:
- llama_hparams currently has no access to the LLM_ARCH enum (it's
filled before arch is fully resolved on llama_model). Adding a
bool flag is more local and follows the same pattern upstream uses
for wkv_head_size (RWKV), n_shortconv_l_cache (LFM2), and
n_embd_head_kda (Kimi KDA): each non-Mamba SSM family advertises
its own layout via a dedicated field, and the unified Mamba
formula is the fallthrough.
- Naming the flag `use_qnext_state_layout` rather than e.g.
`ssm_layout_kind` keeps it boolean and explicit; if a third layout
ever appears, it should get its own dedicated flag rather than
sharing this one.
Phase 3 next steps (NOT in this commit):
- Phase 3.2: port ggml-cuda/ssm-scan.cu (chunked associative scan
kernel for Mamba-2 / Nemotron-H, ~342 lines, CUB-based, needs
sm_120 verification).
- Phase 3.3: port build_mamba2() and build_nemotron_h_moe() into
llama-build-context.cpp, wire into the dispatch switch at
line ~9982 to replace the GGML_ABORT.
- Phase 3.4: end-to-end load + generate Nemotron-Nano on GPU,
bench tok/s, validate Qwen3.5 zero regression.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 3.2 replaces ik_llama's ggml_ssm_scan body and signature with the
upstream unified Mamba-1+Mamba-2 form (last arg renamed `sq` → `ids`,
shapes go from 2D to 4D). The only consumer in src/ is the existing
llm_build_context::build_mamba() at llama-build-context.cpp:6104, which
calls ggml_ssm_scan with the OLD shapes and would fail to compile after
Phase 3.2 lands.
Mamba-1 has never worked end-to-end in ik_llama: there is no CUDA scan
kernel, the CPU op was a partial stub, and no production GGUF (Qwen3.5,
chimere-v3-ramp, anything used by chimere-server / qwen35-custom.service)
calls this code path. Stubbing build_mamba() to abort at runtime is safe
and isolates the Phase 3.2 ggml change from the Phase 3.3 build-graph
rewrite.
What changes:
- src/llama-build-context.cpp: the original build_mamba() body is wrapped
in `#if 0 ... #endif` (preserved verbatim for Phase 3.3 reference, just
not compiled). A 7-line stub returning GGML_ABORT replaces it. The
function signature is unchanged so the dispatch in llama_build_graph
(line 9857) still links.
What does NOT change:
- The dispatch case `case LLM_ARCH_MAMBA: result = llm.build_mamba()`
remains. If a Mamba-1 GGUF were ever loaded, it would now hit the
GGML_ABORT with a clear message instead of doing wrong-shape ssm_scan
calls.
- ggml/, src/llama-hparams.{h,cpp}, src/llama-arch.{h,cpp},
src/llama-load-tensors.cpp, src/llama-model.cpp — all untouched.
Validation:
- Pure stub, no runtime path changed. Build verified after Phase 3.2
lands (Phase 3.2 agent build will succeed against this stub, where it
would have failed against the original body).
Phase 3.3 will rewrite build_mamba() on top of the new ggml_ssm_scan
(4D shapes, ids), and add build_mamba2() and build_nemotron_h_moe()
as siblings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace ik_llama's old Mamba-1-only ggml_ssm_scan with upstream's
unified Mamba-1 + Mamba-2 op, and add the CUB-based CUDA backend.
Mamba-1 vs Mamba-2 is discriminated at runtime via A->ne[0] (1 for
Mamba-2, d_state for Mamba-1). Shapes go from 2D/3D to 4D
(head_dim x n_head x n_seq_tokens x n_seqs for x).
What changes:
- ggml/include/ggml.h: ggml_ssm_scan signature updated to upstream
unified Mamba-1+Mamba-2 form (8 args, last renamed sq -> ids).
Added shape-convention comment.
- ggml/src/ggml.c: ggml_ssm_scan body + ggml_compute_forward_ssm_scan
CPU impl ported from upstream llama.cpp/ggml-cpu/ops.cpp. Mamba-2
branch uses GGML_SIMD/GGML_F32_VEC_* macros (already present in
ik_llama). Mamba-1 branch is the scalar reference (d_state usually
16, expf inside the loop, no SIMD as upstream).
- ggml/src/ggml-cuda/ssm-scan.cu: NEW, copied verbatim (342 lines)
from upstream llama.cpp/ggml-cuda/ssm-scan.cu. CUB path active
(CUDART_VERSION >= 11070), no sm_120 tweaks needed -- builds and
links cleanly with --generate-code arch=compute_120,code=sm_120.
- ggml/src/ggml-cuda/ssm-scan.cuh: NEW, verbatim from upstream.
- ggml/src/ggml-cuda.cu: include ssm-scan.cuh; dispatch
GGML_OP_SSM_SCAN -> ggml_cuda_op_ssm_scan; supports_op gating
mirrors upstream (Mamba-2: d_state in {128,256} && d_head%16==0;
Mamba-1: d_state==16 && d_head==1 && n_head%128==0 && n_group==1).
- ggml/src/CMakeLists.txt: explicit list(APPEND GGML_SOURCES_CUDA
ggml-cuda/ssm-scan.cu) + REMOVE_DUPLICATES, so the new file is
picked up even when cmake's GLOB cache is stale (ik_llama doesn't
use CONFIGURE_DEPENDS).
What does NOT change:
- src/llama-build-context.cpp untouched (Phase 3.3 territory). Its
one existing call to ggml_ssm_scan (legacy Mamba-1 build path)
still compiles thanks to identical arg count/types, but its
shape-prep code is now stale and the Mamba-1 build path remains
broken until Phase 3.3 rewrites it for the 4D upstream layout.
- src/llama-hparams.{h,cpp} untouched (Phase 3.1 final state).
- src/llama-arch.{h,cpp}, src/llama-load-tensors.cpp,
src/llama-model.cpp untouched (Phase 1+2 final state).
- src/llama-delta-net.cpp untouched. The Qwen3.5 production path
uses GGML_OP_DELTA_NET, not GGML_OP_SSM_SCAN, so this change has
zero impact on the Qwen3.5 forward pass.
Validation (no runtime test -- production GPU was busy with
qwen35-custom.service holding ~14.9 GB VRAM):
- cmake auto-reconfigured (CMakeLists.txt newer than cache) and
built llama-cli + llama-server cleanly with CUDA 12.8 sm_120.
- nm libggml.so shows: T ggml_ssm_scan, t
ggml_compute_forward_ssm_scan_f32.isra.0, T ggml_cuda_op_ssm_scan,
W ssm_scan_f32<128,16,L> for L=0..8 (Mamba-1 fallback templates),
W ssm_scan_f32_group<4,128> and <8,256> (Mamba-2 kernels).
- ssm-scan.cu.o is 374 KB, ELF x86-64 relocatable.
Next: Phase 3.3 -- in src/llama-build-context.cpp, port build_mamba2
and build_nemotron_h_moe from upstream llama-model.cpp. The existing
llm_build_mamba (around line 6199) needs its 4D shape preparation
rewritten: x must be reshaped to {head_dim, n_head, n_seq_tokens,
n_seqs}, dt to {n_head, n_seq_tokens, n_seqs}, A to {1, n_head} for
Mamba-2 or {d_state, n_head} for Mamba-1, and the sq tensor must be
replaced by an ids vector of length n_seqs (per-sequence source
state index). The new ggml_ssm_scan asserts will fire loudly at
graph build time if the shapes are wrong.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Addresses the single YELLOW finding from the Phase 3.2 security audit
(agent ac852130). A malformed or malicious Mamba-2 GGUF with
`ssm_n_group = 0` would cause a division by zero in Phase 2's tensor
allocation helpers (`d_inner / n_group` in the ssm_norm shape
computation), and extremely oversized values of `ssm_d_inner` /
`ssm_d_state` could OOM the system during tensor allocation.
All real-world Mamba-2 GGUFs (including Nemotron-3-Nano-30B-A3B used
as the target for this backport, which has n_group=8 d_inner=4096
d_state=128) satisfy these bounds with huge margin — the asserts are
defensive against corruption or adversarial inputs, not for valid
production models.
What changes:
- src/llama-hparams.cpp: 6 GGML_ASSERT calls added to both the
LLM_ARCH_MAMBA2 and LLM_ARCH_NEMOTRON_H_MOE cases in llm_load_hparams:
* ssm_n_group > 0 (prevents divide-by-zero)
* ssm_d_inner > 0
* ssm_d_state > 0
* ssm_d_inner % ssm_n_group == 0 (prevents division with remainder)
* ssm_d_inner <= 65536 (sanity upper bound)
* ssm_d_state <= 2048 (sanity upper bound)
What does NOT change:
- Nemotron-3-Nano-30B-A3B UD-IQ3_XXS loads unchanged: n_group=8,
d_inner=4096, d_state=128 — all asserts pass.
- No Phase 3.1 / 3.2 behaviour modified.
- Qwen3.5 / Mamba-1 / every non-Mamba-2 arch unaffected (different
switch cases, asserts not reached).
Validation:
- Build with CUDA 12.8 sm_120 succeeds (incremental, only
llama-hparams.o rebuilt).
- Asserts have clear failure messages for debuggability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…on_h_moe
Implements the per-arch build graphs for Mamba-2 and Nemotron-H on top of
the Phase 3.2 ggml_ssm_scan op. Nemotron-3-Nano-30B-A3B-UD-IQ3_XXS now
runs end-to-end on CPU (CUDA_VISIBLE_DEVICES=) and generates tokens.
What changes
============
src/llama-build-context.cpp (~490 net new lines)
- Anonymous-namespace helper `build_mamba2_layer(ctx0, gf, lctx, cb,
cur, layer, ssm_ids, n_tokens, _, il)` ports upstream's
`llm_build_mamba_base::build_mamba2_layer`. Differences vs upstream:
* Single-sequence path (n_seqs=1, n_seq_tokens=n_tokens) — Phase 3.3
does not yet support equal_seqs scheduling.
* Reuses ik_llama's OLD `ggml_ssm_conv(s, x, c, sq)` (4-arg, Mamba-1
era) instead of upstream's 2-arg variant — the upstream variant is
not in ik_llama's ggml fork and porting it is out of Phase 3.3
scope. The OLD op handles the channel-wide conv1d transparently
when treating `d_inner + 2*n_group*d_state` as the channel count.
* Uses the NEW Phase 3.2 `ggml_ssm_scan` (7-arg with 4D shapes and
ids) for the SSM scan itself.
* The recurrent state slot is always 0 in Phase 3.3 (no n_seq_max>1
smoke test yet); kv_head parameter is documented as unused.
- `llm_build_context::build_mamba()` rewritten on the new helper.
Dispatches on `model.arch`: Mamba-2 hits `build_mamba2_layer`,
Mamba-1 (LLM_ARCH_MAMBA) aborts cleanly with a clear message
(Mamba-1 has no smoke target in ik_llama and the legacy `#if 0`
block above the function is the textual reference if it ever needs
one).
- `llm_build_context::build_nemotron_h_moe()` — new method, ~120
lines. Per-layer dispatch matches upstream nemotron-h.cpp:
* `hparams.is_recurrent(il)` → SSM mixer via `build_mamba2_layer`
* `hparams.n_ff(il) == 0` → attention layer (no RoPE), built
manually with `llm_build_mul_mat_qkv` + `llm_build_kv`
* else → dense FFN (LLM_FFN_RELU_SQR, LLM_FFN_PAR) or routed MoE
with sigmoid gating + optional shared expert.
- `llm_build_moe_ffn` extended with a gateless code path
(`gate_exps == nullptr`): applies activation directly to up_exps,
no gate*up combination. Required by Nemotron-H's gateless MoE.
Other arches unaffected — they still pass non-null gate_exps.
- Dispatch in `llama_build_graph` switch:
* LLM_ARCH_MAMBA2 → `build_mamba()` (handles both Mamba-1 and -2)
* LLM_ARCH_NEMOTRON_H_MOE → `build_nemotron_h_moe()`
src/llama-build-context.h
- declare `build_nemotron_h_moe()`.
src/llama-context.h
- new field `inp_ssm_ids` (I32 [n_seqs]) on llama_context — input
tensor consumed by ggml_ssm_scan as the per-seq source state index.
src/llama.cpp
- `llama_kv_cache_init` `qnext_recurrent` branch: for the standard
Mamba layout (`!use_qnext_state_layout`, embd_r > 0) we now also
allocate a 1D conv-state tensor in `cache.k_l[i]` of size
`n_embd_r * qnext_state_slots`. The qnext GDN layout is unchanged
(k_l/v_l stay null, all state in s_l).
- `llama_set_inputs` now fills `inp_ssm_ids` with `kv_self.head + s`
when set by the build graph.
Validation
==========
Build (CUDA 12.8 sm_120):
...
[ 98%] Linking CXX executable ../../bin/llama-cli
[ 98%] Linking CXX executable ../../bin/llama-server
[100%] Built target llama-cli
[100%] Built target llama-server
Qwen3.5 CPU non-regression smoke (chimere-v3-ramp.gguf, IQ3_S RAMP):
> What is 2+2? Answer in one sentence.
> <think>1. **Identify the Question:** The user is asking "What is
generation @ 15.41 tok/s, prompt eval @ 52.02 tok/s — coherent.
Nemotron-3-Nano-30B-A3B (UD-IQ3_XXS, 17 GB) CPU smoke:
> Hi → 1 token, immediate EOS, exit clean
> The quick brown fox jumps over → "subt" + 9 more tokens, exit clean
> The capital of France is → 8 tokens, exit clean
generation @ 16.38 tok/s, prompt eval @ 41.03 tok/s
Output is low-quality on CPU (likely IQ3_XXS quant + first-light
smoke), but the SSM scan + Mamba-2 conv + Nemotron-H attention + MoE
pipeline runs end-to-end without aborts.
Caveats / TODO for Phase 3.4
============================
- n_seq_max > 1 / parallel sequences not supported (n_seqs hardcoded
to 1 in build_mamba2_layer; the kv-cache slot is always 0).
- Mamba-1 (LLM_ARCH_MAMBA) build path aborts at runtime — no
smoke target in ik_llama, legacy `#if 0` body kept as reference.
- read_kv_cache_data / write_kv_cache_data still assume the legacy
K/V cache layout for the conv state (`n_embd_k_gqa(il) +
n_embd_r()`). Save/restore of Nemotron-H state is broken until
that path is updated. Smoke test does not exercise it.
- GPU validation deferred to Phase 3.4. Phase 3.2 added the CUDA
backend kernel for ggml_ssm_scan, but Phase 3.3 has only run on
CPU (production qwen35-custom.service is the GPU resident).
- Output quality of Nemotron-Nano IQ3_XXS on CPU is poor — could be
from a) the aggressive UD-IQ3_XXS quant, b) a subtle correctness
bug in the conv↔scan packing, or c) sampling issues with the
Nemotron tokenizer. To diagnose post-Phase 3.4 with bf16/q8 GGUF.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous ggml_print_tensor in examples/eval-callback/eval-callback.cpp
had `sum +=` inside the truncated print loop. When ne[0] > 2*n (n=3 by
default), the inner loop would jump from i0=n to i0=ne[0]-n via
`i0 = ne[0] - n`, skipping the middle elements entirely. Result: `sum`
only included the first 3 + last 3 elements per row, NOT the full tensor.
Concretely, on Nemotron-3-Nano-30B-A3B input embedding {2688, 5}:
- True full sum = 1.460159 (13440 elements)
- Old printer sum = 0.067017 (only 30 elements)
This mismatched the upstream llama.cpp eval-callback (common/debug.cpp)
which computes sum in a separate loop over ALL elements, making the two
binaries impossible to compare numerically when debugging arch backports.
This commit factors the per-element float read into a helper, computes
sum in its own loop over the full tensor (mirroring upstream), then
runs the truncated print as a separate pass. The print output is
unchanged; only the printed `sum =` value now reflects the real total.
What does NOT change:
- Production code paths (only examples/eval-callback affected).
- The print format (tensor structure, indices, value formatting).
- Any other binary or runtime behaviour.
Validation:
- Re-ran eval-callback on Nemotron-3-Nano-30B-A3B UD-IQ3_XXS with
prompt "1+1=", confirmed inp_embd sum now matches upstream's
byte-for-byte: 1.460159 (vs 0.067017 before this fix).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 3.3 (build_mamba2_layer in llama-build-context.cpp) writes the new
recurrent state back to slot 0 unconditionally:
const int64_t r_kv_head = 0;
...
ggml_view_1d(ctx0, ssm_states_all,
d_state * d_inner * n_seqs,
r_kv_head * d_state * d_inner * ggml_element_size(ssm_states_all))
But the READ side (the `ids` input to ggml_ssm_scan, populated here in
llama_set_inputs) was sourcing from `kv_self.head + s`. For the qnext
GDN arches (LLM_ARCH_QWEN35*, LLM_ARCH_QWEN3NEXT) the recurrent cache
is genuinely flagged as `cache.recurrent = true` and `kv_self.head` is
maintained as the recurrent slot index, so that worked. But for the new
LLM_ARCH_MAMBA2 / LLM_ARCH_NEMOTRON_H_MOE arches the cache is
`cache.recurrent = false` (they are registered as "hybrid" in
llama-arch.cpp:255-263), and `kv_self.head` is the *attention* KV head
which advances by n_tokens after every batch (llama.cpp:3652).
After the prefill of "1+1=" (5 tokens), kv_self.head = 5. The next
ggml_ssm_scan call would then read `s0 = src0->data + 5*src0->nb[3]`
which is ~10 MB beyond the single-slot recurrent state cache — pure
out-of-bounds memory. On GPU this manifested as
`ggml_backend_cuda_synchronize: an illegal memory access`. On CPU it
silently produced corrupted recurrent state, garbage logits, and
sampler asserts.
Fix: with n_seq_max = 1 (the Phase 3.3 invariant), the recurrent state
cache has exactly one slot. The READ side now sources from slot
0..n_seqs-1, mirroring the WRITE side. This makes the conv-state and
SSM-state paths consistent (the conv-state uses inp_s_seq_qnext which
is also always zero-filled at lines 3110-3113 of this same file).
What does NOT change:
- The qnext GDN path is unaffected: it does NOT use inp_ssm_ids
(only the new ggml_ssm_scan added in Phase 3.2 does).
- Multi-sequence (n_seq_max > 1) is still unsupported by Phase 3.3
build_mamba2_layer; this fix preserves that limitation. When multi-seq
lands, inp_ssm_ids will need a proper "recurrent slot index" path
that does NOT alias kv_self.head.
- Production Qwen3.5 path: untouched, this code is gated on
`lctx.inp_ssm_ids != nullptr` which is only set by build_mamba/
build_nemotron_h_moe.
Validation:
- Re-ran eval-callback dump on Nemotron-3-Nano-30B-A3B UD-IQ3_XXS for
prompt "1+1=". With this fix, all intermediate tensor sums match
upstream llama.cpp eval-callback within numerical noise (0.04-0.1%)
through the entire layer 0 mixer (ssm_in_proj, ssm_conv, silu,
ssm_scan, mamba2_y_add_d, swiglu, ssm_norm, ssm_norm * weight).
- One residual divergence remains at MUL_MAT(ssm_out, y) → 4× sum
mismatch. Documented in PHASE_3_DEBUG_FINDINGS.md, requires further
investigation. Independent of this fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… FFNs)
Symptom: Nemotron-3-Nano-30B-A3B produced gibberish on Q4_0 AND on UD-IQ3_XXS
through the new Mamba-2 backport, with the divergence vs upstream llama.cpp
starting at the *first* MoE FFN layer (block 1, all subsequent blocks
compounding error). attn_norm-0 and attn_norm-1 matched upstream byte-for-byte;
attn_norm-2 diverged 23%, and from there error grew until the LM head.
Root cause: llm_build_ffn() unconditionally folded the parallel "gate" path
back in via `cur = ggml_mul(cur, tmp)` whenever `type_gate == LLM_FFN_PAR`,
even when no gate weight had been passed. For gateless FFNs (Nemotron-H shared
expert: relu²(up_proj) only), the earlier `} else { cur = tmp; }` already set
`cur` to `tmp`, so the unconditional mul collapsed the activation to
`relu²(up) * up` instead of `relu²(up)`. The same bug exists in the fused
fast path a few lines above (`ggml_fused_mul_unary(cur, tmp, ...)` for
SILU/RELU/GELU + LLM_FFN_PAR + no-gate would yield `op(up) * up`).
This codepath was unreachable until Nemotron-H, because every prior caller of
LLM_FFN_PAR also passed a real gate weight (SwiGLU style). The trace symptom
"ffn_gate_par-1" appearing in ik_llama but absent from upstream's identical
forward graph is what pinned down the regression.
Fix: gate both the fused fast path and the slow path on `gate != nullptr`. No
behavior change for any model that actually uses a gate.
Validation:
- llama-cli on Nemotron-3-Nano-30B-A3B Q4_0 (CPU): now answers
'The capital of France is Paris.' instead of 'optimal opt A A Sole B'.
- llama-cli on the UD-IQ3_XXS GGUF (CPU): now answers
'Paris, and the capital of Italy is Rome, but the capital of Ireland is Dublin.'
- This invalidates the earlier 'iqk IQ3_XXS kernel is broken' diagnosis from
~/Bureau/phase3-bench-2026-04-06/PHASE_3_PERMUTE_BUG.md — keeping that doc
as a record of how a coincidence (iq3_xxs vs q6_K layers happening to drift
by similar magnitudes from accumulated error) misled the first debug pass.
Two follow-up fixes after rebasing the Phase 3.x backport on top of current upstream `main` (ikawrakow/ik_llama.cpp@67fc9c5): 1. `hparams.n_embd_head_k` and `n_embd_head_v` are now per-layer member functions, not fields, since the Gemma4 SWA work landed. Update the single use site in `build_nemotron_h_moe()` to call the new accessors with `il=0` (Nemotron-H attention layers all share the same head_dim and the model has no SWA, so any layer index is representative). 2. `llama_model::cache_size()` was updated upstream to read recurrent layers via `hparams.n_embd_v_s()` for Gemma4-style hybrids; the Phase 2 rename of `n_embd_k_s/v_s -> n_embd_r/s` did not catch this new caller because it was added by upstream after the rename was originally written. Switch the call to `hparams.n_embd_s()`. No semantic change for either Qwen3.5 GDN (no SWA, recurrent path uses n_embd_s = packed conv+ssm size as before) or Nemotron-H (no SWA, all attention layers share head_dim). Build is green again on top of upstream main; the rest of the Phase 3.x cherry-picks apply cleanly.
Adds `bin/test-nemotron.rs` — a ~90-line standalone binary that proves chimere-server's `LlamaForward` (the libllama FFI layer that backs `GenericModel`) is fully wired against the new `ik_llama` Mamba-2 + `nemotron_h_moe` backport (PR ikawrakow/ik_llama.cpp#1593). What it does: 1. `chimere_deltanet::llama_backend::from_env()` — same entry point used by `GenericModel::from_env()`. No `Qwen35Model`, no `cudarc_shell`. 2. tokenize → forward_prefill → greedy argmax → forward_token loop → detokenize. 3. Prints generation rate + decoded text. Validation on Nemotron-3-Nano-30B-A3B Q4_0 (sm_120 CUDA, NCMOE=30, ctx=2048): prompt: "The capital of France is" prefill: 199 ms (30 tok/s prompt) gen: 45 tok/s, 333 ms / 15 tokens output: "The capital of France is Paris.\" No extra text. Must not add anything else. Just that sentence" Confirms end-to-end: - Phase 3.3 graph builder (build_nemotron_h_moe) works on GPU - Mamba-2 SSM scan CUDA kernel works - Hybrid SSM + GQA + gateless RELU² MoE FFN all chained correctly - chimere-server's tokenize/detokenize FFI bindings work for non-Qwen vocab Remaining work (out of scope for this commit, tracked separately): - bin/chimere-server.rs still hardcodes Qwen35Model. Wiring the arch-detection dispatch (Qwen35Model vs GenericModel) requires AppState polymorphism — Step 7 of the multi-arch refactor plan. - CHIMERE_TOKENIZER is mandatory (no GGUF embedded fallback). For Nemotron, download tokenizer.json from `unsloth/Nemotron-3-Nano-30B-A3B` via huggingface-cli. Usage: LD_LIBRARY_PATH=/path/to/ik_llama/build/{ggml/src,src} \ CHIMERE_MODEL=/path/to/Nemotron-3-Nano-30B-A3B-Q4_0.gguf \ CHIMERE_TOKENIZER=/path/to/Nemotron tokenizer.json \ CHIMERE_NCMOE=30 CHIMERE_KV_MAX_SEQ=2048 \ cargo run --release --bin test-nemotron Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…F card draft Phase C of the Step 7 / multi-arch dispatch milestone. Captures the state of chimere as of 2026-04-07 evening, after the 10 atomic Step 7 commits and PR ikawrakow/ik_llama.cpp#1593 landed end-to-end. ## README.md (full rewrite, 199 -> ~570 lines) - New 1-line pitch mentions both architecture families - Highlights box: 8 bullets (multi-arch dispatch, Qwen3.5 prod stack, Mamba-2 / Nemotron-H via PR #1593, engram, sm_120, OpenAI-compat HTTP) - Supported architectures table with arch / variant / GGUF metadata / status / perf / notes - Quick start: 5 commands covering Qwen3.5 prod, Nemotron via swap, test-nemotron smoke, curl examples - ASCII architecture diagram (HTTP -> AppState -> Mutex<AppStateModel> -> {Qwen35Model | GenericModel} -> libllama -> ik_llama -> CUDA sm_120) - Features detailed table grounded against chimere-server source - Full CHIMERE_* env var table from the source (not the README) - Performance section with the 3 narrative corrections inline: * MTP "+49.5%" softened to "infrastructure present, gated, fix planned" with explicit reference to bench_mtp.rs:104-167 SKIPPED * Engram "+quality" reframed as "domain knowledge injection via Cuckoo n-gram logit bias", with explicit disclaimer about the GPT-2 wikitext-2 -13.39% being out of distribution * Nemotron 45 tok/s cited with full reproduction conditions (NCMOE=30, ctx 2048, sm_120 RTX 5060 Ti, test-nemotron binary) - Backend section linking to PR #1593 + describing the Phase 3.x backport (Mamba-2 unified ggml_ssm_scan, nemotron_h_moe arch, FFN gateless+PAR fix, sm_120 CUDA SSM scan kernel) - Models tested + broader Mamba-2 ecosystem hints (Granite 4.0, Falcon Mamba, Codestral Mamba, Bamba, Hymba — "should work, untested") - Roadmap: Step 7.5 (FFI tokenizer fallback, multi-agent on Generic, streaming Generic), Mamba-3 backport (cite arXiv:2603.15569), NVFP4 - Citations: papers + repos + our PR ## CHANGELOG.md (NEW, Keep-a-Changelog format) [0.2.0] - 2026-04-07 — the Step 7 release. Documents: - Added: AppStateModel enum, GenericModel, test-nemotron, new env vars (CHIMERE_ENGRAM_DIR, CHIMERE_GENERIC_EOS, CHIMERE_FORCE_QWEN35) - Changed: AppState::model field type, InferenceState::Generic variant, llm_build_ffn gateless+PAR guard - Backend: ik_llama.cpp pinned to PR #1593 head - Known limitations (Step 7 caveats: single-agent Generic, lossy streaming Generic, no engram on non-Qwen, n_seqs=1 hardcode) - Validated runtime numbers - Migration guide (CHIMERE_ENGRAM_DIR + Nemotron env vars) ## chimere-server/docs/STEP7_MULTI_ARCH.md (NEW, 426 lines) Technical reference for the dispatch architecture: - Why (Mamba-2 / Nemotron support requires non-Qwen35 code path) - Approach (closed enum vs trait object — preserves Qwen35Model inherent methods) - Diagram of the dispatch flow - The Send-across-await constraint discovered during commit 7/10 implementation: &Qwen35Model is !Send because Qwen35Model is !Sync (RefCells), so the typical "match &*model { Qwen35(qwen) => ... .await ... }" pattern breaks the axum Handler bound - Fix pattern: resolve agent_id BEFORE locking the model, then pattern-match without any await inside the arms - Tradeoffs table per Step 7 limitation with the Step 7.5 fix planned - ChimereModel trait + variant inventory - Code pointers (real line numbers from HEAD) ## chimere-server/docs/HF_MODEL_CARD_UPDATE.md (NEW, draft) Draft sections for the Kevletesteur/Qwen3.5-35B-A3B-Chimere-v3-GGUF HuggingFace model card, to be applied via `hf` CLI: - Compatible runtimes section (NEW): chimere-server first, stock llama-server as fallback - Quick start REPLACE: chimere-server example replaces stock - Backend section (NEW): ik_llama.cpp fork + PR #1593 + chimere link - Multi-architecture support section (NEW): mention Nemotron-H works on the same backend at ~45 tok/s - 3 narrative corrections applied consistently with the README The doc agent flagged 3 stale doc-comments in chimere_model.rs:28-37 and bin/chimere-server.rs:5-31 as "noticed but not fixed" — to address in a follow-up cosmetic commit. GH topics updated separately via gh api: 11 -> 18, added mamba2, mamba, nemotron-h, hybrid-ssm, ffi, sm120, state-space-models. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The fork's main branch tracks upstream `ikawrakow/ik_llama.cpp` and inherits its README verbatim. Add a top-level `FORK_NOTES.md` so visitors landing on github.com/AIdevsmartdata/ik_llama.cpp can tell at a glance: - this is a fork (not the upstream) - what was added (12-commit Phase 3.x backport: Mamba-2 + Nemotron-H MoE arch support, with all the ik_llama perf optimizations preserved) - where to find the clean PR-ready branch (mamba2-nemotron-h-backport) - the upstream PR being offered (ikawrakow#1593) - end-to-end validation results (Nemotron-3-Nano-30B-A3B Q4_0 + UD-IQ3_XXS, ~45 tok/s on RTX 5060 Ti sm_120) - caveats / scope (n_seqs=1, state save/restore not adapted, Mamba-1 stubbed, OLD ggml_ssm_conv reused) - downstream consumer (chimere-server) - build instructions for sm_120 (Blackwell consumer) - sync policy - contact path Single doc-only commit on top of upstream main (67fc9c5). Does not touch any source file. Safe to delete if upstream merges PR ikawrakow#1593. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Thank you for the PR. This project does not have a formal AI policy (yet), but that does not mean that AI generated PRs are encouraged or appreciated. There have been very few AI generated pieces accepted so far. They have been either very limited in scope, where it has been easy to spot the bugs, or they have been rewritten (the Qwen-3.5 architecture started from an AI-generated PR before I rewrote most of it). This PR adds a lot of new code, so my questions to you are
And now a question to everybody else: is anyone actually using this model? I mean, we all know that everybody is curious to try when a new model comes out, but based on a quick look at the download counts on HF, Nemotron3-Nano has been download an order of magnitude less than Qwen-3.5-35B-A3B, for instance. |
I personally don't use Nvidia AI models at all. There are several reasons for this:
Nemotron-Cascade-2 looks attractive, but I don't think its size is worth it. Gemma4 and Qwen3.5 look more interesting. |
|
Bonjour,
j'écris en français ma langue natale.
Merci pour votre considération, je vous fais le retour complet juste aprèe
avec l'IA (Claude) pour vous donner ma vision long terme et utile pour
l'open source.
Bonne journée.
Le mar. 7 avr. 2026 à 16:52, Mikhail Komarov ***@***.***> a
écrit :
… *hardWorker254* left a comment (ikawrakow/ik_llama.cpp#1593)
<#1593 (comment)>
Thank you for the PR.
This project does not have a formal AI policy (yet), but that does not
mean that AI generated PRs are encouraged or appreciated. There have been
very few AI generated pieces accepted so far. They have been either very
limited in scope, where it has been easy to spot the bugs, or they have
been rewritten (the Qwen-3.5 architecture started from an AI-generated PR
before I rewrote most of it).
This PR adds a lot of new code, so my questions to you are
- How well do you understand this code?
- Has there been any real testing beyond AI generated slop?
- Without looking at the code, it is not immediately clear which
backend(s) are supported. Have they been tested?
- Where are the performance figures?
And now a question to everybody else: is anyone actually using this model?
I mean, we all know that everybody is curious to try when a new model comes
out, but based on a quick look at the download counts on HF, Nemotron3-Nano
has been download an order of magnitude less than Qwen-3.5-35B-A3B, for
instance.
I personally don't use Nvidia AI models at all. There are several reasons
for this:
1. Size. Even the Q2_K_L quant is larger than 16 gigabytes
(Nemotron-3-Nano-30B-A3B), which excludes its use on systems with small
resources.
2. I don't think Nvidia has trained its models on datasets that have
Russian, which is important enough for me to support.
Nemotron-Cascade-2 looks attractive, but I don't think its size is worth
it. Gemma4 and Qwen3.5 look more interesting.
—
Reply to this email directly, view it on GitHub
<#1593 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BG4RVADFUTY6D2RJHOW2IM34UUI2NAVCNFSM6AAAAACXOVF2FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOJZHA4TGOJSHE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Draft reply to ikawrakow on PR #1593
Hi Iwan, thank you for reading the PR and for the honest review — it's exactly the kind of feedback I was hoping for. I want to be upfront: yes, this PR was written with AI assistance (Anthropic Claude Opus 4.6, used as a pair-programmer over a few intensive debugging sessions). I'll answer your four questions directly, then I'll explain why I opened the PR knowing the bar for AI-assisted work in this project, and what I plan to do next. 1. How well do I understand the codeHonestly: well enough to have hit the bugs, debugged them with comparative
What I am NOT pretending to fully understand: the qnext / GDN cache layout decisions you made for Qwen-3.5, the multi-slot attention KV scheduler, and the deeper iqk matmul template metaprogramming. I deliberately did not touch any of those — Phase 1 added new arch tags, Phase 2 added new tensor allocators, Phase 3 added a new graph builder next to the existing one. The qnext path is gated behind a 2. Real testingEnd-to-end on Nemotron-3-Nano-30B-A3B (
For Qwen-3.5-35B-A3B Chimere v3 RAMP (the model I run in production every day, qnext path), I ran a 4-prompt regression test against a baseline captured before the PR — 4/4 PASS, no regression. The Qwen-3.5 hot path is byte-for-byte unchanged in this PR by construction; I was paranoid enough to verify it anyway because that model is what I actually use. Beyond that there has been no formal MMLU / perplexity / multi-language sweep — I'd want to do that before the PR is mergeable, and the bar should be "matches upstream 3. Backend coverage (and which were tested)
This is a real gap in the PR and you're right to point it out. If the project gates merges on multi-backend coverage, this PR cannot land as-is — at minimum it needs Metal / Vulkan equivalents from people who can test them. 4. Performance figuresOn RTX 5060 Ti (sm_120, Blackwell consumer), CUDA 12.8, build flags New Mamba-2 / Nemotron-H path (this PR)
Qwen-3.5-35B-A3B (qnext path, untouched by this PR — included as non-regression sanity check)
ik_llama vs stock b8125 head-to-head deltas on the same GGUF / same hardware (these are part of why I picked your fork as the backend in the first place):
The dense gains are modest (+8-10 % gen) but the MoE gains (+18 % to +32 % gen, +80 % to +165 % prefill) are exactly the reason ik_llama is the right backend for the desktop-MoE niche. This is also the reason the Mamba-2 backport is worth doing in your fork specifically rather than just using stock — once the new arch lands, hybrid SSM-MoE models like Granite-4.0 H-Tiny, Falcon-H1, Bamba, Hymba, eventually Jamba inherit the same +18-32 % MoE advantage. The full table with exact ncmoe / ctx / VRAM figures across all quants is at https://github.com/AIdevsmartdata/chimere/blob/main/benchmarks/benchmark-qwen35-2026-03-07.md. Hardware bottleneck — why we target 15-17 GB modelsThe RTX 5060 Ti is a consumer Blackwell with PCIe 5.0 ×8 (it physically has 16 lanes wired but the Ti SKU is x8-electrical). For MoE models with I have not yet run a head-to-head Nemotron benchmark of this PR vs stock Quality benchmarks and what actually moved the needle on this hardwareBeyond raw throughput, here are the quality numbers we have on the production stack — and, crucially, which optimizations actually paid off for us on consumer hardware versus which were null results. The honest delta between MTP / DFlash (negative or zero) and Engram logit-bias (clear positive) is part of why I want this PR landed: it shifts the focus from speculative-decoding tricks (which break on hybrid SSM-attention models because of the GDN state-rollback barrier) to what does work on a single 16 GB consumer GPU. Qwen-3.5-35B-A3B Chimere v3 RAMP (
|
| Benchmark | Score | Source |
|---|---|---|
| IFEval (15 instruction tests) | 100 % (15/15) | hand harness |
| GSM8K CoT 8-shot (1319 questions) | 84.0 % flexible / 83.2 % strict | lm_eval gsm8k_cot 8shot v3 |
| HumanEval (30 problems, executed) | 83 % (25/30) | custom agent harness |
| BFCL tool-calling (20 questions) | 75 % | custom agent harness |
| GPQA-diamond zero-shot | 47.98 % | lm_eval |
| ARC-C | 59.3 % / 62.2 % norm | lm_eval |
| Adversarial edge cases (15 tests) | 100 % | prompt-injection resistance |
| Speed (chimere-server HTTP path) | ~80 tok/s gen, 789 tok/s prefill | RTX 5060 Ti, NCMOE=3, ctx 64K |
The full v3 model card with comparison to v1 (Kevletesteur/Qwen3.5-35B-A3B-Chimere-Distilled-GGUF, 1591 downloads) and the BF16 source is at https://huggingface.co/Kevletesteur/Qwen3.5-35B-A3B-Chimere-v3-GGUF. The v1 → v3 delta is: IFEval 67 % → 100 %, GSM8K 52 % → 84 %, HumanEval 97 % → 83 % (regression on code, intentional tradeoff for the IFEval+GSM8K gain), BFCL 90 % → 75 % (same tradeoff). v1 stays the right choice for code-heavy use; v3 is the new general-purpose default.
Engram (n-gram logit bias) — the optimization that actually worked for us
I run an n-gram logit-bias overlay on top of the C++ fast sampler (Cuckoo filter, 4 prebuilt domain tables: kine 20 MB, code, cyber, general — ~21 MB total). The interesting bit is the production ablation on our internal ODO 10-task benchmark:
| Engram config | ODO bench (10 tasks) |
|---|---|
v1 α = 0.35, applied during <think> blocks and response |
77 % ← previous nightly default |
| OFF α = 0 (baseline) | 85 % |
| v2 α = 0.1, response-only, NEST adaptive per-token | 88 % ← current production |
Net: +11 pp over no-engram, +3 pp over the OFF baseline, on a benchmark suite where the baseline was already passing. The root cause of the v1 regression was bias-during-thinking and α too high; v2 fixed both with tighter, response-phase-only application. This is small in absolute terms but it's the only inference-time optimization that gave us a positive measurable delta on this hardware.
What did NOT pay off (negative results we logged)
Two optimizations we instrumented and benchmarked at length on the same hardware, both with negative net result, both because of constraints specific to consumer-class GPUs:
| Optimization | Acceptance / quality | Wall-clock | Verdict |
|---|---|---|---|
| MTP (multi-token prediction) for Qwen-3.5-35B-A3B MoE | 84.8 % token acceptance rate | 0.51× speedup (i.e. 2× slower) | Negative. The MTP head is itself MoE (256 experts on CPU), so each draft costs as much as a main forward — speculative decoding can't win against itself. |
| DFlash (block diffusion drafter) | τ = 6.06 (comparable to the original DFlash paper's τ=6.4 on holdout) | 0.73× wall-clock (i.e. 27 % slower) | Negative. The target model is too fast (93 tok/s on bare ik_llama) for speculative decoding to amortize the drafter cost. Independently, the GDN State Barrier makes draft rollback structurally impossible on hybrid SSM-attention models — recurrent state cannot be re-wound after a draft rejection. |
We ran 8 DFlash architectures over 27 days. The DFlash result is reported in detail in our paper draft (link in the chimere-system landing card) and the negative wall-clock is the headline finding — speculative decoding is not the right lever on this class of hardware for this class of model. What works is reducing per-token compute and avoiding PCIe round-trips, not predicting more tokens per step. Engram + custom mixed-precision quantization (RAMP) + a tight 15-17 GB GGUF that fits with zero offload + the ik_llama iqk MoE matmul deltas above — that's the lever set that actually moves the needle.
The combination of (a) the 15-17 GB sweet spot, (b) the ik_llama iqk MoE perf, (c) Engram on Qwen-style vocabularies, and (d) the upcoming Mamba-3 work, is what makes this Mamba-2 PR a foundation rather than a one-off Nemotron port for me. The PR on its own doesn't close the loop; it unblocks the model architecture for the next quant + fine-tune + Mamba-3 cycle.
On your meta question — is anyone using Nemotron-3-Nano?
You're right that the HF download counts are an order of magnitude lower than Qwen-3.5-35B-A3B. I wouldn't argue otherwise. To be blunt: Nemotron-3-Nano was a proof-of-concept for me, not the destination. I picked it for the backport because (a) it has both Mamba-2 mixers AND a hybrid attention-MoE structure, so getting it to work end-to-end exercises basically every code path I needed for the real target, and (b) it has a public reference implementation in ggml-org/llama.cpp so I had a working oracle to debug against.
I share your assessment of the model itself: it's larger than necessary for what it does, and on a 16 GB consumer GPU it's at the edge of the envelope. I personally use Qwen-3.5-35B-A3B in production every day (the Chimere v3 RAMP variant on my server), not Nemotron. If I'm honest the Nemotron support is fungible — what I actually wanted was the underlying ggml_ssm_scan op + the build_mamba2_layer helper, because that's what unblocks the next generation of models I want to run on my desktop.
What I'm actually trying to build, and where this PR fits
I'm working towards a frontier-class single-desktop pipeline targeting consumer Blackwell (RTX 5060 Ti class, 16 GB VRAM). The end-to-end stack I have in flight:
-
Base model: Qwen-3.5-27B dense (15 GB Q4_K_S, already running on the same hardware, ~23 tok/s on the nightly pipeline). I share your view that Gemma-4 and Qwen-3.5 are the right targets for desktop-class hardware. Nemotron and Qwen-3.5-35B-A3B are interesting upper bounds, but the 27B dense is the sweet spot.
-
Reasoning + tool-calling fine-tune: Dr.GRPO agentic rewards on top of an SFT BF16 LoRA (xLAM-2 + Nemotron-Agentic + Opus traces). Target ~$40-50 on rented GPU for the full pipeline.
-
OPSDC distillation (reasoning compression): -64% tokens on the chain-of-thought without losing accuracy. I have a working pipeline and a small dataset on disk.
-
TurboQuant KV cache (arXiv:2504.19874, ICLR 2026): you already shipped
k_cache_hadamardin this very project, which is the rotation step. I plan to enable it on the 27B nightly withQ4_0keys, generate the d=256 codebook for Qwen-3.5-27B'shead_dim=256(currently missing), and follow upstream's TBQ3_0 / TBQ4_0 work in PR #21089. This would let me push context from 8K to 16-32K with the freed VRAM. -
Mamba-3 upcycle (this is the part where this current PR matters most): the Mamba-3 paper landed on arXiv on 2026-03-16 (
arXiv:2603.15569, accepted at ICLR 2026, by the same Tri Dao + Albert Gu lineage). It's three concrete changes on top of Mamba-2: exponential-trapezoidal discretization (which lets the implicit convolution replace the explicit causal conv1d — i.e. removes the very Phase-3-hackggml_ssm_convreuse this PR ships), complex-valued state (data-dependent rotary equivalent, unblocks state-tracking benchmarks Mamba-2 cannot solve), and MIMO state update (matmul-based, +1.8 pts at 1.5B, half the latency at fixed quality). My Phase 3.x work here is deliberately the foundation for that — about 90 % of what I added (arch dispatch, cache layout, hparams plumbing, build_xxx_layer pattern) carries over to Mamba-3 directly; only the scan kernel changes. -
MoE upcycling of the 27B dense into a small-MoE (à la upcycling literature) so the same 16 GB budget can host more total parameters with sub-active compute.
-
Self-improving loop: this is the part of the system that I've already shipped and that I think is genuinely novel. My downstream Rust HTTP server (
chimere-server) carries an n-gram logit-bias overlay ("Engram"), 4 prebuilt tables (kine 20 MB, code, cyber, general) loaded via Cuckoo filter into the C++ fast sampler. There's a nightly read/write loop (engram-write-nightly.timer) that ingests scored production traffic, decays unused n-grams, and re-publishes the table. An orchestrator ("ODO", port 8084) sits in front and routes between intent classification, RAG, web search, dynamic engram, multi-agent pipelines and the model itself. The 27B nightly teacher pipeline runs at 00:30 every night on the same hardware, pre-empting the production server, training/scoring, then handing the GPU back. The whole point of having a fast Mamba-2 / Mamba-3 backend on consumer hardware is so this loop can keep running without renting cloud GPUs. -
Four-benchmark frontier target: I'm aiming for a single 27B-class model on a single 16 GB consumer GPU that posts competitive numbers on τ²-bench (agentic tool calling), BFCL-V4 (function calling), GSM8K / GPQA (reasoning) and LiveCodeBench v6 (code). Baseline today on the unmodified Qwen-3.5-27B: τ² ≈ 79 %, BFCL-V4 ≈ 68.5 %. Target after the SFT + Dr.GRPO + OPSDC + TurboQuant + (eventually) Mamba-3 upcycle: τ² in the 82-87 % range with the model fitting in 15 GB.
So in plain terms: this PR is the Mamba-2 foundation under the Mamba-3 + 27B-frontier work I'm actually trying to do. If you'd rather I close it and re-submit in two months as a smaller, more focused, multi-backend, benchmark-backed PR after the Mamba-3 work is done, I will. That might honestly be the right call for the project — by then the conv1d hack would be gone (replaced by Mamba-3's implicit convolution), the Metal / Vulkan kernels could be upstreamed by people who have the hardware, and there would be a real benchmark sweep behind it.
Or, if you'd prefer a different scope (just LLM_ARCH_MAMBA2, no Nemotron-H, no FFN gateless fix, no eval-callback printer fix; or only the FFN fix as a standalone bug-fix PR), I'm happy to split this one into smaller pieces. The 12-commit history was deliberately structured to be walkable so each step can be reviewed or dropped independently.
Whatever you decide, thank you for taking the time to look. The honest review is more valuable than a silent merge would have been.
Best,
Kévin Rémondière (@AIdevsmartdata) — and yes, Claude Opus 4.6 helped me write this reply too.
|
I use Qwen3.5 for vision but indeed I prefer Qwen coder next for other stuff;(little tricky thing I found out is that seems those recurrent hybrids can reuse cache between each other via checkpoints or something else I don't know now thus...) so basically I won't use those 30b moe in any case... I think that impl is too complicated for a single model support and hard to maintain the consequence, around 400~600 lines change maybe be something more manageable... just a users opinion. |
|
I had a quick look at the change, it looks mostly OK. Still, nobody seems excited. From the plans it looks like the hope is that Mamba-3 will get used in new frontier models, and for that it would be useful to have this change set. This seems like the strongest argument for accepting the PR so far (but one could of course also just wait-and-see, and when new Mamba-3 based frontier models start coming out, add the changes then). |
|
I'd like to run Nemotron-Cascade-2-30B-A3B and it's future successors on desktop (linux x86_64 with an RTX 5060 Ti 16GB) because of it's math + statistics + reasoning strength that AFAICT exceeds that of Qwen3.6-35B-A3B and Qwen3-Coder-Next-80B-A3B, while also being smaller and hopefully faster when getting away with lower --n-cpu-moe. Gemma-4-26B-A4B was rather disappointing for my uses; Qwen3.6-35B-A3B can be annoyingly verbose when thinking; and the dense models in the 30B range (Qwen3.6-27B, Gemma-4-31B) are much slower and more power hungry than their A3B MoE siblings. Many thanks for the ik_llama project and the everyday value of it's efficiency ! |
5 participants
Summary
Brings Mamba-2 and NVIDIA Nemotron-H MoE support to
ik_llama.cppby porting the upstreamggml-org/llama.cppMamba-2 work and adapting it to the existing ik_llama plumbing (iqk SIMD matmul, sm_120 CUDA, qnext-style packed recurrent cache, custom IQK quants). Validated end-to-end on Nemotron-3-Nano-30B-A3B (Unsloth UD-IQ3_XXS and Q4_0) producing coherent French/English text viallama-cli. Qwen3.5-A3B GDN remains byte-for-byte working (no regression).The motivation: ik_llama's iqk perf path + custom quants are unique, but until now the project did not support hybrid SSM architectures. This PR closes that gap so Nemotron-H, Mamba-2, and other recent state-space hybrids can run on top of ik_llama's optimized backend instead of having to fall back to stock llama.cpp.
What's in this PR (12 commits)
Phased so each step is independently reviewable:
feat(arch): Phase 1 stub for Mamba-2 + nemotron_h_moe metadataLLM_ARCH_MAMBA2,LLM_ARCH_NEMOTRON_H_MOEarch tags + GGUF kv name tablefeat(arch): Phase 2 — hparams loading + tensor allocationcreate_mamba2_tensors()+create_nemotron_h_moe_tensors()in load-tensorsrefactor(hparams): rename n_embd_k_s/v_s → n_embd_r/s, move to .cppfeat(hparams): use_qnext_state_layout flagfeat(hparams): defensive SSM bounds for Mamba-2 + Nemotron-H loadfeat(ggml): port upstream ggml_ssm_scan op + CUDA backendggml_ssm_scan(signature + CPU + CUDA kernel underggml/src/ggml-cuda/ssm-scan.cu). Old 4-argggml_ssm_convis reused as-is for the conv1d step (Phase 3.3 hack — works for n_seqs=1)chore(build_mamba): stub before Phase 3.2 ggml_ssm_scan signature changebuild_mamba()body so the old Mamba-1 path doesn't break the buildfeat(build-context): Phase 3.3 — build_mamba (Mamba-2) + build_nemotron_h_moebuild_mamba2_layer()mirrors upstream'smamba-base.cppline-for-line;build_nemotron_h_moe()dispatches per-layer between Mamba-2 SSM, GQA attention, and gateless RELU² MoE/dense FFNfix(eval-callback): compute sum over full tensor, not just printed sliceexamples/eval-callbackprinter summed only the displayed elements, which made bisecting numerical divergence vs upstream impossible. Now sums the whole tensor in a separate loop, then prints the truncated head/tail. Saved hours of debuggingfix(llama): inp_ssm_ids must read recurrent slot 0, not kv_self.headkv_self.headis the attention KV head which advances per-token. Reading atkv_self.head * src0->nb[3]walked out of bounds after the first prefill. Fix: writedata[s] = (int32_t) s(i.e. slot index, not position)fix(build-ffn): guard parallel-gate fold on gate != nullptr (gateless FFNs)llm_build_ffnunconditionally foldedcur = mul(cur, tmp)forLLM_FFN_PAR, even when nogateweight was passed. For gateless FFNs (Nemotron-H shared expert:relu²(up_proj)only) this collapsed the activation torelu²(up) * upinstead ofrelu²(up). Every previous caller ofLLM_FFN_PARhappened to also pass a gate (SwiGLU style), so the gateless path was unreachable until Nemotron-H exercised itfix(api): catch up with current upstream API drift (Gemma4)mainsince the backport began:n_embd_head_k/v→ per-layer functions, and one newcache_size()caller ofn_embd_v_s()→n_embd_s()Validation
Both architectures tested with the same
llama-clibinary built from this branch (CPU only,-ngl 0, prompt\"The capital of France is\"):The capital of France is Paris.\" Provide exactly that line, no extra text. Must not add anyThe capital of France is Paris, and the capital of Italy is Rome, but the capital of Ireland is Dublin.The capital of France is Paris, a city withThe Nemotron pass exercises new code; the Qwen3.5 pass exercises everything that was already working (qnext recurrent layout, MTP path, IQK matmul, sm_120 CUDA path) to confirm zero regression.
The numerical validation method that finally pinned down the FFN bug was a side-by-side
eval-callbacksum diff between this branch and upstreamggml-org/llama.cppon the same Q4_0 GGUF.attn_norm-0andattn_norm-1matched byte-for-byte,attn_norm-2started diverging by 23%, and the trace showed an extraffn_gate_par-1op in ik_llama that was absent from upstream — the smoking gun for the gateless+PAR bug.Caveats / scope
These are intentional Phase 3 limits, all clearly documented in the code, all easy to lift in a follow-up:
n_seqs == 1only.build_mamba2_layerwrites the new SSM state back to slot 0 unconditionally and readsinp_ssm_ids[s] = s. Multi-sequence parallel decoding for Nemotron-H is not yet wired (single-slot recurrent cache only). Qwen3.5 GDN multi-seq is unaffected.read/write_kv_cache_datastill walks the legacy K-cache layout for the hybrid path. Persistent sessions (--cache-reuse) are broken for Nemotron-H; regular fresh-prompt inference is fine.build_mamba()aborting on the legacy branch and reused the OLDggml_ssm_conv4-arg op as-is for the Mamba-2 conv1d step. Cleaning that up would mean porting the new 2-argggml_ssm_convfrom upstream, which is independent of this PR.ggml_ssm_convas a 4-arg call (one state + xBC + weight + state_seq) instead of porting the upstream 2-argconcat(state, transpose(xBC)) → ssm_conv(conv_x, weight)rewrite. The result is numerically identical forn_seqs=1but adds 23 saved ops vs upstream (one per SSM layer). Can be reworked if it bothers anyone.Tested models / quants
Test plan for reviewer
cmake -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 ..(no warnings expected from new files)llama-cli -m <Nemotron-Q4_0> -ngl 0 -n 20 -p \"The capital of France is\"→ coherentllama-cli -m <Qwen3.5-RAMP> -ngl 0 -n 5 -p \"...\"→ coherent (regression check)-ngl 99(CUDA SSM scan + iqk matmul path on the same model)Happy to split this into smaller PRs (Phase 1, Phase 2, Phase 3.x) if you prefer, and to rebase on a moving target. The 12-commit history is intentionally walkable so each can be reviewed/dropped independently.
🤖 Backport done with Claude Code