Two months of follow-up work changed the picture substantially. The May conclusions (kept below as the historical record) were correct for their stack but several of their blockers are now resolved. Current honest state:
We patched OGL to wire mixed precision into the Multigrid preconditioner (
precision double|mixed|single; was BJ-only). Withprecision single(all-float preconditioner) the tuned Multigrid (V-cycle + CG coarse-solver) beats CPU GAMG: at 17.2M, 20.9 s/step (np=16) vs GAMG 22.1 (−6%), and 22.4 vs 25.9 at np=8 (−14%); at 7.1M 7.95 vs 8.5–9.3. Same ~13 iterations as double (no accuracy penalty), and −17 to −27% VRAM (ceiling20M→28-30M). This is the first clear GPU-CFD pressure-solve win on Battlemage. (Double precision alone was only a near-tie — see maps.) Full-float solve (for 34M + more bandwidth) is the next step. Seeknowledge/performance-maps.md,findings/code/ogl-patches/mixed-precision-multigrid.patch.Tuned Ginkgo Multigrid (V-cycle, Jacobi smoother, CG coarse-solver) on a 7.1M mesh: ~13 iters, ~9 s/step = ~1.17× CPU GAMG (down from ILU's 3×). But the B70 is under-fed at 7.1M — compute engine only ~46% busy, copy ~30% (CPU-assembly/transfer bound, which is why more MPI ranks help). So the next test is a larger ~18M mesh to feed the GPU above the ~10M-cells/GPU win threshold (case prepared:
Testcase-mid). VRAM ceiling for MG ≈ 20M (double); a mixed-precision OGL patch (not yet wired by OGL) would lift it to ~25–30M. The GPU-CFD win is now a feeding + VRAM problem, not a preconditioner dead-end. Seescripts/next-session-plan.md.Platform fit (researched): the B70 has strong FP64 (~1335 GFLOPS measured, ~1:8 — not the bottleneck; our earlier weak-FP64 guess was wrong), and our CG + matrix-AMG + FP64 approach matches community best practice. We are, per the public record, the only ones running OGL/Ginkgo OpenFOAM on a consumer Battlemage GPU — bleeding edge; the gap is the Ginkgo SYCL preconditioner software, not the silicon. Classical Ruge-Stüben AMG (the real fix for the iter-count floor) just landed in Ginkgo
developbut is CPU-only so far (no SYCL kernels yet). Seeknowledge/intel-platform-fit.md.The May verdict's central claim — "no viable GPU preconditioner exists" — no longer holds. A standalone Ginkgo 2.0 SYCL sweep (findings/26) confirms four previously-blocking bugs are fixed:
find_blocksunderflow (BJ>1),add_candidatesSIGABRT (ICT),lower_trsNotImplemented (ILU, via Ginkgo PR #2023), and Multigrid PGM. BJ(1/2/4/8/16), ILU, ISAI and Multigrid all run single-process up to 36M rows. OGL was migrated to Ginkgo 2.0 with two small patches (findings/24).CR 26.14–26.18 abort during multi-process Level-Zero
zeInit(gmm_helper/resource_info.cpp:15) whenever ≥2 ranks share the GPU — the normal mode for MPI CFD. Pure-Level-Zero minimal reproducer + full analysis in findings/29; upstream as intel/compute-runtime#922 (open, no fix, persists through CR 26.18 + kernel 7.0.0-22).User-side
LD_LIBRARY_PATHto an extracted CR 26.05libze_intel_gpu, no sudo, no system change (findings/27,scripts/cr2605-shell.sh). On a freshly-booted GPU this ran BJ(1) multi-rank at 34M cells again (~94 s/step).A freshly-booted GPU + the CR 26.05 LD-switch finally produced deterministic multi-rank numbers on the 34M case (findings/30):
- BJ(1): ~51.5 s/step steady-state (the earlier "~94 s/step" was the setup-inclusive first step). Every pressure solve hits the 201-iteration cap — BJ(1) never converges. → ~1.44× slower than CPU GAMG (35.7 s/step).
- ILU: VRAM OOM → DEVICE_LOST at peak 31.5 GB, in the
ParIlufactorization'sCsr::convert_to(Coo)— does not fit at 34M cells on the 32 GB B70 with the OGL distributed overhead. (GPU recovered on its own, no reboot.)- BJ(2–16): still the
find_blockssize_t underflow in the OGL distributed path (cleanAllocationError, GPU unharmed).So no strong GPU preconditioner currently works on this mesh at this VRAM: the one that runs (BJ1) is too weak, the ones strong enough to win either don't fit (ILU/MG) or hit the distributed
find_blocksbug (BJ>1). The B70 ran 8-way multi-rank GPU solves cleanly — this is a software-stack limit, not hardware.To remove VRAM from the equation we remeshed the case at half resolution (7.1M cells, parallel snappyHexMesh on 8 cores) and ran CPU GAMG vs GPU ILU on the same mesh / decomposition / initial condition (findings/30, Addendum 2):
per step GAMG (CPU np=8) ILU (GPU B70 np=8) wall-clock ~7.7 s ~22–24 s pressure CG iters 3–5 160–201 peak VRAM host 10.7 GB (fits easily) ILU on the GPU is ~3× slower than CPU GAMG even where VRAM is a non-issue, because it needs ~40× more CG iterations (ILU is a local preconditioner; GAMG is algebraic multigrid with ~O(N) convergence).
Testing every available preconditioner on the 7.1M mesh (np=8) changed the conclusion: Ginkgo's algebraic Multigrid runs, fits (10.4 GB), and converges in 55–101 iterations at ~14 s/step — only ~1.8× slower than CPU GAMG (7.7 s), vs ILU's ~3× (160–201 iters) and BJ/ISAI never converging. And this is the untuned default (V-cycle, Jacobi smoother).
preconditioner (7.1M, np=8) iters/solve s/step VRAM GAMG (CPU) 3–5 7.7 s host Multigrid (Ginkgo GPU) 55–101 ~14 s 10.4 GB ILU (GPU) 160–201 ~23 s 11.0 GB ISAI / BJ(1) (GPU) 201 (cap) — — ICT / BJ(>1) (GPU) crash — — "Hardware great, software not yet ready" — but the path is now clear and matches the field. BJ/ILU are one-level preconditioners that lose to multigrid (textbook; AmgX/Hypre prove GPU-AMG is the answer). On this stack the GPU-AMG exists — Ginkgo Multigrid — and already lands within ~1.8× of CPU GAMG untuned at 7.1M. The remaining work is concrete and bounded: (1) tune Ginkgo Multigrid (W-cycle, stronger smoother, more levels) to close the ~1.8× gap, and (2) make it VRAM-viable at production mesh sizes (~1027 bytes/row today). Until then: CPU GAMG for production; but a real B70 GPU-CFD win is now a tuning problem, not a dead end.
Short answer: No, not with the current software stack (May 2026).
At equal solver settings (nNonOrth=2, maxIter=200, all equations on the same hardware path):
- CPU GAMG np=16: 35.7 s/step
- GPU OGL BJ np=16: ~50 s/step
GPU only "wins" (30.6 s/step) with reduced quality settings (nNonOrth=1, maxIter=80) that would also speed up CPU GAMG to an estimated ~24 s/step.
| Factor | Impact |
|---|---|
| NOT a bottleneck — earlier "100 µs" claim was wrong | |
| No GPU-aware MPI: forced D2H→MPI→H2D round-trip per AllToAll | 5-10× extra communication cost |
| No GPU-side Multigrid: BJ vs GAMG = O(√N) vs O(log N) iterations | algorithmic disadvantage |
| SYCL preconditioner gaps: IC/ILU/IRILU/Hybrid not implemented or broken | Limited tuning options |
| Block-Jacobi maxBlockSize=1 only stable choice | Weak preconditioning |
- LLM Inference: 32 GB ECC VRAM enables serious local AI work (Gemma 27B + Qwen2.5-Coder 32B simultaneously loaded)
- ParaView Visualization: ANARI helide backend (CPU Embree currently; GPU helide_gpu would need SDL3+glslang)
- Future CFD potential: SYCL Graph + GPU-aware MPI could 5-10× improve the CFD path — but those are 12-24 months out
| Timeframe | Expected Change |
|---|---|
| Now | Use CPU GAMG, GPU for LLMs |
| 6-12 months | Level Zero command-list/Graph optimizations land in icpx |
| 12-18 months | SYCL Graph integrated in Ginkgo + OGL |
| 18-24 months | GPU-aware OpenMPI/UCX with Level Zero support |
| 24-36 months | Production-grade GPU CFD on Battlemage class hardware |
In this repo:
- lspci PCIe speed reporting bug (xe driver, SR-IOV PF mode)
- Ginkgo SYCL Block-Jacobi
find_blocksunderflow with maxBlockSize > 1 (fixed in Ginkgo 2.0, see June update) - OGL preconditioner sub-dict syntax requirement (undocumented)
- ONEAPI_DEVICE_SELECTOR syntax pitfall
- SYCL preconditioner support matrix (only BJ at maxBS=1 stable)
Plus:
- xe driver in OpenFOAM apt-paraview pulls in broken
pvserver --version(hangs) - Hybrid matrix format claimed in OGL README, not implemented in distributed mode
ICTpreconditioner causes GPUDEVICE_LOSTduring ParICT generate
For production CFD on this hardware in May 2026:
- p: CPU GAMG, np=16 (all P+E cores)
- U/k/ω: CPU PBiCGStab/DILU
- GPU: Reserved for LLM inference (vLLM/llama.cpp with SYCL backend)
Revisit GPU-CFD acceleration in 12–18 months when SYCL Graph lands in OGL or when AMD/NVIDIA-style GPU-aware MPI becomes available for Intel Xe Consumer.
This is still pioneering work — and the stack DOES build and run. We have:
- Working OGL/Ginkgo/SYCL build on Battlemage with patches
- Verified bandwidth profile (508 GB/s VRAM, 15 GB/s PCIe)
- Reproducible benchmark methodology (Time=8-10 mean)
- Identified specific bugs upstream can fix
- Clear path forward (Ginkgo 2.0, SYCL Graph, GPU-aware MPI)
That's a useful foundation for the next iteration when the software catches up to the hardware.
We spent significant effort here. The result is a negative result in the sense that GPU doesn't beat CPU for THIS case TODAY. That's still valuable — it's an honest data point that other Battlemage CFD users can build on instead of repeating our 30+ failed runs.
The hardware is great. The software is not yet ready. ¯_(ツ)_/¯
After exhaustive testing, the situation is clear:
No viable GPU preconditioner exists in Ginkgo 1.10 SYCL for this mesh.
The only working option (BJ maxBlockSize=1) is mathematically equivalent to diagonal scaling — far too weak for a 34M-cell CFD pressure system. It never converges; the solver always hits maxIter=200 cap.
Path forward: OGL rebuild with Ginkgo 2.0
Ginkgo 2.0 includes:
- Improved SYCL BJ generate (potential fix for maxBlockSize>1 OOM)
- Better distributed multigrid
- More complete SYCL preconditioner support
This rebuild is the only remaining avenue for meaningful GPU acceleration.
Outcome (June 2026): This path was taken and the Ginkgo-side bugs are indeed fixed — see the June update at the top of this file. The remaining blocker moved to the Intel Compute Runtime (multi-process
zeInitabort), with a working CR 26.05 LD-switch workaround. The "does it actually beat CPU GAMG with a strong preconditioner" question is now testable for the first time but not yet answered.