Add 8-wave warp-pipeline GEMM kernels (a16w16 / a8w8 / a4w4)

README.md

@@ -10,10 +10,10 @@ The headline result: on `a16w16` (FP16, 4096×4096×8192), the naive Gluon basel
 
 ## Start Here
 
-If this is your first time in the repo, open **[`kernels/gemm/a16w16/README.md`](kernels/gemm/a16w16/README.md)** and follow the Acts I–IV narrative from `v0_naive` to `v9_beyond_hotloop`. Each version isolates one concept — a layout, a pipeline stage, a scheduling decision. Read it alongside the code, then run the kernel:
+If this is your first time in the repo, open **[`kernels/gemm/intra_wave/a16w16/README.md`](kernels/gemm/intra_wave/a16w16/README.md)** and follow the Acts I–IV narrative from `v0_naive` to `v9_beyond_hotloop`. Each version isolates one concept — a layout, a pipeline stage, a scheduling decision. Read it alongside the code, then run the kernel:
 
 ```bash
-cd kernels/gemm/a16w16
+cd kernels/gemm/intra_wave/a16w16
 python bench.py --version 0 --K 8192 --dtype fp16
 ```
 
@@ -45,10 +45,11 @@ Once you're comfortable there, the FP8 and MXFP4 kernels (`a8w8`, `a4w4`) show h
 
 | If you want to…                                          | Look in                     |
 |----------------------------------------------------------|-----------------------------|
-| Learn the full optimization workflow end to end          | `kernels/gemm/a16w16/`      |
-| Apply the same design to FP8                             | `kernels/gemm/a8w8/`        |
-| Understand microscaling (MXFP4) and scale pipelines      | `kernels/gemm/a4w4/`        |
+| Learn the full optimization workflow end to end          | `kernels/gemm/intra_wave/a16w16/`      |
+| Apply the same design to FP8                             | `kernels/gemm/intra_wave/a8w8/`        |
+| Understand microscaling (MXFP4) and scale pipelines      | `kernels/gemm/intra_wave/a4w4/`        |
 | Understand the block-level design philosophy             | `docs/performance_philosophy.md` |
+| Understand warp-pipelining (the 8-wave kernels' theory)  | `docs/warp_pipelining.md`   |
 | Visualize a blocked / dot operand / LDS layout as a PDF  | `layout_plot/`              |
 | Build a mental model for LDS or HBM throughput           | `docs/`                     |
 | Automate rocprof, collect counters, generate perf tables | `scripts/`                  |

ROADMAP.md

@@ -31,6 +31,7 @@ Build a comprehensive collection of optimized GPU kernels using Gluon, with:
 | 4-bit + Scales | Implement baseline 4-bit MoE kernel in Gluon | :calendar: |
 | 4-bit + Scales | Optimize with llirSched + amdgcnas | :calendar: |
 | 4-bit + Scales | Document preshuffling and related optimizations | :calendar: |
+| 8-wave Warp-Pipeline | Ship warp-pipeline (pingpong) GEMM — a16w16, a8w8, a4w4 (`kernels/gemm/*-8wave/`) | :white_check_mark: |
 | **FlashAttention** | | |
 | FAv3 8-Wave | Port existing Triton FAv3 kernel to Gluon | :calendar: |
 | FAv3 8-Wave | Adapt llirSched and amdgcnas for 8-wave solution | :calendar: |
@@ -41,9 +42,8 @@ Build a comprehensive collection of optimized GPU kernels using Gluon, with:
 | LLVM Path | Investigate LLVM scheduling infrastructure | :construction: |
 | LLVM Path | Prototype LLVM-based scheduler | :construction: |
 | **Backlog** | | |
-| Future | 8-wave pingpong for GEMM kernels | :grey_question: |
 | Future | 4-wave solution for FA | :grey_question: |
 
 ---
 
-*Last updated: 2026-04-16*
+*Last updated: 2026-07-07*

docs/lds_throughput.md

@@ -13,7 +13,7 @@ Both are important,
 but they are solved in different ways and by different parts of the kernel design.
 
 In this tutorial, we focus on building an accurate mental model of `ds_read` throughput. 
-Latency is discussed separately in §3, and worked through concretely in the [`v5_local_prefetch`](../kernels/gemm/a16w16/v5_local_prefetch/README.md) kernel tutorial.
+Latency is discussed separately in §3, and worked through concretely in the [`v5_local_prefetch`](../kernels/gemm/intra_wave/a16w16/v5_local_prefetch/README.md) kernel tutorial.
 
 ## 1. The basic mental model for throughput
 
@@ -130,7 +130,7 @@ Only when the FIFO fills does back-pressure force the issue rate down to the LDS
 The same FIFO also buffers `ds_write` requests, not just `ds_read`s.
 A slow `ds_write` at the head of the queue can block subsequent `ds_read`s behind it —
 a detail that matters for kernels that mix the two,
-see [a4w4 §3.5–3.6](../kernels/gemm/a4w4/README.md#35-where-to-place-lwlr-for-scales) for a worked example where this dynamic drives the pipeline design.
+see [a4w4 §3.5–3.6](../kernels/gemm/intra_wave/a4w4/README.md#35-where-to-place-lwlr-for-scales) for a worked example where this dynamic drives the pipeline design.
 
 With SQ issuing every 4 cycles and the FIFO depth at 8, the first eight `ds_read`s enter the FIFO without waiting — they issue at the SQ rate of one every 4 cycles, filling the FIFO over ~28 cycles.
 The first completion from LDS arrives ~50–70 cycles after the first issue (this is the LDS pipeline latency, not the 16-cycle steady-state service rate), and subsequent completions arrive every 16 cycles.
@@ -242,7 +242,7 @@ It reads from LDS exactly like `ds_read`, but the hardware transposes the data d
 Its throughput profile is identical to `ds_read_b*` — the same SP-to-LDS pipeline, the same bank-conflict rules, and the same 8-entry FIFO — so everything in this doc applies unchanged.
 The choice to use it is a layout-conversion decision, not a throughput decision.
 The MXFP4 kernel uses `ds_read_tr` for the LR step of its scale pipeline;
-see [a4w4 §2.5](../kernels/gemm/a4w4/README.md#25-ds_read_tr-hardware-assisted-layout-conversion-for-scales) for the full treatment.
+see [a4w4 §2.5](../kernels/gemm/intra_wave/a4w4/README.md#25-ds_read_tr-hardware-assisted-layout-conversion-for-scales) for the full treatment.
 
 
 ## 5. LDS bank conflicts and their impact on throughput
@@ -253,7 +253,7 @@ In practice, this assumption does not always hold, and when it does not,
 the effect on performance can be dramatic.
 
 For a practical demonstration of how layout design affects bank conflicts,
-see the [v3_lds kernel tutorial](../kernels/gemm/a16w16/v3_lds/README.md).
+see the [v3_lds kernel tutorial](../kernels/gemm/intra_wave/a16w16/v3_lds/README.md).
 
 LDS is organized into 64 banks, each servicing one 4-byte access per cycle. 
 When a `ds_read` instruction reaches LDS, the 64 threads in a wave are not serviced individually. 
@@ -347,9 +347,9 @@ Whether that contract *should* change at all is the design question — not just
 
 ## 7. See Also
 
-- [v3_lds kernel tutorial](../kernels/gemm/a16w16/v3_lds/README.md) — Practical application of the throughput model to evaluate LDS layout designs (raw, swizzling, padding).
-- [v5_local_prefetch kernel tutorial](../kernels/gemm/a16w16/v5_local_prefetch/README.md) — Where latency hiding (the other side of the latency-vs-throughput distinction in §3) is worked through concretely.
-- [a4w4 §2.5](../kernels/gemm/a4w4/README.md#25-ds_read_tr-hardware-assisted-layout-conversion-for-scales) — Full treatment of `ds_read_tr` in the MXFP4 scale pipeline.
-- [a4w4 §3.5–3.6](../kernels/gemm/a4w4/README.md#35-where-to-place-lwlr-for-scales) — Worked example of how the SP-to-LDS FIFO (shared between `ds_read` and `ds_write`) drives the scale pipeline design.
+- [v3_lds kernel tutorial](../kernels/gemm/intra_wave/a16w16/v3_lds/README.md) — Practical application of the throughput model to evaluate LDS layout designs (raw, swizzling, padding).
+- [v5_local_prefetch kernel tutorial](../kernels/gemm/intra_wave/a16w16/v5_local_prefetch/README.md) — Where latency hiding (the other side of the latency-vs-throughput distinction in §3) is worked through concretely.
+- [a4w4 §2.5](../kernels/gemm/intra_wave/a4w4/README.md#25-ds_read_tr-hardware-assisted-layout-conversion-for-scales) — Full treatment of `ds_read_tr` in the MXFP4 scale pipeline.
+- [a4w4 §3.5–3.6](../kernels/gemm/intra_wave/a4w4/README.md#35-where-to-place-lwlr-for-scales) — Worked example of how the SP-to-LDS FIFO (shared between `ds_read` and `ds_write`) drives the scale pipeline design.
 - [LDS throughput validation experiment](../experiments/lds_throughput_validation) — Microbenchmark + ATT traces validating the steady-state model.
 - [Performance philosophy](performance_philosophy.md) — The broader block-level programming model that the LDS throughput model fits inside.

docs/performance_philosophy.md

@@ -18,7 +18,7 @@ Neither problem is hard because the hardware is hard. They are hard because the
 Gluon is a **block-level** programming model. Kernels operate on tiles and express pipelines in terms of `DOT`, `local_load`, `async_copy`, and related block-level ops. Layouts are explicit. The kernel author designs, at block level, the things a traditional compiler tries to recover from thread-level IR:
 
 - **Dependencies are a design decision.** When a Gluon author writes a 3-stage pipeline where `DOT`, `local_load`, and `buffer_load` are independent within an iteration, that independence is a *structural property of the kernel*, not a fact to be recovered. Downstream, `mfma`, `ds_read`, and `buffer_load_to_lds` inherit that independence and can be interleaved freely based on throughput — not on dependency analysis.
-- **Register usage has a closed form.** At block level, register requirements are arithmetic: `(M × N × elemType × sharing_factor) / (num_warps × waveSize)` per tile. The kernel author evaluates the formula up front, budgets registers against the SIMD's 512 VGPRs, and slices along M or N if the budget does not fit (see [v7_sliceN](../kernels/gemm/a16w16/v7_sliceN/README.md)). Allocation is not graph coloring at this level — it is bookkeeping.
+- **Register usage has a closed form.** At block level, register requirements are arithmetic: `(M × N × elemType × sharing_factor) / (num_warps × waveSize)` per tile. The kernel author evaluates the formula up front, budgets registers against the SIMD's 512 VGPRs, and slices along M or N if the budget does not fit (see [v7_sliceN](../kernels/gemm/intra_wave/a16w16/v7_sliceN/README.md)). Allocation is not graph coloring at this level — it is bookkeeping.
 
 > [!IMPORTANT]
 > The methodological shift: **what used to be compiler problems become kernel design problems.** And kernel design problems are tractable — the author has full block-level visibility and can evaluate register formulas, pipeline depths, and dependency chains by hand.
@@ -53,7 +53,7 @@ See [kernels/gemm/README.md §2.1](../kernels/gemm/README.md#21-triton-build-and
 The goal is not to keep `llirSched`, `force-agpr`, and `amdgcnas` outside the standard Triton/LLVM flow forever. The goal is to fold their ideas into the LLVM backend in three phases, smallest-lift first:
 
 1. **`llirSched` → an LLVM backend scheduling pass**, gated on backend and kernel shape. This retires most of the friction: users on stock Triton + LLVM reach the O(n)-interleaving regime without loading a plugin.
-2. **`force-agpr` → LLVM's AMDGPU register allocator.** The flags are small, local changes; the challenge is making the policy *selective* — the `RewriteMFMAFormStage` pass, which chooses AGPR vs. VGPR form per MFMA by register pressure so it need not fall back to the blunt all-AGPR form for kernels where the epilogue is a larger share of runtime (see [a16w16 v7 §4.3](../kernels/gemm/a16w16/v7_sliceN/README.md#43-register-allocation-workaround)).
+2. **`force-agpr` → LLVM's AMDGPU register allocator.** The flags are small, local changes; the challenge is making the policy *selective* — the `RewriteMFMAFormStage` pass, which chooses AGPR vs. VGPR form per MFMA by register pressure so it need not fall back to the blunt all-AGPR form for kernels where the epilogue is a larger share of runtime (see [a16w16 v7 §4.3](../kernels/gemm/intra_wave/a16w16/v7_sliceN/README.md#43-register-allocation-workaround)).
 3. **`amdgcnas` → an LLVM AMDGPU MachineInstr-level pass.** The biggest engineering lift and the smallest measured impact on FP16/BF8 (~2pp MFMA efficiency); may remain a prototype indefinitely.
 
 This work is in progress in collaboration with LLVM engineers. When phases 1 and 2 land, upstream Triton + stock LLVM will produce most of what the three components produce today on a pinned build, and this tutorial's out-of-tree plugin dependency can retire.

docs/regenerating_ir_dumps.md

@@ -2,9 +2,9 @@
 
 Two kernel versions in this tutorial bundle compiler dump artifacts so the README narrative can link to specific lines:
 
-- `kernels/gemm/a16w16/v3_lds/ir_dump_K4096_fp16/` (three subdirectories: `no_swizzling/`, `swizzling_8-2-8/`, `padding_512-16/`)
-- `kernels/gemm/a16w16/v5_local_prefetch/ir_dump_K4096_fp16/`
-- `kernels/gemm/a16w16/v5_local_prefetch/ir_dump_K4096_fp16_llirSched/`
+- `kernels/gemm/intra_wave/a16w16/v3_lds/ir_dump_K4096_fp16/` (three subdirectories: `no_swizzling/`, `swizzling_8-2-8/`, `padding_512-16/`)
+- `kernels/gemm/intra_wave/a16w16/v5_local_prefetch/ir_dump_K4096_fp16/`
+- `kernels/gemm/intra_wave/a16w16/v5_local_prefetch/ir_dump_K4096_fp16_llirSched/`
 
 Each directory contains four files: `.ttgir` (Triton GPU IR), `.llir` (LLVM IR), `.amdgcn` (final assembly as Triton emits it, with debug labels), and `.s` (the same assembly with `.loc` directives and `.Ltmp` labels stripped, which is what the READMEs link to so the cited line numbers stay stable). All artifacts in this repository were produced against the [`gfx950-tutorial-v1.0`](https://github.com/triton-lang/triton/releases/tag/gfx950-tutorial-v1.0) tag in `triton-lang/triton`. To verify them, or regenerate after a Triton bump, follow the steps below.
 
@@ -75,7 +75,7 @@ For each variant:
 # Choose the variant by editing v3_lds/matmul_kernel.py's matmul() launcher
 # (see the comment near line 244)
 
-cd kernels/gemm/a16w16
+cd kernels/gemm/intra_wave/a16w16
 export TRITON_CACHE_DIR=/tmp/triton_cache_v3_<variant>
 rm -rf "$TRITON_CACHE_DIR"
 python bench.py --version 3 --K 4096 --dtype fp16
@@ -90,7 +90,7 @@ emit "$SRC" v3_lds/ir_dump_K4096_fp16/<variant> <kernel_name>
 ## v5_local_prefetch — base, llirSched, and llirSched+amdgcnas variants
 
 ```bash
-cd kernels/gemm/a16w16
+cd kernels/gemm/intra_wave/a16w16
 
 # (uses the emit() helper defined above)