sparse-attention

[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.

Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper

Fast Multi-dimensional Sparse Attention

Trainable fast and memory-efficient sparse attention

[ICML2025, NeurIPS2025 Spotlight] Sparse VideoGen 1 & 2: Accelerating Video Diffusion Transformers with Sparse Attention

[NeurIPS 2025] Radial Attention: O(nlogn) Sparse Attention with Energy Decay for Long Video Generation

Speed Always Wins: A Survey on Efficient Architectures for Large Language Models

SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse–Linear Attention

[ICML 2025 Spotlight] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

Efficient triton implementation of Native Sparse Attention.

Code for paper: [ICLR2025 Oral] FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference

[CoLM'25] The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

NVIDIA Sol-Attn for ComfyUI / Triton kernel on SM89 - SM121, with zero-copy MiniMax H3 nodes: memory-efficient attention, scheduled tau with graph preview, and feed-forward chunking. Measured 1.14–1.44× vs SageAttention and −37% MLP peak VRAM on H3

🦀 Decoder-only LLM built from scratch in pure Rust using Candle — no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE, native video/document understanding, long-horizon tool agents, quantization-aware training. Scales: Tiny (25M) to Large (1.3B).

Vortex: Programmable Sparse Attention for Agents as Algorithm Designers

Advancing the frontier of efficient AI

[ICLR 2026] SparseD: Sparse Attention for Diffusion Language Models

Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers

Piecewise-Taylor Attention