v0.2.1

What's Changed

Code Structure Updates

• Re-organized C++ source code into csrc folder with better modualized function managements.
• Exposed C++ interface to Python with pybind11.

Feature Updates

• Added batch support in major inference process.

• Auto weight loading for Llama-ish models.

• Offline benchmarking with naive warmup.

• Ceval and mmlu tests.

• Improved service support.

• Paged Attention using InfiniCore operators.

• Paged Attention using Flash Attention backend.

• CUDA Graph for Nvidia and a few other CUDA-ish or CUDA-supported platforms (requires paged attention).

• Quantized Inference: w4a16 awq fp16.

New Supported Platforms

• Ali PPU
• Metax MACA
• QY

New Supported Models

• 9g8b thinking llama
• New Supported Models in the Classic Inference in src:
  Qwen3-VL - text only
  Qwen2.5-AWQ
  Qwen2.5-GPTQ

Full Changelog: v0.1.0-late-release...v0.2.1

Backend Requirements

InfiniCore@ed61b51

Contributors (in ID alphabetical order)

@AcidProton @Ceng23333 @gongchensu @ma-hang @PanZezhong1725 @pengcheng888 @qinyiqun @scbz4learning @spike-zhu @Susskind115 @xgqdut2016 @voltjia @wooway777 @zhangyue207