Deepseek-OCR-2 Quants Support

[soymh]

Is your feature request related to a problem? Please describe.

I've been running DeepSeek-OCR-2 on a laptop GPU (RTX 3050, 3.68GB VRAM) and after a long debugging journey — including profiling inference with HuggingFace Transformers, measuring true token rates, and testing every quantization path available — I landed on deepseek-ocr.rs as the most promising engine for my hardware. The Rust engine's 1.46× decode speedup over Python is exactly what I need.

There are two blockers I hit back to back:

Blocker 1 — No quantized variants for OCR-2.
deepseek-ocr-2 runs fine in FP16, but there are no quantized variants registered. Trying any of them produces:

error: unknown model id `deepseek-ocr-2-q4k` (baseline `deepseek-ocr-2-q4k`)

The FP16 OCR-2 weights need ~6.3GB VRAM, which exceeds most laptop GPUs. Without quantized variants, OCR-2 is effectively CPU-only on this hardware class — which defeats the entire purpose of using the Rust engine. I even used the existing CLI to successfully generate a .dsq snapshot from my OCR-2 weights, so the tooling works — the model IDs just aren't registered.

Blocker 2 — No CUDA 13 Linux builds.
Compiling with --features cuda fails immediately on Linux with CUDA 13.2:

thread 'main' panicked at cudarc-0.19.1/build.rs:138:5:
Unsupported cuda toolkit version: `13.2`. Please raise a github issue.

CUDA 13.x ships with the latest NVIDIA drivers on Linux (595.x+). Users on current driver stacks have no path to GPU acceleration without either downgrading their entire CUDA toolkit or patching the build manually. The existing Docker images target CUDA 12.5.1, which means the pre-built binaries and containers are also unusable for this driver generation.

I asked Claude (Anthropic's AI assistant) to help me write this issue description clearly after going through this debugging process together, so apologies if it reads a little more structured than usual! 😄

---

Describe the solution you'd like

For OCR-2 quantization:

Register three new model IDs in the built-in registry, mirroring the existing OCR-1 quantized entries:

• deepseek-ocr-2-q4k → kind = "deepseek-ocr-2", snapshot.dtype = "Q4_K"
• deepseek-ocr-2-q6k → kind = "deepseek-ocr-2", snapshot.dtype = "Q6_K"
• deepseek-ocr-2-q8k → kind = "deepseek-ocr-2", snapshot.dtype = "Q8_0"

These should behave identically to deepseek-ocr-q4k etc. — load FP16 base weights from deepseek-ai/DeepSeek-OCR-2 and patch linear layers with the DSQ snapshot. The good news is that the OCR-2 decoder is identical to OCR-1's (both are the DeepSeek-V2 3B MoE), so the existing DSQ adapter should apply without modification. The layer names to target are the same: model.layers.*.self_attn.{q,k,v,o}_proj, model.layers.*.mlp.experts.*.{gate,up,down}_proj, and lm_head.weight. This should be a small registry change rather than a new backend.

For CUDA 13 Linux support:

• Bump cudarc to a version that recognises CUDA 13.x, or add CUDA 13.x to the supported version list in build.rs
• Publish Linux CUDA 13 builds in the build-binaries workflow alongside the existing macOS Metal and Windows bundles — ideally targeting cuda13.x-cc86 (Ampere, covers RTX 30xx laptop GPUs) as a starting point
• Update the Docker image tags to include a cuda13.x variant

---

Describe alternatives you've considered

• BitsAndBytes NF4 via HuggingFace Transformers: works but runs at ~16 t/s on RTX 3050 due to Python/GIL overhead — the whole reason I'm here.
• vLLM: doesn't support BitsAndBytes NF4/INT8, and AWQ conversion fails on this custom architecture due to the SAM+Qwen2 vision stack.
• AWQ conversion: AutoAWQ can't reliably quantize the DeepseekOCR2 custom model class.
• CUDARC_CUDA_VERSION=12.8 workaround: compiles but untested for correctness; not a solution for users who just want to grab a binary and run.
• CPU FP16 via this engine: works but ignores the GPU entirely and ends up slower than quantized GPU inference.

---

Additional context

Hardware: NVIDIA GeForce RTX 3050 Laptop GPU (3.68GB VRAM, Ampere SM8.6), Intel i5-12500H, 16GB DDR5, Driver 595.71.05, CUDA 13.2, Arch Linux.

When loaded via HuggingFace NF4, OCR-2 sits at 2.12GB VRAM — so a Q4K DSQ variant should fit comfortably within 3.68GB with headroom for KV cache. This hardware class is genuinely viable for GPU inference on OCR-2, it just needs the registry entries and a working CUDA build to unlock it.

I have the .dsq file I generated from my OCR-2 weights and I'm happy to share it for testing, or validate any builds you put together. Thanks for the great project — the DSQ quantization work and the OCR-1 pipeline are exactly the kind of thing that makes local inference actually usable on constrained hardware! 🙏