TD-005-instruct-2507-variant.md

id	TD-005
title	Use the Instruct-2507 variant (256K native) instead of the original with YaRN
status	accepted
date	2026-06-09
supersedes
superseded_by
tags	model context moe

TD-005: Use the Instruct-2507 variant (256K native) instead of the original with YaRN

Context

Qwen3-30B-A3B comes in several variants. The original (May 2025) has 32K native, extendable to 131K only with YaRN configured by hand, which however degrades performance on short context. The "2507" variants (July 2025) have 256K native with no tricks.

Decision

We use Qwen3-30B-A3B-Instruct-2507 (or Thinking-2507 for reasoning). The ref in carbon.toml is qwen3:30b-a3b-instruct-2507-q4_K_M.

Rationale

256K native without YaRN: no penalty on short context, no fragile manual configuration. On this box the practical limit is still RAM (~32K comfortably), but starting from a model with a long native context avoids degradation and gotchas.

Consequences

• Positive: no YaRN config; no degradation on short text; context headroom if RAM/swap is increased in the future.
• Negative: the exact tag must be confirmed at build time against the Ollama/HF registry; the 1M variant exists but needs ~240 GB → excluded.

Alternatives Considered

Original + YaRN to reach 131K

Rejected: YaRN must be configured by hand and degrades short context, which is the common use case.

The 1M variant

Rejected: ~240 GB of RAM, out of scale for this laptop.