Developmental Atlas of Attention Head Specialization: Spacing, Stranding, and the Capacity Tax of BPE Tokenization. ~50% of attention heads in standard BPE are mandatory whitespace boundary recovery. Merge barriers eliminate the tax.
transformer tokenization spacing bpe attention-heads gpt-neox mechanistic-interpretability developmental-interpretability merge-barriers capacity-tax
-
Updated
Jul 9, 2026 - Python