Optimizes BPE Tokenization

[apaniukov]

Optimizes BPE tokenization in two hot paths:

• Uses a fixed-array linear merge path for pretokens with up to 32 symbols, retaining the priority queue for longer inputs.
• Replaces the node-based result cache with a contiguous, open-addressed table. Cache hits compare string_view directly and avoid constructing an owned string.

The existing cache capacity, synchronization, token IDs, and full-key collision checks are preserved.

Performance

Benchmarked with Qwen/Qwen3-Reranker-0.6B, 2,000 ShareGPT prompts, batch size 1.

Short merge path

   Metric                                 Baseline            Optimized    Change
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━  ━━━━━━━━
   LATENCY BPE mean                       0.458 ms             0.448 ms     -2.2%
  ────────────────────────────  ───────────────────  ───────────────────  ────────
   LATENCY sync throughput       1,795.7 prompts/s    1,857.6 prompts/s     +3.4%
  ────────────────────────────  ───────────────────  ───────────────────  ────────
   THROUGHPUT BPE mean                    0.588 ms             0.565 ms     -3.9%
  ────────────────────────────  ───────────────────  ───────────────────  ────────
   THROUGHPUT sync throughput      572.4 prompts/s      573.7 prompts/s     +0.2%

These results are from single before/after runs.

Open-addressed result cache

Five interleaved before/after pairs per performance hint:

   Metric                                String Cache    Open-Addressed Cache    Mean Change       Paired 95% CI
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━
   LATENCY BPE mean                       0.453548 ms             0.443789 ms         -2.13%    -4.68% to +0.42%
  ─────────────────────────────  ─────────────────────  ──────────────────────  ─────────────  ──────────────────
   LATENCY BPE p50                        0.208900 ms             0.200200 ms         -4.16%    -5.93% to -2.38%
  ─────────────────────────────  ─────────────────────  ──────────────────────  ─────────────  ──────────────────
   LATENCY sync throughput        1,844.467 prompts/s     1,879.074 prompts/s         +1.88%    +1.29% to +2.46%
  ─────────────────────────────  ─────────────────────  ──────────────────────  ─────────────  ──────────────────
   THROUGHPUT BPE mean                    0.563152 ms             0.536678 ms         -4.70%    -5.46% to -3.94%
  ─────────────────────────────  ─────────────────────  ──────────────────────  ─────────────  ──────────────────
   THROUGHPUT BPE p50                     0.334000 ms             0.306500 ms         -8.23%    -9.78% to -6.68%
  ─────────────────────────────  ─────────────────────  ──────────────────────  ─────────────  ──────────────────
   THROUGHPUT async throughput    3,572.159 prompts/s     3,671.403 prompts/s         +2.80%    +0.96% to +4.63%

BPE p50 improved in all ten paired comparisons. No process-level peak-RSS regression was measurable.

Validation

• All benchmark outputs matched Hugging Face.
• The interleaved cache benchmark validated 40,000 outputs in aggregate.
• Tests cover merge ordering, rank/token-ID differences, the 32/33-symbol boundary, collision chains, saturation, long values, repeated hits, and concurrent inference.
• Also fixes fuse_unk handling and adds coverage for byte fallback and end suffixes.
• CI separates pass-rate tests from functional tests so functional failures cannot be masked.