v2.3.0 — NIAH (Needle in a Haystack) Benchmark

New Benchmark: NIAH (Needle in a Haystack)

Tests LLM long-context retrieval by inserting a "needle" fact at varying depths in a long "haystack" document.

Features

• 3 data sources: Kamradt Original (EN), NeedleBench (ZH+EN), LongBench (ZH)
• 3 scoring modes: substring match (default), exact match, token-level F1
• Custom dataset generator: --generate-niah CLI tool to create NIAH tests from your own text
• Config template: config.niah.template.yaml
• 49 tests: full coverage of extractor, scorer, generator, presets, example datasets

Usage

evaluation:
  dataset_paths:
    - "datasets/example/niah/kamradt/"
  evaluation_method: "niah"

Generator

twinkle-eval --generate-niah \
    --haystack my_docs.txt \
    --needle "The secret code is 42." \
    --question "What is the secret code?" \
    --answer "42" \
    --context-lengths 1024,4096,16384 \
    --needle-depths 0,25,50,75,100

Other Changes

• CLAUDE.md: require full pytest tests/ run before PR submission (§6.6, §13)

Full Changelog: v2.2.0...v2.3.0