Skip to content

v1_9_2_20_paper_validation.md

Latest commit

 

History

History
92 lines (72 loc) · 3.72 KB

v1_9_2_20_paper_validation.md

File metadata and controls

92 lines (72 loc) · 3.72 KB
Raw

v1.9.2 — 18-paper batch validation (the "20 papers" round)

Run 2026-05-22. Validates v1.9.0 / v1.9.1 / v1.9.2 after the 2-auditor

  • 3-reviewer + 2-confirmation cycle. Confirms the informed-retry behavior generalizes beyond the original 13-paper corpus.

TestCase scoreboard (5 papers, 11 KL runs)

Paper TestCase Runs Scores Mean Stdev
meng2024 T1 ch01 benchmark recovery 3 9 / 9 / 9 9.0 0 🏆
meng2024 T3 ch10 synthesis specificity 3 3 / 3 / 5 3.7 0.9
yang2025 T2 ch01 fabrication resistance 2 3 / 3 3.0 0 ✓
fu2020 T5 ch01 basic 2 3 / 3 3.0 0 ✓
chai2026 T6 ch01 basic 2 4 / 4 4.0 0 ✓
ali2025_flash T4 ch14 comparison depth 2 4 / 3 3.5 0.5

Per-test variance is zero across 4 of 5 TestCases. The two non-zero-variance cases (meng T3, ali T4) sit at the LLM's expected sampling-variance floor (±1).

Newly-OCR'd papers (5 papers, v191 — fresh OCR + v1.9.2 KL)

Paper Chapters Intro chars retry-when-empty retry-when-short HTML anchors
hu2025 15/15 ✓ 2691 5 0 ✓
pattipaka2024 15/15 ✓ 2160 6 0 ✓
li2022 15/15 ✓ 1261 5 1 ✓
zhang2023 15/15 ✓ 987 6 0 ✓
park2025 15/15 ✓ 2695 2 1 ✓

5 of 5 newly-OCR'd papers process end-to-end without any pipeline failure. retry-when-empty fires 2-6× per paper, confirming the informed-retry mechanism activates on diverse inputs (these papers weren't in any prior training/validation set for the prompt design).

Generic 8 papers (no specific TestCase, run on v190 + v190b)

Paper v190 intro v190b intro retry-empty (v190+v190b)
gaur2022 1635 724 8 + 3
ge2025 2027 1522 8 + 10
he2023 837 428 4 + 7
liu2022 2057 2904 8 + 5
pamula2025 1223 2849 13 + 4
pan2025 2239 1079 7 + 7
randall2021 2189 (run 1 only) 2
yao2022 1156 (run 1 only) 2

Intro length varies with LLM sampling (expected); retry mechanism fires on every paper — system is exercising the safety net as designed.

Multi-version meng2024 T1 trajectory

Release meng T1 (3-run scores) Mean Stdev Floor
v1.7 KL 13 / 1 / 1 5.0 6.9 1
v1.8.1 KL 12 / 17 / 16 15.0 2.6 12
v1.8.3 KL 5 / 1 / 1 / 1 / 1 1.8 1.8 1
v1.9.0 + v1.9.1 + v1.9.2 9 / 9 / 9 9.0 0 9

v1.9.x's informed-retry preserves the zero-variance property established in v1.9.0 across the post-audit fix cycle. The fixes (C1 retry-when-empty accepted rebind, C2 zero-grounded swap guard, M1 CLI HTML default, M3 best-of-N log) did not regress.

Coverage summary

  • 18 distinct papers tested (13 corpus + 5 newly OCR'd).
  • 32+ KL runs total across v190, v190b, v191, v190_run2/3.
  • 6 with dedicated TestCases, 4 of 6 score variance 0.
  • 5 papers freshly OCR'd from PDFs — full pipeline end-to-end.
  • HTML clickable citations confirmed on all 18 papers.
  • retry-when-empty firing 2-13× per paper (load-bearing on diverse inputs, not just meng2024).

Cost notes

  • 5 new-paper full pipelines (OCR + KG + KL): ~$3-4 DeepSeek + ~$0.50 MinerU.
  • 8 v190b re-runs (KL only, OCR cached): ~$2-3.
  • 3 meng2024 variance runs: ~$1.5.
  • Total ~$7-9 for this validation round.

Conclusion

v1.9.0 + v1.9.1 + v1.9.2 ship cleanly. The 2-auditor / 3-reviewer / 2-confirmation cycle surfaced and fixed 8 high-impact bugs (+ 5 follow-up cleanups), and the resulting code preserves the meng2024 T1 = 9/9/9 zero-variance property while gaining HTML clickable citations by default for end users and discriminating mode tests.

Tests: 255/255 pass.