v2.2.0 — BFCL v1 Function-Calling Evaluation

What's New

BFCL v1 — Berkeley Function-Calling Leaderboard

新增 BFCL v1 評測，支援兩種模式評估模型的 function-calling 能力：

• FC mode (bfcl_fc)：使用 OpenAI tool_calls API，評估結構化 function-calling 輸出
• Prompting mode (bfcl_prompt)：將 function schema 注入 system prompt，從文字輸出中解析 function call（支援 reasoning model 的 <think> block）
• 支援 simple / multiple / parallel 三種 function-calling 子類型
• AST-based 結構比對評分（function name + argument matching）
• Dataset converter 可將 raw BFCL 資料轉換為評測格式
• 70 個 pytest 全部通過

安裝

pip install twinkle-eval[tool]

Config 範例

evaluation:
  dataset_paths:
    - "datasets/bfcl_v1/simple/"
  evaluation_method: "bfcl_fc"   # or "bfcl_prompt"

Full Changelog: v2.1.0...v2.2.0

v2.1.0 — IFEval & IFBench Instruction-Following Evaluation

What's New

新增兩個 instruction-following 評測 benchmark，讓 Twinkle Eval 支援完整的指令遵循能力評估。

IFEval — Google Instruction Following Evaluation

• 25 種可驗證指令類型（change_case、keywords、length_constraints 等）
• 移植自 Google Research 官方實作（Apache 2.0）
• 4 個指標：strict/loose × prompt/instruction
• evaluation_method: "ifeval"
• pip install twinkle-eval[ifeval]

IFBench — AllenAI Instruction Following Benchmark (OOD)

• 58 種 out-of-distribution 指令類型，分為 7 大類別（count, ratio, words, sentence, format, custom, repeat）
• 移植自 AllenAI IFBench（Apache 2.0, NeurIPS 2025）
• 與 IFEval 共用 strict/loose 評分框架
• evaluation_method: "ifbench"
• pip install twinkle-eval[ifbench]

Score Parity（與官方工具對比）

Benchmark	Metric	Twinkle Eval	Official	Diff
IFEval (541 rows)	prompt_strict	89.65%	89.65%	+0.00% ✅
IFEval (541 rows)	instruction_strict	92.93%	92.93%	+0.00% ✅
IFBench (294 rows)	prompt_strict	44.22%	44.22%	+0.00% ✅
IFBench (294 rows)	instruction_strict	47.20%	47.16%	+0.04% ✅

Other Changes

• Evaluator 支援 IFEval（JSON string）與 IFBench（原生 list/dict）兩種資料集格式
• CLAUDE.md 新增 §6.6：每個新 benchmark 必須附 tests/test_{name}.py
• 新增 43 個 IFBench pytest + 31 個 IFEval pytest

Full Changelog: v2.0.0...v2.1.0

v2.0.0 — 模組化架構重構：Extractor/Scorer 拆分

⚠️ Breaking Change 重大版本

本版本為完整架構重構，不向下相容 v1.x 的 import 路徑。若有自訂程式碼使用舊路徑，請依照下方遷移說明更新。

---

主要變更

Extractor / Scorer 拆分（#37）

將原本的 EvaluationStrategy 拆解為兩個獨立介面：

介面	職責	實作
Extractor	從 LLM 輸出中抽取答案字串	PatternExtractor、BoxExtractor、LogitExtractor、MathExtractor、CustomRegexExtractor
Scorer	正規化並判斷答案是否正確	ExactMatchScorer、MathRulerScorer

兩者透過 PRESETS 登錄表組合為具名的 evaluation_method，使用者也可自行組合任意 Extractor + Scorer 傳入 Evaluator。

套件目錄整理

所有根目錄散落的模組已遷移至正確的子套件：

舊路徑（已刪除）	新路徑
twinkle_eval/config.py	twinkle_eval/core/config.py
twinkle_eval/logger.py	twinkle_eval/core/logger.py
twinkle_eval/validators.py	twinkle_eval/core/validators.py
twinkle_eval/evaluators.py	twinkle_eval/runners/evaluator.py
twinkle_eval/benchmark.py	twinkle_eval/runners/benchmark.py
twinkle_eval/finalize.py	twinkle_eval/runners/finalize.py
twinkle_eval/hf_uploader.py	twinkle_eval/integrations/huggingface.py

CLI --init 改版

不再產生單一 config.yaml，改為建立 configs/ 目錄，內含兩份範本：

• configs/config.multiple_choice.template.yaml（適用 pattern / box / logit）
• configs/config.math.template.yaml（適用 math 評測）

新增 Notebooks

notebooks/ 目錄提供兩份教學文件：

• notebooks/01_multiple_choice.ipynb：選擇題評測完整教學
• notebooks/02_math.ipynb：數學評測教學

---

遷移指南（v1.x → v2.0）

# 舊寫法（v1.x）
from twinkle_eval.evaluation_strategies import PatternMatchingStrategy
from twinkle_eval.evaluators import Evaluator

evaluator = Evaluator(evaluation_strategy=PatternMatchingStrategy())

# 新寫法（v2.0）
from twinkle_eval.metrics.extractors.pattern import PatternExtractor
from twinkle_eval.metrics.scorers.exact import ExactMatchScorer
from twinkle_eval.runners.evaluator import Evaluator

evaluator = Evaluator(extractor=PatternExtractor(), scorer=ExactMatchScorer())

config.yaml 格式不受影響，evaluation_method 字串仍完全相容。

---

已修復

• box 評測 max_tokens 不足導致推理截斷（建議 math/box 場景設 4096）
• mathruler 缺少傳遞依賴說明（#35），統一引導使用 pip install twinkle-eval[math]

v1.4.0

What's New

feat: Logit-based evaluation strategy (#7)

Implements lm-evaluation-harness-style logit evaluation for MCQ benchmarks.

How it works:

• Instead of parsing generated text, computes log P(choice | context) for each option separately
• Context ends with "\nAnswer:" and each option uses a leading space (e.g., " A"), matching the lm-harness MMLU template format
• Uses /v1/completions with echo=True — compatible with vLLM, llama.cpp, and OpenAI legacy completions

Parallel execution advantage:
All (question × option) calls are submitted simultaneously to ThreadPoolExecutor, allowing vLLM's scheduler to process them in maximum batch size — significantly faster than lm-harness's sequential per-option evaluation.

Usage: set evaluation_method: logit in config.yaml (no extra dependencies required)

feat: Example benchmark subsets (#24)

Pre-sampled evaluation subsets committed directly into the repo under datasets/example/:

Dataset	Source	Questions	Method
gsm8k/	openai/gsm8k	20	math
aime2025/	MathArena/aime_2025	30	math
tmmluplus/	ikala/tmmluplus	20	box
mmlu/	cais/mmlu	20	box
mmlu_pro/	TIGER-Lab/MMLU-Pro	20	box

v1.3.0 — Slurm 分散式評測與 HuggingFace 上傳支援

新功能

Slurm 多節點分散式評測

每個節點/rank 輸出獨立 shard，評測完成後執行 finalize 自動合併：

# 1. 各節點各自評測（由 Slurm 腳本驅動）
sbatch scripts/run_slurm_full.sh

# 2. 合併碎片並上傳至 HuggingFace
twinkle-eval --finalize-results 20260316_1200 \
  --hf-repo-id my-org/my-benchmark-logs-and-scores \
  --hf-variant v1

HuggingFace 上傳

pip install twinkle-eval[slurm]

twinkle-eval --config config.yaml \
  --hf-repo-id my-org/my-benchmark-logs-and-scores \
  --hf-variant experiment-1

MMLU 格式自動正規化

dataset.py 現在自動將 HuggingFace MMLU 格式（choices list + 整數 answer）轉為 A/B/C/D 具名欄位，無需額外處理。

安裝

pip install twinkle-eval          # 基本版
pip install twinkle-eval[math]    # 含數學評測
pip install twinkle-eval[slurm]   # 含 HuggingFace 上傳
pip install twinkle-eval[math,slurm]  # 全功能

完整 CHANGELOG

請見 CHANGELOG.md

Credits

Slurm 分散式設計原始作者：@whats2000

v1.2.0 — 數學評測策略與 pass@k 支援

新功能

數學評測策略

evaluation:
  evaluation_method: math
  system_prompt:
    zh: "請用 \\boxed{} 包住最終答案。"

安裝：

pip install twinkle-eval[math]

• 從 \boxed{...} 提取答案（支援巢狀大括號）
• 使用 mathruler 進行語意等價判斷
• 支援 LaTeX 指令大小寫正規化
• 支援逗號分隔解集合的無序比對

pass@k 支援

evaluation:
  samples_per_question: 10
  pass_k: 3

per-dataset 覆蓋設定

evaluation:
  evaluation_method: box
  dataset_overrides:
    "datasets/math/":
      evaluation_method: math
      max_tokens: 8192

升級注意事項

evaluate_file() 回傳值由 (path, float, results_path) 改為 (path, dict, results_path)。若你在程式碼中直接呼叫此方法，請更新為：

file_path, metrics, results_path = evaluator.evaluate_file(...)
accuracy = metrics["accuracy"]

完整 CHANGELOG

請見 CHANGELOG.md

v1.1.6 — fix: 推理模型 think tag 自動剝離

Bug Fixes

• fix(evaluators): 統一推理輸出解析，自動處理 inline think tag 與 content=null (#26)
  • 情境 A：content 含完整 <think>...</think> 區塊（如 Ollama 等 backend）→ 自動剝離，從結尾提取答案
  • 情境 B：content=null（如推理模型 skip_special_tokens=true）→ fallback 至 reasoning_content
  • 支援 <think>、<reason>、<reasoning> 三種 tag
  • 截斷的 tag（只有結尾 tag、缺開頭）視為格式不合格，不處理
  • 不需要任何 config 設定，完全自動偵測

v1.1.5 — fix: 推理模型 content=null 時答案提取失敗

Bug Fixes

• fix(evaluators): fallback to reasoning_content when content is null (#23)
  • Reasoning models with skip_special_tokens=true (vLLM default) may return message.content=null, placing the answer in reasoning_content instead
  • Previously, extract_answer(None) was called silently, marking every question incorrect with no error
  • Fix: when content is null or empty, extraction now falls back to reasoning_content
  • If both are null, an error is logged and the question is scored as incorrect (no crash)

v1.1.4 — fix: 多檔評測 JSONL 結果被覆蓋

Bug Fixes

• fix(evaluators): use append mode to prevent JSONL overwrite across multiple files
  • evaluate_file() previously opened the JSONL output with open(..., 'w'), causing the second file in a dataset directory to overwrite the first file's per-question records
  • Fixed by changing to open(..., 'a') so results from all files accumulate correctly
  • Accuracy statistics in results_{timestamp}.json were unaffected (computed in-memory); only the per-question eval_results_{timestamp}_runN.jsonl detail log was affected
  • Experimentally verified: 2 × 150-question files now produce 300 lines instead of 150

Testing

• Added tests/test_pr17_overwrite_bug.py with unit and end-to-end mock-LLM coverage

v1.1.3 — fix: get_info() NameError、版號同步、評測靜默失敗

Bug Fixes

• fix(package): resolve NameError: __email__ is not defined in get_info() (#10)

  • Removed undefined __email__ reference from get_info() and __all__

• fix(version): sync __init__.py __version__ with pyproject.toml

  • Both now consistently report 1.1.3
  • Added TestVersionConsistency test to catch future drift

• fix(eval): raise EvaluationError instead of silently returning empty results (#6)

  • Single-dataset evaluation failures now raise a clear exception instead of exiting with code 0 and no output
  • Partial failures (some datasets succeed, some fail) still continue correctly

Testing

• Added pytest infrastructure (tests/ directory)
• New test modules: test_package.py, test_issue6.py