feat(#231): LLM benchmark v2 — typed assertions, skill-trace, LLM-as-Judge, CI migration

[ZhouChaunge]

Summary / 概述

Implements #231: LLM-as-Judge benchmark evaluation framework — v2 assertions, skill-hit tracing, LLM judge evaluator, scenario schema upgrade, and CI migration from llm-integration to llm-benchmark.

---

What Changed / 变更内容

New files / 新增文件

File	Purpose (EN)	用途 (中文)
tests/llm-benchmark/lib/skill-trace.cjs	Extracts skill match results from Agent messages by parsing detect_structure_type tool call responses	从 Agent messages 中解析 detect_structure_type 工具调用，提取 skillId 和 structureType
tests/llm-benchmark/lib/judge.cjs	LLM-as-Judge evaluator for natural_language assertions (temperature=0, timeout=30s, max_tokens=500, model via LLM_JUDGE_MODEL)	natural_language 断言的 LLM 评判器，参数固定（temperature=0, 30s 超时），模型通过 LLM_JUDGE_MODEL 配置
.github/workflows/llm-benchmark.yml	CI workflow for the benchmark suite; uploads benchmark-results.json artifact; triggered by /test-llm-benchmark comment	基准测试 CI workflow，上传 benchmark-results.json artifact，通过 /test-llm-benchmark 评论触发

Refactored / 重构文件

File	Change (EN)	变更说明 (中文)
tests/llm-benchmark/lib/evaluate.cjs	Converted to async; type-dispatched evaluators for all 6 v2 assertion types; v1 scenarios auto-upgraded	改为 async；按断言 type 分发到类型化评估器（6 种 v2 类型）；v1 场景自动升级
tests/llm-benchmark/runner.cjs	Supports maxRetries: retries failed scenarios up to N times, uses final result	支持 maxRetries：失败后最多重试 N 次，取最终结果

Schema v1 → v2 upgrade / 场景格式升级

All 5 scenario files updated from flat v1 fields to the new v2 structure with category, tags, maxRetries, skills, and typed assertions array. Backward compatible — v1 format still works.

5 个场景文件从扁平 v1 字段升级为新的 v2 结构，含 category、tags、maxRetries、skills 和类型化 assertions 数组。向后兼容 — v1 格式仍可使用。

CI Migration / CI 迁移

• Added llm-benchmark.yml: full benchmark CI with artifact upload / 新增 llm-benchmark.yml：完整基准测试 CI，含 artifact 上传
• Updated llm-integration.yml: routing-only (deterministic, no LLM key needed) / 更新 llm-integration.yml：仅 routing 类别（确定性，CI 无需 LLM Key）

---

V2 Assertion Types / V2 断言类型

Type	Description (EN)	说明 (中文)
structural_type	Checks state.structuralTypeKey	检查结构类型
has_model	Verifies node/element counts	验证节点/单元数量
has_analysis	Checks analysis result presence	检查分析结果
has_report	Checks report markdown length > 100 chars	检查报告长度
skill_match	Parses detect_structure_type tool call to verify routing	解析工具调用验证路由
natural_language	LLM-as-Judge evaluates free-form criterion	LLM 评判自然语言标准

---

Testing / 测试

# Module load check / 模块加载检查
node -e "require('./tests/llm-benchmark/lib/skill-trace.cjs'); require('./tests/llm-benchmark/lib/judge.cjs'); require('./tests/llm-benchmark/lib/evaluate.cjs'); console.log('OK')"
# → All modules load OK

# Run benchmark (requires LLM_API_KEY)
node tests/runner.mjs llm-benchmark
node tests/runner.mjs llm-benchmark --scenario beam-static-6m

# Run routing tests (no LLM key needed)
node tests/runner.mjs llm-integration

---

Done When / 完成标准 ✅

• llm-benchmark supports all v2 assertion types including natural_language
• Skill-hit tracking verifies skill routing accuracy
• v1 scenario format backward compatible
• CI migrated from llm-integration to llm-benchmark
• AGENTS.md test documentation updated

Closes structureclaw#231

[gemini-code-assist]

Code Review

This pull request enhances the LLM benchmarking suite and improves Python environment management. Key updates include the introduction of v2 assertions for benchmarks—supporting LLM-as-Judge and skill-hit tracing—and a more robust Python dependency synchronization check that parses requirements files and verifies installed versions. Feedback focuses on improving the JSON extraction logic for nested objects in the judge component, addressing the lack of support for nested requirements files in the parser, and mitigating potential command-line length limitations when running the Python check script on Windows.

[Copilot]

Pull request overview

Implements the v2 LLM benchmark framework (typed assertions + skill-hit tracing + LLM-as-Judge), upgrades benchmark scenarios to the v2 schema with retries, migrates CI coverage from llm-integration to a dedicated llm-benchmark workflow, and tightens CLI analysis-Python setup by checking pinned requirement versions.

Changes:

• Added skill-tracing + LLM-as-Judge modules and refactored benchmark evaluation to async, type-dispatched v2 assertions (with v1 auto-upgrade).
• Upgraded benchmark scenarios to v2 schema (category, tags, maxRetries, skills, typed assertions).
• Introduced llm-benchmark CI workflow and simplified llm-integration workflow to routing-only; enhanced CLI Python setup to detect requirements drift.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/runner.mjs	Updates CLI help text for routing-only llm-integration and new v2 llm-benchmark command details.
tests/llm-benchmark/scenarios/beam.json	Migrates beam scenarios to v2 schema, adds retries and natural_language assertions.
tests/llm-benchmark/scenarios/double-span-beam.json	Migrates scenario to v2 schema and adds skill routing assertions.
tests/llm-benchmark/scenarios/frame.json	Migrates frame scenarios to v2 schema, adds retries and natural_language assertions.
tests/llm-benchmark/scenarios/portal-frame.json	Migrates portal-frame scenarios to v2 schema with routing assertions.
tests/llm-benchmark/scenarios/truss.json	Migrates truss scenario to v2 schema with routing assertions.
tests/llm-benchmark/runner.cjs	Adds per-scenario retry support and awaits async evaluation.
tests/llm-benchmark/lib/evaluate.cjs	Implements v2 assertion dispatch, v1→v2 upgrade, and async natural_language evaluation.
tests/llm-benchmark/lib/skill-trace.cjs	Extracts skillId / structureType from detect_structure_type tool result messages.
tests/llm-benchmark/lib/judge.cjs	Adds HTTP-based LLM judge client + prompt building and JSON response parsing.
scripts/cli/runtime.js	Adds requirements parsing + runtime check helpers to detect Python pinned-version drift.
scripts/cli/main.js	Uses the new requirements drift checks when determining whether analysis Python env is “ready”.
backend/tests/analysis-python-setup.test.mjs	Adds tests covering requirements drift reinstalls + requirements parsing/script generation.
AGENTS.md	Documents new benchmark runner commands and clarifies routing-only integration tests.
.github/workflows/llm-integration.yml	Renames to routing-only and hardwires CI execution to routing.
.github/workflows/llm-benchmark.yml	Adds a dedicated benchmark workflow with PR-comment trigger, artifact upload, and PR reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.