Skip to content

Commit 999e082

Browse files
Gui-YueGui-Yue
committed
refactor(benchmark): simplify to SWE-Verified + TB2 official flow, fix TB2 scoring, and trim dead TAU
 assets - unify benchmark pipeline around --benchmark=swe|tb2|both - switch SWE runner to verified-only instances - add official TB2 harbor wrapper runner (run-tb2-official.ts) - fix TB2 score parsing to ignore top-level summary result.json - remove legacy mini/full SWE + TAU + HTML benchmark code paths - delete unused TAU domain fixtures under tests/benchmark/tau/domains - update benchmark result docs (EN/ZH) to confirmed latest scores
1 parent 70d3de3 commit 999e082

36 files changed

Lines changed: 1110 additions & 4001 deletions

‎.github/workflows/benchmark.yml‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
name: Benchmark Full Suite
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
benchmark:
7+
description: "Which benchmark to run"
8+
type: choice
9+
required: true
10+
default: both
11+
options:
12+
- both
13+
- swe
14+
- tb2
15+
provider:
16+
description: "SWE provider filter"
17+
type: choice
18+
required: true
19+
default: all
20+
options:
21+
- all
22+
- anthropic
23+
- openai
24+
- gemini
25+
tb2_model:
26+
description: "TB2 model in provider/model format"
27+
type: string
28+
required: true
29+
default: openai/glm-5
30+
31+
env:
32+
NODE_VERSION: "20"
33+
34+
permissions:
35+
contents: read
36+
37+
jobs:
38+
benchmark:
39+
name: Benchmark
40+
runs-on: ubuntu-latest
41+
timeout-minutes: 360
42+
env:
43+
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
44+
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
45+
46+
steps:
47+
- name: Checkout
48+
uses: actions/checkout@v4
49+
50+
- name: Setup Node.js
51+
uses: actions/setup-node@v4
52+
with:
53+
node-version: ${{ env.NODE_VERSION }}
54+
cache: npm
55+
56+
- name: Setup uv
57+
uses: astral-sh/setup-uv@v4
58+
59+
- name: Login to Docker Hub (optional)
60+
if: ${{ env.DOCKERHUB_USERNAME != '' && env.DOCKERHUB_TOKEN != '' }}
61+
uses: docker/login-action@v3
62+
with:
63+
username: ${{ env.DOCKERHUB_USERNAME }}
64+
password: ${{ env.DOCKERHUB_TOKEN }}
65+
66+
- name: Install dependencies
67+
run: npm ci
68+
69+
- name: Create benchmark environment
70+
run: |
71+
cat > .env.test << 'EOT'
72+
ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
73+
ANTHROPIC_MODEL_ID=${{ vars.ANTHROPIC_MODEL_ID }}
74+
ANTHROPIC_BASE_URL=${{ vars.ANTHROPIC_BASE_URL }}
75+
76+
OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY }}
77+
OPENAI_MODEL_ID=${{ vars.OPENAI_MODEL_ID }}
78+
OPENAI_BASE_URL=${{ vars.OPENAI_BASE_URL }}
79+
80+
GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
81+
GEMINI_MODEL_ID=${{ vars.GEMINI_MODEL_ID }}
82+
GEMINI_BASE_URL=${{ vars.GEMINI_BASE_URL }}
83+
84+
BENCHMARK_DOCKER_PROXY=${{ vars.BENCHMARK_DOCKER_PROXY }}
85+
BENCHMARK_TIMEOUT_MS=${{ vars.BENCHMARK_TIMEOUT_MS }}
86+
EOT
87+
88+
- name: Run unified benchmark command
89+
run: |
90+
mkdir -p tests/tmp
91+
args=(
92+
--benchmark=${{ inputs.benchmark }}
93+
--tb2-model=${{ inputs.tb2_model }}
94+
--tb2-agent=oracle
95+
--tb2-runner=uvx
96+
--tb2-python=3.12
97+
--tb2-jobs-dir=./tests/tmp/jobs
98+
--output=json
99+
--output-file=tests/tmp/benchmark-report.json
100+
)
101+
102+
if [[ "${{ inputs.provider }}" != "all" && "${{ inputs.benchmark }}" != "tb2" ]]; then
103+
args+=(--provider=${{ inputs.provider }})
104+
fi
105+
106+
npm run test:benchmark -- "${args[@]}"
107+
108+
- name: Write step summary
109+
if: ${{ always() }}
110+
run: |
111+
node - <<'NODE' >> "$GITHUB_STEP_SUMMARY"
112+
const fs = require('fs');
113+
function readJson(p) {
114+
if (!fs.existsSync(p)) return null;
115+
try { return JSON.parse(fs.readFileSync(p, 'utf8')); } catch { return null; }
116+
}
117+
118+
const report = readJson('tests/tmp/benchmark-report.json');
119+
console.log('## Benchmark Report');
120+
console.log('');
121+
122+
if (!report) {
123+
console.log('- report not found');
124+
process.exit(0);
125+
}
126+
127+
if (Array.isArray(report.swe) && report.swe.length > 0) {
128+
console.log('### SWE-bench-Verified');
129+
console.log('');
130+
console.log('| Provider / Model | Resolved | Rate |');
131+
console.log('|---|---:|---:|');
132+
for (const r of report.swe) {
133+
const name = `${r.provider.id} / ${r.provider.model}`;
134+
const resolved = `${r.summary.resolved}/${r.summary.total}`;
135+
const rate = `${(r.summary.rate * 100).toFixed(1)}%`;
136+
console.log(`| ${name} | ${resolved} | ${rate} |`);
137+
}
138+
console.log('');
139+
}
140+
141+
if (report.tb2) {
142+
const tb2 = report.tb2;
143+
console.log('### Terminal Bench 2.0');
144+
console.log('');
145+
console.log(`- Agent: \`${tb2.agent}\``);
146+
if (tb2.model) console.log(`- Model: \`${tb2.model}\``);
147+
console.log(`- Passed: **${tb2.passed}/${tb2.total}**`);
148+
console.log(`- Rate: **${(tb2.rate * 100).toFixed(1)}%**`);
149+
console.log('');
150+
}
151+
NODE
152+
153+
- name: Upload benchmark artifacts
154+
if: ${{ always() }}
155+
uses: actions/upload-artifact@v4
156+
with:
157+
name: benchmark-artifacts-${{ github.run_id }}
158+
if-no-files-found: warn
159+
path: |
160+
tests/tmp/benchmark-report.json
161+
tests/tmp/jobs/*/result.json

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,7 @@ See [docs/en/guides/architecture.md](./docs/en/guides/architecture.md) for detai
146146
| [Providers](./docs/en/guides/providers.md) | Model provider configuration |
147147
| [Database](./docs/en/guides/database.md) | SQLite/PostgreSQL persistence |
148148
| [Resume & Fork](./docs/en/guides/resume-fork.md) | Crash recovery & branching |
149+
| [Benchmark Results](./docs/en/guides/benchmark-results.md) | Confirmed benchmark score tables |
149150
| **Project** | |
150151
| [Contribution Guide](./docs/en/contribution.md) | How to contribute |
151152
| **Reference** | |

‎README.zh-CN.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ npm run example:room # 多Agent协作
106106
| [Provider 配置](./docs/zh-CN/guides/providers.md) | 模型 Provider 配置 |
107107
| [数据库存储](./docs/zh-CN/guides/database.md) | SQLite/PostgreSQL 持久化 |
108108
| [恢复与分叉](./docs/zh-CN/guides/resume-fork.md) | 崩溃恢复与分支 |
109+
| [Benchmark 结果](./docs/zh-CN/guides/benchmark-results.md) | 已确认的跑分结果表格 |
109110
| **项目** | |
110111
| [贡献指南](./docs/zh-CN/contribution.md) | 提交 PR 的要求与流程 |
111112
| **参考** | |

‎docs/en/guides/benchmark-results.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Benchmark Results (Confirmed)
2+
3+
Last updated: 2026-02-26
4+
5+
## SWE-bench-Verified
6+
7+
| Provider / Model | Instances | Resolved | Rate | Avg Tokens | Avg Duration |
8+
|---|---:|---:|---:|---:|---:|
9+
| openai / glm-5 | 12 | 12/12 | 100.0% | 17.2k | 134.5k ms |
10+
11+
Source: local full run log (`2026-02-25__21-06-21`).
12+
13+
## Terminal Bench 2.0
14+
15+
| Agent / Model | Passed | Parseable | Unknown | Rate (parseable) | Notes |
16+
|---|---:|---:|---:|---:|---|
17+
| oracle / glm-5 | 1 | 31 | 58 | 3.2% | From the same full run; many tasks ended with runtime/timeout errors. |
18+
19+
## Reproduce
20+
21+
```bash
22+
npm run test:benchmark -- \
23+
--benchmark=both \
24+
--tb2-model=openai/glm-5 \
25+
--tb2-agent=oracle \
26+
--tb2-runner=uvx \
27+
--tb2-jobs-dir=./tests/tmp/jobs \
28+
--output=json \
29+
--output-file=tests/tmp/benchmark-report.json
30+
```
31+
32+
The JSON report includes both `swe` and `tb2` sections.

0 commit comments

Comments
 (0)