Skip to content

Commit 9d3cd77

Browse files
ravenSansteteclaude
andcommitted
Extract shared scorer utilities, move AgentDojo tools, refresh Chinese README
- Extract _run_coro_sync() to snowl/scorer/_sync_bridge.py (dedup model_judge + grade_judge) - Extract render_judge_prompt() to snowl/scorer/_prompt.py (dedup template rendering) - Move AgentDojo banking/travel tools to snowl/benchmarks/agentdojo/tools.py (P1.5 fix) - Update fortress and toolemu scorers to use shared _sync_bridge - Full Chinese README refresh aligned with English version - Update governance.md: mark P1.3, P1.5, P2.1, P2.2 as RESOLVED Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent cd26602 commit 9d3cd77

11 files changed

Lines changed: 865 additions & 1112 deletions

File tree

README.zh-CN.md

Lines changed: 96 additions & 333 deletions
Large diffs are not rendered by default.

docs/governance.md

Lines changed: 16 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -105,16 +105,16 @@ tool.py ─────────→ errors
105105
| **MEDIUM** | `snowl/runtime/policy.py:9` | Runtime imports `BenchmarkConcurrencyProfile` from benchmarks (reversed dependency direction) |
106106
| **MEDIUM** | `snowl/runtime/policy.py:90-97` | Runtime queries benchmark registry at runtime (deferred circular import) |
107107
| **MEDIUM** | `snowl/runtime/container_providers.py:25` | Runtime imports `OSWorldContainerLauncher` from a specific benchmark |
108-
| **MEDIUM** | `snowl/runtime/engine.py:961` | Engine checks `output.get("osworld_score")` — benchmark-specific logic in generic engine |
109-
| **LOW** | `snowl/tools/stateful_executor.py:53-408` | AgentDojo-specific tool implementations in shared `snowl/tools/` package |
108+
| **MEDIUM** | `snowl/runtime/engine.py:961` | ~~Engine checks `output.get("osworld_score")`~~ **RESOLVED**: replaced with generic `_get_extra_payload_keys()` that reads from benchmark registry |
109+
| **LOW** | `snowl/tools/stateful_executor.py:53-408` | ~~AgentDojo-specific tool implementations in shared `snowl/tools/` package~~ **RESOLVED**: moved to `snowl/benchmarks/agentdojo/tools.py` |
110110
| **LOW** | `snowl/benchmarks/base_adapter.py:46-48` | `benchmark_info()` does deferred import of registry (semantic coupling) |
111111

112112
### 1.5 Duplicate Abstractions
113113

114114
| Duplication | Locations | Fix |
115115
|-------------|-----------|-----|
116-
| `_run_coro_sync()` | `scorer/model_judge.py:30-52`, `scorer/grade_judge.py:20-40` | Extract to `scorer/_sync_bridge.py` |
117-
| Template rendering | `scorer/model_judge.py` (`_render_template`), `scorer/grade_judge.py` (`render_prompt_template`) | Unify to single implementation in `scorer/grade_judge.py` |
116+
| `_run_coro_sync()` | ~~`scorer/model_judge.py:30-52`, `scorer/grade_judge.py:20-40`~~ **RESOLVED**: extracted to `scorer/_sync_bridge.py` |
117+
| Template rendering | ~~`scorer/model_judge.py` (`_render_template`), `scorer/grade_judge.py` (`render_prompt_template`)~~ **RESOLVED**: unified to `scorer/_prompt.py:render_judge_prompt()` |
118118
| `_tool_schemas()` | `benchmarks/bfcl/adapter.py`, `benchmarks/agentdojo/adapter.py` | Extract to `benchmarks/utils.py` |
119119

120120
### 1.6 Overly Large Files
@@ -342,16 +342,12 @@ Adapter tests should:
342342
- **Validation**: `pytest tests/test_emulated_tool.py -v`
343343
- **Public behavior change**: Type annotation only
344344

345-
#### P1.3: Remove benchmark-specific logic from runtime engine
345+
#### P1.3: ~~Remove benchmark-specific logic from runtime engine~~ RESOLVED
346346

347-
- **Problem**: `engine.py:961` checks `output.get("osworld_score")` — benchmark-specific in generic engine
347+
- **Problem**: ~~`engine.py:961` checks `output.get("osworld_score")` — benchmark-specific in generic engine~~
348+
- **Resolution**: Replaced with generic `_get_extra_payload_keys()` that reads key names from benchmark registry at runtime. No benchmark-specific key names hardcoded in engine.
348349
- **Files**: `snowl/runtime/engine.py`
349-
- **Why it matters**: Every benchmark shouldn't need engine modifications
350-
- **Violates core/adapters?**: Yes — runtime depends on benchmark details
351-
- **Risk level**: Medium
352-
- **Strategy**: Move OSWorld score extraction to OSWorld's scorer; use generic payload passthrough in engine
353-
- **Validation**: `pytest tests/test_osworld_benchmark.py tests/test_runtime_engine.py -v`
354-
- **Public behavior change**: OSWorld scores may appear in different payload location (document in CHANGELOG)
350+
- **Remaining concern**: Engine still does deferred import of benchmark registry (MEDIUM coupling noted in Part 1.4)
355351

356352
#### P1.4: Remove benchmark-specific container providers from runtime
357353

@@ -364,28 +360,21 @@ Adapter tests should:
364360
- **Validation**: `pytest tests/test_container_runtime_providers.py -v`
365361
- **Public behavior change**: Container provider registration becomes explicit
366362

367-
#### P1.5: Move AgentDojo tool implementations out of shared tools/
363+
#### P1.5: ~~Move AgentDojo tool implementations out of shared tools/~~ RESOLVED
368364

369-
- **Problem**: `stateful_executor.py` contains AgentDojo banking/travel tools
370-
- **Files**: `snowl/tools/stateful_executor.py`, `snowl/benchmarks/agentdojo/`
371-
- **Why it matters**: Adapter-specific code in shared infrastructure package
372-
- **Violates core/adapters?**: Yes
373-
- **Risk level**: Low
374-
- **Strategy**: Move tool implementations to `snowl/benchmarks/agentdojo/tools.py`; `stateful_executor.py` keeps only the generic sentinel pattern
375-
- **Validation**: `pytest tests/test_tool_middleware.py -v`
376-
- **Public behavior change**: Import path changes for AgentDojo tool implementations
365+
- **Problem**: ~~`stateful_executor.py` contains AgentDojo banking/travel tools~~
366+
- **Resolution**: Moved to `snowl/benchmarks/agentdojo/tools.py`. `stateful_executor.py` retains backward-compatible `__getattr__` re-exports.
367+
- **Files**: `snowl/tools/stateful_executor.py`, `snowl/benchmarks/agentdojo/tools.py`
377368

378369
### P2 — Good Cleanup
379370

380-
#### P2.1: Extract duplicated `_run_coro_sync()` to shared utility
371+
#### P2.1: ~~Extract duplicated `_run_coro_sync()` to shared utility~~ RESOLVED
381372

382-
- **Files**: `snowl/scorer/model_judge.py`, `snowl/scorer/grade_judge.py`
383-
- **Strategy**: Create `snowl/scorer/_sync_bridge.py` with shared implementation
373+
- **Resolution**: Created `snowl/scorer/_sync_bridge.py` with shared `run_coro_sync()`. Both `model_judge.py` and `grade_judge.py` now import from it.
384374

385-
#### P2.2: Unify template rendering logic
375+
#### P2.2: ~~Unify template rendering logic~~ RESOLVED
386376

387-
- **Files**: `snowl/scorer/model_judge.py`, `snowl/scorer/grade_judge.py`
388-
- **Strategy**: Use `render_prompt_template` from `grade_judge.py` as the canonical implementation; have `model_judge.py` delegate to it
377+
- **Resolution**: Created `snowl/scorer/_prompt.py` with shared `render_judge_prompt()`. Both `model_judge.py` and `grade_judge.py` now import from it.
389378

390379
#### P2.3: Extract shared `_tool_schemas()` to benchmarks/utils
391380

0 commit comments

Comments
 (0)