Skip to content

DESIGN_NOTES.md

Latest commit

 

History

History
149 lines (109 loc) · 6.3 KB

DESIGN_NOTES.md

File metadata and controls

149 lines (109 loc) · 6.3 KB
Raw

Eval Harness Generalization — Design Notes

What Was Built

The tax-eval-harness evaluation framework was generalized from a tax-specific implementation to a skill-agnostic evaluation harness. Any skill can now ship evals as a first-class part of its package.

New Files Created

File Purpose
src/tax_eval/eval_defs.py EvalDefinition dataclass, YAML loading, auto-discovery
src/tax_eval/eval_builder.py Converts definitions → SDK hooks + post-run evals
src/tax_eval/skill_agent.py Generic SkillAgent + BatchSkillRunner + CLI
.claude/skills/tax-analyzer/evals.yaml Tax evals migrated to declarative YAML
.claude/skills/tax-analyzer/evals/pii_check.py Example function eval

Modified Files

File Changes
src/tax_eval/skill_loader.py Added evals_dir, evals_yaml_path, has_evals, build_generic_prompt()
src/tax_eval/__init__.py Added new exports
pyproject.toml Added pyyaml dep, skill-eval CLI entry point

Backward Compatibility

All existing code untouched: eval_hooks.py, tax_agent.py, run_example.py. Legacy API still works.

Five Eval Modes

Mode Trigger Field Default Event Default Layer
Descriptive prompt stop semantic
Script script post_run deterministic
Function function pre_tool (if tool set) / post_run deterministic
regex_deny check: regex_deny pre_tool deterministic
path_restrict check: path_restrict pre_tool deterministic

Smart defaults minimize YAML boilerplate. Explicit values always override defaults.

Eval Package Structure

.claude/skills/any-skill/
├── SKILL.md
├── evals.yaml          # Declarative eval definitions
├── evals/              # Auto-discovered scripts/functions
│   ├── validate_output.py   # Must export evaluate(context) -> (bool, str)
│   └── check_format.sh      # Exit 0 = pass, non-zero = fail
├── references/
└── scripts/

Auto-discovery: Any .py with evaluate() or .sh in evals/ is found automatically, even without evals.yaml. YAML definitions take precedence over auto-discovered ones with the same name.

Multi-Skill Orchestration — Open Design Questions

Current State: 1 Skill = 1 Agent

The current design is 1:1 — one SkillAgent runs one skill with its evals. Hooks are registered globally on the SDK client.

The Problem

A plugin or orchestrating agent may need multiple skills simultaneously (e.g., a financial-analysis plugin using tax-analyzer + budget-planner + report-generator). Each skill ships its own evals, so:

  • Evals must be scoped to the skill that defined them (tax write-restriction shouldn't block the report generator)
  • Evals must activate/deactivate as the agent switches between skills
  • Hooks need skill context (which skill is currently executing)

What the Claude SDK Provides

Explored the SDK internals and found:

  • No SkillDefinition type — skills exist only at the Cowork/Claude Code app level as filesystem conventions (SKILL.md + manifest.json)
  • AgentDefinition — has description, prompt, tools, model but no hooks field
  • SdkPluginConfig — just type + path, a pointer not a definition
  • Global hook registry — no per-skill isolation
  • SubagentStart / SubagentStop hooks — lifecycle boundaries that could serve as scoping anchors
  • session_id in hooks — available but not skill-scoped

Proposed Architecture (Not Yet Built)

SkillDefinition — extends AgentDefinition semantics with eval/hook support:

@dataclass
class SkillDefinition:
    name: str
    agent: AgentDefinition          # prompt, tools, model
    eval_definitions: list[EvalDefinition]
    hooks: dict[str, list]          # skill-scoped hooks
    root_dir: Path

PluginRunner — multi-skill orchestrator:

class PluginRunner:
    skills: dict[str, SkillDefinition]
    active_skill: str | None

    # Uses SubagentStart/SubagentStop to scope evals:
    # - SubagentStart → activate that skill's hooks
    # - SubagentStop → deactivate, record results
    # - Global hooks (e.g., PII check) always active

Key design decisions needed:

  1. How does the orchestrating agent decide which skill to invoke? (explicit routing vs. LLM choice)
  2. Should skills share eval reports or have isolated ones?
  3. How to handle cross-skill evals (e.g., "final output incorporates all skills' work")?
  4. Should this be a wrapper on top of the SDK, or a PR to the SDK itself?

The Two-Layer Gap

┌─────────────────────────────────────────────┐
│ Cowork App / Claude Code                     │
│  Rich skill packages: SKILL.md, references,  │
│  scripts, evals, manifest.json               │
│  BUT: no programmatic representation         │
├─────────────────────────────────────────────┤
│ Claude Agent SDK (Python)                    │
│  AgentDefinition: prompt, tools, model       │
│  SdkPluginConfig: type, path                 │
│  NO: hooks, evals, references, scripts       │
└─────────────────────────────────────────────┘

Our SkillPackage + EvalDefinition pipeline is essentially constructing what a native SDK SkillDefinition would provide. The natural path forward is either:

  • Build it as a library layer on top of the SDK (what we can do now)
  • Propose it as an SDK primitive (requires Anthropic SDK team alignment)

Test Status

77 tests passing across all test files. Fixed a Python module caching issue where importlib.import_module returned stale cached modules across test fixtures with different tmp_path values — solved by using importlib.util.spec_from_file_location with unique module names.

uv run pytest -v                    # All 77 pass
uv run skill-eval inspect           # CLI works
uv run tax-eval inspect             # Legacy CLI works