Skip to content

README.md

Latest commit

 

History

History
85 lines (62 loc) · 2.43 KB

README.md

File metadata and controls

85 lines (62 loc) · 2.43 KB
Raw

Evaluation

中文说明

This directory keeps the reusable evaluation pipeline and the supplementary evaluation examples.

Environment

From the repository root, sync the uv workspace first:

uv sync --all-packages

This installs both the evaluation environment and the workspace-linked circle dependency into the shared repo-root .venv.

Stable Entry Points

  • uv run python evaluation/cli.py case ...
    • single-case orchestrator over prepare / gt / circle / baseline / eval
  • uv run python evaluation/cli.py single-eval ...
    • evaluation-only on an already prepared case root
  • uv run python evaluation/cli.py batch-circle ...
    • dataset-scale CiRCLE runner
  • uv run python evaluation/cli.py batch-baseline ...
    • dataset-scale baseline runner
  • uv run python evaluation/cli.py batch-eval ...
    • dataset-scale evaluation/report runner

Main Pieces

  • src/: single-case discovery, loading, evaluation, and report generation
  • script/: batch entrypoints and wrapper scripts
  • paper-cases/: supplementary evaluation cases
  • dataset/: dataset layout, minicase index, and smoke/report pointers
  • docs/: detailed evaluation notes and public limitations

Public Example

The repository-wide public example lives at ../example/.

Fast smoke test:

bash example/run.sh report

Direct invocation:

uv run python evaluation/cli.py case --case-root example --steps eval

Small explicit-case dataset demo:

uv run python evaluation/cli.py batch-eval \
  --case-root example \
  --case-root evaluation/paper-cases/Motivating-Example \
  --case-root evaluation/paper-cases/More-Cases/nested_structure \
  --dataset-family osprey \
  --report-group osprey_mini \
  --report-root example/dataset_eval/osprey_mini \
  --tools hyres,resym,retypd,trex,typeforge,circle \
  --circle-variants no_llm_full

For canonical Osprey runs, batch-circle and batch-baseline create evaluation/result/osprey/process/<binary>/ on demand, and batch-eval consumes that shared result tree.

Public Metric Scope

The shipped public relation tables currently cover these field-level relation shapes:

  1. PointerToStruct
  2. NestedStruct
  3. ArrayInStruct

Root-level array shapes stay outside the public relation tables. See docs/limitations.md for the current boundary.