cwm-execution-tracer

Mines runtime observables from SWE-bench Verified instances, builds a "predict the runtime behavior" benchmark of LLM tasks from them, evaluates models on that benchmark, and renders reports.

The published dl4c artifacts live as three HuggingFace datasets under JetBrains-Research: cwm-benchmarks-dl4c-environments (tracing inputs), cwm-benchmarks-dl4c-benchmark (435 prompt + ground-truth samples), and cwm-benchmarks-dl4c-traces (the raw trace observables). To reproduce those artifacts, start with reproduction/.

Reproducing the paper results

Two turn-key scripts, run from the repo root (see reproduction/README.md for details):

# 1. Re-mine the 435 traces and compare against the published dataset (needs Docker)
uv run python reproduction/reproduce_traces.py

# 2. Evaluate a model against the published benchmark (needs an API key)
uv run python reproduction/reproduce_evaluation.py --model claude-sonnet-4-6

Layout

execution_tracer/
├── tracers/                       # in-container instrumentation (injected per test)
│   ├── injectable_tracer.py       # sys.settrace / sys.monitoring: outcome +
│   │                              #   exception + line/function time & memory
│   ├── memory_test_profiler.py    # sys.setprofile + tracemalloc.reset_peak per call
│   ├── cprofile_test_profiler.py  # cProfile wrapper: per-method exclusive wall time
│   └── test_timer.py              # perf_counter timer, no profiler attached
│
├── legacy_swebench_harness/       # Docker-side orchestration (extends swebench harness)
│   ├── run_traced.py              # outcome + line-trace pass (one or two-sided)
│   ├── run_memprof.py             # memory-only pass
│   ├── run_cprofile.py            # method-time pass
│   └── run_walltime.py            # clean walltime pass
│
├── benchmark/
│   ├── builder.py                 # trace JSON + repo snapshot → BenchmarkSample
│   ├── samples.py                 # BenchmarkSample dataclass + on-disk layout
│   └── scoring.py                 # per-sample metrics + cross-sample aggregation
│
└── scripts/                       # CLI drivers + reporting
    ├── run_single.py              # mine one instance
    ├── run_batch_traced.py        # batch trace miner (+ memprof/cprofile/walltime)
    ├── build_benchmark.py         # traces + snapshots → benchmark samples
    ├── build_hf_datasets.py       # local artifacts → the three HF datasets
    ├── run_evaluation.py          # LLM evaluation runner (OpenAI / Anthropic /
    │                              #   OpenAI-compatible base_url for self-hosted)
    └── build_report.py            # eval results → self-contained HTML report

reproduction/                      # turn-key scripts to reproduce the dl4c artifacts
tests/                             # unit tests for the tracers + profilers + harness
data/                              # on-disk artifacts (traces, samples, eval_results)

Data flow

SWE-bench Verified
      │
      ▼  one Docker container per instance, gold-patched; inject one pass:
      │    • run_traced   → trace_output.json[.gz]     (outcome, lines)
      │    • run_memprof  → memprof_output.json        (peak, methods)
      │    • run_cprofile → cprofile_output.json       (method times)
      │    • run_walltime → walltime_output.json       (test wall time)
      ▼
trace_results/<inst>/  (+ memprof_/cprofile_/walltime_results/)
      │
      ▼  build_benchmark.py  (cross-pass ground-truth extraction)
benchmark_samples/<task>/*.json   (one prompt + GT per (instance, test, side))
      │
      ▼  run_evaluation.py  (per model)
eval_results/<model>/responses/*.json + summary.json
      │
      ▼  build_report.py
experiment_report.html

Each profiling sub-task is mined in its own pass so attached instrumentation never inflates the very numbers it measures.

Quick start (underlying pipeline)

uv sync

# 1. Mine one instance (smoke test)
uv run python -m execution_tracer.scripts.run_single \
    --instance_id pytest-dev__pytest-10356 --trace_level line

# 2. Mine a batch (dual-sided pre + post)
uv run python -m execution_tracer.scripts.run_batch_traced \
    --dual_trace --trace_level line --memory_tracking both --max_workers 4

# 3. Build benchmark samples
uv run python -m execution_tracer.scripts.build_benchmark \
    --trace_dir trace_results --out_dir benchmark_samples --context_strategy smart

# 4. Evaluate an LLM
uv run python -m execution_tracer.scripts.run_evaluation \
    --samples_dir benchmark_samples --out_dir eval_results \
    --model gpt-5-mini --workers 8

# 5. Build the HTML report
uv run python -m execution_tracer.scripts.build_report \
    --root eval_results --out experiment_report.html

Apple Silicon

SWE-bench images are published for linux/amd64. Export DOCKER_DEFAULT_PLATFORM=linux/amd64 before any docker pull or trace run; Docker Desktop's emulation handles the rest.

Tasks the benchmark evaluates

Each sample asks the model to predict, without executing the code, the runtime behavior of a single test:

Sub-task	Prediction	Metric
outcome	passed / failed (AssertionError) / error + exception + line	P/R/F1, exception exact-match
peak_bytes	Peak memory above entry baseline, bytes	log10 linear fit + log10 MAE
wall_ms	Total test wall-clock time, milliseconds	log10 linear fit + log10 MAE
hot_methods_time	Top-20 in-project functions by exclusive wall time	NDCG@5 + Recall@5
hot_methods_alloc	Top-20 in-project functions by exclusive allocation	NDCG@5 + Recall@5
hot_lines_time	Top-20 path/file.py:line strings by wall time	NDCG@5 + Recall@5
hot_lines_alloc	Top-20 path/file.py:line strings by allocation	NDCG@5 + Recall@5

All seven sub-tasks are emitted as a single combined JSON prompt + GT per sample; the model returns one JSON object scored against each independently.

Per-level measurement strategy (one pass per task):

	Test-level	Method-level	Line-level
Time	walltime pass: perf_counter, no profiler	cprofile pass: cProfile exclusive self-time per qualified function name	trace pass: deltas between consecutive sys.settrace line events
Memory	memprof pass: max of peak tracemalloc rise, peak RSS rise, largest per-method peak	memprof pass: max of summed per-call tracemalloc peaks and summed RSS deltas	trace pass: per-line tracemalloc + RSS deltas between line events

memprof calls tracemalloc.reset_peak() on every project-frame entry so per-call exclusive peaks are accurate, ratcheting the test-level high-water mark before each reset.

Tests

uv run python -m pytest tests/

Notable files: test_memprof.py (test-level peak ratchet), test_tracer.py / test_tracer_features.py (line + function events, gzip output, sysmonitoring backend), test_harness.py (Docker-less harness scaffolding).

Dependencies

• swebench — Docker harness, dataset loader, and instance specs
• datasets — HuggingFace datasets (SWE-bench Verified + the dl4c datasets)
• openai, anthropic — LLM clients (openai also drives OpenAI-compatible endpoints via --base_url)
• docker — container management