SuperQode Harness Optimization Demo

A runnable demo of self-improving harness optimization with SuperQode.

This repository shows how to:

• mine structured failures from a harness eval result
• persist those failures into a repo-local logbook
• export a bounded harness-improvement project
• audit candidate harness changes before applying them
• preserve both accepted and rejected harness attempts in a candidate ledger
• inspect held-in and held-out eval splits

The demo is deterministic and does not require an API key. It uses a small fixture eval result and runs with --allow-ungated for the candidate-audit step so you can try the loop offline.

Background

This demo is inspired by Lilian Weng's post, Harness Engineering for Self-Improvement. It turns the harness self-improvement loop into a concrete SuperQode workflow: mine failures, persist logbook memory, export bounded improvement evidence, audit candidates, and preserve accepted/rejected attempts.

Quick Start

git clone https://github.com/SuperagenticAI/superqode-harness-optimization-demo.git
cd superqode-harness-optimization-demo
./demo/run-demo.sh

By default the script uses the published release:

uvx superqode==0.2.8

To run against a local checkout of SuperQode instead:

SUPERQODE_CMD="uv run superqode" ./demo/run-demo.sh

The script writes generated artifacts into _run/.

What You Should See

The demo creates:

_run/failures.json
_run/.superqode/self-improve/logbook/failure_patterns.yaml
_run/improve-demo/trace-evidence.md
_run/.superqode/self-improve/candidates.jsonl

The final ledger should show one accepted candidate and one rejected candidate:

candidate       decision  surfaces  violations
cand_...        accepted  context   -
cand_...        rejected  ...       protected_surface_change,...

Why This Matters

SuperQode is not trying to be your daily coding agent. It is a harness engineering and optimization framework for the agents you already use. The harness is the layer that controls context, tools, workflow, permissions, evals, memory, and candidate acceptance.

This demo shows that harness improvement can be:

• file-backed
• auditable
• regression-aware
• protected by permission/check gates
• repeatable by other developers

Files

File	Purpose
demo/harness.yaml	Baseline SuperQode harness with optimization policy
demo/tasks.yaml	Held-in and held-out eval task split
demo/eval-failed.json	Fixture failed eval result used for weakness mining
demo/candidate-safe.yaml	Candidate that makes a narrow context edit
demo/candidate-unsafe.yaml	Candidate that widens permissions and disables checks
demo/run-demo.sh	End-to-end runnable demo
demo/expected-output.md	Representative output from the demo
artifacts/	Example generated artifacts

Production Notes

The offline demo uses --allow-ungated because it does not call a live model. In a production harness optimization workflow, run a real held-out eval and audit with --require-heldout.

Example:

uvx superqode==0.2.8 harness eval \
  --spec harness.yaml \
  --variant candidate.yaml \
  --tasks tasks.yaml \
  --split held-out \
  --live \
  --json > heldout.json

uvx superqode==0.2.8 harness audit-candidate \
  --base harness.yaml \
  --candidate candidate.yaml \
  --tasks tasks.yaml \
  --eval-result heldout.json \
  --require-heldout \
  --record