Skip to content

DEMO.md

Latest commit

 

History

History
253 lines (184 loc) · 6.95 KB

DEMO.md

File metadata and controls

253 lines (184 loc) · 6.95 KB
Raw

Falsification Engine — 3-Minute Demo

Hands-off option

Run ./demo.sh for an auto-narrated walkthrough that runs the full PASS → tamper → FAIL → guard-block story end-to-end and restores the fixture when it's done. Everything below is the manual version if you want to pace the demo yourself.

What you'll see

Pre-registration + CI for AI-agent claims: lock a hypothesis, run the experiment, get a deterministic PASS/FAIL verdict, and let guard block any later text that contradicts the result.

Prerequisites

  • Python 3.11+
  • pip install pyyaml (inside a venv if your system blocks global pip)
  • Clone this repo, cd into it.

Step 1 — Lock a hypothesis (calibration sample)

mkdir -p .falsify/calibration
cp examples/calibration_sample/spec.yaml .falsify/calibration/spec.yaml
python3 falsify.py lock calibration

Expected:

✓ Locked calibration @ 97c1b9bc4a8c
  claim: brier_score below 0.25

The spec's canonical YAML is SHA-256 hashed and stored in .falsify/calibration/spec.lock.json. Any semantic edit to spec.yaml after this invalidates the lock and blocks run with exit 3.

Step 2 — Run the experiment

python3 falsify.py run calibration

Expected:

✓ Run 20260421T184500_000000Z (0.05s)

experiment.command executed with a 5-minute timeout; stdout, stderr, a copy of the lock, and a metadata file (host, python version, duration, return code) are written under .falsify/calibration/runs/<timestamp>/. The run is reproducible from those artifacts alone.

Step 3 — Get the verdict

python3 falsify.py verdict calibration

Expected:

Verdict: PASS
  observed brier_score = 0.214265
  threshold: below 0.25

The metric_fn (examples.calibration_sample.metric:brier_score) was imported and checked against the locked criterion. PASS → exit 0, FAIL → exit 10. verdict.json is written so downstream tools (list, guard, the claim-auditor subagent) can read it without re-running the experiment.

Step 4 — Dashboard (list)

python3 falsify.py list

Expected:

NAME  LOCKED        LAST RUN                 VERDICT  OBSERVED
calibration  97c1b9bc4a8c  20260421T184500_000000Z  PASS     0.214265

For CI or scripts, python3 falsify.py list --json emits the same data as a JSON array.

Step 5 — Guard (CI) and the commit-msg hook

# Scan mode: non-zero if any claim is FAIL or STALE (> spec drift).
python3 falsify.py guard

# Text mode: block affirmative language that contradicts a logged verdict.
python3 falsify.py guard "we've proven the model predictions is well-calibrated"

Because the calibration verdict is PASS, the text call exits 0 — the affirmative language matches a passing claim. If the latest verdict had been FAIL or INCONCLUSIVE, the same sentence would have exited 11 with a BLOCKED: breakdown on stderr.

Install the hook so every commit gets the same check automatically:

python3 falsify.py hook install

Installs the commit-msg guard; backs up any pre-existing hook. Now a commit message that asserts a falsified claim is rejected before it enters history — the same mechanism CI uses on every push.

Bonus — health check

python3 falsify.py doctor

End-to-end diagnostic — what's OK, what needs attention.

Bonus — detect a tampered spec

sed -i 's/threshold: 0.25/threshold: 0.20/' .falsify/calibration/spec.yaml
python3 falsify.py diff calibration

diff proves the spec was tampered with — you see exactly what changed before re-locking. Exit 3 on any drift; --force required to relock.

Bonus — see all verdicts at once

python3 falsify.py stats
python3 falsify.py stats --json
python3 falsify.py stats --html > dashboard.html && open dashboard.html

One-shot dashboard across every locked spec in .falsify/ — name, state, metric, observed value, threshold, sample size, age in days. Aggregate counts on the last line. The JSON form is CI-friendly; the --html form is a self-contained dark-mode-aware page for demos and quick browser review (no external assets).

Bonus — audit trail

python3 falsify.py export > audit.jsonl
wc -l audit.jsonl

One JSON line per event (lock / verdict; add --include-runs for run records too). Append-only. Deterministic — same .falsify/ contents produce byte-identical output. Share with a peer to reproduce your verdicts from nothing but this file plus the original spec.yaml.

Bonus — project-wide honesty score

python3 falsify.py score
python3 falsify.py score --format shields

A single number across every claim in .falsify/. The shields form is a JSON endpoint you can drop straight into your README for a live badge that turns red when claims start failing.

Bonus — spotting drift

python3 falsify.py trend calibration

ASCII sparkline of the metric across every recorded run, with the threshold line marked and the trajectory classified (improving / degrading / flat / mixed). Useful for catching a slow regression before it tips a claim into FAIL.

Bonus — explaining a verdict

python3 falsify.py why calibration

Output (STALE case, right after someone edits the spec without re-locking):

claim: calibration
state: STALE
reasoning: the spec has been edited (sha256:1038219d75a8) but no run
  exists against this hash. Last run was against sha256:164f619d4860.
locked: yes (sha256:164f619d4860, 2h ago)
last run: 2026-04-22T02:10:17+00:00 (2h ago)
next action: `falsify run <name>` to produce a fresh verdict against
  the current spec.

Verdict tells you PASS/FAIL; why tells you why — and, more importantly, what the next honest action is. Always exits 0.

Bonus — proving reproducibility

python3 falsify.py replay <run-id>

Runs the metric again against the same dataset. Exits 0 only if the number matches bit-for-bit (or within --tolerance) — the claim isn't just locked, it's reproducible.

Bonus — verify a peer's audit trail

python3 falsify.py verify audit.jsonl

Walks the JSONL and checks that each verdict's locked_hash chains back to a preceding lock's canonical_hash, that timestamps are monotonic per spec, and that no records were reordered. If anyone tampered with the file after export, the hash chain breaks and verify refuses it with exit 10.

Self-measuring

python3 falsify.py bench --runs 3

Spawns each of init, --help, list, stats, score under a fresh temp directory and prints per-command latency (min / median / p95 / max / mean / stddev). Handy as a "did we regress startup time?" sanity check before a release.

See also

  • tests/smoke_test.sh — full pipeline in one bash driver.
  • .github/workflows/falsify.yml — unittest + smoke + calibration e2e in CI.
  • .claude/skills/ — falsify (orchestrator), hypothesis-author (NL → locked spec), claim-audit (fast text audit + subagent).
  • .claude/agents/claim-auditor.md — semantic PR/release auditor.