CI + verify harness + agentkit niceties + CONTRIBUTING

#1 GitHub Actions CI (uv sync, pytest, validate all skills, selftests,
   README lint, deploy-config parse) + live CI badge in the README.
#2 tests/verify.sh (the no-model matrix CI runs) + Makefile targets
   (verify/smoke/evals/validate/cheatsheet/gifs) + tools/lint_readmes.py.
#3 agentkit, backward-compatible: ChatResult gains a .tier field;
   route.cascade() degrades gracefully when the strong tier has no key
   (keeps the cheap answer, records why) instead of crashing; evals --cases N
   writes benchmark.partial.json so a limited run can't clobber the committed
   full benchmark. +4 no-model route unit tests (23 pass).
#4 CONTRIBUTING.md — the house conventions as an add-your-own-starter guide.

Verified: tests/verify.sh ALL GREEN; model-matchmakah still +1.00 after the
route change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: holzerjm

.github/workflows/ci.yml

@@ -0,0 +1,49 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      - name: Sync dependencies
+        run: uv sync
+
+      # Mirrors tests/verify.sh — the no-model matrix. The model-dependent
+      # checks (each project answering, eval pass-rate deltas) need a local
+      # model and run on a laptop via `make smoke`, not in CI.
+      - name: Unit tests
+        run: uv run pytest tests/ -q
+
+      - name: Eval engine selftest
+        run: uv run python -m agentkit.evals --selftest
+
+      - name: Validate every skill
+        run: |
+          fail=0
+          for d in $(find projects -name SKILL.md -exec dirname {} \;); do
+            echo "validating $d"
+            uv run agentskills validate "$d" || fail=1
+          done
+          exit $fail
+
+      - name: Deterministic script selftests
+        run: |
+          uv run python projects/everyday/paypah-trail/skills/expense-report/scripts/parse_receipt.py --selftest
+          uv run python projects/fun/dungeon-mastah/skills/campaign-rules/scripts/roll.py --selftest
+
+      - name: README lint
+        run: uv run python tools/lint_readmes.py
+
+      - name: Deploy config parses
+        run: uv run python -c "import tomllib,pathlib; tomllib.loads(pathlib.Path('projects/chief-of-staff/chief-of-stuff/deploy/zeroclaw/config.toml').read_text())"

.gitignore

@@ -6,3 +6,4 @@ __pycache__/
 *.egg-info/
 dist/
 .pytest_cache/
+**/evals/benchmark.partial.json

CONTRIBUTING.md

@@ -0,0 +1,86 @@
+# Contributing — add your own starter
+
+This repo is a **GitHub template** (click *Use this template*) and a set of
+worked examples that all follow one shape. Adding your own starter — for Build
+Day or after — means copying that shape. Here it is.
+
+## Anatomy of a project
+
+```
+projects/<theme>/<name>/
+├── agent.py            # the entry point + the run() contract
+├── skills/<skill>/SKILL.md   # one validated skill (+ optional scripts/ references/ assets/)
+├── evals/evals.json    # the house eval method (with vs without skill)
+├── fixtures/           # offline fallback for every networked tool
+└── docs/MODEL_SELECTION.md + docs/ADLC.md
+```
+
+The gentlest reference is [`fridge-whisperer`](projects/everyday/fridge-whisperer/)
+(no network, ~80 lines). Copy it and rename.
+
+## The one contract
+
+Every `agent.py` exposes:
+
+```python
+def run(task: str, skill=DEFAULT_SKILL): ...   # returns an AgentResult (or anything with .text)
+```
+
+`skill=None` is the **without-skill baseline** — the same code path, no second
+branch. That's what the eval runner toggles. Every `agent.py` also starts with
+the self-locating bootstrap so `python agent.py "..."` works from anywhere:
+
+```python
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(next(p for p in HERE.parents if (p / "agentkit").is_dir())))
+from agentkit import run_agent, tool
+```
+
+## The harness (`agentkit/`)
+
+```python
+from agentkit import run_agent, tool, chat, load_skill, cascade
+
+@tool                       # schema comes from the signature + docstring (the docstring is a prompt)
+def get_thing(query: str):
+    """What it does, and WHEN to use it."""
+    return f"sentence-shaped result for {query}"   # small models parse sentences better than bare values
+
+r = run_agent("the task", tools=[get_thing], skill=DEFAULT_SKILL)   # the loop
+r.text, r.cost_usd, r.usage, r.tool_calls                            # everything measured
+```
+
+`chat()` for one-shot calls, `cascade()` for cheap-first routing, `run_agent()`
+for the tool loop. Provider is one env var (`MODEL_PROVIDER`); see
+[docs/SETUP.md](docs/SETUP.md).
+
+## House rules (these earn rubric points)
+
+- **Zero/low-arg tools, sentence-shaped returns.** A 3B local model is the
+  default target; keep tools simple.
+- **The fixture rule.** Every networked tool checks `AGENT_OFFLINE` and falls
+  back to `fixtures/` on *any* exception, with a one-line notice to stderr.
+  Evals always run offline so numbers are reproducible.
+- **System prompt = role only.** All format/rules live in the SKILL.md, never
+  in `agent.py`.
+- **Skill anatomy:** `name` == directory name; a trigger-rich `description`
+  (what + when); body sections `## Defaults` / `## Workflow` / `## Gotchas` /
+  `## Output template`. Validate: `uv run agentskills validate skills/<name>`.
+- **Assert outcomes, not rituals.** Eval the brief's *content*, not a magic
+  phrase the model happens to print.
+- **Budget:** ~300 lines of code per project (the flagship gets ~500). Readable
+  beats clever.
+
+## Before you push
+
+```bash
+make verify     # the no-model matrix (tests, validate, selftests, lint) — also what CI runs
+make smoke      # every project answers offline on your local model
+make evals      # full eval pass-rate deltas (needs a local model; slow)
+```
+
+CI runs `make verify` on every push and PR. Keep your skill's eval delta ≥ 0,
+fill `docs/MODEL_SELECTION.md` with the numbers you measured, and you've cleared
+the bar. Then add a row to the picker table in the top-level [README](README.md).
+
+By contributing you agree your work is MIT-licensed, like the rest of the repo.

Makefile

@@ -0,0 +1,27 @@
+.PHONY: help verify smoke evals validate cheatsheet gifs
+.DEFAULT_GOAL := help
+
+help:  ## List targets
+	@grep -E '^[a-z-]+:.*##' $(MAKEFILE_LIST) | sed -E 's/:.*## / — /' | sort
+
+verify:  ## No-model verification matrix (what CI runs)
+	@bash tests/verify.sh
+
+smoke:  ## Run every project offline on the local model
+	@bash tests/smoke.sh
+
+evals:  ## Run all project evals (needs local model; slow)
+	@for d in projects/*/*/; do \
+		[ -f "$$d/evals/evals.json" ] && echo "── $$d" && uv run python -m agentkit.evals "$$d"; \
+	done
+
+validate:  ## Validate every skill against the agentskills spec
+	@for d in $$(find projects -name SKILL.md -exec dirname {} \;); do \
+		uv run agentskills validate "$$d"; \
+	done
+
+cheatsheet:  ## Rebuild docs/cheat-sheet.pdf
+	@uv run --with reportlab python tools/make_cheatsheet.py
+
+gifs:  ## Re-record the README terminal GIFs (needs vhs + ffmpeg)
+	@vhs tools/dungeon-mastah.tape && vhs tools/chief-of-stuff.tape

README.md

@@ -4,6 +4,7 @@
 
 # AgentDay-Example
 
+[![CI](https://github.com/holzerjm/AgentDay-Example/actions/workflows/ci.yml/badge.svg)](https://github.com/holzerjm/AgentDay-Example/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-3DA639)](LICENSE)
 ![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white)
 ![TOA Agent Build Day](https://img.shields.io/badge/TOA%20Agent%20Build%20Day-Jun%2027%2C%202026-CC0000)
@@ -111,7 +112,8 @@ before the event; fork them during it.
 [Lane 2 playbook (fork BriefingClaw)](docs/LANE2-PLAYBOOK.md) ·
 [Ten more project ideas](docs/ideas-appendix.md) ·
 [Printable cheat sheet (PDF)](docs/cheat-sheet.pdf) ·
-[Templates](templates/)
+[Templates](templates/) ·
+[Contributing — add your own starter](CONTRIBUTING.md)
 
 Reference architecture for the event:
 [BriefingClaw](https://github.com/holzerjm/GACEP-Spring-2026-demo) — the

agentkit/evals.py

@@ -204,8 +204,12 @@ def run_evals(
                 bench["modes"]["with_skill"]["pass_rate"]
                 - bench["modes"]["without_skill"]["pass_rate"], 2)
         }
-    (project_dir / "evals" / "benchmark.json").write_text(json.dumps(bench, indent=2), encoding="utf-8")
-    print(f"\nwrote {project_dir / 'evals' / 'benchmark.json'}")
+    # A limited run (--cases N) must not clobber the committed full benchmark.
+    partial = limit is not None and limit < len(spec["cases"])
+    bench["partial"] = partial
+    out = project_dir / "evals" / ("benchmark.partial.json" if partial else "benchmark.json")
+    out.write_text(json.dumps(bench, indent=2), encoding="utf-8")
+    print(f"\nwrote {out}" + ("  (partial — committed benchmark.json left intact)" if partial else ""))
     for mode, s in bench["modes"].items():
         print(f"  {mode:>14}: {s['passed']}/{s['total']} pass  (${s['cost_usd']}, avg {s['latency_s_avg']}s)")
     if "delta" in bench:

agentkit/llm.py

@@ -89,6 +89,7 @@ class ChatResult(NamedTuple):
     usage: dict  # {"prompt_tokens": int, "completion_tokens": int}
     latency_s: float
     cost_usd: float
+    tier: str = "default"  # which tier served this call — build routing tables for free
 
 
 def _load_dotenv() -> None:
@@ -186,6 +187,7 @@ def chat_raw(
         usage=usage,
         latency_s=latency,
         cost_usd=estimate_cost(handle.model, usage),
+        tier=handle.tier,
     )
 
 

agentkit/route.py

@@ -58,10 +58,18 @@ def cascade(
     first = llm.chat(task, system=system, provider=cheap[0], tier=cheap[1])
     confidence, grade = self_grade(task, first.text, provider=cheap[0])
     attempts = [first, grade]
-    escalated = confidence < threshold
-    if escalated:
-        final = llm.chat(task, system=system, provider=strong[0], tier=strong[1])
-        attempts.append(final)
+    escalated, note = False, "confidence above threshold — stayed cheap"
+    if confidence < threshold:
+        try:
+            final = llm.chat(task, system=system, provider=strong[0], tier=strong[1])
+            attempts.append(final)
+            escalated, note = True, "escalated to strong tier"
+        except Exception as e:  # noqa: BLE001
+            # Graceful degradation: a missing strong-tier key (or an
+            # unreachable endpoint) keeps the cheap answer instead of
+            # crashing the run — the table_row note records what happened.
+            note = (f"wanted to escalate ({confidence:.2f} < {threshold}) but "
+                    f"strong tier unavailable: {type(e).__name__}")
     text = attempts[-1].text if escalated else first.text
     cost = round(sum(a.cost_usd for a in attempts), 6)
     return RouteResult(
@@ -76,5 +84,6 @@ def cascade(
             "confidence": round(confidence, 2),
             "cost_usd": cost,
             "latency_s": round(sum(a.latency_s for a in attempts), 2),
+            "note": note,
         },
     )

tests/smoke.sh

@@ -105,3 +105,49 @@ def test_script_assertion(tmp_path):
     (tmp_path / "ck.py").write_text("import sys,json; c=json.loads(sys.argv[1]); sys.exit(0 if 'Report' in sys.stdin.read() and c['id']=='t' else 1)\n")
     got = check(OUT, {"type": "script", "path": "ck.py"}, project_dir=tmp_path, case={"id": "t"})
     assert got.ok, got.evidence
+
+
+# ── routing (no live model — monkeypatch llm.chat) ─────────────────────────
+
+from agentkit import ChatResult, route  # noqa: E402
+
+
+def _cr(text):
+    return ChatResult(text=text, model="stub", provider="local",
+                      usage={"prompt_tokens": 1, "completion_tokens": 1}, latency_s=0.0, cost_usd=0.0)
+
+
+def test_chatresult_has_tier_default():
+    assert _cr("x").tier == "default"
+
+
+def test_cascade_stays_cheap_when_confident(monkeypatch):
+    seq = iter([_cr("the cheap answer"), _cr('{"confidence": 0.95}')])
+    monkeypatch.setattr(route.llm, "chat", lambda *a, **k: next(seq))
+    r = route.cascade("2+2?", threshold=0.7)
+    assert not r.escalated and r.text == "the cheap answer" and r.confidence >= 0.7
+
+
+def test_cascade_escalates_when_unsure(monkeypatch):
+    seq = iter([_cr("weak"), _cr('{"confidence": 0.2}'), _cr("the strong answer")])
+    monkeypatch.setattr(route.llm, "chat", lambda *a, **k: next(seq))
+    r = route.cascade("hard", threshold=0.7)
+    assert r.escalated and r.text == "the strong answer"
+
+
+def test_cascade_degrades_gracefully_without_strong_key(monkeypatch):
+    calls = {"n": 0}
+
+    def fake(*a, **k):
+        calls["n"] += 1
+        if calls["n"] == 1:
+            return _cr("local answer")
+        if calls["n"] == 2:
+            return _cr('{"confidence": 0.1}')
+        raise RuntimeError("ANTHROPIC_API_KEY is not set")  # no strong-tier key
+
+    monkeypatch.setattr(route.llm, "chat", fake)
+    r = route.cascade("hard", threshold=0.7)
+    assert not r.escalated  # never crashes
+    assert r.text == "local answer"  # keeps the cheap answer
+    assert "unavailable" in r.table_row["note"].lower()