Learn harness engineering by building Claude Code from scratch, from a bare-bones agent loop to a swarm of cloud agents.
Open-source harness engineering course
by Decoding AI in collaboration with Modal, Opik (by Comet), and Kitaru (by ZenML).
📖 Read Lesson 1 (17 min) · 🎬 Watch Lesson 1 · 📚 See all 8 lessons
Try the finished agent first — 5 minutes, $0:
git clone https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-course.git cd building-a-coding-agent-from-scratch-course make install cp .env.example .env # set LLM API key uv run decodeThen type
/demo-and pick a demo. See what they do below. Full setup guide.
Type /demo- and the six demos are one keystroke away.
In LangChain's Terminal-Bench experiment, changing only the harness (with the same model) moved a coding agent from ~30th place into the top 5: the harness, not the model, is what makes a coding agent good.
agent = Agent(
build_model(settings.llm_provider), # gemini | openrouter | modal
deps_type=AgentDeps, # cwd, event sink, permission gate
output_type=[str, DeferredToolRequests], # final answer, or tools paused for approval
)
register_tools(agent) # read, edit, bash, grep, ...
async with agent.iter(prompt, message_history=history) as run:
async for node in run: # model request → tool calls → repeat
stream_events(node)That's the entire tool-calling agent. Everything else in this repo (the tools, skills, permission layer, sandbox, steering queue, memory, compaction, session recording & replay, remote execution, subagent fan-out, and evals) is the harness. That's what you're here to build.
A fresh session powered by Qwen3.6-35B hosted on Modal
We spent months under the hood of Claude Code (via its leaked source), OpenCode, Pi, and Aider, then distilled what we learned into 8 articles and 6 videos where you'll build decode, your own coding agent, from scratch. One headless core hooked to two modes: an interactive TUI and Modal serverless functions running N copies in parallel, fired by CLI, webhook, or cron.
Warning
Building a coding harness from scratch can make you dangerously good at building any other AI product, whether in finance, medicine, or e-commerce.
The harness architecture of the coding agent you will build during this course.
The finished agent ships with demo skills under .decode/skills/. Open the TUI, type /demo-, pick one, and watch the harness you're about to build do real work:
Implement the Skills Standard
Type /demo- and the six demos are one keystroke away.
Capable of Creating Games |
Fetching Data & Creating Dashboards |
Extracting Ontologies & Rendering Graphs |
|
And the infra that powers the agents.
Record & Replay for AI Agents |
Remote Sandboxing |
Powered by Open-Source Models |
Adding AI Evals & Observability |
- Design a coding agent harness from scratch (article 1 · video 1)
- Implement the coding agent loop as a headless harness (article 2 · video 1)
- Attach the headless harness to multiple interfaces: a terminal UI, a CLI, or even a remote runtime (article 6 · video 4)
- Execute the agent's tools within local Docker or remote Modal sandboxes (article 3 · video 2)
- Host open-source models as SGLang servers on Modal (article 2 · video 1)
- Implement guardrails by adding a permission layer (article 1 · video 1)
- Build essential context engineering techniques: memory, compaction, skills (article 4 · video 3)
- Hook up an LSP server for faster feedback loops (article 4 · video 3)
- Implement a configurable agents catalog: build, plan, code reviewer, and exploration agents (article 5)
- Deploy the headless harness on Modal, triggering remote agents via the CLI, a webhook, or a cron job (article 6 · video 4)
- Spawn parallel subagents via fan-out strategies (article 5)
- Add observability (article 2 · video 1)
- Design an eval harness for benchmarking the agent and checking for regressions (article 7 · video 5 soon)
- Organically grow your regression suite from failed agent traces (article 8 · video 6 soon)
- Reproduce agent failures and check for regressions when changing your prompts or models by replaying traces with Kitaru (article 8 · video 6 soon)
Plan mode, live: the agent breaks the Snake demo into a task list with the todo tool — [x] done, [~] in progress.
The code is written in Python, with the following frameworks and libraries:
- Agent Framework: Pydantic AI
- LLM Providers: Modal (open weights you serve yourself via SGLang), OpenRouter (open weights as a service), or Gemini (proprietary).
- Session Recording & Replays: Kitaru
- Observability & Evals: Opik
- Sandboxing: local Docker & remote Modal sandboxes
- Deploying: Modal as headless agents (fired by hand, by cron, or by webhook)
Otherwise, we build all the functionality from scratch to teach you the foundations that last, not frameworks that abstract away the hard parts.
For the full experience, go through the articles and videos that cover what the code can't. The why behind every decision.
- What the essential components of a coding agent are, and what is optional. (article 1 · video 1)
- Why we have a headless harness and two interface modes: TUI + Remote. (article 1 · video 1)
- Why we plugged in 9 tools, no more, no less. (article 2 · video 1)
- What guardrails are actually useful. (article 3 · video 2)
- Why compaction fires at ~80% of the window instead of at the limit. (article 4 · video 3)
- Why subagents are context engineering: scoped windows instead of one bloated context. (article 5)
- Why your agents should keep working after you close your laptop lid. (article 6 · video 4)
- Why you need both benchmarks and regression tests. (article 7 · video 5 soon)
- Why we record every run, and what a replay buys you that a re-run doesn't. (article 8 · video 6 soon)
| Lesson | Written Lesson | Video Lesson | Running the Code |
|---|---|---|---|
| 1 Building a Coding Agent From Scratch Sketch the full harness: a headless core, six modules, TUI and remote modes, and an evals layer. ⏱ 17-min read |
 Article 1 |
 Video 1 |
01_install_and_usage.md · 02_modal_endpoints.md |
| 2 The Bare-Bones Coding Agent Loop Your agent loops over 9 tools and 3 swappable LLM providers in a steerable TUI. ⏱ 25-min read |
 Article 2 |
01_install_and_usage.md · 02_modal_endpoints.md | |
| 3 From a Raw Shell to a Sandboxed Coding Agent Your agent's tools run in a Docker or Modal sandbox, never on your host. ⏱ 13-min read |
 Article 3 |
 Video 2 |
01_install_and_usage.md · 02_modal_endpoints.md · 03_sandboxing.md |
| 4 Context Engineering for Coding Agents Your agent loads memory, invokes skills, reads LSP diagnostics, and auto-compacts at 80%. ⏱ 15-min read |
 Article 4 |
 Video 3 |
01_install_and_usage.md · 02_modal_endpoints.md |
| 5 Subagents Are Context Engineering Your agent fans out parallel Explore subagents and switches between build, plan, and code-reviewer personas. ⏱ 14-min read |
 Article 5 |
- | 01_install_and_usage.md · 02_modal_endpoints.md |
| 6 Deploy a Headless Coding Agent Harness to Modal Your agent runs headless on Modal, fired by the CLI, a webhook, or a cron job, and ships branches. ⏱ 15-min read |
 Article 6 |
 Video 4 |
01_install_and_usage.md · 02_modal_endpoints.md · 03_sandboxing.md · 04_deploy.md |
| 7 AI Evals Foundations: Benchmarks, Regression and Online Score your agent on a 19-task benchmark and a regression suite in Opik. ⏱ 19-min read |
 Article 7 |
🎬 Video 5 — coming soon | 01_install_and_usage.md · 02_modal_endpoints.md · 05_evals.md |
| 8 AI Evals on Steroids via Replays Record runs in Kitaru, cluster failures into cohorts, and replay them to verify fixes. ⏱ 16-min read |
 Article 8 |
🎬 Video 6 — coming soon | 01_install_and_usage.md · 02_modal_endpoints.md · 06_evals_replays.md |
📖 Read all the articles on Substack · 🎬 Watch all the videos on YouTube
Join 45k+ engineers subscribed to the Decoding AI Magazine to learn to build coding agents from scratch.
Engineers who learn by building. You finish with a working coding agent that teaches you harness engineering patterns to steal for your own agentic applications.
Best for ML/AI engineers who want to level up their craft and for software engineers and data scientists who want to transition into building agentic systems from scratch.
| Category | Requirements |
|---|---|
| Skills | - Python (Intermediate) - LLMs & agents (Beginner) |
| Hardware | Any modern machine will do. No GPU required, as we run all the LLMs in the cloud. |
| Level | Intermediate (but with a little sweat and patience, anyone can do it) |
| Time | ~4–8 hours for the whole course — 4 if you read and watch, 6–8 if you run everything. |
Running the code costs $0 if you stick to free tiers:
| Service | Cost |
|---|---|
| Gemini API (default provider — easy setup, but limited API requests) | free tier (Google AI Studio) |
| Modal (recommended provider, remote sandbox, remote agents) | $30 free credits — enough to run the course |
| OpenRouter (alternative provider) | $0 on :free models (optional $10 credit raises the daily cap) |
| Opik (tracing + evals) | free tier |
| Kitaru (recording + replays) | free — the OSS server runs on your laptop (make kitaru-local) |
Only reading? Everything's free!
As an open-source course, it is entirely self-paced and based on this repository, plus the attached lessons that walk you through the code. No paywall. No platform.
Read the lessons on the Decoding AI Magazine, watch the videos on the Decoding AI Channel, run the code on your own machine, break it, fix it, and learn from the process.
One Python package; each module maps to one part of the architecture:
.
├── docs/
│ ├── adr/ # Architecture Decision Records — the "why" of every choice
│ └── glossary.md # one canonical name per concept
├── evals/ # benchmark tasks + regression cases + the Opik harness
├── scripts/ # operator scripts: Kitaru bootstrap + eval table generation
├── tests/{unit,integration}/ # mirrors src/ 1:1; milestone capstones prove each milestone
└── src/decode/
├── cli.py # Click entrypoint → launches the TUI
├── tui/ # input: prompt_toolkit · output: Rich
├── harness/ # message queue + priority gate around the loop
├── agent/ # the Pydantic-AI ReAct loop (LLM ⇄ tools)
├── agents/ # agents catalog: Build / Plan / Code-Reviewer + Explore subagent
├── tools/ # file I/O, bash, web, todo, skills dispatch, LSP, ask_user
├── permissions/ # allow/ask/deny · modes · settings.json
├── sandbox/ # bash + file tools seam: none (host) / docker / modal
├── services/lsp/ # hand-rolled stdio LSP client (ty)
├── runtime/ # plain headless decode run + the Recording Seam
├── remote/ # decode remote deploy|run|attempts|logs — the Modal Headless App (CLI, webhook, cron)
├── context/ # compaction + conversation log (JSONL)
├── memory/ # AGENTS.md / MEMORY.md loading + write-back
├── observability/ # Opik tracing
└── config/, entities/ # settings singleton · shared models
The guides live under running_the_code/. Follow them in order; each ends with a link to the next:
| Guide | What's inside |
|---|---|
| 00_troubleshooting.md | Every known failure, and its fix |
| 01_install_and_usage.md | Start here: install, one key, first session |
| 02_modal_endpoints.md | Serving open models on Modal |
| 03_sandboxing.md | Docker (local) / Modal (remote) sandboxing + the sandbox git token |
| 04_deploy.md | The headless harness on Modal — CLI, webhook, cron |
| 05_evals.md | Benchmarks and regression cases on Opik |
| 06_evals_replays.md | Kitaru on your laptop: record, replay & the full evals loop |
Special thanks to Modal, Opik (by Comet), and Kitaru (by ZenML) for sponsoring this open-source course and keeping it free!
Opik and Kitaru are open source. Consider starring their repositories: Opik on GitHub · Kitaru on GitHub.
Open a GitHub issue for course questions, setup trouble, or concept clarifications. Known gotchas are documented in the running_the_code/ guides.
Do I need a paid API key?
No. The default Gemini provider has a free tier, OpenRouter routes across :free models, and Modal gives $30 in credits — see Cost Structure.
Why Python and not TypeScript or Go? Accessibility: our audience knows Python. The course focuses on the design decisions, which transfer to any language.
Why build from scratch instead of extending Pi, DeepAgents, or an existing harness? Because adding custom logic to an existing harness is the easy part. Knowing what to add requires understanding the internals. Those fundamentals are what still make AI engineers valuable. Build a coding agent once and you're equipped to build a custom agent for any use case.
Found a bug and know the fix? Fork, fix, run make ci (no API key needed), and open a pull request. Future readers will thank you 🤗
Paul Iusztin |
Senior AI Engineer, Educator & Founder of Decoding AI. Author of the best-selling LLM Engineer's Handbook. |
Join 45k+ engineers subscribed to the Decoding AI Magazine to learn to build coding agents from scratch.
If you found this course useful, consider starring the repository so others can find it too.
Released under Apache-2.0. Clone, fork, and build on it; keep the LICENSE and credit this repo.