container.md

The lifecycle of a task container

Every task in panopticon runs in its own container — a throwaway Docker container with a claude agent inside it, working on a private clone of the repo. This doc explains that container's life from the operator's chair: how one comes up, the statuses you watch on the dashboard, what each means, and how a container recovers or is torn down. It's about observable behaviour from the operator's side.

For the token a container authenticates with, see auth; for the image it's built from, layers; for a task's branch/clone and provisioning, tasks.

One task, one container, one session

A task's work happens in exactly one place: its container. Nothing else in panopticon runs an LLM — the task service (control plane) and the session service (runner) are deterministic host processes that never call claude. That's the determinism invariant: the container is the only LLM-bearing component, so everything the agent does is scoped to it.

Concretely, each running task has:

• a container named panopticon-<task-id>, spawned by the session service on the host's Docker daemon;
• a tmux session (panopticon-<task-id>) whose pane runs the agent — this is what you attach to with t from the dashboard;
• a per-task clone of the repo, mounted read-write at /workspace, on the task's own branch.

The session service (the per-host runner) owns the container's whole lifecycle — it claims the task, builds the image, starts the container, and later heals or cleans it up. The task service only ever records and displays what the runner reports; it spawns nothing itself.

The status you see on the dashboard

The dashboard shows one container status per task — a single word the task service computes by folding three signals together: the spawn phase the runner is reporting, whether the container has an open registration (its live connection), and whether the runner itself is still connected. First match wins, so a status higher in this table always overrides a lower one.

Status	What it means	What you do
queued	Non-terminal but unclaimed — no runner has picked it up yet.	Wait; if it sticks, check a runner is actually running (make start).
claiming	A runner just claimed it; the spawn is about to start.	Nothing — transient.
preparing	Readying the per-task clone / workspace.	Nothing — transient.
building	Composing and docker build-ing the image. The slow step on a first run for a repo (later runs hit Docker's cache).	Wait; first build of a repo image can take minutes.
starting	docker run and the tmux session are coming up.	Nothing — transient.
awaiting	Container and tmux are up; waiting for the agent to open its /live connection.	Nothing — transient; if it lingers, see When it goes wrong.
live	A container registration is open — the agent is running and reachable.	Attach with t to watch or steer.
down	Claimed, the runner is alive, but the container is gone and unregistered. It came up and vanished, or never reported.	Respawn from the dashboard with R (the runner also self-heals — see below).
failed	A spawn step raised an error (hover / inspect for the detail).	Read the detail; fix the cause (bad image layer, missing secret) and respawn.
disconnected	Claimed by a runner that's no longer connected to the task service.	Bring that runner's host back, or the task stays stranded until its claim is released.
–	Terminal task (COMPLETE / DROPPED) — no container concept.	Nothing.

The dashboard only displays this status; it does no liveness guessing of its own. The five middle statuses (claiming→awaiting) are the spawn phases the runner pushes as it works; queued, live, down, and disconnected are derived by the task service from registration + runner liveness, so the runner never invents them. (See compose_container_status in core/models.py for the exact precedence.)

Coming up: the spawn sequence

When a new task appears, the per-host session service brings its container up in five steps, reporting each as a status above.

1. Claim (claiming). A runner claims the unclaimed task first — a compare-and-set that 409s if another host got there first. This is the spawn gate: exactly one host ever runs a given task, even with several runners watching the same task service.

2. Prepare (preparing). The runner makes the task's workspace: a git clone --local of a per-repo cache clone into a task-private directory, mounted read-write at /workspace. The clone is self-contained (hard-linked objects, so it's cheap), and its origin is pointed at the real forge rather than the local cache.

3. Build (building). The runner composes and docker builds the task's image. It's tagged per (workflow, repo), so once built it's cached — only the first task for a repo pays the full build cost, which is why building can be slow that first time. See layers for how the image is composed.

4. Start (starting). The runner does docker run --detach (injecting the repo's secrets via --env-file, mounting /workspace, and a small config volume that persists the agent's history across respawns), then creates the tmux session whose pane execs the agent inside the container.

5. Await (awaiting). The container's entrypoint starts up. It first remaps its baked panopticon user to the invoking host's uid/gid (PANOPTICON_PUID/PGID) and drops privileges via gosu — so files the agent writes under /workspace are owned by you on the host, not root. Then it connects to the task service and holds the connection open, registering that this container is working on this task.

Once that registration is open, the status flips to live and the agent is off.

Provisioning: naming the task gives it a branch

A fresh task starts on whatever its clone checked out; once the agent sets a slug, the branch appears. The mechanics happen on the runner, where the container actually is (so they stay correct even when the runner is remote): the session service sees the slug land and runs git checkout -b panopticon/<slug> on the per-task clone, with origin already pointed at the forge. The task service only records the result — it touches no filesystem.

For what a slug, branch, clone, and provisioned mean as task concepts, see tasks.

While it runs

• Liveness is connection-based, not a heartbeat. The container holds one long-lived /live connection to the task service. While it's open the task reads live; if it drops (crash, kill, network blip) the registration clears and the task stops being live. The container reconnects with backoff across transient blips.
• The agent. In the tmux pane, the launcher wires everything the agent needs and then execs claude: it renders the workflow's skills and operations as slash-commands, points the agent's MCP client at the task service (so it can read/write artifacts and drive its own state), and puts the workflow's state-machine overview in the system prompt.
• The task memo is pre-filled. On a task's first spawn the runner pastes the task's description into the agent's input box (unsent), so you see the ask waiting when you attach.
• Turn and blocked. As the agent and user hand the work back and forth, the task's turn flips (agent ↔ user) via in-container hooks; a separate blocked marker is a deliberate "waiting on something" flag the agent sets. Both are shown on the dashboard.
• Auth. The agent authenticates from a CLAUDE_CODE_OAUTH_TOKEN injected from the repo's env-file — see auth.

Resource priority: tasks yield to you

A task container is where unbounded work happens — an agent running a repo's whole test suite, a docker build inside a dind task, six tasks at once — on the same machine as your editor, your shell, and panopticon's own control plane. So every container is spawned deprioritized: it loses CPU and disk races against normally-weighted processes, and under real memory pressure the kernel kills it before anything of yours.

This is priority, not a cap. Nothing limits how much CPU or RAM a task may use when nobody else wants it, so on an idle host tasks run at full speed.

Concretely, each container gets --cpu-shares 2 (docker's floor — cgroup v2 cpu.weight 1), --blkio-weight 10 (the floor; disk contention is what actually makes a desktop stutter) and --oom-score-adj 500. The agent pane raises its own OOM score too: oom_score_adj is per-process and inherited across fork, and the pane is started with docker exec (forked from the Docker daemon, not from the container's PID 1), so the flag on docker run would otherwise miss the one process that actually eats memory. The cgroup levers need no such trick — an exec'd process joins the container's cgroup. The workspace-cleanup sweep runs deprioritized as well.

If a task does get OOM-killed you'll see it: the runner reads OOMKilled off docker inspect before the exit code, so the task shows failed with an out-of-memory detail rather than a mystery.

Tuning it per host

The knobs are read from the runner host's environment, so a big build box and a laptop can differ. Set any of them to off (or empty) to drop that flag entirely — the argv is then exactly what panopticon emitted before any of this existed.

Variable	Default	What it sets
PANOPTICON_CONTAINER_CPU_SHARES	2	CPU weight (--cpu-shares)
PANOPTICON_CONTAINER_BLKIO_WEIGHT	10	block-IO weight (--blkio-weight)
PANOPTICON_CONTAINER_OOM_SCORE_ADJ	500	OOM-killer preference, for the container and the agent pane
PANOPTICON_CONTAINER_CGROUP_PARENT	unset	parent cgroup (--cgroup-parent) — see below
PANOPTICON_HOST_NICE	19	nice for a shell task's host session (no container to weight)

A bad value warns and falls back to the default instead of failing the spawn, and values are clamped to what docker and the kernel accept (a negative OOM adjustment — shielding a task at the host's expense — is refused).

Hosts that refuse the flags. Some daemons reject --cpu-shares/--blkio-weight outright: a nested Docker daemon whose cgroup is in threaded mode answers any of them with unable to apply cgroup configuration. A spawn is never lost to that — the runner retries once without the cgroup-backed flags (keeping --oom-score-adj, which needs no controller), logs one warning, and remembers the answer for the rest of its life.

Going further with a slice. With docker's systemd cgroup driver, containers land in system.slice/docker-<id>.scope while your own processes sit in user.slice, and cgroup weights are compared between siblings — so weight 1 on a container's scope deprioritizes it within system.slice but doesn't by itself make it lose to user.slice. If you want that enforced at the hierarchy level, create a low-weight slice and point panopticon at it:

# /etc/systemd/system/panopticon.slice
[Slice]
CPUWeight=1
IOWeight=1

sudo systemctl daemon-reload
export PANOPTICON_CONTAINER_CGROUP_PARENT=panopticon.slice   # on the runner host

Installing a unit is your call, so per-container weights stay the zero-setup default.

When it goes wrong

A container can disappear out from under a live task — an OOM kill, a host reboot, a docker rm. The system distinguishes a few cases:

• down — the container is gone but its runner is alive. The runner's reconcile pass notices the container has vanished and clears the stale spawn phase, so the task composes to down rather than lying at awaiting.
• Self-heal. The runner also heals orphans automatically: a task it still owns whose tmux session is gone gets respawned through the same idempotent spawn path (the agent resumes from its persisted history). A crash-loop cap (a handful of respawns within a short window) stops a hopelessly failing task from respawning forever — past the cap it's left down for you to look at.
• Manual respawn. You can always respawn a down task yourself from the dashboard with R. This is also how you pick up a changed secret or env-file — respawn to restart the container with the new values.
• disconnected — the runner is gone, not just the container. The task is stuck claimed by an absent host. Bring that host back (its runner reclaims and heals), or release the claim so another runner can take over.
• failed — a spawn step raised. The status carries a detail string (e.g. a broken image layer or a missing secret). Fix the underlying cause, then respawn.

Teardown

When a task reaches a terminal state (COMPLETE or DROPPED), its container is no longer needed. The runner's cleanup pass stops the container, releases the claim, and removes the per-task /workspace clone. If a delete is blocked (e.g. by a file left root-owned by a nested build), it escalates — an as-root sweep, then quarantining the directory aside — so a stuck workspace never wedges the runner. After cleanup the task shows –: no container, nothing to attach to.

See also

• auth — giving a repo its claude token.
• layers — how a task's image is composed (base → workflow → repo).
• tasks — slug, branch, clone, and provisioning as task concepts.