RL Environment - Distributed Control Plane Benchmark

A realistic Reinforcement Learning environment for evaluating frontier coding agents on distributed-systems debugging tasks, built using the Harbor framework - a framework for agent evaluation, benchmark development, and RL rollout generation that also powers Terminal-Bench 2.0 and 3.0.

The centerpiece is control-plane - a Go-based multi-controller control plane containing intentionally seeded recovery regressions designed to require architectural root-cause analysis across controller ownership boundaries. Evaluated against claude-sonnet-5 across 5 independent rollouts: 0/5 solved (0% pass rate).

What I Built

This repository contains my work on designing and evaluating a realistic agentic debugging environment using Harbor. The project includes:

• a multi-container distributed control-plane environment
• PostgreSQL-backed durable state
• event-driven controllers and reconciliation loops
• intentionally seeded recovery regressions
• an oracle solution and no-op baseline
• behavioral and architecture-aware verification
• five independent Claude Sonnet 5 rollout artifacts and analyses

What is this?

This project designs and ships a realistic software-engineering environment and task that is hard for a frontier coding model to solve reliably. The environment mirrors real-world distributed systems work: multi-container services with PostgreSQL-backed durable state, event-driven reconciliation loops, optimistic concurrency, fault recovery, and controller ownership semantics.

Rather than a toy bug-hunt, the difficulty comes from multi-bug causal reconstruction across ownership boundaries. Each controller mostly looks locally reasonable, normal operations continue to work, fixing one subsystem improves some behavior, but full convergence requires understanding how scheduler, provision, and retry semantics compose across the whole system. The challenge is to continue investigating after the first plausible fix and determine whether the system has actually recovered end-to-end.

The control-plane Task

A distributed control plane at /workspace/environment/ is not behaving as intended. The system manages the lifecycle of external resources through multiple cooperating components.

Observed behavior:

• The control plane remains available and continues processing normal workloads.
• Newly created resources complete their lifecycle successfully.
• Most previously managed resources remain healthy.
• A small subset of existing resources fails to fully converge after transient operational interruptions, even though no persistent infrastructure failures remain.
• The affected resources remain stable but do not naturally recover to their expected steady state.
• Individual components appear to behave reasonably when examined on their own, making the underlying behavioral issues difficult to isolate.

The agent must restore the system to its intended behavior while preserving the existing architecture, component responsibilities, reconciliation behavior, correctness guarantees, and overall system behavior.

Environment Topology (Docker Compose)

Service	Purpose	Ports
main	Agent-facing workspace with Go + Docker toolchain	-
postgres	Durable resource state, provider state, event history	5432
api	User-facing resource API	8080
controller	Scheduler, event bus, worker, reconciliation controllers, metrics	9090

Recovery Properties Exercised by the Task

The benchmark evaluates recovery behavior across several cooperating subsystems. End-to-end convergence requires all of the following invariants to be satisfied, not just whichever one is most locally salient:

• Scheduled retry recovery - Overdue durable scheduled work must eventually resume after an interruption rather than being trusted forever as "already scheduled."
• Provisioning recovery - Interrupted resource lifecycle transitions must safely return to a reconcilable state instead of taking invalid shortcuts that can strand provider-side reconciliation.
• Dependency recovery - When a dependency recovers, downstream retry-owned state must be durably unpinned and resume correctly, producing the expected retry transition evidence.

Rollout Results

Pass rate across 5 independent rollouts of claude-code with anthropic/claude-sonnet-5 in default mode: 0/5 (0%).

Across the five rollouts, agents consistently converged on a plausible local explanation before establishing whether recovery was complete end-to-end. Two rollouts correctly repaired the scheduler regression, while other rollouts pursued plausible retry or Provision-side failure modes that did not resolve the full recovery contract.

All five rollouts voluntarily terminated with substantial time remaining, suggesting that the dominant failure mode was premature confidence after local validation rather than exhaustion of the allotted debugging budget.

Trial	Reward	Tests failed	Rollout (s)
control-plane__4Pp2iTz	0.0	5 recovery + invariants tests	558.2
control-plane__7LggQ6a	0.0	6 recovery + scheduler tests	208.8
control-plane__7poPv9G	0.0	5 recovery + invariants tests	298.1
control-plane__LQshexr	0.0	6 recovery + scheduler tests	552.3
control-plane__Z2eBQgH	0.0	6 recovery + scheduler tests	1418.5

Dominant Failure Mode: Local Anchoring + Premature Completion

• Scheduler anchoring (2 trials) - Correctly found and repaired the scheduler regression, then stopped without investigating the remaining recovery paths.
• Retry anchoring (2 trials) - Identified suspicious retry/dependency behavior and developed plausible controller-local fixes, but the patches did not restore the intended dependency recovery semantics.
• Provision anchoring (1 trial) - Went furthest dynamically: built Docker containers, crash-injected the controller, reproduced and experimentally fixed a genuine Provision-side issue. However, that issue was not sufficient to restore the benchmark's intended recovery behavior, and the rollout did not continue investigating scheduler and retry paths.

The common stopping rule was local confidence: once the agent had a plausible explanation, a clean patch, and build/vet success, it declared done instead of running the full environment-level verifier or systematically auditing the remaining controllers.

Full per-trial analysis and writeup in submission/REPORT.md.

Repository Layout

RL-Agent-Eval-Env/
├── README.md                  <- this file
├── scripts/
│   ├── qa.sh                  <- static checks
│   ├── grade.py               <- evaluate a rollout job
│   └── submit.py              <- package a graded job into submission/
├── submission/
│   ├── REPORT.md              <- full task writeup + rollout analysis
│   ├── grading/
│   │   ├── grade.json         <- scorecard: pass rate, oracle/nop, rubric tally
│   │   └── rubric-review.json <- rubric review (harbor check)
│   └── job/                   <- 5 rollouts: per-trial rewards + analysis
├── ci_checks/                 <- static check scripts
├── rubrics/                   <- harbor cli rubrics
└── tasks/
    ├── hello-world/           <- runnable reference task (Harbor format demo)
    │   ├── task.toml          <- metadata, resource limits, timeouts
    │   ├── instruction.md     <- the prompt the agent receives
    │   ├── environment/       <- sandbox the agent runs in
    │   │   └── Dockerfile
    │   ├── solution/solve.sh  <- reference solution scoring 1.0
    │   └── tests/             <- verifiers
    │       ├── Dockerfile
    │       ├── test.sh
    │       └── test_state.py
    └── control-plane/         <- the main benchmark task
        ├── task.toml
        ├── instruction.md
        ├── environment/       <- Go control plane source + Compose stack
        │   ├── Dockerfile
        │   ├── docker-compose.yaml
        │   ├── cmd/api/, cmd/controller/
        │   ├── internal/      <- controllers, scheduler, events, store, …
        │   ├── go.mod / go.sum
        │   ├── schema.sql
        │   └── seed.sql
        ├── solution/
        │   ├── solve.sh
        │   └── files/         <- scheduler.go.src, provision.go.src, retry.go.src
        └── tests/
            ├── Dockerfile
            ├── docker-compose.yaml
            ├── test.sh
            └── test_outputs.py

Getting Set Up

1. Install Harbor

uv tool install harbor==0.14.0
harbor --version

2. Install Docker

A running Docker daemon is required for local trials.

docker ps   # should not error

3. (Optional) Set an OpenRouter API key

Needed only for running real model rollouts (not required for oracle/nop runs).

cp .env.example .env
# then edit .env with your OpenRouter API key

Running Locally

From the repo root:

# Validate the reference task scores 1.0 via the Oracle agent (runs solve.sh).
harbor run -p tasks/hello-world -a oracle

# Validate the verifier rejects a no-op agent (expects reward 0.0).
harbor run -p tasks/hello-world -a nop

# Run control-plane oracle (reference solution -> reward 1.0).
harbor run -p tasks/control-plane -a oracle

# Run control-plane nop (no-op -> reward 0.0).
harbor run -p tasks/control-plane -a nop

# Run a real model end-to-end via OpenRouter.
harbor run --env-file .env \
  -p tasks/control-plane \
  -a claude-code \
  -m anthropic/claude-sonnet-5

# Run N independent rollouts to estimate a pass rate.
harbor run --env-file .env -p tasks/control-plane \
  -a claude-code \
  -m anthropic/claude-sonnet-5 \
  --n-attempts 5

Each command writes a trial directory under ./jobs/<timestamp>/. The final reward is in <trial>/verifier/reward.txt.

Interactive Debugging

Open a shell inside the task's environment container - useful while developing:

harbor tasks start-env -p tasks/control-plane -e docker -a -i

Grading and QA

scripts/qa.sh                              # static checks
python3 scripts/grade.py jobs/<job-name>   # oracle/nop + rubric review + analysis

Building a New Task

1. Scaffold the directory. Either copy tasks/hello-world/ or use Harbor's scaffolder:

   harbor tasks init my-task -p tasks/

2. Configure timeouts and resources. In task.toml, set [agent].timeout_sec, [verifier].timeout_sec, and [environment] resources appropriate for the task's horizon and infrastructure.

3. Verify the reference solution. Run the oracle and confirm reward 1.0:

   harbor run -p tasks/my-task -a oracle

4. Measure difficulty and iterate. Run a real agent n times:

   harbor run --env-file .env -p tasks/my-task \
     -a claude-code \
     -m anthropic/claude-sonnet-5 \
     --n-attempts 5