Close Harbor integration gaps: timer daemon, verifier isolation, artifact collection

[surelyMersad]

Closes #8. Replaces #33 with @hrdkbhatnagar's review feedback addressed
end-to-end on Modal.

Stacked on #36 ("Repin vllm and inspect_evals so the Dockerfile
builds"). The end-to-end Modal verification below was run with #36's
Dockerfile fix in place. If #36 merges first, this PR's diff against
add_harbor_support will become net-clean. If #36 hasn't merged yet,
the diff here will include #36's two-line vllm/inspect_evals repin —
please review them as part of #36, not here.

What changed since #33

• Pre-agent timer is a real daemon, not a sentinel-file. The
  sentinel-file approach was correctly flagged as broken — the agent
  could (and would) influence its start time. The new design starts a
  background timer at container boot via three redundant mechanisms
  (ENTRYPOINT, /etc/bash.bashrc, BASH_ENV profile script), all
  idempotent via a PID file. start_epoch is captured by the daemon's
  first iteration before any agent code runs.

• tests/test.sh restructured around a single fail() helper that
  always writes reward.txt and metrics.json and snapshots the timer
  daemon's state into /logs/verifier/timer/ — so reviewer-visible
  evidence lands on every exit path, not just the success path.

• Artifact collection moved out of test.sh onto Harbor's top-level
  artifacts: config in the generated job.yaml. Fixes a bug from Close Harbor integration gaps: verifier isolation, artifact collection #33
  where the in-script find … -exec cp block was unreachable on early
  exits (e.g. missing final_model) — exactly when artifacts are most
  useful.

• Anti-tamper shebang check added: test.sh verifies timer.sh and
  entrypoint.sh still start with their expected shebangs.

• Harbor version bumped to ≥ 0.2.0 in install instructions. Harbor
  0.2.0 fixed two upstream issues that were blocking the integration:

  • Image.from_dockerfile(path) now passes context_dir=environment_dir,
    so COPY directives resolve against the Dockerfile's directory
    regardless of where harbor run was invoked.
  • File transfers use 4 MB chunks (was 8 KB) and parallelise downloads
    via an asyncio semaphore.

End-to-end verification on Modal

Real Modal run, gsm8k+qwen3-1.7b, --agent nop, TASK_BUDGET_SECS=60:

verifier/test-stdout.txt:
=== Verifying evaluate.py integrity ===
OK: evaluate.py integrity verified (02b97287d5cd2179...)
OK: timer.sh and entrypoint.sh shebangs intact
OK: timer state snapshotted
=== GPU Check ===
NVIDIA H100 80GB HBM3 ...
=== Checking final_model ===
FAIL: final_model directory not found

verifier/reward.txt: 0
verifier/metrics.json: {"error":"final_model directory not found","accuracy":0}
verifier/timer/start_epoch: 1777071723
verifier/timer/budget_secs: 60
verifier/timer/timer.pid: 5
verifier/timer/alert_5min ← all three sub-budget alerts fired,
verifier/timer/alert_10min ← proving the daemon loop ticked at least
verifier/timer/alert_30min ← once before the verifier ran

pid=5 and pre-fired alerts together prove the daemon was started by
the entrypoint before the agent or verifier ran — which is the
property "we want a pre-agent hook" was asking for.

Test plan

• Task generation: all 28 combinations generate without error
• Modal e2e with --agent nop: build → verifier → reward.txt + timer state captured (output
  above), result.json.exception_info: null
• Modal e2e with --agent claude-code and short timeout — skipped for smoke; can attach if
  requested
• Modal SHA256 tamper variant — verified locally; can attach Modal log if requested

[hrdkbhatnagar]

Thanks for putting this together, and especially for the Modal verification. Going point by point, since I've pushed a substantial round of changes to add_harbor_support since you opened this (full update on #8) and want to make sure we converge:

1. Daemon-based timer (ENTRYPOINT + /etc/bash.bashrc + BASH_ENV)

Already fixed on the branch via Harbor's native healthcheck. The old design was actually broken for a different reason than tampering:

START_FILE="$(dirname "$0")/.timer_start" resolves relative to invocation cwd, so every new directory the agent cd'd into got a fresh sentinel and the timer reset.

New design uses [environment.healthcheck] in task.toml to write /timer_start (absolute path) once before agent setup begins. Tamper-resistant for the same reason yours is — agent isn't running yet — but two advantages over the daemon:

• Tighter start time. Daemon starts at container boot, ~3-5mins before the agent, eating into budget. Healthcheck fires immediately before agent setup, ~5s grace at most.

• No prompt change. Your daemon design seems to require a modified agent prompt, which diverges from the original PTB run setting. Healthcheck keeps the prompt identical.

Drops the three redundant idempotency mechanisms + PID file in favor of something native, which matters for us - divergence from the original codebase is a parity claim we have to defend in the paper.

2. fail() helper writing reward.txt + metrics.json on every exit path

Promising. I'm testing whether it's still needed once Harbor's native failure capture is wired in, there's overlap. If it covers a real gap, I'll port it; just want to understand the gap before adding a divergent code path. Lmk what you think

3. Artifact collection via Harbor's top-level artifacts: config

Done on the branch! task.toml now declares both final_model/ and the full agent workspace/ (with sensible excludes) as top-level artifacts.

4. Anti-tamper shebang check on timer.sh / entrypoint.sh

I'd skip this. Preserving a shebang while tampering is trivial, so it's a canary for accidental destruction (agent does > timer.sh), not a defense against an adversarial agent.

More importantly, the file tampering vector it's nominally guarding is already closed on the branch via /tests/ placement: evaluate.py, templates/, evaluation_code/, contamination_judge.py, and metadata.json are now uploaded to /tests by Harbor after the agent process exits, so the agent has no write access to the copies the verifier reads. The shebang check guards files that are no longer reachable as an attack surface. This is also native to Harbor so we dont have to do anything special.

5. evaluate.py SHA256 integrity check

Same - the version of evaluate.py the verifier executes now lives in /tests, untouchable by the agent. The hash check would fire only on a vector that's already closed, and wouldn't fire on the residual one (package tampering).

The residual concern :agent pip install-ing tampered packages that the verifier later imports, is being fixed natively upstream in harbor-framework/harbor#1607 ([verifier_environment] support, fresh sandbox for verification). I think that's the right place to solve it; doing it client side via shebang/hash checks wouldn't actually address it imo.

Happy to discuss, let me know what you think