Skip to content

fix(sandbox/recover): revert Hermes probe boundary enforcement #5547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sandl99 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(sandbox/recover): revert Hermes probe boundary enforcement #5547

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
91d51db
Revert "fix(sandbox/recover): enforce Hermes env-file secret boundary…
sandl99 Jun 19, 2026
65d4669
test(e2e): relax workflow selector timeout
sandl99 Jun 19, 2026
f7f832b
Merge remote-tracking branch 'origin/main' into pr5547-merge-main
sandl99 Jun 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions docs/reference/commands-nemohermes.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,11 +501,6 @@ If the gateway is already running, the command exits zero with a probe message a
nemohermes my-assistant recover
```

`recover` re-evaluates the documented Hermes secret boundary against `/sandbox/.hermes/.env` on every run, including when the gateway is already healthy.
If the file contains raw secret-shaped values (for example a pasted Telegram, Discord, or Slack bot token in place of the expected `openshell:resolve:env:<name>` placeholder), the command stops the running gateway, exits non-zero, and prints the offending key.
Replace each flagged value with the `openshell:resolve:env:<name>` placeholder and re-run.
Older Hermes sandbox images that predate the standalone validator are detected and left untouched: `recover` prints a `[boundary]` warning naming the sandbox and noting that `/sandbox/.hermes/.env` was not re-evaluated, then proceeds with the rest of the recovery path rather than blocking the gateway. Re-image the sandbox to a current Hermes build to enable the per-run boundary re-evaluation described above.

### `nemohermes <name> status`

Show sandbox status, health, and inference configuration.
Expand Down
9 changes: 0 additions & 9 deletions docs/reference/commands.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -630,15 +630,6 @@ If the gateway is already running, the command exits zero with a probe message a
$$nemoclaw my-assistant recover
```

<AgentOnly variant="hermes">

`recover` re-evaluates the documented Hermes secret boundary against `/sandbox/.hermes/.env` on every run, including when the gateway is already healthy.
If the file contains raw secret-shaped values (for example a pasted Telegram, Discord, or Slack bot token in place of the expected `openshell:resolve:env:<name>` placeholder), the command stops the running gateway, exits non-zero, and prints the offending key.
Replace each flagged value with the `openshell:resolve:env:<name>` placeholder and re-run.
Older Hermes sandbox images that predate the standalone validator are detected and left untouched: `recover` prints a `[boundary]` warning naming the sandbox and noting that `/sandbox/.hermes/.env` was not re-evaluated, then proceeds with the rest of the recovery path rather than blocking the gateway. Re-image the sandbox to a current Hermes build to enable the per-run boundary re-evaluation described above.

</AgentOnly>

### `$$nemoclaw <name> status`

Show sandbox status, health, and inference configuration.
Expand Down
108 changes: 0 additions & 108 deletions src/lib/actions/sandbox/connect-flow.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,6 @@ type ConnectHarnessOptions = {
wasRunning?: boolean;
recovered?: boolean;
forwardRecovered?: boolean;
secretBoundaryRefused?: boolean;
secretBoundaryReason?: "raw-secret" | "inconclusive";
};
spawnStatus?: number | null;
};
Expand Down Expand Up @@ -224,110 +222,4 @@ describe("connectSandbox flow", () => {
);
expect(exitSpy).toHaveBeenCalledWith(1);
});

it("probe-only mode exits with raw-secret remediation when the Hermes boundary refuses recovery", async () => {
const harness = createConnectHarness({
processCheck: {
checked: true,
wasRunning: true,
recovered: false,
forwardRecovered: false,
secretBoundaryRefused: true,
secretBoundaryReason: "raw-secret",
},
});
const agentRuntime = requireDist("../../../../dist/lib/agent/runtime.js");
vi.spyOn(agentRuntime, "getSessionAgent").mockReturnValue({ name: "hermes" });
vi.spyOn(agentRuntime, "getAgentDisplayName").mockReturnValue("Hermes");
const errorSpy = vi.spyOn(console, "error");

await expect(harness.connectSandbox("alpha", { probeOnly: true })).rejects.toThrow(
"process.exit(1)",
);

expect(harness.runAutoPairSpy).not.toHaveBeenCalled();
expect(harness.spawnSyncSpy).not.toHaveBeenCalledWith(
"openshell",
["sandbox", "connect", "alpha"],
expect.any(Object),
);
const errorOutput = errorSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
expect(errorOutput).toContain(
"Probe failed: refused to confirm Hermes gateway in 'alpha' — /sandbox/.hermes/.env contains raw secret-shaped values.",
);
expect(errorOutput).toContain(
"Replace raw secret values with openshell:resolve:env:<name> placeholders and re-run.",
);
const logOutput = harness.logSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
expect(logOutput).not.toContain("Probe complete");
expect(exitSpy).toHaveBeenCalledWith(1);
});

it("non-probe connect exits before Ollama/inference-route/auto-pair when the Hermes boundary refuses", async () => {
const harness = createConnectHarness({
processCheck: {
checked: true,
wasRunning: true,
recovered: false,
forwardRecovered: false,
secretBoundaryRefused: true,
secretBoundaryReason: "raw-secret",
},
});
const agentRuntime = requireDist("../../../../dist/lib/agent/runtime.js");
vi.spyOn(agentRuntime, "getSessionAgent").mockReturnValue({ name: "hermes" });
vi.spyOn(agentRuntime, "getAgentDisplayName").mockReturnValue("Hermes");
const errorSpy = vi.spyOn(console, "error");

await expect(harness.connectSandbox("alpha")).rejects.toThrow("process.exit(1)");

expect(harness.ensureOllamaAuthProxySpy).not.toHaveBeenCalled();
expect(harness.runAutoPairSpy).not.toHaveBeenCalled();
expect(harness.spawnSyncSpy).not.toHaveBeenCalledWith(
"openshell",
["sandbox", "connect", "alpha"],
expect.any(Object),
);
const errorOutput = errorSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
expect(errorOutput).toContain(
"Connect failed: refused to confirm Hermes gateway in 'alpha' — /sandbox/.hermes/.env contains raw secret-shaped values.",
);
expect(errorOutput).toContain(
"Replace raw secret values with openshell:resolve:env:<name> placeholders and re-run.",
);
expect(exitSpy).toHaveBeenCalledWith(1);
});

it("probe-only mode exits with inconclusive guidance when the Hermes boundary check could not run", async () => {
const harness = createConnectHarness({
processCheck: {
checked: true,
wasRunning: true,
recovered: false,
forwardRecovered: false,
secretBoundaryRefused: true,
secretBoundaryReason: "inconclusive",
},
});
const agentRuntime = requireDist("../../../../dist/lib/agent/runtime.js");
vi.spyOn(agentRuntime, "getSessionAgent").mockReturnValue({ name: "hermes" });
vi.spyOn(agentRuntime, "getAgentDisplayName").mockReturnValue("Hermes");
const errorSpy = vi.spyOn(console, "error");

await expect(harness.connectSandbox("alpha", { probeOnly: true })).rejects.toThrow(
"process.exit(1)",
);

expect(harness.runAutoPairSpy).not.toHaveBeenCalled();
const errorOutput = errorSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
expect(errorOutput).toContain(
"Probe failed: secret-boundary check did not complete for Hermes gateway in 'alpha'.",
);
expect(errorOutput).toContain(
"Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.",
);
const logOutput = harness.logSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
expect(logOutput).not.toContain("Probe complete");
expect(exitSpy).toHaveBeenCalledWith(1);
});
});
38 changes: 2 additions & 36 deletions src/lib/actions/sandbox/connect.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import { findReachableOllamaHost, probeLocalProviderHealth } from "../../inferen
import { ensureOllamaAuthProxy, probeOllamaAuthProxyHealth } from "../../inference/ollama/proxy";
import { LOCAL_INFERENCE_TIMEOUT_SECS } from "../../onboard/env";
import { resolveSandboxGatewayName } from "../../onboard/gateway-binding";
import { getSandboxTargetGatewayName } from "./gateway-target";
import { isWsl } from "../../platform";
import { ROOT } from "../../runner";
import * as sandboxVersion from "../../sandbox/version";
Expand All @@ -52,7 +53,6 @@ import {
import { preflightVllmModelEnvOrExit } from "./connect-vllm-preflight";
import { isDockerRuntimeDown, printDockerRuntimeDownGuidance } from "./gateway-failure-classifier";
import { ensureLiveSandboxOrExit, printGatewayLifecycleHint } from "./gateway-state";
import { getSandboxTargetGatewayName } from "./gateway-target";
import { printGatewayWedgeDiagnostics } from "./gateway-wedge-diagnostics";
import { checkAndRecoverSandboxProcesses, executeSandboxExecCommand } from "./process-recovery";
import { applyOpenShellVmDnsMonkeypatch, shouldApplyVmDnsMonkeypatch } from "./vm-dns-monkeypatch";
Expand Down Expand Up @@ -183,33 +183,6 @@ export function parseSandboxConnectArgs(
return options;
}

function exitOnSecretBoundaryRefusal(
sandboxName: string,
agentName: string,
processCheck: Record<string, unknown>,
contextLabel: "Probe" | "Connect",
): never {
console.error("");
const reason =
"secretBoundaryReason" in processCheck
? (processCheck.secretBoundaryReason as "raw-secret" | "inconclusive" | undefined)
: undefined;
if (reason === "raw-secret") {
console.error(
` ${contextLabel} failed: refused to confirm ${agentName} gateway in '${sandboxName}' — /sandbox/.hermes/.env contains raw secret-shaped values.`,
);
console.error(
" Replace raw secret values with openshell:resolve:env:<name> placeholders and re-run.",
);
} else {
console.error(
` ${contextLabel} failed: secret-boundary check did not complete for ${agentName} gateway in '${sandboxName}'.`,
);
console.error(" Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.");
}
process.exit(1);
}

function runSandboxConnectProbe(sandboxName: string): void {
const processCheck = checkAndRecoverSandboxProcesses(sandboxName, { quiet: true });
const agent = agentRuntime.getSessionAgent(sandboxName);
Expand All @@ -220,9 +193,6 @@ function runSandboxConnectProbe(sandboxName: string): void {
);
process.exit(1);
}
if ("secretBoundaryRefused" in processCheck && processCheck.secretBoundaryRefused) {
exitOnSecretBoundaryRefusal(sandboxName, agentName, processCheck, "Probe");
}
if (processCheck.wasRunning) {
ensureSandboxInferenceRoute(sandboxName, { quiet: true });
// Defense-in-depth scope-upgrade approval on the probe-only / `recover`
Expand Down Expand Up @@ -891,11 +861,7 @@ export async function connectSandbox(
/* non-fatal — don't block connect on session detection failure */
}

const processCheck = checkAndRecoverSandboxProcesses(sandboxName);
if ("secretBoundaryRefused" in processCheck && processCheck.secretBoundaryRefused) {
const agentName = agentRuntime.getAgentDisplayName(agentRuntime.getSessionAgent(sandboxName));
exitOnSecretBoundaryRefusal(sandboxName, agentName, processCheck, "Connect");
}
checkAndRecoverSandboxProcesses(sandboxName);
// Ensure Ollama auth proxy is running (recovers from host reboots)
ensureOllamaAuthProxy();

Expand Down
124 changes: 1 addition & 123 deletions src/lib/actions/sandbox/process-recovery.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,6 @@ import {
runOpenshell,
} from "../../adapters/openshell/runtime";
import { OPENSHELL_PROBE_TIMEOUT_MS } from "../../adapters/openshell/timeouts";
import {
buildHermesEnvFileBoundaryStandaloneCheck,
SECRET_BOUNDARY_OK_MARKER,
SECRET_BOUNDARY_REFUSED_MARKER,
SECRET_BOUNDARY_VALIDATOR_MISSING_MARKER,
} from "../../agent/hermes-recovery-boundary";
import * as agentRuntime from "../../agent/runtime";
import { G, R } from "../../cli/terminal-style";
import { DASHBOARD_PORT } from "../../core/ports";
Expand Down Expand Up @@ -488,115 +482,12 @@ function recoverHermesDashboardProcessIfEnabled(sandboxName: string): boolean |
return recoverHermesDashboardProcess(sandboxName, { executeCommand: executeSandboxCommand });
}

function isHermesAgent(agent: ReturnType<typeof agentRuntime.getSessionAgent>): boolean {
return !!agent && agent.name === "hermes";
}

type SecretBoundaryRefusalReason = "raw-secret" | "inconclusive";

type HermesSecretBoundaryEnforcement =
| { refused: false }
| { refused: true; reason: SecretBoundaryRefusalReason; stderr: string };

function printValidatorStderr(stderr: string): void {
if (!stderr.trim()) return;
for (const line of stderr.split(/\r?\n/)) {
if (line.trim()) console.error(` ${line}`);
}
}

/**
* Re-run the Hermes env-file secret-boundary validator against a running
* gateway, before the probe path returns control to the caller. The
* relaunch path already runs the same validator inline as part of
* `buildRecoveryScript`, but the probe path returns early as soon as the
* gateway is reported healthy, so a poisoned `.env` injected after cold
* start would otherwise never be re-evaluated. The check is invoked via
* `openshell sandbox exec` (root) so the validator's kill snippet can
* actually signal the gateway-user process when refusing — a sandbox-user
* SSH shell cannot (test/e2e-gateway-isolation.sh test 13). Every
* refusal diagnostic — validator `[SECURITY]` stderr, the helper's own
* context line, and the remediation hint — is written to `console.error`
* unconditionally, so the offending key (e.g. `TELEGRAM_BOT_TOKEN (line
* N)`) and the reason for refusal always reach the operator, including
* on the quiet probe/recover path. Returns `null` only when the persisted
* sandbox registry entry is not Hermes (no boundary to enforce). When
* the registry says Hermes but the in-memory agent definition failed to
* load (`getSessionAgent()` returned `null` from its catch path), the
* helper fails safe with an inconclusive refusal rather than silently
* skipping the boundary. A running Hermes gateway whose root exec
* channel is unreachable is also treated as a fail-safe inconclusive
* refusal rather than a healthy path. Non-zero validator status without
* a `SECRET_BOUNDARY_REFUSED` marker is reported as inconclusive, not as
* a raw-secret refusal, so a shell or validator crash does not
* masquerade as a poisoned env file.
*/
function enforceHermesSecretBoundaryOnRunningGateway(
sandboxName: string,
agent: ReturnType<typeof agentRuntime.getSessionAgent>,
): HermesSecretBoundaryEnforcement | null {
const persistedAgent = registry.getSandbox(sandboxName)?.agent;
if (persistedAgent !== "hermes") return null;
if (!isHermesAgent(agent)) {
console.error("");
console.error(
` ${R}Hermes agent definition could not be loaded for sandbox '${sandboxName}'.${R}`,
);
console.error(" Refusing recovery to keep the validator-enforced boundary intact.");
return { refused: true, reason: "inconclusive", stderr: "" };
}
const script = buildHermesEnvFileBoundaryStandaloneCheck();
const result = executeSandboxExecCommand(sandboxName, script, 30000);
if (!result) {
console.error("");
console.error(
` ${R}Secret-boundary check could not run against the Hermes gateway in '${sandboxName}'.${R}`,
);
console.error(" Refusing recovery to keep the validator-enforced boundary intact.");
return { refused: true, reason: "inconclusive", stderr: "" };
}
const stdoutMarker = result.stdout
.split(/\r?\n/)
.reverse()
.find((line) => line.trim().startsWith("SECRET_BOUNDARY_"));
if (stdoutMarker === SECRET_BOUNDARY_REFUSED_MARKER) {
printValidatorStderr(result.stderr);
console.error("");
console.error(
` ${R}Secret-boundary check refused recovery of Hermes gateway in '${sandboxName}'.${R}`,
);
console.error(" /sandbox/.hermes/.env contains raw secret-shaped values. Replace them with");
console.error(
" openshell:resolve:env:<name> placeholders and re-run `nemoclaw <sandbox> recover`.",
);
return { refused: true, reason: "raw-secret", stderr: result.stderr };
}
if (stdoutMarker === SECRET_BOUNDARY_OK_MARKER) {
return { refused: false };
}
if (stdoutMarker === SECRET_BOUNDARY_VALIDATOR_MISSING_MARKER) {
console.error(
` [boundary] Hermes secret-boundary validator missing in sandbox '${sandboxName}'; recover proceeded without re-evaluating /sandbox/.hermes/.env. Re-image the sandbox to enable per-run enforcement.`,
);
return { refused: false };
}
printValidatorStderr(result.stderr);
console.error("");
console.error(
` ${R}Secret-boundary check did not complete cleanly for Hermes gateway in '${sandboxName}'.${R}`,
);
console.error(
" Refusing recovery; inspect the validator output above before re-running `nemoclaw <sandbox> recover`.",
);
return { refused: true, reason: "inconclusive", stderr: result.stderr };
}

/**
* Detect and recover from a sandbox that survived a gateway restart but
* whose OpenClaw processes are not running. Also re-establishes the
* host-side dashboard port-forward when it has gone dead independently
* of the gateway. Returns an object describing the outcome:
* `{ checked, wasRunning, recovered, forwardRecovered, secretBoundaryRefused?, secretBoundaryReason? }`.
* `{ checked, wasRunning, recovered, forwardRecovered }`.
Comment thread
sandl99 marked this conversation as resolved.
*/
export function checkAndRecoverSandboxProcesses(
sandboxName: string,
Expand All @@ -608,19 +499,6 @@ export function checkAndRecoverSandboxProcesses(
}
const recoveryAgent = agentRuntime.getSessionAgent(sandboxName);
const recoveryPort = resolveSandboxDashboardPort(sandboxName);
if (running) {
const enforcement = enforceHermesSecretBoundaryOnRunningGateway(sandboxName, recoveryAgent);
if (enforcement?.refused) {
return {
checked: true,
wasRunning: true,
recovered: false,
forwardRecovered: false,
secretBoundaryRefused: true,
secretBoundaryReason: enforcement.reason,
};
}
}
if (running) {
// Gateway is alive but the host-side forward can still be dead or
// owned by another sandbox. Probe and re-establish only when
Expand Down
Loading
Loading