Skip to content

feat: explain native stage mismatches with clawctl status --native - #264

Draft
paulcam206 wants to merge 1 commit into
paulcam206-native-stage-admissionfrom
paulcam206-native-stage-diagnostics
Draft

paulcam206 wants to merge 1 commit into
paulcam206-native-stage-admissionfrom
paulcam206-native-stage-diagnostics

Conversation

@paulcam206

@paulcam206 paulcam206 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Related: #222

What Problem This Solves

Fixes: when a launch refused a stale or damaged native dependency stage, nothing showed which packages differed, whether a package resolved from the stage or the immutable package, or what setup would change. That left #222-style mismatches hard to diagnose from logs alone. Affects clawctl status.

User Impact

User impact: clawctl status --native (and --native --json) reports whether the agent's staged native dependencies match the installed package. The report gives file counts, content IDs and, for each package, source and staged versions, short digests, and whether the package resolves from the stage or the immutable package. A mismatch exits nonzero with clawctl setup as the repair. Plain clawctl status is unchanged and does not inspect the stage.

Why This Change Was Made

This is the second of two stacked PRs. It builds on #262, which refuses a mismatched stage before Node.js starts, and has clawctl setup publish a new verified generation without disturbing held stages.

The opt-in inspection runs read-only under the agent identity, using the same inventory and verification as admission. It reports no paths, stages nothing, loads no native code, and never walks a link the verifier rejected. A rejected layout, or a recorded stage it cannot lease, reports resolvesFrom: unknown rather than guessing. The helper request and report are a narrow versioned contract with source-generated JSON, bounded to 64 packages.

Evidence

Head b04a39d on #262's head 7413c50 (base origin/main c6ef88e). The installed proof ran at this head and covers both PRs.

Installed upgrade on an isolated Windows guest (packages built at b04a39d)

Environment: one disposable Hyper-V guest (Windows build 29683 x64, 4 vCPU / 8 GiB) from a clean template plus the official VC++ 14.52 x64 redistributable, which is a prerequisite here, not under test. Packages: two MSIXes built by scripts\Build-LocalMSIX.ps1 from a clean tree at b04a39d (sourceTreeDirty: false) with the published payloads, test-signed in the guest only, and installed with Add-AppxPackage as an in-place update. A′ is 0.1.10000.21 with payload v2026.10.6 (fe06181d, fs-safe 0.25.0, where only the architecture package is native-bearing). B′ is 0.1.10000.22 with payload v2026.11.1 (4ffdd204, fs-safe 0.26.5, where @openclaw/fs-safe gains a helper .exe). Both use Node.js 24.22.1. Every package command ran in the interactive session; results are recorded as exit codes and JSON.

Upgrade, A′ then B′:

  • Under A′: clawctl setup, companion prepare, and gateway-service start succeeded, recording stage generation 35decda5….
  • After updating to B′:
    • gateway-service start adopted the running Gateway (disposition: adopted, the same attempt as before).
    • openclaw --version, clawctl pwsh --command …, and clawctl backup --json all exited 1 before starting anything. Text output said staged from different package content (expected 8f55ca68…, found d3511a8e…) [stale-source; 209 of 931 files verified, 697 missing, 28 unexpected, 25 changed] … Run 'clawctl setup' to refresh them., and backup --json returned error.type: "native_stage_stale". setup.json was unchanged by the refusals.
    • clawctl status --native --json reported state: stale-source, provenance: package-inventory, repair: clawctl setup. It showed the OpenClaw 2026.10.6 — Gateway exit code 78; SQLite migration blocked by fs-safe helper-unavailable #222 split: @openclaw/fs-safe 0.26.5 resolvesFrom: immutable (no staged copy), while @openclaw/fs-safe-win32-x64-msvc was staged at 0.25.0 against source 0.26.5.
  • clawctl setup (17 s) kept the same sandbox ID and published generation 7037cd44…. The running Gateway, the same listener process before and after, kept 35decda5…; both generations were present after repair.
  • After repair:
    • status --native reported current, 931/931 verified, with both fs-safe packages staged at 0.26.5.
    • A Node.js probe through the packaged redirect loaded the fs-safe native parent and ran its isolated worker (parentNative: true, isolatedWorker: true). Under A′ the same probe reported the worker as helper-unavailable.
    • clawctl backup --json succeeded with a verified archive.
  • After gateway-service restart, the next clawctl setup reclaimed 35decda5….

Damage behind a retained marker, with a shell held open:

  • A clawctl pwsh session stayed running on generation 7037cd44…. The fs-safe worker script was then deleted from that stage, leaving its marker in place.
  • openclaw --version exited 1 with [incomplete; 930 of 931 files verified, 1 missing, 0 unexpected, 0 changed]. status --native reported incomplete, with @openclaw/fs-safe resolving mixed. gateway-service restart --json returned error.type: "native_stage_stale".
  • clawctl setup (10 s) kept the same sandbox and published efa71e0b…. The held shell was still running and both generations were present.
  • The probe again showed parentNative: true and isolatedWorker: true, and gateway-service start succeeded. The released shell exited 0.
  • With the superseded generation's marker set read-only, clawctl setup still succeeded and left that generation whole. After the marker was made writable, the next setup reclaimed it.

User state: the same sandbox ID before the update, after each repair, and at the end. Hashes of the agent's openclaw.json and three synthetic markers with distinct random contents (DB, config, jobs) were identical before and after each repair.

Cleanup: clawctl teardown --force and package removal left 0 packages, agent accounts, profiles, and scheduled tasks; the guest was then destroyed.

Observed and unrelated to this change: when two clawctl pwsh invocations share one console, the second exits only after the first ends, although its command has already completed. The host's console-input relay (MxcSdkSessionClient) is unchanged here. Separate consoles complete independently.

Local lanes

  • .\scripts\Test-DotNetQuality.ps1: passed at 7413c50 and b04a39d.
  • dotnet test .\OpenClaw.Gateway.MSIX.slnx --configuration Release: 1711/1711 at 7413c50, 1716/1716 at b04a39d.
  • .\scripts\Test-NativeAotCli.Tests.ps1: passed at both heads. At b04a39d it also round-trips --inspect-native and the status nativeStage JSON under NativeAOT.
  • .\scripts\Test-SessionNativeStagerEfs.Tests.ps1 -ApplicationDirectory <EFS-encrypted fixture>\app: passed at both heads.
  • .\scripts\Test-DocReferences.ps1: 0 findings. git -c core.whitespace= diff --check: clean.

Tests added here:

  • NativeStageUpgradeScenarioTests:
    • status --native --json reports stale-source, with @openclaw/fs-safe resolving from immutable and no agent paths;
    • it changes nothing;
    • plain status does not inspect;
    • it reports current after repair.
  • SessionNativeStageInspectorTests:
    • a cyclic junction in a staged package reports unsupported-layout with resolvesFrom: unknown, without being walked;
    • a recorded stage whose marker is gone reports missing with resolvesFrom: unknown. This test was mutation-checked: it fails if resolution is inferred from skipped reads.
  • ClawCtlConsoleTests: human rendering.
  • AOT smoke: --inspect-native and the status JSON.

Not run: ARM64 lanes and an ARM64 installed upgrade.

@paulcam206
paulcam206 added this pull request to stack #265 November 17, 2026 02:25
@clawsweeper

clawsweeper Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Nov 17, 2026
@clawsweeper

clawsweeper Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

Adds optional native-dependency inspection to clawctl status, with human and JSON reports.

Example: The installed fs-safe package is 0.26.5, but its staged architecture package remains at 0.25.0.

  • Before: Status does not identify the native-package mismatch.
  • After: status --native --json reports stale-source, the package versions and resolution locations, and clawctl setup as the repair.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) Useful, focused diagnostics with installed behavior evidence and the prior blocker resolved.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (terminal): Exact-head installed x64 Windows evidence exercises status --native through the packaged CLI before and after an in-place payload update, stage damage, and setup repair; reported versions, resolution locations, counts, and states match the changed inspector. Supplied quality, Release tests, NativeAOT, EFS, and documentation lanes passed; ARM64 installed execution remains unrun.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Feature · Worth it: Yes · Fix scope: Complete
User problem: Users cannot readily identify which staged native packages differ from the installed application.
Reason: The collaborator-authored direction addresses a demonstrated support need through a bounded opt-in report; it uses Related wording for the broader recovery incident.

Merge readiness

✅ Ready for maintainer review

Keep open: this collaborator-authored feature is absent from main, and the previous reporting defect is fixed; no new blocking defect was found.

Priority: P2
Reviewed head: b04a39d6b4d4cf1d284fcb69e95121b093939b2f

Before merge

None.

Findings

None.

Agent review details

How this fits together

The launcher sends an inspection request to the existing isolated-session helper, which compares installed inventory with the recorded native stage and returns facts for status rendering.

flowchart LR
  A[clawctl status native] --> B[Launcher]
  B --> C[Session protocol]
  C --> D[Agent inspector]
  E[Installed inventory] --> D
  F[Recorded native stage] --> D
  D --> G[Human and JSON output]
Loading

Technical review

Best possible solution:

Retain this focused diagnostic extension on the shared verifier and land it after its admission-and-repair dependency.

Do we have a high-confidence way to reproduce the issue?

The reporting need is supported by the linked incident and supplied installed-upgrade output; this review inspected source and evidence without running target code.

Is this the best way to solve the issue?

The change extends the existing command tree, session executor, protocol, verifier, and renderers while keeping repair under setup.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against c6ef88e098ae.

Provenance checked

  • Status results and rendering keeps the original intent (b904c90: Centralize semantic results and readable Spectre.Console output instead of composing presentation in command handlers.)
  • Recorded-session inspection keeps the original intent (19d1f85: Make isolated-session ownership and lifecycle explainable; the new operation requires the existing staged helper rather than staging one during status.)
  • Native-stage verification keeps the original intent (7413c50: The base PR verifies staged dependencies before launch and reserves repair for explicit setup; this PR consumes that decision without changing admission.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

Likely related people:

  • Paul Campbell: Raw commit b904c90 adds src/OpenClaw.Launcher/ClawCtlConsole.cs:280 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: b904c90c1439; files: src/OpenClaw.Launcher/ClawCtlConsole.cs)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.
  • remove rating: 🦪 silver shellfish: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This opt-in diagnostic improves troubleshooting of native-stage mismatches without changing ordinary status or launch admission.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • proof: sufficient: Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle)
  • reviewed 2026-11-17T02:41:01.289Z sha 9a6f928 :: needs changes before merge. :: [P2] [P2] Preserve unknown resolution when the stage cannot be leased

Reviewed November 16, 2026, 10:54 PM ET / November 17, 2026, 03:54 UTC (Revision 2).

When a launch reports native-stage-stale, or a native helper fails inside
the session, there was no way to see how the agent's staged native copies
compared with the installed package without opening an agent shell and
probing by hand.

clawctl status --native asks the session helper, under the agent identity,
to run the same inventory and verification launch admission uses and report
the state, file counts, content identifiers, and for each native-bearing
package its source and staged versions and digests and whether it resolves
from the stage or the immutable package. It never stages, repairs, loads
native code, or reports paths; when no fresh inspection is possible it says
why instead of claiming a state. A mismatch exits 1 and names clawctl setup.
Plain status is unchanged.

Related: #222
@paulcam206
paulcam206 force-pushed the paulcam206-native-stage-diagnostics branch from 9a6f928 to b04a39d Compare November 17, 2026 03:23
@paulcam206
paulcam206 changed the base branch from paulcam206-native-stage-repair to paulcam206-native-stage-admission November 17, 2026 03:23
@paulcam206
paulcam206 added this pull request to stack #267 November 17, 2026 03:23
@paulcam206

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant