ci(e2e): retry live vitest failures

[cv]

Summary

Add automatic retries for live Vitest E2E projects so transient external-service failures get a bounded second chance in CI. The default is 2 retries in CI (3 total attempts), while local runs stay single-shot unless NEMOCLAW_E2E_RETRIES is set.

Changes

• Add resolveE2ERetryCount() to centralize E2E retry defaults and overrides.
• Configure e2e-scenarios-live and e2e-branch-validation Vitest projects to use the resolved retry count.
• Add support tests for retry configuration and document the retry behavior in the E2E Vitest fixture guide.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Verification

• PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
• Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
• Targeted tests pass for changed behavior
• Full npm test passes (broad runtime changes only)
• Tests added or updated for new or changed behavior
• No secrets, API keys, or credentials committed
• Docs updated for user-facing behavior changes
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

• Tests

  • Implemented configurable E2E retry behavior to improve reliability, including CI vs local defaults.
  • Added validation to ensure retry settings apply only to live E2E projects, with invalid overrides ignored and values clamped to a safe maximum.

• Documentation

  • Updated E2E scenario documentation to explain how to enable local retries for external-service flakiness using NEMOCLAW_E2E_RETRIES and how CI retry defaults work.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e53843a6-dca4-4965-bdc5-01a55e4bf67c

📥 Commits

Reviewing files that changed from the base of the PR and between 2058099 and af29295.

📒 Files selected for processing (3)

• test/e2e-scenario/docs/README.md
• test/e2e-scenario/support-tests/e2e-live-project-config.test.ts
• test/helpers/e2e-retries.ts

✅ Files skipped from review due to trivial changes (1)

• test/e2e-scenario/docs/README.md

🚧 Files skipped from review as they are similar to previous changes (2)

• test/helpers/e2e-retries.ts
• test/e2e-scenario/support-tests/e2e-live-project-config.test.ts

---

📝 Walkthrough

Walkthrough

Adds a resolveE2ERetryCount utility function that derives an E2E retry count from NEMOCLAW_E2E_RETRIES, CI environment flags, or a local default. The function is imported and applied as retry in the e2e-branch-validation Vitest project config. New test cases validate the function's behavior and project-level retry gating. The README is updated with matching usage instructions.

Changes

E2E Retry Configuration

Layer / File(s)	Summary
resolveE2ERetryCount utility and Vitest wiring test/helpers/e2e-retries.ts, vitest.config.ts	Creates resolveE2ERetryCount with CI/local defaults and NEMOCLAW_E2E_RETRIES override parsing. Imports it in vitest.config.ts, derives e2eRetryCount, sets retry: e2eRetryCount on the e2e-branch-validation project, and adds explanatory comments in the e2e-scenarios-live section.
Test assertions and README documentation test/e2e-scenario/support-tests/e2e-live-project-config.test.ts, test/e2e-scenario/docs/README.md	Extends ProjectConfig.test with optional retry, adds test cases verifying per-project retry gating and resolveE2ERetryCount output for all environment combinations. Updates the README example command to set NEMOCLAW_E2E_RETRIES=2 and documents CI vs local retry defaults.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

area: e2e, chore

Poem

A rabbit hops through CI lanes,
Where flaky tests bring fleeting pains.
"Retry!" it cries, with NEMOCLAW_E2E_RETRIES=2,
Three chances now to see things through.
No more false failures, no more dread—
The carrot waits at green ahead! 🥕✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'ci(e2e): retry live vitest failures' directly describes the main objective of the PR—adding automatic retry functionality for live Vitest E2E projects to handle transient failures in CI environments, which is the core change across all modified files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/e2e-vitest-retries

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the ci/e2e-vitest-retrie... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.

File	main	ci/e2e-vitest-retrie... cfff4f5	+/-
nemoclaw/src/se...cret-scanner.ts	—	100%	—
nemoclaw/src/commands/slash.ts	—	100%	—
nemoclaw/src/li...bprocess-env.ts	—	100%	—
nemoclaw/src/bl...eprint/state.ts	—	98%	—
nemoclaw/src/onboard/config.ts	—	98%	—
nemoclaw/src/bl...int/snapshot.ts	—	97%	—
nemoclaw/src/bl...print/runner.ts	—	95%	—
nemoclaw/src/co...ration-state.ts	—	94%	—
nemoclaw/src/bl...ate-networks.ts	—	94%	—
nemoclaw/src/index.ts	—	94%	—

TypeScript / code-coverage/cli

The overall coverage in the ci/e2e-vitest-retrie... branch is 46%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.

File	main	ci/e2e-vitest-retrie... cfff4f5	+/-
src/lib/state/o...oard-session.ts	—	90%	—
src/lib/inference/local.ts	—	76%	—
src/lib/sandbox/config.ts	—	72%	—
src/lib/actions...dbox/rebuild.ts	—	67%	—
src/lib/onboard/preflight.ts	—	64%	—
src/lib/actions...licy-channel.ts	—	56%	—
src/lib/state/sandbox.ts	—	55%	—
src/lib/policy/index.ts	—	49%	—
src/lib/onboard...er-gpu-patch.ts	—	44%	—
src/lib/onboard.ts	—	18%	—

---

_{Updated June 20, 2026 15:43 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.}

[github-actions]

E2E Advisor Recommendation

Required E2E: None
Optional E2E: e2e-vitest-support, diagnostics-vitest

Dispatch hint: diagnostics-vitest

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• None. No required live E2E is recommended because the changes are limited to E2E Vitest harness retry configuration, support tests, and documentation; they do not change installer/onboarding, credentials, sandbox lifecycle, network policy, inference routing, deployment, or assistant user-flow runtime code.

Optional E2E

• e2e-vitest-support (low): Optional fast confidence check for the changed Vitest E2E project configuration and retry helper contract; it directly includes the modified support test without provisioning live infrastructure.
• diagnostics-vitest (medium): Optional smoke check that a free-standing live Vitest E2E job still starts under the updated e2e-scenarios-live project configuration and CI retry setting. Not merge-blocking because the product/runtime paths are unchanged.

New E2E recommendations

• None.

Dispatch hint

• Workflow: e2e-vitest-scenarios.yaml
• jobs input: diagnostics-vitest

[github-actions]

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: e2e-scenarios-all
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

• gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

• e2e-scenarios-all: The PR changes shared Vitest live E2E project configuration by adding retry behavior in vitest.config.ts and the shared retry helper used by live E2E projects. This affects execution semantics for all live registry-backed scenarios, so the full Vitest scenario fan-out should run.
  • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Optional Vitest E2E scenarios

• None.

Relevant changed files

• test/e2e-scenario/docs/README.md
• test/e2e-scenario/support-tests/e2e-live-project-config.test.ts
• test/helpers/e2e-retries.ts
• vitest.config.ts

[github-actions]

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 0 still apply, 0 new items found

Consider writing more tests for

• **Runtime validation** — Exercise a minimal Vitest project with a test that fails once and then passes to confirm the live-project `retry` setting produces the expected additional attempt under CI-like env.. The helper and Vitest project wiring have good targeted tests. Because the changed behavior affects live infrastructure projects, lightweight runtime validation of actual Vitest retry semantics would further increase confidence, without relying on external E2E job status here.
• **Runtime validation** — Add a lightweight contract around the branch-validation project showing retries do not bypass the existing opt-in gate when Brev credentials or the workflow sentinel are absent.. The helper and Vitest project wiring have good targeted tests. Because the changed behavior affects live infrastructure projects, lightweight runtime validation of actual Vitest retry semantics would further increase confidence, without relying on external E2E job status here.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

[github-actions]

Vitest E2E Scenario Results — ❌ Some jobs failed

Run: 27844111051
Workflow ref: ci/e2e-vitest-retries
Requested scenarios: (default — all supported)
Requested jobs: (default — all free-standing when no scenarios are requested)
Summary: 30 passed, 15 failed, 0 skipped

Job	Result
bedrock-runtime-compatible-anthropic-vitest	✅ success
channels-add-remove-vitest	✅ success
cloud-inference-vitest	✅ success
cloud-onboard-vitest	✅ success
common-egress-agent-vitest	✅ success
concurrent-gateway-ports-vitest	✅ success
credential-migration-vitest	✅ success
credential-sanitization-vitest	❌ failure
diagnostics-vitest	✅ success
double-onboard-vitest	✅ success
full-e2e-vitest	✅ success
gateway-drift-preflight-vitest	✅ success
gateway-guard-recovery	❌ failure
gateway-health-honest-vitest	✅ success
generate-matrix	✅ success
gpu-double-onboard-vitest	✅ success
hermes-e2e-vitest	❌ failure
hermes-root-entrypoint-smoke-vitest	✅ success
inference-routing-vitest	✅ success
issue-2478-crash-loop-recovery-vitest	✅ success
issue-4434-tui-unreachable-inference-vitest	❌ failure
issue-4462-scope-upgrade-approval-vitest	✅ success
launchable-smoke-vitest	❌ failure
live-scenarios	❌ failure
messaging-compatible-endpoint-vitest	✅ success
messaging-providers-vitest	❌ failure
model-router-provider-routed-inference-vitest	❌ failure
network-policy-vitest	❌ failure
onboard-negative-paths-vitest	✅ success
onboard-repair-vitest	✅ success
onboard-resume-vitest	✅ success
openclaw-inference-switch-vitest	✅ success
openclaw-skill-cli-vitest	✅ success
openclaw-tui-chat-correlation-vitest	❌ failure
openshell-version-pin-vitest	✅ success
rebuild-openclaw-vitest	❌ failure
runtime-overrides-vitest	✅ success
sandbox-rebuild-vitest	❌ failure
sandbox-survival-vitest	❌ failure
sessions-agents-cli-vitest	✅ success
shields-config-vitest	❌ failure
skill-agent-vitest	❌ failure
snapshot-commands-vitest	✅ success
state-backup-restore-vitest	✅ success
token-rotation-vitest	✅ success

Failed jobs: credential-sanitization-vitest, gateway-guard-recovery, hermes-e2e-vitest, issue-4434-tui-unreachable-inference-vitest, launchable-smoke-vitest, live-scenarios, messaging-providers-vitest, model-router-provider-routed-inference-vitest, network-policy-vitest, openclaw-tui-chat-correlation-vitest, rebuild-openclaw-vitest, sandbox-rebuild-vitest, sandbox-survival-vitest, shields-config-vitest, skill-agent-vitest. Check run artifacts for logs.

[github-actions]

Vitest E2E Scenario Results — ❌ Some jobs failed

Run: 27844111051
Workflow ref: ci/e2e-vitest-retries
Requested scenarios: (default — all supported)
Requested jobs: (default — all free-standing when no scenarios are requested)
Summary: 30 passed, 15 failed, 0 skipped

Job	Result
bedrock-runtime-compatible-anthropic-vitest	✅ success
channels-add-remove-vitest	✅ success
cloud-inference-vitest	✅ success
cloud-onboard-vitest	✅ success
common-egress-agent-vitest	✅ success
concurrent-gateway-ports-vitest	✅ success
credential-migration-vitest	✅ success
credential-sanitization-vitest	❌ failure
diagnostics-vitest	✅ success
double-onboard-vitest	✅ success
full-e2e-vitest	✅ success
gateway-drift-preflight-vitest	✅ success
gateway-guard-recovery	❌ failure
gateway-health-honest-vitest	✅ success
generate-matrix	✅ success
gpu-double-onboard-vitest	✅ success
hermes-e2e-vitest	❌ failure
hermes-root-entrypoint-smoke-vitest	✅ success
inference-routing-vitest	✅ success
issue-2478-crash-loop-recovery-vitest	✅ success
issue-4434-tui-unreachable-inference-vitest	❌ failure
issue-4462-scope-upgrade-approval-vitest	✅ success
launchable-smoke-vitest	❌ failure
live-scenarios	❌ failure
messaging-compatible-endpoint-vitest	✅ success
messaging-providers-vitest	❌ failure
model-router-provider-routed-inference-vitest	❌ failure
network-policy-vitest	❌ failure
onboard-negative-paths-vitest	✅ success
onboard-repair-vitest	✅ success
onboard-resume-vitest	✅ success
openclaw-inference-switch-vitest	✅ success
openclaw-skill-cli-vitest	✅ success
openclaw-tui-chat-correlation-vitest	❌ failure
openshell-version-pin-vitest	✅ success
rebuild-openclaw-vitest	❌ failure
runtime-overrides-vitest	✅ success
sandbox-rebuild-vitest	❌ failure
sandbox-survival-vitest	❌ failure
sessions-agents-cli-vitest	✅ success
shields-config-vitest	❌ failure
skill-agent-vitest	❌ failure
snapshot-commands-vitest	✅ success
state-backup-restore-vitest	✅ success
token-rotation-vitest	✅ success

Failed jobs: credential-sanitization-vitest, gateway-guard-recovery, hermes-e2e-vitest, issue-4434-tui-unreachable-inference-vitest, launchable-smoke-vitest, live-scenarios, messaging-providers-vitest, model-router-provider-routed-inference-vitest, network-policy-vitest, openclaw-tui-chat-correlation-vitest, rebuild-openclaw-vitest, sandbox-rebuild-vitest, sandbox-survival-vitest, shields-config-vitest, skill-agent-vitest. Check run artifacts for logs.