Skip to content

Restore missing filesystem store parent directories at runtime - #3200

Closed
zpzjzj wants to merge 1 commit into
TraceMachina:mainfrom
zpzjzj:codex/filesystem-parent-recovery
Closed

zpzjzj wants to merge 1 commit into
TraceMachina:mainfrom
zpzjzj:codex/filesystem-parent-recovery

Conversation

@zpzjzj

@zpzjzj zpzjzj commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Restore missing parent directories at runtime in FilesystemStore, then retry the failed filesystem operation once:

  • Temporary CAS file creation recovers its parent on NotFound.
  • Publishing an upload and retiring an evicted blob recover the destination parent only after confirming that the source still exists.
  • Failed publication does not insert a partial blob or replace an existing generation. Failed eviction preserves an existing content file; successful eviction preserves outstanding readers and last-reference cleanup.

This handles directory disappearance without claiming why the directory disappeared, reconstructing deleted blobs, or changing missing-source error classification.

Includes filesystem regression tests for digest and string keys, failure boundaries, retry limits, reader lifetime, and fast/slow cache fill, plus filesystem-store troubleshooting documentation.

How was this verified?

  • Based on upstream main b30bb2d4a5252f153285f7b5ec9f09f22e3a33e4.
  • Isolated Linux preflight passed for exact patch 645afa9a28677afdc62c80b94ca7fb3abeee94be; the CI carrier was checked to differ only in its CI configuration.
  • git diff --check passed.
  • Changed Rust files passed rustfmt --edition 2024 --check using the repository-pinned nightly-2026-08-26 formatter.
  • Actual builds passed: cargo +1.97.1 build --workspace --locked and bazelisk build //:nativelink --config=nl-toolchain --config=nl-quiet --lockfile_mode=error --verbose_failures --jobs=8 (Bazel 9.3.1).
  • Strict Cargo clippy passed with unchanged workspace lints and -Dwarnings: all workspace libraries/binaries, then the filesystem_store_test and fast_slow_store_test targets.
  • Both Bazel test targets passed with the unchanged upstream rust_clippy_aspect/rustfmt_aspect, their check output groups, --@rules_rust//:clippy_flag=-Dwarnings, and --nocache_test_results.
  • Cargo ran both complete affected suites: filesystem 64 passed, fast/slow 49 passed. They passed again after the baseline comparison.
  • Baseline comparison retained the new tests and replaced only the filesystem implementation with the baseline source. All four selected regressions compiled and failed: uploads_restore_removed_parent_directories, unref_restores_missing_temp_dir, publish_retries_once_and_preserves_previous_generation, and slow_fill_restores_removed_filesystem_directories. The exact patched source was restored and checked before rerunning the complete suites.
  • Broader Cargo clippy --workspace --all-targets --locked -- -Dwarnings was attempted but is blocked by an unchanged S3 test's deprecated BehaviorVersion::v2026_01_12() call. No lint allowance or unrelated source fix was added.
  • Documentation snippets, boundaries, and navigation checks passed.
  • No local Rust compilation or tests were run.
  • Local documentation anchor checking could not run because frontend dependencies are not installed. This preflight does not replace upstream's full CI; maintainer approval to run the fork workflows and the upstream documentation checks remain required.

Risk

Retries are bounded to one and limited to NotFound. Rename failures are ambiguous, so recovery checks source existence first. This does not make directory deletion or mount loss safe: concurrent external changes may still cause an ordinary I/O error. No configuration or protocol changes.

AI assistance

OpenAI Codex assisted with the implementation, regression tests, documentation, PR description, and diff review under my direction. I have reviewed and understand the changes in this PR, including the bounded directory-recovery retries, source-existence checks, and publication and eviction failure behavior. I take responsibility for this contribution and can explain the changes during review.

@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Nov 16, 2026 9:04am UTC
nativelink-aidm Ready Ready Preview Nov 16, 2026 9:04am UTC

Request Review

This branch was successfully deployed

2 active deployments
Preview – nativelink — 645afa9a Deployed Nov 16, 2026 by vercel[bot]
Preview – nativelink-aidm — 645afa9a Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant