Skip to content

Add store topology and fault-injection conformance test harness - #3010

Open
matdexir wants to merge 4 commits into
mainfrom
test-store-conformance-harness
Open

matdexir wants to merge 4 commits into
mainfrom
test-store-conformance-harness

Conversation

@matdexir

@matdexir matdexir commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Store wrappers and leaf backends have primarily been tested in isolation, often with non-default settings such as block_size: 1 on FilesystemSpec, allowing contract mismatches across composed store tiers to slip through. This change introduces store_conformance_test, a fault-injecting topology conformance harness that exercises a matrix of single-tier and multi-tier store stacks with production defaults (including FilesystemSpec's default 4 KiB block_size).

Architecture & Topology Matrix

                   +-----------------------------------------------+
                   |         6 Conformance Contract Suites         |
                   |  1. Round-trip & exact `has()` content size   |
                   |  2. Cold fast-tier fill across slow backends  |
                   |  3. Torn / invalid upload rejection & safety  |
                   |  4. Retryable error code preservation         |
                   |  5. Self-healing `remove()` cache repair      |
                   |  6. Concurrent fill & 4 KiB eviction stress   |
                   +-----------------------+-----------------------+
                                           |
                                           v
                   +-----------------------------------------------+
                   |                  VerifyStore                  |
                   |     (size + hash check, triggers remove())    |
                   +-----------------------+-----------------------+
                                           |
                                           v
                   +-----------------------------------------------+
                   |              ExistenceCacheStore              |
                   +-----------------------+-----------------------+
                                           |
                                           v
                   +-----------------------------------------------+
                   |                 FastSlowStore                 |
                   +-----------+-----------------------+-----------+
                               |                       |
              fast tier (cache)|                       |slow tier (durable)
                               v                       v
       +-------------------------------+       +-------------------------------+
       |  CacheMetrics(allow_remove)   |       |       CompressionStore        |
       |  Shard / SizePartitioning     |       |         (LZ4 frames)          |
       |  RefStore                     |       +---------------+---------------+
       +---------------+---------------+                       |
                       |                                       v
                       v                       +-------------------------------+
       +-------------------------------+       |          DedupStore           |
       |      FaultInjectingStore      |       |     (FastCDC index + chunks)  |
       |  - TruncateReadAtBytes        |       +---------------+---------------+
       |  - FailUpdateAfterBytes       |                       |
       |  - FailGetAfterBytes          |                       v
       +---------------+---------------+       +-------------------------------+
                       |                       |      FaultInjectingStore      |
                       v                       +---------------+---------------+
       +-------------------------------+                       |
       |  FilesystemStore (4 KiB blk)  |                       v
       |  or MemoryStore               |       +-------------------------------+
       +-------------------------------+       |  FilesystemStore (4 KiB blk)  |
                                               |  or MemoryStore               |
                                               +-------------------------------+

It covers six shared contracts across the topology matrix:

  1. Round-trip & has() content size: Exact has() content length and full/partial get_part round-trips across sub-block (1 B, 146 B), exact-block (4096 B), multi-block (4097 B), and multi-chunk LZ4 (68,869 B) payload boundaries.
  2. Cold fast-tier cache fills: Cold fast tier populated from every slow-tier backend (FilesystemStore, RefStore -> FilesystemStore, CompressionStore, ShardStore, and DedupStore).
  3. Torn & invalid upload safety: Rejection of truncated uploads, 0-byte uploads for non-empty digests, and torn overwrites without clobbering previously committed blobs.
  4. Retryable error code preservation: Preservation of retryable gRPC error codes (Unavailable, ResourceExhausted, Aborted) on mid-stream upload and download failures through wrapper stacks.
  5. Self-healing cache repair (StoreDriver::remove()): Truncated fast-tier reads trigger DataLoss and evict the bad fast copy via remove(), while keeping protected durable slow-tier copies intact so the next read refills cleanly.
  6. Concurrent fill & eviction stress: Concurrent cache fills and has() queries against an evicting 4 KiB-block FilesystemStore fast tier.

How was this verified?

Ran bazel test //nativelink-store:integration_tests/store_conformance_test_test under nix develop (all 6 conformance tests pass, along with Clippy, rustfmt, and pre-commit hooks).

Risk

Low. Test-only addition (nativelink-store/tests/store_conformance_test.rs and nativelink-store/BUILD.bazel) with no production code changes.

AI assistance

Built the conformance harness with Cursor Agent; reviewed and verified against the Bazel test suite.

@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 23, 2026 11:25pm UTC
nativelink-aidm Ready Ready Preview Oct 23, 2026 11:25pm UTC

Request Review

@matdexir

Copy link
Copy Markdown
Contributor Author

Store Conformance Harness Overview

                   +-----------------------------------------------+
                   |         6 Conformance Contract Suites         |
                   |  1. Round-trip & exact `has()` content size   |
                   |  2. Cold fast-tier fill across slow backends  |
                   |  3. Torn / invalid upload rejection & safety  |
                   |  4. Retryable error code preservation         |
                   |  5. Self-healing `remove()` cache repair      |
                   |  6. Concurrent fill & 4 KiB eviction stress   |
                   +-----------------------+-----------------------+
                                           |
                                           v
                   +-----------------------------------------------+
                   |                  VerifyStore                  |
                   |     (size + hash check, triggers remove())    |
                   +-----------------------+-----------------------+
                                           |
                                           v
                   +-----------------------------------------------+
                   |              ExistenceCacheStore              |
                   +-----------------------+-----------------------+
                                           |
                                           v
                   +-----------------------------------------------+
                   |                 FastSlowStore                 |
                   +-----------+-----------------------+-----------+
                               |                       |
              fast tier (cache)|                       |slow tier (durable)
                               v                       v
       +-------------------------------+       +-------------------------------+
       |  CacheMetrics(allow_remove)   |       |       CompressionStore        |
       |  Shard / SizePartitioning     |       |         (LZ4 frames)          |
       |  RefStore                     |       +---------------+---------------+
       +---------------+---------------+                       |
                       |                                       v
                       v                       +-------------------------------+
       +-------------------------------+       |          DedupStore           |
       |      FaultInjectingStore      |       |     (FastCDC index + chunks)  |
       |  - TruncateReadAtBytes        |       +---------------+---------------+
       |  - FailUpdateAfterBytes       |                       |
       |  - FailGetAfterBytes          |                       v
       +---------------+---------------+       +-------------------------------+
                       |                       |      FaultInjectingStore      |
                       v                       +---------------+---------------+
       +-------------------------------+                       |
       |  FilesystemStore (4 KiB blk)  |                       v
       |  or MemoryStore               |       +-------------------------------+
       +-------------------------------+       |  FilesystemStore (4 KiB blk)  |
                                               |  or MemoryStore               |
                                               +-------------------------------+

Related Fix PRs (split out from the initial conformance run)

@matdexir

Copy link
Copy Markdown
Contributor Author

Follow-Up Roadmap: Stopping the Regression Whack-a-Mole

An audit of the last ~150 commits and recent issues shows that recurring regressions fall into five main clusters, largely because existing tests exercise individual components against mocks at their immediate boundaries or override production defaults (like block_size: 1 on FilesystemSpec).

This PR establishes Pillar 1 (Store Topology & Fault-Injection Conformance). Below is the follow-up plan to cover the remaining failure modes across the stack:

+-------------------------------------------------------------------------+
| Pillar 1: Store Topology & Fault-Injection Conformance (This PR #3010)  |
| - Multi-layer store compositions with production defaults (4 KiB block) |
| - Expand `conformance_self_healing_cache_repair_across_wrappers` as     |
|   wrapper `remove()` PRs land (#2970/#2986, #2971/#2988, #2972/#2989,   |
|   #2973) and add `GrpcStore <-> ByteStreamServer` resume/cut fault cases|
+------------------------------------+------------------------------------+
                                     |
                                     v
+-------------------------------------------------------------------------+
| Pillar 2: In-Process Full-Stack RBE & Chaos Harness                     |
| Wire real services in-process without mocks at boundary seams:          |
|   Client -> CasServer / ByteStreamServer / ExecutionServer              |
|          -> SimpleScheduler -> WorkerApiServer                          |
|          -> LocalWorker -> RunningActionsManagerImpl                    |
|                                                                         |
| Targets recurring clusters:                                             |
| - ByteStream mid-upload disconnects, `QueryWriteStatus` resume (#2888), |
|   and `GrpcStore` slow-tier upstream restarts (#2915)                   |
| - Worker <-> Scheduler split-brain, backwards wall-clock steps (#2862), |
|   keepalive reconciliation (#2925), and command-start timeout anchoring |
|   (#2827)                                                               |
| - Concurrent CAS fast-tier eviction while `prepare_action` stages       |
|   inputs and hardlinks (#2926, #2928, #2932)                            |
| - High file-count `upload_results` under tight `OPEN_FILE_SEMAPHORE`    |
|   limits and concurrent `cleanup_action` load (#2636, #2672)            |
+------------------------------------+------------------------------------+
                                     |
                                     v
+-------------------------------------------------------------------------+
| Pillar 3: Real-Action Worker Sandbox, Namespace & Process-Tree Suite    |
| Execute real child processes through `RunningActionsManagerImpl`        |
| (instead of raw `Command::new("sh")` helpers):                          |
|                                                                         |
| Targets recurring clusters:                                             |
| - `use_mount_namespace` working directory mount mismatch causing        |
|   relative-to-absolute `rename(2)` `EXDEV` failures (#3002)             |
| - `use_namespaces` PID namespace vs. `/proc` mount mismatch (#3003)     |
| - Subprocesses escaping the action's process group via `set -m`         |
|   (e.g. Bazel's `test-setup.sh`), evading resource sampling and         |
|   timeout/kill signals (#2921 / #2922)                                  |
+-------------------------------------------------------------------------+

@matdexir
matdexir marked this pull request as ready for review October 22, 2026 14:24

@b7r6 b7r6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this conformance harness. Exercising composed stores with the default filesystem block size is useful coverage. I would like to address two gaps before merging: the repair test can pass without actually evicting the fast-tier entry, and the invalid-upload suite does not exercise oversized uploads despite describing that contract. Details and suggested assertions are inline.

Review was based on the diff and relevant store implementations; I did not run the tests.

@deus-ex-b7r6-0x02 deus-ex-b7r6-0x02 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above.

@matdexir

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Both points are addressed in 7cbf174.

This branch was successfully deployed

2 active and 1 inactive deployments
Preview – nativelink — 7cbf1747 Deployed Oct 23, 2026 by vercel[bot]
Preview – nativelink-aidm — 7cbf1747 Deployed Oct 23, 2026 by vercel[bot]
cluster-ci-benchmark — 7cbf1747 Deployed Oct 23, 2026 by matdexir via Build shared Nix package #9301
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants