nfsproxy: isolate file-handle caches by sandbox/mount point #3555

Symptom

The NFS proxy currently uses a single global file-handle cache shared by every sandbox mounted on the same node.

When one sandbox lists a directory containing a large number of files, READDIR and READDIRPLUS generate or access many file handles in the shared LRU. This can evict handles that are still being used by unrelated sandboxes.

If an affected sandbox is mounted with client-side caching enabled, the client may later reuse one of those evicted handles. The NFS proxy can no longer resolve it and returns nfs.NFSStatusStale (NFS3ERR_STALE).

As a result, filesystem activity in one sandbox can cause stale-handle errors in another sandbox.

Root cause

The caching handler is created once for the entire NFS proxy:

handler = helpers.NewCachingHandler(handler, cacheLimit)

helpers.CachingHandler maps opaque NFS file handles to their corresponding filesystem and path through an LRU cache. Because this handler wraps the global proxy handler, all sandboxes and mount points share:

• The same cache capacity.
• The same eviction policy.
• The same cache synchronization and locks.
• The same handle lookup and LRU maintenance overhead.

There is no ownership boundary between cache entries belonging to different sandboxes. A workload that lists or accesses many files can therefore consume most of the cache and evict entries owned by other sandboxes.

The current mitigation in #3549 makes the cache limit configurable and increases its size. This reduces the probability of eviction, but it does not provide workload isolation.

Increasing the global cache also has a performance cost. Concurrent directory listings from multiple sandboxes contend on the same cache locks, while operations such as handle lookup and LRU maintenance become more expensive as the shared cache grows.

Why it matters

A global cache creates a noisy-neighbor problem between otherwise independent sandboxes:

• A large directory listing in one sandbox can cause NFSStatusStale errors in another sandbox.
• Cache sizing must account for the aggregate peak workload of every sandbox on the node.
• Increasing the cache size only postpones eviction rather than eliminating cross-sandbox interference.
• A larger shared cache increases lock contention and lookup overhead during concurrent file operations.
• Performance becomes less predictable as sandbox density and filesystem concurrency increase.

A preliminary per-sandbox cache prototype showed a significant improvement. With a cache limit of 64 * 1024 and directories containing more than 5,000 files, concurrent ls operations completed approximately 30 times faster than with the global-cache implementation.

The exact result depends on concurrency and directory layout, but it demonstrates that cache isolation addresses both correctness and scalability rather than only increasing capacity.

Proposed fix

Allocate an independent file-handle cache for each sandbox or mount point:

• Use the sandbox or mount-point identity as the cache shard key.
• Route handle creation and lookup to the corresponding cache.
• Prevent cache eviction and lock contention from affecting other sandboxes.
• Remove the cache when the sandbox or mount point is released.