remotemount.refresh: live mount updates without unmounting#3330
Open
cpuhrsch wants to merge 1 commit intometa-pytorch:mainfrom
Open
remotemount.refresh: live mount updates without unmounting#3330cpuhrsch wants to merge 1 commit intometa-pytorch:mainfrom
cpuhrsch wants to merge 1 commit intometa-pytorch:mainfrom
Conversation
7409367 to
f5df95c
Compare
Summary: Add MountHandler.refresh(sourcepath) for atomically updating FUSE mount data while the filesystem stays mounted. Open file handles remain valid and subsequent reads see the new data immediately. ## Walkthrough **Rust (chunked_fuse.rs):** Extract FsData struct (metadata + chunks + chunk_size) and wrap in Arc<RwLock<FsData>> shared between ChunkedFuseFs and PyMountHandle. All FUSE methods acquire a read lock (non-blocking with each other). PyMountHandle.refresh() acquires the write lock and atomically swaps the data. TTL is set to zero so the kernel always re-validates metadata after refresh. **Rust (tls_sender.rs):** Fix pre-existing flake where tcp shutdown(Write) failed with ENOTCONN when the receiver closed first. This is benign (data already sent) — now ignored instead of fatal. **Python (remotemount.py):** FUSEActor.mount() now mounts an empty FUSE filesystem and calls _do_refresh() to populate it, sharing the flush/cache/swap logic with FUSEActor.refresh_mount(). MountHandler uses a shared _sync() method for open() and refresh() that packs the source directory, diffs block hashes, transfers dirty blocks, then mounts or refreshes. refresh(sourcepath) requires the caller to pass the source path again to prevent accidentally refreshing with a forgotten path. **Tests:** 5 FUSE unit tests (TestFuseRefresh) covering data refresh, metadata refresh, open file handles across refresh, repeated refreshes, and end-to-end pack+refresh+read. 4 integration tests with actors covering refresh, open handles, no-change, and add-file scenarios. **Benchmark:** bench_fuse_read.py measures FUSE read latency and throughput vs tmpfs baseline. Reviewed By: zdevito Differential Revision: D98953121
f5df95c to
d5485de
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Add MountHandler.refresh(sourcepath) for atomically updating FUSE mount
data while the filesystem stays mounted. Open file handles remain valid
and subsequent reads see the new data immediately.
Walkthrough
Rust (chunked_fuse.rs): Extract FsData struct (metadata + chunks +
chunk_size) and wrap in Arc<RwLock> shared between ChunkedFuseFs
and PyMountHandle. All FUSE methods acquire a read lock (non-blocking
with each other). PyMountHandle.refresh() acquires the write lock and
atomically swaps the data. TTL is set to zero so the kernel always
re-validates metadata after refresh.
Rust (tls_sender.rs): Fix pre-existing flake where tcp
shutdown(Write) failed with ENOTCONN when the receiver closed first.
This is benign (data already sent) — now ignored instead of fatal.
Python (remotemount.py): FUSEActor.mount() now mounts an empty FUSE
filesystem and calls _do_refresh() to populate it, sharing the
flush/cache/swap logic with FUSEActor.refresh_mount(). MountHandler uses
a shared _sync() method for open() and refresh() that packs the source
directory, diffs block hashes, transfers dirty blocks, then mounts or
refreshes. refresh(sourcepath) requires the caller to pass the source
path again to prevent accidentally refreshing with a forgotten path.
Tests: 5 FUSE unit tests (TestFuseRefresh) covering data refresh,
metadata refresh, open file handles across refresh, repeated refreshes,
and end-to-end pack+refresh+read. 4 integration tests with actors
covering refresh, open handles, no-change, and add-file scenarios.
Benchmark: bench_fuse_read.py measures FUSE read latency and
throughput vs tmpfs baseline.
Reviewed By: zdevito
Differential Revision: D98953121