Decouple the flush path from the read path

[armstrong0704]

Problem

Previously, flushing the EphemeralFile to disk held a write lock on the struct. During slow disk IO (10ms - 50ms+), this blocked all concurrent readers (GetPage), causing latency spikes and throughput collapse.

See #12172 for the full context.

Summary of changes

1. Decouple the 'Frozen' flush buffer from the 'Mutable' tail.
2. Use arc_swap::ArcSwap to publish the frozen buffer atomically. Readers access it wait-free.
3. Add disk_committed_offset to distinguish between data in the buffer vs. disk, preventing race conditions.
4. Add writer_lock to serialize concurrent writers (WAL append-only integrity) without blocking readers.

Verification & Benchmarks (Local NVMe + Synthetic 50ms Latency):

• Read Throughput: ~7.4x improvement (43k -> 323k ops/sec).
• Write Throughput: ~1600-2300 MB/s (Comparable to baseline).
• Fixes pageserver: improve IO concurrency for in-memory layers #12172

[VladLazar]

Hey. Thanks for the PR. Appreciate you taking an interest in this and taking the time to work on this.

In the current form the changes here are very broad. This part of the code base is rather delicate (there's a number of non-obvious invariants) and we've tried to incrementally evolve it over time with particular attention to correctness. As it stands, convincing myself this is all correct would take a substantial amount of time and it carries a lot of risk.

If you're interested I suggest a rework that preserves the overall code structure and focuses on the issue of holding the writer lock while reading from the flush buffer.
I think changing BufferedWriter::maybe_flushed to ArcSwap<Option<FullSlice<B::IoBuf>>> and making BufferedWriter::inspect_maybe_flushed to return the current value in the arc swap would be sufficient. Then you could drop the write lock after reading from the mutable buffer.

[armstrong0704]

@VladLazar Thanks for the review and recommendations. I have updated the PR based on your feedback.

• Refactor: I moved the ArcSwap logic inside BufferedWriter instead of rewriting EphemeralFile. This preserves the existing write pipelining and flow control while exposing the flushing buffer wait-free.

• Performance: The results are even better with this cleaner abstraction. Benchmarks under simulated load (50ms disk latency) show a ~28x improvement in read throughput compared to main (readers remain completely non-blocking during flushes).