fetch_fx_minute: write cache parquets atomically

[zachisit]

User story

As someone building a multi-GB local FX cache with scripts/fetch_fx_minute.py,
I want each year's parquet to be written atomically,
so that a Ctrl-C, OOM kill, or crash mid-write cannot leave a truncated file that later runs silently trust as a valid cache hit.

Context

scripts/fetch_fx_minute.py was added in #253 (approved by @shardul0701, review). He raised this as a non-blocking follow-up:

Two possible future niceties, not blockers: retrying transient HTTP 5xx responses the same way as 429s, and atomic parquet writes to protect against a process kill mid-write.

Current behaviour

load_pair() treats existence of the parquet as proof of validity:

path = os.path.join(cache_dir, f"{pair.replace(':', '_')}_{year}.parquet")
if os.path.exists(path):
    frames.append(pd.read_parquet(path))
    continue
...
df.to_parquet(path)

to_parquet writes in place. If the process dies partway through, a partial file is left at the final path — and the next run takes the os.path.exists branch and reads it.

Why it matters

This is the failure mode I'd worry about most, because of how it fails:

• A full 2010–2026 pull across 14 symbols is ~1.4 GB across 238 files, taking ~15 minutes. Interrupting it is normal, not exceptional.
• The cache is explicitly designed to be resumable — re-running the same command is the documented recovery path. That behaviour is what makes a truncated file dangerous: it will never be re-fetched.
• A truncated parquet either raises a confusing decode error far from the cause, or — worse — reads back short, silently handing back a year with missing bars. Research built on a silently-short year is wrong in a way that is very hard to trace back to a fetch that was interrupted weeks earlier.

Not hypothetical for this codebase: the 1-min files are large enough that a write is a multi-second window per file.

Proposed change

Write to a temporary file in the same directory, then os.replace() into place. os.replace is atomic on POSIX and Windows when source and destination are on the same filesystem — hence same-directory temp, not /tmp.

Roughly:

tmp = f"{path}.tmp.{os.getpid()}"
df.to_parquet(tmp)
os.replace(tmp, path)          # atomic; readers see old-or-new, never partial

With cleanup of the temp file if to_parquet raises, so a failed write doesn't leave .tmp litter in the cache dir.

Acceptance criteria

• Each year's parquet is written via temp-file + os.replace() in the same directory
• A failure during to_parquet leaves no file at the final path and no orphaned temp file
• An interrupted write does not produce a path that a later load_pair() treats as a cache hit
• Concurrent fetchers writing the same (pair, year) cannot corrupt each other (PID or tempfile.mkstemp in the temp name)
• Existing cache-hit / cache-miss behaviour and the tz round trip are unchanged
• Tests added to tests/test_fetch_fx_minute.py — simulate to_parquet raising and assert no final file and no temp residue; extend the TestLoadPair group
• pytest tests/ stays green

Notes

• Considered and rejected: validating cache files on read (e.g. re-opening and checking row counts). That costs I/O on every cache hit to defend against a rare write failure, and it can't distinguish "truncated" from "genuinely short year" — Polygon coverage is legitimately non-uniform (C:XAUUSD has no 2012 data at all). Fixing the write is cheaper and unambiguous.
• Empty years already correctly write no file (test_empty_year_is_skipped_and_not_cached), so this only concerns the non-empty write path.

Related

• feat: cached 1-minute Polygon FX bar fetcher #253 — PR that introduced the script
• fetch_fx_minute: retry transient HTTP 5xx like 429s #254 — sibling follow-up from the same review: retry transient HTTP 5xx