caveman-compress truncates target file to 0 bytes with no backup on Windows (non-ASCII content)

[duyvo3001]

Bug: /caveman-compress truncates target file to 0 bytes with no backup on Windows (non-ASCII content)

Summary

On Windows, caveman-compress can destroy the file it is compressing — leaving it empty (0 bytes) with no .original.md backup — whenever the file (or the compressed output) contains any character outside cp1252 (e.g. ⚠️, →, —, smart quotes). The file is unrecoverable unless the user has a copy elsewhere.

Environment

• OS: Windows 11
• Python: 3.14.3
• Plugin: caveman @ commit 0d95a81d35a9
• Path: no ANTHROPIC_API_KEY set → claude --print CLI fallback path
• File: a personal RTK.md (~964 bytes) containing ⚠️ and →

Repro

1. Take any prose .md containing a non-cp1252 char (e.g. a line with ⚠️ or an → arrow).
2. Run /caveman-compress <file> on Windows (default codepage, no PYTHONUTF8).
3. Observed:

   Processing: ...\RTK.md
   Compressing with Claude...
   ❌ Error: 'charmap' codec can't encode characters in position 648-649: character maps to <undefined>
   

4. The target file is now 0 bytes. No *.original.md backup exists. Re-running then reports ❌ Refusing to compress: file is empty or whitespace-only.

Root cause

scripts/compress.py correctly pins encoding="utf-8" on the subprocess call to claude (lines 157–165, per the issue #152 comment) — but every file I/O call omits encoding=, so on Windows they default to cp1252:

• L249 original_text = filepath.read_text(errors="ignore")
• L303 backup_path.write_text(original_text)
• L304 backup_path.read_text(errors="ignore")
• L313 filepath.write_text(compressed)
• L331 filepath.write_text(original_text) (the "restore original" path — itself fails on Windows)
• L340 filepath.write_text(compressed)

Two distinct defects follow:

1. Data loss (0-byte file). Path.write_text() opens the file in "w" mode, which truncates immediately, and then encodes the string. When the string contains a char cp1252 can't encode, .write() raises UnicodeEncodeError after truncation — leaving a 0-byte file. Because the same missing-encoding bug hits the backup write (L303) too, the backup either never completes or is corrupt, so there is no recovery copy. The restore original fallback (L331) has the identical flaw, so even the built-in recovery path fails on Windows.

2. Silent corruption (latent). L249 reads a UTF-8 file as cp1252 with errors="ignore", producing mojibake / dropped bytes. Even when a write doesn't crash, the "original" that gets backed up and the body sent to Claude are already corrupted.

Suggested fix

Add encoding="utf-8" to every read_text / write_text in compress.py — the same fix already applied to the subprocess:

original_text = filepath.read_text(encoding="utf-8", errors="ignore")   # L249
backup_path.write_text(original_text, encoding="utf-8")                  # L303
backup_readback = backup_path.read_text(encoding="utf-8", errors="ignore")  # L304
filepath.write_text(compressed, encoding="utf-8")                        # L313, L340
filepath.write_text(original_text, encoding="utf-8")                     # L331

Optional hardening so a crash can never truncate the primary:

• Write to a temp file in the same dir, fsync, then os.replace() onto the target (atomic on Windows and POSIX). A failed encode then leaves the original untouched.
• Drop errors="ignore" on the source read, or warn — silently dropping bytes corrupts the "original" backup.

Impact

High. Runs against a user's CLAUDE.md, todos, or notes — exactly the prose files the skill targets — and warehouse/dev docs routinely contain →, —, ✓, emoji. On Windows this is a straight-up unrecoverable delete of the input.

[scythe-327]

I looked into this and I think the UTF-8 encoding fix is necessary, but it may not be sufficient to prevent future data-loss issues.

I'd like to implement:

• Explicit encoding="utf-8" for all read_text() / write_text() calls.
• Replace direct writes to the target file with an atomic write (tempfile + os.replace) so encoding errors or interrupted writes cannot truncate - the original file.
• Remove errors="ignore" on the initial source read (or fail with a clear error if the file isn't valid UTF-8).

This would make the fix address both the Windows encoding issue and the broader risk of in-place file corruption.

If this approach sounds good, I'd be happy to open a PR.