Data loss prevention

[notABot101010]

As mentioned in #1 , there is a possibility of data loss if the process / node crash when the data is buffered in memory.

It was mentioned that a Write Ahead Log (WAL) is planned.

I would also recommend considering SQLite which seems way easier to implement, and can easily be migrated to a WAL later if using the correct abstraction.

SQLite's performance allows to reduce batch interval to something like 1-5 ms and will allow creating parquet files with even better optimized size, reducing the need for further compaction.

[xe-nvdk]

Good point, we actually started digging into this yesterday, and I completely agree with you.
We already use SQLite internally, so extending it for this purpose shouldn’t be too complex.

[xe-nvdk]

Hey @notABot101010, good news, we also implemented WAL. There is some details that you can find in the repo too.

Current WAL Implementation

Arc currently implements a custom binary WAL with the following characteristics:

• Per-worker WAL files: Each worker has its own WAL file to avoid lock contention
• Binary format: MessagePack serialization with CRC32 checksums
• Configurable sync modes: fsync (maximum safety), fdatasync (balanced), async (fastest)
• Automatic rotation: Based on size (100MB) or age (1 hour)
• Disabled by default: To maximize throughput

Performance Impact

Tested on M3 Max (14 cores) with MessagePack binary protocol:

Configuration	Throughput	Data Loss Risk
WAL Disabled (default)	2.01M rec/s	0-5 seconds
WAL async	1.67M rec/s (-17%)	<1 second
WAL fdatasync	1.63M rec/s (-19%)	Near-zero
WAL fsync	1.67M rec/s (-17%)	Zero

Next Steps

We're evaluating a hybrid approach:

• Implement SQLiteWAL backend for simplicity and reliability
• Keep BinaryWAL for extreme throughput scenarios (>1M rec/s per worker)
• Make it configurable: wal_backend = "sqlite" or wal_backend = "binary"

Would you be interested in testing an early implementation? Real-world feedback would be invaluable.

Thanks for the excellent suggestion!