Skip to content

CHANGELOG.md

Latest commit

 

History

History
1488 lines (1270 loc) · 83.9 KB

CHANGELOG.md

File metadata and controls

1488 lines (1270 loc) · 83.9 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Added

  • Hash derive on FlushPolicy and DurabilityPolicy. Both enums now implement Hash, enabling them as HashMap/HashSet keys for callers that bucket or deduplicate by policy.
  • Nightly benchmark CI workflow (.github/workflows/bench-nightly.yml). Scheduled GitHub Actions job that runs cargo bench --features encryption on master with criterion baseline caching between runs, enabling performance regression detection between releases.
  • jscpd code-duplication gate. A duplication job in CI (.github/workflows/ci.yml) that fails when library-code duplication exceeds 2%. Configured via .jscpd.json; wired into scripts/verify-gate.sh as the jscpd gate.

Fixed

  • fuzz_flush_policy no longer fails to compile on current nightly. The &u8 as usize casts in the fuzz target's parameter decoding were removed by a recent nightly toolchain (E0606). Replaced with usize::from(*b). The nightly Fuzz CI job was red because of this.

Changed

  • scripts/verify-gate.sh now includes a jscpd gate (19 gates total).

[0.6.0] - 2026-08-10

Default durability policy flipped to Throughput. Since v0.5.0 the default was Segment with a documented plan to flip after one release. Seven releases later, this is the flip. The crate's target use case is the local throughput buffer in front of cloud sync — the cloud is the durable layer, the local disk is a throughput buffer, and skipping fsync entirely is the correct default for that workload.

This is a behavioral change (hence the minor bump): upgrading users who relied on the Segment default (fsync file data on every flush) will see a wider crash-loss window (~30s OS dirty window instead of the ~5-30s rename window). Existing on-disk segment files are unaffected — the policy is encode-time only, all existing files decode correctly.

Migration

If you relied on the old default and need file-level fsync, set the policy explicitly:

let config = SegmentConfig::builder()
    .durability(DurabilityPolicy::Segment)
    .build();

For standalone-queue deployments where this buffer is the last durable copy, Maximal (fsync file + directory after rename) is recommended over Segment.

Changed

  • Default DurabilityPolicy flipped from Segment to Throughput (src/lib.rs): the #[default] attribute moves from Segment to Throughput. SegmentConfig::default() now produces a config that skips fsync on every flush. Maximal and Segment remain fully selectable.
  • All documentation updated: rustdoc, README crash-behavior table, docs/DOMAIN_LANGUAGE.md, docs/LIMITATIONS.md, docs/PERFORMANCE.md, FEATURES.md, AGENTS.md.

Added

  • cargo publish --dry-run --features encryption added to the release runbook (AGENTS.md) to catch Cargo.lock version-sync failures before the tag push.

0.5.7 - 2026-08-10

Performance tuning and documentation. The default zstd compression level changes from 3 to 1 based on a full compression-level sweep — level 1 is ~2x faster to encode with negligible ratio loss (3.2x vs 3.1x on text payloads). For a throughput buffer where segments are short-lived, encode speed matters more than ratio. New benchmarks, a LIMITATIONS page visible on docs.rs, and CI/gate hardening round out the release.

Changed

  • Default compression_level changed from 3 to 1 (src/lib.rs): the compression-level sweep (docs/perf/2026-08-10_compression-level-sweep.tsv) proved level 1 is ~2x faster than level 3 across all payload kinds (uniform, text, json, random) with negligible ratio loss. This is a behavioral change: existing segments at any level still decode correctly (the level is encode-only), but new segments will use level 1 unless explicitly overridden via .compression_level(N). Users who need the previous default should set .compression_level(3) explicitly.

Added

  • docs/LIMITATIONS.md — comprehensive documentation of all design limitations (no WAL, no cursor persistence, no backpressure policy, etc.) with rationale. Linked from the crate-level rustdoc # Limitations section (renders on docs.rs) and from the README.
  • benches/bench_concurrent_append.rs — MPMC throughput benchmark covering 1/2/4/8 threads with both append and append_all. Key finding: append_all is 3.6x faster than append at 8 threads because it acquires the mutex once per batch.
  • Latency percentiles (p50/p95/p99) in examples/scaling.rs — surfaces tail latency from inline flush for both load and drain phases.
  • --dir, --encrypted, and --keep flags in examples/scaling.rs — enables real-disk (non-tmpfs) testing, XChaCha20-Poly1305 encryption at scale, and keeping the buffer directory for inspection after runs.
  • scripts/compression-sweep.sh — sweeps all 22 zstd levels across 4 payload kinds, outputs TSV for analysis.
  • Expanded benchmark ranges: bench_append_all batch sizes expanded to 10/50/100/500/1k/5k/10k; bench_read_from scan cache segment counts expanded to 10/100/1k/10k.
  • Benchmark baseline snapshot in docs/PERFORMANCE.md with all 10 criterion targets' medians.
  • CI hardening: clippy(fuzz) added to test job, RUSTDOCFLAGS=-D warnings on doc build, clippy components on MSRV job.
  • scripts/verify-gate.sh rewritten with --list and --only= flags, 17 gates (was 15), slug validation.
  • scripts/check-changelog-links.sh — GITHUB_TOKEN support, HTTP 403 graceful degradation.

0.5.6 - 2026-08-10

API polish and test-coverage expansion. The SegmentStore trait is now sealed (feature-gated under loom; no impact on default-feature consumers), SegmentConfig gains PartialEq/Eq, FlushPolicy gains a reusable validate() method, and BufferStats/DurabilityPolicy/SegmentConfig gain human-readable Display impls. Test coverage expands with 2 loom tests, 2 property tests, 1 stress test, 1 fuzz target, and 2 criterion benchmarks. No on-disk format change, no new dependency.

Added

  • PartialEq + Eq impls for SegmentConfig (src/lib.rs): enables direct assert_eq! on configs instead of field-by-field comparison. The cipher field is compared by Arc::ptr_eq (pointer identity) because the SegmentCipher trait does not require PartialEq — comparing key material is a security concern. Two configs sharing the same cipher Arc are equal; two configs with distinct-but-equivalent cipher instances are not.
  • FlushPolicy::validate() method (src/lib.rs): centralises the debug_assert!s for BatchOrIntervalMin constraints (min_batch <= batch_size, interval <= max_interval) into a reusable public method, callable from both the builder and open(). The builder's flush_at_batch_or_interval_min and SegmentConfigBuilder::build both call it, and open_internal validates the final policy before use.
  • Sealed SegmentStore trait (src/store.rs): the trait now requires a private Sealed supertrait marker, preventing downstream implementations. Only RealStore and in-tree test mocks can implement it. The seal marker (SegmentStoreSealed) is re-exported under the loom feature so loom mock stores can opt in.
  • Display impls for DurabilityPolicy, BufferStats, and SegmentConfig (src/lib.rs): all three types were Debug-only. Now they produce stable, human-readable single-line summaries suitable for logging. SegmentConfig's Display masks the cipher ([set]/[none]) to prevent key leakage into logs. BufferStats formats byte values in human-readable binary units (4.0KB, 1.0MB). Format follows the existing FlushPolicy::Display pattern.
  • #[must_use] on BufferStats (src/lib.rs): the stats snapshot is meaningless if discarded — the lint now catches that at compile time.
  • #[doc(alias = "backlog")] on pending_count() (src/lib.rs): discoverability — users searching for "backlog" in rustdoc now find the right method.
  • Property tests for compute_store_pressure (src/property_tests.rs): 4 new properties covering zero-max, [0,1] clamping, monotonicity in bytes, and saturation at 1.0.
  • Property tests for percentile_of_sorted edge cases (src/property_tests.rs): 4 new properties covering empty slice, all-equal values, "result is an actual element" (no interpolation), and p0/p100 boundary correctness.
  • segment_size_stats_works_with_xchacha20_encrypted_segments test (src/tests.rs): parity test for the XChaCha20-Poly1305 cipher, mirroring the existing AES-GCM test.
  • Loom tests for for_each_from and iter_from (tests/loom.rs): 2 new tests proving the v0.5.5 snapshot-then-release-lock pattern (Phase 2 of for_each_from) and the materialising iter_from path are safe under every interleaving with a concurrent append. Loom suite: 12 → 14 tests.
  • Property test for publish_disk_stats correctness (src/property_tests.rs): verifies both atomic counters (approx_disk_bytes and segment_count) match the directory truth after sync_disk_bytes and after recover (re-open), across arbitrary append/flush/delete sequences.
  • Property test for delete_acked idempotency under concurrent append (src/property_tests.rs): scales the loom proof beyond two threads — two concurrent deleters with overlapping ack ranges + an appender, proving the live segment_count counter self-heals to directory truth after sync_disk_bytes, head_seq <= pending_start after settling, and the stats snapshot is never torn.
  • Stress test for segment_size_stats under concurrent flush + delete_acked (src/tests.rs): proves the O(n_segments) scan is panic-free and structurally valid under arbitrary concurrent mutation, and exactly correct after mutation settles.
  • Fuzz target for for_each_from (fuzz/fuzz_targets/): arbitrary start_seq + limit with interleaved flush/delete_acked/append operations. Asserts no panic, correct seq → item mapping, and strict ascent.
  • bench_segment_size_stats benchmark (benches/): quantifies the O(n_segments) scan cost at 100 / 1k / 10k segments.
  • bench_cipher benchmark (benches/, --features encryption): measures AES-256-GCM and XChaCha20-Poly1305 overhead vs the no-cipher baseline in the full flush() encode pipeline.

Changed

  • seq_to_index helper extraction (src/lib.rs): three duplicated usize::try_from(seq.saturating_sub(base)).unwrap_or(usize::MAX) call sites in read_from/for_each_from replaced with a single private seq_to_index helper. No behaviour change.

  • Internal deduplication of library + test boilerplate (src/lib.rs, src/property_tests.rs): no user-facing change. Five computations that were inlined at multiple call sites are now named, reused methods: BufferInner::pending_count(), BufferInner::latest_sequence(), and BufferInner::pending_start() (const fn where applicable) replace duplicated sequence arithmetic in stats(), pending_count(), latest_sequence(), read_from, for_each_from, and delete_acked. Buffer::compute_store_pressure replaces the verbatim pressure formula in store_pressure() and stats(). Buffer::publish_disk_stats replaces the two-atomic-store pattern in sync_disk_bytes and recover. In the property-test module, prop_item, prop_config, prop_buffer, concurrent_test_config, and count_segments helpers replace ~100 lines of repeated inline construction. Behaviour is unchanged; all 145 tests (default features) + 12 loom tests pass.

Documentation

  • Version-label corrections across living docs (FEATURES.md, README.md, AGENTS.md, ROADMAP.md): the post-v0.5.5 docs-health pass corrected every stale version reference — _(unreleased)_ labels → _(v0.5.5)_, "current release is v0.5.4" → v0.5.5, "All 8 versions" → "All 12 versions", gate count 14 → 15, and property-test count 25 → 28. No code change.
  • CONTRIBUTING.md lint-architecture correction: the unchecked_time_subtraction lint was described as a nursery-group lint "not listed explicitly"; it is actually a pedantic-group lint and is explicitly listed in Cargo.toml [lints.clippy].
  • TODO_LIST rebuilt from 32-report harvest: 7 → 24 items across 7 categories. ROADMAP gained a "Tooling direction" section. All items verified against code.
  • 32 archived status/planning reports annotated and moved to docs/{status,planning}/archived/. 6 high-value files received inline ~~strikethrough~~ done at <hash> markers on concrete action items.
  • Crate-level # Guarantees section (src/lib.rs): documents the panic-free public API contract (Clippy-enforced), single-process flock enforcement, and filename-based crash recovery — previously implicit in CI but invisible to API consumers.
  • Examples table in crate-level rustdoc (src/lib.rs): added missing batch_or_interval_min and segment_tuning rows (2 of 14 examples were undocumented).
  • FEATURES.md examples inventory (FEATURES.md): expanded from 3 to all 14 examples.
  • AGENTS.md archive convention (AGENTS.md): documented that resolved reports move to docs/{status,planning}/archived/.

0.5.5 - 2026-08-04

Non-breaking release: panic-free public API, live segment_count, segment_size_stats tuning primitive, scan-cache TOCTOU fix, strict Clippy lint architecture, and expanded concurrency-property coverage. No API break, no on-disk format change, no new dependency.

Added

  • for_each_from under concurrent delete_acked property test (src/property_tests.rs): exercises the delete-ack race window through the lending-iterator callback path, proving it returns no wrong, out-of-order, or payload-mismatched items.

  • iter_from under concurrent flush + delete property test (src/property_tests.rs): races the materialising iterator against a front deleter and an in-memory flusher simultaneously, proving the wrapper does not introduce new failure modes.

  • High-concurrency segment_count stress test (src/tests.rs): segment_count_stress_4_writers_2_deleters runs 4 writers and 2 deleters in parallel, asserting the live atomic counter never panics and converges to the actual directory count after sync_disk_bytes.

  • SegmentSizeStats + segment_size_stats() (src/lib.rs): a new on-demand size-distribution query returning count / min / max / mean / p50 / p90 byte sizes of the on-disk segment files — the tuning primitive for FlushPolicy::Batch(N). Like sync_disk_bytes, it is an O(n_segments) directory scan performed outside the buffer mutex, and is a pure query (it does not mutate the cached counters). Percentiles use the nearest-rank method. The new SegmentSizeStats struct is #[non_exhaustive]; non-breaking.

  • Deterministic Barrier-based regression test for the scan-cache TOCTOU (src/tests.rs): scan_cache_toctou_mtime_guard_forces_rescan_after_mid_scan_rename forces the exact scan → rename → scan-returns-stale interleaving via a HookedStore wrapping RealStore with two std::sync::Barrier sync points. No thread::sleep, no retry loop — deterministic. Verified by temporarily reverting the fix: the test fails (10 items instead of 11), confirming it catches the regression.

  • Loom coverage for scan_segments (tests/loom.rs): two new loom tests (read_from_concurrent_flush_scan_cache_no_corruption and read_from_concurrent_delete_acked_scan_cache_no_corruption) — the first loom tests to exercise read_from (the scan-cache populate path) under concurrent mutation. Loom suite is now 12 tests (was 9). Proves no deadlocks, no panics, data integrity, and eventual consistency across every interleaving of the scan-cache populate/invalidate surface.

  • Live segment_count in BufferStats (src/lib.rs): a new segment_count: u64 field tracked incrementally alongside approx_disk_bytes — incremented on flush, decremented on delete_acked, recalibrated by recover and sync_disk_bytes. Gives callers a live on-disk segment-file count without a directory scan, unlike the one-time RecoveryReport::segment_count snapshot. Non-breaking: BufferStats is #[non_exhaustive].

  • Display impl for FlushPolicy (src/lib.rs): human-readable output for all variants (batch(256), interval(5s), batch_or_interval_min(batch=256, min=10, interval=5s, max=60s), manual). Stable format for log-scraping.

  • Edge-case tests for BatchOrIntervalMin (src/tests.rs): three boundary conditions — min_batch == 0 (always flushes at interval), max_interval == interval (min_batch irrelevant), min_batch == batch_size (interval arm reduces to batch arm).

  • Cipher equivalence tests (src/tests.rs): prove new(&[u8; 32]) and from_slice(&[u8]) constructors produce interchangeable ciphers for both AES-GCM and XChaCha20-Poly1305.

  • Concurrency stress test with BatchOrIntervalMin (src/tests.rs): 4 writers × 2 500 events with auto-flush at batch_size=1000. Proves the new policy is safe under contention.

  • Standalone example for BatchOrIntervalMin (examples/): demonstrates the tiny-segment suppression pattern with a burst-then-drip scenario.

  • Fuzz target for flush-policy parameters (fuzz/fuzz_targets/): fuzz_flush_policy exercises should_flush over arbitrary parameter combinations (variant, sizes, durations). Never panics on any input.

  • bacon in the default Nix devShell (flake.nix): live clippy feedback during development.

  • publish.yml idempotency guard (.github/workflows/): queries crates.io API before publishing; skips if the version already exists. Prevents red CI when a tag workflow re-runs after a successful publish.

  • CONTRIBUTING.md lint architecture section (CONTRIBUTING.md): documents the two-tier Clippy strategy so contributors understand why library code denies unwrap/expect while test code allows them.

  • last_flush initialization timing docs (src/lib.rs): documents that the interval clock starts at open(), not at the first append().

  • BatchOrIntervalMin in tradeoffs matrix (docs/DOMAIN_LANGUAGE.md): completes the tradeoffs documentation with the tiny-segment suppression knob.

  • CHANGELOG link-validation script (scripts/check-changelog-links.sh): validates that every GitHub tag URL in CHANGELOG.md points to a real tag. Wired into scripts/verify-gate.sh (with --no-changelog-links skip flag for offline runs).

  • curl in the Nix devShell (flake.nix): check-changelog-links.sh uses curl for GitHub API tag validation. Added to devShells.default buildInputs so the gate runs reproducibly under nix develop without relying on a system curl.

  • Cargo.lock drift check in CI (.github/workflows/ci.yml): cargo fetch --locked step catches unintended transitive dep bumps.

  • Release runbook (AGENTS.md): step-by-step release procedure with gotchas (idempotent publish, GitHub release API, tag-before-push ordering).

  • 26 historical status reports archived (docs/status/archived/): resolved reports from July 2026 moved out of the active docs/status/ directory to reduce noise. 6 current reports remain.

  • 4 older status reports annotated (docs/status/archived/): the 2026-07-22/23 batch received ## Resolution appendices closing their open items.

  • Consistency-model property tests (src/property_tests.rs): five property tests (16 → 21 total) formalising the two documented read_from race windows. Three deterministic tests verify data correctness for every generated state (surviving items after delete_acked, disk/memory split, gap-closes-after-flush); two concurrent tests exercise the actual race windows with generated parameters, including a completeness assertion that the transient gap closes once the flusher settles.

  • changelog-links CI job (.github/workflows/ci.yml): CI now runs scripts/check-changelog-links.sh, closing the split brain where the local verify-gate.sh gate validated CHANGELOG tag-links but CI did not — a broken link could ship while CI stayed green.

  • segment_count coverage (src/tests.rs, src/property_tests.rs, tests/loom.rs): three new tests close gaps in the live segment-count atomic. A property test (segment_count_matches_disk_across_flush_delete_ops) asserts the incrementally-maintained counter equals the real on-disk file count after every append/flush/delete_acked op; a loom test (segment_count_self_heals_after_concurrent_flush_and_delete) proves the counter never panics and is recalibrated by sync_disk_bytes after a concurrent flush+delete (which can momentarily wrap it past zero); and append_all_auto_flush_increments_segment_count asserts the counter after an append_all-triggered auto-flush.

  • for_each_from + dual-mutation concurrency property tests (src/property_tests.rs): a property test exercises the for_each_from lending iterator under concurrent flush (the same Phase 1/Phase 2 gap as read_from, different code path), and read_from_invariant_under_concurrent_delete_acked_and_flush races a reader against a deleter AND a flusher at once (previously only single-mutation races were property-tested).

  • Standalone segment_tuning example (examples/): demonstrates the full segment_size_stats() tuning loop — measure a small-batch baseline, sweep candidate batch sizes against a target segment-size window, and pick the first whose p50 lands inside it. The crate's stated tuning use case now has a runnable demonstration.

  • Loom absence justification for segment_size_stats (tests/loom.rs): documents in the module-level "What this does NOT cover" section why segment_size_stats has no loom test (pure query, reuses the already-covered scan_segments path, acquires no lock the hot path does not already acquire).

Changed

  • Panic-free public API: re-entrancy deadlock eliminated at the root (src/lib.rs): for_each_from no longer holds the buffer mutex across the user callback — in-memory pending items are snapshotted under the lock, then the lock is released before the callback runs. This removes the only panic path in the library: the assert_not_reentered guard, the iteration_in_progress flag, and the IterationGuard RAII type are deleted. Re-entrant calls from inside a for_each_from callback (append, stats, delete_acked, ...) are now safe instead of panicking. The public API is now provably panic-free with no qualifications. No on-disk format change; no API signature change. The Phase 2 clone is bounded by limit, not the whole backlog. Three tests in src/tests.rs cover the new behaviour (re-entrant reads, re-entrant mutation, buffer-usable-after-panicking-callback).

  • Strict Clippy lint architecture (Cargo.toml, src/lib.rs): a declarative [lints.clippy] section in Cargo.toml denies pedantic + nursery + restriction lints (as_conversions, arithmetic_side_effects, unwrap_used, expect_used, indexing_slicing, string_slice, panic_in_result_fn, panic, exit, todo, unimplemented, unchecked_time_subtraction, unreachable) across all targets. The crate-level #![deny(...)] in src/lib.rs is retained as belt-and-braces documentation of the library-only panic-prevention surface. Test, bench, and example modules carry #![allow(...)] overrides so assertions stay direct. Library code is fully clippy-clean under the entire strict set — provably panic-free on every public API path. Inspired by namtao's "Strict Lints" philosophy.

  • cargo-nextest in the default Nix devShell (flake.nix): faster and clearer local test execution with failure isolation. Not a CI dependency — CI continues to use cargo test.

  • Property test for unwrap_envelope boundary (src/property_tests.rs): 16th property test, exercising the ENVELOPE_LEN byte boundary where the pre-rewrite code could panic on untrusted input.

  • unwrap_envelope rewritten with bounds-checked access (src/segment.rs): four direct indexing/slicing operations on untrusted bytes (raw[..], raw[range], raw[idx]) replaced with .get() pattern matching. Provably panic-free on any input — including truncated or corrupted segment files.

  • Cipher new() constructors made infallible (src/cipher.rs): AesGcmCipher::new and XChaCha20Poly1305Cipher::new now use KeyInit::new(GenericArray::from(&key)) instead of new_from_slice(&key).expect(...). The signature and behavior are identical; the # Panics doc sections were removed since the constructors can no longer panic.

  • Cloud-sync example's unreachable!() replaced (examples/cloud_sync.rs): the retry-exhaustion path now returns a graceful Err(...) instead of unreachable!(), so copied integration code stays defensive even if the retry control flow changes.

Fixed

  • iter_from sequence-number bug with gaps (src/lib.rs): the wrapper enumerated returned items from start_seq + i, which is wrong when a deleted segment leaves a gap (e.g. start_seq = 0 but the first surviving segment starts at 11). The implementation now reuses for_each_from, which derives each seq from the segment's actual start or the pending-window base, so the returned (seq, item) pairs are always correct. No API signature change.

  • Floating-point false-positive in percentile property test (src/property_tests.rs): percentile_of_sorted_matches_nearest_rank_for_all_pct used (pct / 100.0 * n).ceil() as the reference, which can round the product across an integer boundary (pct=55, n=100 produced expected=56 instead of 55). The reference now uses integer div_ceil, eliminating the flake.

  • Scan-cache TOCTOU under concurrent flush (src/lib.rs): scan_segments captured the directory mtime after its readdir. A segment rename landing mid-scan could then pair a post-rename mtime with a pre-rename (stale) segment list in the cache, so the mtime staleness guard failed to detect it and "a retry sees them" did not hold until the next directory mutation. The mtime is now captured before the scan, so any mid-scan rename leaves the cached mtime stale and forces a re-scan on the next call. Surfaced by the new read_from_invariant_under_concurrent_flush property test. (Effective on filesystems where mtime advances; the mtime_supported == false path still relies solely on explicit invalidation.)

  • MAPFILE → mapfile bug in check-changelog-links.sh: the script used uppercase MAPFILE, which is not recognized as a builtin on GNU bash 5.x. The script would have always failed with command not found — it was dead code that had never run successfully before being wired into the gate.

  • HEAD tag skip in check-changelog-links.sh: the [Unreleased] compare link uses v0.5.4...HEAD per Keep-a-Changelog convention. HEAD resolves to the default-branch tip on GitHub but 404s on the tag-ref API. The script now skips HEAD refs, preventing a false failure on every run with unreleased changes.

  • verify-gate.sh stop-on-first exited 0 on failure (scripts/verify-gate.sh): run() captured the command's status with local rc=$? after an if "$@"; then ...; fi with no else. By POSIX such an if returns 0 on a false condition, so rc was always 0 — the default (stop-on-first) mode printed FAIL (rc=0) and exited 0 even when a gate failed. Fixed by capturing the real status via "$@" || rc=$?. The orchestrator also now runs under set -euo pipefail (was set -u only); run() returns 0 in --all mode so set -e does not abort after the first failure (the summary still exits non-zero).

  • verify-gate.sh --help range was hardcoded: sed -n '2,22p' drifted on every header edit. Replaced with an awk filter that prints the comment block dynamically (skips the shebang, prints # lines, stops at the first non-comment line) — self-maintaining.

  • Loom test sentinel polluted the assertion (tests/loom.rs): read_from_concurrent_delete_acked_scan_cache_no_corruption appended id: 99 to invalidate the scan cache but did not filter it from the final assertion. The sentinel is now a named constant (SENTINEL_ID) and is filtered out so the asserted set contains only the real items under test.

Documentation

  • Updated test counts and descriptions (FEATURES.md, AGENTS.md): unit tests now 116 (was 115), property tests now 28 (was 26), with descriptions expanded to cover the new for_each_from + delete_acked, iter_from + flush + delete, and segment_count stress tests. The iter_from seq-number fix is also documented in the Owned-item iterator capability note.

  • Verification-discipline refresh (AGENTS.md): rule 4 now cites the canonical scripts/verify-gate.sh gate and documents the set -euo pipefail

    • run() rewrite that captures real exit status, replacing the four-command hand-rolled subset.

  • Clarified pending_count() vs unflushed distinction (src/lib.rs): the rustdoc now states explicitly that "pending" means not yet acknowledged (the total backlog of on-disk segments plus in-memory items), not "not yet flushed." It notes that flush() leaves the count unchanged and that the on-disk / in-memory split is not exposed separately by the public API. No API change (option (c) from the 2026-08-02 report's Q3).

  • segment_count underflow / wrap contract (src/lib.rs): the field's rustdoc now documents the two benign wrap scenarios (external file removal; concurrent flush+delete where fetch_sub lands before fetch_add) and that sync_disk_bytes/recover recalibrate to the authoritative directory scan.

  • recover is open-time only (src/lib.rs): the private recover method now documents that it runs exactly once inside open/open_with_store/ open_with_report before the buffer is shared, so it cannot race read_from/flush/delete_acked — there is no scan-cache/recovery interleaving window to test. Resolves the open "loom test for scan_segments + recover" item: the race is impossible by construction.

  • Pre-encoded MockStore investigation (tests/loom.rs): the loom module doc now records why a decode-skipping mock is not tractable without compromising fidelity — encode is already skipped (the store receives pre-encoded bytes), and decode is the read path loom exists to exercise; bypassing it would need a production test-hook this crate rejects. Resolves the open "investigate pre-encoded MockStore" item.

0.5.4 - 2026-08-02

Non-breaking batch: a new FlushPolicy variant for tiny-segment suppression, a counting-allocator regression guard, expanded concurrency tests, and broader documentation. No API break, no on-disk format change, no new dependency.

Added

  • FlushPolicy::BatchOrIntervalMin (src/lib.rs): a new flush policy variant that suppresses tiny segment files during low-throughput periods. Flushes immediately at batch_size, at interval only if at least min_batch items are pending, or unconditionally at max_interval (safety valve for crash-recovery latency). Added SegmentConfigBuilder::flush_at_batch_or_interval_min convenience setter with debug_assert! validation guarding against contradictory configs (min_batch > batch_size, interval > max_interval). Covered by pure decision tests (no wall-clock dependency), an integration test for the batch-size trigger path, and an exhaustive proptest asserting should_flush matches the documented formula across all parameter combinations.

  • Allocation-count regression guard (tests/alloc_guard.rs): a counting allocator asserting fixed heap-allocation budgets on the hot paths (warm append, read_from in-memory, stats, append+flush). Machine-independent — catches tail-latency regressions (extra clones, Vec growth, format! in hot loops) without the CI hardware variance that makes absolute-latency thresholds flaky.

  • MPMC boundary stress tests (src/tests.rs): two tests proving that read_from never returns corrupt data under concurrent operation: concurrent_read_and_delete_never_corrupts (spurious Io(NotFound) when a segment is deleted between scan and read) and concurrent_read_and_flush_never_corrupts (transient gaps when items leave unflushed for a segment file the scan already missed).

  • Domain Language expansions (docs/DOMAIN_LANGUAGE.md):

    • Consistency model: canonical single-consumer guarantees (read-your-writes, monotonic reads, contiguous result, at-least-once) vs concurrent MPMC behavior (the two race windows above), with retry guidance.
    • Tradeoffs matrix: four tradeable knobs (DurabilityPolicy, FlushPolicy::Batch, compression_level, read_from vs for_each_from) and four non-tradeable invariants, with worked examples for both extremes.
    • Schema evolution of T: the two-versioning-layer split (SBF1 envelope vs CBOR payload), compatible-change patterns, and migration strategies.

  • Crate-level rustdoc (src/lib.rs): "Delivery guarantees" and "Schema evolution of T" summary sections linking to the corresponding domain language entries.

  • Percentile-latency baseline (docs/perf/2026-07-23_percentile-latency-baseline.md): documents where criterion's p99/p99.9 data lives and why allocation-count budgets are the CI-stable regression signal.

  • Book-insights mapping (docs/book-insights-mapping.md) and action plan (docs/planning/2026-07-23_15-50_book-insights-action-plan.md): theory-to-practice analysis mapping seven distributed-systems books against the codebase, with a Pareto execution plan and execution log.

Changed

  • SegmentConfig examples in docs now use Default::default() + field reassignment instead of struct-literal syntax, which fails for external consumers due to #[non_exhaustive].

  • FlushPolicy time-based tests rewritten as pure decision tests. The should_flush method is a pure function — the tests now call it directly with synthetic Duration values instead of thread::sleep + file I/O. Eliminates all CI flakiness from wall-clock dependency and reduces test runtime from ~3s to ~0.02s.

Documentation

  • docs/PERFORMANCE.md tuning guide: added BatchOrIntervalMin callout in the FlushPolicy section as the write-amplification alternative for low-throughput producers who cannot use Manual + append_all.

0.5.3 - 2026-07-22

Changed

  • chacha20poly1305 0.10 → 0.11: migrated XNonce::from_slice → XNonce::from / try_into in the XChaCha20-Poly1305 cipher and the bring_your_own_cipher example, mirroring the same migration already done for aes-gcm 0.11. The on-disk format is unchanged.

Fixed

  • docs/CIPHERS.md nonce snippet drifted from its runnable counterpart (examples/bring_your_own_cipher.rs) — the markdown still showed the deprecated Nonce::from_slice pattern after the 0.11 migration. Both encrypt and decrypt snippets now match the compiled code.

0.5.2 - 2026-07-22

Performance-focused batch: no API break, no on-disk format change, no new dependency. Three levers that make the cloud-sync deployment target faster by default and show producers how to remove the flush path from the append hot path.

Added

  • Performance tuning guide (docs/PERFORMANCE.md § "Tuning for your workload"). Surfaces the four config-only Tier 0 levers (DurabilityPolicy, FlushPolicy, compression_level, for_each_from) in impact order, each with a code snippet and a "when NOT to use" guardrail. The README crash-behavior table now cross-links to the section.
  • examples/background_flush.rs — the recommended pattern for p99-sensitive producers: FlushPolicy::Manual + a caller-owned timer thread with an atomic shutdown flag and a final synchronous flush before exit. Achieves the same append/flush decoupling a library-internal worker would, without adding a per-buffer thread, a channel, or delayed error propagation to the crate. See docs/planning/2026-07-21_08-26_flush-worker-and-tier-0-levers.md § "Addendum" for the design rationale.

Changed

  • flush() now recycles the unflushed Vec capacity across flushes. Previously std::mem::take left the field at zero capacity, forcing the next batch's append() calls through ~log2(N) reallocs. The new path reserves the previous batch's capacity inside the same lock scope as the take — one upfront allocation replaces incremental growth. No public API change, no new lock acquisition, no change to the concurrency contract. Covered by a new unit test flush_preserves_unflushed_capacity_for_next_batch.

Documentation

  • README expansion covering primitives that shipped in v0.4.1 / v0.5.0 but were only documented in rustdoc and FEATURES.md: a Cargo features table (default = [], encryption, loom, fuzz with semver-stability notes), an append_all one-liner in Quickstart, an iter_from example next to the drain loop, and an open_with_report crash-recovery example. The contents block and section anchors were updated to match.
  • SegmentBuffer rustdoc: added a # Concurrency section documenting MPMC semantics, the single-process flock invariant, the mutex-never-held-across-I/O rule, the loom-proven delete_acked + append interleaving, and the re-entrancy panic. Added #[doc(alias = "queue" | "spool" | "wal" | "writeahead" | "log")] for rustdoc search discoverability from other ecosystems. Cross-linked all examples/*.rs from the crate-root rustdoc with a one-line-per-example table.

CI / tooling

  • actionlint step added to scripts/verify-gate.sh (with --no-actionlint skip flag) and a matching actionlint job added to .github/workflows/ci.yml. YAML parse is the floor; actionlint catches ${{ }} expression syntax errors, needs: cycles, deprecated action versions, and runner/os typos that the YAML parser accepts silently.
  • lychee redirect URLs documented. The two redirects lychee reports on the README (docs.rs/segment-buffer and docs.rs/segment-buffer/badge.svg) are idiomatic docs.rs patterns; added an explanatory comment to .github/lychee.toml so future maintainers don't "fix" them and break the badge convention.
  • Historical doc annotations (update-old-docs pass). 12 of the 23 docs/{status,planning,perf}/2026-07-2* files annotated with non-destructive resolution notes: inline corrections for stale TL;DR claims (MSRV 1.85→1.86 resolution, "Cargo.toml still at 0.4.2" → shipped, refuted anchor "may be broken" alarm), and end-of-file resolution appendices for self-review reports whose TODO items have since been completed. 11 files left untouched (already self-resolving, still-current design docs, or already annotated by a prior session). No historical file was rewritten — all annotations are non-destructive per the update-old-docs skill contract.

0.5.1 - 2026-07-20

Metadata-only patch: fixes the crates.io discovery surface (description + keywords) and the crate-root rustdoc, which still carried the pre-v0.5.0 "durable bounded queue" framing after the v0.5.0 release shipped. No API change, no on-disk format change, no migration.

Changed

  • Cargo.toml description and keywords updated to match the v0.5.0 reframing. The description now leads with "high-throughput local buffer for cloud sync" (was "durable bounded queue"); keywords now include cloud-sync and spool (dropped the generic disk and durable). This fixes the crates.io search/landing surface — the #1 discovery path for new users — which still showed the pre-reframing positioning after v0.5.0 shipped.

Fixed

  • Crate-root rustdoc (src/lib.rs //!) and the SegmentBuffer<T> struct doc comment still described the crate as a "durable bounded queue". Both now lead with the cloud-sync positioning, matching the README and AGENTS.md product-positioning section. Visible on the docs.rs module page and in IDE hovers.

0.5.0 - 2026-07-20

The v0.5.0 cloud-sync throughput batch — the first release that makes the 2026-07-20 reframing (single-process throughput buffer for cloud sync, with optional performant encryption, durability-configurable, at-least-once delivery) literally true rather than aspirational. Breaking changes are batched so users upgrade once. See docs/planning/2026-07-20_03-40_v0.5.0-cloud-sync-throughput-batch.md for the full Pareto plan and docs/planning/2026-07-20_05-50_envelope-v2-design-and-v0.6-deferrals.md for the deferred-to-v0.6 items with rationale.

Added (v0.5.0)

  • DurabilityPolicy enum (Maximal / Segment / Throughput) — selects per-flush fsync behavior. Segment (today's behavior) stays the default for one release for backward compatibility; cloud-sync deployments switch to Throughput to eliminate the per-flush fsync from the hot path (~12% faster than Segment, ~26% faster than Maximal on the bench_durability_policy benchmark). Maximal adds dir.sync_all() after the rename, closing the rename-window gap that today's Segment already has. Threaded through SegmentStore::write_atomic as a third parameter; the trait signature change is a breaking change (the trait is documented as not-stable-semver-surface under the loom feature, but external implementors must update).
  • flock-based single-process lock — open() acquires an exclusive flock on <dir>/.segment-buffer.lock via fs4::FileExt::try_lock, fail-fast with the new SegmentError::Locked { path } if another process holds it. The lock file handle is held for the lifetime of the buffer and released by an explicit Drop impl (the kernel would release it on fd close regardless). Loom tests bypass the lock via open_with_store (loom does not model the filesystem).
  • XChaCha20Poly1305Cipher under the encryption feature (alongside the legacy-compatible AesGcmCipher). Writes `[24-byte nonce][ciphertext
    • 16-byte Poly1305 tag]. Eliminates AES-GCM's 2³²-message-per-key limit via the 24-byte extended nonce; constant-time in software (no AES-NI dependency). The SegmentConfigBuilder::recommended_cipher(key)helper installsXChaCha20Poly1305Cipher— the recommended choice for new buffers. Legacy AES-GCM segments still read viaAesGcmCipher`.
  • IoSite enum (Dir / Segment(PathBuf) / Unknown) replaces the Option<PathBuf> on SegmentError::Io. Makes the "no path" case explicit (was an overload of None covering both "directory-level failure" and "no context attached yet"); with_path and the new with_dir tag Unknown Io errors at high-value call sites.
  • SegmentError::Locked { path } — the typed error returned when the single-process lock is contended. Distinct from Io so callers can pattern-match the contention case without sniffing error strings.
  • SegmentIter<'_, T> — owned-item iterator yielded by SegmentBuffer::iter_from(start, limit). Returns (seq, item) pairs; works with standard Iterator combinators (.take, .filter, .map) and the for loop. Materialises up to limit items eagerly; the existing for_each_from lending iterator stays for the zero-copy in-memory tail.
  • mtime capability probe for the scan cache — open() writes a sentinel file twice with a 15ms sleep and checks whether the kernel updated its mtime. On filesystems where the probe succeeds (ext4/xfs/ btrfs/apfs/ntfs defaults), scan_segments stat-checks the directory mtime against the cached value and re-scans if it moved (detects external manipulation: backup tools, manual rm, operator quarantine). On filesystems where the probe fails (some FUSE, network fs with coarse granularity), the cache stays warm until an in-process mutation invalidates it — the safe default that avoids the 0 == 0 false-positive.
  • Pooled read-side zstd Decompressor — symmetric to the write-side Compressor pooling that landed in v0.4.x. Cloud-sync drain loops are read-heavy, so the DCtx pooling matters symmetrically. Falls back to zstd::decode_all only when the frame header lacks a content size (legacy or externally-written files).
  • Three new examples:
    • examples/cloud_sync.rs — runnable at-least-once drain loop with both a ReliableUploader (happy path) and a FlakyUploader (transient failure + retry demonstration).
    • examples/cloud_sync_disk_full.rs — the metrics-not-policy pattern: producer applies backpressure via store_pressure() > threshold, NEVER evicts unacked segments (the at-least-once hard no).
    • examples/idempotent_server.rs — the consumer-side (producer_id, seq) dedup pattern the at-least-once model requires on the server.
  • bench_durability_policy — criterion A/B benchmark comparing Maximal vs Segment vs Throughput on a 1000-event flush. Sizes the headline perf claim for the reframed positioning.

Changed (v0.5.0 — BREAKING)

  • SegmentStore::write_atomic signature now takes policy: DurabilityPolicy as a third parameter. The trait is reachable only under the loom feature (documented as not-stable-semver-surface); external implementors must update.
  • SegmentError::Io field rename — path: Option<PathBuf> is now site: IoSite. Pattern-matching callers must update; the with_path method keeps its name but now produces Segment(path) site. New with_dir method tags Unknown Io errors as Dir site.
  • SegmentConfig.cipher type changed from Option<Box<dyn SegmentCipher>> to Option<Arc<dyn SegmentCipher + Send + Sync>>. Callers using .cipher(Box::new(cipher)) must update to .cipher(Arc::new(cipher)). The benefit: SegmentConfig and SegmentConfigBuilder are now Clone (M12) — the cipher Arc is shared between clones rather than deep-copied.
  • SegmentConfig new field pub durability: DurabilityPolicy — #[non_exhaustive] struct, so direct struct-literal construction was already forbidden externally; the builder pattern is unaffected. Internal callers using struct literals must add durability: DurabilityPolicy::Segment (or Default::default()).

Internal (v0.5.0)

  • fs4 dep added for cross-platform advisory file locking (replaces the unmaintained fs2). Pure-Rust via rustix, no libc dep.
  • chacha20poly1305 dep added under the encryption feature.
  • Nix CI runs on macOS (aarch64-darwin via macos-latest runner) in addition to linux — catches platform-specific regressions earlier.
  • Dependabot auto-merge enabled for github-actions, cargo, and fuzz workspaces. PRs auto-squash-merge after required status checks pass (prevented 8-PR pile-up during the 2026-07-20 CI-broken window).
  • Latency histogram stress test — stress_8_writers_4_readers_latency_histogram reports p50/p90/p99/p99.9 per-append latency so tail-regressions are visible in CI output. p99 soft guard at 50ms (debug-mode CI).
  • Commit signing verification — gpg.ssh.allowedSignersFile now configured globally so git verify-commit succeeds (was failing with a confusing "needs to be configured" error).

Deferred to v0.6+ (with rationale)

See docs/planning/2026-07-20_05-50_envelope-v2-design-and-v0.6-deferrals.md for the full migration path. Summary:

  • Streaming CBOR deserialise + early-stop at limit (M14) — blocked by ciborium's private Deserializer struct. The format itself encodes no item count; the clean early-stop path requires either forking ciborium or changing the envelope. v0.6's envelope v2 includes an item-count field that retires this.
  • Per-segment Blake3 checksum (M17) — v1's 3 reserved bytes are too small for a useful checksum at scale. v0.6's envelope v2 ships a trailing-checksum design that covers this.
  • Compression-algorithm negotiation, metadata block, streaming cipher, async I/O, SegmentStore second impl — all deferred pending a concrete consumer request that forces the surface area; the envelope v2 design doc explains each in detail.

Earlier changes in the v0.5.0 release window

These items shipped between v0.4.2 and the v0.5.0 batch; all are part of the v0.5.0 release.

Fixed

  • CI hang: stress_8_writers_2_readers_throughput and concurrency_4_writers_1_reader_10k_events used FlushPolicy::Batch(4), creating 20 000 and 2 500 segment files respectively. Under parallel test execution this caused pathological I/O that hung CI for hours. Both now use FlushPolicy::Manual — items stay in-memory during the concurrent phase, testing mutex contention instead of the filesystem.
  • CI compile failure: The README encryption example doctest referenced AesGcmCipher but cargo test (default features) does not enable the encryption feature. The doctest is now #[cfg(feature = "encryption")]-gated via the hidden-fn main() pattern so it compiles under both feature sets.
  • Nix CI: cachix/cachix-action failed because the binary cache does not exist. Added continue-on-error: true so builds proceed without caching.
  • Broken README doctest leaked into cargo test --doc: the crate-root #![doc = include_str!("../README.md")] embedded the README, whose cloud-sync example called an undefined cloud_upload fn — turning cargo test --doc (and the Nix test check) red on master. Fixed by removing the embedding entirely (see Removed).
  • Nix doc check: a recent crane version began emitting --no-deps itself, duplicating the flake's explicit --no-deps and turning cargo doc into a hard error ("--no-deps cannot be used multiple times"). Removed the redundant arg from cargoExtraArgs. This had been blocking nix flake check.

Added

  • SegmentStore trait + RealStore impl (src/store.rs) — the I/O boundary of SegmentBuffer is now an injectable trait object. Production code constructs a RealStore internally via open()/open_with_report() (signatures unchanged); the trait is only reachable externally under the loom Cargo feature, where SegmentBuffer::open_with_store(dir, config, store) accepts a caller-supplied store. The trait has seven methods covering exactly the former std::fs surface (create_dir_all, scan, clean_tmp, segment_size, remove_segment, write_atomic, read_bytes). RealStore's implementations are extracted verbatim from the pre-refactor segment.rs — on-disk behaviour is byte-identical, the tmp→sync→rename atomicity of write_atomic is preserved, and remove_segment is idempotent on NotFound. Cost: ~5 ns per I/O call via the vtable (negligible next to zstd+CBOR+file I/O).
  • Loom coverage of delete_acked + append interleaving — four new loom tests in tests/loom.rs exhaustively enumerate the delete_acked + append interleaving that was previously covered only statistically by the stress test. The tests prove head_seq <= pending_start (the "honest backlog" invariant — if it ever broke, pending_count would under-report, silently dropping items from a durable queue) across every schedule. The mock is backed by loom::sync::Mutex<HashMap<SegmentRange, Vec<u8>>> and faithfully models write atomicity, remove idempotency, and scan ordering.
  • #[track_caller] on assert_not_reentered and all 9 public methods that call it — re-entrancy panics now point to the user's callback code instead of the internal guard function.
  • supply-chain-report.yml — a new informational, non-gating GitHub Actions workflow (weekly cron + manual dispatch) that runs cargo supply-chain publishers to report which crates.io accounts can publish crates in the dependency tree. This is the publisher-attribution layer that neither cargo audit (vulnerabilities) nor cargo deny (policy) provides; it surfaces ownership concentration and new-publisher events (the compromised-maintainer vector). Every step is continue-on-error, so it never gates a release.

MSRV

  • Bumped 1.85 → 1.86. Driven by criterion 0.8 (requires rustc 1.86) and rand 0.10 (edition 2024). Trait-upcasting coercion (stable in 1.86) lets CipherError::source() upcast Arc<dyn Error + Send + Sync> to &dyn Error directly — the ErrorExt workaround trait that existed solely to bridge this gap under MSRV 1.85 has been deleted. See docs/MSRV.md.

Changed

  • Three-layer separation in src/: segment.rs (byte-level format) is now pure — its only remaining functions operate on &[u8] / Vec<u8>, never touching std::fs. The former write/read functions are now encode_segment/decode_segment. The I/O layer (scan, clean_tmp, write's tmp+sync+rename, read's fs::read) moved to RealStore in the new src/store.rs. lib.rs orchestrates lock + flush policy and delegates every filesystem call through Arc<dyn SegmentStore>. No public API change; every existing test passes unchanged.
  • aes-gcm 0.10 → 0.11: Nonce::from_slice → Nonce::from / try_into (upstream deprecation). No public API change.
  • rand 0.9 → 0.10: RngCore import → Rng (the fill_bytes provider moved to Rng in 0.10). No public API change.
  • criterion 0.5 → 0.8 (dev-only): switched criterion::black_box → std::hint::black_box (criterion deprecated the re-export) and adopted 0.8 now that MSRV is 1.86 (criterion 0.8 requires it).
  • GitHub Actions: bumped all workflows to latest major versions (actions/checkout v4→v7, actions/upload-artifact v4→v7, peter-evans/create-pull-request v6→v8, cachix/install-nix-action v27→v31, cachix/cachix-action v15→v17).
  • Relaxed SegmentBuffer's T bound: dropped the redundant T: 'static (the bound is implied by T: DeserializeOwned, since a borrowed type cannot satisfy for<'de> Deserialize<'de>; parking_lot::Mutex only needs T: Send for Send+Sync). Strictly more permissive — a semver-minor API widening.
  • Nix flake links system libzstd: commonArgs now sets ZSTD_SYS_USE_PKG_CONFIG = "1", so zstd-sys probes the Nix-provided libzstd via pkg-config (preferring a static link) instead of compiling its bundled C on every cold build. Safe because zstd-sys 2.0.16 ships +zstd.1.5.7 and nixpkgs provides exactly zstd 1.5.7. Eliminates the ~30-file C compile that dominated cold nix builds.

Removed

  • #![doc = include_str!("../README.md")] — the crate-root rustdoc no longer embeds the README. It caused the broken-doctest leak (see Fixed) and required a postUnpack band-aid because craneLib.cleanCargoSource strips README.md from the Nix sandbox. The hand-written crate-root doc already links to the README on GitHub; docs.rs renders the README via the readme field in Cargo.toml regardless. The dead postUnpack copy in flake.nix was removed alongside it.

Internal

Process and verification hardening added after the 2026-07-20 session discovered that v0.4.1/v0.4.2 had shipped with CI silently broken for 48+ hours (see docs/status/2026-07-20_01-05_*).

  • Stress test regression guard — stress_8_writers_2_readers_throughput now asserts zero .zst segment files are created during the concurrent phase under FlushPolicy::Manual. Catches any future reintroduction of the Batch(4) config that hung CI (commit 80257a0).

  • Stress throughput re-measured under the corrected Manual config: ~2.29M events/sec (was ~397k, which was actually captured under Batch(4) and mislabeled). Both numbers documented in docs/perf/2026-07-19_v0.4.1_stress_throughput.md with correct attribution.

  • scripts/verify-gate.sh — the full local verification gate (fmt + clippy×3 + test×2 + doc + cargo deny + cargo audit + loom) in one command. Encodes AGENTS.md verification rules 4–6 as an executable script.

  • AGENTS.md verification rule 9 — before git tag for a release, the most recent CI + Nix runs on the target branch must be green (gh run list --limit 4). Local-only green is not release-ready.

  • docs/RELEASE.md pre-tag step now requires the same gh run list green-check.

  • CONTRIBUTING.md gained an MSRV-check subsection: dependency bumps must be verified against the declared MSRV before pushing (lesson from the criterion 0.8 MSRV violation).

  • dependabot.yml criterion ignore retired — the criterion >= 0.6 block is removed because MSRV is now 1.86 and criterion 0.8 is adopted.

  • publish.yml gained a cargo publish --dry-run job that runs on PRs touching Cargo.toml, surfacing packaging issues before a real tag.

  • ErrorExt workaround deleted — src/cipher.rs no longer carries the ErrorExt trait + blanket impl that existed only to bridge the pre-1.86 trait-upcasting gap. CipherError::source() now uses native trait-upcasting coercion. ~20 lines removed.

  • MSRV audit: cargo check --all-targets --features encryption on Rust 1.86.0 passes — no transitive dependency in Cargo.lock exceeds the declared MSRV.

  • v0.4.2 status report annotated (docs/status/2026-07-19_22-48_*) — the false "all green / Health 9/10" claims are now inline-corrected with pointers to the real fixes, per the update-old-docs non-destructive annotation discipline.

Performance

Profile-guided optimisation session on 2026-07-20, captured in docs/perf/2026-07-20_hot-path-flamegraph.md and docs/perf/2026-07-20_read-from-scan-cache.md.

  • Pooled zstd CCtx on SegmentBuffer — flamegraph showed 66% of flush CPU time was inside __memset_avx512_unaligned_erms, called from ZSTD_CCtx_init_compressStream2. The cause: segment::encode_payload called zstd::encode_all per flush, and zstd::encode_all allocates and memsets a fresh ~200 KB CCtx on every call. Fixed by carrying a Mutex<zstd::bulk::Compressor<'static>> on SegmentBuffer, allocated once in open_with_report and reused for every subsequent flush. Measured speedups on bench_append: batch_1 15.09 µs → 7.75 µs (2.07×), batch_100 28.06 → 20.84 µs (1.32×), batch_10000 1.21 ms → 1.06 ms (1.11×). For the v0.1.0→v0.2.0 small-batch regression documented in docs/perf/2026-07-19_v0.1.0-vs-v0.2.0.md, this is now a 2.3× net speedup vs v0.1.0 (was a 30–65% slowdown).
  • read_from_scan_cache benchmark group — new criterion group in benches/bench_read_from.rs measuring cold-vs-warm read_from across 10/100/1000 on-disk segment files. Quantifies the v0.4.0 scan-cache win: 6–9% faster at 10 and 100 segments (the bounded-queue design regime). At 1000 segments the readdir cost is no longer dominant and the cache benefit is lost in noise.
  • SmallVec<[T; 16]> for unflushed rejected. A/B benchmarked against the post-compressor-pooling baseline: batch_1 regressed 3.2%, batch_1000 regressed 8.5%, batch_100/batch_10000 within noise. SmallVec's spill-tracking overhead exceeds the saved initial heap allocation. No dependency added; the trade-off analysis is captured in the flamegraph doc.
  • examples/hotpath_profile.rs — standalone driver for flamegraph profiling of the append+flush hot path, used to capture the perf record data behind the CCtx-pooling fix. Kept in-tree so future profiling runs reproduce the same workload.

0.4.2 - 2026-07-19

The "process debt + semver-leak closure" release. All changes are additive or internal (no breaking changes; drop-in upgrade from v0.4.1). Driven by the brutally honest v0.4.1 self-review, which uncovered four critical gaps (fuzz_hooks semver leak, no CI loom job, missing dual audit+deny gate, missing domain-language docs).

Added

  • fuzz Cargo feature — opt-in feature exposing the fuzz_hooks module (parse_filename, wrap_envelope, unwrap_envelope, SegmentRange) for out-of-tree fuzz targets. Items reachable through this feature are not part of the semver contract and may change in any release without a major bump. The in-tree fuzz crate enables this feature in fuzz/Cargo.toml.
  • CI loom job — RUSTFLAGS="--cfg loom" cargo test --features loom --release --test loom runs on every push and PR. The #![cfg(loom)] test file is invisible to cargo test by default and rotted silently between v0.4.0 and v0.4.1 (the v0.4.0 FlushPolicy change removed fields the loom test still referenced); this job prevents that class of regression.
  • Fuzz target: fuzz_append_all — fuzzes append_all over arbitrary iterator behavior (empty, single, large) with 4 invariants: never panics, pending_count grows by batch size, last_seq advances correctly, follow-up empty iterator is a no-op. 771k+ runs / 16s, zero crashes.
  • Property test: append_all_assigns_contiguous_sequences_across_batches — varies batch sizes (0–50) across multiple append_all calls and asserts seqs stay contiguous at the batch boundary. Catches off-by-one regressions in the next_seq counter.
  • Property test: sync_disk_bytes_matches_actual_disk_usage — after every mutation cycle (flush + append), sync_disk_bytes() must bring stats().approx_disk_bytes into exact agreement with the sum of .zst file sizes on disk. Catches reconciliation drift.
  • docs/DOMAIN_LANGUAGE.md — glossary for segment, head_seq, next_seq, acked_seq, envelope, flush, recover. Codifies the ubiquitous vocabulary for issues, doc comments, and commit messages.
  • docs/CIPHERS.md — cipher internals + worked bring-your-own-AEAD examples: ChaCha20-Poly1305 (via the chacha20poly1305 crate), no-op cipher (testing only), and an explanation of what the cipher does and does not see (item boundaries, filename, envelope).
  • docs/perf/2026-07-19_v0.4.1_stress_throughput.md — v0.4.1 stress test baseline: ~397k events/sec under 8-writer × 2-reader contention with FlushPolicy::Manual. Reproduction command + interpretation included.

Changed

  • fuzz_hooks is now #[cfg(any(test, feature = "fuzz"))] instead of #[doc(hidden)] pub. This closes a v0.4.1-introduced semver leak. #[doc(hidden)] hides items from rustdoc but does NOT remove them from the public API surface; the cfg gate does both. See CONTRIBUTING.md → "Internal hooks: #[cfg] over #[doc(hidden)]" for the rationale.
  • Cargo.toml description rewritten for crates.io search clarity: "Durable bounded queue: batch-spills to zstd+CBOR segment files with ack-based deletion and filename-based crash recovery. No WAL, no metadata db."
  • CI supply-chain job renamed to cargo audit + cargo deny for discoverability (no behavior change).

Fixed

  • Broken AesGcmCipher doc link warning under default features (the [AesGcmCipher] intradoc link failed to resolve when the encryption feature was off). Replaced with a prose reference to "the AesGcmCipher behind the encryption feature". RUSTDOCFLAGS="-D warnings" cargo doc is now clean under all feature combinations.

Internal

  • AGENTS.md verification discipline gained two new hard rules:
    • Rule 5: "The supply-chain gate is BOTH cargo audit AND cargo deny check." They pull from different advisory sources in edge cases.
    • Rule 6: "The loom gate is RUSTFLAGS='--cfg loom' cargo test --features loom --test loom --release." #![cfg(loom)] files are invisible to default cargo test and silently rot.
  • CONTRIBUTING.md gained a new section: "Internal hooks: #[cfg] over #[doc(hidden)]" — codifies the lesson from the v0.4.1 semver leak so the next agent doesn't repeat it.

0.4.1 - 2026-07-19

The "safety + trust depth" release. All changes are additive (no breaking changes). On-disk format, encryption contract, and API shapes are unchanged from v0.4.0.

Added

  • for_each_from re-entrancy guard — calling any &self method on the buffer from inside a for_each_from callback now panics with a clear message ({method}: cannot call from within a for_each_from callback) instead of silently deadlocking. The guard is Drop-cleared for panic safety, so a panicking callback does not brick the buffer. (Closes the v0.4.0 footgun.)
  • append_all<I: IntoIterator<Item = T>> — batch append under a single lock acquisition. Returns the last assigned sequence number. The whole batch gets contiguous seqs atomically; flush is checked once at the end. Bench: benches/bench_append_all.rs quantifies the lock-acquisition saving vs a loop of append.
  • SegmentBuffer::path() — returns &Path to the segment directory. Removes the need to Debug-parse the buffer to reach the directory.
  • SegmentBuffer::config() — returns &SegmentConfig the buffer was opened with. Lets callers inspect the flush policy, compression level, and cipher presence without re-deriving them.
  • SegmentBuffer::sync_disk_bytes() — re-stats the segment directory and stores the authoritative total. Corrects drift when an external process (backup, compaction, manual cleanup) touches the directory.
  • fuzz_hooks module (#[doc(hidden)]) — exposes parse_filename, unwrap_envelope, wrap_envelope, and SegmentRange for fuzz targets. Not part of the public API.
  • Two new fuzz targets: fuzz_parse_filename (17M+ runs / 16s, zero crashes) and fuzz_envelope (15M+ runs / 16s, zero crashes; fuzzer discovered the SBF1 magic dictionary entry organically).
  • Property tests: FlushPolicy::Manual never auto-flushes (up to 499 appends); read_from(start, limit) ⊆ read_from(start, larger_limit); delete_acked pending_count is monotone non-increasing; for_each_from visits the same items as read_from.
  • Throughput stress test: 8 writers × 2 readers × 80k events, reports events/sec under contention.
  • Loom test: append_all batch atomicity under concurrent append.
  • CI workflows: nightly cargo-fuzz (fuzz.yml), weekly flake.lock update (update-flake-lock.yml), cargo-audit + cargo-deny supply-chain job, dependabot.yml for GitHub Actions + cargo.
  • Docs: docs/PERFORMANCE.md (methodology), docs/RELEASE.md (runbook), docs/MSRV.md (policy).

Changed

  • packages.default in flake.nix now builds with Rust 1.85 (the declared MSRV) via craneLibMsrv, proving the package builds on its floor — not just on whatever nixpkgs stable ships.
  • dtolnay/rust-toolchain pinned to a commit hash in all CI workflows (supply-chain hygiene).
  • nix.yml cachix-action guarded to the canonical repo + optional token, so forks don't attempt uploads to a cache that doesn't exist.
  • README perf paragraph now carries the methodology caveat inline ("single- run, single-machine; see docs/PERFORMANCE.md").
  • README comparison table now carries a freshness disclaimer.
  • AGENTS.md session-end checklist gains "release scope approval" and "draft release notes before tagging" items (process guard against the v0.4.0 failure).

Fixed

  • Loom test was broken since v0.4.0 — referenced removed max_batch_events / flush_interval_secs fields and had an inner attribute inside a function body. Now uses FlushPolicy::Manual via the builder API. The breakage was invisible because #![cfg(loom)] skips compilation unless --cfg loom is set.

Internal

  • SegmentRange fields are now pub (were pub(crate)) to support the fuzz_hooks re-export. The segment module itself stays private; the fields are only reachable through #[doc(hidden)] fuzz_hooks.

0.4.0 - 2026-07-19

The "API ergonomic + perf" release. Breaking because it removes two SegmentConfig fields (max_batch_events, flush_interval — replaced by FlushPolicy), changes the SegmentError::Io variant from tuple to struct, and renames the now-private flush_interval_secs builder method. On-disk format, encryption contract, and trait shape are unchanged.

Added

  • SegmentConfig::builder() — fluent builder over Default + setters. Removes the Default + field reassignment workaround every external caller had to use under #[non_exhaustive]. Convenience setters: flush_policy, flush_at_batch_size, flush_at_interval, flush_at_batch_or_interval, flush_manually, max_size_bytes, compression_level, cipher.
  • FlushPolicy enum (Batch(usize) / Interval(Duration) / BatchOrInterval { batch_size, interval } / Manual). Replaces the silent OR-combination of max_batch_events + flush_interval_secs that callers had no way to disable.
  • RecoveryReport + SegmentBuffer::open_with_report() — returns (SegmentBuffer<T>, RecoveryReport) so callers can inspect what recovery found (segment count, head/next seq, disk bytes, removed tmp files) programmatically. open() is unchanged and delegates internally.
  • for_each_from(start, limit, F) lending iterator — the zero-clone counterpart to read_from. Benched ~21× faster on 1000 in-memory items (1.2 µs vs 26 µs). Documented deadlock warning: the closure must not re-enter buffer methods while iterating the in-memory tail.
  • examples/crash_recovery.rs — demonstrates that flushed segments survive a process restart and unflushed ones do not, plus the new open_with_report API.
  • examples/mpmc.rs — 4 writers × 1 reader sharing one Arc<SegmentBuffer>, draining via read_from + delete_acked.
  • .github/workflows/nix.yml — CI workflow running nix flake check, nix build .#default, the test check, and treefmt.
  • deny.toml — cargo-deny config (advisories, licenses, bans, sources). All four pass green as of release.
  • renovate.json — weekly dependency updates, with nix and github-actions enabled alongside cargo.
  • release.toml — cargo-release config (no auto-push; tags via sign-tag, hand-curated GitHub releases via gh).
  • Display snapshot test for SegmentError::Io with path: Some(...) and a test for with_path (the path-attach helper).

Changed

  • SegmentConfig lost max_batch_events and flush_interval, replaced by a single flush_policy: FlushPolicy field. Migration: 
    // before
SegmentConfig { max_batch_events: 256, flush_interval_secs: 5, ..Default::default() }
// after
SegmentConfig::builder()
    .flush_at_batch_or_interval(256, Duration::from_secs(5))
    .build()
  • SegmentError::Io is now a struct variant: Io { path: Option<PathBuf>, source: std::io::Error }. The bare From<io::Error> impl preserves ? ergonomics with path: None; the with_path helper and direct construction attach path context at high-value call sites (write_segment, read_segment, scan_segments). Display: "I/O error: {source}" when path is None, or "I/O error for {path}: {source}" when set.
  • approx_disk_bytes is now AtomicU64 outside BufferInner. flush() no longer re-acquires the mutex just to bump one u64; store_pressure() loads the atomic without locking at all.
  • scan_segments() results are cached — invalidated by flush, delete_acked, recover. read_from followed by delete_acked no longer pays the directory-scan cost twice.
  • Tracing fields standardized — every event now carries path, seq, and bytes where they make sense, replacing the inconsistent head_seq / next_seq / disk_bytes / start / end mix.

Fixed

  • The pre-v0.4.0 SegmentError::Io variant dropped path context. Operators saw "I/O error: ..." with no file. Now the offending path is carried whenever it is in scope.

0.3.0 - 2026-07-19

This release closes the v0.2.0 semver/honesty debt identified in the post-v0.2.0 self-reviews. It is a breaking release because BufferStats and SegmentConfig are now #[non_exhaustive] — downstream code that uses struct literals to construct either type must switch to Default::default() + field reassignment (or, in v0.4.0, the planned SegmentConfig::builder()). The break is intentional and minor: the on-disk format, the trait shape, the error types, and the encryption contract are all unchanged from v0.2.0.

Added

  • Debug impl for SegmentBuffer<T> — mirrors the BufferStats field set plus the directory path. Does NOT print in-memory unflushed items, so T: Debug is not required. Snapshot test in src/tests.rs.
  • CipherError::with_source doc-test — the source()-chaining constructor now has a runnable example in its rustdoc.
  • Display snapshot tests for every SegmentError variant and both CipherError constructors (msg + with_source) — locks the operator-facing format strings so a careless thiserror-attribute edit shows up as a test failure rather than silently shifting log output.
  • benches/bench_stats.rs — criterion micro-bench comparing stats() (single lock + 7-field snapshot, ~12 ns) to three individual accessors (~31 ns). The "cheaper" doc claim now cites measured numbers.
  • docs/perf/2026-07-19_v0.1.0-vs-v0.2.0.md — controlled baseline captured via git worktree v0.1.0 vs HEAD: append 30–65% slower on small batches (envelope + stats bookkeeping has a per-write cost), recover 40–45% faster (recovery refactor paid off). README Status section cites this and the trade-off is honest.
  • Verification discipline section in AGENTS.md — four hard rules and a session-end checklist, installed after three same-day sessions produced self-reviews that claimed success without running the verification gate, fabricated working-tree state, and invented baselines.
  • rust-overlay integration in flake.nix with two new devShells: nix develop .#msrv (pinned Rust 1.85.0) and nix develop .#fuzz (nightly for cargo-fuzz). All three MSRV checks (cargo check, cargo test, cargo clippy -- -D warnings) now run locally on the declared MSRV; both fuzz targets now run locally for ≥60s each.

Changed

  • BufferStats and SegmentConfig are now #[non_exhaustive] — paying down the v0.2.0-introduced semver debt. Downstream struct-literal construction must switch to Default::default() + field reassignment. In-crate construction (tests, examples, benches) is unaffected. This is the breaking change that motivates cutting 0.3.0.

Fixed

  • Corrected the "auto-staging Crush hook" myth in the 03-14 and 04-22 self-reviews. Investigation during the v0.3.0 planning session found no such hook exists; the only Crush hook is commit-diff-context.sh, which fires when a commit runs (to inject diff context) and does not stage or commit. The three sessions' "lost track of working-tree state" was a real pattern, but the attribution was wrong — the cause was the assistant not running git status/git log before claiming state. Now codified as Verification discipline rule 1 in AGENTS.md.
  • PROPTEST_CASES=256 pinned in CI — removes a flaky-machine variable and matches the release-build default explicitly.

0.2.0 - 2026-07-19

This release hardens the format envelope's legacy-detection contract (the headline correctness fix), makes CipherError a real error type with source chaining, adds several API ergonomics (len/is_empty/stats/BufferStats), and refactors recover() so the mutex is no longer held across filesystem metadata calls. It is a breaking release because the SegmentError variant shape and CipherError field visibility changed; bump your dependency with cargo update -p segment-buffer.

Added

  • Format envelope: segment files now carry an 8-byte header (SBF1 magic + 1-byte version + 3-byte reserved), making the on-disk format forward-evolvable without breaking existing readers. Legacy files are auto-detected; existing monitor365 segments keep working with zero migration. The envelope is stripped before decryption, so cipher byte-compatibility is unchanged.
  • Envelope hardening (correctness): legacy detection now requires the SBF1 magic and the 3 reserved bytes to all be zero. This drops the false-positive rate on legacy encrypted files from 2⁻³² per file (~1 silent mis-detection per 7 full monitor365 deployments) to 2⁻⁵⁶ (negligible across the entire 597M-segment corpus). Existing on-disk files written by this crate still parse (they always wrote zeros for the reserved bytes).
  • CipherError is now opaque with private fields and two constructors — CipherError::msg (no cause) and CipherError::with_source (preserves the underlying AEAD error for std::error::Error::source() chaining). The previous pub String field is private; cipher implementations no longer need to know about segment paths (the I/O layer still attaches them when promoting to SegmentError::Cipher). AesGcmCipher now routes its underlying AEAD failure through source() instead of flattening it into a format!.
  • len() and is_empty() standard collection methods on SegmentBuffer (semantic aliases of pending_count() == 0 for idiomatic call sites).
  • BufferStats struct + SegmentBuffer::stats() — point-in-time snapshot of pending_count, latest_sequence, head_sequence, next_sequence, approx_disk_bytes, max_size_bytes, and store_pressure captured under a single mutex acquisition (no torn reads between calls).
  • SegmentRange::new(start, end) constructor that debug_asserts the start <= end invariant at construction. Parse-time validation stays loose so legacy files in the wild are surfaced, not dropped.
  • Static Send + Sync assertion on SegmentBuffer<T> — turns the documented MPMC thread-safety guarantee into a compile-time contract.
  • #[must_use] on latest_sequence, pending_count, len, is_empty, store_pressure, is_overloaded, and stats so accidental discards surface as warnings.
  • Property-based tests (proptest): filename bijection, payload bijection, envelope identity, encrypted roundtrip with a varied key (256 cases per cargo test), plus CI-runnable analogues of both cargo-fuzz targets (corrupted-segment read never panics; recovery over arbitrary directory contents never panics).
  • Encrypted-legacy read coverage: the headline monitor365 byte-compatibility guarantee (a [nonce][ciphertext] segment file written without the SBF1 envelope) is now covered by a regression test; previously the entire encrypted-legacy read path had zero coverage.
  • Error-matching doc-test: error.rs now shows how to match on SegmentError::Cbor { path, phase, .. } to recover the offending file path and quarantine it.
  • Fuzz scaffold (cargo +nightly fuzz): two targets — fuzz_corrupted_read (reading bytes-corrupted segments never panics) and fuzz_recovery (opening over a directory of arbitrary garbage never panics). See fuzz/README.md.
  • FEATURES.md — honest feature inventory by status.
  • ROADMAP.md — long-term direction and explicit non-goals.
  • flake.nix — reproducible devShell with zstd, pkg-config, and the Rust toolchain (nix develop).
  • Shared benches/support.rs module consolidating the benchmark helpers previously duplicated across all four criterion targets.

Changed

  • Typed errors (breaking): SegmentError::Cbor, Cipher, and Integrity variants now carry structured context (path: PathBuf, phase: &'static str or reason: &'static str) instead of opaque String payloads. Operators see exactly which file failed and why, without spelunking through logs.
  • CipherError field visibility (breaking): the previous pub String field is now private. Use Display or the new constructors.
  • Extracted src/segment.rs: the on-disk format (filename contract, envelope, CBOR→zstd→cipher encode/decode pipeline, segment scan, tmp cleanup) now lives in its own module. SegmentBuffer focuses purely on in-memory orchestration and locking.
  • Renamed the private BufferInner::pending field to unflushed for precision: it holds items not yet written to a segment file, distinct from the public pending_count() backlog metric.
  • recover() no longer holds the mutex across filesystem metadata calls. All fs::metadata I/O now happens before the lock is taken; the lock is held only long enough to publish the rebuilt head_seq/next_seq/approx_disk_bytes. Restores the invariant that the mutex is never held across file I/O.

Fixed

  • Envelope false-positive on legacy encrypted files. As shipped in 0.1.0 unreleased, the envelope magic-only check (2⁻³² false-positive rate) would silently mis-detect roughly 1 in 7 monitor365 deployments' encrypted segments as enveloped, producing spurious SegmentError::Cipher errors. Requiring the 3 reserved bytes to also be zero drops the rate to 2⁻⁵⁶.
  • delete_acked(acked_seq) no longer under-reports pending_count() when called while items are still buffered in memory. head_seq is now clamped to the in-memory window so the backlog count stays honest even when the ack cannot remove unflushed items.
  • SegmentBuffer::open doc corrected: recovery reads filenames only, so it returns SegmentError::Io on failure — not Cbor/Integrity (those surface at read_from time).
  • fuzz/fuzz_targets/fuzz_recovery.rs parser had a dead-code let _ = rest; and convoluted split/peek logic; rewrote cleanly.

Security

  • Continuing the 0.1.0 baseline: extracted from monitor365 and proven on 597M+ events in production.

0.1.0 - 2026-07-19

Added

  • SegmentBuffer<T> — durable bounded queue backed by zstd-compressed CBOR segment files. Generic over any T: Serialize + DeserializeOwned + Clone + Send + 'static.
  • SegmentConfig with tunable batch size, flush interval, max disk usage, and compression level.
  • SegmentBuffer::open(dir, config) constructor with filename-based crash recovery.
  • append(), flush(), read_from(), delete_acked(), latest_sequence(), pending_count(), store_pressure(), is_overloaded() public API.
  • SegmentCipher trait for pluggable at-rest encryption.
  • AesGcmCipher (AES-256-GCM with random 12-byte nonce prefix) behind the encryption feature.
  • SegmentError (thiserror-based) with Io, Cbor, Cipher, Integrity variants.
  • 26 unit tests + 2 doc tests covering: basic CRUD, partial reads, limits, crash recovery, concurrent writers/readers (10K events, 4 writers + 1 reader), time-based auto-flush, error paths (corrupted zstd, truncated encrypted, wrong key, no key), encryption roundtrips, and pressure/overload boundary conditions.

Fixed

  • Concurrency bug in flush(): sequence numbers (start_seq, end_seq) are now computed atomically inside the mutex lock alongside taking the pending events. Previously, a race between concurrent append() calls could corrupt segment filenames by computing the sequence range from a stale next_seq read in a second lock acquisition.
  • Same race in append(): the returned sequence number is now captured under the same lock as the push, not re-read after releasing the lock.

Security

  • Extracted from monitor365 and proven on 597M+ events in production.