Skip to content

git-sync-bench: exit non-zero on failed runs, drop -1 sentinels#91

Open
Soph wants to merge 1 commit into
mainfrom
fix/bench-exit-code-and-sentinels
Open

git-sync-bench: exit non-zero on failed runs, drop -1 sentinels#91
Soph wants to merge 1 commit into
mainfrom
fix/bench-exit-code-and-sentinels

Conversation

@Soph

@Soph Soph commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Problems

Two reporting bugs in cmd/git-sync-bench:

  1. Always exited 0. run() returned nil after printing the report regardless of outcome, so the process exited 0 even when every benchmark run failed — a failure silently passed for success in CI.
  2. -1 sentinels in output. The aggregate Min* fields start at -1 as an "unset" marker updated by the first qualifying run. With no successful (or no batched) run, the sentinel survived into the report — emitting "minWallMillis": -1, "minSyncElapsedMillis": -1, etc.

Fix

  • Print the report (JSON or text) as before, then return an error when Aggregate.FailedRuns > 0, so main exits non-zero. The report stays the useful artifact; the exit code now reflects reality.
  • Clear the Min* sentinels to 0 when there were no qualifying runs (the batch-count fields are omitempty, so 0 drops them entirely).

Tests

TestSummarizeRunsAllFailedHasNoSentinels runs an all-failed set and asserts the counts, that the Min* fields are 0, and that the marshaled JSON contains no -1. The existing TestSummarizeRuns (mixed success) still passes.

🤖 Generated with Claude Code


Note

Low Risk
Benchmark CLI reporting and exit behavior only; no sync/runtime logic changes.

Overview
git-sync-bench now prints the full report (JSON or text) and then returns an error when any benchmark run failed, so main exits 1 instead of always succeeding in CI.

summarizeRuns resets aggregate Min* fields from the internal -1 “unset” sentinel to 0 when there are no successful runs (wall/sync mins) or no batched runs (batch mins), so JSON/text reports no longer show minWallMillis: -1 and similar.

Adds TestSummarizeRunsAllFailedHasNoSentinels to lock in all-failed aggregation and JSON without -1.

Reviewed by Cursor Bugbot for commit db22241. Configure here.

Two reporting bugs:

  - run() returned nil after printing the report regardless of outcome, so the
    process exited 0 even when every benchmark run failed — a failure could
    pass for success in CI. Return an error (after printing the report) when
    any run failed.
  - The aggregate Min* fields start at -1 as an "unset" sentinel updated by the
    first qualifying run. With no successful (or no batched) run the sentinel
    survived into the report, emitting minWallMillis=-1 etc. Clear the
    sentinels to 0 when there were no qualifying runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: fde63c6432fc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant