Repository navigation
feat(storage): commit-stamped create sequence (arrival_seq) on every backend (BD-5) - #7912
Conversation
…types.Issue BD-5, part 1 of 3: the column and its read path, with nothing writing it yet. arrival_seq is the commit-ordered create sequence the events journal will stamp on a wisp-plane row: the seq of the row's own create record, stored on the row in the same transaction, so a consumer can recover create order from rows after the journal records are pruned. - Migration 0073 adds a nullable BIGINT to issues and wisps (guarded INFORMATION_SCHEMA + PREPARE, 0060/0067 shape). The column only means something on wisps, which is clone-local; issues carries it for shape parity and it stays NULL, the wisps.row_lock/current_revision precedent. The CLI bundle gets the direct-DDL override (dolthub/dolt#11345) and the substitute is listed in cliSubstituteAssumesWispTables. - Ignored migration 0029 carries the wisps column to fresh clones and creates the clone-local UNIQUE index uk_wisps_arrival_seq. - TestLatestVersionIncludesMigration0073 supersedes 0072's pin, the way each slot's pin has superseded the previous one. - IssueBaseColumns and IssueBaseColumnsLite select it; ScanIssueFrom and ScanIssueLiteFrom read it into types.Issue.ArrivalSeq (json:"-" until the wire half lands). The legacy SQLite projection gains its placeholder. - TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace now stops the ignored pass right after 0023 (SetMigrateStepFaultHookForTest) and compares SHOW CREATE TABLE across that step alone. It used to run every ignored migration and require nothing to change, which ignored/0029 now breaks by design: it adds uk_wisps_arrival_seq to wisps. The test still proves what it names, that 0023 is a no-op on a healthy workspace. - TestFullChainFromPreWispsAndMissingDependenciesIDConvergesThroughDoltCLI replays the ignored series through the dolt CLI, which no-ops 0029's PREPARE'd ADD COLUMN (the dolthub/dolt#11345 limitation 0006 already works around there), so 0029's PREPARE'd unique index would then fail on a missing column. The test applies 0029's effect as Go-decided direct DDL before the frozen text, exactly as it does for 0006, and its convergence and re-open checks cover wisps.arrival_seq and uk_wisps_arrival_seq. Ported from gascity/bd-enterprise#204 (12f0a7360), renumbered from main 0070 / ignored 0028 to the next free upstream slots; the Postgres half of that commit stays downstream. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BD-5, part 2 of 3: the writer. RecordEventInTx now decides, once, whether a create stamps: the op is a create, the row was read from wisps, and it carries no stamp yet. When it does, insertEventRow draws the seq, writes it onto the row (stampArrivalSeqInTx), and only then marshals the post-image, so the record's seq, the row's arrival_seq and the post-image agree. There is no second counter: the stamp IS the journal seq, so it inherits the counter's commit-ordered allocation on every engine, and a workspace that does not journal pays nothing and stamps nothing. - The stamp writes arrival_seq alone (updated_at self-assigned against Dolt's ON UPDATE CURRENT_TIMESTAMP), so row_lock, current_revision, updated_at and content_hash are untouched and a caller's post-create CAS still holds. It must match exactly one row. - A collision on the journal seq OR on uk_wisps_arrival_seq goes through the same single heal-and-retry; the retry moves a stamp the failed attempt already wrote. isSeqCollision also recognizes the embedded engine's unique-key error kind. - healEventSeqCounter raises the counter past MAX(arrival_seq) too, and falls back to the journal alone on a drifted clone with no wisps column. - PrepareIssueForInsert and the unit-of-work Insert clear ArrivalSeq, and the generic insert/upsert column lists never name it: no caller-supplied value is ever stored, and a plane move leaves the row unstamped. Guards: TestArrivalSeqIsWrittenOnlyByTheJournalSeam (AST scan of every SQL literal under internal/storage), TestIssueInsertAndUpsertColumnsExcludeArrivalSeq, TestOnlyRecordEventInTxRecordsCreates; the row_lock and version completeness guards learn the stamp is sink bookkeeping. Dolt-server and embedded tests pin commit order under overlapping and racing writers, uniqueness, the untouched-columns rule, both heal orders, the missing-column fallback, and that many NULLs coexist under the index. The 40-shard embedded storage manifest places the new embedded tests. Ported from gascity/bd-enterprise#204 (9ca7815c5); the Postgres refusal and its open-time reconcile stay downstream. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BD-5, part 3 of 3: the member on every Issue-shaped answer, the contract cases on every leg, and the interchange boundary. - types.Issue.ArrivalSeq is now `arrival_seq,omitempty`: absent on every unstamped row, so a reader that predates it sees what it saw before, and outside the content hash. `bd show`/`list`/`ready --json` carry it. - The spec adds an optional int64 arrival_seq to Issue, BatchGetIssue, IssueWithCounts, IssueDetails, IssueWithDependencyMetadata, TreeNode and BlockedIssue, documents the post-image carrying it on EventRecord.issue, and lists the new build-level behavior token issues.arrivalSeq. Whether rows are being stamped follows journal activation, which events.list's 409 already reports. apigen and the wire-shape golden are regenerated; the change is additive, so wire_revision is unchanged. The client mirrors the token (wire.CapArrivalSeq). - bd export and auto-export strip clone-local fields from every record (stripCloneLocalFields), held by an AST guard over every exportIssueRecord builder. - backend/conformance gains nine Journal-role cases (definition, batch order, every read path, immutability, journal-off, plane, create order recovered after the journal is pruned past it, no reuse across delete, supplied values ignored) with hooks bound to the public roles. They run on the dolt, embedded, unit-of-work and served-HTTP legs. The Importer role is optional (only the unit-of-work provider hands one out); the two store legs drive the import cases through CreateIssuesWithFullOptions, the seam `bd import` runs on them, and the served leg leaves the explicit-id batch, the claiming create and import to the store legs. - docs/reference/events-journal.md documents "Arrival order on the row". Ported from gascity/bd-enterprise#204 (cf9f1d431 and 7bb0899e3); the ImportIssue member, the HTTP importer's strip and the Postgres leg stay downstream, where the import operation and Postgres live. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both heal tests wound the counter back to high-1. The first draw is then high, which is already the counter's own value, so a heal that consulted only the journal (GREATEST(next_seq, MAX(seq))) also lands on high+1 and the tests pass without the MAX(arrival_seq) term they exist to pin. Wind back three instead, with the journal empty: a journal-only heal leaves the counter in place and the retry collides with the next surviving stamp. Verified with a go test -overlay mutant that drops the arrival term from healEventSeqCounter: TestArrivalSeq_EmbeddedHealRaisesTheCounterPastTheStamps now fails with "duplicate unique key given: [2]" and passes on the real code. The domain/db twin takes the same change (Dolt-container tier). Ported from gascity/bd-enterprise#204 (8616c3cf4). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… stamp RecordEventInTx stamps only a wisp row that has no stamp yet (issue.ArrivalSeq == 0). That clause is what keeps the synthetic create records an import upsert and an id rename journal for an existing row from restamping it, and nothing tested it: an overlay mutant that drops the clause passed every embedded Journal, Import, Rename and Wisp test. Three Journal-role cases, declared before EveryMutationKindLandsARow and wired on every leg: - ArrivalReimportKeepsTheStamp: re-importing a stamped wisp leaves the stamp, journals a create with its own higher seq, and that record's post-image carries the original stamp. A real flow: bd export keeps no_history wisps. - ArrivalFirstCreateRecordStampsAnUnstampedRow: a wisp created with the journal off is stamped by the first create record journaled for it, with that record's seq. - ArrivalRenameKeepsTheStamp: the create a rename journals carries the row's stamp. Driven through a new optional JournalArrivalHooks.Rename, bound by conformance.ArrivalRenameHook on the dolt and embedded legs (the stores that expose UpdateIssueID); the served and unit-of-work legs skip it by name. The import cases run on the dolt and embedded legs through their bulk upsert (ArrivalUpsertImportHook) and on the unit-of-work leg through its Importer; the served leg skips them by name, since the wire publishes no import. Ported from gascity/bd-enterprise#204 (1efd4d571). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- TestArrivalSeq_HealFallsBackWithoutTheWispColumn drops wisps.arrival_seq on the suite-shared database and restored it with `_, _ =`. wisps is dolt_ignored, so SetupTest's DOLT_RESET never repairs it, and a failed restore would surface as unrelated failures later. The cleanup now reports each failure against this test. - TestArrivalSeq_JournalCollisionMovesTheStamp required its counter reset to clear every surviving stamp, which held only because no earlier test left a stamp above the counter. It now lifts the counter past MAX(arrival_seq) before seeding, so its records sit above every surviving stamp by construction; the Require stays as a setup check. Ported from gascity/bd-enterprise#204 (7962a4598). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… deadline TestArrivalSeq_EmbeddedConcurrentWritersAreUnique gave each of its 12 creates a fixed 60s context, but the engine serializes them, so the last one waits on all the others: the farm's -race build took about 30s, a 2x margin. The writers now share one context bounded by the test binary's deadline less 30s of grace to report a failure in, and by nothing else when no deadline is set. Ported from gascity/bd-enterprise#204 (b2b158a18). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The "Commit order" guarantee was stated unconditionally, but the documented first-record rule breaks it: a row created with the journal off and later renamed or re-imported takes a stamp above rows created after it. And "no stamp and no cost" with the journal off overstated it, since uk_wisps_arrival_seq is maintained on every wisp insert and delete regardless, NULL entries included. The guarantee now reads "ordered by the commit of the create record that stamped the row" and says when that record is not the insert; "no cost" becomes "no extra statement", naming the index. The types.Issue doc and the CHANGELOG entry carry the same caveat (the spec's Issue.arrival_seq description already does). Ported from gascity/bd-enterprise#204 (c15da9a09). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ed issue The storage create APIs write the stored row back onto the issue they are given: CreateIssue, CreateIssues and Transaction.CreateIssue on every store (the calls the public beads.Storage exposes, and the ones gascity's native store maps its create result from), and the unit-of-work repository's Insert. PrepareIssueForInsert and Insert zero ArrivalSeq on the way in so a supplied value is never stored, but nothing put the stamp back: the row carried arrival_seq 1..N while the caller's struct still said 0, so a consumer that keeps its create result reads every stamped wisp as unstamped until it reads the row again. Only the public Lifecycle roles, which re-read the row, returned it. RecordCreateEventInTx journals the create exactly as RecordEventInTx does (both share recordIssueEventInTx, which still makes the one stamp decision) and copies the journaled snapshot's arrival_seq onto the issue. Both create seams use it: issueops.finishCreateIssueInTx, which every store's create and import funnels through, and domain/db Insert. With the journal off nothing is recorded and the field stays zero, as on the row. Red before the fix: TestArrivalSeq_EmbeddedCreateLeavesTheStampOnTheCallersIssue (the caller's issue carried 0 against rows 1..4) and, on a Dolt sql-server, TestDomainDB/TestArrivalSeq_InsertLeavesTheStampOnTheCallersIssue (0 against 17 and 18). The structural guards follow the move: the stamping insertEventRow call now lives in recordIssueEventInTx, and the unit-of-work journal-completeness guard counts RecordCreateEventInTx as an emitter. The embedded storage shard manifest places the new test. Ported from gascity/bd-enterprise#204 (3d12c16dd). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… stamp When bd_events_seq goes missing, the ignored series' cursor floor replays it from ignored/0022 (replayFloor 21), which re-creates the counter and seeds it from the journal's MAX(seq) alone. With the journal pruned past every record that seed is 0 while stamps survive on wisps, and the values swept or deleted wisps held are free: the next create takes one without touching the unique index, so the writer's collision heal never fires and the new row sorts before older surviving rows. ignored/0029 re-runs in that same replay, after 0022, so its third step now raises bd_events_seq past MAX(wisps.arrival_seq) (GREATEST, guarded on the counter table and the stamp column existing). It changes nothing on an in-place upgrade (every stamp NULL), on a fresh clone, or on a healthy re-run. TestArrivalSeq_EmbeddedReplayedCounterSeedClearsTheStamps drives the path end to end on the embedded engine: three stamped wisps, the two oldest deleted, the journal emptied, the counter table dropped, MigrateUp, then a new wisp must stamp above the survivor. Without the step the replay re-seeds at 0 and the test fails on the counter check. TestIgnoredMigration0029RaisesTheSeqCounterPastTheStamps pins the statement, its place after the column and index, and that bd_events_seq's replay floor still sits below 0022. Follow-up from the gascity/bd-enterprise#204 review (finding 4). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The storage contract cases prove the stamp on every backend, but a consumer that shells out to bd reads the CLI's own rendering, which marshals types.Issue, IssueWithCounts or IssueDetails per command; a command that moved to a projection of its own would drop the member without any storage test noticing. TestEmbeddedArrivalSeqOnEveryJSONRead creates an ephemeral wisp, a no_history wisp and a durable bead with the journal on, and checks that bd create --json, bd show, bd list --include-ephemeral, bd ready --include-ephemeral and bd query --json carry each wisp's stamp equal to the seq of its create record in bd events export, that the durable bead carries no member, and that bd export never writes one. The 100-shard embedded cmd manifest places it. Follow-up from the gascity/bd-enterprise#204 review (finding 5). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rential TestVersionedHistoryOffMatchesPreStackGolden (embedded, server and unit-of-work legs) failed on issues.arrival_seq, wisps.arrival_seq and the two new bookkeeping rows: migration 0073 and ignored/0029 add schema the goldens predate. Flag-off storage leaves it untouched (issues never carries a stamp, and wisps carries one only while the workspace journals, which the scenario does not), so it takes its own schemaAdditions entry, as the versioned-history stack's columns did, and the goldens stay as recorded. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
This PR has been added to the review queue. Track progress: https://factory.gascity.com/city/maintainer-city/runs/gcg--9223372036836401312 |
|
Triage: |
…s its index Maintainer fixup for review iteration 1 of #7912. Every issue read now selects arrival_seq, and a server-mode read-only open neither migrates nor checks a behind schema, so after an upgrade every list, show and ready on a v72 database failed on the unknown column. An open that does not migrate now refuses such a database at open with SchemaBehindError (schema.CheckReadableDrift). It does so only when the database lacks a column the shared issue projection selects; a behind database that still has every column keeps serving reads (bd-578h9.5, #5920). The open reads the version once, as the forward-drift check it replaces did, and lists the issues columns (SHOW COLUMNS) only when the database is behind. Three paths do not migrate: - server mode: dolt.Config.RequireReadableSchema. The root pre-run sets it for every open except a preview, and so do the routed and foreign read factories (cgo and nocgo). Through a gateway the refusal names the server administrator's remedy; - proxied mode: the read-through after a refused migration; - embedded: a read-only command's open after a tolerated gate or dirty-table refusal. 0073's down drops wisps.arrival_seq and uk_wisps_arrival_seq with it, but leaves ignored/0029 recorded, so a re-applied 0073 restored the column without the index. 0073's up now creates the index as well, under 0029's guard, and its CLI bundle substitute does the same. Docs: a CHANGELOG upgrade note. "Never reused" is narrowed to values the counter, the journal or a surviving row still holds. A bead moved onto the wisp plane is documented as carrying no stamp. The 1054 and Kind.Is comments are corrected. Tests: - CheckReadableDrift unit cases; - a server-tier open matrix (behind with every column, a missing column, the gateway remedy, the escape hatch); - a 0073 rollback and re-apply through the real migration pass; - a bd list/show/ready e2e on a v72 database, converging with bd migrate schema; - proxied and embedded read-through refusals. Each test fails against its mutant. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…s its index Maintainer fixup for review iteration 1 of #7912. Every issue read now selects arrival_seq, and a server-mode read-only open neither migrates nor checks a behind schema, so after an upgrade every list, show and ready on a v72 database failed on the unknown column. An open that does not migrate now refuses such a database at open with SchemaBehindError (schema.CheckReadableDrift). It does so only when the database lacks a column the shared issue projection selects; a behind database that still has every column keeps serving reads (bd-578h9.5, #5920). The open reads the version once, as the forward-drift check it replaces did, and lists the issues columns (SHOW COLUMNS) only when the database is behind. Three paths do not migrate: - server mode: dolt.Config.RequireReadableSchema. The root pre-run sets it for every open except a preview, and so do the routed and foreign read factories (cgo and nocgo). Through a gateway the refusal names the server administrator's remedy; - proxied mode: the read-through after a refused migration; - embedded: a read-only command's open after a tolerated gate or dirty-table refusal. 0073's down drops wisps.arrival_seq and uk_wisps_arrival_seq with it, but leaves ignored/0029 recorded, so a re-applied 0073 restored the column without the index. 0073's up now creates the index as well, under 0029's guard, and its CLI bundle substitute does the same. Docs: a CHANGELOG upgrade note. "Never reused" is narrowed to values the counter, the journal or a surviving row still holds. A bead moved onto the wisp plane is documented as carrying no stamp. The 1054 and Kind.Is comments are corrected. Tests: - CheckReadableDrift unit cases; - a server-tier open matrix (behind with every column, a missing column, the gateway remedy, the escape hatch); - a 0073 rollback and re-apply through the real migration pass; - a bd list/show/ready e2e on a v72 database, converging with bd migrate schema; - proxied and embedded read-through refusals. Each test fails against its mutant. The proxied shard manifest (.github/scripts/proxied-cmd-test-shards.txt) places the new TestServerMode* test on shard 16 (gen_proxied_shard_manifest.py 34 --weights=duration --write). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2428830 to
c6c4ec4
Compare
Maintainer Adoption ReviewThanks for the contribution, @julianknutsen! This PR adds This PR was reviewed and adopted with maintainer fixes pushed directly to the PR branch. Original PR ReviewDecision: request_changes resolved to approve. Specific gaps fixed:
Review findings addressed:
Non-blocking follow-ups (shared for visibility — NOT completed in this PR): Filed as follow-up beads:
Not filed (below the follow-up bar):
Maintainer ChangesMaintainer fixes were pushed directly to the PR branch as one commit on top of your 12:
The same commit adds the generated proxied shard-manifest line for the new end-to-end test, which cleared the Final Review StatusReady for the merge queue: final head CI: https://github.com/gastownhall/beads/actions/runs/40562215797 Review Iterations2 review passes (Claude and Codex): the first requested changes on the two majors above, the maintainer commit fixed them, and the second approved. Claude also ran the Dolt-backed and Dolt-server tiers that PR CI skips on a same-repository branch. A delta review then confirmed the CI repair added only the generator's own manifest line. Adopted via |
|
A push to main that includes this PR is failure in Main at 5011528: https://github.com/gastownhall/beads/actions/runs/40568706768 |
…(BD-6) (gastownhall#7931) * refactor(dolt): drop the dead per-table id generator generateIssueIDInTable in internal/storage/dolt had no caller left: every id the dolt store mints goes through issueops.GenerateIssueIDInTable. The duplicate checked only its own table, so a future caller would have minted ids the shared generator refuses (for example, an id a retaining sweep is about to retire). Remove it with the helpers only it used (nextCounterIDTx, seedCounterFromExistingIssuesTx, isCounterModeTx, generateHashID, getAdaptiveIDLengthFromTable) and the two unit tests of the last one; issueops carries the live copies of each. Ported from gascity/bd-enterprise#210 (818610dc1). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(schema): id tombstone tables for retaining sweeps (main 0074, ignored 0030) Add the two tables a retaining retention sweep writes and every create path will read (BDP idempotency-expired): id_tombstones, versioned like issues, and wisp_id_tombstones, clone-local like wisps (the existing wisp_% dolt_ignore pattern covers it). Each row is the compact (id, closed_at) pair, keyed on id, nothing else from the swept row. - Main 0074 creates both, guarded and idempotent; ignored 0030 materializes the clone-local one on fresh clones in the identical shape. The slots are the next free ones, directly above arrival_seq's (gastownhall#7912: 0073, ignored 0029). 0073's latest-version pin moves here. - wisp_id_tombstones is a sentinel floored table (floor 29): a clone that lost it re-creates it on the next write-intent open instead of failing every create closed. Re-created EMPTY, the wisps-plane tombstones are gone while the wisps rows may survive, so migrate now prints a loud, unconditional notice for a lost sentinel table that carries one (cursor above its floor) — the only signal a consumer gets that the clone is a new logical scope for wisps-plane idempotency keys. - bd doctor requires id_tombstones and treats wisp_id_tombstones as clone-local; the Dolt schema-parity test pins the pair to one shape. - The history-off differential (historyoffequiv) sets both tables and their bookkeeping rows (74, ignored 30) aside: flag-off storage leaves them empty. - Three of BD-5's tests assume 0073 is the latest migration and follow the slot: the 0073 rollback test rolls back what sits above 0073 first, as an operator must (a row 74 left behind would hold the cursor past 73); the server-mode read refusal rebuilds the v72 shape without 0074 too, so the refusal names v74; and the embedded lenient-gate test regresses the cursor below 0073 rather than one migration back, so the pending 0073 still meets the dirty-table guard. Ported from gascity/bd-enterprise#210 (b60aa6a42). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(storage): retaining sweeps retire ids; every create door refuses a retired id A retention sweep can now ask to retain ids (SweepRequest.RetainIDs): in the same transaction that deletes the closed rows it writes one permanent (id, closed_at) tombstone per row into its own plane's tombstone table. From then on no write path commits that id again (BDP idempotency-expired): - every create door — Lifecycle.Create, BatchApplier and BatchCreator items, the stores' upsert CreateIssue/CreateIssues, a transaction's CreateIssue, the unit of work's domain create with or without CreateOnly — refuses it with the typed *issueops.IdempotencyExpiredError (ErrIdempotencyExpired, re-exported from storage, backend and beads); - an import skips and reports it (BatchCreateOptions.OnRetiredID, which the unit of work's Importer surfaces as ImportBatchResult.RetiredIDs; ConflictSkip, the auto-import path, skips it silently); - the generators treat it as taken: hash mode folds the tombstone check into its one collision read, counter mode walks past retired values with one probe each, the uow mint likewise. The race a create loses to the sweep of its id is closed by a lock, not by isolation: every path that decides whether an id may be written writes the id's create coordination cell (local_metadata, 4096 shards, sorted, one statement) BEFORE it reads occupancy, and a retaining sweep — the shared SweepInTx and the unit of work's retiring delete — writes the cells of every id it retires BEFORE it deletes. On a READ COMMITTED door the create that comes second waits out the sweep and then reads its tombstone; a sweep that comes second fails serialization or meets a write-write conflict and retries over the created row. Occupancy is one statement over both row tables and both tombstone tables; a live row wins. A CreateOnly batch no longer reads tombstones it never consults. Only a retaining sweep retires: a plain prune or purge, the Deleter and bd gc's decay leave ids reusable, and the Sweeper and Deleter docs say so. The IDRetention conformance contract (17 cases: opt-in, both planes, each door, the no-history re-create through every door, dry run, narrowing, history, and three staggered races — create-only, upsert and import — asserting no interleaving ever leaves a live row beside a tombstone) runs on the dolt, embeddeddolt and uow legs; the dolt races run on a multi- connection store. The http leg waives it by name until the wire lands next in this series. The two store legs hand out no Importer role, so they drive the import cases through conformance.IDRetentionUpsertImporter, over CreateIssuesWithFullOptions: the seam `bd import` writes through on them (ArrivalUpsertImportHook's precedent). The unit-of-work leg uses its Importer, which now reports RetiredIDs. The http client cannot send RetainIDs yet (apigen.SweepRequest has no retain_ids member), so it refuses a retaining sweep before the dial under a new refuse row, W-SweepRequest.RetainIDs, rather than dropping the field and running a sweep that leaves the ids reusable; the wire commit retires the row. TestSweepResultCarriesEveryWireMember skips RetainIDs by name until then. httpapi's TestSweepResponseCarriesEveryRoleField counts issueops.SweepResult against the wire's SweepResult; RetainIDs is waived there by name until the wire commit projects it, so this commit is green on its own. The history-off differential's goldens (historyoffequiv) are re-recorded from this series, as its package doc prescribes for a deliberate change to what flag-off storage writes: the coordination cells of the scenario's explicit-id upsert creates add rows to local_metadata (six on the embedded and server legs, for s1-s4, sw and t1; three on the unit-of-work leg, for u1, uw and u2). A recording of the base is byte-identical to the committed goldens, and no other table's digest moves. The digests were recorded on bd-enterprise#210, whose base carries the same goldens as main, and recorded_from says so. The createbatchequiv retired scenario stamps its seeded rows' participation (stampParticipation), as depadd and waitsfor do, so the batch's upsert of eq-rl mints under the participation-generation fence as it did when its golden was recorded. TestIDRetentionContractRaces is pinned to dolt_server_full_test's shard 16 (server-storage-test-shards.txt). Left to the name hash it lands in shard 6, which already takes about the whole 15m test timeout under load, and pushes that shard past it. Ported from gascity/bd-enterprise#210 (58d8b1527). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(cli): bd prune/purge --retain-ids, the typed create refusal, and retired rows on import - `bd prune --retain-ids` and `bd purge --retain-ids` ask the sweep to retire every id it deletes. --json gains retain_ids: true; text says "Retired N id(s)", or "Would retire N id(s) ... would be refused" in a preview; the confirm hint carries the flag so the run it suggests is the sweep that was previewed. - `bd create --id X` of a retired id renders one shared typed refusal on the direct and proxied routes: --json prints {"error","code":"idempotency_expired","id","closed_at"}, text says the create that used the id already happened and never runs again; exit 1. - `bd import` skips a retired row and reports it in retired_ids and a stderr note, on the classic and proxied routes, and `--dry-run` now agrees with the run it previews: it reads the would-create ids' tombstones through an optional store seam (RetiredIssueIDs, under the CLI's decorators) and reports them as retired instead of counting them as creates. - The confirm hint is sweepConfirmHint's, so --retain-ids joins the selectors TestSweepConfirmHintCarriesEverySelector pins. - docs/workflows/wisps.md documents --retain-ids and that only a retaining sweep retires; CHANGELOG entry for the whole series. Tests: the embedded CLI end to end (prune and purge, both envelopes, import and its dry run), the proxied route's envelope (gated on the proxied-server tier), and the dry-run classifier in both stale modes. Ported from gascity/bd-enterprise#210 (4116bc2a5). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(wire): retain_ids on issues:sweep, 410 idempotency_expired, and the served IDRetention leg The wire half of the retaining sweep. - issues:sweep takes retain_ids (and echoes it), behind the new behavior token issues.sweep.retainIds; the token also promises that every create path on that server refuses a retired id. The client refuses RetainIDs locally, sending nothing, when the server does not advertise it: a sweep that dropped the member to get past an older server would leave every swept id reusable. The interim refuse row W-SweepRequest.RetainIDs the storage commit added is retired (kept as history, like the S4 rows). - createIssue and applyBatch answer a retired id with 410 idempotency_expired (param naming the id, or the batch item's members, plus closed_at), classified ahead of the validation and already-exists arms; the client maps the code back to the typed *issueops.IdempotencyExpiredError. - The http leg wires the IDRetention contract through the served fixture and retires the series' interim waivers. Five cases stay waived by name: the in-transaction one (no transaction spans requests), the two that drive a store's upsert-shaped CreateIssue, which the wire does not serve (createIssue is create-only), and the two import cases: the wire publishes no import operation, so the client hands out no Importer. All five run on the store legs. - apigen is regenerated from the spec; the wire-shape golden follows. Ported from gascity/bd-enterprise#210 (c6e938737). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(storage): create-only batches take every id's shard cell once, sorted A create-only batch (ExecuteCreateBatch, a CreateOnly CreateIssues) took its coordination cells one item at a time, in item order, through each row's EnsureIssueIDAvailableInTx, and an apply-batch took them per create item through ExecuteCreate. Retaining sweeps, upsert batches and imports lock large sorted cell sets, so on a backend that runs these doors read committed (bd-enterprise's Postgres store) such a batch could hold a later shard while it waited on an earlier one and deadlock against them (40P01). A SERIALIZABLE envelope or an importer retries that; a single-attempt, read-committed CreateIssues or CreateIssuesWithFullOptions, which `bd import` calls, would surface it. - newCreateBatchCache coordinates every id the batch names in one sorted write for CreateOnly batches too, and a row whose cell the batch already holds runs only the occupancy probe (checkIssueIDAvailableInTx). A row the cache never saw (a generated id) still takes its own cell. - ApplyBatchInTx takes the cells of every explicit create id in one sorted write before its first item. - CoordinateIDCreatesInTx's doc no longer says sorted order "cannot deadlock" unconditionally: it holds between callers that take all their cells in one call, and names the residual (a generated id minted mid-transaction). Tests (sqlmock statement order): a CreateOnly batch locks both ids in one sorted write before its reads and its rows probe without re-locking; a row outside the up-front set still locks first; an apply-batch's first statement is the sorted lock of its explicit create ids. Ported from gascity/bd-enterprise#210 (0802d1640). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(uow): guard only caller-named ids against retirement in the upsert Insert domain/db's upsert Insert ran RefuseRetiredIDInTx for every create, including ids the use case had just minted and child-counter ids, where issueops' create body guards only explicit ids. Every node of an applyGraph, a createMany, a graph apply or a molecule pour through the proxied server paid a coordination cell write and a four-table probe for nothing (the generators already skip retired ids), and on a Dolt server two concurrent 30-node graph applies shared a cell about a fifth of the time, so the loser retried its whole unit of work. InsertIssueOpts gains ExplicitID, set by the create use case from createIDNamedByCaller (ExplicitID, or an id preset on the issue with no parent whose child counter replaces it), and the upsert Insert coordinates and probes only then. The CreateOnly branch is unchanged: it runs the create-only guard for every id, as issueops does. Tests: a generated-id upsert's first statement is its row write; an explicit-id upsert coordinates and probes first; the explicitness rule over ExplicitID, preset, parent and minted ids. Ported from gascity/bd-enterprise#210 (8452e0a74). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(storage): persistence moves and wisp promotion take the id's shard cell A persistence move and a wisp promotion re-insert a row into the other plane without the coordination cell or a fresh look at the row. Raced against a retaining sweep of the same closed row on Dolt, both deleted the source row (identical deletes merge cleanly), the move's insert landed in the other plane, and the sweep's tombstone committed beside it: a live row next to a tombstone, which the contract says never happens. Read-committed doors (bd-enterprise's Postgres store) behaved the same way. Creates still answered 409, so nothing re-executed, but the swept row came back. - moveIssuePersistenceInTx coordinates the id before it reads where the row lives, and a row found in neither plane (a sweep deleted it after the caller read it) is refused with ErrNotFound instead of re-inserting the caller's earlier snapshot. - PromoteFromEphemeralInTx coordinates the id before it reads the wisp. So a move or promotion that comes second behind the sweep finds no row, and one that comes first makes the sweep conflict (Dolt) or fail serialization (a SERIALIZABLE envelope) and retry. Tests (sqlmock statement order): a move's and a promotion's first statement is the id's coordination write; a move of a row in neither plane is ErrNotFound and writes nothing. Ported from gascity/bd-enterprise#210 (6c5f9e5c3). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(dolt): replay a sweep that lost the commit-time merge; state the lock footprint The Dolt store's sweeper ran its write in withWriteTx, a single attempt. A retaining sweep writes one create-coordination cell per shard it retires, and the cells are 4096 shards shared by unrelated ids, so on a sql-server any create, upsert or import committing in one of those shards first conflicts the sweep (1213/1205 or the exact 1105 autocommit rollback). At a limit of 1000 that is about 22% of all shards, and at a load of about 5.8 creates/s per shard most such sweeps would have failed outright. - The sweeper writes through withRetryTx, as the deleter does: a rollback-guaranteed conflict replays the whole body, candidates re-read and the result reset at the top of each attempt. The uow sweeper (RunTxResult) already replays; the embedded store serializes its transactions. - SweepRequest.RetainIDs documents the footprint: about 4096·(1-e^(-N/4096)) shard cells held until commit, what that does to creates on a read-committed backend (they wait) and on a Dolt sql-server (one side conflicts and replays), and to size Limit around 100 and loop on Remaining. Test: a sweep whose first SQL COMMIT is refused with 1213 is replayed and succeeds (fake driver; two COMMITs). Ported from gascity/bd-enterprise#210 (0b1ac36c2). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(conformance): retire rows through the wisps-plane retaining sweep on every leg No IDRetention case swept with tier wisps-plane, the exact request an orchestrator sweeping its operation rows sends (wisps-plane, retain_ids, protect_referenced off, a limit). That tier has its own candidate path on every backend (SearchWispsPlaneInTx; sweepCandidatesInUOW on uow), so the durable and ephemeral cases did not cover it. RunIDRetentionAWispsPlaneRetainingSweepRetiresItsRows seeds a closed no-history row and a closed ephemeral row, sweeps them with that request, and asserts both tombstones land in wisp_id_tombstones with the rows' closed_at, that Remaining is 0, and that a re-create of either is refused through Lifecycle.Create and, where served, the upsert-shaped create. It is in the role bundle and wired on the dolt, embedded, uow and served (http) legs. Ported from gascity/bd-enterprise#210 (76bcf085c). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(cli): report each retired import row once and never as an update The import's accounting of retired rows disagreed with itself in places: - assembleImportResult listed a retired id once per row carrying it; the report and the spec say RetiredIDs is deduplicated. - A row the pre-filter planned as an update (live when it read it) that a retaining sweep retired before the batch wrote was reported both in UpdatedIssues/Updated and in RetiredIDs. The batch skipped it, so it is now reported retired only. - The dry-run classifier checked retired ids with slices.Contains per row, quadratic in the rows of an export full of retired op rows; it now uses sets. - Under ConflictSkip a retired row is skipped silently and still counted, as every row ConflictSkip carries is. That is kept (ConflictSkip is the auto-import recovery path, which counts the rows it submitted) and now stated on BatchCreateOptions.OnRetiredID. Test: assembleImportResult over a file with a duplicated retired row and a planned update that turned out retired. Ported from gascity/bd-enterprise#210 (6c34e757e). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: state where the IDRetention race cases bite and how to size a retaining sweep - The IDRetention race cases say where they can fail: only on a backend whose upsert and import doors run READ COMMITTED (bd-enterprise's Postgres legs). The embedded leg and the http leg served over it serialize transactions, and on a Dolt sql-server snapshot reads plus commit-time row conflicts refuse the interleavings anyway: with CoordinateIDCreatesInTx made a no-op, the uow leg (doltserver lane, remote) still passes, as the embedded leg did for the reviewer. Green on those legs is no evidence for the race fix. - CHANGELOG and docs/workflows/wisps.md say a retaining sweep holds the coordination cells of the ids it retires until it commits, and to run large sweeps in batches of about 100 (bd purge --limit; bd prune has no limit, so narrow it with --older-than). Ported from gascity/bd-enterprise#210 (b0873bffa). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(storage): refuse a rename onto a retired id UpdateIssueIDInTx renamed a live row onto any id no live row held, so a rename (bd rename, bd rename-prefix, the storage-level UpdateIssueID) could commit a live row beside a retaining sweep's tombstone — the state the CHANGELOG says no write path produces. It was deterministic: on the embedded store, UpdateIssueID(live -> retired) returned nil and left one live row and one tombstone. The rename now calls RefuseRetiredIDInTx(newID) before anything else: it takes newID's create-coordination cell and probes the tombstones, so a retired target is refused with *IdempotencyExpiredError and nothing is written, racing sweep or not. Tests: - TestRenameOntoARetiredIDIsRefusedBeforeItWrites (sqlmock): the cell and the probe come first and a retired target stops the rename. - New IDRetention case RunIDRetentionRefusesARenameOntoARetiredID (both planes; the refused rename leaves the source row in place), with a new IDRetentionFixture.Rename door wired on the dolt and embeddeddolt legs and waived by name on uow and http, which serve no identity change. - CHANGELOG names the rename refusal. Ported from gascity/bd-enterprise#210 (556ecf460). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(dolt): take the coordination cell before demoting a row to wisps DoltStore.demoteToWispInTx — the Dolt store's own persistence move, reached from the storage-level UpdateIssue and UpdateIssueChecked when the updates carry wisp or no_history — re-inserted the row into wisps without the id's create-coordination cell. On a Dolt sql-server a retaining durable sweep and the demotion could both commit: their identical deletes of the issues row and aux rows merge cleanly, and the wisps insert and the tombstone insert touch different tables, leaving a live wisp beside an id_tombstones row (review finding 6's state). The demotion now calls CoordinateIDCreatesInTx for its id before its first read of the row, as moveIssuePersistenceInTx and PromoteFromEphemeralInTx do. A retaining sweep takes the same cell before it deletes, so the two conflict on the cell and one replays: the demotion either commits first (and the sweep then finds no closed issues row) or finds the row gone and fails its update. Test: TestDemoteToWispCoordinatesBeforeItTouchesTheRow (sqlmock) pins the cell write as the demotion's first statement. Ported from gascity/bd-enterprise#210 (437ea2ca8). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(import): count every row of a retired id out of Created The unit of work's Importer set Created to the request's rows minus the stale-rejected ids minus the DISTINCT retired ids. A request may carry an id more than once, and the batch skips every row of a retired id, so a request carrying a retired id twice overcounted Created by one per extra copy, contradicting ImportBatchResult.Created's "excluding ... retired rows". Created now subtracts the request rows whose id is in the retired set. The set is per attempt and keyed by id, so a hook that fires again on a replay still counts each row once. Stale-rejected rows keep their existing per-id accounting; this change does not touch it. The store legs' upsert importer (conformance.IDRetentionUpsertImporter) counts the same way. Test: the IDRetention import case now carries the retired row twice and still wants Created == 1 (on enterprise it failed with Created = 2 before the fix). Ported from gascity/bd-enterprise#210 (195e730e9). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(storage): skip retired child suffixes and retired auto-import rows Review fixes for gastownhall#7931, iteration 1. The child id generators (issueops.GetNextChildIDTx and domain/db's NextChildID) took the next suffix from max(child counter, highest live child) and read no tombstone table. Once a retaining sweep deletes the highest child, its tombstone is the only record that the suffix was used, and the counter can lag it: a database written before GH#4750, the uow route's explicit P.N create, or a re-created wisp parent whose counter row was deleted with it. bd create --parent then refused the minted id with idempotency_expired on every retry, and the uow upsert door (proxied bd q --parent) wrote a live row beside its tombstone. Both generators now fold both tombstone tables into their one high-water read (issueops.UsedChildIDsInTx), and the IDRetention contract gains NeverMintsARetiredChildID on every leg. Under ConflictSkip a retired row was skipped before OnRetiredID could fire, so the embedded auto-import (ImportJSONLData) and the server-mode fallback (importFromLocalJSONLConflictSkip) counted rows they never wrote as imported. OnRetiredID now fires under ConflictSkip too. ImportJSONLData registers it and returns the retired ids, and both auto-import paths count only the rows they wrote, report the skipped ones, and stamp a file whose every row was retired, so the next command does not read it again. The chunked import's dependency pass keeps the callback off, as it already does OnStaleRejected. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
The PR head (538cec7) neither rebased nor merged cleanly onto main (6d713a7, 382 commits past its 2026-10-05 merge base): 25 conflict hunks in 12 files. Each pairs two independent changes, and the resolution keeps both: - issueops/journal.go: main's arrival_seq stamp (#7912) marshals the issue snapshot inside insertEventRow's record closure, after the stamp. The PR's shape-adaptive INSERT is now built there, by journalInsert, from that snapshot. - dolt, embeddeddolt and uow stores: main replaced versionedHistoryEnabled with the deferred VersionedHistorySwitch (#7743). The PR's journalShape and journalActivationErr fields and its shape scope sit beside it. The uow provider keeps main's proxied-server-mode (#7481) and server-log-rotation (#7735) options and the PR's working-set-reconcile option. - cmd/bd factories: main's versioned-history activation defer (#6661) stays registered first, so it still runs after the journal's; the journal defer carries the PR's bead-writing posture. main folded newDoltStoreFromConfig into openDoltStoreFromConfig (#7751), which also serves newExistingDoltStoreFromConfig. Both open for writes, so it passes true, as the PR's newDoltStoreFromConfig did. - migrate_personal.go: main opens the planning store through the routed-write factory (#7751), which now activates with true, so the PR's own change to the removed dolt.New open is carried there. - doctor/fix/store.go, main.go, uow_factory.go, uow_factory_test.go: both sides' changes. - embedded-storage-test-shards.txt: main's file (40-shard block, #7440), with the PR's four embedded tests placed by gen_embedded_storage_shard_manifest.py 40 --weights=duration --write. Co-Authored-By: Claude <noreply@anthropic.com>
…srcs TestArrivalSeqIsWrittenOnlyByTheJournalSeam walks every non-test Go file under internal/storage, but under Bazel it sees only issueops_test's data, a hand-picked package list. That list left out historyoffequiv, so the guard's false positive there (gastownhall#8025) failed Main's macOS `go test` job on every push since gastownhall#7912 while the gating Bazel lane stayed green. Adding historyoffequiv alone still leaves 26 packages (81 non-test files) where a real arrival_seq write would go unseen by Bazel. Declare //internal/storage:tree_go_srcs instead. tools/bazel/go_srcs.py generates it and TestBazelTreeGoSrcsListsEveryPackage holds it to every package under the tree, so the Bazel view now matches `go test` (520 non-test files, against 439 for the list) and a new package is scanned without an edit here. //internal/storage:storage_test declares it the same way. It subsumes the historyoffequiv entry this PR added, this package's own :go_srcs, and the sibling packages TestAllIssueRowWritesStampRowLock and TestEpochCASTestsBuildEveryAddressWithEpochAddress read. Cost: the target also re-runs when Go files change in those 26 packages (143 of the tree's 1337 Go files); the list already held the busiest ones (dolt, embeddeddolt, uow, issueops). Agent-Signature: claude-code-unknown-model-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hall#8033) * fix(issueops): distinguish SQL in arrival sequence guard (ocb-85cac) Use the existing Vitess tokenizer to exclude prose, comments and quoted values while retaining SQL write fixtures and the writer allowlist. Declare historyoffequiv scan data for Bazel. Go tests and Bazel lint are unrun: required exact module versions are not cached and acquisition is forbidden for this source-preparation task. Managed Beads precommit ran separately under the recorded standing hook exception. Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Vishnu Jayavel * fix(issueops): retain arrival sequence fragment detection (ocb-85cac.3) Match tokenized write keywords and the column throughout each statement, including format-prefixed writes and upsert fragments. Add 15 positive and exclusion fixtures and log the scan coverage. The four selected offline tests passed with 51 classifier cases. Bazel lint and full tests are unrun under the admitted resource scope. Managed Beads precommit ran separately under the recorded standing hook exception; prepare-commit-msg remains enabled. Agent-Signature: codex-gpt-6.1-sol-high on behalf of Vishnu Jayavel * fix: require SQL origin in arrival sequence guard (ocb-85cac.4) Keep the existing SQL and fragment fixtures while excluding prose with write words or a later parenthesis. Add the three review counterexamples. Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Vishnu Jayavel * test(issueops): scan all of internal/storage under Bazel via tree_go_srcs TestArrivalSeqIsWrittenOnlyByTheJournalSeam walks every non-test Go file under internal/storage, but under Bazel it sees only issueops_test's data, a hand-picked package list. That list left out historyoffequiv, so the guard's false positive there (gastownhall#8025) failed Main's macOS `go test` job on every push since gastownhall#7912 while the gating Bazel lane stayed green. Adding historyoffequiv alone still leaves 26 packages (81 non-test files) where a real arrival_seq write would go unseen by Bazel. Declare //internal/storage:tree_go_srcs instead. tools/bazel/go_srcs.py generates it and TestBazelTreeGoSrcsListsEveryPackage holds it to every package under the tree, so the Bazel view now matches `go test` (520 non-test files, against 439 for the list) and a new package is scanned without an edit here. //internal/storage:storage_test declares it the same way. It subsumes the historyoffequiv entry this PR added, this package's own :go_srcs, and the sibling packages TestAllIssueRowWritesStampRowLock and TestEpochCASTestsBuildEveryAddressWithEpochAddress read. Cost: the target also re-runs when Go files change in those 26 packages (143 of the tree's 1337 Go files); the list already held the busiest ones (dolt, embeddeddolt, uow, issueops). Agent-Signature: claude-code-unknown-model-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What
arrival_seq: a commit-ordered create sequence stamped on wisp-plane rows. While a workspace journals, the create of anephemeralorno_historybead writes theseqof its owncreatejournal record onto the row, in the same transaction. Sorting surviving rows byarrival_seqthen reproduces the order their creates committed in, even after the journal records are pruned. It works on every backend: Dolt sql-server, embedded Dolt, the unit-of-work (proxied) path, andbd serve/HTTP.Why
A consumer that tails the events journal and falls behind the retained window gets a 410 or truncation and re-baselines from rows. Rows alone do not carry their create order:
created_atis the writer's clock at one-second precision. So the create's journal sequence has to live on the row. The stamp follows FoundationDB's versionstamp idea. It is platform-owned, never a producer field, and it costs no second counter, because the stamp is the journal seq.The first consumer is gascity's conversation-lane op journal. Ops are wisps, and the lane orders them by arrival. The gascity design references, on the remote-worker design branch, not yet on gascity main:
engdocs/design/conversation-operation-journal.md§3.5 "Ordering": "the create's journal sequence must be stamped on the row asarrival_seq… gc reads it from the row column (OJ-10a)". Decision DP8.engdocs/design/BUILD-PLAN.md: the BD-5 row, gate G-BD5 ("BD-5 in a bd release, plus the gc pin bump that takes it"), and "Reserve those slots before BD-5 or BD-6 merges into enterprise".This was built and reviewed downstream in gascity/bd-enterprise#204 (private). This PR is its OSS storage core. The Postgres backend, the HTTP import operation's
ImportIssuemember and the enterprise HTTP import role stay downstream. #204 is being renumbered onto exactly these slots, with byte-identical migration files.Ordering semantics (what a consumer can rely on)
createrecord that stamped them. For a row created while the workspace journals, that record is its insert. The value never changes while the row stays on the wisp plane: updates, closes, reopens, labels, comments, dependency edges and claims leave it alone.bd exportnever writes them, and the replicatedissuesplane carries the column for shape only (always NULL), so Dolt push, pull and merge never see a value.seq. The loser fails the commit-time merge and retries. The counter row alone does not decide it, because two increments to the same value merge cleanly; the journal PK does.bd serve: inherits the guarantee from the store behind it.GetIssue,SearchIssues, list and brief list.CreateIssue,CreateIssuesandTransaction.CreateIssuewrite the stamp back onto the caller's struct (RecordCreateEventInTx).issues.arrivalSeq. The token says the server knows the member. Whether rows are stamped follows journal activation, whichevents.listalready reports with409 events_journal_disabled.bd create/show/list/ready/query --jsoncarry it, omitted when 0.stampArrivalSeqInTx.Migration slots: main 0073, ignored 0029
0073_add_arrival_seq(up/down): a nullableBIGINTonissuesandwisps, with guarded PREPAREs (0060/0067 shape) and a CLI-bundle direct-DDL substitute.ignored/0029_add_wisps_arrival_seq: does three things.UNIQUEindexuk_wisps_arrival_seq. NULLs coexist.bd_events_seqpastMAX(arrival_seq). This last step closes a gap found in review: whenbd_events_seqis lost, the ignored series replays from 0022, which re-seeds the counter from the journal alone. With a pruned journal, the next create could take a free value below rows that are still there, with no collision for the heal to catch.CONFLICTINGagainst main and waiting on a maintainer contract decision (whether labels and comments remint the token), so this PR takes the next contiguous slots. Whichever of the two lands second renumbers. fix: make row_lock an aggregate revision #5738's maintainer-side rebase has to move past main anyway, so if it lands first, this PR moves to 0074/0030 (and bd migrate irregular version reporting, disk i/o error, renames database, ignores config.json #204 with it).scripts/check-migration-hygiene.shpasses againstorigin/main.Evidence it works
Bazel only, on rbe-west, at head
b663791cb4:bazel test //... --config=ci--config=embedded--config=doltserver--config=dolt-race--config=doltserver-integration--config=integration//cmd/bd:bd_dolt_server_test --config=doltserver-cmd--config=doltserver-proxiedmake ci-pr-lint(nogo native plus windows/darwin)scripts/check-migration-hygiene.sh(baseorigin/main)What the tests pin:
Conformance. 12 Journal-role arrival cases, run on the dolt, embedded, unit-of-work and served-HTTP legs:
The two store legs drive import through
CreateIssuesWithFullOptions, the seambd importruns on them. The unit-of-work leg uses itsImporter. The served leg skips import (no import operation on the wire) and the explicit-id batch.Embedded engine. Racing writers are unique; sequential creates stamp in commit order; both heal orders work, including this engine's own unique-key error kind; many NULLs coexist under the index; creates hand the stamp back to the caller; the replayed counter seed clears surviving stamps.
Dolt sql-server (
domain/db). Overlapping writers, the untouched-columns rule (row_lock, current_revision and content_hash are left alone), and the missing-column heal fallback.Schema. The slot pins (0073 supersedes 0072's), the frozen bytes, the CLI substitute, and full-chain CLI convergence including
uk_wisps_arrival_seq. The history-off differential sets the new columns aside (schemaAdditions).Wire. The spec and wire-shape golden are additive (
wire_revisionunchanged), the create response carries the stamp, and the behavior-token vocabulary is mirrored client-side.CLI.
TestEmbeddedArrivalSeqOnEveryJSONRead:bd create/show/list/ready/query --jsoncarry each wisp's stamp, equal to itsbd events exportcreate seq. The durable bead carries none, andbd exportwrites none.Mutants, killed on the embedded engine:
issue.ArrivalSeq == 0clause fails ReimportKeepsTheStamp and RenameKeepsTheStamp: "matched 0 rows, want 1".Costs: one extra
UPDATEby primary key per stamped create, and the unique index is maintained on every wisp insert and delete. A workspace that does not journal pays no extra statement.Layout
Twelve commits, layered so they can be split into a stack if you prefer one layer per PR:
bd exportstrip4–8. test hardening and docs
Checklist
make testlane (--config=ci, nogo included) and the Dolt tiers pass locally on rbe-west (table above).docs/reference/events-journal.md"Arrival order on the row", plus a CHANGELOG entry.AGENTS.mdchanges.🤖 Generated with Claude Code