Skip to content

feat(storage): commit-stamped create sequence (arrival_seq) on every backend (BD-5) - #7912

Merged
gascity-eddie[bot] merged 13 commits into
mainfrom
feat/arrival-seq-bd5
Oct 27, 2026
Merged

gascity-eddie[bot] merged 13 commits into
mainfrom
feat/arrival-seq-bd5

Conversation

@gascityinc-randy

@gascityinc-randy gascityinc-randy Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What

arrival_seq: a commit-ordered create sequence stamped on wisp-plane rows. While a workspace journals, the create of an ephemeral or no_history bead writes the seq of its own create journal record onto the row, in the same transaction. Sorting surviving rows by arrival_seq then reproduces the order their creates committed in, even after the journal records are pruned. It works on every backend: Dolt sql-server, embedded Dolt, the unit-of-work (proxied) path, and bd serve/HTTP.

Why

A consumer that tails the events journal and falls behind the retained window gets a 410 or truncation and re-baselines from rows. Rows alone do not carry their create order: created_at is the writer's clock at one-second precision. So the create's journal sequence has to live on the row. The stamp follows FoundationDB's versionstamp idea. It is platform-owned, never a producer field, and it costs no second counter, because the stamp is the journal seq.

The first consumer is gascity's conversation-lane op journal. Ops are wisps, and the lane orders them by arrival. The gascity design references, on the remote-worker design branch, not yet on gascity main:

  • engdocs/design/conversation-operation-journal.md §3.5 "Ordering": "the create's journal sequence must be stamped on the row as arrival_seq … gc reads it from the row column (OJ-10a)". Decision DP8.
  • engdocs/design/BUILD-PLAN.md: the BD-5 row, gate G-BD5 ("BD-5 in a bd release, plus the gc pin bump that takes it"), and "Reserve those slots before BD-5 or BD-6 merges into enterprise".

This was built and reviewed downstream in gascity/bd-enterprise#204 (private). This PR is its OSS storage core. The Postgres backend, the HTTP import operation's ImportIssue member and the enterprise HTTP import role stay downstream. #204 is being renumbered onto exactly these slots, with byte-identical migration files.

Ordering semantics (what a consumer can rely on)

  • Order key. Unique within one workspace and never reused, across deletes, re-creates and counter heals. Rows are ordered by the commit of the create record that stamped them. For a row created while the workspace journals, that record is its insert. The value never changes while the row stays on the wisp plane: updates, closes, reopens, labels, comments, dependency edges and claims leave it alone.
  • Gaps are normal. Every journal record consumes a seq. Never treat a gap as a missing row.
  • Scope. Values compare only within one database. bd export never writes them, and the replicated issues plane carries the column for shape only (always NULL), so Dolt push, pull and merge never see a value.
  • Plane. Durable beads carry none. Promotion to the durable plane clears it.
  • Unstamped rows. A row created with the journal off stays at 0 until a later synthetic create stamps it: an import upsert or a rename. It then sorts by that later commit. A synthetic create of an already-stamped row never moves the stamp.
  • Commit order per engine.
    • Dolt sql-server: two writers that draw the same seq both insert journal PK seq. The loser fails the commit-time merge and retries. The counter row alone does not decide it, because two increments to the same value merge cleanly; the journal PK does.
    • Embedded Dolt: writers are serialized.
    • bd serve: inherits the guarantee from the store behind it.
  • Reads.
    • Storage API: GetIssue, SearchIssues, list and brief list. CreateIssue, CreateIssues and Transaction.CreateIssue write the stamp back onto the caller's struct (RecordCreateEventInTx).
    • HTTP: every Issue-shaped schema carries it, announced by the build-level behavior token issues.arrivalSeq. The token says the server knows the member. Whether rows are stamped follows journal activation, which events.list already reports with 409 events_journal_disabled.
    • CLI: bd create/show/list/ready/query --json carry it, omitted when 0.
  • Never client-settable. Create, update and import ignore a supplied value. The generic insert and upsert column lists never name the column, and an AST guard holds that. The only writer is stampArrivalSeqInTx.

Migration slots: main 0073, ignored 0029

Evidence it works

Bazel only, on rbe-west, at head b663791cb4:

Lane Result
bazel test //... --config=ci 198/198
--config=embedded 10/10
--config=doltserver 10/10
--config=dolt-race 12/12
--config=doltserver-integration 2/2
--config=integration 156/156
//cmd/bd:bd_dolt_server_test --config=doltserver-cmd pass (1 target, 32 shards)
--config=doltserver-proxied 2/2
make ci-pr-lint (nogo native plus windows/darwin) pass
scripts/check-migration-hygiene.sh (base origin/main) OK

What the tests pin:

  • Conformance. 12 Journal-role arrival cases, run on the dolt, embedded, unit-of-work and served-HTTP legs:

    • the stamp equals the record seq and the post-image
    • batch item order
    • every read path
    • immutability
    • journal off
    • plane
    • create order recovered after a full prune
    • no reuse across delete
    • supplied values ignored
    • re-import, first create record, rename

    The two store legs drive import through CreateIssuesWithFullOptions, the seam bd import runs on them. The unit-of-work leg uses its Importer. The served leg skips import (no import operation on the wire) and the explicit-id batch.

  • Embedded engine. Racing writers are unique; sequential creates stamp in commit order; both heal orders work, including this engine's own unique-key error kind; many NULLs coexist under the index; creates hand the stamp back to the caller; the replayed counter seed clears surviving stamps.

  • Dolt sql-server (domain/db). Overlapping writers, the untouched-columns rule (row_lock, current_revision and content_hash are left alone), and the missing-column heal fallback.

  • Schema. The slot pins (0073 supersedes 0072's), the frozen bytes, the CLI substitute, and full-chain CLI convergence including uk_wisps_arrival_seq. The history-off differential sets the new columns aside (schemaAdditions).

  • Wire. The spec and wire-shape golden are additive (wire_revision unchanged), the create response carries the stamp, and the behavior-token vocabulary is mirrored client-side.

  • CLI. TestEmbeddedArrivalSeqOnEveryJSONRead: bd create/show/list/ready/query --json carry each wisp's stamp, equal to its bd events export create seq. The durable bead carries none, and bd export writes none.

  • Mutants, killed on the embedded engine:

    • Dropping the counter raise from ignored/0029 fails the replay test: "re-seeded the counter at 0, below the surviving stamp 3".
    • Dropping the issue.ArrivalSeq == 0 clause fails ReimportKeepsTheStamp and RenameKeepsTheStamp: "matched 0 rows, want 1".

Costs: one extra UPDATE by primary key per stamped create, and the unique index is maintained on every wisp insert and delete. A workspace that does not journal pays no extra statement.

Layout

Twelve commits, layered so they can be split into a stack if you prefer one layer per PR:

  1. schema and read path
  2. the journal writer
  3. wire, conformance, bd export strip
    4–8. test hardening and docs
  4. create hand-back
  5. counter raise on replay
  6. CLI JSON test
  7. history-off differential

Checklist

  • The why is documented in this description.
  • make test lane (--config=ci, nogo included) and the Dolt tiers pass locally on rbe-west (table above).
  • User-facing docs: docs/reference/events-journal.md "Arrival order on the row", plus a CHANGELOG entry.
  • No invariant in an AGENTS.md changes.

🤖 Generated with Claude Code

julianknutsen and others added 12 commits October 27, 2026 15:09
…types.Issue

BD-5, part 1 of 3: the column and its read path, with nothing writing it yet.

arrival_seq is the commit-ordered create sequence the events journal will
stamp on a wisp-plane row: the seq of the row's own create record, stored on
the row in the same transaction, so a consumer can recover create order from
rows after the journal records are pruned.

- Migration 0073 adds a nullable BIGINT to issues and wisps (guarded
  INFORMATION_SCHEMA + PREPARE, 0060/0067 shape). The column only means
  something on wisps, which is clone-local; issues carries it for shape
  parity and it stays NULL, the wisps.row_lock/current_revision precedent.
  The CLI bundle gets the direct-DDL override (dolthub/dolt#11345) and the
  substitute is listed in cliSubstituteAssumesWispTables.
- Ignored migration 0029 carries the wisps column to fresh clones and
  creates the clone-local UNIQUE index uk_wisps_arrival_seq.
- TestLatestVersionIncludesMigration0073 supersedes 0072's pin, the way each
  slot's pin has superseded the previous one.
- IssueBaseColumns and IssueBaseColumnsLite select it; ScanIssueFrom and
  ScanIssueLiteFrom read it into types.Issue.ArrivalSeq (json:"-" until the
  wire half lands). The legacy SQLite projection gains its placeholder.
- TestEmbeddedIgnoredMigration0023NoopsOnHealthyWorkspace now stops the
  ignored pass right after 0023 (SetMigrateStepFaultHookForTest) and compares
  SHOW CREATE TABLE across that step alone. It used to run every ignored
  migration and require nothing to change, which ignored/0029 now breaks by
  design: it adds uk_wisps_arrival_seq to wisps. The test still proves what it
  names, that 0023 is a no-op on a healthy workspace.
- TestFullChainFromPreWispsAndMissingDependenciesIDConvergesThroughDoltCLI
  replays the ignored series through the dolt CLI, which no-ops 0029's
  PREPARE'd ADD COLUMN (the dolthub/dolt#11345 limitation 0006 already works
  around there), so 0029's PREPARE'd unique index would then fail on a missing
  column. The test applies 0029's effect as Go-decided direct DDL before the
  frozen text, exactly as it does for 0006, and its convergence and re-open
  checks cover wisps.arrival_seq and uk_wisps_arrival_seq.

Ported from gascity/bd-enterprise#204 (12f0a7360), renumbered from main
0070 / ignored 0028 to the next free upstream slots; the Postgres half of
that commit stays downstream.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BD-5, part 2 of 3: the writer.

RecordEventInTx now decides, once, whether a create stamps: the op is a
create, the row was read from wisps, and it carries no stamp yet. When it
does, insertEventRow draws the seq, writes it onto the row
(stampArrivalSeqInTx), and only then marshals the post-image, so the
record's seq, the row's arrival_seq and the post-image agree. There is no
second counter: the stamp IS the journal seq, so it inherits the counter's
commit-ordered allocation on every engine, and a workspace that does not
journal pays nothing and stamps nothing.

- The stamp writes arrival_seq alone (updated_at self-assigned against
  Dolt's ON UPDATE CURRENT_TIMESTAMP), so row_lock, current_revision,
  updated_at and content_hash are untouched and a caller's post-create CAS
  still holds. It must match exactly one row.
- A collision on the journal seq OR on uk_wisps_arrival_seq goes through
  the same single heal-and-retry; the retry moves a stamp the failed
  attempt already wrote. isSeqCollision also recognizes the embedded
  engine's unique-key error kind.
- healEventSeqCounter raises the counter past MAX(arrival_seq) too, and
  falls back to the journal alone on a drifted clone with no wisps column.
- PrepareIssueForInsert and the unit-of-work Insert clear ArrivalSeq, and
  the generic insert/upsert column lists never name it: no caller-supplied
  value is ever stored, and a plane move leaves the row unstamped.

Guards: TestArrivalSeqIsWrittenOnlyByTheJournalSeam (AST scan of every SQL
literal under internal/storage), TestIssueInsertAndUpsertColumnsExcludeArrivalSeq,
TestOnlyRecordEventInTxRecordsCreates; the row_lock and version
completeness guards learn the stamp is sink bookkeeping. Dolt-server and
embedded tests pin commit order under overlapping and racing writers,
uniqueness, the untouched-columns rule, both heal orders, the
missing-column fallback, and that many NULLs coexist under the index. The
40-shard embedded storage manifest places the new embedded tests.

Ported from gascity/bd-enterprise#204 (9ca7815c5); the Postgres refusal and
its open-time reconcile stay downstream.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BD-5, part 3 of 3: the member on every Issue-shaped answer, the contract
cases on every leg, and the interchange boundary.

- types.Issue.ArrivalSeq is now `arrival_seq,omitempty`: absent on every
  unstamped row, so a reader that predates it sees what it saw before, and
  outside the content hash. `bd show`/`list`/`ready --json` carry it.
- The spec adds an optional int64 arrival_seq to Issue, BatchGetIssue,
  IssueWithCounts, IssueDetails, IssueWithDependencyMetadata, TreeNode and
  BlockedIssue, documents the post-image carrying it on EventRecord.issue,
  and lists the new build-level behavior token issues.arrivalSeq. Whether
  rows are being stamped follows journal activation, which events.list's 409
  already reports. apigen and the wire-shape golden are regenerated; the
  change is additive, so wire_revision is unchanged. The client mirrors the
  token (wire.CapArrivalSeq).
- bd export and auto-export strip clone-local fields from every record
  (stripCloneLocalFields), held by an AST guard over every
  exportIssueRecord builder.
- backend/conformance gains nine Journal-role cases (definition, batch
  order, every read path, immutability, journal-off, plane, create order
  recovered after the journal is pruned past it, no reuse across delete,
  supplied values ignored) with hooks bound to the public roles. They run
  on the dolt, embedded, unit-of-work and served-HTTP legs. The Importer
  role is optional (only the unit-of-work provider hands one out); the two
  store legs drive the import cases through CreateIssuesWithFullOptions,
  the seam `bd import` runs on them, and the served leg leaves the
  explicit-id batch, the claiming create and import to the store legs.
- docs/reference/events-journal.md documents "Arrival order on the row".

Ported from gascity/bd-enterprise#204 (cf9f1d431 and 7bb0899e3); the
ImportIssue member, the HTTP importer's strip and the Postgres leg stay
downstream, where the import operation and Postgres live.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both heal tests wound the counter back to high-1. The first draw is then high,
which is already the counter's own value, so a heal that consulted only the
journal (GREATEST(next_seq, MAX(seq))) also lands on high+1 and the tests pass
without the MAX(arrival_seq) term they exist to pin.

Wind back three instead, with the journal empty: a journal-only heal leaves the
counter in place and the retry collides with the next surviving stamp. Verified
with a go test -overlay mutant that drops the arrival term from
healEventSeqCounter: TestArrivalSeq_EmbeddedHealRaisesTheCounterPastTheStamps
now fails with "duplicate unique key given: [2]" and passes on the real code.
The domain/db twin takes the same change (Dolt-container tier).

Ported from gascity/bd-enterprise#204 (8616c3cf4).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… stamp

RecordEventInTx stamps only a wisp row that has no stamp yet
(issue.ArrivalSeq == 0). That clause is what keeps the synthetic create records
an import upsert and an id rename journal for an existing row from restamping
it, and nothing tested it: an overlay mutant that drops the clause passed every
embedded Journal, Import, Rename and Wisp test.

Three Journal-role cases, declared before EveryMutationKindLandsARow and wired
on every leg:

- ArrivalReimportKeepsTheStamp: re-importing a stamped wisp leaves the stamp,
  journals a create with its own higher seq, and that record's post-image
  carries the original stamp. A real flow: bd export keeps no_history wisps.
- ArrivalFirstCreateRecordStampsAnUnstampedRow: a wisp created with the journal
  off is stamped by the first create record journaled for it, with that
  record's seq.
- ArrivalRenameKeepsTheStamp: the create a rename journals carries the row's
  stamp. Driven through a new optional JournalArrivalHooks.Rename, bound by
  conformance.ArrivalRenameHook on the dolt and embedded legs (the stores that
  expose UpdateIssueID); the served and unit-of-work legs skip it by name.

The import cases run on the dolt and embedded legs through their bulk upsert
(ArrivalUpsertImportHook) and on the unit-of-work leg through its Importer;
the served leg skips them by name, since the wire publishes no import.

Ported from gascity/bd-enterprise#204 (1efd4d571).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- TestArrivalSeq_HealFallsBackWithoutTheWispColumn drops wisps.arrival_seq on
  the suite-shared database and restored it with `_, _ =`. wisps is
  dolt_ignored, so SetupTest's DOLT_RESET never repairs it, and a failed
  restore would surface as unrelated failures later. The cleanup now reports
  each failure against this test.
- TestArrivalSeq_JournalCollisionMovesTheStamp required its counter reset to
  clear every surviving stamp, which held only because no earlier test left a
  stamp above the counter. It now lifts the counter past MAX(arrival_seq)
  before seeding, so its records sit above every surviving stamp by
  construction; the Require stays as a setup check.

Ported from gascity/bd-enterprise#204 (7962a4598).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… deadline

TestArrivalSeq_EmbeddedConcurrentWritersAreUnique gave each of its 12
creates a fixed 60s context, but the engine serializes them, so the last one
waits on all the others: the farm's -race build took about 30s, a 2x margin.
The writers now share one context bounded by the test binary's deadline less
30s of grace to report a failure in, and by nothing else when no deadline is
set.

Ported from gascity/bd-enterprise#204 (b2b158a18).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The "Commit order" guarantee was stated unconditionally, but the documented
first-record rule breaks it: a row created with the journal off and later
renamed or re-imported takes a stamp above rows created after it. And "no
stamp and no cost" with the journal off overstated it, since
uk_wisps_arrival_seq is maintained on every wisp insert and delete regardless,
NULL entries included.

The guarantee now reads "ordered by the commit of the create record that
stamped the row" and says when that record is not the insert; "no cost" becomes
"no extra statement", naming the index. The types.Issue doc and the CHANGELOG
entry carry the same caveat (the spec's Issue.arrival_seq description already
does).

Ported from gascity/bd-enterprise#204 (c15da9a09).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ed issue

The storage create APIs write the stored row back onto the issue they are
given: CreateIssue, CreateIssues and Transaction.CreateIssue on every store
(the calls the public beads.Storage exposes, and the ones gascity's native
store maps its create result from), and the unit-of-work repository's
Insert. PrepareIssueForInsert and Insert zero ArrivalSeq on the way in so a
supplied value is never stored, but nothing put the stamp back: the row
carried arrival_seq 1..N while the caller's struct still said 0, so a
consumer that keeps its create result reads every stamped wisp as
unstamped until it reads the row again. Only the public Lifecycle roles,
which re-read the row, returned it.

RecordCreateEventInTx journals the create exactly as RecordEventInTx does
(both share recordIssueEventInTx, which still makes the one stamp decision)
and copies the journaled snapshot's arrival_seq onto the issue. Both create
seams use it: issueops.finishCreateIssueInTx, which every store's create
and import funnels through, and domain/db Insert. With the journal off
nothing is recorded and the field stays zero, as on the row.

Red before the fix: TestArrivalSeq_EmbeddedCreateLeavesTheStampOnTheCallersIssue
(the caller's issue carried 0 against rows 1..4) and, on a Dolt sql-server,
TestDomainDB/TestArrivalSeq_InsertLeavesTheStampOnTheCallersIssue (0 against
17 and 18). The structural guards follow the move: the stamping
insertEventRow call now lives in recordIssueEventInTx, and the unit-of-work
journal-completeness guard counts RecordCreateEventInTx as an emitter. The
embedded storage shard manifest places the new test.

Ported from gascity/bd-enterprise#204 (3d12c16dd).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… stamp

When bd_events_seq goes missing, the ignored series' cursor floor replays it
from ignored/0022 (replayFloor 21), which re-creates the counter and seeds it
from the journal's MAX(seq) alone. With the journal pruned past every record
that seed is 0 while stamps survive on wisps, and the values swept or
deleted wisps held are free: the next create takes one without touching the
unique index, so the writer's collision heal never fires and the new row
sorts before older surviving rows.

ignored/0029 re-runs in that same replay, after 0022, so its third step now
raises bd_events_seq past MAX(wisps.arrival_seq) (GREATEST, guarded on the
counter table and the stamp column existing). It changes nothing on an
in-place upgrade (every stamp NULL), on a fresh clone, or on a healthy
re-run.

TestArrivalSeq_EmbeddedReplayedCounterSeedClearsTheStamps drives the path
end to end on the embedded engine: three stamped wisps, the two oldest
deleted, the journal emptied, the counter table dropped, MigrateUp, then a
new wisp must stamp above the survivor. Without the step the replay
re-seeds at 0 and the test fails on the counter check.
TestIgnoredMigration0029RaisesTheSeqCounterPastTheStamps pins the statement,
its place after the column and index, and that bd_events_seq's replay floor
still sits below 0022.

Follow-up from the gascity/bd-enterprise#204 review (finding 4).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The storage contract cases prove the stamp on every backend, but a consumer
that shells out to bd reads the CLI's own rendering, which marshals
types.Issue, IssueWithCounts or IssueDetails per command; a command that
moved to a projection of its own would drop the member without any storage
test noticing.

TestEmbeddedArrivalSeqOnEveryJSONRead creates an ephemeral wisp, a
no_history wisp and a durable bead with the journal on, and checks that
bd create --json, bd show, bd list --include-ephemeral, bd ready
--include-ephemeral and bd query --json carry each wisp's stamp equal to
the seq of its create record in bd events export, that the durable bead
carries no member, and that bd export never writes one. The 100-shard
embedded cmd manifest places it.

Follow-up from the gascity/bd-enterprise#204 review (finding 5).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rential

TestVersionedHistoryOffMatchesPreStackGolden (embedded, server and
unit-of-work legs) failed on issues.arrival_seq, wisps.arrival_seq and the
two new bookkeeping rows: migration 0073 and ignored/0029 add schema the
goldens predate. Flag-off storage leaves it untouched (issues never carries
a stamp, and wisps carries one only while the workspace journals, which the
scenario does not), so it takes its own schemaAdditions entry, as the
versioned-history stack's columns did, and the goldens stay as recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gascityinc-randy
gascityinc-randy Bot requested a review from a team as a code owner October 27, 2026 16:01
@github-actions github-actions Bot added the status/needs-triage Awaiting initial PM triage label Oct 27, 2026
@gascityinc-randy gascityinc-randy Bot added the status/needs-review-auto Request automated PR review workflow label Oct 27, 2026
@gascity-eddie

gascity-eddie Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

This PR has been added to the review queue.

Track progress: https://factory.gascity.com/city/maintainer-city/runs/gcg--9223372036836401312

@gascity-eddie gascity-eddie Bot added status/reviewing PR review workflow is running and removed status/needs-review-auto Request automated PR review workflow labels Oct 27, 2026
@gascityinc-sarah gascityinc-sarah Bot added kind/feature New user-facing capability priority/p2 Medium: real problem with lower urgency labels Oct 27, 2026
@gascityinc-sarah

Copy link
Copy Markdown

Triage: kind/feature · priority/p2 — Adds a new platform-owned arrival_seq stamp and its read paths on every backend. Nothing is broken today; it unblocks the named gascity op-journal consumer and gate G-BD5, so medium not high.
Auto-triaged by @gascityinc-sarah. Reply or relabel if this is off.

@gascityinc-sarah gascityinc-sarah Bot removed the status/needs-triage Awaiting initial PM triage label Oct 27, 2026
gascity-eddie Bot added a commit that referenced this pull request Oct 27, 2026
…s its index

Maintainer fixup for review iteration 1 of #7912.

Every issue read now selects arrival_seq, and a server-mode read-only open
neither migrates nor checks a behind schema, so after an upgrade every
list, show and ready on a v72 database failed on the unknown column. An
open that does not migrate now refuses such a database at open with
SchemaBehindError (schema.CheckReadableDrift). It does so only when the
database lacks a column the shared issue projection selects; a behind
database that still has every column keeps serving reads (bd-578h9.5,
#5920). The open reads the version once, as the
forward-drift check it replaces did, and lists the issues columns (SHOW
COLUMNS) only when the database is behind. Three paths do not migrate:
- server mode: dolt.Config.RequireReadableSchema. The root pre-run sets it
  for every open except a preview, and so do the routed and foreign read
  factories (cgo and nocgo). Through a gateway the refusal names the
  server administrator's remedy;
- proxied mode: the read-through after a refused migration;
- embedded: a read-only command's open after a tolerated gate or
  dirty-table refusal.

0073's down drops wisps.arrival_seq and uk_wisps_arrival_seq with it, but
leaves ignored/0029 recorded, so a re-applied 0073 restored the column
without the index. 0073's up now creates the index as well, under 0029's
guard, and its CLI bundle substitute does the same.

Docs: a CHANGELOG upgrade note. "Never reused" is narrowed to values the
counter, the journal or a surviving row still holds. A bead moved onto the
wisp plane is documented as carrying no stamp. The 1054 and Kind.Is
comments are corrected.

Tests:
- CheckReadableDrift unit cases;
- a server-tier open matrix (behind with every column, a missing column,
  the gateway remedy, the escape hatch);
- a 0073 rollback and re-apply through the real migration pass;
- a bd list/show/ready e2e on a v72 database, converging with bd migrate
  schema;
- proxied and embedded read-through refusals.
Each test fails against its mutant.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@blacksmith-sh

This comment has been minimized.

…s its index

Maintainer fixup for review iteration 1 of #7912.

Every issue read now selects arrival_seq, and a server-mode read-only open
neither migrates nor checks a behind schema, so after an upgrade every
list, show and ready on a v72 database failed on the unknown column. An
open that does not migrate now refuses such a database at open with
SchemaBehindError (schema.CheckReadableDrift). It does so only when the
database lacks a column the shared issue projection selects; a behind
database that still has every column keeps serving reads (bd-578h9.5,
#5920). The open reads the version once, as the
forward-drift check it replaces did, and lists the issues columns (SHOW
COLUMNS) only when the database is behind. Three paths do not migrate:
- server mode: dolt.Config.RequireReadableSchema. The root pre-run sets it
  for every open except a preview, and so do the routed and foreign read
  factories (cgo and nocgo). Through a gateway the refusal names the
  server administrator's remedy;
- proxied mode: the read-through after a refused migration;
- embedded: a read-only command's open after a tolerated gate or
  dirty-table refusal.

0073's down drops wisps.arrival_seq and uk_wisps_arrival_seq with it, but
leaves ignored/0029 recorded, so a re-applied 0073 restored the column
without the index. 0073's up now creates the index as well, under 0029's
guard, and its CLI bundle substitute does the same.

Docs: a CHANGELOG upgrade note. "Never reused" is narrowed to values the
counter, the journal or a surviving row still holds. A bead moved onto the
wisp plane is documented as carrying no stamp. The 1054 and Kind.Is
comments are corrected.

Tests:
- CheckReadableDrift unit cases;
- a server-tier open matrix (behind with every column, a missing column,
  the gateway remedy, the escape hatch);
- a 0073 rollback and re-apply through the real migration pass;
- a bd list/show/ready e2e on a v72 database, converging with bd migrate
  schema;
- proxied and embedded read-through refusals.
Each test fails against its mutant.

The proxied shard manifest (.github/scripts/proxied-cmd-test-shards.txt)
places the new TestServerMode* test on shard 16
(gen_proxied_shard_manifest.py 34 --weights=duration --write).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gascity-eddie
gascity-eddie Bot force-pushed the feat/arrival-seq-bd5 branch from 2428830 to c6c4ec4 Compare October 27, 2026 18:40
@gascity-eddie

gascity-eddie Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Maintainer Adoption Review

Thanks for the contribution, @julianknutsen! This PR adds arrival_seq, a commit-ordered create sequence that the events journal stamps on wisp-plane rows on every backend (Dolt sql-server, embedded Dolt, the unit-of-work path and bd serve/HTTP). That lets a consumer that falls behind the journal's retained window re-baseline from rows and still recover the order the creates committed in, without a second counter. It unblocks gascity's conversation-lane op journal (BD-5).

This PR was reviewed and adopted with maintainer fixes pushed directly to the PR branch.

Original PR Review

Decision: request_changes resolved to approve.

Specific gaps fixed:

  • After upgrading bd, server-mode reads (bd list, bd show, bd ready, routed and foreign reads, gateway opens) on a database still at v72 failed with a raw unknown-column error for arrival_seq (reproduced on Dolt 2.6). The final patch refuses those opens with SchemaBehindError when the database lacks a column the issue projection selects, and names the server administrator's remedy for a gateway. A database behind only by its migration cursor still opens. TestDoltNew_RequireReadableSchema and the end-to-end TestServerModeReadRefusesASchemaItsReadsCannotQuery pass on real Dolt, bd migrate schema then converges, and the CHANGELOG carries the upgrade note.
  • Rolling 0073 back and re-applying it brought wisps.arrival_seq back without uk_wisps_arrival_seq, because ignored/0029 stays recorded. 0073's up migration and its CLI-bundle substitute now create the index under 0029's guard. TestMigration0073RollbackAndReapplyRebuildsTheArrivalSeqIndex pins the rebuilt index, a next stamp above every pre-rollback stamp, and a refused duplicate.

Review findings addressed:

  • Change Impact / Blast Radius (major, high, claude, reproduced): server-mode reads of a v72 database failed on the unknown column. Fixed in c6c4ec4.
  • Release Safety (major, high, codex + claude): 0073 down then up lost uk_wisps_arrival_seq. Fixed in c6c4ec4.
  • Contract & Interface Fidelity (minor, high, codex + claude): events-journal.md promised values are never reused across counter heals, which a counter loss after a full prune can break. c6c4ec4 narrows the promise to the guarantee the code keeps.
  • Contract & Interface Fidelity (minor, high, claude, reproduced): a bead moved onto the wisp plane while journaling carries no stamp, which the docs said could not happen. c6c4ec4 documents that case in types.go, the OpenAPI Issue.arrival_seq and events-journal.md.
  • Debuggability & Operability (nit, high, claude): a journal comment named error 1054 where Dolt sql-server reports 1105. c6c4ec4's comments now name the string arms that actually match.

Non-blocking follow-ups (shared for visibility — NOT completed in this PR):

Filed as follow-up beads:

  • bd-3f2m9j: Change Impact / Blast Radius (open risk): a write routed to a gateway workspace whose database is still behind this bd's schema fails with a bare unknown-column error for arrival_seq instead of the gateway SchemaBehindError, because the routed writable factories (cmd/bd/store_factory.go:170, :184) do not set RequireReadableSchema. Routed reads and local opens already refuse correctly.

Not filed (below the follow-up bar):

  • Error Handling & Resilience (minor, high, claude, reproduced), internal/storage/uow/dolt_sql_provider.go:436 and internal/storage/embeddeddolt/store.go:697/:723: on a remote-backed embedded or proxied clone behind v73, a read-only command's tolerated adopt, fork-skew, data-behind or dirty-table refusal now surfaces as the generic SchemaBehindError text instead of the gate's own guidance, and --json callers lose the remote_migrate_gate directive. Not filed: wording. The read refusal itself is correct, the bd migrate schema it names is refused with the gate's own guidance, and the matching-release option it names works.
  • Debuggability & Operability (nit, high, claude), internal/storage/schema/remote_migrate_gate.go:492: the shared-no-remote gate body still says read commands keep working against the current schema, which no longer holds for a database without arrival_seq. Not filed: a note that disagrees with a real run that itself behaves correctly.
  • Behavioral Correctness (minor, high, codex + claude), internal/storage/issueops/journal.go:363/:439: a wisp the unit-of-work upsert inserts while an issues-plane row holds the same id (GH#4455 dual presence) is never stamped. Not filed: already open as bd-u4v5z0, the cross-plane id collision that creates the twin (below the bar, no new bead).
  • Test Evidence Quality (minor, high, codex), backend/conformance/journal_arrival_contract.go:47: no arrival conformance case drives a demotion onto the wisp plane. Not filed: test-only.
  • Test Evidence Quality (residual risk, claude), internal/storage/issueops/arrival_seq_guard_test.go (TestArrivalSeqIsWrittenOnlyByTheJournalSeam): the literal-string writer guard would not catch an INSERT … SELECT built from IssueBaseColumns. Not filed: test-only hardening.
  • Test coverage / CI bookkeeping (nit, high, delta review), .github/scripts/proxied-cmd-test-shards.txt:155: the new manifest line schedules on proxied shard 16 a test that always skips in that tier; it runs in //cmd/bd:bd_dolt_server_test. Not filed: test-only.
  • Release Safety (nit, high, claude), internal/storage/schema/migrations/0073_add_arrival_seq.up.sql and ignored/0029_add_wisps_arrival_seq.up.sql: open PR fix: make row_lock an aggregate revision #5738 claims the same slots (0073 / ignored 0029). Not filed: two-PR merge with fix: make row_lock an aggregate revision #5738. Main is still at 0072 / ignored 0028, so whichever lands second renumbers.
  • Open risk (claude), internal/storage/schema/schema.go:339 (CheckReadableDrift): a server-mode clone at v72 that pulls a peer's v73 history gets cursor 73 and issues.arrival_seq, but its clone-local wisps lacks the column until its next writable open applies ignored/0029, so a read-only bd list in that window can fail on the unknown column. Not filed: unverified (not reproduced; row_lock and storage_class share the window on main).
  • Open risk (claude), internal/storage/schema/schema.go:339: a database with no bd schema (v0, for example a mistyped dolt_database) is now reported on server-mode reads as behind at v0 with "run bd migrate schema". Not filed: outside ordinary use; embedded read-only opens already report v0 the same way.
  • Open risk (claude): BD_IGNORE_SCHEMA_SKEW=1 lets a refused open through but does not make issue reads work, since the refusal fires only when a projected column is missing. Not filed: wording. The message already warns that queries touching the newer schema may fail, and its migrate remedy converges.
  • Open risk (claude): the readable check probes the issues table only, so a future projected column outside it would slip past. Not filed: hardening against a future change.

Maintainer Changes

Maintainer fixes were pushed directly to the PR branch as one commit on top of your 12:

  • c6c4ec4 fix(storage): refuse a behind schema reads cannot query; 0073 rebuilds its index (33 files, +1089/−72)

The same commit adds the generated proxied shard-manifest line for the new end-to-end test, which cleared the //scripts:go_test_sources_test failures Blacksmith reported on the first pushed fix head (run 40558338577).

Final Review Status

Ready for the merge queue: final head c6c4ec467e has passing required GitHub checks, and the merge-ready metadata is in place. I am marking this PR status/merge-ready so the merge queue can pick it up.

CI: https://github.com/gastownhall/beads/actions/runs/40562215797

Review Iterations

2 review passes (Claude and Codex): the first requested changes on the two majors above, the maintainer commit fixed them, and the second approved. Claude also ran the Dolt-backed and Dolt-server tiers that PR CI skips on a same-repository branch. A delta review then confirmed the CI repair added only the generator's own manifest line.


Adopted via /adopt-pr workflow. Original contributor commits preserved.

@gascity-eddie gascity-eddie Bot added status/merge-ready PR is ready for merge workflow status/merge-queued Queued for deterministic PR-review merge and removed status/merge-ready PR is ready for merge workflow labels Oct 27, 2026
@gascity-eddie gascity-eddie Bot removed the status/reviewing PR review workflow is running label Oct 27, 2026
@gascity-eddie
gascity-eddie Bot added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 5011528 Oct 27, 2026
43 checks passed
@gascity-eddie
gascity-eddie Bot deleted the feat/arrival-seq-bd5 branch October 27, 2026 19:23
@gascity-eddie gascity-eddie Bot removed the status/merge-queued Queued for deterministic PR-review merge label Oct 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

A push to main that includes this PR is failure in Main at 5011528: https://github.com/gastownhall/beads/actions/runs/40568706768

Candidates: #7912. Tracked in #7914.

@github-actions github-actions Bot mentioned this pull request Oct 27, 2026
quad341 pushed a commit to versioned-beads/beads that referenced this pull request Oct 28, 2026
…(BD-6) (gastownhall#7931)

* refactor(dolt): drop the dead per-table id generator

generateIssueIDInTable in internal/storage/dolt had no caller left: every
id the dolt store mints goes through issueops.GenerateIssueIDInTable. The
duplicate checked only its own table, so a future caller would have minted
ids the shared generator refuses (for example, an id a retaining sweep is
about to retire). Remove it with the helpers only it used
(nextCounterIDTx, seedCounterFromExistingIssuesTx, isCounterModeTx,
generateHashID, getAdaptiveIDLengthFromTable) and the two unit tests of
the last one; issueops carries the live copies of each.

Ported from gascity/bd-enterprise#210 (818610dc1).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(schema): id tombstone tables for retaining sweeps (main 0074, ignored 0030)

Add the two tables a retaining retention sweep writes and every create
path will read (BDP idempotency-expired): id_tombstones, versioned
like issues, and wisp_id_tombstones, clone-local like wisps (the existing
wisp_% dolt_ignore pattern covers it). Each row is the compact
(id, closed_at) pair, keyed on id, nothing else from the swept row.

- Main 0074 creates both, guarded and idempotent; ignored 0030 materializes
  the clone-local one on fresh clones in the identical shape. The slots are
  the next free ones, directly above arrival_seq's (gastownhall#7912: 0073, ignored
  0029). 0073's latest-version pin moves here.
- wisp_id_tombstones is a sentinel floored table (floor 29): a clone that
  lost it re-creates it on the next write-intent open instead of failing
  every create closed. Re-created EMPTY, the wisps-plane tombstones are
  gone while the wisps rows may survive, so migrate now prints a loud,
  unconditional notice for a lost sentinel table that carries one (cursor
  above its floor) — the only signal a consumer gets that the clone is a
  new logical scope for wisps-plane idempotency keys.
- bd doctor requires id_tombstones and treats wisp_id_tombstones as
  clone-local; the Dolt schema-parity test pins the pair to one shape.
- The history-off differential (historyoffequiv) sets both tables and
  their bookkeeping rows (74, ignored 30) aside: flag-off storage leaves
  them empty.
- Three of BD-5's tests assume 0073 is the latest migration and follow the
  slot: the 0073 rollback test rolls back what sits above 0073 first, as an
  operator must (a row 74 left behind would hold the cursor past 73); the
  server-mode read refusal rebuilds the v72 shape without 0074 too, so the
  refusal names v74; and the embedded lenient-gate test regresses the
  cursor below 0073 rather than one migration back, so the pending 0073
  still meets the dirty-table guard.

Ported from gascity/bd-enterprise#210 (b60aa6a42).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(storage): retaining sweeps retire ids; every create door refuses a retired id

A retention sweep can now ask to retain ids (SweepRequest.RetainIDs): in
the same transaction that deletes the closed rows it writes one permanent
(id, closed_at) tombstone per row into its own plane's tombstone table.
From then on no write path commits that id again (BDP idempotency-expired):

- every create door — Lifecycle.Create, BatchApplier and BatchCreator
  items, the stores' upsert CreateIssue/CreateIssues, a transaction's
  CreateIssue, the unit of work's domain create with or without CreateOnly
  — refuses it with the typed *issueops.IdempotencyExpiredError
  (ErrIdempotencyExpired, re-exported from storage, backend and beads);
- an import skips and reports it (BatchCreateOptions.OnRetiredID, which
  the unit of work's Importer surfaces as ImportBatchResult.RetiredIDs;
  ConflictSkip, the auto-import path, skips it silently);
- the generators treat it as taken: hash mode folds the tombstone check
  into its one collision read, counter mode walks past retired values
  with one probe each, the uow mint likewise.

The race a create loses to the sweep of its id is closed by a lock, not by
isolation: every path that decides whether an id may be written writes the
id's create coordination cell (local_metadata, 4096 shards, sorted, one
statement) BEFORE it reads occupancy, and a retaining sweep — the shared
SweepInTx and the unit of work's retiring delete — writes the cells of
every id it retires BEFORE it deletes. On a READ COMMITTED door the create
that comes second waits out the sweep and then reads its tombstone; a
sweep that comes second fails serialization or meets a write-write
conflict and retries over the created row. Occupancy is one statement over
both row tables and both tombstone tables; a live row wins. A CreateOnly
batch no longer reads tombstones it never consults.

Only a retaining sweep retires: a plain prune or purge, the Deleter and
bd gc's decay leave ids reusable, and the Sweeper and Deleter docs say so.

The IDRetention conformance contract (17 cases: opt-in, both planes, each
door, the no-history re-create through every door, dry run, narrowing,
history, and three staggered races — create-only, upsert and import —
asserting no interleaving ever leaves a live row beside a tombstone) runs
on the dolt, embeddeddolt and uow legs; the dolt races run on a multi-
connection store. The http leg waives it by name until the wire lands next
in this series. The two store legs hand out no Importer role, so they
drive the import cases through conformance.IDRetentionUpsertImporter, over
CreateIssuesWithFullOptions: the seam `bd import` writes through on them
(ArrivalUpsertImportHook's precedent). The unit-of-work leg uses its
Importer, which now reports RetiredIDs.

The http client cannot send RetainIDs yet (apigen.SweepRequest has no
retain_ids member), so it refuses a retaining sweep before the dial under
a new refuse row, W-SweepRequest.RetainIDs, rather than dropping the field
and running a sweep that leaves the ids reusable; the wire commit retires
the row. TestSweepResultCarriesEveryWireMember skips RetainIDs by name
until then.

httpapi's TestSweepResponseCarriesEveryRoleField counts issueops.SweepResult
against the wire's SweepResult; RetainIDs is waived there by name until the
wire commit projects it, so this commit is green on its own.

The history-off differential's goldens (historyoffequiv) are re-recorded
from this series, as its package doc prescribes for a deliberate change to
what flag-off storage writes: the coordination cells of the scenario's
explicit-id upsert creates add rows to local_metadata (six on the embedded
and server legs, for s1-s4, sw and t1; three on the unit-of-work leg, for
u1, uw and u2). A recording of the base is byte-identical to the committed
goldens, and no other table's digest moves. The digests were recorded on
bd-enterprise#210, whose base carries the same goldens as main, and
recorded_from says so. The createbatchequiv retired scenario stamps its
seeded rows' participation (stampParticipation), as depadd and waitsfor
do, so the batch's upsert of eq-rl mints under the
participation-generation fence as it did when its golden was recorded.

TestIDRetentionContractRaces is pinned to dolt_server_full_test's shard 16
(server-storage-test-shards.txt). Left to the name hash it lands in shard 6,
which already takes about the whole 15m test timeout under load, and pushes
that shard past it.

Ported from gascity/bd-enterprise#210 (58d8b1527).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(cli): bd prune/purge --retain-ids, the typed create refusal, and retired rows on import

- `bd prune --retain-ids` and `bd purge --retain-ids` ask the sweep to
  retire every id it deletes. --json gains retain_ids: true; text says
  "Retired N id(s)", or "Would retire N id(s) ... would be refused" in a
  preview; the confirm hint carries the flag so the run it suggests is the
  sweep that was previewed.
- `bd create --id X` of a retired id renders one shared typed refusal on
  the direct and proxied routes: --json prints
  {"error","code":"idempotency_expired","id","closed_at"}, text says the
  create that used the id already happened and never runs again; exit 1.
- `bd import` skips a retired row and reports it in retired_ids and a
  stderr note, on the classic and proxied routes, and `--dry-run` now
  agrees with the run it previews: it reads the would-create ids'
  tombstones through an optional store seam (RetiredIssueIDs, under the
  CLI's decorators) and reports them as retired instead of counting them
  as creates.
- The confirm hint is sweepConfirmHint's, so --retain-ids joins the
  selectors TestSweepConfirmHintCarriesEverySelector pins.
- docs/workflows/wisps.md documents --retain-ids and that only a retaining
  sweep retires; CHANGELOG entry for the whole series.

Tests: the embedded CLI end to end (prune and purge, both envelopes,
import and its dry run), the proxied route's envelope (gated on the
proxied-server tier), and the dry-run classifier in both stale modes.

Ported from gascity/bd-enterprise#210 (4116bc2a5).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(wire): retain_ids on issues:sweep, 410 idempotency_expired, and the served IDRetention leg

The wire half of the retaining sweep.

- issues:sweep takes retain_ids (and echoes it), behind the new behavior
  token issues.sweep.retainIds; the token also promises that every create
  path on that server refuses a retired id. The client refuses RetainIDs
  locally, sending nothing, when the server does not advertise it: a sweep
  that dropped the member to get past an older server would leave every
  swept id reusable. The interim refuse row W-SweepRequest.RetainIDs the
  storage commit added is retired (kept as history, like the S4 rows).
- createIssue and applyBatch answer a retired id with 410
  idempotency_expired (param naming the id, or the batch item's members,
  plus closed_at), classified ahead of the validation and already-exists
  arms; the client maps the code back to the typed
  *issueops.IdempotencyExpiredError.
- The http leg wires the IDRetention contract through the served fixture
  and retires the series' interim waivers. Five cases stay waived by name:
  the in-transaction one (no transaction spans requests), the two that
  drive a store's upsert-shaped CreateIssue, which the wire does not
  serve (createIssue is create-only), and the two import cases: the wire
  publishes no import operation, so the client hands out no Importer.
  All five run on the store legs.
- apigen is regenerated from the spec; the wire-shape golden follows.

Ported from gascity/bd-enterprise#210 (c6e938737).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(storage): create-only batches take every id's shard cell once, sorted

A create-only batch (ExecuteCreateBatch, a CreateOnly CreateIssues) took
its coordination cells one item at a time, in item order, through each
row's EnsureIssueIDAvailableInTx, and an apply-batch took them per create
item through ExecuteCreate. Retaining sweeps, upsert batches and imports
lock large sorted cell sets, so on a backend that runs these doors read
committed (bd-enterprise's Postgres store) such a batch could hold a later
shard while it waited on an earlier one and deadlock against them (40P01).
A SERIALIZABLE envelope or an importer retries that; a single-attempt,
read-committed CreateIssues or CreateIssuesWithFullOptions, which `bd
import` calls, would surface it.

- newCreateBatchCache coordinates every id the batch names in one sorted
  write for CreateOnly batches too, and a row whose cell the batch already
  holds runs only the occupancy probe (checkIssueIDAvailableInTx). A row
  the cache never saw (a generated id) still takes its own cell.
- ApplyBatchInTx takes the cells of every explicit create id in one sorted
  write before its first item.
- CoordinateIDCreatesInTx's doc no longer says sorted order "cannot
  deadlock" unconditionally: it holds between callers that take all their
  cells in one call, and names the residual (a generated id minted
  mid-transaction).

Tests (sqlmock statement order): a CreateOnly batch locks both ids in one
sorted write before its reads and its rows probe without re-locking; a row
outside the up-front set still locks first; an apply-batch's first
statement is the sorted lock of its explicit create ids.

Ported from gascity/bd-enterprise#210 (0802d1640).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(uow): guard only caller-named ids against retirement in the upsert Insert

domain/db's upsert Insert ran RefuseRetiredIDInTx for every create,
including ids the use case had just minted and child-counter ids, where
issueops' create body guards only explicit ids. Every node of an
applyGraph, a createMany, a graph apply or a molecule pour through the
proxied server paid a coordination cell write and a four-table probe for
nothing (the generators already skip retired ids), and on a Dolt server
two concurrent 30-node graph applies shared a cell about a fifth of the
time, so the loser retried its whole unit of work.

InsertIssueOpts gains ExplicitID, set by the create use case from
createIDNamedByCaller (ExplicitID, or an id preset on the issue with no
parent whose child counter replaces it), and the upsert Insert coordinates
and probes only then. The CreateOnly branch is unchanged: it runs the
create-only guard for every id, as issueops does.

Tests: a generated-id upsert's first statement is its row write; an
explicit-id upsert coordinates and probes first; the explicitness rule
over ExplicitID, preset, parent and minted ids.

Ported from gascity/bd-enterprise#210 (8452e0a74).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(storage): persistence moves and wisp promotion take the id's shard cell

A persistence move and a wisp promotion re-insert a row into the other
plane without the coordination cell or a fresh look at the row. Raced
against a retaining sweep of the same closed row on Dolt, both deleted the
source row (identical deletes merge cleanly), the move's insert landed in
the other plane, and the sweep's tombstone committed beside it: a live row
next to a tombstone, which the contract says never happens. Read-committed
doors (bd-enterprise's Postgres store) behaved the same way. Creates still answered 409, so nothing
re-executed, but the swept row came back.

- moveIssuePersistenceInTx coordinates the id before it reads where the
  row lives, and a row found in neither plane (a sweep deleted it after the
  caller read it) is refused with ErrNotFound instead of re-inserting the
  caller's earlier snapshot.
- PromoteFromEphemeralInTx coordinates the id before it reads the wisp.

So a move or promotion that comes second behind the sweep finds no row,
and one that comes first makes the sweep conflict (Dolt) or fail
serialization (a SERIALIZABLE envelope) and retry.

Tests (sqlmock statement order): a move's and a promotion's first
statement is the id's coordination write; a move of a row in neither plane
is ErrNotFound and writes nothing.

Ported from gascity/bd-enterprise#210 (6c5f9e5c3).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(dolt): replay a sweep that lost the commit-time merge; state the lock footprint

The Dolt store's sweeper ran its write in withWriteTx, a single attempt.
A retaining sweep writes one create-coordination cell per shard it
retires, and the cells are 4096 shards shared by unrelated ids, so on a
sql-server any create, upsert or import committing in one of those shards
first conflicts the sweep (1213/1205 or the exact 1105 autocommit
rollback). At a limit of 1000 that is about 22% of all shards, and at
a load of about 5.8 creates/s per shard most such sweeps would have failed outright.

- The sweeper writes through withRetryTx, as the deleter does: a
  rollback-guaranteed conflict replays the whole body, candidates re-read
  and the result reset at the top of each attempt. The uow sweeper
  (RunTxResult) already replays; the embedded store serializes its
  transactions.
- SweepRequest.RetainIDs documents the footprint: about
  4096·(1-e^(-N/4096)) shard cells held until commit, what that does to
  creates on a read-committed backend (they wait) and on a Dolt sql-server
  (one side conflicts and replays), and to size Limit around 100 and loop
  on Remaining.

Test: a sweep whose first SQL COMMIT is refused with 1213 is replayed and
succeeds (fake driver; two COMMITs).

Ported from gascity/bd-enterprise#210 (0b1ac36c2).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(conformance): retire rows through the wisps-plane retaining sweep on every leg

No IDRetention case swept with tier wisps-plane, the exact request an
orchestrator sweeping its operation rows sends (wisps-plane, retain_ids,
protect_referenced off, a limit). That tier has its own candidate path on
every backend (SearchWispsPlaneInTx; sweepCandidatesInUOW on uow), so the
durable and ephemeral cases did not cover it.

RunIDRetentionAWispsPlaneRetainingSweepRetiresItsRows seeds a closed
no-history row and a closed ephemeral row, sweeps them with that request,
and asserts both tombstones land in wisp_id_tombstones with the rows'
closed_at, that Remaining is 0, and that a re-create of either is refused
through Lifecycle.Create and, where served, the upsert-shaped create. It is
in the role bundle and wired on the dolt, embedded, uow and served (http)
legs.

Ported from gascity/bd-enterprise#210 (76bcf085c).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(cli): report each retired import row once and never as an update

The import's accounting of retired rows disagreed with itself in places:

- assembleImportResult listed a retired id once per row carrying it; the
  report and the spec say RetiredIDs is deduplicated.
- A row the pre-filter planned as an update (live when it read it) that a
  retaining sweep retired before the batch wrote was reported both in
  UpdatedIssues/Updated and in RetiredIDs. The batch skipped it, so it is
  now reported retired only.
- The dry-run classifier checked retired ids with slices.Contains per row,
  quadratic in the rows of an export full of retired op rows; it now uses
  sets.
- Under ConflictSkip a retired row is skipped silently and still counted,
  as every row ConflictSkip carries is. That is kept (ConflictSkip is the
  auto-import recovery path, which counts the rows it submitted) and now
  stated on BatchCreateOptions.OnRetiredID.

Test: assembleImportResult over a file with a duplicated retired row and a
planned update that turned out retired.

Ported from gascity/bd-enterprise#210 (6c34e757e).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: state where the IDRetention race cases bite and how to size a retaining sweep

- The IDRetention race cases say where they can fail: only on a backend
  whose upsert and import doors run READ COMMITTED (bd-enterprise's
  Postgres legs). The embedded leg and the http leg served over it
  serialize transactions, and on a Dolt sql-server snapshot reads plus
  commit-time row conflicts refuse the interleavings anyway: with
  CoordinateIDCreatesInTx made a no-op, the uow leg (doltserver lane,
  remote) still passes, as the embedded leg did for the reviewer. Green on
  those legs is no evidence for the race fix.
- CHANGELOG and docs/workflows/wisps.md say a retaining sweep holds the
  coordination cells of the ids it retires until it commits, and to run
  large sweeps in batches of about 100 (bd purge --limit; bd prune has no
  limit, so narrow it with --older-than).

Ported from gascity/bd-enterprise#210 (b0873bffa).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(storage): refuse a rename onto a retired id

UpdateIssueIDInTx renamed a live row onto any id no live row held, so a
rename (bd rename, bd rename-prefix, the storage-level UpdateIssueID)
could commit a live row beside a retaining sweep's tombstone — the state
the CHANGELOG says no write path produces. It was deterministic: on the
embedded store, UpdateIssueID(live -> retired) returned nil and left one
live row and one tombstone.

The rename now calls RefuseRetiredIDInTx(newID) before anything else: it
takes newID's create-coordination cell and probes the tombstones, so a
retired target is refused with *IdempotencyExpiredError and nothing is
written, racing sweep or not.

Tests:
- TestRenameOntoARetiredIDIsRefusedBeforeItWrites (sqlmock): the cell and
  the probe come first and a retired target stops the rename.
- New IDRetention case RunIDRetentionRefusesARenameOntoARetiredID (both
  planes; the refused rename leaves the source row in place), with a new
  IDRetentionFixture.Rename door wired on the dolt and embeddeddolt legs
  and waived by name on uow and http, which serve no identity change.
- CHANGELOG names the rename refusal.

Ported from gascity/bd-enterprise#210 (556ecf460).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(dolt): take the coordination cell before demoting a row to wisps

DoltStore.demoteToWispInTx — the Dolt store's own persistence move,
reached from the storage-level UpdateIssue and UpdateIssueChecked when
the updates carry wisp or no_history — re-inserted the row into wisps
without the id's create-coordination cell. On a Dolt sql-server a
retaining durable sweep and the demotion could both commit: their
identical deletes of the issues row and aux rows merge cleanly, and the
wisps insert and the tombstone insert touch different tables, leaving a
live wisp beside an id_tombstones row (review finding 6's state).

The demotion now calls CoordinateIDCreatesInTx for its id before its
first read of the row, as moveIssuePersistenceInTx and
PromoteFromEphemeralInTx do. A retaining sweep takes the same cell
before it deletes, so the two conflict on the cell and one replays: the
demotion either commits first (and the sweep then finds no closed
issues row) or finds the row gone and fails its update.

Test: TestDemoteToWispCoordinatesBeforeItTouchesTheRow (sqlmock) pins
the cell write as the demotion's first statement.

Ported from gascity/bd-enterprise#210 (437ea2ca8).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(import): count every row of a retired id out of Created

The unit of work's Importer set Created to the request's rows minus the
stale-rejected ids minus the DISTINCT retired ids. A request may carry an
id more than once, and the batch skips every row of a retired id, so a
request carrying a retired id twice overcounted Created by one per extra
copy, contradicting ImportBatchResult.Created's "excluding ... retired
rows".

Created now subtracts the request rows whose id is in the retired set.
The set is per attempt and keyed by id, so a hook that fires again on a
replay still counts each row once. Stale-rejected rows keep their
existing per-id accounting; this change does not touch it. The store
legs' upsert importer (conformance.IDRetentionUpsertImporter) counts the
same way.

Test: the IDRetention import case now carries the retired row twice and
still wants Created == 1 (on enterprise it failed with Created = 2 before
the fix).

Ported from gascity/bd-enterprise#210 (195e730e9).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(storage): skip retired child suffixes and retired auto-import rows

Review fixes for gastownhall#7931, iteration 1.

The child id generators (issueops.GetNextChildIDTx and domain/db's
NextChildID) took the next suffix from max(child counter, highest live
child) and read no tombstone table. Once a retaining sweep deletes the
highest child, its tombstone is the only record that the suffix was used,
and the counter can lag it: a database written before GH#4750, the uow
route's explicit P.N create, or a re-created wisp parent whose counter row
was deleted with it. bd create --parent then refused the minted id with
idempotency_expired on every retry, and the uow upsert door (proxied
bd q --parent) wrote a live row beside its tombstone. Both generators now
fold both tombstone tables into their one high-water read
(issueops.UsedChildIDsInTx), and the IDRetention contract gains
NeverMintsARetiredChildID on every leg.

Under ConflictSkip a retired row was skipped before OnRetiredID could fire,
so the embedded auto-import (ImportJSONLData) and the server-mode fallback
(importFromLocalJSONLConflictSkip) counted rows they never wrote as
imported. OnRetiredID now fires under ConflictSkip too. ImportJSONLData
registers it and returns the retired ids, and both auto-import paths count
only the rows they wrote, report the skipped ones, and stamp a file whose
every row was retired, so the next command does not read it again. The
chunked import's dependency pass keeps the callback off, as it already does
OnStaleRejected.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
gascity-eddie Bot added a commit that referenced this pull request Nov 5, 2026
The PR head (538cec7) neither rebased nor merged cleanly onto main
(6d713a7, 382 commits past its 2026-10-05 merge base): 25 conflict
hunks in 12 files. Each pairs two independent changes, and the
resolution keeps both:

- issueops/journal.go: main's arrival_seq stamp (#7912) marshals the
  issue snapshot inside insertEventRow's record closure, after the stamp.
  The PR's shape-adaptive INSERT is now built there, by journalInsert,
  from that snapshot.
- dolt, embeddeddolt and uow stores: main replaced versionedHistoryEnabled
  with the deferred VersionedHistorySwitch (#7743). The PR's journalShape
  and journalActivationErr fields and its shape scope sit beside it. The
  uow provider keeps main's proxied-server-mode (#7481) and
  server-log-rotation (#7735) options and the PR's working-set-reconcile
  option.
- cmd/bd factories: main's versioned-history activation defer (#6661)
  stays registered first, so it still runs after the journal's; the
  journal defer carries the PR's bead-writing posture. main folded
  newDoltStoreFromConfig into openDoltStoreFromConfig (#7751), which also
  serves newExistingDoltStoreFromConfig. Both open for writes, so it
  passes true, as the PR's newDoltStoreFromConfig did.
- migrate_personal.go: main opens the planning store through the
  routed-write factory (#7751), which now activates with true, so the
  PR's own change to the removed dolt.New open is carried there.
- doctor/fix/store.go, main.go, uow_factory.go, uow_factory_test.go:
  both sides' changes.
- embedded-storage-test-shards.txt: main's file (40-shard block, #7440),
  with the PR's four embedded tests placed by
  gen_embedded_storage_shard_manifest.py 40 --weights=duration --write.

Co-Authored-By: Claude <noreply@anthropic.com>
julianknutsen added a commit to vishnujayvel/beads that referenced this pull request Nov 7, 2026
…srcs

TestArrivalSeqIsWrittenOnlyByTheJournalSeam walks every non-test Go file
under internal/storage, but under Bazel it sees only issueops_test's data,
a hand-picked package list. That list left out historyoffequiv, so the
guard's false positive there (gastownhall#8025) failed Main's macOS `go test` job on
every push since gastownhall#7912 while the gating Bazel lane stayed green. Adding
historyoffequiv alone still leaves 26 packages (81 non-test files) where a
real arrival_seq write would go unseen by Bazel.

Declare //internal/storage:tree_go_srcs instead. tools/bazel/go_srcs.py
generates it and TestBazelTreeGoSrcsListsEveryPackage holds it to every
package under the tree, so the Bazel view now matches `go test` (520
non-test files, against 439 for the list) and a new package is scanned
without an edit here. //internal/storage:storage_test declares it the same
way. It subsumes the historyoffequiv entry this PR added, this package's
own :go_srcs, and the sibling packages TestAllIssueRowWritesStampRowLock
and TestEpochCASTestsBuildEveryAddressWithEpochAddress read.

Cost: the target also re-runs when Go files change in those 26 packages
(143 of the tree's 1337 Go files); the list already held the busiest ones
(dolt, embeddeddolt, uow, issueops).

Agent-Signature: claude-code-unknown-model-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pasogott pushed a commit to pasogott/beads that referenced this pull request Nov 8, 2026
…hall#8033)

* fix(issueops): distinguish SQL in arrival sequence guard (ocb-85cac)

Use the existing Vitess tokenizer to exclude prose, comments and quoted values while retaining SQL write fixtures and the writer allowlist. Declare historyoffequiv scan data for Bazel.

Go tests and Bazel lint are unrun: required exact module versions are not cached and acquisition is forbidden for this source-preparation task. Managed Beads precommit ran separately under the recorded standing hook exception.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Vishnu Jayavel

* fix(issueops): retain arrival sequence fragment detection (ocb-85cac.3)

Match tokenized write keywords and the column throughout each statement, including format-prefixed writes and upsert fragments. Add 15 positive and exclusion fixtures and log the scan coverage. The four selected offline tests passed with 51 classifier cases.

Bazel lint and full tests are unrun under the admitted resource scope. Managed Beads precommit ran separately under the recorded standing hook exception; prepare-commit-msg remains enabled.

Agent-Signature: codex-gpt-6.1-sol-high on behalf of Vishnu Jayavel

* fix: require SQL origin in arrival sequence guard (ocb-85cac.4)

Keep the existing SQL and fragment fixtures while excluding prose with
write words or a later parenthesis. Add the three review counterexamples.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Vishnu Jayavel

* test(issueops): scan all of internal/storage under Bazel via tree_go_srcs

TestArrivalSeqIsWrittenOnlyByTheJournalSeam walks every non-test Go file
under internal/storage, but under Bazel it sees only issueops_test's data,
a hand-picked package list. That list left out historyoffequiv, so the
guard's false positive there (gastownhall#8025) failed Main's macOS `go test` job on
every push since gastownhall#7912 while the gating Bazel lane stayed green. Adding
historyoffequiv alone still leaves 26 packages (81 non-test files) where a
real arrival_seq write would go unseen by Bazel.

Declare //internal/storage:tree_go_srcs instead. tools/bazel/go_srcs.py
generates it and TestBazelTreeGoSrcsListsEveryPackage holds it to every
package under the tree, so the Bazel view now matches `go test` (520
non-test files, against 439 for the list) and a new package is scanned
without an edit here. //internal/storage:storage_test declares it the same
way. It subsumes the historyoffequiv entry this PR added, this package's
own :go_srcs, and the sibling packages TestAllIssueRowWritesStampRowLock
and TestEpochCASTestsBuildEveryAddressWithEpochAddress read.

Cost: the target also re-runs when Go files change in those 26 packages
(143 of the tree's 1337 Go files); the list already held the busiest ones
(dolt, embeddeddolt, uow, issueops).

Agent-Signature: claude-code-unknown-model-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature New user-facing capability priority/p2 Medium: real problem with lower urgency

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant