Repository navigation
feat(storage): carry the git data ref of a git-backed Dolt remote - #6521
Conversation
aad43de to
0c0a57d
Compare
|
Adversarial review before this leaves draft: three rounds under each of two models (six in all), each round with a fresh context in a throwaway worktree of the branch, the diff, the body, and the prior rounds' dispositions. Findings: 33 in total (4 blocker, 7 major, the rest minor or question). 31 accepted and fixed on the branch, 1 acknowledged (CI has not run on the PR until a maintainer approves the workflow), 1 open with the author. The main changes the rounds drove: the missing |
bee-ghosttrack
left a comment
There was a problem hiding this comment.
Reviewed at 0c0a57d45 on main (merge-base f56632adcf). Sorry for the wait on this one.
Ran, all ok: go build ./..., go vet ./internal/storage/..., go test ./internal/storage/ ./internal/storage/versioncontrolops/... ./internal/storage/doltutil/... ./internal/storage/domain/..., -run TestRemoteGitDataRefOnEngine ./internal/storage/embeddeddolt/ (real engine), and with BEADS_TEST_ENV_RUN_DOLT=1 on dolt 2.3.1: TestCLIMirrorFollowsGitDataRef, TestCLIMirrorRefRefusedForNonGitURL, TestPersistedRemotes.
The shape is right and the layering argument in the body holds up: a new method instead of a wider AddRemote, the ref bound as a ? everywhere it goes over SQL, the JSON parsed in Go. Two small things need to change before this stands on its own; the rest are notes.
1. MINOR, please fix — the cold-start guard tells the operator to run a command that does not exist at this commit
internal/storage/dolt/store.go:3671:
"... this bd does not own the server's data directory; re-add it with: bd dolt remote add %s %s --ref %s"bd dolt remote add is cobra.ExactArgs(2) with one flag, --allow-git-origin (cmd/bd/dolt.go:1603-1608); --ref arrives only with #6525, which is a draft. This PR is meant to merge first, and the guard is reachable without #6525 — a ref remote made with dolt remote add --ref directly, on a server this bd does not own, at GH#2118 cold start. That operator is handed unknown flag: --ref as the way out. cli_mirror_ref_test.go:186 asserts the string, so the test pins it.
Either name what works today (the CALL DOLT_REMOTE('add', '--ref', …) against that server, or dolt remote add --ref in the server's directory), or hold the bd spelling for #6525 and change the message there together with its test.
2. MINOR, please fix — "probes left by an interrupted run are swept" is not what the code does
doltutil/remotes.go:264-271 removes a leftover only when it is named bd-ref-probe-<this pid>, and the comment above it says so. A run killed between the add and the remove leaves bd-ref-probe-<old pid> in repo_state.json, and nothing later matches it short of pid reuse. cliRemoteLock is a sync.Mutex, in-process only, so I can see why you did not sweep by prefix — another bd's live probe could be the thing swept (that process would then fail on its own RemoveCLIRemote(probe) before touching the real remote: spurious, but safe). Your call between:
- sweep the prefix and accept that spurious failure, or
- keep own-pid only and correct the body, and say in the comment that a dead run's probe stays until removed by hand.
One related wrinkle: when PersistedRemotes errors, the sweep is skipped silently, and a same-pid leftover then fails the add under the "a dolt sql-server serving this directory refuses remote parameters" wording, which would be the wrong diagnosis.
Notes, not asks
- "Nothing changes for a remote without a ref" has one exception, and I think it is the right one.
EnsureCLIRemote(remotes.go:316-321) andhasMatchingCLIRemote(store.go:3534) now readrepo_state.jsonunconditionally, so a no-ref user with a truncated state file gets an error wheremainwent on to the URL check. Failing closed is correct — an unreadable file could be hiding a custom ref on the mirror, and skipping the read when the expected ref is empty would reopen exactly the wrong-ref push this PR closes. Worth a clause in the body so the claim is exact. PersistedRemotesnow fails whole on one malformedparams(Params map[string]any; a non-object value fails the unmarshal where the field used to be ignored). dolt always writes an object ("params": {}on a plain remote — checked on 2.3.1), so this takes a hand-edited file to hit; the cost if hit is thatfederation.go:249-254warns "assuming none".json.RawMessageplus a per-remote error would keep the blast radius to the one remote. Low priority.- Two sources for one remote.
currentcomes fromdolt remote -v,currentReffrom the state file. On 2.3.1 the listing already carries the params:andgitr git+https://example.com/repo.git {"git_ref": "refs/heads/issue-data"}ListCLIRemotesdrops everything afterparts[1]. Reading the ref from the same listing (mind the space inside the JSON when splitting) would make URL and ref one observation, and it is the proxied server's own answer rather than the file's. I could not construct a divergence that guard 1 does not already cover, so this is a simplification to consider, not a defect. bd dolt remote reset-datais ref-blind (cmd/bd/dolt_remote_reset_data.go:26,249: pinsrefs/dolt/data, takesr.URL, never looks atr.Ref). With several databases in one repository — this PR's own motivating layout — resetting a remote onrefs/dolt/units/<key>would delete the default-ref database's data plane and leave the target's in place. Out of scope here by your own cmd/bd line, and I see #6525 touches that file; I have not read it yet and will check for this there. Flagging it so it is on the record for the stack.ValidateGitDataRefArglets U+00A0 through. Not exploitable (no leading dash, git refuses it); mentioning only because the doc comment says "whitespace".
Checked and clean
- Every path where the ref becomes a
doltargv element validates it:AddCLIRemoteWithRef(which the probe goes through) anddoltCloneArgsviaBootstrapFromRemoteWithDBRef. Everything else binds it as a parameter. - No mirror call site drops the ref (
credentials.go,federation.go:639,store.go:3646,3678). - The ownership proof holds:
localActiveDatabaseDiris empty for gateway, proxied, socket, TLS, non-localhost and env-port configurations, so a client of an external server cannot reach the cold-start SQL re-add. paramsis indolt_remotesin the pinned engine (dtables/remotes_table.go:81) and in 2.3.1'srepo_state.json; the widerSELECTis safe.- The probe-before-remove ordering is real and
TestCLIMirrorRefusedProbeLeavesRemoteInPlacewould catch a reorder.
Requesting changes for 1 and 2 only; both are a few lines. I'll re-read on the push.
0c0a57d to
efe18a8
Compare
|
@bee-ghosttrack Thanks for the careful read, and for running the CLI tests on a real dolt. Pushed as 1. The cold-start remedy. Taken: the message now names what works at this commit, the procedure call over bd's connection to that server: 2. The probe sweep. A sweep, narrower than a prefix, and it now runs before every change to the mirror, not only ahead of a probe. Notes.
Verification on the new head: |
|
Adversarial review of this push, on the delta since Findings: 25 in total, 0 blocker, 4 major, the rest minor or question. 24 accepted and fixed on the branch; 1 question answered without a change, the remedy's quoting under cmd.exe, which follows the Verification on this head, dolt 2.3.2 on PATH: the doltutil suite with the CLI tests, the |
bee-ghosttrack
left a comment
There was a problem hiding this comment.
Re-read at efe18a802 on main at 7ccb961f7. The rebase is clean by range-diff against f56632adc..0c0a57d45: every changed hunk is one of the two asks or the whitespace note, and nothing from main's other store.go / version_control.go hunks leaks into the PR's delta.
Ran on this head, all ok: go build ./...; go vet ./internal/storage/doltutil/ for darwin, linux, windows and js/wasm; with BEADS_TEST_ENV_RUN_DOLT=1 on dolt 2.3.1: ./internal/storage/doltutil/... (the sweep, leftover-shape, unreadable-state, own-removal-failure and both invalid-ref tests included) and -run 'TestColdStart|TestCLIMirror|TestPrepareCLIRoute' ./internal/storage/dolt/; -run TestRemoteGitDataRefOnEngine ./internal/storage/embeddeddolt/ on the real engine.
1. The remedy now names a command that exists and repairs the condition. Checked the premises rather than the wording: bd sql is cobra.ExactArgs(1), so the single-quoted statement is one argument; a CALL goes through ExecContext past CheckReadonly, and the guard fires on the push/pull route decision, not on open, so the store bd sql needs is reachable in the GH#2118 window. DOLT_REMOTE takes --ref in the pinned engine (dprocedures/dolt_remote.go:110), and on a 2.3.1 repo CALL DOLT_REMOTE('add', '--ref', 'refs/heads/issue-data', 'origin', 'git+https://example.com/o''hare/repo.git') records {"git_ref": "refs/heads/issue-data"} in repo_state.json and shows it in dolt remote -v; the second call returns remote already exists, as your reply says. The double-quoted literal form the remedy prints works the same way, with a " and a \ in the URL landing intact (dolt percent-encodes them in the stored URL, which is dolt's normalisation and not this PR's concern). TestColdStartReaddRemedyQuotes pins the shell word.
2. The sweep does what the body now says: the exact bd-ref-probe-<pid> shape in canonical decimal, a dead pid only where the platform can tell (ESRCH alone on unix; OpenProcess plus a zero wait on Windows, WAIT_OBJECT_0 read as exited, access-denied as alive), the own name always, and a failed removal of another run's leftover ignored while a failed own-name removal is returned in its own words. TestCLIMirrorSweepsLeftoverProbes plants all four shapes and checks exactly the own and dead ones go, on the dolt CLI. The PersistedRemotes error is no longer swallowed; TestCLIMirrorSweepReportsUnreadableState covers it. ValidateGitDataRefArg at the top of EnsureCLIRemote closes the mis-wording over an existing remote, and the U+00A0 question is settled the right way round: the comment changed, the check did not, and the test pins both sides.
The notes are disposed as you describe; reset-data I'll read on #6525 as planned.
Approving. CI has not run on this PR (fork workflow awaiting approval), so the Windows path is vet-only from here as well; nothing in it is reached on the two platforms the tests ran on.
|
Triage: |
|
This PR has been added to the review queue. Track progress: https://factory.gascity.com/city/maintainer-city/runs/gcg--9223372036854769982 |
Dolt lets a git-backed remote keep its data on a ref other than refs/dolt/data (the git_ref remote parameter, --ref on dolt remote add, dolt clone, DOLT_REMOTE, and DOLT_CLONE). The storage layer never passed it through, so a host that only accepts pushes under refs/heads/, or one repository holding several databases on refs such as refs/dolt/units/<key>, could not be used. Part of gastownhall#6261; the commands and the config key follow in a stacked PR. - storage.RemoteInfo gains Ref (empty = refs/dolt/data); RemoteRefsMatch, EffectiveGitDataRef, and GitRefFromParamsJSON give every package one reading of it, pinned against dolt's constants. - RemoteStore.AddRemoteWithRef on both stores, and CreateRemoteWithRef on the proxied-server remote use case, pass a non-empty ref as DOLT_REMOTE's --ref. AddRemote is unchanged; UpdateRemote keeps the ref through replacement and restore. - versioncontrolops.DoltCloneWithRef and dolt.BootstrapFromRemoteWithDBRef pass the ref to DOLT_CLONE and to the dolt clone argv. - ListRemotes reads the ref back from dolt_remotes.params (git_ref) in both remote repositories; PersistedRemotes reads it from repo_state.json. Malformed params is an error, never a silent default. - The server-mode CLI mirror is compared on URL and ref and re-materialized with dolt remote add --ref when the ref differs. Where the CLI is proxied to the running server: the check reads the server's own repo_state.json when the listing is empty, the cold-start fallback re-adds a ref remote over SQL (which accepts --ref) so the server knows it, and a throwaway probe remote proves --ref works before an existing mirror remote is removed; probe remotes left by runs no longer alive are swept before any change to the mirror. For a server this bd does not own, the route stops and names the DOLT_REMOTE call to run against that server through bd sql, as one shell-quoted word. - A ref is checked for a space, an ASCII control character, DEL, and a leading dash before it becomes a dolt argument; no other ref-name rule applies at this layer. - Tests carry both ref shapes; the embedded-engine test proves add, list, push, and clone on the real engine; sqlmock tests cover the route decider and the cold-start re-add at the store seam; the dolt-CLI tests skip without a dolt binary.
…astownhall#6521 review) Maintainer review fixup for PR gastownhall#6521. The review found no blocker or major; this is the non-gating minor/nit set plus the test pins the synthesis named as the residual defects. - One reader for git_ref. storage.GitRefFromParams is now the single decoder for both planes (the SQL params column through GitRefFromParamsJSON, and repo_state.json through doltutil.PersistedRemotes), and a git_ref that is present but not a string is an error naming the remote instead of silently the default ref. Reading it as the default is how a mirror pinned to a custom ref would be kept on refs/dolt/data -- the wrong-ref push this PR closes, reached from the opposite direction. - CreateRemoteWithRef checks the ref against what the dolt argv boundary will accept, so the recordable ref set is no longer wider than the routable one and a bad ref is refused at add time rather than diagnosed at push time. UpdateRemote deliberately does not re-check: it carries and restores refs that are already recorded and must stay able to put one back. - BootstrapFromRemoteWithDBRef validates the ref before the already-exists early return, so the same --ref is accepted or refused the same way on a fresh host and on a bootstrapped one. - sweepLeftoverProbes warns when it cannot remove another run's leftover probe remote, following PersistedRemoteInfos' precedent: the remote stays visible in `bd remote list` until some run succeeds, and nothing ever said why. - EnsureCLIRemote's doc comment records that escalating a state-file read error is intentional for default-ref remotes too, and TestCLIMirrorUnreadableStateIsAnError now pins the ref == "" direction that a later "helpful" relaxation would gate away. The suggested URL-match short-circuit is deliberately NOT applied: when the state file is unreadable currentRef is unknown, so returning nil on a URL match alone would route a push onto whatever ref the mirror is secretly pinned to. - AddCLIRemote's doc comment records that it is kept deliberately as the default-ref spelling now that EnsureCLIRemote uses AddCLIRemoteWithRef. Not applied: widening probeCLIRemoteRef to `ref != "" || currentRef != ""` and probing the restore shape. On a proxied server that turns a working custom-ref -> default-ref transition into a hard failure (the target add is plain and succeeds today, while probing the restore shape would be refused), and in the one window that survives calibration -- a non-proxied plain-add failure from disk/lock contention -- a capability probe predicts nothing about it. Each new pin was mutation-proved: reverting the fix it guards reds that pin, and only that pin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
efe18a8 to
0801749
Compare
bee-ghosttrack
left a comment
There was a problem hiding this comment.
R2 review at 0801749 (the maintainer fixup over my R1 approve at efe18a8). Approve.
What changed since R1: rebased onto main c25059a (range-diff: the original commit is =) plus one commit. Both git_ref readers, storage.GitRefFromParamsJSON over the dolt_remotes params column and doltutil.PersistedRemotes over repo_state.json, used to do v, _ := params["git_ref"].(string), so a git_ref that was present but not a string was silently read as the default refs/dolt/data, which is the wrong-ref push this PR exists to close. They now share one reader, storage.GitRefFromParams: missing key or JSON null still means the default; any other non-string value is an error, and PersistedRemotes names the remote in it. The commit also checks the ref in CreateRemoteWithRef before anything is recorded, moves the ref check in BootstrapFromRemoteWithDBRef ahead of the "database already exists" early return, and makes the probe sweep warn instead of silently ignoring a failed removal of another run's leftover probe. Every git_ref read in non-test code now goes through the shared reader; the two callers that still swallow its error (persistedCLIRemote, PersistedRemoteInfos) land on the no-push side. 29 files, all under internal/storage/; merge-tree against main clean; the main delta since the merge-base touches none of them.
Ran here (go1.27.1, dolt 2.3.5), at 0801749: go build ./...; go vet on the touched storage packages; gofmt clean. go test over ./internal/storage/, domain/..., versioncontrolops/..., doltutil/... with BEADS_TEST_ENV_RUN_DOLT=1: all ok, 0 SKIPs. ./internal/storage/dolt/ -run 'TestColdStart|TestCLIMirror|TestPrepareCLIRoute|TestBootstrap|TestDoltCloneArgs': ok, 37 run, 0 SKIPs. Embedded tier (BEADS_TEST_EMBEDDED_DOLT=1 -tags cgo): TestRemoteGitDataRefOnEngine, TestGitRefPinsMatchDolt PASS.
Mutations (each reverted, tree clean):
- Shared reader returns
"", nilon a non-string:remote_ref_test.go:70reds for array, number, bool and object (want an error, not the default ref),TestGitRefFromParamsJSONRefusesNonStringRefreds, andpersisted_remotes_test.go:104reds (PersistedRemotes with a non-string git_ref = [{Name:unit … Ref:}], nil; want an error). - Only the on-disk reader falls back to the silent assert: the doltutil pin reds while the storage package stays green, so each reader has its own pin.
CreateRemoteWithRefcheck removed:remote_test.go:131"should be refused" for-x, space, tab;:135"a refused CreateRemoteWithRef reached the repository".- Bootstrap ref check moved back behind the early return:
bootstrap_test.go:174reds. EnsureCLIRemoteescalates only when a ref was asked for:cli_mirror_ref_test.go:390reds.
Not blocking, for the record: the pin in (5) goes red on message wording (the mutated code still fails, later, in the probe sweep, with a message lacking "recorded ref"); the case that matters, URL match + empty ref + unreadable state file, is not in the fixture. The new sweep warning has no test. One non-string git_ref now fails PersistedRemotes for every remote in that directory, the same fail-closed trade-off noted last round; Dolt only ever writes a string there.
Maintainer Adoption ReviewThanks for the contribution, @sdghbtn! This PR makes a git-backed Dolt remote's git data ref a first-class part of the remote record — carried through the SQL plane ( This PR was reviewed and adopted with maintainer fixes pushed directly to the PR branch. Original PR ReviewDecision: approve. Specific gaps fixed:
Review findings addressed:
Each new pin was mutation-proved: reverting the fix it guards reds that pin, and only that pin. Non-blocking follow-ups (shared for visibility — NOT completed in this PR):
Maintainer ChangesOne maintainer fixup commit, Final Review StatusReady for the merge queue: final head CI: https://github.com/gastownhall/beads/actions/runs/36263641540 Review Iterations1 review pass, by two reviewers (Claude + Codex) who independently validated the PR's premise against the pre-change tree rather than taking the PR body's word for it. No blocker or major survived; the non-gating minor/nit set was applied as the single maintainer fixup above. One coverage caveat worth stating plainly: a third reviewer lane was skipped for this run, so cross-file coverage here is two-reviewer rather than three. Adopted via |
Resolve internal/storage/dolt/credentials.go: main (gastownhall#6454 execenv work) replaced the nested os.Environ()/strings.HasPrefix scan with execenv.ContainsKeyWithPrefix, which prefix-matches the env KEY only and applies Windows key identity. This PR independently threads the git data ref through ensureMatchingCLIRemote. The two changes are orthogonal, so keep both: main scan structure + this PR three-arg call. The auto-merged store.go already carries the three-arg definition, so the two-arg arm would not compile.
…6464, round 2) main advanced two commits while the first merge was being resolved: gastownhall#6521 (carry the git data ref of a git-backed Dolt remote) and gastownhall#6461 (retain explicit null config suppression). gastownhall#6521 is disjoint from this PR and auto-merged. gastownhall#6461 lands on the same routing boundary this PR hardens, so all three conflicts are two descriptions of one boundary rather than competing behavior. Resolutions: - cmd/bd/worktree_cmd.go: union. Keep main's new first paragraph, which records that startup config discovery and the already-selected .beads probes share the boundary while the generic internal/beads and internal/git/gitdir.go planes do not (bd-p4che) -- that is new information from gastownhall#6461. Keep this PR's reviewed three-bullet clause split over main's one-sentence version: it says everything main's sentence said, preserves main's "(init's role probe records the same trade-off)" parenthetical, and additionally distinguishes GIT_CEILING_DIRECTORIES (a stop, so clearing widens the walk) from GIT_DISCOVERY_ACROSS_FILESYSTEM (permit-only, so clearing narrows it). Fold gastownhall#6461's suppression exception into the GIT_CONFIG* bullet, where it belongs: ClearRouting matches GIT_CONFIG by prefix but keeps GIT_CONFIG_NOSYSTEM and the /dev/null spellings, which can blind a read but cannot redirect one. - cmd/bd/config.go: keep this PR's comment naming the four verbs (get/set/unset/set-many). It is the narrower and still-true form, and it is the remedy a review finding asked for; main's "bd config's own reads and writes" phrasing is the superseded generalization. - internal/storage/domain/git/git_test.go: superset. Keep every assertion from both sides -- main's require.Error message, ErrorContains(".gitconfig") and the inherited SetConfig refusal, plus this PR's stricter exec.ExitError/exit-128 pin. The two sides assert the same failure at different strengths, so neither contract is dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…light-terminal-glamour-style * gastownhall/main: (157 commits) fix(hooks): identify the index context behind tracked-file refusals (gastownhall#6457) fix(gitenv): retain explicit null config suppression (gastownhall#6461) feat(storage): carry the git data ref of a git-backed Dolt remote (gastownhall#6521) fix(init): write embedded excludes in the selected project (gastownhall#6462) fix(doctor): avoid a leading separator in empty gitignore (gastownhall#6421) fix(notion): keep notion.token out of the Dolt database (gastownhall#6676) (gastownhall#6724) test(bazel): run every Bazel test against a hermetic, pinned dolt CLI (gastownhall#6853) fix(init): bootstrap Git in the selected project directory (gastownhall#6459) fix(doctor): escape database identifiers instead of rejecting them (gastownhall#6412) test(bazel): hermetic test env + inject bd binary into subprocess tests (gastownhall#6843) fix(doctor): report stealth exclude read failures without repair advice (gastownhall#6427) refactor(git): resolve hooks paths in an explicit read-only context (gastownhall#6436) fix(prime): honor shared-server mode consistently across bd prime's store resolution paths (gastownhall#6589) test(bazel): runfiles helpers + declared inputs for scan-guard tests (gastownhall#6828) fix(test): point tx-minting guard at commitWriteTx (gastownhall#6833) docs(recovery): init-safety re-clone playbook covers damaged-store and fresh-clone gotchas (gastownhall#6431) build(bazel): side-by-side Bazel scaffold + policy tests (gastownhall#6798) Accept uutils coreutils timeout in generated hooks, and check the shims against real implementations (gastownhall#6561) fix(list): expose comment bodies in bd list/show --json (be-73x) (gastownhall#6473) fix: dep add stores an unresolved target as a bare external ref, then readers drop it (gastownhall#6571) ... ; Conflicts: ; CHANGELOG.md ; docs/recovery/init-safety.md
…it repository (gastownhall#7902) bd dolt remote reset-data classified a file:// remote by what is on disk. Dolt makes a file URL a git remote only when it ends in .git (stored as git+file://); any other file:// URL is a native Dolt file store, which Dolt writes among the files of a git repository at that path. Once Dolt had pushed into a checkout, reset-data took the directory for a file store and cleared it, .git and every working file included, then reported success. For a bare repository it deleted no Dolt data refs, force-pushed, and reported a reset that left every old table file in place. reset-data now refuses a file:// remote (or bare path) whose directory is a git repository: any .git entry (a directory, as in a checkout, or a file, as in a linked worktree or submodule), or a bare repository. It gives the replace-the-remote hint it already gives a cloud remote, and the help text says so. Tests: TestClassifyResetDataRemote gains gastownhall#7902's legs (a bare repository with and without a Dolt store, a checkout holding a store by file:// URL and by bare path, a linked worktree whose .git is a file, a checkout without a store); with the old classification the checkout and worktree legs holding a store read as file stores (= 1, want 4), the bare legs as git-backed (= 0, want 4), and the checkout without a store as an error. TestEmbeddedDoltRemoteResetData_FileRemoteInGitRepoRefused pushes to a file:// remote in a bare repository and in a checkout, then checks that reset-data --yes refuses each and leaves the directory, .git, README.md and the git history intact. Run against a bd built from hotfix/1.3.2 (BEADS_TEST_BD_BINARY) it fails: reset-data exits 0. Backport of gastownhall#7902 (c153e3b). [hotfix/1.3.2 backport] Ports item 4 of gastownhall#7902 only (the file:// remote in a git repository); this line predates the endpoint-ownership chain the rest of the PR builds on. - Ported from gastownhall#7902: resetDataFileInGitRepo, isGitRepoDir, the classifyResetDataRemote doc comment and refusal, the RunE refusal message, the help-text line, and the TestClassifyResetDataRemote table. The two existing legs that sent a file:// URL or bare path to a bare repository down the git-backed route now expect the refusal, as on main. - The refusal's hint prints the remote name as this line's cloud-remote refusal does; main quotes it through internal/remedy, which is not on this line. - Added for this line: TestEmbeddedDoltRemoteResetData_FileRemoteInGitRepoRefused, the bare-repository and checkout refusal legs of gastownhall#7902's server-mode TestE2E_ResetDataThroughOwnServerKeepsTheRemotesData, run in embedded mode (the classification runs before any push, in either mode; that E2E itself needs the server route, which is not ported). Its TestEmbedded prefix puts it in CI's embedded cmd shards. - Not ported: - Item 1 (cliDirServesServer requiring doltserver.ServesDatabaseDirOnPort, the databaseLocked probe, testutil.HoldDoltDatabaseLock, and the storage-boundary exception in AGENTS.md and engdocs/PROJECT_CHARTER.md): it narrows gastownhall#7870/gastownhall#7893's host-ownership predicate and leftover-remote refusal, which build on gastownhall#7840/gastownhall#7841/gastownhall#7874; none of them is on this line, where hasCLIDatabase checks only that .beads/dolt/<db>/.dolt exists. - Item 2 (the leftover refusal's wording): gastownhall#7893's remoteOnlyInLeftoverDatabaseError does not exist on this line. - Item 3 (reset-data's server route: a temporary remote and ref, a leased move of refs/dolt/data, and a restart of bd's server): it needs AddRemoteWithRef and RemoteInfo.Ref (gastownhall#6521), doltserver.Restart, DoltStore.ManagedServerDir and internal/remedy, none of which is on this line. Embedded and server mode keep delete-then-force-push. - Item 5 (docs/recovery/history-squash.md): the published runbook is served from main, which already carries the change. - resetDataGitURL's scp-style and port handling, resetDataProgressf (--json-only stdout), CHANGELOG.md (release prep writes it), the BUILD.bazel files and tools/nogo list (no Bazel build on this line), and the command-tree golden (absent on this line). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What
Storage primitives learn which git ref a git-backed Dolt remote keeps its data on, so the application layer can offer
--refin a follow-up PR without reaching across layers.storage.RemoteInfogainsRef, JSONref, omitted when empty. Empty means Dolt's defaultrefs/dolt/data;storage.RemoteRefsMatchandstorage.EffectiveGitDataRefcompare and resolve refs with that meaning,GitRefParam/DefaultGitDataRefname dolt's parameter and default, andstorage.GitRefFromParamsJSONreads the parameter out of adolt_remotesparams value. The proxied-server remote surface (domain.Remote,RemoteSQLRepository.AddRemoteWithRef,DoltRemoteUseCase.CreateRemoteWithRef) carries the ref the same way, sobd initin that mode can use it fromcmd/bd. Federation peers (AddFederationPeer,FederationPeer) stay on the default ref; nothing in this pair of PRs changes them.RemoteStore.AddRemoteWithRefon both stores passes a non-empty ref toDOLT_REMOTE('add', '--ref', ...).AddRemoteis unchanged and isAddRemoteWithRefwith an empty ref.versioncontrolops.DoltCloneWithRefpasses the ref toDOLT_CLONE('--ref', ...);dolt.BootstrapFromRemoteWithDBRefadds--refto thedolt cloneargv on the CLI clone path. The existing functions delegate with an empty ref.versioncontrolops.ListRemotesreads the ref back from theparamscolumn ofdolt_remotes, keygit_ref;doltutil.PersistedRemotesreads it fromrepo_state.json. A remote without the key lists an empty ref. A params value that is not JSON is an error naming the remote, never a silent default.hasMatchingCLIRemote,ensureMatchingCLIRemote,doltutil.EnsureCLIRemote) and re-materialized withdolt remote add --refwhen the ref differs, so a mirror on the default ref cannot push a custom-ref database torefs/dolt/data. Three guards for the layouts bd manages, where the dolt CLI in that directory is proxied to the running sql-server: when the proxied listing shows nothing and a ref is expected, the check reads the server's ownrepo_state.jsonin that directory and treats a matching URL and ref as in place; in the cold-start fallback, where SQL lists nothing and the state file has the remote, a remote with a ref is re-added over bd's SQL connection with--ref, which the procedure accepts where the proxied CLI does not, so the server the transfer delegates to knows the remote; "already exists" is accepted, and the re-add runs only when this bd owns the served directory,localActiveDatabaseDir, the same proofresolveLocalActiveDatabaseDirdemands elsewhere. For any other server a client-local state file is not written into the server: the route stops and names theDOLT_REMOTE("add", "--ref", ...)call to run against that server throughbd sql, as one shell-quoted word with the values as escaped SQL literals. The stacked PR'sbd dolt remote add --refcannot be that remedy: it finds the persisted remote, sees URL and ref match, and returns without a SQL add. And before a ref-carrying re-materialization removes an existing mirror remote, a throwaway remote is added and removed with the same URL and ref, so dolt's refusal of remote parameters over the proxy is returned before anything is deleted. Probe remotes left by an interrupted run are swept before every change to the mirror, whatever ref it moves to: a remote named exactly as the probe names them,bd-ref-probe-<pid>, whose pid is no longer running, or this process's own name; a remote that only shares the prefix, and the probe of a run still alive, are left alone; the liveness check iskill(pid, 0)on unix, only ESRCH meaning dead, andOpenProcessplus a zero-timeout wait on Windows, cross-compiled here and not run; on any other platform only the own name is swept. Removing another run's leftover is best-effort: a removal that fails is tried again at the next change, and a pid an unrelated process has since taken keeps the leftover until that pid exits; the own-name leftover must go, and its removal failing is an error. A state file that cannot be read at sweep time is an error naming the read, not a report of the server refusing parameters, and a ref that fails the argument check below is refused before the mirror is touched. A remote on the default ref keeps today's path in all three places. The route decider and the cold-start re-add are proven at the store seam with sqlmock and a state file; the live transfer through a served directory is not exercised here, there is no dolt sql-server in this environment, and is the one claim in this PR resting on the code walk rather than a run.Choices where more than one shape was possible:
AddRemote: onlyDoltStoreandEmbeddedDoltStoreimplementRemoteStore, no fake does, and the conformance suite calls noRemoteStoremethod, so the addition is free of churn there; keepingAddRemoteas it is keeps everycmd/bdcaller untouched in this PR.doltcommand-line argument it is checked for a space, an ASCII control character, DEL, and a leading dash (doltutil.ValidateGitDataRefArg): the first three are bytes git's ref rules refuse, the dash is so the value can never be mistaken for an option; dolt accepts all four through exec, so this is bd policy at the argv boundary, not a ref-name check. A non-ASCII space such as U+00A0 is a valid ref byte for git and for dolt and passes, since this layer adds no ref-name rules.SELECT name, url, params FROM dolt_remoteswith the JSON parsed in Go rather thanJSON_EXTRACTin SQL, so the query has no dependency on JSON function support in the server the client is talking to. The real-engine test proves the parse on the embedded engine for a remote with no parameters and forgit_refin each shape; the embeddedDOLT_REMOTEprocedure accepts no other parameter, so an unrelated parameter cannot be created there. A params object with an unrelated key, the NULL and"null"renderings, and a malformed value are covered by sqlmock, and the on-disk shape by arepo_state.jsonfixture.repo_state.json, not fromdolt remote -v: the listing's rendering of remote parameters is not a documented format, and at GH#2118 cold start the proxied listing is empty, which is the case where the state file is the only source.Why
Part of #6261. Dolt supports a git remote whose data sits on a ref other than
refs/dolt/data(git_refremote parameter,--refondolt remote add,dolt clone,DOLT_REMOTE, andDOLT_CLONE); bd never passes it through, so a host that only accepts pushes underrefs/heads/cannot be used, and neither can one repository holding several Dolt databases on refs such asrefs/dolt/units/<key>. This PR is the storage half; the commands, thesync.remote-refconfig key, and the docs follow in a stacked PR.Nothing changes for a remote without a ref: the SQL for add, clone, push, and pull is the same text as before, the mirror comparison degrades to today's URL comparison, and
ListRemotesreturns the same rows with an extra empty field. One exception, deliberate: the CLI-mirror check reads the mirror directory'srepo_state.jsonunconditionally and fails closed when the file cannot be parsed, wheremainwent on to the URL comparison. An unreadable file could be hiding a custom ref on the mirror, and skipping the read when the expected ref is empty would reopen the wrong-ref push this PR closes.Verification
The fixtures that exercise ref comparison or parsing carry both shapes,
refs/heads/issue-dataandrefs/dolt/units/team-12542, so both pass through unchanged; tests where the ref is only a bind parameter or an argv element use one.