LEP: Add storage interface for get/put of foreign keys - #169
Closed
mjansson wants to merge 1 commit into
Closed
Conversation
mjansson
force-pushed
the
proto/get-resolved-and-subfragments
branch
from
August 15, 2026 07:21
1f445f4 to
d5393f1
Compare
…trip Callers that address content by a key belonging to some other system -- an asset id, a build id, a database primary key -- need both keyspaces for every access, and the two are serially dependent: a read is mutable_load then get, a write is put then mutable_store, and the second cannot start until the first answers. The latency is two round trips regardless of how much work is queued behind it, which is what decides whether Lore is usable as a cache for foreign-keyed data. Add get_resolved and put_resolved, which resolve a key and act on the content it names within one request. Keys live under KeyType::Resolve, so the type is implied rather than sent. Both operations are available over QUIC and gRPC with the same semantics, and through lore_storage_get_resolved and lore_storage_put_resolved in the C API. A read resolves local-first and falls back to the remote on a miss, caching the mapping it learns; a locally cached mapping that names content this store does not hold defers to the remote rather than reporting a miss. The root fragment arrives with the resolution, so there is no local probe for it -- that is the round trip being saved -- but it is decompressed and verified against the resolved hash exactly as load_fragment does, so a server that resolves a key to the wrong content is caught rather than trusted. Leaves still go through load_fragment, and so are still served from and written back to the local store. Reads take the same streaming option an ordinary read does, delivering one event per leaf fragment so peak memory follows the fragment size rather than the content size. A write stores the content before publishing the key, so a key never names content the store does not hold. Content that fits one fragment publishes from inside the ordinary write pipeline: the upload it was going to perform anyway becomes put_resolved rather than put, so content and mapping travel together and the terminal local write is the only local write, already carrying the durable flag and the local_cache_priority retention decision. A publishing write never joins the in-flight rendezvous -- deduping onto another writer's upload would mean no upload of our own, and the key rides on the upload, so it would silently never ship. Content already durable uploads nothing, so there is no command for its key to ride on and the mapping follows as a mutable_store instead; still one round trip, without re-sending a payload the server holds. A fragment tree cannot fuse: its leaves upload first and the key is published only once the whole tree is durable. Writes report where content landed -- local, remote, or both -- as the intersection across every fragment, so a caller can tell content that reached the remote from content that only reached the local store. A remote leg that fails is warned rather than returned: the local write still stands, and the reported placement says so. A publish carrying no content retracts the key. Publishing is last-writer-wins, deliberately: for the caching this exists for either mapping is valid. Folding in a compare-and-swap is not available, because content must be stored before the key names it and a failed swap would strand what it just stored, worst under exactly the contention that would motivate it. Per-item outcomes travel in-band, as an ItemStatus on both Put and PutResolved. A Status on a streaming response becomes trailers and ends the stream, so one failed fragment would otherwise strand every other fragment in flight on it. The QUIC request carries a reserved 24-bit flags field, so a future option needs no second wire-format change. Unknown bits are rejected rather than ignored, so a caller can tell whether an option took effect. Repositories can now opt into or out of the shared store explicitly instead of inheriting the machine's global setting, which also isolates the test suite from whatever the developer running it has configured. Includes a LEP describing the motivation and the design, transport parity in system-design.md, and integration coverage that exercises every remote storage operation over both QUIC and gRPC -- including a fragment tree whose leaves disagree on placement, which inverting the fold to a union makes fail. Note what is not covered: no test detects the single-round-trip fusion's absence. Disabling it is semantically invisible -- the mapping simply follows as its own command and every assertion still holds; only the round-trip count changes, and nothing measures that. Guarding it needs a per-command counter on the test server, which neither transport exposes. Signed-off-by: Mattias Jansson <mjansson@gmail.com>
mjansson
force-pushed
the
proto/get-resolved-and-subfragments
branch
from
August 15, 2026 07:26
d5393f1 to
094b919
Compare
|
Imported as Lore CR-363. |
epic-lore-bot Bot
pushed a commit
that referenced
this pull request
Aug 19, 2026
## Motivation Callers that address content by a key belonging to some other system -- an asset id, a build id, a database primary key -- need both keyspaces for every access, and the two are serially dependent: a read is mutable_load then get, a write is put then mutable_store, and the second cannot start until the first answers. The latency is two round trips regardless of how much work is queued behind it, which is what decides whether Lore is usable as a cache for foreign-keyed data. ## Summary Add get_resolved and put_resolved, which resolve a key and act on the content it names within one request. Keys live under KeyType::Resolve, so the type is implied rather than sent. Both operations are available over QUIC and gRPC with the same semantics, and through lore_storage_get_resolved and lore_storage_put_resolved in the C API. A read resolves local-first and falls back to the remote on a miss, caching the mapping it learns; a locally cached mapping that names content this store does not hold defers to the remote rather than reporting a miss. The root fragment arrives with the resolution, so there is no local probe for it -- that is the round trip being saved -- but it is decompressed and verified against the resolved hash exactly as load_fragment does, so a server that resolves a key to the wrong content is caught rather than trusted. Leaves still go through load_fragment, and so are still served from and written back to the local store. Reads take the same streaming option an ordinary read does, delivering one event per leaf fragment so peak memory follows the fragment size rather than the content size. A write stores the content before publishing the key, so a key never names content the store does not hold. Content that fits one fragment publishes from inside the ordinary write pipeline: the upload it was going to perform anyway becomes put_resolved rather than put, so content and mapping travel together and the terminal local write is the only local write, already carrying the durable flag and the local_cache_priority retention decision. A publishing write never joins the in-flight rendezvous -- deduping onto another writer's upload would mean no upload of our own, and the key rides on the upload, so it would silently never ship. Content already durable uploads nothing, so there is no command for its key to ride on and the mapping follows as a mutable_store instead; still one round trip, without re-sending a payload the server holds. A fragment tree cannot fuse: its leaves upload first and the key is published only once the whole tree is durable. Writes report where content landed -- local, remote, or both -- as the intersection across every fragment, so a caller can tell content that reached the remote from content that only reached the local store. A remote leg that fails is warned rather than returned: the local write still stands, and the reported placement says so. A publish carrying no content retracts the key. Publishing is last-writer-wins, deliberately: for the caching this exists for either mapping is valid. Folding in a compare-and-swap is not available, because content must be stored before the key names it and a failed swap would strand what it just stored, worst under exactly the contention that would motivate it. ``` Imported-PR: #169 Imported-From: 094b919 Imported-Base: 132d574 Imported-Merge: d7b7bca Imported-Author: Mattias Jansson (mjansson) Signed-off-by: Mattias Jansson <mjansson@gmail.com> GH-URL: #169 ``` Lore-RevId: 634 Lore-Signature: 7e6e378a4002f9c11871fdb6e27ae1b4cabf8e1795ad5105795326f0d9da3324
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Callers that address content by a key belonging to some other system -- an asset id, a build id, a database primary key -- need both keyspaces for every access, and the two are serially dependent: a read is mutable_load then get, a write is put then mutable_store, and the second cannot start until the first answers. The latency is two round trips regardless of how much work is queued behind it, which is what decides whether Lore is usable as a cache for foreign-keyed data.
Summary
Add get_resolved and put_resolved, which resolve a key and act on the content it names within one request. Keys live under KeyType::Resolve, so the type is implied rather than sent. Both operations are available over QUIC and gRPC with the same semantics, and through lore_storage_get_resolved and lore_storage_put_resolved in the C API.
A read resolves local-first and falls back to the remote on a miss, caching the mapping it learns; a locally cached mapping that names content this store does not hold defers to the remote rather than reporting a miss. The root fragment arrives with the resolution, so there is no local probe for it -- that is the round trip being saved -- but it is decompressed and verified against the resolved hash exactly as load_fragment does, so a server that resolves a key to the wrong content is caught rather than trusted. Leaves still go through load_fragment, and so are still served from and written back to the local store. Reads take the same streaming option an ordinary read does, delivering one event per leaf fragment so peak memory follows the fragment size rather than the content size.
A write stores the content before publishing the key, so a key never names content the store does not hold. Content that fits one fragment publishes from inside the ordinary write pipeline: the upload it was going to perform anyway becomes put_resolved rather than put, so content and mapping travel together and the terminal local write is the only local write, already carrying the durable flag and the local_cache_priority retention decision. A publishing write never joins the in-flight rendezvous -- deduping onto another writer's upload would mean no upload of our own, and the key rides on the upload, so it would silently never ship. Content already durable uploads nothing, so there is no command for its key to ride on and the mapping follows as a mutable_store instead; still one round trip, without re-sending a payload the server holds.
A fragment tree cannot fuse: its leaves upload first and the key is published only once the whole tree is durable. Writes report where content landed -- local, remote, or both -- as the intersection across every fragment, so a caller can tell content that reached the remote from content that only reached the local store. A remote leg that fails is warned rather than returned: the local write still stands, and the reported placement says so. A publish carrying no content retracts the key.
Publishing is last-writer-wins, deliberately: for the caching this exists for either mapping is valid. Folding in a compare-and-swap is not available, because content must be stored before the key names it and a failed swap would strand what it just stored, worst under exactly the contention that would motivate it.
Auxiliary
Add shared store control to be able to override the machine global config in tests and when running CLI/API commands. Previously tests targeting shared store would behave differently depending on what config the machine had, not isolated to the test.
Tests
Updates the integration test harness to be able to test both QUIC and gRPC transports, and to work without pre-generated certs. Also leverage the new shared store control to ensure tests are run in a controlled environment without interference from system configuration.