Skip to content

LEP: Add storage interface for get/put of foreign keys - #169

Closed
mjansson wants to merge 1 commit into
mainfrom
proto/get-resolved-and-subfragments
Closed

LEP: Add storage interface for get/put of foreign keys#169
mjansson wants to merge 1 commit into
mainfrom
proto/get-resolved-and-subfragments

Conversation

@mjansson

Copy link
Copy Markdown
Collaborator

Motivation

Callers that address content by a key belonging to some other system -- an asset id, a build id, a database primary key -- need both keyspaces for every access, and the two are serially dependent: a read is mutable_load then get, a write is put then mutable_store, and the second cannot start until the first answers. The latency is two round trips regardless of how much work is queued behind it, which is what decides whether Lore is usable as a cache for foreign-keyed data.

Summary

Add get_resolved and put_resolved, which resolve a key and act on the content it names within one request. Keys live under KeyType::Resolve, so the type is implied rather than sent. Both operations are available over QUIC and gRPC with the same semantics, and through lore_storage_get_resolved and lore_storage_put_resolved in the C API.

A read resolves local-first and falls back to the remote on a miss, caching the mapping it learns; a locally cached mapping that names content this store does not hold defers to the remote rather than reporting a miss. The root fragment arrives with the resolution, so there is no local probe for it -- that is the round trip being saved -- but it is decompressed and verified against the resolved hash exactly as load_fragment does, so a server that resolves a key to the wrong content is caught rather than trusted. Leaves still go through load_fragment, and so are still served from and written back to the local store. Reads take the same streaming option an ordinary read does, delivering one event per leaf fragment so peak memory follows the fragment size rather than the content size.

A write stores the content before publishing the key, so a key never names content the store does not hold. Content that fits one fragment publishes from inside the ordinary write pipeline: the upload it was going to perform anyway becomes put_resolved rather than put, so content and mapping travel together and the terminal local write is the only local write, already carrying the durable flag and the local_cache_priority retention decision. A publishing write never joins the in-flight rendezvous -- deduping onto another writer's upload would mean no upload of our own, and the key rides on the upload, so it would silently never ship. Content already durable uploads nothing, so there is no command for its key to ride on and the mapping follows as a mutable_store instead; still one round trip, without re-sending a payload the server holds.

A fragment tree cannot fuse: its leaves upload first and the key is published only once the whole tree is durable. Writes report where content landed -- local, remote, or both -- as the intersection across every fragment, so a caller can tell content that reached the remote from content that only reached the local store. A remote leg that fails is warned rather than returned: the local write still stands, and the reported placement says so. A publish carrying no content retracts the key.

Publishing is last-writer-wins, deliberately: for the caching this exists for either mapping is valid. Folding in a compare-and-swap is not available, because content must be stored before the key names it and a failed swap would strand what it just stored, worst under exactly the contention that would motivate it.

Auxiliary

Add shared store control to be able to override the machine global config in tests and when running CLI/API commands. Previously tests targeting shared store would behave differently depending on what config the machine had, not isolated to the test.

Tests

Updates the integration test harness to be able to test both QUIC and gRPC transports, and to work without pre-generated certs. Also leverage the new shared store control to ensure tests are run in a controlled environment without interference from system configuration.

@mjansson
mjansson force-pushed the proto/get-resolved-and-subfragments branch from 1f445f4 to d5393f1 Compare August 15, 2026 07:21
…trip

Callers that address content by a key belonging to some other system -- an asset
id, a build id, a database primary key -- need both keyspaces for every access,
and the two are serially dependent: a read is mutable_load then get, a write is
put then mutable_store, and the second cannot start until the first answers. The
latency is two round trips regardless of how much work is queued behind it, which
is what decides whether Lore is usable as a cache for foreign-keyed data.

Add get_resolved and put_resolved, which resolve a key and act on the content it
names within one request. Keys live under KeyType::Resolve, so the type is
implied rather than sent. Both operations are available over QUIC and gRPC with
the same semantics, and through lore_storage_get_resolved and
lore_storage_put_resolved in the C API.

A read resolves local-first and falls back to the remote on a miss, caching the
mapping it learns; a locally cached mapping that names content this store does
not hold defers to the remote rather than reporting a miss. The root fragment
arrives with the resolution, so there is no local probe for it -- that is the
round trip being saved -- but it is decompressed and verified against the
resolved hash exactly as load_fragment does, so a server that resolves a key to
the wrong content is caught rather than trusted. Leaves still go through
load_fragment, and so are still served from and written back to the local store.
Reads take the same streaming option an ordinary read does, delivering one event
per leaf fragment so peak memory follows the fragment size rather than the
content size.

A write stores the content before publishing the key, so a key never names
content the store does not hold. Content that fits one fragment publishes from
inside the ordinary write pipeline: the upload it was going to perform anyway
becomes put_resolved rather than put, so content and mapping travel together and
the terminal local write is the only local write, already carrying the durable
flag and the local_cache_priority retention decision. A publishing write never
joins the in-flight rendezvous -- deduping onto another writer's upload would
mean no upload of our own, and the key rides on the upload, so it would silently
never ship. Content already durable uploads nothing, so there is no command for
its key to ride on and the mapping follows as a mutable_store instead; still one
round trip, without re-sending a payload the server holds.

A fragment tree cannot fuse: its leaves upload first and the key is published
only once the whole tree is durable. Writes report where content landed -- local,
remote, or both -- as the intersection across every fragment, so a caller can
tell content that reached the remote from content that only reached the local
store. A remote leg that fails is warned rather than returned: the local write
still stands, and the reported placement says so. A publish carrying no content
retracts the key.

Publishing is last-writer-wins, deliberately: for the caching this exists for
either mapping is valid. Folding in a compare-and-swap is not available, because
content must be stored before the key names it and a failed swap would strand
what it just stored, worst under exactly the contention that would motivate it.

Per-item outcomes travel in-band, as an ItemStatus on both Put and PutResolved. A
Status on a streaming response becomes trailers and ends the stream, so one
failed fragment would otherwise strand every other fragment in flight on it.

The QUIC request carries a reserved 24-bit flags field, so a future option needs
no second wire-format change. Unknown bits are rejected rather than ignored, so a
caller can tell whether an option took effect.

Repositories can now opt into or out of the shared store explicitly instead of
inheriting the machine's global setting, which also isolates the test suite from
whatever the developer running it has configured.

Includes a LEP describing the motivation and the design, transport parity in
system-design.md, and integration coverage that exercises every remote storage
operation over both QUIC and gRPC -- including a fragment tree whose leaves
disagree on placement, which inverting the fold to a union makes fail.

Note what is not covered: no test detects the single-round-trip fusion's absence.
Disabling it is semantically invisible -- the mapping simply follows as its own
command and every assertion still holds; only the round-trip count changes, and
nothing measures that. Guarding it needs a per-command counter on the test
server, which neither transport exposes.

Signed-off-by: Mattias Jansson <mjansson@gmail.com>
@mjansson
mjansson force-pushed the proto/get-resolved-and-subfragments branch from d5393f1 to 094b919 Compare August 15, 2026 07:26
@mjansson mjansson added the ready-to-import Approved by Epic staff for import into Lore label Aug 15, 2026
@epic-lore-bot epic-lore-bot Bot added imported Imported into Lore for internal review and removed ready-to-import Approved by Epic staff for import into Lore labels Aug 15, 2026
@epic-lore-bot

epic-lore-bot Bot commented Aug 15, 2026

Copy link
Copy Markdown

Imported as Lore CR-363.

epic-lore-bot Bot pushed a commit that referenced this pull request Aug 19, 2026
## Motivation
Callers that address content by a key belonging to some other system -- an asset id, a build id, a database primary key -- need both keyspaces for every access, and the two are serially dependent: a read is mutable_load then get, a write is put then mutable_store, and the second cannot start until the first answers. The latency is two round trips regardless of how much work is queued behind it, which is what decides whether Lore is usable as a cache for foreign-keyed data.

## Summary
Add get_resolved and put_resolved, which resolve a key and act on the content it names within one request. Keys live under KeyType::Resolve, so the type is implied rather than sent. Both operations are available over QUIC and gRPC with the same semantics, and through lore_storage_get_resolved and lore_storage_put_resolved in the C API.

A read resolves local-first and falls back to the remote on a miss, caching the mapping it learns; a locally cached mapping that names content this store does not hold defers to the remote rather than reporting a miss. The root fragment arrives with the resolution, so there is no local probe for it -- that is the round trip being saved -- but it is decompressed and verified against the resolved hash exactly as load_fragment does, so a server that resolves a key to the wrong content is caught rather than trusted. Leaves still go through load_fragment, and so are still served from and written back to the local store. Reads take the same streaming option an ordinary read does, delivering one event per leaf fragment so peak memory follows the fragment size rather than the content size.

A write stores the content before publishing the key, so a key never names content the store does not hold. Content that fits one fragment publishes from inside the ordinary write pipeline: the upload it was going to perform anyway becomes put_resolved rather than put, so content and mapping travel together and the terminal local write is the only local write, already carrying the durable flag and the local_cache_priority retention decision. A publishing write never joins the in-flight rendezvous -- deduping onto another writer's upload would mean no upload of our own, and the key rides on the upload, so it would silently never ship. Content already durable uploads nothing, so there is no command for its key to ride on and the mapping follows as a mutable_store instead; still one round trip, without re-sending a payload the server holds.

A fragment tree cannot fuse: its leaves upload first and the key is published only once the whole tree is durable. Writes report where content landed -- local, remote, or both -- as the intersection across every fragment, so a caller can tell content that reached the remote from content that only reached the local store. A remote leg that fails is warned rather than returned: the local write still stands, and the reported placement says so. A publish carrying no content retracts the key.

Publishing is last-writer-wins, deliberately: for the caching this exists for either mapping is valid. Folding in a compare-and-swap is not available, because content must be stored before the key names it and a failed swap would strand what it just stored, worst under exactly the contention that would motivate it.

```
Imported-PR: #169
Imported-From: 094b919
Imported-Base: 132d574
Imported-Merge: d7b7bca
Imported-Author: Mattias Jansson (mjansson)
Signed-off-by: Mattias Jansson <mjansson@gmail.com>
GH-URL: #169
```

Lore-RevId: 634
Lore-Signature: 7e6e378a4002f9c11871fdb6e27ae1b4cabf8e1795ad5105795326f0d9da3324
@mjansson mjansson closed this Aug 19, 2026
@mjansson
mjansson deleted the proto/get-resolved-and-subfragments branch August 19, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

imported Imported into Lore for internal review

Development

Successfully merging this pull request may close these issues.

1 participant