Skip to content

Commit 1af363a

Browse files
committed
LEP: Unify store get/query/exist interfaces and a define a contract for the store trait
## Problem `ImmutableStore` had four ways to ask whether a store held an address, each written many times for the store types, and they did not agree. The AWS store resolved the same address two ways and only one of them knew what obliteration was; the replicas sent one request under two names; the remote store downloaded a payload to answer a question about metadata; the composite carried four copies of one fan-out. Nothing said what any of it meant, so each implementation decided for itself, which is why they differed. ## Proposal Query is unified with exist and becomes a batch interface, responding the best match level found in the store, with a clear contract what the store must uphold in terms of data partitioning. Both get and metadata reads lose the match level parameter as it was mostly unused and better served by an answer of the best level found - with one `StoreGetData` carrying the fragment, the level and an optional payload, so a caller that loads content learns whether the association was its own. The store contract is written into the trait and enforced by a battery every store runs: never over-report, may under-report, obliterated never matches, reads do not under-serve and agree with each other, and a match names where it was found. ## Validation This change also carries the implementation to show what it looks like in practice - the reduced and unified store interface along with the fixes the contract enforcement caught. ``` Imported-PR: #164 Imported-From: 0813eec Imported-Base: d09b123 Imported-Merge: b83f073 Imported-Author: Mattias Jansson (mjansson) Signed-off-by: Mattias Jansson <mjansson@gmail.com> GH-URL: #164 ``` Lore-RevId: 555 Lore-Signature: 2acc021e9fee932a40cb45fd89d98db1175bdc3a98380a1cba9e3b78c542f5f7
1 parent 66fbef3 commit 1af363a

65 files changed

Lines changed: 5121 additions & 3538 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/proposals/2026-08-03-fragment-metadata-on-the-s3-object.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -461,7 +461,7 @@ re-put that sees `MatchFull` and does nothing.
461461
`error!` turn a silent miss into an alarmable one. *Implemented, on both reads that can observe
462462
it: `get` and `get_metadata`. Doing it on only one would have made detection depend on which call
463463
a client happened to make, and `get_metadata` is the cheaper one. The counter is
464-
`store.immutable.missing_payload`.*
464+
`urc.store.immutable.aws.missing_payload`.*
465465

466466
This is also why a failed `DynamoDB` read must never be reported as not-found. Repair acts on
467467
that signal, so mapping a throttle to a miss would let overload be recorded as data loss and

docs/proposals/2026-08-04-unify-store-existence-and-metadata.md

Lines changed: 790 additions & 0 deletions
Large diffs are not rendered by default.

lore-aws/src/s3.rs

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ use aws_sdk_s3::operation::list_object_versions::ListObjectVersionsOutput;
2222
use aws_sdk_s3::operation::put_object::PutObjectError;
2323
use aws_sdk_s3::operation::put_object::PutObjectOutput;
2424
use aws_sdk_s3::primitives::ByteStream;
25+
use bytes::Bytes;
2526
use lore_telemetry::InstrumentProvider;
2627
use lore_telemetry::METRICS_OPERATION_LATENCY_METRIC_NAME;
2728
use lore_telemetry::observe::Observe;
@@ -197,22 +198,22 @@ impl S3Impl {
197198
/// always observes the metadata that was written with the bytes it is reading. Writing the
198199
/// two together is what makes them impossible to tear apart.
199200
#[tracing::instrument(name = "S3Impl::put_object", skip_all)]
200-
pub async fn put_object<T>(
201+
/// The body is [`Bytes`] rather than something convertible to a `Vec<u8>`: callers already hold
202+
/// the payload in a refcounted buffer, and `ByteStream` takes one directly, so the bytes reach
203+
/// the SDK without being copied on the way.
204+
pub async fn put_object(
201205
&self,
202206
bucket: &str,
203207
key: &str,
204-
body: T,
208+
body: Bytes,
205209
metadata: Option<HashMap<String, String>>,
206-
) -> Result<PutObjectOutput, AwsError<SdkError<PutObjectError>>>
207-
where
208-
T: Into<Vec<u8>> + 'static,
209-
{
210+
) -> Result<PutObjectOutput, AwsError<SdkError<PutObjectError>>> {
210211
self.client
211212
.put_object()
212213
.bucket(bucket)
213214
.key(key)
214215
.set_metadata(metadata)
215-
.body(ByteStream::from(Into::<Vec<u8>>::into(body)))
216+
.body(ByteStream::from(body))
216217
.send()
217218
.observe(
218219
self.instruments.operation_latency_histogram.clone(),

0 commit comments

Comments
 (0)