Skip to content

feat(store): add Alibaba Cloud OSS through the S3-compatible API - #3149

Merged
palfrey merged 15 commits into
TraceMachina:mainfrom
zpzjzj:feat/oss-store
Nov 16, 2026
Merged

palfrey merged 15 commits into
TraceMachina:mainfrom
zpzjzj:feat/oss-store

Conversation

@zpzjzj

@zpzjzj zpzjzj commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Add an oss cloud object provider using Alibaba Cloud's S3-compatible API and the existing S3Store transfer implementation. Configuration requires region, bucket, access_key_id, and secret_access_key per store. Optional fields include session_token, endpoint, cname, key_prefix, retry, and multipart_max_concurrent_uploads (default: 2). Credentials support shell expansion, do not fall back to the AWS credential chain, and are redacted from config Debug output.

The default service-endpoint mode uses virtual-hosted addressing. With cname: true, an explicit HTTPS endpoint is used directly without adding the bucket to the hostname or path. A custom endpoint resolver preserves the S3 SigV4 signing region, service name, and path-encoding settings. The domain binding, DNS, and TLS certificate must already be configured by the operator. This addresses the client's CNAME-addressing gap relevant to Alibaba Cloud's default-public-endpoint restrictions for newer OSS accounts in mainland China; it does not bypass that policy.

Optional trailing checksums are disabled to avoid aws-chunked uploads. The adapter uses the existing shared HTTP connector and system DNS resolver; no new dependencies, TTL renewal, read proxy, or worker changes are included.

Two shared S3 transfer fixes are also included: correct the inclusive range end and bound in-flight multipart requests as well as queued parts. Zero-length reads return EOF without making a request. These fixes also affect existing S3-backed providers.

How was this verified?

Latest public revision: 168ff6ac8e5ea4af2398f0aba65a4b6b23df9886.

A separate CI runner passed an actual server build, workspace/all-targets Clippy with -D warnings, and 93 focused tests using Rust 1.97.1. Validation revision 5ae278885aeb752aacb945fa606bf401d1895d54 had identical tracked source to this public revision, with only the additional CI harness excluded from the PR. The job checked source-tree identity before running the commands. No local compilation was used.

cargo +1.97.1 build --bin nativelink --locked
cargo +1.97.1 clippy --workspace --all-targets --locked --keep-going -- -D warnings
cargo +1.97.1 test -p nativelink-config --test deserialization_test --test json5_test --locked
cargo +1.97.1 test -p nativelink-store --test oss_store_test --test s3_store_test --test oci_store_test --test r2_store_test --test common_s3_utils_test --locked

The 93 tests cover config deserialization (40), generated JSON5 examples (1), OSS (20), S3 (20), OCI (8), R2 (2), and shared S3 HTTP transport (2). CNAME regressions verify the final host and path, independently recompute SigV4 signatures, and cover encoded keys, Range, temporary credentials, small PUTs, multipart completion, and abort cleanup. Existing service-endpoint and credential-isolation coverage remains in place.

Before the latest upstream-main merge, a separate live test ran the actual Rust OSS adapter against an existing OSS bucket. The OSS/CNAME implementation is unchanged by the merge. Both cname: false with a service endpoint and cname: true with a fully qualified system bucket endpoint passed small-object and 6 MiB multipart uploads, HEAD, full downloads, ranged downloads, and SHA-256 readback. The four test objects were deleted and verified absent, and a scoped multipart listing confirmed no uploads remained. Validation revision 00bdeb708728b2b44ea8b3d73b87ad59ef2a07fc retained the same provider source; its additional files were the CI/live-test harness.

Live-test scope: the available bucket has no bound custom domain. This verifies real OSS transfers through both client addressing modes, but does not establish bound-CNAME compatibility, custom-domain TLS configuration, or access from an account affected by the public-endpoint policy.

Changed-file rustfmt checks passed with nightly-2026-08-26, as did diff checks, documentation snippet/anchor/link/navigation/boundary checks, and Vale 3.17.1 (zero errors and warnings for the three checked source/guide files). The store example and main config reference were regenerated from the config source/schema, and CI confirmed the store-example generator leaves a clean diff.

These are separate-runner Cargo and live-backend results. A full workspace test run and upstream Bazel/PR checks are not claimed as passed for this head.

Risk

The provider uses the OSS S3-compatible API, not the native OSS REST protocol. Automatic Alibaba RAM role discovery is not supported. CNAME addressing requires a valid bucket/domain binding and certificate; a bound custom domain has not been tested live.

The shared transfer path retains these limitations:

  • Multipart concurrency is bounded per upload, but there is no global byte budget, upload admission queue, or disk-backed replay. Assembly/retry buffering can still consume substantial memory; this PR does not resolve the existing OOM concern.
  • The SDK attempt timeout is configured, but the custom HTTP client ignores SDK connector settings, including connection timeout. Receive, body-stream, and consumer waits have no total transfer deadline.
  • Returned multipart failures attempt abort; dropping an upload future does not guarantee bounded cleanup.
  • Transport and store retries can multiply attempts, and some permanent service errors can be treated as retryable.
  • Strict Content-Range/ignored-Range/416 handling, streamed output bounds, reused-writer accounting, control-response budgets, and transfer-budget diagnostics need further work.

The shared range and concurrency fixes can change request sizes and throughput for existing S3-backed providers. Their configuration defaults are unchanged.

AI assistance

OpenAI Codex authored the adapter, CNAME support, tests, and documentation, reviewed the changes, and ran the separate CI and live Rust-adapter validation. Human review and upstream CI validation remain pending. The live test used a system bucket domain in direct-host mode; it is not presented as a test of a bound custom domain.

@vercel

vercel Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Nov 16, 2026 10:50am UTC
nativelink-aidm Ready Ready Preview Nov 16, 2026 10:50am UTC

Request Review

@modernmedici
modernmedici changed the base branch from main to migrate-to-thin-GitHub-runners November 9, 2026 08:49
@github-actions

This comment has been minimized.

@zpzjzj
zpzjzj changed the base branch from migrate-to-thin-GitHub-runners to main November 9, 2026 08:52
@palfrey

palfrey commented Nov 9, 2026

Copy link
Copy Markdown
Member

OSS uses a shared Hickory resolver/cache based on system DNS configuration. Cacheable concurrent lookups of the same name are coalesced; query attempts, lookup duration, cache size, active distinct names, and positive/negative TTL retention are bounded. Host headers, TLS SNI, and signing remain unchanged. Select dns_resolver: "system" to use the existing connector; other providers retain their existing resolver behavior. This adds hickory-resolver and tower-service as direct dependencies; both were already in Cargo.lock, with no package version changes.

I'm confused why this is part of this PR. Does OSS work without this? This is the sort of thing that should be in a separate PR, and either we swap over to this for everything if it's better, or leave them all as-is.

@zpzjzj

zpzjzj commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, that makes sense. OSS works with the existing shared HTTP connector and system DNS resolver; Hickory was an optional hardening change, not a requirement for OSS support.

I've removed the Hickory integration, resolver configuration, dependencies, and related tests from this PR, and updated the description. Any DNS/cache improvement can be proposed separately and evaluated across the shared clients.

@palfrey palfrey left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good overall, some docs changes.

  • web/apps/docs/content/docs/how-to/stores/s3-and-compatible.mdx needs changes as that doesn't refer to the OSS store yet

Comment thread nativelink-config/src/stores.rs Resolved
Describe OSS in the S3 overview and clarify public endpoint restrictions.
Keep test examples independent of environment credentials and synchronize
the generated store configuration. Remove redundant deployment caveats.
@zpzjzj

zpzjzj commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Updated the S3 and compatible overview in d6f213d to include OSS in the provider table and add an OSS section with its configuration fields, a link to the OSS guide, and endpoint restrictions. The common-options section now explicitly applies to AWS, R2, and ONTAP, not OSS.

@CLAassistant

CLAassistant commented Nov 12, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Build results for this fork PR at 65fd57d311e9:

Job Result Links
pull-request Passed Open the build · check run

Fork PRs cannot stream build events, so a trusted workflow publishes them after CI finishes and updates this comment as each job is published. Opening a build needs a Trace Machina sign-in; the check run lists the failing tests and their logs.

@zpzjzj
zpzjzj requested a review from palfrey November 13, 2026 01:23

@palfrey palfrey left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work here

@palfrey
palfrey merged commit e262950 into TraceMachina:main Nov 16, 2026
23 checks passed

This branch was successfully deployed

2 active and 1 inactive deployments
public-ci — 65fd57d3 Deployed Nov 16, 2026 by palfrey via NativeLink / Linux (PR) #745
Preview – nativelink — 65fd57d3 Deployed Nov 16, 2026 by vercel[bot]
Preview – nativelink-aidm — 65fd57d3 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants