Repository navigation
feat(store): add Alibaba Cloud OSS through the S3-compatible API - #3149
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This comment has been minimized.
This comment has been minimized.
I'm confused why this is part of this PR. Does OSS work without this? This is the sort of thing that should be in a separate PR, and either we swap over to this for everything if it's better, or leave them all as-is. |
|
Thanks, that makes sense. OSS works with the existing shared HTTP connector and system DNS resolver; Hickory was an optional hardening change, not a requirement for OSS support. I've removed the Hickory integration, resolver configuration, dependencies, and related tests from this PR, and updated the description. Any DNS/cache improvement can be proposed separately and evaluated across the shared clients. |
palfrey
left a comment
There was a problem hiding this comment.
Looking good overall, some docs changes.
- web/apps/docs/content/docs/how-to/stores/s3-and-compatible.mdx needs changes as that doesn't refer to the OSS store yet
Describe OSS in the S3 overview and clarify public endpoint restrictions. Keep test examples independent of environment credentials and synchronize the generated store configuration. Remove redundant deployment caveats.
78011bc to
d6f213d
Compare
|
Updated the S3 and compatible overview in d6f213d to include OSS in the provider table and add an OSS section with its configuration fields, a link to the OSS guide, and endpoint restrictions. The common-options section now explicitly applies to AWS, R2, and ONTAP, not OSS. |
1e05549 to
2534d18
Compare
|
Build results for this fork PR at
Fork PRs cannot stream build events, so a trusted workflow publishes them after CI finishes and updates this comment as each job is published. Opening a build needs a Trace Machina sign-in; the check run lists the failing tests and their logs. |
What and why
Add an
osscloud object provider using Alibaba Cloud's S3-compatible API and the existingS3Storetransfer implementation. Configuration requiresregion,bucket,access_key_id, andsecret_access_keyper store. Optional fields includesession_token,endpoint,cname,key_prefix,retry, andmultipart_max_concurrent_uploads(default: 2). Credentials support shell expansion, do not fall back to the AWS credential chain, and are redacted from config Debug output.The default service-endpoint mode uses virtual-hosted addressing. With
cname: true, an explicit HTTPSendpointis used directly without adding the bucket to the hostname or path. A custom endpoint resolver preserves the S3 SigV4 signing region, service name, and path-encoding settings. The domain binding, DNS, and TLS certificate must already be configured by the operator. This addresses the client's CNAME-addressing gap relevant to Alibaba Cloud's default-public-endpoint restrictions for newer OSS accounts in mainland China; it does not bypass that policy.Optional trailing checksums are disabled to avoid
aws-chunkeduploads. The adapter uses the existing shared HTTP connector and system DNS resolver; no new dependencies, TTL renewal, read proxy, or worker changes are included.Two shared S3 transfer fixes are also included: correct the inclusive range end and bound in-flight multipart requests as well as queued parts. Zero-length reads return EOF without making a request. These fixes also affect existing S3-backed providers.
How was this verified?
Latest public revision:
168ff6ac8e5ea4af2398f0aba65a4b6b23df9886.A separate CI runner passed an actual server build, workspace/all-targets Clippy with
-D warnings, and 93 focused tests using Rust 1.97.1. Validation revision5ae278885aeb752aacb945fa606bf401d1895d54had identical tracked source to this public revision, with only the additional CI harness excluded from the PR. The job checked source-tree identity before running the commands. No local compilation was used.The 93 tests cover config deserialization (40), generated JSON5 examples (1), OSS (20), S3 (20), OCI (8), R2 (2), and shared S3 HTTP transport (2). CNAME regressions verify the final host and path, independently recompute SigV4 signatures, and cover encoded keys, Range, temporary credentials, small PUTs, multipart completion, and abort cleanup. Existing service-endpoint and credential-isolation coverage remains in place.
Before the latest upstream-main merge, a separate live test ran the actual Rust OSS adapter against an existing OSS bucket. The OSS/CNAME implementation is unchanged by the merge. Both
cname: falsewith a service endpoint andcname: truewith a fully qualified system bucket endpoint passed small-object and 6 MiB multipart uploads, HEAD, full downloads, ranged downloads, and SHA-256 readback. The four test objects were deleted and verified absent, and a scoped multipart listing confirmed no uploads remained. Validation revision00bdeb708728b2b44ea8b3d73b87ad59ef2a07fcretained the same provider source; its additional files were the CI/live-test harness.Live-test scope: the available bucket has no bound custom domain. This verifies real OSS transfers through both client addressing modes, but does not establish bound-CNAME compatibility, custom-domain TLS configuration, or access from an account affected by the public-endpoint policy.
Changed-file rustfmt checks passed with
nightly-2026-08-26, as did diff checks, documentation snippet/anchor/link/navigation/boundary checks, and Vale 3.17.1 (zero errors and warnings for the three checked source/guide files). The store example and main config reference were regenerated from the config source/schema, and CI confirmed the store-example generator leaves a clean diff.These are separate-runner Cargo and live-backend results. A full workspace test run and upstream Bazel/PR checks are not claimed as passed for this head.
Risk
The provider uses the OSS S3-compatible API, not the native OSS REST protocol. Automatic Alibaba RAM role discovery is not supported. CNAME addressing requires a valid bucket/domain binding and certificate; a bound custom domain has not been tested live.
The shared transfer path retains these limitations:
The shared range and concurrency fixes can change request sizes and throughput for existing S3-backed providers. Their configuration defaults are unchanged.
AI assistance
OpenAI Codex authored the adapter, CNAME support, tests, and documentation, reviewed the changes, and ran the separate CI and live Rust-adapter validation. Human review and upstream CI validation remain pending. The live test used a system bucket domain in direct-host mode; it is not presented as a test of a bound custom domain.