@@ -15,6 +15,110 @@ Dragonfly client written in Rust. It can serve as both a peer and a seed peer.
1515
1616You can find the full documentation on the [ d7y.io] ( https://d7y.io ) .
1717
18+ ## S3-aware proxy
19+
20+ ` dfdaemon ` can classify authenticated S3 API requests received through its
21+ HTTP/HTTPS proxy. Only GetObject is eligible for P2P; listings, metadata
22+ requests, writes, and unknown S3 subresources are sent directly to the origin.
23+ SigV4 headers and presigned URLs are preserved.
24+
25+ The feature is disabled by default:
26+
27+ ``` yaml
28+ proxy :
29+ server :
30+ # Required when sharing authenticated S3 responses.
31+ basicAuth :
32+ username : trusted-client
33+ password : change-me
34+ s3 :
35+ enable : true
36+ detectAwsEndpoints : true
37+ # Safe default is false. Unversioned requests also require one strong
38+ # If-Match value, which becomes part of the task identity.
39+ allowUnversionedObjects : true
40+ # Safe default is false. Enable only for callers in one trust boundary.
41+ allowSharedAuthenticatedRequests : true
42+ # Safe default is false. Anonymous S3 policies can still vary by request
43+ # context or be revoked after a task is cached.
44+ allowSharedAnonymousRequests : false
45+ # Declare nonstandard credential headers used by custom endpoints.
46+ authenticationHeaders :
47+ - X-Company-Auth
48+ # Optional S3-compatible endpoints. Exact hosts use path-style addressing;
49+ # subdomains of a suffix use virtual-hosted-style addressing.
50+ hosts :
51+ - minio.example.com
52+ hostSuffixes :
53+ - storage.example.com
54+ # Safe default is false. With no include prefixes, custom hosts remain
55+ # direct because they can expose admin APIs on the S3 listener.
56+ allowAllCustomKeys : false
57+ # Key scoping is required for custom endpoints unless all keys are
58+ # explicitly allowed. Exclusions take precedence.
59+ includeKeyPrefixes :
60+ - models/
61+ - datasets/
62+ excludeKeyPrefixes :
63+ - datasets/mutable/
64+ ` ` `
65+
66+ Clients can use the proxy without application-specific download APIs:
67+
68+ ` ` ` shell
69+ export HTTPS_PROXY=http://127.0.0.1:4001
70+ export AWS_CA_BUNDLE=/etc/ssl/certs/dragonfly-proxy-ca.pem
71+ ```
72+
73+ By default, only a non-null ` versionId ` is considered an immutable cache
74+ identity. ` allowUnversionedObjects ` must be enabled explicitly, and each
75+ unversioned request must carry one strong ` If-Match ` entity tag. That tag is
76+ included in the task identity and validated on source requests, preventing
77+ local or peer pieces from different object generations from being combined.
78+
79+ ` X-Dragonfly-Use-P2P: false ` , ` Cache-Control: no-store ` , `Cache-Control:
80+ no-cache` , and request freshness directives such as ` Cache-Control: max-age`
81+ explicitly bypass P2P. Conditional requests other than a single strong
82+ ` If-Match ` on S3 GetObject, SSE-C, requester-pays, and expected-owner requests
83+ also bypass P2P because a completed Dragonfly task cannot reproduce their
84+ origin semantics. A strong ` If-Match ` is included in task identity and validated
85+ against origin ETags; unversioned sharing requires both it and
86+ ` allowUnversionedObjects ` . Checksum-mode requests use separate whole-object or exact-range task
87+ identities; whole-object checksum metadata is preserved, while checksum headers
88+ are omitted from reconstructed partial responses unless the origin returned
89+ that exact compact range.
90+
91+ Origin responses with ` private ` , ` no-store ` , ` no-cache ` , freshness lifetimes,
92+ ` Expires ` , ` Vary ` , or ` Set-Cookie ` are sent directly instead of being retained
93+ as shared tasks. Origin errors and unsupported ranges also fall back to direct
94+ proxying so S3 SDKs receive the original status, headers, redirects, and XML
95+ error body.
96+
97+ The proxy is an acceleration layer, not an S3 authorization enforcement point.
98+ Authenticated requests bypass P2P unless ` allowSharedAuthenticatedRequests ` is
99+ enabled explicitly and proxy Basic authentication is configured. When enabled,
100+ a completed task can be served without revalidating the caller's S3
101+ credentials, so the proxy credential defines the trusted cache boundary; do
102+ not share it across mutually untrusted tenants. Origin credentials remain
103+ local to the fetching daemon and are removed from scheduler registration.
104+ Proxy Basic authentication protects only proxy ingress: every scheduler and
105+ piece-serving port participating in authenticated sharing must also be isolated
106+ to the same trust domain or protected by authenticated peer transport.
107+
108+ List custom S3 credential headers in ` authenticationHeaders ` . Configured
109+ global backend request headers are also treated as authentication for S3
110+ routing. Cross-origin backend redirects forward only a small representation
111+ header allowlist, so custom credentials cannot leak to the redirect target.
112+
113+ Anonymous requests have the same revalidation limitation: bucket policies can
114+ depend on request context and public access can later be revoked. They bypass
115+ P2P unless ` allowSharedAnonymousRequests ` is enabled for objects intended to
116+ remain readable by every proxy caller for the cached task's lifetime.
117+
118+ HTTPS origins are verified against the host's native trust store. Install the
119+ CA for private S3-compatible endpoints (such as an internal MinIO deployment)
120+ in that trust store before enabling P2P.
121+
18122## Community
19123
20124Join the conversation and help the community grow. Here are the ways to get involved:
0 commit comments