docs: add Python example for native S3 range reads - #2042
Open
YQ-Wang wants to merge 1 commit into
Open
Conversation
Signed-off-by: Yiqing Wang <yiqing@wangemail.com>
YQ-Wang
marked this pull request as ready for review
September 9, 2026 05:18
YQ-Wang
requested review from
CooooolFrog,
Zephyrcf,
gaius-qi,
hhhhsdxxxx and
yyzai384
September 9, 2026 05:18
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2042 +/- ##
=======================================
Coverage 64.50% 64.50%
=======================================
Files 101 101
Lines 30203 30203
=======================================
Hits 19482 19482
Misses 10721 10721 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a seekable Python S3 reader using the existing v2 gRPC API and generated
dragonfly-apibindings. It preserves whole-object task identity, validates and trims complete pieces, and obtains AWS credentials (including STS tokens) through boto3.The example includes an optional PyAV video-window consumer, focused tests, and a path-scoped Python 3.11/3.12 CI workflow. PyAV is not required for byte-range reads. The README documents the trusted-socket and immutable-object assumptions, memory limits, and small-range scheduling behavior.
Related Issue
Related to #1821. This demonstrates native gRPC integration for callers that can be adapted; it does not resolve signed HTTP proxy compatibility or close that issue.
Motivation and Context
Python data loaders need selected object bytes in process, including seekable reads for large videos. The existing daemon API can serve this use case while keeping normal task/piece caching and P2P behavior. No daemon or protocol changes are needed.
Validation
Validated the example files in
517d43b8370a6eb571d8e3045959b7fa9865a249against upstreamd5bccb022:git diff --check.ALWAYSand fell back to source withAUTO.The live tests used MinIO, not AWS S3. Cache hits can still issue metadata HEADs, and these results are functional checks, not throughput benchmarks. Local test deployments and generated logs are excluded from this PR.
AI Assistance
AI tools assisted implementation and testing. I reviewed the code and the testing process.