Skip to content

docs: add Python example for native S3 range reads - #2042

Open
YQ-Wang wants to merge 1 commit into
dragonflyoss:mainfrom
YQ-Wang:feat/python-s3-range-example
Open

docs: add Python example for native S3 range reads#2042
YQ-Wang wants to merge 1 commit into
dragonflyoss:mainfrom
YQ-Wang:feat/python-s3-range-example

Conversation

@YQ-Wang

@YQ-Wang YQ-Wang commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Description

Add a seekable Python S3 reader using the existing v2 gRPC API and generated dragonfly-api bindings. It preserves whole-object task identity, validates and trims complete pieces, and obtains AWS credentials (including STS tokens) through boto3.

The example includes an optional PyAV video-window consumer, focused tests, and a path-scoped Python 3.11/3.12 CI workflow. PyAV is not required for byte-range reads. The README documents the trusted-socket and immutable-object assumptions, memory limits, and small-range scheduling behavior.

Related Issue

Related to #1821. This demonstrates native gRPC integration for callers that can be adapted; it does not resolve signed HTTP proxy compatibility or close that issue.

Motivation and Context

Python data loaders need selected object bytes in process, including seekable reads for large videos. The existing daemon API can serve this use case while keeping normal task/piece caching and P2P behavior. No daemon or protocol changes are needed.

Validation

Validated the example files in 517d43b8370a6eb571d8e3045959b7fa9865a249 against upstream d5bccb022:

  • Python 3.11 and 3.12 in Linux Docker: all 13 tests passed, including real protobuf/gRPC streaming and video decoding. A clean Python 3.11 install without PyAV passed the 12 core tests and skipped the optional video test.
  • Ruff syntax/import checks and formatting; git diff --check.
  • Real integration with two official dfdaemon v1.5.5 containers, separate caches, manager/scheduler, and MinIO: signed piece GETs returned 206; changing Range after signing in the control request returned 403.
  • Repeated ranges fetched 8 MiB from the remote peer without a source GET; overlapping ranges retained the same task ID and hit the daemon's local cache. Small reads used remote peers with ALWAYS and fell back to source with AUTO.
  • Two MinIO STS sessions succeeded; invalid credentials on cold objects failed. URL-encoded keys and empty objects passed.
  • Two one-minute windows at the two-hour mark of a synthetic three-hour indexed MP4 matched sequential local decoding. The reader transferred 9,923,151 of 39,283,279 object bytes in three RPCs; a fresh reader repeated the windows entirely from daemon cache with no new source GET.

The live tests used MinIO, not AWS S3. Cache hits can still issue metadata HEADs, and these results are functional checks, not throughput benchmarks. Local test deployments and generated logs are excluded from this PR.

AI Assistance

AI tools assisted implementation and testing. I reviewed the code and the testing process.

Signed-off-by: Yiqing Wang <yiqing@wangemail.com>
@YQ-Wang
YQ-Wang marked this pull request as ready for review September 9, 2026 05:18
@YQ-Wang
YQ-Wang requested review from a team as code owners September 9, 2026 05:18
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.50%. Comparing base (d5bccb0) to head (517d43b).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2042   +/-   ##
=======================================
  Coverage   64.50%   64.50%           
=======================================
  Files         101      101           
  Lines       30203    30203           
=======================================
  Hits        19482    19482           
  Misses      10721    10721           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant