Skip to content

script: adjust download script defaults#329

Merged
mrpollo merged 2 commits intomainfrom
mrpollo/update_download_script
Jan 13, 2026
Merged

script: adjust download script defaults#329
mrpollo merged 2 commits intomainfrom
mrpollo/update_download_script

Conversation

@mrpollo
Copy link
Contributor

@mrpollo mrpollo commented Jan 12, 2026

Add rate limiting and safer defaults to download_logs.py

Summary

This PR updates download_logs.py to respect server-side rate limits and prevent accidental bulk downloads that can impact infrastructure costs.

Background

Flight Review has been experiencing increased scraper activity, with bots bulk-downloading large portions of the log database. This drives up AWS costs (network egress, storage I/O) that the Dronecode Foundation pays for.

The server now has rate limiting in place (10 requests/minute per IP on the /download endpoint), and this script update ensures compatibility with those limits.

Changes

Safer defaults

  • --max-num now defaults to 10 (was unlimited). Users who need more must explicitly request it.
  • Warning prompt for large downloads: If requesting >100 files, the script displays a warning with estimated download time and asks for confirmation. Use --yes to skip the prompt.

Rate limit compliance

  • 6-second delay between downloads by default (configurable via --delay). This keeps requests under the 10/minute server limit.
  • 503 handling: Detects rate limiting and backs off exponentially instead of hammering the server.
  • 403/444 handling: If IP is blocked, exits immediately with a clear error message instead of retrying endlessly.
  • Connection error handling: Warns user that connection failures may indicate an IP block.

New command-line options

  • --delay: Adjust delay between downloads (default: 6 seconds)
  • --yes / -y: Skip confirmation prompt for large downloads

Example output

Default behavior (10 files):

Fetching database info...
Found 330195 total public logs in database.
330195 logs match your filter criteria.
Limiting to 10 files (use --max-num to change).
Downloading 1/10 (4c4e36fb-593d-4d9a-9cf4-0716580cd233)
...

Large download warning (>100 files):

============================================================
WARNING: You are about to download 500 files.
Estimated time: 0:50:00 (at 6s between downloads)
============================================================

The server has rate limits in place. Bulk downloading without
appropriate delays may result in your IP being blocked.

Network and storage costs for Flight Review are funded by the
Dronecode Foundation. If you find this service useful, please
consider supporting the project: https://www.dronecode.org/membership/

To download more files, use: --max-num 500 --yes

Continue with download? [y/N]:

IP blocked error:

============================================================
ERROR: Your IP address has been blocked (HTTP 403).
This may be due to excessive download requests.

If you believe this is an error, please contact:
https://github.com/PX4/flight_review/issues
============================================================

Breaking changes

Users who relied on the default behavior to download all logs will now need to explicitly pass --max-num -1 --yes. This is intentional.

Testing

Tested locally on macOS against production server:

  • Default 10-file download works
  • Large download warning displays correctly
  • Rate limiting delays work as expected

@mrpollo mrpollo requested a review from bkueng January 12, 2026 18:53
@mrpollo
Copy link
Contributor Author

mrpollo commented Jan 12, 2026

I'll fix CI and push a new commit before merging

Copy link
Member

@bkueng bkueng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check CI? Otherwise looks good.

* warn users trying to download the whole database
* max number of logs by default is 10
* increase delay between downloads considering rate limiting server side

if you are using this script to download tons of logs please consider
supporting Dronecode with a membership to offset the costs of the
network bandwidth

Signed-off-by: Ramon Roche <mrpollo@gmail.com>
@mrpollo mrpollo force-pushed the mrpollo/update_download_script branch from 165c4fe to a1bacb9 Compare January 13, 2026 22:22
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
@mrpollo mrpollo merged commit 1871fad into main Jan 13, 2026
5 checks passed
@mrpollo mrpollo deleted the mrpollo/update_download_script branch January 13, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants