Merged
Conversation
julianoes
approved these changes
Jan 12, 2026
Contributor
Author
|
I'll fix CI and push a new commit before merging |
bkueng
reviewed
Jan 13, 2026
Member
bkueng
left a comment
There was a problem hiding this comment.
Can you check CI? Otherwise looks good.
* warn users trying to download the whole database * max number of logs by default is 10 * increase delay between downloads considering rate limiting server side if you are using this script to download tons of logs please consider supporting Dronecode with a membership to offset the costs of the network bandwidth Signed-off-by: Ramon Roche <mrpollo@gmail.com>
165c4fe to
a1bacb9
Compare
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add rate limiting and safer defaults to download_logs.py
Summary
This PR updates
download_logs.pyto respect server-side rate limits and prevent accidental bulk downloads that can impact infrastructure costs.Background
Flight Review has been experiencing increased scraper activity, with bots bulk-downloading large portions of the log database. This drives up AWS costs (network egress, storage I/O) that the Dronecode Foundation pays for.
The server now has rate limiting in place (10 requests/minute per IP on the
/downloadendpoint), and this script update ensures compatibility with those limits.Changes
Safer defaults
--max-numnow defaults to 10 (was unlimited). Users who need more must explicitly request it.--yesto skip the prompt.Rate limit compliance
--delay). This keeps requests under the 10/minute server limit.New command-line options
--delay: Adjust delay between downloads (default: 6 seconds)--yes/-y: Skip confirmation prompt for large downloadsExample output
Default behavior (10 files):
Large download warning (>100 files):
IP blocked error:
Breaking changes
Users who relied on the default behavior to download all logs will now need to explicitly pass
--max-num -1 --yes. This is intentional.Testing
Tested locally on macOS against production server: