Skip to content

Limit the number of simultaneous batch chunk reqs #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
UmanShahzad opened this issue Sep 10, 2021 · 0 comments · Fixed by #39
Closed

Limit the number of simultaneous batch chunk reqs #34

UmanShahzad opened this issue Sep 10, 2021 · 0 comments · Fixed by #39
Assignees

Comments

@UmanShahzad
Copy link
Contributor

When we send a batch request for X amount of IPs, and X > 1000, we start chunking X into groups of 1000, and make requests for each group separately to the IPinfo batch endpoint, because the endpoint does not accept a size greater than 1000.

However, we do this all at once for all chunks: if there were 500k IPs as input, 500 simultaneous, parallel requests would be made. The issue with this is we may exceed the file descriptor limit of the user; we had this issue with e.g. the CLI.

Implement an upper-limit on the number of simultaneous, parallel requests being made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants