_http_get_json in core/scholar.py returns None for any non-200, so a rate-limit response is indistinguishable from "paper not found":
if resp.status_code != 200:
return None
That matters more than it looks, because lookup_metadata routes arXiv ids exclusively to Semantic Scholar (OpenAlex has no arXiv id scheme, so it is DOI-only):
if doi.lower().startswith("arxiv:"):
...
"https://api.semanticscholar.org/graph/v1/paper/arXiv:"
Unauthenticated Semantic Scholar shares one global request pool across all anonymous callers, and it returns 429 readily. The 1.1s per-host courtesy delay in _HOST_DELAY paces our own loop but does nothing about the shared pool. When that happens, every arXiv note in the pass gets None, sources score reports them as unenriched, and nothing in the output distinguishes "rate limited, try again" from "no such paper". On an arXiv-heavy vault the result is a scoring pass that quietly produces zero citation data, which then propagates into source ranking and the quality gates that consume it.
There is also no way to raise the ceiling: the request sends only a User-Agent, so a user holding a Semantic Scholar API key cannot use it.
Reproduction:
- Point
api.semanticscholar.org at anything returning 429 (a local stub server is enough), or run hpr sources score against a vault of arXiv sources while the anonymous pool is saturated.
- Every arXiv note comes back unenriched, with no error and no retry.
Suggested fix (what we ship in our fork): retry only on 429, up to 3 attempts with a short bounded backoff (2s, then 4s), leaving every other failure soft exactly as it is today. Separately, read S2_API_KEY / SEMANTIC_SCHOLAR_API_KEY and send it as x-api-key only when the request host is semanticscholar.org, so the key cannot leak to OpenAlex or Crossref. We verified live that scoring converges across passes once the retry is in place, since successes cache in api_cache. Regression tests cover the 429-then-200 sequence and the host-scoping of the key. Happy to send the PR.
_http_get_jsonincore/scholar.pyreturnsNonefor any non-200, so a rate-limit response is indistinguishable from "paper not found":That matters more than it looks, because
lookup_metadataroutes arXiv ids exclusively to Semantic Scholar (OpenAlex has no arXiv id scheme, so it is DOI-only):Unauthenticated Semantic Scholar shares one global request pool across all anonymous callers, and it returns 429 readily. The 1.1s per-host courtesy delay in
_HOST_DELAYpaces our own loop but does nothing about the shared pool. When that happens, every arXiv note in the pass getsNone,sources scorereports them as unenriched, and nothing in the output distinguishes "rate limited, try again" from "no such paper". On an arXiv-heavy vault the result is a scoring pass that quietly produces zero citation data, which then propagates into source ranking and the quality gates that consume it.There is also no way to raise the ceiling: the request sends only a User-Agent, so a user holding a Semantic Scholar API key cannot use it.
Reproduction:
api.semanticscholar.orgat anything returning 429 (a local stub server is enough), or runhpr sources scoreagainst a vault of arXiv sources while the anonymous pool is saturated.Suggested fix (what we ship in our fork): retry only on 429, up to 3 attempts with a short bounded backoff (2s, then 4s), leaving every other failure soft exactly as it is today. Separately, read
S2_API_KEY/SEMANTIC_SCHOLAR_API_KEYand send it asx-api-keyonly when the request host is semanticscholar.org, so the key cannot leak to OpenAlex or Crossref. We verified live that scoring converges across passes once the retry is in place, since successes cache inapi_cache. Regression tests cover the 429-then-200 sequence and the host-scoping of the key. Happy to send the PR.