Skip to content

sources score treats a Semantic Scholar 429 as a permanent failure: arXiv sources silently score as unenriched #70

Description

@fduple

_http_get_json in core/scholar.py returns None for any non-200, so a rate-limit response is indistinguishable from "paper not found":

if resp.status_code != 200:
    return None

That matters more than it looks, because lookup_metadata routes arXiv ids exclusively to Semantic Scholar (OpenAlex has no arXiv id scheme, so it is DOI-only):

if doi.lower().startswith("arxiv:"):
    ...
    "https://api.semanticscholar.org/graph/v1/paper/arXiv:"

Unauthenticated Semantic Scholar shares one global request pool across all anonymous callers, and it returns 429 readily. The 1.1s per-host courtesy delay in _HOST_DELAY paces our own loop but does nothing about the shared pool. When that happens, every arXiv note in the pass gets None, sources score reports them as unenriched, and nothing in the output distinguishes "rate limited, try again" from "no such paper". On an arXiv-heavy vault the result is a scoring pass that quietly produces zero citation data, which then propagates into source ranking and the quality gates that consume it.

There is also no way to raise the ceiling: the request sends only a User-Agent, so a user holding a Semantic Scholar API key cannot use it.

Reproduction:

  1. Point api.semanticscholar.org at anything returning 429 (a local stub server is enough), or run hpr sources score against a vault of arXiv sources while the anonymous pool is saturated.
  2. Every arXiv note comes back unenriched, with no error and no retry.

Suggested fix (what we ship in our fork): retry only on 429, up to 3 attempts with a short bounded backoff (2s, then 4s), leaving every other failure soft exactly as it is today. Separately, read S2_API_KEY / SEMANTIC_SCHOLAR_API_KEY and send it as x-api-key only when the request host is semanticscholar.org, so the key cannot leak to OpenAlex or Crossref. We verified live that scoring converges across passes once the retry is in place, since successes cache in api_cache. Regression tests cover the 429-then-200 sequence and the host-scoping of the key. Happy to send the PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions