Skip to content

fix(sync): authenticate doc fetches and back off on rate limits#27

Merged
SPIKESPIGEL404 merged 1 commit into
mainfrom
fix/docs-sync-rate-limit
Jul 7, 2026
Merged

fix(sync): authenticate doc fetches and back off on rate limits#27
SPIKESPIGEL404 merged 1 commit into
mainfrom
fix/docs-sync-rate-limit

Conversation

@SPIKESPIGEL404

Copy link
Copy Markdown
Contributor

Problem

The docs build (pnpm buildprebuildsync-docs.mjs) fails intermittently with:

sync-docs: fetching bitrouter/bitrouter@main:docs via GitHub
Error: raw 429 for docs/concepts/mcp.zh.md

Two compounding causes:

  1. No auth in CIci.yml runs pnpm build with no GITHUB_TOKEN in env, so every fetch is unauthenticated.
  2. Fragile transport — the script pulls each doc file individually from raw.githubusercontent.com, whose rate limit is per-IP and trips fast on shared Actions runners. Unauthenticated × N raw requests → 429.

This is pre-existing fragility; the recent get-started restructure merge (more files + a docs-changed dispatch) just made it trip reliably.

Fix

  • Read blobs via the authenticated git blob API (n.url from the tree response) with Accept: application/vnd.github.raw, instead of raw.githubusercontent.com. The API honors GITHUB_TOKEN for a 5000/hr budget and isn't subject to raw's per-IP throttle.
  • Add ghFetch() with backoff on 429 (and 403 with the rate-limit budget exhausted), honoring Retry-After / X-RateLimit-Reset. Applied to both the tree call and each blob read.
  • Pass GITHUB_TOKEN to pnpm build in CI. The automatic secrets.GITHUB_TOKEN can read the public bitrouter repo; sync-changelog.mjs (which also fetches) benefits too.

Verification

Ran the rewritten sync-docs.mjs against bitrouter@main with a token:

sync-docs: wrote 111 file(s) into content/docs

Exit 0, no 429s. (It also emits the usual non-blocking "N stale translation(s)" warning, unchanged behavior.)

🤖 Generated with Claude Code

The docs build fetched every source file individually from
raw.githubusercontent.com, unauthenticated (CI passed no token). On
shared Actions runners that endpoint's per-IP limit trips fast, failing
prebuild with `raw 429 for docs/concepts/mcp.zh.md`.

- Read blobs via the authenticated git blob API (n.url) instead of
  raw.*, so GITHUB_TOKEN buys a 5000/hr budget with no per-IP throttle.
- Add ghFetch() with backoff on 429 / 403-exhausted, honoring
  Retry-After and X-RateLimit-Reset.
- Pass GITHUB_TOKEN to `pnpm build` in CI (the automatic token can read
  the public bitrouter repo; sync-changelog benefits too).

Verified end to end against bitrouter@main: 111 files, no 429.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@SPIKESPIGEL404
SPIKESPIGEL404 merged commit 0ae2822 into main Jul 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant