Skip to content

Commit cc1524c

Browse files
zkochanclaude
andauthored
ci: build every locale separately (#889)
* ci: build every locale separately Docusaurus builds locales one after another in a single process, so the 13 locales of this site cost 13 sequential builds in one Vercel builder: 4 minutes on a 32-core machine, considerably more on a build container. Build each locale in its own CI job instead. Every job downloads only its own language from Crowdin and runs `docusaurus build --locale <locale>`; the resulting trees are stitched back together and handed to Vercel with `vercel deploy --prebuilt`, so the site is still one atomic deployment. The wall clock is now the slowest single locale rather than the sum of all of them. A build narrowed down to one locale drops the `/<locale>/` base URL segment, which suits multi-domain deployments but not this site, so the segment is pinned in the config. The assembled output was compared against a full `pnpm build`: same 12116 files, and all 4303 pages identical. Vercel's git integration is turned off, since building from CI and building from a push would otherwise deploy the site twice. Pull requests keep a preview deployment, English-only to stay fast. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: address the review comments CodeQL asks every workflow to limit what the GITHUB_TOKEN can do; the deploy workflow only reads the repository. The preview deployment is also skipped when no Vercel credentials are configured, instead of failing the build test that ran before it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: pin the Vercel CLI to an exact version `vercel@59` picks whatever 59.x is current when the job runs, in the one job that holds the deploy token. An exact version is immutable on the registry. Also spell out in the README that pull requests from forks get the build test but no preview, since their token cannot read the Vercel credentials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: run the Vercel CLI through pnpm `pnpm dlx` keeps the exact version pin without installing anything globally. Its esbuild dependency needs its install script, which pnpm blocks by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: keep the Vercel token out of the build step The credentials moved to the job environment so that the preview step could skip itself when they are missing, but that also handed them to the install and build steps, which run the code of the pull request being tested. Detect them in a step of their own instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: drop the preview deployment from pull requests Deploying from a pull request means running the code under review with the credentials of the live site: a branch could rewrite the build command in vercel.json and read the token out of the environment. Branch-push access is not deploy access, so pull requests get the build test alone. `pnpm dlx` no longer runs anywhere on the pull request path, and the assembly script now only accepts the artifacts of the deploy workflow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent b727d3a commit cc1524c

9 files changed

Lines changed: 298 additions & 13 deletions

File tree

.github/workflows/ci.yml

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
11
name: Build Test
22

33
on: [pull_request]
4+
5+
permissions:
6+
contents: read
7+
8+
env:
9+
COREPACK_ENABLE_AUTO_PIN: 0
10+
411
jobs:
512
deploy:
613
runs-on: ubuntu-latest
714
environment: deploy
8-
env:
9-
COREPACK_ENABLE_AUTO_PIN: 0
1015
steps:
1116
- name: Checkout Commit
1217
uses: actions/checkout@v4
@@ -16,9 +21,16 @@ jobs:
1621
uses: pnpm/action-setup@v4.1.0
1722
with:
1823
standalone: true
24+
- name: Setup Node
25+
uses: actions/setup-node@v4
26+
with:
27+
node-version: 22
28+
cache: pnpm
1929
- name: Install dependencies
2030
run: pnpm install
2131
- name: Build
32+
# English only: the translations are not part of a pull request, and
33+
# building them would cost as much as a deploy.
2234
run: pnpm build
2335
env:
2436
LOCALE_CI: en

.github/workflows/deploy.yml

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
name: Deploy
2+
3+
# The site used to be built by Vercel's git integration, in a single build that
4+
# walked the 13 locales one after another — Docusaurus has no parallel mode.
5+
# Here every locale is built by its own job instead, and only the finished
6+
# static files are handed to Vercel (`vercel deploy --prebuilt`), so the wall
7+
# clock is the slowest single locale rather than the sum of all of them.
8+
9+
on:
10+
push:
11+
branches: [main]
12+
workflow_dispatch:
13+
inputs:
14+
production:
15+
description: Publish on pnpm.io. Turn this off to get a preview URL instead.
16+
type: boolean
17+
default: true
18+
19+
# A deploy always ships the whole site, so an in-flight one is pointless as
20+
# soon as a newer commit lands.
21+
concurrency:
22+
group: deploy
23+
cancel-in-progress: true
24+
25+
# Nothing here writes to the repository; the deploy goes out through Vercel.
26+
permissions:
27+
contents: read
28+
29+
env:
30+
COREPACK_ENABLE_AUTO_PIN: 0
31+
32+
jobs:
33+
locales:
34+
runs-on: ubuntu-latest
35+
outputs:
36+
locales: ${{ steps.read.outputs.locales }}
37+
steps:
38+
- uses: actions/checkout@v4
39+
- id: read
40+
run: echo "locales=$(jq -c '[.[].locale]' locales.json)" >> "$GITHUB_OUTPUT"
41+
42+
build:
43+
needs: locales
44+
runs-on: ubuntu-latest
45+
environment: deploy
46+
strategy:
47+
# One broken locale shouldn't hide the state of the other twelve; the
48+
# deploy job below refuses to ship an incomplete site anyway.
49+
fail-fast: false
50+
matrix:
51+
locale: ${{ fromJSON(needs.locales.outputs.locales) }}
52+
name: build (${{ matrix.locale }})
53+
steps:
54+
- uses: actions/checkout@v4
55+
with:
56+
# `showLastUpdateTime` reads the dates from the git history.
57+
fetch-depth: 0
58+
- uses: pnpm/action-setup@v4.1.0
59+
with:
60+
standalone: true
61+
- uses: actions/setup-node@v4
62+
with:
63+
node-version: 22
64+
cache: pnpm
65+
- run: pnpm install
66+
- name: Copy the current docs into the latest version
67+
# Crowdin maps the translations of the latest docs version onto this
68+
# copy, so it has to exist before the download below.
69+
run: pnpm copy-docs
70+
- name: Download the translations
71+
if: matrix.locale != 'en'
72+
run: |
73+
language=$(jq -r --arg locale "$LOCALE" '.[] | select(.locale == $locale) | .crowdinLanguage // empty' locales.json)
74+
test -n "$language" # locales.json must name the language for Crowdin
75+
pnpm exec crowdin download --language "$language" --no-progress --no-colors
76+
env:
77+
LOCALE: ${{ matrix.locale }}
78+
CROWDIN_PROJECT_ID: ${{ vars.CROWDIN_PROJECT_ID || '302994' }}
79+
CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}
80+
- name: Build
81+
run: node scripts/build-with-fallback.mjs --locale "$LOCALE"
82+
env:
83+
LOCALE: ${{ matrix.locale }}
84+
- uses: actions/upload-artifact@v4
85+
with:
86+
name: site-${{ matrix.locale }}
87+
path: build
88+
retention-days: 1
89+
# The files are already minified; compressing them again costs more
90+
# time than it saves on the transfer.
91+
compression-level: 0
92+
93+
deploy:
94+
needs: build
95+
runs-on: ubuntu-latest
96+
environment: deploy
97+
steps:
98+
- uses: actions/checkout@v4
99+
- uses: pnpm/action-setup@v4.1.0
100+
with:
101+
standalone: true
102+
- uses: actions/setup-node@v4
103+
with:
104+
node-version: 22
105+
- name: Collect every locale
106+
uses: actions/download-artifact@v4
107+
with:
108+
pattern: site-*
109+
# Each locale occupies its own subtree, so they merge into the single
110+
# directory the site is served from.
111+
merge-multiple: true
112+
path: .site-artifacts
113+
- name: Deploy to Vercel
114+
# `vercel build` doesn't rebuild the site: vercel.json points its build
115+
# command at scripts/assemble-site.mjs, which just moves the artifacts
116+
# into place. It is still needed to turn vercel.json into a deployment.
117+
run: |
118+
# esbuild, which the CLI depends on, has to run its install script.
119+
vercel () { pnpm dlx --allow-build=esbuild vercel@59.1.4 "$@"; }
120+
if [ "$PRODUCTION" = 'true' ]; then
121+
environment=production
122+
prod=--prod
123+
else
124+
environment=preview
125+
prod=
126+
fi
127+
vercel pull --yes --environment="$environment" --token="$VERCEL_TOKEN"
128+
vercel build $prod --token="$VERCEL_TOKEN"
129+
url=$(vercel deploy --prebuilt $prod --token="$VERCEL_TOKEN" | tail -n1)
130+
echo "Deployed $url" >> "$GITHUB_STEP_SUMMARY"
131+
env:
132+
PRODUCTION: ${{ github.event_name != 'workflow_dispatch' || inputs.production }}
133+
VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
134+
VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
135+
VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}

.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,9 @@ versioned_docs/version-11.x
3131

3232
# local Claude Code settings
3333
.claude/settings.local.json
34+
35+
# assembled by scripts/assemble-site.mjs when deploying
36+
.site-artifacts
37+
38+
# Vercel CLI state, pulled during deploys
39+
.vercel

README.md

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,28 @@ pnpm start
1111

1212
## How to publish
1313

14-
Push to the default branch, the website will be deployed automatically.
14+
Push to the default branch, the website will be deployed automatically by the
15+
[Deploy workflow](.github/workflows/deploy.yml).
16+
17+
Docusaurus builds one locale after another, and this site has 13 of them, so the
18+
workflow builds each locale in its own job instead: every job downloads only its
19+
own translations from Crowdin and runs `docusaurus build --locale <locale>`. The
20+
resulting trees are stitched back together by `scripts/assemble-site.mjs` and
21+
shipped to Vercel with `vercel deploy --prebuilt`, as a single deployment.
22+
23+
Because of that, Vercel's own git integration is turned off (see
24+
`git.deploymentEnabled` in [vercel.json](vercel.json)) and the workflow needs
25+
these secrets in the `deploy` environment: `VERCEL_TOKEN`, `VERCEL_ORG_ID`,
26+
`VERCEL_PROJECT_ID`, and `CROWDIN_PERSONAL_TOKEN`.
27+
28+
Pull requests get an English-only build test rather than a preview
29+
deployment: deploying from a pull request would mean handing the credentials
30+
of the live site to the code being reviewed. To see a change served, start the
31+
Deploy workflow by hand with the "Publish on pnpm.io" box unticked, which
32+
returns a preview URL instead of publishing.
33+
34+
The locales are listed in [locales.json](locales.json), together with the name
35+
Crowdin uses for each of them. Adding a locale there adds a build job for it.
1536

1637
## Algolia Search
1738

docusaurus.config.ts

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import path from 'node:path';
22
import { themes } from 'prism-react-renderer';
33
import progress from "./scripts/progress_lang.json" with { type: "json" };
4+
import locales from "./locales.json" with { type: "json" };
45
import type { Config } from '@docusaurus/types';
56
import type * as Preset from '@docusaurus/preset-classic';
67

@@ -11,10 +12,29 @@ const TRANSLATE_URL = "https://translate.pnpm.io";
1112
const CRYPTO_DONATIONS_HREF = '/crypto-donations';
1213
const LOCALE_CI = process.env.LOCALE_CI;
1314
const DEFAULT_LOCALE = 'en';
14-
const LOCALE_FULL_CODE: Record<string, string> = {
15-
zh: 'zh-CN',
16-
pt: 'pt-BR',
17-
es: 'es-ES',
15+
// The locales live in locales.json because the deploy workflow builds one
16+
// locale per job and needs to read the same list.
17+
const LOCALES_CONFIG: { locale: string, crowdinLanguage?: string }[] = locales;
18+
const LOCALES = LOCALES_CONFIG.map(({ locale }) => locale);
19+
// Crowdin names some languages differently from Docusaurus (`zh-CN` vs `zh`).
20+
const LOCALE_FULL_CODE: Record<string, string> = Object.fromEntries(
21+
LOCALES_CONFIG.flatMap(({ locale, crowdinLanguage }) =>
22+
crowdinLanguage ? [[locale, crowdinLanguage]] : [])
23+
);
24+
25+
// Docusaurus infers `/<locale>/` as the base URL of a localized site, but only
26+
// when a build covers several locales at once. A build narrowed down to one
27+
// locale with `--locale` drops the segment, which suits multi-domain
28+
// deployments but not this site: every locale is built by its own CI job and
29+
// the results are stitched back together under one domain. Pinning the base
30+
// URL keeps both build shapes identical.
31+
function withLocaleBaseUrls<T extends Record<string, object>> (localeConfigs: T): T {
32+
return Object.fromEntries(
33+
Object.entries(localeConfigs).map(([locale, localeConfig]) => [
34+
locale,
35+
locale === DEFAULT_LOCALE ? localeConfig : { baseUrl: `/${locale}/`, ...localeConfig },
36+
])
37+
) as T;
1838
}
1939

2040
const PROJECT_NAME = 'pnpm.io'
@@ -336,8 +356,8 @@ const docusaurusConfig = {
336356
} satisfies Preset.ThemeConfig,
337357
i18n: {
338358
defaultLocale: DEFAULT_LOCALE,
339-
locales: LOCALE_CI ? [LOCALE_CI] : ['en', 'it', 'zh', 'ja', 'ko', 'pt', 'zh-TW', 'ru', 'uk', 'fr', 'tr', 'es', 'id'],
340-
localeConfigs: {
359+
locales: LOCALE_CI ? [LOCALE_CI] : LOCALES,
360+
localeConfigs: withLocaleBaseUrls({
341361
en: { label: "English" },
342362
it: { label: `Italiano (${progress["it"].translationProgress}%)` },
343363
zh: { label: `简体中文 (${progress["zh-CN"].translationProgress}%)` },
@@ -360,7 +380,7 @@ const docusaurusConfig = {
360380
// hu: { label: `Magyar (${progress["hu"].translationProgress}%)` },
361381
// pl: { label: `Polski (${progress["pl"].translationProgress}%)` },
362382
// de: { label: `Deutsch (${progress["de"].translationProgress}%)` },
363-
},
383+
}),
364384
},
365385
} satisfies Config;
366386

locales.json

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[
2+
{ "locale": "en" },
3+
{ "locale": "it", "crowdinLanguage": "it" },
4+
{ "locale": "zh", "crowdinLanguage": "zh-CN" },
5+
{ "locale": "ja", "crowdinLanguage": "ja" },
6+
{ "locale": "ko", "crowdinLanguage": "ko" },
7+
{ "locale": "pt", "crowdinLanguage": "pt-BR" },
8+
{ "locale": "zh-TW", "crowdinLanguage": "zh-TW" },
9+
{ "locale": "ru", "crowdinLanguage": "ru" },
10+
{ "locale": "uk", "crowdinLanguage": "uk" },
11+
{ "locale": "fr", "crowdinLanguage": "fr" },
12+
{ "locale": "tr", "crowdinLanguage": "tr" },
13+
{ "locale": "es", "crowdinLanguage": "es-ES" },
14+
{ "locale": "id", "crowdinLanguage": "id" }
15+
]

scripts/assemble-site.mjs

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
import { cpSync, existsSync, readdirSync, readFileSync, renameSync, rmSync } from 'node:fs'
2+
import path from 'node:path'
3+
4+
// Vercel builds this site by running this script (see `buildCommand` in
5+
// vercel.json) instead of building Docusaurus, because the build itself
6+
// happens in CI: .github/workflows/deploy.yml builds every locale in its own
7+
// job — the default locale at the root of the tree, the other ones under their
8+
// own `/<locale>/` segment — and downloads all of those artifacts into
9+
// ARTIFACTS_DIR. All that is left here is to move them where Vercel looks for
10+
// them, so that `vercel build` still turns vercel.json into a deployment.
11+
12+
const ARTIFACTS_DIR = path.resolve('.site-artifacts')
13+
const OUT_DIR = path.resolve('build')
14+
15+
function isPopulated (dir) {
16+
try {
17+
return readdirSync(dir).length > 0
18+
} catch {
19+
return false
20+
}
21+
}
22+
23+
function fail (message) {
24+
console.error(message)
25+
process.exit(1)
26+
}
27+
28+
if (isPopulated(ARTIFACTS_DIR)) {
29+
console.log(`Assembling the site from ${path.relative(process.cwd(), ARTIFACTS_DIR)}`)
30+
rmSync(OUT_DIR, { recursive: true, force: true })
31+
try {
32+
renameSync(ARTIFACTS_DIR, OUT_DIR)
33+
} catch (err) {
34+
// The artifacts may sit on a different filesystem than the checkout.
35+
if (err.code !== 'EXDEV') throw err
36+
cpSync(ARTIFACTS_DIR, OUT_DIR, { recursive: true })
37+
rmSync(ARTIFACTS_DIR, { recursive: true, force: true })
38+
}
39+
// A locale whose build job was skipped or whose artifact failed to download
40+
// would silently disappear from the site, so make sure they all arrived.
41+
const [defaultLocale, ...locales] = JSON.parse(readFileSync('locales.json', 'utf-8'))
42+
.map(({ locale }) => locale)
43+
const missing = locales.filter(locale => !existsSync(path.join(OUT_DIR, locale, 'index.html')))
44+
if (missing.length > 0) {
45+
fail(`These locales are missing from the assembled site: ${missing.join(', ')}.`)
46+
}
47+
console.log(`Assembled ${defaultLocale} and ${locales.length} translated locales.`)
48+
} else {
49+
fail(`Nothing to deploy: ${path.relative(process.cwd(), ARTIFACTS_DIR)} is empty.
50+
51+
The site is built by the "Deploy" GitHub Actions workflow, one job per locale,
52+
and shipped from there with \`vercel deploy --prebuilt\`. Re-run that workflow
53+
instead of building from the Vercel dashboard. To build the whole site locally,
54+
run \`pnpm build\`.`)
55+
}
56+
57+
if (!existsSync(path.join(OUT_DIR, 'index.html'))) {
58+
fail(`${path.relative(process.cwd(), OUT_DIR)} has no index.html: the build of the default locale is missing.`)
59+
}

scripts/build-with-fallback.mjs

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,15 @@ import path from 'node:path'
55
const MAX_RETRIES = 50
66
const I18N_DIR = path.resolve('i18n')
77

8+
// Extra arguments are handed to `docusaurus build` untouched, so that CI can
9+
// build one locale per job with `--locale <locale>`.
10+
const BUILD_ARGS = process.argv.slice(2)
11+
// When a single locale is built, only that locale's translations can be at
12+
// fault, so the scan for broken files below is narrowed down to it.
13+
const BUILT_LOCALES = BUILD_ARGS.flatMap((arg, i) =>
14+
arg === '--locale' || arg === '-l' ? [BUILD_ARGS[i + 1]] : []
15+
).filter(Boolean)
16+
817
function extractBrokenI18nFile (output) {
918
// Match file paths inside the i18n directory from the build error output.
1019
// Docusaurus / MDX / webpack errors typically include the full file path.
@@ -30,7 +39,8 @@ function extractBrokenI18nFile (output) {
3039
function extractMismatchedI18nIdFile (output) {
3140
if (!output.includes('Invalid sidebar file')) return null
3241
if (!existsSync(I18N_DIR)) return null
33-
for (const locale of safeReaddir(I18N_DIR)) {
42+
const locales = BUILT_LOCALES.length > 0 ? BUILT_LOCALES : safeReaddir(I18N_DIR)
43+
for (const locale of locales) {
3444
const localeDocsRoot = path.join(I18N_DIR, locale, 'docusaurus-plugin-content-docs')
3545
if (!existsSync(localeDocsRoot)) continue
3646
for (const versionDir of safeReaddir(localeDocsRoot)) {
@@ -103,9 +113,9 @@ function escapeRegExp (string) {
103113
let removedFiles = []
104114

105115
for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {
106-
console.log(`\n🔨 Build attempt ${attempt}${removedFiles.length > 0 ? ` (${removedFiles.length} broken translation(s) removed so far)` : ''}...\n`)
116+
console.log(`\n🔨 Build attempt ${attempt}${BUILT_LOCALES.length > 0 ? ` for ${BUILT_LOCALES.join(', ')}` : ''}${removedFiles.length > 0 ? ` (${removedFiles.length} broken translation(s) removed so far)` : ''}...\n`)
107117
try {
108-
execSync('docusaurus build', {
118+
execSync(['docusaurus', 'build', ...BUILD_ARGS].join(' '), {
109119
stdio: ['inherit', 'inherit', 'pipe'],
110120
encoding: 'utf-8',
111121
maxBuffer: 50 * 1024 * 1024,

0 commit comments

Comments
 (0)