Skip to content

WIP: ~35x faster agent inner-loop build (3m30s → ~6s); ~5x faster cold buildAll - #5517

Draft
inodb wants to merge 6 commits into
cBioPortal:masterfrom
inodb:cbio-ci-speedup
Draft

inodb wants to merge 6 commits into
cBioPortal:masterfrom
inodb:cbio-ci-speedup

Conversation

@inodb

@inodb inodb commented Apr 14, 2026

Copy link
Copy Markdown
Member

Context

Experimental branch to speed up the webpack build so coding agents get a near-instant inner feedback loop. Not for review/merge yet — pushed as a PR purely to exercise CircleCI.

Measured impact

Build scenario Before After Speedup
Agent inner loop (yarn run buildQuick warm) ~3m30s ~6s ~35x
Full cold buildAll with warm caches 6m46s ~90s ~5x
CI build_frontend buildAll step (cache cold, first run) ~390s 236s 1.7x
CI build_frontend buildAll step (cache hit, second run) ~390s 72s 5.4x

First CI run on a new branch is slightly slower overall than master p50 because the new webpack/DLL filesystem caches have to populate. From the second run onward the v1-webpack-cache-<branch>- fallback restores the cache and the buildAll step drops to ~72s.

Changes

Always-on

  • webpack 5 filesystem cache at .webpack_cache/ (main) + .webpack_cache_dll/ (vendor DLL) with config-file invalidation. Warm main webpack rebuild: ~3m30s → ~5s.
  • TerserPlugin({parallel: true}) — was hardcoded false.
  • lerna run build --concurrency=4 — parallelize the 10 workspace packages (respects the dep graph; level-0 packages now build in parallel).
  • scripts/build-packages-if-stale.js — skip Rollup rebuild of packages whose dist/ is up-to-date vs src/. ~40ms when all fresh; builds only the stale package(s) otherwise.
  • yarn run buildQuick — wraps webpack with the env vars needed to skip clean/DLL rebuild and hit the cache. The agent inner-loop command.
  • CircleCI cache — restore/save .webpack_cache + .webpack_cache_dll with branch-prefix fallback keys so every CI run gets the closest available cache (branch-specific revision, then branch prefix, then master).
  • AGENTS.md / CLAUDE.md — symlinks into .github/copilot-instructions.md where a new "Fast build loop" section documents how to avoid yarn run buildAll for agent iteration.

Fast transpilers — opt-in, not default

Decided to keep these behind env-var toggles rather than flipping the default, so the produced bundle is byte-identical to the current Babel+ts-loader output unless explicitly requested.

  • USE_SWC=true → swc-loader. Cold main webpack 3m30s → 1m28s (2.4x). Handles this codebase cleanly (MobX decorators, useDefineForClassFields, JSX — all verified). Good candidate for flipping the default after a round of real-world validation.
  • USE_ESBUILD=true → esbuild-loader. Faster than swc but its TSX parser mishandles chained < / > comparisons (hits a parse error in QuerySummary.tsx among likely others). Left in as an experiment; would need source-level rewrites to adopt.

CI validation

All functional jobs green on rev 434b8f3:

  • build_frontend, check_typescript, unit_tests_main, unit_tests_packages, api_sync_cbioportal, api_sync_dependencies, prettier, forgotten_spec / incorrect_import checks

E2e baseline (flaky on master):

Job Total Failed Pass rate
e2e_tests 580 10 98.3%
e2e_localdb_tests 346 4 98.8%

Master's recent e2e_tests run had 5 failures / 575 (99.1%). This branch's failures are all screenshot drift (comparison tab venn diagrams, alteration enrichments), and 4 of the 10 are identical to master failures (TMB H biomarker annotation, session mutation tab, no session mutation tab) — pre-existing flakes unrelated to the build changes. None of the always-on optimizations change the actual bundle output.

inodb and others added 2 commits April 14, 2026 03:10
- Enable TerserPlugin parallel minification (was disabled)
- Add webpack 5 filesystem cache at .webpack_cache/ with config-file
  invalidation — cuts warm webpack rebuild from ~3m30s to ~9s
- Parallelize lerna buildModules with --concurrency=4 (respects the
  workspace dep graph; 4 level-0 packages now build in parallel)
- Document the fast build loop in copilot-instructions.md and add
  AGENTS.md / CLAUDE.md symlinks so all agent conventions share it

Measured: cold buildAll 6m46s (was ~8m), warm `npx webpack` 14s total
(9s webpack). Use `yarn start` for HMR or `npx webpack` for production
artifacts — not `yarn run buildAll` which runs clean + DLL rebuild.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
USE_SWC=true swaps babel-loader + ts-loader for swc-loader. Cuts cold
main webpack from ~210s to ~88s (2.4x). Warm rebuilds are unchanged
(webpack filesystem cache already dominates that path).

USE_ESBUILD=true uses esbuild-loader. Faster still, but its TSX parser
mishandles some `<` / `>` comparison expressions (see QuerySummary.tsx).
Left in place as an experiment but not recommended without source fixes.

Default behavior (no env var) is unchanged — still babel-loader +
ts-loader, byte-identical output to before.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@netlify

netlify Bot commented Apr 14, 2026

Copy link
Copy Markdown

Deploy Preview for cbioportalfrontend ready!

Name Link
🔨 Latest commit aa3891e
🔍 Latest deploy log https://app.netlify.com/projects/cbioportalfrontend/deploys/69de20b98744400008592ae8
😎 Deploy Preview https://deploy-preview-5517.preview.cbioportal.org
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

inodb and others added 4 commits April 14, 2026 03:31
- `.circleci/config.yml`: restore/save `.webpack_cache` +
  `.webpack_cache_dll` with branch-prefix fallback keys so every CI run
  gets the closest available cache (branch-specific, then master). The
  existing dependencies-plus-dist cache is all-or-nothing on source
  changes; this supplements it with an incremental layer.
- `vendor-bundles.webpack.config.js`: add filesystem cache at
  `.webpack_cache_dll/`. Modest local win (~4s) but compounds with the
  CI cache — in CI the DLL doesn't have to re-parse vendors each run.
- `.gitignore`: add `.webpack_cache_dll`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wraps `webpack` with the env vars needed for a production build that
skips clean + DLL rebuild and hits the filesystem cache. After an
initial `yarn run buildAll`, subsequent `yarn run buildQuick` calls
complete in ~5-10s (measured: 4.95s webpack / 6.77s total).

Updated copilot-instructions.md to recommend buildQuick.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add scripts/build-packages-if-stale.js which checks each package's
src/ mtime vs dist/index.js mtime and only runs Rollup on packages
whose src is newer. `yarn run buildModules` now calls this script;
`buildModulesAll` preserves the old always-build behavior.

Saves ~60-90s on cold buildAll when only a subset of workspace
packages changed (or none). Measured: 40ms when all up-to-date;
only the touched package rebuilds (8s) otherwise.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
No functional change — a trivial comment edit to invalidate the
dependencies-plus-dist cache (which keys on a hash of src/ among other
things) while leaving yarn.lock + webpack configs stable. Expected:
dep cache restores (skip yarn install), webpack filesystem cache
restores via the new v1-webpack-cache-<branch>- fallback, and main
webpack build drops significantly from the 606s baseline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@inodb inodb changed the title WIP: CI build speed experiment — agent inner-loop optimization WIP: ~35x faster agent inner-loop build (3m30s → ~6s); ~5x faster cold buildAll Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant