Conversation
Runs against https://www.cbioportal.org by default, inside the pinned Playwright Docker image, sharded 4x. Triggered on frontend-path PRs, pushes to master/rc, and manual workflow_dispatch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The filter still pointed at the pre-move subfolder, so the workflow wouldn't self-trigger on changes to the file itself. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
GitHub Actions defaults run: steps to /bin/sh (dash) inside container: jobs, but the script used bash arrays. Switch to an explicit shell: bash and replace the array with a simple if/else so the command is portable either way. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous run: slowest shard was shard 4 at 15.6m on a 131-test split; total wall-clock 16.8m. With 8 shards the slowest partition should be roughly ~68 tests, projected ~8m per shard. No change to runner size or per-shard parallelism. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Experiment: if the ubuntu-latest-4-cores label resolves on this account, per-shard CPU should roughly double vs the default 2-vCPU runner, shaving ~half off each shard's wall time. Will revert if the label isn't available. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ubuntu-latest-4-cores label isn't available on this account — image-tag sat queued for minutes with runner_name: null. Back to the default 2-vCPU runner. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous run had shard 7 at 8m51s with 72 tests from just two screenshot-heavy files (results.spec.ts + screenshot-misc.spec.ts). Playwright --shard splits the enumerated test list into contiguous chunks, so going to 12 shards forces those test blocks to split further. Projected slowest-shard wall time ≈ 6m. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
✅ Deploy Preview for cbioportalfrontend ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Using cBioPortal org's Default runner group 4-core runner to roughly halve per-shard CPU time. image-tag stays on ubuntu-latest (2-core) — it's just a node -p call and doesn't benefit from a larger runner, and keeping it on the default removes a potential label-mismatch block between jobs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Swap the e2e matrix from the (unavailable) ubuntu-latest-4-cores GitHub larger-runner label to Blacksmith's blacksmith-4vcpu-ubuntu-2404. Should roughly double per-shard CPU vs the default 2-vcpu runner and drop slowest-shard wall time from ~8m to ~4m. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
With Blacksmith 4-vcpu each shard has double the CPU of the original 2-vcpu baseline, so 12 shards was over-partitioned — half the runners die with startup overhead or 0-28s test work. 6 shards per run should hold wall-clock in the 5-7 min range while cutting runner-minutes and image-pull concurrency roughly in half. Also fixes a stale '/4' in the step name (was already wrong at 12). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6 shards was slower wall-clock (9m55 vs 7m30) because serial test execution within a shard dominates, and the extra CPU per 4-vcpu runner goes unused with workers=1. 12 shards restored to unblock 4-min wall-clock target; further gains will come from balancing the shard distribution. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With workers=1 each Blacksmith 4-vcpu shard leaves ~3 vcpu idle — Playwright runs tests serially within a shard. Bumping to workers=2 splits per shard across two process-isolated browser contexts, roughly halving per-shard wall-clock if the test suite tolerates it. Experiment: watch for screenshot pixel flakes. Screenshot tests are CPU-sensitive due to font antialiasing + animation timing; two concurrent Chromiums on the same host may cause drift. Fall back to workers=1 if flake rate climbs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Just-for-fun test to see if a bigger runner tier squeezes more out of the workers=2 split. Blacksmith's standard tiers are 2/4/8/16/32 so 6 may not resolve — if so, will revert or bump to 8. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6-vcpu label didn't resolve (Blacksmith tiers are 2/4/8/16/32); bumping to the next-up 8-vcpu. Expected payoff is modest since Playwright is largely I/O-bound against the remote backend, but worth measuring for the comparison. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… speedup) Measured: total runner-minutes nearly identical at 4-vcpu (32.17m) vs 8-vcpu (32.35m), confirming the suite is network-bound against cbioportal.org rather than CPU-bound. Bigger runner just doubles cost without moving wall-clock. 4-vcpu stays. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Playwright's default --shard=X/N groups a file's tests together, which on this suite left shard 11 with 315s of screenshot-heavy work while shard 8 got 0 tests. Both failure modes together — one slow shard bounds wall-clock and one empty shard wastes a runner. Replace --shard with an LPT-packed manifest of spec files per shard, built from per-file timings captured from a prior 12-shard run. All shards now target ~153-168s of sequential test work; at workers=2 this projects to ~2m wall-clock on the slowest shard. The manifest will need periodic regeneration as tests are added or durations shift — current plan is to do it manually from the latest run's artifacts. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Found 8 test failures on Blacksmith runners: Failures
|
With the duration-balanced manifest, the slowest shard holds just one heavy file (studyview.spec.ts, ~168s sequential). Playwright's fullyParallel=false makes in-file tests serial inside one worker, so workers=2 couldn't help that shard — it only bloated the other shards' per-runner billing and reintroduced the oncoprinter race flake. Back to workers=1 per shard. Projected: ~3m20s wall-clock, fewer race flakes, slightly cheaper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Also drops the follow-up workers=1 retry. Measured: manifest + workers=2 ran 4m49s for $0.31, manifest + workers=1 ran 4m37s for $0.36 with 3 failures — both worse than default --shard=X/12 + workers=2 at 4m31s for $0.26. The manifest's per-file balance is defeated by fullyParallel=false: the slowest shard ends up pinned to one heavy single-file bucket (studyview/screenshot-results-excluding-unprofiled) that can't use the extra worker anyway. Back to the best-measured config. Further wins need fullyParallel=true in playwright.config.ts or a split of the heavy spec files — both touch the suite, out of scope for this workflow PR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds test.describe.configure({ mode: 'parallel' }) to spec files that
each cold-load their own URL and don't share state across tests. With
--workers=2 in the workflow, these files' tests can now run in parallel
within a shard instead of serializing through one worker.
Picked conservatively: only files where the author's docstring or
structure makes independence explicit. Files with test.describe.serial,
beforeAll-driven shared state, or cumulative-state tests (config,
studyview, screenshot-results-*, comparison-*, home, etc.) are left on
the default serial mode.
Covers: redirect, patient-logic, genomic-evolution-logic,
patient-screenshot, oncoprint-screenshot-catalog, oncoprinter-screenshot,
group-comparison-misc-screenshot, screenshot-misc, mutation-mapper-tool,
quick-search, mutations-tab, plots.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ison
Splits the 15-test helper describe block into two exported functions:
- runResultsTabTests (9 tests) — independent main-view tab snapshots
- runResultsComparisonTests (6 tests) — comparison-flow tests that
share backend session state (patient_enrichments=true set by
alterations-patient-mode is needed by mrna enrichments ranking)
Each of the 3 existing config spec files now runs ONLY the comparison
subset. Three new *-tabs.spec.ts files run the tab subset with
test.describe.configure({ mode: 'parallel' }) — they were previously
all bundled in a single spec per config that couldn't shard or
parallelize.
Moves 27 tab snapshot PNGs under new __snapshots__/...-tabs.spec.ts/
paths (snapshotPathTemplate is keyed by test file).
Shardability: 3 bundled files → 6 shardable files. Expected
slowest-shard wall-clock drop for the results-screenshots portion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.github/workflows/playwright-e2e.ymlto run the Playwright suite (end-to-end-test-playwright/) in GitHub Actionsmcr.microsoft.com/playwright:v1.59.1-jammyimage, version resolved fromend-to-end-test-playwright/package.jsonfail-fast: false; each shard uploads its HTML report + traces on any outcomepull_requestandpush(master/rc) on frontend paths, plus manualworkflow_dispatchwithcbioportal_urlandgrepinputse2e-stabilize-flakesbecause the Playwright suite lives on that branch — PR diff is just the workflow fileStatus of testing so far
Validated on
alisman/cbioportal-frontend:mcr.microsoft.comimage-pull throttling (retried 3× by GitHub, all blocked); other 11 passedNext iteration
Once merged (or via workflow_dispatch), we want to try a larger runner. The
ubuntu-latest-4-coreslabel wasn't configured on the alisman fork so we reverted; if cBioPortal has larger runners set up in org settings, we can patchruns-on:to that label.Test plan
image-tagresolves tov1.59.1-jammyplaywright-report-shard-N) upload on success and failure paths