Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how the dashboard scoring works #667

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,38 @@ Scores for each focus area are computed in [results-analysis](https://github.com

The "pipeline" for updating scores is as follows:

- TODO
- All of WPT is run on Chrome, Edge, Firefox, and Safari across two different CI systems: Taskcluster and Azure Pipelines. The cadence varies:

- Chrome Canary: [every commit](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L188-L193)
foolip marked this conversation as resolved.
Show resolved Hide resolved
- Chrome Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L206-L211)
- Edge Dev: [every 3 hours](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L380)
- Edge Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L344)
- Firefox Nightly: [every commit](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L163-L168)
- Firefox Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L175-L180)
- Safari Technology Preview: [every 3 hours](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L491)
- Safari Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L451)

Runs take 1-3 hours to complete. Results are uploaded to wpt.fyi and are made available within minutes of completion.
foolip marked this conversation as resolved.
Show resolved Hide resolved

Delay: 1-6 hours for experimental, 1-27 hours for stable

- [results-analysis-cache](https://github.com/web-platform-tests/results-analysis-cache) downloads all runs from wpt.fyi and coverts them into a more efficient format to speed up results scoring. This runs [hourly](https://github.com/web-platform-tests/results-analysis-cache/blob/683fe10f5f58c096e2a0dc3214c53ce949192c39/.github/workflows/results.yml#L7-L9) and takes less than 10 minutes.

Delay: 10-70 minutes

- [results-analysis](https://github.com/web-platform-tests/results-analysis) fetches results from `results-analysis-cache` and produces the CSV files used by the dashboard. This runs [every 3 hours](https://github.com/web-platform-tests/results-analysis/blob/67c603a9b306c71da37c4798301935318949c45e/.github/workflows/update_gh_pages.yml#L7-L9) and takes a few minutes.

Delay: 0-3 hours
foolip marked this conversation as resolved.
Show resolved Hide resolved

- The wpt.fyi frontend always fetches the latest CSV files, so there is no additional delay before the changes appear on the dashboard.

## Troubleshooting

It's normal for experimental results to take up 10 hours to update and stable results over 30 hours. If things take longer than this it's time to troubleshoot.

First check if new runs are appearing in [wpt.fyi/runs](https://wpt.fyi/runs)? If not, the runs are probably not completing in Taskcluster or Azure Pipelines. Diagnosing why this happens is not always easy, so please [file an issue](https://github.com/web-platform-tests/wpt/issues/new) and poke someone for help.

Once wpt.fyi has the results it's rare for anything to go wrong, but it could be in one of the GitHub Actions workflows:

- [results-analysis-cache](https://github.com/web-platform-tests/results-analysis-cache/actions)
- [results-analysis](https://github.com/web-platform-tests/results-analysis/actions)