Skip to content

Commit 83e7e51

Browse files
trallardpeytondmurraydrammock
authored
MAINT - Ensure Playwright tests use test sites and are run in CI (#2133)
Ensures playwright is always available for tests following conversations in #2119 (comment) Note, however, that the `test_version_switcher_highlighting` test is currently failing due to changes to the version switcher component (as part of recent a11y improvements) and due to RTD version switcher no longer being flushed into the sidebar (#2034). I could try and add an alternative test that checks perhaps that we have at least a latest and a dev version in the version switcher? --------- Co-authored-by: Peyton Murray <[email protected]> Co-authored-by: Daniel McCloy <[email protected]>
1 parent 404c3c8 commit 83e7e51

File tree

12 files changed

+220
-71
lines changed

12 files changed

+220
-71
lines changed

pyproject.toml

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,6 @@ doc = [
8181
"ipywidgets",
8282
"graphviz",
8383
]
84-
test = ["pytest", "pytest-cov", "pytest-regressions", "sphinx[test]"]
8584
dev = [
8685
"pyyaml",
8786
"pre-commit",
@@ -90,7 +89,14 @@ dev = [
9089
"pandoc",
9190
"sphinx-theme-builder[cli]",
9291
]
93-
a11y = ["pytest-playwright"]
92+
test = [
93+
"pytest",
94+
"pytest-cov",
95+
"pytest-regressions",
96+
"sphinx[test]",
97+
"pytest-playwright"
98+
]
99+
a11y = ["pydata-sphinx-theme[test]"]
94100
i18n = ["Babel", "jinja2"]
95101

96102
[project.entry-points]

tests/sites/version_switcher/Makefile

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Minimal makefile for Sphinx documentation
2+
#
3+
4+
# You can set these variables from the command line, and also
5+
# from the environment for the first two.
6+
SPHINXOPTS ?=
7+
SPHINXBUILD ?= sphinx-build
8+
SOURCEDIR = .
9+
BUILDDIR = _build
10+
11+
# Put it first so that "make" without argument is like "make help".
12+
help:
13+
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
14+
15+
.PHONY: help Makefile
16+
17+
# Catch-all target: route all unknown targets to Sphinx using the new
18+
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
19+
%: Makefile
20+
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
[
2+
{
3+
"version": "dev",
4+
"url": "https://pydata-sphinx-theme.readthedocs.io/en/latest/"
5+
},
6+
{
7+
"name": "0.16.1 (stable)",
8+
"version": "v0.16.1",
9+
"url": "https://pydata-sphinx-theme.readthedocs.io/en/stable/"
10+
}
11+
]

tests/sites/version_switcher/conf.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
"""Test conf file - basic site with version switcher."""
2+
3+
# -- Project information -----------------------------------------------------
4+
project = "PyData Tests"
5+
copyright = "2020, Pydata community"
6+
author = "Pydata community"
7+
8+
root_doc = "index"
9+
10+
# -- General configuration ---------------------------------------------------
11+
html_theme = "pydata_sphinx_theme"
12+
13+
html_static_path = ["_static"]
14+
15+
# Add version switcher
16+
html_theme_options = {
17+
"switcher": {
18+
"json_url": "_static/switcher.json",
19+
"version_match": "dev",
20+
},
21+
"navbar_start": ["navbar-logo", "version-switcher"],
22+
}
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Simple site with version switcher
2+
=================================
3+
4+
.. toctree::
5+
:caption: Caption 1
6+
:numbered:
7+
8+
page1
9+
page2
10+
section1/index
11+
12+
Other content
13+
-------------
14+
15+
Nothing to see here
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
Page 1
2+
======
3+
4+
**normal link**
5+
6+
- https://pydata-sphinx-theme.readthedocs.io/en/latest/
7+
8+
**GitHub**
9+
10+
.. container:: github-container
11+
12+
https://github.com
13+
https://github.com/pydata
14+
https://github.com/pydata/pydata-sphinx-theme
15+
https://github.com/pydata/pydata-sphinx-theme/pull/1012
16+
https://github.com/orgs/pydata/projects/2
17+
18+
**GitLab**
19+
20+
.. container:: gitlab-container
21+
22+
https://gitlab.com
23+
https://gitlab.com/gitlab-org
24+
https://gitlab.com/gitlab-org/gitlab
25+
https://gitlab.com/gitlab-org/gitlab/-/issues/375583
26+
https://gitlab.com/gitlab-org/gitlab/issues/375583
27+
https://gitlab.com/gitlab-org/gitlab/-/issues/
28+
https://gitlab.com/gitlab-org/gitlab/issues/
29+
https://gitlab.com/gitlab-org/gitlab/-/issues
30+
https://gitlab.com/gitlab-org/gitlab/issues
31+
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84669
32+
https://gitlab.com/gitlab-org/gitlab/-/pipelines/511894707
33+
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6788
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
:html_theme.sidebar_secondary.remove: true
2+
3+
Page :math:`\beta`
4+
==================
5+
6+
.. code-block:: python
7+
8+
import pydata_sphinx_theme as pst
9+
raise RuntimeError('Test of pygments highlighting')
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Section 1 index
2+
===============
3+
.. toctree::
4+
5+
page1
6+
https://google.com
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Section 1 page1
2+
===============

tests/test_a11y.py

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
"""Using Axe-core, scan the Kitchen Sink pages for accessibility violations."""
1+
"""
2+
Using Axe-core, scan the Kitchen Sink pages for accessibility violations.
3+
Note that in contrast with the rest of our tests, the accessibility tests in this file
4+
are run against a build of our PST documentation, not purposedly-built test sites.
5+
"""
26

37
from urllib.parse import urljoin
48

@@ -288,17 +292,6 @@ def test_notebook_ipywidget_output_tab_stop(page: Page, url_base: str) -> None:
288292
expect(ipywidget).to_have_attribute("tabindex", "0")
289293

290294

291-
def test_breadcrumb_expansion(page: Page, url_base: str) -> None:
292-
"""Foo."""
293-
# page.goto(urljoin(url_base, "community/practices/merge.html"))
294-
# expect(page.get_by_label("Breadcrumb").get_by_role("list")).to_contain_text("Merge and review policy") # noqa: E501
295-
page.set_viewport_size({"width": 1440, "height": 720})
296-
page.goto(urljoin(url_base, "community/topics/config.html"))
297-
expect(page.get_by_label("Breadcrumb").get_by_role("list")).to_contain_text(
298-
"Update Sphinx configuration during the build"
299-
)
300-
301-
302295
@pytest.mark.a11y
303296
def test_search_as_you_type(page: Page, url_base: str) -> None:
304297
"""Search-as-you-type feature should support keyboard navigation.

tests/test_playwright.py

Lines changed: 81 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
"""Build minimal test sites with sphinx_build_factory and test them with Playwright."""
1+
"""
2+
Build minimal test sites with sphinx_build_factory and test them with Playwright.
3+
When adding new tests to this file, remember to also add the corresponding test site
4+
to `tests/sites/` or use an existing one.
5+
"""
26

37
from pathlib import Path
48
from typing import Callable
@@ -21,6 +25,7 @@
2125
test_sites_dir = repo_path / "docs" / "_build" / "html" / "playwright_tests"
2226

2327

28+
# ------------------------- Helper functions -------------------------
2429
def _is_overflowing(element):
2530
"""Check if an element is being shortened via CSS due to text-overflow property.
2631
@@ -42,7 +47,8 @@ def _build_test_site(site_name: str, sphinx_build_factory: Callable) -> None:
4247

4348
def _check_test_site(site_name: str, site_path: Path, test_func: Callable):
4449
"""Make the built test site available to Playwright, then run `test_func` on it."""
45-
test_sites_dir.mkdir(exist_ok=True)
50+
# Need to ensure parent directories exist in CI
51+
test_sites_dir.mkdir(exist_ok=True, parents=True)
4652
symlink_path = test_sites_dir / site_name
4753
try:
4854
symlink_path.symlink_to(site_path, True)
@@ -55,65 +61,37 @@ def _check_test_site(site_name: str, site_path: Path, test_func: Callable):
5561
test_sites_dir.rmdir()
5662

5763

58-
def test_version_switcher_highlighting(page: Page, url_base: str) -> None:
64+
# ------------------------- Test functions: style -------------------------
65+
def test_version_switcher_highlighting(
66+
sphinx_build_factory: Callable, page: Page, url_base: str
67+
) -> None:
5968
"""
6069
In sidebar and topbar - version switcher should apply highlight color to currently
6170
selected version.
6271
"""
63-
page.goto(url=url_base)
64-
# no need to include_hidden here ↓↓↓, we just need to get the active version name
65-
button = page.get_by_role("button").filter(has_text="dev")
66-
active_version_name = button.get_attribute("data-active-version-name")
67-
# here we do include_hidden, so sidebar & topbar menus should each have a
68-
# matching entry:
69-
entries = page.get_by_role("option", include_hidden=True).filter(
70-
has_text=active_version_name
71-
)
72-
assert entries.count() == 2
73-
# make sure they're highlighted
74-
for entry in entries.all():
75-
light_mode = "rgb(10, 125, 145)" # pst-color-primary
76-
# dark_mode = "rgb(63, 177, 197)"
77-
expect(entry).to_have_css("color", light_mode)
78-
79-
80-
def test_breadcrumb_expansion(page: Page, url_base: str) -> None:
81-
"""Test breadcrumb text-overflow."""
82-
# wide viewport width → no truncation
83-
page.set_viewport_size({"width": 1440, "height": 720})
84-
page.goto(urljoin(url_base, "community/topics/config.html"))
85-
expect(page.get_by_label("Breadcrumb").get_by_role("list")).to_contain_text(
86-
"Update Sphinx configuration during the build"
87-
)
88-
el = page.get_by_text("Update Sphinx configuration during the build").nth(1)
89-
expect(el).to_have_css("overflow-x", "hidden")
90-
expect(el).to_have_css("text-overflow", "ellipsis")
91-
assert not _is_overflowing(el)
92-
# narrow viewport width → truncation
93-
page.set_viewport_size({"width": 150, "height": 720})
94-
assert _is_overflowing(el)
95-
96-
97-
def test_breadcrumbs_everywhere(
98-
sphinx_build_factory: Callable, page: Page, url_base: str
99-
) -> None:
100-
"""Test breadcrumbs truncate properly when placed in various parts of the layout."""
101-
site_name = "breadcrumbs"
72+
site_name = "version_switcher"
10273
site_path = _build_test_site(site_name, sphinx_build_factory=sphinx_build_factory)
10374

104-
def check_breadcrumb_truncation():
105-
page.goto(
106-
urljoin(url_base, f"playwright_tests/{site_name}/hansel/gretel/house.html")
75+
def check_version_switcher_highlighting():
76+
page.goto(urljoin(url_base, f"playwright_tests/{site_name}/index.html"))
77+
# no need to include_hidden here ↓↓↓, we just need to get the active
78+
# version name
79+
button = page.get_by_role("button").filter(has_text="dev")
80+
active_version_name = button.get_attribute("data-active-version-name")
81+
82+
# here we do include_hidden, since we are not adding this in the sidebar
83+
# we should only get one entry
84+
entries = page.get_by_role("option", include_hidden=True).filter(
85+
has_text=active_version_name
10786
)
108-
# sidebar should overflow
109-
text = "In the oven with my sister, so hot right now. Soooo. Hotttt."
110-
el = page.locator("#main-content").get_by_text(text).last
111-
assert _is_overflowing(el)
112-
# footer containers never trigger ellipsis overflow because min-width is content
113-
el = page.locator(".footer-items__center > .footer-item")
114-
assert not _is_overflowing(el)
87+
assert entries.count() == 1
88+
# make sure they're highlighted
89+
for entry in entries.all():
90+
light_mode = "rgb(10, 125, 145)" # pst-color-primary
91+
# dark_mode = "rgb(63, 177, 197)"
92+
expect(entry).to_have_css("color", light_mode)
11593

116-
_check_test_site(site_name, site_path, check_breadcrumb_truncation)
94+
_check_test_site(site_name, site_path, check_version_switcher_highlighting)
11795

11896

11997
def test_colors(sphinx_build_factory: Callable, page: Page, url_base: str) -> None:
@@ -143,3 +121,53 @@ def check_colors():
143121
expect(el).to_have_css("color", hover_color)
144122

145123
_check_test_site(site_name, site_path, check_colors)
124+
125+
126+
# ------------------------- Test functions: layout & components -----------------------
127+
128+
129+
def test_breadcrumb_expansion(
130+
sphinx_build_factory: Callable, page: Page, url_base: str
131+
) -> None:
132+
"""Test breadcrumb text-overflow."""
133+
site_name = "breadcrumbs"
134+
site_path = _build_test_site(site_name, sphinx_build_factory=sphinx_build_factory)
135+
136+
def check_breadcrumb_expansion():
137+
# wide viewport width → no truncation
138+
page.set_viewport_size({"width": 1440, "height": 720})
139+
page.goto(urljoin(url_base, "community/topics/config.html"))
140+
expect(page.get_by_label("Breadcrumb").get_by_role("list")).to_contain_text(
141+
"Update Sphinx configuration during the build"
142+
)
143+
el = page.get_by_text("Update Sphinx configuration during the build").nth(1)
144+
expect(el).to_have_css("overflow-x", "hidden")
145+
expect(el).to_have_css("text-overflow", "ellipsis")
146+
assert not _is_overflowing(el)
147+
# narrow viewport width → truncation
148+
page.set_viewport_size({"width": 150, "height": 720})
149+
assert _is_overflowing(el)
150+
151+
_check_test_site(site_name, site_path, check_breadcrumb_expansion)
152+
153+
154+
def test_breadcrumbs_everywhere(
155+
sphinx_build_factory: Callable, page: Page, url_base: str
156+
) -> None:
157+
"""Test breadcrumbs truncate properly when placed in various parts of the layout."""
158+
site_name = "breadcrumbs"
159+
site_path = _build_test_site(site_name, sphinx_build_factory=sphinx_build_factory)
160+
161+
def check_breadcrumb_truncation():
162+
page.goto(
163+
urljoin(url_base, f"playwright_tests/{site_name}/hansel/gretel/house.html")
164+
)
165+
# sidebar should overflow
166+
text = "In the oven with my sister, so hot right now. Soooo. Hotttt."
167+
el = page.locator("#main-content").get_by_text(text).last
168+
assert _is_overflowing(el)
169+
# footer containers never trigger ellipsis overflow because min-width is content
170+
el = page.locator(".footer-items__center > .footer-item")
171+
assert not _is_overflowing(el)
172+
173+
_check_test_site(site_name, site_path, check_breadcrumb_truncation)

tox.ini

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ env_list =
1515
# helping contributors run common tasks without needing to call all the steps
1616
# For example to run the tests: tox run -m tests
1717
labels =
18-
tests = compile-assets, i18n-compile, py312-tests
18+
tests = compile-assets, i18n-compile, py312-tests
1919
a11y = compile-assets, i18n-compile, py312-docs, a11y-tests
2020
i18n = i18n-extract, i18n-compile
2121
live-server = compile-assets, i18n-compile, docs-live
@@ -61,15 +61,20 @@ description = "Run tests Python and Sphinx versions. If a Sphinx version is spec
6161
# need to ensure the package is installed in editable mode
6262
package = editable
6363
extras =
64-
test # install dependencies - defined in pyproject.toml
64+
test # install dependencies, includes pytest-playwright - defined in pyproject.toml
65+
pass_env = GITHUB_ACTIONS # so we can check if this is run on GitHub Actions
6566
deps =
6667
coverage[toml]
6768
py39-sphinx61-tests: sphinx~=6.1.0
6869
py312-sphinxdev: sphinx[test] @ git+https://github.com/sphinx-doc/sphinx.git@master
6970
depends =
7071
compile-assets,
7172
i18n-compile
73+
allowlist_externals=
74+
playwright
75+
bash
7276
commands =
77+
bash -c 'if [[ "{env:GITHUB_ACTIONS:}" == "true" ]]; then playwright install --with-deps; else playwright install; fi'
7378
py3{9,10,11,12}{,-sphinx61,-sphinxdev,}-tests: coverage run -m pytest -m "not a11y" {posargs}
7479
py3{9,10,11,12}{,-sphinx61,-sphinxdev,}-tests-no-cov: pytest -m "not a11y" {posargs}
7580

@@ -84,8 +89,7 @@ description = "run accessibility tests with Playwright and axe-core"
8489
base_python = py312 # keep in sync with tests.yml
8590
pass_env = GITHUB_ACTIONS # so we can check if this is run on GitHub Actions
8691
extras =
87-
test
88-
a11y
92+
test # install dependencies, includes pytest-playwright - defined in pyproject.toml
8993
depends =
9094
compile-assets,
9195
i18n-compile

0 commit comments

Comments
 (0)