Repository navigation

Selenium Tests #860

Workflow file for this run

	name: Selenium Tests

	on:
	schedule:
	- cron: '0 10 * * *'
	# develop is the head of the deploy chain: this suite runs first, then the
	# Terraform workflow (workflow_run on success), then the test-site deploy.
	push:
	branches: [ master, develop ]
	# Run on PRs targeting master or develop (branches = the PR's base branch).
	pull_request:
	branches: [ master, develop ]
	workflow_dispatch:
	permissions:
	contents: read
	# Pulling the private image mirror and pushing what the run built, both on
	# ghcr.io. See docker-compose-run-tests.yml's header.
	packages: write

	jobs:
	# THE IMAGE MIRROR, BEFORE EITHER STACK IS BUILT. Everything the two jobs
	# below pull or build FROM comes from private copies on ghcr.io, so that a
	# reset connection to Docker Hub (the 2026-09-27 scheduled run died three
	# seconds in on one) cannot take the suite down. `missing` copies only an
	# image the mirror does not have yet — added to .github/image-mirror.txt in
	# this commit, say — so a complete mirror costs one manifest request per
	# image and no request to an upstream registry at all.
	#
	# A job here rather than a call to mirror-images.yml, deliberately: a called
	# workflow that declares packages: write fails OUTRIGHT when the caller's
	# token is read-only, which it is on a Dependabot or a fork pull request.
	# Inline, the token is simply downgraded, and with a complete mirror this
	# job only reads.
	mirror:
	runs-on: ubuntu-latest
	steps:
	- name: Checkout Repository
	uses: actions/checkout@v7
	- name: Log in to GitHub Container Registry
	run: echo "${{ secrets.GITHUB_TOKEN }}" \|
	docker login ghcr.io -u "${{ github.actor }}" --password-stdin
	- name: Copy any image the mirror does not have yet
	run: .github/scripts/mirror-images.sh missing

	# Plain suite: full battery in containers, no coverage. Wraps
	# ./docker-run-tests.sh, which builds the stack (docker-compose-run-tests.yml),
	# runs the Selenium suite, and always tears the stack down.
	test:
	needs: mirror
	runs-on: ubuntu-latest
	env:
	# The name every image this job builds is tagged with, and pushed under
	# (the api, the client, the mock STS, the tests and the WS-Federation
	# side-car; common.sh derives STS_IMAGE from it).
	IMAGE_TAG: ${{ github.sha }}
	steps:
	- name: Checkout Repository
	uses: actions/checkout@v7
	with:
	# sts/ is a submodule (https://github.com/rcbj/iya-sts.git). The STS
	# image is built from it and tests/Dockerfile copies a module out of it,
	# so without it both builds fail — saying "failed to read dockerfile"
	# and "COPY sts/common/vendored/bbs2023.js: not found", neither of which names a
	# submodule. The launcher initialises it too (requireMockStsCheckout in
	# common/common.sh); asking for it here keeps a fetch failure on the
	# checkout step, where it reads as one.
	# `recursive`, not `true`, and the difference is load-bearing: there
	# are now submodules INSIDE submodules. api/node-ldapjs is the LDAP
	# library the api's client uses, and sts/node-ldapjs is the same
	# library inside the mock STS, which its embedded directory is built
	# on. `true` stops one level short of the second, and an uninitialised
	# submodule is an EMPTY DIRECTORY — so the sts image builds, npm
	# installs a package with no main, and the container dies at startup
	# with `Cannot find module 'ldapjs'`, naming a package rather than a
	# submodule.
	submodules: recursive
	- name: Install docker-compose
	uses: KengoTODA/actions-setup-docker-compose@v1
	with:
	version: '5.2.0'
	# THE IMAGE MIRROR AND THE BUILT IMAGES ARE PRIVATE PACKAGES on ghcr.io.
	# GITHUB_TOKEN carries packages: write (above), and a package this
	# repository's workflows created grants it access. Compose runs WITHOUT
	# sudo on a runner, so this login is the one it uses.
	- name: Log in to GitHub Container Registry
	run: echo "${{ secrets.GITHUB_TOKEN }}" \|
	docker login ghcr.io -u "${{ github.actor }}" --password-stdin
	# The script's docker_compose() (common/common.sh) invokes compose with
	# sudo itself, so it runs unprivileged here.
	- name: Run Selenium test suite
	# See the coverage job below for why this is not the mock's default
	# of `debug`.
	env:
	STS_LOG_LEVEL: info
	run: ./docker-run-tests.sh
	# What this run built, to ghcr.io: under the commit and under the
	# branch name. Not from a pull request, whose GITHUB_TOKEN may not write
	# packages and whose images are nobody's to keep.
	- name: Push built images
	if: (success() \|\| failure()) && github.event_name != 'pull_request'
	run: .github/scripts/push-stack-images.sh "${{ github.ref_name }}"
	# The tests container writes tests/report as root via a bind mount; hand it
	# back to the runner user before the upload step reads it.
	- name: Normalize report permissions
	if: always()
	run: sudo chown -R "$(id -u):$(id -g)" tests/report \|\| true
	- name: Upload Test Report
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: test-report
	path: tests/report/latest
	if-no-files-found: warn
	retention-days: 30

	# Coverage suite: same battery WITH frontend (Istanbul) + API (c8) coverage.
	# Wraps ./run-coverage.sh, which layers docker-compose-coverage.yml on top,
	# runs the suite, renders the coverage reports, and tears the stack down.
	# TEST_CONCURRENCY IS DELIBERATELY NOT SET HERE. It defaults to 3, and 4
	# and 6 were both tried on 2026-08-31 and both made the run RED.
	#
	# 3 1,396s wall 281 passed, 0 failed
	# 4 1,306s wall 278 passed, 3 FAILED
	# 6 1,266s wall 277 passed, 4 FAILED
	#
	# The binding constraint is ONE JOB, not the pool. [109] Digital
	# Signature takes 594s of a 900s TEST_JOB_TIMEOUT_MS watchdog at 3 --
	# already two thirds of its budget -- and adding a single slot pushes it
	# PAST 900s, where it is killed outright. These runners have four cores
	# and that job is CPU-bound crypto in a browser; the rest of the failures
	# are the same contention seen from the other side, browser jobs whose
	# waitTime is 2 seconds reporting a race as an assertion about the page.
	#
	# THAT LAST SENTENCE IS ALSO WHAT WENT WRONG AT 3 ON 2026-09-01, so read it
	# before raising this again: 288 of 289 jobs passed and [08] OAuth2
	# Authorization Code (public, PKCE=false) failed on a two-second wait for
	# `#token_client_id` after the Keycloak redirect — an instrumented page's
	# load time, reported as an assertion about the page. run-coverage.sh now
	# exports TEST_WAIT_TIME_MS=10000 for this run alone, and the client's log
	# level went to `info` with it. Neither touches the pool: the binding
	# constraint above is still one job, and this number is still 3.
	#
	# So raising this number buys ~7% of wall clock and costs the run. The
	# levers that would actually work are SPLITTING job 109, or a runner with
	# more cores. Fix one of those before touching this again.

	# THE COVERAGE JOB IS SWITCHED OFF (2026-10-02). On run 36984561174 the
	# hosted runner lost contact with GitHub about a third of the way through
	# the instrumented suite and was failed an hour later, with no step after
	# the suite run. The plain test job on the same commit passed. A
	# standard ubuntu-latest runner cannot be given more memory or cores, so
	# until this job has a larger runner, run ./run-coverage.sh locally when
	# a coverage report is wanted. To restore it, uncomment the block below.

	# coverage:
	# needs: mirror
	# runs-on: ubuntu-latest
	# env:
	# # Its own tag: these images are built with COVERAGE=true, so they are
	# # not the same images as the test job's under the same names.
	# IMAGE_TAG: ${{ github.sha }}-coverage
	# steps:
	# - name: Checkout Repository
	# uses: actions/checkout@v7
	# with:
	# # The mock STS submodule — see the test job above. This run uses
	# # --abort-on-container-exit, so a service that cannot be built takes the
	# # whole coverage run down with it.
	# # `recursive`, not `true`, and the difference is load-bearing: there
	# # are now submodules INSIDE submodules. api/node-ldapjs is the LDAP
	# # library the api's client uses, and sts/node-ldapjs is the same
	# # library inside the mock STS, which its embedded directory is built
	# # on. `true` stops one level short of the second, and an uninitialised
	# # submodule is an EMPTY DIRECTORY — so the sts image builds, npm
	# # installs a package with no main, and the container dies at startup
	# # with `Cannot find module 'ldapjs'`, naming a package rather than a
	# # submodule.
	# submodules: recursive
	# - name: Install docker-compose
	# uses: KengoTODA/actions-setup-docker-compose@v1
	# with:
	# version: '5.2.0'
	# # THE IMAGE MIRROR AND THE BUILT IMAGES ARE PRIVATE PACKAGES on ghcr.io.
	# # GITHUB_TOKEN carries packages: write (above), and a package this
	# # repository's workflows created grants it access. Compose runs WITHOUT
	# # sudo on a runner, so this login is the one it uses.
	# - name: Log in to GitHub Container Registry
	# run: echo "${{ secrets.GITHUB_TOKEN }}" \|
	# docker login ghcr.io -u "${{ github.actor }}" --password-stdin
	# - name: Run test suite with coverage
	# # `info`, not the mock's own default of `debug`. That log is the record
	# # of everything the mock issued and is why debug is the default for a
	# # run somebody is watching — but nobody reads a passing CI run's log,
	# # and run-coverage.sh's own header measures what it costs: about half
	# # of that service's CPU, 156MB written in sixteen seconds, and `info`
	# # roughly doubles what one instance can answer with several jobs
	# # driving it. The variable outranks whatever env/docker-tests.js says,
	# # so nothing on this machine or anybody else's changes with it.
	# #
	# # THE api AND THE CLIENT ARE AT `info` TOO NOW, and not through a
	# # variable — issue #269's second half was a decision rather than a
	# # knob. Their level comes from the config file alone and the client's
	# # is compiled into the browser bundle by browserify, so no runtime
	# # variable can reach either; `api/env/docker-tests.js` changed on
	# # 2026-08-31 and `client/src/env/docker-tests.js` on 2026-09-01, each
	# # saying so beside the key. `env/local.js` stays at `debug` on both,
	# # which is the stack somebody watches. The client's mattered most:
	# # that one is what every PAGE logs at, in Chrome, while this job is
	# # serving Istanbul-instrumented bundles to a pool of jobs whose every
	# # browser wait is two seconds.
	# env:
	# STS_LOG_LEVEL: info
	# run: ./run-coverage.sh
	# - name: Push built images
	# if: (success() \|\| failure()) && github.event_name != 'pull_request'
	# run: .github/scripts/push-stack-images.sh
	# "${{ github.ref_name }}-coverage"
	# # coverage/ and tests/report are written as root inside the containers.
	# - name: Normalize coverage permissions
	# if: always()
	# run: sudo chown -R "$(id -u):$(id -g)" coverage tests/report \|\| true
	# - name: Upload coverage report
	# if: always()
	# uses: actions/upload-artifact@v7
	# with:
	# name: coverage-report
	# path: coverage/
	# if-no-files-found: warn
	# retention-days: 30
	# # Distinct artifact name from the test job (artifact names must be unique
	# # across a workflow run).
	# - name: Upload Test Report
	# if: always()
	# uses: actions/upload-artifact@v7
	# with:
	# name: coverage-test-report
	# path: tests/report/latest
	# if-no-files-found: warn
	# retention-days: 30

Navigation Menu

Repository navigation

Selenium Tests #860

Workflow file

Selenium Tests #860

Uh oh!

Workflow file for this run