Nightly Comprehensive Tests #301

Workflow file for this run

.github/workflows/nightly.yml at e217ea1

	# Nightly comprehensive test suite with full metrics collection
	# Runs all tests (unit + integration + e2e, including slow/ml_models)
	# Generates comprehensive metrics, reports, and trend tracking
	# See docs/ci/METRICS.md — test metrics and health tracking (Layer 3)
	#
	# Note: Linter warnings about "Unable to resolve action" for actions/checkout@v7
	# and actions/cache@v6 are false positives - these are standard GitHub Actions
	# that exist in the marketplace but the linter cannot resolve them offline.
	#
	# Structure mirrors python-app.yml with separate jobs for isolation:
	# Dependency flow:
	# 1. nightly-lint, nightly-build, preload-ml-models-nightly start in parallel (preload has no lint dep)
	# 2. After lint+build: one parallel segment — nightly-security-quality, nightly-test-unit,
	# nightly-viewer-unit (Vitest; always runs, no path filters)
	# 3. nightly-test-integration and nightly-test-e2e run in parallel (both gate: preload only)
	# 4. nightly-viewer-e2e (Playwright) after pytest E2E succeeds — always runs when workflow runs
	# 5. nightly-only-tests gates: unit, integration, e2e, nightly-viewer-unit, nightly-viewer-e2e
	# 6. nightly-metrics gates: lint, security-quality, build, all test jobs + both viewer jobs, nightly-only
	# 7. nightly-docs gates nightly-metrics (very last step, final validation)
	#
	# Jobs:
	# - preload-ml-models-nightly: Preload and validate ML models
	# - nightly-lint: Fast lint checks (format, lint, markdown, type)
	# - nightly-security-quality: Security and quality checks (security, complexity, deadcode, docstrings, spelling)
	# - nightly-docs: Documentation build
	# - nightly-build: Package build
	# - nightly-test-unit: Unit tests
	# - nightly-viewer-unit: Vitest for web/gi-kg-viewer (same segment as security-quality + test-unit)
	# - nightly-test-integration: Integration tests
	# - nightly-test-e2e: E2E tests (multi_episode mode - same as regular CI)
	# - nightly-viewer-e2e: Playwright + Firefox (Tier-1 + Tier-2, after nightly-test-e2e)
	# - nightly-tier3-validation: Tier-3 real-backend Playwright walk against the
	# in-repo synthetic corpus + in-CI-built LanceDB search index. RFC-086 / ADR-095 / #774.
	# Auto-files a deduped tier3-regression issue on failure (scheduled runs only).
	# - nightly-only-tests: Nightly-specific tests with production models
	# - nightly-metrics: Collect metrics and generate dashboard

	name: Nightly Comprehensive Tests

	on:
	schedule:
	# Run nightly at 2 AM UTC (runs on default branch/main)
	- cron: '0 2 * * *'
	workflow_dispatch: # Allow manual triggering
	# Note: Metrics for main and release branches are handled by python-app.yml (regular CI)
	# Nightly is for comprehensive testing and metrics on main branch only

	permissions:
	contents: write # For publishing metrics to gh-pages branch
	pages: write # For publishing metrics to GitHub Pages
	id-token: write # For GitHub Pages deployment

	concurrency:
	group: "pages-metrics"
	cancel-in-progress: false # Don't cancel, let metrics accumulate

	jobs:
	# ===========================================================================
	# Preload ML models - runs first, validates cache
	# ===========================================================================
	preload-ml-models-nightly:
	runs-on: ubuntu-latest
	timeout-minutes: 30 # May need to download production models
	env:
	HF_HOME: /home/runner/.cache/huggingface
	HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
	# Gated pyannote diarization models download only with a terms-accepted token.
	HF_TOKEN: ${{ secrets.HF_TOKEN }}
	steps:
	- uses: actions/checkout@v7

	- name: Free disk space
	run: \|
	sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
	docker image prune -af \|\| true
	sudo apt-get clean
	sudo rm -rf /var/lib/apt/lists/*
	df -h

	- name: Cache ML models (production + test models for nightly)
	uses: actions/cache@v5
	id: cache-models
	with:
	path: \|
	~/.cache/whisper
	~/.cache/huggingface
	# Fully content-based cache key. `diar2` salt forces a clean re-preload (`diar1` was saved
	# incomplete by a cancelled run); the gated pyannote + MiniLM models get added to the cache.
	key: ml-models-nightly-${{ runner.os }}-diar2-${{ hashFiles('scripts/cache/preload_ml_models.py', 'src/podcast_scraper/config.py') }}
	restore-keys: \|
	ml-models-nightly-${{ runner.os }}-diar2-
	ml-models-nightly-${{ runner.os }}-

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Install ffmpeg (pyannote 4.x / torchcodec needs FFmpeg shared libs to import)
	run: \|
	sudo apt-get update
	sudo apt-get install -y --no-install-recommends ffmpeg
	- name: Install full dependencies (including ML)
	run: \|
	python -m pip install --upgrade pip
	pip install -e ".[dev,ml,llm,search]"

	- name: Validate ML model cache (probe only — preload runs next if incomplete)
	id: validate-cache
	run: \|
	# Single manifest (#917): verify_required_models.py reads
	# model_manifest.ci_artifact_model_ids() -- no duplicated bash arrays.
	# Must NOT exit non-zero: this step only SETS models_complete so the conditional
	# preload below can run. A cache miss (e.g. config.py changed -> cache key changed,
	# as on the #1010 merge) is expected and is handled by the preload + the final
	# validation step; exiting 1 here killed the job before preload could run.
	if python scripts/cache/verify_required_models.py --tier production; then
	echo "models_complete=true" >> "$GITHUB_OUTPUT"
	else
	echo "models_complete=false" >> "$GITHUB_OUTPUT"
	fi
	- name: Preload production ML models (if incomplete)
	if: steps.validate-cache.outputs.models_complete != 'true'
	run: make preload-ml-models-production

	- name: Final cache validation
	run: python scripts/cache/verify_required_models.py --tier production
	- name: Upload ML models artifact (for dependent test jobs in same run)
	uses: actions/upload-artifact@v7
	with:
	name: ml-models
	path: /home/runner/.cache
	retention-days: 1
	include-hidden-files: true

	# ===========================================================================
	# Code quality checks
	# ===========================================================================
	# Fast lint checks - blocks tests to catch issues early
	nightly-lint:
	runs-on: ubuntu-latest
	timeout-minutes: 10 # Fast checks: format, lint, markdown, type
	steps:
	- uses: actions/checkout@v7

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Set up Node.js
	uses: actions/setup-node@v7
	with:
	node-version: "20"

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	python -m pip install -e ".[dev]"

	- name: Install markdownlint
	run: npm install -g markdownlint-cli

	- name: Run fast lint checks
	run: \|
	echo "Running fast lint checks..."
	make format-check
	make lint
	make lint-markdown
	make type

	# Security and quality checks - gated by lint/build, runs independently to nightly-metrics (does not gate tests)
	nightly-security-quality:
	runs-on: ubuntu-latest
	timeout-minutes: 20 # Slower checks: security, complexity, deadcode, docstrings, spelling
	needs: [nightly-lint, nightly-build] # Wait for lint, build to pass (docs removed - doesn't gate security)
	steps:
	- uses: actions/checkout@v7

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	python -m pip install -e ".[dev]"

	- name: Run security checks
	run: \|
	echo "Running security checks..."
	make security

	- name: Run quality checks
	run: \|
	echo "Running quality checks..."
	make quality

	# ===========================================================================
	# Documentation build - validates docs can be built correctly
	# Runs as very last step after all tests and metrics pass (final validation)
	# Docs deployment happens in docs.yml workflow (separate, already gated properly)
	# ===========================================================================
	nightly-docs:
	runs-on: ubuntu-latest
	timeout-minutes: 15
	needs: [nightly-metrics] # Very last step - validates docs after all tests and metrics pass
	steps:
	- uses: actions/checkout@v7

	- name: Free disk space
	run: \|
	sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
	docker image prune -af \|\| true
	sudo apt-get clean
	sudo rm -rf /var/lib/apt/lists/*
	df -h

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: \|
	docs/requirements.txt
	pyproject.toml

	- name: Install doc dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -r docs/requirements.txt
	# No [ml] / ffmpeg: mkdocstrings (griffe) parses source statically and
	# never imports the package, so the heavy ML stack and FFmpeg shared
	# libs aren't needed to build the API docs.
	pip install -e .

	- name: Build docs
	run: make docs

	# ===========================================================================
	# Package build - validates package can be built correctly
	# Builds source distribution (sdist) and wheel distribution
	# Catches packaging issues (pyproject.toml errors, missing files, etc.)
	# Fast check (~2-3 min) that gates test-unit to catch packaging issues early
	# ===========================================================================
	nightly-build:
	runs-on: ubuntu-latest
	timeout-minutes: 10
	steps:
	- uses: actions/checkout@v7

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Install build tools
	run: \|
	python -m pip install --upgrade pip
	pip install build

	- name: Build package
	run: make build

	# ===========================================================================
	# Unit tests - isolated job (no ML dependencies, runs in parallel with preload)
	# ===========================================================================
	nightly-test-unit:
	runs-on: ubuntu-latest
	timeout-minutes: 15
	needs: [nightly-lint, nightly-build] # Wait for lint, build to pass (docs removed - doesn't gate unit tests)
	steps:
	- uses: actions/checkout@v7

	- name: Free disk space
	run: \|
	sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
	docker image prune -af \|\| true
	sudo apt-get clean
	sudo rm -rf /var/lib/apt/lists/*
	df -h

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Install dev dependencies (no ML; includes FastAPI via [dev])
	run: \|
	python -m pip install --upgrade pip
	python -m pip install -e ".[dev]"
	python -m pip install pytest-json-report

	- name: Install ffmpeg (required for audio preprocessing unit tests)
	run: \|
	sudo apt-get update
	sudo apt-get install -y --no-install-recommends ffmpeg

	- name: Verify unit tests can import without ML dependencies
	run: \|
	PYTHONPATH="${PYTHONPATH}:$(pwd)"
	export PYTHONPATH
	python scripts/tools/check_unit_test_imports.py
	env:
	PACKAGE: podcast_scraper

	- name: Run unit tests with coverage (parallel execution)
	run: \|
	PYTHONPATH="${PYTHONPATH}:$(pwd)"
	export PYTHONPATH
	mkdir -p reports
	# Run tests with coverage and capture output and exit code
	set +e
	OUTPUT=$(pytest tests/unit/ -v --tb=short -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --json-report --json-report-file=reports/pytest-unit.json --junitxml=reports/junit-unit.xml --cov=podcast_scraper --cov-report=xml:reports/coverage-unit.xml --cov-report=term-missing --reruns 2 --reruns-delay 1 --durations=20 2>&1)
	PYTEST_EXIT_CODE=$?
	set -e
	echo "$OUTPUT"
	# Verify tests were collected and run
	if echo "$OUTPUT" \| grep -q "no tests collected"; then
	echo "ERROR: No tests were collected!"
	exit 1
	fi
	# Verify minimum test count
	TEST_COUNT=$(echo "$OUTPUT" \| grep -oE "[0-9]+ passed" \| head -1 \| grep -oE "[0-9]+" \|\| echo "0")
	if [ "$TEST_COUNT" -lt 50 ]; then
	echo "ERROR: Only $TEST_COUNT tests passed, expected at least 50 unit tests"
	exit 1
	fi
	# Exit with pytest's exit code
	if [ $PYTEST_EXIT_CODE -ne 0 ]; then
	echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE"
	exit $PYTEST_EXIT_CODE
	fi
	env:
	PACKAGE: podcast_scraper

	- name: Upload unit test reports
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: nightly-unit-reports
	path: reports/
	retention-days: 30

	# GI/KG viewer v2 — Vitest (same parallel segment as nightly-security-quality + nightly-test-unit)
	nightly-viewer-unit:
	runs-on: ubuntu-latest
	timeout-minutes: 5
	needs: [nightly-lint, nightly-build]
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-node@v7
	with:
	node-version: "22"
	cache: "npm"
	cache-dependency-path: web/gi-kg-viewer/package-lock.json
	- name: Install dependencies
	run: \|
	cd web/gi-kg-viewer
	npm ci
	- name: Run Vitest unit tests
	run: \|
	cd web/gi-kg-viewer
	npm run test:coverage

	# ===========================================================================
	# Integration tests - isolated job
	# ===========================================================================
	nightly-test-integration:
	runs-on: ubuntu-latest
	timeout-minutes: 60 # was 30; integration suite outgrew it (hit the 30m cap 2026-08-03)
	needs: [preload-ml-models-nightly] # Wait for model preload only (can run in parallel with unit tests)
	env:
	HF_HOME: /home/runner/.cache/huggingface
	HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
	ML_MODELS_VALIDATED: "true"
	steps:
	- uses: actions/checkout@v7

	- name: Free disk space
	run: \|
	sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
	docker image prune -af \|\| true
	sudo apt-get clean
	sudo rm -rf /var/lib/apt/lists/*
	df -h

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Download ML models from preload job
	uses: actions/download-artifact@v8
	with:
	name: ml-models
	path: /home/runner/.cache

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -e ".[dev,ml,llm,search]"
	pip install pytest-socket pytest-json-report

	- name: Install ffmpeg
	run: \|
	sudo apt-get update
	sudo apt-get install -y --no-install-recommends ffmpeg

	- name: Run integration tests with coverage (network guard, parallel)
	run: \|
	PYTHONPATH="${PYTHONPATH}:$(pwd)"
	export PYTHONPATH
	mkdir -p reports
	# Run integration tests with coverage and network guard
	set +e
	OUTPUT=$(pytest tests/integration/ -v -m integration -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --json-report --json-report-file=reports/pytest-integration.json --junitxml=reports/junit-integration.xml --cov=podcast_scraper --cov-append --cov-report=xml:reports/coverage-integration.xml --cov-report=term-missing --cov-fail-under=42 --disable-socket --allow-hosts=127.0.0.1,localhost --reruns 2 --reruns-delay 1 --durations=20 2>&1)
	PYTEST_EXIT_CODE=$?
	set -e
	echo "$OUTPUT"
	# Verify tests were collected and run
	if echo "$OUTPUT" \| grep -q "no tests collected"; then
	echo "ERROR: No tests were collected!"
	exit 1
	fi
	# Verify minimum test count
	TEST_COUNT=$(echo "$OUTPUT" \| grep -oE "[0-9]+ passed" \| head -1 \| grep -oE "[0-9]+" \|\| echo "0")
	if [ "$TEST_COUNT" -lt 50 ]; then
	echo "ERROR: Only $TEST_COUNT tests passed, expected at least 50 integration tests"
	exit 1
	fi
	# Exit with pytest's exit code
	if [ $PYTEST_EXIT_CODE -ne 0 ]; then
	echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE"
	exit $PYTEST_EXIT_CODE
	fi

	- name: Upload integration test reports
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: nightly-integration-reports
	path: reports/
	retention-days: 30

	# ===========================================================================
	# E2E tests - isolated job, multi_episode mode (same as regular CI)
	# ===========================================================================
	nightly-test-e2e:
	runs-on: ubuntu-latest
	timeout-minutes: 60 # was 45; full e2e suite hit the 45m cap (cancelled 2026-08-03)
	needs: [preload-ml-models-nightly] # Wait for model preload only (can run in parallel with unit tests)
	env:
	HF_HOME: /home/runner/.cache/huggingface
	HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
	ML_MODELS_VALIDATED: "true"
	# E2E test mode - matches regular CI (python-app.yml)
	E2E_TEST_MODE: "multi_episode"
	# Diarization e2e: pyannote provider needs a token to construct; gated models
	# are preloaded in preload-ml-models-nightly. OFFLINE=1 → load from cache only,
	# no 3rd-party traffic at test time (airgapped contract).
	HF_TOKEN: ${{ secrets.HF_TOKEN }}
	HF_HUB_OFFLINE: "1"
	steps:
	- uses: actions/checkout@v7

	- name: Free disk space
	run: \|
	sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
	docker image prune -af \|\| true
	sudo apt-get clean
	sudo rm -rf /var/lib/apt/lists/*
	df -h

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Download ML models from preload job
	uses: actions/download-artifact@v8
	with:
	name: ml-models
	path: /home/runner/.cache

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -e ".[dev,ml,llm,search]"
	pip install pytest-socket pytest-json-report

	- name: Install ffmpeg
	run: \|
	sudo apt-get update
	sudo apt-get install -y --no-install-recommends ffmpeg

	- name: Run E2E tests with coverage (network guard, xdist)
	run: \|
	PYTHONPATH="${PYTHONPATH}:$(pwd)"
	export PYTHONPATH
	mkdir -p reports
	set +e
	OUTPUT=$(E2E_TEST_MODE=multi_episode pytest tests/e2e/ -v -m "e2e and not nightly" -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --json-report --json-report-file=reports/pytest-e2e.json --junitxml=reports/junit-e2e.xml --cov=podcast_scraper --cov-report=xml:reports/coverage-e2e.xml --cov-report=term-missing --cov-fail-under=39 --disable-socket --allow-hosts=127.0.0.1,localhost --reruns 2 --reruns-delay 1 --durations=20 2>&1)
	PYTEST_EXIT_CODE=$?
	set -e
	echo "$OUTPUT"
	# Verify tests were collected and run
	if echo "$OUTPUT" \| grep -q "no tests collected"; then
	echo "ERROR: No tests were collected!"
	exit 1
	fi
	# Verify minimum test count
	TEST_COUNT=$(echo "$OUTPUT" \| grep -oE "[0-9]+ passed" \| head -1 \| grep -oE "[0-9]+" \|\| echo "0")
	if [ "$TEST_COUNT" -lt 50 ]; then
	echo "ERROR: Only $TEST_COUNT tests passed, expected at least 50 E2E tests"
	exit 1
	fi
	# Exit with pytest's exit code
	if [ $PYTEST_EXIT_CODE -ne 0 ]; then
	echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE"
	exit $PYTEST_EXIT_CODE
	fi

	- name: Upload E2E test reports
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: nightly-e2e-reports
	path: reports/
	retention-days: 30

	# GI/KG viewer v2 — Playwright after pytest E2E (matches python-app viewer-e2e placement)
	nightly-viewer-e2e:
	runs-on: ubuntu-latest
	timeout-minutes: 20
	needs: [nightly-lint, nightly-build, nightly-test-e2e]
	if: \|
	always() &&
	needs.nightly-lint.result == 'success' &&
	needs.nightly-build.result == 'success' &&
	needs.nightly-test-e2e.result == 'success'
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-node@v7
	with:
	node-version: "22"
	cache: "npm"
	cache-dependency-path: web/gi-kg-viewer/package-lock.json
	- name: Install dependencies and browsers
	run: \|
	cd web/gi-kg-viewer
	npm ci
	npx playwright install --with-deps firefox
	- name: Run Playwright E2E
	run: \|
	cd web/gi-kg-viewer
	npm run test:e2e

	# ===========================================================================
	# Tier-3 viewer validation (RFC-086 / ADR-095 / GH #774)
	#
	# Runs the real-backend Playwright walk against the in-repo synthetic
	# corpus + an in-CI-built LanceDB search index. Catches drift between mocks
	# (Tier-1 + Tier-2 above) and what the real Python API actually serves.
	# Auto-files a deduped ``tier3-regression`` issue on failure.
	#
	# Previously lived in a separate ``tier3-validation.yml`` workflow with
	# a weekly Sunday cron. Folded into nightly to inherit daily cadence
	# (better drift detection) + the existing dispatch entry + the metrics
	# dashboard. See VIEWER_GRAPH_SPEC.md "Graph handoff orchestrator".
	# ===========================================================================
	nightly-tier3-validation:
	runs-on: ubuntu-latest
	timeout-minutes: 30
	needs: [nightly-lint, nightly-build]
	if: \|
	always() &&
	needs.nightly-lint.result == 'success' &&
	needs.nightly-build.result == 'success'
	permissions:
	contents: read
	issues: write
	steps:
	- uses: actions/checkout@v7

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Set up Node
	uses: actions/setup-node@v7
	with:
	node-version: "22"
	cache: "npm"
	cache-dependency-path: web/gi-kg-viewer/package-lock.json

	- name: Install Python venv + project
	run: \|
	python -m venv .venv
	.venv/bin/python -m pip install --upgrade pip
	# ``[search]`` = sentence-transformers + lancedb + torch — the
	# minimum needed by the indexing step below. ``[dev]`` adds the
	# FastAPI / uvicorn stack the in-process viewer API needs. We
	# deliberately do NOT install ``[ml]`` (whisper / spaCy / etc.) —
	# tier-3 doesn't transcribe or summarise; it only indexes for
	# the digest/search lookup.
	.venv/bin/python -m pip install -e ".[search,dev]"

	- name: Install ffmpeg (torchcodec — pulled in via [search]'s torch — needs FFmpeg shared libs to import)
	run: \|
	# The synthetic-corpus build + indexing import modules that transitively load
	# torchcodec, which auto-loads libavutil/libtorchcodec on import. Without ffmpeg
	# that import raises "Could not load libtorchcodec" and the job dies (it does not
	# decode audio — the import just needs the shared libs present, same as the other
	# nightly ml jobs).
	sudo apt-get update
	sudo apt-get install -y --no-install-recommends ffmpeg

	- name: Install viewer deps
	working-directory: web/gi-kg-viewer
	run: npm ci

	- name: Install Playwright browsers (chromium)
	working-directory: web/gi-kg-viewer
	run: node_modules/.bin/playwright install --with-deps chromium

	- name: Regenerate synthetic validation corpus
	run: \|
	# Deterministic + idempotent. Confirms the script still runs
	# against the latest text fixtures; if the checked-in corpus
	# has drifted from the script output, the diff will surface.
	.venv/bin/python scripts/build_synthetic_validation_corpus.py

	- name: Cache HuggingFace hub (sentence-transformers MiniLM-L6-v2 ~80MB)
	uses: actions/cache@v6
	with:
	path: ~/.cache/huggingface
	key: hf-hub-minilm-l6-v2
	restore-keys: \|
	hf-hub-

	- name: Preload sentence-transformers embedding model
	env:
	# Skip whisper / spaCy / transformers preload — they aren't
	# installed under ``[search,dev]`` and tier-3 doesn't need them.
	# SKIP_GIL stays unset because the GIL evidence stack includes
	# the embedding model (MiniLM-L6-v2) that the LanceDB index build
	# (``cli index-two-tier``, via ``make build-validation-index``) requires.
	SKIP_WHISPER: "1"
	SKIP_SPACY: "1"
	SKIP_TRANSFORMERS: "1"
	run: \|
	# ``index_corpus`` runs with allow_download=False — the model
	# MUST be cached before indexing. preload_ml_models.py is the
	# production way to populate the HF cache for embedding-only use.
	.venv/bin/python scripts/cache/preload_ml_models.py

	- name: Build validation search index + topic clusters (V2/V3/V4/V6 prereq)
	run: \|
	# LanceDB two-tier index (search/lance_index/) + topic_clusters.json
	# under <corpus>/<FIXTURES_VERSION>/search/. Prereq for the digest
	# topic-bands (V2 / P1.3 / P4.2), the dashboard topic-cluster chip
	# (V4) and semantic search (V3 / V6). ``make build-validation-index``
	# is the single source of truth: it resolves the versioned corpus
	# path from tests/fixtures/FIXTURES_VERSION, so it tracks the v2→v3
	# fixture migration (#1148) automatically. The prior hand-rolled
	# ``cli index`` / ``cli topic-clusters`` calls hard-coded the
	# version-less parent, which post-migration walks 0 episodes → empty
	# bands → every index-dependent Tier-3 spec fails.
	make build-validation-index

	- name: Start serve-for-validation (API + UI) in background
	run: \|
	# Background the serve so subsequent steps can hit it. ``make
	# serve-for-validation`` runs ``serve-api`` + ``serve-ui`` in
	# parallel with ``SERVE_OUTPUT_DIR=$PWD`` so the synthetic
	# corpus at ``tests/fixtures/viewer-validation-corpus`` is
	# reachable via the API's path-allowlist check.
	nohup make serve-for-validation > tier3-serve.log 2>&1 &
	echo $! > tier3-serve.pid

	- name: Wait for API + UI to come up
	run: \|
	for i in $(seq 1 60); do
	api=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/api/health \|\| echo 000)
	ui=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:5173/ \|\| echo 000)
	if [ "$api" = "200" ] && [ "$ui" = "200" ]; then
	echo "API + UI ready after ${i}s"
	exit 0
	fi
	sleep 2
	done
	echo "ERROR: serve did not come up within 120s"
	cat tier3-serve.log \|\| true
	exit 1

	- name: Run Tier-3 validation walk
	id: tier3
	working-directory: web/gi-kg-viewer
	run: \|
	# CORPUS_PATH must include the FIXTURES_VERSION subdir — the raw
	# feeds/<feed>/metadata artifacts live under it, and pointing the
	# walk at the version-less parent discovers 0 episodes (#1148).
	VVC="${{ github.workspace }}/tests/fixtures/viewer-validation-corpus/$(cat ../../tests/fixtures/FIXTURES_VERSION)"
	CORPUS_PATH="$VVC" \
	node_modules/.bin/playwright test --config playwright.validation.config.ts \
	--reporter=list 2>&1 \| tee tier3-output.log
	continue-on-error: true

	- name: Stop serve
	if: always()
	run: \|
	if [ -f tier3-serve.pid ]; then
	kill "$(cat tier3-serve.pid)" 2>/dev/null \|\| true
	fi

	- name: Upload artifacts on failure
	if: steps.tier3.outcome == 'failure'
	uses: actions/upload-artifact@v4
	with:
	name: tier3-validation-failures
	path: \|
	web/gi-kg-viewer/validation-results/
	web/gi-kg-viewer/tier3-output.log
	tier3-serve.log
	retention-days: 30

	- name: File auto-issue on regression
	if: steps.tier3.outcome == 'failure' && github.event_name == 'schedule'
	uses: actions/github-script@v9
	with:
	script: \|
	const { owner, repo } = context.repo;
	const date = new Date().toISOString().slice(0, 10);
	const title = `Tier-3 validation regression on ${date}`;
	// Search for an existing open issue with this label to avoid
	// duplicate filings if the failure persists across nights.
	const existing = await github.rest.issues.listForRepo({
	owner, repo, state: 'open', labels: 'tier3-regression',
	per_page: 5,
	});
	if (existing.data.length > 0) {
	const issueNumber = existing.data[0].number;
	await github.rest.issues.createComment({
	owner, repo, issue_number: issueNumber,
	body: `Tier-3 validation failed again on ${date} (run ${context.runId}). ` +
	`[See logs](https://github.com/${owner}/${repo}/actions/runs/${context.runId}).`,
	});
	return;
	}
	await github.rest.issues.create({
	owner, repo, title,
	labels: ['tier3-regression', 'bug', 'viewer'],
	body: [
	`Tier-3 viewer validation walk failed on the scheduled nightly run.`,
	'',
	`Date: ${date}`,
	`Run: [${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})`,
	`Corpus: \`tests/fixtures/viewer-validation-corpus/<FIXTURES_VERSION>\` (synthetic, in-repo)`,
	'',
	'## What this catches',
	'',
	'- V1 — Library row "Open in graph" full 6-point contract',
	'- V2 — Digest topic pill "Open in graph" (search-backed)',
	'- V3 — Search "Show on graph" (search-backed)',
	'- V4 — Dashboard topic-cluster chip',
	'- V5 — Hot-state Library → Library supersession',
	'- + 24 expanded matrix rows in handoff-matrix-real-corpus.spec.ts',
	'',
	'V2 / V4 prerequisites are built in-CI (search index +',
	'``topic_clusters.json``) before the validation walk. See',
	'``tests/fixtures/VIEWER_VALIDATION_CORPUS.md``.',
	'',
	'## Next steps',
	'',
	'Per RFC-086 institutional rule, the fix PR must add a',
	'Tier-2 matrix row under `web/gi-kg-viewer/e2e/handoff-production/`',
	'reproducing the regression before merge.',
	'',
	'## Logs',
	'',
	`- Download artifact: \`tier3-validation-failures\` from the run`,
	`- Local repro: \`make build-validation-index\` (builds the search index +`,
	` topic clusters and prints the exact \`make ci-ui-validation CORPUS=…\``,
	` command with the versioned corpus path); then \`make serve-for-validation\``,
	` (terminal 1) + that printed command (terminal 2).`,
	'',
	'References: RFC-086, ADR-095, #774.',
	].join('\n'),
	});

	- name: Fail the workflow if validation failed
	if: steps.tier3.outcome == 'failure'
	run: \|
	echo "Tier-3 validation regressed. See logs + issue."
	exit 1

	# ===========================================================================
	# Tier-3 consumer-app validation (mirrors nightly-tier3-validation for the
	# viewer). Runs the real-backend Playwright walk against the committed
	# app-validation-corpus/v3 fixture — screenshots + logs upload as an
	# artifact, and a deduped ``tier3-app-regression`` issue is auto-filed
	# on scheduled failures.
	#
	# Separate from ``app-e2e`` in python-app.yml: that's the fast per-PR
	# gate against a preview build. This is the drift detector against a
	# production-shape serve, running sequentially with screenshots on
	# every step for post-hoc inspection.
	# ===========================================================================
	nightly-tier3-app-validation:
	runs-on: ubuntu-latest
	timeout-minutes: 30
	needs: [nightly-lint, nightly-build]
	if: \|
	always() &&
	needs.nightly-lint.result == 'success' &&
	needs.nightly-build.result == 'success'
	permissions:
	contents: read
	issues: write
	steps:
	- uses: actions/checkout@v7

	- name: Set up Python 3.11 (for the real API server)
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Set up Node
	uses: actions/setup-node@v7
	with:
	node-version: "22"
	cache: "npm"
	cache-dependency-path: web/learning-player/package-lock.json

	- name: Install podcast_scraper (for ``podcast_scraper serve``)
	run: \|
	python -m venv .venv
	.venv/bin/pip install --upgrade pip
	# [dev] pulls FastAPI + uvicorn — same install shape as app-e2e.
	# NO [ml] — Tier-3 walks a served corpus, it doesn't index.
	.venv/bin/pip install -e ".[dev]"

	- name: Install app dependencies
	working-directory: web/learning-player
	run: npm ci

	- name: Install Playwright browsers (chromium)
	working-directory: web/learning-player
	run: npx playwright install --with-deps chromium

	- name: Build the app (prod bundle for the preview server)
	working-directory: web/learning-player
	run: npm run build

	- name: Start API server against the app-validation-corpus
	run: \|
	# Background the API so subsequent steps can hit it. The
	# committed app-validation-corpus/v3 is the default; an
	# operator-driven run overrides via APP_CORPUS_PATH env.
	nohup .venv/bin/python -m podcast_scraper.cli serve \
	--output-dir tests/fixtures/app-validation-corpus/v3 \
	--port 8000 --host 127.0.0.1 \
	> tier3-app-api.log 2>&1 &
	echo $! > tier3-app-api.pid
	env:
	APP_OAUTH_PROVIDER: mock
	APP_SESSION_SECRET: tier3-app-secret
	APP_SIGNUP_MODE: open
	APP_DATA_DIR: /tmp/tier3-app-data

	- name: Start app preview on :5175
	working-directory: web/learning-player
	run: \|
	nohup npm run preview -- --port 5175 --strictPort --host 127.0.0.1 \
	> ../tier3-app-preview.log 2>&1 &
	echo $! > ../tier3-app-preview.pid
	env:
	VITE_API_TARGET: http://127.0.0.1:8000

	- name: Wait for API + app preview to come up
	run: \|
	for i in $(seq 1 60); do
	api=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/api/health \|\| echo 000)
	ui=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:5175/ \|\| echo 000)
	if [ "$api" = "200" ] && [ "$ui" = "200" ]; then
	echo "API + app preview ready after ${i}s"
	exit 0
	fi
	sleep 2
	done
	echo "ERROR: services did not come up within 120s"
	cat tier3-app-api.log tier3-app-preview.log \|\| true
	exit 1

	- name: Run Tier-3 app validation walk
	id: tier3app
	working-directory: web/learning-player
	run: \|
	npx playwright test --config playwright.validation.config.ts \
	--reporter=list 2>&1 \| tee ../tier3-app-output.log
	continue-on-error: true

	- name: Stop background servers
	if: always()
	run: \|
	if [ -f tier3-app-api.pid ]; then
	kill "$(cat tier3-app-api.pid)" 2>/dev/null \|\| true
	fi
	if [ -f tier3-app-preview.pid ]; then
	kill "$(cat tier3-app-preview.pid)" 2>/dev/null \|\| true
	fi

	- name: Upload artifacts on failure
	if: steps.tier3app.outcome == 'failure'
	uses: actions/upload-artifact@v4
	with:
	name: tier3-app-validation-failures
	path: \|
	web/learning-player/validation-results/
	tier3-app-output.log
	tier3-app-api.log
	tier3-app-preview.log
	retention-days: 30

	- name: File auto-issue on regression
	if: steps.tier3app.outcome == 'failure' && github.event_name == 'schedule'
	uses: actions/github-script@v9
	with:
	script: \|
	const { owner, repo } = context.repo;
	const date = new Date().toISOString().slice(0, 10);
	const title = `Tier-3 app validation regression on ${date}`;
	// Dedup: comment on existing open ``tier3-app-regression`` issue
	// rather than filing a new one.
	const existing = await github.rest.issues.listForRepo({
	owner, repo, state: 'open', labels: 'tier3-app-regression',
	per_page: 5,
	});
	if (existing.data.length > 0) {
	const issueNumber = existing.data[0].number;
	await github.rest.issues.createComment({
	owner, repo, issue_number: issueNumber,
	body: `Tier-3 app validation failed again on ${date} (run ${context.runId}). ` +
	`[See logs](https://github.com/${owner}/${repo}/actions/runs/${context.runId}).`,
	});
	return;
	}
	await github.rest.issues.create({
	owner, repo, title,
	labels: ['tier3-app-regression', 'bug', 'app'],
	body: [
	`Tier-3 consumer-app validation walk failed on the scheduled nightly run.`,
	'',
	`Date: ${date}`,
	`Run: [${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})`,
	`Corpus: \`tests/fixtures/app-validation-corpus/v3\` (committed synthetic)`,
	'',
	'Screenshots + logs are attached to the run under the',
	'`tier3-app-validation-failures` artifact.',
	].join('\n'),
	});

	- name: Fail the workflow if validation failed
	if: steps.tier3app.outcome == 'failure'
	run: \|
	echo "Tier-3 app validation regressed. See logs + issue."
	exit 1

	# ===========================================================================
	# Nightly-only tests - production models, full podcast suite
	# Only runs after unit, integration, E2E, and viewer tests pass (expensive, run as final stage)
	# ===========================================================================
	nightly-only-tests:
	runs-on: ubuntu-latest
	timeout-minutes: 120 # ~75 min typical sequential run + buffer
	needs:
	- nightly-test-unit
	- nightly-test-integration
	- nightly-test-e2e
	- nightly-viewer-unit
	- nightly-viewer-e2e
	env:
	HF_HOME: /home/runner/.cache/huggingface
	HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
	ML_MODELS_VALIDATED: "true"
	# Nightly mode uses p01-p05 podcasts with production models
	E2E_TEST_MODE: "nightly"
	steps:
	- uses: actions/checkout@v7

	- name: Free disk space
	run: \|
	sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
	docker image prune -af \|\| true
	sudo apt-get clean
	sudo rm -rf /var/lib/apt/lists/*
	df -h

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Download ML models from preload job
	uses: actions/download-artifact@v8
	with:
	name: ml-models
	path: /home/runner/.cache

	- name: Install dependencies
	run: \|
	python -m pip install --upgrade pip
	pip install -e ".[dev,ml,llm,search]"
	pip install pytest-socket pytest-json-report

	- name: Install ffmpeg
	run: \|
	sudo apt-get update
	sudo apt-get install -y --no-install-recommends ffmpeg

	- name: Create reports directory
	run: mkdir -p reports

	- name: Verify test collection before running
	run: \|
	PYTHONPATH="${PYTHONPATH}:$(pwd)"
	export PYTHONPATH
	echo "🔍 Verifying nightly test collection..."
	pytest tests/e2e/ -m "nightly and not llm" --collect-only -q \|\| {
	echo "❌ Test collection failed!"
	pytest tests/e2e/ -m "nightly and not llm" --collect-only -v
	exit 1
	}
	echo "✅ Test collection successful"

	- name: Run nightly-only tests with production models (sequential)
	timeout-minutes: 100 # ~75 min typical + buffer
	run: \|
	PYTHONPATH="${PYTHONPATH}:$(pwd)"
	export PYTHONPATH
	# Uses production models: Whisper base.en, BART-large-cnn, LED-large-16384
	# Processes all 15 episodes across 5 podcasts (p01-p05)
	echo "🚀 Starting nightly tests at $(date)"
	make test-nightly \|\| {
	EXIT_CODE=$?
	echo "⚠️ Nightly tests failed with exit code $EXIT_CODE at $(date)"
	# Try to collect any partial reports
	if [ -d reports ]; then
	echo "📊 Available reports:"
	ls -lah reports/ \|\| true
	fi
	exit $EXIT_CODE
	}
	echo "✅ Nightly tests completed at $(date)"
	continue-on-error: true

	- name: Upload nightly-only test reports
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: nightly-only-reports
	path: reports/
	retention-days: 30

	# ===========================================================================
	# Metrics collection and dashboard generation
	# ===========================================================================
	nightly-metrics:
	runs-on: ubuntu-latest
	timeout-minutes: 30
	needs:
	- nightly-lint
	- nightly-security-quality
	- nightly-build
	- nightly-test-unit
	- nightly-viewer-unit
	- nightly-test-integration
	- nightly-test-e2e
	- nightly-viewer-e2e
	- nightly-tier3-validation
	- nightly-only-tests
	if: always()
	steps:
	- uses: actions/checkout@v7
	with:
	fetch-depth: 0

	- name: Set up Python 3.11
	uses: actions/setup-python@v7
	with:
	python-version: "3.11.8"
	cache: "pip"
	cache-dependency-path: pyproject.toml

	- name: Install jq (for JSON parsing in job summary)
	run: \|
	sudo apt-get update
	sudo apt-get install -y --no-install-recommends jq

	- name: Install minimal dependencies
	run: \|
	python -m pip install --upgrade pip
	# Install coverage for combining reports
	# Install dev tools needed for code quality metrics (radon, interrogate, vulture, codespell)
	# Scripts (generate_metrics.py, generate_dashboard.py) use only stdlib
	pip install coverage[toml]
	pip install radon interrogate vulture codespell

	- name: Create directories
	run: \|
	mkdir -p reports
	mkdir -p metrics

	- name: Download all test reports
	uses: actions/download-artifact@v8
	with:
	path: downloaded-reports
	pattern: nightly-*-reports
	merge-multiple: false
	continue-on-error: true # Some jobs may have failed and not uploaded artifacts

	- name: Merge test reports
	run: \|
	# Copy all reports to a single directory (if any exist)
	if [ -d downloaded-reports ]; then
	find downloaded-reports -name "*.json" -exec cp {} reports/ \; 2>/dev/null \|\| true
	find downloaded-reports -name "*.xml" -exec cp {} reports/ \; 2>/dev/null \|\| true
	fi
	ls -la reports/ \|\| echo "No reports directory yet"

	# Merge pytest JSON reports if multiple exist
	if ls reports/pytest-*.json 1>/dev/null 2>&1; then
	echo "Merging pytest JSON reports..."
	python3 << 'EOF'
	import json
	from pathlib import Path

	json_files = list(Path('reports').glob('pytest-*.json'))
	if not json_files:
	print("No pytest JSON files found")
	exit(0)

	merged_summary = {"total": 0, "passed": 0, "failed": 0, "skipped": 0}
	merged_tests = []
	total_duration = 0.0

	for json_file in json_files:
	try:
	with open(json_file) as f:
	data = json.load(f)
	summary = data.get("summary", {})
	merged_summary["total"] += summary.get("total", 0)
	merged_summary["passed"] += summary.get("passed", 0)
	merged_summary["failed"] += summary.get("failed", 0)
	merged_summary["skipped"] += summary.get("skipped", 0)
	total_duration += data.get("duration", 0)
	merged_tests.extend(data.get("tests", []))
	print(f" ✅ {json_file}")
	except Exception as e:
	print(f" ⚠️ {json_file}: {e}")

	merged_data = {
	"summary": merged_summary,
	"duration": total_duration,
	"tests": merged_tests
	}

	with open('reports/pytest.json', 'w') as f:
	json.dump(merged_data, f, indent=2)
	print(f"✅ Merged pytest JSON: {merged_summary['total']} tests, {total_duration:.1f}s")
	EOF
	fi

	# Merge JUnit XML reports if multiple exist
	if ls reports/junit-*.xml 1>/dev/null 2>&1; then
	echo "Merging JUnit XML reports..."
	python3 << 'EOF'
	import xml.etree.ElementTree as ET
	from pathlib import Path

	xml_files = list(Path('reports').glob('junit-*.xml'))
	if not xml_files:
	print("No JUnit XML files found")
	exit(0)

	# Create base testsuites element
	root = ET.Element('testsuites')
	root.set('tests', '0')
	root.set('failures', '0')
	root.set('time', '0')

	total_tests = 0
	total_failures = 0
	total_time = 0.0

	for xml_file in xml_files:
	try:
	tree = ET.parse(xml_file)
	file_root = tree.getroot()
	# Handle both testsuites and testsuite elements
	for testsuite in file_root.findall('.//testsuite'):
	root.append(testsuite)
	total_tests += int(testsuite.get('tests', 0))
	total_failures += int(testsuite.get('failures', 0))
	total_time += float(testsuite.get('time', 0))
	print(f" ✅ {xml_file}")
	except Exception as e:
	print(f" ⚠️ {xml_file}: {e}")

	root.set('tests', str(total_tests))
	root.set('failures', str(total_failures))
	root.set('time', str(total_time))

	tree = ET.ElementTree(root)
	tree.write('reports/junit.xml', encoding='utf-8', xml_declaration=True)
	print(f"✅ Merged JUnit XML: {total_tests} tests, {total_failures} failures, {total_time:.1f}s")
	EOF
	fi

	# Properly merge coverage reports from all test jobs
	if ls reports/coverage-*.xml 1>/dev/null 2>&1; then
	echo "Merging coverage reports..."
	python3 << 'EOF'
	import xml.etree.ElementTree as ET
	from pathlib import Path

	def parse_coverage_xml(filepath):
	tree = ET.parse(filepath)
	root = tree.getroot()
	coverage_data = {}
	for package in root.findall('.//package'):
	for cls in package.findall('.//class'):
	filename = cls.get('filename', '')
	if not filename:
	continue
	if filename not in coverage_data:
	coverage_data[filename] = {'hits': set(), 'misses': set()}
	for line in cls.findall('.//line'):
	line_num = int(line.get('number', 0))
	hits = int(line.get('hits', 0))
	if hits > 0:
	coverage_data[filename]['hits'].add(line_num)
	coverage_data[filename]['misses'].discard(line_num)
	elif line_num not in coverage_data[filename]['hits']:
	coverage_data[filename]['misses'].add(line_num)
	return coverage_data, root

	xml_files = list(Path('reports').glob('coverage-*.xml'))
	if not xml_files:
	print("No coverage files found")
	exit(0)

	print(f"Merging {len(xml_files)} coverage files...")
	merged_coverage = {}
	base_root = None

	for xml_file in xml_files:
	try:
	coverage_data, root = parse_coverage_xml(xml_file)
	if base_root is None:
	base_root = root
	for filename, data in coverage_data.items():
	if filename not in merged_coverage:
	merged_coverage[filename] = {'hits': set(), 'misses': set()}
	merged_coverage[filename]['hits'].update(data['hits'])
	for miss in data['misses']:
	if miss not in merged_coverage[filename]['hits']:
	merged_coverage[filename]['misses'].add(miss)
	print(f" ✅ {xml_file}")
	except Exception as e:
	print(f" ⚠️ {xml_file}: {e}")

	total_lines = sum(len(d['hits']) + len(d['misses']) for d in merged_coverage.values())
	covered_lines = sum(len(d['hits']) for d in merged_coverage.values())
	coverage_pct = (covered_lines / total_lines * 100) if total_lines > 0 else 0
	print(f"📈 Unified: {coverage_pct:.1f}% ({covered_lines}/{total_lines})")

	if base_root is not None:
	base_root.set('line-rate', str(covered_lines / total_lines if total_lines > 0 else 0))
	ET.ElementTree(base_root).write('reports/coverage.xml', encoding='unicode', xml_declaration=True)
	print("✅ Written to reports/coverage.xml")
	EOF
	fi

	- name: Load metrics history
	env:
	GITHUB_REPOSITORY: ${{ github.repository }}
	run: \|
	mkdir -p metrics
	# Prefer live GitHub Pages URL: deploy-pages often does not update the gh-pages git ref,
	# so "git show gh-pages:metrics/..." stays empty and history never accumulates across runs.
	bash scripts/dashboard/fetch_metrics_file_from_pages.sh history-nightly.jsonl metrics/history-nightly.jsonl
	if [ -s metrics/history-nightly.jsonl ]; then
	python scripts/dashboard/repair_metrics_jsonl.py metrics/history-nightly.jsonl --in-place \|\| true
	fi

	- name: Generate code quality metrics
	run: \|
	mkdir -p reports
	# Generate complexity metrics for dashboard
	radon cc src/podcast_scraper/ -a -s --total-average --json > reports/complexity.json 2>/dev/null \|\| echo '{"total_average": 0}' > reports/complexity.json
	radon mi src/podcast_scraper/ -s --json > reports/maintainability.json 2>/dev/null \|\| echo '[]' > reports/maintainability.json
	# Interrogate 1.7+ / vulture 2.x: use capture script (no invalid --json flags; no \|\| overwrite)
	python scripts/dashboard/capture_quality_for_metrics.py --reports-dir reports --vulture-min-confidence 60

	- name: Create JUnit XML fallback
	run: \|
	if [ ! -f reports/junit.xml ]; then
	echo "⚠️ No JUnit XML found, creating minimal fallback"
	python3 -c "import xml.etree.ElementTree as ET; root = ET.Element('testsuites'); root.set('tests', '0'); root.set('failures', '0'); root.set('time', '0'); tree = ET.ElementTree(root); tree.write('reports/junit.xml', encoding='utf-8', xml_declaration=True)"
	fi

	- name: Create pytest JSON fallback
	run: \|
	if [ ! -f reports/pytest.json ]; then
	echo "⚠️ No pytest JSON found, creating minimal fallback"
	echo '{"summary": {"total": 0, "passed": 0, "failed": 0, "skipped": 0}, "duration": 0, "tests": []}' > reports/pytest.json
	fi

	- name: Collect pipeline metrics
	run: \|
	python scripts/dashboard/collect_pipeline_metrics.py \
	--output reports/output/pipeline_metrics.json \
	--max-episodes 1 \|\| echo "⚠️ Pipeline metrics collection failed (non-blocking)"
	continue-on-error: true

	- name: Generate metrics JSON
	run: \|
	python scripts/dashboard/generate_metrics.py \
	--reports-dir reports \
	--output metrics/latest-nightly.json \
	--history metrics/history-nightly.jsonl \
	--pipeline-metrics reports/output/pipeline_metrics.json \
	--coverage-threshold 70 \
	--slowest-top-n 10

	- name: Validate metrics JSON
	run: \|
	if [ ! -f metrics/latest-nightly.json ]; then
	echo "❌ Error: metrics/latest-nightly.json not found"
	exit 1
	fi
	python3 -c "import json, sys; json.load(open('metrics/latest-nightly.json'))" \|\| exit 1
	echo "✅ Metrics JSON is valid"

	- name: Debug metrics files
	if: failure()
	run: \|
	echo "=== Metrics Files ==="
	if [ -d metrics ]; then
	find metrics -maxdepth 1 -printf '%M %u %s %TF %TR %f\n'
	else
	echo "No metrics directory"
	fi
	echo ""
	echo "=== Latest JSON (first 30 lines) ==="
	head -30 metrics/latest-nightly.json \|\| echo "No latest-nightly.json"
	echo ""
	echo "=== History file (line count) ==="
	wc -l metrics/history-nightly.jsonl \|\| echo "No history-nightly.jsonl"
	echo ""
	echo "=== Reports directory ==="
	if [ -d reports ]; then
	find reports -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' \| head -20
	else
	echo "No reports directory"
	fi

	- name: Update metrics history
	run: \|
	if [ -f metrics/latest-nightly.json ]; then
	if [ -s metrics/history-nightly.jsonl ]; then
	python scripts/dashboard/repair_metrics_jsonl.py metrics/history-nightly.jsonl --in-place \|\| true
	fi
	python scripts/dashboard/append_metrics_history_line.py metrics/latest-nightly.json >> metrics/history-nightly.jsonl
	echo "✅ Appended to nightly history (total lines: $(wc -l < metrics/history-nightly.jsonl))"
	fi

	- name: Generate unified HTML dashboard
	# Generate dashboard HTML for schedule and manual dispatch runs
	# The unified dashboard has a dropdown to select which build to view
	# (Metrics for release branches are handled by regular CI, not nightly)
	env:
	GITHUB_REPOSITORY: ${{ github.repository }}
	run: \|
	bash scripts/dashboard/fetch_metrics_file_from_pages.sh latest-ci.json metrics/latest-ci.json
	if [ ! -s metrics/latest-ci.json ]; then rm -f metrics/latest-ci.json; fi
	bash scripts/dashboard/fetch_metrics_file_from_pages.sh history-ci.jsonl metrics/history-ci.jsonl
	python scripts/dashboard/generate_dashboard.py \
	--unified \
	--output metrics/index.html
	python scripts/dashboard/consolidate_dashboard_data.py \
	--input-dir metrics \
	--output metrics/dashboard-data.json

	- name: Generate job summary
	run: \|
	{
	echo "# 📊 Nightly Test Results"
	echo ""
	# Job status summary
	echo "## Job Status"
	echo "- Lint: ${{ needs.nightly-lint.result }}"
	echo "- Security/Quality: ${{ needs.nightly-security-quality.result }}"
	echo "- Build: ${{ needs.nightly-build.result }}"
	echo "- Docs: Runs after metrics (no dependency to avoid cycle)"
	echo "- Unit Tests: ${{ needs.nightly-test-unit.result }}"
	echo "- Integration Tests: ${{ needs.nightly-test-integration.result }}"
	echo "- E2E Tests: ${{ needs.nightly-test-e2e.result }}"
	echo "- Nightly-Only Tests: ${{ needs.nightly-only-tests.result }}"
	echo ""
	} >> "$GITHUB_STEP_SUMMARY"

	# Extract metrics from pytest JSON reports
	for report in reports/pytest-*.json; do
	if [ -f "$report" ]; then
	TOTAL=$(jq -r '.summary.total // 0' "$report" 2>/dev/null \|\| echo "0")
	PASSED=$(jq -r '.summary.passed // 0' "$report" 2>/dev/null \|\| echo "0")
	FAILED=$(jq -r '.summary.failed // 0' "$report" 2>/dev/null \|\| echo "0")
	SKIPPED=$(jq -r '.summary.skipped // 0' "$report" 2>/dev/null \|\| echo "0")
	DURATION=$(jq -r '.duration // 0' "$report" 2>/dev/null \|\| echo "0")
	REPORT_NAME=$(basename "$report" .json)

	{
	echo "## Test Summary ($REPORT_NAME)"
	echo "- Total Tests: $TOTAL"
	echo "- Passed: ✅ $PASSED"
	echo "- Failed: ❌ $FAILED"
	echo "- Skipped: ⏭️ $SKIPPED"
	echo "- Duration: ${DURATION}s"
	echo ""
	} >> "$GITHUB_STEP_SUMMARY"
	fi
	done

	# Extract coverage from coverage.xml
	if [ -f reports/coverage.xml ]; then
	COVERAGE=$(python -c "import xml.etree.ElementTree as ET; tree = ET.parse('reports/coverage.xml'); root = tree.getroot(); print(f\"{float(root.attrib.get('line-rate', 0)) * 100:.1f}%\")" 2>/dev/null \|\| echo "N/A")
	{
	echo "## Coverage"
	echo "- Overall Coverage: $COVERAGE"
	echo ""
	} >> "$GITHUB_STEP_SUMMARY"
	fi

	# Extract slowest tests from JUnit XML reports
	for junit in reports/pytest-*.json; do
	if [ -f "$junit" ]; then
	echo "## Slowest Tests (Top 10)" >> "$GITHUB_STEP_SUMMARY"
	echo '```' >> "$GITHUB_STEP_SUMMARY"
	python3 -c "
	import json, sys
	try:
	with open('$junit') as f:
	data = json.load(f)
	tests = [(t.get('duration', 0), t.get('nodeid', 'unknown')) for t in data.get('tests', [])]
	tests.sort(reverse=True)
	for time, name in tests[:10]:
	print(f'{time:.2f}s - {name}')
	except Exception as e:
	print(f'Error: {e}', file=sys.stderr)
	" >> "$GITHUB_STEP_SUMMARY" 2>&1 \|\| echo "Unable to parse" >> "$GITHUB_STEP_SUMMARY"
	echo '```' >> "$GITHUB_STEP_SUMMARY"
	break # Only show once
	fi
	done

	# Check for flaky tests (tests that passed on rerun)
	for report in reports/pytest-*.json; do
	if [ -f "$report" ]; then
	# Match generate_metrics.py: pytest-json-report uses outcome=="rerun" + call.passed
	FLAKY_COUNT=$(jq -r '[.tests[] \| select((.outcome == "rerun" and .call.outcome == "passed") or (.outcome == "passed" and .rerun == true))] \| length' "$report" 2>/dev/null \|\| echo "0")
	if [ "${FLAKY_COUNT:-0}" -gt 0 ]; then
	{
	echo ""
	echo "## ⚠️ Flaky Tests Detected"
	echo "- Count: $FLAKY_COUNT tests passed on rerun"
	echo ""
	echo "### Flaky Test Names:"
	echo '```'
	jq -r '.tests[] \| select((.outcome == "rerun" and .call.outcome == "passed") or (.outcome == "passed" and .rerun == true)) \| "\(.nodeid)"' "$report" 2>/dev/null \|\| echo "Unable to extract"
	echo '```'
	} >> "$GITHUB_STEP_SUMMARY"
	fi
	fi
	done

	# Show alerts from metrics (if available)
	if [ -f metrics/latest-nightly.json ]; then
	ALERT_COUNT=$(jq -r '.alerts \| length' metrics/latest-nightly.json 2>/dev/null \|\| echo "0")
	if [ "$ALERT_COUNT" -gt 0 ]; then
	echo "" >> "$GITHUB_STEP_SUMMARY"
	echo "## 🚨 Metric Alerts" >> "$GITHUB_STEP_SUMMARY"
	jq -r '.alerts[] \|
	if .severity == "error" then "🔴 ERROR: \(.message)"
	elif .severity == "warning" then "⚠️ WARNING: \(.message)"
	else "ℹ️ INFO: \(.message)"
	end' metrics/latest-nightly.json 2>/dev/null >> "$GITHUB_STEP_SUMMARY" \|\| echo "Unable to extract alerts"
	fi
	fi

	- name: Upload combined test reports (extended retention for trend tracking)
	uses: actions/upload-artifact@v7
	with:
	name: nightly-test-reports
	path: \|
	reports/
	retention-days: 90

	- name: Upload metrics artifact
	uses: actions/upload-artifact@v7
	with:
	name: nightly-metrics
	path: metrics/
	retention-days: 90

	- name: Post-build cleanup
	if: always()
	run: rm -rf .pytest_cache .mypy_cache .build dist downloaded-reports

	# NOTE: Metrics deployment strategy:
	# - Schedule/manual runs: Generate metrics data + unified dashboard HTML
	# - Regular CI (python-app.yml): Handles metrics for main and release branches
	# - Nightly: Comprehensive testing and metrics on main branch only (via schedule)
	# - The docs.yml workflow deploys everything to GitHub Pages
	# - Nightly metrics are available as workflow artifacts for download
	# - The unified dashboard uses a dropdown to select CI vs Nightly builds

Navigation Menu

Nightly Comprehensive Tests #301

Workflow file

Nightly Comprehensive Tests #301

Uh oh!

Workflow file for this run