Skip to content
Python application

feat: autoresearch batch 2 — Opus silver + per-model tuning + G-Eval finale #1100

Sign in to view logs

feat: autoresearch batch 2 — Opus silver + per-model tuning + G-Eval finale

feat: autoresearch batch 2 — Opus silver + per-model tuning + G-Eval finale #1100

Workflow file for this run

.github/workflows/python-app.yml at d08a9ae
# This workflow installs Python dependencies, runs tests and quality checks.
#
# Dependency flow (path-A split — see ``detect-changes.python`` / ``.viewer``):
#
# Segment 1 — bootstrap (parallel, no cross-deps):
# lint, build, detect-changes, preload-ml-models
# ``preload-ml-models`` skips on viewer-only / docs-only PRs.
#
# Segment 2 — fast checks (parallel, after lint+build):
# security-quality (Python), test-unit (pytest), viewer-unit (Vitest)
# Python-gated jobs (test-unit, security-quality) skip on viewer-only PRs;
# viewer-unit always runs when viewer changes (always cheap).
#
# Segment 3 — heavy Python tests (parallel, after Segment 2):
# test-integration / test-integration-fast / test-e2e / test-e2e-fast
# Gate on test-unit + preload-ml-models for fail-fast; skip on viewer-only.
#
# Segment 4 — fixture-driven post-unit tests (parallel with Segment 3):
# viewer-e2e (Playwright; gates on viewer-unit, no Python deps),
# test-acceptance-fixtures (push-to-main only; gates on test-unit).
#
# Segment 5 — finalisation:
# coverage-unified (gates on Python tests + viewer-e2e; skips on viewer-only),
# docs (gates on coverage-unified + viewer-unit + viewer-e2e; tolerates
# coverage-unified=skipped so docs still validates on viewer-only PRs).
#
# Path-A summary: on viewer-only PRs only viewer-unit + viewer-e2e + lint +
# build + docs run (~5-8 min wallclock). On Python-only PRs viewer-unit /
# viewer-e2e still run (Vitest is ~30 s, Playwright ~5 min) but they're
# independent of Python results. On PRs touching both, everything runs.
#
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
name: Python application
on:
push:
branches: [ "main", "release/2.4", "release/2.5", "release/2.6" ]
paths:
- '**.py'
- '**.j2'
- 'tests/**'
- 'pyproject.toml'
- 'Makefile'
- 'docker/pipeline/Dockerfile'
- 'docker/api/**'
- 'docker/viewer/**'
- '.dockerignore'
- 'web/gi-kg-viewer/**'
- 'config/acceptance/MAIN_ACCEPTANCE_CONFIG.yaml'
- 'config/acceptance/fragments/**'
- 'config/profiles/**'
- 'compose/**'
- 'scripts/tools/validate_profile_docker_tier.py'
- '.github/workflows/python-app.yml'
pull_request:
branches: [ "main", "release/2.4", "release/2.5", "release/2.6" ]
paths:
- '**.py'
- '**.j2'
- 'tests/**'
- 'pyproject.toml'
- 'Makefile'
- 'docker/pipeline/Dockerfile'
- 'docker/api/**'
- 'docker/viewer/**'
- '.dockerignore'
- 'web/gi-kg-viewer/**'
- 'config/acceptance/MAIN_ACCEPTANCE_CONFIG.yaml'
- 'config/acceptance/fragments/**'
- 'config/profiles/**'
- 'compose/**'
- 'scripts/tools/validate_profile_docker_tier.py'
- '.github/workflows/python-app.yml'
permissions:
contents: read
pages: write
id-token: write
# Cancel superseded runs on PRs (only the latest commit matters → fast feedback).
# On main/release pushes do NOT cancel: every commit gets a full CI result, and —
# critically — an in-flight ``preload-ml-models`` job is never killed mid-download,
# which would let ``actions/cache`` save a partial (poisoned) model cache that
# later runs cache-hit. Release / deploy / backup workflows keep
# ``cancel-in-progress: false`` explicitly.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
# Detect what changed so heavy CI segments can skip when irrelevant.
# - ``docs-only`` → only docs/markdown changed; everything heavy skips
# - ``python`` → Python source / tests / packaging / Python Docker /
# workflows changed; gates the Python-heavy jobs
# (preload-ml-models, test-unit, test-integration*,
# test-e2e*, test-acceptance-fixtures,
# security-quality)
# - ``viewer`` → viewer source / viewer Docker changed; viewer-unit
# + viewer-e2e always run when this is true
# - ``code`` → either Python OR viewer changed (legacy alias for
# "anything actionable"); kept for backwards-compat
# with existing gates
detect-changes:
runs-on: ubuntu-latest
outputs:
docs-only: ${{ steps.filter.outputs.docs }}
code: ${{ steps.filter.outputs.code }}
python: ${{ steps.filter.outputs.python }}
viewer: ${{ steps.filter.outputs.viewer }}
stack-profiles: ${{ steps.filter.outputs.stack-profiles }}
stack-compose: ${{ steps.filter.outputs.stack-compose }}
steps:
- uses: actions/checkout@v6
- uses: dorny/paths-filter@v4
id: filter
with:
filters: |
docs:
- 'docs/**'
- 'mkdocs.yml'
- '**/*.md'
- 'LICENSE'
code:
- '**.py'
- '**.j2'
- 'tests/**'
- 'pyproject.toml'
- 'Makefile'
- 'docker/pipeline/Dockerfile'
- '.dockerignore'
- '.github/workflows/**'
- 'web/gi-kg-viewer/**'
python:
- '**.py'
- '**.j2'
- 'tests/**'
- 'pyproject.toml'
- 'Makefile'
- 'docker/pipeline/Dockerfile'
- 'docker/api/**'
- '.dockerignore'
- '.github/workflows/**'
- 'config/acceptance/**'
- 'config/profiles/**'
- 'compose/**'
viewer:
- 'web/gi-kg-viewer/**'
- 'docker/viewer/**'
stack-profiles:
- 'config/profiles/**'
- 'compose/**'
- 'scripts/tools/validate_profile_docker_tier.py'
- '.github/workflows/python-app.yml'
stack-compose:
- 'compose/**'
- 'docker/api/**'
- 'docker/viewer/**'
- 'docker/pipeline/Dockerfile'
- 'compose/docker-compose.jobs-docker.yml'
- '.github/workflows/python-app.yml'
# GI/KG viewer v2 — Playwright E2E
#
# Gates on ``viewer-unit`` (Vitest, ~30 s) — same fail-fast pattern as
# ``test-unit`` gating the Python heavy jobs: if the viewer source has
# type/build/unit failures, don't waste a Playwright run on it.
#
# **Does NOT depend on Python test results.** Viewer e2e specs use
# mocked API responses (``page.route``), not a real Python server.
# Previously this job waited on ``test-e2e``/``test-e2e-fast`` which
# added ~28 min of wallclock to viewer-only PRs for zero functional
# benefit. (See ``detect-changes.viewer`` filter and the path-A split
# rationale.)
viewer-e2e:
runs-on: ubuntu-latest
timeout-minutes: 20
needs: [lint, build, viewer-unit, detect-changes]
if: |
always() &&
needs.lint.result == 'success' &&
needs.build.result == 'success' &&
needs.viewer-unit.result == 'success'
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
node-version: "22"
cache: "npm"
cache-dependency-path: web/gi-kg-viewer/package-lock.json
- name: Install dependencies and browsers
run: |
cd web/gi-kg-viewer
npm ci
npx playwright install --with-deps firefox
- name: Run Playwright E2E
run: |
cd web/gi-kg-viewer
npm run test:e2e
# Fast lint checks - blocks tests to catch issues early
lint:
runs-on: ubuntu-latest
timeout-minutes: 10 # Fast checks: format, lint, markdown, type
steps:
- uses: actions/checkout@v6
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: "20"
- name: Install lint dependencies (no ML packages)
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Check black version
run: |
python -m black --version
pip show black | grep Version
- name: Install markdownlint
run: npm install -g markdownlint-cli
- name: Install actionlint
run: |
# Pinned latest stable from rhysd/actionlint releases. Validates
# .github/workflows/*.yml against the GHA schema + runs shellcheck
# on inline ``run:`` blocks. Catches workflow regressions before
# they hit prod (e.g. typo in branch list, missing input, bad
# if-expression). Local equivalent: ``brew install actionlint``.
bash <(curl -fsSL https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash)
sudo mv actionlint /usr/local/bin/
actionlint -version
- name: Run fast lint checks
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
make format-check
make lint
make lint-markdown
make type
- name: Lint GitHub Actions workflows
# Validates every workflow in .github/workflows/ — fails fast on:
# schema violations, undefined refs, unsafe ${{ }} interpolation,
# AND shellcheck issues in inline run: blocks (re-enabled in #831
# after cleaning the ~195 pre-existing warnings).
run: |
actionlint .github/workflows/*.yml
- name: Self-hosted runner allowlist (ADR-097)
run: make check-test-policy
# Security and quality checks - gated by lint/build, runs independently to coverage-unified (does not gate tests)
security-quality:
runs-on: ubuntu-latest
timeout-minutes: 20 # Slower checks: security, complexity, deadcode, docstrings, spelling
needs: [lint, build, detect-changes] # Wait for lint, build to pass (docs removed - doesn't gate security)
# Python-only gate (path-A): bandit/flake8/codespell only scan Python
# sources, so viewer-only PRs (web/gi-kg-viewer/**) can skip this.
if: needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Run security checks
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
make security
- name: Run quality checks
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
make quality
- name: Code quality report
if: always()
continue-on-error: true
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
{
echo "## Code Quality Report"
echo ""
echo "### Complexity Analysis"
echo '```'
radon cc src/podcast_scraper/ -a -s --total-average 2>&1 || echo "No complexity data available"
echo '```'
echo ""
echo "### Maintainability Index"
echo '```'
radon mi src/podcast_scraper/ -s 2>&1 || echo "No maintainability data available"
echo '```'
echo ""
echo "### Docstring Coverage"
echo '```'
interrogate src/podcast_scraper/ -v 2>&1 || echo "No docstring data available"
echo '```'
echo ""
echo "### Dead Code Detection"
echo '```'
vulture src/podcast_scraper/ .vulture_whitelist.py --min-confidence 80 2>&1 || echo "No dead code detected"
echo '```'
echo ""
echo "### Spell Checking"
echo '```'
codespell src/ docs/ --skip="*.pyc,*.json,*.xml,*.lock,*.mp3,*.whl" 2>&1 || echo "No spelling errors found"
echo '```'
echo ""
} >> "$GITHUB_STEP_SUMMARY"
# Save complexity metrics to file for metrics generation
mkdir -p reports
radon cc src/podcast_scraper/ -a -s --total-average --json > reports/complexity.json 2>/dev/null || echo '{"total_average": 0}' > reports/complexity.json
radon mi src/podcast_scraper/ -s --json > reports/maintainability.json 2>/dev/null || echo '[]' > reports/maintainability.json
python scripts/dashboard/capture_quality_for_metrics.py --reports-dir reports --vulture-min-confidence 80
# Unit tests - fast, no ML dependencies, network isolation enforced
test-unit:
runs-on: ubuntu-latest
timeout-minutes: 15 # Conservative timeout for fast unit tests
needs: [lint, build, detect-changes] # optional verify-stack-profiles + stack compose config steps
# Python-only gate (path-A): viewer-only PRs run viewer-unit (Vitest)
# instead; pytest doesn't apply to web/gi-kg-viewer/**.
if: needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Install dev dependencies (no ML; includes FastAPI via [dev])
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Verify packaged profiles vs Docker pipeline tier (#660)
if: needs.detect-changes.outputs.stack-profiles == 'true'
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)/src"
export PYTHONPATH
make verify-stack-profiles
- name: Validate stack Compose file merges (no build)
if: needs.detect-changes.outputs.stack-compose == 'true'
run: |
export PODCAST_DOCKER_PROJECT_DIR="${GITHUB_WORKSPACE}"
docker compose -f compose/docker-compose.stack.yml config -q
docker compose -f compose/docker-compose.stack.yml -f compose/docker-compose.jobs-docker.yml config -q
- name: Install ffmpeg (required for audio preprocessing unit tests)
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Verify unit tests can import without ML dependencies
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
python scripts/tools/check_unit_test_imports.py
env:
PACKAGE: podcast_scraper
- name: GIL and KG quality metrics on CI fixtures (PRD-017 / PRD-019)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)/src"
export PYTHONPATH
python scripts/tools/gil_quality_metrics.py tests/fixtures/gil_kg_ci_enforce \
--enforce --strict-schema --fail-on-errors \
--min-extraction-coverage 1.0 \
--min-grounded-insight-rate 1.0 \
--min-quote-validity-rate 1.0 \
--min-avg-insights 1 \
--min-avg-quotes 1
python scripts/tools/kg_quality_metrics.py tests/fixtures/gil_kg_ci_enforce \
--enforce --strict-schema --fail-on-errors \
--min-artifacts 1 \
--min-avg-nodes 1 \
--min-avg-edges 0 \
--min-extraction-coverage 1.0
env:
PACKAGE: podcast_scraper
- name: Run unit tests with coverage (network isolation enforced, parallel execution)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
# Create reports directory for coverage output
mkdir -p reports
# Run tests with coverage and capture output and exit code
set +e # Don't exit on non-zero return code yet
OUTPUT=$(pytest tests/unit/ -v --tb=short -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --json-report --json-report-file=reports/pytest-unit.json --junitxml=reports/junit-unit.xml --cov=podcast_scraper --cov-report=xml:reports/coverage.xml --cov-report=term-missing --reruns 2 --reruns-delay 1 2>&1)
PYTEST_EXIT_CODE=$?
set -e # Re-enable exit on error
echo "$OUTPUT"
# Verify tests were collected and run
if echo "$OUTPUT" | grep -q "no tests collected"; then
echo "ERROR: No tests were collected!"
exit 1
fi
# Verify minimum test count (unit tests should have many tests)
# Extract test count from output (handles formats like "229 passed" or "3 failed, 226 passed")
TEST_COUNT=$(echo "$OUTPUT" | grep -oE "[0-9]+ passed" | head -1 | grep -oE "[0-9]+" || echo "0")
if [ "$TEST_COUNT" -lt 50 ]; then
echo "ERROR: Only $TEST_COUNT tests passed, expected at least 50 unit tests"
exit 1
fi
# Exit with pytest's exit code (fails if any tests failed or coverage below threshold)
if [ $PYTEST_EXIT_CODE -ne 0 ]; then
echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE (some tests failed or coverage below threshold)"
exit $PYTEST_EXIT_CODE
fi
env:
PACKAGE: podcast_scraper
- name: Export coverage data for unified merge
if: success()
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
coverage combine || true
mkdir -p reports
if [ -f .coverage ]; then
cp .coverage reports/coverage-data.unit
echo "✅ Exported reports/coverage-data.unit for coverage-unified job"
else
echo "⚠️ No .coverage file after unit tests"
fi
- name: Generate coverage summary
if: always()
run: |
{
echo "# 📊 Test Coverage Report"
echo ""
if [ -f reports/coverage.xml ]; then
COVERAGE=$(python3 -c "import xml.etree.ElementTree as ET; tree = ET.parse('reports/coverage.xml'); root = tree.getroot(); print(f\"{float(root.attrib.get('line-rate', 0)) * 100:.1f}%\")" 2>/dev/null || echo "N/A")
BRANCH_COVERAGE=$(python3 -c "import xml.etree.ElementTree as ET; tree = ET.parse('reports/coverage.xml'); root = tree.getroot(); print(f\"{float(root.attrib.get('branch-rate', 0)) * 100:.1f}%\")" 2>/dev/null || echo "N/A")
echo "## Coverage Summary"
echo "- **Line Coverage**: $COVERAGE"
echo "- **Branch Coverage**: $BRANCH_COVERAGE"
echo ""
THRESHOLD=70
echo "- **Combined Threshold**: ${THRESHOLD}% *(enforced on combined coverage only)*"
echo ""
COVERAGE_NUM="${COVERAGE%\%}"
COVERAGE_NUM="${COVERAGE_NUM%%.*}"
if [ "$COVERAGE_NUM" != "N/A" ] && [ "$COVERAGE_NUM" -ge "${THRESHOLD}" ]; then
echo "✅ Unit coverage meets combined threshold!"
elif [ "$COVERAGE_NUM" != "N/A" ]; then
echo "ℹ️ Unit coverage below combined threshold (expected - combined includes integration/E2E)"
fi
else
echo "⚠️ Coverage report not found"
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload coverage artifacts
if: always()
uses: actions/upload-artifact@v7
with:
name: coverage-unit
path: |
reports/coverage.xml
reports/coverage-data.unit
reports/junit-unit.xml
retention-days: 7
if-no-files-found: ignore
- name: Upload pytest JSON report
if: always()
uses: actions/upload-artifact@v7
with:
name: pytest-unit
path: reports/pytest-unit.json
retention-days: 7
if-no-files-found: ignore
# Network isolation is enforced by pytest-socket (--disable-socket --allow-hosts)
# The dedicated test file was removed as the blocking feature is now disabled
# N-1 corpus compat — current server code × prior-release fixture (#796).
test-corpus-version-compat:
name: N-1 corpus compat (current code × prior fixture)
runs-on: ubuntu-latest
timeout-minutes: 10
needs: [detect-changes, test-unit]
if: needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Install dev dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Run N-1 corpus version compat integration tests
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
pytest tests/integration/server/test_corpus_version_compat.py -v --tb=short
# GI/KG viewer v2 — Vitest unit tests (same parallel segment as security-quality + test-unit)
viewer-unit:
runs-on: ubuntu-latest
timeout-minutes: 5
needs: [lint, build]
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
# Vite 8 requires ^20.19 || >=22.12; bare "20" on the runner can be older and breaks Vite
node-version: "22"
cache: "npm"
cache-dependency-path: web/gi-kg-viewer/package-lock.json
- name: Install dependencies
run: |
cd web/gi-kg-viewer
npm ci
- name: Run Vitest unit tests
run: |
cd web/gi-kg-viewer
npm run test:unit
# Preload ML models - runs when Python code/tests/configs change so models
# are cached for integration and e2e tests. Path-A gate: skipped on
# viewer-only PRs and docs-only PRs (the viewer doesn't use ML models).
preload-ml-models:
runs-on: ubuntu-latest
# Cold cache: production preload (Whisper + many HF + GIL evidence models) can take 20–40+ min;
# cache-hit path still finishes quickly; generous cap avoids killing long downloads.
timeout-minutes: 60
needs: [detect-changes]
if: needs.detect-changes.outputs.python == 'true'
env:
# Ensure consistent cache paths for Hugging Face libraries
HF_HOME: /home/runner/.cache/huggingface
HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
# Gated pyannote diarization models (speaker-diarization-3.1 + segmentation-3.0)
# only download when a token from a terms-accepted account is present. Without
# it the preload skips diarization gracefully (build still green). Optional secret.
HF_TOKEN: ${{ secrets.HF_TOKEN }}
steps:
- uses: actions/checkout@v6
# Restore is ~9GB compressed; without cleanup, cache-hit runs can run out of disk mid-tar
# ("Wrote only N of M bytes"). Always free space before restore, not only on cache miss.
- name: Free disk space (before ML cache restore or download)
if: needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true'
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
docker image prune -af || true
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
df -h
- name: Cache ML models (check first - no Python needed, if cache hits job completes in ~6-12s)
if: needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true'
uses: actions/cache@v5
id: cache-models
with:
path: |
~/.cache/whisper
~/.cache/huggingface
# NOTE: spaCy removed - installed as pip package via .[ml]
# Cache key invalidates when model definitions change; restore-keys let PRs/branches reuse latest cache.
# `diar2` salt: forces a clean re-preload. `diar1` was saved incomplete by a cancelled run
# (missing MiniLM + gated pyannote models), so cache-hits served a broken cache. The broad
# restore-key warm-starts from the prior cache so only the missing models re-download.
key: ml-models-${{ runner.os }}-diar2-${{ hashFiles('scripts/cache/preload_ml_models.py', 'src/podcast_scraper/config.py') }}
restore-keys: |
ml-models-${{ runner.os }}-diar2-
ml-models-${{ runner.os }}-
- name: Set up Python 3.11 (only if cache miss - needed for model download)
if: (needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true') && steps.cache-models.outputs.cache-hit != 'true'
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Install ffmpeg (pyannote 4.x / torchcodec needs FFmpeg shared libs to import)
if: (needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true') && steps.cache-models.outputs.cache-hit != 'true'
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Install full dependencies (including ML)
if: (needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true') && steps.cache-models.outputs.cache-hit != 'true'
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,ml,llm,search]"
- name: Preload ML models (only if cache miss)
if: (needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true') && steps.cache-models.outputs.cache-hit != 'true'
run: make preload-ml-models-production
- name: Verify models were downloaded (only if cache miss)
if: (needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true') && steps.cache-models.outputs.cache-hit != 'true'
run: |
echo "🔍 Verifying models were downloaded..."
MISSING=""
# Check Whisper models (preload-production downloads both tiny.en and base.en)
for model in "tiny.en" "base.en"; do
if [ ! -f "$HOME/.cache/whisper/${model}.pt" ]; then
MISSING="$MISSING whisper:$model"
fi
done
# Check Hugging Face models (preload-production downloads all: bart-base, led-base-16384, bart-large-cnn, led-large-16384)
HF_CACHE="$HOME/.cache/huggingface/hub"
echo "🔍 Debug: Checking HuggingFace cache structure..."
echo " HF_CACHE: $HF_CACHE"
echo " HF_CACHE exists: $([ -d "$HF_CACHE" ] && echo 'yes' || echo 'no')"
if [ -d "$HF_CACHE" ]; then
echo " Contents of HF_CACHE:"
find "$HF_CACHE" -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' | head -20
fi
for model in "facebook/bart-base" "allenai/led-base-16384" "google/long-t5-tglobal-base" "google/flan-t5-base"; do
MODEL_DIR="models--${model//\//--}"
MODEL_PATH="$HF_CACHE/$MODEL_DIR"
echo " Checking $model:"
echo " MODEL_PATH: $MODEL_PATH"
echo " MODEL_PATH exists: $([ -d "$MODEL_PATH" ] && echo 'yes' || echo 'no')"
if [ -d "$MODEL_PATH" ]; then
echo " Contents of MODEL_PATH:"
find "$MODEL_PATH" -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' | head -10
if [ -d "$MODEL_PATH/snapshots" ]; then
echo " snapshots/ exists, contents:"
find "$MODEL_PATH/snapshots" -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' | head -5
else
echo " snapshots/ does NOT exist"
fi
if [ -d "$MODEL_PATH/blobs" ]; then
echo " blobs/ exists, file count: $(find "$MODEL_PATH/blobs" -type f 2>/dev/null | wc -l)"
fi
TOTAL_FILES=$(find "$MODEL_PATH" -type f 2>/dev/null | wc -l)
echo " Total files in MODEL_PATH: $TOTAL_FILES"
fi
# Model is valid if it has files (in snapshots, blobs, or anywhere in the directory)
if [ ! -d "$MODEL_PATH" ] || [ "$(find "$MODEL_PATH" -type f 2>/dev/null | head -1)" = "" ]; then
MISSING="$MISSING hf:$model"
fi
done
if [ -n "$MISSING" ]; then
echo "❌ ERROR: Models not downloaded: $MISSING"
exit 1
fi
echo "✅ All required models downloaded successfully"
echo "📦 Cache will be saved at end of job for test jobs to use"
echo "🔍 Debug: Cache sizes before save:"
du -sh "$HOME/.cache/whisper" 2>/dev/null || echo " Whisper cache not found"
du -sh "$HOME/.cache/huggingface" 2>/dev/null || echo " HuggingFace cache not found"
- name: Upload ML models artifact (for dependent test jobs in same run)
if: needs.detect-changes.outputs.docs-only != 'true' || needs.detect-changes.outputs.code == 'true'
uses: actions/upload-artifact@v7
with:
name: ml-models
path: /home/runner/.cache
retention-days: 1
include-hidden-files: true
# Full integration tests - all integration tests, runs on main branch only
test-integration:
runs-on: ubuntu-latest
# Large ml-models artifact + full integration suite; align headroom with preload growth.
timeout-minutes: 45
# ``test-unit`` added for fail-fast: don't run expensive integration on Python
# unit-test failures. Path-A python gate skips this for viewer-only PRs.
needs: [detect-changes, preload-ml-models, test-unit]
env:
# Ensure consistent cache paths for Hugging Face libraries
HF_HOME: /home/runner/.cache/huggingface
HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
if: |
github.event_name == 'push' &&
(github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6') &&
needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
docker image prune -af || true
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
df -h
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Download ML models from preload job
uses: actions/download-artifact@v8
with:
name: ml-models
path: /home/runner/.cache
- name: Validate ML model cache
id: validate-cache
run: |
echo "🔍 Validating ML model cache..."
MISSING_MODELS=""
MODELS_COMPLETE=true
# Check Whisper models
echo "📦 Whisper models:"
for model in "tiny.en" "base.en"; do
WHISPER_PATH="$HOME/.cache/whisper/${model}.pt"
if [ -f "$WHISPER_PATH" ] && [ -s "$WHISPER_PATH" ]; then
SIZE=$(du -h "$WHISPER_PATH" | cut -f1)
echo " ✅ $model ($SIZE)"
else
echo " ❌ $model - MISSING or empty"
MISSING_MODELS="$MISSING_MODELS whisper:$model"
MODELS_COMPLETE=false
fi
done
# Check Hugging Face models
echo ""
echo "📦 Hugging Face models:"
HF_CACHE="$HOME/.cache/huggingface/hub"
for model in "facebook/bart-base" "allenai/led-base-16384" "google/long-t5-tglobal-base" "google/flan-t5-base"; do
MODEL_DIR="models--${model//\//--}"
MODEL_PATH="$HF_CACHE/$MODEL_DIR"
# Check if model has files anywhere (blobs, snapshots, or root)
# HuggingFace cache structure: files in blobs/, symlinks in snapshots/
TOTAL_FILES=$(find "$MODEL_PATH" -type f 2>/dev/null | wc -l)
if [ -d "$MODEL_PATH" ] && [ "$TOTAL_FILES" -gt 0 ]; then
SIZE=$(du -sh "$MODEL_PATH" | cut -f1)
echo " ✅ $model ($SIZE, $TOTAL_FILES files)"
else
echo " ❌ $model - MISSING or incomplete (0 files found)"
MISSING_MODELS="$MISSING_MODELS hf:$model"
MODELS_COMPLETE=false
fi
done
echo ""
if [ "$MODELS_COMPLETE" = true ]; then
echo "✅ All required ML models are cached!"
echo "models_complete=true" >> "$GITHUB_OUTPUT"
else
echo "⚠️ Some models are missing: $MISSING_MODELS"
echo "models_complete=false" >> "$GITHUB_OUTPUT"
exit 1
fi
- name: Set ML_MODELS_VALIDATED (only if cache validated)
if: steps.validate-cache.outputs.models_complete == 'true'
run: |
echo "ML_MODELS_VALIDATED=true" >> "$GITHUB_ENV"
- name: Install full dependencies (including ML)
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,ml,llm,search]"
pip install pytest-socket
- name: Install ffmpeg (required for Whisper)
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Run all integration tests with coverage (full suite, with network guard)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
# Create reports directory for coverage output
mkdir -p reports
# Run all integration tests with coverage and network guard
set +e # Don't exit on non-zero return code yet
OUTPUT=$(pytest tests/integration/ -v -m integration -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --json-report --json-report-file=reports/pytest-integration.json --junitxml=reports/junit-integration.xml --cov=podcast_scraper --cov-append --cov-report=xml:reports/coverage-integration.xml --cov-report=term-missing --cov-fail-under=42 --disable-socket --allow-hosts=127.0.0.1,localhost --reruns 2 --reruns-delay 1 2>&1)
PYTEST_EXIT_CODE=$?
set -e # Re-enable exit on error
echo "$OUTPUT"
# Verify tests were collected and run
if echo "$OUTPUT" | grep -q "no tests collected"; then
echo "ERROR: No tests were collected!"
exit 1
fi
# Verify minimum test count (all integration tests should have many tests)
# Extract test count from output (handles formats like "229 passed" or "3 failed, 226 passed")
TEST_COUNT=$(echo "$OUTPUT" | grep -oE "[0-9]+ passed" | head -1 | grep -oE "[0-9]+" || echo "0")
if [ "$TEST_COUNT" -lt 50 ]; then
echo "ERROR: Only $TEST_COUNT tests passed, expected at least 50 integration tests"
exit 1
fi
# Exit with pytest's exit code (fails if any tests failed)
if [ $PYTEST_EXIT_CODE -ne 0 ]; then
echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE (some tests failed)"
exit $PYTEST_EXIT_CODE
fi
- name: Export coverage data for unified merge
if: success()
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
coverage combine || true
mkdir -p reports
if [ -f .coverage ]; then
cp .coverage reports/coverage-data.integration
echo "✅ Exported reports/coverage-data.integration for coverage-unified job"
else
echo "⚠️ No .coverage file after integration tests"
fi
- name: Upload coverage artifacts
if: always()
uses: actions/upload-artifact@v7
with:
name: coverage-integration
path: |
reports/coverage-integration.xml
reports/coverage-data.integration
reports/junit-integration.xml
retention-days: 7
if-no-files-found: ignore
- name: Upload pytest JSON report
if: always()
uses: actions/upload-artifact@v7
with:
name: pytest-integration
path: reports/pytest-integration.json
retention-days: 7
if-no-files-found: ignore
- name: Post-build cleanup
if: always()
run: |
# Keep model caches for next run (they're cached via GitHub Actions cache)
rm -rf .pytest_cache .mypy_cache .build dist
# Fast integration tests - critical path only, runs on PRs only
# Full integration tests run on main branch only
test-integration-fast:
runs-on: ubuntu-latest
# Same ml-models artifact as E2E; allow time for download/unpack + integration pytest.
timeout-minutes: 35
# ``test-unit`` added for fail-fast (don't run integration on Python unit
# failures). Path-A python gate skips this for viewer-only PRs.
needs: [detect-changes, preload-ml-models, test-unit]
env:
# Ensure consistent cache paths for Hugging Face libraries
HF_HOME: /home/runner/.cache/huggingface
HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
if: |
github.event_name == 'pull_request' &&
!contains(github.event.pull_request.head.ref, 'docs/') &&
needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
docker image prune -af || true
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
df -h
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Download ML models from preload job
uses: actions/download-artifact@v8
with:
name: ml-models
path: /home/runner/.cache
- name: Validate ML model cache
id: validate-cache
run: |
echo "🔍 Validating ML model cache..."
MISSING_MODELS=""
MODELS_COMPLETE=true
# Check Whisper models
echo "📦 Whisper models:"
for model in "tiny.en"; do
WHISPER_PATH="$HOME/.cache/whisper/${model}.pt"
if [ -f "$WHISPER_PATH" ] && [ -s "$WHISPER_PATH" ]; then
SIZE=$(du -h "$WHISPER_PATH" | cut -f1)
echo " ✅ $model ($SIZE)"
else
echo " ❌ $model - MISSING or empty"
MISSING_MODELS="$MISSING_MODELS whisper:$model"
MODELS_COMPLETE=false
fi
done
# Check Hugging Face models
echo ""
echo "📦 Hugging Face models:"
HF_CACHE="$HOME/.cache/huggingface/hub"
echo "🔍 Debug: Checking HuggingFace cache after restore..."
echo " HF_CACHE: $HF_CACHE"
echo " HF_CACHE exists: $([ -d "$HF_CACHE" ] && echo 'yes' || echo 'no')"
if [ -d "$HF_CACHE" ]; then
echo " Contents of HF_CACHE:"
find "$HF_CACHE" -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' | head -20
echo " Total size: $(du -sh "$HF_CACHE" 2>/dev/null | cut -f1)"
fi
for model in "facebook/bart-base" "allenai/led-base-16384" "google/long-t5-tglobal-base" "google/flan-t5-base"; do
MODEL_DIR="models--${model//\//--}"
MODEL_PATH="$HF_CACHE/$MODEL_DIR"
echo " Checking $model:"
echo " MODEL_PATH: $MODEL_PATH"
echo " MODEL_PATH exists: $([ -d "$MODEL_PATH" ] && echo 'yes' || echo 'no')"
if [ -d "$MODEL_PATH" ]; then
echo " Contents of MODEL_PATH:"
find "$MODEL_PATH" -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' | head -10
if [ -d "$MODEL_PATH/snapshots" ]; then
echo " snapshots/ exists, contents:"
find "$MODEL_PATH/snapshots" -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' | head -5
FILE_COUNT_SNAPSHOTS=$(find "$MODEL_PATH/snapshots" -type f -o -type l 2>/dev/null | wc -l)
echo " File/symlink count in snapshots/: $FILE_COUNT_SNAPSHOTS"
else
echo " snapshots/ does NOT exist"
fi
if [ -d "$MODEL_PATH/blobs" ]; then
echo " blobs/ exists, contents:"
find "$MODEL_PATH/blobs" -maxdepth 1 -printf '%M %u %s %TF %TR %f\n' | head -5
FILE_COUNT_BLOBS=$(find "$MODEL_PATH/blobs" -type f 2>/dev/null | wc -l)
echo " File count in blobs/: $FILE_COUNT_BLOBS"
else
echo " blobs/ does NOT exist"
fi
# Check if model is complete: either files in snapshots (following symlinks) or files in blobs
TOTAL_FILES=$(find "$MODEL_PATH" -type f 2>/dev/null | wc -l)
echo " Total files in MODEL_PATH: $TOTAL_FILES"
fi
# Model is valid if it has files anywhere (blobs, snapshots, or root)
# HuggingFace cache structure: files in blobs/, symlinks in snapshots/
TOTAL_FILES=$(find "$MODEL_PATH" -type f 2>/dev/null | wc -l)
if [ -d "$MODEL_PATH" ] && [ "$TOTAL_FILES" -gt 0 ]; then
SIZE=$(du -sh "$MODEL_PATH" | cut -f1)
echo " ✅ $model ($SIZE, $TOTAL_FILES files)"
else
echo " ❌ $model - MISSING or incomplete (0 files found)"
MISSING_MODELS="$MISSING_MODELS hf:$model"
MODELS_COMPLETE=false
fi
done
echo ""
if [ "$MODELS_COMPLETE" = true ]; then
echo "✅ All required ML models are cached!"
echo "models_complete=true" >> "$GITHUB_OUTPUT"
else
echo "⚠️ Some models are missing: $MISSING_MODELS"
echo "models_complete=false" >> "$GITHUB_OUTPUT"
exit 1
fi
- name: Set ML_MODELS_VALIDATED (only if cache validated)
if: steps.validate-cache.outputs.models_complete == 'true'
run: |
echo "ML_MODELS_VALIDATED=true" >> "$GITHUB_ENV"
- name: Install dev dependencies with ML (pytest-socket for network guard)
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,ml,llm,search]"
pip install pytest-socket
- name: Install ffmpeg (required for Whisper)
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Run fast integration tests with coverage (critical path only, with network guard)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
# Create reports directory for coverage output
mkdir -p reports
# Run tests with coverage and network guard: --disable-socket --allow-hosts=127.0.0.1,localhost
# Critical path tests only for faster CI feedback
# Note: Fast tests don't enforce coverage threshold (they're a subset, full suite enforces threshold)
set +e # Don't exit on non-zero return code yet
OUTPUT=$(pytest tests/integration/ -v -m "integration and critical_path" -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --cov=podcast_scraper --cov-append --cov-report=xml:reports/coverage-integration.xml --cov-report=term-missing --disable-socket --allow-hosts=127.0.0.1,localhost --reruns 2 --reruns-delay 1 --durations=20 2>&1)
PYTEST_EXIT_CODE=$?
set -e # Re-enable exit on error
echo "$OUTPUT"
# Verify tests were collected and run
if echo "$OUTPUT" | grep -q "no tests collected"; then
echo "ERROR: No tests were collected!"
exit 1
fi
# Verify minimum test count (critical path integration tests should have some tests)
# Extract test count from output (handles formats like "229 passed" or "3 failed, 226 passed")
TEST_COUNT=$(echo "$OUTPUT" | grep -oE "[0-9]+ passed" | head -1 | grep -oE "[0-9]+" || echo "0")
if [ "$TEST_COUNT" -lt 5 ]; then
echo "ERROR: Only $TEST_COUNT tests passed, expected at least 5 critical path integration tests"
exit 1
fi
# Exit with pytest's exit code (fails if any tests failed)
if [ $PYTEST_EXIT_CODE -ne 0 ]; then
echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE (some tests failed)"
exit $PYTEST_EXIT_CODE
fi
- name: Export coverage data for unified merge
if: success()
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
coverage combine || true
mkdir -p reports
if [ -f .coverage ]; then
cp .coverage reports/coverage-data.integration
echo "✅ Exported reports/coverage-data.integration for coverage-unified job"
else
echo "⚠️ No .coverage file after fast integration tests"
fi
- name: Upload coverage artifacts
if: always()
uses: actions/upload-artifact@v7
with:
name: coverage-integration-fast
path: |
reports/coverage-integration.xml
reports/coverage-data.integration
retention-days: 7
if-no-files-found: ignore
- name: Post-build cleanup
if: always()
run: |
rm -rf .pytest_cache .mypy_cache .build dist
# Fast E2E tests - critical path only, runs on PRs only
# Full E2E tests run on main branch only
test-e2e-fast:
runs-on: ubuntu-latest
# After preload: large ml-models artifact restore + pip + ffmpeg + pytest; keep headroom vs preload duration.
timeout-minutes: 45
# ``test-unit`` added for fail-fast (don't run e2e on Python unit
# failures). Path-A python gate skips this for viewer-only PRs.
needs: [detect-changes, preload-ml-models, test-unit]
env:
# Ensure consistent cache paths for Hugging Face libraries
HF_HOME: /home/runner/.cache/huggingface
HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
if: |
github.event_name == 'pull_request' &&
!contains(github.event.pull_request.head.ref, 'docs/') &&
needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
docker image prune -af || true
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
df -h
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Download ML models from preload job
uses: actions/download-artifact@v8
with:
name: ml-models
path: /home/runner/.cache
- name: Validate ML model cache
id: validate-cache
run: |
echo "🔍 Validating ML model cache..."
MISSING_MODELS=""
MODELS_COMPLETE=true
# Check Whisper models
echo "📦 Whisper models:"
for model in "tiny.en" "base.en"; do
WHISPER_PATH="$HOME/.cache/whisper/${model}.pt"
if [ -f "$WHISPER_PATH" ] && [ -s "$WHISPER_PATH" ]; then
SIZE=$(du -h "$WHISPER_PATH" | cut -f1)
echo " ✅ $model ($SIZE)"
else
echo " ❌ $model - MISSING or empty"
MISSING_MODELS="$MISSING_MODELS whisper:$model"
MODELS_COMPLETE=false
fi
done
# Check Hugging Face models
echo ""
echo "📦 Hugging Face models:"
HF_CACHE="$HOME/.cache/huggingface/hub"
for model in "facebook/bart-base" "allenai/led-base-16384" "google/long-t5-tglobal-base" "google/flan-t5-base"; do
MODEL_DIR="models--${model//\//--}"
MODEL_PATH="$HF_CACHE/$MODEL_DIR"
# Check if model has files anywhere (blobs, snapshots, or root)
# HuggingFace cache structure: files in blobs/, symlinks in snapshots/
TOTAL_FILES=$(find "$MODEL_PATH" -type f 2>/dev/null | wc -l)
if [ -d "$MODEL_PATH" ] && [ "$TOTAL_FILES" -gt 0 ]; then
SIZE=$(du -sh "$MODEL_PATH" | cut -f1)
echo " ✅ $model ($SIZE, $TOTAL_FILES files)"
else
echo " ❌ $model - MISSING or incomplete (0 files found)"
MISSING_MODELS="$MISSING_MODELS hf:$model"
MODELS_COMPLETE=false
fi
done
echo ""
if [ "$MODELS_COMPLETE" = true ]; then
echo "✅ All required ML models are cached!"
echo "models_complete=true" >> "$GITHUB_OUTPUT"
else
echo "⚠️ Some models are missing: $MISSING_MODELS"
echo "models_complete=false" >> "$GITHUB_OUTPUT"
exit 1
fi
- name: Set ML_MODELS_VALIDATED (only if cache validated)
if: steps.validate-cache.outputs.models_complete == 'true'
run: |
echo "ML_MODELS_VALIDATED=true" >> "$GITHUB_ENV"
- name: Install dev dependencies (pytest-socket for network guard)
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,ml,llm,search]"
pip install pytest-socket
- name: Install ffmpeg (required for Whisper)
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Run fast E2E tests with coverage (critical path only, with network guard)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
export E2E_TEST_MODE=fast
# Create reports directory for coverage output
mkdir -p reports
# Run tests with coverage and network guard: --disable-socket --allow-hosts=127.0.0.1,localhost
# Critical path tests only for faster CI feedback
# Note: Fast tests don't enforce coverage threshold (they're a subset, full suite enforces threshold)
set +e # Don't exit on non-zero return code yet
OUTPUT=$(E2E_TEST_MODE=fast pytest tests/e2e/ -v -m "e2e and critical_path" -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --cov=podcast_scraper --cov-report=xml:reports/coverage-e2e.xml --cov-report=term-missing --disable-socket --allow-hosts=127.0.0.1,localhost --reruns 2 --reruns-delay 1 --durations=20 2>&1)
PYTEST_EXIT_CODE=$?
set -e # Re-enable exit on error
echo "$OUTPUT"
# Verify tests were collected and run
if echo "$OUTPUT" | grep -q "no tests collected"; then
echo "ERROR: No tests were collected!"
exit 1
fi
# Verify minimum test count (critical path E2E tests should have some tests)
# Extract test count from output (handles formats like "229 passed" or "3 failed, 226 passed")
TEST_COUNT=$(echo "$OUTPUT" | grep -oE "[0-9]+ passed" | head -1 | grep -oE "[0-9]+" || echo "0")
if [ "$TEST_COUNT" -lt 3 ]; then
echo "ERROR: Only $TEST_COUNT tests passed, expected at least 3 critical path E2E tests"
exit 1
fi
# Exit with pytest's exit code (fails if any tests failed)
if [ $PYTEST_EXIT_CODE -ne 0 ]; then
echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE (some tests failed)"
exit $PYTEST_EXIT_CODE
fi
- name: Export coverage data for unified merge
if: success()
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
coverage combine || true
mkdir -p reports
if [ -f .coverage ]; then
cp .coverage reports/coverage-data.e2e
echo "✅ Exported reports/coverage-data.e2e for coverage-unified job"
else
echo "⚠️ No .coverage file after fast E2E tests"
fi
- name: Upload coverage artifacts
if: always()
uses: actions/upload-artifact@v7
with:
name: coverage-e2e-fast
path: |
reports/coverage-e2e.xml
reports/coverage-data.e2e
retention-days: 7
if-no-files-found: ignore
- name: Post-build cleanup
if: always()
run: |
rm -rf .pytest_cache .mypy_cache .build dist
# Full E2E tests - all E2E tests, runs on main branch only
test-e2e:
runs-on: ubuntu-latest
# Full E2E suite + same artifact path as fast E2E; needs slack if models keep growing.
timeout-minutes: 60
# ``test-unit`` added for fail-fast. Path-A python gate skips on viewer-only.
needs: [detect-changes, preload-ml-models, test-unit]
env:
# Ensure consistent cache paths for Hugging Face libraries
HF_HOME: /home/runner/.cache/huggingface
HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
# The diarization e2e (ml_models) constructs the pyannote provider, which
# requires a token even when loading gated models from the offline cache.
# When unset (forks), the test skips via its provisioning markers.
HF_TOKEN: ${{ secrets.HF_TOKEN }}
# Airgapped contract: gated models are downloaded once in preload-ml-models;
# this job loads them from cache only. OFFLINE=1 means the HF libs never
# attempt (socket-blocked) network calls — zero 3rd-party traffic at test time.
HF_HUB_OFFLINE: "1"
if: |
github.event_name == 'push' &&
(github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6') &&
needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
docker image prune -af || true
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
df -h
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Download ML models from preload job
uses: actions/download-artifact@v8
with:
name: ml-models
path: /home/runner/.cache
- name: Validate ML model cache
id: validate-cache
run: |
echo "🔍 Validating ML model cache..."
MISSING_MODELS=""
MODELS_COMPLETE=true
# Check Whisper models
echo "📦 Whisper models:"
for model in "tiny.en" "base.en"; do
WHISPER_PATH="$HOME/.cache/whisper/${model}.pt"
if [ -f "$WHISPER_PATH" ] && [ -s "$WHISPER_PATH" ]; then
SIZE=$(du -h "$WHISPER_PATH" | cut -f1)
echo " ✅ $model ($SIZE)"
else
echo " ❌ $model - MISSING or empty"
MISSING_MODELS="$MISSING_MODELS whisper:$model"
MODELS_COMPLETE=false
fi
done
# Check Hugging Face models
echo ""
echo "📦 Hugging Face models:"
HF_CACHE="$HOME/.cache/huggingface/hub"
for model in "facebook/bart-base" "allenai/led-base-16384" "google/long-t5-tglobal-base" "google/flan-t5-base"; do
MODEL_DIR="models--${model//\//--}"
MODEL_PATH="$HF_CACHE/$MODEL_DIR"
TOTAL_FILES=$(find "$MODEL_PATH" -type f 2>/dev/null | wc -l)
if [ -d "$MODEL_PATH" ] && [ "$TOTAL_FILES" -gt 0 ]; then
SIZE=$(du -sh "$MODEL_PATH" | cut -f1)
echo " ✅ $model ($SIZE, $TOTAL_FILES files)"
else
echo " ❌ $model - MISSING or incomplete (0 files found)"
MISSING_MODELS="$MISSING_MODELS hf:$model"
MODELS_COMPLETE=false
fi
done
echo ""
if [ "$MODELS_COMPLETE" = true ]; then
echo "✅ All required ML models are cached!"
echo "models_complete=true" >> "$GITHUB_OUTPUT"
else
echo "⚠️ Some models are missing: $MISSING_MODELS"
echo "models_complete=false" >> "$GITHUB_OUTPUT"
exit 1
fi
- name: Set ML_MODELS_VALIDATED (only if cache validated)
if: steps.validate-cache.outputs.models_complete == 'true'
run: |
echo "ML_MODELS_VALIDATED=true" >> "$GITHUB_ENV"
- name: Install full dependencies (including ML)
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,ml,llm,search]"
pip install pytest-socket
- name: Install ffmpeg (required for Whisper)
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Run all E2E tests with coverage (full suite, with network guard)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
export E2E_TEST_MODE=multi_episode
# Create reports directory for coverage output
mkdir -p reports
# Run all E2E tests with coverage and network guard (single xdist run; no @pytest.mark.serial in suite)
set +e # Don't exit on non-zero return code yet
OUTPUT=$(E2E_TEST_MODE=multi_episode pytest tests/e2e/ -v -m "e2e" -n "$(python3 -c 'import os; print(max(1, (os.cpu_count() or 2) - 2))')" --json-report --json-report-file=reports/pytest-e2e.json --junitxml=reports/junit-e2e.xml --cov=podcast_scraper --cov-report=xml:reports/coverage-e2e.xml --cov-report=term-missing --cov-fail-under=40 --disable-socket --allow-hosts=127.0.0.1,localhost --reruns 2 --reruns-delay 1 2>&1)
PYTEST_EXIT_CODE=$?
set -e # Re-enable exit on error
echo "$OUTPUT"
# Verify tests were collected and run
if echo "$OUTPUT" | grep -q "no tests collected"; then
echo "ERROR: No tests were collected!"
exit 1
fi
# Verify minimum test count (all E2E tests should have many tests)
# Extract test count from output (handles formats like "229 passed" or "3 failed, 226 passed")
TEST_COUNT=$(echo "$OUTPUT" | grep -oE "[0-9]+ passed" | head -1 | grep -oE "[0-9]+" || echo "0")
if [ "$TEST_COUNT" -lt 50 ]; then
echo "ERROR: Only $TEST_COUNT tests passed, expected at least 50 E2E tests"
exit 1
fi
# Exit with pytest's exit code (fails if any tests failed)
if [ $PYTEST_EXIT_CODE -ne 0 ]; then
echo "ERROR: pytest exited with code $PYTEST_EXIT_CODE (some tests failed)"
exit $PYTEST_EXIT_CODE
fi
- name: Export coverage data for unified merge
if: success()
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
coverage combine || true
mkdir -p reports
if [ -f .coverage ]; then
cp .coverage reports/coverage-data.e2e
echo "✅ Exported reports/coverage-data.e2e for coverage-unified job"
else
echo "⚠️ No .coverage file after E2E tests"
fi
- name: Upload coverage artifacts
if: always()
uses: actions/upload-artifact@v7
with:
name: coverage-e2e
path: |
reports/coverage-e2e.xml
reports/coverage-data.e2e
reports/junit-e2e.xml
retention-days: 7
if-no-files-found: ignore
- name: Upload pytest JSON reports
if: always()
uses: actions/upload-artifact@v7
with:
name: pytest-e2e
path: |
reports/pytest-e2e.json
retention-days: 7
if-no-files-found: ignore
- name: Post-build cleanup
if: always()
run: |
# Keep model caches for next run (they're cached via GitHub Actions cache)
rm -rf .pytest_cache .mypy_cache .build dist
# Acceptance runner: full fast matrix with USE_FIXTURES (offline E2E server + mock APIs).
# Same branch gate as full E2E. Parallels ``viewer-e2e`` in the DAG: both are
# fixture-driven post-unit tests that don't share build outputs. Gates on
# ``test-unit`` for fail-fast (same pattern as test-integration / test-e2e).
test-acceptance-fixtures:
runs-on: ubuntu-latest
timeout-minutes: 90
needs: [detect-changes, preload-ml-models, test-unit]
env:
HF_HOME: /home/runner/.cache/huggingface
HF_HUB_CACHE: /home/runner/.cache/huggingface/hub
if: |
github.event_name == 'push' &&
(github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6') &&
needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /home/linuxbrew/.linuxbrew
docker image prune -af || true
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
df -h
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Download ML models from preload job
uses: actions/download-artifact@v8
with:
name: ml-models
path: /home/runner/.cache
- name: Validate ML model cache (same checks as E2E)
run: |
for model in tiny.en base.en; do
test -s "$HOME/.cache/whisper/${model}.pt" || { echo "missing whisper $model"; exit 1; }
done
HF_CACHE="$HOME/.cache/huggingface/hub"
for model in facebook/bart-base allenai/led-base-16384 google/long-t5-tglobal-base google/flan-t5-base; do
MODEL_DIR="models--${model//\//--}"
test -d "$HF_CACHE/$MODEL_DIR" || { echo "missing hf $model"; exit 1; }
test "$(find "$HF_CACHE/$MODEL_DIR" -type f 2>/dev/null | head -1)" || { echo "empty hf $model"; exit 1; }
done
- name: Install dependencies (ML + LLM for full pipeline acceptance)
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,ml,llm,search]"
- name: Install ffmpeg
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Run acceptance fast matrix with E2E fixtures
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
# Fast matrix is four cloud/dev rows (no airgapped Whisper-heavy preset); 900s per config.
TIMEOUT=900 PER_RUN_WALL_SECONDS=600 make test-acceptance-fixtures-fast
- name: Verify GIL Quote vs FAISS transcript chunk offsets (acceptance corpora)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
# #528: each successful acceptance run_* with vector_search + generate_gi must
# align Quote char spans with indexed transcript chunks (feeds nightly / lift contract).
make verify-gil-offsets-after-acceptance
# Documentation build - validates docs can be built correctly
# Runs as very last step after all tests and coverage pass (final validation)
# Docs deployment happens in docs.yml workflow (separate, already gated properly)
docs:
runs-on: ubuntu-latest
needs: [coverage-unified, viewer-unit, viewer-e2e] # Docs after coverage + viewer tests
# Path-A: on viewer-only PRs ``coverage-unified`` is skipped; allow docs
# to still run as long as viewer-unit + viewer-e2e succeeded.
if: |
always() &&
needs.viewer-unit.result == 'success' &&
needs.viewer-e2e.result == 'success' &&
(needs.coverage-unified.result == 'success' || needs.coverage-unified.result == 'skipped')
steps:
- uses: actions/checkout@v6
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: |
docs/requirements.txt
pyproject.toml
- name: Install doc dependencies
run: |
python -m pip install --upgrade pip
pip install -r docs/requirements.txt
# No [ml] / ffmpeg: mkdocstrings (griffe) parses source statically and
# never imports the package, so the heavy ML stack and FFmpeg shared
# libs aren't needed to build the API docs. (The earlier [ml]+ffmpeg
# steps were copied from preload-ml-models during the pyannote-4
# migration on a false premise — verified the docs build is identical
# with torch/transformers/pyannote/torchcodec all blocked.)
pip install -e .
- name: Build docs
run: make docs
# Unified coverage report - combines unit + integration + E2E coverage
# Waits for all test jobs to complete, then combines their coverage reports
# Path-A: skip on docs-only AND viewer-only (no Python coverage to combine).
coverage-unified:
runs-on: ubuntu-latest
needs: [detect-changes, security-quality, test-unit, test-integration, test-integration-fast, test-e2e, test-e2e-fast, viewer-e2e] # Python tests + Playwright before merge report
if: always() && needs.detect-changes.outputs.python == 'true'
steps:
- uses: actions/checkout@v6
with:
# Full history required for wily (code quality trends) on main/release
fetch-depth: ${{ (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6') && 0 || 1 }}
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Install coverage tooling and project (for combine + report parity with local)
run: |
python -m pip install --upgrade pip
pip install coverage[toml]
pip install -e .
# Same tools as nightly metrics + Makefile quality: not in base install — without these,
# `radon` is missing, metrics fall back to complexity 0 / empty MI (flat code-quality chart).
pip install "radon>=5.1.0,<5.2" "interrogate>=1.5.0,<2.0.0" "vulture>=2.10,<3.0.0" "codespell>=2.2.0,<3.0.0"
- name: Download coverage artifacts
uses: actions/download-artifact@v8
with:
pattern: coverage-*
merge-multiple: false
path: coverage-artifacts
continue-on-error: true # Some jobs may not have run (conditional)
- name: Copy JUnit XML from coverage artifacts (slowest tests when JSON lacks timings)
run: |
mkdir -p reports
find coverage-artifacts -type f -name 'junit*.xml' -exec cp -f {} reports/ \; 2>/dev/null || true
ls -la reports/junit*.xml 2>/dev/null || echo "(no junit*.xml in coverage artifacts)"
- name: Download pytest JSON artifacts
uses: actions/download-artifact@v8
with:
pattern: pytest-*
merge-multiple: false
path: pytest-artifacts
continue-on-error: true # Some artifacts may not exist
- name: Combine pytest JSON reports
run: |
mkdir -p reports
# Merge all pytest JSON reports into a single combined report
python3 << 'EOF'
import json
import os
from pathlib import Path
combined = {
"summary": {"total": 0, "passed": 0, "failed": 0, "skipped": 0},
"duration": 0.0,
"tests": []
}
artifacts_dir = Path("pytest-artifacts")
if artifacts_dir.exists():
for json_file in artifacts_dir.rglob("*.json"):
try:
with open(json_file) as f:
data = json.load(f)
summary = data.get("summary", {})
combined["summary"]["total"] += summary.get("total", 0)
combined["summary"]["passed"] += summary.get("passed", 0)
combined["summary"]["failed"] += summary.get("failed", 0)
combined["summary"]["skipped"] += summary.get("skipped", 0)
combined["duration"] += data.get("duration", 0)
combined["tests"].extend(data.get("tests", []))
print(f"✅ Merged {json_file}: {summary.get('total', 0)} tests")
except Exception as e:
print(f"⚠️ Failed to parse {json_file}: {e}")
# Calculate pass rate
total = combined["summary"]["total"]
passed = combined["summary"]["passed"]
combined["summary"]["pass_rate"] = passed / total if total > 0 else 0.0
with open("reports/pytest.json", "w") as f:
json.dump(combined, f, indent=2)
print(f"📊 Combined: {total} tests, {passed} passed, {combined['summary']['failed']} failed")
EOF
- name: Copy sharded pytest JSON into reports/ (slowest-test durations)
run: |
# Merged reports/pytest.json can lack per-test timings; generate_metrics prefers
# pytest-unit.json / pytest-integration.json / pytest-e2e*.json when present.
find pytest-artifacts -type f \( \
-name 'pytest-unit.json' -o -name 'pytest-integration.json' -o \
-name 'pytest-e2e.json' \
\) -exec cp -f {} reports/ \; 2>/dev/null || true
ls -la reports/pytest-*.json 2>/dev/null || true
- name: Combine coverage with coverage.py (matches local make coverage-enforce)
run: |
PYTHONPATH="${PYTHONPATH}:$(pwd)"
export PYTHONPATH
mkdir -p reports
set -e
# Merge unit + integration + E2E using the same engine as `coverage combine` locally.
# Cobertura XML merging inflated denominators vs coverage.py (~63% vs ~70%).
UNIT=$(find coverage-artifacts -name 'coverage-data.unit' 2>/dev/null | head -1 || true)
INT=$(find coverage-artifacts -name 'coverage-data.integration' 2>/dev/null | head -1 || true)
E2E=$(find coverage-artifacts -name 'coverage-data.e2e' 2>/dev/null | head -1 || true)
if [ -z "$UNIT" ] || [ -z "$INT" ] || [ -z "$E2E" ]; then
echo "❌ Missing coverage data files from test-unit / test-integration / test-e2e"
echo "UNIT=$UNIT INT=$INT E2E=$E2E"
ls -laR coverage-artifacts 2>/dev/null || true
exit 1
fi
rm -f .coverage .coverage.*
cp "$UNIT" .coverage.unit
cp "$INT" .coverage.integration
cp "$E2E" .coverage.e2e
coverage combine
coverage report --fail-under=70 --show-missing
coverage xml -o reports/coverage-unified.xml
echo "✅ coverage.xml written to reports/coverage-unified.xml"
- name: Generate unified coverage summary
if: always()
run: |
{
echo "# 📊 Unified Test Coverage Report"
echo ""
if [ -f reports/coverage-unified.xml ]; then
COVERAGE=$(python3 -c "import xml.etree.ElementTree as ET; tree = ET.parse('reports/coverage-unified.xml'); root = tree.getroot(); print(f\"{float(root.attrib.get('line-rate', 0)) * 100:.1f}%\")" 2>/dev/null || echo "N/A")
BRANCH_COVERAGE=$(python3 -c "import xml.etree.ElementTree as ET; tree = ET.parse('reports/coverage-unified.xml'); root = tree.getroot(); print(f\"{float(root.attrib.get('branch-rate', 0)) * 100:.1f}%\")" 2>/dev/null || echo "N/A")
echo "## Unified Coverage Summary"
echo "- **Line Coverage**: $COVERAGE"
echo "- **Branch Coverage**: $BRANCH_COVERAGE"
echo ""
THRESHOLD=70
echo "- **Threshold**: ${THRESHOLD}%"
echo ""
COVERAGE_NUM="${COVERAGE%\%}"
COVERAGE_NUM="${COVERAGE_NUM%%.*}"
if [ "$COVERAGE_NUM" != "N/A" ] && [ "$COVERAGE_NUM" -ge "${THRESHOLD}" ]; then
echo "✅ Coverage meets threshold!"
elif [ "$COVERAGE_NUM" != "N/A" ]; then
echo "⚠️ Coverage below threshold (${THRESHOLD}%)"
fi
else
echo "⚠️ Unified coverage report not found"
fi
} >> "$GITHUB_STEP_SUMMARY"
# Flaky tests (passed after pytest-rerunfailures retry; match generate_metrics.py / nightly.yml)
shopt -s nullglob
for report in reports/pytest*.json; do
if [ -f "$report" ]; then
FLAKY_COUNT=$(jq -r '[.tests[] | select((.outcome == "rerun" and .call.outcome == "passed") or (.outcome == "passed" and .rerun == true))] | length' "$report" 2>/dev/null || echo "0")
if [ "${FLAKY_COUNT:-0}" -gt 0 ]; then
{
echo ""
echo "## ⚠️ Flaky tests (passed on rerun)"
echo "- **Report**: \`$(basename "$report")\` — **count**: $FLAKY_COUNT"
echo ""
echo "### Node IDs"
echo '```'
jq -r '.tests[] | select((.outcome == "rerun" and .call.outcome == "passed") or (.outcome == "passed" and .rerun == true)) | .nodeid' "$report" 2>/dev/null || echo "Unable to extract"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
fi
fi
done
shopt -u nullglob
- name: Note on combined coverage threshold
run: |
# Line threshold is enforced by `coverage report --fail-under=70` in the combine step
# (same semantics as `make coverage-enforce` on a combined .coverage file).
if [ -f reports/coverage-unified.xml ]; then
echo "✅ Unified coverage file present"
fi
- name: Generate code quality metrics
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
run: |
mkdir -p reports
# Generate complexity metrics for dashboard
radon cc src/podcast_scraper/ -a -s --total-average --json > reports/complexity.json 2>/dev/null || echo '{"total_average": 0}' > reports/complexity.json
radon mi src/podcast_scraper/ -s --json > reports/maintainability.json 2>/dev/null || echo '[]' > reports/maintainability.json
python scripts/dashboard/capture_quality_for_metrics.py --reports-dir reports --vulture-min-confidence 80
- name: Load metrics history from gh-pages (for wily trends)
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
env:
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
mkdir -p metrics
bash scripts/dashboard/fetch_metrics_file_from_pages.sh history-ci.jsonl metrics/history-ci.jsonl
if [ -s metrics/history-ci.jsonl ]; then
python scripts/dashboard/repair_metrics_jsonl.py metrics/history-ci.jsonl --in-place || true
fi
- name: Install wily
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
run: pip install wily
continue-on-error: true
- name: Build wily baseline
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
run: |
python -m wily build src/podcast_scraper -n 50 || echo "Wily build failed (non-blocking)"
continue-on-error: true
- name: Generate wily trend reports
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
run: |
mkdir -p reports/wily
python -m wily report src/podcast_scraper/workflow/orchestration.py > reports/wily/orchestration-trends.txt 2>&1 || echo "Wily report failed (non-blocking)"
python -m wily report src/podcast_scraper/ > reports/wily/overall-trends.txt 2>&1 || echo "Wily report failed (non-blocking)"
continue-on-error: true
- name: Generate wily trends JSON for metrics pipeline
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
run: |
mkdir -p reports/wily
python scripts/dashboard/wily_trends_to_json.py --reports-dir reports --history metrics/history-ci.jsonl --output reports/wily/trends.json || echo '{"complexity_trend":"N/A","maintainability_trend":"N/A","files_degrading":[],"files_improving":[]}' > reports/wily/trends.json
continue-on-error: true
- name: Collect pipeline metrics
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
run: |
# Install ML dependencies for pipeline execution
pip install -e ".[ml]" || echo "⚠️ ML dependencies not available, skipping pipeline metrics"
# Run minimal pipeline to collect performance metrics
python scripts/dashboard/collect_pipeline_metrics.py \
--output reports/output/pipeline_metrics.json \
--max-episodes 1 || echo "⚠️ Pipeline metrics collection failed (non-blocking)"
continue-on-error: true # Don't fail if pipeline metrics collection fails
- name: Generate metrics JSON from unified coverage
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
run: |
# Create minimal reports structure for metrics generation
# The unified coverage is already in reports/coverage-unified.xml
# Copy it to reports/coverage.xml for metrics script compatibility
if [ -f reports/coverage-unified.xml ]; then
cp reports/coverage-unified.xml reports/coverage.xml
fi
# Create minimal JUnit XML if not available (metrics script expects it)
if [ ! -f reports/junit.xml ]; then
python3 -c "import xml.etree.ElementTree as ET; root = ET.Element('testsuites'); root.set('tests', '0'); root.set('failures', '0'); root.set('time', '0'); tree = ET.ElementTree(root); tree.write('reports/junit.xml', encoding='utf-8', xml_declaration=True)"
fi
# Create minimal pytest JSON if merge didn't produce one
if [ ! -f reports/pytest.json ]; then
echo "⚠️ No pytest JSON found, creating minimal fallback"
echo '{"summary": {"total": 0, "passed": 0, "failed": 0, "skipped": 0}, "duration": 0, "tests": []}' > reports/pytest.json
else
echo "✅ Using merged pytest.json with $(jq '.summary.total' reports/pytest.json) tests"
fi
# Generate metrics JSON (includes complexity and pipeline metrics)
# Coverage threshold (70%) is for combined coverage only (matches Makefile / enforcement step)
# Save with CI prefix for unified dashboard
python scripts/dashboard/generate_metrics.py \
--reports-dir reports \
--output metrics/latest-ci.json \
--history metrics/history-ci.jsonl \
--pipeline-metrics reports/output/pipeline_metrics.json \
--coverage-threshold 70 \
--slowest-top-n 10 || echo "⚠️ Metrics generation failed (non-blocking)"
- name: Update metrics history
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
env:
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
# Load existing history from live Pages (deploy-pages) or gh-pages git branch
mkdir -p metrics
bash scripts/dashboard/fetch_metrics_file_from_pages.sh history-ci.jsonl metrics/history-ci.jsonl
if [ -s metrics/history-ci.jsonl ]; then
python scripts/dashboard/repair_metrics_jsonl.py metrics/history-ci.jsonl --in-place || true
fi
# Append latest metrics as a single JSONL line (pretty-printed JSON breaks line-based parsers)
if [ -f metrics/latest-ci.json ]; then
python scripts/dashboard/append_metrics_history_line.py metrics/latest-ci.json >> metrics/history-ci.jsonl
echo "✅ Appended to CI history"
fi
- name: Generate HTML dashboard
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
env:
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
bash scripts/dashboard/fetch_metrics_file_from_pages.sh latest-nightly.json metrics/latest-nightly.json
if [ ! -s metrics/latest-nightly.json ]; then rm -f metrics/latest-nightly.json; fi
bash scripts/dashboard/fetch_metrics_file_from_pages.sh history-nightly.jsonl metrics/history-nightly.jsonl
python scripts/dashboard/generate_dashboard.py \
--unified \
--output metrics/index.html || echo "⚠️ Dashboard generation failed (non-blocking)"
python scripts/dashboard/consolidate_dashboard_data.py \
--input-dir metrics \
--output metrics/dashboard-data.json || echo "⚠️ dashboard-data.json merge failed (non-blocking)"
- name: Upload unified coverage report
if: always()
uses: actions/upload-artifact@v7
with:
name: coverage-unified
path: reports/coverage-unified.xml
retention-days: 30
if-no-files-found: ignore
- name: Upload coverage to Codecov
if: always()
uses: codecov/codecov-action@v6
with:
files: reports/coverage-unified.xml
flags: unittests
name: codecov-unified
fail_ci_if_error: false
token: ${{ secrets.CODECOV_TOKEN }}
- name: Upload metrics as artifact
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
uses: actions/upload-artifact@v7
with:
name: metrics
path: metrics/
# Longer retention so `make fetch-ci-metrics` can merge more run-* bundles locally (was 30).
retention-days: 90
- name: Upload wily reports as artifact
if: always() && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release/2.4' || github.ref == 'refs/heads/release/2.5' || github.ref == 'refs/heads/release/2.6')
uses: actions/upload-artifact@v7
with:
name: wily-reports
path: |
reports/wily/
.wily/
retention-days: 90
continue-on-error: true
# NOTE: Metrics deployment strategy:
# - Main branch: Generates metrics data + unified dashboard HTML
# - Release branches: Only generate metrics data (JSON/JSONL), no dashboard HTML
# - The docs.yml workflow deploys everything to GitHub Pages
# - Metrics are available as workflow artifacts for download
# - The unified dashboard uses a dropdown to select CI vs Nightly builds
# Build package - validates package can be built correctly
# Builds source distribution (sdist) and wheel distribution
# Catches packaging issues (pyproject.toml errors, missing files, etc.)
# Fast check (~2-3 min) that gates test-unit to catch packaging issues early
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Set up Python 3.11
uses: actions/setup-python@v6
with:
python-version: "3.11.8"
cache: "pip"
cache-dependency-path: pyproject.toml
- name: Install build tools
run: |
python -m pip install --upgrade pip
pip install build
- name: Build package
run: make build