Ops: Tier 4 re-drain and projection rebuild for PRD 016B

[santhonys]

Why

PRD 016B's schema and UI are live, and production has been backfilled from existing cds_fields. That backfill is only the floor. The new Tier 4 C21/C22 cleaner logic has not been applied to historical Tier 4 artifacts, so the measured Phase 0 ED/EA answerability gains are not fully visible in production yet.

Scope

Re-run Tier 4 extraction for 2024-25+ tier4_docling documents, then rebuild the browser projection so cds_fields and school_browser_rows pick up the improved C21/C22 values.

Recommended operator path

Use an operator machine or self-hosted runner, not the bounded GitHub-hosted daily worker. The hosted ops workflow caps runs at 100 rows and is intended for pending extraction drains, not corpus re-drains.

Suggested steps:

1. Bump the Tier 4 producer version if we want selected-result precedence to distinguish newly cleaned artifacts from older tier4_docling rows.
2. Re-extract 2024-25+ Tier 4 rows with the updated cleaner.
3. Rebuild the browser projection:

python tools/browser_backend/project_browser_data.py --full-rebuild --apply

4. Re-run the PRD 016B audit:

python tools/browser_backend/prd016b_phase0_audit.py

5. Re-run public contract checks:

tests/api/admission_strategy_contract.sh
tests/api/positioning_contract.sh

Acceptance checks

• Remote school_browser_rows ED count answerability moves materially above the current SQL-backfill floor.
• Top-200 ED-offered answerability remains at or above the Phase 0 gate documented in docs/plans/prd-016b-phase-0-findings.md.
• No increase in ed_math_inconsistent rows beyond the known verifier-rejected cases unless manually explained.
• /schools/[school_id] admission strategy card renders ED and wait-list blocks for rows where valid counts are present, even if the explicit checkbox field was missing.

Current production floor after PRD 016B migrations

• ed_applicants=not.is.null: 38 rows
• ed_offered=is.true: 119 rows
• wait_list_policy=is.true: 211 rows
• ea_offered=is.true: 99 rows
• admission_strategy_card_quality=eq.ok: 470 rows

[santhonys]

Tier 4 re-drain production pass is complete from branch santhonys/tier4-redrain-20260502; PR is #40.

Summary:

• queued 343 runnable 2024+ Tier 4 PDF rows; 340 extracted successfully as tier4_docling 0.3.1
• skipped 1 known not_applicable U Maine system row
• 3 rows still fail after retry because the source PDFs are invalid/malformed:
  • Johns Hopkins 2025-26: 4a844d1d-241a-475a-b0d9-0b14374c0640
  • Williams 2025-26: ed6f2933-0896-4099-9b86-300179782f47
  • Williams 2024-25: 5d262e0c-6d68-46b4-94a2-67de77259e92
• rebuilt browser projection: 197,856 fields and 469 browser rows from 502 documents
• final PRD 016B audit: ED true 145, EA true 114, early-plan true 190, ED answerable 104, top-200 answerable 68, waitlist true 210, quality_ok 363

Verification passed:

• python3 -m unittest tools.browser_backend.project_browser_data_test
• PYTHONPATH=tools/extraction_worker python3 -m unittest tools/extraction_worker/test_worker_projection_refresh.py tools/extraction_worker/test_schema_dispatch.py
• tests/api/admission_strategy_contract.sh
• tests/api/positioning_contract.sh