Skip to content

INGESTION.md

Latest commit

 

History

History
104 lines (81 loc) · 5.88 KB

INGESTION.md

File metadata and controls

104 lines (81 loc) · 5.88 KB
Raw

Course & Program Data Ingestion

How course and program data is sourced from the Loyola catalog (catalog.luc.edu) and kept correct.

Principle: the catalog is the single source of truth

Every course number maps to exactly one title. A course title is always looked up from the catalog by its number — never copied from a sibling program file or typed by hand. The Audit tab de-duplicates shared courses across programs by code + title (src/utils/progress.ts), so a divergent title for one number silently breaks credit-sharing. The pipeline and the course integrity test exist to enforce this.

Artifacts

Path What it is Generated by
src/data/dept-courses/<DEPT>.json Per-department course list (topics variants consolidated). Used for the UI and elective-pool population. scripts/fetch-dept-courses.mjs
src/data/course-index.json Flat, unconsolidated code → {title, credits} master map. The authoritative number→name lookup. scripts/fetch-dept-courses.mjs
src/data/course-exceptions.json Documented exceptions to the integrity check (page-less prefixes, known-missing codes). Keep small and shrinking. hand-maintained, tracked in LIKELY_WRONG.md
src/data/<program>.json Major/minor/grad program requirements + roadmaps. hand/AI-authored, titles normalized from the index

Pipeline order

Always ingest courses first, then make program data conform to them.

# 1. Scrape the master course list for every known department (writes
#    src/data/dept-courses/*.json and rebuilds src/data/course-index.json)
node scripts/fetch-dept-courses.mjs
#    …or refresh specific departments (merges into the index):
node scripts/fetch-dept-courses.mjs PHIL MATH

# 2. Rewrite every program's course TITLES from the index (one name per number).
#    Reports any code not found in the catalog. Add --dry-run to preview.
node scripts/normalize-program-courses.mjs

# 3. Verify nothing drifted. The "course integrity" test fails if any program
#    course code is missing from the index or its title differs.
npm test

# 4. Full gate before committing.
npm run typecheck && npm run lint && npm run build

Program ingestion (majors / minors / graduate)

Whole program files (src/data/<id>.json) are reproducible from committed inputs, not hand-typed. Three layers, merged by one generator (each owns disjoint fields):

Input Path Owns
Manifest (seed) scripts/program-manifest.mjs id, name, degree, school, department, kind, catalogUrl, credit-total overrides
Extract (deterministic scrape) src/data/program-extracts/<id>.json courses (code/title/credits), elective pools, roadmap
Overlay (AI, fuzzy semantics) src/data/program-refine/<id>.json choose-N / credit caps: creditsRequired, note, requirementGroup, choiceNote
Supplement (non-catalog) src/data/program-supplements/<id>.json checklist (AP/transfer/milestones), program note, hasCompletionEstimate, credit pins

Pipeline:

node scripts/fetch-program.mjs            # scrape program pages -> program-extracts/<id>.json (+ contentHash)
#   author/refresh program-refine/<id>.json + program-supplements/<id>.json as needed (only fuzzy/non-catalog bits)
node scripts/build-programs.mjs           # merge -> src/data/<id>.json (+ programs.ts at full rollout)
node scripts/normalize-program-courses.mjs
npm test && npm run check:programs        # programIntegrity + byte-reproducibility drift guard
  • Titles/credits in the extract are reconciled against course-index.json, so the catalog stays the source of truth for names. The build is deterministic and offline (it never calls a model — the AI overlay is committed JSON).
  • Migration is incremental. A manifest entry with generated: true is adopted: its src/data/<id>.json must byte-match build-programs.mjs output (npm run check:programs fails otherwise), so hand-editing an adopted file is forbidden — change the manifest/extract/overlay/ supplement instead. Entries without the flag are legacy hand files awaiting their migration wave.
  • Run node scripts/build-programs.mjs --out /tmp/regen <id> and diff against the current file to vet a program before flipping generated: true.

Updating when the catalog rolls over

When LUC publishes a new catalog year (and preserves the current page structure):

  1. Bump CATALOG_YEAR in scripts/fetch-dept-courses.mjs.
  2. Run the pipeline above. Re-run #1 with no arguments to rebuild the whole index.
  3. Run node scripts/normalize-program-courses.mjs — it reports codes not in course-index. For each: it's either a renumbered course (fix the code in the program JSON), a new page-less prefix (add to prefixesWithoutCatalogPage), or a genuinely missing code (add to knownMissingCodes and note it in LIKELY_WRONG.md).
  4. npm test until the course integrity test is green, then npm run build.

Adding a new department: add a { code, name } entry to DEPARTMENTS in scripts/fetch-dept-courses.mjs (override slug only if the catalog URL differs from the lowercase code), then run the pipeline.

Scope & known gaps

  • Titles only. The normalizer does not change credit hours; program credit totals depend on them. Catalog-vs-data credit mismatches are reported in CREDITS_DISCREPANCIES.md.
  • Page-less subjects. A few prefixes (FORS, FRSC, HSRV) have no /course-descriptions/<dept>/ page; their titles live only in program JSON and are exempt from the integrity check.
  • Wrong course numbers. Where a program used the wrong number for a course, normalization makes the displayed name match the (wrong) number — these are tracked in LIKELY_WRONG.md for a code fix, not silently "corrected" by changing the title.
  • Roadmaps reference course IDs (not titles), so they inherit normalized titles automatically; no separate step is needed.