Skip to content

architecture.md

Latest commit

 

History

History
272 lines (188 loc) · 18.9 KB

architecture.md

File metadata and controls

272 lines (188 loc) · 18.9 KB
Raw

Architecture

Overview

Two-phase LLM pipeline. The offline phase runs once (or on demand) to build persistent artifacts. The online phase runs on every query.

Offline (setup)

flowchart TD
    A["Watch History CSVs"] --> B["Ingestion<br/>Netflix / Prime / Apple TV / Manual"]
    B --> C["TMDB Metadata Fetch<br/>cached, title-cleanup fallback"]
    C --> D["Watch Index<br/>TMDB ID + normalized title"]
    D --> E["Claude Haiku Enrichment<br/>only cached on success"]
    E --> F["Engagement Scoring<br/>completion + rewatch + recency"]
    F --> G["Taste Profile - Claude Sonnet<br/>batched, merged prose"]
Loading

Online (query)

flowchart TD
    A["User Query - natural language"] --> B["parse_intent - Claude Sonnet<br/>QueryIntent: genres, moods, similar_to, platforms..."]
    B --> C1["TMDB Discover<br/>structured filter"]
    B --> C2["Claude Suggestions<br/>semantic, taste-aware"]
    C1 --> D["Merge + Deduplicate Candidates"]
    C2 --> D
    D --> E["Filter Watched Titles<br/>content-type-aware"]
    E --> F["Annotate Streaming Availability<br/>TMDB watch providers, cached"]
    F --> G["enrich_batch - Claude Haiku<br/>cached descriptions"]
    G --> H["rank_candidates - Claude Sonnet<br/>query relevance primary, taste profile secondary"]
    WI[("Watch Index")] -. watch history .-> E
    TP[("Taste Profile")] -. secondary signal .-> H
Loading

Components

Ingestion (recommender/ingestion/)

Module Source Key fields
netflix.py raw Netflix export zip with ViewingActivity.csv inside title, duration, timestamp, profile
prime.py raw Prime Video export zip with Viewing History.csv inside title, duration, timestamp
apple_tv.py raw Apple privacy export zip with Video Play Activity.csv inside title, duration, timestamp, storefront
manual.py data/manual/tv.csv + movies.csv title only; synthetic duration + current timestamp

All parsers emit WatchEvent dataclasses (defined in base.py). Manual entries use datetime.now() as timestamp so they score competitively with platform data. Each file type deduplicates independently (TV and movie lists use separate seen sets).

base.py also provides a shared is_bonus_content() filter, wired into all four parsers, that drops clips/trailers/featurettes/sing-alongs/deleted-scene rows before they ever reach TMDB matching.

Signals (recommender/signals.py)

Computes a score per unique title from raw watch events:

  • Completion ratio (50%) — watched_duration / runtime (uses TMDB runtime when available, falls back to 45min TV / 90min movie)
  • Rewatch bonus (30%) — log-scale multiplier for titles watched more than once
  • Recency decay (20%) — true half-life decay: 0.5 ^ (days / half_life_days)

TMDB Client (recommender/tmdb_client.py)

Four roles:

  1. Metadata lookup (get_metadata) — text search by title + content type, returns TmdbMetadata. On miss, tries cleaned title variants (strips parenthetical suffixes, edition markers, episode prefixes, season markers) and falls back to alternate content type. A post-search validator rejects a top-scoring candidate that isn't plausibly related to the source title (title-similarity check, boosted by an original-language bonus) rather than trusting vote/popularity weight alone — this stops generic/short titles like "Don" from matching an unrelated, more popular result. Distinguishes an outright API/search failure from a genuine zero-results response. Cached at recommender/cache/tmdb/.

  2. Candidate discovery (search_by_filters) — calls TMDB Discover endpoint with genre IDs, origin countries, languages, and year range. Page-limited to MAX_DISCOVER_PAGES (20). Failed fetches are logged, not silently swallowed.

  3. Disambiguation candidates (get_disambiguation_candidates) — cross-type ranked candidate search (checks both the hinted content type and the alternate one) used when a manually-added title can't be resolved to a single confident match. Backs the manual-add disambiguation modal (see below).

  4. Watch providers (get_watch_providers) — looks up flatrate streaming availability by region. Cached at recommender/cache/providers/.

Genre name -> TMDB ID mappings are stored as module-level dicts. TV has no "thriller" genre; it maps to Mystery (9648).

Watch Index (recommender/watch_index.py)

Persisted at recommender/cache/watch_index.json.

Content-type-aware dual-key lookup:

  • Primary key — TMDB ID
  • Fallback key — (normalized_title, content_type) tuple

This prevents cross-media false matches (watching the TV show "Fargo" won't block the movie "Fargo" from recommendations).

Each entry also aggregates provenance across all its watch events: platforms (sorted source list, merged on dedup) and last_watched (latest ISO timestamp). This backs the /history source-provider filter and "Recently watched" sort; manual additions carry their own manual source.

Enricher (recommender/enricher.py)

Calls Claude Haiku to generate 2-3 sentence semantic descriptions per title. Only successful LLM responses are cached — fallback descriptions (keyword strings) are not persisted, allowing retry on subsequent runs.

The enrichment index (index.json) uses identity keys: content_type/tmdb_id for resolved titles (e.g. tv/12345) and unknown/slug for unresolved ones. A _title_keyed_enrichments() bridge in setup.py converts identity keys back to display titles for the taste profile builder.

Taste Profile Builder (recommender/taste_profile_builder.py)

Processes ALL enriched titles (no limit) in batches of 200. Each batch produces a mini taste profile, then a merge pass consolidates them into one document covering every taste cluster. Includes rate limit retry with backoff.

Previous profiles are auto-backed up with timestamps before rebuild.

Negative preferences from the feedback system are included in the prompt, generating a "What you don't enjoy" section.

The merged output is capped at 15 clusters. Family/kids/seasonal clusters (Disney, Pixar, Christmas, holiday, children's content, etc.) are sorted after personal-taste clusters so a cap never silently drops a genre the user actually cares about in favor of shared/family viewing. Markdown section headings are renumbered sequentially after merging so a dropped or reordered cluster doesn't leave gaps.

User Store (recommender/user_store.py)

SQLite storage (events.db) for user-managed state:

  • Watchlist (saved_titles with status watchlist) — save/unsave from any UI page, CSV export
  • Dismissed (saved_titles with status dismissed) — excluded from recommendations
  • Ratings (title_ratings) — liked/disliked, applied as score multipliers during profile rebuild
  • Manual archive (manual_archive_entries) — titles added via CLI or web UI
  • Disliked titles inform negative preference prompting in the taste profile

All lookups use TMDB-ID-first matching with normalized-title fallback. UserStateIndex (in user_state.py) provides a read-only snapshot for fast matching in query filtering and UI rendering.

find_conflict(content_type, tmdb_id) checks the watchlist and manual archive for an existing record of a title, used by the manual-add disambiguation flow to warn before a duplicate add. mark_watched_from_watchlist() promotes a watchlist entry straight to the archive.

Note: recommender/feedback.py is deprecated. The original JSON-based feedback was migrated to the SQLite tables above.

Manual-Add Disambiguation (/archive/resolve, /archive/confirm in web.py)

Manually adding a title from the web UI can be ambiguous (multiple TMDB matches, wrong content type, or a title the user already has recorded). The flow:

  1. /archive/resolve — runs TmdbClient.get_disambiguation_candidates() (cross-type ranked search) and renders _archive_disambiguate.html with the candidates.
  2. Each candidate is checked for a conflict two ways: user_store.find_conflict() against the watchlist/manual-archive tables, then — if that finds nothing — a lookup against the loaded watch_index by (content_type, tmdb_id), so titles that only exist in ingested platform history (not the SQL tables) still surface as "Already in your watch history" rather than being silently re-added.
  3. /archive/confirm — commits the user's choice: add to archive, mark watched, or keep as an unmatched manual entry with no TMDB id.

Query Engine (recommender/query_engine.py)

The online pipeline:

1. Intent parsing — Claude Sonnet parses natural language into QueryIntent with validation, defaults, and type coercion. Supports conversational context (refinements like "more like that", "but British"). Detects platform filters ("on Netflix"). All API calls have 30s timeout.

2. Hybrid candidate generation — Two sources run in parallel:

  • TMDB Discover (structured metadata filter)
  • Claude suggestions (semantic, taste-aware — always runs, not just as fallback)

Both TV and movie versions are kept for suggested titles (ranker decides). Results are deduplicated by TMDB ID.

3. Watch filter — Content-type-aware exclusion via watch index.

4. Streaming availability — Each candidate annotated with flatrate providers for the configured region. Optionally filtered to user's subscribed platforms.

5. Ranking — Claude Sonnet ranks with query relevance as primary signal, taste profile as tiebreaker. Returns JSON with title, explanation, and score.

Special modes:

  • "why not X?" — traces a title through the pipeline and explains exactly where it was filtered
  • "abandoned" queries — checks watch history for partial viewing and advises whether to continue

Mood Match Wizard (recommender/wizard.py, recommender/wizard_flow.py)

A guided alternative to free-text search that turns the taste profile into an interactive question loop, then hands a synthesized QueryIntent to the same online pipeline.

The flow is a hybrid of deterministic and LLM-led turns:

  1. Content-type tap (wizard_flow) — an instant first question (movie / TV / either) rendered with no LLM call. It counts as question 1 toward the cap.
  2. Adaptive loop (wizard.next_turn) — one role="reason" call per turn. The model sees the taste profile as a prior plus the answers so far and returns either the next question (chips + optional free text) or a finish signal carrying a QueryIntent and a free-text ranking note.
  3. Soft floor / hard cap — the wizard must keep asking until it has WIZARD_MIN_QUESTIONS answers before it may finish on its own; below the floor an early recommend is rejected and one more question is forced (relenting if the model still will not ask, so the loop is bounded). WIZARD_MAX_QUESTIONS is the hard cap, enforced here independent of what the model returns. The user can always bail early with Show me something now.
  4. Review — a recap surface before finalizing, so answers are visible and editable.
  5. Finalize — the recommendation runs as a background job (job_registry); the page polls /wizard/jobs/<id>/poll (via _polling.html) until results render. When the model finishes, its adaptive intent is merged over the deterministic seed (merge_intent_with_seed).
  6. Refine / replay — /wizard/refine applies deterministic directives (shorter, lighter, more obscure, surprise me, or free text) to the existing intent and re-runs while hard-excluding already-shown titles; /wizard/replay re-runs a stored wizard run from its structured intent (not its recap text).

WizardState is carried in a hidden form field across turns (size- and turn-count-bounded as a malformed-payload guard, not a security boundary). Routes live in web.py (/wizard, /wizard/next, /wizard/jobs/<id>/poll, /wizard/refine, /wizard/replay); templates are wizard.html, _wizard_step.html, _wizard_review.html, _wizard_results.html, and _polling.html.

Web UI (recommender/web.py)

Flask app serving:

  • / — Home: search bar (HTMX-powered), taste profile clusters (expandable, markdown-rendered), archive poster wall
  • /wizard — Mood Match guided wizard (see the wizard section above)
  • /history — Watch archive with switchable views (list, poster grid, compact). Search + type filter + source-provider filter + rating filter + A-Z/Z-A/recently-watched sort.
  • /title/:id — Title detail with poster, TMDB overview, AI analysis, credits, keywords, TMDB link
  • /recommend — Standalone discover page
  • /watchlist — Saved titles rendered as rich cards (cached poster, rating, genres, streaming availability, TMDB/IMDB links), CSV export via /watchlist/export
  • /watchlist/save, /watchlist/unsave, /watchlist/dismiss, /watchlist/remove, /watchlist/watched — HTMX toggle/transition endpoints for inline management from any page
  • /archive/add — Manual add; /archive/resolve and /archive/confirm handle disambiguation when the added title is ambiguous or already recorded (see above)
  • /searches — Query history with user state badges (watchlist, archived, dismissed) per result; recent searches are also surfaced inline in the home search suggestion row

Streaming provider names are consolidated server-side (_consolidate_providers(), mirrored client-side for filter dropdowns) so ad-tier variants and channel resells display under one canonical brand.

LLM Abstraction (recommender/llm.py)

ABC-based LLMClient with AnthropicClient and GeminiClient. Call sites use roles instead of model names:

  • role="fast" — enrichment (high volume, simple descriptions)
  • role="reason" — intent parsing, ranking, taste profile, suggestions (complex reasoning)

Model names are resolved from config.yaml per provider:

models:
  anthropic:
    fast: claude-haiku-4-5-20251001
    reason: claude-sonnet-4-6
  gemini:
    fast: gemini-2.5-flash
    reason: gemini-2.5-pro

Gemini-specific handling: output token scaling (x3), JSON mode via response_mime_type, rate limit retry (429/RESOURCE_EXHAUSTED/504), extended timeout for Pro thinking mode.

Token usage and cost tracking via UsageStats — accumulated per query, printed after results.

Title Overrides (recommender/overrides.py)

data/overrides.json maps raw platform titles to corrected titles, direct TMDB IDs, content type corrections, or skip. Auto-detected at setup time — if the overrides file is newer than the watch index, triggers a rebuild without --refresh-data.

CLI (recommender/main.py)

Rich-powered output with spinners during API calls and panel-formatted results. Stderr/stdout separation for pipe-friendly usage. Interactive REPL with conversational context and inline feedback commands (+liked, +disliked, +add). Token usage and cost printed after each query.

Deploy Tooling (tools/merge_user_state.py)

Streamline runs on more than one machine (local + home server), each accumulating its own watchlist/rating/history changes. This tool merges user-generated state — the db form merges saved_titles/title_ratings/manual_archive_entries between two copies of the app's SQLite user-store DB (data/streamline.db, i.e. config.EVENT_DB_PATH — not recommender/cache/events.db, which is unrelated and always empty); the history form merges two query_history.json files. Both merge "other" into "local" in place, so a deploy never silently clobbers changes made on the other side. Derived/cache data (TMDB metadata, enrichments, provider availability) is untouched since it's rebuilt from setup, not user input.

Cache Layout

recommender/cache/
  tmdb/                          TMDB metadata (JSON, by content_type/tmdb_id)
  enrichments/
    tv/{tmdb_id}.txt
    movie/{tmdb_id}.txt
    unknown/{slug}.txt
    index.json                   identity-keyed description index (content_type/tmdb_id -> text)
  providers/
    {content_type}/{region}/{tmdb_id}.json
  profile_batches/               Intermediate batch profiles (auto-cleaned after merge)
    fingerprint.txt              SHA-256 of scored title list (staleness check)
    batch_01.txt ... batch_NN.txt
  watch_index.json               [{tmdb_id, title, content_type}, ...]
  taste_profile.txt              LLM-generated prose output
  taste_profile_*.txt            Timestamped backups
  feedback.json                  User ratings and additions

Configuration

Secrets come from the environment. Shared application settings live in config.yaml, and optional machine-specific overrides live in config.local.yaml.

Environment variables:

  • TMDB_API_KEY — TMDB v3 API key
  • ANTHROPIC_API_KEY — Anthropic API key
  • GEMINI_API_KEY — Google Gemini API key (AIza* for AI Studio, AQ.* for Vertex AI)
  • OPENAI_API_KEY — OpenAI-compatible API key

.env — optional local convenience for setting those variables (gitignored)

config.yaml — tracked shared settings, organized in sections:

Section Keys Description
(top-level) provider, models.* LLM provider and model assignments (fast/reason roles)
llm.* timeout_*, tokens_*, profile_batch_size, rate_limit_wait Per-call-type timeouts, token limits, batch sizes
wizard.* max_questions, min_questions, max_tokens Mood Match wizard question cap, soft floor, and per-turn output-token ceiling
scoring.* weight_completion, weight_rewatch, weight_recency, default_*_runtime, rewatch_saturation Engagement scoring weights and fallback runtimes
manual.* timestamp, tv_duration_minutes, movie_duration_minutes Synthetic values for manual list titles
(top-level) default_top_n, min_vote_count, recency_half_life_days Recommendation tuning
(top-level) watch_region, streaming_platforms Streaming availability
(top-level) platform_paths.*, manual_*_path, overrides_path Data file locations

config.local.yaml — optional local overrides loaded after config.yaml. Use this for machine-specific values such as platform_paths.* for personal watch-history export zips. config.local.yaml is gitignored; start from config.local.example.yaml.

Provider API keys use the standard environment variable names by default. Add models.<provider>.api_key_env only when a deployment needs a non-standard variable name.

config.py is a thin loader that reads config.yaml, applies config.local.yaml overrides when present, and reads secrets from the environment. All values have sensible defaults — a minimal config.yaml with just provider: works.

Taste Profile Batch Caching

The taste profile builder saves intermediate batch profiles to recommender/cache/profile_batches/ as they complete. A SHA-256 fingerprint of the scored title list detects staleness — if enrichments, scores, or the title set change, cached batches are invalidated. This makes profile rebuilds resumable: if the merge step fails (timeout, rate limit), re-running --refresh-profile loads the cached batches and retries only the merge. Batch files are cleaned up after a successful merge.