Skip to content

refactor: captions/audio pipeline — consolidate the ep5 fix series into clean architecture #227

Description

@SecurityQQ

Context

The ep5 incident produced a series of working but surgical fixes (PR #223, PR #226, issue #225). They ship correct behavior — verified frame-by-frame on formula-care-ep5 — but each one patched a symptom inside an architecture that made those bugs possible. This issue tracks the proper refactor.

What the fix series bolted on (and where it creaks)

  1. Caption timing is corrected in 4 different places instead of one:

    • transformCue (shift/drop/clamp vs clip trim window) in merge-ass.ts
    • monotonizeEntries (non-monotonic whisper words) in ass.ts
    • refineWordTimings (leading/trailing silence vs first/last word) in primitives/silence.ts
    • per-node analysis memoization + refine wiring inside audio-element.ts
      A caption cue is now transformed by up to 3 sequential passes owned by 3 different modules. There should be ONE timing model: words → (refine) → cues → (window remap) → timeline, as a single pure pipeline with one test surface.
  2. Two resolve paths still exist (the root cause of the 429 storm): standalone resolve.ts (~1400 lines, now with memoizedElementResolve bolted on) vs renderers with withDedup/pendingFiles. Stage 3/4 of the graph plan (render = compile + executePlan) was landed in PR refactor: computation-graph IR, render lifecycle, renderers split #223, but async components still generate through the standalone path during resolveLazy, outside the executor and its concurrency limit (semaphore follow-up in async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225 still open). The executor should be the only thing that generates; standalone resolvers should be thin wrappers over it.

  3. Caption pipeline state lives in /tmp with hand-rolled path management. uniqueTempPath fixed the Date.now() collision, but the deeper problem is that renderCaptions/mergeAssFiles communicate through temp files at all. In-memory ASS structures (parse once, transform, serialize once at burn time) would delete the entire class of file-collision/ordering bugs — and most of merge-ass.ts.

    • Related: remaining Date.now()-only temp paths in packshot.ts, slider.ts, swipe.ts, text-measure.ts, render.ts carry the same latent race, just outside the hot path today.
  4. renderCaptions / resolveSrtContent is a 4-way branch stack (srt path / string src / AudioNode src / speech src / native words / whisper fallback) that grew a 5th branch for memoized AudioNode transcripts. Captions should consume ONE input shape (words + text), with a small resolver in front mapping every src type to it.

  5. AudioNode is doing too much (audio-element.ts): element identity, thenable semantics, meta write-back, per-option memoization maps, refine policy, gateway-model fallback. The analysis layer (transcribe/silence/ranges + memo) wants to be its own module with AudioNode as a thin facade.

  6. HoistedCaption/window plumbing (flatten.tsrender.tsmerge-ass.ts) passes trim windows positionally through 3 hops with ! indexing (flatten.hoistedCaptions[i]!). Fold window + offset + emoji shift into one per-clip caption descriptor computed in one place.

  7. Idempotency-Key is client-side only. The API stores the key but (to verify) does not dedupe on it server-side — see async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225. API-side enforcement is the real guarantee; the SDK header is best-effort.

Suggested shape (rough)

  • captions/ module: words → cues → timeline pure pipeline (monotonize + refine + group + window-remap as composable steps), in-memory ASS model, single serializer; renderers only call it
  • generation: executor-only (executePlan), standalone resolvers delegate; resolveLazy-phase generation goes through the same semaphore/dedup (async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225)
  • audio analysis: primitives/analysis.ts owning transcribe/silence/refine/memo; AudioNode delegates
  • kill temp-file IPC between caption stages; uniqueTempPath everywhere a real file is still needed (packshot/slider/swipe/text-measure/render)

References

Non-goal: changing public syntax (vid.audio, speechRange, <Captions src>, audio: "native") — all of it stays; this is internal consolidation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions