From f2461fe2b14f14632bc1e51e4c8d287e7cd504a8 Mon Sep 17 00:00:00 2001 From: Ziyu Huang Date: Mon, 6 Jul 2026 22:04:18 +0800 Subject: [PATCH 1/5] =?UTF-8?q?feat(graphics):=20mlx=5Fimage=20provider=20?= =?UTF-8?q?=E2=80=94=20bridge=20MLX=20native=20gen=20into=20OpenMontage?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the first Apple-Silicon-native image provider. `mlx_image` shells to the sibling mlx-movie-director repo's `run.py image` (Z-Image / Flux2 Klein / Lens) and is auto-discovered by `image_selector` with zero selector edits — the selector-plus-provider seam from the story→image review. Closes OpenMontage's biggest verified image gap: a local, $0 path that advertises the full control surface (controlnet / img2img / reference_image / faceswap / lora / multi_lora) that today only grok_image's edit mode partially covers. The scorer rewards it accordingly — it ranks #1 by weighted score (control=1.0, cost_efficiency=1.0) and survives the edit-mode candidate filter. - tools/graphics/mlx_image.py: provider; action routing (t2i/i2i/controlnet/ faceswap), CLI arg mapping, availability gate (arm64 + MLX_MOVIE_DIRECTOR_DIR + venv + models), JSON_SUMMARY output parsing with dir-scan fallback. - .agents/skills/mlx-movie-director/SKILL.md: Layer-3 bridge (env contract, pipeline choice, controlnet/i2i/faceswap mapping, LoRA stacks). - tests/tools/test_mlx_image.py: 24 tests — registration, capability surface, zero-edit discovery, edit-filter survival, gate (env/venv/full), action routing, arg building, mocked-subprocess execute success + error paths. - docs/REVIEW-story-to-image.md, docs/REVIEW-image-to-video-voice.md: the grounded reviews that motivated this provider (gap analysis + MLX side-by-side). Verified end-to-end: a real Z-Image generation (828 KB PNG, 17.6s, $0) flows through image_selector -> mlx_image -> run.py and lands at output_path with asset_manifest-ready provenance (source_tool=mlx_image, model=mlx-zimage, seed). Two upstream MLX-repo deps (opencv-python, mflux) are missing from its requirements.txt — noted in install_instructions + the skill (tracked upstream). Co-Authored-By: Claude --- .agents/skills/mlx-movie-director/SKILL.md | 109 +++++ docs/REVIEW-image-to-video-voice.md | 532 ++++++++++++++++++++ docs/REVIEW-story-to-image.md | 255 ++++++++++ tests/tools/test_mlx_image.py | 310 ++++++++++++ tools/graphics/mlx_image.py | 535 +++++++++++++++++++++ 5 files changed, 1741 insertions(+) create mode 100644 .agents/skills/mlx-movie-director/SKILL.md create mode 100644 docs/REVIEW-image-to-video-voice.md create mode 100644 docs/REVIEW-story-to-image.md create mode 100644 tests/tools/test_mlx_image.py create mode 100644 tools/graphics/mlx_image.py diff --git a/.agents/skills/mlx-movie-director/SKILL.md b/.agents/skills/mlx-movie-director/SKILL.md new file mode 100644 index 000000000..b9cccc208 --- /dev/null +++ b/.agents/skills/mlx-movie-director/SKILL.md @@ -0,0 +1,109 @@ +--- +name: mlx-movie-director +description: Use when routing image generation to the mlx_image provider (Apple-Silicon-native MLX via the sibling mlx-movie-director repo). Covers pipeline choice (zimage / flux2-klein / lens), ControlNet, i2i, LoRA stacks, faceswap, availability gating, and the MLX_MOVIE_DIRECTOR_DIR / MLX_VENV_PYTHON environment contract. +--- + +# MLX Movie Director (run.py) in OpenMontage + +Use this skill before calling `mlx_image`, and whenever a brief wants a local, +`$0` Apple-Silicon path for ControlNet, true image-to-image, LoRA conditioning, +or BFS face/head swap — the surface no other OpenMontage image provider covers +fully (only `grok_image`'s edit mode partially overlaps). + +## Environment Contract + +`mlx_image` shells out to the sibling MLX repo. Two env vars wire it up (set +them in the OpenMontage deployment environment, not per-call): + +- **`MLX_MOVIE_DIRECTOR_DIR`** — repo root containing + `python/mlx-movie-director/run.py`. Without this the tool is UNAVAILABLE. +- **`MLX_VENV_PYTHON`** — the MLX venv interpreter. Defaults to + `/python/venv/bin/python`. **This venv is per-machine + and NOT auto-created** (per the MLX repo's CLAUDE.md). On a fresh clone or + after `git clean` it is absent; recreate it: + ``` + uv venv python/venv --python 3.12 + uv pip install -r python/mlx-movie-director/requirements.txt --python python/venv/bin/python + ``` + Two runtime deps are not yet in that requirements.txt (tracked as an upstream + MLX-repo gap) — install them into the same venv after the requirements + install: `opencv-python` (the image-quality command imports `cv2`) and + `mflux` (the Z-Image VAE loader). + +Apple Silicon (`arm64`) is required. The provider's `get_status()` returns +UNAVAILABLE with the exact recreate command when the venv is missing — surface +that reason to the user; do not retry silently. + +## Choosing a Pipeline + +The `pipeline` input selects the MLX model family (the `--pipeline` flag passed +through to `run.py image`): + +- **`zimage`** (default) — Moody 12.6 DPO turbo. ~1.5 s/step, 9 steps default. + Best for portraits and general T2I. CFG is opt-in (the model was distilled at + guidance 0.0; set `cfg_scale` > 1.0 only for an experimental dual-forward). +- **`flux2-klein`** — Klein 9B (or 4B). ~10 s/step, 4 steps. Better for + consistent characters; the edit-class backend for angle / i2i / faceswap / + profile. Distilled — CFG inert for T2I. +- **`lens`** — Microsoft Lens 3.8B, pure MLX, high-res. ~0.4 s/step, 20 steps, + 1024² default. `cfg_scale` default 4.0. No LoRA / ControlNet / i2i. + +Resolution: omit `width`/`height` to use the per-pipeline model-optimal default +(zimage/klein 640×960, lens 1024²). Pass `resolution` for a named tier +(`model` / `benchmark` / `large`) or explicit `WxH` (snapped to a multiple of +16) — useful for QA / self-learning at larger sizes. + +## ControlNet, i2i, and Faceswap + +These are the capabilities that justify choosing `mlx_image` over cloud +providers. The provider infers the `run.py image` action from the inputs: + +| Inputs present | run.py action | Notes | +|---|---|---| +| `image`/`image_path` + `face` | `faceswap` | BFS face/head swap (Flux2 Klein + BFS LoRA). Set `face_mode`: `face` (face only) or `head` (head + hair). | +| `image` + `controlnet_type` or `controlnet_strength` | `controlnet` | Z-Image native ControlNet. `controlnet_type` e.g. `pose`. | +| `image` only | `i2i` | Image-to-image. Set `denoise_strength` (0–1; higher = more change). Optional `reference_image` for pose/style conditioning. | +| neither | `t2i` | Text-to-image. | + +For i2i and controlnet, `image`/`image_path` maps to run.py's `--input-image` +(not `--input`). For faceswap it maps to `--input` (the body) and `face` to +`--face` (the source). The provider handles this mapping; callers just pass +`image`/`image_path` and `face`. + +## LoRA Stacks + +Pass `lora_path` (list of `.safetensors` paths) and a paired `lora_scale` list +(same order, one per path). The provider emits `--lora-path`/`--lora-scale` +pairs. If a single scale is given it is broadcast to the first path only — +prefer explicit per-path scales. LoRAs apply to the zimage and flux2-klein +pipelines (lens has no LoRA story). + +## What mlx_image Does NOT Support + +- **No `negative_prompt`** — run.py image T2I has no `--negative-prompt` flag + (Z-Image handles negatives internally). Do not pass one; the selector strips + it. +- **No CUDA / non-Apple-Silicon** — the MLX runtime is MPS-only. On a CUDA box + prefer `comfyui_image` (which bundles FLUX2 NVFP4 for Blackwell/DGX Spark). +- **No `custom_workflow`** — `mlx_image` is not a ComfyUI-graph backend. For an + arbitrary ComfyUI workflow, route to `comfyui_image` instead (the selector's + `_filter_candidates` handles this when `workflow_json`/`workflow_path` + + `output_node` are present). + +## Provenance + +Each result carries `provider: "mlx"`, `model: "mlx-"` (or +`mlx-/` when a transformer is named), `pipeline`, +`action`, `seed`, and `mlx_run_py` (the exact `run.py` path invoked). Treat +seed + pipeline + transformer + prompt as the reproducibility contract; MLX +generation is byte-deterministic for a fixed seed on the same stack. + +## When to Pick mlx_image + +- A brief asks for ControlNet-conditioned generation, true i2i edit, LoRA + character/style conditioning, or face/head swap — and the box is Apple Silicon. +- Cost matters (every cloud provider charges per image; mlx_image is `$0`). +- Offline / air-gapped generation is required. +- The scorer ranks it highly: it advertises the full `control` surface + (controlnet/img2img/reference_image/faceswap/lora), so it wins the `control` + and `cost_efficiency` dimensions decisively. diff --git a/docs/REVIEW-image-to-video-voice.md b/docs/REVIEW-image-to-video-voice.md new file mode 100644 index 000000000..712d2675e --- /dev/null +++ b/docs/REVIEW-image-to-video-voice.md @@ -0,0 +1,532 @@ +# Review — Image → Video / Voice / Music / Compose path in OpenMontage + +> Collected 2026-07-06 from a grounded read of the OpenMontage codebase, with a +> side-by-side against the sibling MLX-on-Apple-Silicon repo +> (`video_generation__image_workflow` / `python/mlx-movie-director`). Scope: every +> post-image stage — motion generation, voice/TTS, music + audio mixing, the +> avatar/lip-sync layer, and the compose stage that finalizes the deliverable. +> This is the second half of the review started in +> `docs/REVIEW-story-to-image.md`. +> +> Companion to `docs/ARCHITECTURE.md`, `docs/PROVIDERS.md`, and the story→image +> review. All excerpts carry `file:line` references. + +## TL;DR + +The image→deliverable half of OpenMontage is **governance-heavy by design** — +this is where a wrong choice costs real money (cloud motion/TTS) or breaks the +creative contract (a runtime swap). Three locks hold the line: + +1. **`render_runtime` lock** — locked at proposal, carried through `edit_decisions` + unchanged; a silent swap is a **CRITICAL** violation, detected in-tool by a + three-source comparator (`video_compose._run_final_review`). +2. **"Present Both Composition Runtimes" HARD RULE** — when both Remotion and + HyperFrames are installed, the agent MUST surface both at proposal; a + single-runtime `options_considered` is a **CRITICAL** reviewer finding. +3. **Motion-required prohibition** — for any `motion_required=true` brief, a + still-image / FFmpeg-Ken-Burns / animatic substitute is forbidden without + explicit user approval of the downgrade. Stated in 5 reinforcing locations. + +The tool layer mirrors the image side's selector-plus-provider seam: +`video_selector`, `tts_selector`, `music_gen`(+`suno_music`/library/stock), and +`audio_mixer` are all registry-auto-discovered. The compose stage dispatches one +tool (`video_compose`) across three runtimes (FFmpeg / Remotion / HyperFrames), +orthogonal to a Templated-vs-Atelier authoring axis. + +The honest gaps (all verified): (1) **seedance dedup race** — fal.ai and Replicate +Seedance share `provider="seedance"`, so the second-registered is invisible to the +selector; (2) **LUFS mismatch** — `audio_mixer` hard-codes `-16 LUFS` while +`sound-design.md` targets `-14` for YouTube; (3) **ducking schema drift** — +`edit_decisions` declares `threshold_db`/`reduction_db`, but `audio_mixer._duck` +consumes `music_volume_during_speech` with no translation step; (4) **no MLX +motion path** — OM's local video providers (wan/hunyuan/ltx/cogvideo) need CUDA +`diffusers`; on Apple Silicon there is no native i2v (the `mlx_video` port fills +this); (5) the **faceswap orphan** stands, refined — the skill is HeyGen-API-backed, +not a dangling local-tool reference. + +--- + +## 1. The chain, stage by stage (post-image) + +The canonical state machine continues from where the story→image review left off +(`AGENT_GUIDE.md:185`): + +``` +… scene_plan → assets → edit → compose +``` + +| Stage | Director skill (explainer) | Canonical artifact | Post-image content | +|---|---|---|---| +| assets | `pipelines/explainer/asset-director.md` | `asset_manifest` | narration (TTS), music/sfx, generated video clips; per-asset `source_tool`/`cost_usd`/`scene_id` | +| edit | `pipelines/explainer/edit-director.md` | `edit_decisions` | cut list, audio layers (narration/music/sfx + ducking), subtitles, **`render_runtime` lock carried forward** | +| compose | `pipelines/explainer/compose-director.md` | render + `final_review` | dispatch on `render_runtime` → FFmpeg/Remotion/HyperFrames; promise-preservation check | + +The **proposal** stage is where the post-image commitments are locked: +`production_plan.render_runtime` + `composition_mode` (the lock), the mandatory +**Music Plan** (`music_source`), and `delivery_promise.motion_required`. These +three are the governance seeds for everything downstream. + +## 2. The motion tool layer + +### 2.1 `video_selector` — the routing entry point + +`tools/video/video_selector.py` — `capability="video_generation"`, +`provider="selector"`. Same auto-discovery seam as `image_selector`: +`registry.get_by_capability("video_generation")` minus itself +(`video_selector.py:135-140`). Adding a provider = drop a file in `tools/video/`. + +Routing (`execute()`, `video_selector.py:175-231`): + +1. discover candidates → `_filter_candidates` (operation-aware: `image_to_video` + needs `supports.image_to_video` or an `image_url`/`reference_image_url` schema + key; `reference_to_video` needs `reference_image_urls`; custom workflows need + `custom_workflow` capability — `video_selector.py:328-366`); +2. score with `lib/scoring.rank_providers` (reused, not reimplemented); +3. honor `preferred_provider` (a booster, see gap §6.4) and `allowed_providers`; +4. read the `VIDEO_GEN_LOCAL_MODEL` env hint (`video_selector.py:252-262`) to + pre-select a local model when `preferred=="auto"`; +5. delegate `tool.execute(adapted)` and annotate `selected_tool`/`selected_provider`/ + `provider_score`/`alternatives_considered` **and** `selected_tool_agent_skills`. + +Supports `operation: "rank"` (score without generating) for preflight. Note the +`fallback_tools` property (`video_selector.py:142-145`) **unconditionally appends +`image_selector`** — a latent footgun for motion-required briefs (§6.7). + +### 2.2 Concrete providers (`tools/video/`) + +17 providers. Local providers gate on `VIDEO_GEN_LOCAL_ENABLED=true` + importable +`diffusers`/`torch` (`_shared.py:183-195`) — i.e. **CUDA-oriented**, not Apple +Silicon native. + +| tool | provider | runtime | cost | i2v/t2v | motion strength | `agent_skills` | key `supports` | +|---|---|---|---|---|---|---|---| +| `wan_video` | wan | LOCAL_GPU | $0 | both | 1.3B 832×480@16fps 81f (8 GB) / 14B 720p (24 GB) | `ltx2` | reference_image, offline, local_gpu | +| `hunyuan_video` | hunyuan | LOCAL_GPU | $0 | both | 848×480@24fps 121f (14 GB) | `ltx2` | reference_image, offline, local_gpu | +| `ltx_video_local` | ltx | LOCAL_GPU | $0 | both | 768×512@30fps 121f (12 GB) | `ltx2` | reference_image, offline, local_gpu | +| `cogvideo_video` | cogvideo | LOCAL_GPU | $0 | both (but `cogvideo-2b` variant is i2v=False — gap §6.3) | 5B 720×480@8fps 49f (12 GB) / 2B (6 GB) | `ltx2` | reference_image, offline, local_gpu | +| `ltx_video_modal` | ltx-modal | API | flat $0.25/gen | both | self-hosted cloud LTX2 (`MODAL_LTX2_ENDPOINT_URL`) | `ltx2` | reference_image, self_hosted_cloud | +| `comfyui_video` | comfyui | HYBRID | $0 | both | server-driven, bundled WAN 2.2 14B FP8 (16 GB) | `comfyui`, `ai-video-gen`, `ltx2` | custom_workflow, custom_output_node, offline | +| `heygen_video` | heygen | API | $0.15–0.50/gen (quality tier) | both | veo3.1/veo3/kling/sora/runway/seedance variants | `ai-video-gen`, `create-video` | reference_image, cloud_generation | +| `seedance_video` | **seedance** | API (fal.ai) | $0.3034/s std, $0.2419/s fast | t2v+i2v+ref | quality_score 0.95; lip_sync, multi_shot, native_audio | `seedance-2-0`, `ai-video-gen` | text/image/reference_to_video, multiple_reference, native_audio, lip_sync, multi_shot | +| `seedance_replicate` | **seedance** | API (Replicate) | via Replicate | t2v+i2v | quality_score 0.95 | `seedance-2-0`, `ai-video-gen` | text_to_video, image_to_video, native_audio, lip_sync, multi_shot | +| `kling_video` | kling | API | $0.30 master / $0.20 pro / $0.10 std per 5s | t2v+i2v | v2.1/v3; native_audio | `ai-video-gen` | text/image_to_video, native_audio, cinematic_quality | +| `minimax_video` | minimax | API | $0.08–0.15/gen | t2v+i2v | hailuo-02/2.3 variants | `ai-video-gen` | text/image_to_video, camera_direction | +| `veo_video` | veo | API | $0.10/s fast, $0.20/s base, $0.40/s 4k, +$0.20–0.40/s audio | t2v+i2v+ref+first_last | veo3/3.1; dialogue_generation, ambient_sound | `ai-video-gen` | text/image/reference/first_last_frame, native_audio, dialogue_generation | +| `runway_video` | runway | API | $0.05–0.30/s by model | t2v+i2v | gen3a/gen4/seedance_2.0; quality_score 0.9 | `seedance-2-0`, `ai-video-gen` | text/image_to_video, professional_control, lip_sync, multi_shot | +| `higgsfield_video` | higgsfield | API | $0.10–0.80/clip | t2v+i2v | multi_model_routing; quality_score 0.9 | `seedance-2-0`, `ai-video-gen` | text/image_to_video, character_consistency, multi_model_routing | +| `grok_video` | grok | API | $0.05/s 480p, $0.07/s 720p, +$0.002/in img | t2v+i2v+ref | 1–15s; lip_sync | `grok-media`, `ai-video-gen` | text/image/reference_to_video, native_audio, lip_sync — **no quality_score (gap §6.5)** | +| `pexels_video` / `pixabay_video` | pexels / pixabay | API | free (SOURCE tier) | stock only | stock footage; not i2v/t2v | — | orientation/size/category filters, free_commercial_use | + +### 2.3 Motion-required governance (the crown jewels) + +For any `delivery_promise.motion_required=true` brief, five locations state the +prohibition consistently: + +- **`AGENT_GUIDE.md:391-406`** — "Critical Rule: Motion-Required Requests": + still-image fallback is forbidden; FFmpeg-only fallback is forbidden when it + changes the deliverable from motion-led to still-led; **silent runtime swap is + forbidden** ("If `render_runtime='hyperframes'` was locked and HyperFrames is + unavailable, do NOT route to Remotion instead. Surface the blocker…"). +- **`skills/core/hyperframes.md:74-82`** — the locked runtime is "a commitment, + not a hint"; Compose MUST NOT downgrade to FFmpeg Ken Burns. +- **`skills/pipelines/cinematic/compose-director.md:15,35`** — silent swap + (including FFmpeg Ken Burns) is a **CRITICAL governance violation**. +- **`skills/pipelines/cinematic/asset-director.md:46-82`** — for motion-required + jobs, use `video_selector` first; `image_selector` does not satisfy the motion + requirement by itself. +- **`skills/pipelines/cinematic/proposal-director.md:30,101`** — preflight + enforcement: if neither generation nor stock can deliver motion, "do not + silently downgrade to still-led… present the constraint honestly." + +## 3. The voice path + +### 3.1 The `script.voice_performance` contract + +`schemas/artifacts/script.schema.json:12-33` — a top-level (optional) object: +`performance_intent`, `pacing_profile` (enum: contemplative/conversational/ +energetic/technical/cinematic/custom), `energy_curve`, `pause_policy`, +`sample_section_id` ("most performance-sensitive section to use for TTS sample +approval"), `provider_notes`. Section-level `delivery_cues` (`:46-68`) carry +`pace`/`energy`/`emphasis_words`/`pause_before_seconds`/`pause_after_seconds`/ +`delivery_note` and **`provider_text`** — the provider-ready string that may +contain SSML `` tags. This is the SSML contract. + +### 3.2 `tts_selector` + 6 providers + +`tools/audio/tts_selector.py` — `capability="tts"`, `provider="selector"`. +Auto-discovers via `registry.get_by_capability("tts")` (`tts_selector.py:124-129`), +reuses `lib/scoring.rank_providers`, supports `operation: "rank"`. Annotates +`selected_tool`/`selected_provider`/`provider_score`/`selected_tool_agent_skills`/ +`alternatives_considered`. + +| provider | runtime | cost | cloning | SSML | `agent_skills` | fallback | +|---|---|---|---|---|---|---| +| `elevenlabs_tts` | API | $0.0003/char (**$300/1M**) | **yes** | voice settings, not tags | `elevenlabs`, `text-to-speech` | openai → piper | +| `openai_tts` | API | $0.000015/char (**$15/1M**) | no | no; `instructions` on `gpt-4o-mini-tts` only | `openai-docs` | piper | +| `google_tts` | API | $4–$160/1M (tiered: Standard→Chirp3-HD) | no | **yes** — `input_type:"ssml"` wraps ``, `` supported | `text-to-speech` | openai → elevenlabs → piper | +| `dashscope_tts` | API | $15/1M | no | no; `instructions` on `qwen3-tts-instruct-flash` | `dashscope` | doubao → elevenlabs → openai → piper | +| `doubao_tts` | API (ASYNC poll) | $15/1M | no | no; **`enable_timestamp`** → word/sentence alignment (subtitle specialist) | `doubao-tts`, `text-to-speech` | google → elevenlabs → openai → piper | +| `piper_tts` | LOCAL | **$0** | no | no; `length_scale` + `sentence_silence` | `text-to-speech` | terminal (universal floor) | + +No `edge-tts`. Piper is the universal offline floor — every API chain terminates +there. ElevenLabs is the only cloning provider. Google is the only first-class +SSML provider. Doubao is the only one returning timestamp metadata. + +### 3.3 `voice-performance-director` + sample-then-batch + +`skills/meta/voice-performance-director.md` sits above the asset-director. Its +thesis (`:3-7`): "make generated narration sound **directed**, not merely read." +It mandates the `voice_performance` contract, translates intent into per-provider +knobs (`:56-71`: OpenAI `instructions` only on `gpt-4o-mini-tts`; Google `ssml`+ +`speaking_rate 0.25..2.0`; ElevenLabs `stability`/`similarity_boost`/`style`), +and defines four failure conditions (`:83-94`): no plan, vague directions, +provider/voice change after sample approval, and generating from raw text when +`provider_text` exists. + +The asset-director runs **sample-first, batch-second** (`asset-director.md:69-82`): +generate the `sample_section_id` (most demanding section, ~$0.03–0.08), play it, +confirm voice/pace/pauses/tone (max 3 iterations), then batch. Each narration +asset records `voice_performance` provenance (`asset_manifest.schema.json:37-53`): +`source_section_id`, `delivery_cues_applied`, `provider_text_used`, +`provider_settings`, `sample_approved`, `sample_path`, `review_notes`. + +## 4. Music + audio mixing + +### 4.1 Generation + sources + +- `tools/audio/music_gen.py` — **ElevenLabs Music only** (`music_v1`), + `~$0.05/30s`, 3–600s, `agent_skills: ["music","sound-effects","elevenlabs"]`. + No `supports` flags. The `force_instrumental=true` mandate lives only in the + usage skill (`skills/creative/music-gen-usage.md:14`); the tool never sends it + (gap §6.8). +- `tools/audio/suno_music.py` — Suno V4/V4_5/V5, ASYNC, BETA, vocals + custom + lyrics; declares `supports = {instrumental, vocals, custom_lyrics, + style_control, long_form}`; `fallback_tools = ["music_gen"]`. +- `tools/audio/music_library.py` — local filesystem scan (read-only), default + `/music_library/`. **No mood/BPM/genre/license metadata** + (`music_library.py:102-122`) — selection is by filename + duration only (gap + §6.9). +- `freesound_music` / `pixabay_music` — stock search/download. + +### 4.2 Mandatory Music Plan + +At proposal, `skills/pipelines/explainer/proposal-director.md:362-397` ("Step 5b: +Music Plan (Mandatory)") requires surfacing music availability in order: +user library → AI generation → stock → none (told to user explicitly). The +decision lands in `proposal_packet.production_plan.music_source` +(`proposal_packet.schema.json:167-178`: `source_type` user_library/ai_generated/ +bring_your_own/none, `track_path`, `provider`, `mood_direction`, `estimated_cost_usd`). +**Policy-mandatory, not schema-enforced** (`music_source` is not in `production_plan.required`). + +### 4.3 `audio_mixer` + ducking + LUFS + +`tools/audio/audio_mixer.py` — pure **FFmpeg** `filter_complex` engine +(`provider="ffmpeg"`, `agent_skills: ["ffmpeg","video-toolkit"]`). Operations: +`mix`, `duck`, `extract`, `full_mix`, `segmented_music`. + +**Ducking = real sidechain compression, not static gain** (`audio_mixer.py:356-362`): + +```python +[1:a]sidechaincompress=threshold=0.02:ratio=9:attack={attack}:release={release}:level_sc=1:mix=0.9[ducked]; +[ducked]volume={music_vol * 3}[music_out]; +[0:a][music_out]amix=inputs=2:duration=longest[out] +``` + +Speech is the sidechain key; music is compressed by it. `full_mix` layers +narration+music+sfx, runs the music subgroup through the same sidechain keyed by +the speech subgroup (`:526-531`), then `loudnorm`. + +**LUFS target = `-16`** (`audio_mixer.py:254,578`): +`loudnorm=I=-16:LRA=11:TP=-1.5`. This is the **podcast/Apple** target, NOT the +YouTube `-14` that `skills/creative/sound-design.md:17,138` targets — a +documented inconsistency (gap §6.1). The mixer does not parameterize the target. + +`segmented_music` (`:606-706`) builds a per-frame `volume` expression for +"music during talking-head, silence during showcase", looping the bed with +`-stream_loop -1`. + +### 4.4 edit-director ducking wiring (and the schema drift) + +`skills/pipelines/explainer/edit-director.md:84-113` configures audio layers: +narration segments + a music bed (volume `0.08`, fade in/out) + a `ducking` +object (`threshold_db: -3`, `reduction_db: -8`, attack 200ms, release 500ms) + +sfx. **But** the `edit_decisions` ducking shape (`threshold_db`/`reduction_db`) +is **not** the shape `audio_mixer._duck` consumes (`music_volume_during_speech`/ +`attack_ms`/`release_ms` + fixed `threshold=0.02`/`ratio=9`). No translation step +exists in the mixer (gap §6.2). Note also: when rendering via **Remotion** (the +default), `audio_mixer` is not used at all — Remotion's `