calesthio · amartya-dev · Jul 10, 2026
@@ -0,0 +1,115 @@
+---
+name: azure-speech-to-text
+description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.
+license: MIT
+compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
+metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
+---
+
+# Azure AI Speech — Speech-to-Text
+
+Transcribe audio to text with **Azure Fast Transcription** — synchronous,
+word-level timestamps, speaker diarization, and multi-language identification.
+In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,
+`provider=azure`). It is an **optional cloud STT provider** — when
+`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local
+`transcriber` tool (faster-whisper) remains the **default offline path** and the
+fallback when Azure is unavailable.
+
+> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)
+
+## Why Fast Transcription (not Batch)
+
+Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because
+the pipeline transcribes **local audio files**:
+
+| Surface | Input | Latency | Needs |
+|---------|-------|---------|-------|
+| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |
+| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |
+| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |
+
+Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just
+`requests` and the two env vars.
+
+## Setup
+
+Create a **Speech** resource in the [Azure portal](https://portal.azure.com);
+copy the key and region from its **Keys and Endpoint** page.
+
+```bash
+export AZURE_SPEECH_KEY=your_speech_resource_key
+export AZURE_SPEECH_REGION=eastus          # your resource's region
+# export AZURE_SPEECH_ENDPOINT=https://...  # optional: overrides region
+```
+
+`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
+`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.
+
+## Using it in a pipeline
+
+Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output
+matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`
+and any stage that consumes a transcript.
+
+```python
+from tools.tool_registry import registry
+registry.discover()
+stt = registry._tools["azure_stt"]
+
+result = stt.execute({
+    "input_path": "projects/my-video/assets/audio/narration.mp3",
+    # "language": "en",          # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID
+    # "diarize": True,           # speaker labels, no HuggingFace token needed
+    # "max_speakers": 4,
+    "output_dir": "projects/my-video/artifacts",
+})
+if result.success:
+    segs = result.data["segments"]          # [{id,start,end,text,words:[...]}]
+    words = result.data["word_timestamps"]  # flat [{word,start,end,probability}]
+```
+
+If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`
+(local whisper) — its `execute` signature and output are identical.
+
+## Parameters that matter
+
+- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it
+  when you know the language; it is faster and more accurate than auto-ID.
+- **`candidate_locales`** — when `language` is omitted, Azure runs language
+  identification across this shortlist. Narrow it to the languages you actually
+  expect; a huge list slows detection and invites misclassification.
+- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,
+  podcasts). Set `max_speakers` to the real upper bound.
+- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.
+
+## Response shape (mapped to the transcriber schema)
+
+The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is
+converted to seconds and the OpenMontage transcript schema:
+
+```json
+{
+  "segments": [
+    {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",
+     "speaker": 1,
+     "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}
+  ],
+  "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],
+  "language": "en-US",
+  "duration_seconds": 2.4,
+  "provider": "azure"
+}
+```
+
+Note: Fast Transcription has no *per-word* confidence, so each word carries the
+**phrase** confidence in `probability`.
+
+## Limits & tips
+
+- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk
+  jobs, use Azure Batch Transcription instead.
+- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if
+  you only need speech — smaller upload, same result.
+- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check
+  the first and last cues against the source audio.
@@ -0,0 +1,115 @@
+---
+name: azure-speech-to-text
+description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.
+license: MIT
+compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
+metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
+---
+
+# Azure AI Speech — Speech-to-Text
+
+Transcribe audio to text with **Azure Fast Transcription** — synchronous,
+word-level timestamps, speaker diarization, and multi-language identification.
+In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,
+`provider=azure`). It is an **optional cloud STT provider** — when
+`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local
+`transcriber` tool (faster-whisper) remains the **default offline path** and the
+fallback when Azure is unavailable.
+
+> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)
+
+## Why Fast Transcription (not Batch)
+
+Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because
+the pipeline transcribes **local audio files**:
+
+| Surface | Input | Latency | Needs |
+|---------|-------|---------|-------|
+| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |
+| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |
+| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |
+
+Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just
+`requests` and the two env vars.
+
+## Setup
+
+Create a **Speech** resource in the [Azure portal](https://portal.azure.com);
+copy the key and region from its **Keys and Endpoint** page.
+
+```bash
+export AZURE_SPEECH_KEY=your_speech_resource_key
+export AZURE_SPEECH_REGION=eastus          # your resource's region
+# export AZURE_SPEECH_ENDPOINT=https://...  # optional: overrides region
+```
+
+`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
+`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.
+
+## Using it in a pipeline
+
+Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output
+matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`
+and any stage that consumes a transcript.
+
+```python
+from tools.tool_registry import registry
+registry.discover()
+stt = registry._tools["azure_stt"]
+
+result = stt.execute({
+    "input_path": "projects/my-video/assets/audio/narration.mp3",
+    # "language": "en",          # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID
+    # "diarize": True,           # speaker labels, no HuggingFace token needed
+    # "max_speakers": 4,
+    "output_dir": "projects/my-video/artifacts",
+})
+if result.success:
+    segs = result.data["segments"]          # [{id,start,end,text,words:[...]}]
+    words = result.data["word_timestamps"]  # flat [{word,start,end,probability}]
+```
+
+If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`
+(local whisper) — its `execute` signature and output are identical.
+
+## Parameters that matter
+
+- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it
+  when you know the language; it is faster and more accurate than auto-ID.
+- **`candidate_locales`** — when `language` is omitted, Azure runs language
+  identification across this shortlist. Narrow it to the languages you actually
+  expect; a huge list slows detection and invites misclassification.
+- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,
+  podcasts). Set `max_speakers` to the real upper bound.
+- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.
+
+## Response shape (mapped to the transcriber schema)
+
+The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is
+converted to seconds and the OpenMontage transcript schema:
+
+```json
+{
+  "segments": [
+    {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",
+     "speaker": 1,
+     "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}
+  ],
+  "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],
+  "language": "en-US",
+  "duration_seconds": 2.4,
+  "provider": "azure"
+}
+```
+
+Note: Fast Transcription has no *per-word* confidence, so each word carries the
+**phrase** confidence in `probability`.
+
+## Limits & tips
+
+- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk
+  jobs, use Azure Batch Transcription instead.
+- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if
+  you only need speech — smaller upload, same result.
+- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check
+  the first and last cues against the source audio.
@@ -56,6 +56,12 @@ UNSPLASH_ACCESS_KEY=         # Unsplash stock images (free developer key)
 
 # --- Analysis ---
 HF_TOKEN=                    # HuggingFace token — enables speaker diarization in transcriber
+# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the
+# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber
+# remains the default offline path.
+AZURE_SPEECH_KEY=            # Azure AI Speech resource key ('Keys and Endpoint' page)
+AZURE_SPEECH_REGION=         # Speech resource region, e.g. eastus
+# AZURE_SPEECH_ENDPOINT=     # Optional: full custom endpoint URL (overrides region)
 
 # --- Avatar (local installs) ---
 # WAV2LIP_PATH=              # Path to cloned Wav2Lip repo (for lip sync)

@@ -678,7 +678,8 @@ The `.agents/skills/` directory is large. When you're not coming in through a to
 | **Image generation** | `bfl-api`, `flux-best-practices` |
 | **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` |
 | **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |
-| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `speech-to-text`, `agents` |
+| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) |
+| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` |
 | **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) |
 | **Visualization** | `beautiful-mermaid`, `d3-viz`, `manim-composer`, `manimce-best-practices`, `manimgl-best-practices` |
 | **Media editing** | `video-edit`, `video-download`, `video-understand`, `video-toolkit`, `visual-style` |

@@ -43,6 +43,10 @@ DOUBAO_SPEECH_API_KEY=       # Volcengine Doubao Speech TTS (strong Mandarin nar
 DOUBAO_SPEECH_VOICE_TYPE=    # Default Doubao speaker/voice type
 DASHSCOPE_API_KEY=           # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)
 
+# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)
+AZURE_SPEECH_KEY=            # Azure AI Speech — Fast Transcription (word-level timestamps)
+AZURE_SPEECH_REGION=         # Speech resource region, e.g. eastus
+
 # MULTI-MODEL GATEWAY (one key, 6+ tools)
 FAL_KEY=                     # FLUX, Recraft, Kling, Veo, MiniMax video
 
@@ -245,6 +249,54 @@ Doubao Speech 2.0 is billed by character package or usage in Volcengine. OpenMon
 
 ---
 
+### Azure AI Speech — Speech-to-Text
+
+> **Cloud transcription.** Azure AI Speech Fast Transcription turns local audio into text with word-level timestamps, speaker diarization, and multi-language identification — no GPU required. Optional: the local faster-whisper `transcriber` remains the default offline STT path. When `AZURE_SPEECH_KEY` is set, the agent prefers `azure_stt` for cloud transcription.
+
+**Tools unlocked:** `azure_stt`
+**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_SPEECH_ENDPOINT`)
+
+#### Setup
+
+1. In the [Azure portal](https://portal.azure.com), create a **Speech** resource (Azure AI services → Speech service).
+2. Open the resource's **Keys and Endpoint** page.
+3. Copy **KEY 1** and the **Location/Region** (e.g. `eastus`).
+4. Add to `.env`:
+   ```bash
+   AZURE_SPEECH_KEY=your-speech-resource-key
+   AZURE_SPEECH_REGION=eastus
+   # AZURE_SPEECH_ENDPOINT=https://<custom>...  # optional, overrides region
+   ```
+
+#### API Notes
+
+OpenMontage uses the **Fast Transcription** REST endpoint, which accepts a local
+audio file directly (multipart upload) and returns a synchronous result — no
+Azure Blob storage, SAS URLs, or async job polling:
+
+```text
+POST https://{region}.api.cognitive.microsoft.com/speechtotext/transcriptions:transcribe?api-version=2024-11-15
+Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}
+```
+
+For files longer than ~2 hours or bulk jobs, use Azure Batch Transcription instead (not wired into OpenMontage).
+
+#### What It Is Best For
+
+- Cloud transcription with word-level timestamps and no local GPU
+- Multi-language auto-detection across a candidate locale set
+- Speaker diarization without a HuggingFace token
+- Subtitle timing metadata that flows straight into `subtitle_gen`
+
+#### Pricing
+
+Azure AI Speech Standard (S0) bills speech-to-text by audio-hour (roughly
+$1.00/audio-hour at time of writing; a free F0 tier includes a limited monthly
+allowance). OpenMontage estimates cost from the transcribed audio duration. See
+[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
+
+---
+
 ### Google — TTS + Imagen + Music + Video (Shared Key)
 
 > **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. Imagen 4 generates high-quality images. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.

@@ -81,7 +81,8 @@ Key capability families to look for in the output:
 | FFmpeg | `core/ffmpeg.md` | Video encoding, filtering, composition | `ffmpeg`, `video-toolkit` |
 | Remotion | `core/remotion.md` | React-based composition, Phase 3+ | `remotion-best-practices`, `remotion` |
 | HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` |
-| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps | `speech-to-text` |
+| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` |
+| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` |
 | Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` |
 | Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` |
 
@@ -306,7 +307,7 @@ Claude Code accesses them via symlinks in `.claude/skills/`.
 |----------|-----------------|--------|
 | **Video Composition** | `remotion-best-practices`, `remotion`, `hyperframes` (router), `hyperframes-core`, `hyperframes-creative`, `hyperframes-media`, `hyperframes-animation`, `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video`, `remotion-to-hyperframes`, `website-to-video` | `remotion-dev/skills`, `digitalsamba/claude-code-video-toolkit`, `heygen-com/hyperframes` (vendored v0.7.17, see `.agents/skills/hyperframes/PROVENANCE.md`) |
 | **Video Processing** | `ffmpeg`, `video-toolkit` | `digitalsamba/claude-code-video-toolkit` |
-| **TTS & Audio** | `text-to-speech`, `speech-to-text`, `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` |
+| **TTS & Audio** | `text-to-speech`, `speech-to-text` (whisper, default STT), `azure-speech-to-text` (optional cloud STT), `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` |
 | **Image Generation** | `flux-best-practices`, `bfl-api`, `grok-media` | `black-forest-labs/skills`, local OpenMontage skill |
 | **Math Animation** | `manimce-best-practices`, `manimgl-best-practices`, `manim-composer` | `adithya-s-k/manim_skill` |
 | **3D Graphics** | `threejs-animation`, `threejs-fundamentals`, `threejs-geometry`, `threejs-interaction`, `threejs-lighting`, `threejs-loaders`, `threejs-materials`, `threejs-postprocessing`, `threejs-shaders`, `threejs-textures` | `cloudai-x/threejs-skills` |