diff --git a/.agents/skills/azure-speech-to-text/SKILL.md b/.agents/skills/azure-speech-to-text/SKILL.md new file mode 100644 index 000000000..60b1b8a40 --- /dev/null +++ b/.agents/skills/azure-speech-to-text/SKILL.md @@ -0,0 +1,115 @@ +--- +name: azure-speech-to-text +description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path. +license: MIT +compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION). +metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}} +--- + +# Azure AI Speech — Speech-to-Text + +Transcribe audio to text with **Azure Fast Transcription** — synchronous, +word-level timestamps, speaker diarization, and multi-language identification. +In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`, +`provider=azure`). It is an **optional cloud STT provider** — when +`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local +`transcriber` tool (faster-whisper) remains the **default offline path** and the +fallback when Azure is unavailable. + +> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview) + +## Why Fast Transcription (not Batch) + +Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because +the pipeline transcribes **local audio files**: + +| Surface | Input | Latency | Needs | +|---------|-------|---------|-------| +| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region | +| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing | +| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package | + +Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just +`requests` and the two env vars. + +## Setup + +Create a **Speech** resource in the [Azure portal](https://portal.azure.com); +copy the key and region from its **Keys and Endpoint** page. + +```bash +export AZURE_SPEECH_KEY=your_speech_resource_key +export AZURE_SPEECH_REGION=eastus # your resource's region +# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region +``` + +`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either +`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set. + +## Using it in a pipeline + +Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output +matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen` +and any stage that consumes a transcript. + +```python +from tools.tool_registry import registry +registry.discover() +stt = registry._tools["azure_stt"] + +result = stt.execute({ + "input_path": "projects/my-video/assets/audio/narration.mp3", + # "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID + # "diarize": True, # speaker labels, no HuggingFace token needed + # "max_speakers": 4, + "output_dir": "projects/my-video/artifacts", +}) +if result.success: + segs = result.data["segments"] # [{id,start,end,text,words:[...]}] + words = result.data["word_timestamps"] # flat [{word,start,end,probability}] +``` + +If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber` +(local whisper) — its `execute` signature and output are identical. + +## Parameters that matter + +- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it + when you know the language; it is faster and more accurate than auto-ID. +- **`candidate_locales`** — when `language` is omitted, Azure runs language + identification across this shortlist. Narrow it to the languages you actually + expect; a huge list slows detection and invites misclassification. +- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews, + podcasts). Set `max_speakers` to the real upper bound. +- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`. + +## Response shape (mapped to the transcriber schema) + +The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is +converted to seconds and the OpenMontage transcript schema: + +```json +{ + "segments": [ + {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world", + "speaker": 1, + "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]} + ], + "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}], + "language": "en-US", + "duration_seconds": 2.4, + "provider": "azure" +} +``` + +Note: Fast Transcription has no *per-word* confidence, so each word carries the +**phrase** confidence in `probability`. + +## Limits & tips + +- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk + jobs, use Azure Batch Transcription instead. +- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if + you only need speech — smaller upload, same result. +- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check + the first and last cues against the source audio. diff --git a/.claude/skills/azure-speech-to-text/SKILL.md b/.claude/skills/azure-speech-to-text/SKILL.md new file mode 100644 index 000000000..60b1b8a40 --- /dev/null +++ b/.claude/skills/azure-speech-to-text/SKILL.md @@ -0,0 +1,115 @@ +--- +name: azure-speech-to-text +description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path. +license: MIT +compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION). +metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}} +--- + +# Azure AI Speech — Speech-to-Text + +Transcribe audio to text with **Azure Fast Transcription** — synchronous, +word-level timestamps, speaker diarization, and multi-language identification. +In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`, +`provider=azure`). It is an **optional cloud STT provider** — when +`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local +`transcriber` tool (faster-whisper) remains the **default offline path** and the +fallback when Azure is unavailable. + +> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview) + +## Why Fast Transcription (not Batch) + +Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because +the pipeline transcribes **local audio files**: + +| Surface | Input | Latency | Needs | +|---------|-------|---------|-------| +| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region | +| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing | +| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package | + +Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just +`requests` and the two env vars. + +## Setup + +Create a **Speech** resource in the [Azure portal](https://portal.azure.com); +copy the key and region from its **Keys and Endpoint** page. + +```bash +export AZURE_SPEECH_KEY=your_speech_resource_key +export AZURE_SPEECH_REGION=eastus # your resource's region +# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region +``` + +`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either +`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set. + +## Using it in a pipeline + +Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output +matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen` +and any stage that consumes a transcript. + +```python +from tools.tool_registry import registry +registry.discover() +stt = registry._tools["azure_stt"] + +result = stt.execute({ + "input_path": "projects/my-video/assets/audio/narration.mp3", + # "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID + # "diarize": True, # speaker labels, no HuggingFace token needed + # "max_speakers": 4, + "output_dir": "projects/my-video/artifacts", +}) +if result.success: + segs = result.data["segments"] # [{id,start,end,text,words:[...]}] + words = result.data["word_timestamps"] # flat [{word,start,end,probability}] +``` + +If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber` +(local whisper) — its `execute` signature and output are identical. + +## Parameters that matter + +- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it + when you know the language; it is faster and more accurate than auto-ID. +- **`candidate_locales`** — when `language` is omitted, Azure runs language + identification across this shortlist. Narrow it to the languages you actually + expect; a huge list slows detection and invites misclassification. +- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews, + podcasts). Set `max_speakers` to the real upper bound. +- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`. + +## Response shape (mapped to the transcriber schema) + +The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is +converted to seconds and the OpenMontage transcript schema: + +```json +{ + "segments": [ + {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world", + "speaker": 1, + "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]} + ], + "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}], + "language": "en-US", + "duration_seconds": 2.4, + "provider": "azure" +} +``` + +Note: Fast Transcription has no *per-word* confidence, so each word carries the +**phrase** confidence in `probability`. + +## Limits & tips + +- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk + jobs, use Azure Batch Transcription instead. +- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if + you only need speech — smaller upload, same result. +- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check + the first and last cues against the source audio. diff --git a/.env.example b/.env.example index ac6f07080..14f2a351b 100644 --- a/.env.example +++ b/.env.example @@ -56,6 +56,12 @@ UNSPLASH_ACCESS_KEY= # Unsplash stock images (free developer key) # --- Analysis --- HF_TOKEN= # HuggingFace token — enables speaker diarization in transcriber +# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the +# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber +# remains the default offline path. +AZURE_SPEECH_KEY= # Azure AI Speech resource key ('Keys and Endpoint' page) +AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus +# AZURE_SPEECH_ENDPOINT= # Optional: full custom endpoint URL (overrides region) # --- Avatar (local installs) --- # WAV2LIP_PATH= # Path to cloned Wav2Lip repo (for lip sync) diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md index feb67699d..2a0b6bd5c 100644 --- a/AGENT_GUIDE.md +++ b/AGENT_GUIDE.md @@ -678,7 +678,8 @@ The `.agents/skills/` directory is large. When you're not coming in through a to | **Image generation** | `bfl-api`, `flux-best-practices` | | **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` | | **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` | -| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `speech-to-text`, `agents` | +| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) | +| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` | | **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) | | **Visualization** | `beautiful-mermaid`, `d3-viz`, `manim-composer`, `manimce-best-practices`, `manimgl-best-practices` | | **Media editing** | `video-edit`, `video-download`, `video-understand`, `video-toolkit`, `visual-style` | diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md index 8cee5bb25..be77ba3f0 100644 --- a/docs/PROVIDERS.md +++ b/docs/PROVIDERS.md @@ -43,6 +43,10 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin nar DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps) +# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default) +AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps) +AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus + # MULTI-MODEL GATEWAY (one key, 6+ tools) FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video @@ -245,6 +249,54 @@ Doubao Speech 2.0 is billed by character package or usage in Volcengine. OpenMon --- +### Azure AI Speech — Speech-to-Text + +> **Cloud transcription.** Azure AI Speech Fast Transcription turns local audio into text with word-level timestamps, speaker diarization, and multi-language identification — no GPU required. Optional: the local faster-whisper `transcriber` remains the default offline STT path. When `AZURE_SPEECH_KEY` is set, the agent prefers `azure_stt` for cloud transcription. + +**Tools unlocked:** `azure_stt` +**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_SPEECH_ENDPOINT`) + +#### Setup + +1. In the [Azure portal](https://portal.azure.com), create a **Speech** resource (Azure AI services → Speech service). +2. Open the resource's **Keys and Endpoint** page. +3. Copy **KEY 1** and the **Location/Region** (e.g. `eastus`). +4. Add to `.env`: + ```bash + AZURE_SPEECH_KEY=your-speech-resource-key + AZURE_SPEECH_REGION=eastus + # AZURE_SPEECH_ENDPOINT=https://... # optional, overrides region + ``` + +#### API Notes + +OpenMontage uses the **Fast Transcription** REST endpoint, which accepts a local +audio file directly (multipart upload) and returns a synchronous result — no +Azure Blob storage, SAS URLs, or async job polling: + +```text +POST https://{region}.api.cognitive.microsoft.com/speechtotext/transcriptions:transcribe?api-version=2024-11-15 +Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY} +``` + +For files longer than ~2 hours or bulk jobs, use Azure Batch Transcription instead (not wired into OpenMontage). + +#### What It Is Best For + +- Cloud transcription with word-level timestamps and no local GPU +- Multi-language auto-detection across a candidate locale set +- Speaker diarization without a HuggingFace token +- Subtitle timing metadata that flows straight into `subtitle_gen` + +#### Pricing + +Azure AI Speech Standard (S0) bills speech-to-text by audio-hour (roughly +$1.00/audio-hour at time of writing; a free F0 tier includes a limited monthly +allowance). OpenMontage estimates cost from the transcribed audio duration. See +[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates. + +--- + ### Google — TTS + Imagen + Music + Video (Shared Key) > **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. Imagen 4 generates high-quality images. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips. diff --git a/skills/INDEX.md b/skills/INDEX.md index c8a050903..dc81f0e68 100644 --- a/skills/INDEX.md +++ b/skills/INDEX.md @@ -81,7 +81,8 @@ Key capability families to look for in the output: | FFmpeg | `core/ffmpeg.md` | Video encoding, filtering, composition | `ffmpeg`, `video-toolkit` | | Remotion | `core/remotion.md` | React-based composition, Phase 3+ | `remotion-best-practices`, `remotion` | | HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` | -| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps | `speech-to-text` | +| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` | +| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` | | Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` | | Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` | @@ -306,7 +307,7 @@ Claude Code accesses them via symlinks in `.claude/skills/`. |----------|-----------------|--------| | **Video Composition** | `remotion-best-practices`, `remotion`, `hyperframes` (router), `hyperframes-core`, `hyperframes-creative`, `hyperframes-media`, `hyperframes-animation`, `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video`, `remotion-to-hyperframes`, `website-to-video` | `remotion-dev/skills`, `digitalsamba/claude-code-video-toolkit`, `heygen-com/hyperframes` (vendored v0.7.17, see `.agents/skills/hyperframes/PROVENANCE.md`) | | **Video Processing** | `ffmpeg`, `video-toolkit` | `digitalsamba/claude-code-video-toolkit` | -| **TTS & Audio** | `text-to-speech`, `speech-to-text`, `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` | +| **TTS & Audio** | `text-to-speech`, `speech-to-text` (whisper, default STT), `azure-speech-to-text` (optional cloud STT), `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` | | **Image Generation** | `flux-best-practices`, `bfl-api`, `grok-media` | `black-forest-labs/skills`, local OpenMontage skill | | **Math Animation** | `manimce-best-practices`, `manimgl-best-practices`, `manim-composer` | `adithya-s-k/manim_skill` | | **3D Graphics** | `threejs-animation`, `threejs-fundamentals`, `threejs-geometry`, `threejs-interaction`, `threejs-lighting`, `threejs-loaders`, `threejs-materials`, `threejs-postprocessing`, `threejs-shaders`, `threejs-textures` | `cloudai-x/threejs-skills` | diff --git a/tests/tools/test_azure_stt.py b/tests/tools/test_azure_stt.py new file mode 100644 index 000000000..49c0cb64f --- /dev/null +++ b/tests/tools/test_azure_stt.py @@ -0,0 +1,242 @@ +"""Focused tests for the Azure AI Speech (Fast Transcription) STT tool. + +No live API calls: the network layer is monkeypatched. Covers the tool +contract, registry discovery, status behavior, the response→transcript +mapping, execute() guardrails, and downstream subtitle compatibility. +""" + +import sys +from pathlib import Path + +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(PROJECT_ROOT)) + +from tools.base_tool import BaseTool, ToolStatus, ToolTier, ToolRuntime +from tools.tool_registry import ToolRegistry +from tools.analysis.azure_stt import AzureSpeechToText +from tools.subtitle.subtitle_gen import SubtitleGen + + +# A representative Fast Transcription response payload. +SAMPLE_PAYLOAD = { + "durationMilliseconds": 2400, + "combinedPhrases": [{"text": "Hello world this is a test"}], + "phrases": [ + { + "offsetMilliseconds": 0, + "durationMilliseconds": 1200, + "text": "Hello world", + "locale": "en-US", + "confidence": 0.97, + "speaker": 1, + "words": [ + {"text": "Hello", "offsetMilliseconds": 0, "durationMilliseconds": 500}, + {"text": "world", "offsetMilliseconds": 500, "durationMilliseconds": 700}, + ], + }, + { + "offsetMilliseconds": 1200, + "durationMilliseconds": 1200, + "text": "this is a test", + "locale": "en-US", + "confidence": 0.95, + "words": [ + {"text": "this", "offsetMilliseconds": 1200, "durationMilliseconds": 300}, + {"text": "is", "offsetMilliseconds": 1500, "durationMilliseconds": 200}, + {"text": "a", "offsetMilliseconds": 1700, "durationMilliseconds": 200}, + {"text": "test", "offsetMilliseconds": 1900, "durationMilliseconds": 500}, + ], + }, + ], +} + + +class _FakeResponse: + def __init__(self, payload, status_code=200, text="ok"): + self._payload = payload + self.status_code = status_code + self.text = text + + def json(self): + return self._payload + + +@pytest.fixture +def azure_env(monkeypatch): + monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key") + monkeypatch.setenv("AZURE_SPEECH_REGION", "eastus") + + +# ---- Contract ---- + +class TestContract: + def test_inherits_base_tool(self): + assert issubclass(AzureSpeechToText, BaseTool) + + def test_identity(self): + t = AzureSpeechToText() + assert t.name == "azure_stt" + assert t.capability == "analysis" + assert t.provider == "azure" + assert t.runtime == ToolRuntime.API + assert t.tier == ToolTier.CORE + assert t.fallback == "transcriber" + assert "azure-speech-to-text" in t.agent_skills + assert len(t.capabilities) > 0 + + def test_get_info_valid(self): + info = AzureSpeechToText().get_info() + assert info["name"] == "azure_stt" + assert info["capability"] == "analysis" + assert "input_path" in info["input_schema"]["properties"] + + def test_estimate_cost_by_duration(self): + t = AzureSpeechToText() + assert t.estimate_cost({"duration_seconds": 3600}) == pytest.approx(1.0) + assert t.estimate_cost({}) == 0.0 + + +# ---- Registry discovery ---- + +class TestDiscovery: + def test_discoverable(self): + reg = ToolRegistry() + reg.discover("tools") + assert reg.get("azure_stt") is not None + + def test_capability_routing(self): + reg = ToolRegistry() + reg.discover("tools") + names = [t.name for t in reg.get_by_capability("analysis")] + assert "azure_stt" in names + + +# ---- Status behavior ---- + +class TestStatus: + def test_unavailable_without_env(self, monkeypatch): + monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False) + monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False) + monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False) + assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE + + def test_available_with_key_and_region(self, azure_env): + assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE + + def test_available_with_key_and_endpoint(self, monkeypatch): + monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key") + monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False) + monkeypatch.setenv("AZURE_SPEECH_ENDPOINT", "https://custom.example.com") + assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE + + def test_key_alone_is_not_enough(self, monkeypatch): + monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key") + monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False) + monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False) + assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE + + +# ---- Response → transcript mapping (the risky logic) ---- + +class TestParsePayload: + def test_maps_segments_words_and_timestamps(self): + data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"]) + assert data["language"] == "en-US" + assert data["duration_seconds"] == 2.4 + assert len(data["segments"]) == 2 + + seg0 = data["segments"][0] + assert seg0["text"] == "Hello world" + assert seg0["start"] == 0.0 and seg0["end"] == 1.2 + assert seg0["speaker"] == 1 + # ms → seconds conversion on words + assert seg0["words"][0] == { + "word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.97, + } + # flat word stream aggregates every phrase + assert len(data["word_timestamps"]) == 6 + + def test_locale_resolution(self): + r = AzureSpeechToText._resolve_locales + assert r({"language": "en"}) == ["en-US"] # ISO → default locale + assert r({"language": "en-GB"}) == ["en-GB"] # explicit locale kept + assert r({"candidate_locales": ["fr-FR"]}) == ["fr-FR"] + assert len(r({})) > 1 # falls back to a shortlist + + def test_duration_falls_back_to_last_segment(self): + payload = {k: v for k, v in SAMPLE_PAYLOAD.items() if k != "durationMilliseconds"} + data = AzureSpeechToText._parse_payload(payload, ["en-US"]) + assert data["duration_seconds"] == pytest.approx(2.4) + + def test_output_feeds_subtitle_gen(self, tmp_path): + """Parsed segments must be a drop-in for the subtitle generator.""" + data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"]) + out = tmp_path / "captions.srt" + res = SubtitleGen().execute( + {"segments": data["segments"], "format": "srt", "output_path": str(out)} + ) + assert res.success + assert out.exists() + assert "Hello world" in out.read_text() + + +# ---- execute() guardrails + mocked success ---- + +class TestExecute: + def test_missing_file(self, azure_env, tmp_path): + res = AzureSpeechToText().execute({"input_path": str(tmp_path / "nope.wav")}) + assert not res.success + assert "not found" in res.error.lower() + + def test_missing_credentials(self, monkeypatch, tmp_path): + monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False) + monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False) + audio = tmp_path / "a.wav" + audio.write_bytes(b"RIFF....") + res = AzureSpeechToText().execute({"input_path": str(audio)}) + assert not res.success + assert "not configured" in res.error.lower() + + def test_success_path_mocked(self, azure_env, tmp_path, monkeypatch): + import requests + + captured = {} + + def fake_post(url, headers=None, files=None, timeout=None): + captured["url"] = url + captured["headers"] = headers + return _FakeResponse(SAMPLE_PAYLOAD) + + monkeypatch.setattr(requests, "post", fake_post) + + audio = tmp_path / "a.wav" + audio.write_bytes(b"RIFF....") + res = AzureSpeechToText().execute( + {"input_path": str(audio), "language": "en", "output_dir": str(tmp_path)} + ) + + assert res.success + assert res.model == "azure-fast-transcription" + assert res.cost_usd == pytest.approx(0.0007, abs=1e-4) # 2.4s at $1/hr + assert res.data["provider"] == "azure" + assert len(res.data["segments"]) == 2 + # transcript JSON artifact was written + assert res.artifacts and Path(res.artifacts[0]).exists() + # correct endpoint + auth header used + assert "transcriptions:transcribe" in captured["url"] + assert captured["headers"]["Ocp-Apim-Subscription-Key"] == "fake-key" + + def test_http_error_surfaced(self, azure_env, tmp_path, monkeypatch): + import requests + + monkeypatch.setattr( + requests, "post", + lambda *a, **k: _FakeResponse(None, status_code=401, text="Unauthorized"), + ) + audio = tmp_path / "a.wav" + audio.write_bytes(b"RIFF....") + res = AzureSpeechToText().execute({"input_path": str(audio), "output_dir": str(tmp_path)}) + assert not res.success + assert "401" in res.error diff --git a/tools/analysis/azure_stt.py b/tools/analysis/azure_stt.py new file mode 100644 index 000000000..38fd9a939 --- /dev/null +++ b/tools/analysis/azure_stt.py @@ -0,0 +1,365 @@ +"""Azure AI Speech transcription tool (Fast Transcription REST API). + +Speech-to-text with word-level timestamps and optional speaker diarization, +served by Azure AI Speech. This is an optional cloud STT provider; when +`AZURE_SPEECH_KEY` is configured the agent may prefer it for cloud +transcription, while the local faster-whisper `transcriber` tool remains the +default offline path. + +Uses the **Fast Transcription** REST endpoint, which accepts a local audio +file directly (multipart upload) and returns a synchronous JSON result — no +Azure Blob storage, SAS URLs, or async job polling required. Output is shaped +to match `transcriber` exactly (segments + word_timestamps) so it is a +drop-in for `subtitle_gen` and every pipeline that consumes a transcript. + +Docs: https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create +""" + +from __future__ import annotations + +import json +import os +import time +from pathlib import Path +from typing import Any + +from tools.base_tool import ( + BaseTool, + Determinism, + ExecutionMode, + ResourceProfile, + ResumeSupport, + RetryPolicy, + ToolResult, + ToolRuntime, + ToolStability, + ToolStatus, + ToolTier, +) + +# Fast Transcription API version (GA). +_API_VERSION = "2024-11-15" + +# Candidate locales used for automatic language identification when the caller +# does not pin a language. Azure Fast Transcription accepts several locales and +# picks the best match per phrase. +_DEFAULT_CANDIDATE_LOCALES = [ + "en-US", + "es-ES", + "fr-FR", + "de-DE", + "it-IT", + "pt-BR", + "hi-IN", + "ja-JP", + "zh-CN", +] + +# Minimal ISO 639-1 -> default BCP-47 locale map so callers can keep passing the +# short codes the whisper transcriber accepts (e.g. "en", "es"). +_ISO_TO_LOCALE = { + "en": "en-US", + "es": "es-ES", + "fr": "fr-FR", + "de": "de-DE", + "it": "it-IT", + "pt": "pt-BR", + "hi": "hi-IN", + "ja": "ja-JP", + "ko": "ko-KR", + "zh": "zh-CN", + "ar": "ar-EG", + "ru": "ru-RU", + "nl": "nl-NL", + "tr": "tr-TR", + "pl": "pl-PL", + "id": "id-ID", +} + + +class AzureSpeechToText(BaseTool): + name = "azure_stt" + version = "0.1.0" + tier = ToolTier.CORE + capability = "analysis" + provider = "azure" + stability = ToolStability.BETA + execution_mode = ExecutionMode.SYNC + determinism = Determinism.DETERMINISTIC + runtime = ToolRuntime.API + + # Availability is decided by get_status() (env var check), mirroring the + # ElevenLabs provider tool — dependencies stays empty. + dependencies = [] + install_instructions = ( + "Set your Azure AI Speech credentials:\n" + " export AZURE_SPEECH_KEY=your_speech_resource_key\n" + " export AZURE_SPEECH_REGION=eastus # your Speech resource region\n" + "Create a Speech resource in the Azure portal " + "(https://portal.azure.com) — the key and region are on its " + "'Keys and Endpoint' page. Optionally set AZURE_SPEECH_ENDPOINT to a " + "full custom endpoint URL instead of the region." + ) + agent_skills = ["azure-speech-to-text"] + + capabilities = [ + "transcribe", + "word_timestamps", + "diarization", + "language_detection", + ] + + best_for = [ + "Cloud transcription with word-level timestamps and no GPU", + "Multi-language auto-detection across a candidate locale set", + "Speaker diarization without a HuggingFace token", + ] + not_good_for = [ + "Fully offline runs (use the whisper `transcriber` fallback)", + "Single files longer than ~2 hours (use Azure Batch transcription)", + ] + + input_schema = { + "type": "object", + "required": ["input_path"], + "properties": { + "input_path": {"type": "string", "description": "Path to audio or video file"}, + "language": { + "type": "string", + "description": ( + "ISO 639-1 code (e.g. 'en') or BCP-47 locale (e.g. 'en-US'). " + "Omit for automatic language identification across common locales." + ), + }, + "candidate_locales": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "BCP-47 locales to consider for language ID when `language` is " + "not set. Defaults to a common-language shortlist." + ), + }, + "diarize": {"type": "boolean", "default": False}, + "max_speakers": {"type": "integer", "default": 4}, + "profanity_filter": { + "type": "string", + "enum": ["None", "Masked", "Removed", "Tags"], + "default": "Masked", + }, + "output_dir": {"type": "string", "description": "Directory for output files"}, + }, + } + + output_schema = { + "type": "object", + "properties": { + "segments": {"type": "array"}, + "word_timestamps": {"type": "array"}, + "language": {"type": "string"}, + "duration_seconds": {"type": "number"}, + }, + } + + resource_profile = ResourceProfile( + cpu_cores=1, + ram_mb=256, + vram_mb=0, + disk_mb=50, + network_required=True, + ) + + retry_policy = RetryPolicy( + max_retries=2, + retryable_errors=["ConnectionError", "Timeout", "429", "503"], + ) + resume_support = ResumeSupport.FROM_START + idempotency_key_fields = ["input_path", "language", "diarize"] + side_effects = ["writes transcript JSON to output_dir", "sends audio to Azure AI Speech"] + fallback = "transcriber" + fallback_tools = ["transcriber"] + user_visible_verification = [ + "Check transcript text against source audio", + "Verify word timestamps align with speech", + ] + + def get_status(self) -> ToolStatus: + if os.environ.get("AZURE_SPEECH_KEY") and ( + os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT") + ): + return ToolStatus.AVAILABLE + return ToolStatus.UNAVAILABLE + + # Azure AI Speech Fast Transcription bills per audio-hour (Standard tier is + # roughly $1.00/audio-hour at time of writing). + COST_PER_AUDIO_HOUR = 1.0 + + def estimate_cost(self, inputs: dict[str, Any]) -> float: + # Duration is unknown until the audio is transcribed; return the actual + # cost by passing duration_seconds, else 0.0 as a pre-call estimate. + seconds = inputs.get("duration_seconds", 0) or 0 + return round((seconds / 3600.0) * self.COST_PER_AUDIO_HOUR, 4) + + def estimate_runtime(self, inputs: dict[str, Any]) -> float: + # Fast Transcription is well under real-time; conservative default. + return 20.0 + + def _endpoint(self) -> str: + endpoint = os.environ.get("AZURE_SPEECH_ENDPOINT") + if endpoint: + return endpoint.rstrip("/") + region = os.environ.get("AZURE_SPEECH_REGION", "").strip() + return f"https://{region}.api.cognitive.microsoft.com" + + @staticmethod + def _resolve_locales(inputs: dict[str, Any]) -> list[str]: + language = inputs.get("language") + if language: + if "-" in language: # already a BCP-47 locale + return [language] + return [_ISO_TO_LOCALE.get(language.lower(), f"{language.lower()}-US")] + candidates = inputs.get("candidate_locales") + return list(candidates) if candidates else list(_DEFAULT_CANDIDATE_LOCALES) + + def execute(self, inputs: dict[str, Any]) -> ToolResult: + input_path = Path(inputs["input_path"]) + if not input_path.exists(): + return ToolResult(success=False, error=f"Input file not found: {input_path}") + + api_key = os.environ.get("AZURE_SPEECH_KEY") + if not api_key or not ( + os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT") + ): + return ToolResult( + success=False, + error="Azure Speech is not configured. " + self.install_instructions, + ) + + start = time.time() + try: + result = self._transcribe(inputs, api_key, input_path) + except Exception as exc: + return ToolResult(success=False, error=f"Transcription failed: {exc}") + + result.duration_seconds = round(time.time() - start, 2) + result.model = "azure-fast-transcription" + result.cost_usd = self.estimate_cost( + {"duration_seconds": result.data.get("duration_seconds", 0)} + ) + return result + + def _transcribe( + self, inputs: dict[str, Any], api_key: str, input_path: Path + ) -> ToolResult: + import requests + + output_dir = Path(inputs.get("output_dir", input_path.parent)) + diarize = inputs.get("diarize", False) + output_dir.mkdir(parents=True, exist_ok=True) + + locales = self._resolve_locales(inputs) + definition: dict[str, Any] = { + "locales": locales, + "profanityFilterMode": inputs.get("profanity_filter", "Masked"), + } + if diarize: + definition["diarization"] = { + "maxSpeakers": inputs.get("max_speakers", 4), + "enabled": True, + } + + url = f"{self._endpoint()}/speechtotext/transcriptions:transcribe?api-version={_API_VERSION}" + headers = {"Ocp-Apim-Subscription-Key": api_key} + + try: + with open(input_path, "rb") as audio_file: + response = requests.post( + url, + headers=headers, + files={ + "audio": (input_path.name, audio_file, "application/octet-stream"), + "definition": (None, json.dumps(definition), "application/json"), + }, + timeout=600, + ) + except requests.RequestException as exc: # network/timeout + return ToolResult(success=False, error=f"Azure Speech request failed: {exc}") + + if response.status_code != 200: + detail = response.text[:500] if response.text else "" + return ToolResult( + success=False, + error=f"Azure Speech returned HTTP {response.status_code}: {detail}", + ) + + try: + payload = response.json() + except ValueError: + return ToolResult( + success=False, + error="Azure Speech returned a non-JSON response.", + ) + + result_data = self._parse_payload(payload, locales) + result_data["provider"] = "azure" + output_path = output_dir / f"{input_path.stem}_transcript.json" + output_path.write_text(json.dumps(result_data, indent=2), encoding="utf-8") + + return ToolResult( + success=True, + data=result_data, + artifacts=[str(output_path)], + ) + + @staticmethod + def _parse_payload(payload: dict[str, Any], requested_locales: list[str]) -> dict[str, Any]: + """Map the Fast Transcription response onto the transcriber output schema.""" + segments: list[dict[str, Any]] = [] + word_timestamps: list[dict[str, Any]] = [] + detected_locale: str | None = None + + for idx, phrase in enumerate(payload.get("phrases", [])): + offset_ms = phrase.get("offsetMilliseconds", 0) + dur_ms = phrase.get("durationMilliseconds", 0) + confidence = phrase.get("confidence") + if detected_locale is None and phrase.get("locale"): + detected_locale = phrase["locale"] + + seg: dict[str, Any] = { + "id": idx, + "start": round(offset_ms / 1000.0, 3), + "end": round((offset_ms + dur_ms) / 1000.0, 3), + "text": (phrase.get("text") or "").strip(), + } + if "speaker" in phrase: + seg["speaker"] = phrase["speaker"] + + words = phrase.get("words") or [] + if words: + seg_words = [] + for w in words: + w_offset = w.get("offsetMilliseconds", 0) + w_dur = w.get("durationMilliseconds", 0) + word_entry = { + "word": w.get("text", ""), + "start": round(w_offset / 1000.0, 3), + "end": round((w_offset + w_dur) / 1000.0, 3), + # Fast Transcription has no per-word confidence; carry the + # phrase confidence so downstream schemas stay populated. + "probability": round(confidence, 3) if confidence is not None else None, + } + seg_words.append(word_entry) + word_timestamps.append(word_entry) + seg["words"] = seg_words + + segments.append(seg) + + duration_ms = payload.get("durationMilliseconds") + if duration_ms is None and segments: + duration_ms = max(int(s["end"] * 1000) for s in segments) + + return { + "segments": segments, + "word_timestamps": word_timestamps, + "language": detected_locale or (requested_locales[0] if requested_locales else None), + "duration_seconds": round((duration_ms or 0) / 1000.0, 3), + }