Skip to content

feat(transcription): Azure OpenAI realtime streaming transcription (BYOK)#1009

Open
bikramjitk wants to merge 1 commit into
OpenWhispr:mainfrom
bikramjitk:bikramjitk-azure-streaming-transcription
Open

feat(transcription): Azure OpenAI realtime streaming transcription (BYOK)#1009
bikramjitk wants to merge 1 commit into
OpenWhispr:mainfrom
bikramjitk:bikramjitk-azure-streaming-transcription

Conversation

@bikramjitk

@bikramjitk bikramjitk commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a bring-your-own-key Azure OpenAI cloud transcription provider that streams dictation over the OpenAI realtime WebSocket protocol exposed by Azure OpenAI / AI Foundry resources. Today, Azure/Foundry users fall back to the slower batch upload path; this gives them the same low latency as the existing OpenAI streaming path.

Why

OpenAIRealtimeStreaming was hardcoded to wss://api.openai.com/v1/realtime, so dictation streaming only worked with OpenAI directly. Azure exposes the identical realtime protocol at a resource-scoped URL with api-key auth, so we can reuse the same client with a generic URL builder.

What changed

  • openaiRealtimeStreaming.js — generic Azure realtime URL builder that normalizes whatever endpoint form the user pastes and targets the correct surface:
    • …/openai/v1 (or bare host) → GA …/openai/v1/realtime
    • …/openai → preview …/openai/realtime with a deployment param
    • http(s)ws(s), trailing /realtime tolerated, intent=transcription
    • api-key auth header; configurable input sample rate.
    • The OpenAI (non-Azure) path is unchanged — the Azure branch is gated entirely on endpoint being present.
  • audioManager.js — gate Azure streaming on the cloud provider selection + presence of the three credentials; stream at 24 kHz (Azure realtime rejects input below 24 kHz) via a rate-aware AudioContext; everything else continues at 16 kHz.
  • ipcHandlers.js / environment.js / preload.js — persist & load the Azure transcribe endpoint, deployment and api-version (.env) and API key (safeStorage), plus a dedicated dictation connect branch. Kept separate from the existing enterprise Azure reasoning keys so the two deployments don't clobber each other.
  • settings store / types / model registry / TranscriptionModelPicker — new "Azure OpenAI" transcription tab with endpoint, deployment, api-version and key fields.

Scope

This targets Azure OpenAI transcription deployments (gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper). Non-OpenAI Foundry speech-to-text (Voice Live API / Azure AI Speech / mai-transcribe) uses a different endpoint, auth and message schema and is intentionally out of scope; the deployment field help text states the supported deployments, and a wrong/non-transcribe deployment surfaces a connection error rather than silently degrading.

Validation

  • Assumptions cross-checked against Microsoft Learn (realtime WS URL forms for GA vs preview, api-key auth for non-browser clients, session.type: transcription schema, ≥24 kHz PCM16).
  • Verified end-to-end against a live Azure AI Foundry endpoint: correct WS URL, 24 kHz capture, accurate transcription, low latency.
  • tsc --noEmit clean, eslint clean, i18n:check consistent across all 10 locales, renderer build passes.

Backward compatibility

No change for existing OpenAI / OpenWhispr Cloud / Corti users — all new behavior is gated behind selecting the Azure provider and supplying credentials.

Related issues

…n (BYOK)

Adds a bring-your-own-key "Azure OpenAI" cloud transcription provider that
streams dictation over the OpenAI realtime WebSocket protocol exposed by
Azure OpenAI / AI Foundry resources, giving Azure users the same low latency
as the existing OpenAI streaming path instead of the slower batch upload.

- openaiRealtimeStreaming: generic Azure realtime URL builder that handles the
  GA (/openai/v1/realtime) and preview (/openai/realtime?api-version&deployment)
  surfaces as well as bare-host endpoints; api-key auth header; configurable
  input sample rate. The OpenAI (non-Azure) path is unchanged, gated on endpoint.
- audioManager: gate Azure streaming on the cloud provider selection plus the
  three credentials; stream at 24 kHz (Azure realtime rejects <24 kHz) via a
  rate-aware AudioContext; everything else continues at 16 kHz.
- ipcHandlers/environment/preload: persist and load the Azure transcribe
  endpoint, deployment and api-version (.env) and API key (safeStorage), with a
  dedicated dictation connect branch.
- settings/types/registry/UI: new "Azure OpenAI" transcription provider tab with
  endpoint, deployment, api-version and key fields, scoped to Azure OpenAI
  transcription deployments (gpt-4o-transcribe / gpt-4o-mini-transcribe / whisper).

Non-OpenAI Foundry transcription (Voice Live / Azure Speech) uses a different
protocol and is intentionally out of scope; the deployment help text states the
supported deployments so a wrong deployment surfaces a clear error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bikramjitk

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants