feat(transcription): Azure OpenAI realtime streaming transcription (BYOK)#1009
Open
bikramjitk wants to merge 1 commit into
Open
feat(transcription): Azure OpenAI realtime streaming transcription (BYOK)#1009bikramjitk wants to merge 1 commit into
bikramjitk wants to merge 1 commit into
Conversation
…n (BYOK) Adds a bring-your-own-key "Azure OpenAI" cloud transcription provider that streams dictation over the OpenAI realtime WebSocket protocol exposed by Azure OpenAI / AI Foundry resources, giving Azure users the same low latency as the existing OpenAI streaming path instead of the slower batch upload. - openaiRealtimeStreaming: generic Azure realtime URL builder that handles the GA (/openai/v1/realtime) and preview (/openai/realtime?api-version&deployment) surfaces as well as bare-host endpoints; api-key auth header; configurable input sample rate. The OpenAI (non-Azure) path is unchanged, gated on endpoint. - audioManager: gate Azure streaming on the cloud provider selection plus the three credentials; stream at 24 kHz (Azure realtime rejects <24 kHz) via a rate-aware AudioContext; everything else continues at 16 kHz. - ipcHandlers/environment/preload: persist and load the Azure transcribe endpoint, deployment and api-version (.env) and API key (safeStorage), with a dedicated dictation connect branch. - settings/types/registry/UI: new "Azure OpenAI" transcription provider tab with endpoint, deployment, api-version and key fields, scoped to Azure OpenAI transcription deployments (gpt-4o-transcribe / gpt-4o-mini-transcribe / whisper). Non-OpenAI Foundry transcription (Voice Live / Azure Speech) uses a different protocol and is intentionally out of scope; the deployment help text states the supported deployments so a wrong deployment surfaces a clear error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a bring-your-own-key Azure OpenAI cloud transcription provider that streams dictation over the OpenAI realtime WebSocket protocol exposed by Azure OpenAI / AI Foundry resources. Today, Azure/Foundry users fall back to the slower batch upload path; this gives them the same low latency as the existing OpenAI streaming path.
Why
OpenAIRealtimeStreamingwas hardcoded towss://api.openai.com/v1/realtime, so dictation streaming only worked with OpenAI directly. Azure exposes the identical realtime protocol at a resource-scoped URL withapi-keyauth, so we can reuse the same client with a generic URL builder.What changed
openaiRealtimeStreaming.js— generic Azure realtime URL builder that normalizes whatever endpoint form the user pastes and targets the correct surface:…/openai/v1(or bare host) → GA…/openai/v1/realtime…/openai→ preview…/openai/realtimewith adeploymentparamhttp(s)→ws(s), trailing/realtimetolerated,intent=transcriptionapi-keyauth header; configurable input sample rate.endpointbeing present.audioManager.js— gate Azure streaming on the cloud provider selection + presence of the three credentials; stream at 24 kHz (Azure realtime rejects input below 24 kHz) via a rate-awareAudioContext; everything else continues at 16 kHz.ipcHandlers.js/environment.js/preload.js— persist & load the Azure transcribe endpoint, deployment and api-version (.env) and API key (safeStorage), plus a dedicated dictation connect branch. Kept separate from the existing enterprise Azure reasoning keys so the two deployments don't clobber each other.TranscriptionModelPicker— new "Azure OpenAI" transcription tab with endpoint, deployment, api-version and key fields.Scope
This targets Azure OpenAI transcription deployments (
gpt-4o-transcribe,gpt-4o-mini-transcribe,whisper). Non-OpenAI Foundry speech-to-text (Voice Live API / Azure AI Speech /mai-transcribe) uses a different endpoint, auth and message schema and is intentionally out of scope; the deployment field help text states the supported deployments, and a wrong/non-transcribe deployment surfaces a connection error rather than silently degrading.Validation
api-keyauth for non-browser clients,session.type: transcriptionschema, ≥24 kHz PCM16).tsc --noEmitclean,eslintclean,i18n:checkconsistent across all 10 locales, renderer build passes.Backward compatibility
No change for existing OpenAI / OpenWhispr Cloud / Corti users — all new behavior is gated behind selecting the Azure provider and supplying credentials.
Related issues
wss://api.openai.comand ignores custom transcription endpoints) — this PR removes that hardcoding for the Azure path. The generic custom OpenAI-compatible proxy case in Realtime streaming hardcodes wss://api.openai.com — ignores custom transcription endpoint #581 is a separate slice and is not addressed here.