Skip to content

feat: add LiteLLM as AI gateway provider#1016

Open
RheagalFire wants to merge 1 commit into
OpenWhispr:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#1016
RheagalFire wants to merge 1 commit into
OpenWhispr:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

Summary

  • Add LiteLLM as a first-class AI inference provider, giving users access to 100+ LLM providers (OpenAI, Anthropic, Google, Azure, Bedrock, Cohere, Mistral, and more) through a single proxy interface.
  • LiteLLM appears as a named provider in the provider selector with the OpenAI-compatible panel for base URL and API key configuration, defaulting to http://localhost:4000/v1.

Motivation

OpenWhispr currently supports 5 cloud providers (OpenAI, Anthropic, Gemini, Groq, plus a generic "Custom" endpoint). Users who want providers not directly supported (Bedrock, Azure OpenAI, Mistral, Cohere, Together AI, etc.) must use the generic Custom provider and manually configure everything.

LiteLLM is an AI gateway that proxies 100+ LLM providers through a single OpenAI-compatible endpoint. With this PR, users can:

  1. Run litellm --model anthropic/claude-sonnet-4-6 (or deploy the proxy server)
  2. Select "LiteLLM" as the provider in OpenWhispr settings
  3. Access any of 100+ models with automatic provider routing, load balancing, and cost tracking

Changes

  • src/services/ai/inferenceProviders/litellm.ts - new litellmProvider implementing InferenceProvider (follows the Groq provider pattern)
  • src/services/ai/inferenceProviders/index.ts - registered litellm in PROVIDER_REGISTRY
  • src/services/ai/providers.ts - added litellm case to AI SDK factory (reuses createOpenAI with custom baseURL)
  • src/services/ReasoningService.ts - added litellm to cloud provider lists, API key handling, and endpoint resolution
  • src/components/ReasoningModelSelector.tsx - added to CLOUD_PROVIDER_IDS, renders OpenAICompatiblePanel with LiteLLM-specific defaults
  • src/components/chat/useChatStreaming.ts - added litellm to cloud provider detection
  • src/stores/settingsStore.ts - added litellm to provider type union

Tests

1. TypeScript type check:

$ npx tsc --noEmit --pretty
# No errors

2. Prettier format check:

$ npx prettier --check src/services/ai/inferenceProviders/litellm.ts src/services/ai/inferenceProviders/index.ts src/services/ai/providers.ts
All matched files use Prettier code style!

Implementation details

  • The LiteLLM provider delegates to ctx.callChatCompletionsApi() (same pattern as Groq) - the shared context handles request formatting, retries, and response parsing.
  • Uses createOpenAI({ apiKey, baseURL }) from @ai-sdk/openai in the AI SDK factory since LiteLLM proxy exposes an OpenAI-compatible API.
  • The OpenAICompatiblePanel renders with defaultBaseUrl="http://localhost:4000/v1" for LiteLLM, so users see the correct default proxy URL.
  • Model names are free-text (like Custom) since available models depend on the user's LiteLLM proxy configuration. The proxy auto-discovers models via /v1/models.

Risk / Compatibility

  • Additive only. Existing providers untouched.
  • No new npm dependencies - reuses existing @ai-sdk/openai and the shared callChatCompletionsApi infrastructure.
  • LiteLLM provider reuses the same API key and base URL storage as the Custom provider (via OpenAICompatiblePanel).

Example usage

  1. Start a LiteLLM proxy:
pip install litellm
litellm --model anthropic/claude-sonnet-4-6
  1. In OpenWhispr Settings > AI Models:
    • Select LiteLLM as the provider
    • Base URL: http://localhost:4000/v1 (default)
    • API Key: your LiteLLM proxy master key (or leave empty for keyless local proxy)
    • Model: anthropic/claude-sonnet-4-6

@gabrielste1n gabrielste1n self-requested a review July 3, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant