Skip to content

feat(llm): add openai_compatible provider#165

Merged
zc277584121 merged 2 commits into
zilliztech:mainfrom
code2tan:improve_code
Jul 7, 2026
Merged

feat(llm): add openai_compatible provider#165
zc277584121 merged 2 commits into
zilliztech:mainfrom
code2tan:improve_code

Conversation

@code2tan

@code2tan code2tan commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

#167

Summary

New openai_compatible LLM provider so [summary] / [description] can target any OpenAI Chat Completions-compatible endpoint (DeepSeek, Moonshot, Qwen, vLLM, Ollama /v1, …) by configuring base_url + api_key + model in TOML. The cloud openai.py is untouched — mirrors how the embeddings registry already splits ollama/mistral/jina into separate providers instead of overloading openai with a base_url switch.

Why

  • provider = "openai" was overloaded (SDK protocol vs. cloud), so wizard hints and DEFAULT_TEXT_MODELS were misleading for custom endpoints.
  • DEFAULT_TEXT_MODELS["openai"] = "gpt-4o-mini" is wrong for vLLM/Ollama/DeepSeek — compatible endpoints have no sensible default model.
  • Per-endpoint quirks (vision support, param handling) had nowhere to live.

Changes

File Change
common/llm/openai_compatible.py new — OpenAICompatibleLlm: base_url required, api_key optional, model per-call; inlines _resolve_secret_ref (env:/file:, avoids common→engine cycle, TODO(secrets))
common/llm/__init__.py register provider; empty default model; get_provider(name, **kwargs) forwards kwargs
config.py [summary] / [description] gain base_url / api_key (default "", read only for this provider)
common/summary.py, common/vlm.py _ensure_llm forwards endpoint config
server/setup_wizard.py new choice + _prompt_openai_compatible_endpoint() (base_url/api_key/model) + _apply writes the fields
common/llm/anthropic.py, common/llm/gemini.py __init__ accepts **_kwargs (tolerate uniform forwarding)
tests/test_llm_registry.py, tests/test_setup_wizard_sections.py new (10 tests) + 3 _apply cases
docs/providers.md, docs/configuration.md provider table + example + OPENAI_API_KEY note

Key decisions

  • No implicit cloud fallback for api_key. Absent api_key would make AsyncOpenAI auto-read OPENAI_API_KEY — silently pointing a custom endpoint at OpenAI cloud creds. A placeholder is substituted instead (locked by a test).
  • Default model is empty, forcing explicit config — endpoint-specific, no default fits.
  • Did not import CredentialService.resolve — it lives in engine, a higher layer; lifting it to common.secrets is a separate refactor, tracked via TODO(secrets).

Backward compatibility

openai.py unchanged (env path preserved); new fields default empty; get_provider's added **kwargs is signature-compatible. Caveat: base_url/api_key are not in the transformation-cache key — a switch without a model change would return cached summaries from the prior endpoint (low risk in practice; a switch is always accompanied by a model change).

Example

[summary]
enabled = true
provider = "openai_compatible"
model = "deepseek-chat"
base_url = "https://api.deepseek.com/v1"
api_key = "env:DEEPSEEK_API_KEY"

code2t added 2 commits July 6, 2026 20:02
This commit adds type hints and uses specific OpenAI message parameters for chat completions, improving code clarity and ensuring proper usage of the OpenAI API.
This commit introduces support for OpenAI-compatible endpoints, including new tests, configuration options, and documentation updates. It also ensures that the `openai_compatible` provider correctly handles `base_url` and `api_key` with credential references.
@zc277584121 zc277584121 merged commit e9b1431 into zilliztech:main Jul 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants