fix(dash): discover served model for configured chat endpoint#97
Open
volen-silo wants to merge 1 commit into
Open
fix(dash): discover served model for configured chat endpoint#97volen-silo wants to merge 1 commit into
volen-silo wants to merge 1 commit into
Conversation
A configured chat URL (OPENAI_BASE_URL / --chat-url) with no explicit model resolved to the "local-model" placeholder, which 404s on servers that register the model under its real id. Query /v1/models on startup to adopt the served model, mirroring the managed-service path. Gated on a reachable probe so an unreachable endpoint adds no fetch timeout to launch, and an explicit --chat-model/config value still wins. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
034e4e1 to
f8c2da1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/v1/modelsat startup to discover the served model name when a chat endpoint is configured viaOPENAI_BASE_URL/--chat-urlbut no model is set.rocm serve … && OPENAI_BASE_URL=… rocmflow was broken — sending a message returned404 Not Found: The model 'local-model' does not exist, making chat unusable out of the box.resolve_llm_config'sDEFAULT_CHAT_MODEL = "local-model"placeholder and built the agent with it. Accepting the consent prompt doesn't rebuild, so the placeholder reached the first request./v1/modelsand adopt the served id (via the existingfetch_first_model/pick_first_modelhelpers), mirroring the managed-service path.Non-obvious decisions
probe_ok) so an unreachable configured endpoint never adds the 3s fetch timeout to startup.--chat-model/ config value always wins — discovery only fills the placeholder (enforced by the purewith_discovered_modelhelper).local-modelfallback when/v1/modelscan't be read — some endpoints ignore the model field or don't expose the listing.Test plan
cargo test -p rocm-dash-tui --lib— 549 passed (3 new unit tests forwith_discovered_model: placeholder replaced, explicit model preserved, failed discovery keeps fallback).cargo clippy -p rocm-dash-tui --all-targets— clean.rocm serve Qwen/Qwen2.5-1.5B-Instruct --engine vllm, thenOPENAI_BASE_URL=http://127.0.0.1:11435/v1 rocm, accept the endpoint in the Chat tab, send a message → expect a normal reply and nolocal-model404.Risk
Low — additive best-effort discovery on a single startup path; existing precedence and the neutral fallback are preserved.