Skip to content

fix(dash): discover served model for configured chat endpoint#97

Open
volen-silo wants to merge 1 commit into
mainfrom
fix/tui-chat-model-discovery
Open

fix(dash): discover served model for configured chat endpoint#97
volen-silo wants to merge 1 commit into
mainfrom
fix/tui-chat-model-discovery

Conversation

@volen-silo

Copy link
Copy Markdown
Collaborator

Summary

  • The dash TUI chat now queries /v1/models at startup to discover the served model name when a chat endpoint is configured via OPENAI_BASE_URL / --chat-url but no model is set.
  • Why: the documented rocm serve … && OPENAI_BASE_URL=… rocm flow was broken — sending a message returned 404 Not Found: The model 'local-model' does not exist, making chat unusable out of the box.
  • Root cause: the managed-service and in-TUI detect-offer paths already discover the served model, but the startup env/CLI path fell straight through to resolve_llm_config's DEFAULT_CHAT_MODEL = "local-model" placeholder and built the agent with it. Accepting the consent prompt doesn't rebuild, so the placeholder reached the first request.
  • Fix: after resolving the config, when the endpoint is reachable and no explicit model was given, fetch /v1/models and adopt the served id (via the existing fetch_first_model / pick_first_model helpers), mirroring the managed-service path.

Non-obvious decisions

  • Gated on a successful TCP probe (probe_ok) so an unreachable configured endpoint never adds the 3s fetch timeout to startup.
  • An explicit --chat-model / config value always wins — discovery only fills the placeholder (enforced by the pure with_discovered_model helper).
  • Kept the local-model fallback when /v1/models can't be read — some endpoints ignore the model field or don't expose the listing.

Test plan

  • cargo test -p rocm-dash-tui --lib — 549 passed (3 new unit tests for with_discovered_model: placeholder replaced, explicit model preserved, failed discovery keeps fallback).
  • cargo clippy -p rocm-dash-tui --all-targets — clean.
  • Manual (deferred, needs a live engine): rocm serve Qwen/Qwen2.5-1.5B-Instruct --engine vllm, then OPENAI_BASE_URL=http://127.0.0.1:11435/v1 rocm, accept the endpoint in the Chat tab, send a message → expect a normal reply and no local-model 404.

Risk

Low — additive best-effort discovery on a single startup path; existing precedence and the neutral fallback are preserved.

A configured chat URL (OPENAI_BASE_URL / --chat-url) with no explicit
model resolved to the "local-model" placeholder, which 404s on servers
that register the model under its real id. Query /v1/models on startup
to adopt the served model, mirroring the managed-service path. Gated on
a reachable probe so an unreachable endpoint adds no fetch timeout to
launch, and an explicit --chat-model/config value still wins.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant