Skip to content

fix(dash): give instances a real Ready status instead of a Running catch-all#106

Open
michaelroy-amd wants to merge 1 commit into
mainfrom
fix/dash-instance-status
Open

fix(dash): give instances a real Ready status instead of a Running catch-all#106
michaelroy-amd wants to merge 1 commit into
mainfrom
fix/dash-instance-status

Conversation

@michaelroy-amd

Copy link
Copy Markdown
Member

Summary

InstanceStatus had no Ready state, so merge_instance hardcoded every discovered instance to Running regardless of whether it had actually passed a readiness probe. Managed services with a "ready" record status, Lemonade endpoints (only ever built after a health probe), and Docker-discovered containers all collapsed into the same Running label, hiding real lifecycle differences from the TUI and any future readiness-aware consumer (e.g. chat endpoint selection).

Root Cause

rocm_dash_core::traits::merge_instance unconditionally set status: InstanceStatus::Running on every Instance it built, and InstanceStatus itself had no variant to represent "passed its readiness probe" as distinct from "process is up." Every discovery source (managed-service registry, Lemonade collector, Docker collector) therefore reported the same status label no matter what it actually knew about the endpoint's readiness.

Changes

  • Add InstanceStatus::Ready and an is_serving() helper (Ready | Running) for call sites that only care about "is this actively serving."
  • Add a status field to DiscoveredService so each discovery source reports what it actually knows; merge_instance now uses svc.status instead of a hardcoded Running.
  • rocm-dash-daemon::registry::discovered_from_record maps a managed-service record's status string (ready/running/starting/recovering) to the matching InstanceStatus. "recovering" was previously dropped from the scrapeable set entirely (a bug relative to apps/rocm/src/main.rs's own "is this service live" checks, which treat recovering as live) — it's now scraped and mapped to Starting.
  • rocm-dash-collectors::lemonade::lemonade_service builds as Ready since every call site only constructs it after a successful /api/v1/health probe.
  • rocm-dash-collectors::docker has no readiness signal of its own, so Docker-discovered instances start Starting; rocm-dash-daemon::runner::run_loop promotes them to Ready on their first successful vLLM Prometheus scrape or Lemonade stats fetch (documented scope: this is the only readiness signal available for Docker-discovered instances).
  • Add Ready arms to the exhaustive status_meta/status_role matches in crates/rocm-dash-tui/src/ui/tabs/instances.rs; switch the == InstanceStatus::Running equality checks in home.rs/dock.rs/launcher.rs to .is_serving().

Test Plan

  • cargo build -p rocm-dash-core -p rocm-dash-collectors -p rocm-dash-daemon -p rocm-dash-tui
  • cargo clippy -p rocm-dash-core -p rocm-dash-collectors -p rocm-dash-daemon -p rocm-dash-tui --all-targets --all-features -- -D warnings
  • cargo test -p rocm-dash-core -p rocm-dash-collectors -p rocm-dash-daemon (all green, including 2 new tests: instance_status_ready_round_trips_as_lowercase_json, instance_status_is_serving_covers_ready_and_running_only, and discovered_from_record_maps_status_string_to_instance_status)
  • cargo test -p rocm-dash-tui -- --test-threads=1 (543 + 16 + 5 tests green)

Relates to EAI-7350

…tch-all

InstanceStatus had no Ready state, so merge_instance hardcoded every
discovered instance to Running regardless of whether it had actually
passed a readiness probe. Managed services with a "ready" record status,
Lemonade endpoints (built only after a health probe), and Docker
containers all collapsed into the same Running label, hiding real
lifecycle differences from the TUI and any future readiness-aware
consumer.

Add InstanceStatus::Ready, thread a status field through
DiscoveredService so each discovery source reports what it actually
knows, and drop merge_instance's hardcoded override in favor of
svc.status. Managed-service records now map status strings (including
"recovering", previously dropped from the scrapeable set entirely)
through to Ready/Running/Starting. Lemonade endpoints are built as
Ready since every call site only constructs them post-health-probe.
Docker-discovered instances have no status source of their own, so they
stay Starting until their first successful vLLM Prometheus (or Lemonade
stats) scrape promotes them to Ready.

Add an is_serving() helper (Ready | Running) for TUI call sites that
only care about "is this actively serving" rather than the precise
state, and add Ready arms to the exhaustive status_meta/status_role
matches in the instances tab.

Relates to EAI-7350

Signed-off-by: Michael Roy <michael.roy@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant