feat(serve): surface a coarse startup phase while a service comes up#107
Open
michaelroy-amd wants to merge 4 commits into
Open
feat(serve): surface a coarse startup phase while a service comes up#107michaelroy-amd wants to merge 4 commits into
michaelroy-amd wants to merge 4 commits into
Conversation
A service that is still coming up showed only a generic STARTING/spinner
with no indication of whether it was pulling weights, loading them, or
warming up — so a slow first-token launch was indistinguishable from a
hang. The serve supervisor already streams the engine's stdout/stderr to
the service log file but never inspected it, and the dashboard had no
field to carry a startup stage.
Add a coarse startup phase (Downloading/Loading/Warmup) end to end:
- rocm-core: add ManagedServiceRecord.startup_phase (Option<String>,
serde-default so old on-disk records still load), cleared once the
service reaches ready.
- rocmd: tail the service log during wait_for_service_ready and classify
each line into a phase, persisting transitions to the record. Parsing
is split into small pure helpers (last_cr_segment collapses \r progress
redraws the same way the dashboard job console does; classify_startup_phase
matches the vLLM/llama.cpp/HF startup vocabulary) so it is unit-testable.
- rocm-dash-core: add a StartupPhase enum and turn InstanceStatus::Starting
into Starting { phase: Option<StartupPhase> }. StartupPhase is fieldless
and Copy, so InstanceStatus stays Copy and the only ripple is pattern
shape. A new InstanceStatus::label() owns the status text (READY /
DOWNLOADING / LOADING / WARMUP / …).
- rocm-dash-daemon registry: read the record's startup_phase and map it
onto Starting { phase } (recovering/starting only); Docker discovery and
the scrape-success promotion keep phase None.
- rocm-dash-tui: status_meta/status_role and the services list render via
label(), so the phase shows on instance cards, the detail pane, and the
services manager. (serving.rs is a static verb menu with no per-instance
status, so the live rendering lives in instances.rs + services_manager.rs.)
Stacked on #106 (EAI-7350).
Relates to EAI-7355
Signed-off-by: Michael Roy <michael.roy@amd.com>
3 tasks
…ing` supervise_service (apps/rocmd/src/lib.rs) flips a managed-service record's on-disk status to "running" immediately after spawning the engine child -- before it starts polling and classifying startup_phase from the serve log -- and only clears startup_phase once the service reaches "ready". So a record can sit at status="running" with startup_phase=Some(...) for the entire download/load/warmup window. registry::instance_status_for_record only read startup_phase in the "starting"/"recovering" arm, so that overlap fell straight through to a bare InstanceStatus::Running and DOWNLOADING/LOADING/WARMUP never rendered during a real `rocm serve <large-model>` cold start. The feature was inert end-to-end despite the plumbing all being in place. Chose the lower-risk fix: teach instance_status_for_record to also honor startup_phase on the "running" arm, rather than delaying the "running" write in supervise_service until the service is ready (rejected: that would leave the record at status="starting" for the whole download, and manifest_service_recovery_reason's 5-minute staleness check keys off "starting"/"recovering" -- a slow cold start could then get misclassified as a stale/stuck launch and trigger an unwanted recovery restart, a regression supervise_service's current "running"-first ordering was implicitly guarding against). The existing registry test for this path constructed a status="starting" + startup_phase record -- a shape the real producer never writes -- which masked the bug. Replaced it with a case driven through the actual load_service_records + discovered_from_record path using the real status="running" + startup_phase shape, plus a starting/recovering parametrized case and a running-with-no-phase steady-state case. Also bump PROTOCOL_VERSION: InstanceStatus::Starting went from a unit variant to a struct variant carrying `phase` in the prior commit on this branch, which changes Event::InstanceDiscovered/Snapshot's external-tag JSON shape for that variant; the version bump was missing. Regenerated THIRD_PARTY_NOTICES.txt (pre-existing ordering drift, unrelated to this change). Relates to EAI-7355 Signed-off-by: Michael Roy <michael.roy@amd.com>
…o dep change) Signed-off-by: Michael Roy <michael.roy@amd.com>
…g instances serving on unknown phase
- Add a cross-crate serde contract test: serialize rocm-core's real
ManagedServiceRecord (with startup_phase set) and deserialize it as the
daemon's ServiceRecord, so a future #[serde(rename)] on either side fails
the test instead of silently dropping the phase (previously the field-name
match rested only on a hand-written JSON literal). Adds rocm-core as a
workspace-local dev-dependency (no third-party dep, TPN unaffected).
- Harden the running-status mapping: a 'running' record with no phase OR an
unrecognized/foreign phase token now stays Running (serving); only the
pinned tokens (downloading/loading/warmup) map to Starting{phase}. Prevents
wrongly downgrading a genuinely-serving instance to non-serving STARTING.
Signed-off-by: Michael Roy <michael.roy@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
While a managed service is coming up, the dashboard showed only a generic
STARTINGlabel (or a spinner) with no indication of what it was doing — pulling weights, loading them onto the device, or warming up. A slow first launch was therefore indistinguishable from a hang. This PR surfaces a coarse startup phase (Downloading / Loading / Warmup) end to end.Root Cause
The serve supervisor (
apps/rocmd) already redirects the engine's stdout/stderr into the per-service log file, butwait_for_service_readyonly polled a healthcheck RPC and never inspected that output.ManagedServiceRecordhad no field to carry a startup stage, and the dashboard'sInstanceStatus::Startingwas a unit variant with nowhere to put one.Changes
ManagedServiceRecord.startup_phase: Option<String>(#[serde(default)]so older on-disk records still deserialize), cleared back toNoneonce the service reachesready.wait_for_service_readynow tails the service log while polling and reports each phase transition, which the caller persists to the record. Parsing is split into small pure, unit-tested helpers:last_cr_segmentcollapses\rprogress redraws (pip/tqdm/HF), the same rule the dashboard job console uses.classify_startup_phasematches the common vLLM / llama.cpp / Hugging Face startup vocabulary into adownloading/loading/warmuptoken.StartupPhaseenum and changeInstanceStatus::StartingtoStarting { phase: Option<StartupPhase> }.StartupPhaseisCopy, soInstanceStatusstaysCopy— the only ripple is pattern shape (Starting { .. }), not aCopy-removal cascade. A newInstanceStatus::label()centralizes the status text (READY/DOWNLOADING/LOADING/WARMUP/ …).startup_phaseontoStarting { phase }forstarting/recoveringrecords; Docker discovery and the scrape-success promotion keepphase: None.status_meta/status_roleand the services-manager list now render vialabel(), so the phase shows on instance cards, the detail pane, and the services list. Note:serving.rsis a static verb menu with no per-instance status rendering, so the live phase display lives ininstances.rs+services_manager.rs(the actual instance-status surfaces).Scope / Notes
STARTING.Startingserde wire shape changes from the bare string"starting"to an externally-tagged object; this only affects the in-version daemon socket/replay stream, and the round trip is covered by a new test.Test Plan
cargo build(workspace)cargo clippy --all-targets --all-features -- -D warnings(workspace, clean)cargo test -p rocm-core -p rocm-dash-core -p rocm-dash-collectors -p rocm-dash-daemon— green, incl. new tests:instance_status_label_surfaces_startup_phase,instance_status_starting_round_trips_through_serde,startup_phase_from_token_round_trips_labels,discovered_from_record_surfaces_startup_phase_while_startingcargo test -p rocmd --lib— 109 green, incl.last_cr_segment_keeps_final_progress_redraw,classify_startup_phase_maps_engine_vocabulary,classify_startup_phase_emits_only_dashboard_known_tokens,read_new_log_phase_advances_and_tracks_latestcargo test -p rocm-dash-tui -- --test-threads=1— 543 + 16 + 5 greenRelates to EAI-7355