Skip to content

Commit 4359a24

Browse files
committed
fix(serving): confirm ONNX session before status
Add a status reconciliation guard to the ONNX serving adapter. After the Rust ONNX session manager reports a successful load, the adapter now lists sessions and verifies the requested model is available before recording backend served status. If the session list does not include the model or cannot be read, serving returns a non-critical provider-load failure instead of publishing stale loaded state. This keeps durable serving state behind the provider-owned session truth. Update the ONNX Runtime serving plan with M4 status, verification evidence, and remaining serving gaps. Verification: cargo fmt --manifest-path rust/Cargo.toml --all -- --check; cargo test --manifest-path rust/crates/pumas-rpc/Cargo.toml serving; cargo test --manifest-path rust/crates/pumas-core/Cargo.toml serving
1 parent 3506464 commit 4359a24

3 files changed

Lines changed: 49 additions & 4 deletions

File tree

docs/plans/onnx-runtime-embedding-serving/execution-and-coordination.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,15 @@ Update during implementation:
454454
secrets, or full model paths. Verification passed:
455455
`cargo fmt --manifest-path rust/Cargo.toml --all -- --check` and
456456
`cargo test --manifest-path rust/crates/pumas-rpc/Cargo.toml openai_gateway`.
457+
- 2026-05-11: Added an ONNX serving status reconciliation guard before durable
458+
served-state updates. After the Rust ONNX session manager reports a successful
459+
load, `serving_onnx.rs` now lists sessions and verifies the requested model is
460+
present before calling `record_served_model`; a mismatch returns a
461+
non-critical provider-load failure instead of publishing stale loaded status.
462+
Verification passed:
463+
`cargo fmt --manifest-path rust/Cargo.toml --all -- --check`,
464+
`cargo test --manifest-path rust/crates/pumas-rpc/Cargo.toml serving`, and
465+
`cargo test --manifest-path rust/crates/pumas-core/Cargo.toml serving`.
457466

458467
## Commit Cadence Notes
459468

docs/plans/onnx-runtime-embedding-serving/milestones.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -578,7 +578,7 @@ state.
578578
- [ ] Move existing Ollama and llama.cpp serving paths behind provider serving
579579
adapters before adding ONNX load/unload so the RPC handler only performs
580580
boundary parsing, validation orchestration, and response shaping.
581-
- [ ] Confirm the Rust ONNX provider status/list includes the model before
581+
- [x] Confirm the Rust ONNX provider status/list includes the model before
582582
recording loaded status.
583583
- [x] Add unload support through the Rust ONNX session manager and served
584584
status removal.
@@ -608,9 +608,11 @@ selected ONNX profile is running and the primary executable artifact is `.onnx`.
608608
Provider behavior drives ONNX artifact compatibility, and ONNX rejects
609609
llama.cpp-specific placement overrides with non-critical domain errors. The RPC
610610
serving boundary now loads/unloads ONNX through the Rust fake session manager
611-
and records/removes backend served status. Real ONNX Runtime execution,
612-
duplicate load/unload idempotency, session status reconciliation before record,
613-
and gateway embedding routing remain open.
611+
and records/removes backend served status. The ONNX serving adapter now
612+
confirms the Rust session manager lists the loaded model before recording
613+
backend served status. Real ONNX Runtime execution, duplicate load/unload
614+
idempotency, and route/profile fallback cleanup remain open. Gateway embedding
615+
routing has started under Milestone 5.
614616

615617
### Milestone 5: Pumas Gateway Routing
616618

rust/crates/pumas-rpc/src/handlers/serving_onnx.rs

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ pub(super) async fn serve_onnx_model(
5050
}
5151
};
5252

53+
let onnx_model_id = load_request.model_id.clone();
5354
let session = match state.onnx_session_manager.load(load_request).await {
5455
Ok(session) => session,
5556
Err(error) => {
@@ -65,6 +66,18 @@ pub(super) async fn serve_onnx_model(
6566
.await;
6667
}
6768
};
69+
if let Err(error) = confirm_onnx_session_loaded(state, &onnx_model_id).await {
70+
warn!("ONNX fake session status confirmation failed: {}", error);
71+
return non_critical_failure_response(
72+
state,
73+
serving_error(
74+
ModelServeErrorCode::ProviderLoadFailed,
75+
"ONNX Runtime loaded the selected model but did not report it as available",
76+
&request,
77+
),
78+
)
79+
.await;
80+
}
6881

6982
let status = ServedModelStatus {
7083
model_id: request.model_id.clone(),
@@ -176,3 +189,24 @@ async fn resolve_onnx_model_path(
176189
}
177190
Ok(Some(onnx_path))
178191
}
192+
193+
async fn confirm_onnx_session_loaded(
194+
state: &AppState,
195+
model_id: &OnnxModelId,
196+
) -> Result<(), String> {
197+
let sessions = state
198+
.onnx_session_manager
199+
.list()
200+
.await
201+
.map_err(|error| error.to_string())?;
202+
if sessions
203+
.iter()
204+
.any(|session| session.model_id.as_str() == model_id.as_str())
205+
{
206+
return Ok(());
207+
}
208+
Err(format!(
209+
"ONNX model '{}' was absent from session list after load",
210+
model_id.as_str()
211+
))
212+
}

0 commit comments

Comments
 (0)