My focus is on inference cost attribution and there are some metric gaps from that perspective.
Gaps
1. Output and cached token counts
request_input_tokens is recorded from len(reqCtx.TokenIDs) after render. However, there are no request_output_tokens or request_cached_tokens histograms. These come from the usage object in the vLLM decode response body, which the coordinator never currently parses.
This is the primary blocker for cost attribution and chargeback: without output token counts at the coordinator layer, per-request cost cannot be computed without going to the EPP — and the EPP only sees a fragment of the request lifecycle in the coordinator model.
2. target_model_name label — model-rewrite visibility
All metrics carry only model_name, the client-supplied name. When an InferenceObjective rewrites the model (e.g. Qwen3-32B → Qwen3-32B-FP8), the EPP records both model_name and target_model_name as separate labels on every EPP metric. The coordinator records only model_name, so it is impossible to detect from coordinator metrics alone that a rewrite is in effect, and PromQL joins between coordinator and EPP metrics will appear to disagree on what model was served.
3. tenant_id, user_id, workload_id - attribution labels on any metric
To enable attribution of inference costs based on tenant, workload and user. (Design doc based on coordinator currently WIP. EPP based proposal can be seen here.)
Proposals
1. Output and cached token counts
Two prerequisites:
- Streaming requests: inject
"stream_options": {"include_usage": true} into the decode request body before forwarding. vLLM omits usage from the SSE stream by default; without this the final chunk carries no token counts.
- Decode response body tap: for non-streaming responses, parse
usage from the JSON body. For streaming, intercept the final data: SSE chunk (the one with "usage": {...}). The natural place for both is the ModifyResponse hook on the httputil.ReverseProxy in decode_proxy.go — already wired for decodeOutcome.Status — using a tee on resp.Body to read without consuming the stream.
Suggested metrics, both reusing TokenCountBuckets:
llm_d_coordinator_request_output_tokens {model_name}
llm_d_coordinator_request_cached_tokens {model_name}
2. target_model_name label
The EPP communicates the rewrite decision via the x-llm-d-model-name-rewrite response header (metadata.ModelNameRewriteKey), which is available in ModifyResponse on the decode leg (same hook as gap 1). Capture it into a new RequestContext.TargetModel field and add target_model_name as a label on request_total, request_error_total, request_duration_seconds, request_input_tokens, and the proposed output/cached token histograms — matching the EPP label set exactly.
Originally posted by @simanadler in #2616 (comment)
My focus is on inference cost attribution and there are some metric gaps from that perspective.
Gaps
1. Output and cached token counts
request_input_tokensis recorded fromlen(reqCtx.TokenIDs)after render. However, there are norequest_output_tokensorrequest_cached_tokenshistograms. These come from theusageobject in the vLLM decode response body, which the coordinator never currently parses.This is the primary blocker for cost attribution and chargeback: without output token counts at the coordinator layer, per-request cost cannot be computed without going to the EPP — and the EPP only sees a fragment of the request lifecycle in the coordinator model.
2.
target_model_namelabel — model-rewrite visibilityAll metrics carry only
model_name, the client-supplied name. When anInferenceObjectiverewrites the model (e.g.Qwen3-32B→Qwen3-32B-FP8), the EPP records bothmodel_nameandtarget_model_nameas separate labels on every EPP metric. The coordinator records onlymodel_name, so it is impossible to detect from coordinator metrics alone that a rewrite is in effect, and PromQL joins between coordinator and EPP metrics will appear to disagree on what model was served.3. tenant_id, user_id, workload_id - attribution labels on any metric
To enable attribution of inference costs based on tenant, workload and user. (Design doc based on coordinator currently WIP. EPP based proposal can be seen here.)
Proposals
1. Output and cached token counts
Two prerequisites:
"stream_options": {"include_usage": true}into the decode request body before forwarding. vLLM omitsusagefrom the SSE stream by default; without this the final chunk carries no token counts.usagefrom the JSON body. For streaming, intercept the finaldata:SSE chunk (the one with"usage": {...}). The natural place for both is theModifyResponsehook on thehttputil.ReverseProxyindecode_proxy.go— already wired fordecodeOutcome.Status— using a tee onresp.Bodyto read without consuming the stream.Suggested metrics, both reusing
TokenCountBuckets:2.
target_model_namelabelThe EPP communicates the rewrite decision via the
x-llm-d-model-name-rewriteresponse header (metadata.ModelNameRewriteKey), which is available inModifyResponseon the decode leg (same hook as gap 1). Capture it into a newRequestContext.TargetModelfield and addtarget_model_nameas a label onrequest_total,request_error_total,request_duration_seconds,request_input_tokens, and the proposed output/cached token histograms — matching the EPP label set exactly.Originally posted by @simanadler in #2616 (comment)