fix(saturation): treat missing KV/queue metrics as saturated, not empty - #1543
Open
UgaTheDev wants to merge 1 commit into
Open
fix(saturation): treat missing KV/queue metrics as saturated, not empty#1543UgaTheDev wants to merge 1 commit into
UgaTheDev wants to merge 1 commit into
Conversation
A replica whose KV cache or queue metric failed to scrape defaulted to 0 and was analyzed as idle with full spare capacity, so a saturated pod with a transient scrape failure could make an unsafe scale-down look safe or suppress a needed scale-up. Collector now tags ReplicaMetrics with KvCacheUsageMissing/QueueLengthMissing and the V1 analyzer treats either flag as a saturation hit, excluding the placeholder from the usage average/max. Fixes llm-d#360 Signed-off-by: Kush Zingade <kush.zingade@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #360.
When a replica's KV cache usage or queue length metric fails to scrape for a
cycle, the collector (
internal/collector/replica_metrics.go) defaults thevalue to 0 so the rest of the pipeline has something to work with. The V1
saturation analyzer (
internal/saturationv1/analyzer.go) then reads that 0exactly like a genuinely idle replica: a pod that is actually saturated but
had a transient scrape failure gets counted as non-saturated with full spare
capacity. That can make an unsafe scale-down look safe, or suppress a
scale-up that should have fired — silently, since nothing distinguishes "0"
from "unknown."
Fix
domain.ReplicaMetricsgains two fields,KvCacheUsageMissingandQueueLengthMissing, set by the collector whenever the correspondingPrometheus query returned no value for a pod this cycle. The 0 placeholder
is still stored (for compatibility with anything already reading the
numeric fields), but it's now paired with a flag saying whether it's real.
saturationv1.Analyzer.analyzeVarianttreats either flag as an automaticsaturation hit for that replica — conservative-by-default, matching the
direction of the risk in the bug report (missing data biases toward
"assume full," which can only cause an extra scale-up or a blocked
scale-down, never the unsafe direction). The replica is excluded from
AvgKvCacheUsage/MaxKvCacheUsageso the 0 placeholder doesn't drag thosedown either.
missing-metrics replica into
SaturatedReplicasnaturally dropsNonSaturatedCount, which trips the existingMinNonSaturatedReplicasForScaleDowngate when too few real replicasremain to redistribute load — the same path that already protects a
single-replica scale-down today.
Two design options were considered (exclude the replica entirely vs. default
it conservatively); the write-up and the case for the conservative default
chosen here are in the issue thread.
Out of scope:
saturation_v2'scomputeReplicaCapacityFallbackalso readsrm.KvCacheUsagedirectly for its fallback capacity estimate and has thesame theoretical exposure. It's a narrower, already-flagged approximation
(comment in that file acknowledges its unit mismatch caveat), and this PR
doesn't touch it — flagging here so it isn't lost.
Testing
New regression tests, verified to fail against the pre-fix
analyzeVariant/AnalyzeModelSaturation(confirmed locally by reverting the analyzer changeand re-running):
internal/saturationv1/analyzer_test.go:TestAnalyzeVariant_MissingMetricsTreatedAsSaturated— a replica withKvCacheUsageMissing/QueueLengthMissingset is classified saturateddespite reading 0, and its placeholder is excluded from the usage average.
TestAnalyzeModelSaturation_MissingMetricsBlockUnsafeScaleDown— a2-replica model where one replica's metrics are missing reports
ScaleDownSafe=false(pre-fix, it readtrue).internal/collector/replica_metrics_test.go: new subtests underTestCollectReplicaMetrics_FreshnessverifyingKvCacheUsageMissing/QueueLengthMissingare set exactly when the corresponding Prometheusquery has no value for the pod, and clear when both are present.
go build ./...andgo vet ./...are clean. The fullgo test ./...hasfour pre-existing failures unrelated to this change
(
internal/actuator,internal/controller,internal/controller/indexers,internal/engines/saturation,test/e2e) — all fail inBeforeSuiteonmissing envtest/kubebuilder binaries or an absent cluster, not on anything
this PR touches.