test: add real P/D reference trace and topology feature A/B result - #2
Open
madhugoutham wants to merge 2 commits into
Open
test: add real P/D reference trace and topology feature A/B result#2madhugoutham wants to merge 2 commits into
madhugoutham wants to merge 2 commits into
Conversation
Adds the first real prefill/decode disaggregated reference trace to benchmarks/traces/ (the existing 3 traces are monolithic-only). 1,500 samples, stratified downsample of 16,186 real requests collected on a Kermit H200 cluster with topology-aware routing enabled. Runs the existing offline_feature_ab.py tool against it for topology_affinity_score. Result: NEUTRAL — 0.6% pinball delta on TTFT, not significant (within seed noise). Confirms, via the repo's own self-test methodology, the same conclusion reached separately with a custom leakage-free residual-correction analysis on the full trace: topology signal exists but is too small on this cluster's uniform InfiniBand fabric to improve p90 TTFT prediction. No changes to the live prediction schema or model. Evaluation/replay only. Signed-off-by: Madhu Goutham Reddy Ambati <mambati@redhat.com>
The checked-in A/B artifact was generated by an earlier, looser version of offline_feature_ab.py and does not reproduce on the current tool: topology is a null feature here (only zone/region tiers, near-identical transfer speed on uniform InfiniBand), so its delta straddles zero and the self-test cannot separate it from overfitting. Publishing a pass/fail artifact for a null feature is misleading, so drop the results and the derived topology_affinity_score column; keep the reference trace and record the neutral finding in the spec. The consuming path is per-class residual correction, not a raw feature (see predictor issue llm-d#30). Signed-off-by: Madhu Goutham Reddy Ambati <mambati@redhat.com>
|
This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
benchmarks/traces/(the existing 3 traces are monolithic-only, confirmed by checkingpod_typeacross all of them)offline_feature_ab.pytool against it fortopology_affinity_scoreResult
NEUTRAL — 0.6% pinball delta on TTFT, not significant (within seed noise). This confirms, via the repo's own self-test methodology, the same conclusion reached separately with a custom leakage-free residual-correction analysis on the full (non-downsampled) trace: topology signal exists but is too small on this cluster's uniform InfiniBand fabric to improve p90 TTFT prediction.
Note: running the tool against the full 16,186-row trace (not checked in here, exceeds the 1MB trace limit) returns
SELF-TEST FAILED: Real improvement (1.17) is less than 2x the shuffled noise floor (3.31)- an even stronger negative result. The downsampled version checked in here passes the self-test but still shows no significant improvement. Both point the same direction.What this is NOT
Known limitation of the trace itself
topology_distance: "host"has 0 samples in the source data, despite the router's topology-affinity-scorer always selecting a same-host prefill candidate when available - an unresolved observability discrepancy on the router side, documented in the companion spec file and tracked separately in the router PR.Test plan
offline_feature_ab.py --trace benchmarks/traces/kermit-pd-topology-h200.jsonl --feature topology_affinity_score --seeds 10runs clean, self-test PASSED, verdict NEUTRALScope note
Sandboxed on my fork, branched off 60 since it depends on
offline_feature_ab.py. Not proposing this upstream yet - this documents a negative/neutral result for predictor issue 30, to be posted there once finalized.