Skip to content

test: add real P/D reference trace and topology feature A/B result - #2

Open
madhugoutham wants to merge 2 commits into
feat/offline-ab-benchmark-templatefrom
feat/kermit-pd-topology-trace
Open

test: add real P/D reference trace and topology feature A/B result#2
madhugoutham wants to merge 2 commits into
feat/offline-ab-benchmark-templatefrom
feat/kermit-pd-topology-trace

Conversation

@madhugoutham

@madhugoutham madhugoutham commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds the first real prefill/decode disaggregated reference trace to benchmarks/traces/ (the existing 3 traces are monolithic-only, confirmed by checking pod_type across all of them)
  • 1,500 samples, stratified downsample of 16,186 real requests collected on a Kermit H200 cluster (CoreWeave CKS) with topology-aware routing enabled
  • Runs the existing offline_feature_ab.py tool against it for topology_affinity_score

Result

NEUTRAL — 0.6% pinball delta on TTFT, not significant (within seed noise). This confirms, via the repo's own self-test methodology, the same conclusion reached separately with a custom leakage-free residual-correction analysis on the full (non-downsampled) trace: topology signal exists but is too small on this cluster's uniform InfiniBand fabric to improve p90 TTFT prediction.

Note: running the tool against the full 16,186-row trace (not checked in here, exceeds the 1MB trace limit) returns SELF-TEST FAILED: Real improvement (1.17) is less than 2x the shuffled noise floor (3.31) - an even stronger negative result. The downsampled version checked in here passes the self-test but still shows no significant improvement. Both point the same direction.

What this is NOT

  • No changes to the live prediction schema or model - evaluation/replay only
  • Not a claim that topology-aware prediction doesn't work anywhere - Kermit is single-region, uniform InfiniBand fabric; this result does not generalize to heterogeneous/cross-region clusters (untested)

Known limitation of the trace itself

topology_distance: "host" has 0 samples in the source data, despite the router's topology-affinity-scorer always selecting a same-host prefill candidate when available - an unresolved observability discrepancy on the router side, documented in the companion spec file and tracked separately in the router PR.

Test plan

  • offline_feature_ab.py --trace benchmarks/traces/kermit-pd-topology-h200.jsonl --feature topology_affinity_score --seeds 10 runs clean, self-test PASSED, verdict NEUTRAL
  • Trace file under the 1MB pre-commit limit (928K)
  • README updated with the new trace's row in the reference table

Scope note

Sandboxed on my fork, branched off 60 since it depends on offline_feature_ab.py. Not proposing this upstream yet - this documents a negative/neutral result for predictor issue 30, to be posted there once finalized.

Adds the first real prefill/decode disaggregated reference trace to
benchmarks/traces/ (the existing 3 traces are monolithic-only). 1,500
samples, stratified downsample of 16,186 real requests collected on a
Kermit H200 cluster with topology-aware routing enabled.

Runs the existing offline_feature_ab.py tool against it for
topology_affinity_score. Result: NEUTRAL — 0.6% pinball delta on TTFT,
not significant (within seed noise). Confirms, via the repo's own
self-test methodology, the same conclusion reached separately with a
custom leakage-free residual-correction analysis on the full trace:
topology signal exists but is too small on this cluster's uniform
InfiniBand fabric to improve p90 TTFT prediction.

No changes to the live prediction schema or model. Evaluation/replay
only.

Signed-off-by: Madhu Goutham Reddy Ambati <mambati@redhat.com>
The checked-in A/B artifact was generated by an earlier, looser version
of offline_feature_ab.py and does not reproduce on the current tool:
topology is a null feature here (only zone/region tiers, near-identical
transfer speed on uniform InfiniBand), so its delta straddles zero and
the self-test cannot separate it from overfitting. Publishing a pass/fail
artifact for a null feature is misleading, so drop the results and the
derived topology_affinity_score column; keep the reference trace and
record the neutral finding in the spec.

The consuming path is per-class residual correction, not a raw feature
(see predictor issue llm-d#30).

Signed-off-by: Madhu Goutham Reddy Ambati <mambati@redhat.com>
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant