From 4b3eb31fc0c6edba277f902e97bfcac597381ab0 Mon Sep 17 00:00:00 2001 From: Chandan Singh Date: Thu, 15 Oct 2026 17:24:31 -0400 Subject: [PATCH 1/2] Add stability_report: a one-call summary of a pipeline's stability stability_report(metrics, compare="modeling") takes the output dict of a metrics Vset (or a dict_to_df DataFrame) and reports, per metric: - a summary of the results (count, mean, std, min, max) - the share of the variance explained by each perturbed Vset on its own, with the rest attributed to interactions - with compare=, how often each choice is best across the combinations of the other perturbations (win rate, mean rank) - the results by each Vset's choices - plain-language headlines The StabilityReport prints as markdown, renders in notebooks, saves as HTML or markdown, and keeps its tables as DataFrames. The README example shows it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Xy8Dg7fAqocHGzMqVCiSK --- README.md | 24 +- docs/agents.html | 19 + docs/index.html | 37 +- docs/report.html | 886 ++++++++++++++++++++++++++++++++++++++ docs/vflow-internals.html | 19 + tests/test_report.py | 127 ++++++ vflow/__init__.py | 4 + vflow/report.py | 389 +++++++++++++++++ 8 files changed, 1503 insertions(+), 2 deletions(-) create mode 100644 docs/report.html create mode 100644 tests/test_report.py create mode 100644 vflow/report.py diff --git a/README.md b/README.md index c0ea07e..4e650a0 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,29 @@ binary_metrics_set = Vset( binary_metrics = binary_metrics_set.evaluate(preds_test, y_test) ``` -Once we've written this pipeline, we can easily measure the stability of metrics (e.g. "Accuracy") to our choice of subsampling or model. +Once we've written this pipeline, one call reports how stable the metrics +(e.g. "Accuracy") are to our choice of subsampling or model: + +```python +from vflow import stability_report + +report = stability_report(binary_metrics, compare="modeling") +print(report) # or report.save("stability.html"), or show it in a notebook +``` + +which starts, for one run of the random data above, with: + +``` +- `binary_metrics` = Acc: 6 results, mean 0.86 (from 0.8 to 0.92). +- `binary_metrics` = Acc: most of the variation comes from `modeling` (76% of the variance). +- `binary_metrics` = Acc: `DT` is best in 3 of 3 perturbations (100%); next is `LR` (0%). +... +``` + +The report gives each metric separately: a summary, the share of the +variance explained by each Vset, how often each model is best across the +other perturbations, and the results by each Vset's choices. Its tables are +DataFrames (`report.variance`, `report.ranking`, ...) for further analysis. # Documentation diff --git a/docs/agents.html b/docs/agents.html index d010819..c087e13 100644 --- a/docs/agents.html +++ b/docs/agents.html @@ -355,6 +355,7 @@

7  Notes and limits

@@ -399,6 +401,23 @@

StabilityReport

+ + +

  • Subkey