Add stability_report: a one-call summary of a pipeline's stability

.gitignore

@@ -16,7 +16,7 @@ env
 **.idea
 *.pth
 **Icon*
-!docs/favicon.png
+!docs/favicon.*
 venv
 .vscode/**
 # Compiled python modules.

README.md

@@ -73,7 +73,29 @@ binary_metrics_set = Vset(
 binary_metrics = binary_metrics_set.evaluate(preds_test, y_test)
 ```
 
-Once we've written this pipeline, we can easily measure the stability of metrics (e.g. "Accuracy") to our choice of subsampling or model.
+Once we've written this pipeline, one call reports how stable the metrics
+(e.g. "Accuracy") are to our choice of subsampling or model:
+
+```python
+from vflow import stability_report
+
+report = stability_report(binary_metrics, compare="modeling")
+print(report)  # or report.save("stability.html"), or show it in a notebook
+```
+
+which starts, for one run of the random data above, with:
+
+```
+- `binary_metrics` = Acc: 6 results, mean 0.86 (from 0.8 to 0.92).
+- `binary_metrics` = Acc: most of the variation comes from `modeling` (76% of the variance).
+- `binary_metrics` = Acc: `DT` is best in 3 of 3 perturbations (100%); next is `LR` (0%).
+...
+```
+
+The report gives each metric separately: a summary, the share of the
+variance explained by each Vset, how often each model is best across the
+other perturbations, and the results by each Vset's choices. Its tables are
+DataFrames (`report.variance`, `report.ranking`, ...) for further analysis.
 
 # Documentation
 

docs/agents.html

@@ -15,7 +15,8 @@
 <style media="print">@media print{#sidebar h1{page-break-before:always}.source{display:none}}@media print{*{background:transparent !important;color:#000 !important;box-shadow:none !important;text-shadow:none !important}a[href]:after{content:" (" attr(href) ")";font-size:90%}a[href][title]:after{content:none}abbr[title]:after{content:" (" attr(title) ")"}.ir a:after,a[href^="javascript:"]:after,a[href^="#"]:after{content:""}pre,blockquote{border:1px solid #999;page-break-inside:avoid}thead{display:table-header-group}tr,img{page-break-inside:avoid}img{max-width:100% !important}@page{margin:0.5cm}p,h2,h3{orphans:3;widows:3}h1,h2,h3,h4,h5,h6{page-break-after:avoid}}</style>
 <link rel="stylesheet" href="github.css">
 <link rel="stylesheet" href="style.css">
-<link rel="icon" type="image/png" href="favicon.png">
+<link rel="icon" type="image/png" sizes="64x64" href="favicon.png">
+<link rel="icon" type="image/svg+xml" sizes="any" href="favicon.svg">
 <script defer src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.1.1/highlight.min.js" integrity="sha256-Uv3H6lx7dJmRfRvH8TH6kJD1TSK1aFcwgx+mdg3epi8=" crossorigin></script>
 <script>window.addEventListener('DOMContentLoaded', () => hljs.initHighlighting())</script>
 </head>
@@ -32,7 +33,7 @@
 </a>
 <p id="site-tagline"><b>vflow</b> &middot; Stability analysis for data-science pipelines, with PCS.</p>
 <a href="index.html" id="site-logo">
-<img src="vflow_logo.svg" alt="vflow">
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 340 124" role="img" aria-label="vflow"><title>vflow</title><g transform="translate(14,10)"><path d="M6,40 L31,100 L56,40 M84,100 V30 Q84,4 110,4 M68,46 H104 M132,2 V82 Q132,100 150,100 M236,40 L254,100 L271,52 L288,100 L306,40" fill="none" stroke="#1F2A33" stroke-width="13" stroke-linecap="round" stroke-linejoin="round"/><g class="vo"><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f85e7" transform="translate(190.0,40.0) rotate(-90.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2d9cc6" transform="translate(211.2,48.8) rotate(-45.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb3a6" transform="translate(220.0,70.0) rotate(0.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb8a0" transform="translate(211.2,91.2) rotate(45.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb3a6" transform="translate(190.0,100.0) rotate(90.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2d9cc6" transform="translate(168.8,91.2) rotate(135.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f85e7" transform="translate(160.0,70.0) rotate(180.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f80ed" transform="translate(168.8,48.8) rotate(225.0)"/></g></g></svg>
 </a>
 <a href="https://github.com/Yu-Group/veridical-flow" class="nav-icon"
 aria-label="View source on GitHub" title="GitHub">
@@ -355,6 +356,7 @@ <h2 id="limits">7 &nbsp;Notes and limits</h2>
 <ul>
 <li><code><a title="vflow.helpers" href="helpers.html">vflow.helpers</a></code></li>
 <li><code><a title="vflow.pipeline" href="pipeline.html">vflow.pipeline</a></code></li>
+<li><code><a title="vflow.report" href="report.html">vflow.report</a></code></li>
 <li><code><a title="vflow.subkey" href="subkey.html">vflow.subkey</a></code></li>
 <li><code><a title="vflow.utils" href="utils.html">vflow.utils</a></code></li>
 <li><code><a title="vflow.vfunc" href="vfunc.html">vflow.vfunc</a></code></li>
@@ -378,6 +380,7 @@ <h2 id="limits">7 &nbsp;Notes and limits</h2>
 <li><code><a title="vflow.init_step" href="utils.html#vflow.utils.init_step">init_step</a></code></li>
 <li><code><a title="vflow.perturbation_stats" href="utils.html#vflow.utils.perturbation_stats">perturbation_stats</a></code></li>
 <li><code><a title="vflow.sep_dicts" href="utils.html#vflow.utils.sep_dicts">sep_dicts</a></code></li>
+<li><code><a title="vflow.stability_report" href="report.html#vflow.report.stability_report">stability_report</a></code></li>
 <li><code><a title="vflow.to_list" href="utils.html#vflow.utils.to_list">to_list</a></code></li>
 <li><code><a title="vflow.to_tuple" href="utils.html#vflow.utils.to_tuple">to_tuple</a></code></li>
 </ul>
@@ -399,6 +402,23 @@ <h4><code><a title="vflow.PCSPipeline" href="pipeline.html#vflow.pipeline.PCSPip
 </ul>
 </li>
 <li>
+<h4><code><a title="vflow.StabilityReport" href="report.html#vflow.report.StabilityReport">StabilityReport</a></code></h4>
+<ul class="two-column">
+<li><code><a title="vflow.StabilityReport.by" href="report.html#vflow.report.StabilityReport.by">by</a></code></li>
+<li><code><a title="vflow.StabilityReport.compare" href="report.html#vflow.report.StabilityReport.compare">compare</a></code></li>
+<li><code><a title="vflow.StabilityReport.headlines" href="report.html#vflow.report.StabilityReport.headlines">headlines</a></code></li>
+<li><code><a title="vflow.StabilityReport.levels" href="report.html#vflow.report.StabilityReport.levels">levels</a></code></li>
+<li><code><a title="vflow.StabilityReport.ranking" href="report.html#vflow.report.StabilityReport.ranking">ranking</a></code></li>
+<li><code><a title="vflow.StabilityReport.results" href="report.html#vflow.report.StabilityReport.results">results</a></code></li>
+<li><code><a title="vflow.StabilityReport.save" href="report.html#vflow.report.StabilityReport.save">save</a></code></li>
+<li><code><a title="vflow.StabilityReport.summary" href="report.html#vflow.report.StabilityReport.summary">summary</a></code></li>
+<li><code><a title="vflow.StabilityReport.to_html" href="report.html#vflow.report.StabilityReport.to_html">to_html</a></code></li>
+<li><code><a title="vflow.StabilityReport.to_markdown" href="report.html#vflow.report.StabilityReport.to_markdown">to_markdown</a></code></li>
+<li><code><a title="vflow.StabilityReport.variance" href="report.html#vflow.report.StabilityReport.variance">variance</a></code></li>
+<li><code><a title="vflow.StabilityReport.wrt" href="report.html#vflow.report.StabilityReport.wrt">wrt</a></code></li>
+</ul>
+</li>
+<li>
 <h4><code><a title="vflow.Subkey" href="subkey.html#vflow.subkey.Subkey">Subkey</a></code></h4>
 <ul class="">
 <li><code><a title="vflow.Subkey.is_matching" href="subkey.html#vflow.subkey.Subkey.is_matching">is_matching</a></code></li>

docs/build_pages.py

@@ -51,6 +51,8 @@ def shared_sidebar(index_html):
 def main():
     index_html = _read("index.html")
     header = shared_block(index_html, '<header id="site-header">', "</header>")
+    # the logo animation is for the home page only
+    header = header.replace('id="site-logo" class="animate"', 'id="site-logo"')
     sidebar = shared_sidebar(index_html)
     footer = shared_block(index_html, '<footer id="footer">', "</footer>")
     for name, (title, extra) in PAGES.items():

docs/helpers.html

@@ -14,7 +14,8 @@
 <style media="print">@media print{#sidebar h1{page-break-before:always}.source{display:none}}@media print{*{background:transparent !important;color:#000 !important;box-shadow:none !important;text-shadow:none !important}a[href]:after{content:" (" attr(href) ")";font-size:90%}a[href][title]:after{content:none}abbr[title]:after{content:" (" attr(title) ")"}.ir a:after,a[href^="javascript:"]:after,a[href^="#"]:after{content:""}pre,blockquote{border:1px solid #999;page-break-inside:avoid}thead{display:table-header-group}tr,img{page-break-inside:avoid}img{max-width:100% !important}@page{margin:0.5cm}p,h2,h3{orphans:3;widows:3}h1,h2,h3,h4,h5,h6{page-break-after:avoid}}</style>
 <link rel="stylesheet" href="github.css">
 <link rel="stylesheet" href="style.css">
-<link rel="icon" type="image/png" href="favicon.png">
+<link rel="icon" type="image/png" sizes="64x64" href="favicon.png">
+<link rel="icon" type="image/svg+xml" sizes="any" href="favicon.svg">
 <script defer src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.1.1/highlight.min.js" integrity="sha256-Uv3H6lx7dJmRfRvH8TH6kJD1TSK1aFcwgx+mdg3epi8=" crossorigin></script>
 <script>window.addEventListener('DOMContentLoaded', () => hljs.initHighlighting())</script>
 </head>
@@ -30,7 +31,7 @@
 </a>
 <p id="site-tagline"><b>vflow</b> &middot; Stability analysis for data-science pipelines, with PCS.</p>
 <a href="index.html" id="site-logo">
-<img src="vflow_logo.svg" alt="vflow">
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 340 124" role="img" aria-label="vflow"><title>vflow</title><g transform="translate(14,10)"><path d="M6,40 L31,100 L56,40 M84,100 V30 Q84,4 110,4 M68,46 H104 M132,2 V82 Q132,100 150,100 M236,40 L254,100 L271,52 L288,100 L306,40" fill="none" stroke="#1F2A33" stroke-width="13" stroke-linecap="round" stroke-linejoin="round"/><g class="vo"><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f85e7" transform="translate(190.0,40.0) rotate(-90.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2d9cc6" transform="translate(211.2,48.8) rotate(-45.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb3a6" transform="translate(220.0,70.0) rotate(0.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb8a0" transform="translate(211.2,91.2) rotate(45.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb3a6" transform="translate(190.0,100.0) rotate(90.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2d9cc6" transform="translate(168.8,91.2) rotate(135.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f85e7" transform="translate(160.0,70.0) rotate(180.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f80ed" transform="translate(168.8,48.8) rotate(225.0)"/></g></g></svg>
 </a>
 <a href="https://github.com/Yu-Group/veridical-flow" class="nav-icon"
 aria-label="View source on GitHub" title="GitHub">

docs/html.mako

@@ -402,7 +402,8 @@
   <style media="print">${css.print()}</style>
   <link rel="stylesheet" href="github.css">
   <link rel="stylesheet" href="style.css">
-  <link rel="icon" type="image/png" href="favicon.png">
+  <link rel="icon" type="image/png" sizes="64x64" href="favicon.png">
+  <link rel="icon" type="image/svg+xml" sizes="any" href="favicon.svg">
 
   % if google_analytics:
     <script>
@@ -443,7 +444,7 @@
   </a>
   <p id="site-tagline"><b>vflow</b> &middot; Stability analysis for data-science pipelines, with PCS.</p>
   <a href="index.html" id="site-logo">
-    <img src="vflow_logo.svg" alt="vflow">
+    <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 340 124" role="img" aria-label="vflow"><title>vflow</title><g transform="translate(14,10)"><path d="M6,40 L31,100 L56,40 M84,100 V30 Q84,4 110,4 M68,46 H104 M132,2 V82 Q132,100 150,100 M236,40 L254,100 L271,52 L288,100 L306,40" fill="none" stroke="#1F2A33" stroke-width="13" stroke-linecap="round" stroke-linejoin="round"/><g class="vo"><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f85e7" transform="translate(190.0,40.0) rotate(-90.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2d9cc6" transform="translate(211.2,48.8) rotate(-45.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb3a6" transform="translate(220.0,70.0) rotate(0.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb8a0" transform="translate(211.2,91.2) rotate(45.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2bb3a6" transform="translate(190.0,100.0) rotate(90.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2d9cc6" transform="translate(168.8,91.2) rotate(135.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f85e7" transform="translate(160.0,70.0) rotate(180.0)"/><path d="M0,-12.540000000000001 C3.1350000000000002,-7.695000000000001 5.7,-3.9899999999999998 5.7,0 A5.7,5.7 0 1 1 -5.7,0 C-5.7,-3.9899999999999998 -3.1350000000000002,-7.695000000000001 0,-12.540000000000001 Z" fill="#2f80ed" transform="translate(168.8,48.8) rotate(225.0)"/></g></g></svg>
   </a>
   <a href="https://github.com/Yu-Group/veridical-flow" class="nav-icon"
      aria-label="View source on GitHub" title="GitHub">