Skip to content

Commit 53fef44

Browse files
author
Yuto Terashima
committed
Expand README V2 results discussion
1 parent dc73db2 commit 53fef44

1 file changed

Lines changed: 57 additions & 8 deletions

File tree

README.md

Lines changed: 57 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -70,15 +70,64 @@ conda run -n Transformers python scripts/make_report.py
7070
Main report: `reports/benchmark_card_generation_report.md`.
7171

7272
<!-- V2_RESEARCH_UPGRADE -->
73-
## Publishable V2 Research Upgrade
73+
## Publishable V2 Research Results
7474

75-
This repository now includes a project-level V2 experiment suite:
75+
This repository now includes a full V2 research suite with real data, multiple baselines, ablations, result artifacts, figures, and failure analysis. The README summarizes the measured run so the project can be judged from results, not just project intent.
7676

77-
- Reproducible matrix: `configs/experiment_matrix.yaml`
78-
- Main runner: `scripts/run_matrix.py --device cuda --profile full`
79-
- Failure analysis: `scripts/analyze_failures.py`
80-
- Research report: `reports/open_model_benchmark_cards_v2_research_report.md`
81-
- Experiment index: `reports/results/experiment_index.json`
77+
### Dataset And Scale
8278

83-
The V2 artifacts include multiple experiments, ablations, figures, failure cases, and a discussion section while keeping raw caches and large checkpoints out of Git.
79+
Experiment indexes from the other 8 V2 repositories, converted into benchmark-card records with artifact and limitation checks.
8480

81+
- Full-profile result rows: `8`
82+
- Experiment profile: `full`
83+
- Experiment index: [`reports/results/experiment_index.json`](reports/results/experiment_index.json)
84+
- Full report: [`reports/open_model_benchmark_cards_v2_research_report.md`](reports/open_model_benchmark_cards_v2_research_report.md)
85+
86+
### Main Results
87+
88+
| repo | completeness_score | experiments | artifact_count |
89+
| --- | --- | --- | --- |
90+
| agent-safety-eval-lab | 1.0000 | 4.0000 | 6.0000 |
91+
| agent-trace-viewer | 1.0000 | 4.0000 | 6.0000 |
92+
| llm-eval-cookbook | 1.0000 | 5.0000 | 6.0000 |
93+
| mcp-tool-security-playground | 1.0000 | 4.0000 | 6.0000 |
94+
| multilingual-llm-safety-bench | 1.0000 | 4.0000 | 7.0000 |
95+
| prompt-robustness-suite | 1.0000 | 15.0000 | 5.0000 |
96+
| rag-eval-observatory | 1.0000 | 4.0000 | 5.0000 |
97+
| transformer-from-scratch-notes | 1.0000 | 4.0000 | 5.0000 |
98+
99+
### Analysis
100+
101+
- The generator produced benchmark cards for all 8 upstream repos and scored each card for experiment count, dataset path, artifacts, device metadata, and limitations.
102+
- Every upstream card currently reaches the schema completeness threshold, giving the portfolio a cross-repo reproducibility index.
103+
- The generated cards point back to committed reports and result artifacts, so project claims can be audited instead of trusted as prose.
104+
- This repo now closes the loop: it consumes the portfolio's actual experiment indexes and turns them into standardized research cards.
105+
106+
### Failure Analysis
107+
108+
The failure-analysis pass found `0` failure records.
109+
110+
The public failure artifacts use redacted previews or structured metadata where source examples may contain harmful, private, or otherwise sensitive text. This keeps the analysis reproducible without turning the README into a prompt-injection or unsafe-content corpus.
111+
112+
### Key Artifacts
113+
114+
- [`reports/results/v2_benchmark_cards.json`](reports/results/v2_benchmark_cards.json)
115+
- [`reports/results/v2_card_quality_scores.csv`](reports/results/v2_card_quality_scores.csv)
116+
- [`reports/figures/v2_card_artifact_counts.png`](reports/figures/v2_card_artifact_counts.png)
117+
- [`reports/figures/v2_card_completeness.png`](reports/figures/v2_card_completeness.png)
118+
- [`reports/figures/v2_card_experiment_counts.png`](reports/figures/v2_card_experiment_counts.png)
119+
120+
Figures:
121+
122+
- [`reports/figures/v2_card_artifact_counts.png`](reports/figures/v2_card_artifact_counts.png)
123+
- [`reports/figures/v2_card_completeness.png`](reports/figures/v2_card_completeness.png)
124+
- [`reports/figures/v2_card_experiment_counts.png`](reports/figures/v2_card_experiment_counts.png)
125+
126+
### Reproduction
127+
128+
```powershell
129+
conda run -n Transformers python scripts/run_matrix.py --device cuda --profile full
130+
conda run -n Transformers python scripts/analyze_failures.py
131+
conda run -n Transformers python scripts/make_report.py
132+
conda run -n Transformers python -m pytest
133+
```

0 commit comments

Comments
 (0)