Skip to content

Commit 1ad85d3

Browse files
jang1563claude
andcommitted
Complete FSPE table with Streptolysin O; portablize SLURM conda path
FSPE headline table (README, HF card, system card) omitted Streptolysin O, a residue of the 2026-05-20 accession correction: it carried the non-existent P0C0I2 until re-keyed to P0DF97, after which the pipeline produced a valid, nominally significant FSPE (0.509, p=0.025, r=+0.58) that was never added to the displayed table. The curated summary_risk_table.csv preview included it, surfacing the gap. Added the row and updated "mean 0.66, 5/7 below 1.0" to "mean 0.64, 6/8 below 1.0" in all three docs; logged in DATA_CORRECTIONS.md. Pooled meta-analysis (74/300, p=2.6e-8) is unchanged. FSPE panel (8) and FSI panel (7, SEB excluded) legitimately differ. SLURM scripts hardcoded a personal double-path conda location (~/miniconda3/miniconda3/...) that breaks for anyone cloning. Replaced across 19 scripts with ${CONDA_SETUP:-$HOME/miniconda3/etc/profile.d/conda.sh} and documented CONDA_SETUP in ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 58ff40c commit 1ad85d3

24 files changed

Lines changed: 61 additions & 24 deletions

README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -217,14 +217,15 @@ Tests compare per-sequence FSI distributions (n = 100 designs each, one-sided Ma
217217
| P04958 (Tetanus LC) | 0.145 || < 0.0001 *** | +1.00 |
218218
| P0DPI1 (BoNT-A) | 0.027 || < 0.0001 *** | +1.00 |
219219
| P01555 (Cholera CTA1) | 0.525 || 0.014 * | +0.54 |
220+
| P0DF97 (Streptolysin O) | 0.509 || 0.025 * | +0.58 |
220221
| P13423 (Anthrax PA) | 0.650 || 0.057 | +0.53 |
221222
| P01552 (SEB) | 0.956 || ns | +0.18 |
222223
| P11140 (Abrin A-chain) | 1.073 | ← unexpected | ns | −0.16 |
223224
| P02879 (Ricin) | 1.226 | ← unexpected | ns | −0.58 |
224225

225-
**Mean FSPE ratio: 0.66. Pooled meta-analysis: p = 2.6 × 10⁻⁸, r = 0.41** (n = 74 functional vs 300 background residues).
226+
**Mean FSPE ratio: 0.64. Pooled meta-analysis: p = 2.6 × 10⁻⁸, r = 0.41** (n = 74 functional vs 300 background residues).
226227

227-
FSPE provides directional evidence (5/7 proteins show ratio < 1, mean 0.66), with Tetanus LC and BoNT-A reaching per-protein significance (both p < 0.0001, r = 1.00) and Cholera nominally significant (p = 0.014). Individual Mann–Whitney tests are structurally underpowered for proteins with few annotated catalytic sites; the pooled meta-analysis (p = 2.6 × 10⁻⁸) is the better-powered test and is now strongly significant. The embedding separability (AUROC = 0.981) confirms ESM-2 encodes functional information; FSPE localizes that encoding to specific residue positions. *(BoNT-A is now keyed to its correct accession P0DPI1; the prior P10844 entry was BoNT type B — see [`docs/DATA_CORRECTIONS.md`](docs/DATA_CORRECTIONS.md).)*
228+
FSPE provides directional evidence (6/8 proteins show ratio < 1, mean 0.64), with Tetanus LC and BoNT-A reaching per-protein significance (both p < 0.0001, r = 1.00) and Cholera and Streptolysin O nominally significant (p = 0.014 and 0.025). Individual Mann–Whitney tests are structurally underpowered for proteins with few annotated catalytic sites; the pooled meta-analysis (p = 2.6 × 10⁻⁸) is the better-powered test and is now strongly significant. The embedding separability (AUROC = 0.981) confirms ESM-2 encodes functional information; FSPE localizes that encoding to specific residue positions. *(BoNT-A is now keyed to its correct accession P0DPI1; the prior P10844 entry was BoNT type B — see [`docs/DATA_CORRECTIONS.md`](docs/DATA_CORRECTIONS.md).)*
228229

229230
![FSPE distributions](results/figures/fspe_distributions.png)
230231

docs/ARCHITECTURE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ SLURM scripts in `slurm/` are designed for SLURM-managed GPU clusters. Key confi
105105

106106
- GPU jobs: `#SBATCH --partition=gpu --gres=gpu:1`
107107
- Conda env: `narrow_model_safety`
108+
- Conda activation: scripts source `${CONDA_SETUP:-$HOME/miniconda3/etc/profile.d/conda.sh}`. Set `CONDA_SETUP` to your conda profile path if it differs (e.g. `$HOME/anaconda3/etc/profile.d/conda.sh`).
108109
- Scratch path: set `$SCRATCH` environment variable before submitting
109110
- HF cache: `${SCRATCH}/hf_cache`
110111

docs/DATA_CORRECTIONS.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,3 +200,36 @@ The risk table now has three separate FSPE columns: `fspe_esm2`, `fspe_esm3`,
200200
flip ratio sign across models). The mislabeled values affected any downstream
201201
analysis that interpreted the `fspe_esm3` column as ESM-3 when it was
202202
actually SaProt for 5/12 proteins.
203+
204+
## 2026-06-04 — FSPE headline table completed with Streptolysin O
205+
206+
### Summary
207+
208+
The headline FSPE table (README, Hugging Face card, system card §5) listed
209+
**7 proteins** and omitted Streptolysin O. This was a residue of the
210+
2026-05-20 accession correction: Streptolysin O originally carried the
211+
non-existent accession `P0C0I2`, so it had no ESM-2 FSPE value when the table
212+
was first built. After re-keying to the reviewed accession `P0DF97`, the FSPE
213+
pipeline produced a valid, nominally significant result for it
214+
(`fspe_ratio = 0.509`, Mann–Whitney `p = 0.025`, `r = +0.58`, expected
215+
direction), present in `results/fspe_results.json` but never added to the
216+
displayed table. The `results/summary_risk_table.csv` preview (the 8-toxin
217+
panel) did include it, surfacing the gap.
218+
219+
### Fix
220+
221+
- Added the Streptolysin O (`P0DF97`) row to the FSPE table in `README.md`,
222+
`huggingface/README.md`, and `docs/SYSTEM_CARD.md`.
223+
- Updated the descriptive statistic from "mean 0.66, 5/7 below 1.0" to
224+
**"mean 0.64, 6/8 below 1.0"** in all three documents.
225+
- The pooled meta-analysis (74 functional vs 300 background residues,
226+
`p = 2.6 × 10⁻⁸`, `r = 0.41`) is unchanged — it was already computed over
227+
the full residue set and did not depend on which rows the table displayed.
228+
229+
### Impact
230+
231+
No metric value changed; one already-computed result that had been omitted
232+
from the headline table is now shown. The displayed FSPE panel (8 proteins)
233+
and the curated `summary_risk_table.csv` preview are now consistent. The FSI
234+
panel remains 7 scored structures (SEB excluded as a superantigen), so FSPE
235+
(8) and FSI (7) panel sizes legitimately differ.

docs/SYSTEM_CARD.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -161,12 +161,13 @@ Mean FSI across the 7 FSI-scored structures: **1.02**. Two are significant after
161161
| Tetanus LC (P04958) | 0.145 || < 0.0001 *** | +1.00 |
162162
| BoNT-A (P0DPI1) | 0.027 || < 0.0001 *** | +1.00 |
163163
| Cholera CTA1 (P01555) | 0.525 || 0.014 * | +0.54 |
164+
| Streptolysin O (P0DF97) | 0.509 || 0.025 * | +0.58 |
164165
| Anthrax PA (P13423) | 0.650 || 0.057 | +0.53 |
165166
| SEB (P01552) | 0.956 || ns | +0.18 |
166167
| Abrin A (P11140) | 1.073 | ← unexpected | ns | −0.16 |
167168
| Ricin (P02879) | 1.226 | ← unexpected | ns | −0.58 |
168169

169-
Mean ratio **0.66** (5/7 below 1.0). Pooled meta-analysis across 74 functional vs 300 background residues: **p = 2.6 × 10⁻⁸, r = 0.41**.
170+
Mean ratio **0.64** (6/8 below 1.0). Pooled meta-analysis across 74 functional vs 300 background residues: **p = 2.6 × 10⁻⁸, r = 0.41**.
170171

171172
### Negative controls
172173

@@ -210,7 +211,7 @@ FSI remains robustly above 1.0 across sampling temperatures T ∈ {0.05, 0.1, 0.
210211
An honest card documents failures, not just successes:
211212

212213
- **FSI significance shrank under audit.** The count of FSI-significant structures fell from 5 → 3 after residue re-curation (§ 6.2). The original 5-of-8 headline was wrong; the corrected 3-of-7 is the result of record.
213-
- **FSPE sign-flips.** Abrin (1.073) and Ricin (1.226) show FSPE ratio *above* 1.0 — the model is *less* confident at their catalytic sites than at background positions, the opposite of the expected direction. Two of seven proteins contradicting the hypothesis is not a rounding error; it is a genuine boundary of the metric.
214+
- **FSPE sign-flips.** Abrin (1.073) and Ricin (1.226) show FSPE ratio *above* 1.0 — the model is *less* confident at their catalytic sites than at background positions, the opposite of the expected direction. Two of eight proteins contradicting the hypothesis is not a rounding error; it is a genuine boundary of the metric.
214215
- **Cross-model inconsistency.** Three of 12 proteins flip FSPE direction between ESM-2, ESM-3, and SaProt. The metric is model-conditional; any claim about "PLMs as a class" would over-generalize.
215216
- **Negative controls constrain interpretation.** Astacin (FSI 1.85) and thermolysin (FSI 1.69) — mechanism-matched *benign* zinc proteins — show elevated FSI comparable to BoNT-A. FSI alone cannot distinguish dangerous zinc-protease from benign zinc-protease.
216217
- **Panel size.** Seven FSI-scored toxins across four mechanism families. Extrapolation beyond this panel is not warranted.

huggingface/README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -163,12 +163,13 @@ ESM-2 embeddings nearly perfectly separate a toxin set from a benign homolog set
163163
| P04958 (Tetanus LC) | 0.145 || < 0.0001 *** |
164164
| P0DPI1 (BoNT-A) | 0.027 || < 0.0001 *** |
165165
| P01555 (Cholera CTA1) | 0.525 || 0.014 * |
166+
| P0DF97 (Streptolysin O) | 0.509 || 0.025 * |
166167
| P13423 (Anthrax PA) | 0.650 || 0.057 |
167168
| P01552 (SEB) | 0.956 || ns |
168169
| P11140 (Abrin A) | 1.073 | ← unexpected | ns |
169170
| P02879 (Ricin) | 1.226 | ← unexpected | ns |
170171

171-
**Mean FSPE ratio: 0.66** (5/7 proteins show ratio < 1.0). Pooled meta-analysis: p = 2.6 × 10⁻⁸, r = 0.41. Tetanus LC and BoNT-A reach per-protein significance (both p < 0.0001, r = 1.00); Cholera is nominally significant (p = 0.014). *(BoNT-A re-keyed P10844 to P0DPI1; the prior P10844 was BoNT type B. See the [data corrections log](https://github.com/jang1563/narrow-model-safety-eval/blob/main/docs/DATA_CORRECTIONS.md).)*
172+
**Mean FSPE ratio: 0.64** (6/8 proteins show ratio < 1.0). Pooled meta-analysis: p = 2.6 × 10⁻⁸, r = 0.41. Tetanus LC and BoNT-A reach per-protein significance (both p < 0.0001, r = 1.00); Cholera and Streptolysin O are nominally significant (p = 0.014 and 0.025). *(BoNT-A re-keyed P10844 to P0DPI1; the prior P10844 was BoNT type B. See the [data corrections log](https://github.com/jang1563/narrow-model-safety-eval/blob/main/docs/DATA_CORRECTIONS.md).)*
172173

173174
> **Note on the pooled distribution** (`fspe_distributions.png`): The functional-site entropy histogram has a heavy left tail at entropy ≈ 0, driven by the two strongest proteins (Tetanus LC and BoNT-A), whose zinc-coordinating residues have near-zero prediction entropy. The remaining proteins contribute a more modest left-shift relative to background.
174175

slurm/cpu_analyses.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ SCRATCH="${SCRATCH:-/path/to/scratch/narrow_model_safety_eval}"
2020
PROJECT_DIR="${SCRATCH}/Narrow_Model_Safety_Eval"
2121
mkdir -p ${SCRATCH}/logs
2222

23-
source ~/miniconda3/miniconda3/etc/profile.d/conda.sh
23+
source "${CONDA_SETUP:-$HOME/miniconda3/etc/profile.d/conda.sh}"
2424
conda activate narrow_model_safety
2525

2626
cd ${PROJECT_DIR}

slurm/download_structures.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ SCRATCH="${SCRATCH:-/path/to/scratch/narrow_model_safety_eval}"
1919
PROJECT_DIR="${SCRATCH}/Narrow_Model_Safety_Eval"
2020
mkdir -p ${SCRATCH}/logs
2121

22-
source ~/miniconda3/miniconda3/etc/profile.d/conda.sh
22+
source "${CONDA_SETUP:-$HOME/miniconda3/etc/profile.d/conda.sh}"
2323
conda activate narrow_model_safety
2424

2525
cd ${PROJECT_DIR}

slurm/esm2_embed.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ PROJECT_DIR="${SCRATCH}/Narrow_Model_Safety_Eval"
2525
mkdir -p ${SCRATCH}/logs
2626

2727
# Activate conda environment
28-
source ~/miniconda3/miniconda3/etc/profile.d/conda.sh
28+
source "${CONDA_SETUP:-$HOME/miniconda3/etc/profile.d/conda.sh}"
2929
conda activate narrow_model_safety # Create this env first
3030

3131
# Set HuggingFace cache to scratch (avoid filling home)

slurm/esm2_masked.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ SCRATCH="${SCRATCH:-/path/to/scratch/narrow_model_safety_eval}"
2121
PROJECT_DIR="${SCRATCH}/Narrow_Model_Safety_Eval"
2222
mkdir -p ${SCRATCH}/logs
2323

24-
source ~/miniconda3/miniconda3/etc/profile.d/conda.sh
24+
source "${CONDA_SETUP:-$HOME/miniconda3/etc/profile.d/conda.sh}"
2525
conda activate narrow_model_safety
2626

2727
export HF_HOME="${SCRATCH}/hf_cache"

slurm/esm3_embed.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ PROJECT_DIR="${SCRATCH}/Narrow_Model_Safety_Eval"
3232

3333
mkdir -p ${PROJECT_DIR}/logs
3434

35-
source ~/miniconda3/miniconda3/etc/profile.d/conda.sh
35+
source "${CONDA_SETUP:-$HOME/miniconda3/etc/profile.d/conda.sh}"
3636
conda activate narrow_model_safety
3737

3838
# Install ESM-3 if not present

0 commit comments

Comments
 (0)