Skip to content

Commit d8098d7

Browse files
jang1563claude
andcommitted
docs(huggingface): add elicitation coverage, external validation, SafeProtein positioning
Mirror the premium-level enhancements made to the GitHub Evaluation Report into the Hugging Face dataset card, adapted for the lighter HF card surface: - §Elicitation Coverage (under Key Results): tested vs untested elicitation axes; reported FSI/FSPE values are conservative under the tested surface. - §External Validation Status: two-priority validation ask (independent re-curation + cross-institution replication); inputs are reproducible from this dataset alone, no access-gated artifact required. - §Related Work: positions this dataset's representation-level signal as complementary to SafeProtein (Fan et al. 2025) generation-level red-teaming. Risk-Tier Mapping was intentionally NOT mirrored — it is a deployment-context artifact that belongs in the Evaluation Report, not a dataset card. 10/10 release-integrity tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 6eefe19 commit d8098d7

1 file changed

Lines changed: 24 additions & 0 deletions

File tree

huggingface/README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,10 @@ High-FSI sequences are **not** more backbone-compatible than low-FSI sequences (
205205
| `data/annotations/functional_sites.json` | Catalytic residue annotations with DOI citations |
206206
| `data/annotations/physical_realizability.json` | 5-dimension barrier scores (Tier 1–4) |
207207

208+
### Elicitation Coverage
209+
210+
Values reported above were measured under the following elicitation conditions: ProteinMPNN sampling temperature swept over T in {0.05, 0.1, 0.2, 0.5} (FSI stable for BoNT-A, Spearman rho = -0.80); ESM-2 masked prediction is deterministic. Adversarial elicitation axes (fixed-chain / bias_AA constraints, multi-seed redesign over diverse backbone conformers, cross-model FSI at controlled temperature) have not yet been swept. Reported FSI and FSPE values should therefore be read as conservative estimates under the tested elicitation surface.
211+
208212
### Release-surface checks
209213

210214
The GitHub repository includes CI checks for withheld generated artifacts,
@@ -214,6 +218,15 @@ and [publishing checklist](https://github.com/jang1563/narrow-model-safety-eval/
214218

215219
---
216220

221+
## External Validation Status
222+
223+
No independent external validation has been performed on this dataset or framework. Two high-priority validation steps are open:
224+
225+
1. **Independent re-curation** of catalytic residues for at least one toxin by a second annotator (catches numbering / accession failure modes like those documented in the FSI numbering audit).
226+
2. **Cross-institution FSI replication** on the same PDB inputs with an independent ProteinMPNN run.
227+
228+
Replication-relevant inputs (sequences, structures, annotations, aggregate result JSONs) are all included in this dataset; reproduction does not require any access-gated artifact. Validation reports, annotation corrections, or replication failures can be opened as GitHub issues with the validation label.
229+
217230
## Usage
218231

219232
### Load result files directly
@@ -269,6 +282,17 @@ python src/01_collect_data.py # downloads sequences + structures
269282

270283
---
271284

285+
## Related Work
286+
287+
This dataset measures representation-level dual-use encoding in protein language and design models. It is complementary to generation-level red-teaming of the same model class.
288+
289+
- Fan et al. (2025), [SafeProtein](https://arxiv.org/abs/2509.03487): red-teaming framework that tests whether protein foundation models generate sequences matching harmful biological targets under adversarial prompting (up to 70% jailbreak ASR on ESM3).
290+
- This work (FSPE/FSI/PRT): measures whether the same model class already encodes dangerous function in its representations, independent of any generation-time prompt.
291+
292+
The two surfaces are orthogonal: a model may pass a generation-time red-team while still encoding the function at representation level, or vice versa.
293+
294+
---
295+
272296
## Ethics & Responsible Use
273297

274298
This dataset is released for **AI safety research, biosecurity policy, and scientific model evaluation purposes only**.

0 commit comments

Comments
 (0)