You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Mirror the premium-level enhancements made to the GitHub Evaluation Report into
the Hugging Face dataset card, adapted for the lighter HF card surface:
- §Elicitation Coverage (under Key Results): tested vs untested elicitation
axes; reported FSI/FSPE values are conservative under the tested surface.
- §External Validation Status: two-priority validation ask (independent
re-curation + cross-institution replication); inputs are reproducible
from this dataset alone, no access-gated artifact required.
- §Related Work: positions this dataset's representation-level signal as
complementary to SafeProtein (Fan et al. 2025) generation-level red-teaming.
Risk-Tier Mapping was intentionally NOT mirrored — it is a deployment-context
artifact that belongs in the Evaluation Report, not a dataset card.
10/10 release-integrity tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Values reported above were measured under the following elicitation conditions: ProteinMPNN sampling temperature swept over T in {0.05, 0.1, 0.2, 0.5} (FSI stable for BoNT-A, Spearman rho = -0.80); ESM-2 masked prediction is deterministic. Adversarial elicitation axes (fixed-chain / bias_AA constraints, multi-seed redesign over diverse backbone conformers, cross-model FSI at controlled temperature) have not yet been swept. Reported FSI and FSPE values should therefore be read as conservative estimates under the tested elicitation surface.
211
+
208
212
### Release-surface checks
209
213
210
214
The GitHub repository includes CI checks for withheld generated artifacts,
@@ -214,6 +218,15 @@ and [publishing checklist](https://github.com/jang1563/narrow-model-safety-eval/
214
218
215
219
---
216
220
221
+
## External Validation Status
222
+
223
+
No independent external validation has been performed on this dataset or framework. Two high-priority validation steps are open:
224
+
225
+
1.**Independent re-curation** of catalytic residues for at least one toxin by a second annotator (catches numbering / accession failure modes like those documented in the FSI numbering audit).
226
+
2.**Cross-institution FSI replication** on the same PDB inputs with an independent ProteinMPNN run.
227
+
228
+
Replication-relevant inputs (sequences, structures, annotations, aggregate result JSONs) are all included in this dataset; reproduction does not require any access-gated artifact. Validation reports, annotation corrections, or replication failures can be opened as GitHub issues with the validation label.
This dataset measures representation-level dual-use encoding in protein language and design models. It is complementary to generation-level red-teaming of the same model class.
288
+
289
+
- Fan et al. (2025), [SafeProtein](https://arxiv.org/abs/2509.03487): red-teaming framework that tests whether protein foundation models generate sequences matching harmful biological targets under adversarial prompting (up to 70% jailbreak ASR on ESM3).
290
+
- This work (FSPE/FSI/PRT): measures whether the same model class already encodes dangerous function in its representations, independent of any generation-time prompt.
291
+
292
+
The two surfaces are orthogonal: a model may pass a generation-time red-team while still encoding the function at representation level, or vice versa.
293
+
294
+
---
295
+
272
296
## Ethics & Responsible Use
273
297
274
298
This dataset is released for **AI safety research, biosecurity policy, and scientific model evaluation purposes only**.
0 commit comments