Commit PLM parity fixtures and gate parity tests on FERRITIN_HF_TESTS (ferritin-100.2) - #171
Merged
Merged
Conversation
… (ferritin-100.2) Commit the small ESM2 and AMPLIFY golden-fixture parity references (8.5 KB combined) into tests/fixtures/ and un-gitignore them, so the parity tests that previously could never run now have a committed reference. Convert those two parity tests from #[ignore] to a FERRITIN_HF_TESTS=1 runtime gate (via support::parity::hf_tests_enabled): a plain `cargo test` exercises the wiring and skips the multi-hundred-MB weight download, while the new nightly workflow sets the env var and runs the full `-- --include-ignored` suite. Fixture generation: fix scripts/generate_amplify_fixtures.py for modern transformers (trust_remote_code, AutoModel per the repo's auto_map, additive attention mask, materialize the meta-device RoPE buffer). Validating the fixtures against the Rust ports surfaced real problems, filed separately (the point of the parity harness): - ferritin-n5g: ESM2 Candle logits diverge from HF on identical input (bug) - ferritin-edm: AMPLIFY 120M distribution divergence (needs confirmation) - ferritin-100.7: run_forward adds no BOS/EOS, so the L+2 test alignment is wrong — folded into the special-token-contract issue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes ferritin-100.2 (P0 under the ferritin-100 PLM-hardening epic).
What this does
Establishes the correctness floor for the PLM parity suite: commits the small ESM2 + AMPLIFY golden-fixture references and makes the parity tests actually run.
esm2_parity.safetensors+amplify_parity.safetensors(8.5 KB combined, budget was ~2 MB), un-gitignored. Generated from canonical HuggingFace models.#[ignore]the parity tests — gated instead onFERRITIN_HF_TESTS=1viasupport::parity::hf_tests_enabled. A plaincargo testexercises the wiring and skips the multi-hundred-MB weight download; the env var enables the real numeric comparison..github/workflows/nightly-parity.yaml(cron +workflow_dispatch, HF download cache) setsFERRITIN_HF_TESTS=1and runscargo test -p ferritin-plms -- --include-ignored.generate_amplify_fixtures.pyfor moderntransformers(trust_remote_code,AutoModelper the repo'sauto_map, additive attention mask, materialize the meta-device RoPE buffer).Fixtures immediately caught real bugs (filed separately)
Validating the committed fixtures against the Rust ports surfaced exactly the silent numerical drift this epic exists to expose:
run_forwardadds no BOS/EOS, so the test'sL+2alignment is wrong; folded into the special-token-contract issue that owns it.The nightly is expected to run red under
FERRITIN_HF_TESTS=1until those are resolved — that is the intended "make drift visible" signal.Test plan
cargo test -p ferritin-plms— green (111 + integration tests); parity tests run and skip cleanly without the env var.FERRITIN_HF_TESTS=1 cargo test -p ferritin-plms— runs the real comparison (currently fails per n5g/edm, as designed).Notes / deviations
crates/ferritin-plms/tests/fixtures/to match theParityFixture::loadharness finalized in ferritin-100.1, rather than the issue's originalferritin-test-datasuggestion.#[ignore]d: its generator needs the externaldauparas/ProteinMPNNrepo onPYTHONPATH, out of scope here.🤖 Generated with Claude Code