Skip to content

Commit PLM parity fixtures and gate parity tests on FERRITIN_HF_TESTS (ferritin-100.2) - #171

Merged
zachcp merged 1 commit into
mainfrom
ferritin-100-plms-hardening
Aug 30, 2026
Merged

Commit PLM parity fixtures and gate parity tests on FERRITIN_HF_TESTS (ferritin-100.2)#171
zachcp merged 1 commit into
mainfrom
ferritin-100-plms-hardening

Conversation

@zachcp

@zachcp zachcp commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Closes ferritin-100.2 (P0 under the ferritin-100 PLM-hardening epic).

What this does

Establishes the correctness floor for the PLM parity suite: commits the small ESM2 + AMPLIFY golden-fixture references and makes the parity tests actually run.

  • Commit fixturesesm2_parity.safetensors + amplify_parity.safetensors (8.5 KB combined, budget was ~2 MB), un-gitignored. Generated from canonical HuggingFace models.
  • Un-#[ignore] the parity tests — gated instead on FERRITIN_HF_TESTS=1 via support::parity::hf_tests_enabled. A plain cargo test exercises the wiring and skips the multi-hundred-MB weight download; the env var enables the real numeric comparison.
  • Nightly workflow.github/workflows/nightly-parity.yaml (cron + workflow_dispatch, HF download cache) sets FERRITIN_HF_TESTS=1 and runs cargo test -p ferritin-plms -- --include-ignored.
  • Fix generate_amplify_fixtures.py for modern transformers (trust_remote_code, AutoModel per the repo's auto_map, additive attention mask, materialize the meta-device RoPE buffer).

Fixtures immediately caught real bugs (filed separately)

Validating the committed fixtures against the Rust ports surfaced exactly the silent numerical drift this epic exists to expose:

  • ferritin-n5g (P1 bug) — ESM2 Candle logits diverge from HF on identical bare-token input (argmax differs; AA logits off by 9-15). Confirmed, no confounder.
  • ferritin-edm (P2 bug) — AMPLIFY 120M distribution divergence (argmax matches; softmax off by ~0.49 on one sequence). Needs confirmation with real xformers / on GPU.
  • ferritin-100.7run_forward adds no BOS/EOS, so the test's L+2 alignment is wrong; folded into the special-token-contract issue that owns it.

The nightly is expected to run red under FERRITIN_HF_TESTS=1 until those are resolved — that is the intended "make drift visible" signal.

Test plan

  • cargo test -p ferritin-plms — green (111 + integration tests); parity tests run and skip cleanly without the env var.
  • Nightly / local FERRITIN_HF_TESTS=1 cargo test -p ferritin-plms — runs the real comparison (currently fails per n5g/edm, as designed).

Notes / deviations

  • Fixtures live in crates/ferritin-plms/tests/fixtures/ to match the ParityFixture::load harness finalized in ferritin-100.1, rather than the issue's original ferritin-test-data suggestion.
  • ProteinMPNN parity fixture not committed / test left #[ignore]d: its generator needs the external dauparas/ProteinMPNN repo on PYTHONPATH, out of scope here.

🤖 Generated with Claude Code

… (ferritin-100.2)

Commit the small ESM2 and AMPLIFY golden-fixture parity references (8.5 KB
combined) into tests/fixtures/ and un-gitignore them, so the parity tests that
previously could never run now have a committed reference. Convert those two
parity tests from #[ignore] to a FERRITIN_HF_TESTS=1 runtime gate (via
support::parity::hf_tests_enabled): a plain `cargo test` exercises the wiring
and skips the multi-hundred-MB weight download, while the new nightly workflow
sets the env var and runs the full `-- --include-ignored` suite.

Fixture generation: fix scripts/generate_amplify_fixtures.py for modern
transformers (trust_remote_code, AutoModel per the repo's auto_map, additive
attention mask, materialize the meta-device RoPE buffer).

Validating the fixtures against the Rust ports surfaced real problems, filed
separately (the point of the parity harness):
 - ferritin-n5g: ESM2 Candle logits diverge from HF on identical input (bug)
 - ferritin-edm: AMPLIFY 120M distribution divergence (needs confirmation)
 - ferritin-100.7: run_forward adds no BOS/EOS, so the L+2 test alignment is
   wrong — folded into the special-token-contract issue.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zachcp
zachcp merged commit 562e839 into main Aug 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant