Skip to content

[NEW SUBMISSION] ProSST PIT tail rank scoring function - #100

Open
emmett-bicker wants to merge 3 commits into
OATML-Markslab:mainfrom
emmett-bicker:prosst-2048-single-pass-delta
Open

[NEW SUBMISSION] ProSST PIT tail rank scoring function#100
emmett-bicker wants to merge 3 commits into
OATML-Markslab:mainfrom
emmett-bicker:prosst-2048-single-pass-delta

Conversation

@emmett-bicker

@emmett-bicker emmett-bicker commented Mar 10, 2026

Copy link
Copy Markdown

This PR adds a new ProSST (K=2048) scoring method called PIT Tail Rank. It keeps the same single wild-type forward-pass setup, but replaces the previous confidence-calibrated delta readout with a minimal equal-weight calibration score.

For each position, ProSST probabilities over the 20 canonical amino acids are converted into three centered residual signals:

  • centered lower-tail residual
  • protein-level PIT calibration residual
  • protein-level rank residual

The submitted score is simply:

score = local_tail_residual + protein_PIT_residual + protein_rank_residual

There are no learned parameters and no tuned scalar weights.

Validation against official vanilla ProSST log-probability delta:

Model Vanilla ProSST PIT Tail Rank
ProSST-1024 0.484672 0.490633
ProSST-2048 ~0.507 0.524495
ProSST-4096 0.498301 0.500819

The submitted model remains ProSST-2048, where this minimal method gives the strongest absolute result. The same scoring rule also improves ProSST-1024 and ProSST-4096 over their vanilla log-probability-delta baselines.

@emmett-bicker emmett-bicker changed the title Add ProSST-2048 single-pass delta baseline [NEW SUBMISSION] Altered scoring function results in significant boost for ProSST models. Applied to ProSST 2048, this tops the benchmark Mar 10, 2026
@emmett-bicker emmett-bicker changed the title [NEW SUBMISSION] Altered scoring function results in significant boost for ProSST models. Applied to ProSST 2048, this tops the benchmark [NEW SUBMISSION] Confidence-Calibrated Delta scoring function Mar 10, 2026
@emmett-bicker

Copy link
Copy Markdown
Author

Hey @pascalnotin -- When you have time, would really appreciate a review of this

1 similar comment
@emmett-bicker

Copy link
Copy Markdown
Author

Hey @pascalnotin -- When you have time, would really appreciate a review of this

Replace the tuned confidence-calibrated delta readout with the minimal equal-weight PIT/tail/rank scorer so the submission matches the validated method.

Co-authored-by: Cursor <cursoragent@cursor.com>
@emmett-bicker emmett-bicker changed the title [NEW SUBMISSION] Confidence-Calibrated Delta scoring function [NEW SUBMISSION] ProSST PIT tail rank scoring function Jun 7, 2026
Align the baseline paths, scripts, and scorer implementation names with the validated minimal PIT/tail/rank method so the PR no longer references the old single-pass delta submission.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant