Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geneformer modelcard and loss eval script #392

Merged
merged 12 commits into from
Nov 6, 2024
Prev Previous commit
Next Next commit
Update docs/docs/models/geneformer.md
Co-authored-by: Peter St. John <pstjohn@nvidia.com>
Signed-off-by: John St. John <jstjohn@users.noreply.github.com>
  • Loading branch information
jstjohn and pstjohn authored Nov 5, 2024
commit dd7e636e00dbda2a99f73a2284da044d235e6425
4 changes: 2 additions & 2 deletions docs/docs/models/geneformer.md
Original file line number Diff line number Diff line change
@@ -208,8 +208,8 @@ The following describes the bert MLM token loss. Like in the original BERT paper
* The model is trained with a 4096 context rather than a 2048 context. When forcing the model to make predictions
with a 2048 context, the MLM loss drops to *2.76*, which is probably unfair because this may be "out of domain" for
training. It is really hard to compare these loss numbers directly is the only take-home here.
* The model was trained on a set of 20275 genes, rather than the older set of 25426 genes. This would also be
expected to give a boost in loss since there are fewer tokens to chose from.
* The model was trained on a set of 20,275 genes, rather than the older set of 25,426 genes. This would also be
expected to give a boost in loss since there are fewer tokens to choose from.

#### Downstream task accuracy

Loading