Skip to content

Commit

Permalink
docs: CHANGELOG.md fix
Browse files Browse the repository at this point in the history
  • Loading branch information
@Titus-von-Koeller
Titus-von-Koeller committed Jul 27, 2024
1 parent 24f7b65 commit e3b2780
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
This release is quite significant as the QLoRA bug fix big implications for higher `seqlen` and batch sizes.

For each sequence (i.e. batch size increase of one) we expect memory savings of:
- 405B: 39GB for seqlen 1024, and 4888GB for 128k
- 70B: 20.1GB for 1024 and 2516GB for 128k
- 405B: 39GB for `seqlen=1024`, and 4888GB for `seqlen=128,00`
- 70B: 10.1GB for `seqlen=1024` and 1258GB for `seqlen=128,00`

This was due to activations being unnecessary for frozen parameters, yet the memory for them was still erroneously allocated due to the now fixed bug.

Expand Down

0 comments on commit e3b2780

Please sign in to comment.