Does DeepSpeed of Pytorch-Lightning support BFloat16 now? #12481

Unanswered

ShaneTian asked this question in DDP / multi-GPU / multi-node

ShaneTian
Mar 28, 2022

How to configure BFloat16 in DeepSpeed of Pytorch-Lightning?

Pass to Trainer only: Trainer(..., precision="bf16", ...)
Configure in DeepSpeed config only:
```
{
  "bf16": {
    "enabled": true
  }
}
```
Both setup Trainer and DeepSpeed config

One more thing:
I did not find anything about bf16 in deepspeed, is it not supported now?
https://github.com/PyTorchLightning/pytorch-lightning/blob/2e5728a4841cb1d3b75123ac941cd36f681f9a11/pytorch_lightning/strategies/deepspeed.py#L652-L667

Replies: 1 comment

rohitgr7
Apr 5, 2022

will be added:
PR: Add bfloat16 support to DeepSpeedStrategy

0 replies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment