Doesn't it support bf16 when using Deepspeed?

[toriving]

Hello.

I am currently trying to train a model using Deepspeed zero-2.
In Deepspeed official repository, can use bf16 when using zero-2, but isn't it currently supported in PL?

[rohitgr7]

if it's supported natively in deepspeed, PL should support it. Can you share some references from deepspeed?

[toriving]

I was able to find a reference at the links below.

• Bfloat16 zero2 deepspeedai/DeepSpeed#1398
• https://www.deepspeed.ai/docs/config-json/#bfloat16-training-options

[rohitgr7]

PR: Add bfloat16 support to DeepSpeedStrategy

[toriving]

Cooool! Thx :)