question about bf16 training #1
Description
Hi,
I have some questions related to the paper: the grad is accumulated in fp32 when using fp32 master weight, so in collage optimizer, is this done in bf16? If so, how can the precision match fp32?
{{ message }}