HF Accelerate FP8 use more gpu memory then FP16 in training LLM

[Liufeiran123]

HF Accelerate FP8 use more gpu memory then FP16 when trianing LLM
torch == 2.5.1
accelerate == 1.3.0
transformers == 4.45.0
transformer-engine == 1.13

[pggPL]

FP8 training stores weights (and gradients, optimizer state etc.) in higher precision and casts them to FP8 before each gemm. It is faster, but it consumes more memory. So it is expected behavior.