Is it possible to train the 1.3B model on a 40G GPU?

[CzzzzH]

Any idea of it?

[cr8br0ze]

It's quite hard. We tried to reproduce the training, and the minimum we found to work is 8x80G VRAM GPUs.
Maybe 4x80G can work if the teacher is swapped to 1.3B.