Can the random seed be fixed to get reproducible training results?

[davidndrio]

Hi,

I noticed that running the same qNEP training twice with identical nep.in and training data gives substantially different results each time: different loss curves, different final RMSE, different nep.txt. For example, the RMSE of energies changed by a factor of ~2 between two otherwise identical runs.

Is there a way to fix the random seed to make results reproducible across fresh runs?

[brucefan1983]

Could you show loss plots?

If you want to enforce reproduction, you can add -DDEBUG to makefile:

CFLAGS = -std=c++14 -O3 $(CUDA_ARCH) -DDEBUG

Then

make clean
make -j 

But due to the use of atomicAdd() function, two trainings will gradaully (perhaps only slightly) diverge with increasing training steps.

Also, this will cause MD part to be deterministic too, which is usually not what you want. So usually you need to remove -DDEBUG to compile gpumd.

[davidndrio]

Thank you for the information!

Here are the loss functions and RMSE of the 2 trainings.

[brucefan1983]

Seem the second run got better accuracy of BEC (Z), but worse for potential.

There was a bug for BEC training with non-fullbatch mode. Which GPUMD version did you use.

Otherwise, I think your dataset have small range of energy and a RMSE of 1 meV/atom is actually not much different from 2 meV/atom in practice.

[davidndrio]

Both were in version 5.2.

Got it.

Thank you very much for the help!

[brucefan1983]

The bug was fixed in GPUMD-5.2, so that's not the reason. I think they are just due to randomess in the training process.