testcorenrn_patstim::coreneuron_gpu_offline test does not produce reliable output #563
Description
Describe the issue
For several days we have seen hard-to-reproduce CI errors when the testcorenrn_patstim::coreneuron_gpu_offline
test in the NEURON CTest suite produces output spikes that do not match NEURON or CoreNEURON with a CPU backend. The failing tests were run with 2 MPI ranks, reducing this to 1 seems to avoid the problem.
To Reproduce
Build NEURON + CoreNEURON-with-GPU-support, make sure the testcorenrn_patstim::coreneuron_gpu_offline
test is using 2 MPI ranks...see if the CTest suite passes. Reproducing this locally has been hard, the CI environment seems to be more prone to it.
Because the CoreNEURON+GPU configuration does not crash, the test that actually fails is testcorenrn_patstim::compare_results
, which compares the spike outputs from different configurations.
Expected behavior
CoreNEURON+GPU should consistent match NEURON/CoreNEURON+CPU/reference file.
System (please complete the following information)
- OS: BB5
- Compiler: NVHPC 21.2
- Version: master
- Backend: GPU