Error when running nrnivmodl -coreneuron

[kf-cuanschutz]

Context

I get an error after building Neuron with gpu, when it comes to run it.
I tried to rebuild it with different nvhpc_sdk and gcc versions as well.

This is following up from this issue

I have issues when calling nrnivmodl -coreneuron

I also have a problem with my x86_64/.libs/libnrnmech.so library.

I get a x86_64/.libs/libnrnmech.so: undefined symbol: use_cachevec error.

I have attached the logs.

[JCGoran]

Hi, can you try removing the x86_64 directory and re-running nrnivmodl -coreneuron .? It's possible that another version of NEURON built the mod files, and now the new version is trying to link it with symbols that that don't exist anymore.

[kf-cuanschutz]

Thanks you! It looks like following what you said, and running nrnivmodl -coreneuron it worked. However, when running mpirun it does not look like I have numpy for some reasons. Should I install numpy with pip now?

[kf-cuanschutz]

Ok I installed numpy and matplotlib with pip and now I am getting this new issue below.`

`

[kf-cuanschutz]

Ok I get the following now when running this command:

mpirun -vvv -n 4 x86_64/special -mpi  /projects/kefo9343/software/spack/linux-rhel8-zen3/gcc-11.2.0/python-3.10.10-ggcucumyfl6hqjbdahvkqkq6h44nxdkj Neuron_Python_SAC_GA_stim_v3_0.py 

Please see the attached log file. Do you think there is an issue with my libnrniv.so and libc.so.6 libraries?

neuron.log

[nrnhines]

Is your model, or a reduced size form of it, runnable (without gpu, or without coreneuron, or without MPI) on your desktop?
I see that the segfault for each of the four processes occurs at

_Z12newpysechelpP7Section+0x2b

and that has nothing to do with mpi or coreneuron but during creation of a python reference to a NEURON section. It is not clear (though it might become clear with gdb) what line that segfault occurred.

[kf-cuanschutz]

Sorry I realize that I did not call python correctly earlier in my command. I don't think the model was working either without gpu or MPI. However, I reran it properly by calling python and I got a "x86_64/special: syntax error" near line 1 of my python executable. I have attached the corresponding log.

neuron_correct_python_call_mpi.log

[nrnhines]

I did not notice the invalid argument to special in your earlier log. In your last log

$ mpirun -vvv -n 4 x86_64/special -mpi $PYTHON_SPACK/bin/python3  Neuron_helloworld.py

the arg that specifies the python executable $PYTHON_SPACK/bin/python3 is being treated in this context as a HOC file and hence the syntax error. If that is the default python, it is unneeded. If not the default python then you can specify it
with -pyexe $PYTHON_SPACK/bin/python3

Also the argument that is needed to use Python for interactive terminal input instead of HOC is -python. Please try

mpirun -vvv -n 4 x86_64/special -mpi -python Neuron_helloworld.py

[kf-cuanschutz]

Thank you! I ran the following:

mpirun -vvv -n 4 x86_64/special -mpi -python Neuron_helloworld.py

python_Neuron_helloworld_.log

I also ran the following:

 mpirun -vvv -n 4 x86_64/special -mpi -pyexe $PYTHON_SPACK/bin/python3 Neuron_helloworld.py

In both cases I got a segfault. I have attached the log with this message.

I also then ran a few to make sure that I had a good build.

For instance I ran this test.py code: https://bluebrain.github.io/CoreNeuron/doxygen/index.html

I got the following error:

python test.py 
python: /projects/kefo9343/nrn/src/nrnpython/nrnpy_nrn.cpp:1057: NPySecObj *newpysechelp(Section *): Assertion `pysec->sec_ == sec' failed.
Aborted (core dumped)

I also tried to run the test_direct.py example from here https://nrn.readthedocs.io/en/latest/coreneuron/examples.html
I got the same error:

 nrniv -python test_direct.py
NEURON -- VERSION 9.0a-488-gdf306bd7b+ master (df306bd7b+) 2025-01-30
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2022
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 "mod/axial.mod" "mod/axial_pp.mod" "mod/bacur.mod" "mod/banocur.mod" "mod/fornetcon.mod" "mod/green.mod" "mod/invlfire.mod" "mod/netmove.mod" "mod/noisychan.mod" "mod/red.mod" "mod/sample.mod" "mod/watchrange2.mod" "mod/watchrange.mod"
nrniv: /projects/kefo9343/nrn/src/nrnpython/nrnpy_nrn.cpp:1057: NPySecObj *newpysechelp(Section *): Assertion `pysec->sec_ == sec' failed.
Aborted (core dumped)

Finally, I did ran the ring tests and it seemed to pass successfully both on CPUs and GPUs: https://github.com/neuronsimulator/ringtest

Does that benchmarking thus indicate that my neuron build was good?

[nrnhines]

For instance I ran this test.py code

I did not experience an error with that test on my ubuntu 24.04 desktop with 9.0a-490-g06f324e38 master.
My default python is 3.13.1
I built with

cmake .. -G Ninja -DCMAKE_INSTALL_PREFIX=install -DNRN_ENABLE_CORENEURON=ON
ninja install

And tested with

export PATH=$HOME/neuron/nrn/build/install/bin:$PATH
export PYTHONPATH=$HOME/neuron/nrn/build/install/lib/python
nrniv -python test.py

If you send me all the files (hoc, py, ses, mod) needed to run your model compressed into a zip file, I can see if it runs on my machine. Send to [email protected]

[kf-cuanschutz]

Thank you very much! I just sent you an invitation to Dropbox.

[nrnhines]

I'm afraid I did not get the invitation. If the zip file is less than 3 MB, just send it to me via email.
Sorry. I fogot I had a login to Dropbox. Downloaded Neuron_files_to_send.zip and will get back to you.

[nrnhines]

Via zoom (with you sharing your screen) we were able to experience a segfault only when using a build with the cmake args -DCORENRN_ENABLE_GPU=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo. With gpu arg off or build type Debug, there was no segfault. The origin of the segfault was traced to line 1048 of nrnpy_nrn.cpp

    if (auto* pv = sec->prop->dparam[PROP_PY_INDEX].get<void*>(); pv) {

where pv was an invalid address 0x1 (instead of 0). We used nrniv -python test.py where test.py is

from neuron import h
h("create dend[4]")
for sec in h.allsec():
    print(sec)

We tried various work arounds, e.g.

void* pv = sec->prop->dparam[PROP_PY_INDEX].get<void*>();
if (pv) {

did not fix the problem. Comparing to nullptr did not fix the problem. And forcing pv to 0 if it is less than 0x10 merely delayed the issue to an assertion error for the next call to NPySecObj* newpysechelp(Section* sec) {

       assert(pysec->sec_ == sec);`

I've installed the NVIDIA HPC SDK on my machine but have been unable to reproduce the issue. Here are my notes in regard to that.

hines@hines-ThinkStation-P5:~/neuron/nrn/bldnv$ /opt/nvidia/hpc_sdk/Linux_x86_64/25.1/compilers/bin/nvc --version

nvc 25.1-0 64-bit target on x86-64 Linux -tp sapphirerapids 

export CC=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/compilers/bin/nvc
export CXX=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/compilers/bin/nvc++

cmake .. -G Ninja -DCMAKE_INSTALL_PREFIX=install -DNRN_ENABLE_CORENEURON=ON
ninja install

nrnenv nrn/bldnv/install # script to set PATH and PYTHONPATH

test.py
-------
from neuron import h
h("create dend[4]")
for sec in h.allsec():
    print(sec)

$ nrniv -python test.py
NEURON -- VERSION 9.0a-496-gcaaab8355 master (caaab8355) 2025-03-03
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2022
See http://neuron.yale.edu/neuron/credits

dend[0]
dend[1]
dend[2]
dend[3]
>>> 

cmake .. -DCORENRN_ENABLE_GPU=ON
ninja install
nrniv -python test.py # still works

The LINK FLAGS differences between GPU on and off are

-DCORENRN_ENABLE_GPU=ON

-- CoreNEURON    | ON
--   PATH        | /home/hines/neuron/nrn/src/coreneuron
--   LINK FLAGS  | -cuda -gpu=cuda12.6,lineinfo,cc70,cc80 -acc -lcorenrnmech  -lcoreneuron-cuda -Wl,-rpath,/home/hines/soft/openmpi/lib /home/hines/soft/openmpi/lib/libmpi.so -ldl

-DCORENRN_ENABLE_GPU=OFF
-- CoreNEURON    | ON
--   PATH        | /home/hines/neuron/nrn/src/coreneuron
--   LINK FLAGS  |  -lcorenrnmech  -Wl,-rpath,/home/hines/soft/openmpi/lib /home/hines/soft/openmpi/lib/libmpi.so -ldl