Issue when running job twice, simultaneously, by accident

[jakobcasa]

Hi

I unfortunately ran the same traj twice (in different jobs with slurm), and after killing one of the two and the other finished normally, it couldn't restart. The QM.err file is shown below. The TRAJ I can send you, of course (please let me know to whom I should send it). I'm still using SHARC 3.0, so I also tried the 2 restart options, but neither worked.

Thanks

Best
Jakob

Traceback (most recent call last):
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 5292, in
main()
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 5256, in main
errorcodes = runjobs(schedule, QMin)
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 2888, in runjobs
saveFiles(WORKDIR, jobset[job])
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 3416, in saveFiles
strings = get_dets_from_cis(f, QMin)
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 3648, in get_dets_from_cis
infos['NVA'] = infos['nbsuse'] - infos['NOA']
KeyError: 'nbsuse'
Traceback (most recent call last):
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 5292, in
main()
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 5265, in main
QMin, QMout = getQMout(QMin)
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 4542, in getQMout
g = getgrad(logfile, QMin)
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 4993, in getgrad
Gfile = open(logfile, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/storage/homefs/jc22p286/m8/sharc/MeCN/Trj/batch2/Singlet_1/TRAJ_00048/QM/SCR/Singlet_1/TRAJ_00048/master_1/ORCA.engrad.ground.grad.tmp'
Traceback (most recent call last):
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 5292, in
main()
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 5265, in main
QMin, QMout = getQMout(QMin)
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 4542, in getQMout
g = getgrad(logfile, QMin)
File "/storage/homefs/jc22p286/applications/sharc/bin/SHARC_ORCA.py", line 4993, in getgrad
Gfile = open(logfile, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/storage/homefs/jc22p286/m8/sharc/MeCN/Trj/batch2/Singlet_1/TRAJ_00048/QM/SCR/Singlet_1/TRAJ_00048/master_1/ORCA.engrad.ground.grad.tmp'

[maisebastian]

Hi Jakob,
the error message indicates that something was wrong with the ORCA job, because the interface complains that it cannot find the number of orbitals in the CIS calculation. Can you check if the last ORCA job ran through?

[jakobcasa]

Hi sebastian

The calculation did not run through: ORCA finished by error termination in SCF gradient
Calling Command: mpirun -np 16 /software.9/software/ORCA/5.0.3-gompi-2021b/bin/orca_scfgrad_mpi ORCA.scfgrad.inp ORCA
[file orca_tools/qcmsg.cpp, line 465]:
.... aborting the run
and the ORCA.scfgrad.inp does not exist, but untill I started by accident the second job, it was fine! is there an option to start with the last step to continue?

But when I copied the ORCA.inp to the home dir (so outside of the SHARC procedure), the job runs fine.

Best

[maisebastian]

In principle, it should be restartable if all restart files are uncorrupted. The fact that you cannot restart indicates to me that maybe something got broken in the restart files...

That ORCA.scfgrad.inp does not exist is likely simply a consequence of the failure of the job.

Running the job twice is generally not something that we considered (obviously), so it is likely that it will break a lot of different things in an unrecoverable way. In SHARC 4, we guard against this by stopping the driver if restart files are present but the restart keyword is not there. This should stop accidental second executions like in your case. But for SHARC 3 and your case, I do not see a good way of salvaging the trajectory, unless it is clear which files got corrupted.

[jakobcasa]

Is there a way to check which file is corrupted?, Should I send you the TRAJ folder? Now also different TRAJ failed unexpectedly (which did not run twice at the same time), but upon a change of the partition, this is an issue you are not able to solve!

I did also checked the QM.log and it shows for all the traj, which crashed:
....
Could not find Orca version!
2025-11-06 12:08:31.979567 cnode16 /storage/homefs/jc22p286/m8/sharc/MeCN/Trj/batch2/Singlet_1/TRAJ_00053/QM

================================================================================
|| ||
|| SHARC - ORCA - Interface ||
|| ||
|| Authors: Sebastian Mai, Lea Ibele, and Moritz Heindl ||
|| ||
|| Version: 3.0 ||
|| Date: 01.04.23 ||
|| ||

/storage/homefs/jc22p286/m8/sharc/MeCN/Trj/batch2/Singlet_1/TRAJ_00053/QM/SCRATCH exists, cannot create a link of the same name!
/storage/homefs/jc22p286/m8/sharc/MeCN/Trj/batch2/Singlet_1/TRAJ_00053/QM/SAVE exists, cannot create a link of the same name!
Could not find Orca version!

--> but when I check the path, the orca files are there. Or is this also only a sign that there is a broken file in the restart?

Thanks

Best

[jakobcasa]

It works—for now—for all the traj in question. I do not know what the issue was. I'm sorry for bothering you!

thanks for the help!

Best
Jakob

[maisebastian]

Could not find Orca version sounds like there was some (maybe temporal) problem with your file system, so that the ORCA installation was not available when the interface was looking for it.

[maisebastian]

OK, great to hear! All the best!

[jakobcasa]

Yes this I thought so too!