Mpi errore & forrtl: severe (157): Program Exception - access violation

[ESH-y]

Hi,
I have a project that I am stuck with. I keep getting different errors on different computers.
It should be noted that other scenarios in the same project have been completed successfully. Only this level 4 fire scenario is stuck.
A level 2 fire scenario with the exact same blowers ended successfully.

We tried many ways to solve it such as running with a different MESH distribution
We also checked vents, blowers and flow directions and found them to be as required.
The run collapse always between the times of 230-495 seconds.
Except when we remove the sprinkler, without sprinkles the run ended successfully and reached the end of the simulation time.

The sprinkle opening time is: 104
The start time of the bellows work is:119
Amount of cells in the project: 5,000,000
The project is divided into 16 MESH

The run line in Windows is: fds_local -p 16 -o 1 test.fds
The FDS version is: Revision : FDS-6.9.1-0-g889da6a-release
MPI version: 3.1
MPI library version: Intel(R) MPI Library 2021.6 for Windows* OS

The various errors received:

forrtl: severe (157): Program Exception - access violation
Image PC Routine Line Source
fds_openmp.exe 00007FF6AEF59E36 Unknown Unknown Unknown
fds_openmp.exe 00007FF6B6362086 Unknown Unknown Unknown
fds_openmp.exe 00007FF6B619C1A2 Unknown Unknown Unknown
fds_openmp.exe 00007FF6B619BAF0 Unknown Unknown Unknown
fds_openmp.exe 00007FF6B62924A9 Unknown Unknown Unknown
fds_openmp.exe 00007FF6B5FD78B9 Unknown Unknown Unknown
fds_openmp.exe 00007FF6B5F95DCB Unknown Unknown Unknown
fds_openmp.exe 00007FF6B694EBB0 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE524184D4 Unknown Unknown Unknown
ntdll.dll 00007FFE54841791 Unknown Unknown Unknown

another error:

ERROR: MPI exchange of particle sizes timed out for MPI process 0 running on FDS-hv1. FDS will abort.
Abort(1) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

I am attaching the file.
I am attaching a file with the details of the computer on which we ran

I would appreciate any help please

Thanks
test.fds.txt

[mcgratta]

I will take a look at your case. Next time, submit as an Issue so that someone can be explicitly assigned.

[mcgratta]

While I am running the case, try using the latest nightly build of FDS.

[mcgratta]

This case is still running, at about 700+ s or simulation time. It is using about 30 GB RAM. I am running this on a single node of a linux cluster. I do not have a Windows computer that can run it. Did you try the latest nightly build?

[ESH-y]

hi
Thank you very much for the quick reply. I would like to know, please, what is the running line you used to run the file? Because I tried to run on a Linux computer and I also got an error and the run crashed after about 300 seconds.
In addition, I would appreciate an explanation regarding the nightly bundle. Is this a valid version?
We are a company that provides customer service. Is it possible to establish results based on the nightly bundle?

Many thanks

[mcgratta]

This is the Slurm script I used to run the job. I compiled the source code myself, which is why the mpirun command points to a path that is not the same as the installed version. You can rewrite the script to point to the installed executable.

#!/bin/bash
#SBATCH -J test
#SBATCH -e /home/mcgratta/Test/test.err
#SBATCH -o /home/mcgratta/Test/test.log
#SBATCH --partition=batch4
#SBATCH --ntasks=16
#SBATCH --cpus-per-task=1
#SBATCH --nodes=1
#SBATCH --time=99-99:99:99
export OMP_NUM_THREADS=1
export I_MPI_DEBUG=5
mpirun  /home/mcgratta/firemodels/fds/Build/impi_intel_linux/fds_impi_intel_linux test.fds

The nightly build is the latest version of the source code. How you use it is up to you. We cannot fix old versions of the source code. We can only reissue the latest version of the code that hopefully fixes whatever problem you are having.

[mcgratta]

test.fds ran successfully.

 Fire Dynamics Simulator

 Current Date     : February 20, 2025  10:01:09
 Revision         : FDS-6.9.1-1904-g83a682a-master
 Revision Date    : Wed Feb 19 09:45:07 2025 -0500
 Compiler         : Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2025.0.4 Build 20241205
 Compilation Date : Feb 19, 2025 10:31:18

 Number of MPI Processes:  16

 MPI version: 3.1
 MPI library version: Intel(R) MPI Library 2021.14 for Linux* OS