🐛 [BUG] - Significant difference in the values of the Hessian for high-valued points between GPU and CPU

[eyalshimony]

Description

A significant difference in the values of the Hessian for high-valued points is observed between simulation run on GPU and on CPU. Thus, while the median, 99th, and 99.9th percentile are still the same, a difference is already observed at the 99.99th percentile (2%) and keeps growing from there - 16% at the 99.999th percentile, and 11 order of magnitude difference between the absolute maximum of the two. This difference was not observed for the sensitivity kernels for alpha, beta and rho. While this issue can be overcome by clipping, it still seems like a problem.

Affected SPECFEM3D version

Latest development version

Your software and hardware environment

gcc 10.1.0, slurm 24.05.05, cluster with gpu

Reproduction steps

Model: 80x80x80 elements, 20 km x 20 km x 20 km, dt = 3 ms, NSTEP = 5000.
CPU run on 50 cores, GPU on 2 GPUs.

Screenshots

![DESCRIPTION](LINK.png)

Logs

OS

Linux