Given the output type and test, is this related to #2591 @DusanJovic-NOAA ?

Given the output type and test, is this related to #2591 @DusanJovic-NOAA ?

PR 2591 only changes how data are written, it does not change any data. The regression test reproduces current outputs. I'll take a look at hafs_regional_atm_thompson_gfdlsf regression test.

The fact that not all points along the boundary rows/columns are zero indicates that the source of the erroneous zero values are due to esmf remapping. Probably because of slight differences in the lat/lon values between the forecast component and write component. Interestingly, this test (hafs_regional_atm_thompson_gfdlsf) is the only test that outputs regional native grid outputs (i.e. cubed sphere) of all hafs tests. And it's compiled with -D32BIT=ON, which means the lat/lon values of the esmf grid on the forecast component are 32-bit values, while the esmf grid on the write grid component are internally 64-bit values (by default).

Just to test this hypothesis, I compiled the model using 64-bit (without -D32BIT=ON cmake option) and ran the hafs_regional_atm_thompson_gfdlsf test, for example by using this rt.conf:

### HAFS tests ###
COMPILE | hafs | intel | -DAPP=HAFS -DCCPP_SUITES=FV3_HAFS_v1_thompson_tedmf_gfdlsf |  | fv3 |
RUN | hafs_regional_atm_thompson_gfdlsf                 |                              | baseline |

and I do not see zeros along the boundary.

@kristinbarton Can you please recompile your code without -D32BIT=ON and rerun your test.

If this works, then the proper solution will be either to truncate the esmf grid coordinates to single precision when 32BIT is ON, or maybe to use esmf redist instead of esmf remap.

Thank you for looking into this.

Our regional Arctic domain also used -D32BIT=ON, so I recompiled both our case and the regression test using 64-bit. Both cases successfully generated output with no zeros.

@kristinbarton Are you planning to troubleshoot this issue and open a PR, or are you looking for one of our teams to assist with this?
@BinLiu-NOAA do you have any suggestions for Kristin?

@gspetro-NOAA For our purposes, using the 64-bit compilation is satisfactory.

I'm testing a code change that will fix this issue with -D32BIT=ON.

@kristinbarton Please try to use the code from this fv3atm branch (https://github.com/DusanJovic-NOAA/fv3atm/tree/cubed_sphere_redist) and corresponding ufs-wm branch (https://github.com/DusanJovic-NOAA/ufs-weather-model/tree/cubed_sphere_redist) to run your regional Arctic domain configuration but with -D32BIT=ON, to see if the issue is fixed.

If it is, I will open a PR.

@DusanJovic-NOAA It looks like this did fix the issue. I tested it on our Arctic configuration with 32-bit on and there were no zeros in the output.