Add bufr2ioda and ens3dvar yaml for ZTD obs

[HuiLiu-NOAA]

This PR is to add a bufr2ioda yaml and an ens3dvar yaml for ZTD obs.

1. The bufr2ioda yaml converts RAP gpsipw dump hourly Bufr data into the IODA files.
2. The initial en3dvar yaml perform assimilation of ideal or real ZTD observations with MPAS-JEDI using the existing UKMO ZTD local operator.

[SamuelDegelia-NOAA]

@HuiLiu-NOAA Instead of adding a new en3dvar yaml in RDASApp/rrfs-test/testinput, could you instead just create a new obs-space yaml in RDASApp/rrfs-test/validated_yamls/templates/obtype_config? It could just be called something like ztd.yaml. And then in that file all you need to add is the section of your original yaml starting with - obs space: . You can see the files in that directory for examples. You'll also need to add distribution: and obs localizations: headers so that the ztd yaml can work with EnKF.

[SamuelDegelia-NOAA]

Also @HuiLiu-NOAA do you know how the time window is set for the ZTD obs? In our current bufr2ioda yamls we create a timeOffset variable and then use a pre-filter in the obs-space yamls such as:

         - filter: Domain Check
           apply at iterations: 0,1
           where:
             - variable:
                 name: MetaData/timeOffset
               minvalue: -0.75
               maxvalue: 0.75

[HuiLiu-NOAA]

Also @HuiLiu-NOAA do you know how the time window is set for the ZTD obs? In our current bufr2ioda yamls we create a timeOffset variable and then use a pre-filter in the obs-space yamls such as:

         - filter: Domain Check
           apply at iterations: 0,1
           where:
             - variable:
                 name: MetaData/timeOffset
               minvalue: -0.75
               maxvalue: 0.75

The "timeOffset" seems only available in prepbufr data, not in the bufr data? The ZTD bufr data seems does not have the variable.

[SamuelDegelia-NOAA]

The "timeOffset" seems only available in prepbufr data, not in the bufr data? The ZTD bufr data seems does not have the variable.

Yes the timeOffset is only in prepbufr. So if the ZTD obs are bufr then this method would not work. What is the time range of the obs in the bufr and IODA files? Is it small enough that we do not need a time window filter?

[HuiLiu-NOAA]

The "timeOffset" seems only available in prepbufr data, not in the bufr data? The ZTD bufr data seems does not have the variable.

Yes the timeOffset is only in prepbufr. So if the ZTD obs are bufr then this method would not work. What is the time range of the obs in the bufr and IODA files? Is it small enough that we do not need a time window filter?

The time of ZTD is right on the hour of every hourly bufr data. So, time window filter might not be needed?

[SamuelDegelia-NOAA]

Okay yeah then we probably do not need a time window filter right now. Thanks for confirming!

[HuiLiu-NOAA]

@HuiLiu-NOAA Instead of adding a new en3dvar yaml in RDASApp/rrfs-test/testinput, could you instead just create a new obs-space yaml in RDASApp/rrfs-test/validated_yamls/templates/obtype_config? It could just be called something like ztd.yaml. And then in that file all you need to add is the section of your original yaml starting with - obs space: . You can see the files in that directory for examples. You'll also need to add distribution: and obs localizations: headers so that the ztd yaml can work with EnKF.

Thanks! done as suggested.

[rrfsbot]

FAILED on hercules

started build_and_test on hercules at UTC time: Wed Feb 19 19:26:05 UTC 2025
finished at UTC time: Wed Feb 19 20:23:32 UTC 2025

Test project /work/noaa/wrfruc/rrfsbot/PRs_RDASApp/242/build/rrfs-test
    Start 2: rrfs_fv3jedi_2024052700_getkf_observer
    Start 5: rrfs_mpasjedi_2024052700_getkf_observer
    Start 1: rrfs_fv3jedi_2024052700_Ens3Dvar
    Start 4: rrfs_mpasjedi_2024052700_Ens3Dvar
    Start 7: rrfs_mpasjedi_2024052700_bumploc
    Start 8: rrfs_bufr2ioda_msonet
1/8 Test #8: rrfs_bufr2ioda_msonet .....................   Passed   53.00 sec
2/8 Test #5: rrfs_mpasjedi_2024052700_getkf_observer ...   Passed  129.51 sec
    Start 6: rrfs_mpasjedi_2024052700_getkf_solver
3/8 Test #2: rrfs_fv3jedi_2024052700_getkf_observer ....   Passed  132.04 sec
    Start 3: rrfs_fv3jedi_2024052700_getkf_solver
4/8 Test #7: rrfs_mpasjedi_2024052700_bumploc ..........   Passed  314.48 sec
5/8 Test #3: rrfs_fv3jedi_2024052700_getkf_solver ......   Passed  208.76 sec
6/8 Test #6: rrfs_mpasjedi_2024052700_getkf_solver .....   Passed  233.86 sec
7/8 Test #1: rrfs_fv3jedi_2024052700_Ens3Dvar ..........***Failed  568.24 sec
8/8 Test #4: rrfs_mpasjedi_2024052700_Ens3Dvar .........***Failed  703.49 sec

75% tests passed, 2 tests failed out of 8

Label Time Summary:
mpi            = 2343.39 sec*proc (8 tests)
rdas-bundle    = 2343.39 sec*proc (8 tests)
script         = 2343.39 sec*proc (8 tests)

Total Test time (real) = 705.64 sec

The following tests FAILED:
	  1 - rrfs_fv3jedi_2024052700_Ens3Dvar (Failed)
	  4 - rrfs_mpasjedi_2024052700_Ens3Dvar (Failed)
Errors while running CTest
Output from these tests are in: /work/noaa/wrfruc/rrfsbot/PRs_RDASApp/242/build/rrfs-test/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

workdir: /work/noaa/wrfruc/rrfsbot/PRs_RDASApp/242

[guoqing-noaa]

@HuiLiu-NOAA Sorry that I used your PR to test the latest RDASApp on Hercules. The failure is NOT due to this PR.

@SamuelDegelia-NOAA
Some ctests failed on Hercules for the latest RDASApp. Do you have time to take a look? Thanks!

[SamuelDegelia-NOAA]

@guoqing-noaa Looks like it was a memory issue. We will probably need a PR to change the ctest configuration on Hercules specifically.

DRPCG end of iteration 18
  Residual norm (18) = 39.64704425434255
  Norm reduction (18) = 0.03950584712510658

  Quadratic cost function: J   (18) = 17693.9201842808
  Quadratic cost function: Jb  (18) = 2989.616654909762
  Quadratic cost function: JoJc(18) = 14704.30352937103

 DRPCG Starting Iteration 19
slurmstepd: error: Detected 1 oom_kill event in StepId=4595349.0. Some of the step tasks have been OOM Killed.
srun: error: hercules-02-53: task 20: Out Of Memory
srun: Terminating StepId=4595349.0
slurmstepd: error: *** STEP 4595349.0 ON hercules-02-53 CANCELLED AT 2025-02-19T14:23:12 ***

[guoqing-noaa]

@SamuelDegelia-NOAA Thanks a lot for the quick response!