Simulated variables not considered in LETKF analysis

[SamuelDegelia-NOAA]

Current behavior (describe the bug)

I am experiencing a problem where simulated variables processed by running LETKF in the split observer/solver mode are not considered in the analysis. I tested this on ATMS brightness temperature obs wherein the observer has obs passing QC but the solver produces zero increments.

Looking into the diag files a bit, I find that the ObsError group has realistic looking error values but the DerivedObsError consists of only masked data. Replacing the values in DerivedObsError with those from ObsError before running the solver leads to reasonable increments.

Question: The DerivedObsError group is initialized as all masked values, but should this group be updated with the values from ObsError? If not, is there a way to tell JEDI to look at ObsError instead of DerivedObsError?

Steps to Reproduce (if applicable)

What computer are you running on?

Hera

Steps to reproduce the behavior

1. Run LETKF on only ATMS obs
2. Inspect increments

Example run directory: /scratch2/BMC/zrtrr/Samuel.Degelia/RDASApp_reduceobspace/RDASApp/expr/mpas_2024052700_radianceenkf

Expected behavior

Nonzero increments from assimilating ATMS obs

Acceptance Criteria (Definition of Done)

I am not sure if this will need a change to the JEDI code or if this can be solved just through changes to the yaml file. TBD.

[keenaneure]

I am experiencing the same issue with zero increments for some preliminary ATMS tests. However, I still see this with the default Ens3DVar configuation as well.

[SamuelDegelia-NOAA]

The issue I am experiencing is due to the ObsError group not existing in the input data which causes some issues with the LETKF (details here).

So far I have not had any issues with ATMS obs in our Ens3DVar test. Do you know if the obs are passing QC? You can check the output for lines like this:

  0: QC atms_npp brightnessTemperature_9: 75 out of bounds.
  0: QC atms_npp brightnessTemperature_9: 57 out of domain of use.
  0: QC atms_npp brightnessTemperature_9: 6062 removed by thinning.
  0: QC atms_npp brightnessTemperature_9: 57 passed out of 6251 observations.
  0: QC atms_npp brightnessTemperature_10: 75 out of bounds.
  0: QC atms_npp brightnessTemperature_10: 57 out of domain of use.
  0: QC atms_npp brightnessTemperature_10: 6062 removed by thinning.
  0: QC atms_npp brightnessTemperature_10: 57 passed out of 6251 observations.

[keenaneure]

Thank you for pointing me in that direction. From what I see, very few observations are passing (what I assume is) the domain checks. My total observation count is also drastically different here:

0: QC atms_npp brightnessTemperature_9: 740811 out of bounds. 0: QC atms_npp brightnessTemperature_9: 1539 out of domain of use. 0: QC atms_npp brightnessTemperature_9: 13978 removed by thinning. 0: QC atms_npp brightnessTemperature_9: 1688 passed out of 758016 observations. 0: QC atms_npp brightnessTemperature_10: 740811 out of bounds. 0: QC atms_npp brightnessTemperature_10: 1539 out of domain of use. 0: QC atms_npp brightnessTemperature_10: 13978 removed by thinning. 0: QC atms_npp brightnessTemperature_10: 1688 passed out of 758016 observations.

Most of the 22 channels have similar numbers for the amount that pass. Even with the obs that pass, I have yet to find a model level with nonzero increments. My experiment YAML was produced with gen_yaml.sh, and the domain check section does not have anything special at the moment.

When I check ncdump, it also shows missing values in DerivedObsError/brightnessTemperature:

[SamuelDegelia-NOAA]

@keenaneure The difference in the total observation counts are probably due to using the non- domain-check version of the ATMS IODA files. The ctests use atms_npp_obs_2024052700_dc.nc where the dc prefix indicates the offline domain check that removes all obs outside the domain (using the smaller FV3-JEDI domain). There is no problem using the normal version without the offline domain check, but we are using the dc version for the ctests since it is easier to compare between FV3-JEDI and MPAS-JEDI.

But it is still odd that you have obs passing QC in EnVar but still zero increments. Would you mind sharing the path to your yaml so I could have a closer look?

[keenaneure]

That makes sense. I can make that switch to the dc version.

Here is the pathway to where I am working:
/scratch1/BMC/wrfruc/eure/RDASApp_20250131/expr/mpas_2024052700_atms/conv.yaml

(despite the 'conv.yaml' name, it is still indeed for ATMS NPP obs)

I am curious if @xyzemc has seen this issue with ATMS obs before.

[SamuelDegelia-NOAA]

@keenaneure It looks like you are trying to use 3Dvar instead of 3DEnVar like we use for the ctest. See the background error: section of your yaml compared to the one here. But the path to the SABER files in your yaml is just the path to our BUMPloc files, so they might not contain all of the static B information needed for 3Dvar.

[xyzemc]

That makes sense. I can make that switch to the dc version.

Here is the pathway to where I am working: /scratch1/BMC/wrfruc/eure/RDASApp_20250131/expr/mpas_2024052700_atms/conv.yaml

(despite the 'conv.yaml' name, it is still indeed for ATMS NPP obs)

I am curious if @xyzemc has seen this issue with ATMS obs before.

@keenaneure I have the result of assimilating atms recorded in the issue #203 It seems I have the analysis increment at the location of assimilating atms data. However, when I checked my HofX file, I found that the DerivedObsError/brightnessTemperature field contains the same missing values as you showed.

[keenaneure]

Thank you for the help, Sam. I get increments similar to Xiaoyan's with the ATMS tests. However, I reproduce the same issue when I attempt it with the LETKF.

[SamuelDegelia-NOAA]

Thank you for confirming @keenaneure! I have a solution to the LETKF problem but it will require reprocessing some of the input data. I will update this issue when that has been handled.

[SamuelDegelia-NOAA]

Update: our current method for creating observation errors in JEDI involves not converting any errors and instead using the Perform Action filter with the assign error filter action. This has worked fine for conventional variables (t, q, ps, uv). But for simulated variables (e.g., brightness temperature), JEDI initializes an empty DerivedObsError group if there is no ObsError group in the input IODA data. This empty group is causing the issue described here.

The LETKF treats obs errors differently from Var which reads the EffectiveError group. For LETKF, the ObsError is normally used instead. However, this empty DerivedObsError group is apparently taking priority over ObsError for simulated variables. Sine DerivedObsError is always empty, these simulated variables are effectively not assimilated in LETKF despite successfully passing the observer.

The easiest solution here is to include an empty ObsError group in the input IODA data for simulated variables. In that case, the empty DerivedObsError group will never get created and the ObsError group will be used during the LETKF solver. I have tested this by both adding ObsError in the bufr2ioda converter and/or adding ObsError via the offline domain check. After making those changes, the LETKF successfully assimilates radiance obs and the increments look realistic.

The bufr2ioda converters for radiance data are still in active development and are not ready to be committed to RDASApp. Thus, the easiest solution for us is to add ObsError via the offline domain check.