Derived variables for forecast data

[joeloskarsson]

It would be highly useful to also be able to derive variables for forecast datasets. The difference with those is that they don't just have a time dimension, but an analysis time and a lead time dimension.

I don't think there is anything inherent in the way derived variables are computed that prevent doing it for forecast datasets, it is just that the current derived variables do not allow for this (analysis time, lead time) pair of dimensions as input (and correctly using these).

[joeloskarsson]

I implemented one way to allow for this here: sadamov@b1901a1 It is quite easy thanks to the flexible design of derived variables are computed 😄

It would be great if someone wanted to take that and turn it into a full-fledged PR. It might need some cleanup + it would be great to get @ealerskans's input on it.

[AdityaGupta716]

@joeloskarsson I am working on this , will base it off sadamov/mllam-data-prep@b1901a1 and will open a pr soon

[leifdenby]

@joeloskarsson I am working on this , will base it off sadamov/mllam-data-prep@b1901a1 and will open a pr soon

I think it would be useful when doing this to consider how we could create a single time argument derived from the lead_time and analysis_time components, and that way not having to change the function signature for all functions creating derived fields. This could for example be done by allowing the config argument that currently looks like this:

...
    derived_variables:
      # derive variables to be used as forcings
      toa_radiation:
        kwargs:
          time: ds_input.time
          lat: ds_input.lat
          lon: ds_input.lon

to allow for just the sum-expression (+) to indicate that two coordinate should be added, like so:

    derived_variables:
      # derive variables to be used as forcings
      toa_radiation:
        kwargs:
          time: ds_input.time + ds_input.lead_time
          lat: ds_input.lat
          lon: ds_input.lon

the time argument would then still be a string, but you would see if contains spaces, split the string, assert three parts, check that second one is + and then resolve the two xr.DataArrays to add and then pass their sum in as the named argument. What do you think of this?

[AdityaGupta716]

hey @leifdenby! @joeloskarsson's implementation added lead_time as an optional kwarg to each function and did time + lead_time inside the function itself,
so the config would just be:
yaml
time: ds_input.analysis_time
lead_time: ds_input.lead_time

works with the existing kwargs resolution as is, no changes needed there.
with the + in config approach i feel like it could get messy — any typo or missing space would only fail at runtime, and since kwargs is Dict[str, str] the parsing would be this hidden thing in main.py that's not obvious at all when reading the code.
was there something about joeloskarsson's approach that didn't work for you? just wanna make sure i'm not missing something