Skip to content

Efficient std computation using DROP#2715

Open
maqsoodrajput wants to merge 1 commit intomainfrom
drop-mean-square
Open

Efficient std computation using DROP#2715
maqsoodrajput wants to merge 1 commit intomainfrom
drop-mean-square

Conversation

@maqsoodrajput
Copy link
Collaborator

Aim: Implement $E[X^2]$

For efficient ssh_variability computation, the idea is to implement $\sigma = E[X^2] - (E[X])^2$, where $X$ is the monthly data frequency.

Specific to AQUA:

  • Check the data frequency. Make sure that the results are consistent with hourly, daily or monthly frequency.
  • Store the $X^2$ on monthly frequency, catalog entry and able to read via Reader.
  • Unit for $X^2$ term.
  • Tests
  • Update the documentation

The full issue is in AQUA-diagnostics issue #164

@maqsoodrajput maqsoodrajput self-assigned this Feb 23, 2026
@maqsoodrajput maqsoodrajput added improvement Improvement to existing functionality do not merge! labels Feb 23, 2026
@maqsoodrajput
Copy link
Collaborator Author

Hi @jhardenberg and @oloapinivad, would it be fine if I make some additions in Reader, Drop and Timstat modules for the sshvariability diagnostics?

@mnurisso
Copy link
Collaborator

Hi @jhardenberg and @oloapinivad, would it be fine if I make some additions in Reader, Drop and Timstat modules for the sshvariability diagnostics?

as you did as a new statisic it makes sense, we cam choose the name but the strcture looks good

@maqsoodrajput
Copy link
Collaborator Author

Discussion:

So the zos monthly variable in High resolution is 15GB for our historical 1990.

<xarray.Dataset> Size: 15GB
Dimensions:  (time: 300, values: 12582912)
Coordinates:
  * time     (time) datetime64[ns] 2kB 1990-01-01 1990-02-01 ... 2014-12-01
    lat      (values) float64 101MB dask.array<chunksize=(3000000,), meta=np.ndarray>
    lon      (values) float64 101MB dask.array<chunksize=(3000000,), meta=np.ndarray>
Dimensions without coordinates: values
Data variables:
    zos      (time, values) float32 15GB dask.array<chunksize=(10, 3000000), meta=np.ndarray>
Attributes: (12/17)
    Conventions:  CF-1.8
    activity:     baseline
    class:        d1
    date:         19900101
    experiment:   hist
    expver:       0001
    ...           ...
    paramId:      263000
    period:       1990
    realization:  1
    stream:       clmn
    type:         fc
    history:      \n2026-03-09 12:06:44 AQUA💧: Retrieved from IFS-NEMO_histor...

If we store $X^2$ it would be adding 15GB more only for monthly data...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge! improvement Improvement to existing functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants