Support Direct Loading of NetCDF Time-Series Data Without Conversion

[j6k4m8]

Continued from slack discussion [deeplink]

Currently, mllam-data-prep netcdf loaders require data to be in single "long" files (i.e., all timestamps in one data element on disk).

In contrast, my dataset consists of NetCDF files with time-series data, where each file represents a single measurement (Dims = [x, y, time], with time always being a single value). Instead of concatenating these files manually, I’m exploring ways to load them directly using a more flexible datastore approach.

Proposed Solution

• Introduce a method to glob NetCDF files in the YAML config, mapping timestamps from filenames to a proper time dimension.
• Alternatively, improve the existing datastore or document using Kerchunk to create a reference-based dataset without redundant copies.

Related Discussions

• Kerchunk-based solutions
• Existing notebook on NetCDF-to-Zarr conversion

Next Steps

• Determine if a more flexible datastore is needed or if an improved documentation approach (e.g., a tutorial) would suffice.
• Evaluate performance trade-offs of different loading methods.

Would love input from others working on similar datasets!

[leifdenby]

Thanks for typing up this issue @j6k4m8!

I have been thinking that we should maybe offer some more options around the file loading in the config.

Maybe we could add some new optional sections to the input definitions:

inputs:
  danra_height_levels:
    path: /my/path/files*.nc
+   open_with:
+     method: "xr.open_mfdataset"
+     kwargs: ...
...

I've been thinking something similar could allow people to supply a S3 bucket credentials in cases where that is required, e.g. something like:

inputs:
  danra_height_levels:
    path: s3://bucket-name/dataset.zarr/
+   open_with:
+     kwargs:
+       storage_options:
+         profile: ewc-lam
+         endpoint_url: "https://object-store.os-api.cci1.ecmwf.int"
...

In the latter example I am trying to use my S3 credentials stored in ~/.aws/credentials to connect to the S3 server at https://object-store.os-api.cci1.ecmwf.int

What do you think to this? This would allow you to experiment with xr.open_mfdataset at least :)

[j6k4m8]

Oh I definitely love the idea of s3-style access! maybe there's a more general form that doesn't explicitly require a top level path,

inputs:
  my_cool_dataset:
+   backend: filesystem
+   kwargs:
+     path: /my/cool/dataset

inputs:
  my_cool_dataset:
+   backend: s3
+   kwargs:
+     path: s3://bucket-name/dataset.zarr/
+     endpoint_url: ...
+     profile: ...

since that could enable things like globbed files:

inputs:
  my_cool_dataset:
+   backend: filesystem
+   kwargs:
+     glob: /my/cool/dataset/*.cool.zarr

---

and maybe even support for BYO loaders,

inputs:
  my_cool_dataset:
+   backend: 'python_module:MyClassName'
+   kwargs:
+     # arbitrary arguments to `MyClassName(**kwargs)`
+     foo: bar

although maybe that starts to step on the toes of the other dataloaders :)

[leifdenby]

since that could enable things like globbed files:

inputs:
my_cool_dataset:

• backend: filesystem
• kwargs:

• glob: /my/cool/dataset/*.cool.zarr
  

You can do globs with the example I gave :) As in, what I was intending was to allow the user to specify that they want to use xarray.open_dataset(...), xarray.open_zarr(...) or xarray.open_mfdataset(...). And with xarray.open_mfdataset(...) you can give a glob pattern in the path (https://docs.xarray.dev/en/stable/generated/xarray.open_mfdataset.html).

So your example would become:

inputs:
  danra_height_levels:
    path: /my/cool/dataset/*.cool.zarr
    open_with:
      method: "xr.open_mfdataset"
...

What do you think to that?

and maybe even support for BYO loaders,

I would like to start with just the functionality in xarray I think, hopefully that will be enough :) But we could in principle allow people to provide any function name they'd like, as long as it returns an xr.Dataset with the provided path argument and kwargs.

[observingClouds]

I like:

inputs:
  danra_height_levels:
    path: s3://bucket-name/dataset.zarr/
+   open_with:
+     kwargs:
+       storage_options:
+         profile: ewc-lam
+         endpoint_url: "https://object-store.os-api.cci1.ecmwf.int"

This would give more flexibility to the fsspec-supported file-protocols that require additional kwargs.

@j6k4m8 I am just experimenting with kerchunk references over at #88 though on the output end. With virtualizarr/kerchunk you should be able to quickly generate a kerchunk file that concatenates your netCDF files and can then be used as an input. If @leifdenby suggestion gets implemented, it should be as easy as:

inputs:
  danra_height_levels:
    path: reference://
+   open_with:
+     kwargs:
+       storage_options:
+         fo:"/path/to/kerchunk.json"

Having the datastore completely virtualized is challenging because it poses chunksize requirements and derived variables, like "time of day", require to write partial data. See #88 for more.

[leifdenby]

I like:

Thanks for engaging with this @observingClouds! I think what is maybe needed is as well as the implementation allowing the inclusion of kwargs are examples of opening different collections of source data (single zarr dataset, single netcdf, multiple netcdfs). If there were examples like in the README that then that might help, what do you think @j6k4m8?

Also, @observingClouds what do you think about the following two things:

1. Defaulting to use xr.open_dataset() rather than xr.open_zarr(), is there any risk here? The main one I can think of is that by default we should probably provide chunks={} as a default argument if the provided path looks like we're opening a zarr dataset (i.e. ends in .zarr or .zarr/) so that xarray uses the zarr chunking for the xarray chunks. As I read xarray.open_dataset by default xarray will otherwise not use dask which I don't think is disirable here, does that ring a bell and make sense?
2. Using the presence of * in the path to call xr.open_mfdataset(...) rather than xr.open_dataset(...). That would allow people to use glob patterns when wanting to open multiple netcdf. Of course this doesn't resolve the situation where xarray doesn't know how to combine the individual datasets from each file (as in, if there is no common coordinate with different coordinate values). Here I would usually provide a function with the preprocess=... argument that then adds the missing coordinate (usually these will part of the path), but I going that far here is probably overkill. I would say if people struggle with using multiple netCDF datasets with xr.open_mfdataset(provided_path) then we should probably point them to creating a kerchunk-based json-index first and then passing the path to that to mllam-data-prep (as you demonstrated above). Thoughts?
   • I think I still prefer an explicit indication to use xarray.open_mfdataset as I originally proposed though. Just interested in people's thoughts

[observingClouds]

@leifdenby

1
Yes, this makes sense and I would support using xr.open_dataset instead of xr.open_zarr as a default.

2
Using xr.open_mfdataset seems fine too

Is xr.open_mfdataset not sufficent (at least if we focus on xarray data loaders)? xr.open_mfdataset allows to provide a single path as well and covers all (?) cases xr.open_dataset and xr.open_zarr cover, making it unnecessary for us to add a method argument to the config. But maybe I miss something here.

[leifdenby]

Is xr.open_mfdataset not sufficent (at least if we focus on xarray data loaders)? xr.open_mfdataset allows to provide a single path as well and covers all (?) cases xr.open_dataset and xr.open_zarr cover, making it unnecessary for us to add a method argument to the config. But maybe I miss something here.

Ah good point! Then there's just the question of the chunking argument. xr.open_zarr() defaults to chunks={} if dask is installed, whereas xr.open_dataset() defaults to chunks=None which skips using dask. It makes sense to just use dask if it's available