Save custom .attrs in SpatialElements to disk

[lucas-diedrich]

Hi! Thanks for the great package!

I am currently working with spatialdata to load and parse microscopy files (particularly in the proprietary Carl Zeiss .czi format). For this purpose, it would be extremely useful to persistently store metadata related to the acquisition of the microscopy data (e.g. magnification, resolution, etc.) in the .attrs attribute of the SpatialElement. While I can manipulate the Elements in memory, it is currently not possible to write them to disk (see example)

While it would be possible to write the metadata to a custom tables object, it feels much more natural to have this data directly associated with the image.

Therefore, it would be fantastic if SpatialElements would support writing the .attrs keys to disk.

Example

import spatialdata as sd
import numpy as np 
import tempfile
import os

dir = tempfile.tempdir
save_path = os.path.join(dir, "blobs.zarr")


# Load dataset and set custom metadata
blobs = sd.datasets.blobs()
blobs.images["blobs_image"].attrs["metadata"] = {
    "info1": 1,
    "info2": "2",
    "info3": np.zeros(shape=(3, 3))
}

sd.models.Image2DModel().validate(blobs.images["blobs_image"])

# Metadata is in memory
assert "metadata" in blobs.images["blobs_image"].attrs

# Writing to disk leads to loss of the information 
blobs.write(save_path)
sdata = sd.read_zarr(save_path)
assert "metadata" not in sdata.images["blobs_image"].attrs

[LucaMarconato]

Hi, thanks for reaching out. In principle we could extend the way in which we write SpatialData.attrs to disk

spatialdata/src/spatialdata/_io/io_zarr.py

Line 141 in 03dbf80

# read attrs metadata

also for the various elements (shapes, points; for tables as you observed the uns are already serialized).

What do you think @giovp @timtreis @melonora?

The only issue that I may see is that in our APIs we currently only take care of forwarding the metadata that we define in the format (for instance after cropping an element, the information on the coordinate system are passed to the new cropped object). Not sure if we should ignore the rest of values of .attrs and have the user manually copying them, or if there should be an automatic function. In such case, should we shallow copy or deep copy?

[LucaMarconato]

Also, it is a good idea to explore what libraries like pandas, xarray and dask do. To this regard, the situation is a bit unfortunate:

• pandas doesn't have a fully specified/clear semantic: DEPR: attrs pandas-dev/pandas#52166. Also relevant: BUG: Copy attrs on pd.merge() pandas-dev/pandas#60357
• for dask, the situation is more unfortunate. The new dask-expr backend for dask dataframes removed .attrs, change that is currently giving us headaches 😅 Dask 2024.5.1 removed .attrs dask/dask#11146
• for xarray we didn't have problem so far, but one needs to look deeper into the code.