cftime.DatetimeNoLeap incorrectly decoded from netCDF file

[TomNicholas]

What happened?

I have been given a netCDF file (I think it's netCDF3) which when I open it does not decode the time variable in the way I expected it to. The time coordinate created is a numpy object array

What did you expect to happen?

I expected it to automatically create a coordinate backed by a CFTimeIndex object, not a CFTimeIndex object wrapped inside another array type.

Minimal Complete Verifiable Example

The original problematic file is 455MB (I can share it if necessary), but I can create a small netCDF file that displays the same issue.

import cftime

time_values = [cftime.DatetimeNoLeap(347, 2, 1, 0, 0, 0, 0, has_year_zero=True)]
time_ds = xr.Dataset(coords={'time': (['time'], time_values)})
print(time_ds)
time_ds.to_netcdf('time_mwe.nc')

<xarray.Dataset>
Dimensions:  (time: 1)
Coordinates:
  * time     (time) object 0347-02-01 00:00:00
Data variables:
    *empty*

ds = xr.open_dataset('time_mwe.nc', engine='netcdf4', decode_times=True, use_cftime=True)
print(ds)

<xarray.Dataset>
Dimensions:  (time: 1)
Coordinates:
  * time     (time) object 0347-02-01 00:00:00
Data variables:
    *empty*

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

cftime 1.6.2
netcdf4 1.6.4
xarray 2023.8.0

[TomNicholas]

This bug is causing me pain, especially in the index. For the array I can just do ds['time'].astype('datetime64[ns]') but I can't actually work out how to coerce the type of the array backing the CFTimeIndex object after it's been created.

I would prefer to fix the bug, but I actually don't know where in xarray to look. Perhaps @kmuehlbauer you might know where to point me? :)

[spencerkclark]

@TomNicholas sorry for not commenting on this earlier. As far as I can tell this is just an issue with the HTML repr, i.e. it should not affect indexing or otherwise interacting with the time coordinate (ds.indexes["time"] returns a CFTimeIndex). It's definitely something we should fix, but it should not require any type coercion to work around.

[TomNicholas]

Thanks @spencerkclark

this is just an issue with the HTML repr,

Oh really?? So I'm trying to coerce unecessarily? But is the dtype of the array backing the CFTIndex supposed to be object?

[TomNicholas]

Like this looks like the wrong type of array

array([cftime.DatetimeNoLeap(347, 2, 1, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(347, 3, 1, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(362, 1, 1, 0, 0, 0, 0, has_year_zero=True)],
       dtype=object)

[dcherian]

That's correct :)

[spencerkclark]

Yes, this is expected. cftime.DatetimeNoLeap objects are not a NumPy dtype, so they have to be stored in an object dtype array.

As an aside, if you need to coerce to datetime64[ns] (though it obviously will not work in this example) I would recommend using ds.indexes["time"].to_datetimeindex(), as ds["time"].astype("datetime64[ns]") is vulnerable to overflow. See also #5107.

[keewis]

(as an aside, numpy has support for writing external dtypes so cftime could make use of that, though it might be worth waiting until the string dtypes in NEP 55 are polished and accepted into numpy's main)

[spencerkclark]

The repr looks similar for other types of pandas indexes too, so I'm guessing this was at some level intentional (or some flexible index internals are leaking through the repr):

Either way it could make sense to revisit that given the confusion.

[kmuehlbauer]

@TomNicholas Sorry Tom, was already AFK. Would also have referred to @spencerkclark's expertise, though.

[TomNicholas]

Thank you everyone for your help! ❤️

I am still confused about what the difference between

cftime.datetime(1999, 1, 1, 0, 0, 0, 0, calendar='noleap', has_year_zero=True)

and

cftime.DatetimeNoLeap(347, 2, 1, 0, 0, 0, 0, has_year_zero=True)

is, but I was now able to do the operation I needed to do.

---

Should we leave this open to track the repr not being super clear, or just close?

[dcherian]

I think we can close.

Though this could be a good opportunity to update our docs to be more clear about CFTime handling :)