Use appropriate types in the time coordinate units.

[ChrisBarker-NOAA]

Use appropriate types in the time coordinate examples.

Moderator

TBA

Moderator Status Review [last updated: YYYY-MM-DD]

Just started ...

Requirement Summary

Ideally, the examples in the CF docs should represent not only valid CF, but also "best practices".

This is because many folks follow examples, rather than reading the spec.

Currently, there are a number of examples that do not follow (IMHO) best practices for the time coordinate data type, specifically:

• use of float, rather than double, for the data type.

• use of a floating point type (rather than an integer) when an integer type would be more appropriate.

Technical Proposal Summary

I propose to update the data type used in a number of the examples in the CF doc.

This is a (belated) follow up to this discussion:

https://github.com/orgs/cf-convention/discussions/383

and there was a (now closed) PR: #557

I closed the PR because it was apparent that more discussion was needed, so hence this issue.

I think that, in fact, the discussion was pretty complete -- here's my summary:

• The examples should never use float -- if a floating point type, they should be double.

• More of the examples should use an integer type.

In the discussion there was disagreement -- I think a floating point type is almost never the best option, and others think that floats are often called for.

So the "best practice" is to use either a double or an integer depending the circumstances -- but the docs currently imply that double (or worse, float) is the best option, so we really should scatter some integer examples in there.

NOTE:

A secondary proposal is that we add some text discussing what data type should be used for tiem coordinate variables -- with the pros and cons for each option. At this point, I don't know that I have the bandwidth to write that, but if someone wants to take that on, that would be great.

Benefits

Anyone looking at the doc for examples to copy from will benefit -- and fewer "bad" datasets will be created.

Status Quo

The time coordinate examples include a lot of "float" types where it's not good, and very few integer types where they might be called for.

Associated pull request

#557

NOTE: that PR is currently closed, I'll re-open if the response is positive. It is also more aggressive about changing float to int that I think fits the current consensus -- I'll fix that if/when I re-open it.

It was also written a good while ago, and will probably need updating to the latest version.

Detailed Proposal

See the Discussion / PR

[ChrisBarker-NOAA]

@pvanlaake: I'd like to hear your take on this -- you've thought more deeply about time representation than most of us, and are working on updating examples in other ways as well.

[pvanlaake]

Hi @ChrisBarker-NOAA and others,

I just re-read discussion https://github.com/orgs/cf-convention/discussions/383 and I am mostly cozying up to @sethmcg in my views on the issue: there is a place for both options, considering the context.

I do agree with most of the comments that the float data type is generally a poor choice due to the decreasing resolution over time scales that are relevant to climate research. So out with float as far as I am concerned.

On the use of int versus double as a data type for time coordinates, one thing that has not been mentioned (explicitly) is that the choice of data type (can) reflect(s) the nature of the data: daily observations or model data at daily resolution such as maximum daily temperature or accumulated precipitation are just that: you can't (or at least shouldn't!) interpolate that, to recall @davidhassell 's comment. Time, effectively, is not a continuous property in the context of such data, it's discrete by definition. Using an int data type with an attribute "units" = "days since ..." in this context would convey that quite nicely. Instantaneous values, such as 2m temperature or precipitation rate, at (say) hourly resolution would use a double data type because here it would be reasonable to interpolate values over the time dimension. Aggregating daily data to coarser time scales (month, season, year, climatology) would similarly call for time coordinates in double data type because the "natural" time interval for the underlying data is no longer reflected in the "units" attribute.

This obviously goes beyond merely making some changes to the examples (which I support). Perhaps a paragraph or two can be added to section 4.4.2 to elaborate on the choice of data type, as per the above argument.

[JonathanGregory]

Thanks, Chris. I agree that it would be useful to add some text explaining why float is generally a bad choice for time coordinates (because of precision over the whole range, although sometimes it's adequate) and why int is sometimes a good choice (because of precision if the coordinates are spaced in integer multiples of some unit of measure of time). I agree that the examples should follow such advice, although it's not a recommendation, because it depends on the needs of particular datasets. However, I think we should be careful not to imply that there's any information about the quantity implied by the choice of the unit of measure of time. It's always assumed that data can be converted to other physically equivalent units without changing its meaning.

[ChrisBarker-NOAA]

However, I think we should be careful not to imply that there's any information about the quantity implied by the choice of the unit of measure of time. It's always assumed that data can be converted to other physically equivalent units without changing its meaning.

Well, these is a challenge we've discussed multiple times, but I don't suggest we do anything about it now. There are actually related "things":

(A) How to express "daily" (or monthly, or whatever) data, which is not really on the time continuum.
(B) How to express the precision of the data.

As for (A) -- maybe it would make sense to suggest that the choice of units does tell one something about the data, as @pvanlaake suggested:

daily observations or model data at daily resolution such as maximum daily temperature or accumulated precipitation are just that: you can't (or at least shouldn't!) interpolate that.

But as @JonathanGregory said: " It's always assumed that data can be converted to other physically equivalent units without changing its meaning."

I agree there -- if we really want to do (A) we will need an explicit way to do so.

But I think (B) is OK as a hint, if not rule. And again, as Jonathan says -- it can be converted -- many folks will convert the time coordinate to an actual datetime type anyway -- I know I do.

Anyway, it would be nice to better address those some day, but for the purposes of this issue, all I'm suggesting we do is select good practice examples.

And, I agree with @pvanlaake : add a bit of discussion.

float vs int

As for a float type versus an integer type -- "floating point is hard!"

There are very many confusions out there that bite people in the @$$ a lot, and it seem that in this discussion there is some confusion. So I'm going to try a different terminology:

The question is not whether we should store a time coordinate as a real number or integer in the mathematical sense, but rather, whether the time coordinate should be stored in a floating point or fixed point type? Using integer seconds, or microseconds, etc. is really a version of fixed point -- not "integer" per say (the computer type is integer, but not the meaning) [*].

As mentioned, time is really a continuum -- a real number. But floating point data types are not a more accurate representation of the real number line than fixed point numbers. And I argue that they are a less accurate way in this particular use case.

The power of floating point, and why it's so useful, is that it stores a given precision in the mantissa, separate from the exponent -- so regardless of the order of magnitude of your values, you get the same amount of relative precision (e.g. significant figures).

But that is also the downside -- the level of precision depends on the order of magnitude of the number (in the numerical analysis world, this is what leads to loss of precision due to catastrophic cancellation). And in a CF-style time coordinate this is really bad, as the order of magnitude of the value can vary a lot within one variable:

days since an_epoch

Will have far more precision near the epoch than far from it. If you combine using a floating point type with either a long time series (you know, climate data...) or use an epoch that is far from your data (folks often use 1970 or 1958 for various reasons) then you have some real issues.

However, if you use an integer type and a unit that reflects your desired level of resolution (second, millisecond) then you will get exactly the same precision regardless of how long your time series is, or what epoch is chosen.

Note that resolution is different than precision -- for example, a model may output every hour (resolution) but you still want the timestamps to be precise to the second (or finer!). This is particularly important when you convert to another type or unit -- for a 25 year model run, there may be no practical difference between 12:00:13 and 12:00:00, but it can make a mess of computation. Why throw away that precision if you don't need to?

What about double?

The introduction of the double type (a LONG time ago!) swept a great deal of floating point issues under the rug -- who cares if I lose half my digits when there are 15 to work with? And that applies here -- with a double, you can get second-level precision for over 280 million years. So for practical purposes, any use of CF can use a double and get second precision (or milisecond for 280 thousand).

But why? what you've got is a lot of wasted bits, and it's unclear which of those is meaningful. You can store a lot of time with a uint32 seconds-since variable, and more than you'd ever need with a int64 seconds since.

And many sources for the data, when writing the files, will be in some integer based (fixed point) data type to begin with -- why convert to double?

Well, there is a may be a reason -- human readability:

If you have data in, say, years since some epoch, then you can look at the numbers and they are meaningful -- 10.3235343234134 years means something to folks -- 325562978 seconds, not so much. But years is problematic anyway -- is 3770.6709 days meaningful? I suppose it would make sense if you have a few days of data, and use days, or a few hours, and use hours, and use an epoch near the start of the data -- but is that common?

But at least a double will give you plenty of precision.

I think the case of needing directly human-readable time coordinates is pretty minimal, but there you have it.

My proposal:

1. Put a very much shortened version of the above in the docs, and suggest that:

float never be used

First choice:

• store in a type that matches what the source uses -- e.g. if a model is using integer seconds, use integer seconds. If the data are in a floating point type, use that.

Second choice:

• An integer type that reflects the resolution (and maybe precision) of your data.

Third choice:

• If your data would be more human readable as a double with a larger unit (e.g. hours or days), then use a double.

Thoughts on recommendations for the epoch, if any? I like to use one that's "close to" the data -- but there are good reasons to use a "standard" one as well. Maybe say nothing?

And the examples should have a smattering of the above.

Another question -- say something about using units that make sense for "aggregate" data -- e.g. daily average in "days since"? Or is that too fraught with confusion.

Sorry for the very long post -- much have this has been said (mostly by me) in previous discussions, but I hope it's helpful to have it all in one place.

NOTE: I got interested in this for a very practical reason -- I was working with some model output files that did not natively use CF -- they added a CF-compliant time coordinate as the request of the community. And they used a float data type.

What I noticed is that while the model output was hourly, I was getting timestamps like: 12:00:13, rather than 12:00:00 and if I wanted the results at 12:00:00, it was giving me an out of range error -- not good. Anyway, the problem was that a float simply couldn't hold second-level precision with the epoch they used. This was a real pain in the $%#$.

I don't know for sure, but I highly suspect that they used float because that was an example they saw, which may have come from somewhere other than the CF doc, but as a number of the examples do use float, it very well could have.

As for double -- I mentioned it in the discussion, but for a while (I think that it's now been addressed) xarray would convert double time to integer nanoseconds-since, and then when it had to write it back out, it would write out nanoseconds, which broke a number of workflows, (e.g python cf-time)

So: not enough precision is bad, and too much precision is also bad :-(

[*] there is precedent for fixed point in netcdf/CF -- Thescale_factor is a way to get fixed point with an integer type. It's talked about as compression, but that's what it is.

[JonathanGregory]

Dear @ChrisBarker-NOAA

Thanks for writing this down. I think such advice and suggestions about data types are indeed useful. I agree with you that it's sensible to have a reference datetime near to or within the range of the time coordinates. I'm surprised we don't already say that somewhere, but I can't find it if we do.

I am not sure that this advice belongs in the CF document, because it's not as strong as recommendations. We do some give advice and suggestions (short of recommendations) in the document about how best to use CF, but not much. I wonder if "a very much shortened version of the above" could be something really short like, "In order to preserve the precision of datetimes, it is very important to give careful consideration to the choice of data type for the time coordinate variable, and the choice of reference datetime for the units. This is further discussed in X," where X is some other document on the CF website. The short version should mention any very important points that need attention too. This remark could be highlighted with a WARNING, like the one in section 4.4.3, which was your suggestion if I remember correctly. We could also put X in the bibliography of the conventions document. The document CF-WKT appears in the bibliography as a "supporting document", of that kind.

Would that be a good approach, or do you think it would not be sufficiently prominent?

Best wishes

Jonathan

[ChrisBarker-NOAA]

"In order to preserve the precision of datetimes, it is very important to give careful consideration to the choice of data type for the time coordinate variable, and the choice of reference datetime for the units. This is further discussed in X," where X is some other document on the CF website.

Yes, that would be great, though only useful if indeed there is another document in the CF website.

Is there precedent for this?

NOTE: I think there are a lot of topics that could use an more explanatory document -- in fact, I started a draft/prototype of one about grids here:

https://github.com/ChrisBarker-NOAA/CF_conventions_notes

I was intending to get it more mature before telling everyone else about it.

So if there is not currently a home for such documents, I think we need a proposal about how we want to handle that, and then we can put one in about time.

For the moment, I'd like to keep (my) momentum going, and focus on updating the examples, but in order to do so, we need a consensus on what we should put in the examples.

I'll not use the contenius term "best practices", and instead use "what we should use in the examples".

See my previous list for what I think we should use as examples.

If there is general agreement, I'll work on starting a PR, and I think we it comes to the specifics, a PR is the best way to discuss it.