Representing datetimes

[pvanlaake]

Consistency in representing datetimes

Moderator

not assigned

Requirement Summary

Throughout the conventions document there are references to datetimes. These should use a consistent representation.

Technical Proposal Summary

Use ISO8601 extended format consistently to represent datetimes.

Benefits

Consistency in the formatting of datetimes will benefit all users of the conventions.

Status Quo

This issue has been flagged in PR #604 on updating section 4.4 of the conventions. At that time it was agreed to make this a separate issue.

Datetimes are mostly used in section 4.4 Time Coordinates but they do appear elsewhere as well, such as in chapter 5 and several appendices. Some examples of different formats used:

Section	Format used	ISO8601
4.4.1	1990-1-1 0:0:0	1990-01-01T00:00:00
4.4.2	1990-1-1	1990-01-01
4.4.2	1992-10-8 15:15:42.5	1991-10-08T15:15:42.5
4.4.2	1992-10-8 09:15:42.5 -6	1992-10-08T09:15:42.5-06
4.4.3	1972-01-01 00:00	1972-01-01T00:00:00
4.4.4	1-7-15 0:0:0	0001-07-15T00:00:00
5.6	2004-06-23T22:00:00Z	Correct!
7.3	1998-4-19 6:0:0 Z	1998-04-19T06:00:00Z

Associated pull request

To be prepared after agreement has been reached.

[larsbarring]

And here is the link to the specific PR conversation: #604 (comment)

[JonathanGregory]

Thanks for raising this issue, Patrick @pvanlaake. I have changed it to enhancement because I think it's a more substantial matter than a defect. It requires some discussion.

To clarify, I believe the issue is concerned with datetimes in time coordinate units strings.

It seems to me that that there are two separate questions:

• Whether the conventions document should always use the same format for datetimes, although there are various possibilities, and the conventions document doesn't express a preference. For instance, leading zeroes in months, days etc. don't have to be included (1998-4-19 6:0:0 = 1998-04-19 06:00:00) and the time doesn't have to be included (it defaults to 0:0:0). If the conventions document permits variants, it is arguable that it's appropriate for it to exhibit some variety.

• If it is agreed that the conventions document should always use the same format, what format should that be? Please could you state your proposal explicitly? You write, "ISO8601 extended format" but I'm not sure what that encompasses or requires. For instance, https://en.wikipedia.org/wiki/ISO_8601#EDTF describes variants in that format, and some of its features are impossible in CF (e.g. uncertainty) or might be inconsistent with UDUNITS syntax (which is the CF convention for units). Since you label 2004-06-23T22:00:00Z in your table above as "correct", I guess that you may mean this format, which I don't think would be an obvious choice, because it's hardly ever used in the CF metadata, where something like 2004-6-23 22:00 is more common. Although UDUNITS allows the T separator, ncdump -t puts a blank by default.

Best wishes

Jonathan

[pvanlaake]

Dear Jonathan,

Thank you for your observations. I'll try to clarify a bit more.

The intent of this issue is to make the conventions document consistent in its representation of datetimes. It is expressly not the intent to modify the permitted formatting of the "units" attribute of time coordinate variables. Whereas the "units" attribute supports multiple formats for expressing the reference datetime, the documentation of the conventions should, in my opinion, apply a consistent representation in its text. ISO8601 is an international standard that offers such consistency and it is easily understood by Man and Machine alike.

The EDTF is an extension, or profile, of ISO8601 and it was developed by the US Library of Congress for bibliographic purposes. As such it is not part of ISO8601. I was not referring to EDTF, but rather to the extended format as opposed to the basic format of ISO8601. The first looks like 2004-06-23T22:00:00Z, which is generally regarded as the most unambiguous and human-friendly, while the basic version of the same datetime would be 22040623T220000Z, i.e. all separators removed.

[JonathanGregory]

Dear Patrick

Thanks for the clarification. My request for clarification was unclear. 😄 I understand that you're not proposing to change the convention itself, but you are proposing to change (almost) all the units strings mentioned in the text and the examples. Is that right? That's what I was assuming when I wrote my comments.

Possibly I am not Man, or Machine, or human-friendly, but personally I think 2004-06-23T22:00:00Z is less clear than 2004-06-23 22:00:00, because the space makes it easier to parse by eye than the T, and the Z is unnecessary.

Best wishes

Jonathan

[taylor13]

Personally, I would find it misleading if all examples in the document conformed to a specific time units string format unless that format were required by the conventions. I'm o.k. with nonuniformity in the documentation format if we allow nonuniformity in the units string itself.

[sethmcg]

I'm strongly in favor of consistently using leading zeroes on all components of the datetime strings in our examples. Although it's not required by the convention, in my experience it can make a big difference for automated parsing. I would even be in favor of adding a recommendation that m/d/h/m/s all be two digits, but if we don't want to go that far, because so many users follow the examples, we can have a significant positive downstream influence if we adhere to that practice ourselves.

Using "T" as a separator between date and time would be a change to the convention. CF currently specifies that the date and time components are separated with a space, so I think strictly speaking, our datetime strings are actually based on RFC 3339, not ISO 8601-2:2019. (However, strictly speaking, CF datetimes are already not ISO8601, and in some cases cannot be, because we allow for non-Gregorian calendars.)

In this case, I think we should stick with consistently using space as a separator. RFC 3339 explicitly allows it for the sake of human readability, and that's very important for netcdf metadata. Plus, y'know, changing it would render all existing files non-compliant, which is something we generally try to avoid doing.

I'm undecided about how consistent we should be about including Z in the examples. I think that the addition of the emphatic recommendation to never ever use timezone offsets has rendered the standard a little incoherent on this front, since we say don't use them but then devote 7 paragraphs to their use, and also you should explicitly declare zero offset even though the default is zero and you shouldn't use a non-zero offset.

[ChrisBarker-NOAA]

This is a trick -- as many (most?) people use examples more that reading the docs, we should pay a lot of attention to examples, and they should represent "best practices" which may be a bit more stringent that official recommendations [*].

Whether the conventions document should always use the same format for datetimes, although there are various possibilities, and the conventions document doesn't express a preference.

A note on that -- since we are all humans, I think there are a lot of cases in the doc where the examples are not carefully chosen among the legal possibilities -- we (I hope!) take care to make sure that the examples are all legal, but I don't think there's been as much care to make sure that the examples represent the range of options. And what I'm proposing is that when someone is willing to do the work (thanks @pvanlaake) we should try to make the examples represent the range of "best practices" (good practices?) which is more confined that "all allowed". Granted, consensus on best procatices may be harder to reach but I think it's worth a try.

But where best practices vary, I'm for having various examples.

So perhaps a definition of "best practices" in this case?

My start on that:

• Use leading zeroes on all components of the datetime strings (thanks @sethmcg)

• Use either "T" or space -- I personally prefer T, but I think this is a case were reasonable people disagree, and we should use both in the examples.

(This is a change to the spec -- but one I think we should make -- I just checked and UDUNITS accepts both, as does python's netCDF4 and cftime libs)

• Always use dashes and colons, not , e.g. "22040623T220000Z" -- which I think is already the case, but it was mentioned.

• As we say now, it's preferable to specify the offset, even when zero -- explicit is always better than implicit, so I think we should do that in all the examples.

Let the debate commence!

[*] I only just noticed that the datetime spec is one place that CF is more specific than simply "what UDUNITS does". but I think we should add the "T" option.

[pvanlaake]

I very much agree with Seth's and Chris's comments on leading by example, especially in the examples but also in the main text. (This applies not just to this issue, but also issues #625 and #626 on the "calendar" attribute and the data type for time coordinates.)

On the formatting, I am fine with RFC3339, i.e. separate date and time parts with a space.

On the time zone designator Z: as per ISO8601 this denotes that the datetime is expressed in UTC (no time zone offset). It should therefore be illegal in the tai calendar (because that's currently 37 seconds ahead of UTC), and used only with the standard and proleptic_gregorian calendars as an alternative to +00.

Again, I am not proposing to make any changes to the "units" attribute but I do propose one change to the conventions: section 4.4.2 currently only describes the UDUNITS formatting rules and I would propose that we add a reference to the validity of ISO8601 or RFC3339 formats (both of which are successfully ingested by UDUNITS) for the reference datetime. Newer software will have support for ISO8601 and RFC3339 and they thus do not have to reformat the reference datetime to match the documented UDUNITS format.

[larsbarring]

Patrick points at that ISO 8601 is specifically (only IIUC) focussing on UTC, which we have to keep in mind when referring to this standard. And I think that RFC3339 does the same. Both these standards are much more far-reaching than what what we need for specifying the datetime stamp in time units.

In the scientific community (and not least funding organisations) there is currently a very strong focus on making data FAIR, where the last two letters, I for Interoperability and R for Reusability, are immediately relevant for this discussion. By adhering to well established standards the Interoperability and Reusability of the data vastly increases.

The changes needed, which @pvanlaake, @sethmcg and @ChrisBarker-NOAA already identified, are to my mind pretty minor and non-intrusive:

• Make sure that a consistent datetime format is used in all examples (where this appears). With this I mean
  • always have leading zeroes in the elements of the datetime stamp.
  • purely for human readability: always use dashes between elements of the date part, and colon between the time elements
  • whether having a "T" or a space, or show both as variation, between the date part and the time part is to me less important
  • whether always showing the full datetime stamp or allow only the date part where relevant is something to discuss and agree on (see comment below).
• In the text make a clear recommendation to follow these formatting guidelines to increase the FAIRness of the data.

Regarding full datetime stamp vs. date stamp only, I would say that CF and UDUNITS goes against well-established good scientific practice when interpreting lack of a datetime element defaults to 0 (for time) and 1 (for day and month). Usually the numeric precision is used an indication of the underlying precision or level of uncertainty, which is how ISO 8601 handles this.

Speaking of UDUNITS, the current version shows a lot of peculiar behaviour and even some outright bugs with respect to how the datetime stamp is handled. I have an updated version in the pipeline awaiting acceptance of some already submitted PRs, but I have no idea if/when they might be merged.

[pvanlaake]

Patrick points at that ISO 8601 is specifically (only IIUC) focussing on UTC

I hope I did not give that impression because it is not the case. ISO8601 deals with representing time in general, UTC is recognised for its pivotal role in civil timekeeping but otherwise it is not treated differently from, say, local time.

The FAIR principles are commendable but they apply to the conventions overall. If there is to be an explicit reference to them in relation to time coordinates, then I would propose to first introduce the principles in section 1.1. Perhaps better left for a separate issue?

[JonathanGregory]

I feel that introducing a concept of "best practice" which isn't the same as "recommendation" would make CF conformance more complicated or confusing. It seems better to me to add more recommendations and make the examples conform to them (Lars says a similar thing arising from FAIR principles).

CF isn't explicit enough about the units syntax, as we've discussed elsewhere. We say it follows UDUNITS, but it's not easy to find out what that is. In 4.4.2 on time coordinate units we are more restrictive than UDUNITS and more explicit than we are for other kinds of units. The format is "y-m-d [H:M:S[ T]]", where "[...]" indicates an optional element.

I'd be happy with recommending that month, day, hour, minute, second (m, d, H, M, S) should have always have two digits (when present). Seth's point that it makes parsing easier is a good one, although you can't depend on a recommendation being followed.

CF 1.13 requires - and : separators (in sect 4.4.2).

CF 1.13 prohibits a non-zero zone time offset T in tai and utc calendars, and recommends against non-zero offsets in all other cases. Since zero is the default, it would be consistent with CF 1.13 to omit the time zone offset (including Z) everywhere in the document (except where we are discussing time zone offsets). I think that's what we should do (it might already be so). CF 1.13 suggests that "a zero offset be stated explicitly to avoid confusion in situations where omitting it might be misunderstood as indicating local time," but that isn't a recommendation.

CF 1.13 doesn't show T as a possible separator between date and time; it shows a space. Software written to process CF data up to 1.13 would be broken by using the T separator. We could introduce T as an alternative to space in CF 1.14. That would be inconsistent with the "Hankin" principle, number 10 in 1.12:

Because all previous versions must generally continue to be supported in software for the sake of archived datasets, and in order to limit the complexity of the conventions, there is a strong preference against introducing any new capability to the conventions when there is already some method that can adequately serve the same purpose (even if a different method would arguably be better than the existing one).

This isn't inviolable, but I think there has to be a strong reason for breaking it. Is there such a strong reason for allowing T, at the cost of tripping up existing software when it processes new data including T? If we do decide to allow it, I think we should recommend the space, for readability and to minimise the inconvenience to existing software.

CF 1.13 shows the time to be optional, and that's always been the case, because the UDUNITS default is 00:00:00. It wouldn't break anything to recommend 00:00:00 to be included and put it in all the examples. However, I don't currently think we should do that, because for a human it's less readable. It's easier not to see the time at all (and understand that its absence means 00:00:00) than to check that all the digits are indeed zero. When the time is shown, it alerts the human reader to the fact that it matters. Machines always pay attention anyway so they don't care.

[ChrisBarker-NOAA]

I feel that introducing a concept of "best practice" which isn't the same as "recommendation" would make CF conformance more complicated or confusing.

Sorry, I wasn't clear as to what I meant -- I am not suggesting that we introduce the concep of "best practice" "officially", or in the doc -- but rather, that those of us working in the docs keep it in mind, and use that as a guide for examples.

It's simply not possible to have examples of every way to do something that's legal, or even every what that's recommended -- so when we choose to put an example in, it's going to be a subset of every recommend practice - so my suggestion is that we be thoughtful about what we do use as examples.

That being said -- I think there may be a recommendation or two that comes out of this discussion anyway.