CF Roadmap: Kicking off work on Provenance and Lineage theme

[erget]

Topic for discussion

Do you have ideas around data provenance and lineage, and would you like to be involved in ensuring that these themes are represented in a way that's beneficial to the CF Roadmap? This is the discussion for you.

At this year's CF Workshop we divided the roadmap preparations into multiple themes. This is one such theme. As you can see from the table, right now we're working on a green field with a blue sky - nothing really more than 2 keywords to start with. Behind that are a whole bunch of ideas and opportunites to shape where we go with it.

I'm setting up a call for later this month. If you want to be involved, please indicate your availability in this survey by the end of 14 Nov 2024.

Topics:

• what are our core concerns?
• who needs to be involved?
• what other technologies / projects should we be aware of and might we align with or adopt?
• anything else we should be aware of?

From there we can agree how to organise the work moving forward.

If you're not available on those dates but still want to be involved, that's no problem - let me know in the comments or drop me a mail and I'll keep you looped in :)

@pagecp

[JonathanGregory]

Does "provenance and lineage" mean discovery metadata?

[erget]

No, this is focused on being able to inspect data and find out how it was made.

[DocOtak]

Would this possibly be a revival of work on ACDD? I've also come across W3C's PROV a few times.

[erget]

I don't think it's got a good connect to ACDD, because it's not discovery but rather lineage. I like W3C's PROV a whole lot and agree that it seems like a good fit but we should analyse and verify.

[JonathanGregory]

Is it related to cell_methods, and the ways in which cell_methods needs to be enhanced, overhauled or replace e.g. to describe climatologies and anomalies wrt climatology? These matters are being discussed in Discussion 305 opened by @TomLav, following discuss issue 252, opened by @sethmcg, now superseded.

(The present discussion so far reads a bit like Twenty questions. 😃)

[sethmcg]

I think it is at least partially related, yes.

[larsbarring]

My understanding is that many of the mentioned issues and external resources have some overlap. I think that Daniel @erget captures it well:

... focused on being able to inspect data and find out how it was made.

At this general level the W3C's PROV that Andrew @DocOtak points at is the comprehensive resource. The abstract of the web page states:
"Provenance is information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness."

For users requiring this kind all-embracing machinery, maybe the best thing CF can do is to point at PROV in the conventions document, and by including a suitable string in theConventions global attribute.

Personally I think that the 2024 CF Workshop presentations by David Huard here, in the Uncertainty session, and by José Manuel Gutiérrez here, in the Statistical Processing session, offers a lot of food for thought.

Many users do not need e.g. manual quality judgement, expert group voting, and much more, but still do need something more than current cell_methods can offer. As Jonathan @JonathanGregory writes

... ways in which cell_methods needs to be enhanced, overhauled or replace ...

I think that by enhancing the cell method machinery we can cover several existing use cases, but I also think that we need to come up with something new that complements cell methods to cover even more complex use cases and data manipulation/processing. I have for some time been playing with the idea of creating some kind of "pseudo-language". That is, something similar to what is now used for parametric vertical coordinates but with freedom to also describe the equations. Well, still just some wild ideas without much substance behind ....

[erget]

Yes, in the end I reckon it's likely lthat PROV is the spec to use, not to re-invent - but it's still not clear to me how users should do that in a netCDF file, or if we think beyond netCDF in other CF-compliant data. So providing some guidance here so that there's 1 clear way to do it would probably be good. That's where I'm thinking the discussion will go - not necessarily rounding this out for before the roadmap exists but in the roadmap we can set forth what approach we want to use.

[larsbarring]

I fully agree! 👍 +1:

[sethmcg]

Over in discussion #305, we reached the conclusion that there are two levels of provenance to consider: there's the level that tells you how the data was changed such that you can understand what it means now, and there's the level that tracks what happened to the data such that you could completely regenerate it from the original sources by following the chain of provenance.

The first is what we're trying to figure out how we can accomplish with an overhaul to cell_methods. The second is like what gets recorded in the history attribute, but with maybe even more detail, and is (I believe) what we're contemplating in this discussion. The important distinction between the two is that the second is much, MUCH more complicated and detailed than the first, and involves not just abstract geophysical quantities, but entities like files and software and versions and configurations.

I agree that it's good to avoid re-inventing the wheel, and that it's probably best to make use of an existing spec like PROV, but if we're talking about recording a chain of provenance that is sufficiently detailed and specific to let you regenerate the end result completely, I would argue that it should not be mandatory, and definitely should NOT be incorporated into the netCDF file itself, because it can very rapidly become entirely unmanageable.

Recording every step in detail works fine if you're talking about a relatively short and linear chain of derivation, but if it's something more complex where it's branched or if you have a large number of input files, then things start to get out of hand really quickly. If you have multiple files as inputs, you have multiple history attributes (or the equivalent) to track / merge / reconcile / deal with. NCO takes the approach of adding them all to a history_of_appended_files attribute, and if you have more than just a couple input files, you end up with a gigantic header that consists almost entirely of history attributes.

This is not an idle concern; I've already run into problem cases where I had to turn off all history tracking while post-processing model output because I had files generated from hundreds if not thousands of parent files, and the resulting histories were preposterous. And that's just for doing data cleanup to make model output CF-compliant; the problem gets exponentially worse if you're doing things like bias-correcting data or calculating derived quantities with multiple inputs and intermediate outputs. (And if you really want to have nightmares, consider what happens if you do any of this in parallel by splitting files apart then recombining them...)

So I think this is an important concern for the roadmap discussion, and that we want to make sure we're not navigating toward the iceberg of metadata that's bigger than the data it describes.

[erget]

I'm totally on board with your concerns here - this is not something to be underestimated. I've been involved in some work in this area that I think might be useful for the community, because we tried to tackle many of the problems that you're referring to. I'm not claiming this is the only way to do it, or that the solutions we developed were bulletproof, but I think that they could be a useful starting point.

For the purpose of the roadmap the important questions in my mind are what we consider in the scope of our roadmap, i.e. what is important, where do we want to develop a stance, and what do we know about how that would look. And then some kind of horizon when we want to be there and whom we engage with.

More than willing to share the mysterious docs I'm referring to but I'd prefer to do that at a time when we meet - surely there'll be a time before end of Q1 2025 - so I can provide some context and we can discuss, otherwise it turns into a data dump.

But yes, my focus for this discussion is more on the history attribute rather than cell_methods, but I could imagine there are synergies there - I'm not an expert on cell_methods in any way!

[erget]

Hi folks, particularly @sethmcg and @pagecp - the survey has spoken, unfortunately there's no appointment for the near-term that fits for all of us but @sethmcg we'll fill you in and hopefully you can join later.

Anybdoy who wants on the invite let me know and I'll add you. @pagecp you're already on it. It'll be 2024-11-27T13:00Z on Teams.

[sethmcg]

@erget Go ahead and send me an invite. It's possible I'll be awake early enough to hop on.

[feggleton]

@erget do you have any notes from the meeting that you can share?

[erget]

Hi @feggleton , @sethmcg - sorry, I got a bit behind here in responding as I was out out office for a bit.

In the end it was just me in the meeting room so I figured I'd reach out again after New Year's when scheduling's a bit easier and give it another go. Hopefully participation will be easier then as well. I take it you're both keen to be involved? Then I'll reach out again and check your availability before setting a time.

[sethmcg]

@erget Yes, i'm keen to be involved. Thanks!

[JonathanGregory]

I'm interested too, especially to see whether it interacts with cell_methods.

[erget]

Hi folks, we had an appointment last year to roadmap what we want to do with provenance and lineage... In the end it was a veritable echo chamber - I was the only one there! 😱

Do we want to try it again?

Mark your availability here by EOB on 12 Feb (next Wednesday) and we'll set a date.

[sethmcg]

Hi Daniel,
I'm interested, but it looks like you're 8 hours ahead of us, which makes scheduling pretty difficult.

I have a couple specific concerns, so maybe it would make more sense for me to just contribute to online discussions?

[davidhassell]

Hi Daniel, I've not expressed an interest before, but I'd like to join now. I've filled in my availability.

I have acquired an interest in how uncertainty can be recorded (if at all!) from different stages of previous data processing. Regardless of the answer to that, it's be great to see progress on this issue.

[erget]

@sethmcg of course - yes and sorry about the time zone issues, I think there's a few slots in the afternoon that might fit your schedule so you're welcome to join if you can. Otherwise yes, it's very useful if you join the online discussions - we need to get things kicked off though and talk about what our priorities are, and who does what, and therefore it makes sense I think to get the working group together before we do the bulk of the work here on GitHub.

@davidhassell - excellent, glad to hae you on board! Will look forward to working with you on it :)

[erget]

Hi all, the survey has spoken - we'll meet on 20 Feb 2025 at 15 UTC. @davidhassell @JonathanGregory @larsbarring I've put you on the invite. Anybody else who wants to be involved send me an email and I'll rope you in!

[erget]

We met yesterday (sorry @sethmcg for leaving you out in the dark, that was definitely not intentional and it's my fault we missed out on you 😥) and had a first meeting to discuss our intents. We are meeting again on 12 March at 16 CET - if you aren't on the invite and would like to be, let me know and I'll add you.

What use cases did we discuss?

We identified the following reasons why somebody using CF data may want to have provenance baked in:

1. For users of data: Checking if the products in their hands are the latest and greatest version
2. For upstream providers: Being credited when their products are incorporated into downstream products
3. For data providers: Potentially restoring processing chains, potentially including processors and data used

Not all of these use cases need to be fulfilled for things to be useful. The W3C PROV standard provides tools that service all of them.

What do we propose incorporating into the CF Roadmap?

In this order:

1. Being able to put in provenance records into CF data. This might be a point-out to an external file. It might be embedded in the data file itself. Dunno.
2. Linking provenance records in different CF files so that we can have chains that aren't overburdened by metadata.
3. Cryptographically signing it so you can be the nonrepudiable, authoritative author of data products.

Any design principles we want to propose?

We want to re-use PROV rather than adapt it or invent something, because using an existing standard has a lot of advantages. The issue that we see here is how to represent the provenance data in a way that works with CF so we don't have a multutide of implementations.

What are we doing between now and the next meeting?

I made some technical documentation available to the people in the meeting to mull over. This is related to a prototype implementation that we had at EUMETSAT for inspiration, it's not something that we could just take off the shelf and use. So we're thinking about

1. Do we have the right people involved?
2. Are there other communities that we should engage with?
3. Are there other examples we could learn from?
4. Is the PROV data model indeed what we want to use?
5. How could the PROV data model be used in the CF data model?

In particular, Lars kindly volunteered to attempt to provide an overview of uses of PROV in IPCC and other settings, this may or not be ready at that point.

Looking forward to seeing you all at the next meeting :)

[davidhassell]

Clearly not a use case for on the fly processing because there is overhead in describing a dataset.

Possibly a use case for the upcoming aggregation conventions provide a way of recording aggregations of files in a CF-netCDF file without copying the data, and removing the need to repeatedly apply your aggregation algorithm every time you access the aggregated dataset.

[TomLav]

Described at the file level, it's more than 66MB of metadata just for the filenames.

Files pertaining to a dataset often have filenames following a convention. The commands run on these files are often the same. I wonder if a simple grammar with regexp and loops could compress most of this repeated information in much fewer characters. This would be less human-readable.

Some time ago I made the loop_wrapper tool. We still use it in our internal processing of files across dates. Maybe documenting the processing of zillion files could be stored in a loop/regexpr form in PROV instead of explicitly. For example:

loop_wrapper --every 1M 18500101 20200101 ./command -t -H -a auxFile.nc -f dataFile_{d:%Y%m}15.nc

would actually stand for

./command -t -H -a auxFile.nc -f dataFile_18500115.nc
./command -t -H -a auxFile.nc -f dataFile_18500215.nc
...
./command -t -H -a auxFile.nc -f dataFile_20191215.nc;

PS : For the avoidance of doubt I support that we first attempt describing operations on Datasets rather than files, as @sethmcg advocates for. But if this is too cumbersome, maybe something along the line of the above, somehow mixed with PROV, could help compress the size of file-level provenance information.

[sethmcg]

I think we should aim for dataset-level provenance unless it proves absolutely untenable, but I think developing a notation for highly-repetitive / patterned file paths as an element of or complement to it would probably be useful regardless.

I think the notation that NCO uses for filename patterns would be a good place to start; if we can come up with something that's based on that, we may be able to get the augmentations incorporated into NCO, and the more we can hand off file-level provenance to the tools, the better.

[TomLav]

I am not a daily user of NCO. Are you referring to -n loop syntax as described here?

I know the purpose of this github discussion is not to design the actual syntax, but if we get there one day I would be happy to contribute.

And I would like to disagree with myself from a post above:

This [using filename patterns] would be less human-readable.

Actually, it could be more human-readable because who will read through 66MB of character strings anyway? The use of patterns and loops could be better for human comprehension.

[sethmcg]

Yep, that's what I was thinking of. Though I also think your loop_wrapper approach looks great; I was just thinking about what would be the path of least resistance to pushing this kind of thing upstream into the existing toolchain.

[davidhassell]

We discussed the use case

3. For data providers: Potentially restoring processing chains, potentially including processors and data used

I'd like to add that this use case could well cover my use case of recording the uncertainty of data used at different parts of the processing chain, assuming that each contributing dataset includes its own uncertainty description (which is not in any way a part of this discussion!).

[JonathanGregory]

To be better informed for this afternoon's meeting, I've studied PROV a bit. The primer gives examples in three notations suitable for text files, namely xml, Turtle and their own PROV-N notation. As Daniel said, we could easily point to a file containing PROV from a global or variable netCDF attribute.

If we wanted to store a PROV description using any of the notations within a CF-netCDF file, we could put it in a string or char attribute, either of the file or of individual variables. That would be a simple but opaque way of enabling the file to carry the provenance information.

The PROV data model is not inconsistent with the CF data model, because the concepts are non-overlapping. Therefore if we wanted to contain the PROV information in the netCDF file in a way which could be manipulated, I think it would be pretty straightforward to store it in netCDF variables with attributes. They would be what we call "container" variables in the CF convention, which have no data (they are scalars with an irrelevant value), because their purpose is simply to contain attributes.

From the xml of the primer's complete set of examples, you can see that there are only two levels in PROV, which would correspond to variable and attribute in netCDF; you never have a thing inside a thing inside a thing. I suppose, for instance, that you could have a container variable for each of the three types and seven relations of Table 2, with a CF attribute indicating what sort of thing it is. They would refer to each other by their variable names contained in attributes indicating the PROV roles. They might want to refer to CF external variables too. Collections and bundles could be done the same way.

[erget]

@davidhassell @JonathanGregory @sethmcg @taylor13 and I had a meeting yesterday and have I believe everything that we need in order to report back into the roadmapping group. Here's my summary, improved a bit thanks to @JonathanGregory . If anybody sees a way to make this better or would like to join the conversation after having not been involved thus far, that is welcome.

Summary

Use cases

In the CF roadmap we want to support 3 use cases in order of decreasing priority:

1. I have a data file and want to check if it's the latest and greatest version of that dataset. E.g. its DOI might bring me to a dataset that gets reprocessed periodically; which version do I actually have? Or in the absence of a DOI, is this using the same creation pipeline as other files from this same data set?
2. I am creating a data file and want to credit upstream providers, who optionally have provided their own provenance data.
3. I want to understand the processing chain that led to the creation of the data file in my hand. This means that I want to look up one or more provenance records that describe how the file was created; because this is machine readable, there might even be systems that can use this metadata to reconstruct the processing chain - however that is optional from our perspective.

In order to realise this we want to use the W3C standard PROV, which allows us to interlink provenance records and, if combined e.g. with verifiable claims could be used for integrity checks and nonrepudiation. These features are outside the scope of CF, as explained below.

Scope

We won't worry about defining PROV. Although we haven't decided for sure, we're unlikely to use a container variable to embed the PROV record in the netCDF file - despite the elegance that that would provide, there are also many advantages of having a completely external PROV record.

Although provenance is related to uncertainty estimates, we will not try to use PROV as the solution for describing uncertainty.

Instead, we want an unequivocal way of referring to PROV records in CF, so that there is one way for data providers and consumers to provide and find that information.

Other notes

We will encourage users to attach PROV to intermediate products rather than having huge history attributes - at a certain point the history attribute just isn't equipped for describing complex data pipelines.

We need some careful work to make it possible to attach PROV records to specific variables, groups, etc. - it's not a given that a PROV record would refer to an entire file, although this should be possible.