-
Notifications
You must be signed in to change notification settings - Fork 32
CEP XXXX: Build provenance metadata #113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jaimergp
wants to merge
16
commits into
conda:main
Choose a base branch
from
jaimergp:provenance
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 10 commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
3a6aa09
Add CEP for build provenance metadata
jaimergp 2ef69bd
ack
jaimergp e25b7f8
lint
jaimergp 6a56d97
Reword for ecosystem wide scope
jaimergp be01bce
Rework CEP requiredness
jaimergp a23fb78
Recommend HTTPS URLs
jaimergp fba233d
Merge branch 'main' of github.com:conda/ceps into provenance
jaimergp 1bd76d7
pre-commit
jaimergp 436d453
Add link to discussion
jaimergp d02e9f4
Update 'updated' date
jaimergp 805d430
Typo
jaimergp 745b3ab
Add some examples
jaimergp 47fd541
Move to rejected ideas
jaimergp ac502b3
Add RFC note
jaimergp e5f3a3d
Amend note
jaimergp 96de929
Wrap, minor editorial comments
jaimergp File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| # CEP XXXX - Build provenance metadata | ||
|
|
||
| <table> | ||
| <tr><td> Title </td><td> CEP XXXX - Build provenance metadata </td> | ||
| <tr><td> Status </td><td> Draft </td></tr> | ||
| <tr><td> Author(s) </td><td> Jaime Rodríguez-Guerra <jaime.rogue@gmail.com></td></tr> | ||
| <tr><td> Created </td><td> Mar 10, 2025</td></tr> | ||
| <tr><td> Updated </td><td> Sep 27, 2025</td></tr> | ||
| <tr><td> Discussion </td><td> https://github.com/conda/ceps/pull/113 </td></tr> | ||
| <tr><td> Implementation </td><td> https://github.com/conda/conda-build/pull/4303, https://github.com/conda-forge/conda-smithy/pull/1577 </td></tr> | ||
| </table> | ||
|
|
||
| ## Abstract | ||
|
|
||
| This CEP aims to standardize how the conda ecosystem leverages free-form recipe metadata to | ||
| annotate build provenance of its published artifacts. | ||
|
|
||
| ## Motivation | ||
|
|
||
| Provenance metadata is useful to assess how and when a conda artifact was built. | ||
|
|
||
| Since late 2023, thanks to conda-smithy 3.28.0 and later, conda-forge feedstocks have been adding | ||
| CI provenance in the produced artifacts. `defaults` also applies the same conventions. This is used | ||
| by apps like `conda-metadata-app` to show provenance information in the build details. See the | ||
| table in this [Python 3.13 | ||
| example](https://conda-metadata-app.streamlit.app/?q=conda-forge%2Flinux-64%2Fpython-3.13.2-hf636f53_101_cp313.conda). | ||
|
|
||
| This was possible thanks to a new `--extra-meta` flag added in | ||
| [conda-build#4303](https://github.com/conda/conda-build/pull/4303/files) and released in 3.21.8. | ||
| Rattler-build also offers the same functionality using the same CLI flag. `--extra-meta` allows | ||
| passing arbitrary key-value pairs that will be added to the `info/about.json`, under the `extra` | ||
| key (as defined in [CEP PR#133](https://github.com/conda/ceps/pull/133)). For example, if a user | ||
| passes `--extra-meta date=2025-03-11`, `about.json` will contain: | ||
|
|
||
| ```js | ||
| { | ||
| // ... | ||
| "extra": { | ||
| "date": "2025-03-11" | ||
| }, | ||
| // ... | ||
| } | ||
| ``` | ||
|
|
||
| Additional provenance metadata can be collected for source origins, which can be useful for efforts | ||
| like dependency mapping across ecosystems. See [PEP 725](https://peps.python.org/pep-0725/) for | ||
| practical applications in the context of PyPI/conda interoperability. This type of provenance is | ||
| out of scope for this CEP and may be discussed separately. | ||
|
|
||
| ## Specification | ||
|
|
||
| Build provenance metadata is optional. If necessary, the following metadata keys MAY be used to | ||
| record the corresponding information: | ||
|
|
||
| - `sha`: String. Full commit hash of the recipe repository being built. | ||
| - `remote_url`: String. CVS URL of the recipe repository being built. HTTP(S) preferred. | ||
| - `flow_run_id`: String. CI-specific identifier for the workflow run. | ||
|
|
||
| For example, ``conda-forge/linux-64::python-3.13.2-hf636f53_101_cp313.conda` has the following | ||
jaimergp marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| provenance metadata: | ||
|
|
||
| ```json | ||
| { | ||
| "sha": "50a4e2d4203f05082fcbb93e14541180de3aa8ac", | ||
| "remote_url": "https://github.com/conda-forge/python-feedstock", | ||
| "flow_run_id": "azure_20250217.3.1" | ||
| } | ||
| ``` | ||
|
|
||
| CI pipelines are strongly encouraged to add this metadata via `--extra-meta` (or equivalent). Local | ||
| workflows may not have this information available, but they are still recommended to burn in the | ||
| metadata with empty strings. | ||
|
|
||
| ## Acknowledgements | ||
|
|
||
| These efforts were spearheaded by Connor Martin and Daniel Bast at Anaconda, and Isuru Fernando at | ||
| conda-forge. | ||
|
|
||
| ## Copyright | ||
|
|
||
| All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there an example on how this value can be used? I dont think conda-metadata-app uses this right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We only link to the commit hash, which on the GH UI should provide enough information for a user to navigate to the CI workflow (and then manually check the workflow run ID or something). I guess I was just lazy to postprocess strings like
github_1234565435orazure_79979955940; we'd require more information to build the full URL.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also since the workflow logs tend to expire, they are not as useful long term. But we could also add other known keys like
flow_run_urlfor build farms to populate.