-
Notifications
You must be signed in to change notification settings - Fork 233
CI: Add a new workflow to cache dvc cache #4184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| DAGSHUB_TOKEN: ${{ secrets.DAGSHUB_TOKEN }} | ||
|
|
||
| - name: Download DVC cache as artifacts from GitHub | ||
| if: steps.dvc-pull.outcome == 'failure' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
weiji14
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! Just a couple of suggestions.
| permissions: {} | ||
|
|
||
| jobs: | ||
| dvc_cache: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could probably this as a separate job under .github/workflows/cache_data.yaml if want to reduce the number of workflow files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These two jobs have different trigger conditions. The cache_data workflow is usually scheduled to run weekly but the cache_dvc workflow needs to run when .dvc files change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yes, ok to keep them separate then.
Co-authored-by: Wei Ji <[email protected]>
For new contributors, it's frustrating to see that so mamy workflows fail due to the DVC issue reported in #4147.
This PR adds a workflow for this issue. It adds a new workflow
cache_dvc.yamlwhich pulls the baseline images from DagsHub and uploads the.dvc/cachefile as GitHub artifacts.For PRs from forks, the
dvc pullcommand fails, so we can download the dvc cache from the GitHub artifact.Below are workflows that use
dvc pulland may fail:ci_tests_dev.yamlci_tests_dev.yaml: The cache won't work in this workflow, since the dvc cache reflect baseline images in the main branch, but this workflow needs baseline images in both the main and the current branch. But it's unlikely that PRs from new contributors will trigger this workflow, since they can't do 'dvc push' either. So no need to apply the workarounddvc-diff.yaml: Doesn't run in PRs, so no need to apply the workaroundrelease-baseline-images.ymlTests
.dvc/cachedirectory was uploaded https://github.com/GenericMappingTools/pygmt/actions/runs/19008233397?pr=4184dvc pullcommand succeeds (https://github.com/GenericMappingTools/pygmt/actions/runs/19008233364/job/54285305945?pr=4184)