-
Notifications
You must be signed in to change notification settings - Fork 234
CI: Add a new workflow to cache dvc cache #4184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
e1442c1
3dbc3ac
61ded6d
8dd8f0d
6d9be57
bd4b348
3476137
cf6a94b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| # Cache baseline images tracked by DVC and upload as artifacts. | ||
| # | ||
| # This workflow downloads dvc cache needed by PyGMT tests and uploads them as workflow | ||
| # artifacts, which can then be accessed by other GitHub Actions workflows. | ||
| # | ||
| # This workflow serves as a workaround for the DagsHub authentication issue that | ||
| # prevents PRs from forks to access the DVC remote. | ||
| # Related issue: https://github.com/GenericMappingTools/pygmt/issues/4147 | ||
| # | ||
| # It is scheduled to run every Sunday at 12:00 (UTC) and on the main branch if there are | ||
| # any changes to the baseline images. | ||
| # | ||
| name: Cache DVC | ||
|
|
||
| on: | ||
| push: | ||
| branches: [ main ] | ||
| paths: | ||
| - 'pygmt/tests/baseline/*.png.dvc' | ||
| pull_request: # For testing only. | ||
| workflow_dispatch: | ||
| # Schedule runs on 12 noon every Sunday | ||
| schedule: | ||
| - cron: '0 12 * * 0' | ||
|
|
||
| permissions: {} | ||
|
|
||
| jobs: | ||
| dvc_cache: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could probably this as a separate job under
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These two jobs have different trigger conditions. The
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah yes, ok to keep them separate then. |
||
| name: Cache DVC baseline images | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| shell: bash -l {0} | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/[email protected] | ||
| with: | ||
| # fetch all history so that setuptools-scm works | ||
| fetch-depth: 0 | ||
| persist-credentials: false | ||
|
|
||
| - name: Setup data version control (DVC) | ||
| uses: iterative/setup-dvc@175771be1dc3d119268e00a896b52a4b77decb5e # v1.2.0 | ||
|
|
||
| - name: Pull baseline image data from dvc remote | ||
| env: | ||
| DAGSHUB_TOKEN: ${{ secrets.DAGSHUB_TOKEN }} | ||
| run: | | ||
| dvc remote modify upstream url https://${DAGSHUB_TOKEN}@dagshub.com/GenericMappingTools/pygmt.dvc --local | ||
| dvc pull --no-run-cache --verbose && ls -lhR pygmt/tests/baseline/ | ||
| - name: Upload DVC cache as artifacts to GitHub | ||
| uses: actions/[email protected] | ||
| with: | ||
| name: dvc-cache | ||
| include-hidden-files: true | ||
| path: .dvc/cache/ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -163,12 +163,20 @@ jobs: | |
|
|
||
| # Pull baseline image data from dvc remote (DAGsHub) | ||
| - name: Pull baseline image data from dvc remote | ||
| id: dvc-pull | ||
| continue-on-error: true | ||
| run: | | ||
| uv run dvc remote modify upstream url https://${DAGSHUB_TOKEN}@dagshub.com/GenericMappingTools/pygmt.dvc --local | ||
| uv run dvc pull --no-run-cache --verbose && ls -lhR pygmt/tests/baseline/ | ||
| env: | ||
| DAGSHUB_TOKEN: ${{ secrets.DAGSHUB_TOKEN }} | ||
|
|
||
| - name: Download DVC cache as artifacts from GitHub | ||
| if: steps.dvc-pull.outcome == 'failure' | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| run: | | ||
| gh run download --name dvc-cache --dir .dvc/cache/ | ||
| uv run dvc checkout --verbose | ||
|
|
||
| # Install the package that we want to test | ||
| - name: Install the package | ||
| run: make install | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.