Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/cache_dvc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Cache baseline images tracked by DVC and upload as artifacts.
#
# This workflow downloads dvc cache needed by PyGMT tests and uploads them as workflow
# artifacts, which can then be accessed by other GitHub Actions workflows.
#
# This workflow serves as a workaround for the DagsHub authentication issue that
# prevents PRs from forks to access the DVC remote.
# Related issue: https://github.com/GenericMappingTools/pygmt/issues/4147
#
# It is scheduled to run every Sunday at 12:00 (UTC) and on the main branch if there are
# any changes to the baseline images.
#
name: Cache DVC

on:
push:
branches: [ main ]
paths:
- 'pygmt/tests/baseline/*.png.dvc'
pull_request: # For testing only.
workflow_dispatch:
# Schedule runs on 12 noon every Sunday
schedule:
- cron: '0 12 * * 0'

permissions: {}

jobs:
dvc_cache:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could probably this as a separate job under .github/workflows/cache_data.yaml if want to reduce the number of workflow files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two jobs have different trigger conditions. The cache_data workflow is usually scheduled to run weekly but the cache_dvc workflow needs to run when .dvc files change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, ok to keep them separate then.

name: Cache DVC baseline images
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l {0}

steps:
- name: Checkout
uses: actions/[email protected]
with:
# fetch all history so that setuptools-scm works
fetch-depth: 0
persist-credentials: false

- name: Setup data version control (DVC)
uses: iterative/setup-dvc@175771be1dc3d119268e00a896b52a4b77decb5e # v1.2.0

- name: Pull baseline image data from dvc remote
env:
DAGSHUB_TOKEN: ${{ secrets.DAGSHUB_TOKEN }}
run: |
dvc remote modify upstream url https://${DAGSHUB_TOKEN}@dagshub.com/GenericMappingTools/pygmt.dvc --local
dvc pull --no-run-cache --verbose && ls -lhR pygmt/tests/baseline/
- name: Upload DVC cache as artifacts to GitHub
uses: actions/[email protected]
with:
name: dvc-cache
include-hidden-files: true
path: .dvc/cache/
8 changes: 8 additions & 0 deletions .github/workflows/ci_tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -163,12 +163,20 @@ jobs:

# Pull baseline image data from dvc remote (DAGsHub)
- name: Pull baseline image data from dvc remote
id: dvc-pull
continue-on-error: true
run: |
uv run dvc remote modify upstream url https://${DAGSHUB_TOKEN}@dagshub.com/GenericMappingTools/pygmt.dvc --local
uv run dvc pull --no-run-cache --verbose && ls -lhR pygmt/tests/baseline/
env:
DAGSHUB_TOKEN: ${{ secrets.DAGSHUB_TOKEN }}

- name: Download DVC cache as artifacts from GitHub
if: steps.dvc-pull.outcome == 'failure'
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run: |
gh run download --name dvc-cache --dir .dvc/cache/
uv run dvc checkout --verbose

# Install the package that we want to test
- name: Install the package
run: make install
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/ci_tests_dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -161,12 +161,20 @@ jobs:

# Pull baseline image data from dvc remote (DAGsHub)
- name: Pull baseline image data from dvc remote
id: dvc-pull
continue-on-error: true
run: |
dvc remote modify upstream url https://${DAGSHUB_TOKEN}@dagshub.com/GenericMappingTools/pygmt.dvc --local
dvc pull --no-run-cache --verbose && ls -lhR pygmt/tests/baseline/
env:
DAGSHUB_TOKEN: ${{ secrets.DAGSHUB_TOKEN }}

- name: Download DVC cache as artifacts from GitHub
if: steps.dvc-pull.outcome == 'failure'
run: |
gh run download --name dvc-cache --dir .dvc/cache/
dvc checkout --verbose

# Download cached remote files (artifacts) from GitHub
- name: Download remote data from GitHub
run: |
Expand Down
1 change: 1 addition & 0 deletions doc/maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ workflow files for more details.

- `benchmarks.yml`: Benchmarks the execution speed of tests to track performance of PyGMT functions
- `cache_data.yaml`: Cache GMT remote data files and uplodas as artifacts
- `cache_dvc.yaml`: Cache baseline images tracked by DVC and upload as artifacts.
- `check-links.yml`: Check links in the repository and documentation
- `ci_docs.yml`: Build documentation on Linux/macOS/Windows and deploy to GitHub
- `ci_doctest.yaml`: Run all doctests on Linux/macOS/Windows
Expand Down
Loading