Feature/verify cli sdk docs

.github/workflows/verify-docs.yml

@@ -0,0 +1,122 @@
+# GitHub Actions workflow for vast-ai/vast-cli or vast-ai/docs repo.
+# Validates that CLI/SDK documentation stays in sync with the actual package.
+#
+# Place this file at .github/workflows/verify-docs.yml in whichever repo
+# should own the check (typically vast-cli, since it's the source of truth).
+
+name: Verify CLI/SDK Docs
+
+on:
+  # Run on PRs that change CLI or SDK code
+  pull_request:
+    paths:
+      - "vastai/**"
+      - "vastai_sdk/**"
+      - "pyproject.toml"
+
+  # Run on pushes to master (after merge)
+  push:
+    branches: [master, main]
+
+  # Run weekly to catch drift even without code changes
+  schedule:
+    - cron: "0 9 * * 1"  # Monday 9am UTC
+
+  workflow_dispatch: {}
+
+jobs:
+  verify-docs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout vast-cli
+        uses: actions/checkout@v4
+
+      - name: Checkout docs repo
+        uses: actions/checkout@v4
+        with:
+          repository: vast-ai/docs
+          path: docs-repo
+          # If docs repo is private, add a token:
+          # token: ${{ secrets.DOCS_REPO_TOKEN }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install vast-cli package
+        run: pip install -e .
+
+      - name: Run inventory check
+        run: |
+          python scripts/verify_cli_sdk_docs.py \
+            --docs-path docs-repo \
+            --check-params \
+            --json > drift-report.json
+
+      - name: Display report
+        if: always()
+        run: |
+          python scripts/verify_cli_sdk_docs.py \
+            --docs-path docs-repo \
+            --check-params
+
+      - name: Upload drift report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: drift-report
+          path: drift-report.json
+
+      - name: Comment on PR if drift detected
+        if: failure() && github.event_name == 'pull_request'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require('fs');
+            const report = JSON.parse(fs.readFileSync('drift-report.json', 'utf8'));
+            let body = '## Documentation Drift Detected\n\n';
+            if (report.cli.undocumented.length) {
+              body += `### CLI commands missing docs (${report.cli.undocumented.length})\n`;
+              report.cli.undocumented.forEach(c => body += `- \`${c}\`\n`);
+            }
+            if (report.cli.stale_docs.length) {
+              body += `### CLI docs for removed commands (${report.cli.stale_docs.length})\n`;
+              report.cli.stale_docs.forEach(c => body += `- \`${c}\`\n`);
+            }
+            if (report.sdk.undocumented.length) {
+              body += `### SDK methods missing docs (${report.sdk.undocumented.length})\n`;
+              report.sdk.undocumented.forEach(m => body += `- \`${m}\`\n`);
+            }
+            if (report.sdk.stale_docs.length) {
+              body += `### SDK docs for removed methods (${report.sdk.stale_docs.length})\n`;
+              report.sdk.stale_docs.forEach(m => body += `- \`${m}\`\n`);
+            }
+            body += '\n> Update docs in [vast-ai/docs](https://github.com/vast-ai/docs) to resolve.';
+            github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+              body: body,
+            });
+
+      - name: Open issue on scheduled drift
+        if: failure() && github.event_name == 'schedule'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require('fs');
+            const report = JSON.parse(fs.readFileSync('drift-report.json', 'utf8'));
+            const counts = [
+              report.cli.undocumented.length,
+              report.cli.stale_docs.length,
+              report.sdk.undocumented.length,
+              report.sdk.stale_docs.length,
+            ].reduce((a, b) => a + b, 0);
+            github.rest.issues.create({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              title: `Docs drift: ${counts} items out of sync`,
+              body: '```json\n' + JSON.stringify(report, null, 2) + '\n```',
+              labels: ['documentation'],
+            });

docs/verify-cli-sdk-docs.md

@@ -0,0 +1,97 @@
+# CLI/SDK Documentation Verification
+
+This guide explains how to verify that the CLI/SDK documentation in
+[vast-ai/docs](https://github.com/vast-ai/docs) matches the actual `vastai`
+CLI commands and SDK methods.
+
+## Prerequisites
+
+- Python 3.10+
+- The `vastai` package installed (from this repo)
+- A clone of the docs repo (or a specific branch/PR)
+
+## Quick Start
+
+```bash
+# 1. Install the vastai package from this repo
+pip install -e .
+
+# 2. Clone the docs repo (or a specific PR branch)
+git clone https://github.com/vast-ai/docs.git /tmp/docs
+
+# To check a specific PR branch instead:
+# git clone --branch <branch-name> --depth 1 https://github.com/vast-ai/docs.git /tmp/docs
+
+# 3. Run the inventory check
+python3 scripts/verify_cli_sdk_docs.py --docs-path /tmp/docs
+
+# 4. Run with parameter-level validation
+python3 scripts/verify_cli_sdk_docs.py --docs-path /tmp/docs --check-params
+
+# 5. Output as JSON (useful for CI or sharing)
+python3 scripts/verify_cli_sdk_docs.py --docs-path /tmp/docs --check-params --json > drift-report.json
+```
+
+## What It Checks
+
+### Inventory (always runs)
+- **CLI commands missing docs**: commands found in `vastai --help` with no
+  matching `cli/reference/<command>.mdx` page
+- **Stale CLI docs**: MDX pages for commands that no longer exist in the CLI
+- **SDK methods missing docs**: public methods on the `VastAI` class with no
+  matching `sdk/python/reference/<method>.mdx` page
+- **Stale SDK docs**: MDX pages for methods that no longer exist in the SDK
+
+### Parameter validation (`--check-params`)
+- **Undocumented flags/params**: flags in `--help` output or method signature
+  parameters not mentioned in the corresponding MDX page
+- **Stale flags/params**: flags/params documented in the MDX page that no
+  longer exist in the CLI/SDK
+
+## Naming Conventions
+
+The script converts between naming conventions for matching:
+
+| Source | Convention | Example |
+|--------|-----------|---------|
+| CLI commands | kebab-case | `show-instances` |
+| SDK methods | snake_case | `show_instances` |
+| Doc filenames | kebab-case | `show-instances.mdx` |
+
+**Note**: If the CLI uses a two-level command structure (e.g., `vastai show
+instances`), the script parses top-level subcommands from `vastai --help`. The
+doc filenames should match the flattened kebab-case form (e.g.,
+`show-instances.mdx`). If this doesn't match, the script will report drift.
+
+## Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | No drift detected |
+| 1 | Drift detected |
+| 2 | Script error (missing files, import failure, etc.) |
+
+## Interpreting Results
+
+Not all drift is a bug:
+
+- **CLI command restructuring** (e.g., flat → two-level commands) will show
+  everything as "stale" under the old names and "undocumented" under the new
+  names. This means the verification script's name-matching logic may need
+  updating to match the new CLI structure.
+- **`kwargs` as undocumented param** means the SDK method accepts flexible
+  keyword arguments. The docs should list the commonly used kwargs, but the
+  script can't extract individual kwargs from `**kwargs`.
+- **Case differences** (e.g., `Id` vs `id`, `COMMAND` vs `command`) are
+  flagged as mismatches. These are usually cosmetic.
+
+## Automation
+
+This check runs automatically via GitHub Actions
+(`.github/workflows/verify-docs.yml`):
+
+| Trigger | Behavior |
+|---------|----------|
+| PR that changes `vastai/` or `vastai_sdk/` | Runs check, comments on PR if drift found |
+| Push to master | Runs check |
+| Weekly (Monday 9am UTC) | Opens a GitHub issue if drift detected |

docs/workflows/verify-docs-on-docs-pr.yml

@@ -0,0 +1,57 @@
+# Place this file at vast-ai/docs/.github/workflows/verify-cli-sdk-docs.yml
+# Runs the CLI/SDK verification script when doc pages change in the docs repo.
+#
+# This complements the workflow in vast-cli that runs when CLI/SDK code changes.
+# Together they catch drift from both directions:
+#   - vast-cli changes → docs repo workflow detects stale docs
+#   - docs changes     → this workflow detects incorrect docs
+
+name: Verify CLI/SDK Docs
+
+on:
+  pull_request:
+    paths:
+      - "cli/reference/**"
+      - "sdk/python/reference/**"
+
+  workflow_dispatch: {}
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout docs repo
+        uses: actions/checkout@v4
+
+      - name: Checkout vast-cli (for verification script)
+        uses: actions/checkout@v4
+        with:
+          repository: vast-ai/vast-cli
+          path: vast-cli
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install vastai package
+        run: pip install -e vast-cli
+
+      - name: Run verification
+        run: |
+          python vast-cli/scripts/verify_cli_sdk_docs.py \
+            --docs-path . \
+            --check-params \
+            --json > drift-report.json || true
+
+          # Always show human-readable output
+          python vast-cli/scripts/verify_cli_sdk_docs.py \
+            --docs-path . \
+            --check-params
+
+      - name: Upload drift report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: drift-report
+          path: drift-report.json