Auto-generate CLI/SDK docs#398
Open
Gobind-Vast wants to merge 20 commits into
Open
Conversation
Introduces automated drift detection between the vastai CLI/SDK and the Mintlify documentation in vast-ai/docs. The verification script compares installed CLI commands and SDK methods against documented MDX pages, flagging missing docs, stale pages, and parameter mismatches. The GitHub Actions workflow runs on PRs (comments on drift), master pushes, and a weekly schedule (opens issues on drift). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update verify_cli_sdk_docs.py to handle two-level CLI commands (e.g., "vastai show instances") by parsing them from --help output and flattening to kebab-case for doc filename matching - Add docs/verify-cli-sdk-docs.md with instructions for running the verification manually - Add docs/workflows/verify-docs-on-docs-pr.yml as a reference GitHub Actions workflow for the docs repo to run verification when CLI/SDK doc pages change Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gobind-Vast
changed the title
Epilog descriptions and ParamField help text contain literal placeholders like <PHONE_NUMBER>, <name of a field>, <bool, int, float, string>. Mintlify parses these as JSX tags, fails the page build, and returns 404 while the overall deploy reports success. 10 CLI pages on preview PR #131 were affected: tfa-activate, tfa-totp-setup, create-ssh-key, list-machine, and the 6 search-* pages. Escape <,> to <,> outside inline backticks; renders identically to the working pages.
Mintlify reports "Deployment: success" even when individual pages fail to build — the failures surface only as HTTP 404 on the affected URLs. We shipped 10 such pages in PR #131 before catching them via manual spot-check (unescaped <PLACEHOLDER> tokens in argparse epilogs). New step waits for the Mintlify check-run on the docs commit we pushed, then curls every cli/reference and sdk/python/reference path from docs.json and fails the workflow on any non-200. ~3 min added serially; runs after the preview/deploy PR is open so failures appear on the source vast-cli PR.
The pipeline only added/modified files, never deleted them. When commands were removed from vast-cli's main.py (cluster/overlay, run-benchmarks, search-instances), their MDX files persisted on the docs preview branch indefinitely — 13 orphans accumulated by 2026-05-20. Wipe everything under cli/reference and sdk/python/reference before the generator runs. Preserve sdk/python/reference/vastai.mdx (hand-authored overview, the only file the generator doesn't produce).
… pages patch_docs_nav.py emitted "no classify rule" warnings for 4 generated pages: accept-price-increase (CLI+SDK) and get-endpt-workers / get-endpoint-workers (CLI+SDK). They stayed out of docs.json nav even though the MDX existed. Add rules: - endpt|endpoint|workergroup|wrkgrp in name → Serverless - accept-price-increase → Instances
Adding/changing classify_new() rules wasn't triggering the workflow because patch_docs_nav.py wasn't listed in the path filter.
vast.ai's public stance is that the platform does not offer network volumes. The network-volume / network-disk CLI surface was written for a single customer and isn't verified to work for general users. Add EXCLUDED_NAMES set checked before the scope filter; covers all 6 network commands (12 pages CLI+SDK): add-network-disk, create-network-volume, list-network-volume, search-network-volumes, show-network-disks, unlist-network-volume. Manifest gains excluded_policy_cli/sdk for visibility.
… new pages load_cli_parser() kept its own hardcoded copy of the command-module import list, which had drifted from the CLI. It omitted the metrics, benchmarks, and price_increase modules, so the generator silently failed to document 7 real CLI commands (metrics gpu/gpu-trends/gpu-locations, run benchmarks, accept/reject price-increase, show pending-price-increases). Reuse the canonical register_all_commands() helper instead so the generator stays in lockstep with the real command set and new modules are picked up automatically. (The verify parity gate already saw these via `vastai --help`; only the introspection-based generator was blind.) Extend classify_new() in patch_docs_nav.py to place the recovered pages: metrics-gpu* -> Host ([Host]-tagged GPU-market analytics), *-deployment -> Serverless (alongside existing deployment pages), and broaden the accept-price-increase exact match to the whole price-increase family (accept/reject/show-pending). CLI count 133 -> 140; all new pages now land in nav and the patch stays idempotent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gobind-Vast
pushed a commit
to vast-ai/docs
that referenced
this pull request
Jun 1, 2026
Source PR: vast-ai/vast-cli#398 Title: Auto-generate CLI/SDK docs Auto-generated by .github/workflows/auto-generate-docs.yml
Gobind-Vast
pushed a commit
to vast-ai/docs
that referenced
this pull request
Jun 4, 2026
Source PR: vast-ai/vast-cli#398 Title: Auto-generate CLI/SDK docs Auto-generated by .github/workflows/auto-generate-docs.yml
Author
|
Concomitant draft PR in the docs repo (automatically generated from this PR): vast-ai/docs#131 |
vastzuby
approved these changes
Jun 23, 2026
Gobind-Vast
pushed a commit
to vast-ai/docs
that referenced
this pull request
Jun 23, 2026
Source PR: vast-ai/vast-cli#398 Title: Auto-generate CLI/SDK docs Auto-generated by .github/workflows/auto-generate-docs.yml
…ning classify_new() returning None caused new CLI/SDK pages to be written to disk but omitted from docs.json nav, with only a WARNING and a green workflow — so a new command could silently never appear in the sidebar (e.g. dump-logs in run #13). - Add a precise rule for dump-logs / self-test (machine diagnostics -> Host). - Route any still-unclassified page into DEFAULT_GROUP ("Instances"), with a deterministic fallback to any managed group of the matching prefix, so a generated page is never silently orphaned from the nav. The WARNING is retained to prompt adding a precise rule later. Verified against docs main docs.json: dump-logs lands in Host, an unknown command falls back to Instances (visible, flagged), patch is idempotent and --check passes. Generator output is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gobind-Vast
pushed a commit
to vast-ai/docs
that referenced
this pull request
Jun 23, 2026
Source PR: vast-ai/vast-cli#398 Title: Auto-generate CLI/SDK docs Auto-generated by .github/workflows/auto-generate-docs.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-generates CLI and SDK reference MDX pages for the docs site (vast-ai/docs) by introspecting the installed
vastaipackage, mirroring the OpenAPI pipeline that lives in thevastbackend repo.Rationale: Hand-authored CLI/SDK docs in drift fast, and manual sync isn't viable at the rate the CLI is changing.
scripts/generate_cli_sdk_docs.py: walks the apwrap subparsers and theVastAIclass to produce one MDX page per command / method (130 CLI, 146 SDK on current source)..github/workflows/auto-generate-docs.yml: PR runs open a preview PR in vast-ai/docs (Mintlify auto-deploys a preview); master merges open the final deploy PR and clean up orphan preview branches.scripts/data/api_scopes.json: snapshot ofvast/web/scope.jsonjoined withpaths.py, keyed by URL pattern. Lets the generator drop commands whose endpoints are gated by internal-only scopes (same filter the OpenAPI generator uses).