CON-1513/CON-1515: Auto-generate Host Self-Test Reference docs

.github/workflows/self-test-reference.yml

@@ -0,0 +1,105 @@
+name: Self-test reference check
+
+on:
+  repository_dispatch:
+    types:
+      - self-test-reference-source-updated
+  pull_request:
+    paths:
+      - 'host/self-test-reference.mdx'
+      - 'host/how-to-self-test.mdx'
+      - 'host/verification-stages.mdx'
+      - 'scripts/generate_self_test_reference.py'
+      - 'docs.json'
+      - 'package.json'
+      - '.github/workflows/self-test-reference.yml'
+  workflow_dispatch:
+    inputs:
+      vast_cli_ref:
+        description: 'vast-ai/vast-cli ref to generate from'
+        required: false
+        default: 'master'
+      self_test_ref:
+        description: 'vast-ai/self-test ref to generate from'
+        required: false
+        default: 'main'
+
+jobs:
+  verify-self-test-reference:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout docs
+        uses: actions/checkout@v4
+        with:
+          path: docs
+
+      - name: Decide whether private source repos can be checked out
+        id: source-access
+        env:
+          SOURCE_TOKEN: ${{ secrets.VAST_DOCS_SOURCE_TOKEN }}
+          HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ] && [ "$HEAD_REPO" != "${{ github.repository }}" ] && [ -z "$SOURCE_TOKEN" ]; then
+            echo "can_verify=false" >> "$GITHUB_OUTPUT"
+            echo "::notice::Skipping self-test reference sync check for fork PR because VAST_DOCS_SOURCE_TOKEN is unavailable. Run workflow_dispatch or use an upstream branch to verify private source repos."
+          else
+            echo "can_verify=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Checkout vast-cli source
+        if: steps.source-access.outputs.can_verify == 'true'
+        uses: actions/checkout@v4
+        with:
+          repository: vast-ai/vast-cli
+          ref: ${{ github.event.client_payload.vast_cli_ref || github.event.inputs.vast_cli_ref || 'master' }}
+          path: vast-cli-source
+          token: ${{ secrets.VAST_DOCS_SOURCE_TOKEN || github.token }}
+
+      - name: Checkout self-test source
+        if: steps.source-access.outputs.can_verify == 'true'
+        uses: actions/checkout@v4
+        with:
+          repository: vast-ai/self-test
+          ref: ${{ github.event.client_payload.self_test_ref || github.event.inputs.self_test_ref || 'main' }}
+          path: self-test-source
+          token: ${{ secrets.VAST_DOCS_SOURCE_TOKEN || github.token }}
+
+      - uses: actions/setup-python@v5
+        if: steps.source-access.outputs.can_verify == 'true'
+        with:
+          python-version: '3.11'
+
+      - name: Regenerate self-test reference
+        if: steps.source-access.outputs.can_verify == 'true'
+        working-directory: docs
+        run: |
+          PYTHONDONTWRITEBYTECODE=1 npm run generate-self-test-reference -- \
+            --vast-cli ../vast-cli-source \
+            --self-test ../self-test-source
+
+      - name: Fail if generated page is out of sync
+        if: steps.source-access.outputs.can_verify == 'true'
+        working-directory: docs
+        run: |
+          if ! git diff --quiet host/self-test-reference.mdx; then
+            echo "::error::host/self-test-reference.mdx is out of sync with vast-ai/vast-cli or vast-ai/self-test."
+            echo "::error::Run 'npm run generate-self-test-reference -- --vast-cli PATH_TO_VAST_CLI --self-test PATH_TO_SELF_TEST' and commit the result."
+            git diff --stat host/self-test-reference.mdx
+            git diff -- host/self-test-reference.mdx
+            exit 1
+          fi
+
+      - name: Fail if Python bytecode was generated
+        if: steps.source-access.outputs.can_verify == 'true'
+        working-directory: docs
+        run: |
+          if find scripts -name '__pycache__' -o -name '*.pyc' | grep -q .; then
+            echo "::error::Generated Python bytecode should not be committed or produced by the docs generation check."
+            find scripts -name '__pycache__' -o -name '*.pyc'
+            exit 1
+          fi
+
+      - name: Report skipped private-source validation
+        if: steps.source-access.outputs.can_verify != 'true'
+        run: |
+          echo "Self-test reference sync check skipped because this fork PR cannot access private source repositories."

README.md

@@ -12,6 +12,36 @@ API endpoint specs live in `api-reference/openapi/yaml/` (one file per endpoint)
 4. Preview locally: `mint dev`.
 5. Commit both your YAML edit AND the regenerated `api-reference/openapi.yaml`, then open a PR. CI re-runs the build and fails if `openapi.yaml` is out of sync with the sources.
 
+## Updating the self-test reference
+
+The host self-test reference page is generated from the Vast CLI diagnostics and the self-test image metadata.
+
+1. Update the relevant self-test source in `vast-ai/vast-cli` or `vast-ai/self-test`.
+2. Regenerate the docs page:
+
+```bash
+npm run generate-self-test-reference -- --vast-cli ../vast-cli --self-test ../self-test
+```
+
+3. Review and commit `host/self-test-reference.mdx` with the source change or in the matching docs PR.
+4. CI re-runs the generator against `vast-ai/vast-cli` and `vast-ai/self-test` and fails if the committed page is out of sync.
+
+For cross-repo private checkouts, configure the docs repository secret `VAST_DOCS_SOURCE_TOKEN` with read access to `vast-ai/vast-cli` and `vast-ai/self-test`. The workflow can also be run manually with custom `vast_cli_ref` and `self_test_ref` inputs while a source PR is under review.
+
+Source repositories can trigger the docs check after self-test metadata changes by sending a repository dispatch event to `vast-ai/docs`:
+
+```json
+{
+  "event_type": "self-test-reference-source-updated",
+  "client_payload": {
+    "vast_cli_ref": "master",
+    "self_test_ref": "main"
+  }
+}
+```
+
+Use the exact branch or SHA being validated when triggering this from a source PR workflow.
+
 ## Mintlify Information
 
 **[Follow the full quickstart guide](https://starter.mintlify.com/quickstart)**

docs.json

@@ -601,6 +601,7 @@
             "pages": [
               "host/verification-stages",
               "host/how-to-self-test",
+              "host/self-test-reference",
               "host/market-metrics",
               "host/datacenter-status",
               "host/payment",

host/how-to-self-test.mdx

@@ -34,6 +34,8 @@ During the self-test, the following components and conditions are verified:
 
 A short test workload will be executed to assess actual runtime performance.
 
+For the exact thresholds, runtime stages, failure codes, and diagnostic bundle behavior, see the [Self-test Reference](/host/self-test-reference).
+
 <Tip>
 **Tip:** Ensure no other jobs or instances are running during the self-test for the most accurate results.
 </Tip>
@@ -90,11 +92,25 @@ If the test fails:
 
 * The CLI will display detailed reasons for failure.
 * Apply the suggested fixes and rerun the test.
+* Review the [Self-test Reference](/host/self-test-reference) for the failed check or failure code.
 
 If the test says the machine is "not found or not rentable":
 
-* Try un-listing your machine, then listing it again.
-* Ensure your machine has no missing data in your machines page, such as upload and download speed, RAM, or ports.
+* Check whether the machine is currently rented, offline, unlisted, below the reliability threshold, missing active offers, or hidden from the API key you are using.
+* Inspect all offers for the machine:
+
+```bash
+vastai search offers 'machine_id=<machine_id> rentable=any rented=any'
+```
+
+* Review the [Self-test Reference](/host/self-test-reference#not-found-or-not-rentable) for the specific root states the CLI can report.
+
+If the test reports no response or a progress timeout:
+
+* Check the external IP and port printed by the CLI, if present.
+* Confirm your router/firewall forwards that external port to the host's LAN IP.
+* If you are running the CLI from the same LAN as the host, retest from another network to rule out NAT hairpinning.
+* Review the diagnostic bundle path printed by the CLI after failure.
 
 ### Optional: Ignore Requirements Mode
 
@@ -118,3 +134,5 @@ Even if the test passes in this mode, your machine does not meet the minimum ver
 <Warning>
 **Important:** Even with `--ignore-requirements`, your machine must have at least three direct open ports, otherwise, the self-test will fail.
 </Warning>
+
+With `--ignore-requirements`, a passing result means the internal runtime workload passed. It does not mean the machine passed the minimum verification requirements.