Self-Host AI

A self-hosted LLM inference server exposing an OpenAI-compatible Chat Completions API, fronted by a Caddy reverse proxy and protected by GPU thermal monitoring.

The repository ships ready-to-use compose templates for a selection of models, but is not limited to them — any model supported by vLLM can be added by following the same pattern.

All services are deployed via Docker Compose.

---

Project Structure

selfhost-ai/
├── .env.example                     # Template for all env variables (copy to .env)
├── requirements.txt                 # Python dependencies for the test client
├── Makefile                         # Root-level management commands (run make help)
├── models/                          # Model weights — one subdirectory per model,
│                                    # mounted read-only into the vLLM container
│
└── vllm/
    ├── deployment-templates/        # Per-model vLLM + Caddy Docker Compose stacks
    │   ├── Caddyfile
    │   └── docker-compose.<model-name>.yml   # one file per model
    │
    ├── testing/                     # API smoke-test script
    │   └── test_llm_api.py
    │
    └── thermal-guard/               # GPU temperature monitoring & auto-shutdown
        └── docker/                  # Docker Compose-based deployment
            ├── Dockerfile
            ├── docker-compose.thermal.yml
            ├── thermal-guard-docker.sh
            └── README-docker.md

---

Tech Stack

Component	Image / Tool
Inference engine	nvcr.io/nvidia/vllm:26.01-py3 (Qwen3.6-27B uses 26.04-py3)
Reverse proxy	caddy:latest
GPU metrics	nvidia/dcgm-exporter:4.5.2-4.8.1-ubuntu22.04
Thermal guard	alpine:latest (custom script)
Orchestration	Docker Compose v2

---

Prerequisites

• Docker and Docker Compose v2
• NVIDIA Container Toolkit installed and configured
• NVIDIA GPU(s) with CUDA support
• Model weights downloaded into models/

---

Quick Start

1. Verify GPU access

make gpu-check

2. Configure environment

make env          # copies .env.example → .env
# Edit .env and set VLLM_API_KEY:
#   openssl rand -hex 32

3. Deploy a model

make deploy-ministral-8b        # shortcut
make deploy MODEL=Qwen3.6-27B   # any model by name

Caddy will not accept traffic until vLLM reports healthy. Allow up to 10 minutes for large models to load.

4. Start thermal monitoring

make thermal-up

5. Test the API

make venv    # first time only — creates .venv and installs requirements
make test

Run make help to see all available commands.

---

vLLM Deployment (OpenAI-Compatible API)

Overview

Each model has its own Docker Compose file under vllm/deployment-templates/. Every stack starts two services:

Service	Image	Description
vllm-server	nvcr.io/nvidia/vllm:26.01-py3	Inference engine, listens on port 8000 (internal)
caddy	caddy:latest	Reverse proxy, exposes HTTP (80) / HTTPS (443)

Caddy forwards all traffic to vLLM with response streaming enabled (flush_interval -1) and a 20-minute read timeout to accommodate slow generation on large models.

Configuration

Create the .env file from the example and set your API key:

cp .env.example .env
# Edit .env and set VLLM_API_KEY

The only required variable is:

# Generate a strong key: openssl rand -hex 32
VLLM_API_KEY="your-secret-key-here"

.env is git-ignored and must never be committed.

Starting a Model

# Using a Makefile shortcut:
make deploy-ministral-8b
make deploy-devstral-123b

# Or any model by name:
make deploy MODEL=Qwen3.6-27B

# Low-level equivalent:
docker compose -f vllm/deployment-templates/docker-compose.<model-name>.yml up -d

vLLM performs a health check at GET /health every 15 seconds with a 10-minute startup grace period. Caddy will not accept traffic until vLLM reports healthy. Allow up to 10 minutes for large models to load.

Stopping a Stack

make undeploy MODEL=<model-name>
# or:
docker compose -f vllm/deployment-templates/docker-compose.<model>.yml down

API Usage

The API is OpenAI-compatible and reachable at http://<host>/v1/.

curl http://localhost/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{
    "model": "Ministral-3-8B-Instruct-2512",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Note for Mistral-family models: The compose files pass --tokenizer_mode mistral, --config_format mistral, and --load_format mistral to use Mistral's native weight format. Without these flags vLLM falls back to the HuggingFace loader, which misidentifies these models as a Pixtral vision model and crashes at startup.

Note for structured output (xgrammar): Ministral models with structured output enabled can generate indefinitely without hitting a natural stop token. Always pass max_tokens in client API calls when using structured output.

Included Model Templates

The following models have ready-to-use compose files. These serve as templates — the repo is not limited to these models.

Model	Compose File	Notes
Ministral-3-3B-Instruct-2512	docker-compose.Ministral-3-3B-Instruct-2512.yml	Mistral native format, tool calling
Ministral-3-8B-Instruct-2512	docker-compose.Ministral-3-8B-Instruct-2512.yml	Mistral native format, tool calling
Ministral-3-14B-Instruct-2512	docker-compose.Ministral-3-14B-Instruct-2512.yml	Mistral native format, tool calling
Devstral-2-123B-Instruct-2512	docker-compose.Devstral-2-123B-Instruct-2512.yml	123B, async scheduling
gpt-oss-20B	docker-compose.gpt-oss-20B.yml	async scheduling
gpt-oss-120B	docker-compose.gpt-oss-120B.yml	async scheduling
Qwen3.6-27B	docker-compose.Qwen3.6-27B.yml	27B VLM, reasoning parser, tool calling, 131K context, requires vLLM 26.04
Qwen3-Coder-Next-FP8	docker-compose.Qwen3-Coder-Next-FP8.yml	80B MoE (3B active), FP8-quantized (~80 GB), tool calling, 256K context, no thinking mode, requires vLLM 26.04

Model weights must be present in the corresponding models/<model-name>/ directory before starting the stack.

Adding a Model

Step 1 — Download the model weights

Use the Hugging Face CLI to download weights into models/:

pip install huggingface-hub
huggingface-cli download <org>/<model-name> --local-dir models/<model-name>

The directory name under models/ is what gets mounted into the container and passed to vllm serve.

Step 2 — Create a compose file

Copy the closest existing template and rename it:

cp vllm/deployment-templates/docker-compose.Ministral-3-8B-Instruct-2512.yml \
   vllm/deployment-templates/docker-compose.<model-name>.yml

Edit the new file and update:

• volumes — change the host path to ../../models/<model-name>
• command — update the model path and any model-specific vLLM flags
  • Mistral-family models require --tokenizer_mode mistral --config_format mistral --load_format mistral
  • Standard HuggingFace models do not need these flags
• mem_limit / memswap_limit — adjust to the model's size
• container_name — keep as vllm unless running multiple stacks simultaneously

Step 3 — Start the stack

make deploy MODEL=<model-name>

---

Thermal Guard — GPU Temperature Monitoring

The thermal guard protects GPU(s) from overheating by stopping the vLLM container when a temperature threshold is exceeded. It does not auto-restart the container after cooling — that decision is left to the operator to prevent thermal oscillation.

Architecture

┌──────────────────┐
│  DCGM Exporter   │  ──►  GPU metrics at http://dcgm-exporter:9400/metrics
└──────────────────┘
         ▲
         │  polls every N seconds
         │
┌──────────────────┐
│  Thermal Guard   │  ──►  Reads DCGM_FI_DEV_GPU_TEMP
└──────────────────┘        └─► docker stop vllm  (if temp >= threshold)
         │
         ▼
┌──────────────────┐
│   vLLM Server    │
└──────────────────┘

See vllm/thermal-guard/docker/README-docker.md for full details.

All thermal commands are available from the repo root via the root Makefile:

Command	Description
make thermal-up	Start dcgm-exporter + thermal-guard
make thermal-down	Stop thermal monitoring
make thermal-logs	Follow all thermal service logs
make thermal-logs-guard	Follow thermal-guard logs only
make thermal-logs-dcgm	Follow DCGM exporter logs only
make thermal-status	Show service status
make thermal-health	Show health check status
make thermal-metrics	Verify DCGM exporter is returning GPU temp data
make thermal-rebuild	Rebuild the thermal-guard image and restart

# Quick reference:
make thermal-up
make thermal-status
make thermal-logs-guard
make thermal-down

Troubleshooting

DCGM exporter won't start

make thermal-logs-dcgm
docker info | grep -i nvidia

Thermal guard shows connection warnings

make thermal-status     # check dcgm-exporter is healthy
make thermal-metrics    # verify metrics endpoint is responding

vLLM keeps stopping

# Check current GPU temperature:
nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader

# Review thermal guard decisions:
make thermal-logs-guard

# If legitimate overtemp: improve cooling, reduce load, or raise THERMAL_THRESHOLD_C in .env

Thermal Guard Environment Variables

Variable	Default	Description
THERMAL_THRESHOLD_C	80	Stop vLLM when any GPU reaches this temperature (°C)
THERMAL_POLL_SECONDS	5	Polling interval in seconds
EXPORTER_URL	http://dcgm-exporter:9400/metrics	DCGM exporter Prometheus endpoint
CONTAINER_NAME	vllm	Name of the container to stop on overtemp
DOCKER_STOP_TIMEOUT	30	Graceful stop timeout in seconds

The DCGM exporter also exposes metrics on host port 9400, which can be scraped by Prometheus.

---

Testing the API

A smoke-test script is provided at vllm/testing/test_llm_api.py. It reads VLLM_API_KEY, API_BASE_URL, and MODEL from the root .env.

make venv    # first time only — creates .venv, installs requirements.txt
make test

Equivalent manual steps:

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python vllm/testing/test_llm_api.py

The script sends a single chat completion request to $API_BASE_URL/v1/chat/completions and prints the HTTP status code and response body.

---

Environment Files

All secrets live in a single .env at the repo root — git-ignored, sourced by all three stacks. On a fresh clone:

cp .env.example .env
# Edit .env and fill in the required values

Variable	Used by	Description
VLLM_API_KEY	deployment, testing	API key for the vLLM server
API_BASE_URL	testing	Base URL of the running vLLM stack (no trailing slash)
MODEL	testing	Model name as reported by /v1/models
THERMAL_THRESHOLD_C	thermal-guard	Stop vLLM when any GPU reaches this temperature (°C)
THERMAL_POLL_SECONDS	thermal-guard	GPU polling interval in seconds
LLM_CONTAINER_NAME	thermal-guard	Name of the container to stop on overtemp

---

Security Notes

• Never commit .env files. They are git-ignored.
• Generate a strong API key: openssl rand -hex 32. Set VLLM_API_KEY in the root .env.
• vLLM binds only on an internal Docker bridge network; Caddy is the sole public-facing ingress point.
• The thermal guard container mounts the Docker socket (/var/run/docker.sock) to stop containers on overtemp. Restrict access to the host socket accordingly.

---

Known Gaps

• TLS (HTTPS) is not configured in the provided Caddyfile. To enable it, update Caddyfile with your domain and Caddy will obtain a certificate automatically via ACME.