Kimi-Linear-48B-A3B: vLLM Docker Image Project (English)

This repository is a pure Docker image project for serving the Kimi-Linear-48B-A3B-Instruct (AWQ-4bit) model via vLLM. vLLM is used as a component inside the image; this project focuses on building, running, and publishing the image.

• Other languages: 简体中文 | 繁體中文 | 日本語

Features

• Based on vllm/vllm-openai:nightly
• Installs fla-core for Kimi-Linear compatibility
• OpenAI-compatible API
• Environment overrides for TP, context length (128K → 1M), GPU memory utilization, and concurrency
• Makefile for convenient build/run/push

Build

docker build -t neosun100/kimi-linear-vllm:latest .

Run

export HF_HOME="$HOME/.cache/huggingface"
export VLLM_DOWNLOAD_DIR="$HOME/vllm_downloads"
mkdir -p "$HF_HOME" "$VLLM_DOWNLOAD_DIR"

docker run --gpus all -d --name kimi48b-awq --restart unless-stopped \
  --ipc=host -p 8002:8000 \
  -v "$HF_HOME":/root/.cache/huggingface \
  -v "$VLLM_DOWNLOAD_DIR":/data/vllm_downloads \
  neosun100/kimi-linear-vllm:latest

Health check & chat

curl http://localhost:8002/v1/models

curl http://localhost:8002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cyankiwi/Kimi-Linear-48B-A3B-Instruct-AWQ-4bit",
    "messages": [{"role":"user","content":"Hi!"}],
    "max_tokens": 64
  }'

Curl testing cookbook

• List models (expect JSON with model id):

curl -sS http://localhost:8002/v1/models | jq .

• Non-stream chat (waits for full response):

curl -sS http://localhost:8002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cyankiwi/Kimi-Linear-48B-A3B-Instruct-AWQ-4bit",
    "messages": [
      {"role": "user", "content": "Give me a one-sentence fun fact about space."}
    ],
    "max_tokens": 128,
    "stream": false
  }' | jq .

• Stream chat (tokens stream incrementally):

curl -sN http://localhost:8002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cyankiwi/Kimi-Linear-48B-A3B-Instruct-AWQ-4bit",
    "messages": [
      {"role": "user", "content": "Write a single witty one-liner."}
    ],
    "stream": true
  }'

Scripted tests (recommended)

# List models
HOST=localhost PORT=8002 ./scripts/test_models.sh

# Non-stream chat (Chinese prompt by default)
HOST=localhost PORT=8002 MAX_TOKENS=128 ./scripts/test_chat.sh

# Stream chat (prints tokens progressively)
HOST=localhost PORT=8002 ./scripts/test_stream.sh

Environment overrides

• MODEL (default: cyankiwi/Kimi-Linear-48B-A3B-Instruct-AWQ-4bit)
• PORT (default: 8000)
• TP (default: 4)
• MAX_LEN (default: 131072) → step up to 1048576 for 1M
• GPU_MEM_UTIL (default: 0.5)
• MAX_NUM_SEQS (default: 64)
• DOWNLOAD_DIR (default: /data/vllm_downloads)

Example:

docker run --gpus all -d --name kimi48b-awq -p 8002:8000 \
  -v "$HF_HOME":/root/.cache/huggingface \
  -v "$VLLM_DOWNLOAD_DIR":/data/vllm_downloads \
  -e MAX_LEN=1048576 -e GPU_MEM_UTIL=0.45 -e MAX_NUM_SEQS=32 \
  neosun100/kimi-linear-vllm:latest

Push

Docker Hub

# docker login
docker push neosun100/kimi-linear-vllm:latest

GitHub Container Registry (optional)

# gh auth login (write:packages)
docker tag neosun100/kimi-linear-vllm:latest ghcr.io/neosun100/kimi-linear-vllm:latest
docker push ghcr.io/neosun100/kimi-linear-vllm:latest

Makefile shortcuts

make build
make run HOST_PORT=8002
make logs
make stop
make push
make ghcr-push

Notes

• fla-core is installed in-image for Kimi-Linear support.
• Start with 128K for stability; escalate to 1M as resources allow.
• We avoid explicit --quantization so vLLM auto-detects AWQ for this model.

Model switching

⚠️ Important: vLLM loads models at startup. To switch models:

• Option 1: Restart container with new MODEL env var
• Option 2: Run multiple containers (one model per container, different ports)

❌ Not possible: Changing model in curl requests won't auto-download/load new models.

See docs/MODEL_SWITCHING.md for details.