SERVER_SHOOTOUT.md

Strix Halo Server Shootout

Status: active benchmark track, started 2026-05-05.

This track answers a practical question:

How should I use a Strix Halo local AI PC, and which local server should run each job?

Single-user llama-bench numbers are useful, but they do not tell you whether the box feels good as a daily AI service. A local AI PC is useful when it can serve chat, coding tools, agents, document search, and background jobs through stable APIs.

Current Answer

As of the May 2026 baseline:

Use case	Start with	Why	Status
Beginner local chat	Ollama Vulkan/RADV	easiest model management and setup	measured single-user baseline
Open WebUI private chat/docs	Ollama first, then llama-server if you need speed	lowest friction, OpenAI-compatible clients work	measured Ollama baseline; RAG not yet benchmarked
Coding assistant or local scripts	llama-server Vulkan/RADV	best measured Qwen3.6 path at 1-4 parallel requests	measured
Multiple local tools/users	Lemonade llamacpp-rocm b1259	best measured Qwen3.6 aggregate throughput at 8-16 parallel requests	measured
vLLM/API appliance	kyuz0, Lemonade vLLM, or AWQ/DFlash ROCm containers	serving-oriented stack with batching and production APIs	small smoke test only; 27B/35B throughput pending locally
Long-context and RAG ingestion	keep both Vulkan/RADV and ROCm/HIP paths available	HIP can win prompt processing while Vulkan still wins generation	partially measured
Image/video generation	kyuz0 ComfyUI toolboxes	separate ROCm container path for diffusion/video workloads	outside this LLM server shootout

The practical split is now clear for Qwen3.6 35B-A3B UD-Q4_K_M: use Vulkan/RADV llama-server for single-user and small parallel workloads, and use Lemonade llamacpp-rocm b1259 when the box is serving many simultaneous local requests. The open question is whether current vLLM/ROCm builds are better for agent-serving, batching, tool APIs, or long-running server workloads.

Runtime caveat (2026-08-30 review): b1259 remains the build measured in the 2026-05-05 concurrency sweep. Lemonade v11.8.0, released 2026-08-27, is current but unqualified here; the 8-16 parallel advice has not been re-benchmarked on that release.

What Counts As A Server

A server is anything that exposes a model to other tools through an HTTP API.

Server	What it is good for	Main tradeoff
Ollama	easy setup, model pulling, local chat, Open WebUI	lower short-context speed than direct llama-server in current tests
llama-server Vulkan/RADV	fastest measured Qwen3.6 path at 1-4 parallel requests	manual model files and flags
llama-server ROCm/Lemonade	strongest measured Qwen3.6 aggregate throughput at 8-16 parallel requests	packaged build, more setup than Vulkan
kyuz0 vLLM toolbox	isolated Strix Halo vLLM container path	heavier stack, throughput still unmeasured here
Lemonade vllm-rocm	portable gfx1151 vLLM build	candidate, not yet measured here
Experimental AWQ/DFlash repos	promising vLLM speculative decoding experiments	not canonical until reproduced locally

Shootout Matrix

The canonical data file is data/server_shootout.csv.

Current measured/pending summary:

Server	Backend	Model	Concurrency	Result	Verdict
llama-server	Vulkan/RADV	Qwen3.6 35B-A3B UD-Q4_K_M	1-16 sweep	58.80 to 189.72 aggregate t/s	best low-concurrency path
llama-server	Vulkan/RADV	Qwen3-Coder 30B-A3B UD-Q4_K_XL	8	173.16 aggregate t/s, 0.382 s mean TTFT	best measured coding API point
Ollama	Vulkan/RADV	Qwen3.6 35B-A3B Q4_K_M	1	50.51 t/s controlled API warm average	easiest useful path
kyuz0 vLLM toolbox	ROCm/TheRock	Qwen3-0.6B	1	OpenAI-compatible smoke test passed	serving works, throughput not proven
Lemonade llamacpp-rocm b1259	ROCm 7.13	Qwen3.6 35B-A3B UD-Q4_K_M	1-16 sweep	48.62 to 207.81 aggregate t/s	best high-concurrency path
Lemonade vllm-rocm 0.20.1 gfx1151	ROCm 7.12	pending	pending	candidate	needs measured comparison
hec-ovi vllm-awq4-qwen	ROCm/TheRock	Qwen3.6-27B AWQ-INT4	pending locally	candidate	important AWQ/DFlash lead; not reproduced here

Backend Workload Split

The server recommendation now has an important nuance: Vulkan/RADV is not the universal winner. A local HIP/Vulkan spot check and an independent same-build Strix Halo study both point to the same workload split:

• Vulkan/RADV remains better for token generation, short responses, coding loops, and low-concurrency chat.
• ROCm/HIP can be better for prompt-processing-heavy work, including long prompts, RAG ingestion, summarization, and some batch/server workloads.

Local evidence: BACKEND_CROSSOVER.md, data/backend_crossover.csv, charts/backend_crossover_prefill.svg, charts/backend_crossover_generation.svg.

Upstream watchlist: ROCM_VLLM_BUGWATCH.md.

Qwen3.6 Full Sweep

Measured 2026-05-05 on the Beelink GTR9 Pro with the normal workstation baseline recorded. Each row is a 5-rep streaming /v1/completions run, 128 generated tokens per request, 4096 context tokens per slot, and 0 throughput errors.

Parallel requests	Vulkan/RADV aggregate t/s	Vulkan request-mean interval p95	Lemonade ROCm aggregate t/s	Lemonade request-mean interval p95	Read
1	58.80	16.2 ms	48.62	20.0 ms	Vulkan wins for single-user chat/scripts
2	96.08	19.8 ms	82.19	23.6 ms	Vulkan still wins
4	138.83	27.4 ms	127.03	30.4 ms	Vulkan still wins
8	170.87	45.2 ms	177.17	43.5 ms	Lemonade wins aggregate and request-mean decode-interval summary
16	189.72	81.9 ms	207.81	74.3 ms	Lemonade is the high-concurrency winner

Raw data: data/raw/2026-05-05/server-shootout/full-sweep-qwen36-workstation-baseline/.

The interval columns are the mean across repeats of each batch's p95 of per-request mean decode intervals: (stream end − first visible content) / (reported tokens − 1). They include the protocol tail and are not the p95 of individual token gaps. Original raw itl field names are retained as historical evidence; authored CSV headers name the actual metric. No individual token-arrival series exists here to reconstruct a token-gap percentile.

Historical May campaigns retain their reported counts and throughput; their saved detail/summary files do not preserve the original SSE events or an explicit count-source field. The reusable harness's former chunk fallback does not prove those counts wrong, but the saved aggregate alone cannot independently establish usage provenance. The corrected harness requires valid server usage and a complete stream, and reports unavailable throughput for any invalid batch.

Repeat validation after workstation connection work:

Server	Parallel	Baseline aggregate t/s	Repeat aggregate t/s	Delta
llama-server Vulkan/RADV	1	58.80	59.56	+1.3%
llama-server Vulkan/RADV	2	96.08	97.34	+1.3%
llama-server Vulkan/RADV	4	138.83	138.12	-0.5%
llama-server Vulkan/RADV	8	170.87	172.96	+1.2%
llama-server Vulkan/RADV	16	189.72	190.77	+0.6%
Lemonade llamacpp-rocm	1	48.62	51.40	+5.7%
Lemonade llamacpp-rocm	2	82.19	84.72	+3.1%
Lemonade llamacpp-rocm	4	127.03	125.75	-1.0%
Lemonade llamacpp-rocm	8	177.17	181.95	+2.7%
Lemonade llamacpp-rocm	16	207.81	212.38	+2.2%

The repeat did not change the recommendation: Vulkan/RADV remains better at 1-4 parallel requests, and Lemonade ROCm remains better at 8-16.

Smoke Runs

These runs validate the workflow and server behavior, but they are not headline benchmark claims.

Date	Server	Model	Condition	Result
2026-05-05	llama-server Vulkan/RADV	Qwen3.6 35B-A3B UD-Q4_K_M	Normal workstation baseline recorded, VM paused	OpenAI-compatible /v1/completions at -np 8: 173.36 aggregate t/s, 0 errors; feature probe returned HTTP 200 for models, completions, chat, streaming chat, and tools schema
2026-05-05	Lemonade llamacpp-rocm b1259	Qwen3.6 35B-A3B UD-Q4_K_M	Normal workstation baseline recorded, VM paused	OpenAI-compatible /v1/completions at -np 8: 176.43 aggregate t/s, 0 errors; feature probe returned HTTP 200 for models, completions, chat, streaming chat, and tools schema

Benchmark Protocol

Keep this boring and reproducible.

0. Confirm the system is clean enough:

scripts/check_benchmark_cleanliness.sh

This script is read-only. It does not stop local services or mutate benchmark state. On the maintainer workstation it also reports protected local workflow dependencies; third-party replicators should record equivalent background state rather than treating those services as part of the public benchmark requirement. Maintainer-only guardrails live in MAINTAINER_NOTES.md.

1. Record host state:
   • kernel
   • Mesa/RADV or ROCm version
   • tuned-adm active
   • GPU clock
   • container image/tag if applicable
   • model name, quant, hash/path
2. Start the server with exact flags.
3. Run a feature probe:

python3 scripts/check_openai_server_features.py \
  --url http://127.0.0.1:8080 \
  --model MODEL_NAME \
  --output data/raw/DATE/SERVER/features.json

4. Run the streaming throughput harness:

python3 scripts/benchmark_openai_server.py \
  --url http://127.0.0.1:8080 \
  --model MODEL_NAME \
  --np 8 \
  --tokens 128 \
  --reps 3 \
  --detail data/raw/DATE/SERVER/detail-np8.csv \
  --summary data/raw/DATE/SERVER/summary-np8.csv

5. Sweep concurrency:
   • 1
   • 2
   • 4
   • 8
   • 16
6. Store raw logs and CSV summaries.
7. Add only defensible rows to data/server_shootout.csv.

Required Metrics

Minimum publishable row:

• server and backend
• exact build or image tag
• model and quant
• context limit
• endpoint tested
• concurrency
• tokens per request
• aggregate tokens/sec
• mean and p95 TTFT
• p95 of per-request mean decode intervals (or actual token-gap percentiles only with retained token-arrival evidence)
• error count
• setup friction
• limitations

Do not compare two rows as "faster" unless model, quant, prompt, endpoint, token count, context, and concurrency are close enough to be meaningful.

The harness defaults to a fixed token-budget benchmark. --completion-mode natural permits a valid shorter natural completion and records a different workload contract. Both require visible output, integer server token usage, a valid finish reason and the stream terminator. New request_wall_tps includes the entire request; historical May 5 request_tps used a post-first-content interval and must not be pooled with it. May 3 native /completion results used full request wall time and explicitly disabled prompt caching; those are a separate harness/campaign contract. Record prompt/cache policy and warmup conditions separately; a cache being enabled does not prove a historical cache hit. Feature probes distinguish HTTP acceptance from validated content or tool emission. Forced and automatic tool selection are separate checks; neither executes the tool or qualifies an end-to-end client task.

Feature Probes

Throughput is not enough for a useful local AI PC. Each server should also be tested for:

Feature	Why it matters
/v1/models	clients can discover models
/v1/completions	simple scripts and benchmark harness
/v1/chat/completions	OpenAI-style chat clients
streaming chat	interactive latency
tool/function calling	agents and coding workflows
long-running stability	daily local server use
metrics endpoint	monitoring and debugging

Tool calling is model-dependent, so failures must be described carefully. A server can support tool-call syntax while a model still chooses not to call a tool.

Validity Rules

• Do not install vLLM, PyTorch, ROCm, or TheRock packages into the host Python environment.
• Test stable container/builds before nightly/latest builds.
• Do not run publishable numbers while remote desktop streaming, video calls, unrelated VMs, or unrelated local AI services are active.
• Record any required local workflow services separately from the benchmark server under test.
• Treat external vLLM/DFlash claims as leads until reproduced locally.
• Do not publish tokens-per-watt until power telemetry is validated.
• Keep failed starts, OOMs, compile hangs, and missing endpoints in the notes. Failure cases are part of the value.

Next Run Order

1. Done: current llama-server Vulkan/RADV baseline through the new feature probe.
2. Done: Lemonade llamacpp-rocm b1259 with the same model/concurrency shape.
3. Done: full sweep for the two best llama.cpp paths with the workstation baseline recorded.
4. Test a comparable vLLM path in a container only: kyuz0 stable, Lemonade vllm-rocm, or hec-ovi AWQ/DFlash if model access and build time are available.
5. Run a true same-build HIP versus Vulkan comparison before promoting backend-crossover claims beyond "spot check".
6. Keep experimental AWQ/DFlash rows clearly labeled until locally reproduced.