Anubis OSS benchmarker now has first-class omlx support and integration

[uncSoft]

As of Anubis-OSS v. 3.8.1, oMLX is natively supported as a backend for LLM testing, downloading models, toggling thinking per requsest and a ton of other stuff. Other benchmarks are great, but Anubis includes full system power metrics as well as a ton of other stuff, and the dataset is open source.

Repo: https://github.com/uncSoft/anubis-oss

Homebrew install:
brew install --cask uncsoft/anubis/anubis-oss

Release Download: https://github.com/uncSoft/anubis-oss/releases/tag/v3.8.1

Leaderboard Site: https://devpadapp.com/leaderboard.html

Open Dataset (filter by omlx backend if you want) https://devpadapp.com/explorer.html

Run Analytics: https://uncsoft.github.io/anubis-oss/analysis.html

Readme: https://devpadapp.com/anubis-oss-readme.html

Homebrew install:
brew install --cask uncsoft/anubis/anubis-oss

Release Download:

⭐ oMLX Spotlight

• First-class oMLX backend. Dedicated stock backend on port 8000 (separate from mlx-lm on 8080), with its own process detection, power/memory attribution, and owned_by: omlx identity verification.
• Accurate, server-reported metrics. Anubis now reads oMLX's own decode-loop numbers - generation tok/s, prefill tok/s, time-to-first-token, prompt-eval / generation duration, total time, model-load time, cached tokens - instead of estimating from wall-clock. This eliminates the occasional wildly-inflated tok/s and 0.00 J/tok readings.
• Verbatim metrics, your call. Session Details has an Anubis | oMLX toggle: see Anubis's hardware-measured view or oMLX's exact server-reported figures (badged "Reported directly by the oMLX server"). The raw usage block is also shown verbatim in the Inference Stats popover and saved with the run.
• Model management built in. Load and unload oMLX models on demand - from the Vault (with real on-disk sizes and loaded state) and via an eject button right next to the model picker on the Benchmark screen. Authenticated automatically with your existing API key.
• Model browser + downloader. Browse trending/popular MLX models and search HuggingFace through oMLX, then download with live progress - no leaving Anubis. Already-installed models are flagged.
• Thinking toggle. Turn a reasoning model's chain-of-thought on/off (enable_thinking) from the Benchmark screen, per-side in Arena, and in Flow "Set Parameters" steps.
• Reliability. Pins localhost connections to IPv4 (oMLX binds 127.0.0.1), and surfaces clear errors when a downloaded model isn't chat-capable (e.g. vision/embedding/experimental quants) instead of a generic failure.

omlx feature gallery

We also introduced flows- automated chaining of LLM benchmarks

What is a Flow in Anubis?

How chains, repetitions, and For Each iteration work

---

What is a Flow?

A Flow is a saved sequence of benchmark steps. Instead of clicking Run by hand for each model, prompt, or repetition, you author the sequence once and play it back. Anubis records every individual run (so it shows up in your normal Run History) and bundles them into a single Flow Report you can share.

Flows are good for:

• Comparing N models on the same prompt
• Repeating one config to get mean ± 95% confidence intervals
• Cold-start vs warm-start measurements
• Sweeping over quantizations (Q4 vs Q8) or prompt variants
• Hands-off batch runs you publish straight to YouTube/socials

---

The Editor

The editor has three panes:

• Left – Palette: Every step type you can add. Click to append; drag onto the step list for precise placement.
• Middle – Step List: Your flow, top-down execution order. Reorder by dragging or using the ↑/↓ chevrons. Containers (Repeat, For Each) nest their children with an indented rail.
• Right – Inspector: Edits the currently selected step. Different controls per step type.

The header chip on the top right tells you how many Run Benchmark invocations your flow will produce, multiplied through all containers (e.g. "15 runs" for 3 models × 5 reps).

---

Step Types at a Glance

Steps execute top-to-bottom. State setters change the running context every later Run uses; action steps produce results; management steps clean up between runs.

Step	Description
Set Backend	Choose Ollama / LM Studio / Apple Intelligence. Stays in effect for every subsequent Run until you change it.
Set Model	Pick a model on the current backend.
Set Prompt	The text the model is asked to generate from. Multi-line.
Set Parameters	Temperature, top-p, max tokens, seed strategy (random vs fixed).
Run Benchmark	Executes one inference using the current context. Produces one BenchmarkSession in Run History.
Repeat × N	Container — runs its child body N times. Reps are aggregated into a BenchmarkRunGroup (mean ± 95% CI).
For Each Model	Container — iterates over a list of model names, running its body once per model.
For Each Prompt	Container — same, but over prompt texts.
Unload Model	Asks Ollama to evict the active model from memory. No-op on other backends.
Reset Connection	Tears down + rebuilds the backend's URLSession.
Cool Down	Pauses execution — either a fixed wait or until thermal state returns to nominal.
Annotate	Free-text marker that shows up in the report and run log.

---

For Each — The Rules

For Each Model and For Each Prompt are how you batch one flow over many values. Two rules to keep in mind:

Rule 1: For Each replaces the corresponding Set step inside the loop body.

Inside a For Each Model, the current model is whatever value the iterator just picked — any prior Set Model becomes irrelevant for that iteration. The lint chip will flag a Set Model that sits right before a For Each Model: it never takes effect.

Rule 2: Nested containers multiply.

Total runs = (outer count) × (inner count) × (deeper). Three models × two prompts × five reps = 30 BenchmarkSessions.

---

Worked Example — Comparing 2 Quantizations × 5 Reps

Set Backend       Ollama
Set Prompt        "Write a haiku about Anubis."
Set Parameters    temp 0.7, max 256
For Each Model    [llama3.2:3b-q4, llama3.2:3b-q8]
  Repeat × 5
    Run Benchmark
  Unload Model

Produces 10 sessions (2 models × 5 reps). Each model gets a BenchmarkRunGroup with its own mean ± 95% CI. The unload between iterations forces a cold start so timings don't bleed across models.

---

Worked Example — Same Prompt Across 3 Models With Cooldown

Set Backend       Ollama
Set Prompt        "Summarise Hamlet in 100 words."
Set Parameters    temp 0.7, max 512
For Each Model    [llama3.2:1b, llama3.2:3b, qwen2.5:3b]
  Repeat × 3
    Run Benchmark
  Cool Down        5 seconds
  Unload Model

9 sessions total. The Flow Report ranks the three models by mean tok/s.

---

Common Mistakes the Lint Chip Will Catch

• Set Model followed by For Each Model — the Set never runs.
• For Each with a blank entry in the list — that iteration fails at runtime.
• Empty For Each body — nothing to execute per iteration.
• Run Benchmark with no prior Set Backend / Set Model / Set Prompt — fails immediately.

---

Common Patterns

Start from a template (sidebar + → New from Template). Each one is a real working flow you can edit or use as-is:

• Quick Smoke Test — the minimum viable flow. Use this as a scaffold.
• Repeated Run (5×) — one model × 5 reps with random seed. Gets mean ± CI without sampler shenanigans.
• Cold vs Warm Start — unload, time a cold run, time a warm run. The classic TTFT comparison.
• Multi-Model Comparison — For Each Model with cooldown + unload between iterations.
• Q4 vs Q8 Quantization — same family, two quants, 5 reps each.
• Prompt Variants — one model, multiple prompts of varying complexity.

---

Warnings and Lint

The editor analyses your flow as you build it. Any step with a problem gets a yellow ⚠ next to its name and an orange border. The header shows a count chip — click it for a popover listing every warning. Each row in the popover is a button that jumps to the offending step.

Note: Warnings don't block running — sometimes you genuinely want a setup-only flow. They're hints, not blockers.

---

Running a Flow + Reading the Report

Hit the green Run button (⌘R) in the editor header. The Run sheet shows live progress: a step tree with checkmarks/spinners, a scrollback log, and a progress bar reading "Run 7 of 30."

When the flow finishes, the View Report button (⇧⌘R) opens the Flow Report — a YouTube-ready 16:9 or 1:1 card with the leaderboard and methodology. Export as PNG (2× retina) or CSV.

Past runs live in the Run History section of the sidebar. Click any row to reopen its report. Right-click to delete one; the section header menu has Clear All History… (the BenchmarkSessions survive — they stay in the Benchmark tab's Run History).

[ilukashin]

I'm already using Anubis and can confirm it's a cool thing!

[uncSoft]

Thanks! You will have to manually grab the newest version with full oMLX support - unfortunately sparkle shipped with a bug in 3.7

But please try out the new features and let me know we can keep improving!

-John