🦙 LlamaHerd

One endpoint. Many llamas. Smarter routing.

Agent-first multi-key proxy for Ollama Cloud: load balancing, usage tracking, client API keys, rate limits, and a live dashboard.

clients --> [ LlamaHerd ] --> { Sub 1 | Sub 2 | Sub N }
              route by slots, session %, weekly %, health

    __    __                      __  __              __
   / /   / /___ _____ ___  ____ _/ / / /__  _________/ /
  / /   / / __ `/ __ `__ \/ __ `/ /_/ / _ \/ ___/ __  / 
 / /___/ / /_/ / / / / / / /_/ / __  /  __/ /  / /_/ /  
/_____/_/\__,_/_/ /_/ /_/\__,_/_/ /_/\___/_/   \__,_/

✨ Features

Sticky session routing — keeps the same conversation on the same upstream subscription to maximize KV-cache reuse. Falls back to hashing the start of the conversation context when clients don't send a session ID.
Multi-key load balancing — routes across N Ollama Cloud keys, preferring freshest billing cycle and least usage
Auto model discovery — polls /v1/models from each key, merges into one list
Live operations dashboard — period KPIs, urgency-sorted account health, per-model usage, and SSE-powered real-time updates
Usage attribution — per-client API keys track which service made each request
Dynamic key management — add/remove subscriptions and client keys via API, no restart needed
Cookie-based usage scraping — tracks session & weekly usage % from ollama.com/settings
429 auto-retry — upstream rate limits trigger automatic retry on the next best key
Overflow queuing — requests queue (up to 60s) instead of failing fast
OpenAI & native Ollama protocol — both /v1/* and /api/* routes supported
Context length metadata — injects correct context windows into /v1/models so clients don't fall back to 128K defaults

Sticky sessions and cache affinity

LlamaHerd implements sticky session routing with the goal of maximizing the chance that repeated turns of the same conversation hit a warm KV/context cache on the upstream Ollama Cloud subscription. This is particularly valuable for long-context and 1M-context models, where re-transmitting the full prompt to a different subscriber on every turn would be expensive.

How sessions are identified (in order):

X-LlamaHerd-Session request header.
llamaherd-session cookie returned by LlamaHerd.
X-Conversation-ID request header.
Hash of the first system + user messages from the request body.

If a client (e.g. OpenClaw using the OpenAI JS client) does not send an explicit session identifier, LlamaHerd falls back to option 4. The TTL is configurable via sticky_ttl_seconds (default 3600s).

On upstream 429/402 or other errors, the sticky mapping for that session is cleared so the next turn can rebalance to a healthier subscription.

🚀 Quick Start

# Install
pip install llamaherd

# Or from source
git clone https://github.com/bennybuoy/llamaherd.git
cd llamaherd
pip install -e .

# Configure
cp config.example.yaml config.yaml
# Edit config.yaml with your Ollama Cloud API keys

# Run
llamaherd --config config.yaml

# Or with env var overrides
LLAMAHERD_ADMIN_TOKEN=my-secret python -m llamaherd.proxy

Then point your OpenAI-compatible client at http://127.0.0.1:8399/v1.

📊 Dashboard

Open http://127.0.0.1:8399/dashboard?token=YOUR_ADMIN_TOKEN in a browser.

The dashboard features:

Operations summary — compact status header and six KPI cards for calls, tokens, in-flight requests, average latency, and recorded-call error rate
Account health rail — the canonical account overview, urgency-sorted with slot pressure, session/weekly usage, billing remaining, request counts, and 429s. It stays alongside the dashboard on wide screens and moves below the main content on narrower displays.
Overview tab — per-client, per-model, and daily breakdowns plus the live call feed
Models tab — all discovered models with context lengths, key availability, and 7-day usage stats. Search and sort.
Accounts tab — add/remove Ollama Cloud accounts, edit cookies for usage tracking, and inspect plan and billing details
Quota Cost and OpenRouter $ tabs — quota-equivalent usage and fallback-provider cost reporting
Time period filtering — Today, Yesterday, Last 7 days, This Week, This Month, Last Month, or Custom dates
SSE live updates — new calls and status changes stream in real-time, no polling

🔑 Client Keys

LlamaHerd uses internal client keys for usage attribution (not real Ollama keys). Manage them via API:

# Create a client
curl -X POST http://127.0.0.1:8399/admin/clients \
  -H "Authorization: Bearer YOUR_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"id": "my-app", "label": "My Application"}'

# List clients
curl http://127.0.0.1:8399/admin/clients \
  -H "Authorization: Bearer YOUR_ADMIN_TOKEN"

Then point your client at http://127.0.0.1:8399/v1 with Authorization: Bearer ocp-my-app-....

🍪 Usage Tracking Cookies

Usage scraping is optional. Install it with pip install 'llamaherd[scraping]' when installing from PyPI/source; the Docker image includes it by default.

To see session and weekly usage percentages, you need browser cookies from each Ollama account:

Log into ollama.com/settings in your browser
Open DevTools → Application → Cookies → ollama.com
Copy __Secure-session (required, per-account), aid, cf_clearance, and __stripe_mid
Add them to config.yaml under each key's cookies section, or via the Accounts tab in the dashboard

🐳 Docker (recommended: docker-compose)

A docker-compose.yml is provided at the repo root. The build context is the repo itself (the Dockerfile lives at the repo root), so a fresh clone just needs config.yaml, data/, and a one-time chown.

Production layout on atrium-services (100.69.208.20) or any fresh host:

  <repo-root>/                     <- this is the git clone
  ├── docker-compose.yml           # tracked in repo
  ├── Dockerfile                   # tracked in repo
  ├── llamaherd/                   # package source (tracked in repo)
  ├── config.yaml                  # NOT in repo (admin_token, upstream keys)
  ├── openrouter_pricing.yaml
  └── data/                        # bind-mounted SQLite + cache files
      ├── proxy.db
      ├── usage.db
      └── nvidia_model_cache.json

First-time setup

# 1. Clone the repo into the deploy directory
git clone https://github.com/bennybuoy/llamaherd.git /srv/services/llamaherd
cd /srv/services/llamaherd

# 2. Copy config.example.yaml to config.yaml and edit (admin_token, upstream keys)
cp config.example.yaml config.yaml

# 3. Create the data dir and pre-create the bind-mounted files
#    (Docker would otherwise create directories at those paths on first mount)
mkdir -p data
touch data/proxy.db data/usage.db data/nvidia_model_cache.json

# 4. Chown the files so the container's llamaherd user (uid 999) can write
sudo chown 999:999 data/proxy.db data/usage.db data/nvidia_model_cache.json

# 5. Build and run
docker compose build && docker compose up -d

Without step 4, chat completions return 500 with sqlite3.OperationalError: attempt to write a readonly database. Without step 3, Docker creates directories at the bind-mount paths, and SQLite / cache loads fail in confusing ways.

Sidecar workflow for code changes

The proxy is the inference backend for every Hermes profile — a broken deploy affects every agent. Don't push code changes directly to prod. The recommended flow is:

Clone a second copy of the repo into /srv/services/llamaherd-sidecar/ (or any other host) and run it on port 8499 with an isolated DB.
Develop on a feature branch in the sidecar.
Smoke-test against :8499 before merging.
After review: merge to main, then docker compose build && up -d on prod.

🤖 CLI (Agent-Friendly)

LlamaHerd has a full CLI designed for agents and scripts. All commands output JSON by default (pipe to jq for filtering). Use --format table for human-readable output.

# List clients with rate limits
llamaherd clients list -f table

# Create a client with rate limits
llamaherd clients create my-app --label "My App" --daily-token-limit 100000 --rpm-limit 30

# Update limits (null = unlimited)
llamaherd clients update my-app --daily-request-limit 500

# Clear all limits (set to unlimited)
llamaherd clients update my-app --clear-limits

# Regenerate a compromised token
llamaherd clients regenerate-token my-app

# Delete a client
llamaherd clients delete my-app

# Show proxy status
llamaherd status -f table

# List upstream keys
llamaherd keys -f table

# List discovered models
llamaherd models -f table

# Usage stats
llamaherd usage --days 7

# Connection options (defaults to 127.0.0.1:8399)
llamaherd --host 10.0.0.1 --port 8399 -t YOUR_ADMIN_TOKEN clients list

Rate Limiting

Each client key can have optional rate limits:

Limit	Field	Unit	Example
Daily tokens	`daily_token_limit`	tokens (in+out) per day	`100000`
Daily requests	`daily_request_limit`	requests per day	`500`
RPM	`rpm_limit`	requests per minute	`30`

When a limit is exceeded, the proxy returns 429 with a JSON body:

{
  "error": "rate_limit_exceeded",
  "detail": "Daily token limit of 100000 exceeded for client 'my-app'",
  "limit_type": "daily_tokens",
  "limit": 100000,
  "used": 104312,
  "reset_at": 1719792000
}

Set limits via API or CLI. null means unlimited (default).

🔌 Connecting Agents

Hermes Agent

Hermes currently uses LlamaHerd through its OpenAI-compatible route. Include /v1 in the base URL:

model:
  default: deepseek-v3.2
  provider: custom
  base_url: http://127.0.0.1:8399/v1
  api_key: ocp-hermes-gateway-XXXXXXXX

Or via CLI:

hermes config set model.provider custom
hermes config set model.default deepseek-v3.2
hermes config set model.base_url http://127.0.0.1:8399/v1
hermes config set model.api_key ocp-hermes-gateway-XXXXXXXX

Create the client key first:

llamaherd clients create hermes-gateway --label "Hermes Gateway" --rpm-limit 60

OpenClaw

OpenClaw should use LlamaHerd's native Ollama API, not the OpenAI-compatible /v1 route. Do not add /v1; using /v1 can break tool calling.

# In your OpenClaw config
baseUrl: http://127.0.0.1:8399
api: ollama
apiKey: ocp-openclaw-gateway-XXXXXXXX

Create the client key first:

llamaherd clients create openclaw-gateway --label "OpenClaw Gateway" --rpm-limit 60

Copy-paste agent setup prompt

If you are asking an agent to configure itself, give it this prompt and the client token privately:

Configure this agent to use LlamaHerd, the local Ollama Cloud proxy.

Endpoint rules:
- If this agent is OpenClaw, use native Ollama protocol: baseUrl http://127.0.0.1:8399, api ollama, no /v1 suffix.
- If this agent is Hermes Agent or another OpenAI-compatible client, use base_url http://127.0.0.1:8399/v1.
- Use the provided client API key as the Authorization bearer/api_key.
- Pick a model from GET /api/tags for native Ollama clients or GET /v1/models for OpenAI-compatible clients.
- Verify with a small chat request and do not print the API key.

Any OpenAI Client

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8399/v1",
    api_key="ocp-my-app-XXXXXXXX",
)

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[{"role": "user", "content": "Hello!"}],
)

📡 API Endpoints

Endpoint	Method	Purpose
`/admin/status`	GET	Key health, slots, usage %, billing data
`/admin/events?token=X`	GET (SSE)	Real-time event stream
`/admin/models`	GET	Discovered models with context length, capabilities, size, family, parameters
`/admin/totals`	GET	All-time or date-filtered totals
`/admin/keys`	GET/POST/PUT/DELETE	Manage subscription keys
`/admin/clients`	GET/POST/PUT/DELETE	Manage client attribution keys
`/admin/usage/*`	GET	Usage breakdowns (by-client, by-model, daily)
`/admin/recent-calls`	GET	Individual call log
`/admin/refresh`	POST	Trigger model discovery
`/admin/poll-subscriptions`	POST	Poll /api/me for all keys
`/admin/scrape-usage`	POST	Scrape usage from ollama.com/settings
`/v1/*`	ANY	OpenAI-compatible proxy routes
`/api/*`	ANY	Native Ollama protocol routes

All /admin/* endpoints require the admin token, either in the Authorization bearer header or as a ?token=... query param.

Date Range Filtering

Usage endpoints support start_date and end_date params (ISO date: 2025-01-15):

GET /admin/usage/by-client?start_date=2025-05-01&end_date=2025-05-03
GET /admin/totals?start_date=2025-05-01&end_date=2025-05-03

SSE Events

Connect to /admin/events?token=YOUR_TOKEN for real-time updates:

Event	When	Data
`status`	After subscription poll or usage scrape	Key health array
`call`	Every proxied request completes	Call record (model, tokens, latency)
`models`	Model registry refreshes	Count, new models list
`heartbeat`	Every 15 seconds	Keepalive

⚙️ Configuration

See config.example.yaml for all options. Key settings:

keys — List of Ollama Cloud subscriptions with token, max_concurrent, cycle_day, label, and optional cookies
clients — Internal attribution keys (can also be managed via API)
admin_token — Secret for dashboard and admin API access
usage_db — SQLite database path for usage tracking
health_check_interval — Seconds between model discovery (default 300)
usage_activity_debounce — Delay after account activity before refreshing its usage (default 30 seconds)
usage_activity_min_interval — Minimum time between refreshes for the same active account (default 300 seconds)

🏗️ Architecture

┌──────────┐     ┌──────────────┐     ┌─────────────┐
│  Hermes  │────▶│  LlamaHerd   │────▶│ ollama.com  │
│  Cron    │     │  :8399       │     │  /v1/*       │
│  Eval    │────▶│              │────▶│  /api/*      │
└──────────┘     │  ┌────────┐  │     └─────────────┘
                 │  │Key Mgr │  │     ┌─────────────┐
                 │  │(rotate)│  │────▶│ Key 1 (pro) │
                 │  └────────┘  │     │ Key 2 (pro) │
                 │  ┌────────┐  │     │ Key 3 (pro) │
                 │  │UsageDB │  │     └─────────────┘
                 │  │(SQLite)│  │
                 │  └────────┘  │
                 │  ┌────────┐  │
                 │  │Dashboard│  │
                 │  │(SSE)    │  │
                 │  └────────┘  │
                 └──────────────┘

📝 License

MIT — see LICENSE.

Name		Name	Last commit message	Last commit date
Latest commit History 117 Commits
.github/workflows		.github/workflows
assets		assets
docs		docs
llamaherd		llamaherd
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
AGENTS.md		AGENTS.md
BRANDING.md		BRANDING.md
CLAUDE.md		CLAUDE.md
DASHBOARD_FEATURES.md		DASHBOARD_FEATURES.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
config.example.yaml		config.example.yaml
docker-compose.yml		docker-compose.yml
openrouter_pricing.yaml		openrouter_pricing.yaml
pyproject.toml		pyproject.toml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

🦙 LlamaHerd

✨ Features

Sticky sessions and cache affinity

🚀 Quick Start

📊 Dashboard

🔑 Client Keys

🍪 Usage Tracking Cookies

🐳 Docker (recommended: docker-compose)

First-time setup

Sidecar workflow for code changes

🤖 CLI (Agent-Friendly)

Rate Limiting

🔌 Connecting Agents

Hermes Agent

OpenClaw

Copy-paste agent setup prompt

Any OpenAI Client

📡 API Endpoints

Date Range Filtering

SSE Events

⚙️ Configuration

🏗️ Architecture

📝 License

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

🦙 LlamaHerd

✨ Features

Sticky sessions and cache affinity

🚀 Quick Start

📊 Dashboard

🔑 Client Keys

🍪 Usage Tracking Cookies

🐳 Docker (recommended: docker-compose)

First-time setup

Sidecar workflow for code changes

🤖 CLI (Agent-Friendly)

Rate Limiting

🔌 Connecting Agents

Hermes Agent

OpenClaw

Copy-paste agent setup prompt

Any OpenAI Client

📡 API Endpoints

Date Range Filtering

SSE Events

⚙️ Configuration

🏗️ Architecture

📝 License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages