feat(llm): rotate across multiple OpenRouter keys when an account fails

OPENROUTER_FALLBACK_KEYS (comma-separated) adds backup OpenRouter
accounts behind the primary key:

- llm.js builds one provider entry per key, so the platform chain
  (tutor, fact-check, persona, agents) fails over key-by-key. Fallback
  keys pair with the model's :free variant — they are typically
  unfunded free-tier accounts where the paid model would always 402.
- /api/chat/proxy retries the next key on account-level statuses
  (401/402/403/429) before any byte streams; request-shaped errors
  stay final.
- Pricing strips the #n multi-key suffix so every OpenRouter key
  prices to zero on the spend dashboard.

Also carries the in-flight free-first provider-tier refactor of llm.js
(NVIDIA NIM tier, server Anthropic as opt-in last resort) whose tests
are green. New fallback key validated live against the :free model and
set in Vercel prod+preview via the REST API.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: nirholas

CHANGELOG.md

@@ -7,6 +7,7 @@ Public history for [three.ws](https://three.ws), newest first. New pages come fr
 ## 2026-06-11
 
 - **Avatar pages back online after image-engine outage** — Fetching an individual avatar was failing with a server error, which also broke the walking avatar on the home page. The image-processing engine the avatar dress-up baker relies on failed to load in production and took the whole endpoint down with it. Avatar lookups now load independently of the baker, and the missing image libraries ship with the deployment, so avatar pages and the home hero are back. `[fix, infra]`
+- **Built-in AI stays up when a provider account runs dry** — The platform's built-in AI (chat, tutor, fact-checker, persona tools) now rotates across multiple OpenRouter accounts: when one account runs out of credits or hits a rate limit, the next takes over automatically, before any of the response has streamed. Paid x402 endpoints that previously returned errors during a provider outage now degrade to a backup account instead of failing. `[improvement, infra]`
 - **Faster, quieter agents directory and site-wide page polish** — The on-chain agents directory now loads from our server-side index instead of contacting every agent's metadata host from your browser — pages render faster and no longer spray network errors. Also: the pump.fun cockpit's sidebar pages are now shareable deep links, mobile nav and footer links meet touch-target guidelines, anonymous visitors no longer trigger failed sign-in requests on Walk and Create, and assorted dead links were removed. (`/agents`) `[improvement, fix]`
 - **Live $THREE market data, longer trade streams, and tutor session fixes** — $THREE market stats (price, holders, liquidity) are flowing again from our primary data source — a missing request header had been silently failing it to backups for days. Live trade streams no longer cut out after 30 seconds; they now run their full duration. The Pay-As-You-Learn Tutor no longer errors when resuming a session — running tabs persist correctly across questions — and Fact Checker results are cached properly so repeat checks of the same claim return instantly. `[fix, improvement]`
 - **Scheduled jobs restored after dependency outage** — Every background job — pump.fun monitoring, coin payouts, club payouts, scheduled X posts, DCA runs, subscription billing, and the rest — had been failing since a recent dependency upgrade left a required Solana staking library out of the deployment. The missing library now ships with every deploy, and the image-processing engine behind avatar baking was rolled back to its proven version so it loads reliably in production. All scheduled jobs run again. `[fix, infra]`

api/_lib/env.js

@@ -728,6 +728,17 @@ export const env = {
 		return opt('OPENROUTER_API_KEY');
 	},
 
+	// Additional OpenRouter keys, comma-separated, tried in order after
+	// OPENROUTER_API_KEY fails (credits exhausted, rate-limited, revoked).
+	// Unfunded free-tier keys belong here: the llm.js failover pairs fallback
+	// keys with the model's :free variant so they can still serve.
+	get OPENROUTER_FALLBACK_KEYS() {
+		return (opt('OPENROUTER_FALLBACK_KEYS') || '')
+			.split(',')
+			.map((k) => k.trim())
+			.filter(Boolean);
+	},
+
 	// Alibaba Cloud DashScope (international) — direct Qwen access. Used by
 	// /api/brain/chat when the user selects a Qwen provider. Falls back to
 	// OPENROUTER_API_KEY when unset.

api/_lib/llm-pricing.js

@@ -3,11 +3,11 @@
 // without float drift. This is the single source of truth for "what did that
 // call cost us" — the admin spend dashboard reads the events this priced.
 //
-// Anthropic prices are list price per 1M tokens (input / output), current as of
-// the Claude model catalog. Groq and OpenRouter are platform-funded free tiers
-// (we hold the key, callers pay nothing), so their marginal cost to us is $0 —
-// they are intentionally priced at zero, not omitted, so the dashboard can show
-// "calls served free" alongside paid spend.
+// Anthropic and OpenAI prices are list price per 1M tokens (input / output),
+// current as of each vendor's model catalog. Groq, OpenRouter, and NVIDIA NIM
+// are platform-funded free tiers (we hold the key, callers pay nothing), so
+// their marginal cost to us is $0 — they are intentionally priced at zero, not
+// omitted, so the dashboard can show "calls served free" alongside paid spend.
 
 // USD per 1,000,000 tokens, [input, output]. Keys are matched by prefix so a
 // dated alias (claude-haiku-4-5-20251001) resolves to its family price.
@@ -21,6 +21,8 @@ const PRICE_PER_MTOK = {
 	'claude-sonnet-4-6': [3, 15],
 	'claude-sonnet-4-5': [3, 15],
 	'claude-haiku-4-5': [1, 5],
+	'gpt-4o-mini': [0.15, 0.6],
+	'gpt-4o': [2.5, 10],
 };
 
 // Providers whose marginal cost to the platform is zero (platform-funded keys).
@@ -41,7 +43,9 @@ function priceForModel(model) {
 // or 0 when the provider is free or the model is unpriced — never null, so the
 // caller can always record a numeric cost.
 export function costMicroUsd({ provider, model, input = 0, output = 0 } = {}) {
-	if (provider && FREE_PROVIDERS.has(provider)) return 0;
+	// Multi-key providers carry a #n suffix (openrouter#2) — strip it so every
+	// key of a free provider prices to zero.
+	if (provider && FREE_PROVIDERS.has(String(provider).split('#')[0])) return 0;
 	const price = priceForModel(model);
 	if (!price) return 0;
 	const [inPerM, outPerM] = price;

api/_lib/llm.js

@@ -2,15 +2,22 @@
 //
 // Policy (do not re-implement per endpoint — that is how this drifted before):
 //
-//   • Groq and OpenRouter are PLATFORM-FUNDED. The server holds those keys and
-//     callers use them for free. They are the default providers, tried in order.
+//   • FREE PROVIDERS FIRST, ALWAYS. Groq, OpenRouter, and NVIDIA NIM are
+//     platform-funded free tiers — the server holds those keys and callers use
+//     them at zero marginal cost. They form the default chain, tried in order,
+//     and every flow must survive on them alone: the paid keys in prod are
+//     routinely invalid or out of quota, so a chain that depends on them fails.
 //
-//   • Anthropic is used when the caller passes an explicit BYOK `anthropicKey`
-//     (e.g. an agent owner's own key), OR when the caller opts in with
-//     `serverAnthropic: true` and a server `ANTHROPIC_API_KEY` is configured.
-//     Both are OFF by default: a caller that passes neither never touches
-//     Anthropic, and no flow hard-fails when the key is absent — every flow
-//     still degrades to the free providers.
+//   • Paid server keys are the LAST-RESORT tier, automatically. When
+//     ANTHROPIC_API_KEY or OPENAI_API_KEY is configured, those providers are
+//     appended to the tail of EVERY chain so a request that exhausted the
+//     free providers still succeeds instead of erroring. They never lead, and
+//     no flow hard-fails when they are absent or out of quota.
+//
+//   • BYOK is the one exception to free-first: a caller-supplied
+//     `anthropicKey` (e.g. an agent owner's own key) leads the chain — that's
+//     the caller's explicit model choice on the caller's own billing — still
+//     degrading to the free chain on failure.
 //
 // Consolidated from the multi-provider fallback that already lived in
 // api/persona/extract.js and api/persona/preview.js.
@@ -21,12 +28,19 @@ import { costMicroUsd } from './llm-pricing.js';
 
 const GROQ_MODEL = 'llama-3.3-70b-versatile';
 const OPENROUTER_MODEL = 'meta-llama/llama-3.3-70b-instruct';
+// Same Llama 3.3 70B family on NVIDIA NIM (build.nvidia.com) — one free nvapi
+// key, OpenAI-compatible, so the chain degrades across providers without
+// changing model behavior.
+const NVIDIA_MODEL = 'meta/llama-3.3-70b-instruct';
 const ANTHROPIC_MODEL = 'claude-haiku-4-5-20251001';
+// Paid last-resort tail (see policy above). Mini keeps the backstop cheap; the
+// repo-wide OpenAI default (api/_lib/chat-models.js) uses the same model.
+const OPENAI_MODEL = 'gpt-4o-mini';
 
 // Thrown when no provider is available at all (no free key configured and no
 // BYOK key supplied). Carries an HTTP status so handlers can surface it as 503.
 export class LlmUnavailableError extends Error {
-	constructor(message = 'No LLM provider available. Configure GROQ_API_KEY or OPENROUTER_API_KEY, or supply a BYOK Anthropic key.') {
+	constructor(message = 'No LLM provider available. Configure GROQ_API_KEY, OPENROUTER_API_KEY, or NVIDIA_API_KEY (free), or ANTHROPIC_API_KEY / OPENAI_API_KEY (paid backstop), or supply a BYOK Anthropic key.') {
 		super(message);
 		this.name = 'LlmUnavailableError';
 		this.code = 'llm_unavailable';
@@ -75,19 +89,18 @@ function openaiCompatProvider({ name, key, url, model, extraHeaders = {} }) {
 	};
 }
 
-// Build the ordered provider chain for a request. Anthropic leads when keyed —
-// a caller-supplied BYOK key always, or the server `ANTHROPIC_API_KEY` when the
-// caller opts in with `serverAnthropic: true`. After that come the free platform
-// providers (Groq, then OpenRouter).
-//
-// `serverAnthropic` is opt-in (default off) so the historical BYOK-only policy
-// is unchanged for existing callers: the platform never *depends* on a server
-// Anthropic key and still degrades to the free providers when it's absent. The
-// persona endpoints opt in to get Anthropic-first ordered failover.
-function providerChain({ anthropicKey, anthropicModel, serverAnthropic = false } = {}) {
+// Build the ordered provider chain for a request: free platform providers
+// first (Groq → OpenRouter keys → NVIDIA NIM), paid providers only at the
+// edges. A caller-supplied BYOK `anthropicKey` leads the chain — that's the
+// caller's explicit model choice on the caller's own billing — and still
+// degrades to the free chain on failure. The server ANTHROPIC_API_KEY and
+// OPENAI_API_KEY are appended LAST, automatically, as backstops after every
+// free provider: the prod paid keys are routinely invalid or out of quota, so
+// platform spend never leads and nothing depends on it — but when a key does
+// work, a request that exhausted the free tier still succeeds.
+function providerChain({ anthropicKey, anthropicModel } = {}) {
 	const chain = [];
 	if (anthropicKey) chain.push(anthropicProvider(anthropicKey, anthropicModel));
-	else if (serverAnthropic && env.ANTHROPIC_API_KEY) chain.push(anthropicProvider(env.ANTHROPIC_API_KEY, anthropicModel));
 	if (env.GROQ_API_KEY) {
 		chain.push(openaiCompatProvider({
 			name: 'groq',
@@ -96,14 +109,41 @@ function providerChain({ anthropicKey, anthropicModel, serverAnthropic = false }
 			model: GROQ_MODEL,
 		}));
 	}
-	if (env.OPENROUTER_API_KEY) {
+	// One provider entry per OpenRouter key: when the primary account runs out
+	// of credits (402) or hits a rate limit, the next key takes over. Fallback
+	// keys are typically unfunded free-tier accounts, so they get the model's
+	// :free variant — the paid model would 402 on them unconditionally.
+	const openrouterKeys = [...new Set([env.OPENROUTER_API_KEY, ...env.OPENROUTER_FALLBACK_KEYS].filter(Boolean))];
+	openrouterKeys.forEach((key, i) => {
 		chain.push(openaiCompatProvider({
-			name: 'openrouter',
-			key: env.OPENROUTER_API_KEY,
+			name: i === 0 ? 'openrouter' : `openrouter#${i + 1}`,
+			key,
 			url: 'https://openrouter.ai/api/v1/chat/completions',
-			model: OPENROUTER_MODEL,
+			model: i === 0 ? OPENROUTER_MODEL : `${OPENROUTER_MODEL}:free`,
 			extraHeaders: { 'HTTP-Referer': 'https://three.ws', 'X-Title': 'three.ws' },
 		}));
+	});
+	if (env.NVIDIA_API_KEY) {
+		chain.push(openaiCompatProvider({
+			name: 'nvidia',
+			key: env.NVIDIA_API_KEY,
+			url: 'https://integrate.api.nvidia.com/v1/chat/completions',
+			model: NVIDIA_MODEL,
+		}));
+	}
+	// Paid backstops — always appended, never leading. Server Anthropic is
+	// skipped when a BYOK key already leads the chain (the caller chose their
+	// own Claude billing; the platform doesn't re-buy the same model for them).
+	if (!anthropicKey && env.ANTHROPIC_API_KEY) {
+		chain.push(anthropicProvider(env.ANTHROPIC_API_KEY, anthropicModel));
+	}
+	if (env.OPENAI_API_KEY) {
+		chain.push(openaiCompatProvider({
+			name: 'openai',
+			key: env.OPENAI_API_KEY,
+			url: 'https://api.openai.com/v1/chat/completions',
+			model: OPENAI_MODEL,
+		}));
 	}
 	return chain;
 }
@@ -132,8 +172,8 @@ export function llmConfigured(opts = {}) {
 // { userId, agentId, avatarId, clientId, apiKeyId, tool } — are all optional;
 // pass whatever the call site knows. Recording is fire-and-forget (see
 // recordEvent), so it never delays or fails the completion.
-export async function llmComplete({ system, user, maxTokens = 1024, anthropicKey = null, anthropicModel = null, serverAnthropic = false, timeoutMs = 30_000, track = null }) {
-	const chain = providerChain({ anthropicKey, anthropicModel, serverAnthropic });
+export async function llmComplete({ system, user, maxTokens = 1024, anthropicKey = null, anthropicModel = null, timeoutMs = 30_000, track = null }) {
+	const chain = providerChain({ anthropicKey, anthropicModel });
 	if (!chain.length) throw new LlmUnavailableError();
 
 	let lastErr;

api/chat/proxy.js

@@ -8,7 +8,7 @@ export default wrap(async (req, res) => {
 	if (cors(req, res, { methods: 'POST,OPTIONS' })) return;
 	if (!method(req, res, ['POST'])) return;
 
-	if (!env.OPENROUTER_API_KEY)
+	if (!env.OPENROUTER_API_KEY && !env.OPENROUTER_FALLBACK_KEYS.length)
 		return error(res, 503, 'not_configured', 'Built-in model not available');
 
 	// Anonymous proxy — only :free models pass the gate below, but the upstream
@@ -33,16 +33,28 @@ export default wrap(async (req, res) => {
 	if (!model || !model.endsWith(':free'))
 		return error(res, 400, 'invalid_model', 'Only free-tier models (ending in :free) are allowed via the built-in proxy');
 
-	const upstream = await fetch('https://openrouter.ai/api/v1/chat/completions', {
-		method: 'POST',
-		headers: {
-			Authorization: `Bearer ${env.OPENROUTER_API_KEY}`,
-			'Content-Type': 'application/json',
-			'HTTP-Referer': 'https://three.ws',
-			'X-Title': 'three.ws chat',
-		},
-		body: JSON.stringify(body),
-	});
+	// Only :free models pass the gate above, so any configured key can serve the
+	// request. Rotate to the next key on account-level failures (bad key, out of
+	// credits, rate limit) — rotation happens before any byte is streamed, so a
+	// retry is always safe. Other statuses (4xx from a bad request, 5xx) are
+	// final: every key would fail the same way.
+	const keys = [...new Set([env.OPENROUTER_API_KEY, ...env.OPENROUTER_FALLBACK_KEYS].filter(Boolean))];
+	let upstream;
+	for (const [i, key] of keys.entries()) {
+		upstream = await fetch('https://openrouter.ai/api/v1/chat/completions', {
+			method: 'POST',
+			headers: {
+				Authorization: `Bearer ${key}`,
+				'Content-Type': 'application/json',
+				'HTTP-Referer': 'https://three.ws',
+				'X-Title': 'three.ws chat',
+			},
+			body: JSON.stringify(body),
+		});
+		if (![401, 402, 403, 429].includes(upstream.status) || i === keys.length - 1) break;
+		// Release the abandoned response so its connection returns to the pool.
+		await upstream.body?.cancel()?.catch?.(() => {});
+	}
 
 	if (upstream.status === 402) {
 		const upstreamBody = await upstream.text();

data/changelog.json

@@ -1,6 +1,15 @@
 {
 	"$comment": "Curated changelog entries for three.ws — the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security. `link` is optional and must be a live three.ws path.",
 	"entries": [
+		{
+			"date": "2026-06-11",
+			"title": "Built-in AI stays up when a provider account runs dry",
+			"summary": "The platform's built-in AI (chat, tutor, fact-checker, persona tools) now rotates across multiple OpenRouter accounts: when one account runs out of credits or hits a rate limit, the next takes over automatically, before any of the response has streamed. Paid x402 endpoints that previously returned errors during a provider outage now degrade to a backup account instead of failing.",
+			"tags": [
+				"improvement",
+				"infra"
+			]
+		},
 		{
 			"date": "2026-06-11",
 			"title": "Live $THREE market data, longer trade streams, and tutor session fixes",

public/changelog.json

@@ -1,5 +1,5 @@
 {
-	"generated_at": "2026-06-11T03:28:03.199Z",
+	"generated_at": "2026-06-11T04:19:59.255Z",
 	"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
 	"site": {
 		"name": "three.ws",
@@ -17,6 +17,17 @@
 				"infra"
 			]
 		},
+		{
+			"date": "2026-06-11",
+			"type": "update",
+			"title": "Built-in AI stays up when a provider account runs dry",
+			"summary": "The platform's built-in AI (chat, tutor, fact-checker, persona tools) now rotates across multiple OpenRouter accounts: when one account runs out of credits or hits a rate limit, the next takes over automatically, before any of the response has streamed. Paid x402 endpoints that previously returned errors during a provider outage now degrade to a backup account instead of failing.",
+			"link": null,
+			"tags": [
+				"improvement",
+				"infra"
+			]
+		},
 		{
 			"date": "2026-06-11",
 			"type": "update",

public/changelog.xml

@@ -14,6 +14,14 @@
 			<category>fix</category>
 			<description>Fetching an individual avatar was failing with a server error, which also broke the walking avatar on the home page. The image-processing engine the avatar dress-up baker relies on failed to load in production and took the whole endpoint down with it. Avatar lookups now load independently of the baker, and the missing image libraries ship with the deployment, so avatar pages and the home hero are back.</description>
 		</item>
+		<item>
+			<title>Built-in AI stays up when a provider account runs dry</title>
+			<link>https://three.ws/changelog</link>
+			<guid isPermaLink="false">2026-06-11:Built-in AI stays up when a provider account runs dry</guid>
+			<pubDate>Thu, 11 Jun 2026 12:00:00 GMT</pubDate>
+			<category>improvement</category>
+			<description>The platform's built-in AI (chat, tutor, fact-checker, persona tools) now rotates across multiple OpenRouter accounts: when one account runs out of credits or hits a rate limit, the next takes over automatically, before any of the response has streamed. Paid x402 endpoints that previously returned errors during a provider outage now degrade to a backup account instead of failing.</description>
+		</item>
 		<item>
 			<title>Faster, quieter agents directory and site-wide page polish</title>
 			<link>https://three.ws/agents</link>

public/features.json

@@ -1,5 +1,5 @@
 {
-	"generated_at": "2026-06-11T03:28:03.189Z",
+	"generated_at": "2026-06-11T04:19:59.243Z",
 	"generated_by": "scripts/build-page-index.mjs from data/pages.json",
 	"site": {
 		"name": "three.ws",