Skip to content

Commit 983177f

Browse files
faraa2mclaude
andcommitted
feat: Wave 3 v1.0.0 rollout — VS Code, Claude Code skill, Mistral+Cohere, latency, playground showcase
Adds the headline differentiation features for v1.0.0. No breaking changes to existing flags or output paths. VS Code / Cursor extension (`packages/vscode/`, new package): - Status bar shows live token count + USD cost for the active editor file across Claude, GPT-4o, and Gemini. - Settings: model, format, warnOnCostAbove threshold (turns the bar background warning-yellow when exceeded). - Commands: Tokenometer: switch model, Tokenometer: show info. - Debounced 200ms recompute on document changes; 1MB file-size gate. - Reuses @tokenometer/core via esbuild bundle. Marketplace publish in Wave 4 Phase I. Claude Code skill (`packages/claude-code-skill/`, new package): - SKILL.md with `name: tokenometer-cost-check` + multi-trigger description. - 13-section body teaching Claude Code agents when and how to invoke `npx tokenometer` for prompt cost analysis: offline + empirical mode, --output json/sarif, --by-file, --image, auto provider detection, .tokenometer.yml config, CI guardrail via `tokenometer init`. - One-line install via `~/.claude/skills/tokenometer/SKILL.md`. Mistral + Cohere providers (`@tokenometer/core`): - Mistral: `mistral-tokenizer-js` for SentencePiece family (Mistral 7B, Mixtral 8x7B/8x22B, Mistral Large 2407, Codestral 22B); chars/4 heuristic for Tekken-tokenizer models (NeMo, Pixtral, Mistral Small 2409+, Devstral, Mistral Medium 2505+, Magistral, Ministral). All marked `approximate: true`. Empirical mode unsupported — Mistral has no public token-count API. - Cohere: chars/4 heuristic offline; empirical via POST /v1/tokenize using bare fetch (Cohere SDK ships only a REST wrapper, no offline tokenizer). - Pricing: Mistral auto-sourced from @tokenlens/models; Cohere via LOCAL_OVERRIDES until tokenlens adds upstream. Total KNOWN_MODELS: 42 → 63. Latency mode (`--latency` flag): - Measures real generation latency: TTFT (time to first token) + total wall-clock + tokens/sec. Reports p50/p95/mean over n trials (default 3, configurable 1-10 via --latency-trials). - Implies --empirical. Bumps default --max-spend ceiling to $0.25 to cover ~200-token generations. - Streamed via SDK for Anthropic/Google; bare fetch SSE/NDJSON for OpenAI/Mistral/Cohere (no extra SDK install footprint). - New table columns + JSON/SARIF carry the latency block. - Differentiation pitch: only LLM cost CLI that also reports latency. Playground showcase (`packages/web/`): - 11 new routes: /diff, /by-file, /sarif, /vision, /config-builder, /init, /models (Cost Atlas with sort/search/filter), /models/<id> (per-model SEO pages), /editor, /claude-code, plus / and /calculator for the existing landing. - react-router-dom 7, top nav with Tools dropdown, footer linking repo + npm + marketplace placeholders. - Cross-page provider robustness: Map<Provider, ...> instead of static Records so playground gracefully extends as new providers land. - Static robots.txt + sitemap.xml; per-page document.title via usePageTitle hook (no react-helmet dependency). Test count: 150 → 227 (across 20 files). Lint, typecheck, root + web builds all clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent cc745d5 commit 983177f

67 files changed

Lines changed: 9318 additions & 267 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
"tokenometer": minor
3+
"@tokenometer/core": minor
4+
---
5+
6+
Add `@tokenometer/claude-code-skill` — a Claude Code skill that teaches
7+
Claude Code agents to invoke `npx tokenometer` for prompt-cost analysis.
8+
Install via `~/.claude/skills/tokenometer/SKILL.md`. Submission to
9+
community skill registry tracked separately.

.changeset/wave-3-latency.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
"tokenometer": minor
3+
"@tokenometer/core": minor
4+
---
5+
6+
Add `--latency` flag — measures real generation latency (TTFT + total ms +
7+
tokens/sec, p50/p95/mean over n trials) alongside token cost. Implies
8+
`--empirical`. Default trials = 3, configurable via `--latency-trials <n>`
9+
(1-10). Bumps default `--max-spend` to $0.25 to cover the n × 200-token
10+
generations. Supported providers: Anthropic, OpenAI, Google, Cohere,
11+
Mistral (latter two are metered).
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
"tokenometer": minor
3+
"@tokenometer/core": minor
4+
---
5+
6+
Add Mistral and Cohere providers.
7+
8+
- Mistral: `mistral-tokenizer-js` for SentencePiece family (Mistral 7B,
9+
Mixtral, Mistral Large 2407, Codestral); `chars/4` heuristic for Tekken
10+
models (NeMo, Pixtral, Mistral Small 2409+, Devstral, Mistral Medium
11+
2505+, Magistral, Ministral). All marked `approximate: true`. Empirical
12+
mode unsupported (Mistral has no public token-count API).
13+
- Cohere: offline heuristic `chars/4` (Cohere SDK is REST-only; no offline
14+
tokenizer ships in JS). Empirical via `POST /v1/tokenize` when
15+
`COHERE_API_KEY` is set.
16+
17+
Pricing for Mistral auto-sourced from `@tokenlens/models/mistral`. Cohere
18+
pricing comes from `LOCAL_OVERRIDES` (`command-r`, `command-r-plus`)
19+
because `@tokenlens/models` does not yet ship a Cohere catalog at v1.3.0.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
"tokenometer": minor
3+
"@tokenometer/core": minor
4+
---
5+
6+
Playground (`https://tokenometer.vercel.app`) gains showcase pages for
7+
every Wave 2 feature: `/diff`, `/by-file`, `/sarif`, `/vision`,
8+
`/config-builder`, `/init`, `/models` (Cost Atlas + per-model SEO pages),
9+
plus placeholder pages for the VS Code extension and Claude Code skill.
10+
</content>
11+
</invoke>
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
"tokenometer": minor
3+
"@tokenometer/core": minor
4+
---
5+
6+
Add VS Code / Cursor extension (`@tokenometer/vscode`). Status bar shows
7+
live token count + USD cost for the active editor file across Claude,
8+
GPT-4o, and Gemini. Reuses `@tokenometer/core`. Marketplace publish
9+
follows in Phase I.

README.md

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -58,14 +58,15 @@ The `Approx` column shows `✓` when the count is a proxy (Anthropic / Google of
5858
5959
## Why this exists
6060

61-
`tiktoken` and `@anthropic-ai/tokenizer` give you a token count for one provider. They don't tell you:
61+
**Cost AND latency in one CLI — the only tool that does both.** `tiktoken` and `@anthropic-ai/tokenizer` give you a token count for one provider. They don't tell you:
6262

6363
- What the same prompt costs across **multiple providers and models**
64+
- How **fast** each provider actually responds (TTFT + tokens/sec) — a real generation, not a synthetic benchmark
6465
- Whether **format conversion** (YAML ↔ JSON ↔ XML ↔ MD) actually moves the needle
6566
- The **empirical** cost — what your provider actually charged on a real call, after prompt caching
6667
- Whether a PR introduced a **prompt-cost regression**
6768

68-
Tokenometer is dev-time, multi-provider, multi-format, optionally empirical, and CI-native.
69+
Tokenometer is dev-time, multi-provider, multi-format, optionally empirical, latency-aware, and CI-native.
6970

7071
## Install
7172

@@ -135,11 +136,13 @@ The CLI also supports `--output json|sarif` for machine-readable output, `--by-f
135136

136137
Tokenometer picks a tokenizer per provider and flags the count as approximate (`approximate: true` in the API result) when the offline path is a proxy:
137138

138-
| Provider | Offline tokenizer | Exactness | Empirical (`--empirical`) |
139-
|-----------|------------------------------|-------------|----------------------------------|
140-
| OpenAI | `gpt-tokenizer` `o200k_base` | exact | same `o200k_base` (matches OpenAI production count) |
141-
| Anthropic | `gpt-tokenizer` `cl100k_base`| approximate | `messages.countTokens` (exact, free) |
142-
| Google | `chars / 4` heuristic | approximate | `model.countTokens` (exact, free) |
139+
| Provider | Offline tokenizer | Exactness | Empirical (`--empirical`) |
140+
|-----------|------------------------------------------------|-------------|----------------------------------|
141+
| OpenAI | `gpt-tokenizer` `o200k_base` | exact | same `o200k_base` (matches OpenAI production count) |
142+
| Anthropic | `gpt-tokenizer` `cl100k_base` | approximate | `messages.countTokens` (exact, free) |
143+
| Google | `chars / 4` heuristic | approximate | `model.countTokens` (exact, free) |
144+
| Mistral | `mistral-tokenizer-js` (V1/V2/V3) · `chars/4` for Tekken | approximate | unsupported (no public token-count endpoint) |
145+
| Cohere | `chars / 4` heuristic | approximate | `POST /v1/tokenize` (exact, free, requires `COHERE_API_KEY`) |
143146

144147
Cost = `tokens / 1000 × per-1k input rate`. Pricing and context windows are sourced from the [`tokenlens`](https://www.npmjs.com/package/tokenlens) registry, with a small set of local overrides for bleeding-edge models the registry hasn't picked up yet — see [`packages/core/src/rates.ts`](packages/core/src/rates.ts) (`RATES_VERSION`).
145148

@@ -153,7 +156,7 @@ Cost = `tokens / 1000 × per-1k input rate`. Pricing and context windows are sou
153156

154157
## Status
155158

156-
Early. v0.0.x — see [milestones](https://github.com/faraa2m/tokenometer/milestones). Roadmap to v1.0.0 in progress: VS Code extension, Claude Code skill, vision-token cost, Mistral + Cohere providers.
159+
Early. v0.0.x — see [milestones](https://github.com/faraa2m/tokenometer/milestones). Roadmap to v1.0.0 in progress: VS Code extension, Claude Code skill, vision-token cost.
157160

158161
## License
159162

benchmarks/run.mjs

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,24 @@
11
#!/usr/bin/env node
2+
//
3+
// Benchmark sweep modes:
4+
//
5+
// node benchmarks/run.mjs # offline drift check vs results.json
6+
// node benchmarks/run.mjs --regenerate # rewrite results.json from current code
7+
// node benchmarks/run.mjs --empirical # countTokens sweep (Anthropic / Google free), writes empirical.json
8+
// node benchmarks/run.mjs --latency # real generation sweep (METERED), writes latency-empirical.json
9+
//
10+
// `--latency` runs `--latency-trials` (default 3) streaming generations per
11+
// (model, format) cell with max_tokens=200. Cost scales linearly with the
12+
// number of cells; respect your `MAX_SPEND_USD` env or use `--filter`.
13+
214
import { readFile, readdir, writeFile } from 'node:fs/promises';
315
import { dirname, join } from 'node:path';
416
import { fileURLToPath } from 'node:url';
517
import {
618
KNOWN_MODELS,
719
RATES_VERSION,
820
allFormats,
21+
measureLatency,
922
tokenize,
1023
tokenizeEmpirical,
1124
} from '@tokenometer/core';
@@ -14,6 +27,7 @@ const HERE = dirname(fileURLToPath(import.meta.url));
1427
const PROMPTS_DIR = join(HERE, 'prompts');
1528
const RESULTS_PATH = join(HERE, 'results.json');
1629
const EMPIRICAL_PATH = join(HERE, 'empirical.json');
30+
const LATENCY_PATH = join(HERE, 'latency-empirical.json');
1731

1832
const loadPrompts = async () => {
1933
const files = (await readdir(PROMPTS_DIR)).filter((f) => !f.startsWith('.')).sort();
@@ -136,8 +150,11 @@ const compareResults = async (results) => {
136150
const readEnv = () => {
137151
const env = {};
138152
if (process.env.ANTHROPIC_API_KEY) env.anthropicApiKey = process.env.ANTHROPIC_API_KEY;
153+
if (process.env.COHERE_API_KEY) env.cohereApiKey = process.env.COHERE_API_KEY;
139154
const googleKey = process.env.GOOGLE_API_KEY ?? process.env.GEMINI_API_KEY;
140155
if (googleKey) env.googleApiKey = googleKey;
156+
if (process.env.MISTRAL_API_KEY) env.mistralApiKey = process.env.MISTRAL_API_KEY;
157+
if (process.env.OPENAI_API_KEY) env.openaiApiKey = process.env.OPENAI_API_KEY;
141158
return env;
142159
};
143160

@@ -225,17 +242,96 @@ const summarizeByProvider = (rows) => {
225242
return summary;
226243
};
227244

245+
const parseLatencyTrials = (args) => {
246+
const idx = args.indexOf('--latency-trials');
247+
if (idx === -1 || !args[idx + 1]) return 3;
248+
const n = Number.parseInt(args[idx + 1], 10);
249+
if (!Number.isFinite(n) || n < 1 || n > 10) {
250+
throw new Error(`--latency-trials must be 1..10, got "${args[idx + 1]}"`);
251+
}
252+
return n;
253+
};
254+
255+
const runLatencySweep = async (prompts, env, models, trials) => {
256+
const formats = [...allFormats()];
257+
const out = {};
258+
let totalTrials = 0;
259+
for (const [name, prompt] of Object.entries(prompts).sort()) {
260+
process.stderr.write(`${name} ... `);
261+
const byModel = {};
262+
for (const modelId of models) {
263+
const byFormat = {};
264+
for (const format of formats) {
265+
try {
266+
const result = await measureLatency({
267+
env,
268+
modelId,
269+
prompt,
270+
trials,
271+
});
272+
byFormat[format] = {
273+
mean: result.mean,
274+
p50: result.p50,
275+
p95: result.p95,
276+
trials: result.trials,
277+
};
278+
totalTrials += trials;
279+
} catch (err) {
280+
byFormat[format] = { error: err.message };
281+
}
282+
}
283+
byModel[modelId] = byFormat;
284+
}
285+
out[name] = byModel;
286+
process.stderr.write('done\n');
287+
}
288+
return {
289+
formats,
290+
generatedAt: new Date().toISOString(),
291+
latency: out,
292+
models,
293+
schemaVersion: 1,
294+
totalTrials,
295+
trialsPerCell: trials,
296+
};
297+
};
298+
228299
const main = async () => {
229300
const args = process.argv.slice(2);
230301
const isRegenerate = args.includes('--regenerate');
231302
const isEmpirical = args.includes('--empirical');
303+
const isLatency = args.includes('--latency');
232304
const filter = parseModelFilter(args);
233305
const models = selectModels(filter);
234306
if (filter) {
235307
console.error(`Filter active — sweeping ${models.length}/${KNOWN_MODELS.length} models.`);
236308
}
237309
const prompts = await loadPrompts();
238310

311+
if (isLatency) {
312+
const env = readEnv();
313+
const trials = parseLatencyTrials(args);
314+
if (
315+
!env.anthropicApiKey &&
316+
!env.googleApiKey &&
317+
!env.openaiApiKey &&
318+
!env.cohereApiKey &&
319+
!env.mistralApiKey
320+
) {
321+
console.error(
322+
'Latency sweep needs at least one of ANTHROPIC_API_KEY / OPENAI_API_KEY / GOOGLE_API_KEY (or GEMINI_API_KEY) / COHERE_API_KEY / MISTRAL_API_KEY.',
323+
);
324+
console.error(
325+
'Each trial is a metered ~200-token chat completion — use --filter to narrow scope.',
326+
);
327+
return 1;
328+
}
329+
const result = await runLatencySweep(prompts, env, models, trials);
330+
await writeFile(LATENCY_PATH, `${JSON.stringify(result, null, 2)}\n`, 'utf8');
331+
console.log(`\nWrote ${LATENCY_PATH} (${result.totalTrials} trials, ${trials} per cell).`);
332+
return 0;
333+
}
334+
239335
if (isEmpirical) {
240336
const env = readEnv();
241337
if (!env.anthropicApiKey && !env.googleApiKey) {

0 commit comments

Comments
 (0)