Skip to content

Commit 7c541fb

Browse files
1bcMaxKillerQueen-Z
1bcMax
and
KillerQueen-Z
committed
fix(tokens): image-token estimator across the codebase (PR #54 + 3 sibling sites)
Lands PR #54 from KillerQueen-Z verbatim plus three sibling sites the review caught — all the same JSON.stringify-content-array-with-image bug class that's been getting fixed site-by-site since 3.15.89. PR #54 changes (preserved as-authored): - tokens.ts:estimateContentPartTokens — walk content array block-by- block; images count as flat 1500 tokens instead of base64 char length. Empirically verified by the contributor: same session with one ~100KB image showed /context=75K/200K (37.8%) pre-fix vs 1.9K/200K (1.0%) post-fix. 40× over-count. - tokens.ts:getAnchoredTokenCount — both return paths hardcoded contextUsagePct: 0, so the renderer's context ring sat at 0% regardless of real fullness. Fixed to compute against the current model's window. - loop.ts — contextPct was integer-rounded so a fresh session at 0.4% rounded to 0 and froze the ring. Now keeps one decimal. Sibling sites I patched in the same merge: - reduce.ts:estimateChars — image base64 inflated the char count and skewed reduceTokens pass decisions toward aggressive collapse. - compact.ts:tool_result preview — JSON.stringify dumped base64 into the summarizer prompt; sliced to 500 chars, the summarizer saw garbage. Now builds preview from text blocks + `[N image block(s)]` marker. - commands.ts:/context tool char count — /context UI showed inflated totalToolChars on image-bearing sessions. Fixed for consistency with the fixed token count. Eight known sites of this bug class total. After this release, every place in the codebase that handles tool_result.content arrays treats image blocks correctly: 3.15.89: optimize.ts:budgetToolResults 3.15.90: reduce.ts:ageToolResults (PR #53) 3.15.90: reduce.ts:deduplicateToolResultLines 3.15.90: reduce.ts:collapseRepetitiveTools 3.15.98: tokens.ts:estimateContentPartTokens (PR #54) 3.15.98: tokens.ts:getAnchoredTokenCount (PR #54) 3.15.98: loop.ts:contextPct rounding (PR #54) 3.15.98: reduce.ts:estimateChars 3.15.98: compact.ts:tool_result preview 3.15.98: commands.ts:/context tool char count Tests: 3 new regression tests in test/local.mjs pin the image-token math against a 140KB synthetic image — main path, text-only path, and the reduce.ts sibling site. 387/387 tests pass. Wallet billing unaffected — gateway has its own working image handling; this fix is observability + decision-making only. Co-Authored-By: KillerQueen-Z <1211904451@qq.com>
1 parent e808245 commit 7c541fb

9 files changed

Lines changed: 462 additions & 13 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,75 @@
11
# Changelog
22

3+
## 3.15.98 — Image-bearing context-token counters across the codebase (PR #54 + 3 missed siblings)
4+
5+
Bundle of related fixes — PR #54 from `KillerQueen-Z` landed verbatim
6+
plus three sibling sites the review caught.
7+
8+
The bug class: any function that turned `tool_result.content` arrays
9+
into strings via `JSON.stringify(part.content)` was counting a 140KB
10+
base64 image as ~70K phantom tokens / chars. The bug had **eight
11+
known sites** as of 2026-05-11 — five fixed in 3.15.89/90, three more
12+
fixed here:
13+
14+
| Site | Function | Effect of bug | Status |
15+
|---|---|---|---|
16+
| `optimize.ts:budgetToolResults` | 32K char trim | destroyed image block | 3.15.89 |
17+
| `reduce.ts:ageToolResults` | age decay | destroyed image | 3.15.90 (PR #53) |
18+
| `reduce.ts:deduplicateToolResultLines` | ANSI/dedupe | destroyed image | 3.15.90 |
19+
| `reduce.ts:collapseRepetitiveTools` | stub old results | destroyed image | 3.15.90 |
20+
| `tokens.ts:estimateContentPartTokens` | /context display | inflated ~40× | **3.15.98 (PR #54)** |
21+
| `reduce.ts:estimateChars` | reduce pass gates | inflated → wrong collapse decisions | **3.15.98** |
22+
| `compact.ts:tool_result preview` | summary prompt | sent base64 to summarizer | **3.15.98** |
23+
| `commands.ts:/context tool char count` | UI display | inflated tool-char count | **3.15.98** |
24+
25+
Empirical proof from PR #54: same session with one 100KB image showed
26+
`/context = 75K/200K (37.8%)` before fix vs `1.9K/200K (1.0%)` after.
27+
28+
### Also in PR #54
29+
30+
- **`getAnchoredTokenCount` returned `contextUsagePct: 0`** on both
31+
return paths. The renderer's context ring sat at 0% regardless of
32+
real fullness because the agent loop emits this value verbatim.
33+
Fixed to compute `(estimated / contextWindow) * 100` using the
34+
current model's window.
35+
- **`loop.ts:contextPct` was integer-rounded.** A 200-message session
36+
at 0.4% rounded to 0 and froze the ring. Now `.toFixed(1)`-style.
37+
38+
### What's now consistent
39+
40+
Every per-call layer treats an image block as ~1500 tokens (close to
41+
Anthropic's `(w*h)/750` billing math — `Read` caps long edge to
42+
1280px so normalized images land near 1050 tokens):
43+
44+
- Context display (`/context`, the renderer ring)
45+
- Compaction trigger (won't fire spuriously on image-bearing turns)
46+
- Reduce passes (won't aggressively dedupe an image-heavy session)
47+
- Summary prompt (no more base64 dumped into the summarizer)
48+
49+
### Tests
50+
51+
Three new in `test/local.mjs`:
52+
53+
1. `estimateContentPartTokens: image block counts as ~1500 tokens,
54+
not base64 char length` — pins the main PR #54 fix against a
55+
140KB synthetic image. Asserts result is <3000 tokens.
56+
2. `estimateContentPartTokens: text-only string content path
57+
unchanged` — guards against regression in the simple path.
58+
3. `estimateChars (reduce.ts): image blocks count as ~6K chars, not
59+
base64 length` — pins the sibling fix. Builds a 12-message
60+
history with a 140KB image, runs `reduceTokens`, asserts the
61+
image base64 survives intact.
62+
63+
387/387 tests pass.
64+
65+
### Credits
66+
67+
`KillerQueen-Z` for PR #54 — both the empirical reproduction (40×
68+
discrepancy on a real session) and the clean three-part fix
69+
(`tokens.ts` walker + `getAnchoredTokenCount` denominator +
70+
`loop.ts` precision). The three sibling fixes were caught during
71+
review by grepping for `JSON.stringify(part.content)` across `src/`.
72+
373
## 3.15.97 — log entries are one physical line (embedded newlines collapse to ↵)
474

575
Format-integrity fix. Real entry from `franklin-debug.log`:

‎docs/release-notes/2026-05-12-context-token-sweep.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# Franklin 3.15.98 — image-bearing context-token counters across the codebase
2+
3+
*May 12, 2026 · 1 patch release · PR #54 + 3 sibling sites caught in review*
4+
5+
`KillerQueen-Z` filed PR #54 with three context-window display fixes
6+
and a clean empirical reproduction. While reviewing, I grepped the
7+
codebase for the same `JSON.stringify(part.content)` pattern and
8+
found three more sites with the same image-token-inflation bug.
9+
Landing all six together.
10+
11+
## The bug class, one final time
12+
13+
Any function that handles `tool_result.content` arrays and falls back
14+
to `JSON.stringify(content)` will tokenize image base64 as text. A
15+
typical normalized image is ~140KB base64 → ~70K phantom
16+
chars / ~35K phantom tokens. Anthropic actually bills `(w*h)/750`
17+
≈ 1100-1500 tokens per image.
18+
19+
We've been fixing this site by site:
20+
21+
| Release | Site | Damage |
22+
|---|---|---|
23+
| 3.15.89 | `optimize.ts:budgetToolResults` | trimmed → **destroyed image** |
24+
| 3.15.90 | `reduce.ts:ageToolResults` (PR #53) | aged → **destroyed image** |
25+
| 3.15.90 | `reduce.ts:deduplicateToolResultLines` | deduped → **destroyed image** |
26+
| 3.15.90 | `reduce.ts:collapseRepetitiveTools` | collapsed → **destroyed image** |
27+
| **3.15.98** | `tokens.ts:estimateContentPartTokens` | inflated /context by 40× |
28+
| **3.15.98** | `reduce.ts:estimateChars` | inflated → wrong collapse decisions |
29+
| **3.15.98** | `compact.ts:tool_result preview` | base64 in summary prompt |
30+
| **3.15.98** | `commands.ts:/context tool char display` | inflated UI char count |
31+
32+
Eight sites total. After this release, every place in the codebase
33+
that touches `tool_result.content` arrays handles them correctly.
34+
35+
## PR #54 (landed verbatim)
36+
37+
### `tokens.ts:estimateContentPartTokens` — main fix
38+
39+
Empirically verified by the contributor: same 4-message session with
40+
one ~100KB image showed:
41+
42+
- Before: `/context` = **75K / 200K (37.8%)**
43+
- After: `/context` = **1.9K / 200K (1.0%)**
44+
45+
That's a 40× over-count. It also triggered premature `/compact`
46+
calls — agent saw 37% "context fullness" on a session that was 1%
47+
full, fired bloat compactions that weren't needed, burned tokens
48+
unnecessarily.
49+
50+
The fix walks the content array block-by-block. Text blocks count as
51+
text. Image blocks count as 1500 tokens (flat). Unknown block types
52+
still stringify, but with `source.data` redacted to `<bytes>` so
53+
future block kinds (audio? video?) don't regress.
54+
55+
### `getAnchoredTokenCount` — `contextUsagePct: 0` always
56+
57+
Both return paths of this function hardcoded the field. The agent
58+
loop emits this via `kind: 'usage'` events to the renderer, so the
59+
desktop/extension's context ring was stuck at 0% regardless of how
60+
full the context actually was.
61+
62+
Fix: compute `(estimated / contextWindow) * 100` using the current
63+
model's window from `getContextWindow(_currentModel)`.
64+
65+
### `loop.ts` — integer rounding froze the ring
66+
67+
```ts
68+
contextPct: Math.round(contextUsagePct),
69+
```
70+
71+
A 200-message session at 0.4% rounded to 0 and froze the renderer.
72+
Now `Math.round(contextUsagePct * 10) / 10` keeps one decimal.
73+
74+
## Sibling sites (caught during PR #54 review)
75+
76+
### `reduce.ts:estimateChars`
77+
78+
This function gates `reduceTokens`'s passes (dedupe, collapse,
79+
normalize). When an image inflates the char count by ~140K, the
80+
reduce decisions trigger aggressive collapsing — including the
81+
image-bearing tool_result, which (because of the 3.15.90 array-aware
82+
fix) survives the collapse but only after needlessly burning the
83+
reduce pass.
84+
85+
Fix walks blocks: text blocks count text length; image blocks count
86+
~6000 chars (the char-equivalent of 1500 tokens at the 4-chars/token
87+
rule).
88+
89+
### `compact.ts:tool_result preview`
90+
91+
When the agent's summarizer needs to compress old turns, it builds a
92+
preview of each tool_result for the summary prompt:
93+
94+
```ts
95+
const content = typeof part.content === 'string'
96+
? part.content
97+
: JSON.stringify(part.content);
98+
const truncated = content.length > 500 ? content.slice(0, 500) + '...' : content;
99+
textParts.push(`[Tool result: ${truncated}]`);
100+
```
101+
102+
For an image-bearing result, `JSON.stringify` produces a string that
103+
starts with `[{"type":"text","text":"..."},{"type":"image","source":{"type":"base64","data":"AAA...`.
104+
Slicing to 500 chars gives the summarizer a useless preview of base64
105+
garbage.
106+
107+
Fix builds the preview from text blocks only, then appends `[N image
108+
block(s)]` to mark their presence:
109+
110+
```ts
111+
const pieces: string[] = [];
112+
let imageCount = 0;
113+
for (const block of part.content) {
114+
if (block.type === 'text') pieces.push(block.text);
115+
else if (block.type === 'image') imageCount++;
116+
}
117+
if (imageCount > 0) pieces.push(`[${imageCount} image block(s)]`);
118+
```
119+
120+
The summarizer now sees `[Tool result: Image file: /tmp/scene.png [1 image block]]`
121+
instead of 500 chars of base64.
122+
123+
### `commands.ts:/context tool char count`
124+
125+
`/context` displays "Total tool result chars: X" alongside the token
126+
estimate. Pre-fix, X included the base64 bytes — so a user with the
127+
fixed token count seeing `/context = 1.9K/200K (1.0%)` would also see
128+
"Total tool result chars: 142,847" and be confused. Now the char
129+
count walks blocks the same way.
130+
131+
## Tests
132+
133+
Three new in `test/local.mjs`:
134+
135+
1. **`estimateContentPartTokens: image block counts as ~1500 tokens, not
136+
base64 char length`** — pin the main PR #54 fix against a 140KB
137+
synthetic image. Asserts result is `< 3000` tokens and `> 1000`
138+
(not silently zero either).
139+
2. **`estimateContentPartTokens: text-only string content path
140+
unchanged`** — 4000-char string body → ~2000 tokens (within ±25%).
141+
Guards against regression in the simple path.
142+
3. **`estimateChars (reduce.ts): image blocks count as ~6K chars, not
143+
base64 length`** — build a 12-message history with one image
144+
carrying 140KB base64, run `reduceTokens`, assert the image base64
145+
survives. Pre-fix, the inflated char count triggered aggressive
146+
collapse that would have destroyed the image.
147+
148+
387/387 tests pass.
149+
150+
## What didn't change
151+
152+
- **Wallet billing**: unchanged — the gateway has its own (working)
153+
image accounting and uses its own input estimate. PR #54 explicitly
154+
notes this.
155+
- **3.15.95's `cacheCreationInputTokens` / `cacheReadInputTokens`
156+
capture** — independent, complementary fix for wallet-truth
157+
accounting (different layer).
158+
- **The `Read` tool's `sharp` normalization** (3.15.90) is unchanged.
159+
It caps long-edge at 1280px which is why "image ≈ 1500 tokens flat"
160+
is a good estimate.
161+
162+
## Credits
163+
164+
`KillerQueen-Z` (PR #54) — empirical reproduction (40× discrepancy
165+
on a real session) and the clean three-part fix. Same contributor as
166+
PR #53 (vision token explosion). Two sharp diagnostics in a row.
167+
168+
## Behavioral implications
169+
170+
After this release:
171+
172+
- `/context` shows the actual context fullness on image-bearing
173+
sessions. Pre-fix, it could read 37% on a 1% session.
174+
- The desktop/extension context ring updates correctly. Pre-fix, it
175+
was stuck at 0% regardless of fullness.
176+
- Compaction triggers fire on real fullness, not on image-token
177+
inflation. Fewer spurious `/compact` events on vision workflows.
178+
- The summarizer (when compaction does fire) sees text previews
179+
marked with image counts instead of base64 garbage. Marginally
180+
better summaries, marginally lower summary-call costs.
181+
182+
If you've been seeing `/context` numbers that don't match your gut
183+
sense of session length — they should match now.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@blockrun/franklin",
3-
"version": "3.15.97",
3+
"version": "3.15.98",
44
"description": "Franklin — The AI agent with a wallet. Spends USDC autonomously to get real work done. Pay per action, no subscriptions.",
55
"type": "module",
66
"exports": {

‎src/agent/commands.ts‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -201,8 +201,23 @@ const DIRECT_COMMANDS: Record<string, (ctx: CommandContext) => Promise<void> | v
201201
if ('type' in part) {
202202
if (part.type === 'tool_result') {
203203
toolResults++;
204-
const c = typeof part.content === 'string' ? part.content : JSON.stringify(part.content);
205-
totalToolChars += c.length;
204+
// Sibling of PR #54's tokens.ts fix: image base64 must NOT
205+
// count toward the displayed char total — `/context` would
206+
// otherwise show ~70K chars per attached image and confuse
207+
// the user about why the ring is at 1% but "total tool
208+
// chars" is huge.
209+
if (typeof part.content === 'string') {
210+
totalToolChars += part.content.length;
211+
} else if (Array.isArray(part.content)) {
212+
for (const block of part.content) {
213+
const t = (block as { type?: string }).type;
214+
if (t === 'text') {
215+
totalToolChars += ((block as { text?: string }).text || '').length;
216+
} else if (t === 'image') {
217+
totalToolChars += 6000; // ~1500 tokens × 4 chars/tok
218+
}
219+
}
220+
}
206221
}
207222
if (part.type === 'thinking') thinkingBlocks++;
208223
}

‎src/agent/compact.ts‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -472,7 +472,32 @@ function formatForSummarization(messages: Dialogue[]): string {
472472
textParts.push(`[Called tool: ${part.name}(${JSON.stringify(part.input).slice(0, 200)})]`);
473473
break;
474474
case 'tool_result': {
475-
const content = typeof part.content === 'string' ? part.content : JSON.stringify(part.content);
475+
// Sibling of PR #54's tokens.ts fix: when content is a
476+
// [{text}, {image}] array, JSON.stringify dumps base64
477+
// bytes into the summary prompt — bloats the summarizer's
478+
// input and produces a useless preview ("[Tool result:
479+
// [{\"type\":\"text\",\"text\":\"Image file: ...\"},{\"type\":\"image\",\"source\":{\"type\":\"base64\",\"data\":\"...").
480+
// Build the preview from text blocks only; mark images
481+
// explicitly so the summarizer knows they exist.
482+
let content: string;
483+
if (typeof part.content === 'string') {
484+
content = part.content;
485+
} else if (Array.isArray(part.content)) {
486+
const pieces: string[] = [];
487+
let imageCount = 0;
488+
for (const block of part.content) {
489+
const t = (block as { type?: string }).type;
490+
if (t === 'text') {
491+
pieces.push((block as { text?: string }).text || '');
492+
} else if (t === 'image') {
493+
imageCount++;
494+
}
495+
}
496+
if (imageCount > 0) pieces.push(`[${imageCount} image block${imageCount > 1 ? 's' : ''}]`);
497+
content = pieces.join(' ');
498+
} else {
499+
content = JSON.stringify(part.content);
500+
}
476501
const truncated = content.length > 500 ? content.slice(0, 500) + '...' : content;
477502
textParts.push(`[Tool result${part.is_error ? ' (ERROR)' : ''}: ${truncated}]`);
478503
break;

‎src/agent/loop.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1750,7 +1750,11 @@ export async function interactiveSession(
17501750
tier: routingTier,
17511751
confidence: routingConfidence,
17521752
savings: routingSavings,
1753-
contextPct: Math.round(contextUsagePct),
1753+
// Preserve sub-1% precision: a fresh session at 0.4% would
1754+
// round to 0 and freeze the renderer's context ring until the
1755+
// conversation grows past ~1k tokens. Match `/context`'s
1756+
// `.toFixed(1)` fidelity.
1757+
contextPct: Math.round(contextUsagePct * 10) / 10,
17541758
});
17551759

17561760
// Record usage for stats tracking (franklin stats command).

‎src/agent/reduce.ts‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -485,7 +485,25 @@ function estimateChars(history: Dialogue[]): number {
485485
if ('type' in p) {
486486
if (p.type === 'text') total += p.text.length;
487487
else if (p.type === 'tool_result') {
488-
total += typeof p.content === 'string' ? p.content.length : JSON.stringify(p.content).length;
488+
// Sibling of PR #54's tokens.ts fix: JSON.stringify-ing a
489+
// [{text}, {image}] array counts the base64 `data` field as
490+
// text and inflates the char count by ~70K per image. That
491+
// skews every reduce-pass decision (when to dedupe, when to
492+
// collapse) toward "save chars by collapsing the image-
493+
// bearing result" — exactly wrong. Walk blocks instead.
494+
if (typeof p.content === 'string') {
495+
total += p.content.length;
496+
} else if (Array.isArray(p.content)) {
497+
for (const block of p.content) {
498+
if ((block as { type?: string }).type === 'text') {
499+
total += ((block as { text?: string }).text || '').length;
500+
} else if ((block as { type?: string }).type === 'image') {
501+
// Mirror tokens.ts: image ≈ 1500 tokens ≈ ~6K chars
502+
// at the 4-chars/token rule estimateTokens uses.
503+
total += 6000;
504+
}
505+
}
506+
}
489507
} else if (p.type === 'tool_use') {
490508
total += JSON.stringify(p.input).length;
491509
}

0 commit comments

Comments
 (0)