fix(codebuddy): collect traces as authoritative token source, fix 90% undercounting - #399
fix(codebuddy): collect traces as authoritative token source, fix 90% undercounting#399HandSonic wants to merge 6 commits into
Conversation
CodeBuddy CLI writes authoritative token usage to
~/.codebuddy/traces/<pid>/trace_*.json (and ~/.workbuddy/traces/ for
WorkBuddy). Each trace JSON carries trace.modelInfo.{totalInputTokens,
totalOutputTokens,totalCachedTokens,callCount} with 100% field stability
(verified on 286 real traces). The existing jsonl reader only sees
~10% of messages (90% lack providerData.rawUsage in newer CodeBuddy
versions), causing massive undercounting.
This change:
1. resolveCodebuddyProjectFiles: scan traces/ dir, return {path, kind}
objects (jsonl | trace | log)
2. parseCodebuddyIncremental: add trace JSON branch — read modelInfo,
dedup by traceId, skip jsonl messages for traced sessions
(tracedSessionIds) to avoid double-counting
3. resolveWorkbuddyProjectFiles: same traces/ scan
4. parseWorkbuddyIncremental: same trace JSON branch
5. workbuddy rawUsage fallback: when providerData.rawUsage is absent,
fall back to top-level usage field (camelCase convention)
6. Extension-log parsing: default disabled (TOKENTRACKER_CODEBUDDY_LOG_FALLBACK=1
to enable) — it overlaps with jsonl and uses a different dedup key
namespace, causing 3x inflation
Verified on real machine (2026-07-31):
- Before: 15.3M (jsonl only, 90% missing) or 45.9M (3x inflated)
- After: 129.5M (traces + non-traced jsonl sessions)
- traces alone: 103.8M (modelInfo.input + output, includes cache)
- 10-day total: 386M tokens across codebuddy + workbuddy traces
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| let traceObj; | ||
| try { | ||
| const raw = fssync.readFileSync(filePath, "utf8"); |
|
|
||
| let traceObj; | ||
| try { | ||
| const raw = fssync.readFileSync(filePath, "utf8"); |
…nd} return type
resolveCodebuddyProjectFiles and resolveWorkbuddyProjectFiles now return
{path, kind} objects (jsonl | trace | log) instead of bare strings, to
support the new trace JSON branch. Update the two affected tests to map
entries to .path before asserting.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughCodeBuddy and WorkBuddy discovery now returns typed file entries and includes trace JSON files. Both parsers aggregate trace usage, support newer usage fields, deduplicate trace data, and persist bounded state. CodeBuddy log fallback is opt-in. ChangesTrace ingestion
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant CodeBuddyDiscovery
participant CodeBuddyParser
participant TraceJSON
participant DedupState
CodeBuddyDiscovery->>CodeBuddyParser: provide typed trace entries
CodeBuddyParser->>TraceJSON: read finalized trace totals
TraceJSON-->>CodeBuddyParser: return model and token usage
CodeBuddyParser->>DedupState: check and persist trace and session state
sequenceDiagram
participant WorkBuddyDiscovery
participant WorkBuddyParser
participant TraceJSON
participant DedupState
WorkBuddyDiscovery->>WorkBuddyParser: provide typed trace entries
WorkBuddyParser->>TraceJSON: read trace totals
TraceJSON-->>WorkBuddyParser: return normalized token usage
WorkBuddyParser->>DedupState: check and persist trace IDs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
test/rollout-parser.test.js (1)
7079-7082: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd tests for trace discovery and trace aggregation.
The resolvers now emit
kind: "trace"entries, and the cursor schema gainedseenTraceIdsandtracedSessionIds. No test creates atraces/<pid>/trace_*.jsonfile. Add cases that:
- assert the resolver returns a
traceentry fortraces/<pid>/trace_*.json;- run the parser twice over the same trace file and assert the second run aggregates nothing;
- assert a session with both a trace file and a JSONL file is counted once.
The third case directly covers the double-count risks raised in
src/lib/rollout.js.Based on the coding guideline: "Run parser and synchronization tests, including consecutive
syncruns after changingsync.jsor cursor schema, to detect state pollution."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/rollout-parser.test.js` around lines 7079 - 7082, Add tests in the rollout parser suite for trace handling: create a traces/<pid>/trace_*.json fixture and assert the resolver returns an entry with kind "trace"; parse the same trace twice and verify the second run adds nothing; and combine that trace with a session JSONL file to assert the session is counted only once. Include cursor state fields seenTraceIds and tracedSessionIds as needed, and cover consecutive parser runs without state pollution.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/rollout.js`:
- Line 7978: The rollout aggregation currently counts trace and JSONL totals for
the same session. In src/lib/rollout.js lines 7978-7978, update the file sort to
place kind: "trace" entries before kind: "jsonl" entries. In
parseWorkbuddyIncremental at src/lib/rollout.js lines 8727-8797, maintain a
persisted tracedSessionIds set, add each trace.sessionId in the trace branch,
and skip JSONL usage records whose session IDs are already tracked.
- Around line 8828-8832: Update both WorkBuddy provider accesses in the loop
around the messageId check and model usage to use optional chaining, matching
the CodeBuddy branch’s safe access pattern. Preserve the topUsage fallback while
ensuring records with usage but no providerData do not throw when evaluating
provider.messageId or provider.model.
- Around line 8387-8411: Update the CodeBuddy usage extraction around rawUsage
and topUsage so reasoning tokens come from completion/output details rather than
prompt/input details. Subtract reasoningTokens from completionTokens before
assigning the billable output_tokens value, and avoid adding those tokens again
through reasoning_output_tokens; preserve the five-column billing model and
existing fallback behavior.
---
Nitpick comments:
In `@test/rollout-parser.test.js`:
- Around line 7079-7082: Add tests in the rollout parser suite for trace
handling: create a traces/<pid>/trace_*.json fixture and assert the resolver
returns an entry with kind "trace"; parse the same trace twice and verify the
second run adds nothing; and combine that trace with a session JSONL file to
assert the session is counted only once. Include cursor state fields
seenTraceIds and tracedSessionIds as needed, and cover consecutive parser runs
without state pollution.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 098b7987-72e1-46ff-943f-d8d3028754e5
📒 Files selected for processing (2)
src/lib/rollout.jstest/rollout-parser.test.js
| } | ||
|
|
||
| files.sort((a, b) => a.localeCompare(b)); | ||
| files.sort((a, b) => a.path.localeCompare(b.path)); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift
Trace and JSONL sources are counted additively for the same session. Both providers ingest workflow-level trace totals and message-level JSONL totals. The shared root cause is that trace-based suppression is not in effect before JSONL aggregation runs.
src/lib/rollout.js#L7978-L7978: orderkind: "trace"entries beforekind: "jsonl"entries sotracedSessionIdsis populated before the CodeBuddy JSONL branch checks it at Line 8369.src/lib/rollout.js#L8727-L8797: add a persistedtracedSessionIdsset toparseWorkbuddyIncremental, recordtrace.sessionIdin the trace branch, and skip JSONL usage records for those sessions.
📍 Affects 1 file
src/lib/rollout.js#L7978-L7978(this comment)src/lib/rollout.js#L8727-L8797
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lib/rollout.js` at line 7978, The rollout aggregation currently counts
trace and JSONL totals for the same session. In src/lib/rollout.js lines
7978-7978, update the file sort to place kind: "trace" entries before kind:
"jsonl" entries. In parseWorkbuddyIncremental at src/lib/rollout.js lines
8727-8797, maintain a persisted tracedSessionIds set, add each trace.sessionId
in the trace branch, and skip JSONL usage records whose session IDs are already
tracked.
…ace sort order Address CodeRabbit review feedback on PR xiufengsun#399: 1. Reasoning tokens: read from completion_tokens_details (not prompt_tokens_details) — verified on real data that reasoning lives in completion_details (999) not prompt_details (0). Also subtract reasoning from completionTokens to avoid double-counting (completion already includes reasoning). 2. WorkBuddy provider null pointer: use optional chaining (provider?.messageId) — provider (entry.providerData) may be null when only top-level usage is present (topUsage fallback path). 3. File sort order: sort trace files before jsonl files so tracedSessionIds is populated before jsonl parsing — prevents double-counting sessions that have both a trace file and a jsonl file. 4. Add trace-specific tests (test/codebuddy-traces.test.js): - trace discovery under traces/<pid>/trace_*.json - idempotent re-parse (traceId dedup) - trace + jsonl same session counted once (tracedSessionIds dedup)
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/codebuddy-traces.test.js`:
- Around line 75-78: Update the environment cleanup block in the test teardown
to check saved.CODEBUDDY_HOME instead of saved.CODEX_HOME, preserving deletion
only when CODEBUDDY_HOME was originally undefined and otherwise restoring its
saved value.
- Around line 13-58: Update the traceJson and jsonlLine fixture builders to use
nullish fallback semantics for optional numeric fields, replacing || with ?? for
values such as totalTokens, miInput, miOutput, miCached, promptTokens,
completionTokens, cachedTokens, and reasoningTokens. Preserve explicit zero
values while retaining defaults for null or undefined inputs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: db922595-236e-499b-971a-aacae9e31ec8
📒 Files selected for processing (2)
src/lib/rollout.jstest/codebuddy-traces.test.js
🚧 Files skipped from review as they are similar to previous changes (1)
- src/lib/rollout.js
… in fixtures Address remaining CodeRabbit review feedback: - Fix saved.CODEX_HOME → saved.CODEBUDDY_HOME (copy-paste typo in cleanup) - Replace || with ?? in test fixtures to preserve explicit zero values
….messageId Adversarial review found two more provider dereferences without optional chaining in the workbuddy branch: - provider.model (model extraction) - provider.messageId (dedup key, second access after the typeof check) Both are safe in practice (guarded by typeof checks) but use provider?. for consistency with the codebuddy branch and to be robust against future refactors that might remove the guard.
…ting bug) The workbuddy sqlite fallback branch (dbExists) computed total_tokens = inputTokens only, dropping completionTokens + cacheRead + cacheCreation + reasoningTokens. This is a pre-existing bug in the upstream codebase (present in main), not introduced by this PR, but fixed here since we're already touching the workbuddy collector.
xiufengsun
left a comment
xiufengsun
left a comment
There was a problem hiding this comment.
Reviewed exact head 6016d916e96f08c11a4b5bd30aae7aac0363849f merged onto current main 739e759950e4a470a3886feaee9868b5bf4ba5b6.
NO-MERGE: the targeted suite passes (234/234), but there are still deterministic accounting regressions not covered by it.
-
Upgrading an existing CodeBuddy cursor double-counts historical JSONL usage.
tracedSessionIdsonly suppresses JSONL lines parsed after this code lands (src/lib/rollout.js:8049-8057,8373-8380). It never retracts the same session's totals already persisted incursors.hourly. Reproduction: parse a JSONL session totaling 20,500 on the old parser, then parse a 10,000-token trace for that same session on this head. The queued bucket becomes 30,500; if the trace is authoritative it must be 10,000. This needs a safe one-time migration/rebuild, plus an upgrade regression test; a same-run test is not sufficient. -
WorkBuddy double-counts trace + JSONL even on a fresh install. Its trace branch (
8760-8825) has no session-level suppression before the JSONL branch (8834+). A synthetic 10,000-token trace plus the same session's 10,000-token JSONL produces 20,000. Please apply equivalent source precedence and add a WorkBuddy overlap test. -
Older CodeBuddy installs can silently lose all new usage.
resolveCodebuddyProjectFilesnow disables extension logs unlessTOKENTRACKER_CODEBUDDY_LOG_FALLBACK=1(7909-7918), but those logs are the only source for versions with neither traces nor JSONLrawUsage. An undocumented opt-in is not a safe fallback. Source selection needs to happen automatically per session/source with cross-source deduplication, and needs a regression fixture for that older layout.
Please address all three paths and include upgrade-state fixtures, not only clean-cursor fixtures.
3 participants
Summary
CodeBuddy CLI writes authoritative token usage to
~/.codebuddy/traces/<pid>/trace_*.json(and~/.workbuddy/traces/for WorkBuddy). Each trace JSON carriestrace.modelInfo.{totalInputTokens, totalOutputTokens, totalCachedTokens, callCount}with 100% field stability (verified on 286 real traces).The existing jsonl reader only sees ~10% of messages — 90% of assistant messages in newer CodeBuddy versions (glm-5.2) lack
providerData.rawUsagein~/.codebuddy/projects/**/*.jsonl. This causes massive undercounting.Additionally, the extension-log path overlaps with the jsonl path using a different dedup key namespace, causing 3x inflation when both are active.
Root cause
resolveCodebuddyProjectFilesonly scannedprojects/and extension logs, nevertraces/providerData.rawUsageon most assistant messages; token data only exists in tracesentry.uuidvscodebuddy:extension-log:agentId:sec), so the same LLM round-trip is counted twiceproviderData.modelwithoutrawUsage; some carry the camelCase top-levelusagefield insteadFix
resolveCodebuddyProjectFiles: scantraces/dir, return{path, kind}objects (jsonl | trace | log)parseCodebuddyIncremental: add trace JSON branch — readmodelInfo, dedup bytraceId, skip jsonl messages for traced sessions (tracedSessionIds) to avoid double-countingresolveWorkbuddyProjectFiles: sametraces/scan (WorkBuddy is the renamed successor of CodeBuddy, same trace schema)parseWorkbuddyIncremental: same trace JSON branchproviderData.rawUsageis absent, fall back to top-levelusagefield (camelCase convention)TOKENTRACKER_CODEBUDDY_LOG_FALLBACK=1to enable) — overlaps with jsonl, different dedup namespace, causes 3x inflationtracedSessionIdsis populated before jsonl parsing (prevents double-counting sessions with both trace and jsonl)completion_tokens_details(notprompt_tokens_details), subtract fromcompletionTokensto avoid double-countingtotal_tokens = inputTokens→total_tokens = input + output + cacheRead + cacheCreation + reasoning(pre-existing bug in main)Token math (trace branch)
This matches the jsonl branch convention (cached is separate from input).
Scope
src/lib/rollout.js)test/codebuddy-traces.test.js,test/rollout-parser.test.js)Verification (real machine, 2026-07-31)
Tests
test/codebuddy-traces.test.js: 3/3 pass (trace discovery, idempotent re-parse, trace+jsonl dedup)test/rollout-parser.test.js: 230 pass / 1 fail ([bug]: Zed Agent的用量无法统计到 #154 pre-existing, unrelated)test/install-resolver.test.js: 28/28 passtest/dual-install-integration.test.js: 1/1 passtest/codex-wsl-shadow.test.js: 1/1 passCommits (6, based on main)
db578b1fe52757{path, kind}return typecfd0eac5039510??nullish fallback8f1be2cprovider.model/messageIdoptional chaining6016d91total_tokensformula fix (pre-existing bug)CodeRabbit review
All actionable comments addressed:
||→??nullish fallbackChecklist
fix:,test:)Note
This PR is independent of #397 (codex WSL shadowing fix). Both touch
src/lib/rollout.jsbut in different functions (codebuddy/workbuddy collectors vs codex sources builder). If #397 merges first, this should rebase cleanly.