Skip to content

Add token counts, timestamps, and model to rollouts #1583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

bl-ue
Copy link

@bl-ue bl-ue commented Jul 15, 2025

This PR adds per-message token count info and timestamps and per-chat selected model info to the rollout JSONL files. Note that this change is only in the Rust version; I didn't add this information to the TypeScript version's JSON rollout files, but I can if desired.

Codex support in Splitrail is now implemented and waiting for this PR.

Related: #1572

Copy link

github-actions bot commented Jul 15, 2025

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@bl-ue bl-ue force-pushed the enhance-recorded-chat-history-071525 branch from 30edd03 to 9bf17f4 Compare July 15, 2025 15:41
@bl-ue
Copy link
Author

bl-ue commented Jul 15, 2025

I have read the CLA Document and I hereby sign the CLA

github-actions bot added a commit that referenced this pull request Jul 15, 2025
@bl-ue bl-ue marked this pull request as draft July 16, 2025 01:21
@bl-ue bl-ue marked this pull request as ready for review July 17, 2025 20:30
@bl-ue
Copy link
Author

bl-ue commented Jul 17, 2025

Ready for review.

@bl-ue bl-ue marked this pull request as draft July 21, 2025 15:17
…rollout files

This ensures that the metadata is updated as well as the contents, so that file watchers can get immediate updates.
@bl-ue bl-ue marked this pull request as ready for review July 21, 2025 16:09
@bl-ue
Copy link
Author

bl-ue commented Jul 21, 2025

Hi @bolinfest! This PR adds token, timestamp, and model information to rollout files. This enables our new tool, Splitrail, to track token usage, cost, and throughput for Codex users. It also makes it easier for other tools to do the same. Do you mind taking a look? Thank you!

@bl-ue
Copy link
Author

bl-ue commented Jul 26, 2025

cc @aibrahim-oai

@aibrahim-oai
Copy link
Collaborator

aibrahim-oai commented Jul 26, 2025

Tools can already calculate tokens used from the rollout info. I don't think there is much benefit in adding them in response items. What is the use?

@bl-ue
Copy link
Author

bl-ue commented Jul 26, 2025

Hi @aibrahim-oai! Thank you for reviewing so quickly. Yes, that makes sense, but automatic input caching makes accurate calculations impossible. When input exceeds 1024 tokens, inputs are automatically cached (see here), so there's no way to determine which part of the input was cached and which wasn't; therefore, we can't calculate cost accurately.

In addition, rollouts don't currently store model information, so it's not possible to determine which tokenizer to use, nor can we determine model/token cost for accurate cost calculation. This PR stores model information to address this.

Last but not least, Codex can be used with custom providers, and it would be difficult to perform tokenization on rollouts that use custom models/providers; with open-source models in Ollama, you'd have to download and use a tokenizer, and with providers that don't distribute their tokenizers, you'd have to use an API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants