RuntimeError(StackOverflow) with tiktoken.encoding_for_model('gpt-4').encode(txt, disallowed_special=())

[sumitKpandey]

Hi,

we are using tiktoken 0.5.2 to calculate the token count. It is working most of the time, but we are seeing the error: RuntimeError(StackOverflow) with one of the runs.

Here is the part of the stack trace:
SystemLog: File "/azureml-envs/azureml_40acd812113c64de92b8cc12dc98c336/lib/python3.11/site-packages/tiktoken/core.py", line 120, in encode
SystemLog: return self._core_bpe.encode(text, allowed_special)

I also checked the text size and it's pretty big (around 2 GB) is there any limitation on the size?

[ktrnka]

Ran into this today. Adding some more info in case it helps:

Line of code panicing:
embedding_size = len(self.tokenizer.encode(content.decode('utf-8', 'ignore'), disallowed_special=()))

Context:
tokenizer_name='o200k_base' self.tokenizer = tiktoken.get_encoding(tokenizer_name)

Input files that trigger it (from https://github.com/microsoft/TypeScript):

• tests/cases/fourslash/reallyLargeFile.ts
• tests/baselines/reference/hugeDeclarationOutputGetsTruncatedWithError.types
• tests/baselines/reference/tsserver/textStorage/should-be-able-to-return-the-file-size-when-a-JS-file-is-too-large-to-load-into-text.js

tiktoken version 0.9.0 on Ubuntu 24.04

---

I'm not surprised that it fails on these files, just adding extra info in case anyone wants to try some examples or adjust the way these errors are handled