In local with up-to-date main :

  File "/home/zero/gitingest/src/server/query_processor.py", line 91, in process_query
    summary, tree, content = ingest_query(query)
                             ^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/src/gitingest/ingestion.py", line 85, in ingest_query
    return format_node(root_node, query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/src/gitingest/output_formatter.py", line 52, in format_node
    token_estimate = _format_token_count(tree + content)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/src/gitingest/output_formatter.py", line 188, in _format_token_count
    total_tokens = len(encoding.encode(text, disallowed_special=()))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zero/gitingest/.venv/lib/python3.11/site-packages/tiktoken/core.py", line 124, in encode
    return self._core_bpe.encode(text, allowed_special)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: RuntimeError(StackOverflow)

Tiktoken seems to panic, this is a rust error.

I'll try to investigate this more tomorrow.

bug: stackoverflow error with some repos #375

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions