[FIX] v1.33.0 can not chat in stream mode

[thinker007]

Describe the bug

A clear and concise description of what the bug is. Please include what you were expecting to happen vs. what actually happened.
/khoj/src/khoj/processor/conversation/openai/utils.py:103 in │
│ completion_with_backoff │
│ │
│ 100 │ # Calculate cost of chat │
│ 101 │ input_tokens = chunk.usage.prompt_tokens if hasattr(chunk, "usa │
│ else 0 │
│ 102 │ output_tokens = chunk.usage.completion_tokens if hasattr(chunk, │
│ chunk.usage else 0 │
│ ❱ 103 │ cost = chunk.usage.model_extra.get("estimated_cost") or 0 # Es │
│ returned by DeepInfra API │
│ 104 │ tracer["usage"] = get_chat_usage_metrics(model_name, input_toke │
│ tracer.get("usage"), cost) │
│ 105 │ │
│ 106 │ # Save conversation trace │
╰───────────────────────────────────────────────────────────────────────────╯
AttributeError: 'NoneType' object has no attribute 'model_extra'

To Reproduce

Steps to reproduce the behavior:

After model setting ,just chat no return

Screenshots

If applicable, add screenshots to help explain your problem.

Platform

• Server:
  • Cloud-Hosted (https://app.khoj.dev)
  • Self-Hosted Docker
  • Self-Hosted Python package
  • Self-Hosted source code
• Client:
  • Obsidian
  • Emacs
  • Desktop app
  • Web browser
  • WhatsApp
• OS:
  • Windows
  • macOS
  • Linux
  • Android
  • iOS

If self-hosted

• Server Version [e.g. 1.0.1]:

Additional context

Add any other context about the problem here.

[thinker007]

AttributeError: 'NoneType' object has no attribute 'model_extra' still exist

[debanjum]

Hey @thinker007, can you share more details?

• Which chat model are you seeing this issue with?
• Which client (e.g Emacs, Obsidian, Web) are you using to chat?
• What does your Khoj setup look like (OS, Self-hosting Setup method)?
• Was the same chat model streaming responses before v1.33.0 or did you only try it after v1.33.0 was released?

[thinker007]

[15:21:31.254144] ERROR khoj.processor.conversation.openai.utils: Error in llm_thread: 'NoneType' object has no attribute 'model_extra' utils.py:227
╭───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────╮
/khoj/src/khoj/processor/conversation/openai/utils.py:218 in │
│ llm_thread │
│ │
│ 215 │ │ # Calculate cost of chat │
│ 216 │ │ input_tokens = chunk.usage.prompt_tokens if hasattr(chunk, "usage") and │
│ chunk.usage else 0 │
│ 217 │ │ output_tokens = chunk.usage.completion_tokens if hasattr(chunk, "usage") and │
│ chunk.usage else 0 │
│ ❱ 218 │ │ cost = chunk.usage.model_extra.get("estimated_cost") or 0 # Estimated costs │
│ returned by DeepInfra API │
│ 219 │ │ tracer["usage"] = get_chat_usage_metrics(model_name, input_tokens, │
│ output_tokens, tracer.get("usage"), cost) │
│ 220 │ │ │
│ 221 │ │ # Save conversation trace │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'NoneType' object has no attribute 'model_extra'

only after v1.33.0 was released

khoj macos. Self-hosting Setup method
using Web chat to gpt-4o-mini model

[thinker007]

It seems "cost = chunk.usage.model_extra.get("estimated_cost") or 0 # Estimated costs returned by DeepInfra API"

this is the key, after comment it out fix this error

[thinker007]

Hey @thinker007, can you share more details?

• Which chat model are you seeing this issue with?
• Which client (e.g Emacs, Obsidian, Web) are you using to chat?
• What does your Khoj setup look like (OS, Self-hosting Setup method)?
• Was the same chat model streaming responses before v1.33.0 or did you only try it after v1.33.0 was released?

@debanjum I think I got it
Some OpenAI compatible API do not have usage in stream mode
So the response object does not have usage property