-
Notifications
You must be signed in to change notification settings - Fork 2.8k
fix(models): handle mixed tool responses and text/media in LiteLLM ad… #4109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
36dfe40
fix(models): handle mixed tool responses and text/media in LiteLLM ad…
Akshat8510 1b93ecb
fix: refactor content conversion to support multipart and mixed tool …
Akshat8510 01c3773
docs: improve docstring and remove dead code as per AI review
Akshat8510 4768ddf
docs: improve docstring and remove dead code as per AI review
Akshat8510 fcd150d
Merge branch 'main' into fix-litellm-multipart
Akshat8510 827487d
docs: improve docstring and remove dead code as per AI review
Akshat8510 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -444,24 +444,36 @@ def _extract_cached_prompt_tokens(usage: Any) -> int: | |
|
|
||
|
|
||
| async def _content_to_message_param( | ||
| content: types.Content, | ||
| *, | ||
| provider: str = "", | ||
| content: types.Content, | ||
| *, | ||
| provider: str = "", | ||
| ) -> Union[Message, list[Message]]: | ||
| """Converts a types.Content to a litellm Message or list of Messages. | ||
|
|
||
| Handles multipart function responses by returning a list of | ||
| ChatCompletionToolMessage objects if multiple function_response parts exist. | ||
| This function processes a `types.Content` object, which may contain multiple | ||
| parts, and converts them into a format suitable for LiteLLM. It handles | ||
| mixed content, such as tool responses alongside text or other media, by | ||
| generating a list of messages. | ||
|
|
||
| - `function_response` parts are converted into `tool` role messages. | ||
| - Other parts (text, images, etc.) are grouped and converted into a | ||
| single `user` or `assistant` message. | ||
| - If the content contains only tool responses, it may return a single | ||
| message or a list, depending on the number of responses. | ||
|
|
||
| Args: | ||
| content: The content to convert. | ||
| provider: The LLM provider name (e.g., "openai", "azure"). | ||
| provider: The LLM provider name (e.g., "openai", "azure"), used for | ||
| provider-specific content handling. | ||
|
|
||
| Returns: | ||
| A litellm Message, a list of litellm Messages. | ||
| A single litellm Message, a list of litellm Messages, or an empty list if | ||
| the content is empty. | ||
| """ | ||
|
|
||
| # 1. Separate tool results (function_response) from other parts | ||
| tool_messages = [] | ||
| other_parts = [] | ||
|
|
||
| for part in content.parts: | ||
| if part.function_response: | ||
| response = part.function_response.response | ||
|
|
@@ -477,11 +489,64 @@ async def _content_to_message_param( | |
| content=response_content, | ||
| ) | ||
| ) | ||
| if tool_messages: | ||
| else: | ||
| other_parts.append(part) | ||
|
|
||
| # 2. Optimization: If there are ONLY tool messages, return them immediately | ||
| if tool_messages and not other_parts: | ||
| return tool_messages if len(tool_messages) > 1 else tool_messages[0] | ||
|
|
||
| # Handle user or assistant messages | ||
| role = _to_litellm_role(content.role) | ||
| # 3. Process the "other_parts" (text, images, reasoning, function_calls) | ||
| extra_message = None | ||
|
|
||
| if other_parts: | ||
| role = _to_litellm_role(content.role) | ||
|
|
||
| if role == "user": | ||
| user_parts = [part for part in other_parts if not part.thought] | ||
| message_content = await _get_content(user_parts, provider=provider) or None | ||
| if message_content: | ||
| extra_message = ChatCompletionUserMessage(role="user", content=message_content) | ||
|
|
||
| else: # assistant/model | ||
| tool_calls = [] | ||
| content_parts: list[types.Part] = [] | ||
| reasoning_parts: list[types.Part] = [] | ||
|
|
||
| for part in other_parts: | ||
| if part.function_call: | ||
| tool_calls.append( | ||
| ChatCompletionAssistantToolCall( | ||
| type="function", | ||
| id=part.function_call.id, | ||
| function=Function( | ||
| name=part.function_call.name, | ||
| arguments=_safe_json_serialize(part.function_call.args), | ||
| ), | ||
| ) | ||
| ) | ||
| elif part.thought: | ||
| reasoning_parts.append(part) | ||
| else: | ||
| content_parts.append(part) | ||
|
|
||
| message_content = await _get_content(content_parts, provider=provider) or None | ||
| reasoning_content = "\n".join([p.thought for p in reasoning_parts]) if reasoning_parts else None | ||
|
|
||
| extra_message = ChatCompletionAssistantMessage( | ||
| role="assistant", | ||
| content=message_content, | ||
| tool_calls=tool_calls if tool_calls else None, | ||
| thought=reasoning_content, | ||
| ) | ||
|
|
||
| # 4. COMBINE AND RETURN | ||
| final_messages = tool_messages + ([extra_message] if extra_message else []) | ||
|
|
||
| if not final_messages: | ||
| return [] | ||
|
|
||
| return final_messages if len(final_messages) > 1 else final_messages[0] | ||
|
||
|
|
||
| if role == "user": | ||
| user_parts = [part for part in content.parts if not part.thought] | ||
Akshat8510 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There appears to be a bug in the construction of
reasoning_content. It's currently usingp.thought, which seems to be a boolean flag, instead ofp.text, which holds the actual reasoning text. This will result in incorrect content for thethoughtfield of theChatCompletionAssistantMessage.