Tool Calls Occasionally Fail Silently or Return Incomplete Outputs (Groq + Pydantic AI Integration)

[Rikhil-Nell]

Question

Hi team,

I've been working with pydantic_ai for some time now and integrating it into an agentic architecture using both Groq (LLaMA 3.3 70B Versatile) and OpenAI providers.

Here's the repository where the full setup is implemented:
🔗 https://github.com/Rikhil-Nell/Multi-Agentic-RAG

Also here is a deployed streamlit link:
🔗 Streamlit App
(Please note: I'm a student with very limited API credits, so please be mindful if testing.)

The agent is fairly minimal right now — it uses:

• A RAG vector search tool (retrieve_relevant_documentation)
• A dictionary lookup tool (call_dictionary using the Merriam-Webster API)

❗ The Problem

Tool usage is highly inconsistent. At times, everything works perfectly — the agent recognizes intent, calls the right tool, parses arguments, and returns the result correctly.

However, during certain stretches (seemingly random), the model stops making actual tool calls and instead returns placeholder-style outputs like:

<function=call_dictionary({"word": "suburbs"})</function>

This happens despite:

• The tool functions being cleanly defined using @tool decorators
• Well-structured prompts that explicitly direct the model to use tools when needed
• Confirmed success of the same codebase and logic at other times

A sample system prompt looks like this:

Be concise, reply professionally. Use the tool call_dictionary when asked to define a word. Use retrieve_relevant_documentation for queries out of scope. Never respond with the tool call string — only invoke tools directly and wait for the result. Never fabricate an answer. Always begin with RAG if unsure.

I’ve verified that:

• The code path is not skipping function execution
• There are no exceptions thrown during successful runs
• This issue does not seem to stem from the tool logic itself

It feels like either:

• The model isn't parsing the system prompt consistently
• Or there's something flaky in the tool call orchestration layer with pydantic_ai or Groq integration

💡 Questions / Help Needed

• Is this a known limitation when using Groq models through pydantic_ai?
• Are there internal retry mechanisms or validation steps I can hook into?
• Are tool calls non-deterministic across inference providers?
• How can I debug cases where the model outputs a tool call string instead of invoking the function?

I'm not sure if this is a problem with the model, the inference provider, pydantic_ai, or my orchestration — but I'd greatly appreciate any pointers or support.

---

yes I used AI for this, I am too emotional and frustrated right now to be able to write a good issue, so I do apologize, I will provide any and all information required for diagnosis if it means my code works please.

Additional Context

Pydantic_ai version: 0.1.10
Python version: 3.13

[MarianPetlovanyi]

+1
Hi, I have the same issue.

[Rikhil-Nell]

+1 Hi, I have the same issue.

thanks a lot for at least letting me know that I am not the only one

[DouweM]

@Rikhil-Nell Unfortunately, this looks like an issue with Llama 3.3 that's also affecting providers other than Groq, like Bedrock: #1649.

PydanticAI passes available tools to the provider over the API (not directly in the prompt), and reads tool calls from the API response (not directly from the model response text). The provider is responsible for properly prompting the model to generate tool calls in a format the provider can then parse out of the model response. In this case, it looks like the model is not consistently following the provider's instructions, meaning the provider fails to parse out the tool calls from the model response, and they end up being passed back to the API user (PydanticAI and your code) as if there were a simple text response.

As PydanticAI relies on the provider's prompting/parsing behavior, and indirectly on the model's ability to actually follow those rules, a failure at either of those stages is not something that can be addressed directly in PydanticAI and would need to be resolved with better provider-specific scaffolding around the model. I suggest reporting the issue to Groq, as this is something you'd see when using the Groq API with Llama 3.3 with or without PydanticAI (and as mentioned above, even with providers that support Llama 3.3. other than Groq).