Issues using non-"OpenAIModel" models with Azure AI Foundry

[schmidt-marvin]

Question

✅ Question / request still persists on most up-to-date version of PydanticAI.
✅ I checked open/closed issues and pull requests before opening this issue.

Problem description

Hey everyone! This is the first issue I'm opening, so pardon any rookie mistakes on the style / reporting side.

Within my application, I want to support various different LLMs hosted within Azure AI Foundry. For OpenAIModels (eg. gpt-4.1), this works really well following the examples from your documentation page on Azure AI Foundry.

I am however having issues using other models. Most of my testing was on the Mistral-Large-2411 model, so I'll just describe my issue implementing this model.

I assumed that usage of MistralModel and MistralProvider are required for using this particular model. As a reference, I used the template from #1617:

model = MistralModel(
    "XXX-mistral-large-2411",
    provider=MistralProvider(
            api_key=chat_settings.AZURE_MISTRAL_LARGE_KEY,
            base_url=chat_settings.AZURE_MISTRAL_LARGE_ENDPOINT,
        ),
)

(Notes: XXX is the project name which I redacted here; The endpoint, as well as model names are known-good and work in a separate implementation not based on PydanticAI.)

The result to using it like this yields a HTTPValidationError:

Stack trace here

Traceback (most recent call last):
  File "/app/app/chat/router.py", line 232, in ai_completion
    await task
  File "/app/.venv/lib/python3.12/site-packages/pydantic_graph/graph.py", line 166, in run
    async for _node in graph_run:
  File "/app/.venv/lib/python3.12/site-packages/pydantic_graph/graph.py", line 809, in __anext__
    return await self.next(self._next_node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pydantic_graph/graph.py", line 782, in next
    self._next_node = await node.run(ctx)
                      ^^^^^^^^^^^^^^^^^^^
  File "(...)/chat.py", line 63, in run
    async with chat_agent.run_stream(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pydantic_ai/agent.py", line 996, in run_stream
    async with node._stream(graph_ctx) as streamed_response:  # pyright: ignore[reportPrivateUsage]
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pydantic_ai/_agent_graph.py", line 309, in _stream
    async with ctx.deps.model.request_stream(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pydantic_ai/models/instrumented.py", line 190, in request_stream
    async with super().request_stream(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pydantic_ai/models/wrapper.py", line 36, in request_stream
    async with self.wrapped.request_stream(messages, model_settings, model_request_parameters) as response_stream:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pydantic_ai/models/mistral.py", line 169, in request_stream
    response = await self._stream_completions_create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pydantic_ai/models/mistral.py", line 232, in _stream_completions_create
    response = await self.client.chat.stream_async(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/mistralai/chat.py", line 739, in stream_async
    raise models.HTTPValidationError(data=data)
mistralai.models.httpvalidationerror.HTTPValidationError: {}

Related issues

This request is similar to #940 which has been closed.

Question

• Am I using the model wrong here or is this a limitation of PydanticAI?
• For the case of other models also available from Azure AI Foundry, for instance llama-3-3-70B, how am I able to find a reference on how to use them? I wasn't able to find a LlamaModel or MetaModel or something similar.

Additional Context

No response

[DouweM]

I assumed that usage of MistralModel and MistralProvider are required for using this particular model.

@schmidt-marvin It's a bit confusing, but model classes implement specific APIs, and provider classes implement API base URLs and authentication. So in, your case, no matter which actual model you're using, you'd want to use the OpenAIModel because the Azure API is OpenAI-compatible, and the AzureProvider because you're not actually using OpenAI. So the example on https://ai.pydantic.dev/models/openai/#azure-ai-foundry should work also for non-OpenAI models.

The only issue you may run into is that while the Azure API is OpenAI-compatible, not all models have the same requirements of e.g. tool JSON schemas as OpenAI's do. So if it doesn't work right, you'd need to override the JSON schema transformation. We don't have a clean solution for that yet, but are designing one in #1782. If you run into issues with AzureProvider + OpenAIModel + Mistral-Large-2411, let us know and we can help you work around them.

[schmidt-marvin]

Ah, I see - Thank you so much for the swift response and explanation!

[ttamg]

@DouweM Related to this, am I right in understanding that the Azure AI Inference library which we are now starting to use instead of the OpenAI library with the AzureOpenAI provider is then a different Provider which is not yet implemented in Pydantic AI Agents?

This python package seems to be a bit different from the AzureOpenAI package and I think might have a slightly different url with /model after the URL. It also has different method calls

Seems Microsoft is keen we use this Python library now. Here's a link in case it hasn't crossed your radar. Thanks https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-inference

[DouweM]

@ttamg Using a different SDK would require a new Model class, which (as you can see from OpenAIModel) is quite a bit of work to build and maintain as we add new features to PydanticAI. So if a provider has an OpenAI-compatible API that works with the OpenAI SDK, we prefer using the existing OpenAIModel and just adding a Provider class that sets a base URL and handles authentication (like the existing AzureProvider) instead of adding a whole new Model.

If Microsoft is pushing in that direction and their OpenAI-compatible API will start falling behind, we'll have to do that at some point, but for the moment I'm hoping all the providers keep seeing the value of being OpenAI-compatible which saves us a bunch of work.