How could we implement provider-specific features like grounding for Gemini?

[lamberttraccard]

Hi,

I was wondering how you think we could access specific provider features like search grounding or prompt caching that are offered for specific providers like Gemini?

see https://ai.google.dev/gemini-api/docs/google-search

[lamberttraccard]

@ilvalerione no suggestions ? 🙂

[ilvalerione]

I discussed this topic with other developers, because the output and the logic of this provider-specific features are quite different. For example when you add the provider's web_search tool, they reply with a completely different response format.

• Gemini: https://ai.google.dev/gemini-api/docs/google-search
• Anthropic: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/web-search-tool
• OpenAI: https://platform.openai.com/docs/guides/tools-web-search

It's a completely different output from the same API endpoint. OpenAI requires you to use a different API endpoint (called "Response API") to use we search tool, and it's quite interesting, becuase you could eventually create a dedicated provider like OpeAIResponseAPI, with it's own message mapper and so on.

With Anthropic and Gemini it should be found a way to harmonize the output. But this is just my feeling, other point of views are appreciated.

[ilvalerione]

A change of the Neuron Message system may emerge to add more abstractions and flexibility. Becuase these kind of outputs also have an impact on streaming.

[ilvalerione]

@lamberttraccard look at #291

[michelandres]

So at the moment, is it not possible to use provider-native tools directly?
For example, with Gemini I tried to use google_search and configured my agent like this:

protected function tools(): array
{
    return [
        ProviderTool::make(
            type: 'google_search'
        ),
        $this->toolkit
    ];
}

…but I only get an error when running it. Am I missing some configuration, or is this not yet supported?

[ilvalerione]

Nope. It's the provider that does not support it. Gemini doesn't allow to list both.

Tools and Toolkits are the most reliable ways to attach capabilities to your agent.