Antropic chat / completion translation driver is mistaken in tool_choice format.

[HarrievG]

kong/kong/llm/drivers/anthropic.lua

Line 166 in b3380fd

Should this not be in line with the chat completions api?

kong/kong/llm/drivers/anthropic.lua

Line 148 in b3380fd

messages.tool_choice = request_table.tool_choice

Currently, when using the chat completions api with tools :

{"type":"error","error":{"type":"invalid_request_error","message":"tool_choice: Input should be a valid dictionary or object to extract fields from"}}

[spacewander]

What does "anthropic chat completion" mean? There are only v1/complete and /v1/messages for Anthropic.

[HarrievG]

Are you implying that Kong does not do what I thought it does, and that I am assuming it is configured as being a cross-api gateway that takes openAi-api requests and route them to target platform API, but is set to passthrough?

Besides that the code I am referencing, is directly about /completions.

[HarrievG]

Oke, I have figured it out, and got it to work, with openAI request to antropic.

For open-ai api documentation states:

tool_choice
string or object

Optional
How the model should select which tool (or tools) to use when generating a response. See the tools parameter to see how to specify which tools the model can call.

so the documentation states it can be :

{ "tool_choice" : "auto"} OR { "tool_choice" : null }

Where antropic only accepts
{ "tool_choice" : { "type" : "auto" }}

This is where i am thinking the driver is not unifying things, but i need to have the whole request translated before i know for sure.

To test the tool_choice difference, I modified my own back end, to send the { "tool_choice" : { "type" : "auto" } } with the openai-api request to antropic enpoint and all worked as intended.

Is there a way to check if it gets translated at all or its passthrough and this is just the normal behaviour with openai compatibility for anthropic..?

[spacewander]

@HarrievG
Thanks for your detailed report. I will submit a PR to fix it soon.

[HarrievG]

I'm sorry for the late reply; unfortunately I do not have the means to build and install a custom package/ build.

[spacewander]

It's fine. I will find someone else to take a look at this.

[subnetmarco]

@spacewander what's the latest on this?

[fffonion]

There's another PR working in progress by @ttyS0e and can be backported later.

[tysoekong]

Yeah this is all being fixed in a piece of work already ongoing, this will get patched.

There is a massive revamp to tool_use mappings coming next release, as well as response_format support, to fix all these issues.

[spacewander]

@subnetmarco
Here is the fix: #14694

@tysoekong
Would you review #14694? Many thanks!