Support for Tool Use

[felixdivo]

Does the sglang frontend support tool use yet? Tool use allows providing a list of functions with calling schemas and descriptions. Supported LLMs (among them Llama 3.2 and OpenAI GPT-4-turbo and newer) can then call them, which typically suspends generations (by detecting special tokens) and allows the client caller code to execute the function and provide the result back to the LLM. Building this yourself or with control flow is different/significantly less flexible since one has to specify the positions where such (constrained) function calling generation can happen.

Further reading:

• https://medium.com/@zilliz_learn/function-calling-with-ollama-llama-3-2-and-milvus-ac2bc2122538
• https://platform.openai.com/docs/guides/function-calling#fine-tuning-may-help-improve-accuracy-for-function-calling

[felixdivo]

There is #573

[felixdivo]

And #2544

[r3lativo]

I'd really like to dive more into tool integration, where can I found more info about this? For example a list of LLMs with tools (I tried tinyllama as a toy and it said me it doesn't have it), or even better an example of use.