Autogen + ToolDec

[akashicMarga]

Adding ToolDec in auto gen code execution will be really helpful for making API calls and other commands that are critical and need to follow certain structures. As Per the ToolDec paper (Syntax Error-Free and Generalizable Tool Use for LLMs via Finite-State Decoding)

" We evaluated multiple prior methods and their ToolDec-enhanced versions on a variety of tasks involving tools like math functions, knowledge graph relations, and complex real-world RESTful APIs. Our experiments show that ToolDec reduces syntactic errors to zero, consequently achieving significantly better performance and as much as a 2x speedup. We also show that ToolDec achieves superior generalization performance on unseen tools, performing up to 8x better than the baselines"

https://arxiv.org/abs/2310.07075

[sonichi]

Interesting. How does it fit in the autogen framework? What needs to be changed besides the code execution part?

[akashicMarga]

As of now, I think we can add proper function calls to ensure that the write functions are invoked with correct parameters or any API calls with a definite structure. I guess we can have it in code_utils. I haven't gone through the auto gen whole codebase and was testing out a RAG-based two-agent conversation which involved one user proxy who can run code and another agent assistant who can help write code. So, let's say the generated output is not right sometimes in code execution—maybe some variable names are incorrect or the right function is not called. We can add this tool to increase accuracy. If I am missing something, please let me know.

[sonichi]

How about using another agent to fix the code before execution? Then we can easily switch on/off or replace that agent.

[akashicMarga]

yes, that's good. but adding too many agents won't add latency and more steps in execution? it's something that is specific to code execution so can be a parameter of that module itself?

[sonichi]

Making it a parameter is an option, though it could be a growing pain to maintain it. Latency is not necessarily increased if the design is right. The increase of the steps in execution is inevitable when we add this functionality, no matter as an agent or not.