This repo contains the code that was used to fine tune the model during the Mistral Hackthon on 28.12.2026 by team SupaClaw.
The goal of the fine-tuning was to teach a small VLM, that was small enough to run locally on an iPhone, to use tools and interact with Telegram, Google Maps and the weather App, while being able to understand and talk back to the user. As a proxy for this task, our team chose optimising for the Tau2-bench benchmark, which is testing LLMs on tool usage, while using an other LLM as a 'User' that interacts with it.
The repo contains the code for the supervised fine-tuning, done on the Salesforce/APIGen-MT-5k, which is a set of questions/answers of GPT-4o and DeepSeek-V3 to questions tau-bench (first version of the benchmark). This first step helps us teach our model to read/answer questions in the required format and we were able to see quite good results with the full process taking ~3 hours.
For the RL process we decided to go with the GRPO method (popularized by DeepSeek) due to it being a computationally cheaper alternative to PPO. It involves prompting the model n times (in our case 4) and then having a 'Judge' decide which one was better. Providing feedback to our model and helping it learn.
The model went through the training dataset of Tau2-bench and used Mistral Large 3 as a 'User' that also gave feedback on the multiple responses. Results of the Reward graph per step can be seen below.
As the gains made by the model were very sparse, we decided to amend the reward funciton at step ~150 to add more rewards, not just for fully completed steps but also for partially completed ones (ie correct usage of tools). Around step ~400, the model started diverging with KL converge going up, therefore some parameters were tuned to make it's behaviour less consistent and fight overfitting on some edge solutions (as it was trying to go for the small wins, using tools but never actually answering the prompt correctly). The full RL process laster for ~5 hours.
A big mistake done during the RL process was to not iterate over tasks from different categories of questions from Tau2 (Airline, Telecom and Retail). Each time we restarted training, the model first had to go through all the Airline-specific questions before hitting the other categories. Thus, after 2 interruption and 5 hours of training, the model has not seen a single task from the Retail class and only a few from Telecom.
As the evaluation on Tau2 is done in tandem with another LLM, it takes quite some more time than we expected at first. Even when running 3 tasks in parallel, it took us 2 hours to run the full evaluation on the final model. This came at a surprise to us in the end and explains why not using LoRA has been a mistake, doing RL for just 5 hours for a 2h benchmark is obviously not enough time. In the end, since we have mainly done RL on the Airline class, the model did very well (for its size) on it while failing at the other 2 classes. But it was with great pleasure that we saw, that for the one class we actually trained for, our 3B model was able to outperform DeepSeek-R1 after just a few hours of training on a single GPU!
All training was done on an Nvidia H200 GPU provided by the Nvidia team. With 140GB VRAM, we decided to use its full capacity and always trained the model on full parameters, skipping LoRA techinques. This might have proven to be a mistake in the end, especially for RL since we did not have enough time to go over the whole dataset as the hackathon came to and end. Mistral API credits for Mistral Large 3 were provided by the Mistral team.
It was with quite some surprise that we realised once our evaluation was finished that Mistral Large 3 actually performs quite poorly on Tau2 benchmark (hitting 24% vs ~80% for top performing LLMs) which probably negatively affected our results during training, since we were distilling from a quite poor model. Even though, the role of a 'User' is supposed to be easier than that of the trainee.
Since evaluating on Tau2-bench showed us the importance of training on the actual benchmark (learning on Airline did not help us much on Retail or Telecom) we were able to conclude that small models indeed have good potential for agentic tasks, but with a caveat, the RL process has to be done on a set of tasks that most closly match the functions that the model is to be doing in the future. As for our application SupaClaw, this would mean building a synthetic dataset with complex scenarios, where an agent has to perform tasks while getting new information in real time (new messages from contacts, new quetsions from the user, feedback from tools etc).
In anyway, it was a real pleasure to compete in this hackathon with our great team and build such a cool project. I also want to express my sincere gratitude to the team of Iterate for organizing the event, Mistral team for the technical support and open models (and API credits!) and the Nvidia and Weights & Biases team for the provided hardware.
Sources: https://arxiv.org/abs/2506.07982 - Tau2-bench https://huggingface.co/datasets/Salesforce/APIGen-MT-5k https://arxiv.org/pdf/2402.03300 - origin of GRPO by DeepSeek https://hal.cs.princeton.edu/taubench_airline - DeepSeek-R1 performance on Tau-bench Airline https://huggingface.co/supateam/ministral-3-3B-RL-300_steps/settings - Final model