[BUG] Cannot run math_500/mmlu_pro on models without chat template

[Giuseppe5]

Describe the bug

When running math_500/mmlu_pro (and potentially other tasks) on a model without a chat template, I get the following error:

TypeError: can only concatenate list (not "tuple") to list

at the following line.

To Reproduce

lighteval accelerate "model_name=HuggingFaceTB/SmolLM2-135M" "mmlu_pro|0"

Expected behavior

Eval to run

Version info

lighteval==0.13
transformers==4.57.3

[NathanHB]

hey ! thanks for the report, i think you just need to convert the stop_tokens to a list at the line you mentioned

[Giuseppe5]

Happy to open a quick PR if you want, or let you guys handle it.

The only question would be, how do I know, assuming no errors are raised, if the accuracy numbers I get are correct?
Shall I just find a small model with either one of these benchmark results published and compare?

[NathanHB]

yes that's usually how we verify ! We also look at the details of the eval to make sure inpu, output and metrics are correct :)

Happy to open a quick PR if you want, or let you guys handle it.

We would ber happy to review if you oppened a PR