What is num_fewshot in eval configuration icl_tasks?

[kmcnayr]

Hi all, naive question: in the eval data configuration the icl_task entries use a variable num_fewshot which is used in the base_eval script. What does this parameter do? The icl_task entries use either 0 or 10.

I'm trying to use a custom eval dataset by adding an icl_task entry to config.yaml in eval_bundle. I'm just wondering what parameters I should use. The custom eval data is similar to world_knowledge/bigbench_qa_wikidata.jsonl which is context -> continuation and uses num_fewshot=[10] so I assume I would use the same.

Thanks!

[svlandeg]

Hi! num_fewshot refers to how many examples are given up-front to the LLM. In core_eval you can see that task_meta['num_fewshot'] number of such examples are sampled from the dataset (ensuring, of course, that the current example we're testing is not being sampled).

For evaluation purposes, it matters, because a zero-shot setting will typically produce worse results than let's say a 5-shot setting. If you're defining your own evaluation/dataset, you might want to try both to see the difference in performance.