per checkpoint evaluation metrics for OLMoE

[Muennighoff]

Question from @SherinBojappa

In Table 13 of the OLMoE paper (https://arxiv.org/pdf/2409.02060), you report DCLM eval metrics for the checkpoints at 1.2M steps, 1.22M steps, and the annealed model. I was wondering whether you might be willing to share (or point me to) the corresponding evaluation metrics for earlier intermediate checkpoints of OLMoE-1B-7B.

We did not run the DCLM evals for other checkpoints I think but you should be able to run them easily by following these instructions: https://github.com/allenai/OLMoE?tab=readme-ov-file#after-pretraining & using the many pretraining ckpts at https://huggingface.co/allenai/OLMoE-1B-7B-0924/tree/main

[SherinBojappa]

Thank you!

[arumshisky]

@Muennighoff Quick question - the OLMoE paper mentions (Table 11 in the appendix) that 11 benchmarks were used to track downstream performance during pre-training, although Figures 3 and 25 only show the performance plots for 6 benchmarks. Would you be able to share the plots for the remaining 7 benchmarks?

[Muennighoff]

sure you should be able to find all on wandb e.g. https://wandb.ai/ai2-llm/olmoe/reports/OLMoE-1B-7B-0924--Vmlldzo4OTcyMjU3 & then clicking on the individual runs etc