When will you open-source the evaluation code?

[LIRENDA621]

We attempted to reproduce the paper's results on the video-mme dataset but were unable to achieve comparable performance. As shown in the figure, there is a significant gap in accuracy. Providing evaluation code for the dataset would greatly benefit the open-source community.

[Zplusdragon]

Thank you for your interest in our work. May I ask which models you are using? In our default setup, we employ Qwen2.5-VL-7B and DeepSeek-R1-250120. Additionally, when evaluating on VideoMME, the temporal scopes of each memory need to be specifically adjusted. Furthermore, the demo we released is designed for open-ended Q&A, and since most benchmarks are based on multiple-choice questions, our evaluation approach also differs slightly. We will organize and release the complete evaluation code after the NeurIPS conference concludes.

[zrrr333]

We follow your hyperparameter settings in the appendix (like T_c, T_f for the Video-MMe benchmark). We also used Qwen2.5VL-7B and Deepseek-R1 as the VLM and LLM models, respectively. The only difference might be the evaluation script. Could you please tell the differences between the evaluation and the demo?

[jihwankwak]

@Zplusdragon Thanks for the hard work, I am wondering whether the evaluation code targeted for multiple choice QA is now available. Also, is the spliced video for EgoMem uploaded (as described in the Readme) or can you specify the tools that you used for splicing (would concat with ffmpeg be fine)?

[Yiyuan0902]

same question here

[lvgd]

Same issue. Thank you for the promising work. Just wondering when the eval scripts of the mentioned benchmarks will be released.