To launch the benchmarking

[Gpoxolcku]

Hi! Awesome work and datasets collection! Is there a way (or plan to release such a script) to launch a model's benchmark evaluation on the full set of data and obtain a comprehensive report on all the metrics?

[danielz02]

Thanks a lot for your interest! We are working on it :) The end goal would be to support common Hugging Face models. Do you have any model in mind?

[Gpoxolcku]

Thank you for quick answer! Do you know any approximate release time?) Just developing yet another model, interested in metrics to track the progress :)

[danielz02]

I'm thinking of some time around ICLR, which is early May, but I can definitely adjust the priority if there is a need for evaluating new models. What interface does your model use? Is it a Hugging Face pipeline or a Llava-like interface?

[Gpoxolcku]

That would be very nice of you, thank you! I use Llava-like interface on a local machine

[Gpoxolcku]

Hi, is there any success in finalizing the eval scripts? such benchmark would be very helpful for my projects, thanks :)

[alievrusik]

Hello! I would also greatly appreciate such script, right now it's not very convenient to compare different EO VLMs on this benchmark. Do you still plan to release it?