Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

43 Open 223 Closed
43 Open 223 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

ValueError: Image features and image tokens do not match
#710 opened Jan 4, 2025 by MoyusiteruIori
MMT-Bench Assertion Error
#709 opened Jan 4, 2025 by MoyusiteruIori
请问 CORE_MM benchmark 是否已经不支持了呢？
#707 opened Jan 4, 2025 by lihytotoro
本地评估模型相关疑问 awaiting confirm
#693 opened Dec 24, 2024 by Leke-G
@FangXinyu-0913
2
ChartQA/DocVQA 分数异常
#683 opened Dec 23, 2024 by OrlandoBloom16
@PhoenixZ810
1
MME-realworld benchmark如何设置local path？ awaiting confirm
#676 opened Dec 19, 2024 by ZihaoZheng98
2
AUTO_SPLIT=1 如何搭配CUDA_VISIBLE_DEVICES=使用
#674 opened Dec 19, 2024 by ZihaoZheng98
@FangXinyu-0913
2
The image quality in VLMEvalKit is compressed and lower than that provided by Benchmark itself.
#665 opened Dec 13, 2024 by Sync-yxh
1
Local judge LLM?
#664 opened Dec 12, 2024 by lyzhongcrd
3
报错
#661 opened Dec 12, 2024 by 0704highlight
@PhoenixZ810
1
AutoScoringJudge文件
#651 opened Dec 5, 2024 by wksuixin
@PhoenixZ810
1
LLAVA-one-vision MMMU 性能评测差别很大
#645 opened Dec 4, 2024 by linxid
2
用Chartqa测评LLama3.2-11b-vision时，结果全部为0
#644 opened Dec 4, 2024 by GoogleAlphaZero
@PhoenixZ810
1
Does VLMEvalKit support Paged Attention evaluation?
#641 opened Dec 2, 2024 by weiaicunzai
@PhoenixZ810
1
Discrepancy in MM Math Evaluation Results: Possible Issue with Answer Extraction in VLMEvalkit
#638 opened Dec 1, 2024 by mantle2048
6
The evaluation device_map setting is not optimized for quantized models using the ExLlamaV2 backend.
#629 opened Nov 26, 2024 by WeiweiZhang1
Ovis1.5-Llama3-8B在Hallusion Bench上的指标和榜单上的指标差距过大
#595 opened Nov 13, 2024 by LIRENDA621
1
ChartQA eval bug
#584 opened Nov 7, 2024 by lemonliu1992
Corrupted MathVista testmini data WIP
#578 opened Nov 5, 2024 by LeoDu0314
1
AI2d gpt和claude3.5官方分数非常高 awaiting confirm
#577 opened Nov 5, 2024 by Violettttee
3
LLaVA v1.5 7b very low performance on TextVQA
#554 opened Oct 28, 2024 by wufeim
Different evaluate result between different VLMEvalKit version
#523 opened Oct 16, 2024 by terry-for-github
@kennymckormick
5
Are images compressed in tsv files?
#517 opened Oct 12, 2024 by ywwynm
3
textonly benchmarks
#497 opened Sep 30, 2024 by JcWang20
@PhoenixZ810
1
[Help Wanted] the alignment with official accuracy in llama3.2-vision Feature Request Extra attention is needed
#493 opened Sep 29, 2024 by droidXrobot
8
ProTip! Mix and match filters to narrow down what you’re looking for.