Where to download RefVIE-Bench and OpenVE-Bench evaluation datasets?

[Timchonggao]

The README mentions evaluation on RefVIE-Bench and OpenVE-Bench benchmarks:

python infer.py \
  --ckpt_path path_to_ckpt \
  --bench openve \  # or refvie
  --max_frame 81 \
  --max_pixels 921600 \
  --save_dir ./infer_results/exp_name/

And notes that: Default benchmark paths in inference scripts assume datasets are under ./benchmark/....
However, I cannot find the download links for these benchmark datasets anywhere in the README, DATASET.md, or the HuggingFace model/dataset pages.

Could you please provide:

1. Download links for RefVIE-Bench and OpenVE-Bench evaluation datasets
2. Expected directory structure under ./benchmark/ after downloading
3. Data format of the benchmark (e.g., video paths, prompts, reference images, etc.)

What I've Checked

1. README.md: Only mentions the inference command and eval scripts, no benchmark download link
2. DATASET.md: Only covers training data (Ditto-1M, OpenVE-3M, ReCo, etc.), not evaluation benchmarks
3. HuggingFace (linyq collections):
   Only found model weights and training metadata, no benchmark datasets
4. Paper (arxiv.org/abs/2603.02175):
   Describes the benchmarks but doesn't provide explicit download URLs

[linyq17]

Hi! Thanks for the detailed report.

Here are the download links for the evaluation benchmark datasets:

• OpenVE-Bench: https://huggingface.co/datasets/Lewandofski/OpenVE-Bench
• RefVIE-Bench: https://huggingface.co/datasets/linyq/RefVIE-Bench

After downloading, the expected directory structure under ./benchmark/ is:

benchmark/
├── OpenVE-Bench/
│   ├── videos/
│   ├── benchmark_videos.csv
│   └── README.md
└── RefVIE-Bench/
    ├── ref_images/
    ├── source_videos/
    └── refvie_bench.yaml

These links and the directory structure have now been added to the README and DATASET.md on GitHub. Sorry for the confusion!

[Timchonggao]

Thanks for providing the benchmark download links! 🎉

However, I noticed that the infer.py script mentioned in the README for benchmark evaluation:

python infer.py \
  --ckpt_path path_to_ckpt \
  --bench openve \
  --max_frame 81 \
  --max_pixels 921600 \
  --save_dir ./infer_results/exp_name/

But infer.py does not exist in the repository. After searching through the codebase, I found that test_benchmark.py provides the same functionality. So the correct command should be like this?

python test_benchmark.py \
  --ckpt_path path_to_ckpt \
  --bench openve \
  --max_frame 81 \
  --max_pixels 921600 \
  --save_dir ./infer_results/exp_name/

[linyq17]

You are correct — infer.py was an older script that has since been replaced by test_benchmark.py. The command you provided is accurate:

python test_benchmark.py \
    --ckpt_path path_to_ckpt \
    --bench openve \
    --max_frame 81 \
    --max_pixels 921600 \
    --save_dir ./infer_results/exp_name/

We'll update the README to reflect this. Thanks for pointing it out!

[Timchonggao]

I ran the RefVIE-Bench evaluation using test_benchmark.py (since infer.py is not included in the repo) with the following command:

python test_benchmark.py \                                 
  --ckpt_path ./models/Wan-AI/linyq/wan2.2_ti2v_5b_qwen25vl_3b_stage2_img_vid_720x1280_81f/model.safetensors \
  --bench refvie \
  --dataset_file ./demo_data/benchmark/RefVIE-Bench/refvie_bench.yaml \
  --data_root ./demo_data/benchmark/RefVIE-Bench/ \
  --max_frame 81 \
  --max_pixels 921600 \
  --save_dir ./infer_results/stage2_refvie/

However, the results are noticeably worse than expected. I have a few questions:

1. Is this the correct checkpoint for RefVIE-Bench?
   I used wan2.2_ti2v_5b_qwen25vl_3b_stage2_img_vid_720x1280_81f, which is a Stage 2 checkpoint.
2. Any differences in inference hyperparameters?
   I noticed that in test_benchmark.py, the 5B model branch uses default values for several parameters (e.g., mllm_max_frame=16, mllm_max_pixels_per_frame=512*512, num_video_queries=512, num_ref_queries=768). Are these the correct settings for reproducing the paper results, or should different values be used?

Environment
Python 3.10, PyTorch 2.7, CUDA 12.8
transformers==4.57.0
Model: Wan2.2-TI2V-5B + Qwen2.5-VL-3B-Instruct
I did not modify any inference code

Thanks for your help! 🙏

https://github.com/user-attachments/assets/6b5669f2-3443-426b-9aa5-cc8bbffec1a9
https://github.com/user-attachments/assets/10675bd4-7095-41a7-a654-eb891b6f5083
https://github.com/user-attachments/assets/ab2bc907-a293-4c8e-805a-ad97220ab28e

I also ran the evaluation on OpenVE-Bench with the same checkpoint and settings:

python test_benchmark.py \
  --ckpt_path ./models/Wan-AI/linyq/wan2.2_ti2v_5b_qwen25vl_3b_stage2_img_vid_720x1280_81f/model.safetensors \
  --bench openve \
  --dataset_file ./demo_data/benchmark/OpenVE-Bench/benchmark_videos.csv \
  --data_root ./demo_data/benchmark/ \
  --max_frame 81 \
  --max_pixels 921600 \
  --save_dir ./infer_results/stage2/

The OpenVE-Bench results look reasonable and much better than the RefVIE-Bench results above.
Here are some OpenVE-Bench samples:

https://github.com/user-attachments/assets/443e54ad-be3a-4ca4-b47b-7ab1e8324c76
https://github.com/user-attachments/assets/abf18fd8-cec5-443a-915f-74fb9cbd9bf6
https://github.com/user-attachments/assets/2d3e9b40-189a-43b6-b337-7bbb81391264

[linyq17]

Thanks for the detailed report!

For RefVIE-Bench evaluation, please use the Stage 3 checkpoint instead of Stage 2. The Stage 2 checkpoint was not trained on reference data, so it won't perform well on reference-based tasks like RefVIE — this explains the noticeably worse results you observed.

Regarding your second question on inference hyperparameters, the default values in test_benchmark.py should be correct for reproducing the paper results.

Please let me know if you encounter any further issues. Thanks!

[Timchonggao]

Thanks for the clarification! That makes sense — I'll switch to the Stage 3 checkpoint for RefVIE-Bench evaluation.

 python test_benchmark.py \
  --ckpt_path ./models/Wan-AI/linyq/wan2.2_ti2v_5b_qwen25vl_3b_stage3_img_vid_refvid_720x1280_81f/model.safetensors \                 
  --bench refvie \                                                     
  --dataset_file ./demo_data/benchmark/RefVIE-Bench/refvie_bench.yaml \
  --data_root ./demo_data/benchmark/RefVIE-Bench/ \
  --max_frame 81 \     
  --max_pixels 921600 \                    
  --save_dir ./infer_results/stage3_refvie/

[Timchonggao]

Update: I've now tested with the Stage 3 checkpoint (wan2.2_ti2v_5b_qwen25vl_3b_stage3_img_vid_refvid_720x1280_81f) on RefVIE-Bench:

 python test_benchmark.py \
  --ckpt_path ./models/Wan-AI/linyq/wan2.2_ti2v_5b_qwen25vl_3b_stage3_img_vid_refvid_720x1280_81f/model.safetensors \                 
  --bench refvie \                                                     
  --dataset_file ./demo_data/benchmark/RefVIE-Bench/refvie_bench.yaml \
  --data_root ./demo_data/benchmark/RefVIE-Bench/ \
  --max_frame 81 \     
  --max_pixels 921600 \                    
  --save_dir ./infer_results/stage3_refvie/

The results are still noticeably worse than expected, any guidance would be greatly appreciated! 🙏

concat_0_Replace.th_walking.1.mp4

[linyq17]

Thanks for the update and for testing with the Stage 3 checkpoint!

The case you are evaluating on is actually one of the hardest samples in our benchmark, so lower performance on it alone may not be representative of the overall results. We'd recommend testing on more samples across the benchmark to get a better picture of the model's performance. Thanks!

[linyq17]

Additionally, for best reference editing performance, you can also try this model: https://huggingface.co/linyq/wan2.2_ti2v_5b_qwen25vl_3b_stage3_refvid_only_720x1280_81f_pad_first

This model is only trained with reference data at stage 3.

[Timchonggao]

Thanks for the guidance!
I'll expand my testing to a broader set of samples from the benchmark. I'll also try the refvid_only checkpoint you suggested and compare the results.