Cross-View Completion Models are
Zero-shot Correspondence Estimators

Honggyu An^1* · Jinhyeon Kim^2* · Seonghoon Park³ · Jaewoo Jung¹
Jisang Han¹ . Sunghwan Hong² . Seungryong Kim^1†

¹KAIST    ²Korea University    ³Samgsung Electronics
*: Equal Contribution      †: Corresponding Author

Prerequisites

• Environment setup

  conda create -n CVC python=3.10.15
  conda activate CVC
  pip install -r requirements.txt

• Dataset preparation

  bash download_ETH3D.sh
  bash download_hpatches.sh

• CVC model checkpoint preparation

  • CroCo
  • DUSt3R
  • MASt3R

• Create admin/local by running the following command and update the paths to the dataset

python -c "from admin.environment import create_default_local_file; create_default_local_file()"

Zero-shot Evaluation

The scripts folder contains multiple bash files for evaluating models on either the HPatches or ETH3D datasets. Most experiments were conducted on HPatches. For each model, you can perform zero-shot evaluation of geometric matching performance using one of three methods:

Available Methods

1. Encoder Correlation: Uses encoder features to build a correlation
2. Decoder Correlation: Uses decoder features to build a correlation
3. Cross-Attention Maps: Uses cross-attention maps for correlation

For detailed explanations of each method, please refer to our paper.

Example Commands

# HPatches (Original Resolution) - CroCov2
bash scripts/run_hp_crocov2_Largebase.sh

# HPatches (240 Resolution) - CroCov2
bash scripts/run_hp240_crocov2_LargeBase.sh

# ETH3D - CroCov2 
bash scripts/run_eth3d_crocov2_LargeBase.sh

Script Configuration Details

Each evaluation script contains several key parameters that can be customized:

# Example evaluation script
CUDA=3  # Specify GPU device rank
CUDA_VISIBLE_DEVICES=${CUDA} python -u eval_matching.py \
    --seed 2024                     # Random Seed Selection
    --dataset hp                    # Dataset (hp: HPatches, hp-240: HPatches (240x240), eth3d: ETH3D)
    --model_img_size 224 224        # CVC model's Input image dimensions
    --model crocov2                 # Model type [crocov1, crocov2, dust3r, mast3r]
    --pre_trained_models croco      # Pre-trained model type
    --croco_ckpt /path/to/croco/ckpts/CroCo_V2_ViTLarge_BaseDecoder.pth
    --output_mode ca_map            # Correlation method choose from [enc_feat, dec_feat, ca_map]
    --output_ca_map                 # Enable cross-attention map output
    --reciprocity                   # Enable reciprocal cross-attention map
    --save_dir /path/to/save/images/for/visualisation/  

Acknowledgements

This code is heavily based on DenseMatching, We highly appreciate the authors for their great work.

Citation

If you found this code useful, please consider citing our paper.

@article{an2024cross,
  title={Cross-View Completion Models are Zero-shot Correspondence Estimators},
  author={An, Honggyu and Kim, Jinhyeon and Park, Seonghoon and Jung, Jaewoo and Han, Jisang and Hong, Sunghwan and Kim, Seungryong},
  journal={arXiv preprint arXiv:2412.09072},
  year={2024}
}